When companies move their development over to scrum they feel they’ve found a holy grail. For a while, anyway. Then the realities of scrum and agile development creep in. The separate Operations division finds the response time they can manage and the training of developers on using repositories, CICD servers, packer, ansible, tooling, plus trying to deploy those tools falls short of what is needed. So they drop a developer or sysadmin into the developer team, call him Devops, to learn the Operations side and support his team. This pushes the cross-functional mantra from Scrum. Ops is still determining for the most part what tools are deployed and things start to work. Ops does POCs with some input from their Devops guy. Maybe.
That Devops developer is buried in small tasks related to bringing developers into the tool set deployed by Operations. The end result is a valuable resource busy bandaging things together as an operational basis. Automation is a maybe, the focus moves to making this all work and to deliver. Operations is still on the hook for the developer tools, and the Devops guy is following behind filling in the missing pieces as fast as he can.
And CICD gets almost done. There are pieces still done by checklist and by hand. There are bash scripts doing the deployment, Jenkins is configured by the Devops guy, who also spends a lot of time troubleshooting builds and adjusting configuration for developers and building small tools to smooth the way when he finds breathing room. We dropped Release Engineering and QA to the side, and that makes the business somewhat happy, but there are a lot of hands keeping this thing working.
What if…
What if instead you set Devops off as a team building tooling. Responsible to developers, but also to Operations, and separate from both. Responsive from a much higher level to the business. And you let those guys loose to build an automated build pipeline.
They support their tools. They train developers to use those tools. They POC and determine best of breed. Automation is first, consistency and repeatability and infrastructure as code is a workable mantra and you start to see true CICD, with “Continuous Integration Continuous Delivery” becoming “Continuous Integration Continuous Deployment”.
Here’s the thing about automation. Six months from now if I need to know how we built a website down to the last detail, I have that. I can kick off a build that constructs and provisions the exact server at that version. I have in a repo every detail of that knowledge of how to build that server. I have institutional knowledge, and I can be certain I have that for every single product the company produces. Let that sink in.
I know. Exactly. How that server went together.
I don’t have to discover it. I already have it, in an accessible and repeatable form.
That documentation lives with the code for the application, whether that app is a server or a program. Same.
Bash becomes a language for one-offs and POCs and code starts being written in supportable languages such as python or go.
Imagine a world where developers commit code, instantiate a pull request, which triggers a build and which is accepted AFTER the checkin is built and tested and passes in a regression branch of his team fork. That acceptance of the PR builds and tests again and deploys to DEV/QA/STAGE – which is all the same and spun up and down as demand requires in a cloud service (insert Azure, Digital Ocean, AWS, etc). Once that builds and tests integration, a PR can be instantiated to bring that code into PROD. The code builds, this time in PROD, as the PR is created. Once it successfully completes, the PR can be accepted and the code can deploy, into a new cloud stack, or expanding the servers to accommodate the new servers alongside the old, then gradually shrinking them as monitoring confirms they are in good shape.
No one works after hours on deployment. No human does anything to forward the progress of a build other than develop code, check it in and review and accept pull requests.
Rollback in cloud is automated.
But to get to that point you have to trust that your Devops guys can produce a pipeline with a standard of testing and quality that allows confidence. And you have to give him the space and time and team to pull it off. And do not drop your Devops Engineer into a lion pit of needy developers…