Before I share some thoughts on DevOps, here's a quick refresher on how DevOps is defined. Wikipedia has a very telling definition:

"DevOps is a methodology in the software development and IT industry. Used as a set of practices and tools..." DevOps - Wikipedia

The first thing that struck me with this definition is the focus on the practices and tools. Culture isn't mentioned at first and I had this "Aha! Gotcha!" moment. As you get through the article, however, there is a short section that mentions culture.

I'm homing in on the culture component because, in my career experience to date, the tooling seems to always come first. I worked at an insurance company for nearly ten years. When I started there, App Services and Tech Services were two distinct departments. Thankfully, they were both under the same Vice President, but they each had their own directors and management staff. The two departments were not always on friendly terms. The various conflicts between the departments led to the codification of request procedures with required fields and agreed-upon response times.

As the industry moved forward, practices were developed and refined. Tools were created and monetized to support those practices. The insurance company I was at was old and not very large. Their project management processes were out of date. When I joined, there weren't consistent code reviews. No agile, no scrum, and no git. A modern developer would likely be quite confounded trying to work at this insurance company back in 2012.

Eventually, IT leadership started to make some moves. If we had stayed in "just keep the lights on" mode, we would have run out of road. It would have gotten more difficult to keep and retain talent. It would have been increasingly difficult to support our aging software solutions.

I mentioned that it seems like the tools come first, but it may be more accurate to say buying tools is easier than changing an organization's culture. We brought in consultants to evaluate where we were and where we wanted to be. Plans were created and refined. We invested in tools like Visual Studio, Team Foundation Server, Jira, Bitbucket, Puppet, and more. Then, we trained the individual contributors and leaders on the new way of doing things. We adopted Agile and Scrum practices. We did team-building exercises, hosted mock pointing exercises, and came up with clever & fun team names. It started to look like a DevOps shop. It was very rough around the edges, but all the pieces were there.

Changing the culture to a DevOps one was the largest roadblock facing the company. The company's first unique challenge was the unusually low turnover rate in IT. A large portion of the IT staff had worked in the department for ten, fifteen, twenty years or more. They had "old school" skills and mentalities that had served them well for a long time. Some were not eager to learn a new way of doing things. These long-tenured staff members also had long memories. For some of them, Agile and DevOps were just the latest buzzwords; just another fad from IT leadership.

I also noticed that some of the team members were responsive to being directed, but weren't comfortable with self-directing. The contractors that introduced Agile and Scrum practices tried to get team members to "think outside the box", adapt to changing situations, and stretch themselves. Some were afraid of making mistakes (due to the lack of psychological safety) while some seemed to lack the experience and skill needed to work in a more flexible and ambiguous environment. At the time, I had a real concern that DevOps would never work until the staff and leaders who couldn't make the turn quit or retired.

I left the insurance company in 2021. The IT department made great strides in improving its tech stack and agile practices. However, based on the cultural component of DevOps, I wouldn't be surprised if they were still "in transition" to this day.

I eventually moved to a staffing company that has only been around since 1997. I was very excited about my new role. The company had already migrated to using cloud technologies. They used tools like Azure DevOps, Terraform, OpsGenie, and ServiceNow. The company even had job titles like DevOps Engineer and Cloud Infrastructure Architect. The insurance company I was at previously was only poking a stick at cloud anything when I left. I was looking forward to seeing how a modern, cloud-based IT shop operates.

On my development team, everything was going as I had hoped. We used Scrum to iterate on work, we used PRs to review and approve code changes, and our deployments were automated. We had a Change Advisory Board (CAB) that handled reviewing and approving changes before deployment. I had a good working relationship with one of the DevOps managers. He and I had worked together to get my team's Azure access adjusted and we refactored some code so that manual steps by a DevOps admin were eliminated from our normal deployments.

Then, one fateful day, my team needed to deploy a brand new solution to production. On the day of the release, someone happened to notice that we were using Bicep, a language from Microsoft used to deploy Azure resources, instead of Terraform to set up the solution's infrastructure. A few hours before the release, the Director of Tech Services informed me that my release was not to go live with Bicep. Terraform was the only allowed technology and I needed to push the release to convert to Terraform. I offered to deploy with Bicep that night and switch to Terraform during the following sprint, but the director would not compromise. He indicated that in his experience development teams don't follow through with such things and his team gets stuck with the clean-up. He said "We are working toward a true DevOps culture...", but I certainly wasn't feeling it at that moment.

Establishing a DevOps culture is more of a journey than a destination. The tooling part of DevOps seems like a solved problem to me; an organization just has to decide which tools they can afford to use. The real work for establishing a DevOps culture is with the people in all departments (not just IT). Old department silos need to be torn down. The people need good processes, good leaders, and a clear vision. I wonder now if forgiveness is also key to making the transition successfully.

Cheers!