My team had a software release the other day. One of the items we were releasing was an Azure Function. The old version of the function was a .NET Core 3.1 solution running in Azure Function Runtime 3.x. We updated the solution to .NET 6 and Azure Function Runtime 4 as a maintenance task along with a feature change.

We had trouble with getting this function updated previously due to changes with .NET and the Azure Function Runtime that we didn't fully understand. After our most recent fixes, it had been running in QA for over a week with no additional issues. When we deployed it to production, however, the Azure resource seemed to having a problem, like the environment was constrained somehow. The Azure portal even had difficulty loading information and seemed to be lagging.

Most mature dev shops have a plan for this kind of thing. Code is released to production and then the changes are verified. If it isn't working, gather information (like error logs, screenshots, etc.), and then initiate a rollback, restoring the previous code to the production environment. There are variations, of course, depending on what capabilities are available.

In this case, my team had already rolled this code back once and then invested a substantial amount of time with the upgrade changes. During the deployment, my team wanted to merge the feature changes into the older .NET Core 3 version of the function and release that because they knew that would work. So, we did. I knew how much effort they had put in and I trusted my team.

We deviated from the normal process, made a code change, pushed it through the pipeline to production, and delivered the functionality our customers expected. I keep track of how clean our deployments are. I marked this deployment as "unclean" and reported the issues on the change tickets in ServiceNow. I also published written accolades for two of my developers who got the job done on an internal board. The release wasn't clean, but I didn't hold that against the team. There was an opportunity for a bit of heroics and I let them take it. In this case, I felt like the opportunity was more valuable than strictly following the process.

I hope you get a chance to feel heroic today. Cheers!