3 Must-Avoid Deployment Anti-Patterns!
And how you should get rid of them?
Software release days are usually nail-bitingly tense affairs. Every team member is on the edge of their seat, hoping that the new release will go off without a hitch. But, all too often, things don't go according to plan.
A failed release impacts everyone in the software development hierarchy
Managers are worried about potentially negative customer experiences
Project Managers fear the onset of post-release incidents
Architects wonder whether the design decisions will survive contact with actual users
Developers are skeptical whether the application will work as it did during testing
Ops engineers are suspicious about the completeness of the deployment process documented by the dev team
But why should this be the case?
Why, despite so many people suffering, software releases are prone to problems?
The reason is that most software shops are stuck following one of the three anti-patterns of deploying software.
These anti-patterns not only make software releases a bigger deal than they should be, but they also make the teams less agile in fixing issues that might arise after the release.
Let’s look at these anti-patterns and how you can avoid them.
Antipattern#1 - Manual Deployment
I feel manual deployment is one of the biggest sins in modern software deployment. And that doesn’t mean I haven’t done it.
Everyone is guilty of manually deploying some artifact at one time or the other.
It might have worked in earlier days when applications were not complicated.
However, modern applications are complex pieces of software. These days, even a simple web application such as a blog might have multiple components and many moving parts.
And yet, many organizations are still stuck releasing software manually.
Imagine the plight of an ops team following a bunch of manual steps written in a document while deploying a distributed application comprised of multiple services, integration systems, database and caching solutions and front-end artifacts.
Each step needs to be handled separately. Maybe, different steps need to be performed by a different individual or even separate teams based on access rules and so on. Things can become difficult if the order of the steps or the timeline is altered. Manual decisions must be made at each step making them prone to human error.
There are some tell-tale signs that your team is caught in the rut of manual deployments.
You have an extensive document that describes the steps to be taken to deploy the software to production
Every time there’s a release, you rely on some experts to manually test the application
You are getting frequent calls from the ops team on release day to figure out how certain things work
After every release, you are forced to update the extensive document because you find something that was missing in the document
Despite all the care, the release has an unpredictable outcome and you are forced to rollback
More often than not, you end up sitting in front of a monitor at 3 AM the day after release, trying to figure out how to make things work
Antipattern#2 - No Production Deployments before the Big Day
Don’t get me wrong on this one. Of course, you would let the users use your application only after you are done developing it. But that does not mean you can’t deploy your application to a production-like environment during the course of development.
What are the signs of this particular anti-pattern? Here are a few:
Testers haven’t got a chance to test your application. Even if they did perform some tests, it was done on development machines
The Ops team doesn’t have any exposure to the application before the release day. Only on the release date, the development team gathers the application, configuration files, database migrations and documentation, and hands them off to the operations team.
Even the developers have no idea how the application is going to behave in an actual production-like environment
Following this antipattern leads to a plethora of problems.
For example, the application might be relying on the file system of the dev machine to do something but the same approach might not work in a clustered production environment.
The longer the release cycle, the longer the development team has to make incorrect assumptions and the longer it will take to fix them.
How do you get rid of this anti-pattern?
You need to integrate the testing, deployment and release activities into a seamless development process. Make it normal to deploy your code into a production-like environment as early as possible so that it becomes ingrained in your team’s work culture.
After all, practice makes one perfect. Handling deployments is no different.
Antipattern#3 - Manual Configuration Management
In my personal view, this is the most dangerous antipattern. Unfortunately, it is also very common.
Big and small organizations manage the configuration of their production environments through a team of ops people.
Changes such as database connection settings or increasing the number of threads on a server are done manually on the production server.
More often than not, the changes and how to make them are not well documented. Knowledge is restricted to a bunch of individuals who have access to the production servers. If the responsible person goes on leave or quits the organization, the entire process grinds to a halt.
Your team is in the grips of this anti-pattern when your production deployments fail and they are fixed by tweaking a particular configuration setting manually.
To avoid this antipattern, you need to ensure that all configuration aspects of testing, staging and production environments must be handled from version control systems by means of an automated process.
You should be able to recreate the production environment in an automatic manner whenever needed. Changes must be audited and reviewed using the version control tool so that it is possible to roll back to an earlier version in case of issues.
That’s it
You can’t make your software available to end-users without deployment. It’s a fundamental step in the big picture.
But you seriously hamper your ability to roll out new features if your deployment process is plagued by one of these disastrous anti-patterns.
The trouble is that these anti-patterns are quite sticky. Over time, teams tend to create habits that tend to support the anti-patterns rather than removing them. It takes a concerted effort to get rid of them.
Of course, you won’t be able to do it in one big push. You need to follow an iterative process that takes into account the capabilities of your team and the organization.
As a first step, I recommend looking into CI/CD practices along with configuration management. If things get mature, you can start exploring something like Infrastructure as Code. But more on that in future posts.
Are you suffering from one of the anti-patterns discussed in this post? Or do you have another problem that may not be similar to what we have discussed?
Don’t hesitate to write your thoughts in the comments section below.
If you found the post useful, consider sharing it with your friends or colleagues.