Lots of people have asked me how we handle rollbacks in our Continuous Delivery pipeline. The short answer is we can redeploy an older MSDeploy package to revert to an older version of our websites, services and other applications.
Inevitably, the next question is what about your databases? We can actually rollback our databases as well by rerunning our MSDeploy package because our database packages consist of a DACPAC in most cases.
A data-tier application (DAC) is a logical database management entity that defines all of the SQL Server objects – like tables, views, and instance objects, including logins – associated with a user’s database. A DAC is a self-contained unit of SQL Server database deployment that enables data-tier developers and database administrators to package SQL Server objects into a portable artifact called a DAC package, also known as a DACPAC.
It’s basically a snapshot of the schema which when deployed will change the target to match the snapshot thus rolling back to a previous state. Checkout my blog posts on SQL database deployments with MSDeploy for more information:
https://www.dotnetcatch.com/2016/02/10/deploying-a-database-project-with-msdeploy/
But there is still a risk of data loss which is why although technically we CAN, we almost never rollback a deployment. We do practice the rollbacks periodically in a non-live environment to make sure they still work but I can only think of one instance across all our products in the last 2 years where we used them to revert a deployment.
What’s the alternative?
I discussed many options for recovering from deployment failure in a previous blog post: Continuous Delivery – Resilience. We should always assume deployments will fail and have a way to disable the new code easily and quickly. Our teams typically use service versioning and feature flags as our primary mechanisms for recovery.
- Service Versioning – Instead of changing services we deploy changes as a new version. To “rollback”, clients only need to go back to referencing the old version.
- Feature Flags – For most code changes, we wrap the changes with a feature flag so we can easily enable/disable the change. We utilize ReallySimpleFeatureToggle with a custom centralized persistance service to manage our feature flags.
These options work 99% of the time for us. The third option it to roll forward with the deployment issue and quickly go back to fix it in the code and push the change through the pipeline to fix the issue up through production. This is possible because we have invested in our continuous delivery pipeline and can push changes out quickly/safely.
The last option we consider is actually rolling back the deployment.
This process has worked very well for us over the last three years. Granted it has taken time to build up a culture and capabilities to support, this investment has paid dividends many times over the initial cost. This seems like a big/scary change but I challenge you to try it. Our teams had all those same concerns but now would never go back to not having these capabilities.
What do you think? Have you tried this? Share your successes/failures/questions in the comments below.