How do we manage Web Application deployments with thousands of users on production

Thumb Rule: Simple processes that encourage developers to follow them yet matured enough to ensure production is stable. It is OK to take time and think twice than bringing production down. This is what we do here at Systango.

Bare Minimum:

1. Continuous integration server integrated with source control repository in our case GIT. For CI servers we use Jenkins, Teamcity, Bamboo internally.
2. Checkout Test Suite: Minimum but sufficient tests that runs every morning (or mid night) to ensure production is sane and healthy.
3. Auto Deploys: Deployment scripts that auto deploys on QA and Production. No manual management of configs. My favorite tool is Capistrano.
4. Process Health Monitor, Error, Performance and load monitoring tools: Have process monitors installed, which could if needed restart server, escalate if there are any issues to ensure 99.99% uptime. Airbrake, New Relic, JMeter, GOD are the tools we use for error, server monitoring and load testing respectively. GOD is to monitor services/processes and restart, scale if needed.
5. Source Code Repository: Ensure to have source code repository and commit/branch strategy in place irrespective of how small/big project is.
6. Database backup and migrations: Always have back up of the database before altering its schema and always alter them with migrations.
7. Server environment: Update server environment frequently to ensure latest and stable patches, libraries and packages are installed.

 

Different server environments one should have:

Development: Dev server is where every commit gets deployed and is replica of what is on master.

Stage/QA: For us staging is the test bed. This is where all the integration and testing happens. This server environment allows QAs to test the future releases and report/debug in case of issues. This server needs to be replica of production environment to ensure testing and decisions made are also valid in production environment.

Production: This is the live environment for end users to access the product/service. Infrastructure and resources are tweaked to maximum required capability for proper scaling and high performance for users. This is where checkout test suite should run every day.

Deployment Cycles

Dev: Deployment with every commit. Reflection of what developer is working on.

Staging: Once or may be twice a week for feature updates. For bug fixes it’s pretty much on the daily basis.

Production: Once or may be twice a week for feature updates. For bug fixes it’s pretty much on the daily basis.

Avoid deploying on production on Friday irrespective of how much confidence you have on your code!

Code Repository

Good developer will never find managing code repository tedious. Over the time they tend to realize its benefits. In the long run it saves not only efforts but if managed properly also minimizes downtime.

Branch Strategy:

1. master: Keep the latest stable code to be released.

2. live: What is on the production at any point of time.

3. last_live: previous live to roll back if needed on production.

4. Every deployment has a version and use that as Tag to tag the commits deployed on live.

Ensure to keep your commits atomic with clean and relevant commit message. Atomic commits keep development organized and make bug tracking much more efficient. Commit messages make navigating history much easier

There are several more things we could do such as using Chef etc but as an IT consultancy these choices are also dependent on the budget our clients have. Never the less we do atleast this in every project we do to ensure we don’t have to be awake all night to manage a bad deployment once every month.