Software versioning and release

Objective

This RFC aims to propose the process for software versioning and release. The objective is to release only stable builds to production. We can achieve this by ensuring each build is tested extensively by QA team before release. In the hopefully rare event there is an issue due to production release, we want a way to revert to the last stable build.

Motivation

In the absence of a clear process for release, untested builds get released to production. This leads to production outages. It damages the credibility of our development team and software reliability. We want to ensure high availability of our software. Over time we should be able to back our claim of high availability with SLAs. This would aid customer satisfaction and product growth.

Design Proposal

Unified GitHub Workflow for staging and production release

All releases go through a unified pipeline.

• The release is auto-deployed to staging in the absence of staging freeze.
• QA team can perform testing on the build in staging. This would initiate staging deployment freeze until testing completes.
• If it passes all the tests, this release gets QA sign-off. After sign-off, deployment freeze would be lifted.
• On QA sign-off, this release becomes eligible for production release. At this stage, a release tag would be associated to this commit/release.
• Product manager can review the QA sign-off and promote the release to production.

If the release doesn't pass QA checks on staging, the steps can be repeated with newer commit/release.

All the commit SHA released to production will have release tags

In order to have a history of stable builds readily available for backup, we will add a release tag with each build that gets QA sign-off.

Identifying software releases using Git commit SHA for logging and tracing

Since not all builds get QA sign-off, not all builds will have a release tag. Release tags are reserved for builds that get QA sign-off and potentially released to production. For logging and tracing, we will use Git commit SHA on both staging and production for consistency. Git commit SHA can be injected via the environment variable at the time of deployment.

Weekly release cadence and staging deployment freeze for QA testing - Monday

At the time of writing our automation test suite is not very exhaustive. Due to this, we need to perform manual testing during working hours. In order to facilitate this, we would freeze staging deployment on Monday. During the freeze, no PRs can be merged and main branch would be locked. QA will use this window for testing the builds and providing sign-off for eligible builds.

Drawbacks

• Due to lack of exhaustive automation testing, we are choosing slower release cadence. This is to facilitate manual testing.
• Deployment freeze may lead to slower feedback loop for developers. The alternative is to have separate AWS accounts for development and staging. Since this would significantly increase the cost, we chose deployment freeze.