The internal team at Silverstripe (aka CMS Squad)responsible for the development of Silverstripe CMS is currently in the process of rethinking how we approach our release process.
Today, we’re going to share some of the changes we’re thinking about implementing and how this might impact the community. We’ll love to hear your feedback on what you’re thinking and how this might impact you.
TL;DR
We want to minimise the time we spend tagging minor releases and make it easier for the community to help us test Silverstripe CMS and plan their upgrades.
For upcoming minor releases, we’re thinking of following this process:
- 1 month of beta for testing and patching bugs
- 2 weeks of release candidate for a security audit and to plan upgrades
- then the official release.
Patch releases would continue following the current process.
What the sticking points are and what we’re hoping to achieve
Our core goals are:
- Minimise the time we spend on releases
- Have a clearer definition of what each pre-stable release is meant to accomplish
- Have a more transparent release process
Every quarter, we put out a new minor core release of Silverstripe CMS and a matching minor release of the Common Web Platform (CWP). Internally, the CMS Squad will spend several weeks of developer time reviewing and preparing each release. This is valuable time we’re not spending fixing issues, implementing new features or helping the community.
In the lead up to a stable release, we’ll some time put up an alpha release, a beta release or a release candidate. Currently, we do not have a formal definition of what each of these terms mean and the distinction between these concepts in the context of Silverstripe CMS is not very clear.
Those pre-stable releases are currently being created for mostly internal reasons which are not clearly communicated to the wider Silverstripe Community. So we might put out a beta release, but we’re not telling the community how long the beta period will be or what we expect them to do with the beta release … or even why we decided to put a beta release this time.
What drives the timing of our releases
Our quarterly releases are audited by an independent third party for security purposes. The auditors expect the release candidate to be provided to them on a specific date. This part of the release process can not be moved because it involves a third party who has to set aside time to review our release.
The other release milestones are more flexible and can be pushed around a bit. In practice our release process is driven by the delivery of this release candidate to our auditors, because that’s the one thing that is out of our control.
What the new release timeline will look like
Beta period (about 1 month before the audit)
The CMS Squad institutes a feature freeze on the next minor release. We will release a first beta release and create a new minor branch for the upcoming minor release. Internally, the CMS Squad will run a series of functional tests to identify any regressions.
Throughout the month leading up to the audit, the main focus of the CMS Squad will be to identify and fix issues in the beta. This is a part where the community can be really helpful by testing the beta and helping us identify issues.
If critical issues in the beta are holding back testers, we might put out additional betas once fixes are available.
Release Candidate period (a few days before the audit)
At this stage, we expect to have identified and fixed all critical or high priority issues in the beta release. We tag a release candidate. This is a good opportunity for early adopters in the community to try upgrading their projects and start planning their production deployment for when the stable release is available.
RC delivery to auditors
We deliver a release candidate to the auditors. A point to note is that the release candidate we deliver to the auditors might be slightly different from the publicly available one. That’s because our auditors need to review fixes to undisclosed security issues.
Auditors deliver their report (2 weeks after the audit)
We receive a report from the auditors. We’ll take a few days to go through and discuss it with stakeholders. If auditors find vulnerabilities, we will develop a fix to include in the final release.
Quarterly release (a few days after the audit report)
We do the official stable release. We may also tag patch releases for older minor branches at the same time. We’ll also announce the end-of-life date for some previous minor releases as well.
Some proposed definitions
Beta release
A beta release will contain new features. It is likely to contain regressions and new bugs.
The purpose of a beta release is to stabilise the Silverstripe CMS codebase for an upcoming stable release and showcase new features. It’s primarily aimed at testers who want to help identify bugs and regressions.
Once an initial beta is released, the matching minor release branch enter a feature freeze sate.
Release Candidate
A release candidate will not introduce any new features compared to the previous beta release. The development team is confident a release candidate is production quality and does not introduce new regressions.
The purpose of a release candidate is to allow third party and early adopters to audit the upcoming release and confidently plan their upgrade.
Once an initial release candidate is released, only critical bug fixes can be added to the upcoming release.
Feature freeze
A “Feature freeze” is an unreleased minor branch state. This state allows the development team to stabilise the upcoming release and reach production-quality.
When in “Feature freeze”, the following changes can be merged in the branch:
- Bug fixes, regardless of severity,
- API changes if they fix a bug and are compliant with our existing semver commitments,
- Merge-ups from older minor branches.
The following changes can NOT be merged in a branch in feature freeze:
- Any PR that introduce new functionality,
- API changes not targeted to fixing a bug or that alters an existing API in a stable release.
What about patch releases
This updated process won’t really affect how we do patch releases. We’ll still be doing occasional patch releases for specific modules and recipes in between quarterly releases to deliver high value bug fixes.
Another point to note, is that after delivering a release candidate to our auditors, we try our best to avoid adding any extra bug fixes into the quarterly release. There’s a class of Silverstripe CMS customers for which using officially audited releases is a very important concern. Adding extra commits on top of what’s in the release candidate undermines that.
That’s why sometimes we’ll quickly deliver a patch release right after a quarterly release. Customers who only want to run audited code can use the quarterly release. Customers who are comfortable running some unaudited code to get the latest bug fixes can run the latest patch release.
Why not do alpha releases
We considered doing alpha releases, but after some discussion we concluded that it would only be helpful if we needed to coordinate ongoing development with external stakeholders.
We concluded that the time that would be required to do regular alpha releases would be better spent on other tasks. If we ever get to the point where we can automate releases, that calculation might change.
Members of the community who have an interest in running pre-release code can still do this by targeting the development branches in their composer files (e.g. 4.x-dev).