Backup Exec journey from year+ release cycle to frequent releases

The world is changing quite rapidly, software companies are becoming agile and moving to frequent release model to cater to their customer needs. Backup Exec supports different applications and operating systems that have started releasing regular updates. Technology has changed at a rapid pace, thereby affecting customers. With all the changes, the Backup Exec team anticipated that frequent product releases would help serve the changing customer needs. The team decided to adopt Agile due to the benefits it offered. This article focuses on the Agile method's key aspects that helped the Backup Exec team in releasing more frequent and regular product updates.

During pre-Agile days, the engineering organization consisted of component teams-any feature development would happen through a virtual team composed of members of different component teams. Each virtual team member would work on their part or the interfacing component, instead of the feature. After each component team finished, the virtual team then performed integration testing.

The virtual team would develop these features on separate branches and then merge them into the main branch or the trunk. Once all the feature development concluded, the system team started the hardening cycle, which would eventually lead to beta and further milestones. 

The development cycle consisted of requirement analysis, design, development, and test phase. Escalations were worked on by a dedicated team, which was known as the customer focus team. 

When Veritas implemented Agile, the Backup Exec product was more than 30 years old. Therefore we adopted what we call "Vintage Agile Delivery Framework," which provides flexibility to accommodate the needs without compromising Agile principles. 

The release starts with Inception Planning (IP), where the product team, including developers, QA, PM, and other support functions, meet to plan upcoming releases. Typically, Product Manager (PM), Architect, and Support exhibit content for two upcoming releases. The release content is classified as the base and extended. Base content refers to the release items that are a stop-ship, without which a release cannot happen. The view of two releases during an IP helps the team to identify long poles for the second release. It helps to start work on long pole items early. 

The main goals of the IP is to ensure that: 

  • Base content is signed up by scrum teams, and we identify long poles, risks, and external dependencies. 
  • Estimated time for the releases is made available.

After the IP, scrum teams start working on feature development. If there is an area that is unexplored or needs more information, the teams work on discovery. Developers contribute in their area of expertise or even newer fields depending upon their skills, interest, or need. Scrum teams are responsible for working on escalations as well, so they sign up for escalations regularly during feature development. 

The scrum team meets for daily standup, which helps to discover impediments earlier, enabling the team to solve them quickly. The sprint duration is two weeks, and at the end of the sprint, there is a demo for stakeholders. The demo provides an opportunity for stakeholders, including customers, to review the sprint work done by the scrum team and provide feedback. Once the scrum team completes all the features signed up in a release, the Product Test Team (PTT) starts with the hardening cycle. The hardening cycle mainly covers regression scenarios. The hardening cycle eventually leads you to the beta milestone and subsequent milestones in a release. 

One of the significant changes was the team structure, with the engineering organization now consisting of scrum teams composed of different component experts. It enabled the scrum team to take complete feature development responsibility. Other stakeholders who closely work with the team are support, escalation program manager, and info developer. This team composition has made an enormous difference. It has helped to eliminate waste, get early inputs, and focus on the release goal. Unlike component teams, members work in any area, thereby ensuring that the workload is balanced, and they are working on priority items. It also helps to reduce friction typically seen in the component team model as they work towards a common goal as a scrum team. Release CycleRelease Cycle

A scrum team is responsible for the complete development of the feature, including testing, automation, and unit testing. After the scrum team completes all the feature development, the main thing that remains is hardening, which is executed by the Product Test Team (PTT). The team has automated many regression test cases, which has helped to reduce the hardening cycle. Automation has also contributed to making last moment changes without having a major effect on the release schedules. It is one of the reasons the agility has increased. 

From a prioritization perspective, there is clarity on the priority of items in the backlog. It helps in ensuring that the team stays focused on higher priority items. We have also adopted the philosophy of developing features on the main branch by design. All feature development work takes place on the trunk or main branch and remains pristine at any given time. To achieve a pristine branch, we thoroughly test any code that goes into the branch. The single branch eliminates all the efforts that are related to merging and keeps the team going. When all the feature development work is complete, then the release branch is created. One thing to note, the work happens in parallel for multiple releases. While development for one release is in progress, the other release is in hardening or later phases, as indicated in the Release Cycle diagram. 

BranchingBranchingWhile Agile has offered many benefits, reducing the "release cycle" has been one of the key ones. It has helped the team to provide frequent updates, which added regular incremental value. The team has delivered 17 releases in five and a half years. That is, on average, one release in less than four monthsI don't know if one could have imagined in the past, releasing Backup Exec so frequently and regularly. But with proper training, open mindset, persistent efforts, and, most importantly, support from leadership, the team has achieved it.