Summary

Release Management is essential for all organizations, as it is the bridge between the development or delivery of software and services and live operation. Every organization uses IT services, with regular updates to those services. These updates can be minor, such as applying patches, or major, such as adding new services. Every update has an associated risk. Release management helps to manage those risks, balancing the risks against the time and cost of addressing them. Hence, release management should be used whether you develop your own software, buy it or use software as a service, customizing the approaches for each situation. Best-practice release management relies on a good understanding of the discipline, knowledge of the services and the underpinning technologies; an awareness of the capabilities of the supporting tools; and intelligence.

What is a release?

Your company has an IT helpdesk to provide support services to assist your employees in addressing questions, issues and provisioning needs related to the technology devices and services that support day-to-day business operations. In modern business environments, technology plays an important role in enabling productivity, managing company and customer data, and facilitating business processes. When the technology breaks, the customer needs access to something new, or if something changes in the technology environment, it can have a big impact on your employees’ ability to do their jobs effectively. The goal of your IT helpdesk is to provide information, resources and knowledgeable staff to help your employees resolve their technical issues and get back to their normal work activities quickly.

What is release management?

Release management is the discipline responsible for planning and controlling the build, test, deployment, and acceptance of releases. The objective of release management is to ensure the integrity of the live environment is protected and the intended components are included in the release. In some respects, release management can be considered a combination of techniques from two other disciplines, project management, and risk management. Project management techniques are used for planning and controlling while risk management is used to reduce the risks to the live environment and to customize appropriately the release-management approach to suit different scenarios.

What are the relationships between release management and other ITSM processes?

Release management techniques can be applied when deploying changes to all of the environments typically used in the path to live. These include environments and servers used for development, system testing, pre-production quality assurance, and staging and live-production environments. Although different people and teams may be responsible for different environments, it is a good idea to have an overarching release management process as this can help with end-to-end planning. ITSM personnel should own this end-to-end process, with oversight and coordination of local processes, such as those development and testing teams use. Release management should be used in all sizes of organization, including those with separate teams for design, development, testing, deployment and ITSM and those with just one team, such as DevOps, and any other combination. The scope of release management includes changes to the code, purchased applications, servers, networks, other infrastructure, processes, documentation, and organizational structures.

Change management

Release management is the implementer of changes. The change management process is responsible for approving the deployment of a change into an environment, including approval of the scheduling. Release management must consider changes well before they are approved, so it can control and schedule the changes to honor the desired deployment date. Release management must also plan for deployment and acceptance of an approved change. Change management is responsible for change approval and is just one small part of the lifecycle of a release.

Configuration and asset management

Configuration and asset management provides detailed information to release management about the individual items in a release or the release could affect and, hence, must be considered during planning. Release management relies on the integrity of the data and information, which configuration and asset management provide, to reduce the risks of deployment. These risks include baselines that provide the capability to return safely to the last-known working configuration if major issues are encountered after the deployment of a release.

Incident management

Release management uses incident management following the deployment of a release to ensure any issues are recorded. Release management can then use this information to decide if the release must be rolled back. Incident management can also provide release management with information on any current tickets that could affect a decision to proceed with a deployment. For instance, the start of a release deployment could be delayed if there is an active major incident with one of the release components.

What is the relationship between release management and software development?

For releases that include changes to applications, software development is responsible for delivering working code and any supporting documentation. The detail of how the software is developed is external to release management, but the outputs must align with the requirements of release management. Software development, therefore, must be given clear definitions of what is required from it, and when, to ensure seamless integration between the processes. To support effective planning, there must be regular communications between the two disciplines. Best practice is that release management and software development act as part of the same team, as both have the goal of delivering working software to the customer. Release-management-process activities may need to be customized to fit best with the development processes. For example, if development uses a continuous-delivery approach, then this can overlap with testing and packaging activities that are part of release management.

Why should you implement release management?

Release management is an important component of service transition. Applying best practices for managing releases can improve success and significantly reduce the risks of delivering changes and new services. Without effective release management, even supposed small changes can result in significant adverse impacts to the business of an organization, including unexpected system outages and bugs that disrupt users. These can result in large costs of recovery, loss of business credibility and even lost business. For example, a provider of online holiday and hotel booking services had an immature release management process. It deployed what was considered to be a minor change to all of its servers without testing it and without having a robust fallback plan. The minor change resulted in a total loss of all online services for 3 days. The provider estimated it had lost a million dollars in revenue as customers went to a competitor.

What activities are included in best-practice release management?

There are a number of different activities associated with release management best-practices. These are described below, but it is important to understand many of these activities must be customized to suit the circumstances of a particular organization and the type of release. All releases should be managed in some manner, but the depth of application of each activity should depend on the risk of the release to the business. What is appropriate for managing a major release where there are significant changes to multiple applications and supporting infrastructure will be too onerous for minor updates, such as updating a single Website page. Best-practice release management achieves a balance between managing risks and the costs and time of that management, by customizing and applying these phases and activities:

Create a release policy

A release policy for release management should be created, used and maintained. The policy defines a set of rules for defining, planning, building and deploying releases. It should contain a definition of the different types of releases; how the release-management-process activities and responsibilities vary for each type; standards for various items, such as documentation; and how releases are categorized for urgency and impact. Examples of release types are software patches, new functionality releases, maintenance releases, major releases, infrastructure upgrades, Software as a Service (SaaS) releases and emergency releases. The release policy should also include naming conventions for releases, and any release-identification scheme, such as release numbers and versions.

Plan the release

Every release should have a release plan covering its entire lifecycle, from design to successful implementation. The plan’s complexity and level of detail will depend on the characteristics of each release, but each of the following activities in this section should be considered and included in some manner. The plan should include the expected outputs from each phase, milestone dates for the completion of each phase and responsibility for delivery of the outputs. It’s good practice to create template plans for each type of release in your organization. Even though responsibilities for how some tasks are done are outside the scope of release management, for instance, software development, the release plan should include stages and milestone dates for all necessary activities. These include agreeing requirements and designing the solution and developing it. The release plan must be current and used to track progress.

Communicate the release

There should be an available release plan that includes all planned releases and maintenance activities. A good plan will use visual techniques to indicate different types of releases and have the capability for different stakeholders to see different layers. For example, users may only be interested in releases that directly affect them. The plan should also include any significant business events that releases could impact; many organizations impose a “freeze” leading to the busy Christmas period when only emergency releases can be deployed. The release plan, which release management maintains, can identify potential clashes between releases and these critical events and also between different releases from different parts of IT. Release management should hold regular planning meetings with all parties involved where these clashes can be discussed.

Design and agree on acceptance criteria

There should be acceptance criteria for each phase in the release plan. These should be designed and agreed before the release-management activities commence. This avoids the need for any rework or disagreements which can arise if the release-acceptance criteria are left until later in the release lifecycle. The criteria should be designed to match the desired outcomes of release management: to protect the integrity of the live environment and ensure the intended components are included in the release. Acceptance criteria must be Specific, Measurable, Achievable, Relevant and Time-bound (SMART). A common reason for failures in release management is asking people to assess if a release is OK to be deployed without having any objective measures on which to base the decision.

Build components

The development or procurement of individual components is outside the scope of release management; however, it must be kept aware of progress against the plan, including any potential changes to the deliverables.

Package release

Once the individual components have been delivered, they must be assembled into a release package. The package should include all that is needed for a successful deployment, and should include all documentation associated with the release. This is called a release note, which release management creates. It contains the following information: any associated design documentation; training notes; test results; outstanding defects; known errors; new and changed functionality; any resolved problems; and operating instructions for IT, users and the service desk.

Create deployment plan

IT and release management should agree on the method for deploying the release, and according to the release policy. A plan should be created for managing the release deployment, including timings, responsibilities, activities, and actions to be taken if issues are experienced requiring a roll-back to the pre-deployment situation.

Conduct Quality Assurance (QA

During this phase, the quality of the release is assessed. This includes checking that the requirements have been fulfilled, any standards have been followed and regression testing has been completed to verify the release will not have any adverse impact on the current live service. Good-practice release management includes non-functional testing, which assesses any impacts on resilience, performance, and capacity. Some organizations use independent test teams, operating under the control of release management. If possible, then the deployment plan, including a roll-back, should also be tested. Final QA tests should be performed in a live-like, pre-production environment. Release management should update the release note with the results of this QA phase.

Approve release deployment

The related change management process is used to approve the deployment of a release into an environment. The release-management function should provide the relevant information to assist approval, including the release note, deployment plan and an assessment of the risks to live service.

Communicate and train

All parties the release affects should be made aware of it, including IT, ITSM, suppliers, and users. Communications should include any impacts to them during and following release deployment. Where training is required to use any new or changed functionality, release management should ensure it is completed before deployment.

Deploy the release

In this phase, the approved release package is deployed into the target environment for use, following the steps agreed in the deployment plan. It is good release-management practice to provide enhanced support during and immediately following deployment, so any issues can be quickly identified and resolved, and in the worst case, by executing a managed roll-back of the release.

Service acceptance and release closure

Before the plan for managing the release can be closed, an assessment of the success of the release should be made using the previously-agreed acceptance criteria. For large releases and where supplier payment depends on success, it may be some time after deployment until some criteria can be satisfied. Where this is the case, release management uses a two-stage acceptance process: the release deployment is signed off if no roll-back was required, but the full release is not signed off until all criteria have been satisfied. These criteria should include verification that all service levels are being satisfied, all high-impact defects have been resolved and customers are satisfied with the release.

How should an organization implement release management?

A good approach for implementing release management is to introduce it one release at a time. Design your release-management process first. It is a good idea to base this on something already in use, either published best-practice for release management or the existing process your development or project management teams use to manage releases. Make sure your new release-management process includes all of the steps described here. Also, create your draft-release policy. Then, talk with your development team to identify the next release work in its schedule. Work with it and project management to use your new release-management process, noting what works well and what needs improvement. Do not be tempted to change your process too soon. After this first release has been deployed, review the lessons learned with the development and project management teams, and make any necessary amendments to the release-management process and the release policy. Use the new versions to implement another release, ideally a different type as described in your release policy, once again learning as you proceed through the process. Continue this until you have tested your release-management process with a wide range of different types of releases. Once you and the other teams are satisfied with the process and the policy, then formally launch release management for use with all new releases. 

What should I consider in a release-management tool?

A good tool for release management should have the following features:"

How can an organization apply release management if it uses Software as a Service (SaaS) applications?

If an organization chooses to use the cloud to provide SaaS services, then it will have to use release management in a different manner for these services. The differences should be described in the release policy by defining a new type of release for SaaS services. With SaaS, the supplier will perform all of the release-management activities itself. The supplier will build and then deploy the release itself. The release-management process that belongs to the customer organization must reflect this; however, the supplier must still perform some activities to manage SaaS releases. Users and the service desk may require training on the contents of the release, which is the responsibility of release management. Some SaaS suppliers make the new release available for customers to try before making it live. In this case, the customer organization’s release-management process still must include stages for testing, acceptance and also deployment, even though this may be as simple as setting a flag for using the new SaaS release.

How can you improve the velocity of your releases?

It is important that release management doesn’t slow the flow of releases into live use. Increasing the velocity of releases using effective release management can deploy changes into production sooner, use fewer resources and increase the number of deployments. If a focus is not maintained on efficiency, then the delivery of much-needed functionality and fixes can be delayed, and the resources required for release management may increase with demand. Business today requires rapid response from IT to remain competitive in the digital age. Techniques borrowed from Lean thinking should be applied to release management to satisfy these demands.

Value-stream mapping

Value-stream mapping uses visual flowcharting to depict information flows and helps to identify activities that add no value, which are called “waste.” The goal is to remove all waste to improve the velocity of the flow. For release management, all steps should be fully mapped and then analyzed to verify that every step contributes value to the delivery of a working release. Value can include ensuring the release does not disrupt normal operations, verifying legislative compliance requirements have been fulfilled and the risks of deployment are known and minimized.

Remove inefficient hand-offs

Release-management processes nearly always require activities from multiple teams. This means information and, sometimes, control must be passed between teams. These transfers are known as hand-offs. Every hand-off adds time delays, increases the risk of information corruption and transposition, adds non-value-added steps to check the information received and the hand-off itself adds no value. Analysis of the value stream of release management should question every hand-off if it is required. One strategy to reduce them is to merge teams. For example, in DevOps, one team is responsible for executing most, if not all of the release-management activities, including specifying requirements, building the solution, packaging the release and andoffh

Shared knowledge and information

Best-practice release management encourages the sharing of knowledge and information between all parties involved in the release, including customers and users. This aids rapid and effective transfer to support the release-management process. The tools and techniques of knowledge management are used to support this sharing. Information may need to be customized to be understandable and relevant to particular groups of consumers, as they will have different requirements. For example, technical teams will be primarily interested in technical information, users will be interested in how to use the new functionality, and the business managers in any expected disruption to the business during deployment that release management cannot eliminate.

Accessible release plans

Improving the visibility and readability of release plans, which release management creates, can provide information on project timelines and progress against key milestones. This can help to prevent surprises and keep all those involved working to the same plan. Having effective project-management skills within release management is essential to maintaining progress on the individual steps.

Automation

Automation of repetitive, low-risk and regularly executed activities can greatly improve the velocity of release management. Tools can be used to automate activities, such as release packaging, maintaining repositories of release components and release packs, sharing information and knowledge between teams and automated deployment of releases between environments.

Runbooks

Where automation is not possible, repetitive tasks associated with release management, especially with code deployment, should be documented in a runbook. Staff responsible for executing the deployment activities can then use the runbook instead of relying on tribal knowledge. This can help to improve the velocity of release management by removing thinking time.

How can you plan for major releases in advance?

Major releases are those that contain a large number of changed components, a large number of changes to a few components, a change to a critical component, or a combination of these. Your release policy should closely define what is a major release to avoid ambiguity. By their very nature, major releases can present a high risk to the organization. To provide sufficient time to eliminate or at least reduce the risks to an acceptable level, the planning for major releases should start well before the intended deployment date. Major releases are also more likely to require an outage for deploying the release. By starting to plan early, release management will have more time to negotiate the necessary outage with the business. Businesses that rely on IT systems 24x7 can require at least 6 months to prepare. Despite early planning by release management, it may still be necessary to delay deployment if the risks remain too large to be acceptable, often because insufficient time was allowed for testing and fixing identified defects. These techniques can be useful when planning major releases:

Communicate early, and continue to communicate

Release management should communicate with the business, IT, affected departments and any suppliers as soon as the need for a major release is identified. The initial communication doesn’t need any detail, just information about what services are likely to be affected and when the release is likely to be deployed. This doesn’t have to be a specific date, or time, just enough information so discussions can start with the business and IT about when would be most suitable for all involved. Communications should then continue at regularly planned intervals, adding more and more detail until the complete information is available.

Release plans

Many release plans only show releases change management has approved. These often omit major changes that IT knows are likely to happen, but release management hasn’t yet sought change approval as the details are not yet known. A release should appear on a release plan as soon as it’s suspected it will happen, to start communication and discussions. An example is major releases of commercial database solutions, where it is probable there will be an annual major release. Instead of waiting until it is announced, release management should add an entry in the release plan, showing the following year’s major release and when the new one is expected to be deployed. A useful release-management technique is to classify entries in the release plan as “fixed,” “firm” and “tentative.” A tentative entry is one where the date is possible. A firm entry is one where the date is highly likely, but release management has not yet obtained deployment approval from change management. A fixed date is one where all approvals have been given.

Re-use and change

Apart from the very first time an organization has a major release, there will always be some release management artifacts from previous releases that can be used as the basis for the next major release. After every major release, release management should hold a “lessons-learned” review. This should identify what went well, which should be kept for the next release, and what requires improvement, which should trigger changes. It is also possible to learn from other parts of the same organization and different organizations. One good place is the manufacturing industry, where some businesses make major changes regularly.

How can you prevent and quickly recover from failed releases?

Even with release management best-practices, there can be failed releases. Failure can include unexpected outages during and after deployment; release deployments that exceeded the time release management planned; increased incidents after a release; functionality that worked in a test, but doesn’t work in live; and functionality that wasn’t meant to change, but did. How you recover from failure will depend precisely on the nature of the failure. Often, it is less disruptive to users and the business to “fail forward,” prioritizing finding and fixing the causes instead of returning the release to the pre-deployment status. There are some release management techniques that can help you avoid a roll-back, or can help you successfully achieve one if it is the best option.

Controlled source-code repository/snapshot copies

To be able to recover successfully with a roll-back, you must be able to replace the newly deployed code with the last known working version. This can be achieved by restoring from backup, but this can be a time-consuming and, sometimes, unsuccessful approach. There are two techniques that can help release management recover from failures, both based on technologies. Tools exist that provide a controlled repository for source code, which is baselined and versioned every time code is changed or added. Recovery from failure is achieved by deploying the last known working version from this repository. There are also tools which operate in a similar manner at the storage level, where you can take a snapshot of everything, including the applications and data before starting the deployment, and then very quickly restore the snapshot if the release fails.

Recovery rehearsals

Wherever possible, testing and rehearsing recovery should be executed under release management control. Confidence in both execution and timing increases as more and more successful rehearsals occur. This can make taking a decision to roll-back much easier.

Deployment plans

A best-practice deployment plan will include all steps that must be executed and written as operating instructions that can be easily followed. Many release failures are caused by omitting important steps, due to relying on personal knowledge. The plan should also include a timing plan for managing the release. This plan must include a key milestone time at which a decision should be made to either continue with or roll-back a release if issues are encountered during deployment. This milestone must allow enough time to recover the service within the time period agreed with users.

Fire breaks

Many deployments don’t run exactly as planned. It is a good idea to include “fire breaks” in a deployment plan. These are time slots where nothing is happening, providing contingency time that can be used without further approval from release management. It is important, though, to have a formal go/no-go gateway at the end of each fire break. If tasks overrun beyond the agreed contingency time in the fire break, then release management must provide approval before the deployment can continue.

Failover

Where a disaster recovery (DR) environment exists, then recovery can be achieved by executing the disaster-recovery plan to switch services to this environment. How much time is required depends on how DR has been designed. High-availability services can switch within seconds while other services can take days. Release management must be aware of any DR arrangements, and should ensure that DR testing occurs before a major release. One common approach for major releases is to do final release testing in the DR environment, then switch to it, so it is now live. The new code is then deployed to the previously live environment, now DR, to keep it in step. If this approach is taken, then the responsibility should be with release management, and not with service continuity.

Canaries

Where it is technically feasible, a good approach is first to deploy new code to a small subset of the infrastructure or users. This reduces the impact of any failures, and if a rollback is necessary, then the user impact during the roll-back is also reduced. This technique is called “canaries,” after the use of caged birds in underground mines to determine the presence of noxious gasses before they would affect miners.

Smoke tests

Immediately after a deployment, tests can be run to verify that all is working as planned. These tests are often conducted before users are allowed to use the system again, or after a release is deployed during times of low usage, such as weekends. The tests allow issues to be identified and addressed before users start reporting them.

Release management and continuous integration 

Continuous integration (CI) is an approach that has emerged from Agile and DevOps. Historic techniques for release management often meant that it took considerable time to build, package, test and deploy every release. This led to long release cycles, typically with a major release every 12 months with quarterly maintenance releases between each major release, delaying delivery of new functionality and non-urgent fixes to users. CI aims to address this through automating with tools the release management activities of packaging, testing, and deployment, coupled with robust controls for adding and changing source code in the release package. Using CI can result in a fully-tested release and ready for deployment every time individual code changes have been completed. The same team still executes every stage of release management, and with as little human intervention as possible. This approach can significantly increase service quality and confidence in the services and in release management.

Other IT Resources