on
Delivery Performance
Delivery performance is defined by Forsgren et al. [1] as a combination of three metrics:
- frequency of deployment,
- time from commit to production, and
- mean time to recovery.
This definition seeks to avoid the problems of previous performance metrics, such as lines of code or velocity. These previous metrics often suffer from two drawbacks: focus on outputs rather than outcomes and focus on local results rather than in global ones. On the other hand, Forsgren et al. defined delivery performance in such way to focus on global outcome, ensuring teams aren’t pitted against each other.
Based on a survey with 27,000 responses in 2017, Forsgren et al. applied cluster analysis to these metrics and discovered three groups: high, medium, and low performers.
High performers:
- have multiple deployments per day;
- have commits taking less than 1 hour to reach production;
- and repair incidents in less than 1 hour.
Medium performers:
- deploy between once per week and once per month;
- have a time from commit to production between one week and one month;
- and take less than one day to repair incidents.
Low performers:
- deploy between once per week and once per month;
- have a time from commit to production between one week and one month;
- and take between one day and one week to repair incidents.
The research of Forsgren et al. has also discovered that the level of delivery performance correlates to the achievement of higher-level goals of the organizations, both for commercial and noncommercial purposes.
In our research, at ccsl/devops, we are not interested in distinguishing medium from lower performers.
We are interested only in identifying high performers and non-high performers.
But the above clusters are problematic for our purpose since there is a gap
between high and medium performers cluster.
We circumvent this problem by considering an organization as high performer if i) it is within the boundaries limiting the cluster of high performers above defined or ii) it violates at most one high-performance threshold by only one point in the scale adopted for the metric.
Here follows the scale for each metric.
Frequency of deployment:
- on demand (multiple deploys per day)
- between once per hour and once per day
- between once per day and once per week
- between once per week and once per month
- between once per month and once every six months
- fewer than once every six months
Time from commit to production:
- less than one hour
- less than one day
- between one day and one week
- between one week and one month
- between one month and six months
- more than six months
Mean time to recovery:
- less than one hour
- less than one day
- between one day and one week
- between one week and one month
- between one month and six months
- more than six months
[1] Nicole Forsgren, Jez Humble, and Gene Kim. 2018. Measuring Performance. In Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations. IT Revolution Press.