APPENDIX B

Want to know what we’ve found from a statistical standpoint? Here is one place that lists it all, organized by category.

As a reminder:

Correlation looks at how closely two variables move together (or don’t) but it doesn’t tell us if one variable’s movement predicts or causes the movement in another variable. Two variables moving together can always be due to a third variable or, sometimes, just chance.

Prediction talks about the impact of one construct on another. Specifically, we used inferential prediction, one of the most common types of analysis conducted in business and technology research today. It helps us understand the impact of HR policies, organizational behavior, and motivation, and helps us measure how technology affects such outcomes as user satisfaction, team efficiency, and organizational performance. Inferential design is used when purely experimental design is not possible and field experiments are preferred—for example, in business, where data collection happens in complex organizations, not in sterile lab environments, and companies won’t sacrifice profits to fit into control groups defined by the research team. Analysis methods used to test prediction include simple linear regression and partial least squares regression, described in Appendix C.

ORGANIZATIONAL PERFORMANCE

• High performers are twice as likely to exceed organizational performance goals as low performers: profitability, productivity, market share, number of customers.
• High performers are twice as likely to exceed noncommercial performance goals as low performers: quantity of products/ services, operating efficiency, customer satisfaction, quality of products/services, achieving organizational/mission goals.
• In a follow-up survey to the initial 2014 data collection effort, we gathered stock ticker data and performed additional analysis on responses from just over 1,000 respondents across 355 companies who volunteered the organization they worked for. For those who worked for publicly traded companies, we found the following (this analysis was not replicated in later years because our dataset was not large enough):

• High performers had 50% higher market capitalization growth over three years compared to low performers.

SOFTWARE DELIVERY PERFORMANCE

• The four measures of software delivery performance (deploy frequency, lead time, mean time to restore, change fail percentage) are good classifiers for the software delivery performance profile. The groups we identified—high, medium, and low performers—are all significantly different across all four measures each year.
• Our analysis of high, medium, and low performers provides evidence that there are no trade-offs between improving performance and achieving higher levels of tempo and stability: they move in tandem.
• Software delivery performance predicts organizational performance and noncommercial performance.
• The software delivery performance construct is a combination of three metrics: lead time, release frequency, and MTTR. Change fail rate is not included in the construct, though it is highly correlated with the construct.
• Deploy frequency is highly correlated with continuous delivery and the comprehensive use of version control.
• Lead time is highly correlated with version control and automated testing.
• MTTR is highly correlated with version control and monitoring.
• Software delivery performance is correlated with organizational investment in DevOps.
• Software delivery performance is negatively correlated with deployment pain. The more painful code deployments are, the poorer the software delivery performance and culture.

QUALITY

• Unplanned work and rework:

• High performers reported spending 49% of their time on new work and 21% on unplanned work or rework.
• Low performers spend 38% of their time on new work and 27% on unplanned work or rework.
• There is evidence of the J-curve in our rework data. Medium performers spend more time on unplanned rework than low performers, with 32% of their time spent on unplanned work or rework.

• Manual work:

• High performers report the lowest amount of manual work across all practices (configuration management, testing, deployments, change approval process) at statistically significant levels.
• There is evidence of the J-curve again. Medium performers do more manual work than low performers when it comes to deployment and change approval processes, and these differences are statistically significant.
• See Table B.1 for manual work percentages in high, medium, and low performers.

Table B.1 Manual Work Percentages

* Differences are not statistically significant between medium and low performers for configuration management and testing.

BURNOUT AND DEPLOYMENT PAIN

• Deployment pain is negatively correlated with software delivery performance and Westrum organizational culture.
• The five factors most highly correlated with burnout are Westrum organizational culture (negative), leaders (negative), organizational investment (negative), organizational performance (negative), and deployment pain (positive).

TECHNICAL CAPABILITIES

(Architecture capabilities are in their own section, below.)

• Trunk-based development:

• High performers have the shortest integration times and branch lifetimes, with branch life and integration typically lasting hours or a day.
• Low performers have the longest integration times and branch lifetimes, with branch life and integration typically lasting days or weeks.

• Technical practices predict continuous delivery, Westrum organizational culture, identity, job satisfaction, software delivery performance, less burnout, less deployment pain, and less time spent on rework.
• High performers spend 50% less time remediating security issues than low performers.

ARCHITECTURE CAPABILITIES

• There was no correlation between a particular type of system (e.g., system of engagement or system of record) and software delivery performance.
• Low performers were more likely to say that the software they were building—or the set of services they had to interact with—was “custom software developed by another company (e.g., an outsourcing partner).”
• Low performers were more likely to be working on mainframe systems.
• Having to integrate against mainframe systems was not a statistically significant indicator of performance.
• Medium and high performers have no significant correlation between system type and software delivery performance.
• A loosely coupled, well-encapsulated architecture drives IT performance. In the 2017 dataset, this was the biggest contributor to continuous delivery.
• Among those who deploy at least once per day, as the number of developers on the team increases we found:

• Low performers deploy with decreasing frequency.
• Medium performers deploy at a constant frequency.
• High performers deploy at a significantly increasing frequency.

• High-performing teams were more likely to respond positively to the following statements:

• We can do most of our testing without requiring an integrated environment.
• We can and do deploy/release our applications independently of other applications/services they depend on.
• It is custom software that uses a microservices architecture.

• We found no significant differences according to which type of architecture teams were building or integrating against.

LEAN MANAGEMENT CAPABILITIES

• Lean management capabilities predict Westrum organizational culture, job satisfaction, software delivery performance, and less burnout.
• Change approvals:

• Change advisory boards are negatively correlated with software delivery performance.
• Approval only for high-risk changes was not correlated with software delivery performance.
• Teams that reported no approval process or used peer review achieved higher software delivery performance.
• A lightweight change approval process predicts software delivery performance.

LEAN PRODUCT MANAGEMENT CAPABILITIES

• The ability to take an experimental approach to product development is highly correlated with the technical practices that contribute to continuous delivery.
• Lean product development capabilities predict Westrum organizational culture, software delivery performance, organizational performance, and less burnout.

ORGANIZATIONAL CULTURE CAPABILITIES

• These measures are strongly correlated to culture:

• Organizational investment in DevOps
• The experience and effectiveness of team leaders
• Continuous delivery capabilities
• The ability of development, operations, and infosec teams to achieve win-win outcomes
• Organizational performance
• Deployment pain
• Lean management practices

• Westrum organizational culture predicts software delivery performance, organizational performance, and job satisfaction.
• Westrum organizational culture is negatively correlated with deployment pain. The more painful code deployments are, the poorer the culture.

IDENTITY, EMPLOYEE NET PROMOTER SCORE (ENPS), AND JOB SATISFACTION

• Identity predicts organizational performance.
• High performers have better employee loyalty, as measured by employee Net Promoter Score (eNPS). Employees in high-performing organizations were 2.2 times more likely to recommend their organization as a great place to work.
• eNPS was significantly correlated with:

• The extent to which the organization collects customer feedback and uses it to inform the design of products and features
• The ability of teams to visualize and understand the flow of products or features through development all the way to the customer
• The extent to which employees identify with their organization’s values and goals, and the effort they are willing to put in to make the organization successful

• Employees in high-performing teams are 2.2 times more likely to recommend their organization as a great place to work.
• Employees in high-performing teams are 1.8 times more likely to recommend their team as a great place to work.
• Job satisfaction predicts organizational performance.

LEADERSHIP

• We observed significant differences in leadership characteristics among high-, medium-, and low-performing teams.

• High-performing teams reported having leaders with the strongest behaviors across all dimensions: vision, inspirational communication, intellectual stimulation, supportive leadership, and personal recognition.
• Low-performing teams reported the lowest levels of all five leadership characteristics.
• These differences were all at statistically significant levels.

• Characteristics of transformational leadership are highly correlated with software delivery performance.
• Transformational leadership is highly correlated with employee Net Promoter Score (eNPS).
• Teams with the top 10% of reported transformational leadership characteristics were equally or even less likely to be high performers, compared to the entire population of teams represented in survey results.
• Leadership is predictive of Lean product development capabilities (working in small batches, team experimentation, gathering and implementing customer feedback) and technical practices (test automation, deployment automation, trunk-based development, shift left on security, loosely coupled architecture, empowered teams, continuous integration).

DIVERSITY

• Of the total respondents, 5% self-identified as women in 2015, 6% in 2016, and 6.5% in 2017.
• 33% of our respondents report working on teams with no women.
• 56% of our respondents report working on teams that are less than 10% female.
• 81% of our respondents report working on teams that are less than 25% female.
• Gender

• 91% Male
• 6% Female
• 3% Non-binary or other

• Underrepresented

• 77% responded no, I do not identify as underrepresented.
• 12% responded yes, I identify as underrepresented.
• 11% responded that they preferred not to respond or NA.

OTHER

• Investment in DevOps was highly correlated to software delivery performance.
• Percentage of people reporting working in DevOps teams has grown over the last four years:

• 16% in 2014
• 19% in 2015
• 22% in 2016
• 27% in 2017

• DevOps is happening across all operating systems. We first started asking about this in 2015.

• 78% of respondents are widely deployed on 1-4 different operating systems, the most popular being: Enterprise Linux, Windows 2012, Windows 2008, Debian/Ubuntu Linux.

• Figure B.1 shows the Firmographics from the 2017 data. We note that high, medium, and low performers see representation from all groups. That is, there are large enterprises in the high-, medium-, and low-performing groups. We also see startups in high-, medium-, and low-performing groups. Highly regulated industries (including financial, healthcare, telecommunications, etc.) are also found in the high-, medium-, and low-performing groups. What matters is not what industry you’re in or how big you are; even large, highly regulated organizations are able to develop and deliver software with high performance, and then leverage these capabilities to deliver value to their customers and their organization.

Figure B.1: Firmographics: Organization Size, Industry, Number of Servers in 2017