New Forrester research, “The Definitive Software Quality Metrics for Agile + DevOps,” reveals what software quality metrics and practices differentiate DevOps/Agile leaders from DevOps/Agile laggards. The study was commissioned by Tricentis and led by Forrester VP and Principal Analyst Diego Lo Giudice.
The study found that companies with the most successful Agile and DevOps initiatives do a number of things differently than their peers:
- They are transforming software testing into continuous testing by adopting five core practices, such as automating end-to-end functional testing and integrating testers into cross-functional teams.
- They are almost twice as likely to consider automating the software quality process to be a “critical business differentiator.”
- They are significantly more likely to have high levels of automation for key testing and QA processes (test case design, functional test automation, test data management, etc.).
However, only 26 percent of firms with active DevOps adoptions are currently practicing the five core continuous testing practices shown to separate leaders from laggards. And, as the infographic below highlights, many CXOs have a severely inflated sense of their firms’ DevOps and continuous testing maturity.
Different DevOps Quality Metrics
The study also identified the 20 most important Agile and DevOps quality metrics that separate DevOps/Agile experts from their less advanced peers. Forrester examined 94 quality metrics that were classified into four categories. Key findings include:
- Understanding of business risk is the most important determining factor of DevOps and Agile maturity.
- Experts focus primarily on contextual metrics (e.g., requirements coverage) while others focus on “counting” metrics (e.g., number of tests).
- Experts are more likely to measure the user experience across an end-to-end transaction while others rely on application-specific or team-specific metrics.
To determine the “most important” metrics, the research surveyed 603 global enterprise participants responsible for their firms’ Agile and/or DevOps strategies. Participants were first asked a variety of questions to assess the maturity of their Agile/DevOps practices. Next, they were asked what quality metrics they actually used, then asked to rank the value of each metric they regularly measured. Results from the Agile/DevOps experts were later separated from, and compared to, those of the “other” respondents. The most important metrics were determined by analyzing the experts’ value rankings for the metrics that they actually used.
The Risk ‘Blind Spot’
Risk was a prevalent thread throughout the various parts of the report. Although risk-related metrics did not rank high in overall popularity, Agile and DevOps experts measure them significantly more frequently than the non-experts did. In fact, Agile and DevOps experts consistently ranked risk-related metrics among their top three most valuable metrics in different phases and categories.
Interestingly, most firms (80 percent) believe they deliver within acceptable business risk, but fewer than a quarter state that their QA and testing processes completely cover business risk. Only 15 percent of respondents say that their test suites reliably provide a good indication of business risk.
From Tricentis’ analysis of Global 2000 companies’ test case portfolios, we’ve found that most test suites actually cover about 40 percent of the organization’s business risks. However, these test suites have a 67 percent level of redundancy on average—meaning that over two-thirds of the tests don’t actually increase their business risk coverage. This might be one cause of the disconnect: people assume that more tests means more business risk coverage, but that’s often not the case.
Another likely cause is that most test results focus on pass/fail status, which doesn’t provide the required insight into whether the release has an acceptable level of risk. As companies move towards continuous delivery and automated release processes, real-time insight into business risk becomes critical for promoting acceptable releases as rapidly as possible while stopping potentially-damaging ones from proceeding down the release pipeline.
Details and Additional Findings
If you want to learn more about how Forrester arrived at these findings—and see how the 94 quality metrics ranked—you can read the complete “Definitive Software Quality Metrics for Agile+DevOps” on the Tricentis website.
