Enterprises seeking to speed up their delivery of software and IT services to customers and internal users are turning to DevOps methods in droves. Unfortunately, a few recent studies show that for many of these nimbler DevOps teams, maintaining software quality at speed remains a work in progress.
Scratch beneath most typical digital transformation stories at organizations, and you’re likely to hear CIOs reference their transition to DevOps practices as key to helping them quickly jump on market trends and improve customer satisfaction. This closer collaboration of development and operations teams, with the help of ample automation and cloud tooling, has helped organizations speed up the rate at which organizations update their technology capabilities.
In theory, DevOps practices are supposed to bake quality assurance (QA) testing into the entire software development lifecycle. Best-in-class organizations tend to embed QA engineers in their DevOps teams to design automated testing that developers can run themselves throughout the lifecycle. But in most real-world situations organizations struggle to come up with an effective testing model that works for their DevOps teams.
“While the majority of organizations have shifted to DevOps-driven, Agile development methodologies, and the majority of development teams are keenly aware of the important role testing plays in those processes, they are still behind in actually implementing effective continuous testing throughout the software development lifecycle,” explained the Continuous Testing Benchmark Report, released today by Sauce Labs.
only about 6% of organizations achieve a benchmark for excellence in continuous testing
That study shows that only about 6% of organizations achieve a benchmark for excellence in continuous testing across the four major categories of test quality, test run time, test platform coverage, and test concurrency. For example, when it comes to test quality, fewer than one in five organizations pass 90% of the tests they run.
Still, the increase of automated testing that modern DevOps teams have brought to the table is improving the predictability of builds. Another report released by Mabl last week found that DevOps teams do twice as much automated testing as traditional software development teams. The study found that teams that automate little to none of their tests are 13% more likely to find bugs after a release than those that automate most or all of their tests, indicating “that teams who automate more have more predictable builds,” the report said.
However, that study also showed that teams that automate most or all of their tools were also more dissatisfied with their testing process than those with little to no automation.
“A big contributor to that could be the maintenance burden of automated functional and end-to-end tests, the most popular types of tests to be automated as we saw earlier,” the report showed.
Nevertheless, experts in software quality say that increased automation is crucial to achieve improved end-user satisfaction, which was named by organizations as the number one business objective of QA testing by respondents in the most recent World Quality Report by Capgemini. That study showed that test automation still remains at somewhere between 14% to 18% at most enterprises, with authors stating that the “low level of automation is the number-one bottleneck for maturing testing in enterprises.”
