Does Your Test Automation Arrive Too Late?

Traditional test automation arrives one release behind the new features – after the manual QA team has spent at least two weeks testing the new features, and long after the developers have moved on to other things.

This is too late to deliver real value; at a minimum the automation should be available as soon as the code is delivered to QA; ideally the automation should be delivered to the developers in-sprint. If it is delivered at the same time as the code, the manual QA team has a lot less work to do; if it is delivered in-sprint, the developers can run all of the tests before their changes are even merged into the Release branch.

Real World Examples

At General Motors I came in to help a test automation team that was doing things the traditional way – automating the manual test cases using Java and Selenium and delivering the automation one release behind the developers. A year later they became the first QA team in all of GM IT to deliver automated tests to the developers while they were still writing the code. For another website they went from delivering one release behind to delivering two or three days after the code was put into an integrated environment – and they went from using eight automation engineers to maintain and update the code, to one-half of an automation engineer, plus one manual QA person to run the tests.

At United Technologies I worked with a team of developers writing data science applications in Python; a year later they had transitioned from 100% manual testing to almost all automation, and were delivering the automated tests in-sprint. Because the developers were able to run fully automated regression tests plus fully automated tests for the new functionality before they opened a pull request, they quit delivering bugs. Code reviews were no longer about finding bugs; they were about ensuring that the code was clean, well designed, and easily maintained. That team went on to deliver just two bugs into Production over the next year!

How is this possible?

It requires a fundamental paradigm shift, from traditional narrative requirements to Specification by Example/BDD, and it requires a full-team effort:

  • The Product Owners and Business Analysts must learn how to write requirements that are more detailed and more explicit than traditional narrative requirements – Specification by Example says that all requirements must include concrete examples.
  • The Developers must learn how to collaborate with the Product Owners and BAs, to write the requirements together.
  • The QAs must learn how to help write the requirements, and how to automate early and reliably. This means that test automation code must be first-class code, not second-class code. At United Technologies the Developers reviewed every pull request opened by the QA team, and used the same review standards as for production application code.

I can teach your teams how to do this; contact me and let’s schedule some time to talk.

Leave a Comment