How to reduce the cost of software quality
Practical steps to shift left Last month we published “Do you know how much your quality costs?” where we discussed the importance of shifting left and shortening the feedback loop as it relates to the cost of quality. According to Bank of America, discovering a bug in development would cost about 5 hours to fix […]
Practical steps to shift left
Last month we published “Do you know how much your quality costs?” where we discussed the importance of shifting left and shortening the feedback loop as it relates to the cost of quality. According to Bank of America, discovering a bug in development would cost about 5 hours to fix and $390. Discovering a bug in QA cost about x30. 127 hours to fix, due to to the time it takes to reproduce, the back and forth between dev and QA and the need to trace back the code. Fixing a bug in production – Estimated at $72,000. Shifting left allows testing close to development as possible and a fast and continuous feedback loop.
Four key ingredients to achieving shift left:
- Test automation – Traditional QA processes would result in months of testing before you get feedback. A productive tester can get about 20 tests done in a single day. Assuming your test suite includes dozens of tests there are two ways to get the feedback loop down to minutes: Hire hundreds of testers or automate your tests. Automated tests don’t get tired, take days off, go out for lunch etc.
- High concurrency – Shifting left means giving near real time feedback i.e minutes. Running dozens of tests, each taking roughly 30-60 seconds, with the expectations of getting the results back in, say 5 min., can only be done if you are running your tests in parallel. The higher your level of parallelism the shorter the feedback loop is. Running your tests in parallel required a grid, a cloud based infrastructure that includes virtual environments allowing you to spin in and out different browsers.
- Build-test-deploy automation – Shifting left means you are optimizing the handover across the different stages of the development process. The development cycle includes dozens of handovers and human intervention can always delay that process hence you want to automate the handover using Continuous Integration (CI) servers. CI servers monitor events through your development cycle, such as a new build, and can be programed to trigger a set of actions once a certain event occurred, such as triggering your automated tests.
- Suites – It is best practice to optimize your quality assurance processes by segmenting your tests into suites and running different suites at different stages of your development cycle. For example you full regression might include thousands of tests. Running all those tests every time a develop push code would be time and resource consuming. If 2-3 key scenarios failed, there is no point in waiting for the entire suite to be over. You might want to create a small subset of tests, composed of your high priority flows, and run that “sanity” suite often, while running the more extensive suites once if sanity suite ran successfully.
A few weeks ago, we talked about automating tests. Now it’s time to take a look at six of the best Python test automation tools. The good news is that the Python standard library already includes excellent unit testing tools. You can go a long way toward setting up robust test automation using the language’s […]
Presenting a perfect product to the customer is the end goal of every organization. But did you know that there was a time when testing wasn’t even a part of the software development life cycle (SDLC)? Nothing puts off customers more than bug-filled user experience. So when enterprises realized this, they began to include testing […]
We’ve already undergone quite a journey to evolve from “classical testing” and test automation to continuous testing. Nevertheless, when we look into the future, it’s clear that even continuous testing won’t be sufficient. Certainly, digital transformation happens in test automation, as it does in every field. We’ve come from manual testing to a robust automated […]