Our client is the market leading data authority for the hybrid cloud. They provide a full range of hybrid cloud data services that simplify application management and data across cloud and on-site environment to accelerate digital transformation. Their solution empowers global organizations to unleash the full potential of their data to expand customer engagement, foster greater innovation, and optimize their operations.
The Challenge – Achieve CI/CD
Their application was complex and included many user flows, requiring the creation of thousands of functional tests with the goal of shifting left. In addition, they needed to test as close to development as possible.
Selenium was their first choice and a team of over a dozen testers was assembled to begin the task. After a few months it became apparent that it was going to take more time and man power to achieve their goal of CI/CD. The tests were complex and took three days to author. To make things worse, the tests would often break, leading the team to spend extra time maintaining and fixing the tests. Shifting left would require teaching the developers Selenium as well.
The lack of tools to troubleshoot a failed test (e.g screenshots to compare, details error messages pointing to the right step, test history or the parameters over the flows) led to long time-to-resolution involving a number of team members. A lot of time was wasted, not only trying to figure out why the test failed, but to also explain the discoveries to the developers.
“Maintaining tests took a lot of my time. When developers run tests that fail it becomes more of a
distractionthan confidence in bug prevention.Both groups had to stop what they are doing to figure
out if its the functionality or if it’s the test. We found ourselves spending more time trying to stabilize
the tests than actually testing.” – Company QA Manager
“Today we have hundreds of tests running on every pull, giving the developers feedback to their code
within minutes. The developers themselves easily update test scenarios so QA can focus on increasing
coverage. We also signiﬁcantly reduced our cost of quality: The rich information we get allows us to
reduce the time to troubleshoot by 80%.”
As a rapidly growing enterprise, they needed a way to optimize processes through automation. This plays a signiﬁcant role in their move to agile development. Within a couple months of using Testim, The team was able to create hundreds of their UI tests scenarios. Today, they are proud to say that they are authoring tests in under an hour compared to the three days it was originally taking.
“Before Testim it would take 3 days to author a single test in Selenium, now even for the less
experiencedtester, writing tests takes under an hour, developers can update tests on the ﬂy and
ﬁgure out where the tests are failing without any additional help.”
Now, troubleshooting a failed tests takes a fraction of the time. There is an indication on which step failed, including JS syntaxes, screenshots comparing prior runs, and access to the DOM with detailed error messages.
However, the biggest impact was the reduction on maintenance. Tests are stable and trustworthy so when a test fails the user knows it is either due to a bug or the test requires a change in the flow. The team focused most of its time on increasing coverage knowing that the tests adapt to UI changes by the development team.
Today, we are proud to say that they are fully CI/CD, testing on every code push, and running thousands of tests every day.