As we work with our clients, some themes are recurring. The most common theme is that development organizations are short staffed–they cannot hire enough developers. At the same time the business pressure to ship product features does not let up and pressure is high. So what does the head of engineering do? None of the options are good:
- Delay the release – Not acceptable to the business.
- Reduce the feature set and only deliver a subset of what the customers expect – Risk or failing to meet customer commitments and losing their business.
- Prioritize new feature development over quality and hope for the best.
Unfortunately #3 is what many R&D teams are forced into. Many development executives prioritize the expansion of the development team, and not the QA team. For many growing development teams, adding QA engineers is a luxury. And using sophisticated frameworks like Selenium require skilled engineers.
Yet not doing UI testing invites disaster. The cost of fixing bugs increases over time and so capturing bugs later in the release cycle becomes expensive to fix. Relying on manual UI testing, while better than nothing, provides poor coverage and scale. Many people rationalize their decision not to test the UI, hoping that developers test their own code and whatever their code affects, and maybe we can live without automated UI testing. This often works, until a bug creeps in that stops all forward movement. Suddenly you’re on a fire drill taking valuable resources from multiple departments in the organization. Afterall, developers only test a specific set of the app, on a specific environment.
Does Selenium fix this problem? Not really. The results are OK, but it requires dedicated, skilled engineers to set it up and, more importantly, to maintain. Unmaintained tests lose their value the minute the application changes. And the application is always changing.
The challenge our customers face is how to increase coverage and maintain acceptable quality levels, update their tests to fit code changes and keep up with fast release cycles. If only there was a way to automate the maintenance of tests… possibly leveraging machine learning to ensure the tests keep up with the pace of software delivery…and that the team can grow in this area without adding people.