smoke testing


Thank you to everyone who participated in our round table discussion on Choosing the Right DevOps Strategy: How far left should I shift? We had a solid turnout with lots of great questions from the audience. If you missed the live event, don’t worry…

You can watch the recorded session any time:

Tanya Kravtsov, Director of QA at Audible, and Bob Crews, President of Checkpoint Technologies shared their lessons learned from four different types of enterprise DevOps transformations.

They covered:

  • Practical suggestions for assessing your operational reality
  • Major pitfalls in DevOps transformations and how to avoid them
  • How to organize your team to increase productivity
  • Impact on development processes, ownership, tool stack and KPIs

Some of the audience questions they answered:

  • How do you ensure that security requirements are always considered in DevOps?
  • Which KPIs are important to track to ensure successful DevOps transformation?
  • What is an acceptable amount of technical debt? How do we know when it becomes a problem?
  • What are some metrics for measuring risk?

There were several questions that we were not able to address during the live event so I followed up with the panelist afterwards to get their answers.

Q: Why was continuous deployment listed twice on the diagrams?

Francis: Great catch. That was actually a typo, a marketing design bug slipped through to production… The “Continuous Deployment” label shown to the upper left, within the square encompassing the “Build” and “Test” phases in both the upper (Continuous Delivery) and bottom (Continuous Deployment) cycles, should have been labeled “Continuous Integration.”

Q: In reference to CD /CI what are the main benefits of doing smoke testing after production deployment? And, what could be an example of smoke testing?  

Tanya: Smoke test validating the most critical functionality of the product should be executed after every deployment to test the staging and production environments. This will help you uncover any deployment related issues and rollback the release if necessary without impacting the customer.  

Bob: Examples of “smoke testing” would be quick test to check for and validate successful login with valid data, login error message with invalid data, checking for broken links, tests to quickly validate the number and state of objects on a webpage, etc. These are very simple, but important tests. Typically they are run in production as one last check since many, perhaps most, organizations do not have a test environment that is an EXACT duplicate of production.

Q: Are automated tests the entire QA solution across all these organizations? Do the panelists see any value in manual, exploratory QA testers?

Tanya: Automated tests are an important component of the Continuous Delivery solution. It enables exploratory QA testers to focus on what they do best and actually test the new features of the product  by eliminating the need for manually running the same regression tests every time new code is committed.  

Bob: Absolutely. I believe it’s critical to carve out an approach to perform some quick manual, exploratory tests. These can be done as a separate initiative, apart from the validation taking place during these phases.

Q: What approach we should follow to test DevOps solutions?

Tanya: Any scripted DevOps solutions should be treated just like any code,  accompanied by unit and integration tests which run continuously in a test environment before the changes get promoted to the mainline.  In addition,  closely tracking KPIs discussed in the Webinar will alert us to any issues with the current solutions such as build up of technical debt or reduction in quality,  which should be acted upon immediately.   

Bob: I would need a little more information in order to provide a good answer. Knowing the current state of the organization’s testing approach, their objectives, the tools already in place, the skill set of current team members, etc. are just a few questions I would have before being able to provide any insight. If starting from scratch I would say start with becoming entrenched in an Agile approach. If you are already Agile then the next step would be to look at tackling Continuous Integration. Please feel free to connect with me Linkedin and I would be happy to chat more about your question.

About Tanya Kravtsov
Tanya Kravtsov is the Director of QA at Audible. She is building a new QA organization to support innovative web product development at scale. Previously, as head of automation and continuous delivery at ROKITT, senior QA manager at Syncsort, and VP at Morgan Stanley, Tanya focused on quality, automation, and DevOps practices, working with internal and external customers to transform development and testing processes. Tanya is passionate about process automation, continuous integration, and continuous delivery. She is a founder of the DevOpsQA NJ Meetup group and a frequent speaker at STAREAST, QUEST, and other conferences and events. Follow her on Twitter @DevOpsQA.

About Bob Crews
Bob Crews is the President of Checkpoint Technologies. He is a consultant and trainer with almost three decades of I.T. experience including full life-cycle development involving development, requirements management and software testing. He is also the President of the Tampa Bay Quality Assurance Association. He has consulted and trained for over 200 different organizations in areas such as effectively using automated testing solutions, test planning, implementing automated frameworks and developing practices which ensure the maximum return-on-investment with automated solutions.

Be smart & save time...
Simply automate.

Get Started Free