Thank you to everyone who participated in our round table discussion on Cracking Your Test Automation Code: The Path to CI/CD. We had a great turnout with lots of solid questions from the audience. If you missed the live event, don’t worry… Just click here to watch the recorded session. There were several questions that we were not able to answer during the live event so I followed up with the panelist afterwards to get their answers.
Q: How do you implement CI/CD without using any tools?
Bas Dijkstra: Without any tools at all it will not be possible to do CI/CD. Maybe technically there are ways to do it, but it will be massively inefficient. You will need tools that help you do version control, building software and deploying it on a specific environment. If we just look at doing CI/CD without testing tools, that is possible in theory, but you will need to do the required amount of checking by hand anyway to make sure that the software you delivered has a certain level of quality. Doing that manually will very likely be less efficient and act as a bottleneck in the process.
How can I improve backend testing of a mobile app beyond unit testing?
Oren Rubin: You can expand beyond the unit tests by adding API testing. This is also considered integrations tests, as you testing several units as one big unit. Consider that one big state that you need to initial, and the API calls (either REST, SOAP, etc) are like function calls, some getters, some setters which modify the state.
Q: What are the things we need to cover during API tests?
Bas Dijkstra: In short: connectivity (can I talk to the API at all?), syntax (is the data returned formatted correctly, are headers and status codes OK?), semantics (is the data returned the data I expected?), functionality (does the API call have the required side effects (data stored, processes triggered, etc.), as well as no unintended side effects), performance and security (authorization, authentication).
How should one avoid overlapping in functional and unit test automation?
Oren Rubin: It’s easy to overlap the two, but it’s almost unavoidable as each unit is tested on it’s own. They are all tested via the integration tests, which usually doesn’t care about a specific unit but mostly on part of a user story/flow. Each action (e.g. click) can drive a lot of units and check them and the integrations. Remember that E2E tests come at a high price, so choose them wisely.
Is it good to separate manual test cases from automated test cases?
Bas Dijkstra: Yes. Especially since translating your ‘manual’ test cases (where you’ll likely check a number of things in the process) to automated test cases (which should ideally check one thing only) 1-to-1 will likely lead to inefficient automation. Determining what to automate and what to leave for testers is an art in itself, and simply handing someone a batch of manual test cases to automate isn’t likely to give you good results.
Oren Rubin: There is no difference. Unit tests allow you to test a single unit (e.g. a controller of a component) and mock everything else; use spies, stubs, and mocks. An integration test, which might test the entire UI, where you can check unit loading order works the CSS (which is global by definition, if you exclude future shadow dom), and integrations between the different units. If you don’t mock the servers, then it’s actually End-to-End testing.
About the panelists
Bas is an independent test automation professional who has been in the test automation and service virtualization field for over 10 years now, designing and developing test automation and service virtualization solutions that enhance and improve test teams and test processes. Find out more information about Bas on his LinkedIn profile. For questions and more information you can always send him an email at firstname.lastname@example.org or give me a nudge via @_basdijkstra on Twitter.
Oren has over 20 years of experience in the software industry, building mostly test-related products for developers at IBM, Wix, Cadence, Applitools, and Testim.io. In addition to being a busy entrepreneur, Oren is a community activist and and the co-organizer of the Selenium-Israel meetup and the Israeli Google Developer Group meetup. He has taught at Technion University, and mentored at the Google Launchpad Accelerator.