What did the team do this week and what was the impact?
Do we have enough test coverage in the sprint?
With Testim’s new Managerial Reports, you never have to worry about getting answers to these questions again. Get deep insight into project status and quality metrics which provide granular details into execution cycles, active runs, run durations and success rates – all available online or sent weekly to your inbox.
These reports, dashboards and KPIs quickly summarize the team’s effort invested over the course of the week, identify tests that require attention as well as if additional effort is required to improve your quality score. Easily track trends week over week and see how your quality coverage is improving.
For the next two months Testim is offering these reports free of charge. Moving forward, additional licensing will be required to take advantage of these new insights. Sign into Testim to see the new reports now or contact your account manager with any questions.
According to the US Software Engineering Institute, 50% of defects are due to bad specifications.
Are you managing a software team or responsible for looking at their end-to-end delivery processes holistically?
Then, join this round-table discussion with Sam Hatoum, CEO of Xolv.io as he shares how he’s implemented proper specifications throughout projects he’s worked on to increase velocity by up to 35% and reduce defect rates down to 2%.
Thank you to everyone who participated in our round table discussion on The Future of Test Automation: How to prepare for it? We had a fantastic turnout with lots of solid questions from the audience. If you missed the live event, don’t worry…
You can watch the recorded session any time:
Alan Page, QA Director at Unity Technologies and Oren Rubin, CEO of Testim shared their thoughts on:
The current state of test automation
Today’s test automation challenges
Trends that are shaping the future
The future of test automation
How to create your test automation destiny
In this session they also covered:
Tips and techniques for balancing end to end vs. unit testing
How testing is moving from the back end to the front end
How to overcome mobile and cloud testing challenges
Insights into how the roles of developers and testers are evolving
Skills you should start developing now to be ready for the future of testing
Some of the audience questions they answered:
How do we know what is the right amount of test coverage to live with a reasonable amount of risk?
What is the best way to get developers to do more of the testing?
How do you deal with dynamic data, is the best practice to read a DB and compare the results to the front end?
Does test automation mark the end of manual testing as we know it?
There were several questions that we were not able to address during the live event so I followed up with the panelist afterwards to get their answers.
Q: What is Alan’s idea of what an automated UI test should be?
As much as I rant about UI Automation, I wrote some a few weeks ago. The Unity Developer Dashboard provides quick access to a lot of Unity services for game developers. I wrote a few tests that walk through workflows and ensure that the cross-service integration is working correctly.
The important bit is, that I wrote tests to find issues that could only be found with UI automation. If validation of the application can be done at any lower level, that’s where the test should be written.
Q: The team I work on complex machines with Android UI and separate backend. What layer would you suggest to concentrate more testing effort on?
I’d weight my testing heavily on the backend and push as much logic as possible out of the Android UI and into the backend, where I can test more, and test faster.
Q: Some legacy applications are really difficult to unit test. What are your suggestions in handling these kind of applications?
Q: How do you implement modern testing to compliment automation efforts?
My mantra in Modern Testing is, Accelerate the Achievement of Shippable Quality. As “modern” testers, we sometimes do that by writing automated tests, but more often, we look at the system we use to make software – everything from the developer desktop all the way to deployment and beyond (like getting customer feedback), and look for ways we can optimize the system.
For example, as a modern tester, I make sure that we are running the right tools (e.g. static analysis) as part of the build process, that we are taking unit testing seriously and finding all the bugs that can be found by unit tests during unit testing. I try to find things to make it easier for the developers I work with to create high quality and high value tests (e.g. wrappers for templates, or tools to help automate their workflow). I make sure we have reliable and efficient methods for getting feedback from our customers, and that we have a tight loop of build-measure-learn based on that feedback.
Q: Alan Page could you give an example of a test that would be better tested (validated) at a lower level (unit) as opposed to UI level?
It would be easier to think of a test that would not be better validated at that level. Let’s say your application has a sign-in page. One could write UI automation to try different combinations of user names, email addresses, and passwords, but you could write tests faster (and run them massively faster) if you just wrote API tests to sign up users. Of course, you’d still want to test the UI in this case, but I’d prefer to write a bunch of API tests to verify the system, and then exploratory test the UI to make sure it’s working well with the back end, has a proper look and feel, etc.
Q: How critical is today for a QA person to be able to code? In other words, if you are a QA analyst with strong testing/automation skills, but really have not had much coding experience, what would be the best way to incorporate some coding into his or her profile? Where would you start?
Technology is evolving in a rapid pace and the same applies to tools and programming languages as well. This being said, it would be good for testers to know the basics of some programming language in order to keep up with this pace. I would not say this is critical but it will definitely be good to have and with so many online resources available, it is even easier for testers to gain technical knowledge.
Some of the best ways I have known to incorporate coding into his/her profile would be via:
Online Tutorials and courses (Udemy, Coursera, Youtube videos)
Pairing with developers when they are programming. You can ask them basic questions of how things work, as and when they are coding. This is a nice way to learn
Attending code reviews helps to gain some insight into how the programming language works
Reading solutions to different problems on Stack Overflow and other forums
Volunteering to implement a simple feature in your system/tool/project by pairing with another developer
Organizing/Attending meetups and lunch ‘n’ learns focused on a particular programming language and topic
Choose a mentor who could guide you and give you weekly assignments to complete. Set clear goals and deadlines for deliverables
Q: My developers really like reusing cucumber steps, but I couldn’t make them write these steps. The adoption problem is getting the budget reallocated. Any advice for what I should do?
Reusing cucumber steps may not be necessarily a bad thing. It could also mean that the steps you may have written are really good and people can use them for other scenarios. In fact, this is a good thing in BDD (Behavior Driven Development) and helps in easier automation of these steps.
But if the developers are lazy and then reusing steps which do not make sense in a scenario, then we have a problem. In this case, what I would do is try to make developers understand why a particular step may not make sense for a scenario and discuss how you would re-write them. This continuous practice of spot feedback would help to instill the habit of writing good cucumber steps. Also, I would raise this point in retrospective and team meetings, and discuss it with the entire team. This will help to come to a common understanding of the expectations.
In terms of budget reallocation, I would talk to your business folks and project manager on the value of writing cucumber steps and how it helps to bring clarity in requirements, helps to catch defects early and saves a lot of time and effort which would otherwise be spent on re-work of stories due to unclear requirements and expectations for a feature.
Q: Can we quickly Capture Baseline Images using AI?
What exactly do you want the AI part to do? Currently, it’s not there are tools (e.g. Applitools and Percy.io) which can create a baseline very fast. I would expect AI to help in the future with setting the regions that must be ignored (e.g. field showing today’s date), and the closest thing I know is Applitools’ layout comparison (looking and comparing the layout of a page rather than the exact pixels, so the text can differ and the number of lines change, but still have a match).
Q: What are your thoughts on Automatic/live static code analysis?
Code analysis is great! It can help prevent bugs and add to code consistency inside the organization. The important thing to remember is that it never replaces functional testing and it’s merely another (orthogonal) layer which also helps.
Q: When we say ‘Automated Acceptance tests’, do they mean REST API automated tests which automation tool is good to learn?
No. They usually mean E2E (functional) tests, though acceptance tests should include anything related to approving a new release, and in some cases, this includes load/stress testing and even security testing.
Regarding good tools, For functional testing, I’m very biased toward Testim.io, but many prefer to code and choose Selenium or its mobile version Appium (though Espresso and Earl grey are catching on in popularity).
For API testing, there are too many, from the giant HP (Stormrunner) to medium sized Blazemeter, to small and cool solutions like APIfortress, Postman, Loadmill, and of course Testim.io.
Why not to call it full stack tests instead of e2e?
Mostly because e2e is used more often in the industry – but I actually prefer to use the google naming conventions, and just call tests small, medium, or large. Full stack / end-to-end tests fall in the large category.
Growing up as Testers in the software testing industry, we often go through a lot of thoughts and have several questions in mind such as:
How do I learn about software testing?
I like my job but how do I get to the next level?
Am I good at my job?
There are so many testing jargons people are using and I do not understand any of them
You are trying to communicate effectively but people still do not understand you
Based on testing software for a over a decade now, reading articles and blogs, interacting with practitioners from all over the world and analyzing my success and failures as a tester in the past several years; I discovered everything comes down to 3 Key Factors that paves the path to becoming a strong tester. These factors form the Strong Tester Model Stability as shown below.
Factor 1 – Motivation
“Run Your Own Race” – As Testers, we constantly keep comparing ourselves with other people, try to do more without paying attention to what our goals are and finally end up getting stressed as a result of being overworked and concentrating on lesser priority things. In life and in testing, we need to remember that, the ONLY person we are competing with; is OURSELVES. We need to identify our strengths and answer to our conscience, rather than comparing ourselves with others who totally have different set of goals.
Embrace Your Talents – Recognize your strengths and weaknesses. Embrace them and work to your skill sets with well defined goals and deadlines. Hold yourself accountable.
Go Explore – We will find our true passion only when we explore different things and take chances. Everyone starts from somewhere and no one is an overnight success. So start your journey and exploration. Try anything at least once with full dedication and determination. Remember “If you take a chance in life sometimes good things happen and sometimes bad things happen. But if you don’t take a chance, nothing happens.”
Tips and tricks for sustained and continuous motivation:
Have inspirational quotes to get you going when you are down or feel lost. Everyone has a trigger point, what is yours?
Have a Testing Notebook to note down all things testing when you read, explore and talk to people. Looking at your notes will spark different ideas for new techniques, strategies, articles, talks and so on. The opportunities are endless.
Use Mind Maps to visualize your thoughts and goals. This gives you something concrete to think about and helps in prioritizing each one of them.
Listen to inspiring podcasts and read motivational books.
Do deep work.
Have trusted Mentors to help you out in your journey. They help to challenge ideas, brainstorm solutions and guide you. Meet with them regularly via Skype or on a 1:1 basis.
Factor 2 – Communication
Intercultural communication – In a corporate world, we work with people from different cultures and regions. That being said, be cognizant of the cultural differences, usage of idioms and phrases and help them adapt to these differences. The learning goes both ways in terms of people from different cultures having an open mind and learning from the local people and vice versa.
Body Language – About 55% of our communication is through body language. Thus, having effective body language is important when working with other people. Be a good listener, have proper eye contact and pay attention.
Tone of Voice – Raising our voice in meetings to express our opinions or concerns does not work. When we raise our voice, the point does not get communicated across to the other person. The only thing noticeable during these times is the fact that, the other person is shouting and it automatically makes people react in a less amicable way.
Mental Models – People create their own mental models about objects, systems, ideas, description and so on. It is important to notice this and be open to hearing other people’s ideas.
Know your audience – Different people need different kinds of information. We need to modify our testing story based on our audience.
Safety Language – Prevent digging a hole for yourself by using safety language. Use word fillers like “could be”, “may have”, “would it be” etc. For Example – You can say “This defect could be reproduced under these conditions” instead of saying “This defect will be reproduced only in this way.”
Factor 3– Education
Validated Learning – The“Lean startup” principles of Build->Measure->Learn holds good in testing as well. Always build a Minimal Viable Product or Proof of Concept, then solicit feedback. Based on the feedback keep making the product better. Follow an iterative approach.
Developer – Tester pairing
Pairing while writing unit tests helps to identify gaps in development testing and these gaps can be addressed by Testers during their testing process.
Pairing in code reviews helps to find vulnerabilities in code.
Pairing also helps in learning the programming language.
Pair Testing with Testers/Developers
Paired Testing helps in continuous exchange of ideas/observations and helps to better explore the system
We gain experience only by committing mistakes. Remember “Things are never as bad as they feel or good as they sound.”
The track sessions help to learn about different topics relevant to the industry. If you do not like one session feel free to go to another one. A lot of money is invested by you and your company, so take advantage of it.
Do your research – Before going to a conference, identify people you want to network with, then meet up with them during the conference to learn and exchange ideas.
Hallway conversations and networking – A lot of learning takes place outside the conferences rooms in the hallways and networking events during the conference. Ensure you exchange business cards; in the back of the cards note down hints about the person and follow up with him/her after the conference.
Share your ideas, thoughts and problems with the community. Use blogs, LinkedIn and Twitter to help other people like you.
“If you want different results, you need to be willing to do things differently and different things” (from 12 Week Year)
As many yoga instructors do, they encouraged students to find balance. It’s an effective bit of advice in that it puts the onus on the practitioner. It’s also a popular concept for software testing, as industry experts often recommend that software teams find the balance between automation and manual testing practices. Another similarity is the trajectory of their adoption rates. One does not dive into yoga, but slowly adopts it over time until it becomes a part of their daily routine. The same goes for test automation: you can’t expect to start automating everything from scratch. Good test automation is the culmination of work over time.
Why not all manual?
Manual testing is fine, and was once the status quo, but with high adoption of Agile and continuous delivery methodologies, it just doesn’t scale. For software development, every enhancement or new feature that’s rolled out for an app must have a corresponding set of tests to ensure the new functionality works and does not break the code from the previous version. In this regard, to check all of the file directories, database, workflows, integrations, rules and logic manually would be extremely time consuming.
For companies looking to improve time to market and increase test coverage, automation provides significant advantages over manual testing. A typical environment can host thousands of test cases of varying complexity to be executed effortlessly and nonstop. As a result, automation dwindles the time required to perform mundane, repetitive test cases from days to hours and even minutes if you run in parallel. That inherent velocity increase is what attracts teams to automated testing.
Why not all automation?
The benefits of automation are obvious. Automation provides the ability to quickly react to ever-changing business requirements and generate new tests continuously. Is it reasonable then to assume that the more you automate the more benefits you reap? Then why not just automate everything?
Traditionally you would automate only once your platform is stable and the user flow is pretty consistent. This was primarily due to the amount of time it took to write a test, sometimes hours and even days. Plus the amount of time it took to update and maintain tests as code changed. The ROI was not justified up to the point your user flow is stable.
Feedback from our customers such it has changed. We recently heard a number of our clients who automate everything and even as early as wireframes are ready. We reduced authoring time by about 70% of the time and maintenance time by about 90%. Creating a user flow now takes a few minutes with Testim and so is updating the user flow. Reducing the time per test to a few minutes instead of hours and days makes it much more affordable to automate, providing immediate ROI from feedback loop benefits.
So should you strive to 100% automation? Well no but you can get to 80%-90% which was unheard of until recently. There are still scenarios that only humans can do such as Face ID. There are also other aspects of your Software Development Lifecycle (SDLC) that automation is dependent on.
There is an ongoing struggle to keep up with the rate dictated customer pressure and competition to produce on a continuous basis. And not just produce for production’s sake, but a quality product that’s been thoroughly tested. That’s why we created Testim…