Tag

software quality

Browsing

One of the most important factors related to automated tests is Maintenance. A lot of effort is spent on maintaining the tests than writing actual tests.  A recent study suggested about 30% of testers time is spent on maintenance.This leads to wastage of valuable time and effort by the resources, which they could have rather spent on testing the actual application.

Imagine a world where the software can maintain tests without human interaction? This world has become a reality with Testim.io. We use Artificial Intelligence (AI) underneath the hood, which provides self-healing maintenance i.e problems are detected by the AI and automatically fixed without human intervention.

Testim.io also help to speed up the maintenance of tests by providing the follow features within our platform-

  1. Version Control

At any given time, it is important to have logs of what changes were made to a particular test. This way we can always revert back to an older version of test as and when required. Our platform provides this functionality by showing all the version history by going to the Properties panel of the setup step and clicking on “See old revisions”

  1. Branching

At Testim.io, we firmly believe in the “Shift Left Paradigm” where Development and Testing must start in parallel as early as possible in the software development lifecycle. Keeping this in mind, we provide the functionality to teams to create separate branches for each team member and work on the same projects and tests. This way, no one can overwrite the changes of the other team members and teams can work on the same code base at any instant of time

In our platform, we just need to select “Fork” to create a new branch and we can also switch between existing branches


        3.  Scheduler

Users have the option of scheduling their tests. This helps to run the tests automatically at a certain day and time without any manual intervention. We can also get notified via email in case of any errors

 

Troubleshooting

As testers, we spend considerable amount of time troubleshooting issues. To help in troubleshooting, our platform offers different options to the user to narrow down the scope of the problem. These options  are as follows-

  1. Screenshots

The screenshot feature explained in the “Authoring and Execution” section helps users to know what was the baseline image and what was the actual image found.

  1.   Properties Panel

The properties panel helps to capture the error messages and display it to the user. The user also has the option of interacting with DOM and see what objects were extracted during the run

  1. Test Logs

Logs are a rich source of information on what happened underneath the UI. We provide test logs when the user runs the tests on our grid or a 3rd party grid. The option can be found in the in top section of editor

  1. Bug Reporting

One of the most time consuming aspects of testing is after finding a bug, we need to report it to the developer with relevant information, to speed up the troubleshooting and fixing of issues.

With Testim.io you can do this with a single click with the help of our chrome extension. All the details related to the bug are automatically generated for you.

  1. Documentation

We put in a lot of effort to document most of the features of the tool in our User Documentation found under the “Educate” tab.

We also have detailed videos on how to troubleshoot your tests quickly

Troubleshooting Part 1- Element is not visible

Troubleshooting Part 2 – Element not found

Troubleshooting Part 3 – Timing issues

Troubleshooting Part 4 – Issues related to mouse hover

With the above features, Testim.io helps to create stable tests that are highly maintainable.

The below posts gives more in depth analysis of Testim in terms of different features that make authoring and execution of tests really simple and how to create reusable components that improves the extensibility of the tool

https://blog.testim.io/bringing-simplicity-to-authoring-and-execution-of-automated-tests/

https://blog.testim.io/how-to-make-reusable-and-extensible-code-using-testim-io/

 

Artificial Intelligence (AI) and machine learning (ML) are advancing at a rapid pace. Companies like Apple, Tesla, Google, Amazon, Facebook and others have started investing more into AI to solve different technological problems in the areas of healthcare, autonomous cars, search engines, predictive modeling and much more. Applying AI is real. It’s coming fast. It’s going to affect every business, no matter how big or small. This being the case how are we as Testers going to adapt to this change and embrace AI? Here is the summary of different things you need to know about using AI in software testing.

Let’s summarize how the testing practice has evolved over the last 4 decades

  • In the 1980’s majority of software development was waterfall and testing was manual
  • In the 1990’s, we had bulky automation tools which were super expensive, unstable and had really primitive functionality. During the same time, there were different development approaches being experimented like Scrum, XP, RAD (Rapid Application Development)
  • From 2000, the era of open source frameworks began
    • People wanted to share their knowledge with the community
    • Started encouraging innovation and asking community of like minded people to help out in improving testing
    • Agile became a big thing, XP, Scrum, Kanban became a standard process in the SDLC
    • There were need for faster release cycles as people wanted more software features delivered faster
  • In the 2010’s, it was all about scale, how to write tests fast and find bugs faster
    • Crowdtesting started
      • Encouraging other people to give feedback on the application. Free and Paid services
    • Cloud testing started
      • People started realizing they need more
        • Server space
        • Faster processing
      • Started to realize the problem of maintenance. How expensive it is to buy hardware and software for maintaining your tests
      • Then we have
        • DevOps
        • Continuous Testing
        • CI/CD integration
  • I believe the Future will be about Autonomous Testing using Machine Learning and AI

 

Basics of AI

Let’s start by de-mystifying some of the terminologies related to AI

  • Artificial Intelligence (AI) is an area of computer science that emphasizes the creation of intelligent machines that work and react like humans
  • Machine Learning (ML) evolved from the study of pattern recognition and computational learning theory (studying design and analysis of ML algorithms) in AI. It is a field of study that gives computers ability to learn without being explicitly programmed
  • Deep Learning(DL) is one of the many approaches to ML. Other approaches include decision tree learning, inductive logic programming, clustering and Bayesian networks. It is based on neural networks in the human body. Each neuron keeps learning and interconnects with other neurons to perform different actions based on different responses

 

There are 3 types of widely used ML algorithms

  • Supervised Learning – We are giving the right training data (input/output combinations) for the algorithm to learn
    • Examples
      • Give bunch of e-mails and identify spam e-mails
      • Extracting text from audio
      • Fill out a loan application and find the probability of the user repaying the loan
      • How to make user click on ads by learning their behavior
      • Recommendation engines on Amazon, Netflix where customer is recommended products and movies
      • Amazon uses AI for logistics
      • Car Optimization
      • Autonomous cars
  • Un-supervised learning – We give a bunch of data and see what we can find
    • Examples
      • Taking a single image and creating a 3D model
      • Market Segmentation
  • Reinforced learning – Based on concept of reward function. Rewarding Good/Bad behavior and making the algorithm learn from it. E.g. Training a Dog

 

Real life AI applications to visually see how it works

  • Quick Draw from Google
  • Weka is an open source project where they are using ML algorithms for data mining

 

What challenges can AI solve?

Let’s discuss the challenges the industry faced while transitioning to agile and what’s still remains a challenge:

How can we use AI to solve testing problems?

There are many companies taking multiple approaches to solve different problems related to software testing and automation. Testim.io is one such company

Testim.io uses Dynamic Locators, The Artificial Intelligence (AI) underneath the platform in real time, analyzes all the DOM objects of a page and extracts the objects and its properties. Finally, the AI decides the best location strategy to locate a particular element based on this analysis. Due to this, even if a developer changes the attribute of an element, the test still continues to run and this leads to more stable tests. As a result, the authoring and execution of automated tests are much faster and more stable.

 

Here is the detailed insight of how our AI works – https://www.softwaretestinghelp.com/testim-io-tool-tutorial/

One of the good practices of writing automated tests is creating reusable components that can be used in different parts of our test suite.

Why is this important?

Creating reusable components is important because it

  • Helps to increase the readability of the automated tests
  • Saves effort by not repeating the same set of steps in different parts of the tests
  • Any changes to the reusable step needs to be done only in one place and it is reflected throughout the tests, across different projects
  • Makes the automated tests more extensible

 

Testim.io helps to ensure Reusability by “Grouping” and “Parameterization”.

  • Grouping

Any number of related steps can be grouped into one reusable component.

For Example – The “Login” scenario is one of the most commonly used steps in any application. The way we can create a reusable “Login” step would be to select the steps we want to group together and clicking on “Add new Group” as shown below

  1. Parameterization

Our platform gives the option of testing application through various input combinations via Parameterization.

This can be achieved in various ways. One way to do this is to give all the input parameters we would need to test the application in the form of a JSON file in the Setup step (The first step of our tests) as shown below

Then add the variable names used in the json file in the appropriate fields of the step as show below

 

Another important aspect of automation is building your tests such that it is extensible.

Why is this important?

As the product and teams grow, there will be need to test more complex functionalities which would require building upon already existing tests. This being the case, the automation suites need to be simple, understandable and should be easy to add more tests to already existing test suites with low coupling and high cohesion.

Testim.io gives the flexibility for organizations to extend the functionalities of our platform using JavaScript and HTML. This way, any functionality our platform does not handle; the user can write their own code to build a robust automation framework

For Example – Say we want to validate the “Select Destination” button from our previous examples. The way to do this would be.

  • Click on “Add custom action”
  • Give a name to the New Step and click on “Confirm”
  • Click on “PARAMS” and Select “HTML” for this example
  • Add Custom Code

The new step with Custom Code gets added to the list of already existing steps

The above features help to make the automation suite more reusable and extensible.

The below posts gives more in depth analysis of Testim in terms of different features that make authoring and execution of tests really simple and easy to maintain

https://blog.testim.io/bringing-simplicity-to-authoring-and-execution-of-automated-tests/

https://blog.testim.io/maintenance-of-tests-made-easy-with-testim-io/

Do you ever ask yourself these questions?

  • How do I know if my quality is improving?
  • What did the team do this week and what was the impact?
  • Do we have enough test coverage in the sprint?

With Testim’s new Managerial Reports, you never have to worry about getting answers to these questions again. Get deep insight into project status and quality metrics which provide granular details into execution cycles, active runs, run durations and success rates – all available online or sent weekly to your inbox.

Testim Test Automation Managerial Reports

These reports, dashboards and KPIs quickly summarize the team’s effort invested over the course of the week, identify tests that require attention as well as if additional effort is required to improve your quality score. Easily track trends week over week and see how your quality coverage is improving.

Testim Test Automation Managerial Reports

For the next two months Testim is offering these reports free of charge. Moving forward, additional licensing will be required to take advantage of these new insights. Sign into Testim to see the new reports now or contact your account manager with any questions.

Guide: Evaluating UI and Mobile Automated Testing Solutions

According to the US Software Engineering Institute, 50% of defects are due to bad specifications.

Are you managing a software team or responsible for looking at their end-to-end delivery processes holistically?

Then, join this round-table discussion with Sam Hatoum, CEO of Xolv.io as he shares how he’s implemented proper specifications throughout projects he’s worked on to increase velocity by up to 35% and reduce defect rates down to 2%.

Date: Tuesday, April 24
Time: 12:00pm PT
RESERVE MY SEAT

Why you should attend this session?

  • Get tips for balancing the tradeoff between velocity and quality
  • Learn how to smooth the transition between development phases while ensuring full alignment
  • See how your test automation strategy becomes a result of your upfront specification planning
  • Succeed in reducing defects before a single line of code is written
  • Minimize the handoffs between Engineering, Quality and Product
  • Capture some quick wins to implement quality practices that will
  • Influence cultural change, speed up release cycles and improve your customer’s experience

Thank you to everyone who participated in our round table discussion on The Future of Test Automation: How to prepare for it? We had a fantastic turnout with lots of solid questions from the audience. If you missed the live event, don’t worry…

You can watch the recorded session any time:

Alan Page, QA Director at Unity Technologies and Oren Rubin, CEO of Testim shared their thoughts on:
  • The current state of test automation
  • Today’s test automation challenges
  • Trends that are shaping the future
  • The future of test automation
  • How to create your test automation destiny
In this session they also covered:
  • Tips and techniques for balancing end to end vs. unit testing
  • How testing is moving from the back end to the front end
  • How to overcome mobile and cloud testing challenges
  • Insights into how the roles of developers and testers are evolving
  • Skills you should start developing now to be ready for the future of testing

Some of the audience questions they answered:

  • How do we know what is the right amount of test coverage to live with a reasonable amount of risk?
  • What is the best way to get developers to do more of the testing?
  • How do you deal with dynamic data, is the best practice to read a DB and compare the results to the front end?
  • Does test automation mark the end of manual testing as we know it?

There were several questions that we were not able to address during the live event so I followed up with the panelist afterwards to get their answers.

Q: What is Alan’s idea of what an automated UI test should be?

As much as I rant about UI Automation, I wrote some a few weeks ago. The Unity Developer Dashboard provides quick access to a lot of Unity services for game developers. I wrote a few tests that walk through workflows and ensure that the cross-service integration is working correctly.

The important bit is, that I wrote tests to find issues that could only be found with UI automation. If validation of the application can be done at any lower level, that’s where the test should be written.

Q: The team I work on complex machines with Android UI and separate backend. What layer would you suggest to concentrate more testing effort on?

I’d weight my testing heavily on the backend and push as much logic as possible out of the Android UI and into the backend, where I can test more, and test faster.

Q: Some legacy applications are really difficult to unit test.  What are your suggestions in handling these kind of applications?

Read Working Effectively with Legacy Code by Michael Feathers, and you’ll find that adding unit tests to legacy code isn’t as hard as you thought it was.

Q: How do you implement modern testing to compliment automation efforts?

My mantra in Modern Testing is, Accelerate the Achievement of Shippable Quality. As “modern” testers, we sometimes do that by writing automated tests, but more often, we look at the system we use to make software – everything from the developer desktop all the way to deployment and beyond (like getting customer feedback), and look for ways we can optimize the system.

For example, as a modern tester, I make sure that we are running the right tools (e.g. static analysis) as part of the build process, that we are taking unit testing seriously and finding all the bugs that can be found by unit tests during unit testing. I try to find things to make it easier for the developers I work with to create high quality and high value tests (e.g. wrappers for templates, or tools to help automate their workflow). I make sure we have reliable and efficient methods for getting feedback from our customers, and that we have a tight loop of build-measure-learn based on that feedback.

Q: Alan Page could you give an example of a test that would be better tested (validated) at a lower level (unit) as opposed to UI level?

It would be easier to think of a test that would not be better validated at that level. Let’s say your application has a sign-in page. One could write UI automation to try different combinations of user names, email addresses, and passwords, but you could write tests faster (and run them massively faster) if you just wrote API tests to sign up users.

Of course, you’d still want to test the UI in this case, but I’d prefer to write a bunch of API tests to verify the system, and then exploratory test the UI to make sure it’s working well with the back end, has a proper look and feel, etc.

Q: How critical is today for a QA person to be able to code? In other words, if you are a QA analyst with strong testing/automation skills, but really have not had much coding experience, what would be the best way to incorporate some coding into his or her profile? Where would you start?

Technology is evolving in a rapid pace and the same applies to tools and programming languages as well. This being said, it would be good for testers to know the basics of some programming language in order to keep up with this pace. I would not say this is critical but it will definitely be good to have and with so many online resources available, it is even easier for testers to gain technical knowledge.

Some of the best ways I have known to incorporate coding into his/her profile would be via:

  • Online Tutorials and courses (Udemy, Coursera, Youtube videos)
  • Pairing with developers when they are programming. You can ask them basic questions of how things work, as and when they are coding. This is a nice way to learn
  • Attending code reviews helps to gain some insight into how the programming language works
  • Reading solutions to different problems on Stack Overflow and other forums
  • Volunteering to implement a simple feature in your system/tool/project by pairing with another developer
  • Organizing/Attending meetups and lunch ‘n’ learns focused on a particular programming language and topic
  • Choose a mentor who could guide you and give you weekly assignments to complete. Set clear goals and deadlines for deliverables

Q: My developers really like reusing cucumber steps, but I couldn’t make them write these steps. The adoption problem is getting the budget reallocated. Any advice for what I should do?

Reusing cucumber steps may not be necessarily a bad thing. It could also mean that the steps you may have written are really good and people can use them for other scenarios. In fact, this is a good thing in BDD (Behavior Driven Development) and helps in easier automation of these steps.

But if the developers are lazy and then reusing steps which do not make sense in a scenario, then we have a problem. In this case, what I would do is try to make developers understand why a particular step may not make sense for a scenario and discuss how you would re-write them. This continuous practice of spot feedback would help to instill the habit of writing good cucumber steps. Also, I would raise this point in retrospective and team meetings, and discuss it with the entire team. This will help to come to a common understanding of the expectations.

In terms of budget reallocation, I would talk to your business folks and project manager on the value of writing cucumber steps and how it helps to bring clarity in requirements, helps to catch defects early and saves a lot of time and effort which would otherwise be spent on re-work of stories due to unclear requirements and expectations for a feature.

Q: Can we quickly Capture Baseline Images using AI?

What exactly do you want the AI part to do? Currently, it’s not there are tools (e.g. Applitools and Percy.io) which can create a baseline very fast. I would expect AI to help in the future with setting the regions that must be ignored (e.g. field showing today’s date), and the closest thing I know is Applitools’ layout comparison (looking and comparing the layout of a page rather than the exact pixels, so the text can differ and the number of lines change, but still have a match).

Q: What are your thoughts on Automatic/live static code analysis?

Code analysis is great! It can help prevent bugs and add to code consistency inside the organization. The important thing to remember is that it never replaces functional testing and it’s merely another (orthogonal) layer which also helps.

Q: When we say ‘Automated Acceptance tests’, do they mean REST API automated tests which automation tool is good to learn?

No. They usually mean E2E (functional) tests, though acceptance tests should include anything related to approving a new release, and in some cases, this includes load/stress testing and even security testing.

Regarding good tools, For functional testing, I’m very biased toward Testim.io, but many prefer to code and choose Selenium or its mobile version Appium (though Espresso and Earl grey are catching on in popularity).

For API testing, there are too many, from the giant HP (Stormrunner) to medium sized Blazemeter, to small and cool solutions like APIfortress, Postman, Loadmill, and of course Testim.io.

Why not to call it full stack tests instead of e2e?

Mostly because e2e is used more often in the industry – but I actually prefer to use the google naming conventions, and just call tests small, medium, or large. Full stack / end-to-end tests fall in the large category.

Time is money. How many times have you heard that?

We are truly a “Startup nation” constantly racing against the clock to deliver multiple features and execute sprint content, to meet our customer’s demands. This intense pace is an everyday reality. As a QA Engineer that has worked with and in different organizations, I have experienced it up-close and personal.

On one side there are owners and investors – they want to see growth. On the other side, there are customers – they want features and capabilities that work. And then there is us, Testers – we want to deliver quality.

Where does software quality lie in the release process?

Let’s start by defining software quality and how would you measure it?  In the context of software engineering,  software quality refers to how well it meets functional and non-functional requirements.  In terms of how well you can measure it; there are various metrics associated with it and is complex. This is like asking, “how would you define a high-quality watch, car, or a clothing item?”

Could it be that from using the product you can feel that its creator/maker used good materials (even if it means that the price would be higher)? If you use it for a long time, would it still be preserved from standard wear and tear? Is it designed to be comfortable? Fun to use? Does it break or overheat when you accelerate to a high speeds or drive long distances?

With that said, a Mercedes costs 10 times more than a Honda. Does that mean that a Honda is not a good quality car?

All of these examples teach us that quality is based on a perceived notion of price to value.

  • Does the product serve its purpose in a good way?
  • Can it stand up to our “use demands” in a reliable, long-lasting way?

Price can be interpreted in different ways as well – for example – implementation and maintenance time.  Don’t be fooled,  breaking this perception is a lot easier than building it in the eyes of our users. I can think of more than one car brand that has managed to break our perception of it in the last decade or so.

The farther you run, the more expensive it is to go back.

What I’m about to write will not be a shock to anyone, and still, you will be surprised to hear the number of organizations I see that just don’t seem to assimilate that idea. The earlier you will incorporate quality practices into the product’s life cycle – the less money you will spend on fixing it in the long runI have seen how a wide aspect feature is being invented, designed and developed, only to understand that it is different from the poet’s intention, or does not serve its purpose the best way.

What can possibly go wrong? Nothing much, just:

  • Features developed contrary to the customer’s tastes/needs
  • Modules that do not carry out the action for which they were designed
  • Time-consuming debates on whether it was a bug or maybe the characterization was not unequivocal and clear enough
  • Going back and forth by working on/fixing the same module for several times
    • As a result: lack of time to perform all of the evaluated “sprint content” = less features, less progress
  • Features that aren’t clear enough for the user and result in complaints or support issues
  • Unsatisfied customers
  • Bad user experience
  • Bugs in production
  • Wasting time = money

Make a decision, start the change!

Start by understanding the importance of having a wide range testing infrastructure in your organization. Sure, it will not happen overnight. Yet still, it is an investment that produces huge value.

Suit up, sneakers on, and start the process:

  • Start designing a work procedure that includes clear written requirements. People never completely understand one another. If there is no written documentation, it can become challenging to examine what to develop and what to test. I am realistic, there is no place or time for detailed requirements in a startup. However, we can design a format which is short enough to be written in our race and clear enough for the developer to understand what to perform.  
  • Implement the use of a knowledge base and management tools that will suit your needs. Lets define knowledge base.  As far as I’m concerned, it can be a Google-Drive but you still need one place that I can go to if I need to know how a feature works. Now, about management tools, lets just say that it is a topic of a whole another article and still what you need to understand is that there are free requirement management, bug management and process driven tools that could help you organize your everyday tasks.
  • Start building a work process which will suit your company’s needs and corporate culture. The same as there are rules to running a marathon, there has to be rules to running a startup. When do you start the run? When can the tester join? What is the criteria that indicates that the lap is over? What can and can’t be done in the middle of the sprint? Every game needs its set of rules.
  • Implement productive communication between product and QA. How? Well for one thing here is personal example. Start by implementing a productive and positive discussion. Every individual requires an approach so they will not “lock up” to a defensive position and observe your words. Take some time to learn your environment and how to approach each member.
  • Assign a quality lead that will provide you with the added value of a productive process.
  • Incorporate quality practices as early on as possible in the concept, design, and development. You will be surprised how effective it is to review requirements (even before one line of code has been written) and how much time it can save you in the long run.  
  • Make an organizational level decision to make quality an high priority issue, and let QA do their job with the appropriate amount of authority.

Two points I would like to add to all that has been written:

  1. It is a process and conception that takes time, but it will return its investment.
  2. It’s not magic, there will always be bugs! The question is in what amount and what severity?

Remember the question we asked at the beginning of the article? (How do we fit in this never-ending race?)

I would like to wrap up this article with a quote by the famous Tester Michael Bolton:

“I wish more testers understood that testing is not about “building confidence in the product”. That’s not our job at all. If you want confidence, go ask a developer or a marketer. It’s our job to find out what the product is and what it does, warts and all. To investigate. It’s our job to find out where confidence isn’t warranted where there are problems in the product that threaten its value.”

Practical steps to shift left

Last month we published “Do you know how much your quality costs?” where we discussed the importance of shifting left and shortening the feedback loop as it relates to the cost of quality. According to Bank of America, discovering a bug in development would cost about 5 hours to fix and $390. Discovering a bug in QA cost about x30. 127 hours to fix, due to to the time it takes to reproduce, the back and forth between dev and QA and the need to trace back the code. Fixing a bug in production – Estimated at $72,000. Shifting left allows testing close to development as possible and a fast and continuous feedback loop.

Direct cost to find/fix a defect

Four key ingredients to achieving shift left:

  1. Test automation – Traditional QA processes would result in months of testing before you get feedback. A productive tester can get about 20 tests done in a single day. Assuming your test suite includes dozens of tests there are two ways to get the feedback loop down to minutes: Hire hundreds of testers or automate your tests. Automated tests don’t get tired, take days off, go out for lunch etc.
  2. High concurrency – Shifting left means giving near real time feedback i.e minutes. Running dozens of tests, each taking roughly 30-60 seconds, with the expectations of getting the results back in, say 5 min., can only be done if you are running your tests in parallel. The higher your level of parallelism the shorter the feedback loop is. Running your tests in parallel required a grid, a cloud based infrastructure that includes virtual environments allowing you to spin in and out different browsers.
  3. Build-test-deploy automation – Shifting left means you are optimizing the handover across the different stages of the development process. The development cycle includes dozens of handovers and human intervention can always delay that process hence you want to automate the handover using Continuous Integration (CI) servers. CI servers monitor events through your development cycle, such as a new build, and can be programed to trigger a set of actions once a certain event occurred, such as triggering your automated tests.
  4. Suites – It is best practice to optimize your quality assurance processes by segmenting your tests into suites and running different suites at different stages of your development cycle. For example you full regression might include thousands of tests. Running all those tests every time a develop push code would be time and resource consuming. If 2-3 key scenarios failed, there is no point in waiting for the entire suite to be over. You might want to create a small subset of tests, composed of your high priority flows, and run that “sanity” suite often, while running the more extensive suites once if sanity suite ran successfully.

 

Guide: Evaluating UI and Mobile Automated Testing Solutions

Be smart & save time...
Simply automate.

Get Started Free