Author

Raj Subramanian

Browsing

Selenium has been a popular automation testing framework for the past several decades. But, as applications have become more complex in the past several years, especially with the use of popular JavaScript frameworks such as Angular.js, Vue.js, React.js and Ember.js for building web applications; Selenium has found it hard to adapt to these technologies.

For example– If you are a currently using Selenium, have you ever experienced any of the below problems-

  • Spending majority of your valuable testing time fixing flaky tests?
  • Unable to make automation progress due to the lack of skilled programmers to write automated tests?
  • Not finding enough support in the open source community when new libraries and updates break existing tests and you have no idea what to do?
  • Need visual validation when a step fails, to visually understand the exact reason for the failure?
  • Insufficient logging information when your tests fail?
  • Finding it hard to seamlessly integrate your automated tests within your CI/CD pipeline?

If you answered YES, to any of the above questions, then you are not alone!!! According to new Gartner research, “Selenium is the de facto standard for front-end test automation of modern web technologies due to the flexible and powerful browser automation capabilities of WebDriver… It’s a sophisticated tool that isn’t easy to learn. It requires your team to have programming expertise and familiarity with object-oriented languages.” So, Selenium is good if you have the necessary programming expertise in your team, if not it becomes an obstacle.

There is no need to fear as there are good alternatives to Selenium. Below are the top 3 widely used alternatives currently available to give you and your team the flexibility to build robust automation test suites/frameworks.

  1. Cucumber

This is an open source testing tool that focuses on BDD (Behavior Driven Development). It emphasizes writing requirements in plain english in the form of Given, When and Then statements. This is commonly referred to as “Gherkin” syntax. You then convert these GWT statements into code using Java, JavaScript, Ruby or Kotlin.  This helps to enforce collaboration and bring more clarity to requirements.

On the flip side, it still needs someone with programming background to write the binding code to convert these GWT statements into usable actions. Also, the GWT format leads to a maintenance nightmare especially when many people collaborate and start making changes to different steps. Finally, it does not have any visual validation and has insufficient logging features making it hard to troubleshoot errors.

  1. SikuliX

This is an open source GUI based automation tool. It uses image recognition powered by OpenCV to automate anything with a UI. This includes desktop and web applications. It supports multiple programming languages including Python, Ruby and JavaScript.

Although this tool has robust UI validation functionalities, it lacks a lot of necessary features that are needed for stable automation such as locating elements based on attributes, creating reusable components and modularizing your tests for easier maintenance.

  1. Testim

Testim uses artificial intelligence (AI) for the authoring and execution of automated tests. The focus is on functional testing, end-to-end testing and UI testing. The Dynamic Location strategy used within the tool sets it year’s apart from its competitors. The Artificial Intelligence (AI) underneath the platform in real time, analyzes all the DOM objects of a page and extracts the objects and its properties. Finally, the AI decides the best location strategy to locate a particular element based on this analysis. Due to this, even if a developer changes the attribute of an element, the test still continues to run and this leads to more stable tests. This is the major advantage of using Testim compared to other frameworks like Selenium which uses static locators. Also, the “self-healing” mechanism of the AI helps to proactively detect problems in your test and fixes it automatically for you. As a result, the authoring and execution of automated tests are much faster and more stable.

Testim is not a completely code-less tool; you can use JavaScript and HTML to write complex programming logic (if needed) for your applications. This helps to make the automation suite more extensible. It provides easy integration with your CI/CD pipeline and most importantly helps to involve the whole team in automation including technical and non-technical people.

In summary, Selenium has really good capabilities but knowing there are other alternatives out there helps to give your more options to make your automation effort more easier and stable.

 

We recently hosted a webinar on AI and its influence on test automation with an awesome panel consisting of Jason Arbon, Oren Rubin, Dionny Santiago and me being the moderator. There were lot of great discussions on this topic and we wanted to share it with the community as well.

Below you will find the video recording of the webinar, the list of questions and answers that we couldn’t get to during the webinar and different resources to learn about AI and testing. Please feel free to reach out to me in case of any questions at raj@testim.io or any of the panel members.

Video Recording

 

Q&A

I am a performance engineer and am working on AI for quality gates in load testing results…that needs to be a high priority for the “future” which is “now”.How do you think bots can be used in this area?
@Jason: UI-Bots can help generate user-like load directly via the application.  Though, for most load testing problems, would recommend something like charles proxy, or internal ways to spin up load, and only use the ‘expensive’ UI-based bots to see how the app works E2E for the user under load.

With rapid changes in agile requirements, how do we make the machines learn or adapt to the changes every time?
@Jason: The ai bots most folks are working on these days (vendors) will auto discover new features in the app and exercise them.  At test.ai we have a set of 5k+ tests written for common flows in apps, so if you add a new feature to your app that looks like something similar on another app, the bots will auto apply that test case to the new build.

@Raj: The more tests you run, the smarter the AI becomes in detecting changes in the application. It will start detecting changes in application’s UI, element attributes and start adapting the tests automatically to these changes due to its self-healing mechanism. It can identify flaky tests, optimize waits in between steps and also proactively fix issues for us before they occur.

With BDD model and shift left and demand for testing at unit and service/api layer where does E2E testing stand?
@Jason: Dionny’s work can help generate valid permutations of existing API test case parameters/flows.  Also, clustering methods can help identify misbehaving servers via logs of activity during api testing or production.

Where can we find Dionny’s paper on AI testing, you were talking about?
Dionny’s paper

Lot of automation test scripts fail due to test data issues, can we use AI to tackle those kind of issues?
@Jason: Thats a broad category of failure types, but yes, ‘AI’ can be taught to auto associate correct data with the right application states.  Google also shared some ‘test selection’ findings using ML to help decide what to do with all those failing tests:  https://testing.googleblog.com/2018/09/efficacy-presubmit.html

I wanted to understand, what does really mean by AI in testing?if it mean by machine will perform testing? if machine will test then if they will be already defined with scenarios which needs to be tested? is it same as automation testing as there also we don’t need manual intervention?
@Jason: Generally, AI in testing, means applying machine learning / AI techniques to test applications.  There is also ‘Testing AI’ which refers to approaches to test AI/ML-based products and features. There are a variety of ways to apply AI to testing, some leverage pre-written test cases and the AI is used to automatically execute the tests, create variations, or analyze the results.  Some AI based systems are trained to mimic general human behavior and can execute basic ‘flow testing’ for many apps, without pre-written test scenarios.  The bots we build at test.ai can read BDD/Scenarios and execute them against a set of applications.  As for need for human intervention, like automation, there is still the need for plenty of human intervention in AI-based testing approaches these days 🙂 Humans gather oracle/training data for the AI. Humans measure the correctness of the ‘AI’, and humans evaluate the significance of the AI-based test results as they relate to business/shipping decision.

@Raj: In addition to what @Jason was saying,  I wanted to mention that, AI can have a positive impact on several facets of software testing especially test automation. There have been so many different tools and frameworks that have come up trying to solve different kinds of problems related to test automation but one problem that has been a constant challenge till date, is the aspect of “maintenance”. One of the main reasons for this is the use of static locators. With static locators, we use only one attribute of an element to uniquely identify it on a page and if this changes, the test breaks and as testers we end up spending a considerable amount of time troubleshooting the problem and fixing it. Based on research, about 30% of testers’ time is consumed in just maintaining tests. Can you imagine the opportunity cost associated with this effort? It is mind blowing. Testers’ time is valuable and better spent on actually exploring the application and providing information to help stakeholders make informed decisions about the product. With AI based testing we can overcome this problem by using dynamic locators. Dynamic Locators is a concept where we use multiple attributes of an element to locate it on on the page instead of a single attribute. This way, even if one attribute changes, the element can still be successfully located with the help of other attributes that have already been extracted from the DOM by the AI.

Can you guys elaborate on how do AI-based tests learn acceptance criteria that normally has to be defined by humans?
@Jason: Depends on the AI system being used.  The bots at test.ai execute human-written test cases.  Acceptance tests are written at a very high level of abstraction, and the bots do all the test execution.  Reporting is as normal for test automation.  In summary, just tell the bots what your acceptance criteria are.

What an automation tester needs to learn to align with the future of ai in testing ?
@Jason: A good set of links to learn are here:  https://www.linkedin.com/pulse/links-ai-curious-jason-arbon/ . You can also start to leverage/experiment with “AI” via the current testing vendors. If you are already familiar with selenium/appium like testing, there is a new open source API that uses AI for element selection that you can use today:  https://medium.com/testdotai/adding-ai-to-appium-f8db38ea4fac?sk

Is AI platform dependent. like desktop application or web/mobile?
@Jason: 
Depends on the AI approach/solution.  Many are platform dependent.  The bots we build at test.ai though are not platform dependent–a key feature.  The bots are platform-independent as the machines are trained to recognize UI elements much like humans do, and humans are platform dependent ;).

Is there an Open Source project that allows to apply AI to locate the elements?
@Jason: Yes, for appium today and likely Selenium soon:  https://medium.com/testdotai/adding-ai-to-appium-f8db38ea4fac?sk

How can AI be used for improving test coverage ?
@Jason: AI can help generate many more validate test scenarios than a human could create. AI also enabled re-use of test artifacts so a test written for one app, can also execute on a similar app with no human intervention.

@Raj:  Now with AI, you can also connect your production apps to the testing cycle. This means that we can create tests based on actual flows done by the user in production. Also, the AI can observe and find repeated steps and cluster them to make reusable components in your tests. For Example – Login, Logout scenarios. So now we have scenarios that are actually created based on real production data instead of us assuming what the user will do in production. In this way, we also get good test coverage based on real data.

Will AI testing replace selenium appium and all tools and technologies?
@Jason: Asymptotically.

Is AI really better?
@Dionny: Traditional testing teams focus on either a single app, or a small set of apps; whereas, AI can learn from millions of different examples and apps. The more data we show the AI, the better it gets. Also, the AI never gets tired!

What are the immediate benefits of using AI?
@Raj: Apart from the benefits already mentioned in the answers above, AI can also help in increasing team collaboration. The field of test automation has historically been a technical tester focused community. This stigma can also change with AI. What this means is, non-technical resources no longer need to fear code and technology, rather AI will help to bridge the gap between the technical know-how and authoring and execution of tests making life easier for teams.

Will our jobs be replaced?
@Raj: Over the past decade technologies have evolved drastically, there have been so many changes happening in the technology space but one thing constant is human testers’ interaction with them and how we use them for our needs. The same holds true for AI as well. Secondly, to train the AI, we need good data combinations (which we call a training dataset). So to work with modern software we need to choose this training dataset carefully as the AI starts learning from this and starts creating relationships based on what we give to it. Also, it is important to monitor how the AI is learning as we give different training datasets. This is going to be vital to how the software is going to be tested as well. We would still need human involvement in training the AI. Finally, it is important to ensure while working with AI the security, privacy and ethical aspects of the software are not compromised. All these factors contribute to better testability of the software. We need humans for this too.

In summary, we will continue to do exploratory testing manually but will use AI to automate processes while we do this exploration. It is just like automation tools which do not replace manual testing but complement it. So, contrary to popular belief, the outlook is not all ‘doom-and-gloom;’ being a real, live human does have its advantages. For instance, human testers can improvise and test without written specifications, differentiate clarity from confusion, and sense when the ‘look and feel’ of an on-screen component is ‘off’ or wrong. Complete replacement of manual testers will only happen when AI exceeds those unique qualities of human intellect. There are a myriad of areas that will require in-depth testing to ensure safety, security, and accuracy of all the data-driven technology and apps being created on a daily basis. In this regard, utilizing AI for software testing is still in its infancy with the potential for monumental impact.

Will intelligence machines take over the world?
@Raj: Hollywood movies do have an influence on our lives don’t they 🙂 ? At most of the conferences I speak at, there is this weird notion that, in 3 years, AI powered robots are going to take over the world and we will become slaves to them. Which sounds interesting on paper but in reality I don’t think that is going to be the case.

Currently there are are some section of the people who believe fully developed AI that can react and think like humans , will be developed by 2055 and there are are other sections of people who think it will take several hundred years for that to happen. No one knows the exact answer yet. That being said, there are several organizations trying to ensure the AI currently being developed is safe for the human society. For example – The future of life institute was formed for the exact same purpose and has the brightest minds in the AI field working in that group on AI safety research. We also have groups like the World Economic forum keeping a close eye on the impact of AI on society.

So, I do not think machines will take over the world,  just yet!!! 🙂

AI Resources

Courses

And there are more courses available online. Just google search for “Deep Learning courses”, “Machine Learning courses” as keywords.

 

Free Resources/Courses

 

Books

Introduction

We work hard to improve the functionality and usability of our autonomous testing platform to support your software quality initiatives. This month we’re thrilled to release a few of your most requested features; Shared Group Indicator, Numbered Test Steps, New Base URL Parameter. Check them out and let us know what you think.

Shared Group Indicator

What is it?

When trying to change a Shared step the users will now get a notification that they are editing a shared step. Further clicking on “See affected tests” takes the user to the list of tests that are using the shared step.

 

 

Why should I care?

You no longer have to worry about someone changing a shared step unknowingly, as you now see the shared group indicator letting you know the effects of a change before it is done. This is useful when teams are collaborating to build test suites and when multiple people are working on the same set of tests. Now individuals have more visibility to how their changes might impact overall testing.

 

Base URL as a Parameter

What is it?

Users now have the ability to access the base url through a variable within your custom actions. The new variable that automatically stores the url value is named BASE_URLLearn More

Why should I care?

You no longer have to add extra code to get the url value of the web page used in the test. Instead, you just use the BASE_URL parameter and perform any actions necessary inside our custom actions. For example – If we want to print out the url of the web page to ensure the same page is still displayed after certain number of validations, you could just say

console.log(“The current base url is” + BASE_URL)

 

Numbered Test Steps

What is it?

Step numbers help to uniquely identify each step in a test. You now have the step number displayed next to the name of each and every step that is added to your test.

Why should I care?

Having numbered steps help to easily refer to a particular step in a test. This is helpful in cases where you want

  • To edit a particular step
  • To collaboratively work on a particular step of a test with other team members
  • To talk to our support team to debug a particular step in a test

Introduction

We work hard to improve the functionality and usability of our autonomous testing platform to support your software quality initiatives. This month we’re thrilled to release a few of your most requested features; Result Labels, Test Run Navigation Icon, Grid Management. Check them out and let us know what you think.

Result Labels

What is it?

The “Result Labels” allows you to name each remote run. On the “Suite Runs” and “Test runs” pages, you can easily filter your runs by choosing a result label.

Testim Result Labels

Why should I care?  

You now have the ability to label your runs. This is especially useful when you need to drill down into specific runs based on environment, application version, sprint numbers etc. For example you can label your runs as “nightly-scheduler”, “v1.42.34”, “Jenkins”, “Troubleshooting”, “Staging”.

Result labels can be added to the CLI using the parameter –result-label “<user defined name of the run>”. Learn more

Test Run Navigation Icon

What is it?

The new navigation icon opens the results of a test in a new tab.

Testim test run navigation

Why should I care?

You now have  the ability to switch back and forth between test and the test runs via the tabs.

Grid Management

What is it?

To run your tests remotely, you need to integrate either with Testim grid, your own local grid or other 3rd party grids like Sauce Labs and Browserstack. Learn more

testim grid management

Why should I care?  

Grid management now offers the ability to easily manage multiple grids providing an abstraction layer for your devops. The grid information is automatically added to the CLI based on the already configured grids and will appear in this format –grid “<grid name>”.

Customers have access to these new features now. Check it out and let us know what you think. If you’re not a customer, sign up for a free trial to experience autonomous testing. We’d love to hear what you think of the new features. Please share your thoughts on Twitter, LinkedIn or Facebook.

Introduction

We work hard to improve the functionality and usability of our autonomous testing platform to support your software quality initiatives. This month we’re thrilled to release a few of your most requested features; Hidden Parameters, Data Driven testing  via config files and Element Text condition. Check them out and let us know what you think.

Hidden Parameters

What is it?

When you use parameters in your tests, the values that are passed in during run time are saved and shown in the UI. Sometimes this information is sensitive and you may want the value to be hidden. This is now possible using the hidden parameters option available in the project settings page of the Testim editor. Learn more

Why should I care?  

You no longer have to worry about revealing sensitive information in your tests. This is especially true if your application is related to banking, security, insurance or any other domain that handles a large amount of sensitive data.

Data Driven testing now supports CSV, database and other external sources

What is it?

Now users have the ability to pass data sets at run time via config files. The newly added “overrideTestData” parameter in the beforeSuite hook will allow users to pass in multiple parameters to multiple tests at the same time. The same parameter can also be used to extract data from external sources such as CSV, Databases etc.

Why should I care?  

Data Driven testing is no longer just restricted to passing a json file within the tests. Now, you have the flexibility to pass this data at run time through a single config file. Also, you can extract data from external sources and use it within your tests. Everything happens automatically for you during run time. This makes test data setup much more extensible and reusable. Learn more

Talking about working with excel;  we already have detailed documentation of an alternate way to import excel data into Testim. You can learn more about here.

Element Text condition

What is it?

Testim provides several predefined conditions (“if statements”) to be used with steps.  For example, whether an element is visible or not. We just introduced a new condition which checks whether an element has a specific text. Just pass in a string, regex, or a js statement (you can use variables too!).

Why should I care?
Now you have the flexibility to add conditions based on element text instead of just checking for element being visible on the screen. Learn more

 

Customers have access to these new features now. Check it out and let us know what you think. If you’re not a customer, sign up for a free trial to experience autonomous testing. We’d love to hear what you think of the new features. Please share your thoughts on Twitter, LinkedIn or Facebook.

 

One of the most important factors related to automated tests is Maintenance. A lot of effort is spent on maintaining the tests than writing actual tests.  A recent study suggested about 30% of testers time is spent on maintenance.This leads to wastage of valuable time and effort by the resources, which they could have rather spent on testing the actual application.

Imagine a world where the software can maintain tests without human interaction? This world has become a reality with Testim.io. We use Artificial Intelligence (AI) underneath the hood, which provides self-healing maintenance i.e problems are detected by the AI and automatically fixed without human intervention.

Testim.io also help to speed up the maintenance of tests by providing the follow features within our platform-

  1. Version Control

At any given time, it is important to have logs of what changes were made to a particular test. This way we can always revert back to an older version of test as and when required. Our platform provides this functionality by showing all the version history by going to the Properties panel of the setup step and clicking on “See old revisions”

  1. Branching

At Testim.io, we firmly believe in the “Shift Left Paradigm” where Development and Testing must start in parallel as early as possible in the software development lifecycle. Keeping this in mind, we provide the functionality to teams to create separate branches for each team member and work on the same projects and tests. This way, no one can overwrite the changes of the other team members and teams can work on the same code base at any instant of time

In our platform, we just need to select “Fork” to create a new branch and we can also switch between existing branches


        3.  Scheduler

Users have the option of scheduling their tests. This helps to run the tests automatically at a certain day and time without any manual intervention. We can also get notified via email in case of any errors

 

Troubleshooting

As testers, we spend considerable amount of time troubleshooting issues. To help in troubleshooting, our platform offers different options to the user to narrow down the scope of the problem. These options  are as follows-

  1. Screenshots

The screenshot feature explained in the “Authoring and Execution” section helps users to know what was the baseline image and what was the actual image found.

  1.   Properties Panel

The properties panel helps to capture the error messages and display it to the user. The user also has the option of interacting with DOM and see what objects were extracted during the run

  1. Test Logs

Logs are a rich source of information on what happened underneath the UI. We provide test logs when the user runs the tests on our grid or a 3rd party grid. The option can be found in the in top section of editor

  1. Bug Reporting

One of the most time consuming aspects of testing is after finding a bug, we need to report it to the developer with relevant information, to speed up the troubleshooting and fixing of issues.

With Testim.io you can do this with a single click with the help of our chrome extension. All the details related to the bug are automatically generated for you.

  1. Documentation

We put in a lot of effort to document most of the features of the tool in our User Documentation found under the “Educate” tab.

We also have detailed videos on how to troubleshoot your tests quickly

Troubleshooting Part 1- Element is not visible

Troubleshooting Part 2 – Element not found

Troubleshooting Part 3 – Timing issues

Troubleshooting Part 4 – Issues related to mouse hover

With the above features, Testim.io helps to create stable tests that are highly maintainable.

The below posts gives more in depth analysis of Testim in terms of different features that make authoring and execution of tests really simple and how to create reusable components that improves the extensibility of the tool

https://blog.testim.io/bringing-simplicity-to-authoring-and-execution-of-automated-tests/

https://blog.testim.io/how-to-make-reusable-and-extensible-code-using-testim-io/

 

Artificial Intelligence (AI) and machine learning (ML) are advancing at a rapid pace. Companies like Apple, Tesla, Google, Amazon, Facebook and others have started investing more into AI to solve different technological problems in the areas of healthcare, autonomous cars, search engines, predictive modeling and much more. Applying AI is real. It’s coming fast. It’s going to affect every business, no matter how big or small. This being the case how are we as Testers going to adapt to this change and embrace AI? Here is the summary of different things you need to know about using AI in software testing.

Let’s summarize how the testing practice has evolved over the last 4 decades

  • In the 1980’s majority of software development was waterfall and testing was manual
  • In the 1990’s, we had bulky automation tools which were super expensive, unstable and had really primitive functionality. During the same time, there were different development approaches being experimented like Scrum, XP, RAD (Rapid Application Development)
  • From 2000, the era of open source frameworks began
    • People wanted to share their knowledge with the community
    • Started encouraging innovation and asking community of like minded people to help out in improving testing
    • Agile became a big thing, XP, Scrum, Kanban became a standard process in the SDLC
    • There were need for faster release cycles as people wanted more software features delivered faster
  • In the 2010’s, it was all about scale, how to write tests fast and find bugs faster
    • Crowdtesting started
      • Encouraging other people to give feedback on the application. Free and Paid services
    • Cloud testing started
      • People started realizing they need more
        • Server space
        • Faster processing
      • Started to realize the problem of maintenance. How expensive it is to buy hardware and software for maintaining your tests
      • Then we have
        • DevOps
        • Continuous Testing
        • CI/CD integration
  • I believe the Future will be about Autonomous Testing using Machine Learning and AI

 

Basics of AI

Let’s start by de-mystifying some of the terminologies related to AI

  • Artificial Intelligence (AI) is an area of computer science that emphasizes the creation of intelligent machines that work and react like humans
  • Machine Learning (ML) evolved from the study of pattern recognition and computational learning theory (studying design and analysis of ML algorithms) in AI. It is a field of study that gives computers ability to learn without being explicitly programmed
  • Deep Learning(DL) is one of the many approaches to ML. Other approaches include decision tree learning, inductive logic programming, clustering and Bayesian networks. It is based on neural networks in the human body. Each neuron keeps learning and interconnects with other neurons to perform different actions based on different responses

 

There are 3 types of widely used ML algorithms

  • Supervised Learning – We are giving the right training data (input/output combinations) for the algorithm to learn
    • Examples
      • Give bunch of e-mails and identify spam e-mails
      • Extracting text from audio
      • Fill out a loan application and find the probability of the user repaying the loan
      • How to make user click on ads by learning their behavior
      • Recommendation engines on Amazon, Netflix where customer is recommended products and movies
      • Amazon uses AI for logistics
      • Car Optimization
      • Autonomous cars
  • Un-supervised learning – We give a bunch of data and see what we can find
    • Examples
      • Taking a single image and creating a 3D model
      • Market Segmentation
  • Reinforced learning – Based on concept of reward function. Rewarding Good/Bad behavior and making the algorithm learn from it. E.g. Training a Dog

 

Real life AI applications to visually see how it works

  • Quick Draw from Google
  • Weka is an open source project where they are using ML algorithms for data mining

 

What challenges can AI solve?

Let’s discuss the challenges the industry faced while transitioning to agile and what’s still remains a challenge:

How can we use AI to solve testing problems?

There are many companies taking multiple approaches to solve different problems related to software testing and automation. Testim.io is one such company

Testim.io uses Dynamic Locators, The Artificial Intelligence (AI) underneath the platform in real time, analyzes all the DOM objects of a page and extracts the objects and its properties. Finally, the AI decides the best location strategy to locate a particular element based on this analysis. Due to this, even if a developer changes the attribute of an element, the test still continues to run and this leads to more stable tests. As a result, the authoring and execution of automated tests are much faster and more stable.

 

Here is the detailed insight of how our AI works – https://www.softwaretestinghelp.com/testim-io-tool-tutorial/

One of the good practices of writing automated tests is creating reusable components that can be used in different parts of our test suite.

Why is this important?

Creating reusable components is important because it

  • Helps to increase the readability of the automated tests
  • Saves effort by not repeating the same set of steps in different parts of the tests
  • Any changes to the reusable step needs to be done only in one place and it is reflected throughout the tests, across different projects
  • Makes the automated tests more extensible

 

Testim.io helps to ensure Reusability by “Grouping” and “Parameterization”.

  • Grouping

Any number of related steps can be grouped into one reusable component.

For Example – The “Login” scenario is one of the most commonly used steps in any application. The way we can create a reusable “Login” step would be to select the steps we want to group together and clicking on “Add new Group” as shown below

  1. Parameterization

Our platform gives the option of testing application through various input combinations via Parameterization.

This can be achieved in various ways. One way to do this is to give all the input parameters we would need to test the application in the form of a JSON file in the Setup step (The first step of our tests) as shown below

Then add the variable names used in the json file in the appropriate fields of the step as show below

 

Another important aspect of automation is building your tests such that it is extensible.

Why is this important?

As the product and teams grow, there will be need to test more complex functionalities which would require building upon already existing tests. This being the case, the automation suites need to be simple, understandable and should be easy to add more tests to already existing test suites with low coupling and high cohesion.

Testim.io gives the flexibility for organizations to extend the functionalities of our platform using JavaScript and HTML. This way, any functionality our platform does not handle; the user can write their own code to build a robust automation framework

For Example – Say we want to validate the “Select Destination” button from our previous examples. The way to do this would be.

  • Click on “Add custom action”
  • Give a name to the New Step and click on “Confirm”
  • Click on “PARAMS” and Select “HTML” for this example
  • Add Custom Code

The new step with Custom Code gets added to the list of already existing steps

The above features help to make the automation suite more reusable and extensible.

The below posts gives more in depth analysis of Testim in terms of different features that make authoring and execution of tests really simple and easy to maintain

https://blog.testim.io/bringing-simplicity-to-authoring-and-execution-of-automated-tests/

https://blog.testim.io/maintenance-of-tests-made-easy-with-testim-io/

Authoring and Execution of tests is an important aspect of test automation. Tests should be simple to write, understand and execute across projects. The automation framework or tool chosen should give the flexibility to record and playback tests as well as, write custom code to extend the functionalities of the automation framework.

This is where Testim.io can help you out. We follow a  Hybrid Approach and make authoring and execution of tests really simple in such a way that both technical and non-technical members can collaborate and start writing automated tests quickly. This is achieved with the use of “Dynamic Locators”.

What are Dynamic Locators?

The Artificial Intelligence (AI) underneath the platform in real time, analyzes all the DOM objects of a page and extracts the objects and its properties. Finally, the AI decides the best location strategy to locate a particular element based on this analysis. Due to this, even if a developer changes the attribute of an element, the test still continues to run and this leads to more stable tests. As a result, the authoring and execution of automated tests are much faster and more stable.

As we can notice from the above image, the AI parses through all the DOM objects, lists them in the Properties Panel along with the rankings of each and every location strategy for that particular element. In this way, even if the attribute of an element changes, the AI can use a different location strategy from the already parsed list of DOM objects.

Thus, the user does not have to worry about flaky tests.

Some of the basic authoring and execution features Testim.io provides to its customers, are explained below.

  1. How to create a test

We create a new Test by clicking on “Create New” or “New Test”

 

  1. Recording and Saving a test

Once we click the “Record” button, we can record different user actions in our application. After recording different actions, click on “Stop Recording” button to finish recording our tests. Use the “Save” button to save the tests.

 

  1. Validations and Assertions

Our platform helps to make validation of different attributes of an element and API’s really simple. We provide various options for users such as

  • Adding Custom Validations using JavaScript and HTML
  • Validate element visibility
  • Validate element text
  • Pixel level validation
  • API level validation

 

  1. Screenshots

While each test is recorded, the platform takes screenshots of all the Pass and Failed results of each and every step. As a result, users find it easier to troubleshoot problems and understand what happens underneath the hood.

 

  1. Feedback on each step

The user also gets feedback on each step in terms of whether the tests Passed or Failed by showing a “Green” or “Red icon” on the top left portion of each step as shown below

 

  1. Labeling tests

Testim.io provides the feature to label each and every test a user creates. There are 2 reasons why we may want to label a test

  • Helps to identify the reason the test was created in the first place
  • Helps to run tests with the same label all at once through our CLI feature

The way we create labels is by clicking on the “Label” button and either select an existing a label or create a new one.

 

  1. User Documentation

At Testim.io, we took the effort to provide users with all the documentation they will need to use different features of our platform. Most of the answers about using our platform can be found by clicking on the “Educate” tab and Visiting our documentation site as shown below

With the above features, Testim.io helps to make the authoring and execution of tests really fast and simple for our users. Within a matter of seconds a user can record, replay and save the tests. This is surprisingly one of the most overlooked aspects of test automation and our platform takes care of it for our users.

The below posts gives more in depth analysis of Testim in terms of creating reusable components, extensibility and maintenance

https://blog.testim.io/how-to-make-reusable-and-extensible-code-using-testim-io/

https://blog.testim.io/maintenance-of-tests-made-easy-with-testim-io/

 

When we hear the phrase “Record and Playback”, a majority of the people cringe with fear and skepticism as they relate it to primitive, unstable and flaky tests. Organizations have viewed it as a sign of vulnerability in automation and have continued to discourage teams from doing record and playback tests for many years now. The main reason for this is, it leads to-

  • Higher maintenance of tests
  • Lesser stability of tests as it breaks if any element changes
  • Unclear test coverage
  • Tests are highly coupled

This is not a new phenomenon, this has been the case for the past 20 years within which the state of automation has evolved by leaps and bounds.

I am not going to refute the above points as it is true in some cases but people fail to realize there is a time and place for everything which includes “record and playback tests”. These type of tests are valuable to-

  • Do fuzz testing (a.k.a monkey testing), which involves recording large amounts of random data through vast number of valid and invalid actions/assertions and observe the application under test. This helps to uncover issues like memory leaks, unexpected crashes and helps to evaluate the system under extreme conditions that otherwise may be hard to do with normal structured automated tests.

Monkey Testing

  • Perform automated exploratory testing, where the user tries out multiple scenarios and records multiple actions while simultaneously learning about the application and the tool used for automated testing.

a b testing

  • Help in load testing, by quickly recording a bunch of tests and simulate thousands of users concurrently performing the same set of recorded actions on the application.

load testing

  • It helps to get the whole team involved in test automation irrespective of their skillsets.

Now, you may think, “Why am I highlighting the advantages and disadvantages of Record and Playback tests”? The answer is, we at Testim.io, recognized these factors and came up with a hybrid approach to solve the problems with record and playback, by building a platform based on Artificial Intelligence (AI).

Testim.io follows a hybrid approach where we give organizations and users the ability to record and playback tests; while at the same time giving the users flexibility to programmatically manipulate these recorded tests. These tasks can be performed easily using inbuilt functionalities of the platform. It also gives teams the freedom to add their own wrappers around the platform (if needed) by using Javascript and HTML.

hybrid testing approach

To increase stability of tests irrespective of the way the tests are written, Testim.io uses Dynamic Locators. The AI underneath the platform in real time, analyzes all the DOM objects of a page and extracts the object trees and its properties. Finally, it decides on the best location strategy to locate a particular element based on this analysis. The more tests you run, the smarter the AI becomes in increasing the stability of the automated tests. So, even if your strategy for automation is only record and playback, re-running the recorded tests multiple times helps to make those tests stable even if the attribute of an element changes in the future. As a result, the authoring and execution of tests are much faster.

In summary, there are various approaches to test automation. Each approach has its own merits/demerits. Understanding and using the approach that makes more sense in the context of the project is crucial to help in better testing using automation tools, platforms and frameworks. As these options continue to mature, it will become all the more important to follow the hybrid approach to cater to different type of skill sets, needs and expectations of teams and organizations. Hybrid approach to testing is the new era of test automation.

Curious to see how we implement the hybrid approach? Sign up for our free trial.

Be smart & save time...
Simply automate.

Get Started Free