Category

AI

Category

Testim is #1 in innovation among all other competitors for several years now. This can be seen from the new features we have been releasing constantly to make automation faster, more stable and much more collaborative than ever. In continuation with this, we are excited to bring you our next big feature which we are calling Group Context. Imagine you have one or more components (E.g. gallery of images) within a page or across several pages, and you need to easily state on which component to perform the group of actions. Although this is relatively doable in coding (via the Page Object design pattern), this was extremely hard to implement in codeless tools until now with the release of Group Context. Testim is the only codeless platform that can currently support this action

For those of you not familiar with Testim, let’s start by defining “Context”?

“Context” is key in real life and in coding. You provide the necessary information in order to perform an action. For Example – Say you make a reservation at a restaurant for a party of four people, you would be required to provide a name for the reservation. Here the name is “contextual information” and you making a reservation is the “context”. Similarly, in test automation, it is important to know the context of elements from the DOM level in order to perform an action.

A Context in Test Automation

Context is all the more important when you have reusable components in your automation suite. For Example – let’s take the below web page.

 

It is a simple web page, containing a gallery of items. In this case, it has names of places with description and an option to book them as part of a reservation. In each item in the gallery, you have similar elements such as an image, text, and button, this is because the code for generating those instances is the same. So, on a high level, all these gallery items are exactly the same except for the different data showing up for each element in the gallery item.

Say we create a reusable group to:

  1. Validate whether an image and text is present in the gallery item 1 which is “Madan”
  2. Validate whether there is a “Book” button in the gallery item
  3. Click on the “Book” button
  4. Do some validations in the “Checkout” screen

It would look something like this:

 

Now, what if I want to use group on Item 2 of the gallery which is “Shenji” instead of “Madan” (which was the 1st item) ?

 

Typically, we would have to create another reusable group to make it work for gallery item 2, which is time consuming and does not make sense when the whole gallery shares the same DOM structure.

When using code (e.g. Selenium), you can use the Page Object design pattern, and just pass the context in the constructor, either as a locator or a WebElement (see slides 40 and 41 here).

In codeless/low-code platforms, this wasn’t possible until now with the release of our new feature “Group Context”. Now you can maximize reuse by assigning entire groups to different elements of the page and across pages within a single click. Taking the above example, you would not have to reassign all the elements/properties of steps within the group when you want to perform same set of actions, on another item in the gallery, with the exact same DOM structure. This means, we can use the same group on gallery item 2 which is “Shenji” without having to do any rework by just choosing the Context-> Custom option from the properties panel as shown below

 

Use Cases for Group Context

Group Context can be used in many other scenarios, such as:

  • Repeating elements: when you have similar elements repeating in the page and want to execute the same steps on all of them.
  • Table rows: when you want to execute the same steps on different rows in a table.
  • Tabs or frames: when you want to reuse a group of steps recorded on one tab on other tabs in the same or different page

 

Summary

We constantly keep working hard to improve the functionality and usability of our autonomous testing platform to support your software quality initiatives. We strongly believe this feature will benefit teams and help in making automation much smarter. Group Context is currently in beta and we would love your feedback.

Please use our chat support to give us feedback or e-mail us at product@testim.io. Happy Testing!!!

Selenium has been a popular automation testing framework for the past several decades. But, as applications have become more complex in the past several years, especially with the use of popular JavaScript frameworks such as Angular.js, Vue.js, React.js and Ember.js for building web applications; Selenium has found it hard to adapt to these technologies.

For example– If you are a currently using Selenium, have you ever experienced any of the below problems-

  • Spending majority of your valuable testing time fixing flaky tests?
  • Unable to make automation progress due to the lack of skilled programmers to write automated tests?
  • Not finding enough support in the open source community when new libraries and updates break existing tests and you have no idea what to do?
  • Need visual validation when a step fails, to visually understand the exact reason for the failure?
  • Insufficient logging information when your tests fail?
  • Finding it hard to seamlessly integrate your automated tests within your CI/CD pipeline?

If you answered YES, to any of the above questions, then you are not alone!!! According to new Gartner research, “Selenium is the de facto standard for front-end test automation of modern web technologies due to the flexible and powerful browser automation capabilities of WebDriver… It’s a sophisticated tool that isn’t easy to learn. It requires your team to have programming expertise and familiarity with object-oriented languages.” So, Selenium is good if you have the necessary programming expertise in your team, if not it becomes an obstacle.

There is no need to fear as there are good alternatives to Selenium. Below are the top 3 widely used alternatives currently available to give you and your team the flexibility to build robust automation test suites/frameworks.

NOTE:  Selenium is an open source community. People volunteer to make the framework better so the limitations of Selenium is not their fault as it is the responsibility of the user using the framework to extend its functionalities and make it better for his/her own automation needs.

  1. Cucumber

This is an open source testing tool that focuses on BDD (Behavior Driven Development). It emphasizes writing requirements in plain english in the form of Given, When and Then statements. This is commonly referred to as “Gherkin” syntax. You then convert these GWT statements into code using Java, JavaScript, Ruby or Kotlin.  This helps to enforce collaboration and bring more clarity to requirements.

On the flip side, it still needs someone with programming background to write the binding code to convert these GWT statements into usable actions. Also, the GWT format leads to a maintenance nightmare especially when many people collaborate and start making changes to different steps. Finally, it does not have any visual validation and has insufficient logging features making it hard to troubleshoot errors.

  1. SikuliX

This is an open source GUI based automation tool. It uses image recognition powered by OpenCV to automate anything with a UI. This includes desktop and web applications. It supports multiple programming languages including Python, Ruby and JavaScript.

Although this tool has robust UI validation functionalities, it lacks a lot of necessary features that are needed for stable automation such as locating elements based on attributes, creating reusable components and modularizing your tests for easier maintenance.

  1. Testim

Testim uses artificial intelligence (AI) for the authoring and execution of automated tests. The focus is on functional testing, end-to-end testing and UI testing. The Dynamic Location strategy used within the tool sets it year’s apart from its competitors. The Artificial Intelligence (AI) underneath the platform in real time, analyzes all the DOM objects of a page and extracts the objects and its properties. Finally, the AI decides the best location strategy to locate a particular element based on this analysis. Due to this, even if a developer changes the attribute of an element, the test still continues to run and this leads to more stable tests. This is the major advantage of using Testim compared to other frameworks like Selenium which uses static locators. Also, the “self-healing” mechanism of the AI helps to proactively detect problems in your test and fixes it automatically for you. As a result, the authoring and execution of automated tests are much faster and more stable.

Testim is not a completely code-less tool; you can use JavaScript and HTML to write complex programming logic (if needed) for your applications. This helps to make the automation suite more extensible. It provides easy integration with your CI/CD pipeline and most importantly helps to involve the whole team in automation including technical and non-technical people.

In summary, Selenium has really good capabilities but knowing there are other alternatives out there helps to give your more options to make your automation effort more easier and stable.

 

In the current era, where applications are becoming more complex due to the varied functionalities to support growing customer demands, it becomes all the more important to have a sophisticated IDE to write tests and automate these types of applications.

Selenium IDE was once popular with their Firefox and Chrome extensions but as applications became more complex in the past several years, there have been continual problems while using the Selenium IDE, such as-

  • Inability to handle dynamically changing elements on the web page
  • No inbuilt functionalities to automate actions that involve multiple tabs, hovers, scroll and other complex user actions
  • The need to give explicit and implicit waits manually to make tests more stable
  • Flakiness with the inbuilt options of the IDE such as Record/Playback where it seizes to work in some situations for no reason

Knowing these limitations; it is good to be open to other alternatives. Below are the top 3 alternatives that are currently popular among teams and organizations.

NOTE:  Selenium is an open source community. People volunteer to make the framework better so the limitations of Selenium is not their fault as it is the responsibility of the user using the framework to extend its functionalities and make it better for his/her own automation needs.

  1. Unified Functional Testing (UFT)

UFT (originally QTP – Quick Test Professional) was part of Mercury Interactive and was quite popular in the testing community. It was acquired by Hewlett Packard (HP) in 2006, then was renamed to UFT before it got sold to MicroFocus in 2016.

UFT is an automated function testing tool used for functional and regression testing. Using UFT, you can automate different user actions on multiple desktop and mobile browsers with different users and data sets, on various Windows OS versions.The down side of UFT is that, the IDE is built mainly for Windows OS. It has limited support for Mac and no support for Linux OS. Also, the execution of tests is really slow, especially when there is parallelism. So, there seems to be a huge gap between the amount of features offered and the price you pay for the tool given its limitations.

  1.  Protractor

This framework was born as a result of the inabilities of Selenium to handle javascript frameworks such as Angular.js. It is an end to end testing framework to specifically test Angular and Angular JS applications.  It has built in functions to identify different types of elements in these applications. One of the coolest feature of the framework is its support for parallel testing, where you can run several browser instances simultaneously.

Although protractor has good features to do web automation it does have its limitations. It only supports JavaScript and automating Angular.js applications. It does not support mobile apps. Also, the biggest complaint from users is, there is not enough developers in the open source community working to fix the current issues with the framework. As a result. it is really time consuming for teams and organizations to wait on fixes which becomes a big obstacle in automation progress and affects release cycles.

  1. Testim

Testim uses artificial intelligence (AI) for the authoring and execution of automated tests. The focus is on functional testing, end-to-end testing and UI testing.The Dynamic Location strategy  used within the tool sets it year’s apart from its competitors. The Artificial Intelligence (AI) underneath the platform in real time, analyzes all the DOM objects of a page and extracts the objects and its properties. Finally, the AI decides the best location strategy to locate a particular element based on this analysis. Due to this, even if a developer changes the attribute of an element, the test still continues to run and this leads to more stable tests. This is the major advantage of using Testim compared to other frameworks like Selenium which uses static locators. Also, the “self-healing” mechanism of the AI helps to proactively detect problems in your test and fixes it automatically for you. As a result, the authoring and execution of automated tests are much faster and more stable.

Testim is not a completely code-less tool; you can use JavaScript and HTML to write complex programming logic (if needed) for your applications. This helps to make the automation suite more extensible. It provides easy integration with your CI/CD pipeline and most importantly helps to involve the whole team in automation including technical and non-technical people. It supports multiple browsers – Firefox, Chrome, Edge, IE and Safari, multiple OS versions in Windows, Mac and Linux and also supports parallel testing on both web and mobile applications.

In summary, Selenium IDE does have great functionalities but unless like minded developers in the open source community come together and fix the existing issues with the IDE, users will have to look for better alternatives that are highlighted in this post.

We recently hosted a webinar on AI and its influence on test automation with an awesome panel consisting of Jason Arbon, Oren Rubin, Dionny Santiago and me being the moderator. There were lot of great discussions on this topic and we wanted to share it with the community as well.

Below you will find the video recording of the webinar, the list of questions and answers that we couldn’t get to during the webinar and different resources to learn about AI and testing. Please feel free to reach out to me in case of any questions at raj@testim.io or any of the panel members.

Video Recording

 

Q&A

I am a performance engineer and am working on AI for quality gates in load testing results…that needs to be a high priority for the “future” which is “now”.How do you think bots can be used in this area?
@Jason: UI-Bots can help generate user-like load directly via the application.  Though, for most load testing problems, would recommend something like charles proxy, or internal ways to spin up load, and only use the ‘expensive’ UI-based bots to see how the app works E2E for the user under load.

 

With rapid changes in agile requirements, how do we make the machines learn or adapt to the changes every time?
@Jason: The ai bots most folks are working on these days (vendors) will auto discover new features in the app and exercise them.  At test.ai we have a set of 5k+ tests written for common flows in apps, so if you add a new feature to your app that looks like something similar on another app, the bots will auto apply that test case to the new build.

@Raj: The more tests you run, the smarter the AI becomes in detecting changes in the application. It will start detecting changes in application’s UI, element attributes and start adapting the tests automatically to these changes due to its self-healing mechanism. It can identify flaky tests, optimize waits in between steps and also proactively fix issues for us before they occur.

 

With BDD model and shift left and demand for testing at unit and service/api layer where does E2E testing stand?
@Jason: Dionny’s work can help generate valid permutations of existing API test case parameters/flows.  Also, clustering methods can help identify misbehaving servers via logs of activity during api testing or production.

 

Where can we find Dionny’s paper on AI testing, you were talking about?
Dionny’s paper

 

Lot of automation test scripts fail due to test data issues, can we use AI to tackle those kind of issues?
@Jason: Thats a broad category of failure types, but yes, ‘AI’ can be taught to auto associate correct data with the right application states.  Google also shared some ‘test selection’ findings using ML to help decide what to do with all those failing tests:  https://testing.googleblog.com/2018/09/efficacy-presubmit.html

 

I wanted to understand, what does really mean by AI in testing?if it mean by machine will perform testing? if machine will test then if they will be already defined with scenarios which needs to be tested? is it same as automation testing as there also we don’t need manual intervention?
@Jason: Generally, AI in testing, means applying machine learning / AI techniques to test applications.  There is also ‘Testing AI’ which refers to approaches to test AI/ML-based products and features. There are a variety of ways to apply AI to testing, some leverage pre-written test cases and the AI is used to automatically execute the tests, create variations, or analyze the results.  Some AI based systems are trained to mimic general human behavior and can execute basic ‘flow testing’ for many apps, without pre-written test scenarios.  The bots we build at test.ai can read BDD/Scenarios and execute them against a set of applications.  As for need for human intervention, like automation, there is still the need for plenty of human intervention in AI-based testing approaches these days 🙂 Humans gather oracle/training data for the AI. Humans measure the correctness of the ‘AI’, and humans evaluate the significance of the AI-based test results as they relate to business/shipping decision.

@Raj: In addition to what @Jason was saying,  I wanted to mention that, AI can have a positive impact on several facets of software testing especially test automation. There have been so many different tools and frameworks that have come up trying to solve different kinds of problems related to test automation but one problem that has been a constant challenge till date, is the aspect of “maintenance”. One of the main reasons for this is the use of static locators. With static locators, we use only one attribute of an element to uniquely identify it on a page and if this changes, the test breaks and as testers we end up spending a considerable amount of time troubleshooting the problem and fixing it. Based on research, about 30% of testers’ time is consumed in just maintaining tests. Can you imagine the opportunity cost associated with this effort? It is mind blowing. Testers’ time is valuable and better spent on actually exploring the application and providing information to help stakeholders make informed decisions about the product. With AI based testing we can overcome this problem by using dynamic locators. Dynamic Locators is a concept where we use multiple attributes of an element to locate it on on the page instead of a single attribute. This way, even if one attribute changes, the element can still be successfully located with the help of other attributes that have already been extracted from the DOM by the AI.

 

Can you guys elaborate on how do AI-based tests learn acceptance criteria that normally has to be defined by humans?
@Jason: Depends on the AI system being used.  The bots at test.ai execute human-written test cases.  Acceptance tests are written at a very high level of abstraction, and the bots do all the test execution.  Reporting is as normal for test automation.  In summary, just tell the bots what your acceptance criteria are.

 

What an automation tester needs to learn to align with the future of ai in testing ?
@Jason: A good set of links to learn are here:  https://www.linkedin.com/pulse/links-ai-curious-jason-arbon/ . You can also start to leverage/experiment with “AI” via the current testing vendors. If you are already familiar with selenium/appium like testing, there is a new open source API that uses AI for element selection that you can use today:  https://medium.com/testdotai/adding-ai-to-appium-f8db38ea4fac?sk

 

Is AI platform dependent. like desktop application or web/mobile?
@Jason: 
Depends on the AI approach/solution.  Many are platform dependent.  The bots we build at test.ai though are not platform dependent–a key feature.  The bots are platform-independent as the machines are trained to recognize UI elements much like humans do, and humans are platform dependent ;).

 

Is there an Open Source project that allows to apply AI to locate the elements?
@Jason: Yes, for appium today and likely Selenium soon:  https://medium.com/testdotai/adding-ai-to-appium-f8db38ea4fac?sk

 

How can AI be used for improving test coverage ?
@Jason: AI can help generate many more validate test scenarios than a human could create. AI also enabled re-use of test artifacts so a test written for one app, can also execute on a similar app with no human intervention.

@Raj:  Now with AI, you can also connect your production apps to the testing cycle. This means that we can create tests based on actual flows done by the user in production. Also, the AI can observe and find repeated steps and cluster them to make reusable components in your tests. For Example – Login, Logout scenarios. So now we have scenarios that are actually created based on real production data instead of us assuming what the user will do in production. In this way, we also get good test coverage based on real data.

 

Will AI testing replace selenium appium and all tools and technologies?
@Jason: Asymptotically.

 

Is AI really better?
@Dionny: Traditional testing teams focus on either a single app, or a small set of apps; whereas, AI can learn from millions of different examples and apps. The more data we show the AI, the better it gets. Also, the AI never gets tired!

 

What are the immediate benefits of using AI?
@Raj: Apart from the benefits already mentioned in the answers above, AI can also help in increasing team collaboration. The field of test automation has historically been a technical tester focused community. This stigma can also change with AI. What this means is, non-technical resources no longer need to fear code and technology, rather AI will help to bridge the gap between the technical know-how and authoring and execution of tests making life easier for teams.

 

Will our jobs be replaced?
@Raj: Over the past decade technologies have evolved drastically, there have been so many changes happening in the technology space but one thing constant is human testers’ interaction with them and how we use them for our needs. The same holds true for AI as well. Secondly, to train the AI, we need good data combinations (which we call a training dataset). So to work with modern software we need to choose this training dataset carefully as the AI starts learning from this and starts creating relationships based on what we give to it. Also, it is important to monitor how the AI is learning as we give different training datasets. This is going to be vital to how the software is going to be tested as well. We would still need human involvement in training the AI. Finally, it is important to ensure while working with AI the security, privacy and ethical aspects of the software are not compromised. All these factors contribute to better testability of the software. We need humans for this too.

In summary, we will continue to do exploratory testing manually but will use AI to automate processes while we do this exploration. It is just like automation tools which do not replace manual testing but complement it. So, contrary to popular belief, the outlook is not all ‘doom-and-gloom;’ being a real, live human does have its advantages. For instance, human testers can improvise and test without written specifications, differentiate clarity from confusion, and sense when the ‘look and feel’ of an on-screen component is ‘off’ or wrong. Complete replacement of manual testers will only happen when AI exceeds those unique qualities of human intellect. There are a myriad of areas that will require in-depth testing to ensure safety, security, and accuracy of all the data-driven technology and apps being created on a daily basis. In this regard, utilizing AI for software testing is still in its infancy with the potential for monumental impact.

 

Will intelligence machines take over the world?
@Raj: Hollywood movies do have an influence on our lives don’t they 🙂 ? At most of the conferences I speak at, there is this weird notion that, in 3 years, AI powered robots are going to take over the world and we will become slaves to them. Which sounds interesting on paper but in reality I don’t think that is going to be the case.

Currently there are are some section of the people who believe fully developed AI that can react and think like humans , will be developed by 2055 and there are are other sections of people who think it will take several hundred years for that to happen. No one knows the exact answer yet. That being said, there are several organizations trying to ensure the AI currently being developed is safe for the human society. For example – The future of life institute was formed for the exact same purpose and has the brightest minds in the AI field working in that group on AI safety research. We also have groups like the World Economic forum keeping a close eye on the impact of AI on society.

So, I do not think machines will take over the world,  just yet!!! 🙂

 

AI Resources

Courses

And there are more courses available online. Just google search for “Deep Learning courses”, “Machine Learning courses” as keywords.

 

Free Resources/Courses

 

Books

Be smart & save time...
Simply automate.

Get Started Free