Tag

Test automation

Browsing

Testing has been in the spotlight of software development, as organizations continually seek to optimize development cycles. Continuous Delivery and its promises of frequent releases, even as often as hourly, is a big factor driving executives  to find ways  to shave time off of any eligible technical processes. As enterprises embark  on their DevOps transformational journeys, delivering at lightning speed without compromising on quality takes more than cultural changes and process improvements. Test automation is key in helping project teams write and manage the smallest number of test conditions, but get the maximum coverage.

The case for automation: If the scenario is highly quantitative, technology is superior to humans. That’s clearly the case in the current technology landscape, where according to the 2015 OpenSignal report, massive fragmentation has resulted in over 24,000 distinct Android devices that all run on different versions of the Android OS, plus the several variations of iOS devices currently in use. Web is no different. It’s impossible to test every scenario manually, so automation must be leveraged. But therein lies the main point, that automated testing is a tool to be leveraged as needed instead of relied upon to dictate an entire testing strategy.Although logic would deem that if automating a few test scenarios yield fast results, that automating any and every eligible scenario will shorten the test cycle even more; but alas,  that’s usually not how it goes. Automation endeavors take a good deal of planning and forethought in order to add value at the anticipated level. More often than not the overhead of maintenance, especially in CI/CD environments where tests are automatically triggered and one needs to analyze the reports and fix locators, a task that , could  take hours.  This won’t work for organizations who are truly DevOps.  If full stack developers are going to be fully responsible to take the code to production, then they will need tools to automate the process through infrastructure, testing, deployment and monitoring.  This continuous framework enables  weekly , daily and hourly releases.  Leading DevOps organizations like Netflix and Amazon are said to deploy hundreds to thousands times per day.

What’s more, studies reveal that a high percentage of projects utilizing automated testing fall short of the anticipated ROI or fail altogether. These transformations fall short due to their duration, ramp up time, skill-set gap and maintenance overhead.  If the benefits of automated testing aren’t significant enough to mitigate risk then speedy releases could become more of a liability than an asset.

There are varying levels of automation just as there are companies of all different shapes and sizes. Instead of one or the other, it’s more fitting that automated and manual testing are looked at as complementary. Agile and DevOps have created new testing challenges for QA professionals who must meet the requirements for rapid delivery cycles and automated delivery pipelines.

DEMANDS OF CONTINUOUS TESTING

Testing used to have the luxury of having its own phase or at least a set timeframe to stabilize new code prior to pushing to production. No longer. In the era of DevOps, QA must adopt a truly agile mindset and continually be prepared to shift focus to testing new features and functionality as they become available – that could be weekly, daily, or hourly.

From web, mobile and connected devices to IoT, a quick inventory of current technology will reveal a multitude of different devices and literally thousands of potential ways that these technologies can be combined. Across networks, apps, operating systems, browsers and API libraries, the number of  combinations with each requiring its own verification and testing support.  As the push towards Devops continues, the upward trend towards continuous testing is inevitable. As a result, testers and/or developers will need to deliver in brief and rapid iterations.

Beyond automation, what does this look like? QA, Engineering, or whomever is responsible for testing will need to shift left and engage in rounding out user story acceptance criteria to ensure accuracy and completion. In addition, active participation in sprint plannings will help to ensure that user stories are truly decoupled and independent. Or if dependencies are necessary, confirming that story precursors and relationships are in-place. Partnership with Product Owners to confirm that the scope for a given sprint or iteration is realistic.

And finally in support of shifting left, collaboration among Dev and Ops to ensure the necessary support in the event that a code check-in causes a bunch of automated tests to fail. The culprit for these tests will need to be checked manually so it’s important that QA is prepared so as not to pose a blocker for the release. All of these activities is largely what comprises testing outside of automation. It’s a broader role than the traditional QA Specialist that requires initiative and the willingness to embrace an always-on mentality that’s necessary to support DevOps efforts.

SUMMARY

Software development practices have changed and the change brings with it new challenges for the testing arena. The growing need for test automation in addition to the profusion of new testing scenarios will constantly evolve the role of testing and QA.  Yes, the world is changing for testers—in a good way. The days of exhaustive manual testing are certainly numbered, and automated testing is an essential part of the delivery pipeline. But testing has never been more important.

SOURCES

  1. IT Revoloution DevOps Guide
  2. World Quality Report 2016-17
  3. Open Signal: Android Fragmentation

 

Introduction
We work hard to improve the functionality and usability of our platform, constantly adding new features. Our sprints are weekly with minor updates being released sometimes every day, so a lot is added over the course of a month. We share updates by email and social media but wanted to provide a recap of the latest enhancements, summarizing the big and small things we delivered to improve your success with the product. We’d love to hear your feedback.

Test History

Is past performance an indicator of future success? The folks on Wall Street don’t think so… But we do! This is why we are pleased to offer the new Test History feature in Testim.  


The new Test History feature allows users to see slide and dice test results to get more actionable insights.

Why should I care?
This gives users much more insight from the test results. Quickly analyze how many tests ran during the selected time frame, how many succeeded and the average duration of each run, how your tests performs across multiple environments and whether certain environments or scenarios consistently fail. This allows project team to see improvement trends across scenarios or environments over time. Click here to learn more.

What’s included?
Test History allows you to filter based on the following parameters:

  • Run time
  • Specific tests / All tests
  • Last X runs
  • Status
  • Browser

Create Dependencies Between Tests
This new capability allows users to create different aspects of dependencies between tests. As a best practice, we recommend to keep the tests as isolated as possible. However we recognize that sometimes you need the ability to create dependencies between tests.

Why should I care?
Now users can create a logical sequence of tests. By working closely with our customers, we’ve seen project teams required to set a sequence of activities. A general testing framework does not allow you to create a sequence, forcing users to create one long test which may result in performance issues and failures. By creating dependencies during test plans you can create shorter discrete actions, order them in sequence and set dependencies and share data between tests. Click here to learn more.

Setting up Cookies
Cookies is a reserved parameter name for specifying a set of cookies you would like Testim to set for your application.  You can set cookies for each browser session before the test is started or before the page is even loaded. Use cookies to set browser data to be available for your tests. The cookies parameter is set as an array, where each cookie object needs to contain a name and value properties.

Why should I care?
Websites use cookies to personalize user experience. By using this feature, you can test different types of personalized experiences. Click here to learn more.

Improved Scrolling
More flexible way of scrolling through your page.

Why should I care?
Traditionally scrolling in your test automation would be set by pixels so you would test your script to skip 60 pixels. Offering a more flexible scrolling ability makes your tests more adaptive. An  example is to scroll to the element, mouse wheel. Click here to learn more.

How do I get my hands on these hot new features?
Customers have access to these feature now, check it out and let us know what you think. If you’re not a customer, test drive the new features by signing up for a free trial to experience code and maintenance free test automation.

Testing is one of the key processes in software development. As the requirements to support multiple browsers and devices increases, it is driving the demand to automate the functional testing of applications. While teams embark on their DevOps transformational journeys, the handoffs between traditional application lifecycle silos begin to dismantle, enabling a continuous process of updating their applications.

The key challenges for software teams in DevOps include:

  • how to achieve maximum coverage testing with shorter time frames?
  • how to figure out what to automate vs. leave manual?
  • how to transform from testing and validation before go live to more predictive forms of testing and real-time quality monitoring?

With the increased adoption of agile methodologies, test automation is a key piece of continuous testing. According to the 2015-2016 World Quality Report, research show that the average percentage of test case automation has increased from 28% to 45%. An enabler of this trend has been the advancements in development and testing tools which have helped organizations build a solid infrastructure for automation.

Testim’s stable, self-healing, end-to-end test automation solution is recognized by Software Testing Help as a Top 20 Best Tool to help developers and/or testers create their test suites without any coding, script management or maintenance of the tests. Using machine learning, Testim provides the maximum coverage with the fewest amount of test scenarios allowing teams to optimize their test suite. This also eliminates the need of trying to figure out what to automate vs. leave manual since tests are automatically created as the user uses the application. Additionally, Testim helps teams transform from testing and validation by leveraging historical data to prioritize tests.

“We are humbled by making this list,” says Oren Rubin, CEO for Testim. “This is a testament to our commitment to helping project teams eliminate the complexity of building their test automation suite.”

About SoftwaretestingHelp.com
STH is one of the most popular blogs focusing on Software Testing and Quality Assurance topics. This blog is growing up so fast and currently, we have around thousands of testing professionals who visit every day and gain help from this blog. The topics which we cover on this blog include – software testing tutorials, methodologies, manual testing, automation testing, testing tools, interview questions, web testing, testing templates, quality assurance, testing certifications, books, career guidance, job openings, latest testing trends, news, and much more that cannot be listed here on a single page.

About Testim.io
Testim.io leverages machine learning for the authoring, execution and maintenance of automated test cases. We use dynamic locators and learn with every execution. The outcome is super fast authoring and stable tests that learn, thus eliminating the need to continually maintain tests with every code change. Netapp, Verizon Wireless, Wix.com and others run over 300,000 tests using Testim.io every month.

Unit Tests are in the house!
I’m a big advocate for TDD (Test Driven Development). Research shows TDD has a high correlation to quality code, as it prevents bugs from forming in the first place. Also, the cost of authoring tests and fixing bugs is lower

This is true for the back-end. As for the front-end, although MVC and MVVM based frameworks are doing a great job providing easier tools writing unit testing, we still see low adoption. Some claim it stems from frontend comprising mostly glue-code, and some say it’s harder to write test DOM and event (asynchronous) code.

The need for frontend automation rises
Nowadays, a huge part of any project resides in the frontend. Not only did past challenges remain, the plot thickens as we have more environments (e.g. mobile and tablet), more Operating Systems (Window & Mac are both popular), and more browsers to support (IE11, Chrome, Safari, Firefox, Opera, Edge, not to mention old versions of IE.. may it rest in peace already).

Percentage of core logic – backend vs. frontend

Since more R&D teams shifted to agile, QA has become the bottleneck. A big part of being agile is the frequent releases. When a six month project comprises a three week testing cycle it’s one thing, but when a team wants to deploy weekly or even daily without bugs, QA is challenged with shrinking that to a cycle that takes 1-2 hours. The feedback to the developer should be instantaneous, as research show that longer the feedback cycle is, the longer it takes to fix the problem.

Why testing frontend is difficult
With the move to frontend, high maintenance overhead resurfaced. The maintenance for frontend test automation was as high as 50% twenty years ago. It has only improved marginally in recent years and is still as high as 30%.

To mimic a user interaction, you must first locate the element you wish to act upon. There are two major approaches to locate an element:

  • Visual based – look at the pixels
    Frameworks such as Sikuli use that technique, but it’s super fragile, as render per OS/browser/display-adapter will generate different pixels. Some display adapters are non-deterministic in nature. To have stable tests you must be a computer vision guru.
  • Object based – look at the object hierarchy (DOM).
    Frameworks such as Selenium spread like fire. As opposed to twenty years ago, better software design pattern emerged (such as the Page Object) which helped separate the tests’ business logic from the implementation, and urged reuse.

Nonetheless, both approaches result in flaky and unstable tests. They require high maintenance. or every UI (code) change, a test change is required as well (as both rely on the UI). It’s easier for those practicing the shift left paradigm, where quality is inherent in the development cycle, as finding the cause is easier, but they suffer from the same amount of failures. For those with a QA team, it’s still catastrophic. As almost every morning starts with understanding why tests fail and whether it’s a bug or merely a code change.

Crowdsourcing

Crowdsourcing

For End-to-End testing, some companies turn to crowdsourcing to alleviate the pain. This works great for regression tests which require no prior knowledge of the app. The required ramp up is just to write down the instructions in plain English. The big advantage is the low cost of people willing to click all day long. The problem arises when one tries to implement the shift left paradigm. This requires testing not only in the main branch, but also in all dozens of feature branches. Not only does this makes it not cost effective, but developers are custom to get feedback in seconds or minutes, not hours.
Image a developer promoting a hotfix to be released to fix a critical bug yet having to wait for someone in the other side of the world to wake up and test the new version.
Communication is also challenging at the developer and tester have no experience working together.
As for exploratory testing, crowdsourcing still great. It allows one to test different devices and different networks from all around the world. The rose of the garden is the seductive business model, paying only for proven bugs, comprising a thorn where every little bug is reported and the devs find themselves trying to prove that some of them are not real bugs. Until today, crowdsourcing does not provide a sustainable solution when shifting to continuous delivery.

Today I joined Testim.io as President and COO. Testim has the potential to disrupt a massive, $8B+ market for web and mobile QA automation. Testim has built an insanely powerful product that seeks to alleviate the pain associated with test automation — current solutions suck. Testim has achieved product market fit (you might want to read more about that below), has a great team and culture spanning our Israel and San Francisco offices, and needed a business leader to help scale the company to the next level.

How I met Oren Rubin

Throughout the years, I’ve seen hundreds of customers struggle to transform agile. QA was the main bottleneck, specifically the creation and maintenance of automated test cases. I’ve seen organizations fire managers and vendors as a result of failure to meet automation goals. Scripting full test suites took months, during which organizations would depend on one or two skilled automation leads. This had impact on time and money. This is a problem worth solving.

Several months ago, a friend told me about an interesting tool they were assessing. He suggested that I have a call with Oren, the founder and CEO. A few weeks later, I was driving on 101, when Oren called me, The next 45 minutes were invigorating. His description of the need was music to my ears. The way he described Testim’s approach to the maintenance of test scripts intrigued me.

A few weeks later I stopped by Heavybit, a San Francisco based accelerator for developer related startups, to meet Oren. We immediately hit it off. He showed me a demo and I loved it! It was exactly the type of tool that was required to help companies automate release cycles generally and quality assurance specifically. It was cloud based, required little scripting and was visual and easy to use. And developers loved it! It took 3 minutes to create a test. I started firing questions at Oren: about Testim’s product roadmap, current customers, personas, integration with other devops tools, etc. Over the following weeks, we had a number of similar discussions.

So why did I join Testim?

Over the last few years, I’ve met hundreds of startups. There are a few ingredients that make a great startup.

Deep and identifiable Pain Point — “Make sure your product is a pain killer. Not a vitamin.” If you help customers address a great need that is making their job more difficult, they will give you the time of day, and more often than not, will be ready to pay for it. I’ve seen that pain point around Continuous Delivery & Continuous Integration and specifically around test automation: the time to author a single test case, the effort required to maintain these test suites and the challenges around the required expertise. Test automation is a big pain point to a vast majority of the development teams I’ve met.

Superb Team with Domain Expertise — Many investors have articulated the qualities of a great team and more importantly a great founder. It is difficult to define one until you meet one. Oren is that kind of founder. Oren has spent the past two decades living and breathing the problem which Testim is trying to solve. He experienced the challenges of agile transformation and built a product based on his unique insights. He largely bootstrapped the company for the first two years with his own money and that of a few prominent Israel angels. He was able to attract a strong, highly technical team of engineers. Based on his product vision and early customer traction, he was also able to attract a prominent group of Silicon Valley and Israel venture funds. And Oren believed that for the company to realize its full business potential, the corporate HQ would have to be based in Silicon Valley. So he packed up and moved to San Francisco.

Oren also possessed certain founder attributes that I value, including being a good listener, open to feedback, and actually implementing product suggestions based on our discussions. The discussions I had with each person on the team also reinforced my excitement. My discussion with Gil, our VP R&D who leads our Israel development office, lasted two hours, where we covered product roadmap and the rapidly changing market. The meeting with Michael Neril, the Founding Partner of Spider Capital, one of Testim’s investors, was a joy. Michael is an ideal investor in my view: He is constantly looking to add value with introductions to consultants, partners, customer prospects and candidates. We speak frequently and he is a great sounding board on strategy.

Clear Product/Market fit — Over the last couple of weeks, I spent time talking to Testim’s customers as well as folks I know and respect. I remember a specific conversation that happened while waiting in line at a coffee shop. A friend who was also in line for coffee introduced me to his CTO. We somehow got to talk about quality and Testim. “I love the product. It’s awesome. They are onto something.” I heard similar reactions from customers. Just today, we received the following note from a new customer: “This is really a good application you have here. It’s a big help to us QAs.” P.S.: Developers love our product even more!

Scalable Channel to Market — A pain-point, a talented team and a great product are not enough. You need a scalable channel to market. Testim has hundreds of active customers and dozens of paying ones with zero money spent on sales, marketing, or a structured process. That’s why I’m excited to be part of this journey. With a structured sales process, an understanding of the pain points, clear definition of the target persona and organization, well-crafted messaging, and a few additional A players on our team, we have the building blocks to build a truly transformative company in the test automation space.

Breakthrough Technology – Many companies approached the challenge of QA automation by focusing on simplifying authoring using GUI based play and record. Testim’s approach, focusing on stability and reliability first is refreshing. The evolution of real-time machine learning powers the ability to analyze hundreds of elements and thousands of attributes, enabling Testim to deeply understand the Document Object Model (DOM) in order to create dynamic test suites which are adaptive to changes in the application. 

That brings me to my last two comments:

1.If you are looking for a powerful, stable, yet simple automation tool — please reach out.

2.I’m also hiring. If you or someone you know is talented, highly productive, thrives in a fast moving startup, and is a team player — please reach out.

About Testim.io

Testim uses machine learning for the authoring, execution and self-maintenance of automated test suites. Testim learns from every   execution, self-improving the stability of test cases resulting in a test suite that doesn’t break on every code change.

A developer can author test cases in minutes. Testim powers the transition to agile and shift left, giving organizations the tools to maintain maximum coverage over every release. Testim, headquartered in San Francisco, is backed by a leading group of U.S. and Israel venture capital funds.

Be smart & save time...
Simply automate.

Get Started Free