Tag

Test automation

Browsing

Should the goals and principles of DevOps be the same for every company? Is it possible for every company to reach CI/CD? 

Software delivery teams come in all different shapes and sizes. Each team has its own DNA that has organically evolved through generations of diverse experiences, skill sets, tools, technologies and sets of processes.

So, if every project team looks entirely different than the next, where do you start?

RESERVE YOUR SEAT for  this roundtable discussion where we will cover different types of organizational situations and the common as well as unique challenges each situation may face in their DevOps transition. We will review ways to address specific inefficiencies in each situations development process as well as how to plan accordingly and minimize risk along the way.

Date: Wednesday, November 15
Time: 9:00am PT

We will cover:

  • How to transition to DevOps and align your development, testing and release processes
  • How to plan for your DevOps transition based on your current operational reality
  • A breakdown of common as well as unique DevOps obstacles
  • A method for systematically resolving your development issues to enable test automation

 RESERVE YOUR SEAT and bring your questions for the panelists:

  • Tanya Kravtsov is the Director of QA at Audible. She is building a new QA organization to support innovative web product development at scale. Previously, as head of automation and continuous delivery at ROKITT, senior QA manager at Syncsort, and VP at Morgan Stanley, Tanya focused on quality, automation, and DevOps practices, working with internal and external customers to transform development and testing processes. Tanya is passionate about process automation, continuous integration, and continuous delivery. She is a founder of the DevOpsQA NJ Meetup group and a frequent speaker at STAREAST, QUEST, and other conferences and events. Follow her on Twitter @DevOpsQA.
  • Bob Crews is the President of Checkpoint Technologies. He is a consultant and trainer with almost three decades of I.T. experience including full life-cycle development involving development, requirements management and software testing. He is also the President of the Tampa Bay Quality Assurance Association. He has consulted and trained for over 200 different organizations in areas such as effectively using automated testing solutions, test planning, implementing automated frameworks and developing practices which ensure the maximum return-on-investment with automated solutions.

We’re excited to announce that Testim has raised $5.6M in Series A funding, led by Lightspeed Venture Partners. In the last year, we’ve raised a total of $8 million from early investors including Foundation Capital, Spider Capital and joined Heavybit.

This is a big deal for us and we’re incredibly thrilled by what it means. The funds will support our mission to help your engineering team make application testing autonomous and integrative to their agile development cycle. We started this company because we’ve used plenty of automation tools and found them hard to use with little confidence in the stability of the tests. I’m sure you’ve had similar experiences. Together, we’ve built a family of customers including Wix.com, NetApp and Walmart, grown our team to 20+ employees, and become the fastest growing provider of autonomous testing with a 34% compound monthly growth rate.

This funding is the fuel we need to continue investing in our customer’s success by growing our engineering team, expanding our customer success team and supporting your evolving processes. In the last 4 months we’ve added tons of enhancements to help teams automate their testing even faster.

In the next year, and with your ongoing input, we’re excited to tackle:

  • Make testing mobile native applications a breeze
  • Add more self-learning capabilities to our algorithms, making our tests even more stable
  • Integrate Testim.io with more development tools (well, we already support Slack, Jira, Git, CI tools and other software development tools)

We’d love to hear what you think of our product.  Test drive Testim today by signing up for a free trial to experience code and maintenance free test automation.

Our client is the market leading data authority for the hybrid cloud. They provide a full range of hybrid cloud data services that simplify application management and data across cloud and on-site environment to accelerate digital transformation. Their solution empowers global organizations to unleash the full potential of their data to expand customer engagement, foster greater innovation, and optimize their operations.

The Challenge – Achieve CI/CD

Their application was complex and included many user flows, requiring the creation of thousands of functional tests with the goal of shifting left. In addition, they needed to test as close to development as possible.

Selenium was their first choice and a team of over a dozen testers was assembled to begin the task. After a few months it became apparent that it was going to take more time and man power to achieve their goal of CI/CD. The tests were complex and took three days to author. To make things worse, the tests would often break, leading the team to spend extra time maintaining and fixing the tests. Shifting left would require teaching the developers Selenium as well.

The lack of tools to troubleshoot a failed test (e.g screenshots to compare, details error messages pointing to the right step, test history or the parameters over the flows) led to long time-to-resolution involving a number of team members. A lot of time was wasted, not only trying to figure out why the test failed, but to also explain the discoveries to the developers.

Maintaining tests took a lot of my time. When developers run tests that fail it becomes more of a
distraction
than confidence in bug prevention.Both groups had to stop what they are doing to figure
out if its the functionality or if it’s the test. We found ourselves spending more time trying to stabilize
the tests than actually testing.
” – Company QA Manager

The Solution

Testim’s solution that uses machine learning for the authoring, execution and maintenance of automated test cases was implemented. With Testim they were able to capture the scenarios in minutes, as well as complement that with JavaScript syntaxes. The team was able to spend more time creating new tests, and validating the status of their application.

Today we have hundreds of tests running on every pull, giving the developers feedback to their code
within minutes. The developers themselves easily update test scenarios so QA can focus on increasing
coverage. We also significantly reduced our cost of quality: The rich information we get allows us to
reduce the time to troubleshoot by 80%.

The Results

As a rapidly growing enterprise, they needed a way to optimize processes through automation. This plays a significant role in their move to agile development. Within a couple months of using Testim, The team was able to create hundreds of their UI tests scenarios. Today, they are proud to say that they are authoring tests in under an hour compared to the three days it was originally taking.

Before Testim it would take 3 days to author a single test in Selenium, now even for the less
experiencedtester, writing tests takes under an hour, developers can update tests on the fly and
figure out where the tests are failing without any additional help.

Now, troubleshooting a failed tests takes a fraction of the time. There is an indication on which step failed, including JS syntaxes, screenshots comparing prior runs, and access to the DOM with detailed error messages.

However, the biggest impact was the reduction on maintenance. Tests are stable and trustworthy so when a test fails the user knows it is either due to a bug or the test requires a change in the flow. The team focused most of its time on increasing coverage knowing that the tests adapt to UI changes by the development team.

Today, we are proud to say that they are fully CI/CD, testing on every code push, and running thousands of tests every day.

Introduction

We work hard to improve the functionality and usability of our autonomous testing platform, constantly adding new features. Our sprints are weekly with minor updates being released sometimes every day, so a lot is added over the course of a month. We share updates by email and social media but wanted to provide a monthly recap of the month’s latests enhancements, summarizing the big and small things we delivered to improve your success with the Testim. We’d love to hear your feedback.

Customers have access to these feature now, check it out and let us know what you think. If you’re not a customer, sign up for a free trial to experience autonomous testing.

Group Usage Across Tests

What is it?

Groups allow you to create reuseable flows instead of rewriting them again and again. Group usage allows you to quickly filter the tests that are using a specific group.

Why should I care?  

Whether you make changes to a specific group or have a number of tests that failed and want to trace the commonality among them, group usage allows you to quickly filter the relevant tests as well as their results. This feature has been sought out by a large number of our customers and is already getting love from some our of our users. Read more here:

Scheduler –  Email Notifications

What is is?

Easily schedule tests to run on nightly build, or monitor your production app, make sure you’re notified when it fails.

Why should I care?

Plan your schedule and choose your use case:

  • Monitor – Will run the tests on a set interval. Use this option to monitor the health of your application and alert when your service is down.
  • Nightly run – Schedule the tests to run at certain days of the week and time of day. Use this option to automatically trigger test runs such as nightly regression. By the morning you’ll have an email linking to the failed test results.

Read the docs to learn more

Email Scheduler
Email Scheduler

Mobile Web Pilot Program

What is it?

Testim now supports the authoring and execution of tests for mobile web.

Why should I care?  

The number of mobile devices has suppressed the number of desktop computers. We all consume content through our mobile devices and users expect your application to provide superior experience regardless of the medium. Responsive websites look and behave differently than desktop hence require different tests.

We are currently operating a closed beta for a selected number of customers. Reach out to your Testim representative if you are interested in taking part in the program.

We’d love to hear what you think of the new features. Please share your thoughts on Twitter or Facebook.

As many yoga instructors do, they encouraged students to find balance. It’s an effective bit of advice in that it puts the onus on the practitioner. It’s also a popular concept for software testing, as industry experts often recommend that software teams find the balance between automation and manual testing practices. Another similarity is the trajectory of their adoption rates. One does not dive into yoga, but slowly adopts it over time until it becomes a part of their daily routine. The same goes for test automation: you can’t expect to start automating everything from scratch. Good test automation is the culmination of work over time.

Why not all manual?

Manual testing is fine, and was once the status quo, but with high adoption of Agile and continuous delivery methodologies, it just doesn’t scale. For software development, every enhancement or new feature that’s rolled out for an app must have a corresponding set of tests to ensure the new functionality works and does not break the code from the previous version. In this regard, to check all of the file directories, database, workflows, integrations, rules and logic manually would be extremely time consuming.

For companies looking to improve time to market and increase test coverage, automation provides significant advantages over manual testing. A typical environment can host thousands of test cases of varying complexity to be executed effortlessly and nonstop. As a result, automation dwindles the time required to perform mundane, repetitive test cases from days to hours and even minutes if you run in parallel. That inherent velocity increase is what attracts teams to automated testing.

Why not all automation?

The benefits of automation are obvious. Automation provides the ability to quickly react to ever-changing business requirements and generate new tests continuously. Is it reasonable then to assume that the more you automate the more benefits you reap? Then why not just automate everything?

The short answer, its very difficult. According to the World Quality Report 2017-18 Ninth Edition, these are the top challenges enterprise are facing in regards  to test automation.

Traditionally you would automate only once your platform is stable and the user flow is pretty consistent. This was primarily due to the amount of time it took to write a test, sometimes hours and even days. Plus the amount of time it took to update and maintain tests as code changed. The ROI was not justified up to the point your user flow is stable.

Feedback from our customers such it has changed. We recently heard a number of our clients who automate everything and even as early as wireframes are ready. We reduced authoring time by about 70% of the time and maintenance time by about 90%. Creating a user flow now takes a few minutes with Testim and so is updating the user flow. Reducing the time per test to a few minutes instead of hours and days makes it much more affordable to automate, providing immediate ROI from feedback loop benefits.

So should you strive to 100% automation? Well no but you can get to 80%-90% which was unheard of until recently. There are still scenarios that only humans can do such as Face ID. There are also other aspects of your Software Development Lifecycle (SDLC) that automation is dependent on.

Summary

There is an ongoing struggle to keep up with the rate dictated customer pressure and competition to produce on a continuous basis. And not just produce for production’s sake, but a quality product that’s been thoroughly tested. That’s why we created Testim…

To help software teams:

  • Increase coverage by 30% in 3 months
  • Reduce number of missed bugs by 37%   
  • Increase team productivity by 62%
  • Reduce regression testing costs by 80%  
  • Speedup testing schedules by 3 months

Sources

  1. https://techbeacon.com/world-quality-report-2017-18-state-qa-testing
  2. https://dzone.com/articles/automated-vs-manual-testing-how-to-find-the-right
  3. https://blog.qasource.com/the-right-balance-of-automated-vs.-manual-testing-service
  4. http://sdtimes.com/software-testing-is-all-about-automation/#sthash.5Idkps2O.dpuf
  5. http://sdtimes.com/report-test-automation-increasing/#sthash.HlHQmD2l.dpuf
  6. https://appdevelopermagazine.com/4935/2017/2/8/test-automation-usage-on-the-rise/
  7. https://techbeacon.com/devops-automation-best-practices-how-much-too-much

Thank you to everyone who participated in our round table discussion on Cracking Your Test Automation Code: The Path to CI/CD. We had a great turnout with lots of solid questions from the audience. If you missed the live event, don’t worry… Just click here to watch the recorded session. There were several questions that we were not able to answer during the live event so I followed up with the panelist afterwards to get their answers.

Q: How do you implement CI/CD without using any tools?
Bas Dijkstra: Without any tools at all it will not be possible to do CI/CD. Maybe technically there are ways to do it, but it will be massively inefficient. You will need tools that help you do version control, building software and deploying it on a specific environment. If we just look at doing CI/CD without testing  tools, that is possible in theory, but you will need to do the required amount of checking by hand anyway to make sure that the software you delivered has a certain level of quality. Doing that manually will very likely be less efficient and act as a bottleneck in the process.

How can I improve backend testing of a mobile app beyond unit testing?
Oren Rubin: You can expand beyond the unit tests by adding API testing. This is also considered integrations tests, as you testing several units as one big unit. Consider that one big state that you need to initial, and the API calls (either REST, SOAP, etc) are like function calls, some getters, some setters which modify the state.

Q: What are the things we need to cover during API tests?
Bas Dijkstra: In short: connectivity (can I talk to the API at all?), syntax (is the data returned formatted correctly, are headers and status codes OK?), semantics (is the data returned the data I expected?), functionality (does the API call have the required side effects (data stored, processes triggered, etc.), as well as no unintended side effects), performance and security (authorization, authentication).

How should one avoid overlapping in functional and unit test automation?
Oren Rubin: It’s easy to overlap the two, but it’s almost unavoidable as each unit is tested on it’s own. They are all tested via the integration tests, which usually doesn’t care about a specific unit but mostly on part of a user story/flow. Each action (e.g. click) can drive a lot of units and check them and the integrations. Remember that E2E tests come at a high price, so choose them wisely.

Is it good to separate manual test cases from automated test cases?
Bas Dijkstra: Yes. Especially since translating your ‘manual’ test cases (where you’ll likely check a number of things in the process) to automated test cases (which should ideally check one thing only) 1-to-1 will likely lead to inefficient automation. Determining what to automate and what to leave for testers is an art in itself, and simply handing someone a batch of manual test cases to automate isn’t likely to give you good results. 

What is the difference between unit test vs. integration tests in regards to front end testing?
Oren Rubin: There is no difference. Unit tests allow you to test a single unit (e.g. a controller of a component) and mock everything else; use spies, stubs, and mocks. An integration test, which might test the entire UI, where you can check unit loading order works the CSS (which is global by definition, if you exclude future shadow dom), and integrations between the different units. If you don’t mock the servers, then it’s actually End-to-End testing. 

About the panelists 

Bas Dijkstra
Bas is an independent test automation professional who has been in the test automation and service virtualization field for over 10 years now, designing and developing test automation and service virtualization solutions that enhance and improve test teams and test processes. Find out more information about Bas on his LinkedIn profile. For questions and more information you can always send him an email at bas@ontestautomation.com or give me a nudge via @_basdijkstra on Twitter.

Oren Rubin
Oren has over 20 years of experience in the software industry, building mostly test-related products for developers at IBM, Wix, Cadence, Applitools, and Testim.io. In addition to being a busy entrepreneur, Oren is a community activist and and the co-organizer of the Selenium-Israel meetup and the Israeli Google Developer Group meetup. He has taught at Technion University, and mentored at the Google Launchpad Accelerator.

Introduction

We work hard to improve the functionality and usability of our platform, constantly adding new features. Our sprints are weekly with minor updates being released sometimes every day, so a lot is added over the course of a month. We share updates by email and social media but wanted to provide a monthly recap of the month’s latests enhancements, summarizing the big and small things we delivered to improve your success with the Testim. We’d love to hear your feedback.

Customers have access to these feature now, check it out and let us know what you think. If you’re not a customer, sign up for a free trial to experience autonomous testing. If your an existing user, please send us your feedback.

GitHub

Using GitHub? Then checkout these two new advancements we’ve made to the integration.

What is it?

  1. Align your tests for your branches automatically. Testim now supports webhooks: Your tests will automatically be forked with every new branch.
  2. Easily report bugs in Git Issues straight from your browser. All the relevant bug info will automatically be populated in Testim – learn more about bug capture.

testim integration github

Why should I care?
Merging branches in GitHub? Testim will automatically merge tests in Testim so you don’t have to do that manually. Isn’t that what test automation is all about, more automation? Learn more about branches.

In order to integrate testing into your development cycle, it’s important to integrate your test automation platform to your source control. This allows you to align your tests to branches so that you can test different versions of code simultaneously or in different sequences.

UI/UX Facelift

What is it?

As we strive to improve your user experience we try to make everything more convenient by reducing the number of clicks it takes to perform common actions.  Some of these include:

  1. Edit the steps’ properties
  2. View steps’ screenshot
  3. Delete step

Why should I care?

The new UI allows you to do all of this in a single click, just hover and click the appropriate button.

We made the steps bigger so the thumbnails could contain more information. This way you don’t have to open the screenshot to understand the context. The steps description length is also longer.

We tried to remove the clutter and hide uncommon actions (e.g. override timeout) to have a cleaner UI. But don’t worry, none of the actions are gone.

testim.io new UI

  1. Shows and editor with a few steps (e.g. this one)
  2. Hover the 2nd “book” step, and click delete 3 times
  3. Click on the properties button on the step that is now beneath the mouse
  4. Click on the fullscreen button on the step that is now beneath the mouse (don’t show the spinner)

And one more thing… It’s much prettier! We’d love to hear what you think of the new UI. Please share your thoughts on Twitter or Facebook.

Struggling to get answers to these Test Automation questions? Watch the Recorded  Webinar

  • What constitutes an adequate degree of test automation coverage?
  • Does every single function in the application need a corresponding test associated with it or just the core functionality?
  • Does every application need a full end to end automation suite built out or just enough to satisfy QAs and the business? 

Then register for this webinar: Cracking Your Test Automation Code: The path to CI/CD where Bas Dijkstra, Test Automation Consultant, On Test Automation and Oren Rubin, CEO of Testim discuss the Do’s, Don’ts and misconceptions of test automation.

Bring your questions to get answers to:

  • Why your code impacts your approach to testing?
  • What’s the right mix of unit, functional, end-to-end, UI and other types of testing?
  • How to create your test automation strategy and tactical execution plan?
  • What pitfalls to avoid that will increase your cost of quality?

This round-table discussion will include insights from these industry experts:

  • Bas Dijkstra, On Test Automation
  • Oren Rubin, CEO of Testim

About the presenters:

Bas Dijkstra
Bas is an independent test automation professional who has been in the test automation and service virtualization field for over 10 years now, designing and developing test automation and service virtualization solutions that enhance and improve test teams and test processes. Find out more information about Bas on his LinkedIn profile. For questions and more information you can always send him an email at bas@ontestautomation.com or give me a nudge via @_basdijkstra on Twitter.

Oren Rubin
Oren has over 20 years of experience in the software industry, building mostly test-related products for developers at IBM, Wix, Cadence, Applitools, and Testim.io. In addition to being a busy entrepreneur, Oren is a community activist and and the co-organizer of the Selenium-Israel meetup and the Israeli Google Developer Group meetup. He has taught at Technion University, and mentored at the Google Launchpad Accelerator.

Totango is the leading enterprise-grade customer success platform that helps recurring revenue businesses proactively impact business outcomes with customer success. With solutions to empower customer success teams or entire companies, Totango enables every employee to participate in customer success.

The Problem

Like any growing company, Totango was short staffed on engineering resources and each team member owned multiple tasks. Developers were responsible for doing their own testing, mostly focusing on unit level tests while the overlay team supported the first-level testing efforts. While the team supported a variety of different test automation functions, they never had enough time to invest in automating the User Interface level testing. The team considered Selenium and experimented with it.

“We didn’t have the appetite and desire to work with something like Selenium or one of its variants,”
said Oren Raboy, Co-Founder and VP of Engineering for Totango.
“We experimented with it but
realized  
that it’s a steep learning curve and required a lot of effort just to get some basic value out of
building such a system. It was brittle and slow.”

The Solution

Totango believes that everyone on the team should be able to write tests and own code quality. Totango’s R&D team members picked up Testim quickly, using it for front end test automation coverage.

“Testim’s simple user interface makes it easy for our developers to add tests without a steep learning
curve”
says Oren, ”We increased coverage by 30% in 3 months and found that very little maintenance
is required so we can focus on increasing coverage.”

Totango was able to create a UI regression suite that gave the team confidence that things were not breaking while allowing them to add, expand, and grow without investing a ton of resources into their UI test automation. The regression suite is extremely valuable for the app development team as it serves as a safety net they can rely on when they make changes that break things.

Totango is now working to leverage Testim to not only help with code-coverage but run various test scenarios in production, thus giving them an app level production monitoring to mitigate production malfunctions, above & beyond code regressions.

The Results

The engineering team ran for quite some time by doing UI level testing manually. In order to keep up with customer demands and develop additional products to their suite without slowing down their daily releases, the team needed to optimize their frontend testing efforts. Before Testim, Totango would occasionally have a regression at the UI level that forced the team to stop working and go back and fix the issues. Now the team can find the problems quicker and react to them faster, giving them more coverage without slowing down the development pace.

“We had a lot of automated testing, just not at the UI level,” explained Oren. With Testim, we’ve been
able to place quality front and center and make it a company wide priority vs a topic only QA is responsible
for
. The end result is increase in velocity, quality and customer satisfaction.”

 

 

Google recently announced that they released Puppeteer, a Node library which provides an API to control headless Chrome. Within 24hrs they received great feedback from the community;

  • 6,685 stars on Github
  • 2.2K likes and 1.2K shares on Twitter

So why should we care? Here’s a snippet from its GitHub documentation:

Puppeteer’s GitHub documentation Q&A

In Google’s own words, there isn’t much difference from Selenium.

The awesomeness of Selenium is that they convinced ALL browser vendors to support the same low level API (and this took years! try convincing Apple, MS, and Google to work together), and even implemented this API in more than 10 languages (including JS).

Most of Puppeteer’s API is very similar to Selenium (or the Webdriver.io/NightwatchJS alternatives) e.g.;

  • Google’s launch() method vs. Selenium’s init()
  • goto() vs. to url()
  • close() vs. end()
  • type() vs. setValue()
  • click() even stayed the same

Google could have picked the same Selenium API and contributed the changes to the Selenium repo. But the biggest issue isn’t the API. It’s splitting the community and not contributing to the Selenium code base. With Google’s resources and talented developers, they could have contributed to the Selenium project, which currently has a few amazing volunteers, supporting this framework, and some parts are closing for lack of resources.

Selenium is known to be relatively slow compared to operating directly on the browser. This is caused by its architecture, which requires a Selenium Grid or Selenium Standalone server, which acts as a proxy (and even just starting it takes a while). This architecture is great when your tests need to run on multiple browsers in parallel, but you experience the overhead when working with a single browser. By helping the Selenium community speed this up, even if it was just for Chrome, would have been more beneficial than trying to create their own (Chrome only) solution.

Puppeteer is a step in the right direction. Google is an innovative company that pushes the web forward with great ideas and specs, amazing developer tools, and now it seems to help improve UI test automation, which we all know is extremely challenging.

Standardization leads to innovation. With Selenium, not only would you be able to run those tests on other browsers, but the entire industry is relying on those standards to create additional (commercialized) products. For example; Testim.io, Saucelabs, BrowserStack, Applitools, Cross Browser Testing, and the list goes on and on.

I would love to hear your opinion about Puppeteer and Selenium.

Cheers,
Oren Rubin
CEO, Testim.io

Be smart & save time...
Simply automate.

Get Started Free