Tag

continuous delivery

Browsing

Testing has been in the spotlight of software development, as organizations continually seek to optimize development cycles. Continuous Delivery and its promises of frequent releases, even as often as hourly, is a big factor driving executives  to find ways  to shave time off of any eligible technical processes. As enterprises embark  on their DevOps transformational journeys, delivering at lightning speed without compromising on quality takes more than cultural changes and process improvements. Test automation is key in helping project teams write and manage the smallest number of test conditions, but get the maximum coverage.

The case for automation: If the scenario is highly quantitative, technology is superior to humans. That’s clearly the case in the current technology landscape, where according to the 2015 OpenSignal report, massive fragmentation has resulted in over 24,000 distinct Android devices that all run on different versions of the Android OS, plus the several variations of iOS devices currently in use. Web is no different. It’s impossible to test every scenario manually, so automation must be leveraged. But therein lies the main point, that automated testing is a tool to be leveraged as needed instead of relied upon to dictate an entire testing strategy.Although logic would deem that if automating a few test scenarios yield fast results, that automating any and every eligible scenario will shorten the test cycle even more; but alas,  that’s usually not how it goes. Automation endeavors take a good deal of planning and forethought in order to add value at the anticipated level. More often than not the overhead of maintenance, especially in CI/CD environments where tests are automatically triggered and one needs to analyze the reports and fix locators, a task that , could  take hours.  This won’t work for organizations who are truly DevOps.  If full stack developers are going to be fully responsible to take the code to production, then they will need tools to automate the process through infrastructure, testing, deployment and monitoring.  This continuous framework enables  weekly , daily and hourly releases.  Leading DevOps organizations like Netflix and Amazon are said to deploy hundreds to thousands times per day.

What’s more, studies reveal that a high percentage of projects utilizing automated testing fall short of the anticipated ROI or fail altogether. These transformations fall short due to their duration, ramp up time, skill-set gap and maintenance overhead.  If the benefits of automated testing aren’t significant enough to mitigate risk then speedy releases could become more of a liability than an asset.

There are varying levels of automation just as there are companies of all different shapes and sizes. Instead of one or the other, it’s more fitting that automated and manual testing are looked at as complementary. Agile and DevOps have created new testing challenges for QA professionals who must meet the requirements for rapid delivery cycles and automated delivery pipelines.

DEMANDS OF CONTINUOUS TESTING

Testing used to have the luxury of having its own phase or at least a set timeframe to stabilize new code prior to pushing to production. No longer. In the era of DevOps, QA must adopt a truly agile mindset and continually be prepared to shift focus to testing new features and functionality as they become available – that could be weekly, daily, or hourly.

From web, mobile and connected devices to IoT, a quick inventory of current technology will reveal a multitude of different devices and literally thousands of potential ways that these technologies can be combined. Across networks, apps, operating systems, browsers and API libraries, the number of  combinations with each requiring its own verification and testing support.  As the push towards Devops continues, the upward trend towards continuous testing is inevitable. As a result, testers and/or developers will need to deliver in brief and rapid iterations.

Beyond automation, what does this look like? QA, Engineering, or whomever is responsible for testing will need to shift left and engage in rounding out user story acceptance criteria to ensure accuracy and completion. In addition, active participation in sprint plannings will help to ensure that user stories are truly decoupled and independent. Or if dependencies are necessary, confirming that story precursors and relationships are in-place. Partnership with Product Owners to confirm that the scope for a given sprint or iteration is realistic.

And finally in support of shifting left, collaboration among Dev and Ops to ensure the necessary support in the event that a code check-in causes a bunch of automated tests to fail. The culprit for these tests will need to be checked manually so it’s important that QA is prepared so as not to pose a blocker for the release. All of these activities is largely what comprises testing outside of automation. It’s a broader role than the traditional QA Specialist that requires initiative and the willingness to embrace an always-on mentality that’s necessary to support DevOps efforts.

SUMMARY

Software development practices have changed and the change brings with it new challenges for the testing arena. The growing need for test automation in addition to the profusion of new testing scenarios will constantly evolve the role of testing and QA.  Yes, the world is changing for testers—in a good way. The days of exhaustive manual testing are certainly numbered, and automated testing is an essential part of the delivery pipeline. But testing has never been more important.

SOURCES

  1. IT Revoloution DevOps Guide
  2. World Quality Report 2016-17
  3. Open Signal: Android Fragmentation

 

Introduction
We work hard to improve the functionality and usability of our platform, constantly adding new features. Our sprints are weekly with minor updates being released sometimes every day, so a lot is added over the course of a month. We share updates by email and social media but wanted to provide a recap of the latest enhancements, summarizing the big and small things we delivered to improve your success with the product. We’d love to hear your feedback.

Test History

Is past performance an indicator of future success? The folks on Wall Street don’t think so… But we do! This is why we are pleased to offer the new Test History feature in Testim.  


The new Test History feature allows users to see slide and dice test results to get more actionable insights.

Why should I care?
This gives users much more insight from the test results. Quickly analyze how many tests ran during the selected time frame, how many succeeded and the average duration of each run, how your tests performs across multiple environments and whether certain environments or scenarios consistently fail. This allows project team to see improvement trends across scenarios or environments over time. Click here to learn more.

What’s included?
Test History allows you to filter based on the following parameters:

  • Run time
  • Specific tests / All tests
  • Last X runs
  • Status
  • Browser

Create Dependencies Between Tests
This new capability allows users to create different aspects of dependencies between tests. As a best practice, we recommend to keep the tests as isolated as possible. However we recognize that sometimes you need the ability to create dependencies between tests.

Why should I care?
Now users can create a logical sequence of tests. By working closely with our customers, we’ve seen project teams required to set a sequence of activities. A general testing framework does not allow you to create a sequence, forcing users to create one long test which may result in performance issues and failures. By creating dependencies during test plans you can create shorter discrete actions, order them in sequence and set dependencies and share data between tests. Click here to learn more.

Setting up Cookies
Cookies is a reserved parameter name for specifying a set of cookies you would like Testim to set for your application.  You can set cookies for each browser session before the test is started or before the page is even loaded. Use cookies to set browser data to be available for your tests. The cookies parameter is set as an array, where each cookie object needs to contain a name and value properties.

Why should I care?
Websites use cookies to personalize user experience. By using this feature, you can test different types of personalized experiences. Click here to learn more.

Improved Scrolling
More flexible way of scrolling through your page.

Why should I care?
Traditionally scrolling in your test automation would be set by pixels so you would test your script to skip 60 pixels. Offering a more flexible scrolling ability makes your tests more adaptive. An  example is to scroll to the element, mouse wheel. Click here to learn more.

How do I get my hands on these hot new features?
Customers have access to these feature now, check it out and let us know what you think. If you’re not a customer, test drive the new features by signing up for a free trial to experience code and maintenance free test automation.

Today I joined Testim.io as President and COO. Testim has the potential to disrupt a massive, $8B+ market for web and mobile QA automation. Testim has built an insanely powerful product that seeks to alleviate the pain associated with test automation — current solutions suck. Testim has achieved product market fit (you might want to read more about that below), has a great team and culture spanning our Israel and San Francisco offices, and needed a business leader to help scale the company to the next level.

How I met Oren Rubin

Throughout the years, I’ve seen hundreds of customers struggle to transform agile. QA was the main bottleneck, specifically the creation and maintenance of automated test cases. I’ve seen organizations fire managers and vendors as a result of failure to meet automation goals. Scripting full test suites took months, during which organizations would depend on one or two skilled automation leads. This had impact on time and money. This is a problem worth solving.

Several months ago, a friend told me about an interesting tool they were assessing. He suggested that I have a call with Oren, the founder and CEO. A few weeks later, I was driving on 101, when Oren called me, The next 45 minutes were invigorating. His description of the need was music to my ears. The way he described Testim’s approach to the maintenance of test scripts intrigued me.

A few weeks later I stopped by Heavybit, a San Francisco based accelerator for developer related startups, to meet Oren. We immediately hit it off. He showed me a demo and I loved it! It was exactly the type of tool that was required to help companies automate release cycles generally and quality assurance specifically. It was cloud based, required little scripting and was visual and easy to use. And developers loved it! It took 3 minutes to create a test. I started firing questions at Oren: about Testim’s product roadmap, current customers, personas, integration with other devops tools, etc. Over the following weeks, we had a number of similar discussions.

So why did I join Testim?

Over the last few years, I’ve met hundreds of startups. There are a few ingredients that make a great startup.

Deep and identifiable Pain Point — “Make sure your product is a pain killer. Not a vitamin.” If you help customers address a great need that is making their job more difficult, they will give you the time of day, and more often than not, will be ready to pay for it. I’ve seen that pain point around Continuous Delivery & Continuous Integration and specifically around test automation: the time to author a single test case, the effort required to maintain these test suites and the challenges around the required expertise. Test automation is a big pain point to a vast majority of the development teams I’ve met.

Superb Team with Domain Expertise — Many investors have articulated the qualities of a great team and more importantly a great founder. It is difficult to define one until you meet one. Oren is that kind of founder. Oren has spent the past two decades living and breathing the problem which Testim is trying to solve. He experienced the challenges of agile transformation and built a product based on his unique insights. He largely bootstrapped the company for the first two years with his own money and that of a few prominent Israel angels. He was able to attract a strong, highly technical team of engineers. Based on his product vision and early customer traction, he was also able to attract a prominent group of Silicon Valley and Israel venture funds. And Oren believed that for the company to realize its full business potential, the corporate HQ would have to be based in Silicon Valley. So he packed up and moved to San Francisco.

Oren also possessed certain founder attributes that I value, including being a good listener, open to feedback, and actually implementing product suggestions based on our discussions. The discussions I had with each person on the team also reinforced my excitement. My discussion with Gil, our VP R&D who leads our Israel development office, lasted two hours, where we covered product roadmap and the rapidly changing market. The meeting with Michael Neril, the Founding Partner of Spider Capital, one of Testim’s investors, was a joy. Michael is an ideal investor in my view: He is constantly looking to add value with introductions to consultants, partners, customer prospects and candidates. We speak frequently and he is a great sounding board on strategy.

Clear Product/Market fit — Over the last couple of weeks, I spent time talking to Testim’s customers as well as folks I know and respect. I remember a specific conversation that happened while waiting in line at a coffee shop. A friend who was also in line for coffee introduced me to his CTO. We somehow got to talk about quality and Testim. “I love the product. It’s awesome. They are onto something.” I heard similar reactions from customers. Just today, we received the following note from a new customer: “This is really a good application you have here. It’s a big help to us QAs.” P.S.: Developers love our product even more!

Scalable Channel to Market — A pain-point, a talented team and a great product are not enough. You need a scalable channel to market. Testim has hundreds of active customers and dozens of paying ones with zero money spent on sales, marketing, or a structured process. That’s why I’m excited to be part of this journey. With a structured sales process, an understanding of the pain points, clear definition of the target persona and organization, well-crafted messaging, and a few additional A players on our team, we have the building blocks to build a truly transformative company in the test automation space.

Breakthrough Technology – Many companies approached the challenge of QA automation by focusing on simplifying authoring using GUI based play and record. Testim’s approach, focusing on stability and reliability first is refreshing. The evolution of real-time machine learning powers the ability to analyze hundreds of elements and thousands of attributes, enabling Testim to deeply understand the Document Object Model (DOM) in order to create dynamic test suites which are adaptive to changes in the application. 

That brings me to my last two comments:

1.If you are looking for a powerful, stable, yet simple automation tool — please reach out.

2.I’m also hiring. If you or someone you know is talented, highly productive, thrives in a fast moving startup, and is a team player — please reach out.

About Testim.io

Testim uses machine learning for the authoring, execution and self-maintenance of automated test suites. Testim learns from every   execution, self-improving the stability of test cases resulting in a test suite that doesn’t break on every code change.

A developer can author test cases in minutes. Testim powers the transition to agile and shift left, giving organizations the tools to maintain maximum coverage over every release. Testim, headquartered in San Francisco, is backed by a leading group of U.S. and Israel venture capital funds.

Be smart & save time...
Simply automate.

Get Started Free