Author

Shani Shoham

Browsing

Today I had the pleasure of sitting down with Jake Kaldenbaugh, Strategic Exit (M&A) Advisor at GrowthPoint Technology Partners to discuss the state of the software development industry. As a senior banking leader Jake helps growing technology companies with great technology create strategic value.

In this interview Jake covers:

About GrowthPoint Technology Partners

GrowthPoint Technology Partners is an emerging growth investment banking boutique that helps growing technology firms with great technology create strategic value. Our team identifies leading companies in the areas of data, analytics, infrastructure, virtualization, security and systems management and helps lead them through successful value realization strategies that enable entrepreneurs and investors to achieve their best results.

Practical steps to shift left

Last month we published “Do you know how much your quality costs?” where we discussed the importance of shifting left and shortening the feedback loop as it relates to the cost of quality. According to Bank of America, discovering a bug in development would cost about 5 hours to fix and $390. Discovering a bug in QA cost about x30. 127 hours to fix, due to to the time it takes to reproduce, the back and forth between dev and QA and the need to trace back the code.. Fixing a bug in production – Estimated at $72,000. Shifting left allows testing close to development as possible and a fast and continuous feedback loop.

Direct cost to find/fix a defect

Four key ingredients to achieving shift left:

  1. Test automation – Traditional QA processes would result in months of testing before you get feedback. A productive tester can get about 20 tests done in a single day. Assuming your test suite includes dozens of tests there are two ways to get the feedback loop down to minutes: Hire hundreds of testers or automate your tests. Automated tests don’t get tired, take days off, go out for lunch etc.
  2. High concurrency – Shifting left means giving near real time feedback i.e minutes. Running dozens of tests, each taking roughly 30-60 seconds, with the expectations of getting the results back in, say 5 min., can only be done if you are running your tests in parallel. The higher your level of parallelism the shorter the feedback loop is. Running your tests in parallel required a grid, a cloud based infrastructure that includes virtual environments allowing you to spin in and out different browsers.
  3. Build-test-deploy automation – Shifting left means you are optimizing the handover across the different stages of the development process. The development cycle includes dozens of handovers and human intervention can always delay that process hence you want to automate the handover using Continuous Integration (CI) servers. CI servers monitor events through your development cycle, such as a new build, and can be programed to trigger a set of actions once a certain event occurred, such as triggering your automated tests.
  4. Suites – It is best practice to optimize your quality assurance processes by segmenting your tests into suites and running different suites at different stages of your development cycle. For example you full regression might include thousands of tests. Running all those tests every time a develop push code would be time and resource consuming. If 2-3 key scenarios failed, there is no point in waiting for the entire suite to be over. You might want to create a small subset of tests, composed of your high priority flows, and run that “sanity” suite often, while running the more extensive suites once if sanity suite ran successfully.

 

Do you how much quality costs? Have you ever calculated it? If you do, you’ll learn that you spend thousands of dollars and hours of productivity on every bug. From the cost of labor, to the cost of deployment, to the cost of finding the bugs in production, no bug fix is free.

Many companies and academics, from IBM to Microsoft to large banks have performed studies to quantify the cost of finding bugs. It’s been well known for many years that the costs go up the closer to production you go, but the actual numbers are a bit staggering.Their findings show the dramatic financial difference between discovering a bug in the wild, and finding it during development.

  • Fixing a bug in production – Very expensive. These take time to fix, and while repairs take place, your business experiences pain. Your teams are working around the clock to fix something that also requires your developer to rework something they wrote months ago and forgot about.
  • Fixing a bug in QA – Somewhat expensive. The QA person has to report the bug. A senior manager has to determine who to be assigned to this. The developer has to step out of his daily tasks, sometimes impacting the current sprint, attend to the bug, reproduce it, and return to edit some code from the last sprint, which the developer has likely already begun to forget about.
  • Fixing a bug in development – Least expensive. takes a few minutes for the developer to fix. If you have an immediate feedback loop, your developer will get feedback minutes after pushing code. If a test fails, it is still within the current sprint and the developer should recognize the code and its implications as everything is still fresh in their mind. The bug didn’t escalate and the likelihood of on time release is much higher.

Shift left is the term coined for a software development cycle that includes a constant and immediate feedback loop. You may have heard stories about how Facebook has put a billion dollars worth of engineering into creating this feedback loop. The company even wrote a PHP-to-C compiler called HipHop to speed up the loading of its test environments.

If Facebook is willing to spend a billion dollars to get that feedback loop, you can spend a few weeks ensuring your own loop is as fast and reliable as possible. This means fast builds, fast test batteries, and short amounts of time between developer compiling, and developer testing.

This requires certain infrastructure that we’ll get to in a future post but organizations that have deployed it, have seen decrease in cost of quality and increase in on-time delivery. You may even have to disassemble your build pipeline to replace a few slower moving items.

Your goal, here, should be one minute or less between build and test. This may seem impossible, or it may already be the way things work for your team. This can require a lot of work, or a little depending on the existing systems. No matter the environment, there is one fundamental truth for all software development: the faster the developer can see their results in a running binary, the faster they can fix problems and the lower the cost of overall quality.

Want to know how to shift left? Stay tuned.

Sources:

Testing is one of the key processes in software development. As the requirements to support multiple browsers and devices increases, it is driving the demand to automate the functional testing of applications. While teams embark on their DevOps transformational journeys, the handoffs between traditional application lifecycle silos begin to dismantle, enabling a continuous process of updating their applications.

The key challenges for software teams in DevOps include:

  • how to achieve maximum coverage testing with shorter time frames?
  • how to figure out what to automate vs. leave manual?
  • how to transform from testing and validation before go live to more predictive forms of testing and real-time quality monitoring?

With the increased adoption of agile methodologies, test automation is a key piece of continuous testing. According to the 2015-2016 World Quality Report, research show that the average percentage of test case automation has increased from 28% to 45%. An enabler of this trend has been the advancements in development and testing tools which have helped organizations build a solid infrastructure for automation.

Testim’s stable, self-healing, end-to-end test automation solution is recognized by Software Testing Help as a Top 20 Best Tool to help developers and/or testers create their test suites without any coding, script management or maintenance of the tests. Using machine learning, Testim provides the maximum coverage with the fewest amount of test scenarios allowing teams to optimize their test suite. This also eliminates the need of trying to figure out what to automate vs. leave manual since tests are automatically created as the user uses the application. Additionally, Testim helps teams transform from testing and validation by leveraging historical data to prioritize tests.

“We are humbled by making this list,” says Oren Rubin, CEO for Testim. “This is a testament to our commitment to helping project teams eliminate the complexity of building their test automation suite.”

About SoftwaretestingHelp.com
STH is one of the most popular blogs focusing on Software Testing and Quality Assurance topics. This blog is growing up so fast and currently, we have around thousands of testing professionals who visit every day and gain help from this blog. The topics which we cover on this blog include – software testing tutorials, methodologies, manual testing, automation testing, testing tools, interview questions, web testing, testing templates, quality assurance, testing certifications, books, career guidance, job openings, latest testing trends, news, and much more that cannot be listed here on a single page.

About Testim.io
Testim.io leverages machine learning for the authoring, execution and maintenance of automated test cases. We use dynamic locators and learn with every execution. The outcome is super fast authoring and stable tests that learn, thus eliminating the need to continually maintain tests with every code change. Netapp, Verizon Wireless, Wix.com and others run over 300,000 tests using Testim.io every month.

In a world gone mad, where customers will access your application through any damn browser they want, how can you meet their expectations that your app will work perfectly whether it’s running on their phone, their laptop, their tablet, or even their internet enabled refrigerator? Cross-browser testing is essential to ensuring your customer has a consistent user experience, no matter how they access your app. But is just not realistic to manually test all the potential combinations of browsers and systems out there.

However, over the last few years there are a number of vendors who have developed different approaches to cross-browser testing. We have assembled a short list of the 12 most interesting. Each has a slightly different set of capabilities, and a different price level too. Some support mobile, some don’t. Some have security built in, and some do not. Selenium support varies as well.

Most, like Testim.io, have free trials, so you can evaluate which one is the right solution for you.

How to Choose a Provider

There are a number of attributes you should consider when choosing a grid provider. These include:

  • Coverage of browsers, versions and operating systems.
  • Stability – a flaky grid will result in test failure, leading to running those tests again at best and developers spending time figuring out why the test failed at worse. A flaky grid will result in valuable time being lost.
  • Enterprise grade – If your app is internal to the organization you will need tunneling and potentially VPN, encryption or other other security requirements as mandated by your CISO. Not all cross browser testing providers support that.
  • Integration – No organization will change its tool-set due to a grid. Your team is likely using Git, Jira, Slack, Jenkins and/or other tools. Integrating into those tools to trigger your tests and read through the results is essential. Without it, you’ll either have to change the process or end up spending your budget on a tool that will be useless.
  • Features – Do you need screenshots? Single sign-on?
  • Mobile – If you are looking to test your mobile app or web you need to decide whether you settle for emulators or need real devices. Then you have to analyze the variety of devices being offered, including geographies, mobile operators, OS versions etc. Many times you have to drill down to the set of actions the grid supports. Does your app use location sensors? Camera? Fingerprints? Not all grid providers support these actions.

Capability Grid

Cross browser tools
Cross browser tools – Comparison

Today I joined Testim.io as President and COO. Testim has the potential to disrupt a massive, $8B+ market for web and mobile QA automation. Testim has built an insanely powerful product that seeks to alleviate the pain associated with test automation — current solutions suck. Testim has achieved product market fit (you might want to read more about that below), has a great team and culture spanning our Israel and San Francisco offices, and needed a business leader to help scale the company to the next level.

How I met Oren Rubin

Throughout the years, I’ve seen hundreds of customers struggle to transform agile. QA was the main bottleneck, specifically the creation and maintenance of automated test cases. I’ve seen organizations fire managers and vendors as a result of failure to meet automation goals. Scripting full test suites took months, during which organizations would depend on one or two skilled automation leads. This had impact on time and money. This is a problem worth solving.

Several months ago, a friend told me about an interesting tool they were assessing. He suggested that I have a call with Oren, the founder and CEO. A few weeks later, I was driving on 101, when Oren called me, The next 45 minutes were invigorating. His description of the need was music to my ears. The way he described Testim’s approach to the maintenance of test scripts intrigued me.

A few weeks later I stopped by Heavybit, a San Francisco based accelerator for developer related startups, to meet Oren. We immediately hit it off. He showed me a demo and I loved it! It was exactly the type of tool that was required to help companies automate release cycles generally and quality assurance specifically. It was cloud based, required little scripting and was visual and easy to use. And developers loved it! It took 3 minutes to create a test. I started firing questions at Oren: about Testim’s product roadmap, current customers, personas, integration with other devops tools, etc. Over the following weeks, we had a number of similar discussions.

So why did I join Testim?

Over the last few years, I’ve met hundreds of startups. There are a few ingredients that make a great startup.

Deep and identifiable Pain Point — “Make sure your product is a pain killer. Not a vitamin.” If you help customers address a great need that is making their job more difficult, they will give you the time of day, and more often than not, will be ready to pay for it. I’ve seen that pain point around Continuous Delivery & Continuous Integration and specifically around test automation: the time to author a single test case, the effort required to maintain these test suites and the challenges around the required expertise. Test automation is a big pain point to a vast majority of the development teams I’ve met.

Superb Team with Domain Expertise — Many investors have articulated the qualities of a great team and more importantly a great founder. It is difficult to define one until you meet one. Oren is that kind of founder. Oren has spent the past two decades living and breathing the problem which Testim is trying to solve. He experienced the challenges of agile transformation and built a product based on his unique insights. He largely bootstrapped the company for the first two years with his own money and that of a few prominent Israel angels. He was able to attract a strong, highly technical team of engineers. Based on his product vision and early customer traction, he was also able to attract a prominent group of Silicon Valley and Israel venture funds. And Oren believed that for the company to realize its full business potential, the corporate HQ would have to be based in Silicon Valley. So he packed up and moved to San Francisco.

Oren also possessed certain founder attributes that I value, including being a good listener, open to feedback, and actually implementing product suggestions based on our discussions. The discussions I had with each person on the team also reinforced my excitement. My discussion with Gil, our VP R&D who leads our Israel development office, lasted two hours, where we covered product roadmap and the rapidly changing market. The meeting with Michael Neril, the Founding Partner of Spider Capital, one of Testim’s investors, was a joy. Michael is an ideal investor in my view: He is constantly looking to add value with introductions to consultants, partners, customer prospects and candidates. We speak frequently and he is a great sounding board on strategy.

Clear Product/Market fit — Over the last couple of weeks, I spent time talking to Testim’s customers as well as folks I know and respect. I remember a specific conversation that happened while waiting in line at a coffee shop. A friend who was also in line for coffee introduced me to his CTO. We somehow got to talk about quality and Testim. “I love the product. It’s awesome. They are onto something.” I heard similar reactions from customers. Just today, we received the following note from a new customer: “This is really a good application you have here. It’s a big help to us QAs.” P.S.: Developers love our product even more!

Scalable Channel to Market — A pain-point, a talented team and a great product are not enough. You need a scalable channel to market. Testim has hundreds of active customers and dozens of paying ones with zero money spent on sales, marketing, or a structured process. That’s why I’m excited to be part of this journey. With a structured sales process, an understanding of the pain points, clear definition of the target persona and organization, well-crafted messaging, and a few additional A players on our team, we have the building blocks to build a truly transformative company in the test automation space.

Breakthrough Technology – Many companies approached the challenge of QA automation by focusing on simplifying authoring using GUI based play and record. Testim’s approach, focusing on stability and reliability first is refreshing. The evolution of real-time machine learning powers the ability to analyze hundreds of elements and thousands of attributes, enabling Testim to deeply understand the Document Object Model (DOM) in order to create dynamic test suites which are adaptive to changes in the application. 

That brings me to my last two comments:

1.If you are looking for a powerful, stable, yet simple automation tool — please reach out.

2.I’m also hiring. If you or someone you know is talented, highly productive, thrives in a fast moving startup, and is a team player — please reach out.

About Testim.io

Testim uses machine learning for the authoring, execution and self-maintenance of automated test suites. Testim learns from every   execution, self-improving the stability of test cases resulting in a test suite that doesn’t break on every code change.

A developer can author test cases in minutes. Testim powers the transition to agile and shift left, giving organizations the tools to maintain maximum coverage over every release. Testim, headquartered in San Francisco, is backed by a leading group of U.S. and Israel venture capital funds.

Be smart & save time...
Simply automate.

Get Started Free