Tag

continuous delivery

Browsing

This is the third and final part of our series of interviews with Jake Kaldenbaugh, Strategic Exit (M&A) Advisor at GrowthPoint Technology Partners. In the first part of our video series Jake answered questions around players in the DevOps space, the constant evolution of software and the vast growth in mobile. In the second interview, Jake shared his perspective on the DevOps strategies of enterprise giants CA’s, IBM, Amazon, Microsoft and Atlassian. He also touched on what the sale of HPs impact will mean to the industry as well as testing’s role in continuous delivery. 

In this interview Jake covers:

  • Next generation platforms; containers, APIs and microservices
  • Testing’s role in the next generation of platforms
  • Digital transformations challenges
  • Up and coming startups in testing

Checkout the two previous interviews:

About GrowthPoint Technology Partners

GrowthPoint Technology Partners is an emerging growth investment banking boutique that helps growing technology firms with great technology create strategic value. Our team identifies leading companies in the areas of data, analytics, infrastructure, virtualization, security and systems management and helps lead them through successful value realization strategies that enable entrepreneurs and investors to achieve their best results.

Last week we released the first video in our three part series with Jake Kaldenbaugh, Strategic Exit (M&A) Advisor at GrowthPoint Technology Partners. In the first video Jake answered questions around players in the DevOps space, the constant evolution of software and the vast growth in mobile. Checkout the first video in the series.

In the second part of the interview Jake covers:

  • CA’s, IBM, Amazon, Microsoft and Atlassian DevOps strategies
  • The sale of HPs impact on the industry
  • Testing’s role in continuous delivery

About GrowthPoint Technology Partners

GrowthPoint Technology Partners is an emerging growth investment banking boutique that helps growing technology firms with great technology create strategic value. Our team identifies leading companies in the areas of data, analytics, infrastructure, virtualization, security and systems management and helps lead them through successful value realization strategies that enable entrepreneurs and investors to achieve their best results.

Today I had the pleasure of sitting down with Jake Kaldenbaugh, Strategic Exit (M&A) Advisor at GrowthPoint Technology Partners to discuss the state of the software development industry. As a senior banking leader Jake helps growing technology companies with great technology create strategic value.

In this interview Jake covers:

About GrowthPoint Technology Partners

GrowthPoint Technology Partners is an emerging growth investment banking boutique that helps growing technology firms with great technology create strategic value. Our team identifies leading companies in the areas of data, analytics, infrastructure, virtualization, security and systems management and helps lead them through successful value realization strategies that enable entrepreneurs and investors to achieve their best results.

Thank you to everyone who participated in our round table discussion on Choosing the Right DevOps Strategy: How far left should I shift? We had a solid turnout with lots of great questions from the audience. If you missed the live event, don’t worry…

You can watch the recorded session any time:

Tanya Kravtsov, Director of QA at Audible, and Bob Crews, President of Checkpoint Technologies shared their lessons learned from four different types of enterprise DevOps transformations.

They covered:

  • Practical suggestions for assessing your operational reality
  • Major pitfalls in DevOps transformations and how to avoid them
  • How to organize your team to increase productivity
  • Impact on development processes, ownership, tool stack and KPIs

Some of the audience questions they answered:

  • How do you ensure that security requirements are always considered in DevOps?
  • Which KPIs are important to track to ensure successful DevOps transformation?
  • What is an acceptable amount of technical debt? How do we know when it becomes a problem?
  • What are some metrics for measuring risk?

There were several questions that we were not able to address during the live event so I followed up with the panelist afterwards to get their answers.

Q: Why was continuous deployment listed twice on the diagrams?

Francis: Great catch. That was actually a typo, a marketing design bug slipped through to production… The “Continuous Deployment” label shown to the upper left, within the square encompassing the “Build” and “Test” phases in both the upper (Continuous Delivery) and bottom (Continuous Deployment) cycles, should have been labeled “Continuous Integration.”

Q: In reference to CD /CI what are the main benefits of doing smoke testing after production deployment? And, what could be an example of smoke testing?  

Tanya: Smoke test validating the most critical functionality of the product should be executed after every deployment to test the staging and production environments. This will help you uncover any deployment related issues and rollback the release if necessary without impacting the customer.  

Bob: Examples of “smoke testing” would be quick test to check for and validate successful login with valid data, login error message with invalid data, checking for broken links, tests to quickly validate the number and state of objects on a webpage, etc. These are very simple, but important tests. Typically they are run in production as one last check since many, perhaps most, organizations do not have a test environment that is an EXACT duplicate of production.

Q: Are automated tests the entire QA solution across all these organizations? Do the panelists see any value in manual, exploratory QA testers?

Tanya: Automated tests are an important component of the Continuous Delivery solution. It enables exploratory QA testers to focus on what they do best and actually test the new features of the product  by eliminating the need for manually running the same regression tests every time new code is committed.  

Bob: Absolutely. I believe it’s critical to carve out an approach to perform some quick manual, exploratory tests. These can be done as a separate initiative, apart from the validation taking place during these phases.

Q: What approach we should follow to test DevOps solutions?

Tanya: Any scripted DevOps solutions should be treated just like any code,  accompanied by unit and integration tests which run continuously in a test environment before the changes get promoted to the mainline.  In addition,  closely tracking KPIs discussed in the Webinar will alert us to any issues with the current solutions such as build up of technical debt or reduction in quality,  which should be acted upon immediately.   

Bob: I would need a little more information in order to provide a good answer. Knowing the current state of the organization’s testing approach, their objectives, the tools already in place, the skill set of current team members, etc. are just a few questions I would have before being able to provide any insight. If starting from scratch I would say start with becoming entrenched in an Agile approach. If you are already Agile then the next step would be to look at tackling Continuous Integration. Please feel free to connect with me Linkedin and I would be happy to chat more about your question.

About Tanya Kravtsov
Tanya Kravtsov is the Director of QA at Audible. She is building a new QA organization to support innovative web product development at scale. Previously, as head of automation and continuous delivery at ROKITT, senior QA manager at Syncsort, and VP at Morgan Stanley, Tanya focused on quality, automation, and DevOps practices, working with internal and external customers to transform development and testing processes. Tanya is passionate about process automation, continuous integration, and continuous delivery. She is a founder of the DevOpsQA NJ Meetup group and a frequent speaker at STAREAST, QUEST, and other conferences and events. Follow her on Twitter @DevOpsQA.

About Bob Crews
Bob Crews is the President of Checkpoint Technologies. He is a consultant and trainer with almost three decades of I.T. experience including full life-cycle development involving development, requirements management and software testing. He is also the President of the Tampa Bay Quality Assurance Association. He has consulted and trained for over 200 different organizations in areas such as effectively using automated testing solutions, test planning, implementing automated frameworks and developing practices which ensure the maximum return-on-investment with automated solutions.

Our client is the market leading data authority for the hybrid cloud. They provide a full range of hybrid cloud data services that simplify application management and data across cloud and on-site environment to accelerate digital transformation. Their solution empowers global organizations to unleash the full potential of their data to expand customer engagement, foster greater innovation, and optimize their operations.

The Challenge – Achieve CI/CD

Their application was complex and included many user flows, requiring the creation of thousands of functional tests with the goal of shifting left. In addition, they needed to test as close to development as possible.

Selenium was their first choice and a team of over a dozen testers was assembled to begin the task. After a few months it became apparent that it was going to take more time and man power to achieve their goal of CI/CD. The tests were complex and took three days to author. To make things worse, the tests would often break, leading the team to spend extra time maintaining and fixing the tests. Shifting left would require teaching the developers Selenium as well.

The lack of tools to troubleshoot a failed test (e.g screenshots to compare, details error messages pointing to the right step, test history or the parameters over the flows) led to long time-to-resolution involving a number of team members. A lot of time was wasted, not only trying to figure out why the test failed, but to also explain the discoveries to the developers.

Maintaining tests took a lot of my time. When developers run tests that fail it becomes more of a
distraction
than confidence in bug prevention.Both groups had to stop what they are doing to figure
out if its the functionality or if it’s the test. We found ourselves spending more time trying to stabilize
the tests than actually testing.
” – Company QA Manager

The Solution

Testim’s solution that uses machine learning for the authoring, execution and maintenance of automated test cases was implemented. With Testim they were able to capture the scenarios in minutes, as well as complement that with JavaScript syntaxes. The team was able to spend more time creating new tests, and validating the status of their application.

Today we have hundreds of tests running on every pull, giving the developers feedback to their code
within minutes. The developers themselves easily update test scenarios so QA can focus on increasing
coverage. We also significantly reduced our cost of quality: The rich information we get allows us to
reduce the time to troubleshoot by 80%.

The Results

As a rapidly growing enterprise, they needed a way to optimize processes through automation. This plays a significant role in their move to agile development. Within a couple months of using Testim, The team was able to create hundreds of their UI tests scenarios. Today, they are proud to say that they are authoring tests in under an hour compared to the three days it was originally taking.

Before Testim it would take 3 days to author a single test in Selenium, now even for the less
experiencedtester, writing tests takes under an hour, developers can update tests on the fly and
figure out where the tests are failing without any additional help.

Now, troubleshooting a failed tests takes a fraction of the time. There is an indication on which step failed, including JS syntaxes, screenshots comparing prior runs, and access to the DOM with detailed error messages.

However, the biggest impact was the reduction on maintenance. Tests are stable and trustworthy so when a test fails the user knows it is either due to a bug or the test requires a change in the flow. The team focused most of its time on increasing coverage knowing that the tests adapt to UI changes by the development team.

Today, we are proud to say that they are fully CI/CD, testing on every code push, and running thousands of tests every day.

Thank you to everyone who participated in our round table discussion on Cracking Your Test Automation Code: The Path to CI/CD. We had a great turnout with lots of solid questions from the audience. If you missed the live event, don’t worry… Just click here to watch the recorded session. There were several questions that we were not able to answer during the live event so I followed up with the panelist afterwards to get their answers.

Q: How do you implement CI/CD without using any tools?
Bas Dijkstra: Without any tools at all it will not be possible to do CI/CD. Maybe technically there are ways to do it, but it will be massively inefficient. You will need tools that help you do version control, building software and deploying it on a specific environment. If we just look at doing CI/CD without testing  tools, that is possible in theory, but you will need to do the required amount of checking by hand anyway to make sure that the software you delivered has a certain level of quality. Doing that manually will very likely be less efficient and act as a bottleneck in the process.

How can I improve backend testing of a mobile app beyond unit testing?
Oren Rubin: You can expand beyond the unit tests by adding API testing. This is also considered integrations tests, as you testing several units as one big unit. Consider that one big state that you need to initial, and the API calls (either REST, SOAP, etc) are like function calls, some getters, some setters which modify the state.

Q: What are the things we need to cover during API tests?
Bas Dijkstra: In short: connectivity (can I talk to the API at all?), syntax (is the data returned formatted correctly, are headers and status codes OK?), semantics (is the data returned the data I expected?), functionality (does the API call have the required side effects (data stored, processes triggered, etc.), as well as no unintended side effects), performance and security (authorization, authentication).

How should one avoid overlapping in functional and unit test automation?
Oren Rubin: It’s easy to overlap the two, but it’s almost unavoidable as each unit is tested on it’s own. They are all tested via the integration tests, which usually doesn’t care about a specific unit but mostly on part of a user story/flow. Each action (e.g. click) can drive a lot of units and check them and the integrations. Remember that E2E tests come at a high price, so choose them wisely.

Is it good to separate manual test cases from automated test cases?
Bas Dijkstra: Yes. Especially since translating your ‘manual’ test cases (where you’ll likely check a number of things in the process) to automated test cases (which should ideally check one thing only) 1-to-1 will likely lead to inefficient automation. Determining what to automate and what to leave for testers is an art in itself, and simply handing someone a batch of manual test cases to automate isn’t likely to give you good results. 

What is the difference between unit test vs. integration tests in regards to front end testing?
Oren Rubin: There is no difference. Unit tests allow you to test a single unit (e.g. a controller of a component) and mock everything else; use spies, stubs, and mocks. An integration test, which might test the entire UI, where you can check unit loading order works the CSS (which is global by definition, if you exclude future shadow dom), and integrations between the different units. If you don’t mock the servers, then it’s actually End-to-End testing. 

About the panelists 

Bas Dijkstra
Bas is an independent test automation professional who has been in the test automation and service virtualization field for over 10 years now, designing and developing test automation and service virtualization solutions that enhance and improve test teams and test processes. Find out more information about Bas on his LinkedIn profile. For questions and more information you can always send him an email at bas@ontestautomation.com or give me a nudge via @_basdijkstra on Twitter.

Oren Rubin
Oren has over 20 years of experience in the software industry, building mostly test-related products for developers at IBM, Wix, Cadence, Applitools, and Testim.io. In addition to being a busy entrepreneur, Oren is a community activist and and the co-organizer of the Selenium-Israel meetup and the Israeli Google Developer Group meetup. He has taught at Technion University, and mentored at the Google Launchpad Accelerator.

Struggling to get answers to these Test Automation questions? Watch the Recorded  Webinar

  • What constitutes an adequate degree of test automation coverage?
  • Does every single function in the application need a corresponding test associated with it or just the core functionality?
  • Does every application need a full end to end automation suite built out or just enough to satisfy QAs and the business? 

Then register for this webinar: Cracking Your Test Automation Code: The path to CI/CD where Bas Dijkstra, Test Automation Consultant, On Test Automation and Oren Rubin, CEO of Testim discuss the Do’s, Don’ts and misconceptions of test automation.

Bring your questions to get answers to:

  • Why your code impacts your approach to testing?
  • What’s the right mix of unit, functional, end-to-end, UI and other types of testing?
  • How to create your test automation strategy and tactical execution plan?
  • What pitfalls to avoid that will increase your cost of quality?

This round-table discussion will include insights from these industry experts:

  • Bas Dijkstra, On Test Automation
  • Oren Rubin, CEO of Testim

About the presenters:

Bas Dijkstra
Bas is an independent test automation professional who has been in the test automation and service virtualization field for over 10 years now, designing and developing test automation and service virtualization solutions that enhance and improve test teams and test processes. Find out more information about Bas on his LinkedIn profile. For questions and more information you can always send him an email at bas@ontestautomation.com or give me a nudge via @_basdijkstra on Twitter.

Oren Rubin
Oren has over 20 years of experience in the software industry, building mostly test-related products for developers at IBM, Wix, Cadence, Applitools, and Testim.io. In addition to being a busy entrepreneur, Oren is a community activist and and the co-organizer of the Selenium-Israel meetup and the Israeli Google Developer Group meetup. He has taught at Technion University, and mentored at the Google Launchpad Accelerator.

Totango is the leading enterprise-grade customer success platform that helps recurring revenue businesses proactively impact business outcomes with customer success. With solutions to empower customer success teams or entire companies, Totango enables every employee to participate in customer success.

The Problem

Like any growing company, Totango was short staffed on engineering resources and each team member owned multiple tasks. Developers were responsible for doing their own testing, mostly focusing on unit level tests while the overlay team supported the first-level testing efforts. While the team supported a variety of different test automation functions, they never had enough time to invest in automating the User Interface level testing. The team considered Selenium and experimented with it.

“We didn’t have the appetite and desire to work with something like Selenium or one of its variants,”
said Oren Raboy, Co-Founder and VP of Engineering for Totango.
“We experimented with it but
realized  
that it’s a steep learning curve and required a lot of effort just to get some basic value out of
building such a system. It was brittle and slow.”

The Solution

Totango believes that everyone on the team should be able to write tests and own code quality. Totango’s R&D team members picked up Testim quickly, using it for front end test automation coverage.

“Testim’s simple user interface makes it easy for our developers to add tests without a steep learning
curve”
says Oren, ”We increased coverage by 30% in 3 months and found that very little maintenance
is required so we can focus on increasing coverage.”

Totango was able to create a UI regression suite that gave the team confidence that things were not breaking while allowing them to add, expand, and grow without investing a ton of resources into their UI test automation. The regression suite is extremely valuable for the app development team as it serves as a safety net they can rely on when they make changes that break things.

Totango is now working to leverage Testim to not only help with code-coverage but run various test scenarios in production, thus giving them an app level production monitoring to mitigate production malfunctions, above & beyond code regressions.

The Results

The engineering team ran for quite some time by doing UI level testing manually. In order to keep up with customer demands and develop additional products to their suite without slowing down their daily releases, the team needed to optimize their frontend testing efforts. Before Testim, Totango would occasionally have a regression at the UI level that forced the team to stop working and go back and fix the issues. Now the team can find the problems quicker and react to them faster, giving them more coverage without slowing down the development pace.

“We had a lot of automated testing, just not at the UI level,” explained Oren. With Testim, we’ve been
able to place quality front and center and make it a company wide priority vs a topic only QA is responsible
for
. The end result is increase in velocity, quality and customer satisfaction.”

 

Testing has been in the spotlight of software development, as organizations continually seek to optimize development cycles. Continuous Delivery and its promises of frequent releases, even as often as hourly, is a big factor driving executives  to find ways  to shave time off of any eligible technical processes. As enterprises embark  on their DevOps transformational journeys, delivering at lightning speed without compromising on quality takes more than cultural changes and process improvements. Test automation is key in helping project teams write and manage the smallest number of test conditions, but get the maximum coverage.

The case for automation: If the scenario is highly quantitative, technology is superior to humans. That’s clearly the case in the current technology landscape, where according to the 2015 OpenSignal report, massive fragmentation has resulted in over 24,000 distinct Android devices that all run on different versions of the Android OS, plus the several variations of iOS devices currently in use. Web is no different. It’s impossible to test every scenario manually, so automation must be leveraged. But therein lies the main point, that automated testing is a tool to be leveraged as needed instead of relied upon to dictate an entire testing strategy.Although logic would deem that if automating a few test scenarios yield fast results, that automating any and every eligible scenario will shorten the test cycle even more; but alas,  that’s usually not how it goes. Automation endeavors take a good deal of planning and forethought in order to add value at the anticipated level. More often than not the overhead of maintenance, especially in CI/CD environments where tests are automatically triggered and one needs to analyze the reports and fix locators, a task that , could  take hours.  This won’t work for organizations who are truly DevOps.  If full stack developers are going to be fully responsible to take the code to production, then they will need tools to automate the process through infrastructure, testing, deployment and monitoring.  This continuous framework enables  weekly , daily and hourly releases.  Leading DevOps organizations like Netflix and Amazon are said to deploy hundreds to thousands times per day.

What’s more, studies reveal that a high percentage of projects utilizing automated testing fall short of the anticipated ROI or fail altogether. These transformations fall short due to their duration, ramp up time, skill-set gap and maintenance overhead.  If the benefits of automated testing aren’t significant enough to mitigate risk then speedy releases could become more of a liability than an asset.

There are varying levels of automation just as there are companies of all different shapes and sizes. Instead of one or the other, it’s more fitting that automated and manual testing are looked at as complementary. Agile and DevOps have created new testing challenges for QA professionals who must meet the requirements for rapid delivery cycles and automated delivery pipelines.

DEMANDS OF CONTINUOUS TESTING

Testing used to have the luxury of having its own phase or at least a set timeframe to stabilize new code prior to pushing to production. No longer. In the era of DevOps, QA must adopt a truly agile mindset and continually be prepared to shift focus to testing new features and functionality as they become available – that could be weekly, daily, or hourly.

From web, mobile and connected devices to IoT, a quick inventory of current technology will reveal a multitude of different devices and literally thousands of potential ways that these technologies can be combined. Across networks, apps, operating systems, browsers and API libraries, the number of  combinations with each requiring its own verification and testing support.  As the push towards Devops continues, the upward trend towards continuous testing is inevitable. As a result, testers and/or developers will need to deliver in brief and rapid iterations.

Beyond automation, what does this look like? QA, Engineering, or whomever is responsible for testing will need to shift left and engage in rounding out user story acceptance criteria to ensure accuracy and completion. In addition, active participation in sprint plannings will help to ensure that user stories are truly decoupled and independent. Or if dependencies are necessary, confirming that story precursors and relationships are in-place. Partnership with Product Owners to confirm that the scope for a given sprint or iteration is realistic.

And finally in support of shifting left, collaboration among Dev and Ops to ensure the necessary support in the event that a code check-in causes a bunch of automated tests to fail. The culprit for these tests will need to be checked manually so it’s important that QA is prepared so as not to pose a blocker for the release. All of these activities is largely what comprises testing outside of automation. It’s a broader role than the traditional QA Specialist that requires initiative and the willingness to embrace an always-on mentality that’s necessary to support DevOps efforts.

SUMMARY

Software development practices have changed and the change brings with it new challenges for the testing arena. The growing need for test automation in addition to the profusion of new testing scenarios will constantly evolve the role of testing and QA.  Yes, the world is changing for testers—in a good way. The days of exhaustive manual testing are certainly numbered, and automated testing is an essential part of the delivery pipeline. But testing has never been more important.

SOURCES

  1. IT Revoloution DevOps Guide
  2. World Quality Report 2016-17
  3. Open Signal: Android Fragmentation

 

Introduction
We work hard to improve the functionality and usability of our platform, constantly adding new features. Our sprints are weekly with minor updates being released sometimes every day, so a lot is added over the course of a month. We share updates by email and social media but wanted to provide a recap of the latest enhancements, summarizing the big and small things we delivered to improve your success with the product. We’d love to hear your feedback.

Test History

Is past performance an indicator of future success? The folks on Wall Street don’t think so… But we do! This is why we are pleased to offer the new Test History feature in Testim.  


The new Test History feature allows users to see slide and dice test results to get more actionable insights.

Why should I care?
This gives users much more insight from the test results. Quickly analyze how many tests ran during the selected time frame, how many succeeded and the average duration of each run, how your tests performs across multiple environments and whether certain environments or scenarios consistently fail. This allows project team to see improvement trends across scenarios or environments over time. Click here to learn more.

What’s included?
Test History allows you to filter based on the following parameters:

  • Run time
  • Specific tests / All tests
  • Last X runs
  • Status
  • Browser

Create Dependencies Between Tests
This new capability allows users to create different aspects of dependencies between tests. As a best practice, we recommend to keep the tests as isolated as possible. However we recognize that sometimes you need the ability to create dependencies between tests.

Why should I care?
Now users can create a logical sequence of tests. By working closely with our customers, we’ve seen project teams required to set a sequence of activities. A general testing framework does not allow you to create a sequence, forcing users to create one long test which may result in performance issues and failures. By creating dependencies during test plans you can create shorter discrete actions, order them in sequence and set dependencies and share data between tests. Click here to learn more.

Setting up Cookies
Cookies is a reserved parameter name for specifying a set of cookies you would like Testim to set for your application.  You can set cookies for each browser session before the test is started or before the page is even loaded. Use cookies to set browser data to be available for your tests. The cookies parameter is set as an array, where each cookie object needs to contain a name and value properties.

Why should I care?
Websites use cookies to personalize user experience. By using this feature, you can test different types of personalized experiences. Click here to learn more.

Improved Scrolling
More flexible way of scrolling through your page.

Why should I care?
Traditionally scrolling in your test automation would be set by pixels so you would test your script to skip 60 pixels. Offering a more flexible scrolling ability makes your tests more adaptive. An  example is to scroll to the element, mouse wheel. Click here to learn more.

How do I get my hands on these hot new features?
Customers have access to these feature now, check it out and let us know what you think. If you’re not a customer, test drive the new features by signing up for a free trial to experience code and maintenance free test automation.

Be smart & save time...
Simply automate.

Get Started Free