Author

testimio

Browsing

Let’s talk about automated testing tools. Are they valuable? While some people seem to think of them as a mere accessory, that couldn’t be further from the truth.

An old teacher of mine used to say that one year in the IT world was actually worth ten years—such was the pace of innovation and progress. After graduating and becoming a software developer, I could see for myself that he was right. To remain competitive in such a fast-evolving landscape, companies around the world must make use of automation, which includes having a sound automated testing strategy in place.

And to design, implement, and maintain said strategy, you’ve got to have the right tools in your toolbelt. Sure, tools aren’t everything. Like in any field, mastering the fundamentals is always going to be vital. As soon as you get the basics out of the way, though, finding and employing the best tools available becomes imperative. And that’s what this post is all about: helping you on the “finding the best tools” part.

By the end of this post, you should have a mature understanding of the current automated testing tools scene. You’ll then be able to make an educated decision about which tool or tools are the best fit for your needs. Let’s get started!

The 6 Automated Testing Tools You Should Know About

We’re now covering six automated testing tools that you need to be aware of. Keep in mind that our list is going to be a little bit different than other lists you see on similar named posts around the web.

Other posts about automated testing tools tend to focus almost exclusively on UI testing tools. This list, on the other hand, presents more variety. We’ll cover tools that fall into several different categories of automated testing—unit testing, acceptance testing, and yes, UI testing.

When you’re done with this article, you should have a good understanding of the available offerings out there to help you with each step of your automated testing strategy.

The xUnit Family of Unit Test Frameworks

Back in 1999, Kent Beck released a unit test framework for Smalltalk. It was called SmalltalkUnit (or simply SUnit), and its architecture inspired an array of frameworks for unit testing on a variety of languages and platforms.

After SUnit, the first framework to appear was JUnit, which was a port of SUnit to the Java language. Since then, many others followed. For instance:

  • NUnit, for the .NET platform
  • PHPUnit, for the PHP language
  • PyUnit, for Python

Unit testing is one of the most basic types of automated tests, so a xUnit test framework is a tool that deserves a place in your toolbox.

PIT Mutation Testing

PIT is a mutation testing framework for Java. That’s as straightforward a definition as it can get. But the definition is useless if you don’t know what “mutation testing” means.

Mutation testing is the process of deliberately adding defects to your codes. Those faults are called “mutations.” After the framework introduces a mutation, it runs your unit tests. If they fail, we say the mutation was killed. Otherwise, the mutation survived.

The “mutations killed/mutations survived” ratio you get is a reliable indicator of the quality of your unit tests suite. Think about it. If all of your tests still pass after you damaged a portion of the production code, that can only mean one of two things—either your current tests are wrong, or there aren’t enough tests.

You could do all of this by hand, sure. But it would be a prolonged, tedious, and error-prone process.

Cucumber

Cucumber is an interesting item on our list since it wasn’t designed to be an automation testing tool from the start. Instead, Cucumber helps teams—including non-technical people—write specifications and use cases for software features, in a clear, unambiguous way.

Cucumber allows teams to write software using the BDD (behavior driven development) approach, by allowing technical and non-technical members of the team to describe software use cases using a common language.

After you create one or more scenarios for the feature you wish to document, Cucumber allows you to generate automated tests from them. You can do that using test automation libraries that your language already supports. In Java, for instance, you could use JUnit.

Keep in mind that the tests you end up with by employing Cucumber are not unit tests. These are higher-level tests—or so-called “executable specifications.” They describe the way each software feature is meant to behave, from the user’s perspective. Such tests verify a functionality from the UI through the database, passing through every layer.

Unit tests, on the other hand, verify that a single unit works in complete isolation. It gives the developer confidence that their code does what they think it does. But it doesn’t check how the units integrate, or how the different layers work in harmony to fulfill the user’s needs.

In other words, Cucumber doesn’t compete with xUnit style framework; rather, it complements them.

Selenium

Selenium is another interesting case in that it is not, per se, an automated testing tool. Instead, it’s a browser automation tool. With it, you can automate interactions with a web browser. What you do with this capability is up to you.

In practice, what most people use Selenium for is indeed testing. By using the tool, they automate interactions with their web applications, effectively creating automated tests for their web UIs.

Selenium—without the aid of Selenium IDE—isn’t a standalone tool. It’s more like an API, against which you can program—using your favorite language or platform—to create the browser automation that you need.

Serenity

Serenity is a Java framework that operates with BDD tools (e.g., Cucumber) while acting as a wrapper on top of Selenium WebDriver.This tool makes it easier to write BDD-style tests and even Selenium tests. It frees you from writing a lot of boilerplate code and it also comes with useful built-in features, allowing you to take screenshots, easily integrating with Jira, parallelizing tests, and much more.

With Serenity, you can create highly detailed reports. The tool can create documentation that you can use both as test results and documentation for your app.

Testim

Finally, take a look at Testim.

Testim offers a straightforward recording function, allowing non-developers to easily create UI tests. It also enables testers toreport a bug and automatically generate a test that reproduces it. As already covered on this blog, similar tools in the past tended to result in fragile tests. Testim is different since its greatest asset is artificial intelligence.

It uses AI to improve the authoring and execution of tests. How does it do that? It analyzes the DOM objects of the page under test, extracting them along with their properties.

Then, based on this analysis, the AI figures out the optimum strategy to locate a given element. What that implies is as simple as it is powerful—even when a developer changes some attribute of the element, the test still works.

This “dynamic locator” approach is the primary advantage of using Testim compared to other frameworks— authoring and executing tests becomes way faster and stable.

Automated Testing Tools in All Shapes and Sizes: Know Them so You Can Master Them

It took us long enough but we—as in the software industry—are now finally employing automation to improve our own work, at all phases of the process. One of the manifestations of this trend is the modern automated testing movement. For a modern software development shop, having a mature automated testing strategy is no longer a luxury. It’s essential.

And a vital step in order to create such a strategy is to know and master the available tools. Today’s post helped you become aware of some of the tools at your disposal. You can also take a look at our five-step process to learn how to identify the best automation platform for your needs.

The tools we’ve covered today fall into many different categories of automated testing, and this is by design.

You now have a high-level view of not only the tools available but also the types of automated testing they support. The next step is to zoom-in and to learn more about the tools and test style that your repertoire is lacking, in order to improve it even more.

Author bio: This post was written by Carlos Schults. Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.

In today’s advancing and fast-paced world, high-speed implementation is a must. This applies to all products and services. Let’s suppose you were using an application and got stuck because of a bug. After reporting the bug, you, of course, expect the team to fix it as soon as possible. If not, your next move is going to be switching to another service.

Customers want quick fixes and faster updates. For that, most software companies are adopting focused and flexible software testing. If you keep the customers waiting, remember, the competitors are just a few clicks away! 

Saving time and resources and streamlining the entire software development lifecycle is important. That’s why most companies are opting for testing on an agile team. Let’s dig a little deeper to understand what this means. 

In this post, I’m going to explain how shifting from a traditional testing environment to an agile one makes a difference in your project’s budget, resource utilization, and time duration. You will learn about scrum testing, what challenges testers face in an agile environment, and how it’s worthwhile in the end. Consequently, you will gain insights into boosting your enterprise production and customer satisfaction.

What Is Scrum?

Scrum is a framework in which teams resolve complex situations while simultaneously delivering products. The quality of the product delivered has to be high even when the issue is complex. When a problem is complicated, it requires an efficient team collaboration. Scrum is lightweight and easy to understand. But it might be a little tough to master. Unlike popular opinion, scrum isn’t a method. It’s a framework. 

A scrum team comprises of the following:

  • The owner of the product
  • Scrum Master
  • A team of developers and testers

The entire concept of scrum rests on ensuring greater flexibility and creativity and optimizing productivity. The teams are cross-functional. This means that they don’t need to be dependent on other teams to accomplish tasks. Since the teams are self-organized, they don’t need direction from those outside the team. 

The Difference Between Agile and Scrum

Agile comprises of principles that guide through the iterative approach for software processes. But there are certain rules that testers have to follow in an agile environment. This set of rules is called scrum. Scrum is a part of an agile framework. Now, let’s take a look at both agile and scrum in detail for better understanding.

Agile

Agile management is a set of methodologies for software development. These methodologies are incremental and iterative. Agile management includes the rational unified process (RUP), extreme programming (XP), and scrum. Also, agile processes result in need and outcome evolution. This evolution in project development methodology is possible because of the collaboration between teams. 

Agile teams are cross-functional and self-organizing. The analysis, documentation, and development of a new project go hand in hand. Advancement occurs with every iteration. This approach offers ease of accommodation of changes. It also results in better scalability. The flexibility of operations and processes increases. 

Scrum

Scrum is a subset of the agile environment. It’s a technique used to address complex issues and deliver high-quality products simultaneously. If an urgent change is required, the team has the flexibility to adjust. Effective collaboration and frequent communication ensure the success of scrum. Moreover, every sprint introduces better practices to increase efficiency. 

How Is Testing in Agile Different From Traditional Testing?

The Software Development Life Cycle (SDLC) needs a robust approach for quick implementations of ideas. Traditional testing was the mainstream, but efficiency increaseFs when an enterprise makes a shift from traditional to agile testing. Let’s look at how agile testing differs from traditional testing to benefit your organization.

Traditional Testing

Traditional testing aims to understand user needs and develop a product. After development, testers test the product and report bugs before deployment. The development team then works on them and fixes any errors using the best possible solution. Traditional testing works on the assumption that the processes are repetitive and predictable. 

The concept is that the team can get the processes in control during the SDLC. A hierarchy ensures stability at different levels. It standardizes procedures by allotting different tasks to people according to their skills. But while the traditional model seems clear, it lacks flexibility. The procedure is time-consuming as the team completes tasks in a fixed sequence. 

Agile Testing

Agile testing seeks to correct the rigidity rampant in traditional testing. It’s a team-based approach but, unlike traditional testing, it’s interactive and dynamic. As a result, a product’s delivery time shortens. The project is divided into time-boxed tasks called sprints. Every single sprint has a fixed duration. Here, we consider processes unpredictable. Also, the processes might seem a little difficult to comprehend at first. The reason is that the tasks don’t have a clear definition.

But the high level of adaptability and flexibility during the process makes it worthwhile. As soon as users request modifications, the agile testing process is quick to adjust to changes. The iterative cycles make changes according to consistent customer communication and feedback.

What Changes for a Tester With Scrum?

During scrum testing, the team has to test a product and see how it turned out from the customer’s point of view. Some of the main events a tester has to attend in an agile environment include:

  • Sprint-planning sessions 
  • Daily standup meetings
  • Sprint retrospectives 

Instead of leaving testing for the last stage, as they would in a traditional test environment, a tester has to carry it out throughout the process. Testers get to learn a lot of new things in addition to testing like development or business analysis. The work culture becomes different. Now, let’s take a look at some things a tester gains exposure to while working in scrum testing.

Clearer Grasp of Business Logic

Testers are exposed to in-depth knowledge of how domain applications work. They have to work closely with the development team. It enables them to create innovative and effective business case scenarios. The familiarity with architectural diagrams and development terminologies increases. Testers need to have crisp business logic so they can hold discussions with business analytics and developers about the application specifications. 

Adopt Automation for a Speedy Testing

Selenium, Appium, UFT, GitLab, Codeship, Jenkins, etc. are some of the tools testers need to get familiar with. To stay ahead in the industry, they have to welcome changes. Speedy testing requires automation. Even though there are some massive changes a tester has to face during automation, it’s a chance to sharpen their skills. Apart from that, automation reduces risks during regression testing.

Adopting services like Testim is a wise move. It offers automated functional testing using artificial intelligence. Additionally, it speeds up execution, authoring, and maintenance during automated tests.

Adhering to SDLC From the Start

In the waterfall model, testers used to stay behind before the testing begins. But in scrum testing, a tester must adhere to SDLC from the beginning of the process. The test windows broaden and collaboration improves with this approach. This way, testers also gain a deep insight into the procedure. So, no test phases are left out. 

Regular Standup Meetings

It’s mandatory for testers to attend regular standup meetings in an agile environment. These meetings last for 15-30 minutes—usually at the beginning of the day. This is when the manager or the scrum master asks every team member about what they did the day before. In addition to that, they also gather insights on the current day’s tasks and possible roadblocks. Having testers at daily standup meetings eliminates hurdles in the initial stages of a project. The entire team, including testers, remains aware of what is going on. It ensures progress in various tasks. 

What a Tester Needs to Know Before Moving to Agile Testing

For someone used to the waterfall model, adapting to the agile environment is a big change. Here are some things a tester needs to know before moving to agile testing. 

Knowledge in Automation Tools

Repetitive tests for each sprint are a waste of time. Testers need to pace up the regression testing. They also need to have knowledge of automation tools to speed up testing. These tools include Selenium WebDriver, HP UFT, and Appium. JUnit, Cucumber, Pytest, JBehave, etc. are some BDD testing and unit testing tools which is good to learn before moving into Agile testing.

Knowledge of Project Management Tools

There was a time when testers used HP Quality Center to track bugs and report them. Slack, JIRA, and Mantis are some of the versatile tools that serve more than one purpose. Besides identifying bugs, they aid in efficient collaboration and project management. 

What Does Testing Look Like in Scrum?

When I was learning about scrum, I was most surprised that the entire testing procedure is divided into four quadrants. Let’s take a look.

Quadrant 1

The first step is to check the quality of the code. The testers give immediate feedback. Then, based on the feedback, the developers carry on with their tasks. These tasks include unit testing and component architecture testing. The former refers to checking a coding unit to see if it fulfills the requirement, which is often done by the developers. The latter is to ensure that the pieces of code work when integrated.

Quadrant 2

Both the testers and developers get the requirement. Both carry out their respective tasks keeping business objectives in mind. That includes testing possible scenarios. Testers have to perform prototype and wireframe testing while keeping user experience in mind. 

Quadrant 3

Automation testing evaluates the product usage. Despite the product development being incomplete, tests are run. The scheduled demos ensure that development is going on according to business goals. These are the five stages the third quadrant covers:

  • Collaborative testing
  • User acceptance testing
  • Exploratory testing
  • Usability testing
  • Pairwise testing

Quadrant 4

Testers test performance, data migration, infrastructure, stress, and load. Other aspects include security to ensure authentication. The product should have preventive measures for hacking and attacks. Scalability is another factor testers keep in mind.

Strategizing Is the Key to Agile Testing

An impeccable strategy is a must to move from traditional testing to agile. There are four stages to this that keep things organized.

Iteration 0

This stage involves the completion of the initial setup work. Establishing a business case, analyzing requirements, and creating use cases are crucial steps. After estimating costs, the team prepares a preliminary project. 

Development Phase

Each sprint in this phase comprises agile practices. Developers verify unit testing and service integration testing. Also, the testers perform agile acceptance testing. The stakeholder testing team and project testing team together execute test cases. 

Product Deployment

The deployment of product into production comprises four steps. The first step is to train the end-users. Following that, creating data backups comes through. After marketing the release, the documentation of system documents and finalized user takes place. 

Production Support

Production support includes regular testing and reporting bugs. If there are any, the production support team fixes them. 

Challenges a Tester Might Face in Agile Testing

Since the methodologies are different in traditional and agile testing, there are many challenges testers have to face. These include: 

  • Sharp deadlines
  • Learning the development procedure and programming languages
  • Sudden changes requested from the stakeholder
  • Impeccable coordination between teams

However, these challenges are nothing when compared to the huge learning opportunity that agile testing offers. And trust me, once you start working in an agile environment, very soon you will be ready to take on all the challenges that come your way. The agile environment will help a lot in propelling your career as a tester.

Conclusion

Making a move from traditional to agile testing can be overwhelming for a tester at first. But once it settles down, a tester’s learning scope broadens. Consider the change as an opportunity for enhancing your skills and professional growth. Once you get the gist of it, nothing can stop you from progressing in the industry. 

Author bio: This post was written by Arnab Roy Chowdhury.Arnab is a UI developer by profession and a blogging enthusiast. He has strong expertise in the latest UI/UX trends, project methodologies, testing, and scripting.

Are you familiar with the term “test automation tool?” If you’re not, you’ve definitely come to the right place. Before I dive into why, I have a small confession to make: as a software developer, I have a strong dislike for the word “coder.”

I mean, sure, we write code in our jobs. But writing code is more a means to an end than the end itself. Instead of “someone who writes code for a living”, a better description of a software developer would be “someone who employs automation to solve problems as efficiently as possible.” I know, now you’re wondering what this seemingly random digression is all about, but it’s going to make sense soon.

Here’s the thing: developers have long ago figured out that, since their (our) jobs consist of automating all kinds of process, they could do the same to the software development processes itself. Why not automate your app’s build process? Why not go a step further and automate the whole packaging of the product? Heck, why not go even further than that and automate the deploy to final users?

Automate All That’s Automatable

The answer to all the questions above was an obvious “Let’s do it!”. Modern software development relies heavily on automation, on all fronts, from analyzing source code looking for mistakes to testing to the already mentioned build, packaging and deploy process. That’s the scenario where a test automation tool becomes relevant. But as this post title already asked, what is a test automation tool? In this post, we’ll answer the title question and more.

First, we’ll give you a broad definition of test automation. Then, we’ll proceed to define a test automation tool, explaining their use cases and how they fit into the whole test automation scenario, making the process easier. Finally, we’ll show you a few different options of test automation tools, equipping you to make an informed decision.

Let’s get started.

Defining Test Automation

We could easily define test automation as “the automation of test-related activities.” It’d be easy, quick and insufficient. Such a definition would be shallow unless we first define automation itself. And what’s automation?

We could define automation as the technique of performing tasks without human intervention. The justification for doing so is to achieve speed and efficiency levels that greatly surpass those of human beings. Also, in most cases, you’ll apply automation to tasks that are repetitive. As such, they can be extremely error-prone when performed by people.

Ok, that’s a passable definition of automation. In light of that definition, let’s now try to rephrase our first, insufficient definition of test automation:

Test automation is the process of performing software testing activities with little or no human interaction, in order to achieve greater speed and efficiency.

When putting a test automation strategy in place, it’s important for you to remember that usually, the automated part is the running of the tests. Before you’re able to execute your test cases, you first have to create them using some process. This might mean writing code. Or it might mean performing a task while using a window and recording it. Test automation is a broad topic. There is a wide range of techniques and—you’ve guessed it—tools at your disposal.

Speaking of tools, that’s exactly what we’re going to cover on the next section.

Defining Test Automation Tool

Since we’ve already defined both “automation” and then “test automation”, it doesn’t look like defining test automation tool is going to be a lot of trouble for us. Here it goes:

A test automation tool is a piece of software that enables people to define software testing tasks, that are afterwards run with as little human interaction as possible.

Again, it’s important to understand that there are a plethora of different types of test automation tools available. They might differ in the types of application they test (web, desktop, mobile), in the way the test cases are set up (by writing code using a scripting language, writing code in a full programming language, recording steps performed using a GUI) in their licenses (free, freemium, commercial) and many other factors.

Meet Some of the Test Automation Tools at Your Disposal

Now that we’re done defining stuff, it’s time for us to start showing you some of the test automation tools available for you. As we’ve said, there are many different types of tools. We’ll try to give you as broad a sample as possible, so you can experiment with the variety of tools available. Let’s get started.

Katalon Studio

Katalon Studio is a test automation tool that enables you to test your web and mobile apps, as well as APIs. This solution makes use of Selenium and Appium engines, offering users an integrated environment for testers to integrate different frameworks and tools.

UFT

UFT is a commercial tool that originally allowed its users to test desktop, web, and mobile apps. Currently, it also offers features for API testing.

Selenium

Selenium is a very well-known tool when it comes to testing automation. It allows its users to write scripts in a lot of different languages, including Java, C#, Python, Perl, and Ruby. This tool also runs in several operating systems and browsers.

The disadvantage of this tool is that to use it effectively, you must spend a non-trivial amount of time building frameworks, libraries and other tools necessary for the actual automation you’re trying to accomplish.

TestComplete

TestComplete also enables Web, mobile, and desktop testing. It offers its users the choice between JavaScript, VBScript, Python, or C++Script to write scripts.

The tool features an object recognition engine that is able to accurately detect dynamic user interface elements, which makes it particularly useful to test apps whose user interfaces change very often.

Testim

Testim is a test automation tool that employs machine learning to help developers with the authoring, execution, and maintenance of automated tests. This tool allows developers to quickly create test cases and execute them on many web and mobile platforms. The tool learns from data with every execution.

Testim then uses all that learning to improve itself, making test cases more stable. The result of that is a robust test suite, that doesn’t break on every code change.

How to Pick the Right Test Automation Tool

As you can see, there are plenty of options for test automation tools for you to choose from. The tools you’ve just learned about are only but a few of the available tools at your disposal. So, how can you choose?

Giving a definitive, one-size-fits-all answer is hard. So what we’re doing instead is to suggest that you choose based on three main factors: target platform, learning curve, and pricing. The first factor should be easy to understand. If your product is a desktop application, then every automation tool that works only for mobile and web are automatically declassified.

The second factor you have to analyze is the learning curve. A given tool might be widely known and used, but if its learning curve is too steep, that might be a bad sign. How much of a problem is a steep learning curve? It depends on how quickly you want your team to be up and running. Maybe it makes sense for your team to take their time learning the tool because of its benefits. I wouldn’t bet on it, usually. But your mileage may vary.

Last, but not least, we have pricing. Again, it doesn’t matter how popular or sophisticated a tool is if its price is way beyond your team/department/division budget for tooling. Many of the tools presented are free or have a free tier, which allows you to at least try them before making a final decision.

So, based on these three factors—platform, learning curve, and pricing—you should be able to choose a tool that is right for your scenario. Weigh every candidate against the three factors and see how many points they score in each area.

That’s a Wrap

In today’s post, we started by defining a bunch of things. Namely: “automation”, “test automation” and finally “test automation tools”. After that, we proceeded to give you a quick overview of some of the test automation tools at your disposal. Finally, instead of giving you a generic, one-size-fits-all answer to the title question, we gave you an easy framework you can use to evaluate the available tools and find the best match for your reality.

Maybe you’ve heard what test automation is but you’re unsure about how to fit it all together in practice. If that’s you, then you feel just like I did when I was first getting my head around how test automation works. So many conversations about testing quickly become abstract and sometimes we simply want to be practical—just show me the code!

If you feel the same way, then you’re in luck! Today we’re going to cover how automated tests work in detail. To do this, we’ll cover the test pyramid and how it relates to testing, which will set a foundation so we can go through and break down each category of testing—from unit tests to integration tests and finally to journey tests. And we did promise to keep things practical, so we’ll talk through some real-world code examples too.

By the end of this post, you should know what the test pyramid is and how it relates to automation testing. You will also learn in detail what unit, integration, and journey tests are and how they’re written.

What Are Automated Tests?

Before we begin, let’s start with a quick definition:

Automated tests perform machine-based assertions on an application to determine the state of the current application against the assertions.

In simple terms, we ask questions of our systems through testing, and the current state of the system (how it’s coded) informs our test results.

Don’t worry if it still feels abstract, as we’ll be breaking it down.

What Is the Test Pyramid?

Before we dive into our examples on each of our testing steps, let’s define them. A lot of the nuance of how automated tests should be written hinges on the theory around test pyramids, so it’s important to have a strong grasp before continuing.

At its heart, the testing pyramid is about feedback loops, and it forms a recommendation on how to structure your tests to achieve the fastest possible feedback loop.

But, what do I mean by feedback loop? I mean the time it takes from modifying some code until you know that it’s wrong (and, importantly, where it’s wrong). The notion of the pyramid is important as it represents how we should heavily load the bottom layer while keeping the top layer thin in order to retain our pyramid shape (more on balancing testing layers soon!).

Test pyramids can come in slightly different formats. For today, let’s use a simplified version, which breaks down into unit, integration, and journey tests.

Unit Tests

Unit tests are the base of our pyramid—they run fast and validate individual functions work as we expect. However, since we’re only testing at the function level, these tests don’t tell us whether our application works as a whole. But importantly, unit tests run blazingly fast. And when they fail, they tell us exactly what went wrong to the precision of a single function. So while unit tests don’t tell us if the application is broken as a user would deem it, they give us the fastest feedback possible.

Integration Tests

Integration tests form the middle of the testing pyramid. If unit tests together sum up to a single software component, then our integration tests ensure that each software component can work with another. Usually, integration tests run over a network boundary of some form. Integration tests essentially validate whether all the aforementioned functions are put together correctly and that other software components can successfully communicate with the component in question. Integration tests are less about what the functions do (that’s our unit tests), but more that our functions are put together correctly.

Journey Tests

These tests form the last layer and the top of our pyramid. Journey tests typically execute a journey as our user would see it. We don’t tend to cover edge cases in journey tests as they should be covered at base levels of the pyramid (either unit or integration).

And journey tests are merely a few, short tests that cover a lot of code. Because we’re writing so few of them and they cover so much code, they give us a lot of bang-for-your-buck for establishing if our application is working as expected since they are covering core journeys. But—frustratingly—they won’t really tell you what is broken (at least, not to the same degree that a unit test would). Additionally, journey tests can be very slow to run and brittle to maintain (but more on why this is later).

Putting the Pyramid Into Practice

Hopefully, by now you have a decent understanding of the test pyramid and how the layers fit together. Don’t worry if you don’t completely understand at this point. We’re going to break down each layer and show you how it works in practice.

Unit Tests: An Example

Unit tests run against individual pieces of code. So let’s take an example.

The following test is written using the Jest unit test library. Jest gives us the “test” and the “expect” function. With these tools, we can then describe the scenarios we want to run for our tests and also our expected output. Tests can have any number of assertions, but typically you’ll have only one (or a small amount). Keeping assertions small means we can fit them to our descriptions easier.

function isOdd(a) {

    if (typeof a != “number”) {

        throw new Error(“Must pass a number”);

    }

    return a % 2 != 0

}

test(“Passing an odd number returns true”, () => {

    expect(isOdd(1)).toBe(true);

});

test(“Passing an even number returns false”, () => {

    expect(isOdd(1)).toBe(true);

});

test(“Passing a string throws error”, () => {

    expect(() => isOdd(“Hello”)).toThrow(“Must pass a number”);

});

Here we can see that we’ve written:

  • A happy path test
  • An unhappy path test
  • A dynamic type checking and error handling test

The test coverage is therefore 100 percent. However, we may want to write additional cases in the future for edge cases if we believe they add more value. For instance: what should the function do if we pass superfluous arguments? And what about edge cases? Negative numbers? Passing a zero value? Or passing a very large value? All of these are valid tests, but adding more tests has diminishing value over time. If we’re incredibly thorough with our testing at all levels, we likely won’t ship any code. It’s a balance. Choosing whether to add these additional cases will depend on factors such as: how much business impact would there be if the code were faulty? Are there other tests you could write that deliver more value?

Integration Tests: An Example

Building on top of our unit tests we have our integration tests. Below is an example of a test written with SuperTest. SuperTest is a small helper library around Jest (which we used for the unit test example). SuperTest gives us the request functionality (and its chained assertion methods).

The following code takes an endpoint, in our case “/user” makes a request to it, asserting the response to be a 200 success. SuperTest also allows us to simply test HTTP endpoints. Remember when we said integration tests often happen over a network boundary? This is precisely what we meant. We tell our test what our setup criteria are, and we can then assert that we received the correct response, error code, headers, etc.

describe(‘GET /user’, function() {

  it(‘responds with json’, function(done) {

    request(app)

      .get(‘/user’)

      .set(‘Accept’, ‘application/json’)

      .expect(‘Content-Type’, /json/)

      .expect(200, done);

  });

});

But you might be thinking: in the above example are we simply testing that the endpoint responds? Not that it has expected values?

At first, this may seem like a low-value test. However, since we have already covered a lot of the functional testing in our unit tests at the integration test level, we are really only testing that our application links up all of those functions and responds without errors. Of course, we can write more detailed scenarios, but we want to be careful that we’re not replicating unit test code in integration tests. Unit tests run faster, so we should ideally write our tests there when possible.

Journey Tests: An Example

Last up is the journey test. The following is an example of a journey test using the Cypress testing framework. Cypress gives us a test runner and an assertion library. The “describe”, “it”, and “cy” commands all come from the Cypress library.

describe(“When at checkout”, () => {

    it(“Can buy”, () => {

        cy

            .get(“.buy”)

            .click();

        cy

            .get(“.summary”)

            .should(“have.text”, “Thanks for purchasing!”);

    });

});

In our journey test, we are testing the physical interface of the application, which should not only include what the user sees, but also behave as a user would when conducting a task (we call this a journey). In our example, we are navigating to our route, finding a button by its class name, clicking it, and then asserting that we can see our summary.

Can you spot why we said these tests could be brittle? For instance, if a developer decides to change an element class name (which in theory might not break functionality!), this change may result in a test failure. Oh no! We can use clever ways of structuring our code to reduce the fragility of our tests, but we cannot remove the brittleness completely—it’s inherent in the nature of the journey test.

The End of Our Tour of Automated Testing

And that concludes our foray into automated testing today. I hope this post gave you a clearer picture of how automated testing works in practice. We’ve seen how we need to have a balanced pyramid and also some examples of how our pyramid could start to be implemented.

In the real world, automated testing requires constant collaboration communication and continual improvement of our codebase. As we find bugs in our code, we work to ensure our coverage is updated. But in updating our codebase we need to ensure we balance our pyramid to keep our feedback fast.

It’s not easy. But, do it right and automated testing yields incredible results that are far superior to their manual testing counterparts. Learn how to identify the best automation platform for your needs with our simple, five step process. 

And remember, the best way to learn is to get your hands dirty and have a go! So take the examples and attempt to implement your own small application with each layer of the test pyramid!

Author bio: This post was written by Lou Bichard. Lou is a JavaScript full stack engineer with a passion for culture, approach, and delivery. He believes the best products emerge from high performing teams and practices. Lou is a fan and advocate of old-school lean and systems thinking, XP, continuous delivery, and DevOps.

Regression testing is a specific form of testing that verifies whether a given piece of software suffered regressions after undergoing changes. “Regression” here means “going back to a previous undesired state.” So automated regression testing is nothing more than the process of automatically verifying that the application has not regressed to a previous undesired state.

Ideally, regression testing—whether automated or not—should be performed every time a software application is changed in some way, whether by receiving a new feature, an improvement, or a bug fix.

In today’s post, we’ll define and explain what automated regression testing means. We’ll start by explaining what a regression is, how costly they are for software teams, and why you should employ regression testing to avoid them. Since manual regression testing would be both time-consuming and error-prone, we’ll then proceed to cover the need for automation, offering advice and tips on how to actually implement the technique for your teams.

Let’s get started!

Automated Regression Testing: Let’s Come Up With a Definition

Before we dive into the “how” of automated regression testing, we need to clarify its “what” and “why.” What is it, after all? What does automated regression testing mean, and why should we use it?

This section will answer the questions above and more.

What Does “Regression” Mean in “Automated Regression Testing”?

The first step in order to understand “automated regression testing” is to get rid of the “automated” part, at least for now. First, we need to understand what “X” means, then “Automated X” will be self-evident. Or, at the very worst, easier to grok.

If we want to understand what “regression testing” means, the first step is to define “regression.” In this context, is “regression” good or bad? Should we welcome it or fear it? The answer to the questions above is straightforward. Here, regression means essentially the same thing it means in everyday conversation—to go back to a previous state. So, in the context of software development, we say we’ve got a regression when our application unintentionally reverted to a previous state.

Why Do We Need Regression Testing?

As we’ve just defined in the previous section, “regression” means an unintentional reversion to a previous state. However, something really important was left implicit in the definition: software regression, more often than not, means the application went back to a bad previous state.

Have you ever had the experience of making some existing feature stop working after implementing a new one? What about that nasty bug returning months after you thought you had completely eliminated it? These are examples of regression. The application has regressed to a previous inferior state.

Regression Testing Is Vital in Software

The field of software development is disproportionately prone to regression problems. Each addition or change made by any developer has the potential to cause unexpected problems in areas (supposedly) unrelated to the spot where it was performed. Any non-trivial software project, maintained by a team with more than, say, five people, has an incredibly high number of potential regressions during each release.

Developers should always keep in mind that all the changes they make, no matter how small, simple, or insignificant they seem, have the potential to cause surprising side effects. They can break functionalities that don’t have anything to do with the changes being made. By performing regression tests, the developer checks that not only does their change behave as it’s supposed to, but also that it plays well with all of the code that was written up until that point.

That’s where regression testing comes in handy. Regression testing is nothing more than the execution (partial or total) of a test suite in order to verify that a given application hasn’t returned to a previous undesired state. If manually done, though, regression testing can be extremely time-consuming and error-prone, which leads us to our next point.

Regression Testing: Why Automate It?

I often say that everything that’s automatable should be automated. Meaning, if you can automate a process, then you probably should do so. If you can automate a process but you’re still doing it manually, I’m afraid you’re leaving money on the table. Chances are the manual process is slow, time-consuming, tedious, and error-prone. That implies you’re losing money in at least three different ways.

For starters, you waste money by having well-paid professional performing tasks that could be automated. Then, there’s always the opportunity cost. The people who are performing the tests could be doing more valuable tasks. Such tasks could have the potential to generate way more value. Finally, since the manual process is error-prone, then people are bound to make mistakes, which will result in losses.

Automated Regression Testing: How to Put It Into Practice

Now that we’re done defining concepts, it’s time for some practical tips on how to actually implement regression testing. There’s good news. If you already write some kind of automated tests for your application—e.g. unit tests—then you’re already performing regression testing without even knowing about it.

Regression testing is not a “new” category of automated tests. It’s not an alternative to unit tests or integration tests. On the contrary, your automated tests —unit tests, integration tests, and similar—written from day one in the project can and should act as regression tests. After each new change to the codebase,  just re-run all of the relevant tests on the suite to ensure they’re not failing.

Automated Regression Testing Best Practices

Additionally, there are some best practices you should implement when adopting regression testing, regardless of the specifics of your implementation:

  • Adopt a Test Management Software: This is vital for all projects that are larger than a one-developer toy project. You’ll have, realistically, way more tests cases than you’d be able to track and manage individually. Luckily for you, there are lots of test management tools at your disposal.
  • Keep a Testing Schedule: You should maintain a strict testing schedule throughout the entire project. This will ensure the final project is thoroughly tested. Additionally, the schedule will encourage the team to adapt to a frequent testing regimen.
  • Write a New Failing Test for Every New Bug Found: Imagine your code has an unequivocal, reproducible bug. But all of your tests are passing. That means that either the current tests are wrong, or your test suite is lacking tests. If you find yourself in this scenario, write a new failing test to document the bug.
  • Categorize All of Your Tests:  You should split your test suite into smaller categories. Your test management tool will most likely provide you with the ability to categorize your tests. That way, your team members can easily identify each kind of test.

Automated Regression Testing: Put It to Work for Your Team ASAP

Software development is a creative endeavor like no other, but the path to reaping its rewards is full of risks. One of the dangers when writing code is to break existing functionality while adding new ones. Even when fixing bugs, we can make older ones—that we thought were dead—come back to life. Each small and seemingly insignificant change poses the risk of a dreaded regression.

If you want to get rid of regressions—and you should want to, believe me—then a suite of regression tests is the solution you seek.

Author bio: This post was written by Carlos Schults.Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.

In a world where technology moves at breakneck speed, most of us know the importance of keeping pace. That’s why traditional software development models are rapidly giving way to methodologies that are more dynamic.

DevOps is a partnership that emerged from this climate. The DevOps approach emphasizes communication, collaboration, and integration. And it does that by bringing your developers and your operations folks together.

But the term DevOps includes only development and operations in its name, and the reality is that we rely heavily on a third entity as well. To deliver software that’s up to today’s standards, you need to consider your testers an equal part of that equation. And so that’s what we’ll talk about today—why the term DevTestOps is a more fair and accurate representation of how companies deliver great software.

A Brief History

With traditional development models, software releases consisted of a series of handoffs. Specifically, developers coded to technical specifications and then passed their code to QA to test. Then, there was a final handoff to operations, who pushed the code into the production environment.

Now, you might notice that QA—the integral step between development and operations—isn’t reflected in the DevOps title. This is a problem. One of the primary components of DevOps is continuous delivery. And for continuous delivery to be successful, you need a substantial amount of testing.

Now, don’t get me wrong. When the tech industry shifted toward DevOps, it was a great thing. In fact, key findings from the 2016 State of DevOps report show the measurable benefits reported by organizations that have successfully incorporated DevOps:

  • Project teams deploy 200 times more frequently.
  • Teams spend 22% less time on unplanned work and fixes.
  • They also spend 29% more time on new features or code.
  • And finally, teams spend 50% less time on security issues.

The report says it best: “By better integrating information security objectives into daily work, teams achieve higher levels of IT performance and build more secure systems.”

Since many companies are eager to have results like these, we’re seeing an increase in DevOps adoption. After all, who doesn’t want the benefits of an accelerated development cycle? But for your software to be resilient, testers need to ensure that quality is built into the final product.

As you can see, DevOps, as it stands, is missing a key component.

Testers Pull Their Weight

If you’re not convinced that testers have earned their place among development and operations, consider this.

More data than ever is traversing the networks. According to DOMO, there’s an estimated 2.5 quintillion bytes of data created daily. And the world is releasing products at a pace that renders manual testing ineffective.

With all that’s going on, automation is essential. Testing such large amounts of data at fast-paced intervals necessitates it.

If you do smoke/regression testing, performance testing, and static tests, automation does the bulk of the work. That means more time for manual testing when there are subjective factors at play. We all know that sometimes you need human judgment.

However, this may prove to be a tall order. Over 70% of smaller companies have very little automation.

That means that the folks responsible for testing must deal with the challenge of test orchestration and automation on their own—either that or they must try to bear the entire burden with manual testing. Not only must QA teams ensure that code changes are fully functional and integrate well but they also must see that frequent releases don’t render the product unstable.

Yes, your developers and your operations folks are critical. But testers bear equal responsibility for the quality of the software you put out. With all these challenges, it would be hard to argue they don’t pull their weight just as much as the “dev” and “ops” in DevOps do.

Testers Help Break Down Silos, Too

Beyond logging defects, testers in environments work cross-functionally. Sound familiar? It’s what DevOps is all about.

Testers share the spirit of collaboration that defines the DevOps movement. They work to ensure that the entire system, not just an individual feature, is operating at an optimal level. From verifying app performance to validating output results from APIs, testing in continuous development environments covers a broad range of tasks.

And with the added responsibility comes greater visibility. Software testing is no longer limited to that of a supporting role. These folks are partners. QA and testing teams have come to the forefront as key players, in line with development and engineering. They’re breaking down silos as much as anyone else in a DevOps organization.

Based on this type of environment, you can see why the term DevTestOps is more fitting. The trifecta of development, testing, and operations are already working together as equals. In fact, just as the word DevTestOps blends three words into one, this approach blurs the lines between roles and responsibilities. And to a large extent, it also eliminates the need for those previously discussed handoffs—for developers to hand off to QA and then for QA to hand off to operations—in order to promote code into production. Instead, each discipline opens its communication channels and collaborates as one team towards a common goal.

One Final Reason to Put the Test in DevTestOps: Money

We’ve seen plenty of good reasons to call things like they are and adopt the term DevTestOps. The combination of development, testing, and operations brings about many benefits. But let’s get to the heart of why companies would want to make the shift in thinking.

The “test” in DevTestOps adds business value across the board, from rapid delivery to improved security. And herein lies an important point: revenue and profits rely on speedy development. So it’s imperative that you head off problems before they become major events. App crashes must be mitigated. Bugs need to be caught early. And we want the end-to-end user experience to be error-free.

QA is an essential part of making sure this is the case. You have to test in order to save yourself from costly problems like bugs. And the earlier in the development cycle you catch a bug, the cheaper it is to fix it.

So let’s give “test” the weight it deserves. Let’s move to the next step in DevOps evolution: DevTestOps. If we do that, it means we have our priorities in the right place, and we’re going to have more profitable businesses.

If we’re talking convincing reasons, this should be the most convincing one of all.

Moving Into the Future

Testers and QA teams champion their cause, influencing processes and decisions project-wide. A business’s concerns aren’t just matters for development and operations. As key players in continuous delivery, QA organizations have seized the opportunity to evolve, taking testing from error detection to error prevention. Now, it’s our turn to evolve from DevOps to DevTestOps organizations, acknowledging their equal importance in creating great software.

“Automated testing” used to be a phrase that caused developers to scratch their heads. Now, it seems to me that most developers are at least familiar with the notion of automated testing. Fortunately, many devs go beyond familiarity. They turn theoretic notion into practical action by actually employing automated testing on a day-to-day basis. And nowadays there’s immense variety in the types of automated tests that can help you achieve higher software quality. In today’s post, I want to focus on a very specific type of automated testing—UI testing.

We’ll start by briefly defining what UI testing is, and then explain how it can benefit you. Then we’ll proceed to show you, step-by-step, how to get started with UI testing using Testim.

Let’s get started.

UI Testing: What It Is and Why Do You Need It

As you surely know, UI stands for “user interface.” So, defining UI testing doesn’t sound that hard, at first sight—it’s just exercising the application through its user interface. But, is that all there is to it?

Actually, it’s not. First of all, some people seem to think that UI testing isn’t necessary if you already have other types of testing in place, such as unit tests. That couldn’t be farther from the truth. UI testing is not a replacement for other types of testing—instead, it’s a compliment. Unit testing exercises the application’s units in isolation. It’s an important verification to have, but it’s not enough. It’s also important to verify how those units fit together. You have to check how the whole system behaves, from the UI, passing through each layer and then back. So, UI testing can be thought of as more of an “end-to-end” type of testing.

Other people might think that, while UI testing has some importance, you can have one or more developers just click around and see if the application behaves correctly. But that isn’t true. Manual UI testing might work for very small and simple applications. But when it comes to applications with complex business rules and rich user interfaces, manually testing quickly becomes insufficient. It is a tedious, error-prone, expensive, and hard-to-scale process.

At this point, automated UI testing is the best choice.

UI Testing Guide: How To Get Started

It’s time to get your hands dirty. We’re going to offer you a brief guide on how to get started with UI testing. It’ll by no means be an exhaustive tutorial, but it should teach you to create a basic approach upon which you can build and evolve into a more sophisticated approach later.

For this guide, we’re going to use Testim to test a very simple web app. Let’s get started.

An Example of Manual UI Testing

Before we start creating our tests, we need something to test, right? So, let’s create a toy application. It’s going to be a super simple web app, with just two controls—an input box and a button. It will allow the user to input numbers, separated by a comma. After clicking on the button, the application will add all the numbers and display the results. If the user types just one number, the result should be that number. If the input box is left empty, the result should be zero. Simple enough, right?

Creating a Toy Application

Ok, let’s get to it. This application will consist of a single HTML page with a little bit of Javascript. Nothing too fancy. Open up your favorite text editor, create a new file, and add the following content to it:

<!DOCTYPE html>

<html>

<head>

<title>String Calculator</title>

<style type=”text/css”>

label { display:block; margin: 10px;}

</style>

</head>

<body>

<h1>String Calculator</h1>

<form id=”form1″>

<label for=”numbers”>Numbers: <input type=”text” id=”numbers” name=”numbers”/></label>

<label for=”addButton”><input type=”submit” value=”Add” id=”addButton”/>

</form>

<script type=”text/javascript”>

document.getElementById(“addButton”).addEventListener(‘click’, function(){

    var text = document.getElementById(“numbers”).value.trim();

    if (!text) {

        text = ‘0’;

    }    

    var integers = text

        .split(“,”)

        .map(x => parseInt(x));

    var result = integers.reduce(((a, b) => a + b), 0);

    alert(result);

});

</script>

</body>

</html>

After that, save the file with the “.html” extension. Then, double-click the newly saved file. You should see your default browser showing you something like this:

Performing Manual UI Testing

Now let’s do some (manual) UI testing. Click on the input box, type 0 and click on the Add button. You should see this:

Now, redo the test, but leave the input box empty. The result should also be zero:

Repeat the test a few more times, always using a single number. The result should always be the number itself. Then, perform some tests with more than one number, separated by a comma. The result should be the sum of the numbers, as the following example shows:

After this brief session of manual UI testing, I’d be convinced that our little toy app works as intended. But, is that testing enough? For this example application, I’d say yeah, probably. But keep in mind that this silly string calculator is just a placeholder for a much more complex application, closer to the ones you’re likely working on in real life.

Getting Started With Testim

Now it’s time to show you how to get started with Testim so you can see what automated UI testing looks like.

First Steps: Creating an Account and Installing the Testim Browser Extension

First, go to https://testim.io and  click on “Start Trial.” You’ll be asked to sign up for an account. You can do that using your Google or GitHub account. If neither option is OK for you, sign up with an e-mail address and create a password.

After successfully logging in, you’ll see a screen like the following image:

If you click on “Create test” button, you’ll see a message asking you to install the Testim Chrome extension:

After downloading and installing the extension, you’ll be ready to start creating automated tests. This time, though, instead of testing our silly little string calculator, we’ll test a real one—the Google calculator.

Creating Your First Test With Testim

On Chrome, go to the address bar, type “calculator” and hit enter. You should see this:

Just a good old calculator. Now, we’re going to create a test for the “+” button.  While still on the same tab, click on the Testim extension’s icon, as shown in the following image:

After clicking on the extension icon, you should see the following menu:

Click on “CREATE AUTOMATED TEST.” That will open Testim and attach the current tab to it. Then, you’ll return to the calculator tab and see the following message:

What you have to do now is very simple—just use the application and your actions will be recorded. Let’s create a test for the “2 + 2” sum. Perform the sum as you would normally do it. After clicking on the “equals to” button, go back to the tab where Testim is open. You should now see this:

Click on the “SAVE” button. You’ll then see a window asking you for a name and a description for your test:

For the name, enter “Add test”. On “Description”, type “Verify the Add operation” Then click on “OK.” Congratulations, you’ve just created your first test with Testim!

Running Your Test

You’ve just created your first automated UI test using Testim. That’s nice, but how do you actually run the test?

On the left panel, click on the “Automate” button, like in the following image:

After doing so, you’ll see your test list which contains only the test you’ve just created:

To run the test, just mark its checkbox and click on the “play” button:

After clicking on the “Run Selected Tests” button, you’ll see a message warning you to avoid touching the mouse and the keyboard during the test execution:

After clicking on “OK”,  Testim will open up a new Chrome tab and run the test. Testim will execute the exact same steps you’ve recorded. After Testim finishes running the test, it’ll close the window.  Go back to the Testim tab and see the test result:

You’ve successfully run your first test using Testim. Congratulations! But…isn’t there something missing?

Adding a Verification

You’ve probably noticed that the test you’ve just created doesn’t really test anything. Yeah, the test passes but that doesn’t really mean much. In terms of unit testing, it’d be the same as having a test method with no assertions.

Our test lacks something, and that something is a verification. We need our test to check that the result of “2 + 2” is indeed four. How do we do that?

On the left panel, click on the “Automate” button. You’ll go back to the test listing. By clicking on the test name, you’ll reopen the test editor so you can change it:

You can see that the test has six steps, and the last step is clicking on the “=” sign. What we have to do now is add an extra step, whose goal is to perform the validation. To add a new step, click on the “+” button after the sixth step. You’ll see the following options:

Click on the third option and you’ll see the menu change to this:

Now, click on “Validate element text.” Testim will take you back to the calculator tab. There, click on the result field:

After doing that, go back to Testim. Now, you should see that your test has a new step:

Hover over the newly added step and click on the engine icon to show the properties panel:

Now it’s just a matter of editing the “Expected value” field, setting it to ‘4’. After doing that, click on the “SAVE” button again. You’ll see a box prompting you for a message, explaining your change. Type “Add validation step” and click on OK:

If you run the test again after saving it, it will continue to pass. However, Testim is now comparing the expected result to the actual result, just like a unit test assertion does.

UI Testing Goes Way Beyond “Clicking Around”

There are still some misconceptions about UI testing out there. One of them is that UI testing isn’t needed if you already employ different categories of testing. Another myth is the belief that informal manual UI testing—a.k.a. just clicking around—is enough.

Neither of those beliefs is true.

We hope this post has shown you not only that UI testing is indeed necessary, but also that it isn’t hard getting started with it, especially if you employ a tool like Testim.

Thanks for reading, and see you next time!

Author bio: This post was written by Carlos Schults. Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.

Today’s post is going to be all about automated functional testing. What’s this category of tests really about? Why do you need it? How do you go about implementing it? What are the challenges you might face when doing so? With this post, we’re going to answer these questions—and more.

We’ll start out by defining functional testing. Then, we’ll go on to explain the benefits of functional testing so you understand why you need it, even if you already employ other types of tests. After that, we’ll reach the “automated” part of “automated functional testing.” We’ll define automation, explaining how it’s imperative to apply it to testing. Then, we’ll be ready to cover automated functional testing.

With the definitions out of the way, it’s time to get to more specific advice. You’ll see how to get started with automated functional testing, the possible challenges on your journey, and how you can beat them.

Functional Testing: What It Is and Why You Need It

Functional testing, as its name implies, concerns itself with how a given application functions. Does the application meet its functional requirements? Does it solve the problems of its users? These are the kind of questions that should be answered by functional test cases.

Functional testing is, then, a type of black-box testing. In other words, functional testing couldn’t care less whether you’ve used an array or a list to model a given concept. The only thing that matters is that the application’s external behavior meets the requirements.

Unit Tests Are Not Enough

In some software development cycles, there are people who seem to think of unit tests as the ultimate software panacea, the mythical silver bullet that will solve all software crises for good.

There’s no such thing, of course.

Make no mistake: unit tests are awesome. They can either lead to a cleaner, low-coupled design or—in the worst scenario—make you acutely aware of poorly designed, high-coupled code. They often cause developers to learn about good development practices, such as immutability, purity of functions, and separation between logic and presentation. More importantly than all of that, a comprehensive suite of unit tests empower developers to fearlessly refactor, knowing they have a safety net if they break something.

Did you notice who is missing from the previous paragraph? The user.

Let’s Think of the Final Users

The previous section covered some of the benefits of unit testing. You’ll notice, though, that those benefits seem to benefit mainly the developers themselves. Sure, cleaner code and a more maintainable application will indirectly benefit users, no doubt about that. But that’s not enough.

There’s a school of thought that argues that “unit testing” is an unfortunate name for an otherwise great tool, and I agree with that. I’ve seen people use “executable specifications” as an alternative name. The point is this: unit tests aren’t really tests. They’re verifications that developers use in order to check that the code they wrote does what they think it does.

But who’s checking whether the code they wrote does what the user expects it to do? That’s the job of functional testing.

It is possible to write a correct, elegant and efficient solution…to the wrong problem. While unit tests will make you confident you’re solving the problem correctly, functional testing is what ensures that you solve the right problem, for the right users.

Automated Functional Testing: Understanding Its Benefits

Up until now, you’ve seen what functional testing is and why you need it, even if you already use other, also very useful, types of tests. Now it’s finally time to add “automation” into the mix. We’re going to define automation and show why its use is imperative in testing. We’ll then zoom in and talk specifically about automating functional tests.

Automation Is Great. Are We Using It?

As developers—and software development professionals in general—we can consider that automation is the whole raison d’être of our jobs.  Why shouldn’t we use it to improve the efficiency of the software development process itself?

Sure, programmers have been automating aspects of their job since the dawn of software engineering. But in recent years, the industry’s use of automation increased to unprecedented scales. We can see this reflected in the growing use of automated testing and, more generally, in the DevOps movement.

Automation in Testing Is a Must

If there’s one area in the whole software development process that really benefits from automation, that area is testing. That’s due to one of the most unique characteristics of software—even the smallest change in one area of an application can cause unforeseen consequences in a totally different area. Yep, that’s a serious problem. In an ideal application, with sound architecture, something like that should never happen. We don’t live in the ideal world, unfortunately, and things like that do happen frequently.

The implication of this unique software characteristic is crucial. After making some change on a codebase, the only way to be sure the whole application still works properly is to retest the entire app. Just testing your own change isn’t enough. You could’ve broken something and you’d never know.

That means that, in modern software development, applying automation to testing is imperative.

Automated Functional Testing: You Can Have Your Cake and Eat It Too

Automated functional testing is, in a way, the best of both worlds—it brings to the table the benefits that automation provides to all kinds of tests, along with the functional testing ethos of putting the user’s needs first. It takes automated testing to its full potential, making it not only a tool for developer confidence (which is already awesome in itself), but also a tool for customer satisfaction. It bridges the gap between QA professionals and developers, creating a shared space where they can collaborate.

Automated functional testing sounds awesome—and it is—but how do you actually get started with it? I’m glad you asked.

The first step is to have specs. You can’t have tests that verify whether the application conforms to the specifications if there aren’t any. For this first moment, it shouldn’t matter that much whether the specifications are embedded into the code or recorded in an external document. The important thing is that you have them.

Then, it’s time to start thinking about the possible test cases for your application. Start your app, go to its main screen, and look at it critically. What are the tasks the user is supposed to perform here? How could they go wrong? It’s even better if you don’t do this on your own. Instead, bring in some coworkers to help. Do a brainstorm session on the current screen. Write down all of the ideas that you can come up with.

Then do the same with other areas of the application.

After all of this, do you have a reasonably sized list of possible use cases? Great, now it’s time to go shopping for tools. This post is meant to be platform agnostic, so we won’t actually recommend particular testing frameworks. But it doesn’t matter your language or tech stack of choice, rest assured, there are automated functional testing tools out there for you.

Automated Functional Testing Challenges: What They Are and How To Overcome Them

As promised, now it’s finally time to cover some of the common challenges when implementing automated functional testing. What are those? How do you beat them? Keep reading to learn more.

QA Skepticism

One of the most common challenges when implementing all kinds of automated testing is skepticism. It’s 2019 and there are still many professionals that are unfamiliar with automated testing, believe it or not. There are also others who know about automated testing but don’t quite buy into the usefulness. How do you deal with the skeptics? Education.

You must use every kind of material at your disposal—books, online courses, on-site training, and what have you. After some training, make your QA people get their hands dirty. Have them start creating automated functional tests, first on a very small scale. Then, start to gradually increase said scale. In true Agile fashion, use small steps, reviewing your results in short iterations. Rinse and repeat. That’s pretty much it.

Developers Skepticism

When it comes to developers, it’s possible that some of them have jumped on the “unit testing = software panacea” bandwagon. If that’s the case, make them get off the bandwagon. The tool for doing so? You guessed it, education. Hands-on education.

Have your developers collaborate with QA people. Create small, cross-functional teams on which QA and developers can work side by side. As automation experts, developers might be of incredible help, mentoring QA people on the nuances of automated testing frameworks and tools.

This approach will create trust and empathy between developers and QA professionals, bridging an otherwise insurmountable gap, and destroying every trace of skepticism toward automated functional testing on both sides.

Maintenance Burden

Another challenge of automated functional testing—and other types of automated testing—is the maintenance burden. You see, test code is code, after all. It can contain bugs. It might be unreadable. And the list goes on.

How do you overcome this challenge? First of all, education again. Educate yourself and your coworkers on the best practices for writing tests, be them functional or otherwise. Read about ways to keep your test code clean and readable. Learn about the common mistakes people make when writing tests and what you should do to avoid them.

Finally, there are tools that can ease your burden by providing you with test management capabilities that you would otherwise not have, such as comparing versions of the same test, scheduling tests executions, and automating bug reporting.

Automated Functional Testing Helps You Solve the Right Problems at the Right Speed

Software development is a field that evolves at breakneck speed. In order to keep up with such rapid change, companies rely more and more on automation, giving birth to approaches such as DevOps and continuous delivery.

Automated testing plays a crucial role in this scenario. It’s unfeasible for teams to completely retest their application by hand each time someone has made a change. Automated testing is way past the point of being something that’s “nice to have”—software development in the 21st century requires it.

Automated functional testing, in particular, brings the benefits of automation to a type of test that puts the user front and center. It bridges the gap between QA and developers, ensures the application meets the user’s needs, and takes automated testing to its full potential.

Author bio: This post was written by Carlos Schults.Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.

There are two kinds of testing in the world of software—manual and automated. Some types of manual testing, such as discovery testing and usability testing, are invaluable. You can do other kinds of testing—like regression testing and functional testing—manually, but it’s a fairly wasteful practice for humans to keep doing the same thing over and over again. It’s these kinds of repetitive tests that lend themselves to test automation.

Test automation is the practice of running tests automatically, managing test data, and utilizing results to improve software quality. It’s primarily a quality assurance measure, but its activities involve the commitment of the entire software production team. From business analysts to developers and DevOps engineers, getting the most out of test automation takes the inclusion of everyone.

This post will give you a high-level understanding of what test automation is all about. There are all kinds of tests, but not all should be automated; therefore, let’s start with general criteria for test automation.

Criteria for Automation

A test needs to meet some criteria in order to be automated—otherwise, it might end up costing more than it saves. After all, one major goal of automation is to save time, effort, and money. Here are some general criteria for test automation. These are starting points, mind you. Your criteria may differ depending on your circumstances.

Repeatable

The test must be repeatable. There’s no sense in automating a test that can only be run once. A repeatable test has the following three steps:

  1. Set up the test, including data and environment.
  2. Execute the function and measure the result.
  3. Clean up the data and environment.

In the first step, we want to be able to put the environment into a consistent state. In other words, if we have a test for attempting to add an existing user, we need to make sure the user exists before performing the test. Once the test is complete, the environment should be returned to the base state.

Determinant

When a function is determinant, it means that the outcome is the same every time it’s run with the same input. The same is true of tests that can be automated. For example, say we want to test an addition function. We know that 1 + 1 = 2 and that 394.19 + 5.81 = 400.00. Addition is a determinant function.

Software, on the other hand, may use such a high number of variable inputs that it’s difficult to have the same result over time. Some variables may even be random, which may make it difficult to determine the specific outcome. Software design can compensate for this by allowing for test inputs through a test harness.

Other features of an application may be additive; for example, creating a new user would add to the number of users. At least when we add a user we know that the number of users should only grow by one. However, running tests in parallel may cause unexpected results. Isolation can prevent this kind of false positive.

Unopinionated

You cannot automate matters of opinion. This is where usability testing, beta testing, and so forth really shine. User feedback is important, but it just can’t be automated … sorry!

Types of Automated Tests

There are so many types of tests, many of which can be automated, that we can’t really get them all into one post. But here are enough to give you a good starting point.

Code Analysis

There are actually many different types of code analysis tools, including static analysis and dynamic analysis. Some of these tests look for security flaws, others check for style and form. These tests run when a developer checks in code. Other than configuring rules and keeping the tools up to date, there isn’t much test writing to do with these automated tests.

Unit Tests

You can also automate a unit test suite. Unit tests are designed to test a single function, or unit, of operation in isolation. They typically run on a build server. These tests don’t depend on databases, external APIs, or file storage. They need to be fast and are designed to test the code only, not the external dependencies.

Integration Tests

Integration tests are a different kind of animal when it comes to automation. Since an integration test—sometimes called end-to-end tests—needs to interact with external dependencies, they’re more complicated to set up. Often, it’s best to create fake external resources, especially when dealing with resources beyond your control.

If you, for example, have a logistics app that depends on a web service from a vendor, your test may fail unexpectedly if the vendor’s service is down. Does this mean your app is broken? It might, but you should have enough control over the entire test environment to create each scenario explicitly. Never depend on an external factor to determine the outcome of your test scenario.

Automated Acceptance Tests

There are several practices today that use automated acceptance tests (AAT), but they’re basically doing the same thing. Behavior-driven development (BDD) and automated acceptance test-driven development (AATDD) are similar. They both follow the same practice of creating the acceptance test before the feature is developed.

In the end, the automated acceptance test runs to determine if the feature delivers what’s been agreed upon. Therefore, it’s critical for developers, the business, and QA to write these tests together. They serve as regression tests in the future, and they ensure that the feature holds up to what’s expected.

Regression Tests

Without AATs in place, you have to write regression tests after the fact. While both are forms of functional tests, how they’re written, when they’re written, and whom they’re written by are vastly different. Like AATs, they can be driven through an API by code or a UI. Tools exist to write these tests using a GUI.

Performance Tests

Many kinds of performance tests exist, but they all test some aspect of an application’s performance. Will it hold up to extreme pressure? Are we testing the system for high heat? Is it simple response time under load we’re after? How about scalability?

Sometimes these tests require emulating a massive number of users. In this case, it’s important to have an environment that’s capable of performing such a feat. Cloud resources are available to help with this kind of testing, but it’s possible to use on-premises resources as well.

Smoke Tests

What’s a smoke test? It’s a basic test that’s usually performed after a deployment or maintenance window. The purpose of a smoke test is to ensure that all services and dependencies are up and running. A smoke test isn’t meant to be an all-out functional test. It can be run as part of an automated deployment or triggered through a manual step.

General Test Automation Process

Now that we’ve seen criteria for automation and enough types of automated tests to have a feel for things, here’s the general process of test automation. There are three major steps to test automation: prepare, take action, report results.

Prepare

First, we need to prepare the state, the test data, and the environment where tests take place. As we’ve seen, most tests require the environment to be in a certain state before an action takes place. In a typical scenario, this requires some setup. Either the data will need to be manipulated or the application will need to be put into a specific state or both!

Take Action

Once the state and/or environment is in the predefined state, it’s time to take action! The test driver will run the test, either through calling an application’s API or user interface or by running the code directly. The test driver is responsible for “driving” the tests, but the test management system takes on the responsibility of coordinating everything, including reporting results.

Report Results

A test automation system will record and report results. These results may come in a number of different formats and may even create problem tickets or bugs in a work tracking system. The basic result, however, is a pass or fail. Usually, there is a green or red indicator for each test scenario to indicate pass or fail.

Sometimes, tests are inconclusive or don’t run for some reason. When this happens, the automation system will have a full log of the output for developers to review. This log helps them track down the issue. Ideally, they’ll be able to replay the scenario once they’ve put a fix in place.

The Bottom Line

The bottom line is this: test automation helps improve quality with speed. But not all testing can be automated. There’s definitely an investment involved. With so many types of tests, it’s important that you get the right mix. The test pyramid is a simple rule of thumb to help get this right. It says most tests should be unit tests, followed by service tests, then UI tests.

A test automation system coordinates testing concerns, including managing test data, running tests, and tracking results. Test automation is the next step for teams that are becoming overwhelmed by the burden of repeating the same manual tests that should be automated. 

Author bio: This post was written by Phil Vuollet. Phil uses software to automate processes to improve efficiency and repeatability. He writes about topics relevant to technology and business, occasionally gives talks on the same topics, and is a family man who enjoys playing soccer and board games with his children.

When we talk about automation applied to the testing field, there are two expressions you’ll often hear thrown around: “automated testing” and “test automation.” But wait a minute. Aren’t automated testing and test automation the same thing? As it turns out, no. They are related concepts, but each one has a very specific meaning and purpose. And make no mistake: in order for your software organization to thrive, you’ll need both.

If up until now you thought that these two concepts are the same, don’t feel bad. For starters, you’re definitely not alone. People often mistake the two terms for one another. And secondly, your confusion ends today. That’s what this post is all about, after all.

We’ll begin by defining the two terms. Then we’ll dive a little deeper into each one of them, explaining their importance in an organization, also showing how they differently affect each role inside the team. Finally, we part ways by giving some additional tips. Let’s get started!

Defining “Automated Testing”

Right at the start of this post, we claim that “automated testing” and “test automation” aren’t the same thing, even if their names seem to imply otherwise. We’ve also told you that both are critical for your overall testing strategy. Finally, after a brief detour on the importance of testing in general, it’s time to define both approaches, clarifying how they are different—and similar—to one another.

Let’s start out by defining automated testing. And to be fair, that couldn’t be easier. “Automated testing” is just the process of automatically running a specific set of tests and verifying their results, instead of doing it manually. When you run your suite of unit tests on your development machine, you’re performing automated testing. When the CI server runs all of your unit and integration tests when someone pushes new commits to it, it’s also performing automated testing.

That’s pretty much all there is to it. With that out of the way, let’s continue to “test automation”, where things will probably be a little bit trickier.

Defining “Test Automation”

Now it’s time to define “test automation.” What is it and how it differs from the definition you’ve just read in the previous section? While automated testing is basically the running of automated tests, “test automation” is a way broader concept. It refers to completely automating the whole process of managing the different testing needs inside an organization.

Out of context, the line above most likely doesn’t explain much. Things will get a lot easier in the following sections, in which we’ll contextualize the two approaches, explaining how they fit into the overall quality strategy of a software organization and who each one benefits the most.

Automated Testing: Who Does It Help?

To really draw the distinction between automated testing and test automation, it’s important to show how they affect the different roles inside a software organization. Let’s start with automated testing. Which role in an organization benefits the most from this approach? The way I see it, the answer is clear: developers. And why is that?

A Unit Testing Example

For the sake of the argument, let’s focus on unit testing for a while. “Unit tests” is not a really great name if you think about it. No one seems to agree on what “unit” really means, for starters. But the main problem is that they aren’t really “tests.” The best way to think about unit tests is to consider them executable specifications. The term “programmer tests” has also been proposed a few times, and it makes sense. Unit tests are written by programmers, for programmers, in order to “prove”, at least to some degree of confidence, that a given piece of code really does what it’s supposed to do.

Automated Testing Is a Developer Confidence Booster

It isn’t that much of a stretch to expand our definition of “executable specifications” to also include integration tests. A common definition of integration tests would be “unit tests, kind of, but they use the real database/filesystem/etc instead of mocking/stubbing it.” Let’s do just that, then: let’s stretch our definition of “executable specifications” in order to also include integration tests. And let’s go further and pull all of that under the “automated testing” umbrella. What do we get from all of that defining and stretching?

The answer seems clear to me: the main beneficiary of “automated testing” in a software organization is the developer. Or, in other words: automated testing is a tool to achieve developer sanity. It allows developers to fearlessly refactor, knowing that they have a reliable safety net in the form of a comprehensive test suite that will warn them if they break something.

Test Automation: Who Does It Help?

We’ll now turn our focus to test automation and repeat the same analysis we just did with automated testing. Who benefits the most from test automation? The answer, again, is clear: the organization as a whole, with a special focus on the DevOps professionals. To understand why, we have to go back in time, to the (not so) good old days of the waterfall methodology.

Once Upon A Time, During The Dark Ages

In organizations that employ this methodology (and similar ones), testing is nothing but another phase in the whole cycle. And the testing phase tends to happen at the end of the cycle. More often than not, this is too little, too late. The feedback cycles slow down, reducing the usefulness and efficiency of the whole testing strategy. Nowadays, companies are moving toward a world of continuous development, deployment. They can’t afford to wait forever for the feedback from testing. The testing itself has to be continuous so that quality can be ensured at all phases of development. Automation can come in handy in this situation.

Here Comes The Enlightenment (i.e. Continuous Testing)

Automation is the missing link. It can provide the speed and consistency that you need in order to have continuous testing up-and-running. Manually managing the testing needs in such an environment would not only be incredibly hard and error-prone but also time-consuming, which in fact defeats the whole point of doing it in the first place.

By automating the managing of said testing needs, test automation helps organizations keep quality at the highest standards during the whole development cycle. Furthermore, it frees QA professionals from managing the minutia of testing needs, which means they can focus on creating more efficient case tests to ensure the application’s quality.

Automated Testing And Test Automation. You Need Both. ASAP

In today’s post, we’ve clarified the confusion between “Automated Testing” and “Test Automation”. Yep, the naming could use a little less ambiguity. But the concepts themselves are really different. However, they’re equally important if you want your organization to continue delivering software to the highest standards of quality. Automated testing is important, especially for giving your developers the confidence to fearlessly refactor their code,  which contributes to higher quality code, without a doubt.

But test automation becomes critical when your company starts moving towards a continuous development/continuous deployment scenario. Do you want your company to continue thriving in this brave new DevOps world? Put test automation to work for you and your organization.

This post was written by Carlos Schults.Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.

Be smart & save time...
Simply automate.

Get Started Free