Author

testimio

Browsing

Maybe you’ve heard what test automation is but you’re unsure about how to fit it all together in practice. If that’s you, then you feel just like I did when I was first getting my head around how test automation works. So many conversations about testing quickly become abstract and sometimes we simply want to be practical—just show me the code!

If you feel the same way, then you’re in luck! Today we’re going to cover how automated tests work in detail. To do this, we’ll cover the test pyramid and how it relates to testing, which will set a foundation so we can go through and break down each category of testing—from unit tests to integration tests and finally to journey tests. And we did promise to keep things practical, so we’ll talk through some real-world code examples too.

By the end of this post, you should know what the test pyramid is and how it relates to automation testing. You will also learn in detail what unit, integration, and journey tests are and how they’re written.

What Are Automated Tests?

Before we begin, let’s start with a quick definition:

Automated tests perform machine-based assertions on an application to determine the state of the current application against the assertions.

In simple terms, we ask questions of our systems through testing, and the current state of the system (how it’s coded) informs our test results.

Don’t worry if it still feels abstract, as we’ll be breaking it down.

What Is the Test Pyramid?

Before we dive into our examples on each of our testing steps, let’s define them. A lot of the nuance of how automated tests should be written hinges on the theory around test pyramids, so it’s important to have a strong grasp before continuing.

At its heart, the testing pyramid is about feedback loops, and it forms a recommendation on how to structure your tests to achieve the fastest possible feedback loop.

But, what do I mean by feedback loop? I mean the time it takes from modifying some code until you know that it’s wrong (and, importantly, where it’s wrong). The notion of the pyramid is important as it represents how we should heavily load the bottom layer while keeping the top layer thin in order to retain our pyramid shape (more on balancing testing layers soon!).

Test pyramids can come in slightly different formats. For today, let’s use a simplified version, which breaks down into unit, integration, and journey tests.

Unit Tests

Unit tests are the base of our pyramid—they run fast and validate individual functions work as we expect. However, since we’re only testing at the function level, these tests don’t tell us whether our application works as a whole. But importantly, unit tests run blazingly fast. And when they fail, they tell us exactly what went wrong to the precision of a single function. So while unit tests don’t tell us if the application is broken as a user would deem it, they give us the fastest feedback possible.

Integration Tests

Integration tests form the middle of the testing pyramid. If unit tests together sum up to a single software component, then our integration tests ensure that each software component can work with another. Usually, integration tests run over a network boundary of some form. Integration tests essentially validate whether all the aforementioned functions are put together correctly and that other software components can successfully communicate with the component in question. Integration tests are less about what the functions do (that’s our unit tests), but more that our functions are put together correctly.

Journey Tests

These tests form the last layer and the top of our pyramid. Journey tests typically execute a journey as our user would see it. We don’t tend to cover edge cases in journey tests as they should be covered at base levels of the pyramid (either unit or integration).

And journey tests are merely a few, short tests that cover a lot of code. Because we’re writing so few of them and they cover so much code, they give us a lot of bang-for-your-buck for establishing if our application is working as expected since they are covering core journeys. But—frustratingly—they won’t really tell you what is broken (at least, not to the same degree that a unit test would). Additionally, journey tests can be very slow to run and brittle to maintain (but more on why this is later).

Putting the Pyramid Into Practice

Hopefully, by now you have a decent understanding of the test pyramid and how the layers fit together. Don’t worry if you don’t completely understand at this point. We’re going to break down each layer and show you how it works in practice.

Unit Tests: An Example

Unit tests run against individual pieces of code. So let’s take an example.

The following test is written using the Jest unit test library. Jest gives us the “test” and the “expect” function. With these tools, we can then describe the scenarios we want to run for our tests and also our expected output. Tests can have any number of assertions, but typically you’ll have only one (or a small amount). Keeping assertions small means we can fit them to our descriptions easier.

function isOdd(a) {

    if (typeof a != “number”) {

        throw new Error(“Must pass a number”);

    }

    return a % 2 != 0

}

test(“Passing an odd number returns true”, () => {

    expect(isOdd(1)).toBe(true);

});

test(“Passing an even number returns false”, () => {

    expect(isOdd(1)).toBe(true);

});

test(“Passing a string throws error”, () => {

    expect(() => isOdd(“Hello”)).toThrow(“Must pass a number”);

});

Here we can see that we’ve written:

  • A happy path test
  • An unhappy path test
  • A dynamic type checking and error handling test

The test coverage is therefore 100 percent. However, we may want to write additional cases in the future for edge cases if we believe they add more value. For instance: what should the function do if we pass superfluous arguments? And what about edge cases? Negative numbers? Passing a zero value? Or passing a very large value? All of these are valid tests, but adding more tests has diminishing value over time. If we’re incredibly thorough with our testing at all levels, we likely won’t ship any code. It’s a balance. Choosing whether to add these additional cases will depend on factors such as: how much business impact would there be if the code were faulty? Are there other tests you could write that deliver more value?

Integration Tests: An Example

Building on top of our unit tests we have our integration tests. Below is an example of a test written with SuperTest. SuperTest is a small helper library around Jest (which we used for the unit test example). SuperTest gives us the request functionality (and its chained assertion methods).

The following code takes an endpoint, in our case “/user” makes a request to it, asserting the response to be a 200 success. SuperTest also allows us to simply test HTTP endpoints. Remember when we said integration tests often happen over a network boundary? This is precisely what we meant. We tell our test what our setup criteria are, and we can then assert that we received the correct response, error code, headers, etc.

describe(‘GET /user’, function() {

  it(‘responds with json’, function(done) {

    request(app)

      .get(‘/user’)

      .set(‘Accept’, ‘application/json’)

      .expect(‘Content-Type’, /json/)

      .expect(200, done);

  });

});

But you might be thinking: in the above example are we simply testing that the endpoint responds? Not that it has expected values?

At first, this may seem like a low-value test. However, since we have already covered a lot of the functional testing in our unit tests at the integration test level, we are really only testing that our application links up all of those functions and responds without errors. Of course, we can write more detailed scenarios, but we want to be careful that we’re not replicating unit test code in integration tests. Unit tests run faster, so we should ideally write our tests there when possible.

Journey Tests: An Example

Last up is the journey test. The following is an example of a journey test using the Cypress testing framework. Cypress gives us a test runner and an assertion library. The “describe”, “it”, and “cy” commands all come from the Cypress library.

describe(“When at checkout”, () => {

    it(“Can buy”, () => {

        cy

            .get(“.buy”)

            .click();

        cy

            .get(“.summary”)

            .should(“have.text”, “Thanks for purchasing!”);

    });

});

In our journey test, we are testing the physical interface of the application, which should not only include what the user sees, but also behave as a user would when conducting a task (we call this a journey). In our example, we are navigating to our route, finding a button by its class name, clicking it, and then asserting that we can see our summary.

Can you spot why we said these tests could be brittle? For instance, if a developer decides to change an element class name (which in theory might not break functionality!), this change may result in a test failure. Oh no! We can use clever ways of structuring our code to reduce the fragility of our tests, but we cannot remove the brittleness completely—it’s inherent in the nature of the journey test.

The End of Our Tour of Automated Testing

And that concludes our foray into automated testing today. I hope this post gave you a clearer picture of how automated testing works in practice. We’ve seen how we need to have a balanced pyramid and also some examples of how our pyramid could start to be implemented.

In the real world, automated testing requires constant collaboration communication and continual improvement of our codebase. As we find bugs in our code, we work to ensure our coverage is updated. But in updating our codebase we need to ensure we balance our pyramid to keep our feedback fast.

It’s not easy. But, do it right and automated testing yields incredible results that are far superior to their manual testing counterparts. Learn how to identify the best automation platform for your needs with our simple, five step process. 

And remember, the best way to learn is to get your hands dirty and have a go! So take the examples and attempt to implement your own small application with each layer of the test pyramid!

Author bio: This post was written by Lou Bichard. Lou is a JavaScript full stack engineer with a passion for culture, approach, and delivery. He believes the best products emerge from high performing teams and practices. Lou is a fan and advocate of old-school lean and systems thinking, XP, continuous delivery, and DevOps.

Regression testing is a specific form of testing that verifies whether a given piece of software suffered regressions after undergoing changes. “Regression” here means “going back to a previous undesired state.” So automated regression testing is nothing more than the process of automatically verifying that the application has not regressed to a previous undesired state.

Ideally, regression testing—whether automated or not—should be performed every time a software application is changed in some way, whether by receiving a new feature, an improvement, or a bug fix.

In today’s post, we’ll define and explain what automated regression testing means. We’ll start by explaining what a regression is, how costly they are for software teams, and why you should employ regression testing to avoid them. Since manual regression testing would be both time-consuming and error-prone, we’ll then proceed to cover the need for automation, offering advice and tips on how to actually implement the technique for your teams.

Let’s get started!

Automated Regression Testing: Let’s Come Up With a Definition

Before we dive into the “how” of automated regression testing, we need to clarify its “what” and “why.” What is it, after all? What does automated regression testing mean, and why should we use it?

This section will answer the questions above and more.

What Does “Regression” Mean in “Automated Regression Testing”?

The first step in order to understand “automated regression testing” is to get rid of the “automated” part, at least for now. First, we need to understand what “X” means, then “Automated X” will be self-evident. Or, at the very worst, easier to grok.

If we want to understand what “regression testing” means, the first step is to define “regression.” In this context, is “regression” good or bad? Should we welcome it or fear it? The answer to the questions above is straightforward. Here, regression means essentially the same thing it means in everyday conversation—to go back to a previous state. So, in the context of software development, we say we’ve got a regression when our application unintentionally reverted to a previous state.

Why Do We Need Regression Testing?

As we’ve just defined in the previous section, “regression” means an unintentional reversion to a previous state. However, something really important was left implicit in the definition: software regression, more often than not, means the application went back to a bad previous state.

Have you ever had the experience of making some existing feature stop working after implementing a new one? What about that nasty bug returning months after you thought you had completely eliminated it? These are examples of regression. The application has regressed to a previous inferior state.

Regression Testing Is Vital in Software

The field of software development is disproportionately prone to regression problems. Each addition or change made by any developer has the potential to cause unexpected problems in areas (supposedly) unrelated to the spot where it was performed. Any non-trivial software project, maintained by a team with more than, say, five people, has an incredibly high number of potential regressions during each release.

Developers should always keep in mind that all the changes they make, no matter how small, simple, or insignificant they seem, have the potential to cause surprising side effects. They can break functionalities that don’t have anything to do with the changes being made. By performing regression tests, the developer checks that not only does their change behave as it’s supposed to, but also that it plays well with all of the code that was written up until that point.

That’s where regression testing comes in handy. Regression testing is nothing more than the execution (partial or total) of a test suite in order to verify that a given application hasn’t returned to a previous undesired state. If manually done, though, regression testing can be extremely time-consuming and error-prone, which leads us to our next point.

Regression Testing: Why Automate It?

I often say that everything that’s automatable should be automated. Meaning, if you can automate a process, then you probably should do so. If you can automate a process but you’re still doing it manually, I’m afraid you’re leaving money on the table. Chances are the manual process is slow, time-consuming, tedious, and error-prone. That implies you’re losing money in at least three different ways.

For starters, you waste money by having well-paid professional performing tasks that could be automated. Then, there’s always the opportunity cost. The people who are performing the tests could be doing more valuable tasks. Such tasks could have the potential to generate way more value. Finally, since the manual process is error-prone, then people are bound to make mistakes, which will result in losses.

Automated Regression Testing: How to Put It Into Practice

Now that we’re done defining concepts, it’s time for some practical tips on how to actually implement regression testing. There’s good news. If you already write some kind of automated tests for your application—e.g. unit tests—then you’re already performing regression testing without even knowing about it.

Regression testing is not a “new” category of automated tests. It’s not an alternative to unit tests or integration tests. On the contrary, your automated tests —unit tests, integration tests, and similar—written from day one in the project can and should act as regression tests. After each new change to the codebase,  just re-run all of the relevant tests on the suite to ensure they’re not failing.

Automated Regression Testing Best Practices

Additionally, there are some best practices you should implement when adopting regression testing, regardless of the specifics of your implementation:

  • Adopt a Test Management Software: This is vital for all projects that are larger than a one-developer toy project. You’ll have, realistically, way more tests cases than you’d be able to track and manage individually. Luckily for you, there are lots of test management tools at your disposal.
  • Keep a Testing Schedule: You should maintain a strict testing schedule throughout the entire project. This will ensure the final project is thoroughly tested. Additionally, the schedule will encourage the team to adapt to a frequent testing regimen.
  • Write a New Failing Test for Every New Bug Found: Imagine your code has an unequivocal, reproducible bug. But all of your tests are passing. That means that either the current tests are wrong, or your test suite is lacking tests. If you find yourself in this scenario, write a new failing test to document the bug.
  • Categorize All of Your Tests:  You should split your test suite into smaller categories. Your test management tool will most likely provide you with the ability to categorize your tests. That way, your team members can easily identify each kind of test.

Automated Regression Testing: Put It to Work for Your Team ASAP

Software development is a creative endeavor like no other, but the path to reaping its rewards is full of risks. One of the dangers when writing code is to break existing functionality while adding new ones. Even when fixing bugs, we can make older ones—that we thought were dead—come back to life. Each small and seemingly insignificant change poses the risk of a dreaded regression.

If you want to get rid of regressions—and you should want to, believe me—then a suite of regression tests is the solution you seek.

Author bio: This post was written by Carlos Schults.Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.

In a world where technology moves at breakneck speed, most of us know the importance of keeping pace. That’s why traditional software development models are rapidly giving way to methodologies that are more dynamic.

DevOps is a partnership that emerged from this climate. The DevOps approach emphasizes communication, collaboration, and integration. And it does that by bringing your developers and your operations folks together.

But the term DevOps includes only development and operations in its name, and the reality is that we rely heavily on a third entity as well. To deliver software that’s up to today’s standards, you need to consider your testers an equal part of that equation. And so that’s what we’ll talk about today—why the term DevTestOps is a more fair and accurate representation of how companies deliver great software.

A Brief History

With traditional development models, software releases consisted of a series of handoffs. Specifically, developers coded to technical specifications and then passed their code to QA to test. Then, there was a final handoff to operations, who pushed the code into the production environment.

Now, you might notice that QA—the integral step between development and operations—isn’t reflected in the DevOps title. This is a problem. One of the primary components of DevOps is continuous delivery. And for continuous delivery to be successful, you need a substantial amount of testing.

Now, don’t get me wrong. When the tech industry shifted toward DevOps, it was a great thing. In fact, key findings from the 2016 State of DevOps report show the measurable benefits reported by organizations that have successfully incorporated DevOps:

  • Project teams deploy 200 times more frequently.
  • Teams spend 22% less time on unplanned work and fixes.
  • They also spend 29% more time on new features or code.
  • And finally, teams spend 50% less time on security issues.

The report says it best: “By better integrating information security objectives into daily work, teams achieve higher levels of IT performance and build more secure systems.”

Since many companies are eager to have results like these, we’re seeing an increase in DevOps adoption. After all, who doesn’t want the benefits of an accelerated development cycle? But for your software to be resilient, testers need to ensure that quality is built into the final product.

As you can see, DevOps, as it stands, is missing a key component.

Testers Pull Their Weight

If you’re not convinced that testers have earned their place among development and operations, consider this.

More data than ever is traversing the networks. According to DOMO, there’s an estimated 2.5 quintillion bytes of data created daily. And the world is releasing products at a pace that renders manual testing ineffective.

With all that’s going on, automation is essential. Testing such large amounts of data at fast-paced intervals necessitates it.

If you do smoke/regression testing, performance testing, and static tests, automation does the bulk of the work. That means more time for manual testing when there are subjective factors at play. We all know that sometimes you need human judgment.

However, this may prove to be a tall order. Over 70% of smaller companies have very little automation.

That means that the folks responsible for testing must deal with the challenge of test orchestration and automation on their own—either that or they must try to bear the entire burden with manual testing. Not only must QA teams ensure that code changes are fully functional and integrate well but they also must see that frequent releases don’t render the product unstable.

Yes, your developers and your operations folks are critical. But testers bear equal responsibility for the quality of the software you put out. With all these challenges, it would be hard to argue they don’t pull their weight just as much as the “dev” and “ops” in DevOps do.

Testers Help Break Down Silos, Too

Beyond logging defects, testers in environments work cross-functionally. Sound familiar? It’s what DevOps is all about.

Testers share the spirit of collaboration that defines the DevOps movement. They work to ensure that the entire system, not just an individual feature, is operating at an optimal level. From verifying app performance to validating output results from APIs, testing in continuous development environments covers a broad range of tasks.

And with the added responsibility comes greater visibility. Software testing is no longer limited to that of a supporting role. These folks are partners. QA and testing teams have come to the forefront as key players, in line with development and engineering. They’re breaking down silos as much as anyone else in a DevOps organization.

Based on this type of environment, you can see why the term DevTestOps is more fitting. The trifecta of development, testing, and operations are already working together as equals. In fact, just as the word DevTestOps blends three words into one, this approach blurs the lines between roles and responsibilities. And to a large extent, it also eliminates the need for those previously discussed handoffs—for developers to hand off to QA and then for QA to hand off to operations—in order to promote code into production. Instead, each discipline opens its communication channels and collaborates as one team towards a common goal.

One Final Reason to Put the Test in DevTestOps: Money

We’ve seen plenty of good reasons to call things like they are and adopt the term DevTestOps. The combination of development, testing, and operations brings about many benefits. But let’s get to the heart of why companies would want to make the shift in thinking.

The “test” in DevTestOps adds business value across the board, from rapid delivery to improved security. And herein lies an important point: revenue and profits rely on speedy development. So it’s imperative that you head off problems before they become major events. App crashes must be mitigated. Bugs need to be caught early. And we want the end-to-end user experience to be error-free.

QA is an essential part of making sure this is the case. You have to test in order to save yourself from costly problems like bugs. And the earlier in the development cycle you catch a bug, the cheaper it is to fix it.

So let’s give “test” the weight it deserves. Let’s move to the next step in DevOps evolution: DevTestOps. If we do that, it means we have our priorities in the right place, and we’re going to have more profitable businesses.

If we’re talking convincing reasons, this should be the most convincing one of all.

Moving Into the Future

Testers and QA teams champion their cause, influencing processes and decisions project-wide. A business’s concerns aren’t just matters for development and operations. As key players in continuous delivery, QA organizations have seized the opportunity to evolve, taking testing from error detection to error prevention. Now, it’s our turn to evolve from DevOps to DevTestOps organizations, acknowledging their equal importance in creating great software.

“Automated testing” used to be a phrase that caused developers to scratch their heads. Now, it seems to me that most developers are at least familiar with the notion of automated testing. Fortunately, many devs go beyond familiarity. They turn theoretic notion into practical action by actually employing automated testing on a day-to-day basis. And nowadays there’s immense variety in the types of automated tests that can help you achieve higher software quality. In today’s post, I want to focus on a very specific type of automated testing—UI testing.

We’ll start by briefly defining what UI testing is, and then explain how it can benefit you. Then we’ll proceed to show you, step-by-step, how to get started with UI testing using Testim.

Let’s get started.

UI Testing: What It Is and Why Do You Need It

As you surely know, UI stands for “user interface.” So, defining UI testing doesn’t sound that hard, at first sight—it’s just exercising the application through its user interface. But, is that all there is to it?

Actually, it’s not. First of all, some people seem to think that UI testing isn’t necessary if you already have other types of testing in place, such as unit tests. That couldn’t be farther from the truth. UI testing is not a replacement for other types of testing—instead, it’s a compliment. Unit testing exercises the application’s units in isolation. It’s an important verification to have, but it’s not enough. It’s also important to verify how those units fit together. You have to check how the whole system behaves, from the UI, passing through each layer and then back. So, UI testing can be thought of as more of an “end-to-end” type of testing.

Other people might think that, while UI testing has some importance, you can have one or more developers just click around and see if the application behaves correctly. But that isn’t true. Manual UI testing might work for very small and simple applications. But when it comes to applications with complex business rules and rich user interfaces, manually testing quickly becomes insufficient. It is a tedious, error-prone, expensive, and hard-to-scale process.

At this point, automated UI testing is the best choice.

UI Testing Guide: How To Get Started

It’s time to get your hands dirty. We’re going to offer you a brief guide on how to get started with UI testing. It’ll by no means be an exhaustive tutorial, but it should teach you to create a basic approach upon which you can build and evolve into a more sophisticated approach later.

For this guide, we’re going to use Testim to test a very simple web app. Let’s get started.

An Example of Manual UI Testing

Before we start creating our tests, we need something to test, right? So, let’s create a toy application. It’s going to be a super simple web app, with just two controls—an input box and a button. It will allow the user to input numbers, separated by a comma. After clicking on the button, the application will add all the numbers and display the results. If the user types just one number, the result should be that number. If the input box is left empty, the result should be zero. Simple enough, right?

Creating a Toy Application

Ok, let’s get to it. This application will consist of a single HTML page with a little bit of Javascript. Nothing too fancy. Open up your favorite text editor, create a new file, and add the following content to it:

<!DOCTYPE html>

<html>

<head>

<title>String Calculator</title>

<style type=”text/css”>

label { display:block; margin: 10px;}

</style>

</head>

<body>

<h1>String Calculator</h1>

<form id=”form1″>

<label for=”numbers”>Numbers: <input type=”text” id=”numbers” name=”numbers”/></label>

<label for=”addButton”><input type=”submit” value=”Add” id=”addButton”/>

</form>

<script type=”text/javascript”>

document.getElementById(“addButton”).addEventListener(‘click’, function(){

    var text = document.getElementById(“numbers”).value.trim();

    if (!text) {

        text = ‘0’;

    }    

    var integers = text

        .split(“,”)

        .map(x => parseInt(x));

    var result = integers.reduce(((a, b) => a + b), 0);

    alert(result);

});

</script>

</body>

</html>

After that, save the file with the “.html” extension. Then, double-click the newly saved file. You should see your default browser showing you something like this:

Performing Manual UI Testing

Now let’s do some (manual) UI testing. Click on the input box, type 0 and click on the Add button. You should see this:

Now, redo the test, but leave the input box empty. The result should also be zero:

Repeat the test a few more times, always using a single number. The result should always be the number itself. Then, perform some tests with more than one number, separated by a comma. The result should be the sum of the numbers, as the following example shows:

After this brief session of manual UI testing, I’d be convinced that our little toy app works as intended. But, is that testing enough? For this example application, I’d say yeah, probably. But keep in mind that this silly string calculator is just a placeholder for a much more complex application, closer to the ones you’re likely working on in real life.

Getting Started With Testim

Now it’s time to show you how to get started with Testim so you can see what automated UI testing looks like.

First Steps: Creating an Account and Installing the Testim Browser Extension

First, go to https://testim.io and  click on “Start Trial.” You’ll be asked to sign up for an account. You can do that using your Google or GitHub account. If neither option is OK for you, sign up with an e-mail address and create a password.

After successfully logging in, you’ll see a screen like the following image:

If you click on “Create test” button, you’ll see a message asking you to install the Testim Chrome extension:

After downloading and installing the extension, you’ll be ready to start creating automated tests. This time, though, instead of testing our silly little string calculator, we’ll test a real one—the Google calculator.

Creating Your First Test With Testim

On Chrome, go to the address bar, type “calculator” and hit enter. You should see this:

Just a good old calculator. Now, we’re going to create a test for the “+” button.  While still on the same tab, click on the Testim extension’s icon, as shown in the following image:

After clicking on the extension icon, you should see the following menu:

Click on “CREATE AUTOMATED TEST.” That will open Testim and attach the current tab to it. Then, you’ll return to the calculator tab and see the following message:

What you have to do now is very simple—just use the application and your actions will be recorded. Let’s create a test for the “2 + 2” sum. Perform the sum as you would normally do it. After clicking on the “equals to” button, go back to the tab where Testim is open. You should now see this:

Click on the “SAVE” button. You’ll then see a window asking you for a name and a description for your test:

For the name, enter “Add test”. On “Description”, type “Verify the Add operation” Then click on “OK.” Congratulations, you’ve just created your first test with Testim!

Running Your Test

You’ve just created your first automated UI test using Testim. That’s nice, but how do you actually run the test?

On the left panel, click on the “Automate” button, like in the following image:

After doing so, you’ll see your test list which contains only the test you’ve just created:

To run the test, just mark its checkbox and click on the “play” button:

After clicking on the “Run Selected Tests” button, you’ll see a message warning you to avoid touching the mouse and the keyboard during the test execution:

After clicking on “OK”,  Testim will open up a new Chrome tab and run the test. Testim will execute the exact same steps you’ve recorded. After Testim finishes running the test, it’ll close the window.  Go back to the Testim tab and see the test result:

You’ve successfully run your first test using Testim. Congratulations! But…isn’t there something missing?

Adding a Verification

You’ve probably noticed that the test you’ve just created doesn’t really test anything. Yeah, the test passes but that doesn’t really mean much. In terms of unit testing, it’d be the same as having a test method with no assertions.

Our test lacks something, and that something is a verification. We need our test to check that the result of “2 + 2” is indeed four. How do we do that?

On the left panel, click on the “Automate” button. You’ll go back to the test listing. By clicking on the test name, you’ll reopen the test editor so you can change it:

You can see that the test has six steps, and the last step is clicking on the “=” sign. What we have to do now is add an extra step, whose goal is to perform the validation. To add a new step, click on the “+” button after the sixth step. You’ll see the following options:

Click on the third option and you’ll see the menu change to this:

Now, click on “Validate element text.” Testim will take you back to the calculator tab. There, click on the result field:

After doing that, go back to Testim. Now, you should see that your test has a new step:

Hover over the newly added step and click on the engine icon to show the properties panel:

Now it’s just a matter of editing the “Expected value” field, setting it to ‘4’. After doing that, click on the “SAVE” button again. You’ll see a box prompting you for a message, explaining your change. Type “Add validation step” and click on OK:

If you run the test again after saving it, it will continue to pass. However, Testim is now comparing the expected result to the actual result, just like a unit test assertion does.

UI Testing Goes Way Beyond “Clicking Around”

There are still some misconceptions about UI testing out there. One of them is that UI testing isn’t needed if you already employ different categories of testing. Another myth is the belief that informal manual UI testing—a.k.a. just clicking around—is enough.

Neither of those beliefs is true.

We hope this post has shown you not only that UI testing is indeed necessary, but also that it isn’t hard getting started with it, especially if you employ a tool like Testim.

Thanks for reading, and see you next time!

Author bio: This post was written by Carlos Schults. Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.

Today’s post is going to be all about automated functional testing. What’s this category of tests really about? Why do you need it? How do you go about implementing it? What are the challenges you might face when doing so? With this post, we’re going to answer these questions—and more.

We’ll start out by defining functional testing. Then, we’ll go on to explain the benefits of functional testing so you understand why you need it, even if you already employ other types of tests. After that, we’ll reach the “automated” part of “automated functional testing.” We’ll define automation, explaining how it’s imperative to apply it to testing. Then, we’ll be ready to cover automated functional testing.

With the definitions out of the way, it’s time to get to more specific advice. You’ll see how to get started with automated functional testing, the possible challenges on your journey, and how you can beat them.

Functional Testing: What It Is and Why You Need It

Functional testing, as its name implies, concerns itself with how a given application functions. Does the application meet its functional requirements? Does it solve the problems of its users? These are the kind of questions that should be answered by functional test cases.

Functional testing is, then, a type of black-box testing. In other words, functional testing couldn’t care less whether you’ve used an array or a list to model a given concept. The only thing that matters is that the application’s external behavior meets the requirements.

Unit Tests Are Not Enough

In some software development cycles, there are people who seem to think of unit tests as the ultimate software panacea, the mythical silver bullet that will solve all software crises for good.

There’s no such thing, of course.

Make no mistake: unit tests are awesome. They can either lead to a cleaner, low-coupled design or—in the worst scenario—make you acutely aware of poorly designed, high-coupled code. They often cause developers to learn about good development practices, such as immutability, purity of functions, and separation between logic and presentation. More importantly than all of that, a comprehensive suite of unit tests empower developers to fearlessly refactor, knowing they have a safety net if they break something.

Did you notice who is missing from the previous paragraph? The user.

Let’s Think of the Final Users

The previous section covered some of the benefits of unit testing. You’ll notice, though, that those benefits seem to benefit mainly the developers themselves. Sure, cleaner code and a more maintainable application will indirectly benefit users, no doubt about that. But that’s not enough.

There’s a school of thought that argues that “unit testing” is an unfortunate name for an otherwise great tool, and I agree with that. I’ve seen people use “executable specifications” as an alternative name. The point is this: unit tests aren’t really tests. They’re verifications that developers use in order to check that the code they wrote does what they think it does.

But who’s checking whether the code they wrote does what the user expects it to do? That’s the job of functional testing.

It is possible to write a correct, elegant and efficient solution…to the wrong problem. While unit tests will make you confident you’re solving the problem correctly, functional testing is what ensures that you solve the right problem, for the right users.

Automated Functional Testing: Understanding Its Benefits

Up until now, you’ve seen what functional testing is and why you need it, even if you already use other, also very useful, types of tests. Now it’s finally time to add “automation” into the mix. We’re going to define automation and show why its use is imperative in testing. We’ll then zoom in and talk specifically about automating functional tests.

Automation Is Great. Are We Using It?

As developers—and software development professionals in general—we can consider that automation is the whole raison d’être of our jobs.  Why shouldn’t we use it to improve the efficiency of the software development process itself?

Sure, programmers have been automating aspects of their job since the dawn of software engineering. But in recent years, the industry’s use of automation increased to unprecedented scales. We can see this reflected in the growing use of automated testing and, more generally, in the DevOps movement.

Automation in Testing Is a Must

If there’s one area in the whole software development process that really benefits from automation, that area is testing. That’s due to one of the most unique characteristics of software—even the smallest change in one area of an application can cause unforeseen consequences in a totally different area. Yep, that’s a serious problem. In an ideal application, with sound architecture, something like that should never happen. We don’t live in the ideal world, unfortunately, and things like that do happen frequently.

The implication of this unique software characteristic is crucial. After making some change on a codebase, the only way to be sure the whole application still works properly is to retest the entire app. Just testing your own change isn’t enough. You could’ve broken something and you’d never know.

That means that, in modern software development, applying automation to testing is imperative.

Automated Functional Testing: You Can Have Your Cake and Eat It Too

Automated functional testing is, in a way, the best of both worlds—it brings to the table the benefits that automation provides to all kinds of tests, along with the functional testing ethos of putting the user’s needs first. It takes automated testing to its full potential, making it not only a tool for developer confidence (which is already awesome in itself), but also a tool for customer satisfaction. It bridges the gap between QA professionals and developers, creating a shared space where they can collaborate.

Automated functional testing sounds awesome—and it is—but how do you actually get started with it? I’m glad you asked.

The first step is to have specs. You can’t have tests that verify whether the application conforms to the specifications if there aren’t any. For this first moment, it shouldn’t matter that much whether the specifications are embedded into the code or recorded in an external document. The important thing is that you have them.

Then, it’s time to start thinking about the possible test cases for your application. Start your app, go to its main screen, and look at it critically. What are the tasks the user is supposed to perform here? How could they go wrong? It’s even better if you don’t do this on your own. Instead, bring in some coworkers to help. Do a brainstorm session on the current screen. Write down all of the ideas that you can come up with.

Then do the same with other areas of the application.

After all of this, do you have a reasonably sized list of possible use cases? Great, now it’s time to go shopping for tools. This post is meant to be platform agnostic, so we won’t actually recommend particular testing frameworks. But it doesn’t matter your language or tech stack of choice, rest assured, there are automated functional testing tools out there for you.

Automated Functional Testing Challenges: What They Are and How To Overcome Them

As promised, now it’s finally time to cover some of the common challenges when implementing automated functional testing. What are those? How do you beat them? Keep reading to learn more.

QA Skepticism

One of the most common challenges when implementing all kinds of automated testing is skepticism. It’s 2019 and there are still many professionals that are unfamiliar with automated testing, believe it or not. There are also others who know about automated testing but don’t quite buy into the usefulness. How do you deal with the skeptics? Education.

You must use every kind of material at your disposal—books, online courses, on-site training, and what have you. After some training, make your QA people get their hands dirty. Have them start creating automated functional tests, first on a very small scale. Then, start to gradually increase said scale. In true Agile fashion, use small steps, reviewing your results in short iterations. Rinse and repeat. That’s pretty much it.

Developers Skepticism

When it comes to developers, it’s possible that some of them have jumped on the “unit testing = software panacea” bandwagon. If that’s the case, make them get off the bandwagon. The tool for doing so? You guessed it, education. Hands-on education.

Have your developers collaborate with QA people. Create small, cross-functional teams on which QA and developers can work side by side. As automation experts, developers might be of incredible help, mentoring QA people on the nuances of automated testing frameworks and tools.

This approach will create trust and empathy between developers and QA professionals, bridging an otherwise insurmountable gap, and destroying every trace of skepticism toward automated functional testing on both sides.

Maintenance Burden

Another challenge of automated functional testing—and other types of automated testing—is the maintenance burden. You see, test code is code, after all. It can contain bugs. It might be unreadable. And the list goes on.

How do you overcome this challenge? First of all, education again. Educate yourself and your coworkers on the best practices for writing tests, be them functional or otherwise. Read about ways to keep your test code clean and readable. Learn about the common mistakes people make when writing tests and what you should do to avoid them.

Finally, there are tools that can ease your burden by providing you with test management capabilities that you would otherwise not have, such as comparing versions of the same test, scheduling tests executions, and automating bug reporting.

Automated Functional Testing Helps You Solve the Right Problems at the Right Speed

Software development is a field that evolves at breakneck speed. In order to keep up with such rapid change, companies rely more and more on automation, giving birth to approaches such as DevOps and continuous delivery.

Automated testing plays a crucial role in this scenario. It’s unfeasible for teams to completely retest their application by hand each time someone has made a change. Automated testing is way past the point of being something that’s “nice to have”—software development in the 21st century requires it.

Automated functional testing, in particular, brings the benefits of automation to a type of test that puts the user front and center. It bridges the gap between QA and developers, ensures the application meets the user’s needs, and takes automated testing to its full potential.

Author bio: This post was written by Carlos Schults.Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.

There are two kinds of testing in the world of software—manual and automated. Some types of manual testing, such as discovery testing and usability testing, are invaluable. You can do other kinds of testing—like regression testing and functional testing—manually, but it’s a fairly wasteful practice for humans to keep doing the same thing over and over again. It’s these kinds of repetitive tests that lend themselves to test automation.

Test automation is the practice of running tests automatically, managing test data, and utilizing results to improve software quality. It’s primarily a quality assurance measure, but its activities involve the commitment of the entire software production team. From business analysts to developers and DevOps engineers, getting the most out of test automation takes the inclusion of everyone.

This post will give you a high-level understanding of what test automation is all about. There are all kinds of tests, but not all should be automated; therefore, let’s start with general criteria for test automation.

Criteria for Automation

A test needs to meet some criteria in order to be automated—otherwise, it might end up costing more than it saves. After all, one major goal of automation is to save time, effort, and money. Here are some general criteria for test automation. These are starting points, mind you. Your criteria may differ depending on your circumstances.

Repeatable

The test must be repeatable. There’s no sense in automating a test that can only be run once. A repeatable test has the following three steps:

  1. Set up the test, including data and environment.
  2. Execute the function and measure the result.
  3. Clean up the data and environment.

In the first step, we want to be able to put the environment into a consistent state. In other words, if we have a test for attempting to add an existing user, we need to make sure the user exists before performing the test. Once the test is complete, the environment should be returned to the base state.

Determinant

When a function is determinant, it means that the outcome is the same every time it’s run with the same input. The same is true of tests that can be automated. For example, say we want to test an addition function. We know that 1 + 1 = 2 and that 394.19 + 5.81 = 400.00. Addition is a determinant function.

Software, on the other hand, may use such a high number of variable inputs that it’s difficult to have the same result over time. Some variables may even be random, which may make it difficult to determine the specific outcome. Software design can compensate for this by allowing for test inputs through a test harness.

Other features of an application may be additive; for example, creating a new user would add to the number of users. At least when we add a user we know that the number of users should only grow by one. However, running tests in parallel may cause unexpected results. Isolation can prevent this kind of false positive.

Unopinionated

You cannot automate matters of opinion. This is where usability testing, beta testing, and so forth really shine. User feedback is important, but it just can’t be automated … sorry!

Types of Automated Tests

There are so many types of tests, many of which can be automated, that we can’t really get them all into one post. But here are enough to give you a good starting point.

Code Analysis

There are actually many different types of code analysis tools, including static analysis and dynamic analysis. Some of these tests look for security flaws, others check for style and form. These tests run when a developer checks in code. Other than configuring rules and keeping the tools up to date, there isn’t much test writing to do with these automated tests.

Unit Tests

You can also automate a unit test suite. Unit tests are designed to test a single function, or unit, of operation in isolation. They typically run on a build server. These tests don’t depend on databases, external APIs, or file storage. They need to be fast and are designed to test the code only, not the external dependencies.

Integration Tests

Integration tests are a different kind of animal when it comes to automation. Since an integration test—sometimes called end-to-end tests—needs to interact with external dependencies, they’re more complicated to set up. Often, it’s best to create fake external resources, especially when dealing with resources beyond your control.

If you, for example, have a logistics app that depends on a web service from a vendor, your test may fail unexpectedly if the vendor’s service is down. Does this mean your app is broken? It might, but you should have enough control over the entire test environment to create each scenario explicitly. Never depend on an external factor to determine the outcome of your test scenario.

Automated Acceptance Tests

There are several practices today that use automated acceptance tests (AAT), but they’re basically doing the same thing. Behavior-driven development (BDD) and automated acceptance test-driven development (AATDD) are similar. They both follow the same practice of creating the acceptance test before the feature is developed.

In the end, the automated acceptance test runs to determine if the feature delivers what’s been agreed upon. Therefore, it’s critical for developers, the business, and QA to write these tests together. They serve as regression tests in the future, and they ensure that the feature holds up to what’s expected.

Regression Tests

Without AATs in place, you have to write regression tests after the fact. While both are forms of functional tests, how they’re written, when they’re written, and whom they’re written by are vastly different. Like AATs, they can be driven through an API by code or a UI. Tools exist to write these tests using a GUI.

Performance Tests

Many kinds of performance tests exist, but they all test some aspect of an application’s performance. Will it hold up to extreme pressure? Are we testing the system for high heat? Is it simple response time under load we’re after? How about scalability?

Sometimes these tests require emulating a massive number of users. In this case, it’s important to have an environment that’s capable of performing such a feat. Cloud resources are available to help with this kind of testing, but it’s possible to use on-premises resources as well.

Smoke Tests

What’s a smoke test? It’s a basic test that’s usually performed after a deployment or maintenance window. The purpose of a smoke test is to ensure that all services and dependencies are up and running. A smoke test isn’t meant to be an all-out functional test. It can be run as part of an automated deployment or triggered through a manual step.

General Test Automation Process

Now that we’ve seen criteria for automation and enough types of automated tests to have a feel for things, here’s the general process of test automation. There are three major steps to test automation: prepare, take action, report results.

Prepare

First, we need to prepare the state, the test data, and the environment where tests take place. As we’ve seen, most tests require the environment to be in a certain state before an action takes place. In a typical scenario, this requires some setup. Either the data will need to be manipulated or the application will need to be put into a specific state or both!

Take Action

Once the state and/or environment is in the predefined state, it’s time to take action! The test driver will run the test, either through calling an application’s API or user interface or by running the code directly. The test driver is responsible for “driving” the tests, but the test management system takes on the responsibility of coordinating everything, including reporting results.

Report Results

A test automation system will record and report results. These results may come in a number of different formats and may even create problem tickets or bugs in a work tracking system. The basic result, however, is a pass or fail. Usually, there is a green or red indicator for each test scenario to indicate pass or fail.

Sometimes, tests are inconclusive or don’t run for some reason. When this happens, the automation system will have a full log of the output for developers to review. This log helps them track down the issue. Ideally, they’ll be able to replay the scenario once they’ve put a fix in place.

The Bottom Line

The bottom line is this: test automation helps improve quality with speed. But not all testing can be automated. There’s definitely an investment involved. With so many types of tests, it’s important that you get the right mix. The test pyramid is a simple rule of thumb to help get this right. It says most tests should be unit tests, followed by service tests, then UI tests.

A test automation system coordinates testing concerns, including managing test data, running tests, and tracking results. Test automation is the next step for teams that are becoming overwhelmed by the burden of repeating the same manual tests that should be automated. 

Author bio: This post was written by Phil Vuollet. Phil uses software to automate processes to improve efficiency and repeatability. He writes about topics relevant to technology and business, occasionally gives talks on the same topics, and is a family man who enjoys playing soccer and board games with his children.

When we talk about automation applied to the testing field, there are two expressions you’ll often hear thrown around: “automated testing” and “test automation.” But wait a minute. Aren’t automated testing and test automation the same thing? As it turns out, no. They are related concepts, but each one has a very specific meaning and purpose. And make no mistake: in order for your software organization to thrive, you’ll need both.

If up until now you thought that these two concepts are the same, don’t feel bad. For starters, you’re definitely not alone. People often mistake the two terms for one another. And secondly, your confusion ends today. That’s what this post is all about, after all.

We’ll begin by defining the two terms. Then we’ll dive a little deeper into each one of them, explaining their importance in an organization, also showing how they differently affect each role inside the team. Finally, we part ways by giving some additional tips. Let’s get started!

Defining “Automated Testing”

Right at the start of this post, we claim that “automated testing” and “test automation” aren’t the same thing, even if their names seem to imply otherwise. We’ve also told you that both are critical for your overall testing strategy. Finally, after a brief detour on the importance of testing in general, it’s time to define both approaches, clarifying how they are different—and similar—to one another.

Let’s start out by defining automated testing. And to be fair, that couldn’t be easier. “Automated testing” is just the process of automatically running a specific set of tests and verifying their results, instead of doing it manually. When you run your suite of unit tests on your development machine, you’re performing automated testing. When the CI server runs all of your unit and integration tests when someone pushes new commits to it, it’s also performing automated testing.

That’s pretty much all there is to it. With that out of the way, let’s continue to “test automation”, where things will probably be a little bit trickier.

Defining “Test Automation”

Now it’s time to define “test automation.” What is it and how it differs from the definition you’ve just read in the previous section? While automated testing is basically the running of automated tests, “test automation” is a way broader concept. It refers to completely automating the whole process of managing the different testing needs inside an organization.

Out of context, the line above most likely doesn’t explain much. Things will get a lot easier in the following sections, in which we’ll contextualize the two approaches, explaining how they fit into the overall quality strategy of a software organization and who each one benefits the most.

Automated Testing: Who Does It Help?

To really draw the distinction between automated testing and test automation, it’s important to show how they affect the different roles inside a software organization. Let’s start with automated testing. Which role in an organization benefits the most from this approach? The way I see it, the answer is clear: developers. And why is that?

A Unit Testing Example

For the sake of the argument, let’s focus on unit testing for a while. “Unit tests” is not a really great name if you think about it. No one seems to agree on what “unit” really means, for starters. But the main problem is that they aren’t really “tests.” The best way to think about unit tests is to consider them executable specifications. The term “programmer tests” has also been proposed a few times, and it makes sense. Unit tests are written by programmers, for programmers, in order to “prove”, at least to some degree of confidence, that a given piece of code really does what it’s supposed to do.

Automated Testing Is a Developer Confidence Booster

It isn’t that much of a stretch to expand our definition of “executable specifications” to also include integration tests. A common definition of integration tests would be “unit tests, kind of, but they use the real database/filesystem/etc instead of mocking/stubbing it.” Let’s do just that, then: let’s stretch our definition of “executable specifications” in order to also include integration tests. And let’s go further and pull all of that under the “automated testing” umbrella. What do we get from all of that defining and stretching?

The answer seems clear to me: the main beneficiary of “automated testing” in a software organization is the developer. Or, in other words: automated testing is a tool to achieve developer sanity. It allows developers to fearlessly refactor, knowing that they have a reliable safety net in the form of a comprehensive test suite that will warn them if they break something.

Test Automation: Who Does It Help?

We’ll now turn our focus to test automation and repeat the same analysis we just did with automated testing. Who benefits the most from test automation? The answer, again, is clear: the organization as a whole, with a special focus on the DevOps professionals. To understand why, we have to go back in time, to the (not so) good old days of the waterfall methodology.

Once Upon A Time, During The Dark Ages

In organizations that employ this methodology (and similar ones), testing is nothing but another phase in the whole cycle. And the testing phase tends to happen at the end of the cycle. More often than not, this is too little, too late. The feedback cycles slow down, reducing the usefulness and efficiency of the whole testing strategy. Nowadays, companies are moving toward a world of continuous development, deployment. They can’t afford to wait forever for the feedback from testing. The testing itself has to be continuous so that quality can be ensured at all phases of development. Automation can come in handy in this situation.

Here Comes The Enlightenment (i.e. Continuous Testing)

Automation is the missing link. It can provide the speed and consistency that you need in order to have continuous testing up-and-running. Manually managing the testing needs in such an environment would not only be incredibly hard and error-prone but also time-consuming, which in fact defeats the whole point of doing it in the first place.

By automating the managing of said testing needs, test automation helps organizations keep quality at the highest standards during the whole development cycle. Furthermore, it frees QA professionals from managing the minutia of testing needs, which means they can focus on creating more efficient case tests to ensure the application’s quality.

Automated Testing And Test Automation. You Need Both. ASAP

In today’s post, we’ve clarified the confusion between “Automated Testing” and “Test Automation”. Yep, the naming could use a little less ambiguity. But the concepts themselves are really different. However, they’re equally important if you want your organization to continue delivering software to the highest standards of quality. Automated testing is important, especially for giving your developers the confidence to fearlessly refactor their code,  which contributes to higher quality code, without a doubt.

But test automation becomes critical when your company starts moving towards a continuous development/continuous deployment scenario. Do you want your company to continue thriving in this brave new DevOps world? Put test automation to work for you and your organization.

This post was written by Carlos Schults.Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.

So your company is going Agile. If you’re a tester or QA person, that can bring about frightening thoughts about what your world will look like. For example, will you now be expected to test a codebase that changes daily, with no breaks or developer deadlines? Alternatively, you’ve heard that some teams do away with QA and testers completely once they go agile. How will you survive in this scenario?

Fear not, for there’s still a need for QA in our new agile world. However, instead of the old paradigm, where your goal involves finding defects, your new goals include preventing defects and helping ship products faster. And with this guide, you’ll be on your way to creating a robust and healthy QA process.

In this post, we’re going to take a look at what going agile means from a QA perspective and how you can change to thrive in this environment.

What Does Going Agile Mean for QA?

When reviewing the values and principles of agile, we don’t see testing addressed as clearly as we’d like. In fact, we’re not even sure if traditional QA has a place. But we need to read between the lines and picture what this means for us. Here are the four values that came from the Agile Manifesto:

  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan
  • Individuals and interactions over processes and tools

Now let’s look at how these values fit together with QA.

Working Software Over Comprehensive Documentation

For the first value, we learn that working software comes before comprehensive documentation. Now, this doesn’t mean you can’t have any documentation, no matter what your development team tells you.

It means you should create documentation only if it provides value. In the testing world, we should spend less time writing out manual test plan documentation and more time automating the tests.

And we do have some things that will always need documentation. For example, putting together documentation for the customer can be a big part of QA’s job, as they’re most familiar with how to use the system.

As another example, you’ll need to document how to recreate a defect within the application. But don’t spend too much time filling out forms, tickets, or any other unnecessary work. Look for streamlined ways you can clearly communicate the defect to the development team without much overhead.

Whatever you do, make sure it drives the result of having working software.

Customer Collaboration Over Contract Negotiation

In the previous section, we mentioned putting together documentation for customers. That should be one of the last steps. Before the documentation, we should be working closely with our customers to understand how they use the system and why they might run into defects or bugs. Learn what their pain points are so you can share what you’ve learned with the development team.

Overall, your relationship with the customer should look like a partnership, not a criminal case.

Responding to Change Over Following a Plan

One big concern testers have when it comes to delivering software frequently involves keeping up with the testing. Everything changes in agile, and fairly quickly. Stop trying to build a plan for a future that might not come true. Instead, work to put in automated tests and guardrails so that you’re able to react quickly to changing priorities and functionality.

Sprinkle in as much automation as you can so that when change happens, you’re ready.

Individuals and Interactions Over Process and Tools

In our old waterfall world, the QA team would get a few weeks prior to release to learn and test all the new functionality. And, as waterfall usually went, those few weeks would shrink from four to three to two weeks max as software development deadlines slipped. We still needed to ship our product, and the QA team would feel much of this pain.

In this new agile paradigm, QA should be involved throughout the whole process. In fact, testers should interact daily with the team. Whether it’s pairing on a task or discussing how a story should be tested, they’re integral in discussions around testing and quality.

Another important note: agile teams have frequent ad hoc design discussions, so testers should be co-located and available to jump in on those discussions whenever they happen. So don’t create more paperwork and process to share ideas—instead, involve yourself in the whole software delivery process.

What Should QA Do Now?

Now that we’ve covered the agile values, we still might have questions as to how we’re supposed to work when agile. In this section, let’s cover some of the ways that our QA skills can help the team deliver quality software in a continuous way.

From a day-to-day standpoint, involve yourself in stand-ups, retros, and demos. Stay on top of the work the team is doing so you’re aware of dependencies between stories and what additional testing will be necessary.

What else should you do?

Join the Team

The biggest change involves not having a separate QA team. Instead, QA should be part of the development team. Since we’re part of the team, we’re able to assist in continuous testing, instead of doing it all at the end of the sprint.

Also, when we’re on the team from start to finish, we have the chance to affect the delivery of the product much more. Why is that? In an agile team, roles aren’t as clearly defined. You may be able to write your own story by stepping in as a testing coach or automation SME.

When the team focuses on delivering working software faster to the customer, you’ll have the opportunity to provide flexible support to your teammates.

Understand the Big Picture

While the developers dive into specific details of the task they’re working on, you have the responsibility of understanding the big picture. You’ll need deep knowledge of the working system to define tests and scenarios that might not be apparent from the outside. And with that whole-system view, you’ll be able to write the documentation that’s necessary for your customers.

Additionally, keep in mind application design. Understanding your product’s design can help identify interesting edge cases that should be considered.

Understanding the big picture can also help with a blind spot the engineers might miss. The big picture will help you uncover assumptions the team has made about the system. And those assumptions could leave a gaping quality hole in your end product. Therefore, you’ll always want to have a keen understanding of the system as a whole.

Automate Your Test Suite

Since agile testing is done in parallel to development, automation becomes critical. Without it, you’ll find yourself retesting the same functionality three or more times in a sprint or iteration.

Your developers will write unit and integration tests. However, they write those tests to a spec and typically can’t cover the use case from end to end. This is where you come in as a QA professional.

Now, you may think that you’ll need to learn to code to automate all these tests. Learning to code—or at least learning some light scripting—definitely has its benefits. But also take advantage of tools like Testim to easily put together automated functional tests for your product.

These automated tests can be based on the acceptance criteria in developer stories. Create tests that ensure that your product works correctly. And look for where there may be gaps in understanding, and therefore a need for additional tests.

Test Manually for the Right Reasons

Exploratory testing can identify gaps in automated tests. This shouldn’t involve going through the same manual processes over and over.

Instead, this time is for asking those “what if?” questions.

What if the order gets canceled after the shipping paperwork gets printed? What if I lose my internet connection while sending the payment?

Put time into exploratory testing to give your team more confidence that they didn’t miss a critical bug or loss of functionality.

Enable Faster Delivery

As mentioned before, because it’s so critical, we need to enable faster delivery of software. As the QA person on the team, this can mean a lot of different things. For example, help your team define what a completed story looks like. Assist with story creation. Help the team identify what’s missing or out of scope for a particular story. Guide them to ensure that you deliver your product with quality.

But most importantly, don’t be a bottleneck or a block. Enable the team to go faster.

Share Testing Practices

As QA, you have an understanding of good testing practices. Help the team by sharing your experience. Consider how a feature would be easier to test. Perhaps look for ways to make the design more testable.

Continuously Improve

Though quality is the whole team’s job, you should be driving continuous improvement of testing practices. Work to become an expert on agile testing methodologies and strategies. Help the devs create integration tests that aren’t flaky but ensure that the system is working. Help build maintainable and healthy test suites.

Summary

In an agile team, everyone is responsible for quality. And this can be your time to shine by sharing your expertise with the team. Work with your team to build quality into the development process. As I mentioned earlier, the goal no longer involves finding bugs and defects, but preventing them.

Find ways to use agile practices and values to make that happen in collaboration with the team and the customers. You’ll be able to find comfort in your new agile workplace without fear or apprehension.

Author bio: This post was written by Sylvia Fronczak. Sylvia is a software developer that has worked in various industries with various software methodologies. She’s currently focused on design practices that the whole team can own, understand, and evolve over time.

Selenium tests have the reputation of easily becoming fragile tests. We’ll look at some common causes of fragile Selenium tests, how you can alleviate some of these issues, and how Testim can provide extra value and robustness to your UI tests.

What’s a Fragile Test?

A Selenium test can be fragile just like any other automated test. A fragile test is when changes that seemingly shouldn’t influence the test result do so anyway. An example that most of us have encountered is when we change one piece of code and break a test that we believe shouldn’t break. But there can be other influencing factors: test data, the current date and time or other pieces of context, or even other tests. Whenever these factors influence the test outcome, we can say we have a fragile test.

Fragile Selenium Tests

Selenium is a tool to automate the browser, giving us the power to write automated tests for our (web) UI. This entails that we will execute our entire application more or less. A lot of moving parts means these Selenium tests can become fragile more easily than simple unit tests.

Let’s look at some causes of fragile Selenium tests and how we can improve them so they become more robust. The causes we’ll cover are

  • tight coupling to the implementation
  • external services
  • asynchrony
  • fragile data
  • context

Tight Coupling to the Implementation

Just like unit or integration tests, Selenium tests can be written so that they’re tightly coupled to the implementation. However, when the implementation changes, this means our tests need to change too, even if the public or visual behavior hasn’t changed.

For example, let’s say we’ve written our test to click on a button with ID “login-button.” When this ID is changed to something else, for whatever reason, our test will fail, even if the test case still works fine when performed manually.

This is because the test is tightly coupled to the specific implementation. How can we decouple our test from the implementation?

In this specific example, we could improve the test by having the test call a helper method that knows about the implementation. So instead of clicking the button with the ID “login-button,” we can make it call a method named “Login” and pass the necessary parameters. If all of our tests use this helper method, we’ll only have to change one piece of code when the implementation changes.

Now, let’s take this a step further and group our helper methods inside a class per page. This is the Page Object design pattern. If something on a page changes, we know where to update the implementation for our tests—in the corresponding page object.

Can we do better? Yes, we can. Thanks to Dynamic Locators and AI, Testim can identify the page element we need, even if its attributes have changed. Check it out:

A screenshot of a cell phone

Description automatically generated

This means that the QA team can record tests in their browser and they will still run fine if some underlying implementation detail changes.

External Services

Another common cause of fragile tests, especially when testing a complete application, is that we rely on external services to work as we expect. These can be particularly hard to control. But when they don’t behave as they should for the test, or if they aren’t available, the test fails. And this can happen even if we haven’t changed or broken anything on our side.

In this case, we could make our tests more robust by creating a mock service and having our application call that service. Otherwise, we’re also testing the external service and setting ourselves up for fragile tests.

By creating a mock service, we have full control over the responses. We can define what data is returned, but we can also simulate error responses, timeouts, or slow responses. These are things that are harder to imitate when we’re using the real external service.

Asynchrony

When we’re testing an application, we often have test cases where we need to wait for a certain event. For example, we might have an application that makes an AJAX call and renders the result on the page. We now have to tell Selenium to wait until the result is rendered on the page. But for how long?

Luckily, Selenium has explicit and implicit waits. They allow us to tell Selenium to wait until a certain condition is met. In our example, we would wait until the result of our AJAX call has been rendered. When that has happened, we can continue our test.

Fragile Data

Most applications that we test with Selenium will require some data storage. Our Selenium tests can become fragile because of the data for two reasons: either the data changes or certain tests continue with data from previous tests.

Data Changes

When we set up a database for our tests, the amount of data we pump into it can become quite large. Consequently, when we make a small change to this test data for one test, we might break another test that depended on this data. This is especially true for larger test suites where it has become unclear which pieces of data are necessary for which tests.

A possible solution is to populate the database with separate data for each test. However, that can lead to a large amount of test data that can be difficult to maintain over time.

A better option is to populate your database as part of each test. You could start with basic data that doesn’t change, and then have each test add the data that it needs.

Interacting Tests

Another potential issue with data is when one test changes the data of another test. Take a simple example where a test changes some identifier of a record (a product ID for example). At the end of the test, it changes it back so that another test can find that same record. However, if the first test fails to revert the change, the second test won’t find the record and will also fail.

This is a case of interacting tests: a test fails because it has been influenced by another test, even though the failing test runs fine when it’s run individually.

The best solution is to give each test its own set of data. We can achieve this by spinning up a new database for each test. This might sound like a complex solution, but we could leverage containers for this: start a container with a database, fill it with the necessary data, run the test and end by shutting down the container. Rinse and repeat for other tests.

If a separate database for each test isn’t an option, we could look at solutions like in-memory databases or libraries that reset our database to the state before the test (like Respawn for .NET).

Context

The typical example of a fragile Selenium test occurs because of a context, like tests that depend on certain dates or times. This is true for every kind of test, so it also applies to Selenium tests.

When your tests depend on certain dates or times, they might suddenly fail when the conditions (temporarily) don’t apply. This is often the case on leap days, but depending on your business logic, other dates may cause your test to fail too.

In unit tests, the date and time can easily be mocked out, but it’s often much more difficult in Selenium tests. To make matters worse, you’re dealing with two clocks: the “server” clock (where your back-end application is running) and the browser’s clock.

There are many options here, and the solution for you depends on your specific case.

On the server-side, you could wrap calls to the system clock in a service and inject a mock service anywhere you need it in your tests. In production, you would then inject the service that uses the real clock. There’s also libfaketime for Linux and Mac OS X, which might be an easier route.

You can mock the browser’s date and time using TimeShift.js and inject this in your Selenium tests.

Is It Really That Hard?

All of these problems have solutions, but they’re often non-trivial to implement, especially in legacy applications. Also, many of these solutions require time and effort from developers, which you may not be able to spare. But if the tester (a QA department, the product owner, or the end-user) can work together with the developers, then you can achieve solid tests more easily.

Also, take a look at Testim. It can cover more than Selenium does out of the box and it provides a useful platform for developers, the QA team, and end-users to collaborate.

What if you need to verify the contents of a downloaded Word or Excel file? Or if your application has email or SMS integration? And wouldn’t you like to have the tester create your UI tests instead of taking time from your developers?

Testim provides an easy recording function for non-developers to write tests. This also allows non-developers to report a bug and immediately include a test to reproduce the issue. Traditionally, similar recording tools have created fragile tests because they’re too close to the implementation. But as we mentioned, Testim takes a different approach and leads to more robust tests.

For QA professionals, there are powerful features, like running tests with different data sets, conditional steps, and screenshots. Testim also learns about your tests over time, making your tests more stable the more you run the tests, even when implementation details change.

Testim allows you to create robust UI tests for your application that need minimal maintenance and provide maximum value to both developers, QA professionals, and end-users.

Author bio: This post was written by Peter Morlion. Peter is a passionate programmer that helps people and companies improve the quality of their code, especially in legacy codebases. He firmly believes that industry best practices are invaluable when working towards this goal, and his specialties include TDD, DI, and SOLID principles.

We could say automation is the whole raison d’être for software development. As developers, we seek to employ automation in order to solve problems with more efficiency than before. And we solve problems not only for our clients or employers but also for ourselves. We write scripts and software utilities to automate the packaging and deploy of our applications. We employ plugins and other tools that can automatically check our code for common mistakes and even fix some of them.

Another instance of automation is browser automation. And that’s what this post is all about. If the term doesn’t ring a bell, never fear. The post will do justice to its title and answer the question it poses. And after defining the term, we’ll proceed to show scenarios where browser automation is the right tool for the job. Then, to wrap up the article, we’re going to give you tips so you can get started with browser automation ASAP. That’s what, why, and how in just a single post.

Let’s get started.

Browser Automation: Definition

We’ll start by defining browser automation. We could try something like “‘Browser automation’ means to automate the usage of a web browser” and leave it at that. But that would make for a definition that’s both technically correct and useless unless we define automation. That word is one that we often take for granted, so I think it might be useful to actually take a step back and define it.

Defining Automation

Here’s what Merriam Webster has to say about automation:

1: the technique of making an apparatus, a process, or a system operate automatically
2: the state of being operated automatically
3: automatically controlled operation of an apparatus, process, or system by mechanical or electronic devices that take the place of human labor

Interesting. Now take a look at Wikipedia’s definition:

Automation is the technology by which a process or procedure is performed with minimum human assistance

Bots Don’t Get Bored

What do those two definitions have in common? At least for me, the point that’s really obvious is that automation seeks to remove human intervention from the equation. And why would we want to do that? While we humans are great at a lot of things, we’re also terrible at a lot of things—especially tasks of a repetitive nature. When performing repetitive, boring tasks, we tend to get…well, bored. Our mind easily zooms out of focus as we enter autopilot mode, and soon we’re making mistakes.

But since we’re a pretty smart species, we came up with a device that’s way better—and faster—than we are at performing repetitive tasks. And of course, you know I’m talking about the computer. With all of that in mind, here comes my upgraded definition for browser automation:

Browser automation is the process of automatically performing operations on a web browser, in order to achieve speed and efficiency levels that wouldn’t be possible with human intervention.

It’s far from being a perfect definition, but it’s already something we can work with.

Browser Automation: Scenarios for Usage

Why would someone want to automate the operation of a web browser? As it turns out, there are plenty of use cases for browser automation, and that’s what this section will cover.

Automatic Verification of Broken Links

It’s frustrating to click on a link only to see the infamous “404 Not Found” message. If you have a site, then you should definitely fix the broken links in it or, alternatively, delete them. But before you go about doing that, you first need to find them. This might not prove too much of a problem if your site has just a handful of pages. But think about a complex database-backed portal, with hundreds or even thousands of pages, mostly dynamically generated!

Now, what before was a minor nuisance becomes a herculean task. And that’s where browser automation fits in. You can employ tools that will automatically go through your site, verifying every link and reporting the ones that are broken.

Performance Testing

Performance is a huge concern when talking about software development. In this era of high-speed connections, most users will get frustrated if the site they’re trying to access is even slightly slower than they’d expected. Besides, Google itself penalizes slower sites on its search result pages.

Browser automation can also help with that. It’s possible to employ browser automation tools to do load and performance testing on your websites. This way, you can not only verify your web app’s performance on the average case but also predict its behavior under the stress of traffic that’s higher than usual.

Web Data Extraction

When the World Wide Web was invented 30 years ago, its purpose was to allow researchers to easily propagate their works. In other words, humans put stuff on the web for other humans to consume. In the decades that followed, we watched a rise in the non-human use of the web.

Browser automation definitely plays a part in this. Web data extraction, also known as web scraping, is another use case for browser automation tools. From data mining to content scraping to product price monitoring, the sky is the limit for the uses of web data extraction.

Automated Testing

Last but not least, we have what’s probably the poster child of browser automation use cases: automated testing. Yes, we just talked about performance testing and broken link verification, and those things are also automated tests. But here we’re talking of general, end-to-end functional tests. For instance, you might want to check that, when informing an invalid login and/or password at a login screen, an error message is displayed.

Such tests really shine when you can effectively use them as regression tests. If a fixed problem returns in the feature, you have a safety net that will warn you. And that safety net is way faster and more efficient than human testers—at a fraction of the cost.

How to Get Started With Browser Automation

Learning browser automation can seem like a daunting task. It’s an enormous topic, and there’s a lot to know. But it’s no different from any other area in tech. Approach it the way you would approach learning a new programing language or framework: by doing it.

First, think of at least one use case for browser automation in your current organization. We’ve just shown you some, and I’m sure you can think of many more. Some people call this “scratching your own itch,” and it’s an effective way of motivating yourself to learn something.

As soon as you have a small, discrete problem you think you can solve with browser automation, starting looking around for tutorials on how to get started with some of the available tools. When you get stuck, look for help in the documentation of the tool you’re trying to use. You can also search for help on Stack Overflow under the “browser automation” tag. And of course, there’s always Google.

Build a minimum viable example of browser automation in place. As soon as you get something that works, no matter how simple it is, that’s a milestone. You can use it as a foundation upon which to build more sophisticated and complex approaches.

Where to Go From Here?

Today’s post was meant to give you a quick primer on browser automation. We started by defining the term, then proceeded to show some common use cases for the technique. Finally, we gave you tips on how to get started.

As I like to say when writing introductory articles like this one, this was just the tip of the iceberg. There’s much more to browser automation than what could be covered by a single blog post. Where do you go from here then?

There’s no silver bullet: the answer is to keep studying and practicing. Continue to evolve your first minimum test suite and learn from it. You should also keep an eye out for what’s happening in the world. There are interesting developments, such as the use of machine learning to help developers with the creation, running, and maintenance of test cases.

Additionally, stay tuned in this blog for more automation-related content. Thanks for reading and see you next time!

This post was written by Carlos Schults. Carlos is a .NET software developer with experience in both desktop and web development, and he’s now trying his hand at mobile. He has a passion for writing clean and concise code, and he’s interested in practices that help you improve app health, such as code review, automated testing, and continuous build.

Be smart & save time...
Simply automate.

Get Started Free