Principle's of Automation Testing

Well, this is quite a heavy topic that I have picked on to write about.

It's an age-old question on how to build quality software. Over the years testing has become an essential step in building quality software.

The specifics of how to approach testing are still very much in debate and changed over the years. Nonetheless, I believe a few principles have emerged over the years which I would like to share.

Let us look at some of the questions one might ask before one starts a project:

1. When is the right time in the project lifecycle to start testing?

2. Is testing only QA's job?

3. Does the way a developer builds code affects the testability of software?

4. Is it okay to mock stuff? If yes how much?

5. What is the ideal way the tests should look like?

6. How long should a given suite of tests run?

.... etc.

I hope this give's you an idea about how much difficulty there can be when it comes to testing.

So let's start off the crux of the post and delve into a series of points that will answer the questions above :

Testing cannot be an afterthought

This is an important point that needs to be in everyone's mind while starting off a project.

If this is not followed the result of the project will be hard to predict/buggy and over time hard to grow. Even the use of expensive tools will not change the result if testing starts toward the end.

I understand this will be disappointing to a lot of folks but this has been my experience.

So if I say testing cannot be an afterthought does that mean devs also own this? - The answer is Yes! Building quality software is as much a responsibility of a dev as it is of a QA engineer.

Why so?

If you think about it software is a sum of a lot of pieces. Pieces like data structures/functions/classes etc.

Each piece of code can have N different paths of execution. Combine those with other pieces and the complexity increases quite a bit.

I hope that answers the question. Testing should happen right from those individual levels and its combination too. Otherwise, there is no way to have a good level of confidence in the quality of the output.

Developer Approach to Code

Now that we established testing cannot be an afterthought, let's come at it from a dev's perspective as to how to build code that can be tested in the first place. In this area lot of ideas/patterns have emerged the most popular of them being the practice of TDD i.e. Test Driven Development. The basis of the approach is to write a failing test corresponding to a requirement and then write the code to make the failing test pass and then you can refactor the code to do better all the while having the confidence of having the test be green.

This approach has been incredible for my personal workflow while developing code. It produces small well tested abstractions and grows as you go through more requirements. This way you get tests right from the beginning of the project lifecycle. Although this add's to developers' time it saves a ton later in terms of quality. Since bugs in production are a lot harder to debug than on your local system.

Other than that few pointers to keep the code testable:

1. Encapsulate behavior as much as possible in pure functions.

2. Keep the API surface minimal.

3. Make the contracts explicit as much as possible - if you are using a typed language encode that in types to further reduce possible misuse.

4. Grow abstractions in layers.

5. Hideaway imperative/complex bits using encapsulation mechanisms and expose a declarative API.

6. Hoist the parts of the code where side effects are present to the top. And preferably in a singular place.

This is not an exhaustive list but I think it's a good place to start from.

E2E Vs Integration Vs Unit

Now, these terms are used quite frequently in a testing context and usually along with a term called "Testing Pyramid".

The term "Testing Pyramid" refers to the following diagram:

Source: https://www.browserstack.com/guide/testing-pyramid-for-test-automation

So it basically says:

Unit Tests > Integration Tests > E2E Test

But let's define these types of tests in the first place:

Unit Test

A type of test which tests a "unit" of functionality.

the "unit" above could be a lot of things like:

1. function

2. class

3. API route

4. Module

5. React Component

6. ....

So based on your context "unit" could mean a lot of things.

Example:

function add(a, b) {
  return a + b;
}

// add.test.js

test("should add two numbers", () => {
  expect(add(1, 2)).toEqual(3);
});

Trade-Offs:

• Fast feedback loop

• High chance of mocking (reduces the reliability of the test).

Integration Test

A type of test that usually tests a combination of units.

Example:

function add(x, y) {
  return function (x) {
    return x + y;
  };
}

function multiple(x, y) {
  return function (x) {
    return x * y;
  };
}

function doubleAndAddOne(x) {
  const addOne = add(1);
  const double = multiple(2);
  return addOne(double(x));
}

test("should double and add one", () => {
  expect(doubleAndAddOne(5)).toEqual(11);
});

Trade-Offs:

• The typically slower feedback loop

• Typically lesser mocking

E2E Test:

This is where you test your entire application from a user perspective.

If you are in the web dev world, it would look different based on the tools and the language you use to test it.

A sample selenium test using JS:

const By = webdriver.By; // useful Locator utility to describe a query for a WebElement
// open a page, find autocomplete input by CSS selector, then get its value
driver
  .navigate()
  .to("http://path.to.test.app/")
  .then(() => driver.findElement(By.css(".autocomplete")))
  .then((element) => element.getAttribute("value"))
  .then((value) => console.log(value));

Trade-offs:

• Typically very slow feedback loop

• Typically no mocking - more correct.

Let's ponder why the pyramid is structured the way it is.

Given the trade-offs I have mentioned we can tell that the tests have been structured based on feedback loop time (cost):

1. Basically, unit tests run very fast so you can afford to have many of them and not incur many costs and if anything breaks it can be fixed at relatively high speed - correctness can be low if there is too much mocking.

2. Integration tests are just above the hierarchy and are relatively slower to give feedback so we want them to be lesser - but in terms of correctness, they are better since mocking is lesser.

3. in the same vein - E2E are slower to run but in terms of correctness, they are better/best.

The balance to maintain here is correctness and speed.

The pyramid shows the trade-offs involved and gives us a guideline on how to structure our tests.

The point is to look at the trade-offs involved and adapt. Tools like Cypress are good examples of how tools are changing the trade-offs and how we can adapt.

I hope this helps. Thanks for reading.