Scalable testing infrastructure

When a startup is in its very early stages, rapid iteration and dynamism are at the top of its priorities.
The ability to do so, while maintaining a stable and high-quality product, is a big challenge facing the R&D group. We want to release features as quickly as possible, but this rapid velocity cane cause conflicts when writing in-depth, comprehensive tests. To ensure a good flow from development to test to production, it is important to build and maintain an automated, stable, per-change testing infrastructure.

In this blog, we’ll discuss how Lumigo built our testing infrastructure to run two main test methods:
End-to-end (which includes full integration with the serverless BE)
Vanilla UI.

Today, Lumigo has about 700 E2E & UI tests that eventually run per PR as part of our CI pipeline, and that number continues to grow.

Below is an example of a pipeline that contains an E2E tests job:

To keep up with such rapid growth, and to do so efficiently and with proper control, we had to build a customized testing infrastructure.

Customized tests infrastructure is important

Customized test infrastructure is important when setting up testing environments and frameworks.
It saves time and defines standards in terms of architecture, patterns, and code style.

When we started to write our first tests, we had to customize our testing framework to our own needs. This made the first tests slower to write but made all subsequent tests much faster to implement and reduced the amount of work in code reviews. Setting the standards in terms of architecture, patterns and code style is important for readability and helps maintain order when the number of tests is scaling up.

Why Cypress?

Our platform is written in Nx and Angular, and by default, Nx will use Cyprus to create E2E tests when building a new frontend application. Designed to make you more productive, Cypress is a great tool for E2E and UI testing. It records videos of test runs out-of-the-box and provides fast feedback, full access to the application, time travel, and most importantly, a great developer experience.

As part of our infrastructure, we also use some tools we wrote ourselves, in conjunction with tools that Cypress offers:

Flaky tests management

Flaky tests are a serious problem for development teams. Unreliable tests slow down development velocity while teams try to diagnose test failures. Lumigo uses two key tools to prevent this kind of obstacle:

1. Burning tests
   Our team developed a “burn” tool that will run the same test, again and again, to confirm it is flake-free. We present this tool locally on new tests, even before they reach CI.
   
2. Test retries
   By enabling Cypress test retries, you can flag, detect, and track flaky tests from your test runs in your CI/CD pipeline.
   

Save time and money with Parallelization

Running tests on one machine can take a long time if your project has many tests.
In order to reduce running time, save on costs and enable faster development, we use the parallelization tool that Cypress offers.

Cross browser testing

Our customers use a variety of browsers. To ensure the stability of our platform in each of them, we run the tests on top different browsers.
Example:

Writing a maintainable testing infrastructure

For Lumigo’s testing infrastructure we have developed two key additions on top of Cypress:

1. Cypress custom commands that resemble human-readable test steps.
2. A set of custom utils and helper methods that correspond to our application’s key components.

In addition, we document common procedures in testing documentation so that even a developer who wants to write his first test, will be able to do this easily and quickly while maintaining high standards.

Exploring custom utils and helper functions

Often, you see test code like this:

[code lang=”text”]cy.get(‘.filter-container .option-input’).type(‘us-east-1’)
cy.wait(‘someAlias’).its(‘request’).then(({ body }) => {
assert.equal(body.region, ‘us-east-1’);
});
cy.get(‘.grid .container .row span’).should(‘include.text’, ‘us-east-1’)[/code]

Just by looking at this code, you might not be able to understand the action the user is performing. Although there are some hints, it’s not clear what these elements represent so it’s not so easy to draw a conclusion. In all likelihood, if you have returned to this code, it’s because something has failed and you need to solve it fast, and a delay in understanding the context of the text will result in time lost.

Creating key components as part of test scenarios and are worth dedicating their own API.
We found that creating a class to a key component is the best way to maintain such a process.
Each key component has its own file under the “utils” folder.

Encapsulation example

We could declare a Filter class:

[code lang=”text”]export abstract class Filter {
constructor(private selector: string) {
}

abstract apply(term: string): this;

abstract verifyRequeast(alias: string, body: any): this;

abstract verifyGrid(details: string[]): this;[/code]

Apart from making the code more organized, a healthy encapsulation is created between the test writer and the action implementation. As a result, a similar test can be displayed as follows:

[code lang=”text”]new SelectFilter(‘select-region’)
.apply(‘us-east-1’)
.verifyRequest(‘someAlias’, { region: ‘us-east-1’ })
.verifyGrid([‘us-east-1’])[/code]

This code clearly tells the reader which action was performed and what the expected outcome should be,

Custom Commands to save the day
Cypress comes with its own API for creating custom commands and overwriting existing ones. Custom Commands can save you time, clean up code, and make it more readable.

We are declaring the commands in a cypress/support/command.ts file and placing each command in its own file. It’s clearer to separate the function itself from the code that loads the custom command.

For example:

[code lang=”text”]declare global {
namespace Cypress {
interface Chainable {
login(options: LoginOptions): void;

switchProject(projectId: string): void;

getBySel(selector: string): Chainable;
}
}[/code]

[code lang=”text”]Cypress.Commands.add(‘switchProject’, (projectId: string) => {
cy.getBySel(‘projects-menu’).click();
cy.getBySel(`project-${projectId}`).click();
});[/code]

With the help of custom commands, we can perform complex user actions in one-liner items. If you find yourself repeatedly writing the same, long set of steps to perform a necessary user action, there is probably room to create its own command.

Conclusion

We all agree that tests are super important in every CI/CD pipeline. Good test maintenance gives the developer a sense of security when pushing or changing code. With a scalable tests infrastructure, this can be done easily and efficiently, as we did for Lumigo’s serverless backend and vanilla UI.

If you have questions or any more ideas for improving the scalability and stability of a test set, we’d love to hear from you!

Feel free to contact me at ido@lumigo.io or on my LinkedIn profile.