Effective time management on long tests

Posted on by avraampiperidis

This article takes place on the problems and the solutions of the time demanding e2e tests.

Intro

Except for the unit tests and integration tests, that run instantly on every commit/push through a CI circle, e2e opposes problems like a long time to complete, more common failures, and more complex requirements/architecture. Some times is difficult to handle e2e and include it inside the whole CI circle. We going to see how to keep e2e part of the CI circle with transparency without affecting the everyday workflow.

The Night-Time

At night there is a good and very large open window that we can exploit and make the ci server do something useful without our attendance. I always intrigued by the idea while everyone is sleeping, something else is working to our benefit.

E2E could end up requiring a lot of time

Usually, the e2e tests could require a lot of time to complete depending on the different variations they ran like environment initialization and waits for UI elements for better handling. It could lead to many hours for a complete e2e/acceptance test to finish and give back the report.

E2E should NOT affect the development process

At working hours we don’t want to slow down the development process for waiting some long-running e2e to finish every time commit/push is happening. But on the other hand, we want these e2e tests to run against every new code change.

So an ideal ci diagram could look like this..

TestCase. From 10 minutes to 6 hours.

A complete e2e test could take from some minutes to some hours to finish. But let’s start with a simple e2e scenario and see how things can get more time demanding very easily. Let’s start by assuming our e2e takes up 10 minutes to end, not very long right? Well indeed, 10 minutes is a decent time, not something to consider about. On the other hand, developers do have to wait 10 minutes for tests to finish. Now, these 10 minutes are for testing in one browser only. We do actually want to tests other browsers as well so we are having the following browsers for the test. Firefox,Chrome,IE,Edge,Opera,Safari.

We can observe a dramatic increase in time from 10 minutes to 10 x 6 = 60 minutes. Almost one hour for the cross-browser e2e tests to complete, assuming all these running on the same ci server in serial. Its a lot of time comparing for 10 minutes, but not that much.

Next, we can add different resolutions, mobile, HD, full-HD. Now based on this we end up to 3 hours of testing. We can take it one step further by adding different branches or OS, so we would end up with more than 6 hours of testing. And all these from a simple 10 minute e2e.

Nightly test and parallelism to the rescue.

We can consider the below solutions

1)  Execute tests across multiple machines using parallelization.

2)  Execute at night or at hours not bothering anyone using Serial execution.

3)  Combine the above solutions depending on the needs and the available machines/resources.

Parallel execution

With parallel execution, we can execute every step or stage at ‘parallel’. Which means all stages will run simultaneously.

Serial and Parallel comparison.

Serial execution
Parallel execution

Parallel drawback

One big drawback of parallel execution is that it requires a lot of hardware resources. As many parallel executions we have, the hardware resources are getting even higher.

This depending on what the e2e is doing. Some tests are very light but some others are very resource-demanding. It might be a mobile app that requires a VM or a frontend that needs an app inside a docker to be initialized first. Usually, when doing e2e it requires a lot of setup/teardown process. All this depending on the number and size of software we want to test.

For simple/small machines we could use a small number of parallel jobs running at the same time.

For a big machine/data center we could in theory run the 3-hour tests in 10 minutes by dividing every case/step to a different parallel execution. This could even take place to different VM or even better every test to its own dockerized environment.

So worst case scenario 3 or 6 hours. Best case 10 minutes.

Parallel Pipeline script

A sample Jenkins pipeline code that executes tests in different browsers in parallel could look like this. 

//Jenkinsfile should be at your code repo
 pipeline {
   agent any
   stages {
       stage(‘Clone’) {
         steps {
             // prepare
          }
     }
   }
   stage(‘Parallel-Tests’) {
     steps {
       parallel(
         test_chrome:{
           //
         },
         test_firefox:{
           //
         }
         …
       )
     }
   }
 }

The agent section defines the execution context of the pipeline. It can be none(Jenkins default), or for example some custom docker image. For more info for the agent section can be found here.

Final words of wisdom

In conclusion, if you have the resources available go for parallel execution. If not, try to keep the parallel execution number in relation to the available resources.

If the whole night is not enough, we can always increase the time gap from the afternoon to early in the morning.

For example, where I work, we applied a low number of parallel works and the CI server is working a few hours every night, so in the morning we have the reports ready for us. If the hours increase a lot, we could always add some extra resources or machines and increase the parallel works.

last and important remember to always analyze and build proper reports for all this data generated from tests, including helpful resource monitoring.

Software Engineer