@badeball/cypress-parallel

cypress-parallel

Divides your test files into equal buckets and runs a single bucket. This is ideal for parallizing Cypress tests in a CI environment, without relying on external services, such as Cypress' Dashboard Service.

Table of Contents

• Installation
• How it works
• Usage
  • --node <index>:<total>
  • --knapsack <file>
  • --disable-knapsack-output
  • --unweighed-strategy "estimate" | "distribute"
• How to handle knapsack.json
• CI configuration
  • Gitlab CI
  • Other providers

Installation

$ npm install @badeball/cypress-parallel

How it works

1. It will search through your project for test files
2. A knapsack containing file weights is read (defaults to knapsack.json)
3. Tests are divided into N buckets
4. The Ith bucket is executed by passing --spec to Cypress with said bucket of files
5. The knapsack is overwritten with potentially new weights

N and I is determined either by a flag --node or by environment variables for some CI providers.

The overwritten knapsack can then be comitted back into VCS. This will allow the library to always divide your tests somewhat evenly among the nodes.

Usage

Below are all the configuration options and some extended explanation for some of them.

$ npx cypress-parallel --help
Usage: cypress-parallel [options]

Options:
  -v, --version                    output the version number
  --cypress-run-command <cmd>      specifies the command to run cypress (in non-interactive mode), defaults to 'npx cypress
                                   run' or 'yarn cypress run' depending on how invoked
  --node <index>:<count>           specifies number of buckets and which to run
  --knapsack <path>                specifies the path to the knapsack file (default: "knapsack.json")
  --read-knapsack <path>           specifies the path to the knapsack file to read
  --write-knapsack <path>          specifies the path to the knapsack file to write
  --disable-knapsack-output        disables knapsack output (default: false)
  --unweighed-strategy <strategy>  strategy to utilize for unweighed test files ('estimate' (default) | 'distribute')
                                   (default: "estimate")
  -h, --help                       display help for command

Unrecognized arguments are passed along to Cypress, so arguments such as -e / --env can be used as shown below

$ npx cypress-parallel --env foo=bar

--node <index>:<total>

The utility will automatically pick up node configuration for some CI providers. Otherwise you can specify node index and total node count using --node, as shown below.

$ npx cypress-parallel --node 1:5

--knapsack <file>

Specifies the location of the knapsack file. Defaults to knapsack.json.

--read-knapsack <file>

Optionally specified the location of the knapsack file to read.

--write-knapsack <file>

Optionally specified the location of the knapsack file to write.

--disable-knapsack-output

Disables outputting knapsack data to the file system. This is always disabled when you specify --reporter or --reporter-options to Cypress. If you require custom options and still want to obtain the knapsack output, you need to configure cypress-multi-reporters with @badeball/cypress-parallel/knapsack-reporter yourself.

--unweighed-strategy "estimate" | "distribute"

What strategy to utilize if encountering a test file that isn't contained in the knapsack. The "estimate" strategy will estimate expected execution time based off of file length (line numbers). The "distribute" strategy will merely distribute unknown files evenly amongst the nodes.

Custom stragies can be implemented using cusmiconfig, as shown below.

module.export = {
  /** @type {import("@badeball/cypress-parallel").UnweighedStrategy} */
  unweighedStrategy(weighedFiles, unweighedFiles, nodeCount) {
    // Implement me.
  },
};

How to handle knapsack.json

The knapsack contains every test file's weight and is instrumental for dividing tests evenly among nodes. This file needs to be available during the run and it should ideally be kept up-to-date. One way of handling this is to check it into version control somewhat regularly. Another way of handling it is to make your CI provider cache the data and re-surface it on subsequent runs. This last approach requires you to combine the knapsack data for every node before caching it though, which may or may not be so easy depending on your provider.

CI configuration

Below is an example of how to configure Gitlab CI to parallelize Cypress tests. Contributions of similar examples for other providers are welcome.

Gitlab CI

This example illustrate two things, 1) running tests in parallel and 2) combining knapsack data into a single, downloadable artifact. The latter is completely optional and you need to decide for yourself how you want to handle this.

test:
  stage: Test (1)
  parallel: 5
  artifacts:
    when: always
    paths:
      - knapsack-$CI_NODE_INDEX.json
    expire_in: 1 day
  script:
    - npx cypress-parallel --write-knapsack "knapsack-$CI_NODE_INDEX.json"

knapsack:
  stage: Test (2)
  script:
    - cat knapsack-*.json | jq -sS add | tee knapsack.json
  artifacts:
    when: always
    paths:
      - knapsack.json
    expire_in: 1 day

Other providers

If your provider does not provide a keyword such as Gitlab's parallel, then you can always simply just create N explicit jobs, similar to that shown below.

test_1:
  stage: Test
  script:
    - npx cypress-parallel --node 1:5

test_2:
  stage: Test
  script:
    - npx cypress-parallel --node 2:5

test_3:
  stage: Test
  script:
    - npx cypress-parallel --node 3:5

test_4:
  stage: Test
  script:
    - npx cypress-parallel --node 4:5

test_5:
  stage: Test
  script:
    - npx cypress-parallel --node 5:5