@open-xchange/codecept-runner

A runner script for E2E tests. Supports parallel execution, rerunning failed tests, and continuation.

Help

The switch --help (or -h) prints an overview with all supported switches. Example:

yarn codecept-runner --help

CodeceptJS Configuration File

The runner needs to know the location of the configuration file codecept.conf.js. If it is not located in the project root directory, the switch --config (or -c) allows to specify an alternative path.

Example:

yarn codecept-runner --config=./e2e/codecept.conf.js

Filter by Test Titles

The tests to be executed can be selected by full test titles (including scenario names and tags) using the switches --grep (or -g) and --ignore (or -i). Both switches expect a regular expression.

• If --grep is omitted, all existing tests will be selected.
• If --ignore is omitted, no selected test will be skipped.
• If --grep and --ignore are specified, the tests selected by both will be skipped.

Example:

yarn codecept-runner --grep=@stable --ignore=@experimental

will execute all tests with the tag "stable" that do not contain the tag "experimental".

Retry Failed Tests

Failed tests can be rerun automatically using the switch --retries (or -r). The number after the switch specifies the number of additional test runs after the initial test run.

Example:

$ yarn codecept-runner --retries=2
running 1000 tests ...
20 tests failed

retrying 20 tests ...
2 tests failed

retrying 2 tests ...
all tests passed

Parallel Execution

Tests can be executed in parallel with multiple workers using the switch --workers (or -w). The additional switch --retry-workers (or uppercase -W) allows to set a different number of workers for retrying failed tests (usually less than before).

$ yarn codecept-runner --workers=4 --retries=1 --retry-workers=2
running 1000 tests with 4 workers ...
20 tests failed

retrying 20 tests with 2 workers ...
all tests passed

Continuation

The results of a test run will be saved in a JSON file in the configured output directory. After a test run finishes with failed tests, or has been interrupted, the remaining tests (failed and pending) can be restarted using the switch --continue (or uppercase -C).

Example:

$ yarn codecept-runner --grep@stable
running 1000 tests ...
20 tests failed

$ yarn codecept-runner --continue
running 20 tests ...
all tests passed

Dry Run

Using the switch --dry (or -d) the runner will print log messages but will not actually run any test.

Example:

$ yarn codecept-runner --workers=4 --retries=1 --dry
running 1000 tests with 4 workers ...
1000 tests skipped