@open-xchange/codecept-runner

A runner script for E2E tests. Supports parallel execution, rerunning failed tests, and continuation.

CodeceptJS Configuration File

The runner needs to know the location of the configuration file codecept.conf.js. If it is not located in the project root directory, the switch --config (or -c) allows to specify an alternative path.

Example:

yarn codecept-runner --config=./e2e/codecept.conf.js

Filter by Test Titles

The tests to be executed can be selected by full test titles (including scenario names and tags) using the switches --grep (or -g) and --ignore (or -i). Both switches expect a regular expression.

• If --grep is omitted, all existing tests will be selected.
• If --ignore is omitted, no selected test will be skipped.
• If --grep and --ignore are specified, the tests selected by both will be skipped.

Example:

yarn codecept-runner --grep=@stable --ignore=@experimental

will execute all tests with the tag "stable" that do not contain the tag "experimental".

Retry Failed Tests

Failed tests can be rerun automatically using the switch --retries (or -r). The number after the switch specifies the number of additional test runs after the initial test run.

Example:

$ yarn codecept-runner --retries=2
running 1000 tests ...
20 tests failed

retrying 20 tests ...
2 tests failed

retrying 2 tests ...
all tests passed

Parallel Execution

Tests can be executed in parallel with multiple workers using the switch --workers (or -w). The additional switch --retry-workers (or uppercase -W) allows to set a different number of workers (usually less than before) for retrying failed tests.

$ yarn codecept-runner --workers=4 --retries=1 --retry-workers=2
running 1000 tests with 4 workers ...
20 tests failed

retrying 20 tests with 2 workers ...
all tests passed

Continuation

The results of a test run will be saved in a JSON file in the configured output directory. After a test run finishes with failed tests, or has been interrupted, the remaining tests (failed and pending) can be restarted using the switch --continue (or uppercase -C).

Example:

$ yarn codecept-runner --grep@stable
running 1000 tests ...
20 tests failed

$ yarn codecept-runner --continue
running 20 tests ...
all tests passed