e2ed

E2E testing framework over Playwright.

e2ed is designed to quickly parallel run a large number of independent atomic tests (or others tasks) in an arbitrary browser inside a docker container. Tests are written in TypeScript, using explicit dependencies and the concept of page objects. After the run, a detailed HTML report and a summary lite report in JSON format are generated.

Adding e2ed to a project

Prerequisites: node >=16, TypeScript >=4.8.

All commands below are run from the root directory of the project.

Install

Install the latest version of e2ed in devDependencies with the exact version:

npm install e2ed --save-dev --save-exact

Install Playwright browsers (only Chromium for now):

npx e2ed-install-browsers

Initialize

Initialize e2ed in the project; this will add an autotests directory with working sample tests and pageObject-s to the project:

npx e2ed-init

All the code related to e2ed will be in the autotests directory in the root of the project.

Add to TypeScript config

Add the autotests directory in field include of the project's tsconfig.json in the form "./autotests/**/*.ts", to make type checking work in the tests code:

  "include": [
    "./autotests/**/*.ts",
    "..."
  ],

Re-map imports

Also add the re-map of imports from the autotests directory in field paths of the project's tsconfig.json, to use bare imports from the autotests directory (import {...} from 'autotests/...';):

  "paths": {
    "autotests": ["./autotests/index.ts"],
    "autotests/*": ["./autotests/*"]
  },

It is assumed here that the baseUrl field is not specified in the project's tsconfig.json, or that the baseUrl is specified as "baseUrl": ".".

If, for example, baseUrl is equal to ./src, then it will be necessary to correct the path to autotests directory accordingly:

  "paths": {
    "autotests": ["../autotests/index.ts"],
    "autotests/*": ["../autotests/*"]
  },

After that you can run pack with tests in the project locally (sample tests are run on google.com):

E2ED_ORIGIN=https://google.com npx e2ed ./autotests/packs/allTests.ts

Now you can edit tests, pageObject-s and other files in the autotests directory as you need.

Usage

Pack

e2ed always runs packs of tests (or tasks), one pack at a time. A pack is a set of tests (tasks) with their run config, described in one file. e2ed takes one mandatory argument — the path to the pack (absolute or relative to the current directory, which should be the root directory of the project).

Packs are usually stored in the autotests/packs directory.

Run local

To run pack with tests locally for https://google.com:

E2ED_ORIGIN=https://google.com npx e2ed ./autotests/packs/allTests.ts

For convenience, you can add a command to run concrete pack in the scripts field of the package.json:

  "scripts": {
    "e2ed:allTests": "e2ed ./autotests/packs/allTests.ts",
    "..."
  },

After that, you can run the pack like this:

E2ED_ORIGIN=https://google.com npm run e2ed:allTests

Also, when running locally, you can pass additional command-line arguments, such as the path to a specific test file from a pack, to run just that test:

E2ED_ORIGIN=https://google.com npm run e2ed:allTests ./autotests/tests/main/exists.ts

Run in docker

You can download the latest e2ed docker image from https://hub.docker.com/r/e2edhub/e2ed:

docker pull e2edhub/e2ed

And run tests for https://google.com in docker container:

E2ED_ORIGIN=https://google.com ./autotests/bin/runDocker.sh ./autotests/packs/allTests.ts

For convenience, you can add a command to run concrete pack in docker in the scripts field of the package.json:

  "scripts": {
    "e2ed:docker:allTests": "./autotests/bin/runDocker.sh ./autotests/packs/allTests.ts",
    "..."
  },

After that, you can run the pack in docker like this:

E2ED_ORIGIN=https://google.com npm run e2ed:docker:allTests

Personal local pack for development

You can add your local pack with custom settings for developing tests that will not be stored in the repository, under the name local.ts in directory autotests/packs (it's already listed in .gitignore).

This pack might look like this:

import {pack as allTestsPack} from './allTests';

import type {Pack} from 'autotests/configurator';

/**
 * Pack from .gitignore for local development.
 */
export const pack: Pack = {
  ...allTestsPack,
  testIdleTimeout: 10_000,
};

The npx e2ed-init command is already creating such a file as an example.

How to debug tests

For debugging tests, the local run of the pack is intended.

For example, you can use the debug action from e2ed/actions in the test code:

await debug();

When calling the debug action, e2ed will stop the test execution and enter the debug-mode.

After debugging is complete, remember to remove the call of the debug action.

In addition, when run pack locally tests can be debugged using the usual nodejs debugging flags (--inspect-brk, --inspect), as nodejs application (for brevity, we omit the setting of environment variables before commands):

npm run e2ed:allTests ./autotests/tests/main/exists.ts -- --inspect-brk

You can use the debugger instruction to stop execution at the desired line.

Or you can set any non-empty value to the E2ED_DEBUG environment variable, which will also run e2ed in nodejs debug mode (this is equivalentto passing the --inspect-brk flag):

E2ED_DEBUG=true npm run e2ed:allTests ./autotests/tests/main/exists.ts

E2ED_DEBUG also works for run in docker, and allows you to connect a debugger to the e2ed running in docker container.

npm run e2ed:allTests ./autotests/tests/main/exists.ts -- --debug-on-fail

Basic fields of pack config

The pack is a single ts file that exports the pack's config under the name pack.

Here are the basic fields of the pack config.

addLogsWithTags: readonly LogTag[]: array of additional log tags. Logs with a specific tag (in logTag field) will be added only if their tag is specified in this array.

browserFlags: string[]: array of browser flags, like --disable-dev-shm-usage, with which the browser is launched to run tests.

browserName: BrowserName: browser name (one of chromium, firefox, webkit).

concurrency: number: the number of browser windows in which tests will run in parallel.

customPackProperties: CustomPackProperties: a custom set of fields defined within the project. These fields allow, for example, to customize the behavior of hooks for different packs.

deviceScaleFactor: number: device scale factor (aka window.devicePixelRatio).

doAfterPack: ((liteReport: LiteReport) => CustomReportProperties | undefined)[]: an array of functions that will be executed, in order, after the pack completes. The functions accept a lite report object, and can return custom report properties, which in this case will be included in the lite report. Each function can thus access the results of the previous function.

doBeforePack: ((startInfo: StartInfo) => FullPackConfig | undefined)[]: an array of functions that will be executed, in order, before the pack starts. The functions accept a start info object, and can return new full pack config, which in this case will be included in the start info object, and will be used for running pack. Each function can thus access the results of the previous function.

enableCsp: boolean: enables Content-Security-Policy checks in browser.

enableHeadlessMode: boolean: enables headless mode (if browser supports such mode).

enableMobileDeviceMode: boolean: enables Chromium mobile device mode.

enableTouchEventEmulation: boolean: enables touch event emulation. If true, page fires touch events when test interact with the page (instead of click events).

filterTestsIntoPack: (testStaticOptions: TestStaticOptions<TestMeta>) => boolean: this function filters tests (tasks) by their static options — only those tests for which the function returned true get into the pack.

fullMocks: FullMocks | null: functions that specify the "full mocks" functionality.

getTestNamePrefixInUiMode: (testOptions: TestOptions<TestMeta>) => string: get prefix for test name in UI mode by test options.

liteReportFileName: string | null: the name of the file under which, after running the tests, the lite JSON report will be saved in the autotests/reports directory, for example, lite-report.json. If null, the lite report will not be saved.

logFileName: string | null: the name of the file under which, after running the tests, the pack logs will be saved in the autotests/reports directory, for example, pack-logs.log. If null, the log will not be saved.

mapBackendResponseErrorToLog: (response: Response) => Payload | undefined: maps responses with errors from the backend to "red" logs (as errors) during the test. It is assumed that the function will select responses with statuse codes of 400 and higher (client and server errors). Backend responses with errors are accumulated in separate "red" log step (with logEventStatus: 'failed'). Log the responseBody field carefully, as the body of backend response can be very large. If the function returns undefined, the response is not logged (skipped).

mapBackendResponseToLog: (response: Response) => Payload | undefined: maps responses from the backend to logs during the test. Backend responses received during a certain test step are accumulated in an array in the backendResponses field of the log of this step. Log the responseBody field carefully, as the body of backend response can be very large. If the function returns undefined, the response is not logged (skipped).

mapLogPayloadInConsole: (message: string, payload: LogPayload | undefined, logEventType?: LogEventType) => LogPayload | 'skipLog' | undefined: maps log payload for logging in console to clarify, shorten or skip a console log entry. If the mapping returns 'skipLog', the log entry is skipped. If the mapping returns undefined, the log entry is not skipped, but is printed with an empty payload.

mapLogPayloadInLogFile: (message: string, payload: LogPayload | undefined, logEventType?: LogEventType) => LogPayload | 'skipLog' | undefined: maps log payload for logging in file to clarify, shorten or skip a file log entry. If the mapping returns 'skipLog', the log entry is skipped. If the mapping returns undefined, the log entry is not skipped, but is printed with an empty payload.

mapLogPayloadInReport: (message: string, payload: LogPayload | undefined, logEventType?: LogEventType) => LogPayload | null | undefined: maps log payload for logging step in HTML report and lite report to clarify, shorten or skip a report step. If the mapping returns null, the step is skipped. If the mapping returns undefined, the log entry is not skipped, but is printed with an empty payload.

maxRetriesCountInDocker: number: the maximum number of retries to run a test with the command your-project/autotests/bin/runDocker.sh (until the test passes). For example, if it is equal to three, the test will be run no more than three times.

navigationTimeout: number: default timeout for navigation to url (navigateToPage, navigateToUrl actions) in milliseconds.

overriddenConfigFields: PlaywrightTestConfig | null: if not null, then this value will override fields of internal Playwright config.

packTimeout: number: timeout (in millisecond) for the entire pack of tests (tasks). If the test pack takes longer than this timeout, the pack will fail with the appropriate error.

pathToScreenshotsDirectoryForReport: string | null: path to the directory where screenshots will be stored for displaying them in the HTML report. This path must be either relative (from the HTML report file) or absolute (i.e. with http/https protocol). The screenshot directory should be served by the web server with appropriate headers, like a normal static directory. The autotests/reports/screenshots directory from the project should be copied to this directory after the pack is completed, and then screenshots from this directory will be displayed in the HTML report. If null, screenshots will not be displayed in the HTML report.

reportFileName: string | null: the name of the file under which, after running the tests, the HTML report will be saved in the autotests/reports directory, for example, report.html. Also this name is used as the title of the report page. If null, the report will not be saved.

skipTests: SkipTests: this setting allows you to describe a set of skipped tests in a custom form. You can define the SkipTests type and skipTests processing rules in the hook autotests/hooks/isTestSkipped.ts.

takeFullPageScreenshotOnError: boolean: if true, then takes a screenshot of the full page (not just the viewport) at the time of the test error, for display in the HTML report.

takeViewportScreenshotOnError: boolean: if true, then takes a screenshot of the page viewport at the time of the test error, for display in the HTML report.

testFileGlobs: readonly string[]: an array of globs with pack test (task) files. https://www.npmjs.com/package/globby is used for matching globs.

testIdleTimeout: number: timeout (in milliseconds) for each individual test step. If the test step (interval between two log function calls) takes longer than this timeout, the test fails and rerun on the next retry. This parameter can be overridden in the test-specific options.

testTimeout: number: timeout (in milliseconds) for each individual test run. If the test run takes longer than this timeout, the test fails and rerun on the next retry. This parameter can be overridden in the test-specific options.

userAgent: string: userAgent string of browser (device) in tests.

viewportHeight: number: height of viewport of page in pixels.

viewportWidth: number: width of viewport of page in pixels.

waitBeforeRetry: (options: Options) => number: returns how many milliseconds e2ed should wait before running test (for retries).

waitForAllRequestsComplete.maxIntervalBetweenRequestsInMs: number: default maximum interval (in milliseconds) between requests for waitForAllRequestsComplete function. If there are no new requests for more than this interval, then the promise returned by the waitForAllRequestsComplete function will be successfully resolved.

waitForAllRequestsComplete.timeout: number: default timeout (in milliseconds) for waitForAllRequestsComplete function. If the wait is longer than this timeout, then the promise returned by the waitForAllRequestsComplete function will be rejected.

waitForInterfaceStabilization.stabilizationInterval: number: default stabilization interval for waitForInterfaceStabilization function.

waitForInterfaceStabilization.timeout: number: default timeout (in milliseconds) for waitForInterfaceStabilization function. If the wait is longer than this timeout, then the promise returned by the waitForInterfaceStabilization function will be rejected.

waitForRequestTimeout: number: default timeout (in milliseconds) for waitForRequest/waitForRequestToRoute functions. If the wait is longer than this timeout, then the promise returned by the waitForRequest/waitForRequestToRoute function will be rejected.

waitForResponseTimeout: number: default timeout (in milliseconds) for waitForResponse/waitForResponseToRoute functions. If the wait is longer than this timeout, then the promise returned by the waitForResponse/waitForResponseToRoute function will be rejected.

Environment variables

Required environment variables are defined in the ./autotests/variables.env file (they cannot be deleted):

E2ED_DOCKER_IMAGE: the name of the docker image where the tests will run. The image must be based on the e2ed base image.

E2ED_PATH_TO_TS_CONFIG_OF_PROJECT_FROM_ROOT: the path to TypeScript config file of the project from the root directory of the project. The project should have one common TypeScript config for both the application code and the autotest code.

You can pass the following optional environment variables to the e2ed process in any standard way:

E2ED_ORIGIN: origin-part of the url (protocol + host) on which the tests will be run. For example, https://google.com.

E2ED_DEBUG: run e2ed in nodejs debug mode (--inspect-brk=0.0.0.0) if this variable is not empty.

E2ED_DOCKER_DEBUG_PORT: debug port when run in docker (9229 by default).

E2ED_TERMINATION_SIGNAL: the termination signal received by the e2ed process (if any). Typically this value is 'SIGUSR1'.

E2ED_TIMEOUT_FOR_GRACEFUL_SHUTDOWN_IN_SECONDS: default timeout for "graceful shutdown" of e2ed process (in seconds). If the variable is not set, the default value of 16 is used.

License

MIT