Plugin to use Cucumber within Nx workspaces
- Introduction
- Prerequisites
- Cucumber JavaScript API
- Quick start
- Install Nx
- Install dependencies
- Generate Cucumber projects
- Execute Cucumber projects
- Code coverage
- Example projects
- End-to-end tests of this project
- Continuous delivery pipeline
- Debugging and watching scenarios
- Filter Cucumber scenarios
- Special tags
- Profiles
- Logging
- Contribution
- Troubleshooting
This is nx-cucumber, a plugin for Nx which integrates Cucumber to create the best possible connection between these two essential tools for quality software. The main purpose is to provide generators and executors for multiple Cucumber implementations within Nx workspaces. We have just started with Cucumber for JavaScript plus optional Playwright support and there is already more on the roadmap.
The only prerequisite for this software is Node.js. In our continuous delivery pipeline, we test against the latest Node.js LTS version. Other versions are expected to work, but there is no guarantee. Since this is an Nx plugin for Cucumber, these wonderful tools are required too. To provide the best experience for browser automation, the Playwright library is supported out-of-the-box, but is completely optional and must be enabled explicitly. We always support and automatically test the latest Nx, Cucumber and Playwright versions. Additionally, the next Nx and Playwright versions and the previous Nx and Cucumber versions are tested and supported too.
This plugin uses the Cucumber JavaScript API. That means, Cucumber is used in-process and not on the command-line. By using this great piece of software, we achieve optimal performance, stability and testing for executing Cucumber scenarios with Nx.
To immediately use this plugin with a new example React application use the following commands:
# option 1: create a new app inside an existing nx workspace
npm i -D @nx/react
npx nx g @nx/react:application gnu-app --bundler=vite --style=css --e2eTestRunner=none --routing
# option 2: create a new nx workspace with a 'gnu-app' and change to its directory
npx create-nx-workspace gnu-repo --preset=react-monorepo --e2eTestRunner=none --appName=gnu-app --bundler=vite --style=css --nxCloud=skip
cd gnu-repo
# install cucumber and playwright-test
npm i -D @cucumber/cucumber @playwright/test
# install this plugin
npm i -D @gnuechtel/nx-cucumber
# create a cucumber project for gnu-app
npx nx g @gnuechtel/nx-cucumber:project --project=gnu-app --test-runner=playwright
# run cucumber scenarios (implicitly starts gnu-app)
npx nx e2e gnu-app-e2e
# display generated cucumber HTML report
open ./dist/cucumber-js/apps/gnu-app-e2e/reports/html/cucumber-test-report.html
There are two feasible options to install Nx:
- either via
npx create-nx-workspace@latest
to setup a fresh Nx repo or - via
npx nx@latest init
to convert an existing Node.js project into an Nx workspace.
It is never recommended to install all those Nx dependencies manually.
It would just cause too much unnecessary work.
To use this plugin and Cucumber, at least the following two dependencies must be installed in the consuming Nx workspace:
- @gnuechtel/nx-cucumber (of course)
- @cucumber/cucumber (Cucumber for JavaScript)
If we want to use the Playwright test runner, we should also install one of the following dependencies:
- playwright (Playwright library) or
- @playwright/test (Playwright Test framework)
Since Cucumber is our executor, we are using the Playwright library, not the complete Playwright Test framework. However, Playwright Test also provides very useful expect extensions which are usable standalone with the library. So we recommend using @playwright/test
, which also includes the playwright
dependency.
After the installation of the dependencies, we can create a project with Cucumber, for any kind of node application:
# Prerequisite: create application 'my-project' (see quick start for a complete example)
# default core test runner (generates an e2e app 'my-project-e2e')
npx nx g @gnuechtel/nx-cucumber:project --project=my-project
# playwright test runner (also generates an e2e app 'my-project-e2e')
npx nx g @gnuechtel/nx-cucumber:project --project=my-project --test-runner=playwright
# playwright test runner with custom e2e project name (generates an app 'my-project-foo-e2e')
npx nx g @gnuechtel/nx-cucumber:project --project=my-project --name my-project-foo-e2e --test-runner=playwright
# playwright test runner with custom base URL
npx nx g @gnuechtel/nx-cucumber:project --project=my-project --test-runner=playwright --base-url=http://localhost:8765
All possible project generator options may be passed as command-line parameters and are stored in the related schema file. They can also be displayed on the command-line with the following command: npx nx g @gnuechtel/nx-cucumber:project --help
.
After the Cucumber project was generated, we are able to execute it with nx:
npx nx e2e my-project-e2e
The project generator created a project.json
with a default configuration which works out-of-the-box for hello-world projects as described in the quick start.
To change the default configuration, we manually edit the project.json
of the Cucumber project. All available options are stored in the executor schema file. Additionally, a cucumber.js
file can be stored beside the project.json
to tweak Cucumber configuration by custom code (see next section).
Some Cucumber properties are accessible via project.json
.
But not all properties are supported by nx-cucumber and sometimes, we even want to customize the supported ones.
For example, we want to set tags based on environment variables.
To achieve such a custom Cucumber configuration:
-
We remove the related property from
project.json
:"tags": "not @ignore". -
We create a
cucumber.js
file next toproject.json
. For example:const tagsFilter = process.env['CI_STAGE'] !== 'qa' // Some custom logic based on an example stage variable ? 'and not @qa-stage-only' // Exclude all 'qa-stage-only' scenarios : 'and (@qa-stage or @qa-stage-only)'; // Only include 'qa-stage' or 'qa-stage-only' scenarios module.exports = { default: { tags: `not @ignore ${tagsFilter}` }, // Combine default filter with stage filter };
In this example, we used a Cucumber configuration file in CommonJS format, but other formats are supported too. Also we would be able to set all advanced Cucumber options in the cucumber.js
file which are not yet supported in project.json
.
Custom Playwright options which are passed to a new browser context may be customized with nx-cucumber. By default, the project generator already creates custom Playwright options in project.json
:
"additionalPlaywrightConfiguration": { "acceptDownloads": true }
By adding more properties to additionalPlaywrightConfiguration
we are able to customize the Playwright behavior.
By default, the project generator creates a dev server target and a base URL in project.json
:
{
"devServerTarget": "my-project:serve",
"baseUrl": "http://localhost:4200"
}
This will start a dev server with the defined nx target and waits until the baseUrl
is ready with HTTP status code 200.
After the execution of the Cucumber scenarios, the dev server process will be terminated.
For some projects it is also util to start more than one dev server, so nx-cucumber supports this:
"devServerTargets": ["some-dependent-service:serve", "my-project:serve"]
Here, two dev server targets will be started before Cucumber execution and terminated afterwards in reverse order.
To wait for additional URLs there is also another useful configuration:
"additionalDevServerUrls": ["http://localhost:9999/status"]
Here we additionally wait for the Cucumber execution until the URL http://localhost:9999/status
is ready (also with HTTP status code 200).
Sometimes it is not desired to use the dev server facilities of this plugin. For example, the server was started by another script and we just want to execute the Cucumber scenarios. For those use cases, we can skip the dev server in project.json
:
"skipServe": true
By default, the project generator disables video files and attachments. To enable video files, we remove the disableVideoFiles
option from project.json
or set it to false
. Removing that option, would store video files with scenario names in the Cucumber output directory (dist/cucumber-js/application-path/videos
by default). If we also want to include the video files in Cucumber reports as attachments, there is a disableVideoAttachments
option in project.json
. Again, we can remove it or set to false
.
Also by default, screenshots are attached to every scenario which runs with Playwright. We will see them in the Cucumber reports for successful and erroneous scenarios. On the other hand, screenshot files will be created only in case of error. They will also be stored in the Cucumber output directory (dist/cucumber-js/application-path/screenshots
by default). To change the default behavior and create screenshot files for successful scenario too, we set enableSuccessfulScreenshotFiles
to true
. But we can also change the screenshot options in the other direction by setting disableScreenshotFiles
and disableScreenshotAttachments
to true
to disable screenshots completely.
For the Playwright and the core test runner, an additional custom world may be created to extend the default behavior. To achieve this, we add the --custom-world
parameter to the project generator. Currently, the custom world for the core test runner is always generated, even if the parameter is omitted. We are aware of this and will change it in the future. An example path for a generated custom world would be apps/my-project-e2e/src/shared/my-project-world.ts
. The name is the same for both test runners, but there are differences in the implementation.
A custom Playwright world must be derived from the PlaywrightWorld
class of nx-cucumber since a lot of default behavior for the Playwright test runner is implemented there. In theory, it is possible to write an own Playwright world from scratch, but it is absolutely not recommended. To enable a custom Playwright world, it is also required, to set customWorld
to true
in project.json
and to register that world via setWorldConstructor
. The project generator will create this required code for us.
An example Playwright world, which extends the built-in clean-up method looks like this:
import { IWorldOptions } from '@cucumber/cucumber';
import { PlaywrightWorld } from '@gnuechtel/nx-cucumber';
export class CustomPlaywrightWorld extends PlaywrightWorld {
constructor(options: IWorldOptions) {
super(options);
}
override async cleanUp(): Promise<void> {
console.log('Some additional clean-up code in the custom playwright world');
await super.cleanUp();
}
}
A custom core world is written from scratch and should be derived from the default World
provided by Cucumber. An example core world, which adds a get-item and set-item method looks like this:
import { IWorldOptions, World } from '@cucumber/cucumber';
export class CustomWorld extends World {
constructor(options: IWorldOptions) {
super(options);
}
public setItem<T>(key: string, value: T): void {
this._items[key] = value;
}
public getItem<T>(key: string, defaultValue?: T): T {
return (this._items[key] as T) ?? (defaultValue as T);
}
private _items: Record<string, unknown> = {};
}
Code coverage is an important part for all kinds of testing, so nx-cucumber supports it for the Playwright and core test runner. All example projects are configured for code coverage. Playwright code coverage will be enabled by proper application configuration as done in the example react project. Core code coverage will be enabled with the --instrument-test-runner
project generator parameter as done in the example Node.js project.
This repository contains example projects that can be used directly to see nx-cucumber in action.
In the following table, we will see the following information:
- the examples that are split into two projects in each case: one main application project and one cucumber project,
- the Nx commands to create these projects, where the cucumber projects are generated with nx-cucumber,
- the configured code coverage for each project,
- the execution of the cucumber specifications with nx-cucumber (via
nx e2e
), - the Cucumber and coverage reports used.
example react application and cucumber project | example Node.js app and cucumber project | |
---|---|---|
Execute cucumber project | npx nx e2e example-react-app-e2e |
npx nx e2e example-node-app-e2e |
Source cucumber project | apps/example/react-app-e2e | apps/example/node-app-e2e |
Cucumber project file | react-app-e2e/project.json | node-app-e2e/project.json |
Source main project | apps/example/react-app | apps/example/node-app |
Create main project | npx nx g @nx/react:application example/react-app --bundler=vite --style=css --e2e-test-runner=none --routing |
npx nx g @gnuechtel/nx-cucumber:project --project=example-node-app --directory=example/node-app-e2e --instrument-test-runner |
Create cucumber project | npx nx g @gnuechtel/nx-cucumber:project --project=example-react-app --directory=example/react-app-e2e --test-runner=playwright |
npx nx g @gnuechtel/nx-cucumber:project --project=example-node-app --directory=example/node-app-e2e --instrument-test-runner |
Code coverage | configured with the vite istanbul plugin |
--instrument-test-runner parameter and optimizations
|
Cucumber report | cucumber-test-report.html | cucumber-test-report.html |
Coverage report | lcov-report/index.html | lcov-report/index.html |
If we have executed our nx-cucumber end-to-end tests, we got a lot of more example projects, their Cucumber output and coverage reports for free. For example:
We support Cucumber with this plugin, and it would be a shame if we used anything other than Cucumber for our own end to end tests. So, we just do it!
And we are not only using Cucumber alone, we are also using nx-cucumber for our end-to-end tests! This is possible, since nx-cucumber is also usable inside this repository. So we can use nx-cucumber for our tests and example projects.
A great side effect of using nx-cucumber internally is also, that we will be our own and first consumer. So we will always got the first integration test for free after code changes.
To run our own end-to-end tests we simply execute them by:
npm run e2e:nx-cucumber
Our continuous delivery pipeline runs builds and tests for Linux, macOS and Windows on the latest Node.js LTS version.
The end-to-end tests in the continuous delivery pipeline will run on any combination of
- the previous, latest and next Nx version
- the previous and latest Cucumber version
- the latest and next Playwright version
So we have 12 different kinds of end-to-end tests. For performance reasons, all combinations will only tested on Linux. Windows and macOS end-to-end tests will just run on the latest versions of Nx, Playwright and Cucumber.
During the development of Cucumber scenarios, it can be useful to use debugging and watching.
Debugging means, that we can go to the Cucumber scenarios step by step.
Watching means, that scenarios will be executed automatically after file changes.
Debugging is available for the Playwright test runner only.
It enables the Playwright inspector with paused execution in a headed browser.
We achieve debugging by the following command:
npx nx e2e my-project-e2e --debug
In the following screenshot, we can see the debug mode in action:
To watch Cucumber tests, we can use nx workspace watching. By convenience, the project generator creates an e2e-watch
target (see example react app) which we can use with the following command:
npx nx e2e-watch example-react-app-e2e
We can combine watching with other parameters:
npx nx e2e-watch example-react-app-e2e --debug
npx nx e2e-watch example-react-app-e2e --debug --tags=@focus
While development of Cucumber scenarios, we often want to execute only selected scenarios. We can achieve this by using the filter capabilities of Cucumber.
-
Add some tag to some scenario or feature, like
@focus
:@focus Scenario: a user wants to get project information Given any user has found our project home page When the user requests project information Then some advices will be shown
-
Run scenarios with tag parameter:
# Run one-time ... npx nx e2e my-project-e2e --tag=@focus # ... or in combination with watch to continuously run scenarios while development npx nx e2e-watch my-project-e2e --tag=@focus
We can filter scenarios by name with regular expressions. Simple expressions are good to include scenarios (like in the first example). More sophisticated expressions are used when we want to exclude some scenarios (like in the second example).
# execute all scenarios where the name contains 'payment'
npx nx e2e my-project-e2e --name "payment"
# execute all scenarios where the name does not contain 'mobile' nor something like 'top secret'
npx nx e2e my-project-e2e --name '^((?!mobile|top.?secret).)*$'
As seen in the last chapter, Cucumber tags can be used for scenario filtering. Another purpose is to control some technical behavior. Nx-cucumber provides some technical built-in tags which are described here.
In project.json
we can set disableJavaScript
to true
. With that setting, we would disable javascript for all Cucumber scenarios with the Playwright test runner. However, we can enable javascript again for selected scenarios in such a case, by using the @javascript
tag for single scenarios or complete features:
@javascript
Scenario: hidden easter egg
Given a user has found an easter egg on the website
When the user spins the wheel of fortune
Then a random color will be displayed
On the other hand, the default configuration is enabled javascript. But maybe we want to ensure that some scenario is working without javascript. Then we could use the @noscript
tag:
@noscript
Scenario: homepage is displayed without javascript
When any user finds our project home page
Then a welcome message will be shown
By default, all Cucumber scenarios with Playwright run in a headless browser. Some actions like copy-and-paste do not work headless. In such cases, we can add the @headed
tag to scenarios to use a headed browser. When a headless browser is currently running, it will be closed and a headed one will be started and vice versa. Existing headless/headed browser instances will be reused, when the previous scenario also used the same browser mode.
A headed example scenario could be written like this:
@headed
Scenario: Copy search data
Given the user has searched for content
When the users copies the search
Then the search data is stored in the clipboard
The following example implementation of the Cucumber steps reads data from the clipboard.
It only works with the @headed
tag and could be written like this:
import { Given, Then, When } from '@cucumber/cucumber';
import { expect } from '@playwright/test';
import { PlaywrightWorld } from '@gnuechtel/nx-cucumber';
Given(
'the user has searched for content',
async function (this: PlaywrightWorld) {
await this.page.type('#search', mySearch);
await this.page.click('#searchButton');
}
);
When('users copies the search', async function (this: PlaywrightWorld) {
await this.page.click('#copySearchButton');
});
Then(
'the search data is stored in the clipboard',
async function (this: PlaywrightWorld) {
const clipboardText = await readTextFromClipboard(this.page);
expect(clipboardText).toBe(mySearch);
}
);
const mySearch = 'my search';
async function readTextFromClipboard(page: Page) {
await page.context().grantPermissions(['clipboard-read']);
return (await page.evaluate(() =>
window.navigator.clipboard.readText().then((text) => {
return new Promise((resolve) => {
resolve(text);
});
})
)) as string;
}
Nx-cucumber supports Cucumber profiles with custom Cucumber configuration.
An example cucumber.js
file which provides some profiles for special names could look like this:
module.exports = {
homepage: { name: ['a user visits the home page'] },
'project-info': { name: ['a user wants to get project information'] },
};
To define profiles, which will be executed by default, we can define them in project.json
like this:
"profiles": ["homepage"]
We can also run profiles from the command-line, which would override any defined default profiles:
# Use our custom homepage and project-info profiles
npx nx e2e my-project-e2e --profile homepage --profile project-info
# Use built-in Cucumber default profile
npx nx e2e my-project-e2e --profile default
Logging messages are produced by the debug library. The project generator creates a .env file which contains the default logging configuration. By changing the .env
file in the generated Cucumber project, the verbosity of the log output may be decreased (e.g. remove pw:api
or just use e2e:cucumber
) or increased (e.g. use pw:*
). A typical output for a single scenario with Playwright looks like this:
2023-10-31T23:26:23.135Z e2e:cucumber STARTED - scenario - a user visits the home page
2023-10-31T23:26:23.136Z e2e:hooks Before - Cleaning up
2023-10-31T23:26:23.136Z e2e:world Cleaning up world instance
2023-10-31T23:26:23.136Z e2e:world World instance cleaned up
2023-10-31T23:26:23.136Z e2e:hooks Before - Cleaned up
2023-10-31T23:26:23.137Z e2e:cucumber PASSED - hook - prepare
2023-10-31T23:26:23.137Z e2e:hooks Before not @headed - Creating new headless Chromium instance
2023-10-31T23:26:23.139Z pw:api => browserType.launch started
2023-10-31T23:26:23.295Z pw:api <= browserType.launch succeeded
2023-10-31T23:26:23.295Z e2e:hooks Before not @headed - Chromium instance created
2023-10-31T23:26:23.296Z e2e:cucumber PASSED - hook - ensure headless browser [not @headed]
2023-10-31T23:26:23.296Z e2e:hooks BeforeStep
2023-10-31T23:26:23.298Z pw:api => started
2023-10-31T23:26:23.304Z pw:api <= succeeded
2023-10-31T23:26:23.307Z pw:api => browserContext.exposeFunction started
2023-10-31T23:26:23.310Z pw:api <= browserContext.exposeFunction succeeded
2023-10-31T23:26:23.310Z pw:api => started
2023-10-31T23:26:23.311Z pw:api <= succeeded
2023-10-31T23:26:23.311Z pw:api => browserContext.newPage started
2023-10-31T23:26:23.338Z pw:api <= browserContext.newPage succeeded
2023-10-31T23:26:23.340Z pw:api => page.goto started
2023-10-31T23:26:23.341Z pw:api navigating to "http://localhost:4200/", waiting until "load"
2023-10-31T23:26:23.356Z pw:api "commit" event fired
2023-10-31T23:26:23.356Z pw:api navigated to "http://localhost:4200/"
2023-10-31T23:26:23.426Z pw:api "domcontentloaded" event fired
2023-10-31T23:26:23.426Z pw:api "load" event fired
2023-10-31T23:26:23.427Z pw:api <= page.goto succeeded
2023-10-31T23:26:23.427Z e2e:hooks AfterStep - Creating screenshots
2023-10-31T23:26:23.428Z e2e:world Storing screenshots
2023-10-31T23:26:23.430Z pw:api => page.screenshot started
2023-10-31T23:26:23.431Z pw:api taking page screenshot
2023-10-31T23:26:23.655Z pw:api <= page.screenshot succeeded
2023-10-31T23:26:23.655Z e2e:world Attaching image to current step
2023-10-31T23:26:23.656Z e2e:world Image attached
2023-10-31T23:26:23.656Z e2e:world Successful screenshot files are disabled
2023-10-31T23:26:23.656Z e2e:world Screenshots stored
2023-10-31T23:26:23.656Z e2e:hooks AfterStep - Screenshots created
2023-10-31T23:26:23.657Z e2e:cucumber PASSED - step - any user finds our project home page
2023-10-31T23:26:23.657Z e2e:hooks BeforeStep
2023-10-31T23:26:23.662Z pw:api => locator._expect started
2023-10-31T23:26:23.663Z pw:api locator._expect with timeout 5000ms
2023-10-31T23:26:23.665Z pw:api waiting for locator('h1').getByText('Hello there, Welcome gnu-app 👋')
2023-10-31T23:26:23.683Z pw:api locator resolved to <h1>…</h1>
2023-10-31T23:26:23.683Z pw:api <= locator._expect succeeded
2023-10-31T23:26:23.684Z e2e:hooks AfterStep - Creating screenshots
2023-10-31T23:26:23.684Z e2e:world Storing screenshots
2023-10-31T23:26:23.685Z pw:api => page.screenshot started
2023-10-31T23:26:23.686Z pw:api taking page screenshot
2023-10-31T23:26:23.837Z pw:api <= page.screenshot succeeded
2023-10-31T23:26:23.837Z e2e:world Attaching image to current step
2023-10-31T23:26:23.838Z e2e:world Image attached
2023-10-31T23:26:23.838Z e2e:world Successful screenshot files are disabled
2023-10-31T23:26:23.838Z e2e:world Screenshots stored
2023-10-31T23:26:23.838Z e2e:hooks AfterStep - Screenshots created
2023-10-31T23:26:23.839Z e2e:cucumber PASSED - step - a welcome message will be shown
2023-10-31T23:26:23.839Z e2e:hooks After - Collecting coverage
2023-10-31T23:26:23.841Z pw:api => page.evaluate started
2023-10-31T23:26:23.847Z pw:api => started
2023-10-31T23:26:23.848Z pw:api <= succeeded
2023-10-31T23:26:23.849Z pw:api <= page.evaluate succeeded
2023-10-31T23:26:23.849Z e2e:hooks After - Coverage collected
2023-10-31T23:26:23.849Z e2e:hooks After - Cleaning up
2023-10-31T23:26:23.849Z e2e:world Cleaning up world instance
2023-10-31T23:26:23.849Z e2e:world Closing browser context
2023-10-31T23:26:23.850Z pw:api => started
2023-10-31T23:26:23.859Z pw:api <= succeeded
2023-10-31T23:26:23.859Z e2e:world Browser context closed
2023-10-31T23:26:23.859Z e2e:world World instance cleaned up
2023-10-31T23:26:23.859Z e2e:hooks After - Cleaned up
2023-10-31T23:26:23.859Z e2e:cucumber PASSED - hook - clean up
2023-10-31T23:26:23.860Z e2e:cucumber FINISHED - scenario - a user visits the home page
Currently, this project is written and maintained by a single person. The wording of this documentation already shows, that this should not be a one-man-show! Contributing is very welcome! Please open an issue if you want to join.
Nx-cucumber does its best to properly shutdown dev server(s) after Cucumber execution. For some project setups it is difficult or even impossible to kill all processes. For such cases an easy workaround exists:
Before executing the Cucumber project, we just start the dev server in another terminal. For example: npx nx serve my-application
. After that, we execute the Cucumber scenarios as usual: npx nx e2e my-application-e2e
. The nx-cucumber will detect the running application via the base URL. After the execution the server is still running, since nx-cucumber only shutdowns the self-started dev servers.
When trying out the example projects inside this repository, the Playwright inspector highlights incorrect line numbers, when the execution is paused. This is a known issue, but there is not yet any solution.
For example projects which are created by the end-to-end tests of this project or real world projects which use nx-cucumber with npm install
this problem does not exist and line numbers are highlighted correctly as we can see in the screenshot about debugging Playwright.
A common mistake, is to forget the @
before any tag. This would be wrong: npx nx e2e example-react-app-e2e --tag=mytag.
The correct usage is: npx nx e2e example-react-app-e2e --tag=@mytag
.
Nx-cucumber aims to support all important Cucumber product features. Since we are using the Cucumber JavaScript API, the cucumber-js
command-line tool is not available, but fortunately most of the advanced features may be used with custom Cucumber configuration.
Since this is an open source project, you can change this software for your own needs. However, it would be the best choice to contribute to this project by opening an issue.
Moreover, for the next major version, we will implement more generator and executors and consider to fully support cucumber-js
, so there won't be any limitation in the future.