testcafe-browser-provider-browserstack

This plugin integrates TestCafe with the BrowserStack Testing Cloud.

Install

npm i -g testcafe-browser-provider-browserstack

Usage

Before using this plugin, save the BrowserStack username and access key to environment variables BROWSERSTACK_USERNAME and BROWSERSTACK_ACCESS_KEY.

Project name and build name will be displayed in BrowserStack if you set the BROWSERSTACK_PROJECT_NAME and BROWSERSTACK_BUILD_ID environment variables, or the project and build properties in the configuration file.

If you have troubles starting multiple browsers at once, or get browserstack-local related errors like #27, try setting the BROWSERSTACK_PARALLEL_RUNS environment variable to the number of browsers you want to run simultaneously, or to 1 if you want to run just one browser.

You can determine the available browser aliases by running

testcafe -b browserstack

If you run tests from the command line, use the alias when specifying browsers:

testcafe "browserstack:Chrome@53.0:Windows 10" "path/to/test/file.js"

When you use API, pass the alias to the browsers() method:

testCafe
    .createRunner()
    .src('path/to/test/file.js')
    .browsers('browserstack:Chrome@53.0:Windows 10')
    .run();

Tip: you can skip version (@53.0) or/and OS name (:Windows 10).

BrowserStack Proxy Options

Proxy options can be passed via environment variables.

BROWSERSTACK_PROXY - a string that specifies a proxy for the BrowserStack local binary. It should have the following structure: user:pass@proxyHostName:port,
BROWSERSTACK_LOCAL_PROXY - a string that specifies a proxy for the local web server. It should have the following structure: user:pass@proxyHostName:port,
BROWSERSTACK_FORCE_PROXY - if it's not empty, forces all traffic of BrowserStack local binary to go through the proxy,
BROWSERSTACK_FORCE_LOCAL - if it's not empty, forces all traffic of BrowserStack local binary to go through the local machine
BROWSERSTACK_NO_LOCAL - If it's not empty, forces all traffic of BrowserStack to go over public internet
BROWSERSTACK_LOCAL_IDENTIFIER - a string identifier of an open BrowserStack local tunnel. If it's not empty, a new local tunnel is not created. Instead, the browser provider uses an existing local tunnel with the specified identifier.

Example:

export BROWSERSTACK_PROXY="user:p@ssw0rd@proxy.com:8080"
export BROWSERSTACK_LOCAL_PROXY="admin:12345678@192.168.0.2:8080"
export BROWSERSTACK_FORCE_PROXY="1"
export BROWSERSTACK_FORCE_LOCAL="1"
testcafe browserstack:chrome test.js

Other BrowserStackLocal Options

This plugin also allows you to specify the following BrowserStackLocal options via environment variables:

Option	Environment Variable
binarypath	`BROWSERSTACK_BINARY_PATH`
logFile	`BROWSERSTACK_LOGFILE`
verbose	`BROWSERSTACK_VERBOSE`

Example:

export BROWSERSTACK_BINARY_PATH="~/BrowserStack/BrowserStackLocal"
export BROWSERSTACK_LOGFILE="~/BrowserStack/logs.txt"
export BROWSERSTACK_VERBOSE="1"
testcafe browserstack:chrome test.js

BrowserStack JS Testing and BrowserStack Automate

BrowserStack offers two APIs for browser testing:

JS testing supports more types of devices (compare: JS Testing Devices vs Automate Devices), while Automate allows for much longer tests (2 hours vs 30 minutes) and provides some additional features (like the window resizing functionality).

TestCafe uses the JS Testing API by default. In order to use BrowserStack Automate, set the BROWSERSTACK_USE_AUTOMATE environment variable to 1.

Example:

export BROWSERSTACK_USE_AUTOMATE="1"
testcafe browserstack:chrome test.js

Setting Display Resolution

To set the display resolution, use the BROWSERSTACK_DISPLAY_RESOLUTION environment variable or the resolution property in the configuration file. Valid resolutions can be found here.

Remember that this only sets the display resolution and does not resize the browser window. You'll still need to use TestCafe's window resizing API to do so.

Example:

export BROWSERSTACK_DISPLAY_RESOLUTION="1024x768"
testcafe browserstack:chrome test.js

Specifying Chrome Command Line Arguments

To set Chrome command line arguments, use the BROWSERSTACK_CHROME_ARGS environment variable. You can specify multiple arguments by joining them with the space symbol. This option works only if the BrowserStack Automate API is enabled.

Examples:

export BROWSERSTACK_USE_AUTOMATE="1"
export BROWSERSTACK_CHROME_ARGS="--autoplay-policy=no-user-gesture-required"
testcafe browserstack:chrome test.js

export BROWSERSTACK_USE_AUTOMATE="1"
export BROWSERSTACK_CHROME_ARGS="--start-maximized --autoplay-policy=no-user-gesture-required"
testcafe browserstack:chrome test.js

Other BrowserStack Options

BrowserStack Automate allows you to provide options for its internal Selenium Grid in the form of key-value pairs called capabilities.

To specify BrowserStack capabilities via the TestCafe BrowserStack provider, use environment variables or the configuration file. This provider supports the following capabilities:

Capability	Environment Variable
`project`	`BROWSERSTACK_PROJECT_NAME`
`build`	`BROWSERSTACK_BUILD_ID` (`BROWSERSTACK_BUILD_NAME` may also be used)
`resolution`	`BROWSERSTACK_DISPLAY_RESOLUTION`
`name`	`BROWSERSTACK_TEST_RUN_NAME`
`acceptSslCerts`	`BROWSERSTACK_ACCEPT_SSL_CERTS`
`browserstack.debug`	`BROWSERSTACK_DEBUG`
`browserstack.console`	`BROWSERSTACK_CONSOLE`
`browserstack.networkLogs`	`BROWSERSTACK_NETWORK_LOGS`
`browserstack.video`	`BROWSERSTACK_VIDEO`
`browserstack.timezone`	`BROWSERSTACK_TIMEZONE`
`browserstack.geoLocation`	`BROWSERSTACK_GEO_LOCATION`
`browserstack.customNetwork`	`BROWSERSTACK_CUSTOM_NETWORK`
`browserstack.networkProfile`	`BROWSERSTACK_NETWORK_PROFILE`

Refer to the BrowserStack documentation for information about the values you can specify.

Example

export BROWSERSTACK_DEBUG="true"
export BROWSERSTACK_TIMEZONE="UTC"
testcafe browserstack:chrome test.js

Configuration File

You can specify BrowserStack capability options in a JSON configuration file as an alternative to environment variables. Use capability names for configuration file properties. If an option is set in both the configuration file and an environment variable, the environment variable setting takes priority.

To use a configuration file, pass the file path in the BROWSERSTACK_CAPABILITIES_CONFIG_PATH environment variable:

export BROWSERSTACK_CAPABILITIES_CONFIG_PATH="./data/browserstack-config.json"
testcafe browserstack:chrome test.js

browserstack-config.json

{
    "build":                       "build-1",
    "project":                     "my-project",
    "resolution":                  "1024x768",
    "name":                        "Run 1",
    "browserstack.debug":          true,
    "browserstack.console":        "errors",
    "browserstack.networkLogs":    true
}

Exceeding the Parallel Test Limit

When you run tests in multiple browsers or concurrently, you may exceed the maximum number of parallel tests available for your account.

Assume your plan allows 2 parallel tests, and you run one of the following commands:

testcafe 'browserstack:ie@11.0:Windows 10','browserstack:chrome@59.0:Windows 10','browserstack:safari@9.1:OS X El Capitan' tests/acceptance/

testcafe browserstack:ie@11.0:Windows 10 -c3 tests/acceptance/

In this instance, BrowserStack will refuse to provide all the required machines and TestCafe will throw an error:

Unable to establish one or more of the specified browser connections.

To keep within your account limitations, you can run tests sequentially (or in batches), like in the following bash script (credits to @maoberlehner for this example):

browsers=( "browserstack:ie@10.0:Windows 8" "browserstack:ie@11.0:Windows 10" "browserstack:edge@15.0:Windows 10" "browserstack:edge@14.0:Windows 10" "browserstack:firefox@54.0:Windows 10" "browserstack:firefox@55.0:Windows 10" "browserstack:chrome@59.0:Windows 10" "browserstack:chrome@60.0:Windows 10" "browserstack:opera@46.0:Windows 10" "browserstack:opera@47.0:Windows 10" "browserstack:safari@9.1:OS X El Capitan" "browserstack:safari@10.1:OS X Sierra" )

for i in "${browsers[@]}"
do
	./node_modules/.bin/testcafe "${i}" tests/acceptance/
done

Configuring the API Polling Interval for BrowserStack Automate

BrowserStack Automate is based on WebDriver, which forcefully shuts down inactive sessions after an idle timeout expires. This works for WebDriver users, since each page action (clicks, types, etc.) triggers a WebDriver command and thus resets the idle timer.

However, TestCafe is not WebDriver-based. It simulates page actions in a different way and it doesn't trigger WebDriver commands. To prevent test session from being terminated by the BrowserStack WebDriver server due to inactivity, TestCafe triggers a dummy WebDriver command once in a while.

However, if the network connection is unstable, a request that triggers this dummy command can fail. In this instance, the BrowserStack WebDriver server doesn't receive the command before the idle timeout expires, and the test session can be terminated due to inactivity.

If your BrowserStack builds are terminated due to the idle timeout frequently, you can try to decrease the delay before the dummy WebDriver command is sent. In case the first request fails to trigger the command due to a network problem, the next may succeed and thus prevent your test session from being terminated.

Use the TESTCAFE_BROWSERSTACK_API_POLLING_INTERVAL environment variable to control this delay. This variable specifies time (in millisecinds) to pass until an additional request that triggers an dummy WebDriver command is sent to the BrowserStack WebDriver server. The default delay is 80000 millisecinds. If the BrowserStack idle timeout is 90 seconds (or 90000 milliseconds), at least one request is processed by the BrowserStack server in normal network conditions. If you set it to 40000, two requests are processed by the WebDriver server if your network is good. In case of network issues, either request may fail without breaking the build.

Example

export TESTCAFE_BROWSERSTACK_API_POLLING_INTERVAL="40000"
testcafe browserstack:chrome test.js

Author

Developer Express Inc. (https://devexpress.com)