Simple and consistent visual regression testing
robot-eyes is a CI based library for visual regression testing. It works best with docker, since a page renders differently on linux and mac for example. Rest assured that you don't need to know much about the docker to make it work.
Everything revolves around 3 simple commands:
- test: compares the current layout with the previous one.
- report: shows the differences that were found, in an interactive way.
- approve: when there is still no reference image, this is the way to approve the current layout.
Table of contents
Getting started
Run this command in your project's root directory.
npx robot-eyes init
After that, you will probably see a message like this: Alright, you are ready to go!
. If this is the case, this folder structure must be present in your project:
.
├── robot-eyes
│ ├── docker-compose.yml
│ ├── example_app
│ │ ├── about.html
│ │ ├── contact.html
│ │ ├── index.html
│ │ └── robot-eyes.jpg
│ ├── images
│ │ └── reference_images
│ │ ├── contact
│ │ │ └── contact1920x1080.png
│ │ └── home
│ │ └── home1920x1080.png
│ ├── robot-eyes.json
└── └── test.js
The next commands must be executed inside the robot-eyes
folder. The init command that we executed, created a specific docker-compose.yml file for robot-eyes. This file has the 3 commands you will need. Let's start by running the test
.
docker-compose run test
That's OK, we expect some to fail. We did this on purpose so that you learn to use the second command, the report
.
robot-eyes
1) Home
✓ Contact (1146ms)
2) About
1 passing (3s)
2 failing
1) robot-eyes
Home:
Error: 1920x1080: Images are not the same. See difference at /data/images/diff_images/home/home1920x1080.png.
at Promise (/usr/src/compare/index.js:38:14)
2) robot-eyes
About:
Error: 1920x1080: ENOENT: no such file or directory, open '/data/images/reference_images/about/about1920x1080.png'
at Promise (/usr/src/compare/index.js:38:14)
To correct tests when they break, we need to analyze the differences and see if they are expected or not. The best way to do this is to run the report
command.
docker-compose up report
Then go to http://localhost:3000/
and you will be able to see the report page.
Now just use the tools in the report, and if everything is right (in this case a difference similar to the one in the image above should appear), just approve. You should see a message like that in the terminal.
Approved home 1920x1080
If you run docker-compose run test
again, there still one error left. When you do your tests, the first time you run it, as it doesn't have a reference image yet, the report will not work for this test. There comes the last command, the approve
. It is useful in this case, and also when you want to approve an image without seeing the report. So run this:
docker-compose run approve
And if everything went well so far, your tests should pass when you execute docker-compose run test
robot-eyes
✓ Home (711ms)
✓ Contact (1124ms)
✓ About (610ms)
3 passing (2s)
Now let's talk about how to test your application!.
Setting up
After going through all the steps of the getting started section, you are able to configure robot-eyes to your needs. robot-eyes.json
is the file that contains the general settings, which are valid for all tests. Here's a list of the available options:
Property | Description | Default value |
---|---|---|
baseURL |
Link to the main page of your application (shouldn't contain '/'). Example: https://github.com | null |
paths.testImages |
relative path where temporary test image files will be saved. Example './test_images/' | './images/test_images' |
paths.diffImages |
relative path where temporary diff image files will be saved. Example './diff_images/' | './images/diff_images' |
paths.referenceImages |
relative path where the baseline/reference images will be saved. Example './reference_images/' | './images/reference_images' |
viewports |
Array of objects containing the width and height that will be tested | {width: 1920, height: 1080} |
timeout |
Mocha timeout | 40000 |
headless |
Chrome browsing mode. Is important to know that, headless and headed generate different images. | true |
threshold |
Maximum percentage of different pixels for the test to pass | 0.01 |
waitForResourceTimeout |
Maximum time (in ms) to wait for baseURL to be available | 60000 |
resembleOptions |
Config options for Resemble.js. This object is passed directly to Resemble's compareImages function. You can set the parameters it will use to compare the images, for instance, change the default ignore: "nothing" to ignore: "antialiasing" to make it more tolerant to small differences. |
|
Example robot-eyes.json file
{
"baseURL": "https://github.com/mercos/robot-eyes",
"paths": {
"testImages": "./test_images"
}
"viewports": [
{
width: 1920,
height: 1080
},
{
width: 1366,
height: 768
}
],
timeout: 30000,
threshold: 0.02
}
Configure your server
The docker-compose-yml
file comes practically ready, but it lacks the main feature. The default file comes with a simple example of a server using the nginx image, but how you will run your server is up to you.
We suggest that you run your application through the docker too, but if not, here is a tip:
Connect robot-eyes with the application that is running locally: in docker-compose.yml
, maybe change --base-url=http://web:80
with --base-url=host.internal.docker:YOUR_PORT
can work.
Create your tests
Currently we support only mocha. We recommend that you edit the test.js
file that is created by the init
command.
const test = require('robot-eyes/test')
describe('Your website', function () {
it('Login', function () {
return test('/login', this.test.title)
})
})
Aditional options
robot-eyes/test receives an object as 3rd argument. By default, robot-eyes only go to the page, and take a screenshot, with this options you can add some extra behavior. Here is the list of available options:
cookies
: If your application needs authorization, it's the best option. Here's an example:
const test = require('robot-eyes/test')
const getCookies = require('./getCookies')
describe('Your website', function () {
let cookies = []
before(async function () {
cookies = await getCookies()
})
it('Transactions', function () {
return test('/transactions', this.test.title, {cookies})
})
})
If you need help, here's an example of getting cookies with puppeteer: https://github.com/brendonbarreto/robot-eyes/blob/master/examples/getCookies.js
delay
: Wait for some time(ms), or a specific selector to be present. Examples: 1000
, '.card'
.
removeSelectors
: Remove from DOM a list of selectors. Example: ['.container-time', 'container-date']
onReady
: function to be right before the screenshot. It receives puppeteer page, as first argument. Useful for more complex setups:
onReady: async page => {
await page.click('.btn-open-modal')
await page.waitForSelector('.modal.in')
await page.waitForTimeout(150)
}
If you miss any options you can create an issue
Commands
There are 3 commands, test, report and approve. If you need help, you can use --help
and look at the description of each one.
test
robot-eyes test testfile.js
Do all the magic...Capture screenshots and compare with baselines.
-
--grep
: specify a patern to match test name -
--report
: open report after test if it fails
approve
robot-eyes approve "Test name"
Approve the test, in all viewports. You can use --viewport
to specify just one.
report
robot-eyes report
Start a express server where you can look and see and approve differences. Default port is 3000, but you can set it with --port