Webscraper Exporter
A simple yet powerful prometheus
exporter for website performance metrics built using puppeteer
.
Table of contents
CLI
Currently only tested on macOS
and Linux
, should support Windows
Getting started
Installation of the CLI
You can install the cli included to get started quickly with barely no configuration:
npm install --global webscraper-exporter
or
yarn global add webscraper-exporter
You can then check if everything went OK by running:
wsce --version
This should print the package's version
Configuration
In your current working directory, create a file named wsce.config.js
with the following example content:
module.exports = {
scraper: {
urls: ["https://google.com"],
puppeteerOptions: {},
addons: [],
interval: 60_000,
},
exporter: {
port: 9924
},
};
For further information on the configuration, see CONFIGURING.md
Using the CLI
To see all available options for a command, see wsce <command> -h
;
Starting
You can start the exporter by simply executing:
wsce start
You can then access the metrics at http://localhost:9924
If you want more detailled logs, you can pass the -v
argument.
To pass a custom config file, use -c path/to/file
Creating a project
Create a project from one of the included templates by running:
wsce init my-project
You will then be prompted for the project template. You can also directly pass the template to wsce init
, e.g. :
wsce init my-project --template javascript
By default, wsce
includes a typings file for the config so it's easier to fill. If you don't want that, you can pass --typings=false
to the init
command.
Adding, Removing and Listing Templates
wsce template
has 3 subcommands:
-
add
: Add a template from a local directory / remote repository -
remove
: Remove a saved template -
list
: List all saved templates
For more information on templates, see TEMPLATES.md
NodeJS module
Installation of the module
npm install --save webscraper-exporter
or
yarn add webscraper-exporter
Using the module
import { Exporter, Scraper } from "webscraper-exporter";
const scraper = new Scraper({
urls: ["https://cstef.dev"],
addons: [
{
name: "Logger",
when: "before",
run: (browser, page, URL) => {
console.log(`I am running on ${URL}`);
},
},
],
interval: 60_000,
});
scraper.start();
const exporter = new Exporter({
scraper,
port: 9924,
});
exporter.start();
For more examples, see the examples
folder.
Docker
There is a Dockerfile
included in this repository so you can build your own image for wsce
(based on the node:alpine-16
image).
See DOCKER.md for more information.
Exported Data
You can see the metrics that are exposed in: EXPORTED_DATA.md
Customization
You can easily write your own plugins in Javascript to expand webscraper-exporter
's functionalities.
See ADDONS.md for more information.
Contributing
Pull requests are always welcome if you feel like fixing / expanding something in this project !