cloudflare-scraper
Chrome is used to retrieve cloudflare cookies then got is used to perform requests making this solution reliable but also pretty fast.
Version 2 is a complete rewrite:
- it doesn't use puppeteer but vanilla chromium,
- request package was replaced by got ,
- headless support only works on linux out of the box but should be doable on windows or mac os with the help of docker or wsl.
- extra features were removed (captcha bypass, etc..)
Install
npm install cloudflare-scraper
Make sure you alse have xfvb linux package installed
# for ubuntu users
sudo apt-get install xvfb
Quick Example
import got from 'cloudflare-scraper';
(async () => {
try {
const response = await got.get('https://nowsecure.nl');
console.log(response.body);
} catch (error) {
console.log(error);
}
})();
API
Check got documenatation
Env variables
NODE_CHROMIUM_SKIP_INSTALL (boolean)
By default, chromium is downloaded but on npm install
command but you can skip the installation by enabling this variable.
export NODE_CHROMIUM_SKIP_INSTALL=true
CHROME_EXECUTABLE_PATH (string)
Specify a chrome executable
export CHROME_EXECUTABLE_PATH=/path/to/chrome
CF_SCRAPER_HEADLESS (boolean)
Enable/disable headless mode (enabled by default)
Note: headless mode uses "xfvb" and is only available on linux
export CF_SCRAPER_HEADLESS=false
TODO:
- add proxy support
- docker example