scraper-api
Interface to call ScraperAPI.com easily from Node. All current API endpoints and features are implemented in this module. Requires Node 8+.
Install
$ npm install --save scraper-apiUsage
const scraperAPI = // ... options; scraperAPI ;API Documentation
scraper-api
scraperAPI(options) ⇒ ScraperAPI ⏏
Creates new instance of ScraperAPI with the provided options.
Kind: Exported function
Returns: ScraperAPI - New instance of ScraperAPI.
| Param | Type | Default | Description |
|---|---|---|---|
| options | Object |
Optional configuration options to pass into ScraperAPI. | |
| [options.apiKey] | String |
process.env.SCRAPER_API_KEY |
API key for Scraper API. Defaults to pulling API Key from environment variable. |
| [options.renderJs] | Boolean |
false |
Render JavaScript on the page before scraping the HTML for the page. |
| [options.keepHeaders] | Boolean |
false |
Keep headers sent in the request to Scraper API in subsequent request(s) when scraping the provided url. You must set your headers in options.gotOptions.headers. |
| [options.geoCode] | String |
'us' |
Geo code in which to use proxies for when scraping. See documentation for more information. |
| [options.premium] | Boolean |
false |
Whether to use premium proxies. Caution: This will cost 10-25 times more than standard proxies. |
| [options.sessionId] | Number |
A numeric session id to use to maintain the same proxy. See ScraperAPI.session() for more information. | |
| [options.gotOptions] | Object |
{} |
Additional options to pass into got for requests to ScraperAPI. |
Example
const scraperAPI = // options...;scraperAPI.ScraperAPI
Access to the uninstantiated ScraperAPI class.
Kind: static property of scraperAPI
Example
const ScraperAPI = ScraperAPI;const scraperAPI = // options...;ScraperAPI
Kind: global class
- ScraperAPI
- new ScraperAPI([options])
- scraperAPI.session(id, [options]) ⇒
ScraperAPI - scraperAPI.get(url, [options]) ⇒
Promise.<String> - scraperAPI.post(url, data, [options]) ⇒
Promise.<Object>
new ScraperAPI([options])
Constructor for the ScraperAPI class.
| Param | Type | Default | Description |
|---|---|---|---|
| [options] | Object |
{} |
Options for the ScraperAPI class. Options may be overridden on all methods for a single request. See options above for more information. |
scraperAPI.session(id, [options]) ⇒ ScraperAPI
Creates a new instance of ScraperAPI with the specified session id. Sessions allow subsequent requests with the same session id to go through the same proxy. See documentation for more information.
Kind: instance method of ScraperAPI
Returns: ScraperAPI - A new instance of ScraperAPI with sessionId set to the provided id.
| Param | Type | Default | Description |
|---|---|---|---|
| id | Number |
Session ID in which to use for the given session. Must only contain numbers. | |
| [options] | Object |
{} |
Options to override for all subsequent requests to Scraper API. Same as the global options. |
Example
const session = scraperAPI; let result = await scraperAPI;// result -> '<!doctype html> ...'scraperAPI.get(url, [options]) ⇒ Promise.<String>
Calls Scraper API with a GET request to the provided url.
Kind: instance method of ScraperAPI
Returns: Promise.<String> - Promise that resolves with HTML source from requested URL.
| Param | Type | Default | Description |
|---|---|---|---|
| url | String |
The URL in which to scrape. | |
| [options] | Object |
{} |
Options to override for this specific request. |
Example
let result = await scraperAPI;// result -> '<!doctype html> ...'scraperAPI.post(url, data, [options]) ⇒ Promise.<Object>
Calls Scraper API with a POST request to the provided url with the provided data.
Kind: instance method of ScraperAPI
Returns: Promise.<Object> - Promise that resolves with object response from Scraper API. See documentation.
| Param | Type | Default | Description |
|---|---|---|---|
| url | String |
The URL in which to scrape. | |
| data | Object | form-data |
Data in which to post to the provided URL. Must be either a plain object or instance of form-data. | |
| [options] | Object |
{} |
Options to override for this specific request. May be any of the global options and any additional options below. |
| [options.form] | Boolean |
false |
Set true if provided data is form data and should be sent as such. By default, data will be sent as JSON. |
Example
let result = await scraperAPI;More Information
keepHeaders Option
If you would like to pass custom headers through Scraper API to the destination, you may do so by setting your custom headers in options.gotOptions.headers and enabling this option.
let result = await scraperAPI;Tests
Tests are written and provided as part of the module. You may run the tests by calling:
$ npm run testLicense
MIT License. See License in the repository.