NetScrape
Web scraping for Node.js made efficient, simple, and compliant. NetScrape complies with the Robots Exclusion Protocol.
Installation
npm install netscrape
Usage
Netscrape is designed to be simple, but also extensible enough for advanced use cases. The following example demonstrates how to make a simple request to a website.
import Bot from 'netscrape';
const exampleBot = new Bot({ name: 'ExampleBot', version: '1.0' });
try {
const response = await exampleBot.makeRequest('https://www.example.com/path');
console.log(response.body);
} catch (error) {
console.error(error);
}
Bot#constructor
import Bot from 'netscrape';
type BotOptions = {
name: string;
version: string;
minimumRequestDelay?: number;
maximumRequestDelay?: number;
disableCaching?: boolean;
policyURL?: string;
hideLibraryAgent?: boolean;
userAgent?: string;
};
const exampleBot = new Bot({
name: 'ExampleBot' /* required, Name of your bot */,
version: '1.0' /* required, Version of your bot */,
minimumRequestDelay: 1000 /* optional, Minimum delay between requests in milliseconds */,
maximumRequestDelay: 5000 /* optional, Maximum delay between requests in milliseconds (default 10000) */,
disableCaching:
true /* optional, Disable caching of responses (default false) */,
policyURL:
'https://www.example.com/robots.txt' /* optional, URL to robots.txt file (default https://npm.im/netscrape) */,
hideLibraryAgent:
true /* optional, Hide the library agent from the user agent (default false) */,
userAgent:
'ExampleBot/1.0' /* optional, Custom user agent, overrides all other user agent fields */,
});
Bot#makeRequest
import Bot from 'netscrape';
const exampleBot = new Bot({ name: 'ExampleBot', version: '1.0' });
try {
/* Note: Bot#makeRequest automatically requests /robots.txt in the background */
const response = await exampleBot.makeRequest(
'https://www.example.com/path' /* required, well-formatted URL to make request to */,
false /* optional, should you return a byte stream instead of utf8 text */,
);
/* Bot#makeRequest returns the raw npm.im/got package request response */
console.log(response.body);
} catch (error) {
/* Robots.txt rejection, robots.txt 500 error, etc. */
console.error(error);
}
Bot#makeRequestWithBody
import Bot from 'netscrape';
const exampleBot = new Bot({ name: 'ExampleBot', version: '1.0' });
try {
/* Note: Bot#makeRequestWithBody automatically requests /robots.txt in the background */
/* Note: Bot#makeRequestWithBody automatically sets the
Content-Type header to application/json or text/plain based on the body type */
const response = await exampleBot.makeRequestWithBody(
'https://www.example.com/path' /* required, well-formatted URL to make request to */,
{
example: 'body',
} /* required, body to send to server (string or object) */,
{
'x-example-header': 'example header',
} /* optional, headers to send to server */,
'POST' /* optional, HTTP method to use (default POST) */,
);
/* Bot#makeRequestWithBody returns the raw npm.im/got package request response */
console.log(response.body);
} catch (error) {
/* Robots.txt rejection, robots.txt 500 error, etc. */
console.error(error);
}
License
MIT (C) 2023 Russell Steadman. See LICENSE file. Visit Google deps.dev for dependency information.