crawler-request

HTTP request module customized for crawlers.

Installation

npm install crawler-request

Usage

Simple Request

const crawler = require('crawler-request');
 
crawler("https://stackoverflow.com/").then(function(response){
    // handle response
    console.log(response.text.lenght);
});

PDF Parse

const crawler = require('crawler-request');
 
crawler("http://careers.stackoverflow.com/stack_overflow_careers.pdf").then(function(response){
    // handle response
    console.log(response.text.lenght);
});

Extend

const crawler = require('crawler-request');
 
function response_text_size(response){
    response["size"] = response.text.length;
    return response;
}
 
crawler("https://stackoverflow.com/",response_text_size).then(function(response){
    // handle response
    console.log(response.size);
});

Test

mocha or npm test

Check test folder and quickstart.js for extra usages.

Support

I use this package actively myself, so it has my top priority. You can chat on WhatsApp about any infos, ideas and suggestions.

Submitting an Issue

If you find a bug or a mistake, you can help by submitting an issue to GitLab Repository

Creating a Merge Request

GitLab calls it merge request instead of pull request.

• A Guide for First-Timers
• How to create a merge request
• Check Contributing Guide

License

MIT licensed and all it's dependencies are MIT or BSD licensed.