robotto

1.0.16 • Public • Published

Build Status Coverage Status Dependencies Status Code Quality Status This is Awesome

robotto

Easily read, parse and check permissions on robots.txt files!

Installing

You can easily install robotto using npm. Just run:

npm install robotto

Getting Started

After installing robotto you need to require it using:

    var robotto = require('robotto');

Now you can call every single robotto's methods using your robotto variable!

To check if a website allows you to crawl a certain url you just gotta call the canCrawl(userAgent, fullUrl, callback) method. Take a look below for an example on how to do this:

var robotto = require('robotto');
 
robotto.canCrawl('ExampleBot', 'https://www.npmjs.com/browse/keyword/robotto', function(err, isAllowed) {
    if (err) {
        console.error(err);
        return;
    }
 
    if (isAllowed) {
        // Your crawling code goes here...
    } else {
        console.log('I am not allowed to crawl this url.');
    }
});

Simple, isn't it?

Docs

Methods

robotto.getRobotsUrl(url)

Returns the full url to the website's robots.txt.

Parameters
  • url -> Any url
Example
var robotsUrl = robotto.getRobotsUrl('http://www.007.com/characters/the-bonds/');
console.log(robotsUrl);
// --> http://www.007.com/robots.txt

robotto.fetch(url, callback)

Gets the content of the robots.txt file.

Parameters
  • url -> Any url
  • callback(err, content)
    • err -> Errors, if any, otherwise null
    • content -> Content of the robots.txt file
Example
robotto.fetch('https://nodejs.org/robots.txt', function(err, content) {
    if (err) {
        console.error(err);
        return;
    }
 
    console.log('robots.txt content:\n' + content);
});

robotto.parse(text)

Parses the content of a robots.txt file and returns an object with it's rules.

This is an example of the rules object returned:

{
    comments: ["This is a comment", "This is another comment"],
    userAgents: {
        Googlebot: {
            allow: ['/cats/', '/dogs/'],
            disallow: ['/sharks/', '/tigers/']
        },
        Msnbot: {
            allow: ['/technology'],
            disallow: ['/whatever/', '/example/']
        }
    }
}
Parameters
  • text -> The content of a robots.txt file
Example
robotto.fetch('https://nodejs.org/api/cluster.html', function(err, content) {
    if (err) {
        console.error(err);
        return;
    }
 
    var rules = robotto.parse(content);
    // Rules will have an object containing the robots.txt rules for nodejs.org
});

robotto.getRuleDeepness(ruleName, userAgent, urlParam, rulesObj)

Returns the deepness (precision) for a rule, because if an allow and a disallow rule conflict, the most specific is the one to be taken into account.

For example:

  • / is disallowed
  • /magical-route is allowed

In this case we shouldn't be able to crawl any URL except the ones into /magical-route/, because it's the most specific rule.

If the rule you requested is invalid it returns -1. If there isn't any rule for the url you want, it returns 0, otherwise it will return the deepness of the rule (number of parameters specified in robots.txt).

Parameters
  • ruleName -> A string containing allow or disallow
  • userAgent -> The user-agent's name
  • url -> Any url
  • rulesObject -> An object with rules
Example
robotto.fetch('https://twitter.com', function(err, content) {
    if (err) {
        console.error(err);
        return;
    }
 
    var rulesObj = robotto.parse(content);
    var deepness = robotto.getRuleDeepness('disallow', 'myBot', 'https://twitter.com/search/realtime/new', rulesObj);
 
    // Suposing that Twitter's robot.txt file contains:
    // "disallow: /search/realtime/"
    console.log(deepness) // --> 2
 
    // Deepness is two because the routes '/search/' and '/realtime/' where specified
});

robotto.check(userAgent, url, rulesObject)

Checks if an user-agent has permission to crawl an url based on an object with rules. If there are contraditory rules the most specific will be considered.

Parameters
  • userAgent -> The user-agent's name
  • url -> Any url
  • rulesObject -> An object with rules
Example
robotto.fetch('http://www.amazon.com/gp/', function(err, content) {
    if (err) {
        console.error(err);
        return;
    }
 
    var rules = robotto.parse(content);
    // Rules will have an object containing the robots.txt rules for amazon.com
    var permission = robotto.check('MyBotName', 'http://www.amazon.com/gp/goldbox/', rules);
 
    console.log('Permission to Crawl http://www.amazon.com/gp/goldbox/' + permission);
});

robotto.canCrawl(userAgent, url, callback)

Checks if an user-agent has permission to crawl an url.

Parameters
  • userAgent -> The name of your bot (your user-agent)
  • url -> Any url
  • callback(err, isAllowed)
    • err -> Errors, if any, otherwise null
    • isAllowed -> Boolean value indicating if this user-agent is allowed to crawl the url
Example
robotto.canCrawl('https://twitter.com/zenorocha', function(err, isAllowed) {
    if (err) {
        console.error(err);
        return;
    }
 
    if (isAllowed) {
        console.log('I can crawl this url!');
    }
});

Contributing

Feel free to contribute in any way you want. Every help is valid. If you find any issues or even if you have a suggestion please feel free to report it using our issue tracker.

If you want to write some code follow these steps:

  1. Clone this repo using:

    $ git clone https://github.com/trachelas/robotto
    
  2. Go to the folder in which you cloned the repo and install the dev-dependencies:

    $ npm install
    
  3. Write your code

  4. Write tests for it (if needed)

  5. Run gulp build and make sure all the tests pass and that you've got 100% code coverage.

  6. Open a new pull-request

If you have any doubt please let us know. We'll do our best to explain every detail you need to know in order to contribute.

This is the OpenSource Software's magic ✨

Credits

Robotto is proudly brought to you by the Trachelas Team.

Name Github Twitter
Lucas F. da Costa github.com/lucasfcosta @lfernandescosta
Lucas Vieira github.com/vieiralucas

License

MIT License

No need for copyright, live free, buddy, internet is for everyone 😉

Package Sidebar

Install

npm i robotto

Weekly Downloads

32

Version

1.0.16

License

MIT

Last publish

Collaborators

  • lucasfcosta
  • trachelasteam
  • vieiralucas