npm

robots-txt

1.0.0 • Public • Published

robots-txt Build Status

Handles checking of robots.txt. handles concurrent requests and caching.

usage

var robots = require('robots-txt'),
    level  = require('level');
 
var bot = robots({
  db: level('./robots-txt-cache'),
  ttl: 1000 * 60 * 60 * 24 // one day
});
 
bot.isAllowed('woobot', 'http://www.woorank.com/my/path')
  .then(function (isAllowed) {
    // ...
  });
 

This module fetches robots.txt files and optionally caches them in a level-db. parsing and checking of robots.txt files is done by secondary modules robots-txt-parse and robots-txt-guard and can be used separately.

Readme

Keywords

Package Sidebar

Install

npm i robots-txt

Weekly Downloads

4

Version

1.0.0

License

MIT

Last publish

Collaborators

  • woorank-admin
  • janpotoms
  • ndemoor
  • woorank-ci