robots-txt
Handles checking of robots.txt. handles concurrent requests and caching.
usage
var robots = level = ; var bot = ; bot ;
This module fetches robots.txt files and optionally caches them in a level-db. parsing and checking of robots.txt files is done by secondary modules robots-txt-parse and robots-txt-guard and can be used separately.