robotstxt-util
RFC 5234 spec compliant robots.txt builder and parser. 🦾
Install
npm i -D robotstxt-util
Use
Build robots.txt documents on fly:
import RobotsTxt from 'robotstxt-util'
// or const RobotsTXT = require('robotstxt-util')
const builder = new RobotsTXT()
const somebot = builder.newGroup('somebot')
somebot.disallow('/')
// or builder.disallow('/', 'somebot')
const otherbots = builder.newGroup('*')
otherbots.allow('/app')
otherbots.disallow(['/api', '/data'])
// additional products:
builder.newProduct('Sitemap', 'https://test.com/en/sitemap.xml')
builder.newProduct('Sitemap', 'https://test.com/tr/sitemap.xml')
Parse existing robots.txt documents:
const RobotsTXT = require('robotstxt-util')
const data = `
User-agent: Googlebot
Disallow: /nogooglebot/
Disallow: /api
User-agent: *
Allow: /
Disallow: /api
Sitemap: https://test.com/en/sitemap.xml
Sitemap: https://test.com/tr/sitemap.xml
`
const parser = new RobotsTXT(data)
parser.valid() // true|false
parser.share('json') // returns an object 👇
{
groups: {
Googlebot: {
rules: [
{disallow: '/nogooglebot/'},
{disallow: '/api'}
]
},
'*': {
rules: [
{allow: '/'},
{disallow: '/api'}
]
}
},
additional: [
{Sitemap: 'https://test.com/en/sitemap.xml'},
{Sitemap: 'https://test.com/tr/sitemap.xml'}
]
}
parser.share('robotstxt') // returns spec compliant txt 👇
`
user-agent: Googlebot
disallow: /nogooglebot/
disallow: /api
user-agent: *
allow: /
disallow: /api
Sitemap: https://test.com/en/sitemap.xml
Sitemap: https://test.com/tr/sitemap.xml
`
Parse Previously Exported Object
const RobotsTXT = require('robotstxt-util')
const builder = new RobotsTXT()
const obj = {
groups: {
Googlebot: {
rules: [
{disallow: '/nogooglebot/'},
{disallow: '/api'}
]
},
'*': {
rules: [
{allow: '/'},
{disallow: '/api'}
]
}
},
additional: [
{Sitemap: 'https://test.com/en/sitemap.xml'},
{Sitemap: 'https://test.com/tr/sitemap.xml'}
]
}
builder.load(obj)
Contributing
If you're interested in contributing, read the CONTRIBUTING.md first, please.
Version management of this repository done by releaser 🚀
Thanks for watching 🐬