@elioway/spider

1.2.5 • Public • Published

spider

Get your schemon! Tim Bushell

Scrape Schema.org objects into mongoose schema files the elioWay.

This is a requirement of bones but it can also be run as the boilerplate of a web spidering project with scheming intentions.

Install

npm install @elioWay/spider --save

Usage

// yourapp.js
const yourAppSpider = require('@elioWay/spider');
var today = new Date()
// Create schemon the spider.
let schemon = new yourAppSpider(
    version = today.getFullYear() + '.' + today.getMonth() + '.' + today.getDate(), // Do change.
    depth = 2, // The deeper you go, the more objects you get. Go crazy.
    thingsSelector = '#thing_tree', // Don't change - but there is a bigger tree on the page.
    useOjectFields = true // Instead of 1 to 1 relationships to other Things, force String type.
  )
// Let schemon do spider things.
schemon.spider(
  // Wrap what schemon scraped.
  data => Spider.optimize(
    data
  )
)
node yourapp

Seeing is believing

git clone https://gitlab.com/elioschemers/spider/
cd spider
node test_spider

Credits

License

MIT Tim Bushell

elioway

Package Sidebar

Install

npm i @elioway/spider

Weekly Downloads

2

Version

1.2.5

License

MIT

Unpacked Size

76 kB

Total Files

48

Last publish

Collaborators

  • timitee