grab-dans-tes-mains

All the grabbers we need to get content from l'internet mondial.

Install

1. Yarn is a promising node package manager. We recommend to use it. So first make sure you have a global version of it (https://yarnpkg.com/en/docs/install).

2. Install the node_modules with

  yarn

3. Symlink the 'scrap' binary command

  yarn symlink

Run locally

1. Launch just once a watcher that will transpile your babel src files into vanilla javascript lib files anytime you do change in your src grabber files.

  yarn dev

2. You can execute a grab run with the following command, for example

  grab --url=webhose

Or run directly the node scripts/grab.js file.

Add a new provider

1. Follow the example given with grab/webhose, and create your own

2. Don't forget to add a test file in the test folder, and run the test !

  yarn test

example

  grab --name=climate-feedback-users --mongo=mongodb://localhost:27017/reval --save=true