Pillage is a super awesome Node.js library for parsing webpages. It uses a baller algorithm✝ to identify the content region of a webpage with accuracy that's really, really, really, really ... fun. Once we have the content region we can parse out text, images, videos and other media. We also threw in a lot of the easy stuff like OG tags for your convenience.
✝ It basically searches for every text node, then recursively climbs the parent tree, assigning a weighed "score" based on text length to each parent. The value rapidly drops off as we move up the tree. This is done for all text nodes so the weights accumulate to identify the most probable shared parent. Once we have that wrapper we can make assumptions and easily parse out body content.
Install
npm install pillage
Usage
var pillage = ; // Fetch a URL and process; // or, process HTML directlyvar result = ;console;
// Here's the object structure that it will return return title: description: text: images: videos: twitterTags: openGraphTags: articleTags: oEmbed: ;
License
MIT
Author
Mike Holly