a1sabau

get-set-fetch

web crawler, parser and scraper with storage capabilities

get-set-fetch-extension-popup

Responsible for handling toolbar integration. Opens the admin front-end into a new tab.

get-set-fetch-extension-commons

Contains common typescript definitions and utilities used throughout the lerna monorepo.

get-set-fetch-scenario-extract-resources

Extract Resources scenario is used for extracting various resources from the corresponding sites.

get-set-fetch-extension-background

Handles the crawling and scraping logic.

get-set-fetch-extension-admin

React based frontend communicating with the backend (extension background script) via chrome.runtime.sendMessage. Bundled as a single page app with the help of react-router-dom.

get-set-fetch-extension

Bundles the entire monorepo sub-packges into a valid extension folder.

gsf-ext-extract-html-headings

Extract Html Headings scenario is used for extracting H1, H2, H3, H4, H5, H6 text content.

gsf-extension-extract-html-headings

Extract html headings (H1 - H6) content.

get-set-fetch-scenario-extract-html-content

Extract Html Content scenario is used for extracting html nodes text based on dom selectors.

gsf-extension-readability

Extract article content using Mozilla Readability library.

get-set-fetch-scenario-scrape-dynamic-content

extracts text and binary content from dynamic (javascript) pages based on CSS selectors

get-set-fetch-scenario-scrape-static-content

extracts text and binary content from static html pages based on CSS selectors

@get-set-fetch/test-utils

scraping test definitions, launches resources to be scraped under a configurable web server

@get-set-fetch/scraper

Plugin based node.js web scraper. It scrapes, stores and exports data. Supports multiple storage options: SQLite, MySQL, PostgreSQL. Supports multiple browser or dom-like clients: Puppeteer, Playwright, Cheerio, Jsdom.