keywords:web-crawling

robotsforce1

The presidential package of robots.txt generators. A joint task force of elite directives securing the perimeter against unauthorized bots.

alexstevovich

published version 1.0.3, 20 days ago0 dependents licensed under $Apache-2.0

216

mcp-omnisearch

MCP server for integrating Omnisearch with LLMs

spences10

published version 0.0.4, 7 days ago0 dependents licensed under $MIT

247

ayakashi

The next generation web scraping framework

zisismaras

published version 1.0.0-beta8.4, 2 years ago0 dependents licensed under $AGPL-3.0-only

108

quickscraper-sdk

Quick Scraper SDK NodeJS APIs

quickscraper

published version 2.0.8, a year ago0 dependents licensed under $MIT

84

sitemapteam6

The elite unit of sitemap.xml generation—precise, efficient, dominating. If RobotsForce1 is your air defense, this is your recon mission.

alexstevovich

published version 1.0.0, 20 days ago0 dependents licensed under $Apache-2.0

63

crawler-ts-fetch

Lightweight crawler written in TypeScript using ES6 generators.

supergillis

published version 1.1.1, 4 years ago1 dependents licensed under $GPLv3

41

@crawlora/sdk

Crawlora SDK for interacting with the Crawlora platform.

crawlora-soni

published version 1.2.33, 3 months ago1 dependents licensed under $MIT

33

@crawlora/browser

Crawlora SDK for interacting with the Crawlora platform.

crawlora-soni

published version 1.1.3, 3 months ago0 dependents licensed under $MIT

31

crawler-ts-htmlparser2

Lightweight crawler written in TypeScript using ES6 generators.

supergillis

published version 1.1.1, 4 years ago0 dependents licensed under $GPLv3

33

crawler-ts

Lightweight crawler written in TypeScript using ES6 generators.

supergillis

published version 1.1.1, 4 years ago2 dependents licensed under $GPLv3

34

@bluggie/nodescrapy

Web crawler in NodeJS

bluggie

published version 0.1.6, 3 years ago0 dependents licensed under $MIT

34

Crawl-to-markdown is a powerful TypeScript package designed to search search engines for a given keyword, crawl the resulting websites, and deliver the content in clean, readable Markdown format. Additionally, it can directly crawl specified websites for