ipfs_parquet_to_car

IPFS Huggingface Scraper

with interactive prompt:

node scraper.js [source] [model name]

node scraper.js 

import a model already defined:

node scraper.js hf "modelname" (as defined in your .json files)

import all models previously defined:

node scraper.js hf 

IPFS Huggingface Bridge:

for huggingface transformers python library visit: https://github.com/endomorphosis/ipfs_transformers/

for huggingface datasets python library visit: https://github.com/endomorphosis/ipfs_datasets/

for faiss KNN index python library visit: https://github.com/endomorphosis/ipfs_faiss

for transformers.js visit:
https://github.com/endomorphosis/ipfs_transformers_js

for orbitdb_kit nodejs library visit: https://github.com/endomorphosis/orbitdb_kit/

for fireproof_kit nodejs library visit: https://github.com/endomorphosis/fireproof_kit

for ipfs_kit python library visit: https://github.com/endomorphosis/ipfs_kit/

for ipfs_kit nodejs library visit: https://github.com/endomorphosis/ipfs_kit_js/

for python model manager library visit: https://github.com/endomorphosis/ipfs_model_manager/

for nodejs model manager library visit: https://github.com/endomorphosis/ipfs_model_manager_js/

for nodejs ipfs huggingface scraper with pinning services visit: https://github.com/endomorphosis/ipfs_huggingface_scraper/

for ipfs agents visit: https://github.com/endomorphosis/ipfs_agents/

for ipfs accelerate visit: https://github.com/endomorphosis/ipfs_accelerate/

Author - Benjamin Barber QA - Kevin De Haan