npm Postgres Mashup
A cli tool to replecate the npm skimdb into Postgres.
Now Utilizing JSONB!
New in Version 17
- All packages are captured (even those with unicode NULLs)
- All package data is captured in JSONB document just as it is in npm
- Deleted packages are marked accordingly
- Reporting tables are built out via postgres sql queries and JSONB functions
Breaking Changes
Version 17 newness brings a lot of breaking changes.
- Complete schema rethink. ANY PREVIOUS NPM-POSTGRES-MASHUP TABLE WILL BE DROPPED. If you like the old schema don't upgrade or use a different database for future versions.
- A relational model is no longer used. Instead the couchdb documents are stored in JSONB.
- npm-postgres-mashup is only intended to be used as a cli utility, npm2pg. The previous api was lacking, and really if you want something to interface with at this level you should just utilize the follow library directly.
- Download counts have been removed. Scraping the download-count api was slow and I felt like things weren't as efficient as they could have been. Download counts might come back but I need to rethink the approach.
Installation & Usage
npm install npm-postgres-mashup -g
This installs the cli tool "npm2pg". Next create an empty Postgres database.
Prior to replicating the data into the database, you'll need to build the schema. Running this after a major npm-postgres-mashup version change will drop some tables. To build the schema run:
npm2pg -d database -u username -p password --build-schema
Once the schema is built, you can start replicating the data. Do this by running:
npm2pg -d database -u username -p password
npm2pg can also build reporting tables from the package jsonb documents.
Add --reporting-tables parameter to run this after sync is complete.
Otherwise you can run the SQL yourself (see ./db/createReportingTables.sql)
To get to the cli help about additional parameters run:
npm2pg --help
npm-postgres-mashup will automatically add the necessary tables to the postgres database.
License
MIT