@p26e/harvest-npm

harvest-npm

Harvest npm packages for offline use.

You can install harvest-npm using NPM: npm i -g @p26e/harvest-npm, or use the provided docker image (see below).

Usage

Create a pinfile (e.g. pinfile.json) containing the packages you want to download. Each package must have a corresponding semver string, specifying which versions of the package should be downloaded. All versions of a package matching the given semver will be downloaded.

See harvest-npm --help for a complete list of options.

Example

alice@computer$ cd /var/www

alice@computer$ echo '{ "react": "^17 || 16.8.6", "better-sqlite3": ">= 7.4.3"  }' > pinfile.json

alice@computer$ harvest-npm

// pinfile.json
{
	"react": "^17 || 16.8.6",
	"better-sqlite3": ">= 7.4.3"
}

How it works

TL;DR: Every time harvest-npm is executed, it will find all versions satisfying the given semver for each package specified in the pinfile. It will only download package versions that hasn't previously been downloaded (according to the lockfile). The outputed directory can be served as static files to function as a read-only NPM registry.

The key part of harvest-npm is to create a pinfile. The pinfile is a JSON file containing packages and corresponding semver strings (harvest-npm uses this NPM package to resolve matching packages). Let's see an example of a pinfile:

{
	"react": "^17 || 16.8.6",
	"better-sqlite3": ">= 7.4.3"
}

This pinfile contains two packages that harvest-npm should keep track of. The semver string (the value of the key-value style) specifies which versions that should be downloaded. For the first package, react, this resolves to all version with major version 17, and version 16.8.6. Every time harvest-npm is executed, it will find all packages that satisfy this requirement. At the time of writing, this equates to four packages: 17.0.0, 17.0.1, 17.0.2 (latest) and 16.8.6. The second package's semver, >= 7.4.3, resolves to all versions (including major versions) greater than or equal to 7.4.3. At the time of writing 7.4.3 is the latest version, thus only that version would be downloaded.

When package versions are downloaded, all their dependencies (and all their dependencies etc) are also dowloaded. When a package is downloaded its name and version (and other metadata) are written to a lock file (default lockfile.json). Before new packages are downloaded, harvest-npm will consult this file to see if the spesific version has already been downloaded. If it has, the package will be skipped to save bandwith (and time).

The output directory (specified through the --output-dir option) can be served as static files over http and function as a read-only NPM registry. Remember to set the --base-url to the URL you are going to serve the registry under (e.g. https://mydomain.tld/path/to/registry).

Docker

We have created a docker image for ease of use. To use the docker image, mount a folder containing the pinfile at /registry. The files will be written to /registry/packages and the script is automatically executed every time the pinfile changes.

Example:

alice@computer$ mkdir ~/registry

alice@computer$ echo '{ "react": ">= 17" }' > ~/registry/pinfile.json

alice@computer$ docker run -v /home/alice/registry:/registry ghcr.io/p26e/harvest-npm:latest

The following options are available through environment variables:

PINFILE

Default: '/registry/pinfile.json'
Location of the pinfile.

LOCKFILE

Default: '/registry/lockfile.json'
Location of the lockfile.

OUTPUT_DIR

Default: '/registry/packages'
Direcotry to output packages in.

BASE_URL

Default: 'http://registry.npmjs.local/packages'
URL where the downloaded packages will be exposed, used when generating metadata for packages. Only needed if you're going to serve the output directory as static files.

ABI

Default: '93,83,72'
Comma separated list of Node C++ ABI numbers used when downloading pre-built binaries. See the NODE_MODULE_VERSION column at https://nodejs.org/en/download/releases/.

ARCH

Default: 'x64'
Comma separated list of CPU architectures used when downloading pre-built binaries. Valid options: arm, ia32 and x64.

PLATFORM

Default: 'linux,darwin,win32'
Comma separated list of OS platforms used when downloading pre-built binaries. Valid options: linux, darwin, win32, sunos, freebsd, openbsd, and aix.

REGISTRY

Default: 'https://registry.npmjs.org'
NPM Registry URL to download packages from.

TOKEN

Default: ''
Bearer token for the registry when downloading packages.

SCHEDULE

Default: ''
A cron style scheduled interval when the registry should sync. Example: '0 2 * * *' (every night at 2 AM).