@xan105/request

About

HTTP request library based around Node.js' HTTP(S) API interfaces:

• http/https
• http2¹
• undici/fetch (included in Node.js 18)¹

¹ Work in progress

Provides common features such as retry on error, following redirects, progress when downloading file, ...

This library isn't intented to compete nor replace the well known libraries such as got, axios, node-fetch, ... This is merely educational and for informational purposes in order to learn how HTTP requests work under the hood.

This was originally created as request-zero at a time were the module request was the main choice and I didn't quite like it. It had a ton of dependencies, didn't use promises and I needed something very simple.

Example

Simplest call

import { request } from "@xan105/request";
const res = await request("https://www.google.com");
console.log(res.body);

JSON

import { getJSON } from "@xan105/request";

const json = await getJSON("https://jsonplaceholder.typicode.com/todos/1");
console.log(json); 
/*Output:
{ userId: 1, id: 1, title: 'delectus aut autem', completed: false }
*/

//Github API
const json = await getJSON("https://api.github.com/repos/user/repo/releases/latest",{
  headers: {"Accept" : "application/vnd.github.v3+json"}
});
console.log(json);
/*Output:
{ url: '...', tag_name: '0.0.0', target_commitish: 'master', ... }
*/

Download file(s)

import { download, downloadAll } from "@xan105/request";

//Callback example to output progress in the console
function printProgress(percent, speed, file){
  process.stdout.clearLine();
  process.stdout.cursorTo(0);
  process.stdout.write(`${percent}% @ ${speed} kb/s [${file}]`);
}

//Simple download to disk (pipe to stream)
await download(
  "http://ipv4.download.thinkbroadband.com/1GB.zip", 
  "D:/Downloads", 
  printProgress
);

//Download from github ... aws redirection ... content disposition ... but custom filename
const res = await download(
  "https://github.com/user/repo/releases/download/0.0.0/Setup.exe",
  "D:/Downloads/", 
  { filename: "supersetup.exe" }, 
  printProgress
);
console.log(res); 
/*Output:
{ status: 200, message: 'OK', headers: {...}, path: 'D:\\Downloads\\supersetup.exe' }
*/

//Download a list of files one by one
await request.download.all([
  "http://ipv4.download.thinkbroadband.com/5MB.zip",
  "http://ipv4.download.thinkbroadband.com/10MB.zip",
  "http://ipv4.download.thinkbroadband.com/20MB.zip",
  "http://ipv4.download.thinkbroadband.com/50MB.zip"],
  "D:\\Downloads", printProgress);

Download a torrent

import { download } from "@xan105/request/torrent";
download("https://webtorrent.io/torrents/sintel.torrent", "D:\\Downloads");

Misc

import * as h1 from "@xan105/request";

//Head request
const res = await h1.head(`http://ipv4.download.thinkbroadband.com/1GB.zip`);
console.log(res);
/*Output:
{ status: 200, message: 'OK', headers: {...} }
*/

//Manually specify retry on error and redirection to follow
await request("https://steamdb.info/app/220/", { maxRetry: 2, maxRedirect: 2 });

//Upload a single file multipart/form-data
const res = await h1.upload(
  "http://127.0.0.1/upload/test/",
  "Hello world", 
  {name: "file", filename: "hello world.txt"}
);
console.log(res);
/*Output:
{ status: 200, message: 'OK', headers: {...}, body: 'ok' }
*/

Installation

npm install @xan105/request

Optional packages

• webtorrent
  Downloading torrent
  

npm i webtorrent

• xml2js
  XML parser
  

npm i xml2js

API

⚠️ This module is only available as an ECMAScript module (ESM) starting with version 1.0.0.
Previous version(s) are CommonJS (CJS) with an ESM wrapper.

💡 The underlying API used is determined by which namespace you import.
By default this is the http/https (h1) API.
Torrent related are under the torrent namespace.

//Default
import * as h1 from '@xan105/request';

//http/https (h1)
import * as h1 from '@xan105/request/h1';

//http2 (h2)¹
import * as h2 from '@xan105/request/h2';

//Fetch¹
import * as fetch from '@xan105/request/fetch';

//Torrent
import * as torrent from "@xan105/request/torrent";

¹ Work in progress (unavailable at the moment)

Named export

request(href: string, payload?: any, option?: object): Promise<object>

This is the core request function every other functions are helper based on this one (except download, downloadAll and torrent).

The response object tries to be similar whether the request failed or succeeded.

{
  code: string, //HTTP or Node error code
  message: string, //HTTP or Node error message (if any)
  trace: string[], //URL(s) of the request (redirection)
  domain: string, //url domain
  sent: object, //Header sent
  address?: string, //IP address
  family?: string, //IPv4 or IPv6
  protocol?: string, //HTTP protocol (h1, h2, ...)
  security?: string, //TLS (HTTPS)
  port: number, //Network port
  headers?: object, //Response header
  body?: string //Response body
}

💡 In a dual stack network, IPv4 isn't prefered over IPv6 unlike Node's default behavior (Node < 17 ).

💡 When making a HEAD request:

• The promise always resolves no matter the HTTP response code.
• Doesn't follow redirection by design.
  If you need to follow the redirection you can use the headers location from the response and make a new HEAD request.

⚙️ Options

option	type	default	description
method	string	GET	HTTP method: get, post, head, etc
encoding	string	utf8	Response encoding
timeout	number	3000 (ms)	Time before aborting request
maxRedirect	number	3	How many redirections to follow before aborting. Use 0 to not follow redirects
maxRetry	number	0	How many retries on error before aborting. Use 0 to not retry at all
retryDelay	number	200 (ms)	How long to wait before a retry. Use 0 to instantly retry
headers	object	-> Chrome UA and UA Hint if https	Headers of your request
signal	AbortSignal	none	Abort signal

get(url: string, option?: object): Promise<object>

Force the GET method. Since request() default to 'GET' you could just use request() directly. This is here for completeness.

head(url: string, option?: object): Promise<object>

Force the HEAD method.

getJSON(url: string, option?: object): Promise<object>

Parse the response body as a JSON string and return the result.
Force method to GET and the header Accept to "application/json".

• alias: getJson()

postJSON(url: string, obj: object, option?: object): Promise<object>

Send given object payload as a JSON encoded string.
Parse the response body as a JSON string and return the result.
Force method to POST and the headers Accept and Content-Type to "application/json".

getXML(url: string, option?: object): Promise<object>

⚠️ Requires the xml2js module.

Parse the response body as a XML string and return the result.
Force method to GET and the header Accept to "application/xml".

• alias: getXml()

post(url: string, payload: unknown, option?: object): Promise<object>

Force method to POST and write/push given payload.
NB: On HTTP 301, 302, 303 redirection the method will be changed to GET

upload(url: string, payload: unknown, option?: object): Promise<object>

Force method to POST and write/push a multipart/form-data payload.
You can use option {fieldname: string, filename: string} to specify the form field name and the file name.
If you don't they will default respectively to 'file' and Date.now().

download(href: string, destDir: string, option?: object, callbackProgress?: fn): Promise<object>

Download file to destDir.

The response object is like request() minus body and with the addition of a file object:

{
  name: string, //filename
  path: string, //relative
  fullPath: string //absolute
}

This is useful for promise chaining to example unzip an archive, etc.

💡 Progress gives you the following stats: percent, speed, file.
callbackProgress(percent: number, speed: number, file: string)

⚙️ Options

option	type	default	description
timeout	number	3000 (ms)	Time before aborting request
maxRedirect	number	3	How many redirections to follow before aborting. Use 0 to not follow redirects
maxRetry	number	3	How many retries on error before aborting. Use 0 to not retry at all
retryDelay	number	1000 (ms)	How long to wait before a retry. Use 0 to instantly retry
headers	object	-> Chrome UA and UA Hint if https	Headers of your request
signal	AbortSignal	none	Abort signal
filename	string	null	Use this if you want to specify the filename (force rename)
hash	object	null	Verify checksum of downloaded file²

²Checksum option

{
  algo: string, //A Node.js supported crypto algo. eg: "sha1"
  sum: string //Checksum
}

On error or mismatch it will trigger error/retry.

downloadAll(href: string[], destDir: string|string[], option?: object, callbackProgress?: fn): Promise<object>

Download all the files in the list one-by-one to destDir.

If destDir is an array, files[i] will be written to destDir[i] in a 1:1 relation.
In the same fashion you can force the filename of the files with option {filename: [..,..,..]}.
And again same thing for checksum: {hash: [{algo: ..., sum: ...},..,..]}.

Returns an array of download() response object.

Torrent

download(torrent: string, dest: string, option?: object, callbackProgress?: fn): Promise<object>

⚠️ Requires the webtorrent module.

Download files from a torrent url, torrent file, torrent magnet to destDir.

💡 Progress gives you the following stats: percent, speed.
callbackProgress(percent: number, speed: number)

💡 Torrent can be resumed.

Returns an objectect with torrent download location, torrent name, and for every files of the torrent its name, relative path and path.

{
  path: string, //absolute
  name: string, //torrent name
  file: [
    {
      name: string, //filename
      path: string, //relative
      fullPath: string //absolute
    }
  ]
}

⚙️ Options

option	type	default	description
timeout	number	10 (sec)	Time to wait for peers before aborting
exclusion	string[]	none	Exclude files inside the torrent
downloadLimit	number	-1 (none)	Download speed limit
uploadLimit	number	100 (kb/s)	Upload speed limit