A primitive token library used to read and write from a Uint8Array
.
Although it is possible to use this module directly, it is primarily designed to be used with strtok3 tokenizer.
Module: version 5 migrated from CommonJS to pure ECMAScript Module (ESM). JavaScript is compliant with ECMAScript 2019 (ES10).
npm install --save token-types
Usually in combination with strtok3:
npm install --save strtok3
Using TypeScript you should also install @tokenizer/token as a development dependency:
npm install --save-dev @tokenizer/token
import * as strtok3 from 'strtok3';
import * as token from 'token-types';
(async () => {
const tokenizer = await strtok3.fromFile("somefile.bin");
try {
const myNumber = await tokenizer.readToken(token.Float32_BE);
console.log(`My number: ${myNumber}`);
} finally {
tokenizer.close(); // Close the file
}
})();
token-types
supports a wide variety of numeric tokens out of the box:
Token | Number | Bits | Endianness |
---|---|---|---|
UINT8 |
Unsigned integer | 8 | n/a |
UINT16_BE |
Unsigned integer | 16 | big endian |
UINT16_LE |
Unsigned integer | 16 | little endian |
UINT24_BE |
Unsigned integer | 24 | big endian |
UINT24_LE |
Unsigned integer | 24 | little endian |
UINT32_BE |
Unsigned integer | 32 | big endian |
UINT32_LE |
Unsigned integer | 32 | little endian |
UINT64_BE |
Unsigned integer | 64 | big endian |
UINT64_LE * |
Unsigned integer | 64 | little endian |
INT8 |
Signed integer | 8 | n/a |
INT16_BE |
Signed integer | 16 | big endian |
INT16_LE |
Signed integer | 16 | little endian |
INT24_BE |
Signed integer | 24 | big endian |
INT24_LE |
Signed integer | 24 | little endian |
INT32_BE |
Signed integer | 32 | big endian |
INT32_LE |
Signed integer | 32 | little endian |
INT64_BE |
Signed integer | 64 | big endian |
INT64_LE * |
Signed integer | 64 | little endian |
Float16_BE |
IEEE 754 float | 16 | big endian |
Float16_LE |
IEEE 754 float | 16 | little endian |
Float32_BE |
IEEE 754 float | 32 | big endian |
Float32_LE |
IEEE 754 float | 32 | little endian |
Float64_BE |
IEEE 754 float | 64 | big endian |
Float64_LE |
IEEE 754 float | 64 | little endian |
Float80_BE * |
IEEE 754 float | 80 | big endian |
Float80_LE * |
IEEE 754 float | 80 | little endian |
(*) The tokens exceed the JavaScript IEEE 754 64-bit Floating Point precision, decoding and encoding is best effort based.
StringType decoding is implemented using TextDecoder which supports a large number of encodings including but not limited to:
- UTF-8 (the default)
- Windows-1252
- ISO-8859-1
Check out the MDN web docs for the TextDecoder for a complete list
Custom tokens can be added, suitable for reading binary files or network messages:
ExtendedHeader = {
len: 10,
get: (buf, off) => {
return {
// Extended header size
size: Token.UINT32_BE.get(buf, off),
// Extended Flags
extendedFlags: Token.UINT16_BE.get(buf, off + 4),
// Size of padding
sizeOfPadding: Token.UINT32_BE.get(buf, off + 6),
// CRC data present
crcDataPresent: common.strtokBITSET.get(buf, off + 4, 31)
};
}
};