@ton.js/json-parser-rules

---

@ton.js/json-parser

A customizable JSON parser that is 100% compatible with native implementation (JSON.parse()), can be used as a polyfill, adds TC39 source access proposal and additional features, have better security like secure-json-parse and is… 25 times slower…

Rationale

The native implementation of JSON parser in JavaScript (i.e. JSON.parse()) doesn't allow to fully customize the parsing behavior. The JSON specification allows documents to include numbers of arbitrary and unlimited size. However, EcmaScript is using IEEE 754 standard to represent numbers internally and doesn't support numbers of the unlimited size. This leads to the data loss when JSON.parse() is attempting to read big numbers from the JSON documents.

Modern versions of JavaScript support a special BigInt data type specifically designed to represent integer numbers of unlimited size, but there is no way to tell JSON.parse() to use BigInt for parsing numbers.

Web 3.0 community tends to use very big numbers to represent cryptocurrency values, like the number of nanocoins in transaction. This library was designed as a workaround that allows to read big integer numbers from the JSON documents. However, considering that it's a full-fledged customized JSON parser you can use it for other cases as well.

Contents

• Features
• Normal usage (ponyfill)
  • Simple parsing
  • Using native reviver
  • Using reviver with source text access
  • Using key path
  • Throwing on prototypes
• Polyfill usage
  • Prerequisites
  • Simple parsing
  • Using native reviver
  • Using reviver with source text access
  • Using key path
• Parsing by rules
  • Prerequisites
  • Example
  • API
  • Rule format
• Benchmarks
  • Normal dataset
  • Big dataset
• Security
• Contributing
• Support

Features

• modern cross-platform multi-format package that can be used in any JavaScript environment,

• written in pure super-strict TypeScript with 100% type coverage,

• minimum size package with zero dependencies,

• robust security practices,

• 100% compatible with the JSON standard and native JSON.parse() implementation, can be used as a polyfill,

• future-compatible by implementing the Stage-3 TC39 source access proposal and additional features,

• adds special handling for __proto__ and constructor.prototype object properties to implement better security,

• extensively covered by unit tests and tested out on multiple real-life JSON samples,

• parses 1 MB of nested JSON in ~40 ms. (25 times slower than native implementation).

Normal usage (ponyfill)

Ponyfill is a side effects free package that provides an alternative implementation and doesn't affect the native behavior of the system.

Install the package first:

npm install --save @ton.js/json-parser

Please see the examples project for the complete examples.

Simple parsing

import { parseJson } from '@ton.js/json-parser';

interface DocumentType {
 // …
}

const object = parseJson<DocumentType>('{ … }');
// object type will be: DocumentType

Using native reviver

interface DocumentType {
  birthDate: Date;
}

const content = '{ "birthDate": "1989-08-16T10:20:30.123Z" }';

const object = parseJson<DocumentType>(content, (key, value) => (
  (key.endsWith('Date') ? new Date(value) : value)
));

assert(object.birthDate instanceof Date);

Using reviver with source text access

interface DocumentType {
  valueBN: bigint;
}

const content = '{ "valueBN": 12345678901234567890 }';

const object = parseJson<DocumentType>(content, (key, value, context) => (
  (key.endsWith('BN') ? BigInt(context.source) : value)
));

assert.equal(typeof object.valueBN, 'bigint');

Using key path

interface DocumentType {
  foo: {
    bar: {
      value: bigint;
    };
  };
}

const content = '{ "foo": { "bar": { "value": 12345678901234567890 } } }';

const object = parseJson<DocumentType>(content, (key, value, context) => (
  (context.keys.join('.') === 'foo.bar.value' ? BigInt(context.source) : value)
));

assert.equal(typeof object.foo.bar.value, 'bigint');

Throwing on prototypes

In order to prevent parsed object prototype override the JSON parser will automatically skip __proto__ and constructor.prototype properties. However, you can use the throwOnProto: true option to make this behavior more explicit — the parser will throw an error instead:

const content = '{ "foo": true, "__proto__": {} }';

try {
  parseJson(content, undefined, {
    throwOnProto: true,
  });

} catch (error: any) {
  // Forbidden object property name: "__proto__"
  console.log(error);

}

Polyfill usage

The polyfill implements the Stage-3 TC39 source access proposal and adds some additional useful features.

Polyfill is a package that when globally imported, overrides the behavior of the native JSON.parse() method. The polyfill will detect the number of arguments that your reviver function has and will use custom JSON parser implementation only when it has three parameters (i.e. context), in other case the native implementation will be used instead.

Please see the polyfill examples project for the complete examples.

Be advised, that polyfill can't detect if you are using arguments[2] to access the context, so make sure to use a normal function parameter.

Prerequisites

1). Install the polyfill package:

npm install --save @ton.js/json-parse-polyfill

2). Import the package only once as close to the beginning of your program as possible:

import '@ton.js/json-parse-polyfill';

(function main() {
  // …
})();

Simple parsing

interface DocumentType {
 // …
}

// Native implementation will be used
const object = <DocumentType> JSON.parse('{ … }');

Using native reviver

interface DocumentType {
  birthDate: Date;
}

const content = '{ "birthDate": "1989-08-16T10:20:30.123Z" }';

// Native implementation will be used
const object = <DocumentType> JSON.parse(content, (key, value) => (
  (key.endsWith('Date') ? new Date(value) : value)
));

assert(object.birthDate instanceof Date);

Using reviver with source text access

import type { ReviverFunc } from '@ton.js/json-parse-polyfill';

interface DocumentType {
  valueBN: bigint;
}

const content = '{ "valueBN": 12345678901234567890 }';

// Custom implementation will be used
const reviver: ReviverFunc = (key, value, context) => (
  (key.endsWith('BN') ? BigInt(context.source) : value)
);

const object = <DocumentType> (
  JSON.parse(content, reviver as any)
);

assert.equal(typeof object.valueBN, 'bigint');

Using key path

import type { ReviverFunc } from '@ton.js/json-parse-polyfill';

interface DocumentType {
  foo: {
    bar: {
      value: bigint;
    };
  };
}

const content = '{ "foo": { "bar": { "value": 12345678901234567890 } } }';

const reviver: ReviverFunc = (key, value, context) => (
  ((context.keys.join('.') === 'foo.bar.value')
    ? BigInt(context.source)
    : value
  )
);

// Custom implementation will be used
const object = <DocumentType> (
  JSON.parse(content, reviver as any)
);

assert.equal(typeof object.foo.bar.value, 'bigint');

Parsing by rules

A dedicated package is provided to support JSON deserialization using a special rules syntax. This simplifies writing custom revivers for your data types.

Prerequisites

This package requires a JSON.parse implementation that supports TC39 source access proposal and additional features. You can either install the polyfill or specify the parser function directly via the options.

Install the package:

npm install --save @ton.js/json-parser-rules

Example

Please see the parse by rules example project for the complete examples.

import '@ton.js/json-parse-polyfill';

import { strict as assert } from 'node:assert';

import type { Reviver } from '@ton.js/json-parser-rules';
import { parseJsonSchema } from '@ton.js/json-parser-rules';


//===============//
// JSON DOCUMENT //
//===============//

const content = (`
{
  "foo": {
    "hex": {
      "value": "416C6C20796F75206E656564206973206C6F7665"
    },
    "bar": {
      "baz": {
        "valueBig": 11145678901234567890,
        "hex": "466f72204169757221"
      }
    },
    "feeBig": 22245678901234567890,
    "myArray": [0, 1, "2023-01-20T19:30:45.904Z", 3, {
      "hex": {
        "value": "616c6c20796f75722062617365206172652062656c6f6e6720746f207573"
      }
    }]
  }
}
`);


//==================//
// RESULT INTERFACE //
//==================//

interface ParseResult {
  foo: {
    hex: HexBag;
    bar: {
      baz: {
        valueBig: bigint;
        hex: string;
      };
    };
    feeBig: bigint;
    myArray: [number, number, Date, number, {
      hex: HexBag;
    }];
  };
}

interface HexBag {
  value: string;
}


//==========//
// REVIVERS //
//==========//

const bigIntReviver: Reviver = (
  context => BigInt(context.source)
);

const dateReviver: Reviver = (
  context => new Date(context.value)
);

const hexReviver: Reviver = (
  context => Buffer.from(context.value, 'hex').toString()
);


//==================//
// PARSING BY RULES //
//==================//

const object = parseJsonByRules<ParseResult>(content, {
  rules: [{
    pattern: '**.*Big',
    reviver: bigIntReviver,
  }, {
    pattern: [
      '**.hex.value',
      'foo.bar.baz.hex',
    ],
    reviver: hexReviver,
  }, {
    pattern: 'foo.myArray.2',
    reviver: dateReviver,
  }],
});


//=================//
// TESTING RESULTS //
//=================//

assert.equal(typeof object.foo.bar.baz.valueBig, 'bigint');
assert.equal(object.foo.bar.baz.valueBig, 11145678901234567890n);

assert.equal(typeof object.foo.feeBig, 'bigint');
assert.equal(object.foo.feeBig, 22245678901234567890n);

assert.equal(
  object.foo.hex.value,
  'All you need is love'
);

assert.equal(
  object.foo.bar.baz.hex,
  'For Aiur!'
);

assert.equal(
  object.foo.myArray[4].hex.value,
  'all your base are belong to us'
);

assert.equal(
  object.foo.myArray[2].toISOString(),
  '2023-01-20T19:30:45.904Z'
);

If you don't want to use the polyfill you can do this instead:

import { parseJson } from '@ton.js/json-parser';
import { parseJsonByRules } from '@ton.js/json-parser-rules';

const result = parseJsonByRules('{ … }', {
  rules: [],
  parser: parseJson,
});

API

parseJsonByRules<Type>(source, options): Type;

Parses the specified JSON document according to the provided deserialization rules.

Property	Type	Description
* source	string	The JSON document to parse.
* options	Options	An options object.
* options.rules	ReviverRule[]	A list of parsing rules.
options.parser	ParserFunc	An optional JSON.parse compatible implementation.

ReviverRule

Specifies the reviver function that needs to be applied for the specified patterns.

Property	Type	Description
* pattern	string or string[]	A pattern or a list of patterns to match against parsing object keys.
* reviver	Reviver	A reviver function to apply when the pattern is getting matched.

Reviver = (context: ReviverContext) => any

Reviver is a callback function that is getting called to deserialize specific key-value pairs from the JSON document. It must return the deserialized value.

ReviverContext

Reviver function will return a context object with the following information that can be used to deserialize the document data.

Property	Type	Description
value	any	A parsed value.
source	string	An original source text of the value.
path	string	A path of keys delimited by dots.

Rule format

The following rule formats are supported:

Pattern	Description
foo.bar.baz	A direct key path that is getting matched exactly.
foo.*.bar	The * part will match anything except the dot, e.g.: foo.baz.bar, foo.qux.bar
foo.date*	This will match any keys under the foo that start with date, e.g. foo.dateCreated
foo.*Date	This will match any keys under the foo that end with Date, e.g. foo.expirationDate
**	This will apply to all the keys
**.foo	This will match all the foo keys nested in the object, e.g. my.deeply.nested.foo or foo
foo.bar.**	This will match every key under the foo.bar path, e.g. foo.bar.qux.quux
foo.*.bar.**.*Date	This will match all the keys ending with Date under the foo.X.bar, where X could be anything

Benchmarks

Normal dataset

name	ops	margin	percentSlower
native	12535	0.37	0
parse-json	460	0.66	96.33

Big dataset

name	ops	margin	percentSlower
native	29	1.23	0
parse-json	3	1.28	89.66

Security

Please see our security policy.

Contributing

Want to help? Please see the contributing guide.

Support

If you have any questions regarding this library or TON development in general — feel free to join our official TON development Telegram group.

The MIT License (MIT)

Copyright © 2023 TON FOUNDATION

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.