highland-find-json

0.0.1 • Public • Published

highland-find-json

highland-find-json is highland-based library for locating JSON objects in a stream of buffers.

The primary use-case for this library is for dealing with very large (or infinite) streams containing many relatively-small JSON objects, such as JSON-based logging.

It expects as input a stream of buffers, such as would occur if receiving data from a TCP socket. It emits another stream of buffers, but in the output each buffer exactly frames a JSON object from the input stream.

For example, imagine the following buffers show up in an HTTP response:

  • [{"foo":"
  • bar"},{"baz
  • ":4}]

highland-find-json would consume the above and produce a stream with two buffers as follows:

  • {"foo":"bar"}
  • {"baz":4}

This library does not actually parse the JSON; instead, it counts open and close braces so it can find object boundaries with minimal overhead. As long as its input is valid JSON, each of the buffers it emits will contain a valid JSON object.

Usage

var fs = require('fs');
var _ = require('highland');
var findJson = require('highland-find-json');
 
var rawData = fs.createReadStream('some.json');
 
_(rawData).consume(
    findJson()
).map(
    function (buf) {
        return JSON.parse(buf);
    }
).each(
    function (obj) {
        // ... do whatever you want with obj...
        console.log(obj.someProperty);
    }
);

Installation

As usual:

npm install --save highland-find-json

License

Copyright 2016 Say Media, Inc.

This library may be distributed under the terms of the MIT license. For full details, see LICENSE.

Dependencies (1)

Dev Dependencies (2)

Package Sidebar

Install

npm i highland-find-json

Weekly Downloads

1

Version

0.0.1

License

MIT

Last publish

Collaborators

  • apparentlymart