@e2fyi/streams
This NodeJS library provides custom NodeJS streams for specific use cases.
Currently, the following streams are available:
-
DocumentTagger
: A Transform stream to tag an auto-increment field to each object in the stream. Can also mutate the stream objects through a function or default Object. -
MongooseStream
: A Transform stream which will bulk write the objects in the stream to mongodb via a mongoose model.
ChangeLog
- v1.0.0: new streams:
DocumentTagger
,MongooseStream
.
Quick start
Using CommonJS module
// importing DocumentTagger and MongooseStream
const {DocumentTagger, MongooseStream} = require('@e2fyi/streams');
API reference
The API documentation is also available at https://e2fyi.github.io/streams.
@e2fyi/streams
-
@e2fyi/streams
-
.DocumentTagger ⇐
stream.Transform
-
.MongooseStream ⇐
stream.Transform
-
.DocumentTaggerSettings :
Object
-
.MongooseStreamSettings :
Object
-
.DocumentTagger ⇐
stream.Transform
@e2fyi/streams.DocumentTagger ⇐ Transform Object stream (objectMode=true) to tag with an autoIncrement id. An object or function can be optionally provided to mutate each object in the stream.
Kind: static class of @e2fyi/streams
Extends: stream.Transform
Emits: filtered
new DocumentTagger(opts)
Create a new DocumentTagger stream.
Param | Type | Description |
---|---|---|
opts | DocumentTaggerSettings |
Settings for the stream. |
Example
const docTagger = new DocumentTagger({autoIncrement: 'id', mutate: { project: 'test' }});
someReadableStreamFromArray([{text: 'abc'}, {text: 'efg'}])
.pipe(docTagger)
.pipe(process.stdout);
// stdout >
// {"text": "abc", "id": 0, "project": "test"}
// {"text": "efg", "id": 1, "project": "test"}
stream.Transform
@e2fyi/streams.MongooseStream ⇐ A custom NodeJS Transform stream to mongo via mongoose.
Kind: static class of @e2fyi/streams
Extends: stream.Transform
Emits: mongoose-bulk-write
new MongooseStream(opts)
Create a Transform stream which bulkWrite to mongo based on the itemWaterMark. model (mongoose Model) is a required field.
Param | Type | Description |
---|---|---|
opts | MongooseStreamSettings |
Configuration for MongoStream. Default value for itemWaterMark is 50. |
Example
var stream2mongo = new MongooseStream({mode: SomeMongooseModel});
someReadableStreamFromArray([{text: 'abc'}, {text: 'efg'}])
.pipe(stream2mongo) // writes to mongo (while stream are also passthrough)
.pipe(response); // stream same results back to some request
Object
@e2fyi/streams.DocumentTaggerSettings : Settings for DocumentTagger.
Kind: static typedef of @e2fyi/streams
Properties
Name | Type | Description |
---|---|---|
autoIncrement | String |
The auto-increment field to tag onto the stream object. |
ignoreNonObject | Boolean |
If true , no errors will be emitted when the chunk in the stream cannot be parsed into Object . |
readableObjectMode | Boolean |
If true , Object will be emitted from the stream, otherwise a String or Buffer representation or will emitted instead. |
objectMode | Boolean |
If true , Object will be emitted from the stream, otherwise a String or Buffer representation or will emitted instead. Overwrites writableObjectMode . |
filter | function |
Function to filter the objects in the stream. Return true to keep the object. |
mutate |
function | Object
|
Function or Object to mutate the stream. If an Object is provided, each stream object will be mutated with the Object.assign(streamObj, mutateObj) . |
Object
@e2fyi/streams.MongooseStreamSettings : Settings for MongooseStream.
Kind: static typedef of @e2fyi/streams
Properties
Name | Type | Default | Description |
---|---|---|---|
itemWaterMark | Number |
50 |
The number of item collected before writing to mongodb. |
passThrough | Boolean |
If false nothing will be emitted from the stream. |
|
model | mongoose.Model |
mongoose Model. |
Development
Unit testing
npm test
Build documentation
npm run build