stream-generator
Make a readable stream out of any byte generator. An easy way to wrap a random, pseudo-random or just any number generator into a stream of bytes.
Setup
npm install stream-generator
Usage
TL;DR
const StreamGenerator = const byteStream = byteStream
Explanation and working examples
As already said Stream-generator can turn a byte generator into a readable stream.
But what exactly is byte generator?
A hard coded and simplistic byte generator can look like this:
{ // spits out bytes, which correspond to ascii chars: // 'M','a','r','s','h','a','l','l' 77 97 114 115 104 97 108 108}
It should basically yield integers in the range of 0-255, which can be considered bytes. It doesn't have to be a generator function in ES6 terms, it can be any function that returns an iterator on bytes. But generator functions are just a very natural fit: each call they return a brand new iterator in the same exact initial state, so they are natural iterator factories of sorts. We therefore can use such byte generator as constructor parameter for building a reproducible stream.
Let's do it:
const StreamGenerator = const byteStream = byteStream // output> Marshall
Same code as at the top of the page, but now it works, as we got a real byte generator. And because we have chosen a very particular set of bytes we see a nice output. In general that is not always the case as some byte values would be shown as garbage or not be shown at all in regular terminal window.
Deterministic streams
The stream is deterministic (i.e. each instance produces the same exact sequence of data) if underlying byte generator is deterministic. This leads us to a vast range of pseudo-random number generators available to choose from, but not all of them are deterministic and one has to keep that in mind when selecting.
Wrapping pseudo random number generators
OK, we have to feed stream generator with bytes. But many random and pseudo-random number modules can easily produce integers (or array of integers) but not bytes out of the box. Yes, I know, there is neither Integer, nor Byte type in javascript, we do only have Number type for everything. But what we are talking about here is values, not types.
Let's say we have a random number generator that produces integer values in the range of 0...(2^32-1) while we would like it to produce integer values in the range 0...255 which can be interpreted as bytes. The following simple technique can be used to achieve that:
// first we need some module with genuine deterministic// pseudo-random integer generator, e.g. like this oneconst Random = // we then wrap it into a form of ES6 'generator function' { const seed = 123 // hard coded seed for simplicity const mt = Randomengines mt while true } // wrap it further with this helper function: { return { for let int of // Extract each of the 4 bytes of the integer // and yield them one after another int & 0xff int >> 8 & 0xff int >> 16 & 0xff int >> 24 & 0xff }} // Put it all together:const byteGen = const byteStream =
We've just created an endless stream of pseudo-random bytes.
Big files generation
One interesting way of using such a deterministic stream is the ability to reproducibly create files of specified size and same exact content on the fly, which can be helpful when testing big files uploads etc.
Let's create one:
// we would need a way to cut the stream after certain// number of bytes, e.g. with this moduleconst StreamLimiter = const limiter = // cut after 10Mb // prepare destination file streamconst fs = const path = const filepath = pathconst dest = fs // take byteStream from previous listing and pipe itbyteStream dest
Fixed length streams: other way around
In the previous example we've cut the stream during piping. We can do it another way by cutting the sequence of bytes beforehand at the byte generation stage with this generator limiter function:
{ let count = 0 return { for let value of if ++count > size return value }}
And here is the full listing for big file generation with this approach:
// reproducibly create a file of specified size// filled with pseudo-random bytesconst StreamGenerator = const Random = { return { const mt = Randomengines mt while true }} { return { for let int of int & 0xff int >> 8 & 0xff int >> 16 & 0xff int >> 24 & 0xff }} { let count = 0 return { for let value of if ++count > size return value }} const fs = const path = const filepath = pathconst dest = fs const SEED = 123const intGen = const byteGen = const sizeInBytes = 10 * 1024 * 1024const byteGenLtd = const byteStreamLtd = byteStreamLtd dest
Dependencies
For "a stable streams base, regardless of what version of Node you are using" we use readable-stream standalone stream module instead of Node core implementation (read elaboration on this here).