bucket-maker

0.1.3 • Public • Published

bucket-maker

To reduce I/O cost, bucket-maker makes bucket of sliced timline for each unit you requested.
Name of the bucket is defined by its day and slot.

  • day is a number of days from unix epoch
  • slot is a number of unit from 00:00
  • unit is defined by you

This is calculated like as follows:

unit = 5000;                                                // msec
days = Date.now() / (60*60*24*1000);                        // 17764.309366608795
day  = Math.floor(days);                                    // 17764
slot = Math.ceil( (days - day) * (60*60*24*1000) / unit );  // 5346

Demo

$ git clone https://github.com/RayKitajima/Bucket-Maker.git
$ cd Bucket-Maker

$ node demo/timestamp.js

demo/timestamp.js write out timestamp every sec into the bucket.
The bucket is made for every five sec.
So the bucket contains five entries in it.

At a grance

// get 5sec(msec) bucket maker
let bucket = require('bucket-maker').createBucket({
    unit: 5000
});

// put timestamp every sec
setInterval(function(){
    bucket.put(Date.now());
},1000);

// will write out
$ ls 
17764.4761.bucket  17764.4762.bucket

// will contain
$ cat 17764.4762.bucket
1534833405881
1534833406883
1534833407888
1534833408893
1534833409897

Options

Internally bucket is an array. And it will periodically dump and write them by dumper and writer.

name what is
prefix prefix for output bucket name
suffix suffix for output bucket name. default <suffix> is '.bucket'
unit bucket unit as msec
spool where to write out bucket file
dumper how to dump elements in the bucket.
by default assume them as stirng and join() them
writer how to write out dumped object.
by default write out to file named as <prefix><day>.<slot><suffix>.
start whether to start writing at created
cron_time frequency of checking bucket
log how to system log

Practice

const pid  = process.pid;
const host = os.hostname();
const prefix = pid + '.' + host + '.';

let bucket = require('bucket-maker').createBucket({
    prefix : prefix,
    suffix : '.bucket',
    unit   : 5000,
    spool  : '/path/to/buckets',
    dumper : function(items){ return items.map( x => JSON.stringify(x) ).join('\n') },
});

setInterval(function(){
    bucket.put(os.loadavg());
},1000);

Reporting load avarage every sec. And make 5sec bucket.

$ ls 
4825.localhost.17764.4761.bucket  4825.localhost.17764.4762.bucket

More reducing I/O cost with redis

const client = require('redis').createClient();

const writer = function(maker){
    let names = Object.keys(maker.Buckets);
    let now = Date.now();
    for( let i=0; i<names.length; i++ ){
        if( maker.Buckets[names[i]].cutoff > now ){ continue; }   // skip: still putting into the bucket
        let dump = maker.dumper(maker.Buckets[names[i]].buffer);
        let key  = name[i];                                       // wait:17764.4762.bucket
        client.set(key,dump,function(err,res){
            if( err ){
                maker.log(err);
            }else{
                delete maker.Buckets[names[i]];
            }
        });
    }
};

let bucket = require('bucket-maker').createBucket({
    prefix : 'wait:',
    unit   : 5000,
    dumper : function(items){ return items.map( x => JSON.stringify(x) ).join('\n') },
    writer : writer,
});

setInterval(function(){
    bucket.put(os.loadavg());
},1000);

This writes out loadavg report into redis.

const client = require('redis').createClient();

const drain = function(){
    client.keys('wait:*',function(err,keys){
        for( let i=0; i<keys.length; i++ ){
            client.get(keys[i],function(err,content){
                let bucket_name = keys[i].split(':')[1]; // 17764.4762.bucket
                fs.writeFileSync(bucket_name,content);
                client.del(keys[i]);
            });
        }
    });
};

setInterval(drain,5000);

This drains bucket to the file system.

If you would like to implement your own writer, you have to know about internal code, and should write several idiom to define what to write out.

License

MIT

Package Sidebar

Install

npm i bucket-maker

Weekly Downloads

0

Version

0.1.3

License

MIT

Unpacked Size

8.72 kB

Total Files

6

Last publish

Collaborators

  • raykitajima