node-clusterwork
Process arbitrary incoming data on multiple worker processes. This is a helper module that helps setup node's cluster feature for tasks other than network servers.
Why?
The built in cluster module automatically spreads the requests among workers when the main usage is for a nodejs server. BUT there might be other scenarios where a server is not involved, rather work from any arbitrary source is to be distributed among workers for parallel processing on all CPU cores. This modules makes it easy to setup a worker cluster for such scenarios.
Installation
npm install clusterwork --save
Quick Start
const clusterwork = ; { ;} { console;} clusterwork;
When the above script is run, apart from the main (master) process, multiple worker processes, (1 per each CPU core), are automatically spawned.
Master process runs the producer function that can use dispatch()
to send data to worker processes.
The consumer function is invoked in a worker process to handle/process the dispatched data. Workers are selected in a cyclic fashion to evenly distribute processing load.
The above example yields:
master process 14530 sending 0.5805814887272316worker process 14541 got 0.5805814887272316master process 14530 sending 0.3958888451685434worker process 14542 got 0.3958888451685434master process 14530 sending 0.1878265613269785worker process 14547 got 0.1878265613269785master process 14530 sending 0.2966814774610549worker process 14536 got 0.2966814774610549
Usage
The main script should call the init()
function:
clusterwork;
The parameters are explained as follows:
-
producer
- Function called once in master process and is supposed to setup data processing. It receives adispatch
function that can be used to send data object to workers. -
consumer
- Function called in worker process to handle data object. -
respawn
- Optional boolean flag that auto spawns a new worker, if an existing worker terminates somehow.
Notes
- If all worker processes are terminated, the main process throws an error while trying to dispatch data.
Licence
The source code is published under MIT licence.