Knetik Micro Queue

This package adds redis queue support using Bull Queue to any micro-core application as a multi tenant adaptor.

When a customer connects to a micro-core application, the required app_id is passed into the adaptor initializer and used as the context for any jobs created using the micro queue Job class interface.

Setup

Install the @knetik/micro-queue package

yarn add @knetik/micro-queue

Add the Adaptor to your config/environments/{env}.json

"ADAPTORS": [
  "@knetik/micro-queue"
]

Add the REDIS_HOST param as well

"REDIS_HOST": "127.0.0.1"

Job Generator

$ yarn run micro-queue generate {job_name}

A file is generated in app/jobs/{job_name}.job.js

if the {job_name} id some_long_process, the contents of app/jobs/{job_name}.job.js are:

const { JobBase } = require('@knetik/micro-queue');

module.exports = class SomeLongProcessJob extends JobBase {
  static perform(App, params, progress) {
    App.Logger.info('performing SomeLongProcessJob');

    /* ====== EXAMPLE USAGE ========= */
    App.Logger.info('SomeLongProcessJob', params);

    let pending = 100;
    let total = 0;

    // Jobs must return a Promise
    return new Promise((resolve, reject) => {
      const interval = setInterval(() => {
        pending -= 1;
        total += 1;

        App.Logger.info('SomeLongProcessJob progress', pending);

        // use progress to increment the jobs progress status
        progress(total);

        if (!pending) {
          resolve('done');
          clearInterval(interval);
        }
      }, 1000);
    });
    /* ====== END EXAMPLE USAGE ========= */
  }
};

// Set the max concurrency value for this job.
module.exports.concurrency = 50;

Usage

The Adaptor is added to the Micro Core application at App.Queue. Use App.Queue#get to load job classes form the App's initialized jobs.

const Job = App.Queue.get('SomeLongProcessJob');

Call perform_later on the Job class passing in the job data and the job options

const data = { hi: 'there' };
const options = {};

Job.perform_later(data, options);

Queue UI

if the Micro Express Adaptor is installed the Queue management UI is mounted at /arena. Visit it in the browser for handy job management.

Documentation

More info is available in the DOCS