orkid
Reliable and modern Redis-Streams based task queue for Node.js.
Screenshot
Table of Contents
- Features
- Requirements
- Install
- Examples
- API Documentation
- Monitoring and Management UI/Admin Panel
- FAQ
Features
- Orkid lets Redis do the heavy lifting with Redis-Streams.
- Adjustable concurrency per consumer instance for scaling task processing. See example code. See example code.
- Job timeouts and retries. All configurable per consumer. See example code.
- Task Deduplication. If a task is already waiting in the queue, it can be configured to avoid queueing the same task again. (Useful for operations like posting database record updates to elasticsearch for re-indexing. Deduplication is a common pattern here to avoid unnecessary updates). See example code.
- Add tasks in bulk in one call. The producer will handle chunking and optimize round-trips to redis using pipelining.
- Monitoring and management UI for better visibility.
- Rate-limiting workers. (work in progress)
Requirements
- Node.js >= 10
- Redis >= 5
👏 Important: Redis-Streams feature is not available before Redis version 5.
Install
npm install orkid --save
Examples
Basic example of producing and consuming tasks:
Producing tasks:
const Producer = ; // `basic` is the queue name here// We'll use the same name in the consumer to process this taskconst producer = 'basic'; { for let i = 0; i < 10; i++ console; await producer; } ;
Consuming tasks:
const Consumer = ; // Worker Function { let result; /* Do operation on `data` here and store the result in `result` variable Anything you return from this function will be saved in redis and can be viewed in the Orkid UI. Returning nothing is fine too. Throwing error will mark the job as failed, which can be retried too. */ console; return result;} // Consume task from the `basic` queueconst consumer = 'basic' workerFn; // Start processing tasks!// Important: Until you call this method, orkid consumer will do nothing.consumerstart;
👏 More examples are available in the ./examples directory, including how to do task de-duplication, retry on failure, timeout etc. 👏
API Documentation
API Documentation is available here.
Monitoring and Management UI/Admin Panel
You need to run
orkid-ui
separately for the dashboard. Detail instructions on how to runorkid-ui
locally or in production using docker/docker-compose can be found here: https://github.com/mugli/orkid-ui#running-locally
Task/job life-cycle
[TODO: Add a flowchart here]
FAQ
Is this production ready?
This project is under active development right now. API WILL introduce breaking changes until we reach version 1.0. After that semantic versioning will be followed.Why a new job queue for Node.js?
- All the redis-based solutions were created before Redis-Streams (https://redis.io/topics/streams-intro) became available. They all require a lot of work on the queue-side to ensure durability and atomicity of jobs handling operations. Redis-Streams was specifically designed to made this kind of tasks easier, thus allows simpler core in the queue and more reliable and maintainable operations.- None of existing usable job queues in Node.js offers a monitoring option that we liked.
- None of the existing usable task queues support task de-duplication.
How do I set priority in the tasks?
Redis-Streams isn't a right primitive to make a priority queue efficiently on top of it. Orkid doesn't support priority queues now and probably never will.However, as a workaround, you can create a separate queue, keep its workload minimal and use it for high priority jobs with Orkid.
What is the order of job/task delivery?
Jobs are processed in the order they are produced. However, if retry option is turned on and is applicable, failed jobs gets enqueued to the queue at once, along with other newly produced jobs.Maintainer(s)
- Mehdi Hasan Khan (@MehdiHK)
License
MIT