bull vs agenda vs bee-queue vs kue vs node-resque vs bree
Node.js Job Queues and Task Scheduling
搜索包...
bullagendabee-queuekuenode-resquebree类似的npm包:

Node.js Job Queues and Task Scheduling

These libraries enable Node.js applications to handle background tasks, scheduled jobs, and heavy processing without blocking the main event loop. They provide structures for defining work, storing job state, and retrying failed tasks. While some rely on Redis for fast in-memory storage, others use MongoDB for durability or worker threads for local execution. Choosing the right tool depends on your infrastructure, reliability needs, and whether you require distributed processing or simple local scheduling.

npm下载趋势

3 年

GitHub Stars 排名

统计详情

npm包名称
下载量
Stars
大小
Issues
发布时间
License
bull2,245,94716,262309 kB1472 年前MIT
agenda228,6899,712310 kB1910 小时前MIT
bee-queue67,7854,039107 kB461 年前MIT
kue31,0799,418-2869 年前MIT
node-resque26,4421,415720 kB243 个月前Apache-2.0
bree03,29692 kB308 个月前MIT

Node.js Job Queues: Architecture and Implementation Compared

Handling background tasks is a common requirement in modern web applications. Whether you are sending emails, processing images, or running scheduled maintenance, you need a reliable way to manage work without blocking your main server. The packages agenda, bee-queue, bree, bull, kue, and node-resque all solve this problem, but they use different storage engines and architectural patterns. Let's compare how they handle job definition, execution, and storage.

🗄️ Storage Engine: Redis vs MongoDB vs Worker Threads

The underlying storage determines how jobs survive server restarts and how multiple servers share work.

bull and bee-queue rely on Redis.

  • Redis is fast and in-memory, making these queues great for high throughput.
  • Jobs are lost if Redis persists incorrectly, though both support persistence settings.
// bull: Redis connection
const Queue = require('bull');
const queue = new Queue('email', 'redis://127.0.0.1:6379');

// bee-queue: Redis connection
const Queue = require('bee-queue');
const queue = new Queue('email', { redis: { host: '127.0.0.1', port: 6379 } });

agenda uses MongoDB.

  • Jobs are stored as documents, so they survive restarts easily.
  • You can query jobs using standard Mongo queries.
// agenda: MongoDB connection
const Agenda = require('agenda');
const agenda = new Agenda({ db: { address: 'mongodb://127.0.0.1:27017/agenda' } });

bree uses Worker Threads locally.

  • It does not require an external database by default.
  • Jobs are defined as files and run in separate threads within the same process.
// bree: Local configuration
const Bree = require('bree');
const bree = new Bree({
  jobs: [
    { name: 'daily-cleanup', interval: '0 0 * * *', path: './jobs/cleanup.js' }
  ]
});

kue and node-resque also use Redis.

  • They follow older patterns of Redis list management.
  • kue is no longer maintained, so new projects should avoid it.
// kue: Redis connection (Legacy)
const kue = require('kue');
const queue = kue.createQueue();

// node-resque: Redis connection
const { Queue, Worker } = require('node-resque');
const queue = new Queue({ connection: { host: '127.0.0.1', port: 6379 } });

📥 Defining and Processing Jobs

Each library has a different way to define what work needs to be done and how to run it.

bull separates job addition from processing.

  • You define a processor function that runs when a job is available.
  • It supports concurrency limits directly in the processor.
// bull: Process jobs
queue.process('send-email', async (job) => {
  await sendEmail(job.data.to);
});

// bull: Add job
queue.add('send-email', { to: '[email protected]' });

agenda defines jobs by name and schedules them.

  • You define the logic once and can trigger it immediately or later.
  • It feels like a database collection of tasks.
// agenda: Define job
agenda.define('send-email', async (job) => {
  await sendEmail(job.attrs.data.to);
});

// agenda: Schedule job
agenda.now('send-email', { to: '[email protected]' });

bee-queue is similar to Bull but with a simpler API.

  • It focuses on minimal configuration and speed.
  • The processing model is straightforward.
// bee-queue: Process jobs
queue.process(async (job) => {
  await sendEmail(job.data.to);
});

// bee-queue: Create job
const job = queue.createJob({ to: '[email protected]' });
await job.save();

bree uses file-based jobs.

  • You write a script for each job type.
  • Bree manages the timing and execution of these scripts.
// bree: Job file (jobs/cleanup.js)
module.exports = async (config) => {
  await cleanupDatabase();
};

// bree: Start manager
await bree.start();

kue uses a creation chain pattern.

  • You build the job object step-by-step before saving.
  • This is verbose compared to modern alternatives.
// kue: Create job (Legacy)
const job = queue.create('send-email', { to: '[email protected]' });
job.save();

// kue: Process
queue.process('send-email', (job, done) => {
  sendEmail(job.data.to, done);
});

node-resque follows the classic Resque model.

  • You perform jobs based on class names.
  • It requires a separate worker process to listen for jobs.
// node-resque: Perform method
class SendEmail {
  async perform(to) {
    await sendEmail(to);
  }
}

// node-resque: Enqueue
await queue.enqueue('high', 'SendEmail', ['[email protected]']);

🕒 Scheduling and Recurrence

Some libraries handle cron jobs and delayed tasks better than others.

agenda excels at scheduling.

  • It has built-in support for every, schedule, and cron patterns.
  • You can query scheduled jobs easily via MongoDB.
// agenda: Recurring job
agenda.every('10 minutes', 'clear-cache');
agenda.schedule('in 2 hours', 'send-reminder', { userId: 123 });

bull supports delays and repeats.

  • You can set a job to run after a delay or on a cron schedule.
  • It relies on Redis timers for this functionality.
// bull: Delayed job
queue.add('notify', { id: 1 }, { delay: 5000 });

// bull: Repeated job
queue.add('report', {}, { repeat: { cron: '0 0 * * *' } });

bree is built for cron-style jobs.

  • It uses human-readable intervals or cron strings in config.
  • It is designed specifically for time-based execution.
// bree: Cron config
const bree = new Bree({
  jobs: [
    { name: 'backup', cron: '0 0 * * *', path: './jobs/backup.js' }
  ]
});

bee-queue, kue, and node-resque have limited scheduling.

  • They focus on immediate queue processing.
  • You often need an external scheduler (like node-cron) to add jobs at specific times.
// bee-queue: Delay only
const job = queue.createJob({}).delay(5000);
await job.save();

// kue: Delay only
queue.create('task', {}).delay(5000).save();

⚠️ Maintenance and Status

Choosing a library also means choosing its long-term support.

bull and agenda are actively maintained.

  • They receive regular updates and security patches.
  • They have large communities and many tutorials available.

bree is maintained and focused on simplicity.

  • It is a good choice for newer projects that want less infrastructure.
  • It avoids external dependencies like Redis.

bee-queue has slower maintenance cycles.

  • It is stable but updates are less frequent.
  • Suitable for stable projects that do not need cutting-edge features.

kue is deprecated.

  • It should not be used in new development.
  • Security vulnerabilities may not be fixed.

node-resque is stable but legacy.

  • It is best for maintaining older systems.
  • New projects might find bull more aligned with modern Node.js practices.

📊 Summary: Key Differences

Featurebullagendabreebee-queuekuenode-resque
StorageRedisMongoDBLocal/ThreadsRedisRedisRedis
SchedulingBuilt-inBuilt-inBuilt-inLimitedLimitedLimited
StatusActiveActiveActiveStableDeprecatedStable
ComplexityMediumMediumLowLowHighMedium
Best ForDistributedPersistenceCron/LocalSimplicityLegacyRuby Migration

💡 The Big Picture

bull is the safe bet for most distributed Node.js applications. It balances features, performance, and community support. If you need heavy scheduling and already use MongoDB, agenda is a strong contender. For simple cron jobs on a single server, bree removes the need for Redis entirely.

Avoid kue in new projects due to its deprecated status. Use node-resque only if you need compatibility with existing Resque infrastructure. For lightweight Redis needs where Bull feels too heavy, bee-queue remains an option, but verify its maintenance fits your risk tolerance.

Final Thought: The best queue is the one that matches your infrastructure. If you have Redis, use Bull. If you have Mongo, use Agenda. If you have neither, use Bree.

如何选择: bull vs agenda vs bee-queue vs kue vs node-resque vs bree

  • bull:

    Choose bull if you need a mature, feature-rich queue system backed by Redis. It supports priorities, retries, delays, and event listeners out of the box. It is the standard choice for distributed systems requiring high reliability and concurrency, though you should also evaluate BullMQ for modern Redis Stream support.

  • agenda:

    Choose agenda if your stack already uses MongoDB and you need robust scheduling features like cron jobs or recurring tasks. It stores job data in Mongo collections, making it easy to query and inspect jobs using standard database tools. It is well-suited for applications that prioritize data persistence over raw throughput.

  • bee-queue:

    Choose bee-queue if you need a lightweight, Redis-based queue with a focus on simplicity and performance. It has a smaller footprint than Bull and avoids complex Lua scripts. It is a good fit for projects that want Redis storage without the heavier feature set of Bull, though maintenance activity is slower.

  • kue:

    Do not choose kue for new projects. It is deprecated and no longer maintained. While it was popular historically for its Redis backend and UI, it lacks modern security updates and bug fixes. Existing projects should plan migration to bull or agenda.

  • node-resque:

    Choose node-resque if you are migrating from a Ruby on Rails Resque background job system and want to maintain compatibility. It follows the classic Resque pattern strictly. It is stable but less feature-rich than Bull for modern Node.js specific workflows.

  • bree:

    Choose bree if you need to run cron jobs or scheduled tasks using worker threads within a single Node.js process. It is designed for simplicity and does not require Redis or MongoDB. It is ideal for single-server deployments where you want to isolate jobs in separate threads to avoid blocking the main loop.

与bull类似的npm包

bull is a robust queue library for Node.js, designed to handle distributed jobs and messages in a scalable manner. It is built on top of Redis and provides a simple API for creating and managing job queues. Bull is particularly useful for applications that require background processing, such as sending emails, processing images, or handling other time-consuming tasks. With features like job retries, delayed jobs, and concurrency control, Bull makes it easy to manage complex workflows efficiently.

While Bull is a popular choice for job and queue management, there are several alternatives that cater to different needs and preferences:

  • agenda is a lightweight job scheduling library for Node.js that uses MongoDB as its backend. It allows you to schedule jobs to run at specific times or intervals, making it suitable for applications that require cron-like functionality. Agenda is easy to set up and provides a simple API for defining and managing jobs, making it a good choice for applications that already use MongoDB and need a straightforward scheduling solution.

  • bee-queue is another lightweight queue library for Node.js that focuses on simplicity and performance. It is designed for handling jobs in a single Redis instance and provides a minimalistic API for creating and processing jobs. Bee-Queue is particularly well-suited for applications that require fast job processing with low overhead, making it a great alternative for developers looking for a straightforward queuing solution.

  • bullmq is the successor to Bull, offering a more modern and feature-rich approach to job and queue management. It introduces a new architecture and API, providing improved performance and additional features such as rate limiting, job prioritization, and more. If you are starting a new project or looking to upgrade from Bull, BullMQ is the recommended choice for its enhanced capabilities and flexibility.

  • kue is a priority job queue backed by Redis, designed for handling distributed jobs in Node.js applications. It provides a rich set of features, including job prioritization, delayed jobs, and a user interface for monitoring job progress. Kue is suitable for applications that require a more comprehensive job management solution with a focus on monitoring and visualization.

  • node-resque is a Redis-backed library for creating background jobs in Node.js. It supports multiple backends and provides a flexible API for defining and managing jobs. Node-resque is ideal for applications that need a customizable job processing solution and want to leverage the power of Redis for job management.

To explore how these libraries compare, check out the following link: Comparing agenda vs bee-queue vs bull vs bullmq vs kue vs node-resque.

与agenda类似的npm包

agenda is a job scheduling library for Node.js that allows developers to manage and execute background jobs easily. It is built on top of MongoDB, enabling persistent job storage and providing a flexible API for scheduling tasks. With agenda, you can define recurring jobs, manage job priorities, and handle job failures gracefully. This makes it a great choice for applications that require reliable job scheduling and processing.

While agenda is a powerful tool, there are several alternatives available in the Node.js ecosystem that also provide job scheduling capabilities. Here are a couple of notable alternatives:

  • bull is a robust queue and job scheduling library for Node.js that is built on Redis. It offers a rich set of features, including delayed jobs, job retries, and concurrency control. Bull is particularly well-suited for applications that need to handle a high volume of jobs or require advanced job management capabilities. Its performance and reliability make it a popular choice for developers looking to implement job queues in their applications.

  • node-schedule is a flexible job scheduling library for Node.js that allows you to schedule jobs using cron-like syntax. It is lightweight and easy to use, making it ideal for applications that require simple scheduling without the overhead of a persistent job store. Node-schedule is perfect for scenarios where you need to run tasks at specific intervals or times without the need for complex job management features.

To see how agenda compares with bull and node-schedule, check out the comparison: Comparing agenda vs bull vs node-schedule.

与bee-queue类似的npm包

bee-queue is a simple and efficient queue library for Node.js, designed for handling background jobs and tasks. It is built on top of Redis and provides a straightforward API for creating and managing job queues. With its focus on performance and simplicity, bee-queue is particularly well-suited for applications that require a lightweight solution for processing jobs asynchronously.

While bee-queue is a great option, there are several alternatives available in the Node.js ecosystem that offer different features and capabilities:

  • agenda is a job scheduling library for Node.js that uses MongoDB as its backend. It allows you to define jobs and schedule them to run at specific times or intervals. agenda is particularly useful for applications that require recurring tasks or scheduled jobs, making it a good choice for cron-like functionality.
  • bull is a robust queue library for Node.js that also utilizes Redis. It offers advanced features such as job prioritization, delayed jobs, and rate limiting. bull is designed for high-performance applications and is suitable for handling large volumes of jobs efficiently. If your application requires more complex queue management capabilities, bull is an excellent alternative.
  • kue is another Redis-based job queue library for Node.js. It provides a rich set of features, including job retries, event listeners, and a user-friendly UI for managing jobs. While kue is powerful, it is worth noting that it is no longer actively maintained, so developers should consider this when choosing a queue library for new projects.
  • p-queue is a promise-based queue library that allows you to manage asynchronous tasks with ease. It provides features such as concurrency control and priority support, making it suitable for scenarios where you need to limit the number of concurrent operations. p-queue is a good choice for applications that require fine-grained control over task execution.

For a comprehensive comparison of these libraries, check out the following link: Comparing agenda vs bee-queue vs bull vs kue vs p-queue.

与kue类似的npm包

kue is a priority job queue library for Node.js that allows developers to manage and process jobs in a reliable and efficient manner. It provides a simple API for creating, processing, and managing jobs, making it a popular choice for applications that require background processing or task scheduling. While kue is a robust solution for job management, there are several alternatives that developers might consider based on their specific needs:

  • agenda is a lightweight job scheduling library for Node.js that uses MongoDB as its backend. It allows you to define jobs with a simple API and provides features like recurring jobs, job prioritization, and job cancellation. If your application already uses MongoDB and you need a straightforward way to schedule jobs, agenda is a great option that integrates seamlessly with your existing database.
  • bee-queue is a simple and fast job queue library for Node.js that is designed for performance and ease of use. It is built on Redis and focuses on providing a minimalistic API for job processing. bee-queue is ideal for applications that require a lightweight solution for managing background jobs without the overhead of more complex libraries.
  • bree is a job scheduler for Node.js that is designed to be simple and efficient. It allows you to run jobs in the background using worker threads, which can improve performance for CPU-intensive tasks. If you need a scheduler that can handle both background jobs and recurring tasks while leveraging the power of worker threads, bree is an excellent choice.
  • bull is a robust job and message queue library for Node.js that is built on Redis. It offers advanced features such as job retries, delayed jobs, and rate limiting, making it suitable for applications that require more complex job processing capabilities. If you need a powerful and feature-rich solution for managing jobs, bull is a strong contender.
  • node-resque is a job processing library for Node.js that is designed to work with Redis. It provides a flexible API for defining and processing jobs, as well as support for scheduled jobs and job prioritization. If you are looking for a library that offers a balance between simplicity and functionality, node-resque is worth considering.

For a detailed comparison of these job queue libraries, check out the link: Comparing agenda vs bee-queue vs bree vs bull vs kue vs node-resque.

与node-resque类似的npm包

node-resque is a robust job queue library for Node.js applications. It allows developers to manage background jobs and scheduled tasks efficiently, providing a flexible and reliable way to handle asynchronous processing. With support for Redis, node-resque enables distributed job processing, making it suitable for applications that require scalability and resilience. While node-resque is a powerful option for job management, there are several alternatives in the Node.js ecosystem that offer similar functionalities. Here are a few noteworthy alternatives:

  • agenda is a lightweight job scheduling library for Node.js that uses MongoDB as its backend. It provides a simple API for defining jobs and scheduling them to run at specific intervals or times. Agenda is particularly well-suited for applications that already utilize MongoDB and require a straightforward way to manage recurring tasks or delayed jobs. Its ease of use and integration with MongoDB make it a popular choice for developers looking for a minimalistic job scheduler.

  • bee-queue is a simple and fast job queue library for Node.js that is designed for handling jobs in a Redis-backed queue. It focuses on simplicity and performance, making it a great choice for applications that require quick job processing without the overhead of more complex job management systems. Bee-queue is ideal for applications that need to process jobs in real-time and want a straightforward API for managing job queues.

  • bull is a powerful Redis-based job and message queue for Node.js. It offers advanced features such as job prioritization, delayed jobs, and rate-limited processing, making it suitable for applications with complex job processing needs. Bull's robust architecture allows for efficient job management and monitoring, making it a popular choice among developers who require a feature-rich job queue solution.

  • bullmq is the successor to Bull, designed to provide a more modern and flexible API for job management. It builds on the strengths of Bull while introducing new features and improvements, such as support for TypeScript and better performance. Bullmq is ideal for developers looking for a cutting-edge job queue solution that leverages the latest advancements in Node.js and Redis.

  • kue is another job queue library for Node.js that uses Redis as its backend. It offers a rich set of features, including job priority, delayed jobs, and job event listeners. Kue provides a user-friendly dashboard for monitoring job progress and status, making it a suitable choice for applications that require visibility into job processing.

For a comprehensive comparison of these job queue libraries, check out the following link: Comparing agenda vs bee-queue vs bull vs bullmq vs kue vs node-resque.

与bree类似的npm包

bree is a job scheduling library for Node.js that allows developers to run background tasks and scheduled jobs with ease. It is designed to be simple and efficient, providing a straightforward API for scheduling tasks at specific intervals or times. Bree supports both cron-like scheduling and more complex job scheduling scenarios, making it a versatile choice for managing background tasks in Node.js applications.

While Bree is a powerful tool for job scheduling, there are several alternatives in the Node.js ecosystem that offer similar functionalities. Here are a few notable options:

  • agenda is a lightweight job scheduling library for Node.js that uses MongoDB as its persistence layer. It provides a simple API for scheduling jobs and supports recurring jobs, delayed jobs, and more. Agenda is a great choice if you are already using MongoDB in your application and want a seamless way to manage background tasks without introducing additional dependencies.

  • bull is a robust job and message queue library for Node.js that is built on Redis. It offers advanced features such as job prioritization, delayed jobs, and concurrency control. Bull is ideal for applications that require high performance and reliability in job processing, especially when dealing with a large number of jobs or complex workflows.

  • cron is a simple and straightforward library for scheduling tasks using cron syntax. It allows you to run tasks at specific intervals defined by cron expressions. If you are looking for a lightweight solution for running scheduled tasks without the overhead of a full job queue, cron is a solid choice.

  • later is a flexible scheduling library that allows you to define schedules using a variety of formats, including cron-like syntax. It provides a powerful API for scheduling tasks and can handle complex scheduling scenarios. Later is suitable for applications that require advanced scheduling capabilities beyond standard cron expressions.

  • node-cron is another simple cron-based job scheduler for Node.js. It allows you to schedule tasks using cron syntax and is easy to integrate into existing applications. Node-cron is a good option for developers who prefer a straightforward approach to scheduling tasks without additional complexity.

  • node-schedule is a flexible job scheduling library for Node.js that allows you to schedule jobs using cron-like syntax or JavaScript Date objects. It supports recurring jobs and provides a simple API for managing scheduled tasks. Node-schedule is a great choice for applications that need a flexible and easy-to-use scheduling solution.

To see how Bree compares with these alternatives, check out the comparison: Comparing agenda vs bree vs bull vs cron vs later vs node-cron vs node-schedule.

bull的README




The fastest, most reliable, Redis-based queue for Node.
Carefully written for rock solid stability and atomicity.


Sponsors · Features · UIs · Install · Quick Guide · Documentation

Check the new Guide!


🚀 Sponsors 🚀

Dragonfly Dragonfly is a new Redis™ drop-in replacement that is fully compatible with BullMQ and brings some important advantages over Redis™ such as massive better performance by utilizing all CPU cores available and faster and more memory efficient data structures. Read more here on how to use it with BullMQ.

📻 News and updates

Bull is currently in maintenance mode, we are only fixing bugs. For new features check BullMQ, a modern rewritten implementation in Typescript. You are still very welcome to use Bull if it suits your needs, which is a safe, battle tested library.

Follow me on Twitter for other important news and updates.

🛠 Tutorials

You can find tutorials and news in this blog: https://blog.taskforce.sh/

Used by

Bull is popular among large and small organizations, like the following ones:

Atlassian Autodesk Mozilla Nest Salesforce

Official FrontEnd

Taskforce.sh, Inc

Supercharge your queues with a professional front end:

  • Get a complete overview of all your queues.
  • Inspect jobs, search, retry, or promote delayed jobs.
  • Metrics and statistics.
  • and many more features.

Sign up at Taskforce.sh

Bull Features

  • Minimal CPU usage due to a polling-free design.
  • Robust design based on Redis.
  • Delayed jobs.
  • Schedule and repeat jobs according to a cron specification.
  • Rate limiter for jobs.
  • Retries.
  • Priority.
  • Concurrency.
  • Pause/resume—globally or locally.
  • Multiple job types per queue.
  • Threaded (sandboxed) processing functions.
  • Automatic recovery from process crashes.

And coming up on the roadmap...

  • Job completion acknowledgement (you can use the message queue pattern in the meantime).
  • Parent-child jobs relationships.

UIs

There are a few third-party UIs that you can use for monitoring:

BullMQ

Bull v3

Bull <= v2

Monitoring & Alerting

Feature Comparison

Since there are a few job queue solutions, here is a table comparing them:

FeatureBullMQ-ProBullMQBullKueBeeAgenda
Backendredisredisredisredisredismongo
Observables✓
Group Rate Limit✓
Group Support✓
Batches Support✓
Parent/Child Dependencies✓✓
Priorities✓✓✓✓✓
Concurrency✓✓✓✓✓✓
Delayed jobs✓✓✓✓✓
Global events✓✓✓✓
Rate Limiter✓✓✓
Pause/Resume✓✓✓✓
Sandboxed worker✓✓✓
Repeatable jobs✓✓✓✓
Atomic ops✓✓✓✓
Persistence✓✓✓✓✓✓
UI✓✓✓✓✓
Optimized forJobs / MessagesJobs / MessagesJobs / MessagesJobsMessagesJobs

Install

npm install bull --save

or

yarn add bull

Requirements: Bull requires a Redis version greater than or equal to 2.8.18.

Typescript Definitions

npm install @types/bull --save-dev

yarn add --dev @types/bull

Definitions are currently maintained in the DefinitelyTyped repo.

Contributing

We welcome all types of contributions, either code fixes, new features or doc improvements. Code formatting is enforced by prettier. For commits please follow conventional commits convention. All code must pass lint rules and test suites before it can be merged into develop.

Quick Guide

Basic Usage

const Queue = require('bull');

const videoQueue = new Queue('video transcoding', 'redis://127.0.0.1:6379');
const audioQueue = new Queue('audio transcoding', { redis: { port: 6379, host: '127.0.0.1', password: 'foobared' } }); // Specify Redis connection using object
const imageQueue = new Queue('image transcoding');
const pdfQueue = new Queue('pdf transcoding');

videoQueue.process(function (job, done) {

  // job.data contains the custom data passed when the job was created
  // job.id contains id of this job.

  // transcode video asynchronously and report progress
  job.progress(42);

  // call done when finished
  done();

  // or give an error if error
  done(new Error('error transcoding'));

  // or pass it a result
  done(null, { framerate: 29.5 /* etc... */ });

  // If the job throws an unhandled exception it is also handled correctly
  throw new Error('some unexpected error');
});

audioQueue.process(function (job, done) {
  // transcode audio asynchronously and report progress
  job.progress(42);

  // call done when finished
  done();

  // or give an error if error
  done(new Error('error transcoding'));

  // or pass it a result
  done(null, { samplerate: 48000 /* etc... */ });

  // If the job throws an unhandled exception it is also handled correctly
  throw new Error('some unexpected error');
});

imageQueue.process(function (job, done) {
  // transcode image asynchronously and report progress
  job.progress(42);

  // call done when finished
  done();

  // or give an error if error
  done(new Error('error transcoding'));

  // or pass it a result
  done(null, { width: 1280, height: 720 /* etc... */ });

  // If the job throws an unhandled exception it is also handled correctly
  throw new Error('some unexpected error');
});

pdfQueue.process(function (job) {
  // Processors can also return promises instead of using the done callback
  return pdfAsyncProcessor();
});

videoQueue.add({ video: 'http://example.com/video1.mov' });
audioQueue.add({ audio: 'http://example.com/audio1.mp3' });
imageQueue.add({ image: 'http://example.com/image1.tiff' });

Using promises

Alternatively, you can return promises instead of using the done callback:

videoQueue.process(function (job) { // don't forget to remove the done callback!
  // Simply return a promise
  return fetchVideo(job.data.url).then(transcodeVideo);

  // Handles promise rejection
  return Promise.reject(new Error('error transcoding'));

  // Passes the value the promise is resolved with to the "completed" event
  return Promise.resolve({ framerate: 29.5 /* etc... */ });

  // If the job throws an unhandled exception it is also handled correctly
  throw new Error('some unexpected error');
  // same as
  return Promise.reject(new Error('some unexpected error'));
});

Separate processes

The process function can also be run in a separate process. This has several advantages:

  • The process is sandboxed so if it crashes it does not affect the worker.
  • You can run blocking code without affecting the queue (jobs will not stall).
  • Much better utilization of multi-core CPUs.
  • Less connections to redis.

In order to use this feature just create a separate file with the processor:

// processor.js
module.exports = function (job) {
  // Do some heavy work

  return Promise.resolve(result);
}

And define the processor like this:

// Single process:
queue.process('/path/to/my/processor.js');

// You can use concurrency as well:
queue.process(5, '/path/to/my/processor.js');

// and named processors:
queue.process('my processor', 5, '/path/to/my/processor.js');

Repeated jobs

A job can be added to a queue and processed repeatedly according to a cron specification:

  paymentsQueue.process(function (job) {
    // Check payments
  });

  // Repeat payment job once every day at 3:15 (am)
  paymentsQueue.add(paymentsData, { repeat: { cron: '15 3 * * *' } });

As a tip, check your expressions here to verify they are correct: cron expression generator

Pause / Resume

A queue can be paused and resumed globally (pass true to pause processing for just this worker):

queue.pause().then(function () {
  // queue is paused now
});

queue.resume().then(function () {
  // queue is resumed now
})

Events

A queue emits some useful events, for example...

.on('completed', function (job, result) {
  // Job completed with output result!
})

For more information on events, including the full list of events that are fired, check out the Events reference

Queues performance

Queues are cheap, so if you need many of them just create new ones with different names:

const userJohn = new Queue('john');
const userLisa = new Queue('lisa');
.
.
.

However every queue instance will require new redis connections, check how to reuse connections or you can also use named processors to achieve a similar result.

Cluster support

NOTE: From version 3.2.0 and above it is recommended to use threaded processors instead.

Queues are robust and can be run in parallel in several threads or processes without any risk of hazards or queue corruption. Check this simple example using cluster to parallelize jobs across processes:

const Queue = require('bull');
const cluster = require('cluster');

const numWorkers = 8;
const queue = new Queue('test concurrent queue');

if (cluster.isMaster) {
  for (let i = 0; i < numWorkers; i++) {
    cluster.fork();
  }

  cluster.on('online', function (worker) {
    // Let's create a few jobs for the queue workers
    for (let i = 0; i < 500; i++) {
      queue.add({ foo: 'bar' });
    };
  });

  cluster.on('exit', function (worker, code, signal) {
    console.log('worker ' + worker.process.pid + ' died');
  });
} else {
  queue.process(function (job, jobDone) {
    console.log('Job done by worker', cluster.worker.id, job.id);
    jobDone();
  });
}

Documentation

For the full documentation, check out the reference and common patterns:

  • Guide — Your starting point for developing with Bull.
  • Reference — Reference document with all objects and methods available.
  • Patterns — a set of examples for common patterns.
  • License — the Bull license—it's MIT.

If you see anything that could use more docs, please submit a pull request!

Important Notes

The queue aims for an "at least once" working strategy. This means that in some situations, a job could be processed more than once. This mostly happens when a worker fails to keep a lock for a given job during the total duration of the processing.

When a worker is processing a job it will keep the job "locked" so other workers can't process it.

It's important to understand how locking works to prevent your jobs from losing their lock - becoming stalled - and being restarted as a result. Locking is implemented internally by creating a lock for lockDuration on interval lockRenewTime (which is usually half lockDuration). If lockDuration elapses before the lock can be renewed, the job will be considered stalled and is automatically restarted; it will be double processed. This can happen when:

  1. The Node process running your job processor unexpectedly terminates.
  2. Your job processor was too CPU-intensive and stalled the Node event loop, and as a result, Bull couldn't renew the job lock (see #488 for how we might better detect this). You can fix this by breaking your job processor into smaller parts so that no single part can block the Node event loop. Alternatively, you can pass a larger value for the lockDuration setting (with the tradeoff being that it will take longer to recognize a real stalled job).

As such, you should always listen for the stalled event and log this to your error monitoring system, as this means your jobs are likely getting double-processed.

As a safeguard so problematic jobs won't get restarted indefinitely (e.g. if the job processor always crashes its Node process), jobs will be recovered from a stalled state a maximum of maxStalledCount times (default: 1).

npm-compare.com is an independent website designed to help developers choose the most suitable npm packages. We are not affiliated with npm, Inc, the official website for the npm registry. Note that npm is a registered trademark of npm, Inc. For more details, please refer to our Terms & Privacy.