bull vs agenda vs bee-queue vs bree vs kue vs node-resque
Node.js 后台任务队列与调度方案选型指南
bullagendabee-queuebreekuenode-resque类似的npm包:

Node.js 后台任务队列与调度方案选型指南

这些库用于在 Node.js 应用中处理后台任务,例如发送电子邮件、生成报告或处理大量数据。它们帮助开发者将耗时操作从主线程中移走,确保应用响应速度不受影响。agenda 和 bull 适合需要高可靠性和复杂调度的场景;bree 适合需要进程隔离的简单任务;bee-queue 和 node-resque 适合基于 Redis 的轻量级需求;而 kue 已停止维护,不建议在新项目中使用。

npm下载趋势

3 年

GitHub Stars 排名

统计详情

npm包名称
下载量
Stars
大小
Issues
发布时间
License
bull2,245,94716,262309 kB1472 年前MIT
agenda228,6899,712310 kB1910 小时前MIT
bee-queue67,7854,039107 kB461 年前MIT
bree42,9493,29692 kB308 个月前MIT
kue31,0799,418-2869 年前MIT
node-resque26,4421,415720 kB243 个月前Apache-2.0

Node.js 后台任务队列方案深度对比

在构建现代 Web 应用时,前端开发者经常需要与后台任务系统交互。无论是发送确认邮件、处理上传的文件,还是生成复杂的报表,这些操作都不应该阻塞用户界面。agenda、bee-queue、bree、bull、kue 和 node-resque 都是 Node.js 生态中流行的任务处理工具,但它们的设计理念和适用场景各不相同。本文将从架构、代码实现和维护状态三个方面进行深度对比。

🏗️ 核心架构与依赖:数据库 vs 进程隔离

选择任务队列时,首先要看它依赖什么存储系统,以及任务如何执行。这直接影响你的基础设施成本和运维复杂度。

agenda 依赖 MongoDB。

  • 任务数据直接存在 Mongo 集合中。
  • 适合已经使用 Mongo 的项目,无需引入 Redis。
// agenda: 基于 MongoDB
const agenda = new Agenda({ db: { address: 'mongo-uri' } });
agenda.define('sendEmail', async (job) => { /*...*/ });

bull 依赖 Redis。

  • 使用 Redis 列表和排序集合管理任务。
  • 性能极高,支持复杂优先级和速率限制。
// bull: 基于 Redis
const Queue = require('bull');
const queue = new Queue('email', 'redis://localhost');
queue.process(async (job) => { /*...*/ });

bee-queue 依赖 Redis。

  • 比 bull 更轻量,功能较少。
  • 适合简单的 FIFO 队列需求。
// bee-queue: 基于 Redis
const Queue = require('bee-queue');
const queue = new Queue('email', { redis: { host: 'localhost' } });
queue.process(async (job) => { /*...*/ });

node-resque 依赖 Redis。

  • 遵循 Resque 协议,适合多语言互操作。
  • 结构较为传统,基于类实现。
// node-resque: 基于 Redis
const { Worker } = require('node-resque');
const worker = new Worker({ queues: ['email'], connection: { host: 'localhost' } });
worker.perform = async (job, args) => { /*...*/ };

bree 不需要外部数据库。

  • 任务作为独立进程运行,通过文件配置。
  • 天然隔离,单个任务崩溃不影响主进程。
// bree: 基于独立进程
const Bree = require('bree');
const bree = new Bree({ workers: ['send-email'] });
// send-email.js 文件独立运行

kue 依赖 Redis。

  • 早期流行方案,现已停止维护。
  • 架构类似 bull 但功能较旧。
// kue: 基于 Redis (已废弃)
const kue = require('kue');
const queue = kue.createQueue();
queue.process('email', (job, done) => { /*...*/ });

📝 定义与处理任务:代码风格对比

不同的库有不同的 API 设计风格。有的喜欢链式调用,有的喜欢配置对象。了解这些差异有助于团队快速上手。

agenda 使用 define 注册处理函数。

  • 逻辑清晰,支持异步函数。
  • 任务数据通过 job.attrs.data 访问。
// agenda: 定义任务
agenda.define('sendEmail', async (job) => {
  const { to, subject } = job.attrs.data;
  await emailService.send(to, subject);
});

bull 使用 process 注册处理器。

  • 支持并发控制,例如一次处理 5 个任务。
  • 任务数据直接在 job.data 中。
// bull: 定义任务
queue.process('sendEmail', async (job) => {
  const { to, subject } = job.data;
  await emailService.send(to, subject);
});

bee-queue 使用 process 注册处理器。

  • 回调风格或 Promise 风格均可。
  • 需要手动调用 done() 如果使用回调。
// bee-queue: 定义任务
queue.process(async (job) => {
  const { to, subject } = job.data;
  await emailService.send(to, subject);
});

node-resque 通过类方法 perform 定义。

  • 面向对象风格,适合大型系统。
  • 参数直接传递给 perform 方法。
// node-resque: 定义任务
class SendEmail {
  static async perform(args) {
    const { to, subject } = args;
    await emailService.send(to, subject);
  }
}

bree 通过独立文件定义。

  • 每个任务是一个单独的 .js 文件。
  • 默认导出一个函数,接收配置对象。
// bree: 定义任务 (workers/send-email.js)
module.exports = async (opts) => {
  const { to, subject } = opts.data;
  await emailService.send(to, subject);
};

kue 使用 process 注册处理器。

  • 类似 bull,但 API 较旧。
  • 必须调用 done() 完成任务。
// kue: 定义任务 (已废弃)
queue.process('sendEmail', (job, done) => {
  const { to, subject } = job.data;
  emailService.send(to, subject).then(done).catch(done);
});

⏰ 任务调度:即时 vs 延迟 vs 循环

有些任务需要立即执行,有些需要延迟,还有些需要每天定时运行。不同库对调度的支持程度不同。

agenda 支持强大的 schedule 和 every。

  • 可以直接使用 cron 表达式或自然语言。
  • 适合复杂的业务调度需求。
// agenda: 调度任务
await agenda.schedule('in 5 minutes', 'sendEmail', { to: '[email protected]' });
agenda.every('10 minutes', 'cleanupLogs');

bull 支持 delay 和 repeat。

  • 延迟任务简单直接。
  • 重复任务需要配置 cron 表达式。
// bull: 调度任务
await queue.add('sendEmail', data, { delay: 300000 });
await queue.add('cleanup', {}, { repeat: { cron: '*/10 * * * *' } });

bee-queue 支持 delay。

  • 不支持内置的 cron 重复任务。
  • 适合一次性延迟任务。
// bee-queue: 调度任务
const job = queue.createJob(data);
job.delay(300000);
await job.save();

node-resque 支持 enqueueAt。

  • 需要手动计算时间戳。
  • 重复任务需要外部调度器配合。
// node-resque: 调度任务
const time = new Date().getTime() + 300000;
await queue.enqueueAt(time, 'email', 'SendEmail', [{ to: '[email protected]' }]);

bree 通过配置 interval 或 cron。

  • 在初始化配置中定义调度规则。
  • 不需要代码调用,配置即运行。
// bree: 调度任务 (配置文件中)
const bree = new Bree({
  workers: [{ name: 'cleanup', interval: '10m' }]
});

kue 支持 delay。

  • 类似 bee-queue。
  • 重复任务支持较弱。
// kue: 调度任务 (已废弃)
const job = queue.create('sendEmail', data);
job.delay(300000);
job.save();

⚠️ 维护状态与风险提示

在选择库之前,必须确认它是否还在维护。使用废弃的库会给项目带来安全风险和兼容性问题。

  • kue:已废弃。最后更新停留在多年前,社区不再推荐。不要在新项目中使用。
  • bull:维护中。但官方推荐使用新一代 bullmq。如果是新项目,建议评估 bullmq。
  • agenda:活跃维护。适合 Mongo 用户。
  • bree:活跃维护。适合需要进程隔离的场景。
  • bee-queue:低维护。功能稳定但更新缓慢。
  • node-resque:稳定。适合特定协议需求。

📊 总结对比表

特性agendabullbreebee-queuenode-resquekue
存储后端MongoDBRedis无 (文件)RedisRedisRedis
进程隔离否否是否否否
重复任务强大 (cron)支持 (cron)支持 (配置)不支持需外部配合弱支持
维护状态✅ 活跃✅ 活跃✅ 活跃⚠️ 低频✅ 稳定❌ 废弃
学习曲线中中低低高低

💡 最终建议

agenda 是 MongoDB 用户的最佳伙伴 — 如果你已经在用 Mongo,它能让你少维护一个 Redis 实例。它的调度功能非常强大,适合复杂的业务逻辑。

bull 是 Redis 用户的首选 — 性能强劲,功能丰富。但请注意,如果是全新项目,可以看看 bullmq,它是 bull 的现代升级版。

bree 是隔离需求的独特方案 — 如果你担心某个任务内存泄漏会搞挂整个服务,bree 的独立进程模型能让你睡个安稳觉。它不需要额外的数据库,部署简单。

bee-queue 和 node-resque 适合特定场景 — 前者适合简单 Redis 队列,后者适合需要 Resque 协议兼容的环境。

kue 请避免使用 — 它已经完成了历史使命,现在有更安全、更强大的替代品。

核心思考:不要为了用队列而用队列。如果你的任务很简单,setTimeout 或 node-cron 可能就够了。但如果任务需要可靠性、重试机制和监控,那么选择一个维护良好的队列库是必要的投资。

如何选择: bull vs agenda vs bee-queue vs bree vs kue vs node-resque

  • bull:

    如果你需要强大的 Redis 队列功能,包括优先级、速率限制和事件监听,bull 是成熟的选择。注意新版本推荐使用 bullmq,但 bull 依然广泛使用,适合高并发任务处理。

  • agenda:

    如果你已经在使用 MongoDB,并且需要灵活的 cron 表达式或基于日期的调度,agenda 是最佳选择。它将任务数据存储在 Mongo 中,便于查询和管理,适合需要持久化任务状态的场景。

  • bee-queue:

    如果你需要一个基于 Redis 的简单队列,且不需要复杂的优先级或速率限制,bee-queue 是一个轻量级选项。它的 API 非常直观,适合快速搭建小型后台任务系统。

  • bree:

    如果你希望每个任务都在独立的进程中运行,以避免内存泄漏或阻塞主线程,bree 是独特的选择。它不需要 Redis 或 Mongo,仅通过配置文件管理任务,适合 cron 风格的任务调度。

  • kue:

    不建议在新项目中使用 kue。该库已停止维护多年,存在未修复的安全隐患和兼容性问题。请考虑迁移到 bull 或 agenda 以获得长期支持。

  • node-resque:

    如果你需要兼容 Resque 协议或与 Ruby on Rails 的 Resque 系统互操作,node-resque 是合适选择。它基于 Redis,适合需要标准化队列协议的多语言环境。

bull的README




The fastest, most reliable, Redis-based queue for Node.
Carefully written for rock solid stability and atomicity.


Sponsors · Features · UIs · Install · Quick Guide · Documentation

Check the new Guide!


🚀 Sponsors 🚀

Dragonfly Dragonfly is a new Redis™ drop-in replacement that is fully compatible with BullMQ and brings some important advantages over Redis™ such as massive better performance by utilizing all CPU cores available and faster and more memory efficient data structures. Read more here on how to use it with BullMQ.

📻 News and updates

Bull is currently in maintenance mode, we are only fixing bugs. For new features check BullMQ, a modern rewritten implementation in Typescript. You are still very welcome to use Bull if it suits your needs, which is a safe, battle tested library.

Follow me on Twitter for other important news and updates.

🛠 Tutorials

You can find tutorials and news in this blog: https://blog.taskforce.sh/


Used by

Bull is popular among large and small organizations, like the following ones:

Atlassian Autodesk Mozilla Nest Salesforce


Official FrontEnd

Taskforce.sh, Inc

Supercharge your queues with a professional front end:

  • Get a complete overview of all your queues.
  • Inspect jobs, search, retry, or promote delayed jobs.
  • Metrics and statistics.
  • and many more features.

Sign up at Taskforce.sh


Bull Features

  • Minimal CPU usage due to a polling-free design.
  • Robust design based on Redis.
  • Delayed jobs.
  • Schedule and repeat jobs according to a cron specification.
  • Rate limiter for jobs.
  • Retries.
  • Priority.
  • Concurrency.
  • Pause/resume—globally or locally.
  • Multiple job types per queue.
  • Threaded (sandboxed) processing functions.
  • Automatic recovery from process crashes.

And coming up on the roadmap...

  • Job completion acknowledgement (you can use the message queue pattern in the meantime).
  • Parent-child jobs relationships.

UIs

There are a few third-party UIs that you can use for monitoring:

BullMQ

Bull v3

Bull <= v2


Monitoring & Alerting


Feature Comparison

Since there are a few job queue solutions, here is a table comparing them:

FeatureBullMQ-ProBullMQBullKueBeeAgenda
Backendredisredisredisredisredismongo
Observables✓
Group Rate Limit✓
Group Support✓
Batches Support✓
Parent/Child Dependencies✓✓
Priorities✓✓✓✓✓
Concurrency✓✓✓✓✓✓
Delayed jobs✓✓✓✓✓
Global events✓✓✓✓
Rate Limiter✓✓✓
Pause/Resume✓✓✓✓
Sandboxed worker✓✓✓
Repeatable jobs✓✓✓✓
Atomic ops✓✓✓✓
Persistence✓✓✓✓✓✓
UI✓✓✓✓✓
Optimized forJobs / MessagesJobs / MessagesJobs / MessagesJobsMessagesJobs

Install

npm install bull --save

or

yarn add bull

Requirements: Bull requires a Redis version greater than or equal to 2.8.18.

Typescript Definitions

npm install @types/bull --save-dev
yarn add --dev @types/bull

Definitions are currently maintained in the DefinitelyTyped repo.

Contributing

We welcome all types of contributions, either code fixes, new features or doc improvements. Code formatting is enforced by prettier. For commits please follow conventional commits convention. All code must pass lint rules and test suites before it can be merged into develop.


Quick Guide

Basic Usage

const Queue = require('bull');

const videoQueue = new Queue('video transcoding', 'redis://127.0.0.1:6379');
const audioQueue = new Queue('audio transcoding', { redis: { port: 6379, host: '127.0.0.1', password: 'foobared' } }); // Specify Redis connection using object
const imageQueue = new Queue('image transcoding');
const pdfQueue = new Queue('pdf transcoding');

videoQueue.process(function (job, done) {

  // job.data contains the custom data passed when the job was created
  // job.id contains id of this job.

  // transcode video asynchronously and report progress
  job.progress(42);

  // call done when finished
  done();

  // or give an error if error
  done(new Error('error transcoding'));

  // or pass it a result
  done(null, { framerate: 29.5 /* etc... */ });

  // If the job throws an unhandled exception it is also handled correctly
  throw new Error('some unexpected error');
});

audioQueue.process(function (job, done) {
  // transcode audio asynchronously and report progress
  job.progress(42);

  // call done when finished
  done();

  // or give an error if error
  done(new Error('error transcoding'));

  // or pass it a result
  done(null, { samplerate: 48000 /* etc... */ });

  // If the job throws an unhandled exception it is also handled correctly
  throw new Error('some unexpected error');
});

imageQueue.process(function (job, done) {
  // transcode image asynchronously and report progress
  job.progress(42);

  // call done when finished
  done();

  // or give an error if error
  done(new Error('error transcoding'));

  // or pass it a result
  done(null, { width: 1280, height: 720 /* etc... */ });

  // If the job throws an unhandled exception it is also handled correctly
  throw new Error('some unexpected error');
});

pdfQueue.process(function (job) {
  // Processors can also return promises instead of using the done callback
  return pdfAsyncProcessor();
});

videoQueue.add({ video: 'http://example.com/video1.mov' });
audioQueue.add({ audio: 'http://example.com/audio1.mp3' });
imageQueue.add({ image: 'http://example.com/image1.tiff' });

Using promises

Alternatively, you can return promises instead of using the done callback:

videoQueue.process(function (job) { // don't forget to remove the done callback!
  // Simply return a promise
  return fetchVideo(job.data.url).then(transcodeVideo);

  // Handles promise rejection
  return Promise.reject(new Error('error transcoding'));

  // Passes the value the promise is resolved with to the "completed" event
  return Promise.resolve({ framerate: 29.5 /* etc... */ });

  // If the job throws an unhandled exception it is also handled correctly
  throw new Error('some unexpected error');
  // same as
  return Promise.reject(new Error('some unexpected error'));
});

Separate processes

The process function can also be run in a separate process. This has several advantages:

  • The process is sandboxed so if it crashes it does not affect the worker.
  • You can run blocking code without affecting the queue (jobs will not stall).
  • Much better utilization of multi-core CPUs.
  • Less connections to redis.

In order to use this feature just create a separate file with the processor:

// processor.js
module.exports = function (job) {
  // Do some heavy work

  return Promise.resolve(result);
}

And define the processor like this:

// Single process:
queue.process('/path/to/my/processor.js');

// You can use concurrency as well:
queue.process(5, '/path/to/my/processor.js');

// and named processors:
queue.process('my processor', 5, '/path/to/my/processor.js');

Repeated jobs

A job can be added to a queue and processed repeatedly according to a cron specification:

  paymentsQueue.process(function (job) {
    // Check payments
  });

  // Repeat payment job once every day at 3:15 (am)
  paymentsQueue.add(paymentsData, { repeat: { cron: '15 3 * * *' } });

As a tip, check your expressions here to verify they are correct: cron expression generator

Pause / Resume

A queue can be paused and resumed globally (pass true to pause processing for just this worker):

queue.pause().then(function () {
  // queue is paused now
});

queue.resume().then(function () {
  // queue is resumed now
})

Events

A queue emits some useful events, for example...

.on('completed', function (job, result) {
  // Job completed with output result!
})

For more information on events, including the full list of events that are fired, check out the Events reference

Queues performance

Queues are cheap, so if you need many of them just create new ones with different names:

const userJohn = new Queue('john');
const userLisa = new Queue('lisa');
.
.
.

However every queue instance will require new redis connections, check how to reuse connections or you can also use named processors to achieve a similar result.

Cluster support

NOTE: From version 3.2.0 and above it is recommended to use threaded processors instead.

Queues are robust and can be run in parallel in several threads or processes without any risk of hazards or queue corruption. Check this simple example using cluster to parallelize jobs across processes:

const Queue = require('bull');
const cluster = require('cluster');

const numWorkers = 8;
const queue = new Queue('test concurrent queue');

if (cluster.isMaster) {
  for (let i = 0; i < numWorkers; i++) {
    cluster.fork();
  }

  cluster.on('online', function (worker) {
    // Let's create a few jobs for the queue workers
    for (let i = 0; i < 500; i++) {
      queue.add({ foo: 'bar' });
    };
  });

  cluster.on('exit', function (worker, code, signal) {
    console.log('worker ' + worker.process.pid + ' died');
  });
} else {
  queue.process(function (job, jobDone) {
    console.log('Job done by worker', cluster.worker.id, job.id);
    jobDone();
  });
}

Documentation

For the full documentation, check out the reference and common patterns:

  • Guide — Your starting point for developing with Bull.
  • Reference — Reference document with all objects and methods available.
  • Patterns — a set of examples for common patterns.
  • License — the Bull license—it's MIT.

If you see anything that could use more docs, please submit a pull request!


Important Notes

The queue aims for an "at least once" working strategy. This means that in some situations, a job could be processed more than once. This mostly happens when a worker fails to keep a lock for a given job during the total duration of the processing.

When a worker is processing a job it will keep the job "locked" so other workers can't process it.

It's important to understand how locking works to prevent your jobs from losing their lock - becoming stalled - and being restarted as a result. Locking is implemented internally by creating a lock for lockDuration on interval lockRenewTime (which is usually half lockDuration). If lockDuration elapses before the lock can be renewed, the job will be considered stalled and is automatically restarted; it will be double processed. This can happen when:

  1. The Node process running your job processor unexpectedly terminates.
  2. Your job processor was too CPU-intensive and stalled the Node event loop, and as a result, Bull couldn't renew the job lock (see #488 for how we might better detect this). You can fix this by breaking your job processor into smaller parts so that no single part can block the Node event loop. Alternatively, you can pass a larger value for the lockDuration setting (with the tradeoff being that it will take longer to recognize a real stalled job).

As such, you should always listen for the stalled event and log this to your error monitoring system, as this means your jobs are likely getting double-processed.

As a safeguard so problematic jobs won't get restarted indefinitely (e.g. if the job processor always crashes its Node process), jobs will be recovered from a stalled state a maximum of maxStalledCount times (default: 1).