这些库用于在 Node.js 应用中处理后台任务,例如发送电子邮件、生成报告或处理大量数据。它们帮助开发者将耗时操作从主线程中移走,确保应用响应速度不受影响。agenda 和 bull 适合需要高可靠性和复杂调度的场景;bree 适合需要进程隔离的简单任务;bee-queue 和 node-resque 适合基于 Redis 的轻量级需求;而 kue 已停止维护,不建议在新项目中使用。
在构建现代 Web 应用时,前端开发者经常需要与后台任务系统交互。无论是发送确认邮件、处理上传的文件,还是生成复杂的报表,这些操作都不应该阻塞用户界面。agenda、bee-queue、bree、bull、kue 和 node-resque 都是 Node.js 生态中流行的任务处理工具,但它们的设计理念和适用场景各不相同。本文将从架构、代码实现和维护状态三个方面进行深度对比。
选择任务队列时,首先要看它依赖什么存储系统,以及任务如何执行。这直接影响你的基础设施成本和运维复杂度。
agenda 依赖 MongoDB。
// agenda: 基于 MongoDB
const agenda = new Agenda({ db: { address: 'mongo-uri' } });
agenda.define('sendEmail', async (job) => { /*...*/ });
bull 依赖 Redis。
// bull: 基于 Redis
const Queue = require('bull');
const queue = new Queue('email', 'redis://localhost');
queue.process(async (job) => { /*...*/ });
bee-queue 依赖 Redis。
bull 更轻量,功能较少。// bee-queue: 基于 Redis
const Queue = require('bee-queue');
const queue = new Queue('email', { redis: { host: 'localhost' } });
queue.process(async (job) => { /*...*/ });
node-resque 依赖 Redis。
// node-resque: 基于 Redis
const { Worker } = require('node-resque');
const worker = new Worker({ queues: ['email'], connection: { host: 'localhost' } });
worker.perform = async (job, args) => { /*...*/ };
bree 不需要外部数据库。
// bree: 基于独立进程
const Bree = require('bree');
const bree = new Bree({ workers: ['send-email'] });
// send-email.js 文件独立运行
kue 依赖 Redis。
bull 但功能较旧。// kue: 基于 Redis (已废弃)
const kue = require('kue');
const queue = kue.createQueue();
queue.process('email', (job, done) => { /*...*/ });
不同的库有不同的 API 设计风格。有的喜欢链式调用,有的喜欢配置对象。了解这些差异有助于团队快速上手。
agenda 使用 define 注册处理函数。
job.attrs.data 访问。// agenda: 定义任务
agenda.define('sendEmail', async (job) => {
const { to, subject } = job.attrs.data;
await emailService.send(to, subject);
});
bull 使用 process 注册处理器。
job.data 中。// bull: 定义任务
queue.process('sendEmail', async (job) => {
const { to, subject } = job.data;
await emailService.send(to, subject);
});
bee-queue 使用 process 注册处理器。
done() 如果使用回调。// bee-queue: 定义任务
queue.process(async (job) => {
const { to, subject } = job.data;
await emailService.send(to, subject);
});
node-resque 通过类方法 perform 定义。
perform 方法。// node-resque: 定义任务
class SendEmail {
static async perform(args) {
const { to, subject } = args;
await emailService.send(to, subject);
}
}
bree 通过独立文件定义。
.js 文件。// bree: 定义任务 (workers/send-email.js)
module.exports = async (opts) => {
const { to, subject } = opts.data;
await emailService.send(to, subject);
};
kue 使用 process 注册处理器。
bull,但 API 较旧。done() 完成任务。// kue: 定义任务 (已废弃)
queue.process('sendEmail', (job, done) => {
const { to, subject } = job.data;
emailService.send(to, subject).then(done).catch(done);
});
有些任务需要立即执行,有些需要延迟,还有些需要每天定时运行。不同库对调度的支持程度不同。
agenda 支持强大的 schedule 和 every。
// agenda: 调度任务
await agenda.schedule('in 5 minutes', 'sendEmail', { to: '[email protected]' });
agenda.every('10 minutes', 'cleanupLogs');
bull 支持 delay 和 repeat。
// bull: 调度任务
await queue.add('sendEmail', data, { delay: 300000 });
await queue.add('cleanup', {}, { repeat: { cron: '*/10 * * * *' } });
bee-queue 支持 delay。
// bee-queue: 调度任务
const job = queue.createJob(data);
job.delay(300000);
await job.save();
node-resque 支持 enqueueAt。
// node-resque: 调度任务
const time = new Date().getTime() + 300000;
await queue.enqueueAt(time, 'email', 'SendEmail', [{ to: '[email protected]' }]);
bree 通过配置 interval 或 cron。
// bree: 调度任务 (配置文件中)
const bree = new Bree({
workers: [{ name: 'cleanup', interval: '10m' }]
});
kue 支持 delay。
bee-queue。// kue: 调度任务 (已废弃)
const job = queue.create('sendEmail', data);
job.delay(300000);
job.save();
在选择库之前,必须确认它是否还在维护。使用废弃的库会给项目带来安全风险和兼容性问题。
kue:已废弃。最后更新停留在多年前,社区不再推荐。不要在新项目中使用。bull:维护中。但官方推荐使用新一代 bullmq。如果是新项目,建议评估 bullmq。agenda:活跃维护。适合 Mongo 用户。bree:活跃维护。适合需要进程隔离的场景。bee-queue:低维护。功能稳定但更新缓慢。node-resque:稳定。适合特定协议需求。| 特性 | agenda | bull | bree | bee-queue | node-resque | kue |
|---|---|---|---|---|---|---|
| 存储后端 | MongoDB | Redis | 无 (文件) | Redis | Redis | Redis |
| 进程隔离 | 否 | 否 | 是 | 否 | 否 | 否 |
| 重复任务 | 强大 (cron) | 支持 (cron) | 支持 (配置) | 不支持 | 需外部配合 | 弱支持 |
| 维护状态 | ✅ 活跃 | ✅ 活跃 | ✅ 活跃 | ⚠️ 低频 | ✅ 稳定 | ❌ 废弃 |
| 学习曲线 | 中 | 中 | 低 | 低 | 高 | 低 |
agenda 是 MongoDB 用户的最佳伙伴 — 如果你已经在用 Mongo,它能让你少维护一个 Redis 实例。它的调度功能非常强大,适合复杂的业务逻辑。
bull 是 Redis 用户的首选 — 性能强劲,功能丰富。但请注意,如果是全新项目,可以看看 bullmq,它是 bull 的现代升级版。
bree 是隔离需求的独特方案 — 如果你担心某个任务内存泄漏会搞挂整个服务,bree 的独立进程模型能让你睡个安稳觉。它不需要额外的数据库,部署简单。
bee-queue 和 node-resque 适合特定场景 — 前者适合简单 Redis 队列,后者适合需要 Resque 协议兼容的环境。
kue 请避免使用 — 它已经完成了历史使命,现在有更安全、更强大的替代品。
核心思考:不要为了用队列而用队列。如果你的任务很简单,setTimeout 或 node-cron 可能就够了。但如果任务需要可靠性、重试机制和监控,那么选择一个维护良好的队列库是必要的投资。
如果你需要强大的 Redis 队列功能,包括优先级、速率限制和事件监听,bull 是成熟的选择。注意新版本推荐使用 bullmq,但 bull 依然广泛使用,适合高并发任务处理。
如果你已经在使用 MongoDB,并且需要灵活的 cron 表达式或基于日期的调度,agenda 是最佳选择。它将任务数据存储在 Mongo 中,便于查询和管理,适合需要持久化任务状态的场景。
如果你需要一个基于 Redis 的简单队列,且不需要复杂的优先级或速率限制,bee-queue 是一个轻量级选项。它的 API 非常直观,适合快速搭建小型后台任务系统。
如果你希望每个任务都在独立的进程中运行,以避免内存泄漏或阻塞主线程,bree 是独特的选择。它不需要 Redis 或 Mongo,仅通过配置文件管理任务,适合 cron 风格的任务调度。
不建议在新项目中使用 kue。该库已停止维护多年,存在未修复的安全隐患和兼容性问题。请考虑迁移到 bull 或 agenda 以获得长期支持。
如果你需要兼容 Resque 协议或与 Ruby on Rails 的 Resque 系统互操作,node-resque 是合适选择。它基于 Redis,适合需要标准化队列协议的多语言环境。
The fastest, most reliable, Redis-based queue for Node.
Carefully written for rock solid stability and atomicity.
Sponsors · Features · UIs · Install · Quick Guide · Documentation
Check the new Guide!
|
| Dragonfly is a new Redis™ drop-in replacement that is fully compatible with BullMQ and brings some important advantages over Redis™ such as massive better performance by utilizing all CPU cores available and faster and more memory efficient data structures. Read more here on how to use it with BullMQ. |
Bull is currently in maintenance mode, we are only fixing bugs. For new features check BullMQ, a modern rewritten implementation in Typescript. You are still very welcome to use Bull if it suits your needs, which is a safe, battle tested library.
Follow me on Twitter for other important news and updates.
You can find tutorials and news in this blog: https://blog.taskforce.sh/
Bull is popular among large and small organizations, like the following ones:
|
|
|
|
|
Supercharge your queues with a professional front end:
Sign up at Taskforce.sh
And coming up on the roadmap...
There are a few third-party UIs that you can use for monitoring:
BullMQ
Bull v3
Bull <= v2
Since there are a few job queue solutions, here is a table comparing them:
| Feature | BullMQ-Pro | BullMQ | Bull | Kue | Bee | Agenda |
|---|---|---|---|---|---|---|
| Backend | redis | redis | redis | redis | redis | mongo |
| Observables | ✓ | |||||
| Group Rate Limit | ✓ | |||||
| Group Support | ✓ | |||||
| Batches Support | ✓ | |||||
| Parent/Child Dependencies | ✓ | ✓ | ||||
| Priorities | ✓ | ✓ | ✓ | ✓ | ✓ | |
| Concurrency | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Delayed jobs | ✓ | ✓ | ✓ | ✓ | ✓ | |
| Global events | ✓ | ✓ | ✓ | ✓ | ||
| Rate Limiter | ✓ | ✓ | ✓ | |||
| Pause/Resume | ✓ | ✓ | ✓ | ✓ | ||
| Sandboxed worker | ✓ | ✓ | ✓ | |||
| Repeatable jobs | ✓ | ✓ | ✓ | ✓ | ||
| Atomic ops | ✓ | ✓ | ✓ | ✓ | ||
| Persistence | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| UI | ✓ | ✓ | ✓ | ✓ | ✓ | |
| Optimized for | Jobs / Messages | Jobs / Messages | Jobs / Messages | Jobs | Messages | Jobs |
npm install bull --save
or
yarn add bull
Requirements: Bull requires a Redis version greater than or equal to 2.8.18.
npm install @types/bull --save-dev
yarn add --dev @types/bull
Definitions are currently maintained in the DefinitelyTyped repo.
We welcome all types of contributions, either code fixes, new features or doc improvements. Code formatting is enforced by prettier. For commits please follow conventional commits convention. All code must pass lint rules and test suites before it can be merged into develop.
const Queue = require('bull');
const videoQueue = new Queue('video transcoding', 'redis://127.0.0.1:6379');
const audioQueue = new Queue('audio transcoding', { redis: { port: 6379, host: '127.0.0.1', password: 'foobared' } }); // Specify Redis connection using object
const imageQueue = new Queue('image transcoding');
const pdfQueue = new Queue('pdf transcoding');
videoQueue.process(function (job, done) {
// job.data contains the custom data passed when the job was created
// job.id contains id of this job.
// transcode video asynchronously and report progress
job.progress(42);
// call done when finished
done();
// or give an error if error
done(new Error('error transcoding'));
// or pass it a result
done(null, { framerate: 29.5 /* etc... */ });
// If the job throws an unhandled exception it is also handled correctly
throw new Error('some unexpected error');
});
audioQueue.process(function (job, done) {
// transcode audio asynchronously and report progress
job.progress(42);
// call done when finished
done();
// or give an error if error
done(new Error('error transcoding'));
// or pass it a result
done(null, { samplerate: 48000 /* etc... */ });
// If the job throws an unhandled exception it is also handled correctly
throw new Error('some unexpected error');
});
imageQueue.process(function (job, done) {
// transcode image asynchronously and report progress
job.progress(42);
// call done when finished
done();
// or give an error if error
done(new Error('error transcoding'));
// or pass it a result
done(null, { width: 1280, height: 720 /* etc... */ });
// If the job throws an unhandled exception it is also handled correctly
throw new Error('some unexpected error');
});
pdfQueue.process(function (job) {
// Processors can also return promises instead of using the done callback
return pdfAsyncProcessor();
});
videoQueue.add({ video: 'http://example.com/video1.mov' });
audioQueue.add({ audio: 'http://example.com/audio1.mp3' });
imageQueue.add({ image: 'http://example.com/image1.tiff' });
Alternatively, you can return promises instead of using the done callback:
videoQueue.process(function (job) { // don't forget to remove the done callback!
// Simply return a promise
return fetchVideo(job.data.url).then(transcodeVideo);
// Handles promise rejection
return Promise.reject(new Error('error transcoding'));
// Passes the value the promise is resolved with to the "completed" event
return Promise.resolve({ framerate: 29.5 /* etc... */ });
// If the job throws an unhandled exception it is also handled correctly
throw new Error('some unexpected error');
// same as
return Promise.reject(new Error('some unexpected error'));
});
The process function can also be run in a separate process. This has several advantages:
In order to use this feature just create a separate file with the processor:
// processor.js
module.exports = function (job) {
// Do some heavy work
return Promise.resolve(result);
}
And define the processor like this:
// Single process:
queue.process('/path/to/my/processor.js');
// You can use concurrency as well:
queue.process(5, '/path/to/my/processor.js');
// and named processors:
queue.process('my processor', 5, '/path/to/my/processor.js');
A job can be added to a queue and processed repeatedly according to a cron specification:
paymentsQueue.process(function (job) {
// Check payments
});
// Repeat payment job once every day at 3:15 (am)
paymentsQueue.add(paymentsData, { repeat: { cron: '15 3 * * *' } });
As a tip, check your expressions here to verify they are correct: cron expression generator
A queue can be paused and resumed globally (pass true to pause processing for
just this worker):
queue.pause().then(function () {
// queue is paused now
});
queue.resume().then(function () {
// queue is resumed now
})
A queue emits some useful events, for example...
.on('completed', function (job, result) {
// Job completed with output result!
})
For more information on events, including the full list of events that are fired, check out the Events reference
Queues are cheap, so if you need many of them just create new ones with different names:
const userJohn = new Queue('john');
const userLisa = new Queue('lisa');
.
.
.
However every queue instance will require new redis connections, check how to reuse connections or you can also use named processors to achieve a similar result.
NOTE: From version 3.2.0 and above it is recommended to use threaded processors instead.
Queues are robust and can be run in parallel in several threads or processes without any risk of hazards or queue corruption. Check this simple example using cluster to parallelize jobs across processes:
const Queue = require('bull');
const cluster = require('cluster');
const numWorkers = 8;
const queue = new Queue('test concurrent queue');
if (cluster.isMaster) {
for (let i = 0; i < numWorkers; i++) {
cluster.fork();
}
cluster.on('online', function (worker) {
// Let's create a few jobs for the queue workers
for (let i = 0; i < 500; i++) {
queue.add({ foo: 'bar' });
};
});
cluster.on('exit', function (worker, code, signal) {
console.log('worker ' + worker.process.pid + ' died');
});
} else {
queue.process(function (job, jobDone) {
console.log('Job done by worker', cluster.worker.id, job.id);
jobDone();
});
}
For the full documentation, check out the reference and common patterns:
If you see anything that could use more docs, please submit a pull request!
The queue aims for an "at least once" working strategy. This means that in some situations, a job could be processed more than once. This mostly happens when a worker fails to keep a lock for a given job during the total duration of the processing.
When a worker is processing a job it will keep the job "locked" so other workers can't process it.
It's important to understand how locking works to prevent your jobs from losing their lock - becoming stalled -
and being restarted as a result. Locking is implemented internally by creating a lock for lockDuration on interval
lockRenewTime (which is usually half lockDuration). If lockDuration elapses before the lock can be renewed,
the job will be considered stalled and is automatically restarted; it will be double processed. This can happen when:
lockDuration setting (with the tradeoff being that it will take longer to recognize a real stalled job).As such, you should always listen for the stalled event and log this to your error monitoring system, as this means your jobs are likely getting double-processed.
As a safeguard so problematic jobs won't get restarted indefinitely (e.g. if the job processor always crashes its Node process), jobs will be recovered from a stalled state a maximum of maxStalledCount times (default: 1).