bull vs agenda vs bee-queue vs bree vs kue vs node-resque
Node.js 任務排程與佇列管理
bullagendabee-queuebreekuenode-resque類似パッケージ:

Node.js 任務排程與佇列管理

這些 npm 套件主要用於 Node.js 環境中的任務排程和佇列管理,幫助開發者有效地處理背景任務和排程作業。它們各自提供不同的功能和設計理念,適用於不同的使用場景,讓開發者能夠根據需求選擇合適的工具。

npmのダウンロードトレンド

3 年

GitHub Starsランキング

統計詳細

パッケージ
ダウンロード数
Stars
サイズ
Issues
公開日時
ライセンス
bull2,245,94716,262309 kB1472年前MIT
agenda228,6899,712310 kB1910時間前MIT
bee-queue67,7854,039107 kB461年前MIT
bree42,9493,29692 kB308ヶ月前MIT
kue31,0799,418-2869年前MIT
node-resque26,4421,415720 kB243ヶ月前Apache-2.0

機能比較: bull vs agenda vs bee-queue vs bree vs kue vs node-resque

性能

  • bull:

    Bull 是一個高性能的任務佇列,支持多種高級功能,如重試和優先級,適合需要高可靠性的應用。

  • agenda:

    Agenda 是一個輕量級的任務排程工具,專注於簡單的排程需求,適合小型應用。

  • bee-queue:

    Bee-Queue 設計上注重性能,能夠處理高吞吐量的任務,適合需要快速處理的場景。

  • bree:

    Bree 提供了高效的任務排程功能,支持多種排程策略,適合複雜的任務需求。

  • kue:

    Kue 提供了良好的性能,但在高負載下可能會遇到瓶頸,適合中小型應用。

  • node-resque:

    Node-Resque 提供了靈活的性能選擇,適合需要多種後端的應用。

使用場景

  • bull:

    適合需要高可靠性和持久性的應用,如支付處理系統。

  • agenda:

    適合需要簡單任務排程的應用,如定時發送郵件。

  • bee-queue:

    適合需要高效處理大量任務的應用,如即時消息處理。

  • bree:

    適合需要靈活排程的應用,如定期數據備份。

  • kue:

    適合需要監控和管理任務的應用,如任務管理系統。

  • node-resque:

    適合需要多樣化存儲解決方案的應用,如分佈式任務處理。

擴展性

  • bull:

    Bull 提供了強大的擴展性,支持多種高級功能和自定義任務處理。

  • agenda:

    Agenda 的擴展性有限,主要適合簡單的排程需求。

  • bee-queue:

    Bee-Queue 提供了基本的擴展性,但不支持複雜的任務依賴。

  • bree:

    Bree 提供了良好的擴展性,支持多種排程策略和任務類型。

  • kue:

    Kue 的擴展性較好,支持自定義任務類型和處理邏輯。

  • node-resque:

    Node-Resque 提供了靈活的擴展性,支持多種後端和自定義任務處理。

學習曲線

  • bull:

    Bull 的功能強大,但學習曲線較陡,需要一定的時間來熟悉。

  • agenda:

    Agenda 的學習曲線較平緩,適合新手快速上手。

  • bee-queue:

    Bee-Queue 的 API 簡單明瞭,易於理解和使用。

  • bree:

    Bree 的功能較多,學習曲線稍陡,但提供了豐富的文檔支持。

  • kue:

    Kue 的學習曲線較平緩,適合中小型應用的開發者。

  • node-resque:

    Node-Resque 的學習曲線較為平緩,適合需要靈活性的開發者。

維護性

  • bull:

    Bull 的維護性較高,但由於功能複雜,可能需要更多的文檔支持。

  • agenda:

    Agenda 的維護性較好,因為它的設計簡單且易於理解。

  • bee-queue:

    Bee-Queue 的代碼簡潔,維護性良好,適合快速迭代。

  • bree:

    Bree 提供了良好的維護性,代碼結構清晰,易於擴展。

  • kue:

    Kue 的維護性較好,但在高負載下可能需要優化。

  • node-resque:

    Node-Resque 的維護性良好,支持多種後端,適合長期運行的應用。

選び方: bull vs agenda vs bee-queue vs bree vs kue vs node-resque

  • bull:

    Bull 是一個功能強大的任務佇列,適合需要高可靠性和持久性的應用。它支持重試、延遲和優先級等功能,適合複雜的任務處理需求。

  • agenda:

    如果你需要一個簡單易用的任務排程工具,並且希望能夠使用 MongoDB 作為後端存儲,選擇 Agenda 是不錯的選擇。它的 API 設計直觀,適合快速上手。

  • bee-queue:

    當你需要一個高效且輕量的佇列系統,特別是針對高吞吐量的任務處理時,Bee-Queue 是理想的選擇。它專注於性能,並且使用 Redis 作為後端。

  • bree:

    如果你需要一個功能強大且具有靈活性的任務排程工具,Bree 提供了豐富的功能,如定時任務、延遲任務等,並且支持多種後端存儲選項。

  • kue:

    Kue 提供了一個簡單的 API 來處理任務佇列,並且具有可視化的管理界面,適合需要監控和管理任務的應用。

  • node-resque:

    如果你需要一個支持多種後端(如 Redis 和 MongoDB)的任務排程和佇列管理工具,Node-Resque 是一個靈活的選擇,適合需要多樣化存儲解決方案的情況。

bull のREADME




The fastest, most reliable, Redis-based queue for Node.
Carefully written for rock solid stability and atomicity.


Sponsors · Features · UIs · Install · Quick Guide · Documentation

Check the new Guide!


🚀 Sponsors 🚀

Dragonfly Dragonfly is a new Redis™ drop-in replacement that is fully compatible with BullMQ and brings some important advantages over Redis™ such as massive better performance by utilizing all CPU cores available and faster and more memory efficient data structures. Read more here on how to use it with BullMQ.

📻 News and updates

Bull is currently in maintenance mode, we are only fixing bugs. For new features check BullMQ, a modern rewritten implementation in Typescript. You are still very welcome to use Bull if it suits your needs, which is a safe, battle tested library.

Follow me on Twitter for other important news and updates.

🛠 Tutorials

You can find tutorials and news in this blog: https://blog.taskforce.sh/


Used by

Bull is popular among large and small organizations, like the following ones:

Atlassian Autodesk Mozilla Nest Salesforce


Official FrontEnd

Taskforce.sh, Inc

Supercharge your queues with a professional front end:

  • Get a complete overview of all your queues.
  • Inspect jobs, search, retry, or promote delayed jobs.
  • Metrics and statistics.
  • and many more features.

Sign up at Taskforce.sh


Bull Features

  • Minimal CPU usage due to a polling-free design.
  • Robust design based on Redis.
  • Delayed jobs.
  • Schedule and repeat jobs according to a cron specification.
  • Rate limiter for jobs.
  • Retries.
  • Priority.
  • Concurrency.
  • Pause/resume—globally or locally.
  • Multiple job types per queue.
  • Threaded (sandboxed) processing functions.
  • Automatic recovery from process crashes.

And coming up on the roadmap...

  • Job completion acknowledgement (you can use the message queue pattern in the meantime).
  • Parent-child jobs relationships.

UIs

There are a few third-party UIs that you can use for monitoring:

BullMQ

Bull v3

Bull <= v2


Monitoring & Alerting


Feature Comparison

Since there are a few job queue solutions, here is a table comparing them:

FeatureBullMQ-ProBullMQBullKueBeeAgenda
Backendredisredisredisredisredismongo
Observables✓
Group Rate Limit✓
Group Support✓
Batches Support✓
Parent/Child Dependencies✓✓
Priorities✓✓✓✓✓
Concurrency✓✓✓✓✓✓
Delayed jobs✓✓✓✓✓
Global events✓✓✓✓
Rate Limiter✓✓✓
Pause/Resume✓✓✓✓
Sandboxed worker✓✓✓
Repeatable jobs✓✓✓✓
Atomic ops✓✓✓✓
Persistence✓✓✓✓✓✓
UI✓✓✓✓✓
Optimized forJobs / MessagesJobs / MessagesJobs / MessagesJobsMessagesJobs

Install

npm install bull --save

or

yarn add bull

Requirements: Bull requires a Redis version greater than or equal to 2.8.18.

Typescript Definitions

npm install @types/bull --save-dev
yarn add --dev @types/bull

Definitions are currently maintained in the DefinitelyTyped repo.

Contributing

We welcome all types of contributions, either code fixes, new features or doc improvements. Code formatting is enforced by prettier. For commits please follow conventional commits convention. All code must pass lint rules and test suites before it can be merged into develop.


Quick Guide

Basic Usage

const Queue = require('bull');

const videoQueue = new Queue('video transcoding', 'redis://127.0.0.1:6379');
const audioQueue = new Queue('audio transcoding', { redis: { port: 6379, host: '127.0.0.1', password: 'foobared' } }); // Specify Redis connection using object
const imageQueue = new Queue('image transcoding');
const pdfQueue = new Queue('pdf transcoding');

videoQueue.process(function (job, done) {

  // job.data contains the custom data passed when the job was created
  // job.id contains id of this job.

  // transcode video asynchronously and report progress
  job.progress(42);

  // call done when finished
  done();

  // or give an error if error
  done(new Error('error transcoding'));

  // or pass it a result
  done(null, { framerate: 29.5 /* etc... */ });

  // If the job throws an unhandled exception it is also handled correctly
  throw new Error('some unexpected error');
});

audioQueue.process(function (job, done) {
  // transcode audio asynchronously and report progress
  job.progress(42);

  // call done when finished
  done();

  // or give an error if error
  done(new Error('error transcoding'));

  // or pass it a result
  done(null, { samplerate: 48000 /* etc... */ });

  // If the job throws an unhandled exception it is also handled correctly
  throw new Error('some unexpected error');
});

imageQueue.process(function (job, done) {
  // transcode image asynchronously and report progress
  job.progress(42);

  // call done when finished
  done();

  // or give an error if error
  done(new Error('error transcoding'));

  // or pass it a result
  done(null, { width: 1280, height: 720 /* etc... */ });

  // If the job throws an unhandled exception it is also handled correctly
  throw new Error('some unexpected error');
});

pdfQueue.process(function (job) {
  // Processors can also return promises instead of using the done callback
  return pdfAsyncProcessor();
});

videoQueue.add({ video: 'http://example.com/video1.mov' });
audioQueue.add({ audio: 'http://example.com/audio1.mp3' });
imageQueue.add({ image: 'http://example.com/image1.tiff' });

Using promises

Alternatively, you can return promises instead of using the done callback:

videoQueue.process(function (job) { // don't forget to remove the done callback!
  // Simply return a promise
  return fetchVideo(job.data.url).then(transcodeVideo);

  // Handles promise rejection
  return Promise.reject(new Error('error transcoding'));

  // Passes the value the promise is resolved with to the "completed" event
  return Promise.resolve({ framerate: 29.5 /* etc... */ });

  // If the job throws an unhandled exception it is also handled correctly
  throw new Error('some unexpected error');
  // same as
  return Promise.reject(new Error('some unexpected error'));
});

Separate processes

The process function can also be run in a separate process. This has several advantages:

  • The process is sandboxed so if it crashes it does not affect the worker.
  • You can run blocking code without affecting the queue (jobs will not stall).
  • Much better utilization of multi-core CPUs.
  • Less connections to redis.

In order to use this feature just create a separate file with the processor:

// processor.js
module.exports = function (job) {
  // Do some heavy work

  return Promise.resolve(result);
}

And define the processor like this:

// Single process:
queue.process('/path/to/my/processor.js');

// You can use concurrency as well:
queue.process(5, '/path/to/my/processor.js');

// and named processors:
queue.process('my processor', 5, '/path/to/my/processor.js');

Repeated jobs

A job can be added to a queue and processed repeatedly according to a cron specification:

  paymentsQueue.process(function (job) {
    // Check payments
  });

  // Repeat payment job once every day at 3:15 (am)
  paymentsQueue.add(paymentsData, { repeat: { cron: '15 3 * * *' } });

As a tip, check your expressions here to verify they are correct: cron expression generator

Pause / Resume

A queue can be paused and resumed globally (pass true to pause processing for just this worker):

queue.pause().then(function () {
  // queue is paused now
});

queue.resume().then(function () {
  // queue is resumed now
})

Events

A queue emits some useful events, for example...

.on('completed', function (job, result) {
  // Job completed with output result!
})

For more information on events, including the full list of events that are fired, check out the Events reference

Queues performance

Queues are cheap, so if you need many of them just create new ones with different names:

const userJohn = new Queue('john');
const userLisa = new Queue('lisa');
.
.
.

However every queue instance will require new redis connections, check how to reuse connections or you can also use named processors to achieve a similar result.

Cluster support

NOTE: From version 3.2.0 and above it is recommended to use threaded processors instead.

Queues are robust and can be run in parallel in several threads or processes without any risk of hazards or queue corruption. Check this simple example using cluster to parallelize jobs across processes:

const Queue = require('bull');
const cluster = require('cluster');

const numWorkers = 8;
const queue = new Queue('test concurrent queue');

if (cluster.isMaster) {
  for (let i = 0; i < numWorkers; i++) {
    cluster.fork();
  }

  cluster.on('online', function (worker) {
    // Let's create a few jobs for the queue workers
    for (let i = 0; i < 500; i++) {
      queue.add({ foo: 'bar' });
    };
  });

  cluster.on('exit', function (worker, code, signal) {
    console.log('worker ' + worker.process.pid + ' died');
  });
} else {
  queue.process(function (job, jobDone) {
    console.log('Job done by worker', cluster.worker.id, job.id);
    jobDone();
  });
}

Documentation

For the full documentation, check out the reference and common patterns:

  • Guide — Your starting point for developing with Bull.
  • Reference — Reference document with all objects and methods available.
  • Patterns — a set of examples for common patterns.
  • License — the Bull license—it's MIT.

If you see anything that could use more docs, please submit a pull request!


Important Notes

The queue aims for an "at least once" working strategy. This means that in some situations, a job could be processed more than once. This mostly happens when a worker fails to keep a lock for a given job during the total duration of the processing.

When a worker is processing a job it will keep the job "locked" so other workers can't process it.

It's important to understand how locking works to prevent your jobs from losing their lock - becoming stalled - and being restarted as a result. Locking is implemented internally by creating a lock for lockDuration on interval lockRenewTime (which is usually half lockDuration). If lockDuration elapses before the lock can be renewed, the job will be considered stalled and is automatically restarted; it will be double processed. This can happen when:

  1. The Node process running your job processor unexpectedly terminates.
  2. Your job processor was too CPU-intensive and stalled the Node event loop, and as a result, Bull couldn't renew the job lock (see #488 for how we might better detect this). You can fix this by breaking your job processor into smaller parts so that no single part can block the Node event loop. Alternatively, you can pass a larger value for the lockDuration setting (with the tradeoff being that it will take longer to recognize a real stalled job).

As such, you should always listen for the stalled event and log this to your error monitoring system, as this means your jobs are likely getting double-processed.

As a safeguard so problematic jobs won't get restarted indefinitely (e.g. if the job processor always crashes its Node process), jobs will be recovered from a stalled state a maximum of maxStalledCount times (default: 1).