Node.js Job Queues and Task Scheduling
These libraries enable Node.js applications to handle background tasks, scheduled jobs, and heavy processing without blocking the main event loop. They provide structures for defining work, storing job state, and retrying failed tasks. While some rely on Redis for fast in-memory storage, others use MongoDB for durability or worker threads for local execution. Choosing the right tool depends on your infrastructure, reliability needs, and whether you require distributed processing or simple local scheduling.
Npm Package Weekly Downloads Trend
Github Stars Ranking
Stat Detail
Node.js Job Queues: Architecture and Implementation Compared
Handling background tasks is a common requirement in modern web applications. Whether you are sending emails, processing images, or running scheduled maintenance, you need a reliable way to manage work without blocking your main server. The packages agenda, bee-queue, bree, bull, kue, and node-resque all solve this problem, but they use different storage engines and architectural patterns. Let's compare how they handle job definition, execution, and storage.
🗄️ Storage Engine: Redis vs MongoDB vs Worker Threads
The underlying storage determines how jobs survive server restarts and how multiple servers share work.
bull and bee-queue rely on Redis.
- Redis is fast and in-memory, making these queues great for high throughput.
- Jobs are lost if Redis persists incorrectly, though both support persistence settings.
// bull: Redis connection
const Queue = require('bull');
const queue = new Queue('email', 'redis://127.0.0.1:6379');
// bee-queue: Redis connection
const Queue = require('bee-queue');
const queue = new Queue('email', { redis: { host: '127.0.0.1', port: 6379 } });
agenda uses MongoDB.
- Jobs are stored as documents, so they survive restarts easily.
- You can query jobs using standard Mongo queries.
// agenda: MongoDB connection
const Agenda = require('agenda');
const agenda = new Agenda({ db: { address: 'mongodb://127.0.0.1:27017/agenda' } });
bree uses Worker Threads locally.
- It does not require an external database by default.
- Jobs are defined as files and run in separate threads within the same process.
// bree: Local configuration
const Bree = require('bree');
const bree = new Bree({
jobs: [
{ name: 'daily-cleanup', interval: '0 0 * * *', path: './jobs/cleanup.js' }
]
});
kue and node-resque also use Redis.
- They follow older patterns of Redis list management.
kueis no longer maintained, so new projects should avoid it.
// kue: Redis connection (Legacy)
const kue = require('kue');
const queue = kue.createQueue();
// node-resque: Redis connection
const { Queue, Worker } = require('node-resque');
const queue = new Queue({ connection: { host: '127.0.0.1', port: 6379 } });
📥 Defining and Processing Jobs
Each library has a different way to define what work needs to be done and how to run it.
bull separates job addition from processing.
- You define a processor function that runs when a job is available.
- It supports concurrency limits directly in the processor.
// bull: Process jobs
queue.process('send-email', async (job) => {
await sendEmail(job.data.to);
});
// bull: Add job
queue.add('send-email', { to: 'user@example.com' });
agenda defines jobs by name and schedules them.
- You define the logic once and can trigger it immediately or later.
- It feels like a database collection of tasks.
// agenda: Define job
agenda.define('send-email', async (job) => {
await sendEmail(job.attrs.data.to);
});
// agenda: Schedule job
agenda.now('send-email', { to: 'user@example.com' });
bee-queue is similar to Bull but with a simpler API.
- It focuses on minimal configuration and speed.
- The processing model is straightforward.
// bee-queue: Process jobs
queue.process(async (job) => {
await sendEmail(job.data.to);
});
// bee-queue: Create job
const job = queue.createJob({ to: 'user@example.com' });
await job.save();
bree uses file-based jobs.
- You write a script for each job type.
- Bree manages the timing and execution of these scripts.
// bree: Job file (jobs/cleanup.js)
module.exports = async (config) => {
await cleanupDatabase();
};
// bree: Start manager
await bree.start();
kue uses a creation chain pattern.
- You build the job object step-by-step before saving.
- This is verbose compared to modern alternatives.
// kue: Create job (Legacy)
const job = queue.create('send-email', { to: 'user@example.com' });
job.save();
// kue: Process
queue.process('send-email', (job, done) => {
sendEmail(job.data.to, done);
});
node-resque follows the classic Resque model.
- You perform jobs based on class names.
- It requires a separate worker process to listen for jobs.
// node-resque: Perform method
class SendEmail {
async perform(to) {
await sendEmail(to);
}
}
// node-resque: Enqueue
await queue.enqueue('high', 'SendEmail', ['user@example.com']);
🕒 Scheduling and Recurrence
Some libraries handle cron jobs and delayed tasks better than others.
agenda excels at scheduling.
- It has built-in support for
every,schedule, andcronpatterns. - You can query scheduled jobs easily via MongoDB.
// agenda: Recurring job
agenda.every('10 minutes', 'clear-cache');
agenda.schedule('in 2 hours', 'send-reminder', { userId: 123 });
bull supports delays and repeats.
- You can set a job to run after a delay or on a cron schedule.
- It relies on Redis timers for this functionality.
// bull: Delayed job
queue.add('notify', { id: 1 }, { delay: 5000 });
// bull: Repeated job
queue.add('report', {}, { repeat: { cron: '0 0 * * *' } });
bree is built for cron-style jobs.
- It uses human-readable intervals or cron strings in config.
- It is designed specifically for time-based execution.
// bree: Cron config
const bree = new Bree({
jobs: [
{ name: 'backup', cron: '0 0 * * *', path: './jobs/backup.js' }
]
});
bee-queue, kue, and node-resque have limited scheduling.
- They focus on immediate queue processing.
- You often need an external scheduler (like
node-cron) to add jobs at specific times.
// bee-queue: Delay only
const job = queue.createJob({}).delay(5000);
await job.save();
// kue: Delay only
queue.create('task', {}).delay(5000).save();
⚠️ Maintenance and Status
Choosing a library also means choosing its long-term support.
bull and agenda are actively maintained.
- They receive regular updates and security patches.
- They have large communities and many tutorials available.
bree is maintained and focused on simplicity.
- It is a good choice for newer projects that want less infrastructure.
- It avoids external dependencies like Redis.
bee-queue has slower maintenance cycles.
- It is stable but updates are less frequent.
- Suitable for stable projects that do not need cutting-edge features.
kue is deprecated.
- It should not be used in new development.
- Security vulnerabilities may not be fixed.
node-resque is stable but legacy.
- It is best for maintaining older systems.
- New projects might find
bullmore aligned with modern Node.js practices.
📊 Summary: Key Differences
| Feature | bull | agenda | bree | bee-queue | kue | node-resque |
|---|---|---|---|---|---|---|
| Storage | Redis | MongoDB | Local/Threads | Redis | Redis | Redis |
| Scheduling | Built-in | Built-in | Built-in | Limited | Limited | Limited |
| Status | Active | Active | Active | Stable | Deprecated | Stable |
| Complexity | Medium | Medium | Low | Low | High | Medium |
| Best For | Distributed | Persistence | Cron/Local | Simplicity | Legacy | Ruby Migration |
💡 The Big Picture
bull is the safe bet for most distributed Node.js applications. It balances features, performance, and community support. If you need heavy scheduling and already use MongoDB, agenda is a strong contender. For simple cron jobs on a single server, bree removes the need for Redis entirely.
Avoid kue in new projects due to its deprecated status. Use node-resque only if you need compatibility with existing Resque infrastructure. For lightweight Redis needs where Bull feels too heavy, bee-queue remains an option, but verify its maintenance fits your risk tolerance.
Final Thought: The best queue is the one that matches your infrastructure. If you have Redis, use Bull. If you have Mongo, use Agenda. If you have neither, use Bree.
How to Choose: agenda vs bee-queue vs bree vs bull vs kue vs node-resque
- agenda:
Choose
agendaif your stack already uses MongoDB and you need robust scheduling features like cron jobs or recurring tasks. It stores job data in Mongo collections, making it easy to query and inspect jobs using standard database tools. It is well-suited for applications that prioritize data persistence over raw throughput. - bee-queue:
Choose
bee-queueif you need a lightweight, Redis-based queue with a focus on simplicity and performance. It has a smaller footprint than Bull and avoids complex Lua scripts. It is a good fit for projects that want Redis storage without the heavier feature set of Bull, though maintenance activity is slower. - bree:
Choose
breeif you need to run cron jobs or scheduled tasks using worker threads within a single Node.js process. It is designed for simplicity and does not require Redis or MongoDB. It is ideal for single-server deployments where you want to isolate jobs in separate threads to avoid blocking the main loop. - bull:
Choose
bullif you need a mature, feature-rich queue system backed by Redis. It supports priorities, retries, delays, and event listeners out of the box. It is the standard choice for distributed systems requiring high reliability and concurrency, though you should also evaluate BullMQ for modern Redis Stream support. - kue:
Do not choose
kuefor new projects. It is deprecated and no longer maintained. While it was popular historically for its Redis backend and UI, it lacks modern security updates and bug fixes. Existing projects should plan migration tobulloragenda. - node-resque:
Choose
node-resqueif you are migrating from a Ruby on Rails Resque background job system and want to maintain compatibility. It follows the classic Resque pattern strictly. It is stable but less feature-rich than Bull for modern Node.js specific workflows.
Popular Comparisons
README for agenda
Agenda
A light-weight job scheduling library for Node.js
Migrating from v5? See the Migration Guide for all breaking changes.
Agenda 6.x
Agenda 6.x is a complete TypeScript rewrite with a focus on modularity and flexibility:
-
Pluggable storage backends - Choose from MongoDB, PostgreSQL, Redis, or implement your own. Each backend is a separate package - install only what you need.
-
Pluggable notification channels - Move beyond polling with real-time job notifications via Redis, PostgreSQL LISTEN/NOTIFY, or other pub/sub systems. Jobs get processed immediately when saved, not on the next poll cycle.
-
Modern stack - ESM-only, Node.js 18+, full TypeScript with strict typing.
See the 6.x Roadmap for details and progress.
Features
- Minimal overhead job scheduling
- Pluggable storage backends (MongoDB, PostgreSQL, Redis)
- TypeScript support with full typing
- Scheduling via cron or human-readable syntax
- Configurable concurrency and locking
- Real-time job notifications (optional)
- Sandboxed worker execution via fork mode
- TypeScript decorators for class-based job definitions
Installation
Install the core package and your preferred backend:
# For MongoDB
npm install agenda @agendajs/mongo-backend
# For PostgreSQL
npm install agenda @agendajs/postgres-backend
# For Redis
npm install agenda @agendajs/redis-backend
Requirements:
- Node.js 18+
- Database of your choice (MongoDB 4+, PostgreSQL, or Redis)
Quick Start
import { Agenda } from 'agenda';
import { MongoBackend } from '@agendajs/mongo-backend';
const agenda = new Agenda({
backend: new MongoBackend({ address: 'mongodb://localhost/agenda' })
});
// Define a job
agenda.define('send email', async (job) => {
const { to, subject } = job.attrs.data;
await sendEmail(to, subject);
});
// Start processing
await agenda.start();
// Schedule jobs
await agenda.every('1 hour', 'send email', { to: 'user@example.com', subject: 'Hello' });
await agenda.schedule('in 5 minutes', 'send email', { to: 'admin@example.com', subject: 'Report' });
await agenda.now('send email', { to: 'support@example.com', subject: 'Urgent' });
Official Backend Packages
| Package | Backend | Notifications | Install |
|---|---|---|---|
@agendajs/mongo-backend | MongoDB | Polling only | npm install @agendajs/mongo-backend |
@agendajs/postgres-backend | PostgreSQL | LISTEN/NOTIFY | npm install @agendajs/postgres-backend |
@agendajs/redis-backend | Redis | Pub/Sub | npm install @agendajs/redis-backend |
Backend Capabilities
| Backend | Storage | Notifications | Notes |
|---|---|---|---|
MongoDB (MongoBackend) | ✅ | ❌ | Storage only. Combine with external notification channel for real-time. |
PostgreSQL (PostgresBackend) | ✅ | ✅ | Full backend. Uses LISTEN/NOTIFY for notifications. |
Redis (RedisBackend) | ✅ | ✅ | Full backend. Uses Pub/Sub for notifications. |
| InMemoryNotificationChannel | ❌ | ✅ | Notifications only. For single-process/testing. |
Backend Configuration
MongoDB
import { Agenda } from 'agenda';
import { MongoBackend } from '@agendajs/mongo-backend';
// Via connection string
const agenda = new Agenda({
backend: new MongoBackend({ address: 'mongodb://localhost/agenda' })
});
// Via existing MongoDB connection
const agenda = new Agenda({
backend: new MongoBackend({ mongo: existingDb })
});
// With options
const agenda = new Agenda({
backend: new MongoBackend({
mongo: db,
collection: 'jobs' // Collection name (default: 'agendaJobs')
}),
processEvery: '30 seconds', // Job polling interval
maxConcurrency: 20, // Max concurrent jobs
defaultConcurrency: 5 // Default per job type
});
PostgreSQL
import { Agenda } from 'agenda';
import { PostgresBackend } from '@agendajs/postgres-backend';
const agenda = new Agenda({
backend: new PostgresBackend({
connectionString: 'postgresql://user:pass@localhost:5432/mydb'
})
});
Redis
import { Agenda } from 'agenda';
import { RedisBackend } from '@agendajs/redis-backend';
const agenda = new Agenda({
backend: new RedisBackend({
connectionString: 'redis://localhost:6379'
})
});
Real-Time Notifications
For faster job processing across distributed systems:
import { Agenda, InMemoryNotificationChannel } from 'agenda';
import { MongoBackend } from '@agendajs/mongo-backend';
const agenda = new Agenda({
backend: new MongoBackend({ mongo: db }),
notificationChannel: new InMemoryNotificationChannel()
});
Mixing Storage and Notification Backends
You can use MongoDB for storage while using a different system for real-time notifications:
import { Agenda } from 'agenda';
import { MongoBackend } from '@agendajs/mongo-backend';
import { RedisBackend } from '@agendajs/redis-backend';
// MongoDB for storage + Redis for real-time notifications
const redisBackend = new RedisBackend({ connectionString: 'redis://localhost:6379' });
const agenda = new Agenda({
backend: new MongoBackend({ mongo: db }),
notificationChannel: redisBackend.notificationChannel
});
This is useful when you want MongoDB's proven durability and flexible queries for job storage, but need faster real-time notifications across multiple processes.
API Overview
Defining Jobs
// Simple async handler
agenda.define('my-job', async (job) => {
console.log('Processing:', job.attrs.data);
});
// With options
agenda.define('my-job', async (job) => { /* ... */ }, {
concurrency: 10,
lockLimit: 5,
lockLifetime: 10 * 60 * 1000, // 10 minutes
priority: 'high'
});
Defining Jobs with Decorators
For a class-based approach, use TypeScript decorators:
import { JobsController, Define, Every, registerJobs, Job } from 'agenda';
@JobsController({ namespace: 'email' })
class EmailJobs {
@Define({ concurrency: 5 })
async sendWelcome(job: Job<{ userId: string }>) {
console.log('Sending welcome to:', job.attrs.data.userId);
}
@Every('1 hour')
async cleanupBounced(job: Job) {
console.log('Cleaning up bounced emails');
}
}
registerJobs(agenda, [new EmailJobs()]);
await agenda.start();
// Schedule using namespaced name
await agenda.now('email.sendWelcome', { userId: '123' });
See Decorators Documentation for full details.
Scheduling Jobs
// Run immediately
await agenda.now('my-job', { userId: '123' });
// Run at specific time
await agenda.schedule('tomorrow at noon', 'my-job', data);
await agenda.schedule(new Date('2024-12-25'), 'my-job', data);
// Run repeatedly
await agenda.every('5 minutes', 'my-job');
await agenda.every('0 * * * *', 'my-job'); // Cron syntax
Job Control
// Cancel jobs matching a filter (removes from database)
await agenda.cancel({ name: 'my-job' });
await agenda.cancel({ name: 'my-job', data: { userId: 123 } });
// Cancel ALL jobs unconditionally
await agenda.cancelAll();
// Disable/enable jobs globally (by query)
await agenda.disable({ name: 'my-job' }); // Disable all jobs matching query
await agenda.enable({ name: 'my-job' }); // Enable all jobs matching query
// Disable/enable individual jobs
const job = await agenda.create('my-job', data);
job.disable();
await job.save();
// Progress tracking
agenda.define('long-job', async (job) => {
for (let i = 0; i <= 100; i += 10) {
await doWork();
await job.touch(i); // Report progress 0-100
}
});
Stopping / Draining
// Stop immediately - unlocks running jobs so other workers can pick them up
await agenda.stop();
// Drain - waits for running jobs to complete before stopping
await agenda.drain();
// Drain with timeout (30 seconds) - for cloud platforms with shutdown deadlines
const result = await agenda.drain(30000);
if (result.timedOut) {
console.log(`${result.running} jobs still running after timeout`);
}
// Drain with AbortSignal - for external control
const controller = new AbortController();
setTimeout(() => controller.abort(), 30000);
await agenda.drain({ signal: controller.signal });
Use drain() for graceful shutdowns where you want in-progress jobs to finish their work.
Events
agenda.on('start', (job) => console.log('Job started:', job.attrs.name));
agenda.on('complete', (job) => console.log('Job completed:', job.attrs.name));
agenda.on('success', (job) => console.log('Job succeeded:', job.attrs.name));
agenda.on('fail', (err, job) => console.log('Job failed:', job.attrs.name, err));
// Job-specific events
agenda.on('start:send email', (job) => { /* ... */ });
agenda.on('fail:send email', (err, job) => { /* ... */ });
Use fail listeners to capture richer error context, such as stack traces,
without storing large payloads in job.attrs.failReason:
agenda.on('fail', async (err, job) => {
await saveJobError({
jobId: job.attrs._id,
jobName: job.attrs.name,
message: err.message,
stack: err.stack
});
});
Custom Backend
For databases other than MongoDB, PostgreSQL, or Redis, implement AgendaBackend:
import { AgendaBackend, JobRepository } from 'agenda';
class SQLiteBackend implements AgendaBackend {
readonly repository: JobRepository;
readonly notificationChannel = undefined; // Or implement NotificationChannel
async connect() { /* ... */ }
async disconnect() { /* ... */ }
}
const agenda = new Agenda({
backend: new SQLiteBackend({ path: './jobs.db' })
});
See Custom Backend Driver for details.
Documentation
- Full Documentation
- Migration Guide (v5 to v6)
- Custom Backend Driver
- TypeScript Decorators
- API Reference
Related Packages
Official Backend Packages:
- @agendajs/mongo-backend - MongoDB backend
- @agendajs/postgres-backend - PostgreSQL backend with LISTEN/NOTIFY
- @agendajs/redis-backend - Redis backend with Pub/Sub
Tools:
- agendash - Web dashboard for Agenda
- agenda-rest - REST API for Agenda
License
MIT