Managing Background Jobs in Node.js Applications
agenda, bee-queue, bull, bullmq, and kue are libraries designed to handle background tasks and job queues in Node.js. They allow developers to offload time-consuming work — such as sending emails, processing images, or generating reports — from the main application thread. While bull, bullmq, bee-queue, and kue rely on Redis for storage and messaging, agenda uses MongoDB. Choosing the right tool depends on your existing infrastructure, need for persistence, and whether you prioritize modern features or legacy support.
Npm Package Weekly Downloads Trend
Github Stars Ranking
Stat Detail
Managing Background Jobs in Node.js Applications
When building scalable web applications, you often need to run tasks in the background — such as sending welcome emails, processing video uploads, or generating PDF reports. Running these tasks directly in your web server can slow down response times or cause timeouts. The agenda, bee-queue, bull, bullmq, and kue packages solve this by letting you queue jobs and process them separately. Let's compare how they handle setup, job creation, and processing.
⚠️ Important Maintenance Status
Before diving into code, note that some of these libraries are no longer safe for new projects.
kue: Deprecated and archived. Do not use.bee-queue: Deprecated and archived. Do not use.bull: Stable but in maintenance mode. BullMQ is the preferred future.bullmq: Actively maintained. Recommended for Redis users.agenda: Actively maintained. Recommended for MongoDB users.
🔌 Connecting to Storage: Redis vs MongoDB
The biggest difference lies in the database they require. bull, bullmq, bee-queue, and kue need Redis. agenda needs MongoDB.
agenda connects to MongoDB using a connection string.
// agenda: MongoDB connection
const Agenda = require('agenda');
const agenda = new Agenda({ db: { address: 'mongodb://localhost:27017/agenda' } });
bull connects to Redis using a host and port or connection string.
// bull: Redis connection
const Queue = require('bull');
const queue = new Queue('my-queue', { redis: { host: 'localhost', port: 6379 } });
bullmq uses a modern Redis connection option.
// bullmq: Redis connection
const { Queue } = require('bullmq');
const queue = new Queue('my-queue', { connection: { host: 'localhost', port: 6379 } });
bee-queue connects to Redis similarly to bull.
// bee-queue: Redis connection
const Queue = require('bee-queue');
const queue = new Queue('my-queue', { redis: { host: 'localhost', port: 6379 } });
kue creates a queue instance with Redis settings.
// kue: Redis connection
const kue = require('kue');
const queue = kue.createQueue({ redis: { host: 'localhost', port: 6379 } });
📥 Adding Jobs to the Queue
Adding a job looks similar across packages, but the syntax varies slightly. All allow you to pass data payload to the worker.
agenda creates a job with a name and data, then saves it.
// agenda: Create job
const job = agenda.create('send email', { to: 'user@example.com' });
await job.save();
bull adds a job to the queue directly.
// bull: Add job
await queue.add('send email', { to: 'user@example.com' });
bullmq adds jobs similarly but relies on the Queue class.
// bullmq: Add job
await queue.add('send email', { to: 'user@example.com' });
bee-queue creates a job object then saves it.
// bee-queue: Create job
const job = queue.createJob({ to: 'user@example.com' });
await job.save();
kue creates a job with a type and data.
// kue: Create job
const job = queue.create('send email', { to: 'user@example.com' });
job.save();
🏭 Processing Jobs in Workers
Processing defines what happens when a worker picks up a job. This is where your business logic lives.
agenda defines a function linked to a job name.
// agenda: Process job
agenda.define('send email', async (job) => {
console.log(`Sending to ${job.data.to}`);
});
await agenda.start();
bull uses a process method on the queue.
// bull: Process job
queue.process('send email', async (job) => {
console.log(`Sending to ${job.data.to}`);
});
bullmq uses a separate Worker class.
// bullmq: Process job
const { Worker } = require('bullmq');
const worker = new Worker('my-queue', async (job) => {
console.log(`Sending to ${job.data.to}`);
}, { connection: { host: 'localhost', port: 6379 } });
bee-queue registers a process handler.
// bee-queue: Process job
queue.process(async (job) => {
console.log(`Sending to ${job.data.to}`);
});
kue processes jobs by type.
// kue: Process job
queue.process('send email', (job, done) => {
console.log(`Sending to ${job.data.to}`);
done();
});
🛡️ Handling Retries and Failures
Jobs sometimes fail. You need to know how each library handles retries.
agenda sets retry options on the job definition.
// agenda: Retry config
job.attrs.nextRunAt = new Date();
job.attrs.failCount = 0;
// Manual retry logic often required or via plugins
bull supports built-in retry attempts.
// bull: Retry config
await queue.add('task', data, { attempts: 3 });
bullmq supports advanced retry strategies.
// bullmq: Retry config
await queue.add('task', data, { attempts: 3, backoff: { type: 'exponential', delay: 1000 } });
bee-queue allows setting retries on the job.
// bee-queue: Retry config
const job = queue.createJob(data);
job.retries(3);
await job.save();
kue supports attempts on job creation.
// kue: Retry config
const job = queue.create('task', data);
job.attempts(3);
job.save();
📊 Summary Table
| Feature | agenda | bull | bullmq | bee-queue | kue |
|---|---|---|---|---|---|
| Storage | 🍃 MongoDB | 🔴 Redis | 🔴 Redis | 🔴 Redis | 🔴 Redis |
| Status | ✅ Active | ⚠️ Maintenance | ✅ Active | ❌ Deprecated | ❌ Deprecated |
| Worker Model | Defined Function | Queue Process | Separate Worker | Queue Process | Queue Process |
| Retry Support | Manual/Plugin | Built-in | Advanced | Built-in | Built-in |
| Best For | Mongo Stacks | Legacy Redis | Modern Redis | Legacy Light | Legacy Priority |
💡 Final Recommendation
For new projects, your choice mostly comes down to your database. If you already use MongoDB, agenda is a solid choice that keeps your infrastructure simple. If you use Redis or want maximum performance and modern features, bullmq is the clear winner. It is actively developed and designed for today's distributed systems.
Avoid kue and bee-queue entirely for new work. They are deprecated and lack security updates. While bull is still reliable, starting with bullmq saves you a migration later. Choose the tool that matches your stack — but always prioritize maintained software for production systems.
How to Choose: agenda vs bee-queue vs bull vs bullmq vs kue
- agenda:
Choose
agendaif your stack already relies heavily on MongoDB and you need job persistence without adding Redis. It is well-suited for applications where job data must survive server restarts and where you prefer a single database solution. However, be aware that MongoDB locking can impact performance under very high load compared to Redis-based solutions. - bee-queue:
Do not choose
bee-queuefor new projects as it is deprecated and no longer maintained. It was known for being lightweight and fast, but the lack of updates means security vulnerabilities will not be fixed. Only consider this if you are maintaining a legacy system that already depends on it and migration is not currently feasible. - bull:
Choose
bullif you need a stable, battle-tested Redis queue and are not ready to migrate to BullMQ. It has a large ecosystem of plugins and extensive documentation. However, note that development focus has shifted to BullMQ, so long-term projects should plan for eventual migration to the newer library. - bullmq:
Choose
bullmqfor new projects requiring a Redis-based queue with modern features like Redis Streams. It offers better performance, improved event handling, and active maintenance. It is the recommended successor tobulland provides a cleaner API for managing workers and queues in distributed systems. - kue:
Do not choose
kuefor new projects as it is deprecated and archived. It was one of the earliest priority job queues for Node.js but lacks modern features and security updates. Using it today introduces significant risk, and you should evaluatebullmqoragendaas safer alternatives for any new development.
Popular Comparisons
README for agenda
Agenda
A light-weight job scheduling library for Node.js
Migrating from v5? See the Migration Guide for all breaking changes.
Agenda 6.x
Agenda 6.x is a complete TypeScript rewrite with a focus on modularity and flexibility:
-
Pluggable storage backends - Choose from MongoDB, PostgreSQL, Redis, or implement your own. Each backend is a separate package - install only what you need.
-
Pluggable notification channels - Move beyond polling with real-time job notifications via Redis, PostgreSQL LISTEN/NOTIFY, or other pub/sub systems. Jobs get processed immediately when saved, not on the next poll cycle.
-
Modern stack - ESM-only, Node.js 18+, full TypeScript with strict typing.
See the 6.x Roadmap for details and progress.
Features
- Minimal overhead job scheduling
- Pluggable storage backends (MongoDB, PostgreSQL, Redis)
- TypeScript support with full typing
- Scheduling via cron or human-readable syntax
- Configurable concurrency and locking
- Real-time job notifications (optional)
- Sandboxed worker execution via fork mode
- TypeScript decorators for class-based job definitions
Installation
Install the core package and your preferred backend:
# For MongoDB
npm install agenda @agendajs/mongo-backend
# For PostgreSQL
npm install agenda @agendajs/postgres-backend
# For Redis
npm install agenda @agendajs/redis-backend
Requirements:
- Node.js 18+
- Database of your choice (MongoDB 4+, PostgreSQL, or Redis)
Quick Start
import { Agenda } from 'agenda';
import { MongoBackend } from '@agendajs/mongo-backend';
const agenda = new Agenda({
backend: new MongoBackend({ address: 'mongodb://localhost/agenda' })
});
// Define a job
agenda.define('send email', async (job) => {
const { to, subject } = job.attrs.data;
await sendEmail(to, subject);
});
// Start processing
await agenda.start();
// Schedule jobs
await agenda.every('1 hour', 'send email', { to: 'user@example.com', subject: 'Hello' });
await agenda.schedule('in 5 minutes', 'send email', { to: 'admin@example.com', subject: 'Report' });
await agenda.now('send email', { to: 'support@example.com', subject: 'Urgent' });
Official Backend Packages
| Package | Backend | Notifications | Install |
|---|---|---|---|
@agendajs/mongo-backend | MongoDB | Polling only | npm install @agendajs/mongo-backend |
@agendajs/postgres-backend | PostgreSQL | LISTEN/NOTIFY | npm install @agendajs/postgres-backend |
@agendajs/redis-backend | Redis | Pub/Sub | npm install @agendajs/redis-backend |
Backend Capabilities
| Backend | Storage | Notifications | Notes |
|---|---|---|---|
MongoDB (MongoBackend) | ✅ | ❌ | Storage only. Combine with external notification channel for real-time. |
PostgreSQL (PostgresBackend) | ✅ | ✅ | Full backend. Uses LISTEN/NOTIFY for notifications. |
Redis (RedisBackend) | ✅ | ✅ | Full backend. Uses Pub/Sub for notifications. |
| InMemoryNotificationChannel | ❌ | ✅ | Notifications only. For single-process/testing. |
Backend Configuration
MongoDB
import { Agenda } from 'agenda';
import { MongoBackend } from '@agendajs/mongo-backend';
// Via connection string
const agenda = new Agenda({
backend: new MongoBackend({ address: 'mongodb://localhost/agenda' })
});
// Via existing MongoDB connection
const agenda = new Agenda({
backend: new MongoBackend({ mongo: existingDb })
});
// With options
const agenda = new Agenda({
backend: new MongoBackend({
mongo: db,
collection: 'jobs' // Collection name (default: 'agendaJobs')
}),
processEvery: '30 seconds', // Job polling interval
maxConcurrency: 20, // Max concurrent jobs
defaultConcurrency: 5 // Default per job type
});
PostgreSQL
import { Agenda } from 'agenda';
import { PostgresBackend } from '@agendajs/postgres-backend';
const agenda = new Agenda({
backend: new PostgresBackend({
connectionString: 'postgresql://user:pass@localhost:5432/mydb'
})
});
Redis
import { Agenda } from 'agenda';
import { RedisBackend } from '@agendajs/redis-backend';
const agenda = new Agenda({
backend: new RedisBackend({
connectionString: 'redis://localhost:6379'
})
});
Real-Time Notifications
For faster job processing across distributed systems:
import { Agenda, InMemoryNotificationChannel } from 'agenda';
import { MongoBackend } from '@agendajs/mongo-backend';
const agenda = new Agenda({
backend: new MongoBackend({ mongo: db }),
notificationChannel: new InMemoryNotificationChannel()
});
Mixing Storage and Notification Backends
You can use MongoDB for storage while using a different system for real-time notifications:
import { Agenda } from 'agenda';
import { MongoBackend } from '@agendajs/mongo-backend';
import { RedisBackend } from '@agendajs/redis-backend';
// MongoDB for storage + Redis for real-time notifications
const redisBackend = new RedisBackend({ connectionString: 'redis://localhost:6379' });
const agenda = new Agenda({
backend: new MongoBackend({ mongo: db }),
notificationChannel: redisBackend.notificationChannel
});
This is useful when you want MongoDB's proven durability and flexible queries for job storage, but need faster real-time notifications across multiple processes.
API Overview
Defining Jobs
// Simple async handler
agenda.define('my-job', async (job) => {
console.log('Processing:', job.attrs.data);
});
// With options
agenda.define('my-job', async (job) => { /* ... */ }, {
concurrency: 10,
lockLimit: 5,
lockLifetime: 10 * 60 * 1000, // 10 minutes
priority: 'high'
});
Defining Jobs with Decorators
For a class-based approach, use TypeScript decorators:
import { JobsController, Define, Every, registerJobs, Job } from 'agenda';
@JobsController({ namespace: 'email' })
class EmailJobs {
@Define({ concurrency: 5 })
async sendWelcome(job: Job<{ userId: string }>) {
console.log('Sending welcome to:', job.attrs.data.userId);
}
@Every('1 hour')
async cleanupBounced(job: Job) {
console.log('Cleaning up bounced emails');
}
}
registerJobs(agenda, [new EmailJobs()]);
await agenda.start();
// Schedule using namespaced name
await agenda.now('email.sendWelcome', { userId: '123' });
See Decorators Documentation for full details.
Scheduling Jobs
// Run immediately
await agenda.now('my-job', { userId: '123' });
// Run at specific time
await agenda.schedule('tomorrow at noon', 'my-job', data);
await agenda.schedule(new Date('2024-12-25'), 'my-job', data);
// Run repeatedly
await agenda.every('5 minutes', 'my-job');
await agenda.every('0 * * * *', 'my-job'); // Cron syntax
Job Control
// Cancel jobs matching a filter (removes from database)
await agenda.cancel({ name: 'my-job' });
await agenda.cancel({ name: 'my-job', data: { userId: 123 } });
// Cancel ALL jobs unconditionally
await agenda.cancelAll();
// Disable/enable jobs globally (by query)
await agenda.disable({ name: 'my-job' }); // Disable all jobs matching query
await agenda.enable({ name: 'my-job' }); // Enable all jobs matching query
// Disable/enable individual jobs
const job = await agenda.create('my-job', data);
job.disable();
await job.save();
// Progress tracking
agenda.define('long-job', async (job) => {
for (let i = 0; i <= 100; i += 10) {
await doWork();
await job.touch(i); // Report progress 0-100
}
});
Stopping / Draining
// Stop immediately - unlocks running jobs so other workers can pick them up
await agenda.stop();
// Drain - waits for running jobs to complete before stopping
await agenda.drain();
// Drain with timeout (30 seconds) - for cloud platforms with shutdown deadlines
const result = await agenda.drain(30000);
if (result.timedOut) {
console.log(`${result.running} jobs still running after timeout`);
}
// Drain with AbortSignal - for external control
const controller = new AbortController();
setTimeout(() => controller.abort(), 30000);
await agenda.drain({ signal: controller.signal });
Use drain() for graceful shutdowns where you want in-progress jobs to finish their work.
Events
agenda.on('start', (job) => console.log('Job started:', job.attrs.name));
agenda.on('complete', (job) => console.log('Job completed:', job.attrs.name));
agenda.on('success', (job) => console.log('Job succeeded:', job.attrs.name));
agenda.on('fail', (err, job) => console.log('Job failed:', job.attrs.name, err));
// Job-specific events
agenda.on('start:send email', (job) => { /* ... */ });
agenda.on('fail:send email', (err, job) => { /* ... */ });
Use fail listeners to capture richer error context, such as stack traces,
without storing large payloads in job.attrs.failReason:
agenda.on('fail', async (err, job) => {
await saveJobError({
jobId: job.attrs._id,
jobName: job.attrs.name,
message: err.message,
stack: err.stack
});
});
Custom Backend
For databases other than MongoDB, PostgreSQL, or Redis, implement AgendaBackend:
import { AgendaBackend, JobRepository } from 'agenda';
class SQLiteBackend implements AgendaBackend {
readonly repository: JobRepository;
readonly notificationChannel = undefined; // Or implement NotificationChannel
async connect() { /* ... */ }
async disconnect() { /* ... */ }
}
const agenda = new Agenda({
backend: new SQLiteBackend({ path: './jobs.db' })
});
See Custom Backend Driver for details.
Documentation
- Full Documentation
- Migration Guide (v5 to v6)
- Custom Backend Driver
- TypeScript Decorators
- API Reference
Related Packages
Official Backend Packages:
- @agendajs/mongo-backend - MongoDB backend
- @agendajs/postgres-backend - PostgreSQL backend with LISTEN/NOTIFY
- @agendajs/redis-backend - Redis backend with Pub/Sub
Tools:
- agendash - Web dashboard for Agenda
- agenda-rest - REST API for Agenda
License
MIT