agenda vs async vs bee-queue vs bull vs kue vs p-queue vs queue
Job and Task Queues in Node.js
agendaasyncbee-queuebullkuep-queuequeueSimilar Packages:

Job and Task Queues in Node.js

Job and task queues in Node.js are essential for managing background tasks, handling asynchronous operations, and improving the scalability of applications. These queues allow developers to offload time-consuming tasks, such as sending emails, processing images, or performing complex calculations, to be executed separately from the main application flow. This helps in optimizing resource usage, reducing response times, and ensuring a smoother user experience. Libraries like agenda, async, bee-queue, bull, kue, p-queue, and queue provide various features and implementations for managing these tasks efficiently, each with its own strengths and use cases.

Npm Package Weekly Downloads Trend

3 Years

Github Stars Ranking

Stat Detail

Package
Downloads
Stars
Size
Issues
Publish
License
agenda152,8059,635295 kB28 days agoMIT
async028,199808 kB242 years agoMIT
bee-queue04,022107 kB413 months agoMIT
bull016,236309 kB146a year agoMIT
kue09,465-2889 years agoMIT
p-queue04,12976.8 kB52 months agoMIT
queue077619.6 kB203 years agoMIT

Feature Comparison: agenda vs async vs bee-queue vs bull vs kue vs p-queue vs queue

Storage Backend

  • agenda:

    agenda uses MongoDB as its storage backend, allowing for persistent job storage and retrieval. This makes it suitable for applications that require durability and the ability to recover jobs after a restart.

  • async:

    async does not provide a storage backend, as it is a utility library for managing asynchronous operations in memory. It is designed for tasks that can be completed within the application's runtime without the need for persistence.

  • bee-queue:

    bee-queue uses Redis as its storage backend, providing fast and efficient job storage with low latency. It is optimized for high-throughput applications that require quick access to job data.

  • bull:

    bull also uses Redis as its storage backend, offering robust support for job persistence, retries, and delayed jobs. It is designed for applications that need reliable and scalable queue management.

  • kue:

    kue uses Redis for storing job data, including priorities, statuses, and progress. It provides a simple and efficient way to manage jobs with support for persistence and recovery.

  • p-queue:

    p-queue does not use an external storage backend, as it manages tasks in memory. It is designed for lightweight applications that require concurrency control without the overhead of persistent storage.

  • queue:

    queue is an in-memory queue that does not rely on external storage. It is suitable for applications that need simple, fast queueing without the complexity of managing persistent data.

Concurrency Control

  • agenda:

    agenda allows for concurrency control by specifying the number of jobs that can run simultaneously. This helps prevent resource exhaustion and ensures that tasks are processed efficiently.

  • async:

    async provides various methods for controlling concurrency, such as async.parallelLimit and async.queue, which allow developers to limit the number of concurrent tasks. This flexibility makes it easy to manage resource usage while executing asynchronous operations.

  • bee-queue:

    bee-queue supports concurrency by allowing multiple workers to process jobs simultaneously. The concurrency level can be configured to optimize performance based on the application's needs.

  • bull:

    bull offers advanced concurrency control features, including the ability to set the number of concurrent processors for a queue. It also supports job prioritization and rate limiting, making it a versatile choice for managing complex workloads.

  • kue:

    kue allows for concurrency by enabling multiple workers to process jobs in parallel. The concurrency level can be adjusted to balance performance and resource usage.

  • p-queue:

    p-queue is designed for concurrency control, allowing developers to set the maximum number of concurrent tasks. It also supports rate limiting, making it ideal for scenarios where tasks need to be throttled.

  • queue:

    queue provides basic concurrency control by allowing multiple tasks to be processed simultaneously. The concurrency level can be configured to suit the application's requirements.

Job Scheduling

  • agenda:

    agenda excels in job scheduling, particularly for recurring tasks. It allows developers to schedule jobs at specific intervals, making it ideal for applications that require automated, time-based task execution.

  • async:

    async does not provide built-in job scheduling capabilities, as it focuses on managing asynchronous operations. Scheduling would need to be implemented separately using timers or other mechanisms.

  • bee-queue:

    bee-queue supports delayed jobs, allowing tasks to be scheduled for execution after a specified delay. However, it does not have built-in support for recurring jobs.

  • bull:

    bull offers robust job scheduling features, including support for delayed and recurring jobs. It is well-suited for applications that need advanced scheduling capabilities with fine-grained control.

  • kue:

    kue supports delayed jobs, allowing tasks to be scheduled for future execution. However, it lacks native support for recurring jobs, which would need to be implemented manually.

  • p-queue:

    p-queue does not provide job scheduling features, as it is focused on managing the execution of tasks based on concurrency and rate limits. Scheduling would need to be handled separately.

  • queue:

    queue is a simple FIFO queue that does not include built-in scheduling features. Tasks are processed in the order they are added, without support for delays or recurring execution.

Monitoring and UI

  • agenda:

    agenda does not include a built-in UI for monitoring jobs, but there are third-party tools and integrations available. Developers can also create custom dashboards using the MongoDB data.

  • async:

    async does not provide any monitoring or UI features, as it is a utility library for managing asynchronous tasks. Monitoring would need to be implemented separately.

  • bee-queue:

    bee-queue does not have a built-in UI, but it provides APIs for monitoring job status and progress. Developers can create custom dashboards or use third-party tools for visualization.

  • bull:

    bull includes a popular UI called Bull Board for monitoring queues, jobs, and workers. This makes it easy to track job status, view metrics, and manage queues in real-time.

  • kue:

    kue comes with a built-in web UI for monitoring jobs, viewing their statuses, and managing the queue. This feature makes it easy to track job progress and identify issues.

  • p-queue:

    p-queue does not offer any monitoring or UI features, as it is a lightweight library focused on concurrency control. Monitoring would need to be implemented separately.

  • queue:

    queue does not include any built-in monitoring or UI features. It is a simple queue implementation that relies on external tools for tracking and visualization.

Ease of Use: Code Examples

  • agenda:

    agenda provides a simple API for defining and scheduling jobs. Its integration with MongoDB makes it easy to manage job data, and its support for recurring tasks is straightforward to use.

  • async:

    async offers a wide range of functions for managing asynchronous workflows, including queues. Its documentation is comprehensive, making it easy for developers to understand and implement various patterns.

  • bee-queue:

    bee-queue has a simple API for creating and processing jobs. Its focus on performance is reflected in its design, but it remains easy to use for developers familiar with Redis.

  • bull:

    bull provides a rich API with features like job prioritization, retries, and rate limiting. Its documentation is thorough, and the inclusion of Bull Board makes it user-friendly for managing queues.

  • kue:

    kue has an intuitive API for creating jobs and setting priorities. Its built-in UI enhances usability by providing a visual interface for monitoring job status.

  • p-queue:

    p-queue is designed for simplicity, with a clear API for managing concurrency and rate limits. Its promise-based design makes it easy to integrate into modern JavaScript applications.

  • queue:

    queue offers a straightforward API for adding and processing tasks. Its simplicity makes it easy to implement, but it lacks advanced features found in other libraries.

How to Choose: agenda vs async vs bee-queue vs bull vs kue vs p-queue vs queue

  • agenda:

    Choose agenda if you need a MongoDB-backed job scheduler with support for recurring tasks and a simple API. It is ideal for applications that require persistent job storage and scheduling capabilities.

  • async:

    Choose async if you need a comprehensive utility library for managing asynchronous operations, including task queues, parallel execution, and flow control. It is suitable for projects that require fine-grained control over async workflows without relying on external storage.

  • bee-queue:

    Choose bee-queue if you need a high-performance, Redis-backed job queue optimized for fast processing and low latency. It is designed for applications that require efficient handling of large volumes of jobs with minimal overhead.

  • bull:

    Choose bull if you need a feature-rich, Redis-based queue with support for job prioritization, retries, rate limiting, and more. It is ideal for applications that require advanced queue management features and scalability.

  • kue:

    Choose kue if you need a simple, Redis-backed priority job queue with a built-in UI for monitoring. It is suitable for applications that require basic queue functionality with an easy-to-use interface for tracking job status.

  • p-queue:

    Choose p-queue if you need a lightweight, promise-based queue with support for concurrency control and rate limiting. It is ideal for projects that require fine-tuned control over the number of concurrent tasks without the need for external storage.

  • queue:

    Choose queue if you need a simple, in-memory FIFO queue with support for concurrency and task prioritization. It is suitable for lightweight applications that require basic queue functionality without external dependencies.

README for agenda

Agenda

Agenda

A light-weight job scheduling library for Node.js

NPM Version NPM Downloads

Migrating from v5? See the Migration Guide for all breaking changes.

Agenda 6.x

Agenda 6.x is a complete TypeScript rewrite with a focus on modularity and flexibility:

  • Pluggable storage backends - Choose from MongoDB, PostgreSQL, Redis, or implement your own. Each backend is a separate package - install only what you need.

  • Pluggable notification channels - Move beyond polling with real-time job notifications via Redis, PostgreSQL LISTEN/NOTIFY, or other pub/sub systems. Jobs get processed immediately when saved, not on the next poll cycle.

  • Modern stack - ESM-only, Node.js 18+, full TypeScript with strict typing.

See the 6.x Roadmap for details and progress.

Features

  • Minimal overhead job scheduling
  • Pluggable storage backends (MongoDB, PostgreSQL, Redis)
  • TypeScript support with full typing
  • Scheduling via cron or human-readable syntax
  • Configurable concurrency and locking
  • Real-time job notifications (optional)
  • Sandboxed worker execution via fork mode
  • TypeScript decorators for class-based job definitions

Installation

Install the core package and your preferred backend:

# For MongoDB
npm install agenda @agendajs/mongo-backend

# For PostgreSQL
npm install agenda @agendajs/postgres-backend

# For Redis
npm install agenda @agendajs/redis-backend

Requirements:

  • Node.js 18+
  • Database of your choice (MongoDB 4+, PostgreSQL, or Redis)

Quick Start

import { Agenda } from 'agenda';
import { MongoBackend } from '@agendajs/mongo-backend';

const agenda = new Agenda({
  backend: new MongoBackend({ address: 'mongodb://localhost/agenda' })
});

// Define a job
agenda.define('send email', async (job) => {
  const { to, subject } = job.attrs.data;
  await sendEmail(to, subject);
});

// Start processing
await agenda.start();

// Schedule jobs
await agenda.every('1 hour', 'send email', { to: 'user@example.com', subject: 'Hello' });
await agenda.schedule('in 5 minutes', 'send email', { to: 'admin@example.com', subject: 'Report' });
await agenda.now('send email', { to: 'support@example.com', subject: 'Urgent' });

Official Backend Packages

PackageBackendNotificationsInstall
@agendajs/mongo-backendMongoDBPolling onlynpm install @agendajs/mongo-backend
@agendajs/postgres-backendPostgreSQLLISTEN/NOTIFYnpm install @agendajs/postgres-backend
@agendajs/redis-backendRedisPub/Subnpm install @agendajs/redis-backend

Backend Capabilities

BackendStorageNotificationsNotes
MongoDB (MongoBackend)βœ…βŒStorage only. Combine with external notification channel for real-time.
PostgreSQL (PostgresBackend)βœ…βœ…Full backend. Uses LISTEN/NOTIFY for notifications.
Redis (RedisBackend)βœ…βœ…Full backend. Uses Pub/Sub for notifications.
InMemoryNotificationChannelβŒβœ…Notifications only. For single-process/testing.

Backend Configuration

MongoDB

import { Agenda } from 'agenda';
import { MongoBackend } from '@agendajs/mongo-backend';

// Via connection string
const agenda = new Agenda({
  backend: new MongoBackend({ address: 'mongodb://localhost/agenda' })
});

// Via existing MongoDB connection
const agenda = new Agenda({
  backend: new MongoBackend({ mongo: existingDb })
});

// With options
const agenda = new Agenda({
  backend: new MongoBackend({
    mongo: db,
    collection: 'jobs'        // Collection name (default: 'agendaJobs')
  }),
  processEvery: '30 seconds', // Job polling interval
  maxConcurrency: 20,         // Max concurrent jobs
  defaultConcurrency: 5       // Default per job type
});

PostgreSQL

import { Agenda } from 'agenda';
import { PostgresBackend } from '@agendajs/postgres-backend';

const agenda = new Agenda({
  backend: new PostgresBackend({
    connectionString: 'postgresql://user:pass@localhost:5432/mydb'
  })
});

Redis

import { Agenda } from 'agenda';
import { RedisBackend } from '@agendajs/redis-backend';

const agenda = new Agenda({
  backend: new RedisBackend({
    connectionString: 'redis://localhost:6379'
  })
});

Real-Time Notifications

For faster job processing across distributed systems:

import { Agenda, InMemoryNotificationChannel } from 'agenda';
import { MongoBackend } from '@agendajs/mongo-backend';

const agenda = new Agenda({
  backend: new MongoBackend({ mongo: db }),
  notificationChannel: new InMemoryNotificationChannel()
});

Mixing Storage and Notification Backends

You can use MongoDB for storage while using a different system for real-time notifications:

import { Agenda } from 'agenda';
import { MongoBackend } from '@agendajs/mongo-backend';
import { RedisBackend } from '@agendajs/redis-backend';

// MongoDB for storage + Redis for real-time notifications
const redisBackend = new RedisBackend({ connectionString: 'redis://localhost:6379' });
const agenda = new Agenda({
  backend: new MongoBackend({ mongo: db }),
  notificationChannel: redisBackend.notificationChannel
});

This is useful when you want MongoDB's proven durability and flexible queries for job storage, but need faster real-time notifications across multiple processes.

API Overview

Defining Jobs

// Simple async handler
agenda.define('my-job', async (job) => {
  console.log('Processing:', job.attrs.data);
});

// With options
agenda.define('my-job', async (job) => { /* ... */ }, {
  concurrency: 10,
  lockLimit: 5,
  lockLifetime: 10 * 60 * 1000, // 10 minutes
  priority: 'high'
});

Defining Jobs with Decorators

For a class-based approach, use TypeScript decorators:

import { JobsController, Define, Every, registerJobs, Job } from 'agenda';

@JobsController({ namespace: 'email' })
class EmailJobs {
  @Define({ concurrency: 5 })
  async sendWelcome(job: Job<{ userId: string }>) {
    console.log('Sending welcome to:', job.attrs.data.userId);
  }

  @Every('1 hour')
  async cleanupBounced(job: Job) {
    console.log('Cleaning up bounced emails');
  }
}

registerJobs(agenda, [new EmailJobs()]);
await agenda.start();

// Schedule using namespaced name
await agenda.now('email.sendWelcome', { userId: '123' });

See Decorators Documentation for full details.

Scheduling Jobs

// Run immediately
await agenda.now('my-job', { userId: '123' });

// Run at specific time
await agenda.schedule('tomorrow at noon', 'my-job', data);
await agenda.schedule(new Date('2024-12-25'), 'my-job', data);

// Run repeatedly
await agenda.every('5 minutes', 'my-job');
await agenda.every('0 * * * *', 'my-job'); // Cron syntax

Job Control

// Cancel jobs (removes from database)
await agenda.cancel({ name: 'my-job' });

// Disable/enable jobs globally (by query)
await agenda.disable({ name: 'my-job' });  // Disable all jobs matching query
await agenda.enable({ name: 'my-job' });   // Enable all jobs matching query

// Disable/enable individual jobs
const job = await agenda.create('my-job', data);
job.disable();
await job.save();

// Progress tracking
agenda.define('long-job', async (job) => {
  for (let i = 0; i <= 100; i += 10) {
    await doWork();
    await job.touch(i); // Report progress 0-100
  }
});

Stopping / Draining

// Stop immediately - unlocks running jobs so other workers can pick them up
await agenda.stop();

// Drain - waits for running jobs to complete before stopping
await agenda.drain();

// Drain with timeout (30 seconds) - for cloud platforms with shutdown deadlines
const result = await agenda.drain(30000);
if (result.timedOut) {
    console.log(`${result.running} jobs still running after timeout`);
}

// Drain with AbortSignal - for external control
const controller = new AbortController();
setTimeout(() => controller.abort(), 30000);
await agenda.drain({ signal: controller.signal });

Use drain() for graceful shutdowns where you want in-progress jobs to finish their work.

Events

agenda.on('start', (job) => console.log('Job started:', job.attrs.name));
agenda.on('complete', (job) => console.log('Job completed:', job.attrs.name));
agenda.on('success', (job) => console.log('Job succeeded:', job.attrs.name));
agenda.on('fail', (err, job) => console.log('Job failed:', job.attrs.name, err));

// Job-specific events
agenda.on('start:send email', (job) => { /* ... */ });
agenda.on('fail:send email', (err, job) => { /* ... */ });

Custom Backend

For databases other than MongoDB, PostgreSQL, or Redis, implement AgendaBackend:

import { AgendaBackend, JobRepository } from 'agenda';

class SQLiteBackend implements AgendaBackend {
  readonly repository: JobRepository;
  readonly notificationChannel = undefined; // Or implement NotificationChannel

  async connect() { /* ... */ }
  async disconnect() { /* ... */ }
}

const agenda = new Agenda({
  backend: new SQLiteBackend({ path: './jobs.db' })
});

See Custom Backend Driver for details.

Documentation

Related Packages

Official Backend Packages:

Tools:

License

MIT