Background JobsBackground Jobs & Task Scheduling
Not every operation should happen synchronously during an HTTP request. If a user signs up, you don't want them staring at a loading spinner for 4 seconds while your server generates a welcome PDF, resizes their avatar, and sends an email. These tasks should happen in the background — the server responds to the user immediately, and the heavy work runs separately, asynchronously.
What Are Background Jobs?
A background job is a unit of work that is queued and executed outside of the normal HTTP request-response cycle. Instead of doing the work inline, you drop a "task" into a queue, and a separate process (called a worker) picks it up and processes it.
Common use cases for background jobs:
- Email sending: Sending a transactional email takes 200–500ms and can fail. Offload to a queue and retry automatically.
- Image processing: Resizing, compressing, or converting images takes seconds. Don't block the request.
- PDF generation: Generating invoice PDFs, reports, certificates.
- Bulk operations: Sending 50,000 newsletter emails — impossible to do synchronously.
- Sync jobs: Syncing data with third-party APIs on a schedule.
- Cleanup tasks: Deleting expired sessions, archiving old records.
The Queue Architecture
API Request
POST /register
→
Create user
in database
→
Enqueue job
send welcome email
→
Respond
201 Created
Worker process
running separately
→
Dequeues job
from Redis
→
Sends email
via SendGrid
→
Marks job
completed
The API server and the worker process are decoupled. The API doesn't care whether the worker has processed the job or not. If the email fails, the worker can retry automatically without affecting the API server at all.
BullMQ — The Node.js Job Queue
BullMQ is the most popular job queue for Node.js. It uses Redis as the queue storage backend. Jobs are added to a queue and processed by workers asynchronously, with built-in retry logic, delays, priorities, and job progress tracking.
npm install bullmq ioredis
// queue.js — Define the queue
const { Queue } = require('bullmq');
const Redis = require('ioredis');
const connection = new Redis(process.env.REDIS_URL);
const emailQueue = new Queue('email-notifications', { connection });
module.exports = { emailQueue };
// In your API route — add jobs to the queue
const { emailQueue } = require('./queue');
app.post('/auth/register', async (req, res) => {
const { email, password } = req.body;
const user = await createUser(email, password);
// Add a job to the queue — this returns immediately
await emailQueue.add('welcome-email', {
to: user.email,
username: user.username,
userId: user.id
}, {
attempts: 3, // Retry up to 3 times on failure
backoff: {
type: 'exponential',
delay: 2000 // Wait 2s, 4s, 8s between retries
}
});
// Respond immediately — don't wait for the email to send
res.status(201).json({ message: 'Account created' });
});
// worker.js — A separate process that processes jobs
const { Worker } = require('bullmq');
const Redis = require('ioredis');
const nodemailer = require('nodemailer');
const connection = new Redis(process.env.REDIS_URL);
const worker = new Worker('email-notifications', async (job) => {
console.log(`Processing job ${job.id}: ${job.name}`);
if (job.name === 'welcome-email') {
const { to, username } = job.data;
await sendEmail({
to,
subject: `Welcome to VoidX, ${username}!`,
html: '
Your account is ready. Start learning!
'
});
console.log(`Welcome email sent to ${to}`);
}
}, { connection });
worker.on('completed', (job) => console.log(`Job ${job.id} completed`));
worker.on('failed', (job, err) => console.error(`Job ${job.id} failed:`, err));
console.log('Worker is running and listening for jobs...');Scheduled Jobs (Cron Tasks)
Some jobs need to run on a schedule — generate a daily report at midnight, clean up expired tokens every hour, send weekly digest emails every Monday. This is called cron scheduling.
The cron syntax is a string with 5 or 6 fields representing: seconds, minutes, hours, day of month, month, day of week.
const { Queue } = require('bullmq');
const Redis = require('ioredis');
const connection = new Redis(process.env.REDIS_URL);
const scheduleQueue = new Queue('scheduled-tasks', { connection });
// Run every day at midnight
await scheduleQueue.add('daily-report', {}, {
repeat: { cron: '0 0 * * *' }
});
// Run every Monday at 9am
await scheduleQueue.add('weekly-digest', {}, {
repeat: { cron: '0 9 * * 1' } // 1 = Monday
});
// Run every hour
await scheduleQueue.add('cleanup-expired-tokens', {}, {
repeat: { every: 60 * 60 * 1000 }
});
// For simpler use cases, node-cron runs scheduled tasks directly inside the process
const cron = require('node-cron');
// Run every day at midnight UTC
cron.schedule('0 0 * * *', async () => {
console.log('Running daily cleanup...');
const deleted = await prisma.session.deleteMany({
where: {
expiresAt: { lt: new Date(Date.now() - 30 * 24 * 60 * 60 * 1000) }
}
});
console.log(`Deleted ${deleted.count} expired sessions`);
});
// Run every 15 minutes
cron.schedule('*/15 * * * *', async () => {
await syncExchangeRates();
});
Celery — Background Jobs in Python
from celery import Celery
from datetime import datetime
# Initialize Celery with Redis as the broker
app = Celery('tasks', broker=os.environ['REDIS_URL'])
# Define a task
@app.task(bind=True, max_retries=3, default_retry_delay=60)
def send_welcome_email(self, user_id: int, email: str, username: str):
try:
# Do the heavy work here
send_email(
to=email,
subject=f'Welcome, {username}!',
body='Your account is ready.'
)
return {'status': 'sent', 'userId': user_id}
except Exception as exc:
# Retry on failure
raise self.retry(exc=exc)
# In your Flask/FastAPI route, enqueue the task
@app.route('/register', methods=['POST'])
def register():
user = create_user(request.json)
# .delay() queues the job — returns immediately
send_welcome_email.delay(user.id, user.email, user.username)
return jsonify({'id': user.id}), 201