Mastering Delayed::Job – Database-backed Job Queue for Ruby
๐ Table of Contents
- 1. Fundamentals & Core Concepts
- 2. Installation & Setup
- 3. Basic Usage & Workers
- 4. Queue Management & Priority
- 5. Monitoring & Web UI
- 6. Advanced Features
- 7. Troubleshooting & Best Practices
- 8. Interview Questions & Answers
- 9. Real-World Case Studies
- 10. Reference & Commands
- 11. Commands & Concepts Reference Table
๐1. Fundamentals & Core Concepts
Delayed::Job is a database-backed job queue for Ruby that provides a simple and reliable way to handle background job processing. It stores jobs in your application’s database and processes them using worker processes.
What is Delayed::Job?
Delayed::Job is a background job processing system that allows you to move time-consuming tasks out of your main application flow. It stores jobs in your database and processes them using worker processes, making your application more responsive and scalable without requiring additional infrastructure like Redis.
โ Pros
- No additional infrastructure needed (uses your database)
- Simple setup and configuration
- ACID compliant (database transactions)
- Easy to monitor and debug
- Built-in retry mechanism
- Process-based concurrency (better fault isolation)
- Mature and battle-tested
โ Cons
- Lower performance than Redis-based solutions
- Database becomes a bottleneck
- Higher memory usage (process-based)
- Limited advanced features
- No built-in scheduling
- Can impact database performance
๐ Alternatives
- Sidekiq: Higher performance, Redis-backed
- Resque: Redis-backed, excellent monitoring
- Que: PostgreSQL-based, ACID compliant
- ActiveJob: Rails abstraction layer
Why Use Delayed::Job?
- โ To perform tasks asynchronously (emails, notifications)
- โ To avoid blocking the main web request/response cycle
- โ To increase app responsiveness and user experience
- โ To retry failed jobs automatically
- โ To improve scalability and performance
- โ When you want to avoid additional infrastructure
- โ When you need ACID compliance for job processing
Delayed::Job Architecture
๐ How Delayed::Job Works:
- Job Creation: Jobs are enqueued to database table
- Queue Storage: Jobs stored in delayed_jobs table
- Worker Processes: Separate processes poll database
- Job Execution: Workers pick up and execute jobs
- Result Handling: Success/failure logged to database
Key Components
- Database: Your application’s database for job storage
- Workers: Separate processes that execute jobs
- delayed_jobs table: Database table that holds pending jobs
- Jobs: Ruby objects that implement
perform
method - ActiveRecord: ORM for database interactions
โ๏ธ2. Installation & Setup
Prerequisites
- Ruby: 2.0 or higher
- Rails: 4.0 or higher (optional but recommended)
- Database: Any database supported by ActiveRecord
- ActiveRecord: For database interactions
Installation Steps
1. Add Delayed::Job to Gemfile
# Gemfile
gem 'delayed_job'
gem 'delayed_job_active_record' # For ActiveRecord backend
gem 'delayed_job_web' # Optional: for Web UI
2. Install Gems
bundle install
3. Generate Migration
# Generate the delayed_jobs table
rails generate delayed_job:install
# This creates a migration file like:
# db/migrate/YYYYMMDDHHMMSS_create_delayed_jobs.rb
4. Run Migration
rails db:migrate
5. Create Delayed::Job Configuration
# config/initializers/delayed_job.rb
Delayed::Job.destroy_failed_jobs = false
Delayed::Job.max_attempts = 3
Delayed::Job.max_run_time = 5.minutes
# Optional: Configure logging
Delayed::Job.logger = Rails.logger
# Optional: Configure queue names
Delayed::Job.queue_name = 'default'
Rails Integration
Add to Application
# config/application.rb
module YourApp
class Application < Rails::Application
# ... existing code ...
# Configure Delayed::Job as ActiveJob backend
config.active_job.queue_adapter = :delayed_job
end
end
Create Procfile
# Procfile
web: bundle exec rails server
worker: bundle exec rake jobs:work
Database Schema
Delayed Jobs Table Structure
# The migration creates a table with these columns:
create_table :delayed_jobs, force: true do |t|
t.integer :priority, default: 0, null: false
t.integer :attempts, default: 0, null: false
t.text :handler, null: false
t.text :last_error
t.datetime :run_at
t.datetime :locked_at
t.datetime :failed_at
t.string :locked_by
t.string :queue
t.datetime :created_at
t.datetime :updated_at
end
add_index :delayed_jobs, [:priority, :run_at], name: 'delayed_jobs_priority'
Environment Setup
Environment Variables
# .env
DATABASE_URL=postgresql://localhost/your_app
RAILS_ENV=development
QUEUE=default,high,low
Development Setup
# Start Rails server
rails server
# Start Delayed::Job worker (in separate terminal)
bundle exec rake jobs:work
# Or with specific queue
bundle exec rake jobs:work QUEUE=high,default
Verification
Test Database Connection
# In Rails console
rails console
# Test database connection
ActiveRecord::Base.connection.execute("SELECT 1")
# Should return a result
Test Job Enqueue
# Create a test job
class TestJob
def perform
puts "Test job executed!"
end
end
# Enqueue the job
Delayed::Job.enqueue(TestJob.new)
# Should return a Delayed::Job instance
Common Setup Issues
Solution: Check database connection and run
rails db:migrate:status
Solution: Ensure worker is started:
bundle exec rake jobs:work
Solution: Check worker processes:
ps aux | grep delayed_job
๐3. Basic Usage & Workers
Creating Jobs
Basic Job Structure
class EmailJob
def perform(user_id, message)
user = User.find(user_id)
UserMailer.notification(user, message).deliver_now
end
end
Job with Error Handling
class DataProcessingJob
def perform(data_id)
data = Data.find(data_id)
begin
# Process the data
data.process!
data.update(status: 'processed')
rescue => e
data.update(status: 'failed', error: e.message)
raise e # Re-raise to trigger retry
end
end
end
Job with Priority
class HighPriorityJob
def perform
# High priority work
end
end
# Enqueue with high priority
Delayed::Job.enqueue(HighPriorityJob.new, priority: 10)
Enqueueing Jobs
Basic Enqueue
# Enqueue immediately
Delayed::Job.enqueue(EmailJob.new(user.id, "Hello!"))
# Enqueue with delay
Delayed::Job.enqueue(EmailJob.new(user.id, "Hello!"), run_at: 1.hour.from_now)
# Enqueue with priority
Delayed::Job.enqueue(EmailJob.new(user.id, "Hello!"), priority: 5)
Using ActiveJob
class EmailJob < ApplicationJob
queue_as :default
def perform(user_id, message)
user = User.find(user_id)
UserMailer.notification(user, message).deliver_now
end
end
# Enqueue using ActiveJob
EmailJob.perform_later(user.id, "Hello!")
EmailJob.set(wait: 1.hour).perform_later(user.id, "Hello!")
Worker Management
Starting Workers
# Start worker for all queues
bundle exec rake jobs:work
# Start worker for specific queues
bundle exec rake jobs:work QUEUE=high,default
# Start multiple workers
bundle exec rake jobs:work QUEUE=high,default COUNT=3
# Start worker in background
nohup bundle exec rake jobs:work > worker.log 2>&1 &
Worker Configuration
# config/initializers/delayed_job.rb
Delayed::Job.destroy_failed_jobs = false
Delayed::Job.max_attempts = 3
Delayed::Job.max_run_time = 5.minutes
Delayed::Job.sleep_delay = 5
Delayed::Job.read_ahead = 5
# Configure logging
Delayed::Job.logger = Rails.logger
Delayed::Job.logger.level = Logger::INFO
Job Lifecycle
๐ Job Processing Flow:
- Enqueue: Job added to delayed_jobs table
- Wait: Job waits until run_at time
- Lock: Worker locks job for processing
- Execute: Worker calls perform method
- Complete: Job marked as completed or failed
- Retry: Failed jobs retried based on configuration
Error Handling
Retry Configuration
# Global retry settings
Delayed::Job.max_attempts = 3
Delayed::Job.destroy_failed_jobs = false
# Per-job retry settings
class RetryableJob
def perform
# Job logic
end
def max_attempts
5 # Override global setting
end
end
Custom Error Handling
class RobustJob
def perform
begin
# Main job logic
process_data
rescue NetworkError => e
# Retry network errors
raise e
rescue ValidationError => e
# Don't retry validation errors
log_error(e)
return
end
end
def max_attempts
3
end
private
def log_error(error)
Rails.logger.error("Job failed: #{error.message}")
end
end
Monitoring Workers
Check Worker Status
# Check running workers
ps aux | grep delayed_job
# Check job counts
Delayed::Job.count
Delayed::Job.where(failed_at: nil).count
Delayed::Job.where.not(failed_at: nil).count
# Check queue lengths
Delayed::Job.group(:queue).count
Worker Health Check
# Check if workers are processing
Delayed::Job.where('locked_at > ?', 10.minutes.ago).count
# Check for stuck jobs
Delayed::Job.where('locked_at < ?', 1.hour.ago).count
# Check failed jobs
Delayed::Job.where.not(failed_at: nil).count
๐4. Queue Management & Priority
Queue Concepts
Delayed::Job uses a single table for all jobs but supports queue names and priorities to organize job processing. Jobs are processed in order of priority (lower numbers = higher priority) and then by creation time.
Queue Configuration
Setting Queue Names
# Enqueue to specific queue
Delayed::Job.enqueue(EmailJob.new, queue: 'emails')
# Using ActiveJob
class EmailJob < ApplicationJob
queue_as :emails
end
# Enqueue with queue
EmailJob.set(queue: 'high_priority').perform_later
Priority Levels
# High priority (processed first)
Delayed::Job.enqueue(Job.new, priority: 0)
# Normal priority
Delayed::Job.enqueue(Job.new, priority: 10)
# Low priority (processed last)
Delayed::Job.enqueue(Job.new, priority: 20)
Worker Queue Assignment
Processing Specific Queues
# Process all queues
bundle exec rake jobs:work
# Process specific queues
bundle exec rake jobs:work QUEUE=high,default
# Process single queue
bundle exec rake jobs:work QUEUE=emails
# Process multiple queues with priority
bundle exec rake jobs:work QUEUE=high,default,low
Queue Monitoring
# Check queue lengths
Delayed::Job.group(:queue).count
# Check jobs by priority
Delayed::Job.group(:priority).count
# Check failed jobs by queue
Delayed::Job.where.not(failed_at: nil).group(:queue).count
# Check stuck jobs by queue
Delayed::Job.where('locked_at < ?', 1.hour.ago).group(:queue).count
Advanced Queue Management
Queue-specific Workers
# Start dedicated email worker
bundle exec rake jobs:work QUEUE=emails COUNT=2
# Start high-priority worker
bundle exec rake jobs:work QUEUE=high COUNT=1
# Start background worker
bundle exec rake jobs:work QUEUE=default,low COUNT=3
Queue Cleanup
# Clean up old completed jobs
Delayed::Job.where('created_at < ?', 30.days.ago).delete_all
# Clean up failed jobs
Delayed::Job.where.not(failed_at: nil).delete_all
# Clean up stuck jobs
Delayed::Job.where('locked_at < ?', 1.hour.ago).update_all(locked_at: nil, locked_by: nil)
Best Practices
๐ก Queue Management Tips:
- Use descriptive queue names (emails, notifications, reports)
- Set appropriate priorities for different job types
- Monitor queue lengths and processing times
- Use dedicated workers for critical queues
- Implement proper error handling and retries
- Clean up old jobs regularly
๐ง5. Monitoring & Web UI
Built-in Monitoring
Database Queries
# Check overall job status
Delayed::Job.count
Delayed::Job.where(failed_at: nil).count
Delayed::Job.where.not(failed_at: nil).count
# Check queue status
Delayed::Job.group(:queue).count
# Check priority distribution
Delayed::Job.group(:priority).count
# Check processing status
Delayed::Job.where.not(locked_at: nil).count
Delayed::Job.where('locked_at < ?', 10.minutes.ago).count
Worker Status
# Check running workers
ps aux | grep delayed_job
# Check worker processes
pgrep -f "delayed_job"
# Check worker logs
tail -f log/delayed_job.log
Web UI Setup
Install delayed_job_web
# Add to Gemfile
gem 'delayed_job_web'
# Install
bundle install
Configure Routes
# config/routes.rb
Rails.application.routes.draw do
# ... existing routes ...
# Mount the web UI
mount DelayedJobWeb => '/delayed_job'
end
Access Web UI
# Start Rails server
rails server
# Access web UI at:
# http://localhost:3000/delayed_job
Custom Monitoring
Health Check Endpoint
# app/controllers/health_controller.rb
class HealthController < ApplicationController
def delayed_job
stats = {
total_jobs: Delayed::Job.count,
pending_jobs: Delayed::Job.where(failed_at: nil).count,
failed_jobs: Delayed::Job.where.not(failed_at: nil).count,
running_workers: `ps aux | grep delayed_job | grep -v grep | wc -l`.strip.to_i
}
render json: stats
end
end
Monitoring Dashboard
# app/controllers/admin/dashboard_controller.rb
class Admin::DashboardController < ApplicationController
def delayed_job
@stats = {
total_jobs: Delayed::Job.count,
pending_jobs: Delayed::Job.where(failed_at: nil).count,
failed_jobs: Delayed::Job.where.not(failed_at: nil).count,
queue_stats: Delayed::Job.group(:queue).count,
priority_stats: Delayed::Job.group(:priority).count
}
end
end
Logging Configuration
Configure Logging
# config/initializers/delayed_job.rb
Delayed::Job.logger = Rails.logger
Delayed::Job.logger.level = Logger::INFO
# Custom log format
Delayed::Job.logger.formatter = proc do |severity, datetime, progname, msg|
"[#{datetime}] #{severity}: #{msg}\n"
end
Log Levels
# Development - verbose logging
Delayed::Job.logger.level = Logger::DEBUG
# Production - minimal logging
Delayed::Job.logger.level = Logger::WARN
# Custom logging
Delayed::Job.logger.info("Worker started")
Delayed::Job.logger.error("Job failed: #{error.message}")
Alerting
Failed Job Alerts
# config/initializers/delayed_job.rb
Delayed::Job.after_failure do |job, error|
# Send alert
AlertService.notify("Job failed: #{job.id} - #{error.message}")
end
Queue Monitoring
# Check queue health
def check_queue_health
failed_count = Delayed::Job.where.not(failed_at: nil).count
stuck_count = Delayed::Job.where('locked_at < ?', 1.hour.ago).count
if failed_count > 100
AlertService.notify("High failed job count: #{failed_count}")
end
if stuck_count > 50
AlertService.notify("Jobs stuck: #{stuck_count}")
end
end
๐6. Advanced Features
Job Scheduling
Delayed Execution
# Execute after 1 hour
Delayed::Job.enqueue(Job.new, run_at: 1.hour.from_now)
# Execute at specific time
Delayed::Job.enqueue(Job.new, run_at: Time.parse('2024-01-01 09:00:00'))
# Execute every day at 9 AM
class DailyJob
def perform
# Job logic
end
def self.schedule_daily
Delayed::Job.enqueue(new, run_at: Time.current.beginning_of_day + 9.hours)
end
end
Recurring Jobs
# Simple recurring job
class RecurringJob
def perform
# Job logic
# Schedule next execution
Delayed::Job.enqueue(self.class.new, run_at: 1.hour.from_now)
end
end
# Start recurring job
Delayed::Job.enqueue(RecurringJob.new, run_at: 1.hour.from_now)
Job Serialization
Complex Objects
# Jobs with complex parameters
class ComplexJob
def initialize(user, options = {})
@user = user
@options = options
end
def perform
# Access instance variables
user = @user
options = @options
# Job logic
end
end
# Enqueue with complex object
user = User.find(1)
Delayed::Job.enqueue(ComplexJob.new(user, {priority: 'high'}))
ActiveRecord Objects
# Jobs with ActiveRecord objects
class UserNotificationJob
def initialize(user_id)
@user_id = user_id
end
def perform
user = User.find(@user_id)
# Job logic with user
end
end
# Enqueue with user ID
Delayed::Job.enqueue(UserNotificationJob.new(user.id))
Custom Job Classes
Job with Custom Methods
class AdvancedJob
def perform
# Main job logic
process_data
end
def max_attempts
5 # Custom retry attempts
end
def max_run_time
10.minutes # Custom timeout
end
def queue_name
'high_priority' # Custom queue
end
private
def process_data
# Complex processing logic
end
end
Job with Callbacks
class CallbackJob
def perform
# Job logic
end
def before(job)
Rails.logger.info("Starting job #{job.id}")
end
def after(job)
Rails.logger.info("Completed job #{job.id}")
end
def error(job, exception)
Rails.logger.error("Job #{job.id} failed: #{exception.message}")
end
end
Performance Optimization
Database Optimization
# Add database indexes
add_index :delayed_jobs, [:priority, :run_at]
add_index :delayed_jobs, [:queue, :priority, :run_at]
add_index :delayed_jobs, [:locked_at, :locked_by]
add_index :delayed_jobs, [:failed_at]
# Configure read_ahead
Delayed::Job.read_ahead = 10
# Configure sleep_delay
Delayed::Job.sleep_delay = 5
Worker Optimization
# Start multiple workers
bundle exec rake jobs:work COUNT=4
# Use different queues for different workers
# Worker 1: High priority jobs
bundle exec rake jobs:work QUEUE=high COUNT=2
# Worker 2: Background jobs
bundle exec rake jobs:work QUEUE=default,low COUNT=3
Integration with ActiveJob
ActiveJob Configuration
# config/application.rb
config.active_job.queue_adapter = :delayed_job
# Use ActiveJob features
class EmailJob < ApplicationJob
queue_as :emails
def perform(user_id, message)
# Job logic
end
end
# Enqueue with ActiveJob
EmailJob.perform_later(user.id, "Hello!")
EmailJob.set(wait: 1.hour).perform_later(user.id, "Hello!")
EmailJob.set(queue: 'high').perform_later(user.id, "Hello!")
Custom ActiveJob Adapter
# lib/active_job/queue_adapters/delayed_job_adapter.rb
module ActiveJob
module QueueAdapters
class DelayedJobAdapter
def enqueue(job)
Delayed::Job.enqueue(job, queue: job.queue_name)
end
def enqueue_at(job, timestamp)
Delayed::Job.enqueue(job, run_at: Time.at(timestamp))
end
end
end
end
๐ฅ7. Troubleshooting & Best Practices
Common Issues
Jobs Not Processing
Diagnosis: Check if workers are running
Solution: Start workers with
bundle exec rake jobs:work
Jobs Stuck in Database
Diagnosis: Check for crashed workers
Solution: Unlock stuck jobs:
Delayed::Job.where('locked_at < ?', 1.hour.ago).update_all(locked_at: nil, locked_by: nil)
High Memory Usage
Diagnosis: Check for memory leaks in jobs
Solution: Restart workers periodically or use process monitoring
Performance Issues
Slow Job Processing
# Check for slow jobs
Delayed::Job.where('locked_at > ?', 10.minutes.ago).count
# Optimize database queries
add_index :delayed_jobs, [:priority, :run_at]
# Increase worker count
bundle exec rake jobs:work COUNT=4
# Use dedicated workers for slow jobs
bundle exec rake jobs:work QUEUE=slow_jobs COUNT=2
Database Bottleneck
# Monitor database performance
Delayed::Job.where('created_at > ?', 1.hour.ago).count
# Optimize read_ahead setting
Delayed::Job.read_ahead = 5
# Use connection pooling
Delayed::Job.connection_pool = ActiveRecord::Base.connection_pool
Monitoring & Debugging
Job Debugging
# Check job details
job = Delayed::Job.find(job_id)
puts job.handler
puts job.last_error
puts job.attempts
# Check job payload
job_object = YAML.load(job.handler)
puts job_object.class
puts job_object.instance_variables
Worker Debugging
# Check worker logs
tail -f log/delayed_job.log
# Check worker processes
ps aux | grep delayed_job
# Check worker status
Delayed::Job.where.not(locked_at: nil).group(:locked_by).count
Best Practices
๐ก Production Best Practices:
- Use process monitoring (systemd, monit, supervisord)
- Implement proper logging and monitoring
- Set up alerts for failed jobs and stuck workers
- Use database indexes for better performance
- Implement job cleanup strategies
- Test job failure scenarios
- Use appropriate queue names and priorities
- Monitor database performance impact
Security Considerations
Job Security
# Validate job parameters
class SecureJob
def initialize(user_id, data)
@user_id = user_id
@data = data
end
def perform
# Validate inputs
user = User.find(@user_id)
raise "Unauthorized" unless user.can_perform_action?
# Process with validated data
process_secure_data(@data)
end
private
def process_secure_data(data)
# Sanitize and process data
end
end
Worker Security
# Run workers with limited permissions
# Use dedicated user for workers
sudo -u delayed_job_user bundle exec rake jobs:work
# Set up firewall rules
# Limit database access for workers
# Use environment-specific configurations
Delayed::Job.logger.level = Logger::WARN if Rails.env.production?
๐จ8. Interview Questions & Answers
Basic Questions
A: Delayed::Job is a database-backed job queue for Ruby that stores jobs in a database table and processes them using worker processes. Jobs are serialized and stored in the delayed_jobs table, and workers poll the database for pending jobs to execute.
A: No additional infrastructure needed, ACID compliance, simple setup, easy debugging, and no external dependencies. It uses your existing database.
A:
Delayed::Job.enqueue(JobClass.new, priority: 10, run_at: 1.hour.from_now, queue: 'emails')
Intermediate Questions
A: Failed jobs are marked with failed_at timestamp and last_error. Jobs are retried based on max_attempts configuration. You can configure global or per-job retry settings.
A: Priority determines processing order (lower numbers = higher priority). Queue names are used for organizing jobs and can be used to route jobs to specific workers.
A: Use database queries to check job counts, failed jobs, and queue lengths. Use delayed_job_web for web UI. Monitor worker processes and set up alerts for failures.
Advanced Questions
A: Add database indexes, increase worker count, use dedicated workers for different queues, optimize read_ahead and sleep_delay settings, and implement proper job cleanup.
A: Database becomes the bottleneck, limited concurrency due to process-based workers, potential for database locks, and slower performance compared to Redis-based solutions.
A: Use YAML serialization for complex objects, pass IDs instead of full ActiveRecord objects, implement custom serialization for large objects, and be careful with object references.
System Design Questions
A: When you want to avoid additional infrastructure, need ACID compliance, have simple job requirements, or are working in an environment where Redis is not available.
A: Use database partitioning, implement job batching, use multiple worker pools, implement job prioritization, use database clustering, and implement proper monitoring and alerting.
โ9. Real-World Case Studies
E-commerce Platform
Challenge
A large e-commerce platform needed to process order confirmations, inventory updates, and customer notifications without blocking the main application.
Solution
# Order processing jobs
class OrderConfirmationJob
def perform(order_id)
order = Order.find(order_id)
OrderMailer.confirmation(order).deliver_now
InventoryService.update_stock(order)
end
end
class InventoryUpdateJob
def perform(product_id)
product = Product.find(product_id)
product.update_stock_levels
product.notify_low_stock if product.low_stock?
end
end
# Queue configuration
bundle exec rake jobs:work QUEUE=high COUNT=2 # Order confirmations
bundle exec rake jobs:work QUEUE=default COUNT=3 # General processing
bundle exec rake jobs:work QUEUE=low COUNT=1 # Background tasks
Results
- Reduced order processing time from 5 seconds to 200ms
- Handled 10,000+ orders per day
- Improved customer experience with immediate order confirmation
- Reliable inventory management with automatic retries
Content Management System
Challenge
A CMS needed to process large file uploads, generate thumbnails, and send notifications without impacting user experience.
Solution
# File processing jobs
class FileProcessingJob
def perform(file_id)
file = FileUpload.find(file_id)
# Generate thumbnails
file.generate_thumbnails
# Update file status
file.update(status: 'processed')
# Notify user
NotificationJob.perform_later(file.user_id, "File processed")
end
end
class NotificationJob
def perform(user_id, message)
user = User.find(user_id)
UserMailer.notification(user, message).deliver_now
end
end
# Worker configuration
bundle exec rake jobs:work QUEUE=file_processing COUNT=3
bundle exec rake jobs:work QUEUE=notifications COUNT=2
Results
- Processed 5,000+ files per day
- Reduced upload processing time by 80%
- Improved system reliability with automatic retries
- Scalable solution for growing user base
Analytics Platform
Challenge
An analytics platform needed to process large datasets, generate reports, and send scheduled notifications without impacting real-time data collection.
Solution
# Analytics processing jobs
class DataProcessingJob
def perform(dataset_id)
dataset = Dataset.find(dataset_id)
# Process data
results = AnalyticsService.process(dataset)
# Store results
dataset.update(results: results, status: 'completed')
# Schedule report generation
ReportGenerationJob.perform_later(dataset.id)
end
end
class ReportGenerationJob
def perform(dataset_id)
dataset = Dataset.find(dataset_id)
report = ReportService.generate(dataset)
# Send report
ReportMailer.send_report(dataset.user, report).deliver_now
end
end
# Scheduled jobs
class ScheduledReportJob
def perform
# Generate weekly reports
User.find_each do |user|
WeeklyReportJob.perform_later(user.id)
end
# Schedule next run
Delayed::Job.enqueue(self.class.new, run_at: 1.week.from_now)
end
end
Results
- Processed 1TB+ of data daily
- Generated 10,000+ reports per week
- 99.9% uptime for real-time analytics
- Automated report delivery system
๐ข10. Reference & Commands
Configuration Options
Global Configuration
# config/initializers/delayed_job.rb
Delayed::Job.destroy_failed_jobs = false
Delayed::Job.max_attempts = 3
Delayed::Job.max_run_time = 5.minutes
Delayed::Job.sleep_delay = 5
Delayed::Job.read_ahead = 5
Delayed::Job.logger = Rails.logger
Environment Variables
# .env
QUEUE=default,high,low
COUNT=3
RAILS_ENV=production
DATABASE_URL=postgresql://localhost/app_production
Rake Commands
Worker Management
# Start workers
bundle exec rake jobs:work
bundle exec rake jobs:work QUEUE=high,default
bundle exec rake jobs:work QUEUE=* COUNT=4
# Stop workers
pkill -f "delayed_job"
# Restart workers
pkill -f "delayed_job" && bundle exec rake jobs:work
Database Commands
# Generate migration
rails generate delayed_job:install
# Run migration
rails db:migrate
# Check migration status
rails db:migrate:status
Database Schema
Delayed Jobs Table
create_table :delayed_jobs, force: true do |t|
t.integer :priority, default: 0, null: false
t.integer :attempts, default: 0, null: false
t.text :handler, null: false
t.text :last_error
t.datetime :run_at
t.datetime :locked_at
t.datetime :failed_at
t.string :locked_by
t.string :queue
t.datetime :created_at
t.datetime :updated_at
end
add_index :delayed_jobs, [:priority, :run_at], name: 'delayed_jobs_priority'
Monitoring Queries
Job Statistics
# Overall statistics
Delayed::Job.count
Delayed::Job.where(failed_at: nil).count
Delayed::Job.where.not(failed_at: nil).count
# Queue statistics
Delayed::Job.group(:queue).count
Delayed::Job.group(:priority).count
# Worker statistics
Delayed::Job.where.not(locked_at: nil).group(:locked_by).count
Delayed::Job.where('locked_at < ?', 10.minutes.ago).count
Health Checks
# Check for stuck jobs
Delayed::Job.where('locked_at < ?', 1.hour.ago).count
# Check for old failed jobs
Delayed::Job.where('failed_at < ?', 7.days.ago).count
# Check queue health
Delayed::Job.group(:queue).count.each do |queue, count|
puts "#{queue}: #{count} jobs"
end
Useful Gems
Core Gems
- delayed_job: Core job processing
- delayed_job_active_record: ActiveRecord backend
- delayed_job_web: Web monitoring interface
Related Gems
- activejob: Rails job abstraction
- sidekiq: Alternative Redis-based solution
- resque: Alternative Redis-based solution
- que: PostgreSQL-based alternative
Resources
Documentation
- Official Documentation: https://github.com/collectiveidea/delayed_job
- Rails Guide: https://guides.rubyonrails.org/active_job_basics.html
- API Documentation: https://www.rubydoc.info/gems/delayed_job
Community
- GitHub Issues: https://github.com/collectiveidea/delayed_job/issues
- Stack Overflow: Search for "delayed_job"
- Rails Forum: https://discuss.rubyonrails.org/
๐11. Commands & Concepts Reference Table
Core Commands
Command | Description | Usage |
---|---|---|
rails generate delayed_job:install | Generate migration for delayed_jobs table | Initial setup |
rails db:migrate | Run migration to create delayed_jobs table | After generating migration |
bundle exec rake jobs:work | Start worker to process jobs | Production/development |
bundle exec rake jobs:work QUEUE=high | Start worker for specific queue | Queue-specific processing |
bundle exec rake jobs:work COUNT=3 | Start multiple workers | High throughput |
pkill -f "delayed_job" | Stop all delayed_job workers | Maintenance/restart |
ps aux | grep delayed_job | Check running workers | Monitoring |
Ruby/Rails Commands
Command | Description | Usage |
---|---|---|
Delayed::Job.enqueue(Job.new) | Enqueue job immediately | Basic job enqueueing |
Delayed::Job.enqueue(Job.new, run_at: 1.hour.from_now) | Enqueue job with delay | Scheduled jobs |
Delayed::Job.enqueue(Job.new, priority: 10) | Enqueue job with priority | Priority processing |
Delayed::Job.enqueue(Job.new, queue: 'emails') | Enqueue job to specific queue | Queue management |
Delayed::Job.count | Count total jobs | Monitoring |
Delayed::Job.where(failed_at: nil).count | Count pending jobs | Queue monitoring |
Delayed::Job.where.not(failed_at: nil).count | Count failed jobs | Error monitoring |
Delayed::Job.group(:queue).count | Count jobs by queue | Queue analysis |
Database Commands
Command | Description | Usage |
---|---|---|
rails db:migrate:status | Check migration status | Setup verification |
rails db:rollback | Rollback last migration | Development |
rails console | Open Rails console | Debugging/testing |
ActiveRecord::Base.connection.execute("SELECT 1") | Test database connection | Connection testing |
Monitoring Commands
Command | Description | Usage |
---|---|---|
tail -f log/delayed_job.log | Monitor worker logs | Real-time monitoring |
pgrep -f "delayed_job" | Find worker processes | Process monitoring |
Delayed::Job.where('locked_at < ?', 1.hour.ago).count | Check for stuck jobs | Health monitoring |
Delayed::Job.where.not(locked_at: nil).group(:locked_by).count | Check worker distribution | Load balancing |
Core Concepts
Concept | Description | Usage |
---|---|---|
Job | Ruby object with perform method that contains the work to be done | Background task execution |
Worker | Process that polls database and executes jobs | Job processing |
Queue | Named group for organizing jobs (stored in queue column) | Job organization |
Priority | Numeric value determining job processing order (lower = higher priority) | Job ordering |
Handler | YAML-serialized job object stored in database | Job persistence |
Locking | Mechanism to prevent multiple workers from processing same job | Concurrency control |
Retry | Automatic re-execution of failed jobs based on max_attempts | Error handling |
Serialization | Converting job objects to YAML for database storage | Job persistence |
Configuration Options
Option | Description | Default |
---|---|---|
max_attempts | Maximum number of retry attempts for failed jobs | 3 |
max_run_time | Maximum time a job can run before being killed | 5 minutes |
sleep_delay | Seconds to sleep when no jobs are available | 5 |
read_ahead | Number of jobs to read from database at once | 5 |
destroy_failed_jobs | Whether to delete jobs after max_attempts | false |
logger | Logger instance for worker output | Rails.logger |
Database Schema Fields
Field | Type | Description |
---|---|---|
id | integer | Primary key, unique job identifier |
priority | integer | Job priority (lower = higher priority) |
attempts | integer | Number of execution attempts |
handler | text | YAML-serialized job object |
last_error | text | Last error message if job failed |
run_at | datetime | When job should be executed |
locked_at | datetime | When job was locked by worker |
locked_by | string | Worker process identifier |
failed_at | datetime | When job permanently failed |
queue | string | Queue name for job organization |
created_at | datetime | When job was created |
updated_at | datetime | When job was last updated |
Environment Variables
Variable | Description | Example |
---|---|---|
QUEUE | Comma-separated list of queues to process | high,default,low |
COUNT | Number of worker processes to start | 4 |
RAILS_ENV | Rails environment (development, production, etc.) | production |
DATABASE_URL | Database connection URL | postgresql://localhost/app_production |
REDIS_URL | Redis connection URL (if using Redis for other features) | redis://localhost:6379/0 |
Common Job Patterns
Pattern | Description | Use Case |
---|---|---|
Email Job | Send emails asynchronously | User notifications, marketing emails |
Data Processing | Process large datasets in background | Analytics, reports, data imports |
File Processing | Handle file uploads and processing | Image resizing, document processing |
API Calls | Make external API calls asynchronously | Third-party integrations, webhooks |
Cleanup Jobs | Periodic cleanup of old data | Database maintenance, log cleanup |
Scheduled Jobs | Execute jobs at specific times | Daily reports, maintenance tasks |
Learn more aboutย Rails
Learn more aboutย Active Job
Learn more aboutย DevOps
https://shorturl.fm/KtJPc
https://shorturl.fm/vpqgZ
https://shorturl.fm/DCIHC
https://shorturl.fm/9jTGS
https://shorturl.fm/4ll1v
https://shorturl.fm/oC6bG
https://shorturl.fm/1vt9w
https://shorturl.fm/H1oBd
https://shorturl.fm/W7DhQ
https://shorturl.fm/BtcW5
https://shorturl.fm/xaHU6
https://shorturl.fm/0OH9p
https://shorturl.fm/JSBdF
https://shorturl.fm/gPF18
https://shorturl.fm/SOuXE
https://shorturl.fm/tRb73
https://shorturl.fm/aENDu
https://shorturl.fm/Iw4u6
https://shorturl.fm/DBphS