🔍1. Fundamentals & Core Concepts

🏗️ Sidekiq Architecture

🔧 Key Components

⚙️2. Installation & Setup

📦 Installation Steps

1. Add Sidekiq to Gemfile

# Gemfile
gem 'sidekiq', '~> 7.0'
gem 'redis', '~> 5.0'

2. Install Dependencies

# Install gems
bundle install

# Install Redis (Ubuntu/Debian)
sudo apt-get install redis-server

# Install Redis (macOS)
brew install redis

# Start Redis
redis-server

3. Configure Sidekiq

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  config.redis = { url: 'redis://localhost:6379/0' }
end

Sidekiq.configure_client do |config|
  config.redis = { url: 'redis://localhost:6379/0' }
end

4. Create Sidekiq Configuration

# config/sidekiq.yml
:concurrency: 10
:queues:
  - [critical, 5]
  - [high_priority, 3]
  - [default, 2]
  - [low_priority, 1]
  - [mailers, 1]

:max_retries: 3
:retry_interval: 5

5. Configure Rails Integration

# config/application.rb
config.active_job.queue_adapter = :sidekiq

# config/routes.rb
require 'sidekiq/web'
Rails.application.routes.draw do
  mount Sidekiq::Web => '/sidekiq'
end

🚀3. Basic Usage & Workers

📝 Creating Your First Worker

1. Generate Worker

# Generate worker
rails generate sidekiq:worker EmailWorker

# Or create manually
# app/sidekiq/email_worker.rb

2. Basic Worker Structure

class EmailWorker
  include Sidekiq::Worker
  
  def perform(user_id)
    user = User.find(user_id)
    UserMailer.welcome_email(user).deliver_now
  end
end

3. Enqueue Jobs

# Enqueue job
EmailWorker.perform_async(user.id)

# Enqueue with delay
EmailWorker.perform_in(5.minutes, user.id)

# Enqueue at specific time
EmailWorker.perform_at(1.hour.from_now, user.id)

🏷️ Queue Management

Custom Queue Assignment

class UrgentEmailWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'critical'
  
  def perform(user_id)
    # Process urgent email
  end
end

class NewsletterWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'low_priority'
  
  def perform(user_id)
    # Process newsletter
  end
end

🎯 Queue Priority System

📊 Priority Configuration

# config/sidekiq.yml
:concurrency: 10
:queues:
  - [critical, 10]      # Highest priority - processed first
  - [high_priority, 5]  # High priority - processed second
  - [default, 2]        # Normal priority - processed third
  - [low_priority, 1]   # Low priority - processed last
  - [mailers, 1]        # Email jobs - processed last

🔢 Understanding Queue Weights

📊 How Queue Weights Work:

# Configuration
:queues:
  - [critical, 10]      # 10/18 = 55.6% of worker attention
  - [high_priority, 5]  # 5/18 = 27.8% of worker attention
  - [default, 2]        # 2/18 = 11.1% of worker attention
  - [low_priority, 1]   # 1/18 = 5.6% of worker attention

# Total weight: 10 + 5 + 2 + 1 = 18
# Each queue gets its weight / total weight percentage of attention

🎯 Weight Calculation Formula:

Queue Attention % = (Queue Weight / Total Weights) × 100

# Example with [critical, 10]:
# Critical attention = (10 / 18) × 100 = 55.6%
# High Priority attention = (5 / 18) × 100 = 27.8%
# Default attention = (2 / 18) × 100 = 11.1%
# Low Priority attention = (1 / 18) × 100 = 5.6%

⚖️ Weight Guidelines:

🔧 How to Define Queue Weights:

📊 Weight Configuration Examples:

🎯 E-commerce Platform:

# High-priority order processing
:queues:
  - [order_processing, 10]    # 40% - Critical for business
  - [payment_processing, 8]   # 32% - Financial transactions
  - [email_notifications, 4]  # 16% - User communication
  - [analytics, 2]            # 8% - Background analytics
  - [maintenance, 1]          # 4% - Cleanup tasks

📱 Social Media App:

# User experience focused
:queues:
  - [user_actions, 10]        # 40% - Real-time user interactions
  - [content_processing, 8]   # 32% - Media processing
  - [notifications, 4]        # 16% - Push notifications
  - [feed_generation, 2]      # 8% - Content feeds
  - [background_tasks, 1]     # 4% - Maintenance

📊 Analytics Platform:

# Data processing focused
:queues:
  - [data_processing, 10]     # 40% - Core data analysis
  - [report_generation, 6]    # 24% - User reports
  - [data_import, 4]          # 16% - External data
  - [alerts, 3]               # 12% - User alerts
  - [cleanup, 2]              # 8% - Data cleanup

🚨 Common Weight Mistakes:

📈 Monitoring Weight Effectiveness:

# Check queue processing rates
Sidekiq::Queue.all.each do |queue|
  puts "#{queue.name}: #{queue.size} jobs, weight: #{get_queue_weight(queue.name)}"
end

# Monitor processing distribution
stats = Sidekiq::Stats.new
puts "Total processed: #{stats.processed}"
puts "Queue distribution: #{get_queue_distribution}"

🔧 Weight Optimization Tips:

🔍 How Queue Attention Works Behind the Scenes

🔄 Weighted Round-Robin Algorithm:

# Sidekiq's internal queue polling logic (simplified)
class QueuePoller
  def poll_queues
    # Calculate polling frequency based on weights
    queue_weights = {
      'critical' => 10,
      'high_priority' => 5,
      'default' => 2,
      'low_priority' => 1
    }
    
    total_weight = queue_weights.values.sum  # 18
    
    # Poll each queue based on its weight
    queue_weights.each do |queue_name, weight|
      polling_frequency = weight.to_f / total_weight
      
      # Poll this queue 'weight' times for every cycle
      weight.times do
        poll_single_queue(queue_name)
      end
    end
  end
  
  def poll_single_queue(queue_name)
    # Check if queue has jobs
    if jobs_available?(queue_name)
      # Pick up job from this queue
      job = dequeue_job(queue_name)
      process_job(job)
    end
  end
end

⚙️ Detailed Polling Mechanism:

🎯 Step-by-Step Process:

📊 Example Polling Cycle:

# With weights [critical: 10, high_priority: 5, default: 2, low_priority: 1]
# One complete polling cycle looks like:

Cycle 1:  critical → critical → critical → critical → critical → 
          critical → critical → critical → critical → critical → 
          high_priority → high_priority → high_priority → high_priority → high_priority → 
          default → default → 
          low_priority

# This cycle repeats continuously
# Critical gets 10/18 = 55.6% of polling attention
# High Priority gets 5/18 = 27.8% of polling attention
# Default gets 2/18 = 11.1% of polling attention  
# Low Priority gets 1/18 = 5.6% of polling attention

🔍 Redis Implementation Details:

📋 How Sidekiq Stores and Retrieves Jobs:

# Redis data structure for queues
# Each queue is a Redis LIST
redis.lpush("queue:critical", job_data)      # Add job to critical queue
redis.lpush("queue:high_priority", job_data) # Add job to high priority queue
redis.lpush("queue:default", job_data)       # Add job to default queue
redis.lpush("queue:low_priority", job_data)  # Add job to low priority queue

# Sidekiq polls queues in weighted order
def poll_queues_with_weights
  weights = {critical: 10, high_priority: 5, default: 2, low_priority: 1}
  
  weights.each do |queue_name, weight|
    weight.times do
      # Try to get job from this queue
      job_data = redis.brpop("queue:#{queue_name}", timeout: 0.1)
      if job_data
        process_job(job_data)
        break  # Found a job, move to next queue
      end
    end
  end
end

⚡ Worker Thread Behavior:

🎯 How Workers Handle Queue Attention:

# Multiple workers polling simultaneously
Worker 1: [critical] → [critical] → [high_priority] → [default]
Worker 2: [critical] → [high_priority] → [high_priority] → [low_priority]  
Worker 3: [critical] → [default] → [default] → [low_priority]

# Each worker follows the same weighted polling pattern
# But they may pick up different jobs due to timing

📊 Concurrency and Queue Attention:

# With 3 workers and weights [10, 5, 2, 1]
# Total polling attempts per cycle: 3 workers × 18 polls = 54 polls

# Distribution across queues:
# Critical: 3 workers × 10 polls = 30 polling attempts (55.6%)
# High Priority: 3 workers × 5 polls = 15 polling attempts (27.8%)
# Default: 3 workers × 2 polls = 6 polling attempts (11.1%)
# Low Priority: 3 workers × 1 poll = 3 polling attempts (5.6%)

🔧 Advanced Queue Attention Features:

🎯 Dynamic Weight Adjustment:

# Sidekiq can dynamically adjust polling based on queue size
def adaptive_polling
  queue_sizes = get_queue_sizes
  
  queue_sizes.each do |queue_name, size|
    if size > threshold
      # Increase polling frequency for busy queues
      increase_polling_frequency(queue_name)
    end
  end
end

📊 Queue Starvation Prevention:

🚨 Queue Attention Edge Cases:

📈 Monitoring Queue Attention:

🔍 How to Verify Queue Attention is Working:

# Monitor actual queue processing rates
Sidekiq::Queue.all.each do |queue|
  puts "#{queue.name}:"
  puts "  Size: #{queue.size}"
  puts "  Weight: #{get_queue_weight(queue.name)}"
  puts "  Processing rate: #{get_processing_rate(queue.name)}"
end

# Check worker distribution
Sidekiq::Workers.new.each do |worker|
  puts "Worker #{worker['pid']}:"
  puts "  Current job: #{worker['payload']}"
  puts "  Queue: #{worker['queue']}"
  puts "  Busy: #{worker['busy']}"
end

📊 Queue Attention Metrics:

🔧 Tuning Queue Attention:

🎯 Optimization Strategies:

🔍 Priority Processing Rules

🔄 Priority Processing Flow:

1. Queue Order: Sidekiq processes queues in configuration order
2. Job Selection: Takes jobs from highest priority queue first
3. FIFO within Queue: Jobs within same queue processed first-in-first-out
4. Queue Depletion: Only moves to next queue when current queue is empty
5. Round Robin: Cycles through queues based on priority weights

👤 Single Worker Priority

📋 Single Worker Example

# Worker processes jobs in this order:
# 1. All critical jobs (highest priority)
# 2. All high_priority jobs
# 3. All default jobs
# 4. All low_priority jobs
# 5. All mailers jobs (lowest priority)

# Example job processing sequence:
# Critical: Job A, Job B, Job C
# High Priority: Job D, Job E
# Default: Job F, Job G, Job H
# Low Priority: Job I
# Mailers: Job J

⚡ Single Worker Advantages

👥 Multiple Workers Priority

📊 Multiple Workers Example

# Configuration with 3 workers
# config/sidekiq.yml
:concurrency: 3
:queues:
  - [critical, 5]
  - [high_priority, 3]
  - [default, 2]
  - [low_priority, 1]

# Job distribution across workers:
# Worker 1: Critical Job A, High Priority Job D
# Worker 2: Critical Job B, High Priority Job E  
# Worker 3: Critical Job C, Default Job F

# All critical jobs processed first, then high priority, etc.

🔄 Multiple Workers Processing

⚡ Multiple Workers Advantages

🔧 Priority Configuration Strategies

📈 High Priority Strategy

# Prioritize critical jobs
:queues:
  - [critical, 10]      # 50% of worker attention
  - [high_priority, 5]  # 25% of worker attention
  - [default, 3]        # 15% of worker attention
  - [low_priority, 2]   # 10% of worker attention

⚖️ Balanced Strategy

# Equal distribution with slight priority
:queues:
  - [critical, 3]       # 30% of worker attention
  - [high_priority, 3]  # 30% of worker attention
  - [default, 2]        # 20% of worker attention
  - [low_priority, 2]   # 20% of worker attention

📊 Monitoring Priority Processing

# Check queue sizes and priorities
Sidekiq::Queue.all.each do |queue|
  puts "#{queue.name}: #{queue.size} jobs"
end

# Monitor worker queue distribution
Sidekiq::Workers.new.each do |worker|
  puts "Worker #{worker['pid']}: #{worker['queues']}"
end

# Check processing rates by queue
stats = Sidekiq::Stats.new
puts "Processed: #{stats.processed}"
puts "Failed: #{stats.failed}"

🚨 Priority Troubleshooting

🔧 Priority Best Practices

📋4. Queue Management & Priority

🏷️ Queue Management

Custom Queue Assignment

class UrgentEmailWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'critical'
  
  def perform(user_id)
    # Process urgent email
  end
end

class NewsletterWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'low_priority'
  
  def perform(user_id)
    # Process newsletter
  end
end

🎯 Queue Priority System

📊 Priority Configuration

# config/sidekiq.yml
:concurrency: 10
:queues:
  - [critical, 10]      # Highest priority - processed first
  - [high_priority, 5]  # High priority - processed second
  - [default, 2]        # Normal priority - processed third
  - [low_priority, 1]   # Low priority - processed last
  - [mailers, 1]        # Email jobs - processed last

🔢 Understanding Queue Weights

📊 How Queue Weights Work:

# Configuration
:queues:
  - [critical, 10]      # 10/18 = 55.6% of worker attention
  - [high_priority, 5]  # 5/18 = 27.8% of worker attention
  - [default, 2]        # 2/18 = 11.1% of worker attention
  - [low_priority, 1]   # 1/18 = 5.6% of worker attention

# Total weight: 10 + 5 + 2 + 1 = 18
# Each queue gets its weight / total weight percentage of attention

🎯 Weight Calculation Formula:

Queue Attention % = (Queue Weight / Total Weights) × 100

# Example with [critical, 10]:
# Critical attention = (10 / 18) × 100 = 55.6%
# High Priority attention = (5 / 18) × 100 = 27.8%
# Default attention = (2 / 18) × 100 = 11.1%
# Low Priority attention = (1 / 18) × 100 = 5.6%

⚖️ Weight Guidelines:

🔧 How to Define Queue Weights:

📊 Weight Configuration Examples:

🎯 E-commerce Platform:

# High-priority order processing
:queues:
  - [order_processing, 10]    # 40% - Critical for business
  - [payment_processing, 8]   # 32% - Financial transactions
  - [email_notifications, 4]  # 16% - User communication
  - [analytics, 2]            # 8% - Background analytics
  - [maintenance, 1]          # 4% - Cleanup tasks

📱 Social Media App:

# User experience focused
:queues:
  - [user_actions, 10]        # 40% - Real-time user interactions
  - [content_processing, 8]   # 32% - Media processing
  - [notifications, 4]        # 16% - Push notifications
  - [feed_generation, 2]      # 8% - Content feeds
  - [background_tasks, 1]     # 4% - Maintenance

📊 Analytics Platform:

# Data processing focused
:queues:
  - [data_processing, 10]     # 40% - Core data analysis
  - [report_generation, 6]    # 24% - User reports
  - [data_import, 4]          # 16% - External data
  - [alerts, 3]               # 12% - User alerts
  - [cleanup, 2]              # 8% - Data cleanup

🚨 Common Weight Mistakes:

📈 Monitoring Weight Effectiveness:

# Check queue processing rates
Sidekiq::Queue.all.each do |queue|
  puts "#{queue.name}: #{queue.size} jobs, weight: #{get_queue_weight(queue.name)}"
end

# Monitor processing distribution
stats = Sidekiq::Stats.new
puts "Total processed: #{stats.processed}"
puts "Queue distribution: #{get_queue_distribution}"

🔧 Weight Optimization Tips:

🔍 How Queue Attention Works Behind the Scenes

🔄 Weighted Round-Robin Algorithm:

# Sidekiq's internal queue polling logic (simplified)
class QueuePoller
  def poll_queues
    # Calculate polling frequency based on weights
    queue_weights = {
      'critical' => 10,
      'high_priority' => 5,
      'default' => 2,
      'low_priority' => 1
    }
    
    total_weight = queue_weights.values.sum  # 18
    
    # Poll each queue based on its weight
    queue_weights.each do |queue_name, weight|
      polling_frequency = weight.to_f / total_weight
      
      # Poll this queue 'weight' times for every cycle
      weight.times do
        poll_single_queue(queue_name)
      end
    end
  end
  
  def poll_single_queue(queue_name)
    # Check if queue has jobs
    if jobs_available?(queue_name)
      # Pick up job from this queue
      job = dequeue_job(queue_name)
      process_job(job)
    end
  end
end

⚙️ Detailed Polling Mechanism:

🎯 Step-by-Step Process:

📊 Example Polling Cycle:

# With weights [critical: 10, high_priority: 5, default: 2, low_priority: 1]
# One complete polling cycle looks like:

Cycle 1:  critical → critical → critical → critical → critical → 
          critical → critical → critical → critical → critical → 
          high_priority → high_priority → high_priority → high_priority → high_priority → 
          default → default → 
          low_priority

# This cycle repeats continuously
# Critical gets 10/18 = 55.6% of polling attention
# High Priority gets 5/18 = 27.8% of polling attention
# Default gets 2/18 = 11.1% of polling attention  
# Low Priority gets 1/18 = 5.6% of polling attention

🔍 Redis Implementation Details:

📋 How Sidekiq Stores and Retrieves Jobs:

# Redis data structure for queues
# Each queue is a Redis LIST
redis.lpush("queue:critical", job_data)      # Add job to critical queue
redis.lpush("queue:high_priority", job_data) # Add job to high priority queue
redis.lpush("queue:default", job_data)       # Add job to default queue
redis.lpush("queue:low_priority", job_data)  # Add job to low priority queue

# Sidekiq polls queues in weighted order
def poll_queues_with_weights
  weights = {critical: 10, high_priority: 5, default: 2, low_priority: 1}
  
  weights.each do |queue_name, weight|
    weight.times do
      # Try to get job from this queue
      job_data = redis.brpop("queue:#{queue_name}", timeout: 0.1)
      if job_data
        process_job(job_data)
        break  # Found a job, move to next queue
      end
    end
  end
end

⚡ Worker Thread Behavior:

🎯 How Workers Handle Queue Attention:

# Multiple workers polling simultaneously
Worker 1: [critical] → [critical] → [high_priority] → [default]
Worker 2: [critical] → [high_priority] → [high_priority] → [low_priority]  
Worker 3: [critical] → [default] → [default] → [low_priority]

# Each worker follows the same weighted polling pattern
# But they may pick up different jobs due to timing

📊 Concurrency and Queue Attention:

# With 3 workers and weights [10, 5, 2, 1]
# Total polling attempts per cycle: 3 workers × 18 polls = 54 polls

# Distribution across queues:
# Critical: 3 workers × 10 polls = 30 polling attempts (55.6%)
# High Priority: 3 workers × 5 polls = 15 polling attempts (27.8%)
# Default: 3 workers × 2 polls = 6 polling attempts (11.1%)
# Low Priority: 3 workers × 1 poll = 3 polling attempts (5.6%)

🔧 Advanced Queue Attention Features:

🎯 Dynamic Weight Adjustment:

# Sidekiq can dynamically adjust polling based on queue size
def adaptive_polling
  queue_sizes = get_queue_sizes
  
  queue_sizes.each do |queue_name, size|
    if size > threshold
      # Increase polling frequency for busy queues
      increase_polling_frequency(queue_name)
    end
  end
end

📊 Queue Starvation Prevention:

🚨 Queue Attention Edge Cases:

📈 Monitoring Queue Attention:

🔍 How to Verify Queue Attention is Working:

# Monitor actual queue processing rates
Sidekiq::Queue.all.each do |queue|
  puts "#{queue.name}:"
  puts "  Size: #{queue.size}"
  puts "  Weight: #{get_queue_weight(queue.name)}"
  puts "  Processing rate: #{get_processing_rate(queue.name)}"
end

# Check worker distribution
Sidekiq::Workers.new.each do |worker|
  puts "Worker #{worker['pid']}:"
  puts "  Current job: #{worker['payload']}"
  puts "  Queue: #{worker['queue']}"
  puts "  Busy: #{worker['busy']}"
end

📊 Queue Attention Metrics:

🔧 Tuning Queue Attention:

🎯 Optimization Strategies:

🔍 Priority Processing Rules

🔄 Priority Processing Flow:

1. Queue Order: Sidekiq processes queues in configuration order
2. Job Selection: Takes jobs from highest priority queue first
3. FIFO within Queue: Jobs within same queue processed first-in-first-out
4. Queue Depletion: Only moves to next queue when current queue is empty
5. Round Robin: Cycles through queues based on priority weights

👤 Single Worker Priority

📋 Single Worker Example

# Worker processes jobs in this order:
# 1. All critical jobs (highest priority)
# 2. All high_priority jobs
# 3. All default jobs
# 4. All low_priority jobs
# 5. All mailers jobs (lowest priority)

# Example job processing sequence:
# Critical: Job A, Job B, Job C
# High Priority: Job D, Job E
# Default: Job F, Job G, Job H
# Low Priority: Job I
# Mailers: Job J

⚡ Single Worker Advantages

👥 Multiple Workers Priority

📊 Multiple Workers Example

# Configuration with 3 workers
# config/sidekiq.yml
:concurrency: 3
:queues:
  - [critical, 5]
  - [high_priority, 3]
  - [default, 2]
  - [low_priority, 1]

# Job distribution across workers:
# Worker 1: Critical Job A, High Priority Job D
# Worker 2: Critical Job B, High Priority Job E  
# Worker 3: Critical Job C, Default Job F

# All critical jobs processed first, then high priority, etc.

🔄 Multiple Workers Processing

⚡ Multiple Workers Advantages

🔧 Priority Configuration Strategies

📈 High Priority Strategy

# Prioritize critical jobs
:queues:
  - [critical, 10]      # 50% of worker attention
  - [high_priority, 5]  # 25% of worker attention
  - [default, 3]        # 15% of worker attention
  - [low_priority, 2]   # 10% of worker attention

⚖️ Balanced Strategy

# Equal distribution with slight priority
:queues:
  - [critical, 3]       # 30% of worker attention
  - [high_priority, 3]  # 30% of worker attention
  - [default, 2]        # 20% of worker attention
  - [low_priority, 2]   # 20% of worker attention

📊 Monitoring Priority Processing

# Check queue sizes and priorities
Sidekiq::Queue.all.each do |queue|
  puts "#{queue.name}: #{queue.size} jobs"
end

# Monitor worker queue distribution
Sidekiq::Workers.new.each do |worker|
  puts "Worker #{worker['pid']}: #{worker['queues']}"
end

# Check processing rates by queue
stats = Sidekiq::Stats.new
puts "Processed: #{stats.processed}"
puts "Failed: #{stats.failed}"

🚨 Priority Troubleshooting

🔧 Priority Best Practices

⚙️4. Configuration & Optimization

🔧 Advanced Configuration

Redis Configuration

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  config.redis = { url: 'redis://localhost:6379/0' }
end

Sidekiq.configure_client do |config|
  config.redis = { url: 'redis://localhost:6379/0' }
end

Concurrency Settings

# config/sidekiq.yml
:concurrency: 25        # Number of threads per worker
:queues:
  - [critical, 5]       # Highest priority
  - [high_priority, 3]  # High priority
  - [default, 2]        # Normal priority
  - [low_priority, 1]   # Low priority
  - [mailers, 1]        # Email jobs

:max_retries: 3         # Maximum retry attempts
:retry_interval: 5      # Seconds between retries
:timeout: 30            # Job timeout in seconds

Environment-Specific Configuration

# config/environments/production.rb
config.active_job.queue_name_prefix = Rails.env
config.active_job.queue_name_delimiter = '.'

# config/sidekiq.yml
production:
  :concurrency: 50
  :queues:
    - [critical, 10]
    - [high_priority, 5]
    - [default, 3]
    - [low_priority, 1]

development:
  :concurrency: 5
  :queues:
    - [default, 1]

🚀 Performance Optimization

Memory Optimization

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  # Enable garbage collection
  config.on(:startup) do
    GC.auto_compact = true
  end
  
  # Memory monitoring
  config.on(:heartbeat) do
    GC.start if GC.stat[:heap_allocated_pages] > 1000
  end
end

Connection Pooling

# config/initializers/sidekiq.rb
require 'connection_pool'

Sidekiq.configure_server do |config|
  config.redis_pool = ConnectionPool.new(size: 25, timeout: 3) do
    Redis.new(url: ENV['REDIS_URL'])
  end
end

📊5. Monitoring & Web UI

🖥️ Sidekiq Web UI

Setup Web UI

# config/routes.rb
require 'sidekiq/web'

Rails.application.routes.draw do
  # Basic mount
  mount Sidekiq::Web => '/sidekiq'
  
  # With authentication
  authenticate :user, lambda { |u| u.admin? } do
    mount Sidekiq::Web => '/sidekiq'
  end
end

Web UI Features

📊 Monitoring Commands

CLI Monitoring

# Check Sidekiq processes
ps aux | grep sidekiq

# Check Redis connection
redis-cli ping

# Monitor Sidekiq in real-time
bundle exec sidekiqmon

# Check queue sizes
bundle exec sidekiqctl stats

Ruby API Monitoring

# Check queue sizes
Sidekiq::Queue.new.size
Sidekiq::Queue.new('critical').size

# Check worker status
Sidekiq::Workers.new.size
Sidekiq::Workers.new.each do |worker|
  puts "PID: #{worker['pid']}"
  puts "Threads: #{worker['concurrency']}"
  puts "Busy: #{worker['busy']}"
end

# Check failed jobs
Sidekiq::DeadSet.new.size
Sidekiq::RetrySet.new.size

# Get statistics
Sidekiq::Stats.new.processed
Sidekiq::Stats.new.failed

🔍 Advanced Monitoring

Custom Monitoring

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  config.on(:startup) do
    Rails.logger.info "Sidekiq started with #{Sidekiq.options[:concurrency]} threads"
  end
  
  config.on(:shutdown) do
    Rails.logger.info "Sidekiq shutting down"
  end
  
  config.on(:heartbeat) do
    # Custom heartbeat logic
    Rails.logger.info "Sidekiq heartbeat"
  end
end

Health Checks

# lib/sidekiq_health_check.rb
class SidekiqHealthCheck
  def self.healthy?
    begin
      # Check Redis connection
      Sidekiq.redis { |conn| conn.ping }
      
      # Check worker processes
      workers = Sidekiq::Workers.new
      return false if workers.size == 0
      
      # Check queue health
      queues = Sidekiq::Queue.all
      return false if queues.any? { |q| q.size > 1000 }
      
      true
    rescue => e
      Rails.logger.error "Sidekiq health check failed: #{e.message}"
      false
    end
  end
end

🔥6. Advanced Features

📦 Batch Processing

Creating Batches

# Create a batch
batch = Sidekiq::Batch.new
batch.on(:success, BatchCallback, { 'user_id' => user.id })

batch.jobs do
  users.each do |user|
    EmailWorker.perform_async(user.id)
  end
end

Batch Callbacks

class BatchCallback
  def on_success(status, options)
    user_id = options['user_id']
    Rails.logger.info "Batch completed for user #{user_id}"
  end
  
  def on_death(status, options)
    Rails.logger.error "Batch failed: #{status.failures}"
  end
end

⏰ Scheduled Jobs

Delayed Jobs

# Schedule job for later
EmailWorker.perform_in(5.minutes, user.id)

# Schedule job for specific time
EmailWorker.perform_at(1.hour.from_now, user.id)

# Recurring jobs with cron syntax
class DailyReportWorker
  include Sidekiq::Worker
  
  def perform
    # Generate daily report
  end
end

# In initializer
Sidekiq::Cron::Job.create(
  name: 'Daily Report',
  cron: '0 9 * * *',  # Every day at 9 AM
  class: 'DailyReportWorker'
)

🔄 Retry Logic

Custom Retry Strategies

class ApiCallWorker
  include Sidekiq::Worker
  sidekiq_options retry: 5, backtrace: true
  
  def perform(api_url, data)
    response = HTTP.get(api_url, json: data)
    
    if response.status.success?
      process_response(response)
    else
      raise "API call failed: #{response.status}"
    end
  rescue => e
    # Custom retry logic
    if retry_count < 3
      raise e  # Will retry
    else
      # Log final failure
      Rails.logger.error "Final failure: #{e.message}"
    end
  end
end

🔒 Job Locks

Unique Jobs

class UniqueEmailWorker
  include Sidekiq::Worker
  sidekiq_options unique_for: 1.hour
  
  def perform(user_id)
    # Only one job per user per hour
    send_email(user_id)
  end
end

Distributed Locks

class DataSyncWorker
  include Sidekiq::Worker
  
  def perform(data_id)
    # Use Redis for distributed locking
    Sidekiq.redis do |conn|
      lock_key = "sync_lock:#{data_id}"
      
      if conn.set(lock_key, 1, nx: true, ex: 300)  # 5 minute lock
        begin
          sync_data(data_id)
        ensure
          conn.del(lock_key)
        end
      else
        Rails.logger.info "Sync already in progress for #{data_id}"
      end
    end
  end
end

📊 Job Statistics

Custom Metrics

class MetricsWorker
  include Sidekiq::Worker
  
  def perform(metric_name, value)
    # Track custom metrics
    StatsD.increment("sidekiq.job.#{metric_name}", value)
  end
end

# Middleware for automatic metrics
class MetricsMiddleware
  def call(worker, job, queue)
    start_time = Time.current
    
    yield
    
    duration = Time.current - start_time
    StatsD.timing("sidekiq.job.duration", duration)
    StatsD.increment("sidekiq.job.success")
  rescue => e
    StatsD.increment("sidekiq.job.failure")
    raise e
  end
end

🔧7. Troubleshooting & Best Practices

🚨 Common Issues

✅ Best Practices

📋 Production Checklist

🚨 Emergency Procedures

Worker Restart

# Graceful restart
bundle exec sidekiqctl restart

# Force restart
bundle exec sidekiqctl stop
bundle exec sidekiqctl start

# Check worker status
bundle exec sidekiqctl status

Queue Management

# Clear specific queue
Sidekiq::Queue.new('failed').clear

# Clear all queues (emergency only)
Sidekiq::Queue.all.each(&:clear)

# Retry failed jobs
Sidekiq::RetrySet.new.each(&:retry)

# Delete dead jobs
Sidekiq::DeadSet.new.clear

📚8. Reference & Commands

🔧 Sidekiq Commands

📊 Monitoring Commands

Queue Monitoring

# Check all queue sizes
Sidekiq::Queue.all.each do |queue|
  puts "#{queue.name}: #{queue.size} jobs"
end

# Check worker status
Sidekiq::Workers.new.each do |worker|
  puts "PID: #{worker['pid']}, Busy: #{worker['busy']}"
end

# Check failed jobs
puts "Failed: #{Sidekiq::DeadSet.new.size}"
puts "Retrying: #{Sidekiq::RetrySet.new.size}"

Performance Monitoring

# Get processing statistics
stats = Sidekiq::Stats.new
puts "Processed: #{stats.processed}"
puts "Failed: #{stats.failed}"
puts "Enqueued: #{stats.enqueued}"

# Check Redis memory
redis_info = Sidekiq.redis { |conn| conn.info }
puts "Redis Memory: #{redis_info['used_memory_human']}"

🔧 Configuration Examples

Production Configuration

# config/sidekiq.yml
:concurrency: 25
:queues:
  - [critical, 10]
  - [high_priority, 5]
  - [default, 3]
  - [low_priority, 1]
  - [mailers, 1]

:max_retries: 3
:retry_interval: 5
:timeout: 30

# Environment-specific
production:
  :concurrency: 50
  :queues:
    - [critical, 15]
    - [high_priority, 10]
    - [default, 5]
    - [low_priority, 2]

development:
  :concurrency: 5
  :queues:
    - [default, 1]

Redis Configuration

# config/initializers/sidekiq.rb
Sidekiq.configure_server do |config|
  config.redis = {
    url: ENV['REDIS_URL'] || 'redis://localhost:6379/0',
    password: ENV['REDIS_PASSWORD'],
    db: 0,
    network_timeout: 5,
    pool_timeout: 5,
    size: 25
  }
end

Sidekiq.configure_client do |config|
  config.redis = {
    url: ENV['REDIS_URL'] || 'redis://localhost:6379/0',
    password: ENV['REDIS_PASSWORD'],
    db: 0,
    size: 5
  }
end

📚 Additional Resources

❓9. Interview Questions & Answers

🔍 Basic Questions

�� Advanced Questions

🔧 System Design Questions

🏢10. Real-World Case Studies

📧 E-commerce Email System

💡 Solution:

# Queue Configuration
:queues:
  - [critical_emails, 10]    # Order confirmations, password resets
  - [marketing_emails, 3]    # Newsletters, promotions
  - [bulk_emails, 1]         # Mass marketing campaigns

# Worker Classes
class CriticalEmailWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'critical_emails', retry: 3
  
  def perform(user_id, email_type)
    user = User.find(user_id)
    case email_type
    when 'order_confirmation'
      OrderMailer.confirmation(user).deliver_now
    when 'password_reset'
      UserMailer.password_reset(user).deliver_now
    end
  end
end

class MarketingEmailWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'marketing_emails', retry: 2
  
  def perform(user_id, campaign_id)
    user = User.find(user_id)
    campaign = Campaign.find(campaign_id)
    MarketingMailer.newsletter(user, campaign).deliver_now
  end
end

📊 Results:

• ✅ 99.9% email delivery rate
• ✅ Critical emails processed within 30 seconds
• ✅ Marketing emails processed within 5 minutes
• ✅ 50% reduction in email processing time
• ✅ 80% reduction in failed email attempts

🖼️ Image Processing Platform

💡 Solution:

# Priority-based image processing
class ImageProcessingWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'images', retry: 2, backtrace: true
  
  def perform(image_id, priority = 'normal')
    image = Image.find(image_id)
    
    # Process based on priority
    case priority
    when 'urgent'
      process_urgent_image(image)
    when 'normal'
      process_normal_image(image)
    when 'background'
      process_background_image(image)
    end
    
    # Update image status
    image.update!(processed: true, processed_at: Time.current)
  end
  
  private
  
  def process_urgent_image(image)
    # High-quality processing for profile pictures
    generate_thumbnails(image, sizes: [50, 100, 200])
    compress_image(image, quality: 90)
  end
  
  def process_normal_image(image)
    # Standard processing for regular uploads
    generate_thumbnails(image, sizes: [100, 200])
    compress_image(image, quality: 80)
  end
  
  def process_background_image(image)
    # Background processing for old images
    generate_thumbnails(image, sizes: [200])
    compress_image(image, quality: 70)
  end
end

# Batch processing for bulk operations
class BulkImageProcessingWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'bulk_images'
  
  def perform(image_ids)
    images = Image.where(id: image_ids)
    
    batch = Sidekiq::Batch.new
    batch.on(:success, BulkImageCallback)
    
    batch.jobs do
      images.each do |image|
        ImageProcessingWorker.perform_async(image.id, 'background')
      end
    end
  end

📊 Results:

• ✅ Profile pictures processed within 10 seconds
• ✅ Regular uploads processed within 2 minutes
• ✅ Background processing handled 10,000+ images daily
• ✅ 70% reduction in storage costs through compression
• ✅ 95% user satisfaction with image quality

📊 Data Analytics Pipeline

💡 Solution:

# Multi-stage data processing pipeline
class DataCollectionWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'data_collection', retry: 3
  
  def perform(event_data)
    # Collect and validate data
    event = Event.create!(event_data)
    
    # Trigger next stage
    DataProcessingWorker.perform_async(event.id)
  end
end

class DataProcessingWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'data_processing', retry: 2
  
  def perform(event_id)
    event = Event.find(event_id)
    
    # Process and aggregate data
    processed_data = process_event_data(event)
    
    # Store processed data
    ProcessedEvent.create!(processed_data)
    
    # Trigger analytics generation
    AnalyticsWorker.perform_async(event.user_id)
  end
end

class AnalyticsWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'analytics', retry: 1
  
  def perform(user_id)
    user = User.find(user_id)
    
    # Generate user analytics
    analytics = generate_user_analytics(user)
    
    # Update user dashboard
    user.update!(analytics: analytics)
    
    # Send notification if significant changes
    if analytics.significant_change?
      NotificationWorker.perform_async(user_id, 'analytics_update')
    end
  end
end

class ReportGenerationWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'reports', retry: 1
  
  def perform(report_type, date_range)
    # Generate scheduled reports
    report = generate_report(report_type, date_range)
    
    # Store report
    Report.create!(report)
    
    # Send to stakeholders
    ReportMailer.daily_report(report).deliver_now
  end

📊 Results:

• ✅ 5 million data points processed daily
• ✅ Real-time analytics updated within 5 minutes
• ✅ 99.9% data accuracy maintained
• ✅ 60% reduction in processing time
• ✅ Automated report generation for 100+ clients

🔐 Financial Transaction Processing

💡 Solution:

# Secure transaction processing with audit trails
class TransactionProcessingWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'transactions', retry: 0, backtrace: true
  
  def perform(transaction_id)
    transaction = Transaction.find(transaction_id)
    
    # Create audit trail
    AuditLog.create!(
      transaction: transaction,
      action: 'processing_started',
      timestamp: Time.current
    )
    
    begin
      # Validate transaction
      validate_transaction(transaction)
      
      # Check fraud detection
      fraud_check = FraudDetectionWorker.perform_now(transaction.id)
      
      if fraud_check.suspicious?
        raise "Transaction flagged for fraud review"
      end
      
      # Process transaction
      process_transaction(transaction)
      
      # Update status
      transaction.update!(status: 'completed')
      
      # Create success audit log
      AuditLog.create!(
        transaction: transaction,
        action: 'processing_completed',
        timestamp: Time.current
      )
      
      # Send confirmation
      NotificationWorker.perform_async(transaction.user_id, 'transaction_completed')
      
    rescue => e
      # Log failure
      AuditLog.create!(
        transaction: transaction,
        action: 'processing_failed',
        error: e.message,
        timestamp: Time.current
      )
      
      # Update status
      transaction.update!(status: 'failed', error: e.message)
      
      # Send failure notification
      NotificationWorker.perform_async(transaction.user_id, 'transaction_failed')
      
      raise e
    end
  end
end

class FraudDetectionWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'fraud_detection', retry: 1
  
  def perform(transaction_id)
    transaction = Transaction.find(transaction_id)
    
    # Implement fraud detection logic
    risk_score = calculate_risk_score(transaction)
    
    if risk_score > 0.8
      # Flag for manual review
      ManualReviewWorker.perform_async(transaction_id)
      return { suspicious: true, risk_score: risk_score }
    end
    
    { suspicious: false, risk_score: risk_score }
  end
end

class ComplianceReportingWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'compliance', retry: 3
  
  def perform(report_date)
    # Generate compliance reports
    report = generate_compliance_report(report_date)
    
    # Store report
    ComplianceReport.create!(report)
    
    # Submit to regulatory bodies
    submit_to_regulators(report)
  end

📊 Results:

• ✅ 100% transaction audit trail maintained
• ✅ Fraud detection within 2 seconds
• ✅ 99.99% transaction success rate
• ✅ Regulatory compliance reports automated
• ✅ 90% reduction in manual review time

📱 Mobile App Notification System

💡 Solution:

# Intelligent notification system
class NotificationWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'notifications', retry: 2
  
  def perform(user_id, notification_type, data = {})
    user = User.find(user_id)
    
    # Check user preferences
    return unless user.notifications_enabled?
    
    # Check time zone and quiet hours
    return if user.in_quiet_hours?
    
    # Personalize notification
    notification = personalize_notification(user, notification_type, data)
    
    # Send via multiple channels
    send_push_notification(user, notification)
    send_email_notification(user, notification) if user.email_enabled?
    send_sms_notification(user, notification) if user.sms_enabled?
    
    # Track delivery
    NotificationDelivery.create!(
      user: user,
      notification_type: notification_type,
      sent_at: Time.current
    )
  end
end

class BatchNotificationWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'batch_notifications'
  
  def perform(campaign_id)
    campaign = Campaign.find(campaign_id)
    
    # Get target users
    users = campaign.target_users
    
    # Process in batches
    users.in_batches(of: 1000) do |batch|
      batch.each do |user|
        NotificationWorker.perform_async(
          user.id, 
          'campaign', 
          { campaign_id: campaign.id }
        )
      end
    end
  end
end

class EngagementOptimizationWorker
  include Sidekiq::Worker
  sidekiq_options queue: 'engagement', retry: 1
  
  def perform(user_id)
    user = User.find(user_id)
    
    # Analyze user engagement
    engagement_score = calculate_engagement_score(user)
    
    # Adjust notification frequency
    if engagement_score < 0.3
      # Reduce notifications for low-engagement users
      user.update!(notification_frequency: 'reduced')
    elsif engagement_score > 0.8
      # Increase notifications for high-engagement users
      user.update!(notification_frequency: 'increased')
    end
  end

📊 Results:

• ✅ 10 million notifications sent daily
• ✅ 95% delivery rate across all channels
• ✅ 40% increase in user engagement
• ✅ 60% reduction in notification fatigue
• ✅ Personalized delivery timing for each user

📋11. Commands & Concepts Reference Table

Core Commands

Ruby/Rails Commands

Redis Commands

Monitoring Commands

Core Concepts

Configuration Options

Redis Data Structures

Environment Variables

Common Job Patterns