sidekiq-assured-jobs 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 213608148534e316bf24588d51b181572130d503b664db631ce2a0bb5be53f1a
+  data.tar.gz: 5e158bcc84da84d90ed15918ba1216ad6eddcf555aa3f9ca923bf840551e57ad
+SHA512:
+  metadata.gz: e63bf1b133743c43def063eede4e64a94a19d10ab90775f5cf0b36e0cdf9a50dc0597c78e349a4df2c650119b02db9cc8fb165b911ffff4af5cdd7a570df249a
+  data.tar.gz: f41c16f24cf7273f41a76f39c301f4398ed989496355cbe37097ebe7fedf03b798bf623e48c632026d46996caf9e41cf134a19fb6479e914365e026f47df70e3

data/CHANGELOG.md

@@ -0,0 +1,52 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.0.0] - 2025-06-20
+
+### Added
+- **Initial Release**: First public release of sidekiq-assured-jobs
+- **Job Assurance**: Guarantees that critical Sidekiq jobs are never lost due to worker crashes
+- **Automatic Recovery**: Detects and re-enqueues orphaned jobs from crashed workers
+- **Delayed Recovery System**: Configurable additional recovery passes for enhanced reliability
+- **Production Ready**: Designed for high-throughput production environments
+- **Zero Configuration**: Works out of the box with sensible defaults
+- **Sidekiq Integration**: Uses Sidekiq's existing Redis connection pool
+- **Distributed Locking**: Prevents duplicate recovery operations
+- **Minimal Overhead**: Lightweight tracking with configurable heartbeat intervals
+
+### Features
+- **Job Tracking**: Tracks in-flight jobs in Redis with instance identification
+- **Heartbeat System**: Monitors worker instance health with configurable intervals
+- **Orphan Recovery**: Automatically re-enqueues jobs from crashed instances
+- **Background Recovery Threads**: Automatic spawning of delayed recovery threads after startup
+- **Configurable Safety Net**: Multiple recovery passes to catch edge cases and network partition scenarios
+- **Selective Tracking**: Only tracks workers that include the AssuredJobs::Worker module
+- **SidekiqUniqueJobs Integration**: Automatically handles unique job lock clearing
+- **Flexible Redis Configuration**: Uses Sidekiq's Redis by default, supports custom Redis
+
+### Configuration Options
+- `ASSURED_JOBS_INSTANCE_ID`: Unique instance identifier (auto-generated if not set)
+- `ASSURED_JOBS_NS`: Redis namespace for keys (default: "sidekiq_assured_jobs")
+- `ASSURED_JOBS_HEARTBEAT_INTERVAL`: Seconds between heartbeats (default: 15)
+- `ASSURED_JOBS_HEARTBEAT_TTL`: Instance timeout threshold (default: 45)
+- `ASSURED_JOBS_RECOVERY_LOCK_TTL`: Recovery operation lock duration (default: 300)
+- `ASSURED_JOBS_DELAYED_RECOVERY_INTERVAL`: Seconds between delayed recovery passes (default: 300)
+- `ASSURED_JOBS_DELAYED_RECOVERY_COUNT`: Number of delayed recovery passes to run (default: 1)
+
+### Dependencies
+- Ruby >= 2.6.0
+- Sidekiq >= 6.0, < 7
+- Redis ~> 4.0
+
+### Breaking Changes from sidekiq-processing-tracker
+- **Gem Name**: Changed from `sidekiq-processing-tracker` to `sidekiq-assured-jobs`
+- **Module Name**: Changed from `Sidekiq::ProcessingTracker` to `Sidekiq::AssuredJobs`
+- **Worker Mixin**: Changed from `ProcessingTracker::Worker` to `AssuredJobs::Worker`
+- **Sidekiq Option**: Changed from `processing: true` to `assured_jobs: true`
+- **Environment Variables**: All prefixed with `ASSURED_JOBS_` instead of `PROCESSING_`
+- **Default Namespace**: Changed from `sidekiq_processing` to `sidekiq_assured_jobs`
+- **Logging**: Reduced verbose logging for production use

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Sidekiq Assured Jobs Team
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,475 @@
+# Sidekiq Assured Jobs
+
+Reliable job execution guarantee for Sidekiq with automatic orphan recovery.
+
+## Overview
+
+Sidekiq Assured Jobs ensures that your critical Sidekiq jobs are never lost due to worker crashes, pod restarts, or unexpected shutdowns. It provides a robust tracking system that monitors in-flight jobs and automatically recovers any work that was interrupted.
+
+**Perfect for:**
+- Critical business processes that cannot be lost
+- Financial transactions and payment processing
+- Data synchronization and ETL operations
+- Email delivery and notification systems
+- Any job where reliability is paramount
+
+## Key Features
+
+- **🛡️ Job Assurance**: Guarantees that tracked jobs will complete or be automatically retried
+- **🔄 Automatic Recovery**: Detects and re-enqueues orphaned jobs from crashed workers
+- **⏰ Delayed Recovery**: Configurable additional recovery passes for enhanced reliability
+- **⚡ Zero Configuration**: Works out of the box with sensible defaults
+- **🏗️ Production Ready**: Designed for high-throughput production environments
+- **🔗 Sidekiq Integration**: Uses Sidekiq's existing Redis connection pool
+- **🔒 Distributed Locking**: Prevents duplicate recovery operations
+- **📊 Minimal Overhead**: Lightweight tracking with configurable heartbeat intervals
+
+## The Problem
+
+When Sidekiq workers crash or are forcefully terminated (SIGKILL), jobs that were being processed are lost forever:
+
+```mermaid
+sequenceDiagram
+    participant Client
+    participant Queue as Sidekiq Queue
+    participant Worker as Worker Process
+    participant Redis
+
+    Client->>Queue: Enqueue Critical Job
+    Queue->>Worker: Fetch Job
+    Worker->>Redis: Job starts processing
+    Note over Worker: Worker crashes (SIGKILL)
+    Worker--xRedis: Job lost forever
+    Note over Queue: No retry, no error handling
+    Note over Client: Critical work never completed
+```
+
+## The Solution
+
+Sidekiq Assured Jobs tracks in-flight jobs and automatically recovers them:
+
+```mermaid
+graph TB
+    subgraph Cluster["Production Environment"]
+        subgraph W1["Worker Instance 1"]
+            SW1[Sidekiq Worker]
+            HB1[Heartbeat]
+            MW1[Tracking Middleware]
+        end
+
+        subgraph W2["Worker Instance 2"]
+            SW2[Sidekiq Worker]
+            HB2[Heartbeat]
+            MW2[Tracking Middleware]
+        end
+    end
+
+    subgraph Redis["Redis Storage"]
+        HK["Heartbeats<br/>instance:worker-1<br/>instance:worker-2"]
+        JT["Job Tracking<br/>jobs:worker-1<br/>jobs:worker-2"]
+        JP["Job Payloads<br/>job:abc123<br/>job:def456"]
+        RL["Recovery Lock"]
+    end
+
+    Queue[Sidekiq Queue]
+
+    HB1 -->|Every 15s| HK
+    HB2 -->|Every 15s| HK
+
+    MW1 -->|Track Start/End| JT
+    MW1 -->|Store Payload| JP
+    MW2 -->|Track Start/End| JT
+    MW2 -->|Store Payload| JP
+
+    SW2 -->|On Startup| RL
+    SW2 -->|Detect Orphans| JT
+    SW2 -->|Re-enqueue| Queue
+
+    style HK fill:#e8f5e8
+    style JT fill:#fff3e0
+    style JP fill:#e3f2fd
+    style RL fill:#ffebee
+```
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'sidekiq-assured-jobs'
+```
+
+And then execute:
+
+```bash
+bundle install
+```
+
+## Quick Start
+
+### 1. Basic Setup
+
+The gem auto-configures itself when required:
+
+```ruby
+# In your application (e.g., config/application.rb or config/initializers/sidekiq.rb)
+require 'sidekiq-assured-jobs'
+```
+
+### 2. Enable Job Tracking
+
+Include the `AssuredJobs::Worker` module in workers you want to track:
+
+```ruby
+class PaymentProcessor
+  include Sidekiq::Worker
+  include Sidekiq::AssuredJobs::Worker  # Enables job assurance
+
+  def perform(payment_id, amount)
+    # This job will be tracked and recovered if the worker crashes
+    process_payment(payment_id, amount)
+  end
+end
+
+class LogCleanupWorker
+  include Sidekiq::Worker
+  # No AssuredJobs::Worker - not tracked (fine for non-critical work)
+
+  def perform
+    # This job won't be tracked
+    cleanup_old_logs
+  end
+end
+```
+
+### 3. That's It!
+
+Your critical jobs are now protected. If a worker crashes while processing a tracked job, another worker will automatically detect and re-enqueue it.
+
+## Configuration
+
+The gem works with zero configuration but provides extensive customization options. See the [Complete Configuration Reference](#complete-configuration-reference) below for all available options.
+
+
+
+## Complete Configuration Reference
+
+### Core Configuration Options
+
+| Option | Environment Variable | Default | Description |
+|--------|---------------------|---------|-------------|
+| `instance_id` | `ASSURED_JOBS_INSTANCE_ID` | Auto-generated | Unique identifier for this worker instance |
+| `namespace` | `ASSURED_JOBS_NS` | `sidekiq_assured_jobs` | Redis namespace for all keys |
+| `heartbeat_interval` | `ASSURED_JOBS_HEARTBEAT_INTERVAL` | `15` | Seconds between heartbeat updates |
+| `heartbeat_ttl` | `ASSURED_JOBS_HEARTBEAT_TTL` | `45` | Seconds before instance considered dead |
+| `recovery_lock_ttl` | `ASSURED_JOBS_RECOVERY_LOCK_TTL` | `300` | Seconds to hold recovery lock |
+| `delayed_recovery_interval` | `ASSURED_JOBS_DELAYED_RECOVERY_INTERVAL` | `300` | Seconds between delayed recovery passes |
+| `delayed_recovery_count` | `ASSURED_JOBS_DELAYED_RECOVERY_COUNT` | `1` | Number of delayed recovery passes to run |
+
+### Configuration Methods
+
+#### Environment Variables (Recommended for Production)
+```bash
+export ASSURED_JOBS_INSTANCE_ID="worker-pod-1"
+export ASSURED_JOBS_NS="myapp_assured_jobs"
+export ASSURED_JOBS_HEARTBEAT_INTERVAL="30"
+export ASSURED_JOBS_HEARTBEAT_TTL="90"
+export ASSURED_JOBS_RECOVERY_LOCK_TTL="600"
+export ASSURED_JOBS_DELAYED_RECOVERY_INTERVAL="600"
+export ASSURED_JOBS_DELAYED_RECOVERY_COUNT="2"
+```
+
+#### Programmatic Configuration
+```ruby
+Sidekiq::AssuredJobs.configure do |config|
+  config.namespace = "myapp_assured_jobs"
+  config.heartbeat_interval = 30
+  config.heartbeat_ttl = 90
+  config.recovery_lock_ttl = 600
+  config.delayed_recovery_interval = 600
+  config.delayed_recovery_count = 2
+
+  # Advanced: Custom Redis configuration
+  config.redis_options = {
+    url: ENV['ASSURED_JOBS_REDIS_URL'],
+    db: 2,
+    timeout: 5
+  }
+end
+```
+
+### Configuration Guidelines
+
+#### Heartbeat Settings
+- **`heartbeat_interval`**: How often workers send "I'm alive" signals
+  - Lower values = faster orphan detection, higher Redis load
+  - Recommended: 15-30 seconds for production
+- **`heartbeat_ttl`**: How long to wait before considering an instance dead
+  - Should be 2-3x the heartbeat interval
+  - Accounts for network delays and Redis latency
+
+#### Recovery Settings
+- **`recovery_lock_ttl`**: How long one instance holds the recovery lock
+  - Prevents multiple instances from recovering the same jobs
+  - Should be longer than expected recovery time
+- **`delayed_recovery_interval`**: Time between additional recovery passes
+  - Provides safety net for missed orphans
+  - Recommended: 5-10 minutes for most applications
+- **`delayed_recovery_count`**: Number of additional recovery attempts
+  - Balance between reliability and resource usage
+  - Recommended: 1-3 passes for most applications
+
+### Production Recommendations
+
+#### High-Availability Setup
+```ruby
+Sidekiq::AssuredJobs.configure do |config|
+  config.namespace = "#{Rails.application.class.module_parent_name.downcase}_assured_jobs"
+  config.heartbeat_interval = 30      # Balanced load vs detection speed
+  config.heartbeat_ttl = 90           # 3x heartbeat interval
+  config.recovery_lock_ttl = 900      # 15 minutes for large recovery operations
+  config.delayed_recovery_interval = 600  # 10 minutes between passes
+  config.delayed_recovery_count = 2   # Two additional safety passes
+end
+```
+
+#### Resource-Constrained Environment
+```ruby
+Sidekiq::AssuredJobs.configure do |config|
+  config.heartbeat_interval = 60     # Reduce Redis load
+  config.heartbeat_ttl = 180         # 3x heartbeat interval
+  config.delayed_recovery_count = 1  # Single delayed pass
+end
+```
+
+#### Critical Systems (Maximum Reliability)
+```ruby
+Sidekiq::AssuredJobs.configure do |config|
+  config.heartbeat_interval = 15     # Fast orphan detection
+  config.heartbeat_ttl = 45          # Quick failure detection
+  config.delayed_recovery_interval = 300  # 5 minutes between passes
+  config.delayed_recovery_count = 3  # Three additional passes
+end
+```
+
+### Delayed Recovery System
+
+In addition to immediate orphan recovery on startup, the gem provides a configurable delayed recovery system for enhanced reliability:
+
+```ruby
+Sidekiq::AssuredJobs.configure do |config|
+  # Run 2 additional recovery passes, 10 minutes apart
+  config.delayed_recovery_count = 2
+  config.delayed_recovery_interval = 600  # 10 minutes
+end
+```
+
+**How Delayed Recovery Works:**
+
+1. **Immediate Recovery**: On startup, each worker instance performs immediate orphan recovery
+2. **Delayed Passes**: After startup, a background thread runs additional recovery passes
+3. **Configurable Timing**: Control both the interval between passes and total number of passes
+4. **Error Resilience**: Each delayed recovery pass is wrapped in error handling to prevent thread crashes
+
+**Benefits:**
+- **Enhanced Reliability**: Catches jobs that might be missed during startup recovery
+- **Network Partition Recovery**: Handles cases where Redis connectivity issues cause temporary orphaning
+- **Race Condition Mitigation**: Provides additional safety net for edge cases
+- **Zero Application Impact**: Runs in background threads without affecting job processing
+
+**Use Cases:**
+- **High-Availability Systems**: Where maximum job recovery reliability is critical
+- **Network-Unstable Environments**: Where Redis connectivity might be intermittent
+- **Large-Scale Deployments**: Where startup recovery might miss some edge cases
+
+## Advanced Features
+
+### Redis Integration
+
+The gem provides flexible Redis integration options:
+
+#### Default Configuration (Recommended)
+By default, the gem uses Sidekiq's existing Redis connection pool:
+
+```ruby
+# Uses Sidekiq's Redis configuration automatically
+Sidekiq::AssuredJobs.configure do |config|
+  config.namespace = "my_app_assured_jobs"
+end
+```
+
+#### Custom Redis Configuration (Advanced)
+For advanced use cases requiring Redis isolation:
+
+```ruby
+Sidekiq::AssuredJobs.configure do |config|
+  config.namespace = "my_app_assured_jobs"
+  config.redis_options = {
+    url: ENV['ASSURED_JOBS_REDIS_URL'],
+    db: 2,
+    timeout: 5
+  }
+end
+```
+
+### Benefits
+- **Connection Efficiency**: Reuses Sidekiq's connection pool by default
+- **Custom Namespacing**: Efficient key prefixing without external dependencies
+- **Configuration Consistency**: Inherits Sidekiq's Redis settings
+- **Flexible Options**: Support for custom Redis when needed
+
+### SidekiqUniqueJobs Integration
+
+The gem automatically integrates with [sidekiq-unique-jobs](https://github.com/mhenrixon/sidekiq-unique-jobs) to ensure orphaned unique jobs can be recovered immediately:
+
+```ruby
+class UniquePaymentProcessor
+  include Sidekiq::Worker
+  include Sidekiq::AssuredJobs::Worker
+
+  sidekiq_options unique: :until_executed
+
+  def perform(payment_id)
+    # This job will be tracked and can be recovered even with unique constraints
+    process_payment(payment_id)
+  end
+end
+```
+
+**Benefits:**
+- **Immediate Recovery**: Orphaned unique jobs are re-enqueued immediately (no waiting period)
+- **Automatic Detection**: Works seamlessly whether SidekiqUniqueJobs is present or not
+- **Surgical Precision**: Only clears locks for confirmed orphaned jobs
+- **Error Resilience**: Continues operation even if lock clearing fails
+
+## Production Deployment
+
+### Kubernetes Example
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: sidekiq-workers
+spec:
+  replicas: 3
+  template:
+    spec:
+      containers:
+      - name: worker
+        image: myapp:latest
+        env:
+        - name: ASSURED_JOBS_INSTANCE_ID
+          valueFrom:
+            fieldRef:
+              fieldPath: metadata.name  # Use pod name as instance ID
+        - name: ASSURED_JOBS_NS
+          value: "myapp_assured_jobs"
+        - name: ASSURED_JOBS_HEARTBEAT_INTERVAL
+          value: "30"
+        - name: ASSURED_JOBS_HEARTBEAT_TTL
+          value: "90"
+        - name: ASSURED_JOBS_RECOVERY_LOCK_TTL
+          value: "600"
+        - name: ASSURED_JOBS_DELAYED_RECOVERY_INTERVAL
+          value: "600"  # 10 minutes
+        - name: ASSURED_JOBS_DELAYED_RECOVERY_COUNT
+          value: "2"
+```
+
+### Docker Compose Example
+
+```yaml
+version: '3.8'
+services:
+  worker:
+    image: myapp:latest
+    environment:
+      - ASSURED_JOBS_INSTANCE_ID=${HOSTNAME}
+      - ASSURED_JOBS_NS=myapp_assured_jobs
+      - ASSURED_JOBS_HEARTBEAT_INTERVAL=30
+      - ASSURED_JOBS_HEARTBEAT_TTL=90
+      - ASSURED_JOBS_RECOVERY_LOCK_TTL=600
+      - ASSURED_JOBS_DELAYED_RECOVERY_INTERVAL=600
+      - ASSURED_JOBS_DELAYED_RECOVERY_COUNT=2
+    deploy:
+      replicas: 3
+```
+
+## How It Works
+
+1. **Instance Registration**: Each worker instance generates a unique ID and sends periodic heartbeats to Redis
+2. **Job Tracking**: When a tracked job starts, the middleware records the job ID and payload in Redis
+3. **Job Cleanup**: When a job completes (success or failure), tracking data is removed
+4. **Immediate Recovery**: On startup, workers check for jobs tracked by dead instances (no recent heartbeat)
+5. **Safe Recovery**: Using distributed locking, one worker re-enqueues orphaned jobs back to Sidekiq
+6. **Delayed Recovery**: Background threads run additional recovery passes at configurable intervals
+7. **Cleanup**: Orphaned tracking data is removed after successful re-enqueuing
+
+## Use Cases
+
+### Financial Services
+```ruby
+class PaymentProcessor
+  include Sidekiq::Worker
+  include Sidekiq::AssuredJobs::Worker
+
+  def perform(payment_id, amount)
+    # Critical: Payment must be processed
+    process_payment(payment_id, amount)
+  end
+end
+```
+
+### Data Synchronization
+```ruby
+class DataSyncWorker
+  include Sidekiq::Worker
+  include Sidekiq::AssuredJobs::Worker
+
+  def perform(sync_batch_id)
+    # Important: Data consistency depends on completion
+    sync_data_batch(sync_batch_id)
+  end
+end
+```
+
+### Email Delivery
+```ruby
+class CriticalEmailWorker
+  include Sidekiq::Worker
+  include Sidekiq::AssuredJobs::Worker
+
+  def perform(email_id)
+    # Must deliver: Password resets, order confirmations, etc.
+    deliver_critical_email(email_id)
+  end
+end
+```
+
+## Testing
+
+Run the test suite:
+
+```bash
+bundle exec rspec
+```
+
+## Dependencies
+
+### Runtime Dependencies
+- `sidekiq` (>= 6.0, < 7)
+- `redis` (~> 4.0)
+
+### Development Dependencies
+- `rspec` (~> 3.0)
+- `bundler` (~> 2.0)
+- `rubocop` (~> 1.0)
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/example/sidekiq-assured-jobs.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/lib/sidekiq/assured_jobs/middleware.rb

@@ -0,0 +1,68 @@
+# frozen_string_literal: true
+
+module Sidekiq
+  module AssuredJobs
+    class Middleware
+      def call(worker, job, queue)
+        # Only track jobs that have assured_jobs: true option
+        should_track = should_track_job?(worker, job)
+        
+        return yield unless should_track
+
+        jid = job["jid"]
+        instance_id = AssuredJobs.instance_id
+        logger = AssuredJobs.logger
+
+        # Create tracking keys (using custom namespacing)
+        job_tracking_key = AssuredJobs.send(:namespaced_key, "jobs:#{instance_id}")
+        job_data_key = AssuredJobs.send(:namespaced_key, "job:#{jid}")
+
+        begin
+          # Add job to tracking set and store job payload
+          begin
+            AssuredJobs.redis_sync do |conn|
+              conn.multi do |multi|
+                multi.sadd(job_tracking_key, jid)
+                multi.set(job_data_key, job.to_json)
+              end
+            end
+            logger.debug "AssuredJobs started tracking job #{jid} on instance #{instance_id}"
+          rescue => e
+            logger.error "AssuredJobs failed to start tracking job #{jid}: #{e.message}"
+            logger.error e.backtrace.join("\n")
+          end
+
+          # Execute the job
+          yield
+        ensure
+          # Remove job from tracking
+          begin
+            AssuredJobs.redis_sync do |conn|
+              conn.multi do |multi|
+                multi.srem(job_tracking_key, jid)
+                multi.del(job_data_key)
+              end
+            end
+            logger.debug "AssuredJobs stopped tracking job #{jid} on instance #{instance_id}"
+          rescue => e
+            logger.error "AssuredJobs failed to stop tracking job #{jid}: #{e.message}"
+          end
+        end
+      end
+
+      private
+
+      def should_track_job?(worker, job)
+        # Check if the worker class has assured_jobs: true option
+        worker_class = worker.is_a?(Class) ? worker : worker.class
+
+        unless worker_class.respond_to?(:sidekiq_options)
+          return false
+        end
+
+        options = worker_class.sidekiq_options
+        options["assured_jobs"] == true
+      end
+    end
+  end
+end

data/lib/sidekiq/assured_jobs/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Sidekiq
+  module AssuredJobs
+    VERSION = "1.0.0"
+  end
+end

data/lib/sidekiq/assured_jobs/worker.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+module Sidekiq
+  module AssuredJobs
+    module Worker
+      def self.included(base)
+        base.extend(ClassMethods)
+        base.sidekiq_options assured_jobs: true
+      end
+
+      module ClassMethods
+        # Additional class methods can be added here if needed in the future
+      end
+    end
+  end
+end

data/lib/sidekiq-assured-jobs.rb

@@ -0,0 +1,267 @@
+# frozen_string_literal: true
+
+require "sidekiq"
+require "redis"
+require "logger"
+require "securerandom"
+require "set"
+
+require_relative "sidekiq/assured_jobs/version"
+require_relative "sidekiq/assured_jobs/middleware"
+require_relative "sidekiq/assured_jobs/worker"
+
+module Sidekiq
+  module AssuredJobs
+    class Error < StandardError; end
+
+    class << self
+      attr_accessor :instance_id, :namespace, :heartbeat_interval, :heartbeat_ttl, :recovery_lock_ttl, :logger, :redis_options, :delayed_recovery_count, :delayed_recovery_interval
+
+      def configure
+        yield self if block_given?
+        setup_defaults
+        setup_sidekiq_hooks
+      end
+
+      def redis(&block)
+        if redis_options
+          # Use custom Redis configuration if provided
+          redis_client = Redis.new(redis_options)
+          if block_given?
+            result = yield redis_client
+            redis_client.close
+            result
+          else
+            redis_client
+          end
+        else
+          # Use Sidekiq's Redis connection pool
+          Sidekiq.redis(&block)
+        end
+      end
+
+      def redis_sync(&block)
+        # Synchronous Redis operations using Sidekiq's pool or custom config
+        redis(&block)
+      end
+
+      # Helper method to add namespace prefix to Redis keys
+      def namespaced_key(key)
+        "#{namespace}:#{key}"
+      end
+
+      # Clear unique-jobs lock for orphaned jobs to allow immediate re-enqueuing
+      def clear_unique_jobs_lock(job_data)
+        return unless job_data['unique_digest']
+
+        begin
+          # Check if SidekiqUniqueJobs is available
+          if defined?(SidekiqUniqueJobs::Digests)
+            SidekiqUniqueJobs::Digests.del(digest: job_data['unique_digest'])
+            logger.info "AssuredJobs cleared unique-jobs lock for job #{job_data['jid']} with digest #{job_data['unique_digest']}"
+          else
+            logger.debug "AssuredJobs: SidekiqUniqueJobs not available, skipping lock cleanup for job #{job_data['jid']}"
+          end
+        rescue => e
+          logger.warn "AssuredJobs failed to clear unique-jobs lock for job #{job_data['jid']}: #{e.message}"
+        end
+      end
+
+      def reenqueue_orphans!
+        with_recovery_lock do
+          logger.info "AssuredJobs starting orphan job recovery"
+
+          redis_sync do |conn|
+            # Get all job keys and instance keys (using custom namespacing)
+            job_keys = conn.keys(namespaced_key("jobs:*"))
+            instance_keys = conn.keys(namespaced_key("instance:*"))
+
+            # Extract instance IDs from keys
+            live_instances = instance_keys.map { |key| key.split(":").last }.to_set
+
+            orphaned_jobs = []
+
+            job_keys.each do |job_key|
+              instance_id = job_key.split(":").last
+              unless live_instances.include?(instance_id)
+                # Get all job IDs for this dead instance
+                job_ids = conn.smembers(job_key)
+
+                job_ids.each do |jid|
+                  # Get the job payload
+                  job_data_key = namespaced_key("job:#{jid}")
+                  job_payload = conn.get(job_data_key)
+
+                  if job_payload
+                    orphaned_jobs << JSON.parse(job_payload)
+                    # Clean up the job data key
+                    conn.del(job_data_key)
+                  end
+                end
+
+                # Clean up the job tracking key
+                conn.del(job_key)
+              end
+            end
+
+            if orphaned_jobs.any?
+              logger.info "AssuredJobs found #{orphaned_jobs.size} orphaned jobs, re-enqueuing"
+              orphaned_jobs.each do |job_data|
+                # Clear unique-jobs lock before re-enqueuing to avoid lock conflicts
+                clear_unique_jobs_lock(job_data)
+
+                Sidekiq::Client.push(job_data)
+                logger.info "AssuredJobs re-enqueued job #{job_data['jid']} (#{job_data['class']})"
+              end
+            else
+              logger.info "AssuredJobs found no orphaned jobs"
+            end
+          end
+        end
+      rescue => e
+        logger.error "AssuredJobs orphan recovery failed: #{e.message}"
+        logger.error e.backtrace.join("\n")
+      end
+
+      def setup_sidekiq_hooks
+        return unless defined?(Sidekiq::VERSION)
+
+        Sidekiq.configure_server do |config|
+          config.server_middleware do |chain|
+            chain.add AssuredJobs::Middleware
+          end
+
+          # Add startup hook for heartbeat and orphan recovery
+          config.on(:startup) do
+            # Ensure configuration is set up
+            setup_defaults unless @instance_id
+
+            logger.info "AssuredJobs starting up on instance #{instance_id}"
+
+            # Start heartbeat system
+            setup_heartbeat
+
+            # Run orphan recovery on startup only
+            Thread.new do
+              sleep 5 # Give the server a moment to fully start
+              begin
+                reenqueue_orphans!
+                spinup_delayed_recovery_thread
+              rescue => e
+                logger.error "AssuredJobs startup orphan recovery failed: #{e.message}"
+                logger.error e.backtrace.join("\n")
+              end
+            end
+          end
+
+          # Add shutdown hook to clean up
+          config.on(:shutdown) do
+            logger.info "AssuredJobs shutting down instance #{instance_id}"
+            begin
+              # Stop heartbeat thread
+              if @heartbeat_thread&.alive?
+                @heartbeat_thread.kill
+                @heartbeat_thread = nil
+              end
+
+              redis_sync do |conn|
+                # Only clean up instance heartbeat - let orphan recovery handle job cleanup
+                # This ensures that if there are running jobs during shutdown, they will be
+                # detected as orphaned and recovered by the next instance
+                conn.del(namespaced_key("instance:#{instance_id}"))
+
+                # Log tracked jobs but don't clean them up - they should be recovered as orphans
+                job_tracking_key = namespaced_key("jobs:#{instance_id}")
+                tracked_jobs = conn.smembers(job_tracking_key)
+
+                if tracked_jobs.any?
+                  logger.warn "AssuredJobs leaving #{tracked_jobs.size} tracked jobs for orphan recovery: #{tracked_jobs.join(', ')}"
+                  logger.info "AssuredJobs: These jobs will be recovered by the next instance startup"
+                else
+                  logger.info "AssuredJobs: No tracked jobs to leave for recovery"
+                end
+              end
+            rescue => e
+              logger.error "AssuredJobs shutdown cleanup failed: #{e.message}"
+            end
+          end
+        end
+      end
+
+      private
+
+      def setup_defaults
+        @instance_id ||= ENV.fetch("ASSURED_JOBS_INSTANCE_ID") { SecureRandom.hex(8) }
+        @namespace ||= ENV.fetch("ASSURED_JOBS_NS", "sidekiq_assured_jobs")
+        @heartbeat_interval ||= ENV.fetch("ASSURED_JOBS_HEARTBEAT_INTERVAL", "15").to_i
+        @heartbeat_ttl ||= ENV.fetch("ASSURED_JOBS_HEARTBEAT_TTL", "45").to_i
+        @recovery_lock_ttl ||= ENV.fetch("ASSURED_JOBS_RECOVERY_LOCK_TTL", "300").to_i
+        @logger ||= Sidekiq.logger
+        @delayed_recovery_count ||= ENV.fetch("ASSURED_JOBS_DELAYED_RECOVERY_COUNT", "1").to_i
+        @delayed_recovery_interval ||= ENV.fetch("ASSURED_JOBS_DELAYED_RECOVERY_INTERVAL", "300").to_i
+      end
+
+      def setup_heartbeat
+        # Initial heartbeat
+        send_heartbeat
+
+        # Background heartbeat thread
+        @heartbeat_thread = Thread.new do
+          loop do
+            sleep heartbeat_interval
+            begin
+              send_heartbeat
+            rescue => e
+              logger.error "AssuredJobs heartbeat failed: #{e.message}"
+            end
+          end
+        end
+      end
+
+      def send_heartbeat
+        key = namespaced_key("instance:#{instance_id}")
+        redis_sync do |conn|
+          conn.setex(key, heartbeat_ttl, Time.now.to_f)
+        end
+        logger.debug "AssuredJobs heartbeat sent for instance #{instance_id}"
+      end
+
+      def spinup_delayed_recovery_thread
+        Thread.new do
+          @delayed_recovery_count.times do |i|
+            sleep @delayed_recovery_interval
+            begin
+              reenqueue_orphans!
+            rescue => e
+              logger.error(
+                "[AssuredJobs] delayed recovery ##{i+1} failed: #{e.message}"
+              )
+            end
+          end
+        end
+      end
+      def with_recovery_lock
+        lock_key = namespaced_key("recovery_lock")
+        lock_acquired = redis_sync do |conn|
+          conn.set(lock_key, instance_id, nx: true, ex: recovery_lock_ttl)
+        end
+
+        if lock_acquired
+          logger.info "AssuredJobs recovery lock acquired by instance #{instance_id}"
+          begin
+            yield
+          ensure
+            redis_sync { |conn| conn.del(lock_key) }
+            logger.info "AssuredJobs recovery lock released by instance #{instance_id}"
+          end
+        else
+          logger.debug "AssuredJobs recovery lock not acquired, another instance is handling recovery"
+        end
+      end
+    end
+  end
+end
+
+# Auto-setup defaults and Sidekiq hooks when gem is required
+Sidekiq::AssuredJobs.send(:setup_defaults)
+Sidekiq::AssuredJobs.setup_sidekiq_hooks

metadata

@@ -0,0 +1,131 @@
+--- !ruby/object:Gem::Specification
+name: sidekiq-assured-jobs
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- Manikanta Gopi
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2025-06-30 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: sidekiq
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '7'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '6.0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '7'
+- !ruby/object:Gem::Dependency
+  name: redis
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '4.0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: rubocop
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+description: Ensures Sidekiq jobs are never lost due to worker crashes or restarts
+  by tracking in-flight jobs and automatically recovering orphaned work
+email:
+- gopimanikanta50@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE.txt
+- README.md
+- lib/sidekiq-assured-jobs.rb
+- lib/sidekiq/assured_jobs/middleware.rb
+- lib/sidekiq/assured_jobs/version.rb
+- lib/sidekiq/assured_jobs/worker.rb
+homepage: https://github.com/praja/sidekiq-assured-jobs
+licenses:
+- MIT
+metadata:
+  allowed_push_host: https://rubygems.org
+  homepage_uri: https://github.com/praja/sidekiq-assured-jobs
+  source_code_uri: https://github.com/praja/sidekiq-assured-jobs
+  changelog_uri: https://github.com/praja/sidekiq-assured-jobs/blob/main/CHANGELOG.md
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.6.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.4.17
+signing_key: 
+specification_version: 4
+summary: Reliable job execution guarantee for Sidekiq with automatic orphan recovery
+test_files: []