sidekiq_solid_fetch 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 3f9d49a1edc5ffd25c20302adac6f5fea21eb3aacab3c441d7a8722bb42faeca
+  data.tar.gz: 8c137210e8a8febf7409379c7c207270f0c3964ae378898122cc7e1a6f7ae88d
+SHA512:
+  metadata.gz: 2df35c34eed28f7f1c3ed78e853da85b80262a21702e4334699a0c7dadc5ba2f62059dba9dea116882ee71edbfa5c503de1f682b4371e021d580409cd8640322
+  data.tar.gz: a3ffb8214437b7f4b579465b03268b3a4a4a393d481036f5ac4e90b510174ef7220f5d88a93f54038712f6e6731882889536396c7eed71b08e528163da0670e4

data/.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "0.1.1"
+}

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--require spec_helper

data/.standard.yml

@@ -0,0 +1,3 @@
+# For available configuration options, see:
+#   https://github.com/standardrb/standard
+ruby_version: 3.1

data/CHANGELOG.md

@@ -0,0 +1,10 @@
+# Changelog
+
+## [0.1.1](https://github.com/k0va1/sidekiq_solid_fetch/compare/v0.1.0...v0.1.1) (2025-12-04)
+
+
+### Features
+
+* basic implementation ([08809c1](https://github.com/k0va1/sidekiq_solid_fetch/commit/08809c11ff15a623b592721785f76706bb62557d))
+
+## Changelog

data/CLAUDE.md

@@ -0,0 +1,45 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+SidekiqSolidFetch is an open-source Ruby gem that provides a reliable fetch strategy for Sidekiq, similar to Sidekiq Pro's `super_fetch`. It uses Redis LMOVE to atomically move jobs from the queue to a processing queue, ensuring jobs aren't lost if a worker crashes.
+
+## Commands
+
+- **Install dependencies**: `bundle install`
+- **Run all tests**: `bundle exec rake spec`
+- **Run a single test file**: `bundle exec rspec spec/path/to/file_spec.rb`
+- **Run a single test by line**: `bundle exec rspec spec/path/to/file_spec.rb:LINE`
+- **Lint code**: `bundle exec rake standard`
+- **Auto-fix lint issues**: `bundle exec standardrb --fix`
+- **Run all checks (tests + lint)**: `bundle exec rake`
+- **Interactive console**: `bin/console`
+
+## Architecture
+
+### Core Components
+
+- `SidekiqSolidFetch` (lib/sidekiq_solid_fetch.rb) - Main module with `enable!(config)` method to activate the custom fetcher
+- `SidekiqSolidFetch::Fetcher` (lib/sidekiq_solid_fetch/fetcher.rb) - Custom Sidekiq fetcher implementing reliable queue processing
+
+### Fetcher Design
+
+The Fetcher uses Redis LMOVE to atomically move jobs from the source queue to a per-worker processing queue (`processing:queue:{name}:{identity}`). This ensures:
+- Jobs are never lost if a worker crashes mid-execution
+- Unfinished jobs can be requeued via `bulk_requeue`
+
+Key classes:
+- `Fetcher` - Implements Sidekiq's fetch interface with `retrieve_work` and `bulk_requeue`
+- `UnitOfWork` - Struct wrapping a job with `acknowledge` and `requeue` methods
+
+### Queue Modes
+
+- **Strict mode** (`cap.mode == :strict`) - Processes queues in defined order
+- **Weighted mode** (default) - Shuffles queues to distribute load
+
+## Requirements
+
+- Ruby >= 3.1.0
+- Sidekiq >= 7.0

data/Makefile

@@ -0,0 +1,18 @@
+REDIS_URL ?= redis://localhost:6379/0
+
+.PHONY: test lint-fix console install
+
+install:
+	bundle install
+
+console:
+	bin/console
+
+test:
+	REDIS_URL=$(REDIS_URL) bundle exec rspec $(filter-out $@,$(MAKECMDGOALS))
+
+lint-fix:
+	bundle exec standardrb --fix
+
+%:
+	@:

data/README.md

@@ -0,0 +1,75 @@
+# SidekiqSolidFetch
+
+A reliable fetch strategy for Sidekiq that ensures jobs are never lost, even if a worker crashes mid-execution. This is an open-source implementation similar to Sidekiq Pro's `super_fetch`.
+
+## How It Works
+
+SidekiqSolidFetch uses Redis `LMOVE` command to atomically move jobs from the queue to a per-worker processing queue. This approach ensures:
+
+- **No job loss**: Jobs are moved (not copied) to a processing queue before execution. If a worker crashes, the job remains in the processing queue.
+- **Automatic recovery**: On startup, any unfinished jobs from previous runs are automatically requeued.
+- **Graceful shutdown**: When Sidekiq shuts down, in-progress jobs are moved back to their original queues.
+
+### Flow
+
+1. Worker fetches a job → job is atomically moved from `queue:default` to `sidekiq_solid_fetch:processing:queue:default`
+2. Worker processes the job successfully → job is removed from the processing queue
+3. Worker crashes → job stays in the processing queue
+4. On next startup → jobs in processing queues are moved back to their original queues
+
+## Installation
+
+Add to your Gemfile:
+
+```ruby
+gem "sidekiq_solid_fetch"
+```
+
+Then run:
+
+```bash
+bundle install
+```
+
+## Usage
+
+Enable SidekiqSolidFetch in your Sidekiq configuration:
+
+```ruby
+# config/initializers/sidekiq.rb (Rails)
+# or wherever you configure Sidekiq
+
+require "sidekiq_solid_fetch"
+
+Sidekiq.configure_server do |config|
+  SidekiqSolidFetch.enable!(config)
+end
+```
+
+That's it! SidekiqSolidFetch will now handle job fetching with crash recovery.
+
+## Requirements
+
+- Ruby >= 3.1.0
+- Sidekiq >= 7.0
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies.
+
+```bash
+# Run tests
+docker compose up -d
+make test
+
+# Run linter
+make lint-fix
+```
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/k0va1/sidekiq_solid_fetch.
+
+## License
+
+The gem is available as open source under the terms of the MIT License.

data/Rakefile

@@ -0,0 +1,8 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+require "standard/rake"
+
+task default: %i[spec standard]

data/docker-compose.yml

@@ -0,0 +1,16 @@
+services:
+  redis:
+    image: redis:8.4-alpine
+    network_mode: host
+    volumes:
+      - redis:/data
+    ports:
+      - 6379:6379
+    healthcheck:
+      test: redis-cli ping
+      interval: 1s
+      timeout: 3s
+      retries: 30
+
+volumes:
+  redis:

data/lib/sidekiq_solid_fetch/fetcher.rb

@@ -0,0 +1,57 @@
+require "sidekiq"
+require "sidekiq/component"
+require "sidekiq/capsule"
+
+module SidekiqSolidFetch
+  class Fetcher
+    include Sidekiq::Component
+
+    def initialize(cap)
+      raise ArgumentError, "missing queue list" unless cap.queues
+      @config = cap
+      @strictly_ordered_queues = cap.mode == :strict
+      @queues = config.queues.map { |q| "queue:#{q}" }
+      @queues.uniq! if @strictly_ordered_queues
+    end
+
+    def retrieve_work
+      queues_cmd.each do |queue|
+        processing_queue_name = ::SidekiqSolidFetch.processing_queue_name(queue)
+        work = redis do |conn|
+          conn.lmove(queue, processing_queue_name, "RIGHT", "LEFT")
+        end
+
+        return ::SidekiqSolidFetch::UnitOfWork.new(queue, work, config, processing_queue_name) if work
+      end
+      nil
+    end
+
+    def bulk_requeue(*)
+      logger.info("SidekiqSolidFetch: Re-queueing terminated jobs")
+
+      count = 0
+      redis do |conn|
+        queues_cmd.each do |queue|
+          processing_queue_name = ::SidekiqSolidFetch.processing_queue_name(queue)
+          while conn.lmove(processing_queue_name, queue, "RIGHT", "LEFT")
+            count += 1
+            logger.info { "SidekiqSolidFetch: Moving job from #{processing_queue_name} back to #{queue}" }
+          end
+        end
+      end
+      logger.info("SidekiqSolidFetch: Re-queued #{count} jobs")
+    rescue => ex
+      logger.warn("SidekiqSolidFetch: Failed to requeue jobs: #{ex.message}")
+    end
+
+    def queues_cmd
+      if @strictly_ordered_queues
+        @queues
+      else
+        permute = @queues.shuffle
+        permute.uniq!
+        permute
+      end
+    end
+  end
+end

data/lib/sidekiq_solid_fetch/unit_of_work.rb

@@ -0,0 +1,29 @@
+module SidekiqSolidFetch
+  class UnitOfWork
+    attr_accessor :queue, :job, :config, :processing_queue
+
+    def initialize(queue, job, config, processing_queue)
+      @queue = queue
+      @job = job
+      @config = config
+      @processing_queue = processing_queue
+    end
+
+    def acknowledge
+      config.redis { |conn| conn.lrem(processing_queue, -1, job) }
+    end
+
+    def queue_name
+      queue.delete_prefix("queue:")
+    end
+
+    def requeue
+      config.redis do |conn|
+        conn.multi do |multi|
+          multi.lrem(processing_queue, -1, job)
+          multi.lpush(queue, job)
+        end
+      end
+    end
+  end
+end

data/lib/sidekiq_solid_fetch/version.rb

@@ -0,0 +1,3 @@
+module SidekiqSolidFetch
+  VERSION = "0.1.1"
+end

data/lib/sidekiq_solid_fetch.rb

@@ -0,0 +1,39 @@
+require_relative "sidekiq_solid_fetch/version"
+require_relative "sidekiq_solid_fetch/unit_of_work"
+require_relative "sidekiq_solid_fetch/fetcher"
+
+module SidekiqSolidFetch
+  class Error < StandardError; end
+
+  PROCESSING_QUEUE_PREFIX = "sidekiq_solid_fetch:processing"
+
+  def self.enable!(config)
+    config[:fetch_class] = SidekiqSolidFetch::Fetcher
+
+    config.on(:startup) do
+      Sidekiq.logger.info("SidekiqSolidFetch enabled")
+      requeue_not_finished_jobs(config)
+    end
+  end
+
+  def self.requeue_not_finished_jobs(config)
+    Sidekiq.logger.info("SidekiqSolidFetch: Re-queueing not finished jobs from previous runs")
+
+    count = 0
+    Sidekiq.redis do |conn|
+      config.queues.map { |q| "queue:#{q}" }.uniq.each do |queue|
+        processing_queue_name = ::SidekiqSolidFetch.processing_queue_name(queue)
+        while conn.lmove(processing_queue_name, queue, "RIGHT", "LEFT")
+          count += 1
+          Sidekiq.logger.info { "SidekiqSolidFetch: Moved job from #{processing_queue_name} back to #{queue}" }
+        end
+      end
+    end
+
+    Sidekiq.logger.info("SidekiqSolidFetch: Re-queued #{count} jobs from previous runs")
+  end
+
+  def self.processing_queue_name(queue)
+    "#{PROCESSING_QUEUE_PREFIX}:#{queue}"
+  end
+end

data/release-please-config.json

@@ -0,0 +1,16 @@
+{
+  "packages": {
+    ".": {
+      "package-name": "sidekiq_solid_fetch",
+      "include-component-in-tag": false,
+      "changelog-path": "CHANGELOG.md",
+      "release-type": "ruby",
+      "bump-minor-pre-major": true,
+      "bump-patch-for-minor-pre-major": true,
+      "draft": false,
+      "prerelease": false,
+      "version-file": "lib/sidekiq_solid_fetch/version.rb"
+    }
+  },
+  "$schema": "https://raw.githubusercontent.com/googleapis/release-please/main/schemas/config.json"
+}

metadata

@@ -0,0 +1,73 @@
+--- !ruby/object:Gem::Specification
+name: sidekiq_solid_fetch
+version: !ruby/object:Gem::Version
+  version: 0.1.1
+platform: ruby
+authors:
+- Alex Koval
+autorequire:
+bindir: exe
+cert_chain: []
+date: 2025-12-05 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: sidekiq
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '7.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '7.0'
+description: OSS implementation of Sidekiq Pro's `super_fetch`
+email:
+- al3xander.koval@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- ".release-please-manifest.json"
+- ".rspec"
+- ".standard.yml"
+- CHANGELOG.md
+- CLAUDE.md
+- Makefile
+- README.md
+- Rakefile
+- docker-compose.yml
+- lib/sidekiq_solid_fetch.rb
+- lib/sidekiq_solid_fetch/fetcher.rb
+- lib/sidekiq_solid_fetch/unit_of_work.rb
+- lib/sidekiq_solid_fetch/version.rb
+- release-please-config.json
+homepage: https://github.com/k0va1/sidekiq_solid_fetch
+licenses: []
+metadata:
+  homepage_uri: https://github.com/k0va1/sidekiq_solid_fetch
+  source_code_uri: https://github.com/k0va1/sidekiq_solid_fetch
+  changelog_uri: https://github.com/k0va1/sidekiq_solid_fetch/blob/master/CHANGELOG.md
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.1.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 3.5.22
+signing_key:
+specification_version: 4
+summary: OSS implementation of Sidekiq Pro's `super_fetch`
+test_files: []