sidekiq_solid_fetch 0.1.1 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 3f9d49a1edc5ffd25c20302adac6f5fea21eb3aacab3c441d7a8722bb42faeca
4
- data.tar.gz: 8c137210e8a8febf7409379c7c207270f0c3964ae378898122cc7e1a6f7ae88d
3
+ metadata.gz: f17f313f19a6598d87fe5fb8970b68e176389e279c648174c9ec9c92c787aeee
4
+ data.tar.gz: c34a5b15685d7590b3a6acc27132cd7190418cc51c1f96e075e0dc573fb56ab8
5
5
  SHA512:
6
- metadata.gz: 2df35c34eed28f7f1c3ed78e853da85b80262a21702e4334699a0c7dadc5ba2f62059dba9dea116882ee71edbfa5c503de1f682b4371e021d580409cd8640322
7
- data.tar.gz: a3ffb8214437b7f4b579465b03268b3a4a4a393d481036f5ac4e90b510174ef7220f5d88a93f54038712f6e6731882889536396c7eed71b08e528163da0670e4
6
+ metadata.gz: ddd3e40d1b85f1014fc65d9b4a5cd91b8f4959a0c4a11855e95277d38e35e2865ae308510ea4e23fb23ac84285e9a40c746de77a78c56d9f4892cc0e89e39725
7
+ data.tar.gz: 55a487070c122d8696bee38047f47accb1d0518b5b7ad959f30ebbb86b9751dd7344e59b02c426425137c4c9a8a8fa98b1d53755f982d23afdedf335f097a396
@@ -1,3 +1,3 @@
1
1
  {
2
- ".": "0.1.1"
2
+ ".": "0.2.0"
3
3
  }
data/.standard.yml CHANGED
@@ -1,3 +1,3 @@
1
1
  # For available configuration options, see:
2
2
  # https://github.com/standardrb/standard
3
- ruby_version: 3.1
3
+ ruby_version: 3.3
data/CHANGELOG.md CHANGED
@@ -1,5 +1,28 @@
1
1
  # Changelog
2
2
 
3
+ ## [0.2.0](https://github.com/k0va1/sidekiq_solid_fetch/compare/v0.1.2...v0.2.0) (2026-07-29)
4
+
5
+
6
+ ### ⚠ BREAKING CHANGES
7
+
8
+ * require Ruby >= 3.3
9
+
10
+ ### Features
11
+
12
+ * require Ruby >= 3.3 ([9768dc0](https://github.com/k0va1/sidekiq_solid_fetch/commit/9768dc0001ea653220bac678b7000c4f368a4fe2))
13
+
14
+ ## [0.1.2](https://github.com/k0va1/sidekiq_solid_fetch/compare/v0.1.1...v0.1.2) (2026-07-29)
15
+
16
+
17
+ ### Features
18
+
19
+ * add opt-in sidekiq-throttled integration ([4095131](https://github.com/k0va1/sidekiq_solid_fetch/commit/40951313fd7896f406bc8da9c38bee089bb4fa03))
20
+
21
+
22
+ ### Bug Fixes
23
+
24
+ * rework orphan recovery to stop stealing live jobs and losing them ([cf54ebe](https://github.com/k0va1/sidekiq_solid_fetch/commit/cf54ebe7ede6beb73bee2911adbf8eda45c05f2b))
25
+
3
26
  ## [0.1.1](https://github.com/k0va1/sidekiq_solid_fetch/compare/v0.1.0...v0.1.1) (2025-12-04)
4
27
 
5
28
 
data/CLAUDE.md CHANGED
@@ -9,9 +9,9 @@ SidekiqSolidFetch is an open-source Ruby gem that provides a reliable fetch stra
9
9
  ## Commands
10
10
 
11
11
  - **Install dependencies**: `bundle install`
12
- - **Run all tests**: `bundle exec rake spec`
13
- - **Run a single test file**: `bundle exec rspec spec/path/to/file_spec.rb`
14
- - **Run a single test by line**: `bundle exec rspec spec/path/to/file_spec.rb:LINE`
12
+ - **Run all tests**: `bundle exec rake test`
13
+ - **Run a single test file**: `bundle exec rake test TEST=test/path/to/file_test.rb`
14
+ - **Run a single test by name**: `bundle exec ruby -Ilib:test test/path/to/file_test.rb -n /pattern/`
15
15
  - **Lint code**: `bundle exec rake standard`
16
16
  - **Auto-fix lint issues**: `bundle exec standardrb --fix`
17
17
  - **Run all checks (tests + lint)**: `bundle exec rake`
@@ -26,13 +26,23 @@ SidekiqSolidFetch is an open-source Ruby gem that provides a reliable fetch stra
26
26
 
27
27
  ### Fetcher Design
28
28
 
29
- The Fetcher uses Redis LMOVE to atomically move jobs from the source queue to a per-worker processing queue (`processing:queue:{name}:{identity}`). This ensures:
29
+ The Fetcher uses Redis LMOVE to atomically move jobs from the source queue to a per-worker processing queue (`sidekiq_solid_fetch:processing:{capsule}:{identity}:queue:{name}`). This ensures:
30
30
  - Jobs are never lost if a worker crashes mid-execution
31
31
  - Unfinished jobs can be requeued via `bulk_requeue`
32
32
 
33
+ Each fetcher registers its processing queues in a Redis hash (`sidekiq_solid_fetch:registry`) mapping the processing queue key to its original queue and worker identity. Orphan recovery (`SidekiqSolidFetch.requeue_orphaned_jobs`) walks this registry — never key parsing — and requeues jobs whose worker's TTL-based heartbeat key is gone, skipping entries younger than `GRACE_PERIOD`. Recovery runs on `:startup` and every `RECOVERY_INTERVAL` seconds in a background thread (deduplicated across processes by a Redis NX lock).
34
+
33
35
  Key classes:
34
- - `Fetcher` - Implements Sidekiq's fetch interface with `retrieve_work` and `bulk_requeue`
35
- - `UnitOfWork` - Struct wrapping a job with `acknowledge` and `requeue` methods
36
+ - `Fetcher` - Implements Sidekiq's fetch interface with `retrieve_work` and `bulk_requeue`; sleeps `TIMEOUT` seconds when all queues are empty (Sidekiq's processor loop has no delay of its own)
37
+ - `UnitOfWork` - Wraps a job with `acknowledge`, `requeue`, and `requeue_throttled` methods
38
+
39
+ ### sidekiq-throttled integration
40
+
41
+ `lib/sidekiq_solid_fetch/throttled.rb` (opt-in via `require` + `SidekiqSolidFetch::Throttled.setup!`, needs sidekiq-throttled >= 2.0). Never prepend `Sidekiq::Throttled::Patches::BasicFetch` directly: its requeue path LPUSHes a copy of the job while the original stays in the processing queue, leaking duplicates. The integration patches `Strategy#re_enqueue_throttled` to atomically move our `UnitOfWork` back instead.
42
+
43
+ ### Testing notes
44
+
45
+ The suite (Minitest) flushes Redis before every test. It defaults to `redis://localhost:6381/15` (the docker-compose Redis) — never point `REDIS_URL` at a shared Redis.
36
46
 
37
47
  ### Queue Modes
38
48
 
@@ -41,5 +51,5 @@ Key classes:
41
51
 
42
52
  ## Requirements
43
53
 
44
- - Ruby >= 3.1.0
54
+ - Ruby >= 3.3.0
45
55
  - Sidekiq >= 7.0
data/LICENSE.txt ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2025 Alex Koval
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
data/Makefile CHANGED
@@ -1,4 +1,4 @@
1
- REDIS_URL ?= redis://localhost:6379/0
1
+ REDIS_URL ?= redis://localhost:6381/15
2
2
 
3
3
  .PHONY: test lint-fix console install
4
4
 
@@ -9,7 +9,7 @@ console:
9
9
  bin/console
10
10
 
11
11
  test:
12
- REDIS_URL=$(REDIS_URL) bundle exec rspec $(filter-out $@,$(MAKECMDGOALS))
12
+ REDIS_URL=$(REDIS_URL) bundle exec rake test $(addprefix TEST=,$(filter-out $@,$(MAKECMDGOALS)))
13
13
 
14
14
  lint-fix:
15
15
  bundle exec standardrb --fix
data/README.md CHANGED
@@ -7,15 +7,18 @@ A reliable fetch strategy for Sidekiq that ensures jobs are never lost, even if
7
7
  SidekiqSolidFetch uses Redis `LMOVE` command to atomically move jobs from the queue to a per-worker processing queue. This approach ensures:
8
8
 
9
9
  - **No job loss**: Jobs are moved (not copied) to a processing queue before execution. If a worker crashes, the job remains in the processing queue.
10
- - **Automatic recovery**: On startup, any unfinished jobs from previous runs are automatically requeued.
10
+ - **Automatic recovery**: Jobs stranded by dead workers are requeued on startup and periodically afterwards.
11
11
  - **Graceful shutdown**: When Sidekiq shuts down, in-progress jobs are moved back to their original queues.
12
12
 
13
13
  ### Flow
14
14
 
15
- 1. Worker fetches a job job is atomically moved from `queue:default` to `sidekiq_solid_fetch:processing:queue:default`
16
- 2. Worker processes the job successfully → job is removed from the processing queue
17
- 3. Worker crashes → job stays in the processing queue
18
- 4. On next startup jobs in processing queues are moved back to their original queues
15
+ 1. Worker boots it registers its processing queues in a Redis registry (`sidekiq_solid_fetch:registry`)
16
+ 2. Worker fetches a job → job is atomically moved from `queue:default` to a per-worker processing queue (`sidekiq_solid_fetch:processing:{capsule}:{identity}:queue:default`)
17
+ 3. Worker processes the job successfully → job is removed from the processing queue
18
+ 4. Worker crashesjob stays in the processing queue
19
+ 5. Recovery (on startup of any worker, then every 5 minutes) walks the registry, detects workers whose Sidekiq heartbeat has expired, and moves their jobs back to the original queues
20
+
21
+ Recovery never touches queues belonging to live workers: liveness is checked via Sidekiq's TTL-based heartbeat key, and freshly registered queues get a grace period so a worker that just booted can't be mistaken for a dead one. Delivery semantics are at-least-once — a job interrupted mid-execution will run again.
19
22
 
20
23
  ## Installation
21
24
 
@@ -48,15 +51,34 @@ end
48
51
 
49
52
  That's it! SidekiqSolidFetch will now handle job fetching with crash recovery.
50
53
 
54
+ ### Using with sidekiq-throttled
55
+
56
+ Do **not** prepend `Sidekiq::Throttled::Patches::BasicFetch` to the fetcher yourself: sidekiq-throttled's default `:enqueue` requeue pushes a *copy* of the throttled job back to the queue without removing the original from the processing queue, leaking one duplicate per throttle cycle that shutdown or orphan recovery would later re-inject as real jobs.
57
+
58
+ Use the built-in integration instead (requires sidekiq-throttled >= 2.0):
59
+
60
+ ```ruby
61
+ require "sidekiq_solid_fetch/throttled"
62
+
63
+ Sidekiq.configure_server do |config|
64
+ SidekiqSolidFetch.enable!(config)
65
+ SidekiqSolidFetch::Throttled.setup!
66
+ end
67
+ ```
68
+
69
+ `setup!` applies sidekiq-throttled's fetch patches to the fetcher and makes throttled requeues an atomic move back to the queue, so nothing is copied or lost. The `:schedule` requeue strategy works out of the box either way.
70
+
51
71
  ## Requirements
52
72
 
53
- - Ruby >= 3.1.0
73
+ - Ruby >= 3.3.0
54
74
  - Sidekiq >= 7.0
55
75
 
56
76
  ## Development
57
77
 
58
78
  After checking out the repo, run `bin/setup` to install dependencies.
59
79
 
80
+ The test suite flushes its Redis database before every test, so it runs against a dedicated Redis on port 6381 (db 15) by default — never point `REDIS_URL` at a Redis instance a real app uses.
81
+
60
82
  ```bash
61
83
  # Run tests
62
84
  docker compose up -d
data/Rakefile CHANGED
@@ -1,8 +1,11 @@
1
1
  require "bundler/gem_tasks"
2
- require "rspec/core/rake_task"
2
+ require "rake/testtask"
3
3
 
4
- RSpec::Core::RakeTask.new(:spec)
4
+ Rake::TestTask.new(:test) do |t|
5
+ t.libs << "test" << "lib"
6
+ t.pattern = "test/**/*_test.rb"
7
+ end
5
8
 
6
9
  require "standard/rake"
7
10
 
8
- task default: %i[spec standard]
11
+ task default: %i[test standard]
data/docker-compose.yml CHANGED
@@ -1,11 +1,12 @@
1
1
  services:
2
2
  redis:
3
3
  image: redis:8.4-alpine
4
- network_mode: host
5
4
  volumes:
6
5
  - redis:/data
7
6
  ports:
8
- - 6379:6379
7
+ # Dedicated host port so the test suite (which flushes its database)
8
+ # never collides with a dev Redis running on the default 6379.
9
+ - 6381:6379
9
10
  healthcheck:
10
11
  test: redis-cli ping
11
12
  interval: 1s
@@ -6,38 +6,51 @@ module SidekiqSolidFetch
6
6
  class Fetcher
7
7
  include Sidekiq::Component
8
8
 
9
+ # How long to pause when every queue is empty. Mirrors BasicFetch's BRPOP
10
+ # timeout: Sidekiq's processor loop calls retrieve_work with no delay of
11
+ # its own, so blocking here is what keeps idle threads from hot-looping
12
+ # against Redis.
13
+ TIMEOUT = 2
14
+
9
15
  def initialize(cap)
10
16
  raise ArgumentError, "missing queue list" unless cap.queues
11
17
  @config = cap
12
18
  @strictly_ordered_queues = cap.mode == :strict
13
19
  @queues = config.queues.map { |q| "queue:#{q}" }
14
20
  @queues.uniq! if @strictly_ordered_queues
21
+ @identity = cap.identity
22
+ @processing_queues = @queues.uniq.to_h do |queue|
23
+ [queue, ::SidekiqSolidFetch.processing_queue_name(queue, @identity, cap.name)]
24
+ end
25
+ register_processing_queues
15
26
  end
16
27
 
17
28
  def retrieve_work
18
29
  queues_cmd.each do |queue|
19
- processing_queue_name = ::SidekiqSolidFetch.processing_queue_name(queue)
30
+ processing_queue = @processing_queues[queue]
20
31
  work = redis do |conn|
21
- conn.lmove(queue, processing_queue_name, "RIGHT", "LEFT")
32
+ conn.lmove(queue, processing_queue, "RIGHT", "LEFT")
22
33
  end
23
34
 
24
- return ::SidekiqSolidFetch::UnitOfWork.new(queue, work, config, processing_queue_name) if work
35
+ return ::SidekiqSolidFetch::UnitOfWork.new(queue, work, config, processing_queue) if work
25
36
  end
37
+
38
+ sleep(TIMEOUT)
26
39
  nil
27
40
  end
28
41
 
29
42
  def bulk_requeue(*)
30
- logger.info("SidekiqSolidFetch: Re-queueing terminated jobs")
43
+ logger.info("SidekiqSolidFetch: Re-queueing terminated jobs for #{@identity}")
31
44
 
32
45
  count = 0
33
46
  redis do |conn|
34
- queues_cmd.each do |queue|
35
- processing_queue_name = ::SidekiqSolidFetch.processing_queue_name(queue)
36
- while conn.lmove(processing_queue_name, queue, "RIGHT", "LEFT")
47
+ @processing_queues.each do |queue, processing_queue|
48
+ while conn.lmove(processing_queue, queue, "LEFT", "RIGHT")
37
49
  count += 1
38
- logger.info { "SidekiqSolidFetch: Moving job from #{processing_queue_name} back to #{queue}" }
50
+ logger.info { "SidekiqSolidFetch: Moving job from #{processing_queue} back to #{queue}" }
39
51
  end
40
52
  end
53
+ ::SidekiqSolidFetch.unregister_processing_queues(conn, @processing_queues.values)
41
54
  end
42
55
  logger.info("SidekiqSolidFetch: Re-queued #{count} jobs")
43
56
  rescue => ex
@@ -53,5 +66,17 @@ module SidekiqSolidFetch
53
66
  permute
54
67
  end
55
68
  end
69
+
70
+ private
71
+
72
+ # Registering before the first fetch guarantees recovery in other
73
+ # processes can always see who owns a processing queue.
74
+ def register_processing_queues
75
+ redis do |conn|
76
+ @processing_queues.each do |queue, processing_queue|
77
+ ::SidekiqSolidFetch.register_processing_queue(conn, processing_queue, queue, @identity)
78
+ end
79
+ end
80
+ end
56
81
  end
57
82
  end
@@ -0,0 +1,44 @@
1
+ require "sidekiq/throttled"
2
+
3
+ module SidekiqSolidFetch
4
+ # Opt-in integration with sidekiq-throttled (>= 2.0).
5
+ #
6
+ # sidekiq-throttled's stock requeue path treats any non-SuperFetch fetcher
7
+ # like BasicFetch: it LPUSHes a copy of the throttled job back to the queue
8
+ # without removing the original from the processing queue. With this
9
+ # fetcher that leaks one duplicate per throttle cycle, which bulk_requeue
10
+ # or orphan recovery would later re-inject as real jobs.
11
+ #
12
+ # `setup!` prepends sidekiq-throttled's fetch patches to the Fetcher and
13
+ # teaches Strategy#re_enqueue_throttled to atomically MOVE our UnitOfWork
14
+ # back to the target queue instead of copying it.
15
+ #
16
+ # require "sidekiq_solid_fetch/throttled"
17
+ #
18
+ # Sidekiq.configure_server do |config|
19
+ # SidekiqSolidFetch.enable!(config)
20
+ # SidekiqSolidFetch::Throttled.setup!
21
+ # end
22
+ module Throttled
23
+ MINIMUM_VERSION = "2.0.0"
24
+
25
+ module StrategyPatch
26
+ def re_enqueue_throttled(work, target_queue)
27
+ return super unless work.is_a?(::SidekiqSolidFetch::UnitOfWork)
28
+
29
+ target_queue = "queue:#{target_queue}" unless target_queue.start_with?("queue:")
30
+ work.requeue_throttled(target_queue)
31
+ end
32
+ end
33
+
34
+ def self.setup!
35
+ unless Gem::Version.new(Sidekiq::Throttled::VERSION) >= Gem::Version.new(MINIMUM_VERSION)
36
+ raise "SidekiqSolidFetch::Throttled requires sidekiq-throttled >= #{MINIMUM_VERSION} " \
37
+ "(found #{Sidekiq::Throttled::VERSION})"
38
+ end
39
+
40
+ SidekiqSolidFetch::Fetcher.prepend(Sidekiq::Throttled::Patches::BasicFetch)
41
+ Sidekiq::Throttled::Strategy.prepend(StrategyPatch)
42
+ end
43
+ end
44
+ end
@@ -21,7 +21,22 @@ module SidekiqSolidFetch
21
21
  config.redis do |conn|
22
22
  conn.multi do |multi|
23
23
  multi.lrem(processing_queue, -1, job)
24
- multi.lpush(queue, job)
24
+ # RPUSH, like Sidekiq's own requeue: the queue is consumed from the
25
+ # right, so an interrupted job runs next rather than going behind
26
+ # newly enqueued work.
27
+ multi.rpush(queue, job)
28
+ end
29
+ end
30
+ end
31
+
32
+ # Atomically move the job from the processing queue to the BACK of the
33
+ # target queue (LPUSH side), so a throttled job doesn't churn straight
34
+ # through fetch again and no copy is left behind in the processing queue.
35
+ def requeue_throttled(target_queue)
36
+ config.redis do |conn|
37
+ conn.multi do |multi|
38
+ multi.lrem(processing_queue, -1, job)
39
+ multi.lpush(target_queue, job)
25
40
  end
26
41
  end
27
42
  end
@@ -1,3 +1,3 @@
1
1
  module SidekiqSolidFetch
2
- VERSION = "0.1.1"
2
+ VERSION = "0.2.0"
3
3
  end
@@ -1,39 +1,114 @@
1
+ require "json"
2
+
1
3
  require_relative "sidekiq_solid_fetch/version"
2
4
  require_relative "sidekiq_solid_fetch/unit_of_work"
3
5
  require_relative "sidekiq_solid_fetch/fetcher"
4
6
 
5
7
  module SidekiqSolidFetch
6
- class Error < StandardError; end
7
-
8
8
  PROCESSING_QUEUE_PREFIX = "sidekiq_solid_fetch:processing"
9
9
 
10
+ # Registry of every live processing queue. A Redis hash where each field is
11
+ # a processing queue key and each value is a JSON blob with the original
12
+ # queue, the worker identity and the registration timestamp. Recovery reads
13
+ # this registry instead of parsing key names, which would be ambiguous
14
+ # because both queue names and identities can contain colons.
15
+ REGISTRY_KEY = "sidekiq_solid_fetch:registry"
16
+
17
+ RECOVERY_LOCK_KEY = "sidekiq_solid_fetch:recovery_lock"
18
+
19
+ # Entries younger than this are never considered orphaned, so a freshly
20
+ # booted worker whose first heartbeat hasn't landed yet can't have its
21
+ # in-flight jobs stolen by another process running recovery.
22
+ GRACE_PERIOD = 60
23
+
24
+ # How often each process attempts orphan recovery after boot. Runs are
25
+ # deduplicated across processes with RECOVERY_LOCK_KEY.
26
+ RECOVERY_INTERVAL = 300
27
+
10
28
  def self.enable!(config)
11
29
  config[:fetch_class] = SidekiqSolidFetch::Fetcher
12
30
 
13
31
  config.on(:startup) do
14
- Sidekiq.logger.info("SidekiqSolidFetch enabled")
15
- requeue_not_finished_jobs(config)
32
+ config.logger.info("SidekiqSolidFetch enabled")
33
+ requeue_orphaned_jobs(config)
34
+ @recovery_thread = start_recovery_thread(config)
35
+ end
36
+
37
+ config.on(:shutdown) do
38
+ @recovery_thread&.kill
39
+ @recovery_thread = nil
16
40
  end
17
41
  end
18
42
 
19
- def self.requeue_not_finished_jobs(config)
20
- Sidekiq.logger.info("SidekiqSolidFetch: Re-queueing not finished jobs from previous runs")
43
+ # Requeue jobs from processing queues whose worker is dead. A worker is
44
+ # considered dead when its heartbeat key (which Sidekiq maintains with a
45
+ # TTL) no longer exists. Membership in the `processes` set is not used
46
+ # because entries for crashed workers linger there until Sidekiq's
47
+ # rate-limited cleanup runs.
48
+ def self.requeue_orphaned_jobs(config)
49
+ logger = config.logger
50
+ logger.info("SidekiqSolidFetch: Checking for orphaned jobs from dead workers")
21
51
 
22
52
  count = 0
23
- Sidekiq.redis do |conn|
24
- config.queues.map { |q| "queue:#{q}" }.uniq.each do |queue|
25
- processing_queue_name = ::SidekiqSolidFetch.processing_queue_name(queue)
26
- while conn.lmove(processing_queue_name, queue, "RIGHT", "LEFT")
53
+ now = Time.now.to_i
54
+ config.redis do |conn|
55
+ conn.hgetall(REGISTRY_KEY).each do |processing_queue, raw_entry|
56
+ entry = parse_registry_entry(raw_entry)
57
+ unless entry
58
+ conn.hdel(REGISTRY_KEY, processing_queue)
59
+ next
60
+ end
61
+
62
+ next if now - entry["registered_at"].to_i < GRACE_PERIOD
63
+ next if conn.exists(entry["identity"]) > 0
64
+
65
+ queue = entry["queue"]
66
+ while conn.lmove(processing_queue, queue, "LEFT", "RIGHT")
27
67
  count += 1
28
- Sidekiq.logger.info { "SidekiqSolidFetch: Moved job from #{processing_queue_name} back to #{queue}" }
68
+ logger.info { "SidekiqSolidFetch: Moved orphaned job from #{processing_queue} back to #{queue}" }
69
+ end
70
+ conn.hdel(REGISTRY_KEY, processing_queue)
71
+ end
72
+ end
73
+
74
+ logger.info("SidekiqSolidFetch: Re-queued #{count} orphaned jobs")
75
+ end
76
+
77
+ def self.processing_queue_name(queue, identity, capsule_name)
78
+ "#{PROCESSING_QUEUE_PREFIX}:#{capsule_name}:#{identity}:#{queue}"
79
+ end
80
+
81
+ def self.register_processing_queue(conn, processing_queue, queue, identity)
82
+ entry = {"queue" => queue, "identity" => identity, "registered_at" => Time.now.to_i}
83
+ conn.hset(REGISTRY_KEY, processing_queue, JSON.generate(entry))
84
+ end
85
+
86
+ def self.unregister_processing_queues(conn, processing_queues)
87
+ conn.hdel(REGISTRY_KEY, *processing_queues) unless processing_queues.empty?
88
+ end
89
+
90
+ def self.start_recovery_thread(config)
91
+ Thread.new do
92
+ loop do
93
+ sleep(RECOVERY_INTERVAL)
94
+ begin
95
+ requeue_orphaned_jobs(config) if acquire_recovery_lock(config)
96
+ rescue => ex
97
+ config.logger.warn("SidekiqSolidFetch: Orphan recovery failed: #{ex.message}")
29
98
  end
30
99
  end
31
100
  end
101
+ end
32
102
 
33
- Sidekiq.logger.info("SidekiqSolidFetch: Re-queued #{count} jobs from previous runs")
103
+ def self.acquire_recovery_lock(config)
104
+ config.redis { |conn| conn.set(RECOVERY_LOCK_KEY, "1", "NX", "EX", RECOVERY_INTERVAL) }
34
105
  end
35
106
 
36
- def self.processing_queue_name(queue)
37
- "#{PROCESSING_QUEUE_PREFIX}:#{queue}"
107
+ def self.parse_registry_entry(raw_entry)
108
+ entry = JSON.parse(raw_entry)
109
+ return nil unless entry.is_a?(Hash) && entry["queue"] && entry["identity"]
110
+ entry
111
+ rescue JSON::ParserError
112
+ nil
38
113
  end
39
114
  end
metadata CHANGED
@@ -1,14 +1,13 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: sidekiq_solid_fetch
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.1
4
+ version: 0.2.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Alex Koval
8
- autorequire:
9
8
  bindir: exe
10
9
  cert_chain: []
11
- date: 2025-12-05 00:00:00.000000000 Z
10
+ date: 1980-01-02 00:00:00.000000000 Z
12
11
  dependencies:
13
12
  - !ruby/object:Gem::Dependency
14
13
  name: sidekiq
@@ -24,7 +23,9 @@ dependencies:
24
23
  - - ">="
25
24
  - !ruby/object:Gem::Version
26
25
  version: '7.0'
27
- description: OSS implementation of Sidekiq Pro's `super_fetch`
26
+ description: A reliable fetch strategy for Sidekiq that prevents job loss on worker
27
+ crashes by atomically moving jobs to per-worker processing queues and recovering
28
+ orphaned jobs from dead workers. An open-source alternative to Sidekiq Pro's `super_fetch`.
28
29
  email:
29
30
  - al3xander.koval@gmail.com
30
31
  executables: []
@@ -32,26 +33,28 @@ extensions: []
32
33
  extra_rdoc_files: []
33
34
  files:
34
35
  - ".release-please-manifest.json"
35
- - ".rspec"
36
36
  - ".standard.yml"
37
37
  - CHANGELOG.md
38
38
  - CLAUDE.md
39
+ - LICENSE.txt
39
40
  - Makefile
40
41
  - README.md
41
42
  - Rakefile
42
43
  - docker-compose.yml
43
44
  - lib/sidekiq_solid_fetch.rb
44
45
  - lib/sidekiq_solid_fetch/fetcher.rb
46
+ - lib/sidekiq_solid_fetch/throttled.rb
45
47
  - lib/sidekiq_solid_fetch/unit_of_work.rb
46
48
  - lib/sidekiq_solid_fetch/version.rb
47
49
  - release-please-config.json
48
50
  homepage: https://github.com/k0va1/sidekiq_solid_fetch
49
- licenses: []
51
+ licenses:
52
+ - MIT
50
53
  metadata:
51
54
  homepage_uri: https://github.com/k0va1/sidekiq_solid_fetch
52
55
  source_code_uri: https://github.com/k0va1/sidekiq_solid_fetch
53
56
  changelog_uri: https://github.com/k0va1/sidekiq_solid_fetch/blob/master/CHANGELOG.md
54
- post_install_message:
57
+ rubygems_mfa_required: 'true'
55
58
  rdoc_options: []
56
59
  require_paths:
57
60
  - lib
@@ -59,15 +62,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
59
62
  requirements:
60
63
  - - ">="
61
64
  - !ruby/object:Gem::Version
62
- version: 3.1.0
65
+ version: 3.3.0
63
66
  required_rubygems_version: !ruby/object:Gem::Requirement
64
67
  requirements:
65
68
  - - ">="
66
69
  - !ruby/object:Gem::Version
67
70
  version: '0'
68
71
  requirements: []
69
- rubygems_version: 3.5.22
70
- signing_key:
72
+ rubygems_version: 3.6.9
71
73
  specification_version: 4
72
74
  summary: OSS implementation of Sidekiq Pro's `super_fetch`
73
75
  test_files: []
data/.rspec DELETED
@@ -1,2 +0,0 @@
1
- --color
2
- --require spec_helper