sidekiq_solid_fetch 0.1.1 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.release-please-manifest.json +1 -1
- data/.standard.yml +1 -1
- data/CHANGELOG.md +23 -0
- data/CLAUDE.md +17 -7
- data/LICENSE.txt +21 -0
- data/Makefile +2 -2
- data/README.md +28 -6
- data/Rakefile +6 -3
- data/docker-compose.yml +3 -2
- data/lib/sidekiq_solid_fetch/fetcher.rb +33 -8
- data/lib/sidekiq_solid_fetch/throttled.rb +44 -0
- data/lib/sidekiq_solid_fetch/unit_of_work.rb +16 -1
- data/lib/sidekiq_solid_fetch/version.rb +1 -1
- data/lib/sidekiq_solid_fetch.rb +89 -14
- metadata +12 -10
- data/.rspec +0 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: f17f313f19a6598d87fe5fb8970b68e176389e279c648174c9ec9c92c787aeee
|
|
4
|
+
data.tar.gz: c34a5b15685d7590b3a6acc27132cd7190418cc51c1f96e075e0dc573fb56ab8
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: ddd3e40d1b85f1014fc65d9b4a5cd91b8f4959a0c4a11855e95277d38e35e2865ae308510ea4e23fb23ac84285e9a40c746de77a78c56d9f4892cc0e89e39725
|
|
7
|
+
data.tar.gz: 55a487070c122d8696bee38047f47accb1d0518b5b7ad959f30ebbb86b9751dd7344e59b02c426425137c4c9a8a8fa98b1d53755f982d23afdedf335f097a396
|
data/.standard.yml
CHANGED
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,28 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.2.0](https://github.com/k0va1/sidekiq_solid_fetch/compare/v0.1.2...v0.2.0) (2026-07-29)
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
### ⚠ BREAKING CHANGES
|
|
7
|
+
|
|
8
|
+
* require Ruby >= 3.3
|
|
9
|
+
|
|
10
|
+
### Features
|
|
11
|
+
|
|
12
|
+
* require Ruby >= 3.3 ([9768dc0](https://github.com/k0va1/sidekiq_solid_fetch/commit/9768dc0001ea653220bac678b7000c4f368a4fe2))
|
|
13
|
+
|
|
14
|
+
## [0.1.2](https://github.com/k0va1/sidekiq_solid_fetch/compare/v0.1.1...v0.1.2) (2026-07-29)
|
|
15
|
+
|
|
16
|
+
|
|
17
|
+
### Features
|
|
18
|
+
|
|
19
|
+
* add opt-in sidekiq-throttled integration ([4095131](https://github.com/k0va1/sidekiq_solid_fetch/commit/40951313fd7896f406bc8da9c38bee089bb4fa03))
|
|
20
|
+
|
|
21
|
+
|
|
22
|
+
### Bug Fixes
|
|
23
|
+
|
|
24
|
+
* rework orphan recovery to stop stealing live jobs and losing them ([cf54ebe](https://github.com/k0va1/sidekiq_solid_fetch/commit/cf54ebe7ede6beb73bee2911adbf8eda45c05f2b))
|
|
25
|
+
|
|
3
26
|
## [0.1.1](https://github.com/k0va1/sidekiq_solid_fetch/compare/v0.1.0...v0.1.1) (2025-12-04)
|
|
4
27
|
|
|
5
28
|
|
data/CLAUDE.md
CHANGED
|
@@ -9,9 +9,9 @@ SidekiqSolidFetch is an open-source Ruby gem that provides a reliable fetch stra
|
|
|
9
9
|
## Commands
|
|
10
10
|
|
|
11
11
|
- **Install dependencies**: `bundle install`
|
|
12
|
-
- **Run all tests**: `bundle exec rake
|
|
13
|
-
- **Run a single test file**: `bundle exec
|
|
14
|
-
- **Run a single test by
|
|
12
|
+
- **Run all tests**: `bundle exec rake test`
|
|
13
|
+
- **Run a single test file**: `bundle exec rake test TEST=test/path/to/file_test.rb`
|
|
14
|
+
- **Run a single test by name**: `bundle exec ruby -Ilib:test test/path/to/file_test.rb -n /pattern/`
|
|
15
15
|
- **Lint code**: `bundle exec rake standard`
|
|
16
16
|
- **Auto-fix lint issues**: `bundle exec standardrb --fix`
|
|
17
17
|
- **Run all checks (tests + lint)**: `bundle exec rake`
|
|
@@ -26,13 +26,23 @@ SidekiqSolidFetch is an open-source Ruby gem that provides a reliable fetch stra
|
|
|
26
26
|
|
|
27
27
|
### Fetcher Design
|
|
28
28
|
|
|
29
|
-
The Fetcher uses Redis LMOVE to atomically move jobs from the source queue to a per-worker processing queue (`processing:
|
|
29
|
+
The Fetcher uses Redis LMOVE to atomically move jobs from the source queue to a per-worker processing queue (`sidekiq_solid_fetch:processing:{capsule}:{identity}:queue:{name}`). This ensures:
|
|
30
30
|
- Jobs are never lost if a worker crashes mid-execution
|
|
31
31
|
- Unfinished jobs can be requeued via `bulk_requeue`
|
|
32
32
|
|
|
33
|
+
Each fetcher registers its processing queues in a Redis hash (`sidekiq_solid_fetch:registry`) mapping the processing queue key to its original queue and worker identity. Orphan recovery (`SidekiqSolidFetch.requeue_orphaned_jobs`) walks this registry — never key parsing — and requeues jobs whose worker's TTL-based heartbeat key is gone, skipping entries younger than `GRACE_PERIOD`. Recovery runs on `:startup` and every `RECOVERY_INTERVAL` seconds in a background thread (deduplicated across processes by a Redis NX lock).
|
|
34
|
+
|
|
33
35
|
Key classes:
|
|
34
|
-
- `Fetcher` - Implements Sidekiq's fetch interface with `retrieve_work` and `bulk_requeue`
|
|
35
|
-
- `UnitOfWork` -
|
|
36
|
+
- `Fetcher` - Implements Sidekiq's fetch interface with `retrieve_work` and `bulk_requeue`; sleeps `TIMEOUT` seconds when all queues are empty (Sidekiq's processor loop has no delay of its own)
|
|
37
|
+
- `UnitOfWork` - Wraps a job with `acknowledge`, `requeue`, and `requeue_throttled` methods
|
|
38
|
+
|
|
39
|
+
### sidekiq-throttled integration
|
|
40
|
+
|
|
41
|
+
`lib/sidekiq_solid_fetch/throttled.rb` (opt-in via `require` + `SidekiqSolidFetch::Throttled.setup!`, needs sidekiq-throttled >= 2.0). Never prepend `Sidekiq::Throttled::Patches::BasicFetch` directly: its requeue path LPUSHes a copy of the job while the original stays in the processing queue, leaking duplicates. The integration patches `Strategy#re_enqueue_throttled` to atomically move our `UnitOfWork` back instead.
|
|
42
|
+
|
|
43
|
+
### Testing notes
|
|
44
|
+
|
|
45
|
+
The suite (Minitest) flushes Redis before every test. It defaults to `redis://localhost:6381/15` (the docker-compose Redis) — never point `REDIS_URL` at a shared Redis.
|
|
36
46
|
|
|
37
47
|
### Queue Modes
|
|
38
48
|
|
|
@@ -41,5 +51,5 @@ Key classes:
|
|
|
41
51
|
|
|
42
52
|
## Requirements
|
|
43
53
|
|
|
44
|
-
- Ruby >= 3.
|
|
54
|
+
- Ruby >= 3.3.0
|
|
45
55
|
- Sidekiq >= 7.0
|
data/LICENSE.txt
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
The MIT License (MIT)
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2025 Alex Koval
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
|
13
|
+
all copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
|
21
|
+
THE SOFTWARE.
|
data/Makefile
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
REDIS_URL ?= redis://localhost:
|
|
1
|
+
REDIS_URL ?= redis://localhost:6381/15
|
|
2
2
|
|
|
3
3
|
.PHONY: test lint-fix console install
|
|
4
4
|
|
|
@@ -9,7 +9,7 @@ console:
|
|
|
9
9
|
bin/console
|
|
10
10
|
|
|
11
11
|
test:
|
|
12
|
-
REDIS_URL=$(REDIS_URL) bundle exec
|
|
12
|
+
REDIS_URL=$(REDIS_URL) bundle exec rake test $(addprefix TEST=,$(filter-out $@,$(MAKECMDGOALS)))
|
|
13
13
|
|
|
14
14
|
lint-fix:
|
|
15
15
|
bundle exec standardrb --fix
|
data/README.md
CHANGED
|
@@ -7,15 +7,18 @@ A reliable fetch strategy for Sidekiq that ensures jobs are never lost, even if
|
|
|
7
7
|
SidekiqSolidFetch uses Redis `LMOVE` command to atomically move jobs from the queue to a per-worker processing queue. This approach ensures:
|
|
8
8
|
|
|
9
9
|
- **No job loss**: Jobs are moved (not copied) to a processing queue before execution. If a worker crashes, the job remains in the processing queue.
|
|
10
|
-
- **Automatic recovery**:
|
|
10
|
+
- **Automatic recovery**: Jobs stranded by dead workers are requeued on startup and periodically afterwards.
|
|
11
11
|
- **Graceful shutdown**: When Sidekiq shuts down, in-progress jobs are moved back to their original queues.
|
|
12
12
|
|
|
13
13
|
### Flow
|
|
14
14
|
|
|
15
|
-
1. Worker
|
|
16
|
-
2. Worker
|
|
17
|
-
3. Worker
|
|
18
|
-
4.
|
|
15
|
+
1. Worker boots → it registers its processing queues in a Redis registry (`sidekiq_solid_fetch:registry`)
|
|
16
|
+
2. Worker fetches a job → job is atomically moved from `queue:default` to a per-worker processing queue (`sidekiq_solid_fetch:processing:{capsule}:{identity}:queue:default`)
|
|
17
|
+
3. Worker processes the job successfully → job is removed from the processing queue
|
|
18
|
+
4. Worker crashes → job stays in the processing queue
|
|
19
|
+
5. Recovery (on startup of any worker, then every 5 minutes) walks the registry, detects workers whose Sidekiq heartbeat has expired, and moves their jobs back to the original queues
|
|
20
|
+
|
|
21
|
+
Recovery never touches queues belonging to live workers: liveness is checked via Sidekiq's TTL-based heartbeat key, and freshly registered queues get a grace period so a worker that just booted can't be mistaken for a dead one. Delivery semantics are at-least-once — a job interrupted mid-execution will run again.
|
|
19
22
|
|
|
20
23
|
## Installation
|
|
21
24
|
|
|
@@ -48,15 +51,34 @@ end
|
|
|
48
51
|
|
|
49
52
|
That's it! SidekiqSolidFetch will now handle job fetching with crash recovery.
|
|
50
53
|
|
|
54
|
+
### Using with sidekiq-throttled
|
|
55
|
+
|
|
56
|
+
Do **not** prepend `Sidekiq::Throttled::Patches::BasicFetch` to the fetcher yourself: sidekiq-throttled's default `:enqueue` requeue pushes a *copy* of the throttled job back to the queue without removing the original from the processing queue, leaking one duplicate per throttle cycle that shutdown or orphan recovery would later re-inject as real jobs.
|
|
57
|
+
|
|
58
|
+
Use the built-in integration instead (requires sidekiq-throttled >= 2.0):
|
|
59
|
+
|
|
60
|
+
```ruby
|
|
61
|
+
require "sidekiq_solid_fetch/throttled"
|
|
62
|
+
|
|
63
|
+
Sidekiq.configure_server do |config|
|
|
64
|
+
SidekiqSolidFetch.enable!(config)
|
|
65
|
+
SidekiqSolidFetch::Throttled.setup!
|
|
66
|
+
end
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
`setup!` applies sidekiq-throttled's fetch patches to the fetcher and makes throttled requeues an atomic move back to the queue, so nothing is copied or lost. The `:schedule` requeue strategy works out of the box either way.
|
|
70
|
+
|
|
51
71
|
## Requirements
|
|
52
72
|
|
|
53
|
-
- Ruby >= 3.
|
|
73
|
+
- Ruby >= 3.3.0
|
|
54
74
|
- Sidekiq >= 7.0
|
|
55
75
|
|
|
56
76
|
## Development
|
|
57
77
|
|
|
58
78
|
After checking out the repo, run `bin/setup` to install dependencies.
|
|
59
79
|
|
|
80
|
+
The test suite flushes its Redis database before every test, so it runs against a dedicated Redis on port 6381 (db 15) by default — never point `REDIS_URL` at a Redis instance a real app uses.
|
|
81
|
+
|
|
60
82
|
```bash
|
|
61
83
|
# Run tests
|
|
62
84
|
docker compose up -d
|
data/Rakefile
CHANGED
|
@@ -1,8 +1,11 @@
|
|
|
1
1
|
require "bundler/gem_tasks"
|
|
2
|
-
require "
|
|
2
|
+
require "rake/testtask"
|
|
3
3
|
|
|
4
|
-
|
|
4
|
+
Rake::TestTask.new(:test) do |t|
|
|
5
|
+
t.libs << "test" << "lib"
|
|
6
|
+
t.pattern = "test/**/*_test.rb"
|
|
7
|
+
end
|
|
5
8
|
|
|
6
9
|
require "standard/rake"
|
|
7
10
|
|
|
8
|
-
task default: %i[
|
|
11
|
+
task default: %i[test standard]
|
data/docker-compose.yml
CHANGED
|
@@ -1,11 +1,12 @@
|
|
|
1
1
|
services:
|
|
2
2
|
redis:
|
|
3
3
|
image: redis:8.4-alpine
|
|
4
|
-
network_mode: host
|
|
5
4
|
volumes:
|
|
6
5
|
- redis:/data
|
|
7
6
|
ports:
|
|
8
|
-
|
|
7
|
+
# Dedicated host port so the test suite (which flushes its database)
|
|
8
|
+
# never collides with a dev Redis running on the default 6379.
|
|
9
|
+
- 6381:6379
|
|
9
10
|
healthcheck:
|
|
10
11
|
test: redis-cli ping
|
|
11
12
|
interval: 1s
|
|
@@ -6,38 +6,51 @@ module SidekiqSolidFetch
|
|
|
6
6
|
class Fetcher
|
|
7
7
|
include Sidekiq::Component
|
|
8
8
|
|
|
9
|
+
# How long to pause when every queue is empty. Mirrors BasicFetch's BRPOP
|
|
10
|
+
# timeout: Sidekiq's processor loop calls retrieve_work with no delay of
|
|
11
|
+
# its own, so blocking here is what keeps idle threads from hot-looping
|
|
12
|
+
# against Redis.
|
|
13
|
+
TIMEOUT = 2
|
|
14
|
+
|
|
9
15
|
def initialize(cap)
|
|
10
16
|
raise ArgumentError, "missing queue list" unless cap.queues
|
|
11
17
|
@config = cap
|
|
12
18
|
@strictly_ordered_queues = cap.mode == :strict
|
|
13
19
|
@queues = config.queues.map { |q| "queue:#{q}" }
|
|
14
20
|
@queues.uniq! if @strictly_ordered_queues
|
|
21
|
+
@identity = cap.identity
|
|
22
|
+
@processing_queues = @queues.uniq.to_h do |queue|
|
|
23
|
+
[queue, ::SidekiqSolidFetch.processing_queue_name(queue, @identity, cap.name)]
|
|
24
|
+
end
|
|
25
|
+
register_processing_queues
|
|
15
26
|
end
|
|
16
27
|
|
|
17
28
|
def retrieve_work
|
|
18
29
|
queues_cmd.each do |queue|
|
|
19
|
-
|
|
30
|
+
processing_queue = @processing_queues[queue]
|
|
20
31
|
work = redis do |conn|
|
|
21
|
-
conn.lmove(queue,
|
|
32
|
+
conn.lmove(queue, processing_queue, "RIGHT", "LEFT")
|
|
22
33
|
end
|
|
23
34
|
|
|
24
|
-
return ::SidekiqSolidFetch::UnitOfWork.new(queue, work, config,
|
|
35
|
+
return ::SidekiqSolidFetch::UnitOfWork.new(queue, work, config, processing_queue) if work
|
|
25
36
|
end
|
|
37
|
+
|
|
38
|
+
sleep(TIMEOUT)
|
|
26
39
|
nil
|
|
27
40
|
end
|
|
28
41
|
|
|
29
42
|
def bulk_requeue(*)
|
|
30
|
-
logger.info("SidekiqSolidFetch: Re-queueing terminated jobs")
|
|
43
|
+
logger.info("SidekiqSolidFetch: Re-queueing terminated jobs for #{@identity}")
|
|
31
44
|
|
|
32
45
|
count = 0
|
|
33
46
|
redis do |conn|
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
while conn.lmove(processing_queue_name, queue, "RIGHT", "LEFT")
|
|
47
|
+
@processing_queues.each do |queue, processing_queue|
|
|
48
|
+
while conn.lmove(processing_queue, queue, "LEFT", "RIGHT")
|
|
37
49
|
count += 1
|
|
38
|
-
logger.info { "SidekiqSolidFetch: Moving job from #{
|
|
50
|
+
logger.info { "SidekiqSolidFetch: Moving job from #{processing_queue} back to #{queue}" }
|
|
39
51
|
end
|
|
40
52
|
end
|
|
53
|
+
::SidekiqSolidFetch.unregister_processing_queues(conn, @processing_queues.values)
|
|
41
54
|
end
|
|
42
55
|
logger.info("SidekiqSolidFetch: Re-queued #{count} jobs")
|
|
43
56
|
rescue => ex
|
|
@@ -53,5 +66,17 @@ module SidekiqSolidFetch
|
|
|
53
66
|
permute
|
|
54
67
|
end
|
|
55
68
|
end
|
|
69
|
+
|
|
70
|
+
private
|
|
71
|
+
|
|
72
|
+
# Registering before the first fetch guarantees recovery in other
|
|
73
|
+
# processes can always see who owns a processing queue.
|
|
74
|
+
def register_processing_queues
|
|
75
|
+
redis do |conn|
|
|
76
|
+
@processing_queues.each do |queue, processing_queue|
|
|
77
|
+
::SidekiqSolidFetch.register_processing_queue(conn, processing_queue, queue, @identity)
|
|
78
|
+
end
|
|
79
|
+
end
|
|
80
|
+
end
|
|
56
81
|
end
|
|
57
82
|
end
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
require "sidekiq/throttled"
|
|
2
|
+
|
|
3
|
+
module SidekiqSolidFetch
|
|
4
|
+
# Opt-in integration with sidekiq-throttled (>= 2.0).
|
|
5
|
+
#
|
|
6
|
+
# sidekiq-throttled's stock requeue path treats any non-SuperFetch fetcher
|
|
7
|
+
# like BasicFetch: it LPUSHes a copy of the throttled job back to the queue
|
|
8
|
+
# without removing the original from the processing queue. With this
|
|
9
|
+
# fetcher that leaks one duplicate per throttle cycle, which bulk_requeue
|
|
10
|
+
# or orphan recovery would later re-inject as real jobs.
|
|
11
|
+
#
|
|
12
|
+
# `setup!` prepends sidekiq-throttled's fetch patches to the Fetcher and
|
|
13
|
+
# teaches Strategy#re_enqueue_throttled to atomically MOVE our UnitOfWork
|
|
14
|
+
# back to the target queue instead of copying it.
|
|
15
|
+
#
|
|
16
|
+
# require "sidekiq_solid_fetch/throttled"
|
|
17
|
+
#
|
|
18
|
+
# Sidekiq.configure_server do |config|
|
|
19
|
+
# SidekiqSolidFetch.enable!(config)
|
|
20
|
+
# SidekiqSolidFetch::Throttled.setup!
|
|
21
|
+
# end
|
|
22
|
+
module Throttled
|
|
23
|
+
MINIMUM_VERSION = "2.0.0"
|
|
24
|
+
|
|
25
|
+
module StrategyPatch
|
|
26
|
+
def re_enqueue_throttled(work, target_queue)
|
|
27
|
+
return super unless work.is_a?(::SidekiqSolidFetch::UnitOfWork)
|
|
28
|
+
|
|
29
|
+
target_queue = "queue:#{target_queue}" unless target_queue.start_with?("queue:")
|
|
30
|
+
work.requeue_throttled(target_queue)
|
|
31
|
+
end
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
def self.setup!
|
|
35
|
+
unless Gem::Version.new(Sidekiq::Throttled::VERSION) >= Gem::Version.new(MINIMUM_VERSION)
|
|
36
|
+
raise "SidekiqSolidFetch::Throttled requires sidekiq-throttled >= #{MINIMUM_VERSION} " \
|
|
37
|
+
"(found #{Sidekiq::Throttled::VERSION})"
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
SidekiqSolidFetch::Fetcher.prepend(Sidekiq::Throttled::Patches::BasicFetch)
|
|
41
|
+
Sidekiq::Throttled::Strategy.prepend(StrategyPatch)
|
|
42
|
+
end
|
|
43
|
+
end
|
|
44
|
+
end
|
|
@@ -21,7 +21,22 @@ module SidekiqSolidFetch
|
|
|
21
21
|
config.redis do |conn|
|
|
22
22
|
conn.multi do |multi|
|
|
23
23
|
multi.lrem(processing_queue, -1, job)
|
|
24
|
-
|
|
24
|
+
# RPUSH, like Sidekiq's own requeue: the queue is consumed from the
|
|
25
|
+
# right, so an interrupted job runs next rather than going behind
|
|
26
|
+
# newly enqueued work.
|
|
27
|
+
multi.rpush(queue, job)
|
|
28
|
+
end
|
|
29
|
+
end
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
# Atomically move the job from the processing queue to the BACK of the
|
|
33
|
+
# target queue (LPUSH side), so a throttled job doesn't churn straight
|
|
34
|
+
# through fetch again and no copy is left behind in the processing queue.
|
|
35
|
+
def requeue_throttled(target_queue)
|
|
36
|
+
config.redis do |conn|
|
|
37
|
+
conn.multi do |multi|
|
|
38
|
+
multi.lrem(processing_queue, -1, job)
|
|
39
|
+
multi.lpush(target_queue, job)
|
|
25
40
|
end
|
|
26
41
|
end
|
|
27
42
|
end
|
data/lib/sidekiq_solid_fetch.rb
CHANGED
|
@@ -1,39 +1,114 @@
|
|
|
1
|
+
require "json"
|
|
2
|
+
|
|
1
3
|
require_relative "sidekiq_solid_fetch/version"
|
|
2
4
|
require_relative "sidekiq_solid_fetch/unit_of_work"
|
|
3
5
|
require_relative "sidekiq_solid_fetch/fetcher"
|
|
4
6
|
|
|
5
7
|
module SidekiqSolidFetch
|
|
6
|
-
class Error < StandardError; end
|
|
7
|
-
|
|
8
8
|
PROCESSING_QUEUE_PREFIX = "sidekiq_solid_fetch:processing"
|
|
9
9
|
|
|
10
|
+
# Registry of every live processing queue. A Redis hash where each field is
|
|
11
|
+
# a processing queue key and each value is a JSON blob with the original
|
|
12
|
+
# queue, the worker identity and the registration timestamp. Recovery reads
|
|
13
|
+
# this registry instead of parsing key names, which would be ambiguous
|
|
14
|
+
# because both queue names and identities can contain colons.
|
|
15
|
+
REGISTRY_KEY = "sidekiq_solid_fetch:registry"
|
|
16
|
+
|
|
17
|
+
RECOVERY_LOCK_KEY = "sidekiq_solid_fetch:recovery_lock"
|
|
18
|
+
|
|
19
|
+
# Entries younger than this are never considered orphaned, so a freshly
|
|
20
|
+
# booted worker whose first heartbeat hasn't landed yet can't have its
|
|
21
|
+
# in-flight jobs stolen by another process running recovery.
|
|
22
|
+
GRACE_PERIOD = 60
|
|
23
|
+
|
|
24
|
+
# How often each process attempts orphan recovery after boot. Runs are
|
|
25
|
+
# deduplicated across processes with RECOVERY_LOCK_KEY.
|
|
26
|
+
RECOVERY_INTERVAL = 300
|
|
27
|
+
|
|
10
28
|
def self.enable!(config)
|
|
11
29
|
config[:fetch_class] = SidekiqSolidFetch::Fetcher
|
|
12
30
|
|
|
13
31
|
config.on(:startup) do
|
|
14
|
-
|
|
15
|
-
|
|
32
|
+
config.logger.info("SidekiqSolidFetch enabled")
|
|
33
|
+
requeue_orphaned_jobs(config)
|
|
34
|
+
@recovery_thread = start_recovery_thread(config)
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
config.on(:shutdown) do
|
|
38
|
+
@recovery_thread&.kill
|
|
39
|
+
@recovery_thread = nil
|
|
16
40
|
end
|
|
17
41
|
end
|
|
18
42
|
|
|
19
|
-
|
|
20
|
-
|
|
43
|
+
# Requeue jobs from processing queues whose worker is dead. A worker is
|
|
44
|
+
# considered dead when its heartbeat key (which Sidekiq maintains with a
|
|
45
|
+
# TTL) no longer exists. Membership in the `processes` set is not used
|
|
46
|
+
# because entries for crashed workers linger there until Sidekiq's
|
|
47
|
+
# rate-limited cleanup runs.
|
|
48
|
+
def self.requeue_orphaned_jobs(config)
|
|
49
|
+
logger = config.logger
|
|
50
|
+
logger.info("SidekiqSolidFetch: Checking for orphaned jobs from dead workers")
|
|
21
51
|
|
|
22
52
|
count = 0
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
53
|
+
now = Time.now.to_i
|
|
54
|
+
config.redis do |conn|
|
|
55
|
+
conn.hgetall(REGISTRY_KEY).each do |processing_queue, raw_entry|
|
|
56
|
+
entry = parse_registry_entry(raw_entry)
|
|
57
|
+
unless entry
|
|
58
|
+
conn.hdel(REGISTRY_KEY, processing_queue)
|
|
59
|
+
next
|
|
60
|
+
end
|
|
61
|
+
|
|
62
|
+
next if now - entry["registered_at"].to_i < GRACE_PERIOD
|
|
63
|
+
next if conn.exists(entry["identity"]) > 0
|
|
64
|
+
|
|
65
|
+
queue = entry["queue"]
|
|
66
|
+
while conn.lmove(processing_queue, queue, "LEFT", "RIGHT")
|
|
27
67
|
count += 1
|
|
28
|
-
|
|
68
|
+
logger.info { "SidekiqSolidFetch: Moved orphaned job from #{processing_queue} back to #{queue}" }
|
|
69
|
+
end
|
|
70
|
+
conn.hdel(REGISTRY_KEY, processing_queue)
|
|
71
|
+
end
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
logger.info("SidekiqSolidFetch: Re-queued #{count} orphaned jobs")
|
|
75
|
+
end
|
|
76
|
+
|
|
77
|
+
def self.processing_queue_name(queue, identity, capsule_name)
|
|
78
|
+
"#{PROCESSING_QUEUE_PREFIX}:#{capsule_name}:#{identity}:#{queue}"
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
def self.register_processing_queue(conn, processing_queue, queue, identity)
|
|
82
|
+
entry = {"queue" => queue, "identity" => identity, "registered_at" => Time.now.to_i}
|
|
83
|
+
conn.hset(REGISTRY_KEY, processing_queue, JSON.generate(entry))
|
|
84
|
+
end
|
|
85
|
+
|
|
86
|
+
def self.unregister_processing_queues(conn, processing_queues)
|
|
87
|
+
conn.hdel(REGISTRY_KEY, *processing_queues) unless processing_queues.empty?
|
|
88
|
+
end
|
|
89
|
+
|
|
90
|
+
def self.start_recovery_thread(config)
|
|
91
|
+
Thread.new do
|
|
92
|
+
loop do
|
|
93
|
+
sleep(RECOVERY_INTERVAL)
|
|
94
|
+
begin
|
|
95
|
+
requeue_orphaned_jobs(config) if acquire_recovery_lock(config)
|
|
96
|
+
rescue => ex
|
|
97
|
+
config.logger.warn("SidekiqSolidFetch: Orphan recovery failed: #{ex.message}")
|
|
29
98
|
end
|
|
30
99
|
end
|
|
31
100
|
end
|
|
101
|
+
end
|
|
32
102
|
|
|
33
|
-
|
|
103
|
+
def self.acquire_recovery_lock(config)
|
|
104
|
+
config.redis { |conn| conn.set(RECOVERY_LOCK_KEY, "1", "NX", "EX", RECOVERY_INTERVAL) }
|
|
34
105
|
end
|
|
35
106
|
|
|
36
|
-
def self.
|
|
37
|
-
|
|
107
|
+
def self.parse_registry_entry(raw_entry)
|
|
108
|
+
entry = JSON.parse(raw_entry)
|
|
109
|
+
return nil unless entry.is_a?(Hash) && entry["queue"] && entry["identity"]
|
|
110
|
+
entry
|
|
111
|
+
rescue JSON::ParserError
|
|
112
|
+
nil
|
|
38
113
|
end
|
|
39
114
|
end
|
metadata
CHANGED
|
@@ -1,14 +1,13 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: sidekiq_solid_fetch
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.2.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Alex Koval
|
|
8
|
-
autorequire:
|
|
9
8
|
bindir: exe
|
|
10
9
|
cert_chain: []
|
|
11
|
-
date:
|
|
10
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
12
11
|
dependencies:
|
|
13
12
|
- !ruby/object:Gem::Dependency
|
|
14
13
|
name: sidekiq
|
|
@@ -24,7 +23,9 @@ dependencies:
|
|
|
24
23
|
- - ">="
|
|
25
24
|
- !ruby/object:Gem::Version
|
|
26
25
|
version: '7.0'
|
|
27
|
-
description:
|
|
26
|
+
description: A reliable fetch strategy for Sidekiq that prevents job loss on worker
|
|
27
|
+
crashes by atomically moving jobs to per-worker processing queues and recovering
|
|
28
|
+
orphaned jobs from dead workers. An open-source alternative to Sidekiq Pro's `super_fetch`.
|
|
28
29
|
email:
|
|
29
30
|
- al3xander.koval@gmail.com
|
|
30
31
|
executables: []
|
|
@@ -32,26 +33,28 @@ extensions: []
|
|
|
32
33
|
extra_rdoc_files: []
|
|
33
34
|
files:
|
|
34
35
|
- ".release-please-manifest.json"
|
|
35
|
-
- ".rspec"
|
|
36
36
|
- ".standard.yml"
|
|
37
37
|
- CHANGELOG.md
|
|
38
38
|
- CLAUDE.md
|
|
39
|
+
- LICENSE.txt
|
|
39
40
|
- Makefile
|
|
40
41
|
- README.md
|
|
41
42
|
- Rakefile
|
|
42
43
|
- docker-compose.yml
|
|
43
44
|
- lib/sidekiq_solid_fetch.rb
|
|
44
45
|
- lib/sidekiq_solid_fetch/fetcher.rb
|
|
46
|
+
- lib/sidekiq_solid_fetch/throttled.rb
|
|
45
47
|
- lib/sidekiq_solid_fetch/unit_of_work.rb
|
|
46
48
|
- lib/sidekiq_solid_fetch/version.rb
|
|
47
49
|
- release-please-config.json
|
|
48
50
|
homepage: https://github.com/k0va1/sidekiq_solid_fetch
|
|
49
|
-
licenses:
|
|
51
|
+
licenses:
|
|
52
|
+
- MIT
|
|
50
53
|
metadata:
|
|
51
54
|
homepage_uri: https://github.com/k0va1/sidekiq_solid_fetch
|
|
52
55
|
source_code_uri: https://github.com/k0va1/sidekiq_solid_fetch
|
|
53
56
|
changelog_uri: https://github.com/k0va1/sidekiq_solid_fetch/blob/master/CHANGELOG.md
|
|
54
|
-
|
|
57
|
+
rubygems_mfa_required: 'true'
|
|
55
58
|
rdoc_options: []
|
|
56
59
|
require_paths:
|
|
57
60
|
- lib
|
|
@@ -59,15 +62,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
|
59
62
|
requirements:
|
|
60
63
|
- - ">="
|
|
61
64
|
- !ruby/object:Gem::Version
|
|
62
|
-
version: 3.
|
|
65
|
+
version: 3.3.0
|
|
63
66
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
64
67
|
requirements:
|
|
65
68
|
- - ">="
|
|
66
69
|
- !ruby/object:Gem::Version
|
|
67
70
|
version: '0'
|
|
68
71
|
requirements: []
|
|
69
|
-
rubygems_version: 3.
|
|
70
|
-
signing_key:
|
|
72
|
+
rubygems_version: 3.6.9
|
|
71
73
|
specification_version: 4
|
|
72
74
|
summary: OSS implementation of Sidekiq Pro's `super_fetch`
|
|
73
75
|
test_files: []
|
data/.rspec
DELETED