cdc-sidekiq 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +7 -0
- data/README.md +31 -8
- data/benchmark/README.md +12 -6
- data/lib/cdc/sidekiq/configuration.rb +1 -1
- data/lib/cdc/sidekiq/errors.rb +1 -1
- data/lib/cdc/sidekiq/processor_job.rb +13 -6
- data/lib/cdc/sidekiq/runtime.rb +2 -2
- data/lib/cdc/sidekiq/version.rb +1 -1
- data/sig/cdc/sidekiq/configuration.rbs +54 -0
- data/sig/cdc/sidekiq/errors.rbs +27 -0
- data/sig/cdc/sidekiq/processor_job.rbs +155 -0
- data/sig/cdc/sidekiq/runtime.rbs +85 -0
- data/sig/cdc/sidekiq/version.rbs +6 -0
- data/sig/cdc/sidekiq.rbs +22 -0
- data/sig/cdc_sidekiq.rbs +0 -0
- metadata +16 -9
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 70f11806851d7d2e2476a302db30f6ef923c1b503d7571b09618d7606e6e1958
|
|
4
|
+
data.tar.gz: 36d27831e42392428b70d35ae8c4bd9847ae3212db5051dc47b6745fec3e641c
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 4b988f64c6395cd2ea1104cdb2e7b5cc0160dac42de5f3fe1b6125adfedf381144206df6be2bf1bdc3a70dee2a40db85f4f7f2fa4125def3b123e6ec7854f111
|
|
7
|
+
data.tar.gz: 2f1c8cc7ed74c14edae32ce0b96f1cc64d0d6739af7908c5d4d4957a402122d8348b5ba00f09fcb51b5bd81886d6ce76fbe3857d232fb88ea51ecfa0d264f4b6
|
data/CHANGELOG.md
CHANGED
|
@@ -1,8 +1,15 @@
|
|
|
1
1
|
## [Unreleased]
|
|
2
2
|
|
|
3
|
+
## [0.1.1] - 2026-06-12
|
|
4
|
+
|
|
3
5
|
- Improved unit test coverage for runtime selection, configuration copying, class-level job declarations, failure policies, and batch payload behavior.
|
|
6
|
+
- Added RBS signatures and validation coverage for the public cdc-sidekiq API.
|
|
7
|
+
- Fixed RBS validation issues around stdlib `Etc`, `default_runtime`, and dynamic processor-job class helpers.
|
|
8
|
+
- Documented all configuration defaults, including `raise_on_failure` and `batch_payloads`.
|
|
4
9
|
- Expanded benchmark documentation with the 500,000-item Ruby 4.0.5 snapshot, interpretation, and runtime tuning guidance.
|
|
10
|
+
- Clarified benchmark prerequisites, runtime knobs, and snapshot variance notes.
|
|
5
11
|
- Documented runtime-selection guidance and the shared-state correctness boundary between cdc-sidekiq and consumer processors/sinks.
|
|
12
|
+
- Filled missing YARD tag descriptions for void-returning APIs.
|
|
6
13
|
- Updated gem metadata documentation URI and description.
|
|
7
14
|
|
|
8
15
|
|
data/README.md
CHANGED
|
@@ -1,5 +1,10 @@
|
|
|
1
1
|
# cdc-sidekiq
|
|
2
2
|
|
|
3
|
+
[](https://badge.fury.io/rb/cdc-sidekiq)
|
|
4
|
+
[](https://github.com/kanutocd/cdc-sidekiq/actions)
|
|
5
|
+
[](https://www.ruby-lang.org/en/)
|
|
6
|
+
[](https://opensource.org/licenses/MIT)
|
|
7
|
+
|
|
3
8
|
`cdc-sidekiq` integrates Sidekiq with CDC execution primitives.
|
|
4
9
|
|
|
5
10
|
Sidekiq remains the durable job system. It owns scheduling, retries, queues, Redis persistence, and operational behavior.
|
|
@@ -11,6 +16,9 @@ Sidekiq Job
|
|
|
11
16
|
|
|
|
12
17
|
v
|
|
13
18
|
CDC::Sidekiq::ProcessorJob
|
|
19
|
+
|
|
|
20
|
+
+--> :direct
|
|
21
|
+
| sequential processor calls
|
|
14
22
|
|
|
|
15
23
|
+--> cdc-parallel
|
|
16
24
|
| Ractor fan-out / fan-in
|
|
@@ -59,7 +67,7 @@ CDC primitives execute the work.
|
|
|
59
67
|
|
|
60
68
|
The open-source edition focuses on execution primitives.
|
|
61
69
|
|
|
62
|
-
Current
|
|
70
|
+
Current OSS capabilities:
|
|
63
71
|
|
|
64
72
|
- `:direct` runtime
|
|
65
73
|
- `:concurrent` runtime
|
|
@@ -85,7 +93,7 @@ cdc-sidekiq
|
|
|
85
93
|
cdc-orchestrator-pro
|
|
86
94
|
Hybrid runtime
|
|
87
95
|
Nested runtime / worker-local resource pools
|
|
88
|
-
|
|
96
|
+
Orchestration, backpressure, telemetry, tuning
|
|
89
97
|
```
|
|
90
98
|
|
|
91
99
|
This keeps the OSS gem small and useful while leaving operational orchestration to the commercial layer.
|
|
@@ -136,9 +144,9 @@ Runtime support depends on the selected CDC execution substrate:
|
|
|
136
144
|
|
|
137
145
|
| Runtime | Ruby | Required gems |
|
|
138
146
|
| --- | --- | --- |
|
|
139
|
-
| `:direct` | 3.4+ | `cdc-core` |
|
|
140
|
-
| `:concurrent` | 3.4+ | `cdc-core`, `cdc-concurrent` |
|
|
141
|
-
| `:parallel` | 4.0+ | `cdc-core`, `cdc-parallel` |
|
|
147
|
+
| `:direct` | 3.4+ | `sidekiq`, `cdc-core` |
|
|
148
|
+
| `:concurrent` | 3.4+ | `sidekiq`, `cdc-core`, `cdc-concurrent` |
|
|
149
|
+
| `:parallel` | 4.0+ | `sidekiq`, `cdc-core`, `cdc-parallel` |
|
|
142
150
|
|
|
143
151
|
`cdc-parallel` remains optional because it requires Ruby 4+. Ruby 3.4 users can still use `:direct` and `:concurrent`.
|
|
144
152
|
|
|
@@ -151,10 +159,12 @@ gem "cdc-sidekiq"
|
|
|
151
159
|
Runtime gems are installed by the application according to the execution model it uses:
|
|
152
160
|
|
|
153
161
|
```ruby
|
|
154
|
-
gem "cdc-parallel" # for Ractor-backed execution
|
|
155
162
|
gem "cdc-concurrent" # for Async-backed execution
|
|
163
|
+
gem "cdc-parallel" # for Ractor-backed execution
|
|
156
164
|
```
|
|
157
165
|
|
|
166
|
+
Applications that only use `:direct` do not need either optional runtime gem.
|
|
167
|
+
|
|
158
168
|
## Configuration
|
|
159
169
|
|
|
160
170
|
```ruby
|
|
@@ -165,10 +175,24 @@ Sidekiq.configure_server do |_config|
|
|
|
165
175
|
cdc.concurrency = 100
|
|
166
176
|
cdc.timeout = nil
|
|
167
177
|
cdc.preserve_order = true
|
|
178
|
+
cdc.raise_on_failure = true
|
|
179
|
+
cdc.batch_payloads = true
|
|
168
180
|
end
|
|
169
181
|
end
|
|
170
182
|
```
|
|
171
183
|
|
|
184
|
+
Configuration defaults:
|
|
185
|
+
|
|
186
|
+
| Option | Default | Meaning |
|
|
187
|
+
| --- | --- | --- |
|
|
188
|
+
| `default_runtime` | `:concurrent` | Runtime used when a job does not declare `cdc_runtime`. |
|
|
189
|
+
| `parallel_size` | `Etc.nprocessors - 1`, minimum `1` | Ractor worker count for `:parallel` jobs. |
|
|
190
|
+
| `concurrency` | `100` | Async task limit for `:concurrent` jobs. |
|
|
191
|
+
| `timeout` | `nil` | Optional per-item timeout passed to CDC processor pools. |
|
|
192
|
+
| `preserve_order` | `true` | Preserve input order for `:concurrent` result arrays. |
|
|
193
|
+
| `raise_on_failure` | `true` | Raise failed `ProcessorResult` objects so Sidekiq can retry. |
|
|
194
|
+
| `batch_payloads` | `true` | Process array payloads with `process_many` instead of treating the array as one item. |
|
|
195
|
+
|
|
172
196
|
Sidekiq concurrency and CDC runtime concurrency are intentionally separate.
|
|
173
197
|
|
|
174
198
|
```text
|
|
@@ -275,7 +299,6 @@ class BestEffortJob
|
|
|
275
299
|
end
|
|
276
300
|
```
|
|
277
301
|
|
|
278
|
-
|
|
279
302
|
## Benchmarking
|
|
280
303
|
|
|
281
304
|
`cdc-sidekiq` includes `bin/cdc-sidekiq-load`, a benchmark aligned with Sidekiq's `bin/sidekiq-load` style.
|
|
@@ -292,7 +315,7 @@ COUNT=500000 RUNTIME=parallel CDC_PARALLEL_SIZE=7 \
|
|
|
292
315
|
bundle exec bin/cdc-sidekiq-load
|
|
293
316
|
```
|
|
294
317
|
|
|
295
|
-
See [`benchmark/README.md`](benchmark
|
|
318
|
+
See [`benchmark/README.md`](https://github.com/kanutocd/cdc-sidekiq/tree/main/benchmark) for interpretation notes and all benchmark knobs.
|
|
296
319
|
|
|
297
320
|
## Current scope
|
|
298
321
|
|
data/benchmark/README.md
CHANGED
|
@@ -22,6 +22,8 @@ process_many(items)
|
|
|
22
22
|
|
|
23
23
|
This benchmark does **not** replace Sidekiq's Redis-backed load benchmark. It measures the inner execution primitive that a CDC-aware Sidekiq job can use after Sidekiq has already started the job.
|
|
24
24
|
|
|
25
|
+
Run the benchmark from the gem checkout with `bundle exec`. It does not require Redis or a running Sidekiq process, but the selected optional runtime gem must be installed when using `RUNTIME=concurrent` or `RUNTIME=parallel`.
|
|
26
|
+
|
|
25
27
|
## Examples
|
|
26
28
|
|
|
27
29
|
```bash
|
|
@@ -47,10 +49,10 @@ COUNT=500000 RUNTIME=parallel CDC_PARALLEL_SIZE=7 \
|
|
|
47
49
|
| `BATCH_SIZE` | Number of items per `process_many` call | `COUNT` |
|
|
48
50
|
| `RUNTIME` | `direct`, `concurrent`, or `parallel` | `concurrent` |
|
|
49
51
|
| `CDC_CONCURRENCY` | Async task limit for `cdc-concurrent` | `100` |
|
|
50
|
-
| `CDC_PARALLEL_SIZE` | Ractor worker count for `cdc-parallel` | `Etc.nprocessors - 1` |
|
|
51
|
-
| `CDC_TIMEOUT` | Per-item timeout in seconds |
|
|
52
|
-
| `PRESERVE_ORDER` | Preserve result order for concurrent runtime | `true` |
|
|
53
|
-
| `WARMUP` | Warmup items before timing | `min(COUNT / 50,
|
|
52
|
+
| `CDC_PARALLEL_SIZE` | Ractor worker count for `cdc-parallel` | `Etc.nprocessors - 1`, minimum `1` |
|
|
53
|
+
| `CDC_TIMEOUT` | Per-item timeout in seconds | `nil` |
|
|
54
|
+
| `PRESERVE_ORDER` | Preserve result order for the `:concurrent` runtime | `true` |
|
|
55
|
+
| `WARMUP` | Warmup items before timing | `min(COUNT / 50, 10_000)` |
|
|
54
56
|
| `JSON` | Print machine-readable JSON when set to `1` | unset |
|
|
55
57
|
|
|
56
58
|
## Snapshot: 500,000 No-op Items
|
|
@@ -70,10 +72,12 @@ Results:
|
|
|
70
72
|
| Runtime | Knobs | Elapsed | Throughput | GC count |
|
|
71
73
|
| --- | --- | ---: | ---: | ---: |
|
|
72
74
|
| `direct` | default direct execution | `0.085821 sec` | `5,826,083 items/sec` | `0` |
|
|
73
|
-
| `parallel` | `CDC_PARALLEL_SIZE=7
|
|
74
|
-
| `parallel` | `CDC_PARALLEL_SIZE=7
|
|
75
|
+
| `parallel` | `CDC_PARALLEL_SIZE=7`, run 1 | `6.613177 sec` | `75,607 items/sec` | `58` |
|
|
76
|
+
| `parallel` | `CDC_PARALLEL_SIZE=7`, run 2 | `5.830767 sec` | `85,752 items/sec` | `44` |
|
|
75
77
|
| `concurrent` | `CDC_CONCURRENCY=100` | `12.667181 sec` | `39,472 items/sec` | `45` |
|
|
76
78
|
|
|
79
|
+
The duplicate `parallel` rows are separate sample runs with the same settings. Keep that variance in mind when comparing small differences between runtimes.
|
|
80
|
+
|
|
77
81
|
## Interpretation
|
|
78
82
|
|
|
79
83
|
This snapshot is intentionally a no-op workload. It is useful for measuring runtime overhead, not real downstream work.
|
|
@@ -123,6 +127,8 @@ CDC_CONCURRENCY=25
|
|
|
123
127
|
|
|
124
128
|
then increase gradually. A concurrency value of `100` can be reasonable for I/O-bound workloads, but it is pure overhead for no-op work.
|
|
125
129
|
|
|
130
|
+
When comparing results, keep `COUNT`, `BATCH_SIZE`, Ruby version, CPU count, and runtime gem versions fixed. Changing any of those can shift the result more than a runtime tuning change.
|
|
131
|
+
|
|
126
132
|
## Benchmark Rule of Thumb
|
|
127
133
|
|
|
128
134
|
```text
|
|
@@ -32,7 +32,7 @@ module CDC
|
|
|
32
32
|
# @return [Boolean] whether array payloads should be processed with #process_many by default.
|
|
33
33
|
attr_accessor :batch_payloads
|
|
34
34
|
|
|
35
|
-
# @return [void]
|
|
35
|
+
# @return [void] returns nothing.
|
|
36
36
|
def initialize
|
|
37
37
|
@default_runtime = :concurrent
|
|
38
38
|
@parallel_size = [Etc.nprocessors - 1, 1].max
|
data/lib/cdc/sidekiq/errors.rb
CHANGED
|
@@ -17,7 +17,7 @@ module CDC
|
|
|
17
17
|
attr_reader :failures
|
|
18
18
|
|
|
19
19
|
# @param failures [Array<Object>] failed processor results that should be exposed to Sidekiq retry handling.
|
|
20
|
-
# @return [void]
|
|
20
|
+
# @return [void] returns nothing.
|
|
21
21
|
def initialize(failures)
|
|
22
22
|
@failures = failures.freeze
|
|
23
23
|
super("CDC processor failed for #{failures.length} item(s)")
|
|
@@ -30,7 +30,7 @@ module CDC
|
|
|
30
30
|
# Add CDC processor-job class methods to the including job class.
|
|
31
31
|
#
|
|
32
32
|
# @param base [Class] Sidekiq job class including this module.
|
|
33
|
-
# @return [void]
|
|
33
|
+
# @return [void] returns nothing.
|
|
34
34
|
def self.included(base)
|
|
35
35
|
base.extend(ClassMethods)
|
|
36
36
|
end
|
|
@@ -53,8 +53,10 @@ module CDC
|
|
|
53
53
|
private
|
|
54
54
|
|
|
55
55
|
def process_payload(payload)
|
|
56
|
-
|
|
57
|
-
|
|
56
|
+
job_class = self.class
|
|
57
|
+
# @type var job_class: untyped
|
|
58
|
+
runtime = job_class.__cdc_sidekiq_runtime
|
|
59
|
+
if payload.is_a?(Array) && job_class.__cdc_sidekiq_batch_payloads
|
|
58
60
|
runtime.process_many(payload)
|
|
59
61
|
else
|
|
60
62
|
runtime.process(payload)
|
|
@@ -62,9 +64,14 @@ module CDC
|
|
|
62
64
|
end
|
|
63
65
|
|
|
64
66
|
def handle_processor_failures(results)
|
|
65
|
-
|
|
67
|
+
job_class = self.class
|
|
68
|
+
# @type var job_class: untyped
|
|
69
|
+
return unless job_class.__cdc_sidekiq_raise_on_failure
|
|
66
70
|
|
|
67
|
-
failures = Array(results).select
|
|
71
|
+
failures = Array(results).select do |result|
|
|
72
|
+
# @type var result: untyped
|
|
73
|
+
result.respond_to?(:failure?) && result.failure?
|
|
74
|
+
end
|
|
68
75
|
raise ProcessorFailureError, failures unless failures.empty?
|
|
69
76
|
end
|
|
70
77
|
|
|
@@ -179,7 +186,7 @@ module CDC
|
|
|
179
186
|
|
|
180
187
|
def __cdc_sidekiq_processor
|
|
181
188
|
processor = @cdc_processor
|
|
182
|
-
raise MissingProcessorError, "#{name} must declare cdc_processor" unless processor
|
|
189
|
+
raise MissingProcessorError, "#{__send__(:name)} must declare cdc_processor" unless processor
|
|
183
190
|
|
|
184
191
|
processor.is_a?(Class) ? processor.new : processor
|
|
185
192
|
end
|
data/lib/cdc/sidekiq/runtime.rb
CHANGED
|
@@ -14,7 +14,7 @@ module CDC
|
|
|
14
14
|
# @param concurrency [Integer] number of Async tasks used by cdc-concurrent.
|
|
15
15
|
# @param timeout [Float, nil] optional timeout passed to the selected runtime.
|
|
16
16
|
# @param preserve_order [Boolean] whether cdc-concurrent should preserve input order.
|
|
17
|
-
# @return [void]
|
|
17
|
+
# @return [void] returns nothing.
|
|
18
18
|
def initialize(processor:, runtime:, parallel_size:, concurrency:, timeout:, preserve_order:)
|
|
19
19
|
@processor = processor
|
|
20
20
|
@runtime = runtime.to_sym
|
|
@@ -69,7 +69,7 @@ module CDC
|
|
|
69
69
|
# Minimal runtime used for tests and simple sequential execution.
|
|
70
70
|
class DirectPool
|
|
71
71
|
# @param processor [Object] CDC processor object that responds to #process.
|
|
72
|
-
# @return [void]
|
|
72
|
+
# @return [void] returns nothing.
|
|
73
73
|
def initialize(processor)
|
|
74
74
|
@processor = processor
|
|
75
75
|
end
|
data/lib/cdc/sidekiq/version.rb
CHANGED
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
module CDC
|
|
2
|
+
module Sidekiq
|
|
3
|
+
# Runtime defaults shared by CDC-aware Sidekiq jobs.
|
|
4
|
+
#
|
|
5
|
+
# The configuration intentionally describes only the CDC execution layer.
|
|
6
|
+
# Sidekiq still owns queue selection, scheduling, retries, durability, and
|
|
7
|
+
# job concurrency. cdc-sidekiq owns runtime selection for work performed
|
|
8
|
+
# inside a Sidekiq job.
|
|
9
|
+
class Configuration
|
|
10
|
+
@default_runtime: Symbol
|
|
11
|
+
|
|
12
|
+
@parallel_size: Integer
|
|
13
|
+
|
|
14
|
+
@concurrency: Integer
|
|
15
|
+
|
|
16
|
+
@timeout: Float?
|
|
17
|
+
|
|
18
|
+
@preserve_order: bool
|
|
19
|
+
|
|
20
|
+
@raise_on_failure: bool
|
|
21
|
+
|
|
22
|
+
@batch_payloads: bool
|
|
23
|
+
|
|
24
|
+
# @return [Symbol] default runtime used when a job does not declare one.
|
|
25
|
+
attr_accessor default_runtime: Symbol
|
|
26
|
+
|
|
27
|
+
# @return [Integer] default number of Ractor workers for cdc-parallel jobs.
|
|
28
|
+
attr_accessor parallel_size: Integer
|
|
29
|
+
|
|
30
|
+
# @return [Integer] default number of Async tasks for cdc-concurrent jobs.
|
|
31
|
+
attr_accessor concurrency: Integer
|
|
32
|
+
|
|
33
|
+
# @return [Float, nil] default per-item timeout passed to CDC processor pools.
|
|
34
|
+
attr_accessor timeout: Float?
|
|
35
|
+
|
|
36
|
+
# @return [Boolean] default result-ordering policy for cdc-concurrent jobs.
|
|
37
|
+
attr_accessor preserve_order: bool
|
|
38
|
+
|
|
39
|
+
# @return [Boolean] default failure policy for processor jobs.
|
|
40
|
+
attr_accessor raise_on_failure: bool
|
|
41
|
+
|
|
42
|
+
# @return [Boolean] whether array payloads should be processed with #process_many by default.
|
|
43
|
+
attr_accessor batch_payloads: bool
|
|
44
|
+
|
|
45
|
+
# @return [void]
|
|
46
|
+
def initialize: () -> void
|
|
47
|
+
|
|
48
|
+
# Build an immutable copy so job-level overrides cannot mutate globals.
|
|
49
|
+
#
|
|
50
|
+
# @return [Configuration] independent copy of this configuration.
|
|
51
|
+
def dup: () -> Configuration
|
|
52
|
+
end
|
|
53
|
+
end
|
|
54
|
+
end
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
module CDC
|
|
2
|
+
module Sidekiq
|
|
3
|
+
# Base error for all cdc-sidekiq failures.
|
|
4
|
+
class Error < StandardError
|
|
5
|
+
end
|
|
6
|
+
|
|
7
|
+
# Raised when a job declares an unsupported CDC runtime.
|
|
8
|
+
class UnsupportedRuntimeError < Error
|
|
9
|
+
end
|
|
10
|
+
|
|
11
|
+
# Raised when a Sidekiq processor job does not declare a processor.
|
|
12
|
+
class MissingProcessorError < Error
|
|
13
|
+
end
|
|
14
|
+
|
|
15
|
+
# Raised when processor execution returns one or more failed results.
|
|
16
|
+
class ProcessorFailureError < Error
|
|
17
|
+
@failures: untyped
|
|
18
|
+
|
|
19
|
+
# @return [Array<Object>] failed processor results that triggered the error.
|
|
20
|
+
attr_reader failures: Array[untyped]
|
|
21
|
+
|
|
22
|
+
# @param failures [Array<Object>] failed processor results that should be exposed to Sidekiq retry handling.
|
|
23
|
+
# @return [void]
|
|
24
|
+
def initialize: (Array[untyped] failures) -> void
|
|
25
|
+
end
|
|
26
|
+
end
|
|
27
|
+
end
|
|
@@ -0,0 +1,155 @@
|
|
|
1
|
+
module CDC
|
|
2
|
+
module Sidekiq
|
|
3
|
+
# Sidekiq job mixin that executes work through CDC runtime primitives.
|
|
4
|
+
#
|
|
5
|
+
# The job remains a normal Sidekiq job. Sidekiq still owns scheduling,
|
|
6
|
+
# retries, queues, and persistence. cdc-sidekiq only changes how the job
|
|
7
|
+
# executes its payload once Sidekiq has started the job.
|
|
8
|
+
#
|
|
9
|
+
# @example Process many items through cdc-parallel
|
|
10
|
+
# class ReindexUsersJob
|
|
11
|
+
# include Sidekiq::Job
|
|
12
|
+
# include CDC::Sidekiq::ProcessorJob
|
|
13
|
+
#
|
|
14
|
+
# cdc_processor UserIndexer
|
|
15
|
+
# cdc_runtime :parallel
|
|
16
|
+
# end
|
|
17
|
+
#
|
|
18
|
+
# @example Process I/O-heavy items through cdc-concurrent
|
|
19
|
+
# class DeliverWebhooksJob
|
|
20
|
+
# include Sidekiq::Job
|
|
21
|
+
# include CDC::Sidekiq::ProcessorJob
|
|
22
|
+
#
|
|
23
|
+
# cdc_processor WebhookDeliverer
|
|
24
|
+
# cdc_runtime :concurrent
|
|
25
|
+
# cdc_concurrency 250
|
|
26
|
+
# end
|
|
27
|
+
module ProcessorJob
|
|
28
|
+
# Add CDC processor-job class methods to the including job class.
|
|
29
|
+
#
|
|
30
|
+
# @param base [Class] Sidekiq job class including this module.
|
|
31
|
+
# @return [void]
|
|
32
|
+
def self.included: (untyped base) -> untyped
|
|
33
|
+
|
|
34
|
+
# Execute a Sidekiq payload through the configured CDC runtime.
|
|
35
|
+
#
|
|
36
|
+
# Array payloads are processed with #process_many when cdc_batch_payloads
|
|
37
|
+
# is enabled. Other payloads are processed with #process.
|
|
38
|
+
#
|
|
39
|
+
# @param payload [Object, Array<Object>] Sidekiq job payload or batch payload.
|
|
40
|
+
# @return [Object, Array<Object>] CDC processor result or frozen result array.
|
|
41
|
+
# @raise [MissingProcessorError] when the job does not declare a CDC processor.
|
|
42
|
+
# @raise [ProcessorFailureError] when raise-on-failure is enabled and one or more results failed.
|
|
43
|
+
def perform: (untyped payload) -> untyped
|
|
44
|
+
|
|
45
|
+
private
|
|
46
|
+
|
|
47
|
+
def process_payload: (untyped payload) -> untyped
|
|
48
|
+
|
|
49
|
+
def handle_processor_failures: (untyped results) -> (nil | untyped)
|
|
50
|
+
|
|
51
|
+
# Class-level declaration helpers for CDC-aware Sidekiq jobs.
|
|
52
|
+
module ClassMethods
|
|
53
|
+
@cdc_processor: untyped
|
|
54
|
+
|
|
55
|
+
@cdc_runtime: untyped
|
|
56
|
+
|
|
57
|
+
@cdc_parallel_size: untyped
|
|
58
|
+
|
|
59
|
+
@cdc_concurrency: untyped
|
|
60
|
+
|
|
61
|
+
@cdc_timeout: untyped
|
|
62
|
+
|
|
63
|
+
@cdc_preserve_order: untyped
|
|
64
|
+
|
|
65
|
+
@cdc_batch_payloads: untyped
|
|
66
|
+
|
|
67
|
+
@cdc_raise_on_failure: untyped
|
|
68
|
+
|
|
69
|
+
# Declare or read the processor used by this job.
|
|
70
|
+
#
|
|
71
|
+
# @param value [Class, Object, nil] processor class or processor instance.
|
|
72
|
+
# @return [Class, Object, nil] configured processor when called without an argument.
|
|
73
|
+
def cdc_processor: () -> untyped
|
|
74
|
+
| (untyped value) -> untyped
|
|
75
|
+
|
|
76
|
+
# Declare or read the CDC runtime used by this job.
|
|
77
|
+
#
|
|
78
|
+
# @param value [Symbol, String, nil] runtime name, such as :parallel, :concurrent, or :direct.
|
|
79
|
+
# @return [Symbol, nil] configured runtime when called without an argument.
|
|
80
|
+
def cdc_runtime: () -> Symbol?
|
|
81
|
+
| (Symbol | String value) -> Symbol
|
|
82
|
+
|
|
83
|
+
# Declare or read the cdc-parallel worker count for this job.
|
|
84
|
+
#
|
|
85
|
+
# @param value [Integer, nil] number of Ractor workers for this job.
|
|
86
|
+
# @return [Integer, nil] configured worker count when called without an argument.
|
|
87
|
+
def cdc_parallel_size: () -> Integer?
|
|
88
|
+
| (Integer value) -> Integer
|
|
89
|
+
|
|
90
|
+
# Declare or read the cdc-concurrent task concurrency for this job.
|
|
91
|
+
#
|
|
92
|
+
# @param value [Integer, nil] maximum Async task count for this job.
|
|
93
|
+
# @return [Integer, nil] configured concurrency when called without an argument.
|
|
94
|
+
def cdc_concurrency: () -> Integer?
|
|
95
|
+
| (Integer value) -> Integer
|
|
96
|
+
|
|
97
|
+
# Declare or read the runtime timeout for this job.
|
|
98
|
+
#
|
|
99
|
+
# @param value [Float, Integer, nil] timeout in seconds, or nil for no timeout.
|
|
100
|
+
# @return [Float, nil] configured timeout when called without an argument.
|
|
101
|
+
def cdc_timeout: () -> Float?
|
|
102
|
+
| (Numeric? value) -> Float?
|
|
103
|
+
|
|
104
|
+
# Declare or read result ordering for cdc-concurrent.
|
|
105
|
+
#
|
|
106
|
+
# @param value [Boolean, nil] true to preserve input order, false to keep completion order.
|
|
107
|
+
# @return [Boolean, nil] configured ordering policy when called without an argument.
|
|
108
|
+
def cdc_preserve_order: () -> bool?
|
|
109
|
+
| (bool value) -> bool
|
|
110
|
+
|
|
111
|
+
# Declare or read whether array payloads use #process_many.
|
|
112
|
+
#
|
|
113
|
+
# @param value [Boolean, nil] true to batch array payloads, false to process the array as one item.
|
|
114
|
+
# @return [Boolean, nil] configured batching policy when called without an argument.
|
|
115
|
+
def cdc_batch_payloads: () -> bool?
|
|
116
|
+
| (bool value) -> bool
|
|
117
|
+
|
|
118
|
+
# Declare or read whether failed ProcessorResult objects raise.
|
|
119
|
+
#
|
|
120
|
+
# @param value [Boolean, nil] true to raise on failed results so Sidekiq retries the job.
|
|
121
|
+
# @return [Boolean, nil] configured failure policy when called without an argument.
|
|
122
|
+
def cdc_raise_on_failure: () -> bool?
|
|
123
|
+
| (bool value) -> bool
|
|
124
|
+
|
|
125
|
+
# Build the runtime used by one Sidekiq job invocation.
|
|
126
|
+
#
|
|
127
|
+
# @return [Runtime] runtime configured for this job class.
|
|
128
|
+
# @raise [MissingProcessorError] when no processor has been declared.
|
|
129
|
+
def __cdc_sidekiq_runtime: () -> Runtime
|
|
130
|
+
|
|
131
|
+
# @return [Boolean] true when array payloads should use #process_many.
|
|
132
|
+
def __cdc_sidekiq_batch_payloads: () -> bool
|
|
133
|
+
|
|
134
|
+
# @return [Boolean] true when failed results should raise.
|
|
135
|
+
def __cdc_sidekiq_raise_on_failure: () -> bool
|
|
136
|
+
|
|
137
|
+
private
|
|
138
|
+
|
|
139
|
+
def __cdc_sidekiq_processor: () -> untyped
|
|
140
|
+
|
|
141
|
+
def __cdc_sidekiq_runtime_name: () -> untyped
|
|
142
|
+
|
|
143
|
+
def __cdc_sidekiq_parallel_size: () -> untyped
|
|
144
|
+
|
|
145
|
+
def __cdc_sidekiq_concurrency: () -> untyped
|
|
146
|
+
|
|
147
|
+
def __cdc_sidekiq_timeout: () -> untyped
|
|
148
|
+
|
|
149
|
+
def __cdc_sidekiq_preserve_order: () -> untyped
|
|
150
|
+
|
|
151
|
+
def configured_boolean: (untyped value, untyped fallback) -> untyped
|
|
152
|
+
end
|
|
153
|
+
end
|
|
154
|
+
end
|
|
155
|
+
end
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
module CDC
|
|
2
|
+
module Sidekiq
|
|
3
|
+
# Executes a CDC processor through one of the CDC runtime primitives.
|
|
4
|
+
#
|
|
5
|
+
# Runtime is intentionally selected outside Sidekiq's own concurrency
|
|
6
|
+
# setting. Sidekiq concurrency controls how many jobs run at once. This
|
|
7
|
+
# object controls how one CDC-aware job fans work out internally.
|
|
8
|
+
class Runtime
|
|
9
|
+
@processor: untyped
|
|
10
|
+
|
|
11
|
+
@runtime: untyped
|
|
12
|
+
|
|
13
|
+
@parallel_size: untyped
|
|
14
|
+
|
|
15
|
+
@concurrency: untyped
|
|
16
|
+
|
|
17
|
+
@timeout: untyped
|
|
18
|
+
|
|
19
|
+
@preserve_order: untyped
|
|
20
|
+
|
|
21
|
+
# @param processor [Object] CDC processor object that responds to #process.
|
|
22
|
+
# @param runtime [Symbol] execution runtime, currently :parallel, :concurrent, or :direct.
|
|
23
|
+
# @param parallel_size [Integer] number of Ractors used by cdc-parallel.
|
|
24
|
+
# @param concurrency [Integer] number of Async tasks used by cdc-concurrent.
|
|
25
|
+
# @param timeout [Float, nil] optional timeout passed to the selected runtime.
|
|
26
|
+
# @param preserve_order [Boolean] whether cdc-concurrent should preserve input order.
|
|
27
|
+
# @return [void]
|
|
28
|
+
def initialize: (
|
|
29
|
+
processor: untyped,
|
|
30
|
+
runtime: Symbol | String,
|
|
31
|
+
parallel_size: Integer,
|
|
32
|
+
concurrency: Integer,
|
|
33
|
+
timeout: Float?,
|
|
34
|
+
preserve_order: bool
|
|
35
|
+
) -> void
|
|
36
|
+
|
|
37
|
+
# Process one work item through the selected runtime.
|
|
38
|
+
#
|
|
39
|
+
# @param item [Object] work item passed to the processor.
|
|
40
|
+
# @return [Object] processor result returned by the selected runtime.
|
|
41
|
+
def process: (untyped item) -> untyped
|
|
42
|
+
|
|
43
|
+
# Process many work items through the selected runtime.
|
|
44
|
+
#
|
|
45
|
+
# @param items [Array<Object>] work items passed to the processor.
|
|
46
|
+
# @return [Array<Object>] processor results returned by the selected runtime.
|
|
47
|
+
def process_many: (Array[untyped] items) -> Array[untyped]
|
|
48
|
+
|
|
49
|
+
private
|
|
50
|
+
|
|
51
|
+
attr_reader processor: untyped
|
|
52
|
+
|
|
53
|
+
attr_reader runtime: Symbol
|
|
54
|
+
|
|
55
|
+
attr_reader parallel_size: Integer
|
|
56
|
+
|
|
57
|
+
attr_reader concurrency: Integer
|
|
58
|
+
|
|
59
|
+
attr_reader timeout: Float?
|
|
60
|
+
|
|
61
|
+
attr_reader preserve_order: bool
|
|
62
|
+
|
|
63
|
+
def with_pool: () { (untyped) -> untyped } -> untyped
|
|
64
|
+
|
|
65
|
+
def build_pool: () -> untyped
|
|
66
|
+
|
|
67
|
+
# Minimal runtime used for tests and simple sequential execution.
|
|
68
|
+
class DirectPool
|
|
69
|
+
@processor: untyped
|
|
70
|
+
|
|
71
|
+
# @param processor [Object] CDC processor object that responds to #process.
|
|
72
|
+
# @return [void]
|
|
73
|
+
def initialize: (untyped processor) -> void
|
|
74
|
+
|
|
75
|
+
# @param item [Object] work item passed to the processor.
|
|
76
|
+
# @return [Object] processor result returned by the processor.
|
|
77
|
+
def process: (untyped item) -> untyped
|
|
78
|
+
|
|
79
|
+
# @param items [Array<Object>] work items passed to the processor.
|
|
80
|
+
# @return [Array<Object>] processor results returned by the processor.
|
|
81
|
+
def process_many: (Array[untyped] items) -> Array[untyped]
|
|
82
|
+
end
|
|
83
|
+
end
|
|
84
|
+
end
|
|
85
|
+
end
|
data/sig/cdc/sidekiq.rbs
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
module CDC
|
|
2
|
+
# Integration layer between Sidekiq and CDC execution primitives.
|
|
3
|
+
module Sidekiq
|
|
4
|
+
self.@configuration: Configuration
|
|
5
|
+
|
|
6
|
+
# Read the process-wide cdc-sidekiq configuration.
|
|
7
|
+
#
|
|
8
|
+
# @return [Configuration] mutable global configuration object.
|
|
9
|
+
def self.configuration: () -> Configuration
|
|
10
|
+
|
|
11
|
+
# Configure process-wide defaults for CDC-aware Sidekiq jobs.
|
|
12
|
+
#
|
|
13
|
+
# @yieldparam configuration [Configuration] mutable configuration object.
|
|
14
|
+
# @return [Configuration] configured global configuration object.
|
|
15
|
+
def self.configure: () ?{ (Configuration) -> void } -> Configuration
|
|
16
|
+
|
|
17
|
+
# Reset process-wide configuration to defaults.
|
|
18
|
+
#
|
|
19
|
+
# @return [Configuration] new default configuration object.
|
|
20
|
+
def self.reset_configuration!: () -> Configuration
|
|
21
|
+
end
|
|
22
|
+
end
|
data/sig/cdc_sidekiq.rbs
ADDED
|
File without changes
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: cdc-sidekiq
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.1.
|
|
4
|
+
version: 0.1.1
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Ken C. Demanawa
|
|
@@ -13,30 +13,30 @@ dependencies:
|
|
|
13
13
|
name: cdc-core
|
|
14
14
|
requirement: !ruby/object:Gem::Requirement
|
|
15
15
|
requirements:
|
|
16
|
-
- - "
|
|
16
|
+
- - "~>"
|
|
17
17
|
- !ruby/object:Gem::Version
|
|
18
|
-
version:
|
|
18
|
+
version: 0.1.3
|
|
19
19
|
type: :runtime
|
|
20
20
|
prerelease: false
|
|
21
21
|
version_requirements: !ruby/object:Gem::Requirement
|
|
22
22
|
requirements:
|
|
23
|
-
- - "
|
|
23
|
+
- - "~>"
|
|
24
24
|
- !ruby/object:Gem::Version
|
|
25
|
-
version:
|
|
25
|
+
version: 0.1.3
|
|
26
26
|
- !ruby/object:Gem::Dependency
|
|
27
27
|
name: sidekiq
|
|
28
28
|
requirement: !ruby/object:Gem::Requirement
|
|
29
29
|
requirements:
|
|
30
|
-
- - "
|
|
30
|
+
- - "~>"
|
|
31
31
|
- !ruby/object:Gem::Version
|
|
32
|
-
version: '
|
|
32
|
+
version: '8.1'
|
|
33
33
|
type: :runtime
|
|
34
34
|
prerelease: false
|
|
35
35
|
version_requirements: !ruby/object:Gem::Requirement
|
|
36
36
|
requirements:
|
|
37
|
-
- - "
|
|
37
|
+
- - "~>"
|
|
38
38
|
- !ruby/object:Gem::Version
|
|
39
|
-
version: '
|
|
39
|
+
version: '8.1'
|
|
40
40
|
description: |
|
|
41
41
|
Adds CDC-aware processor jobs and runtime selection to Sidekiq,
|
|
42
42
|
allowing selected jobs to execute payloads directly, through cdc-parallel, or through cdc-concurrent.
|
|
@@ -57,6 +57,13 @@ files:
|
|
|
57
57
|
- lib/cdc/sidekiq/runtime.rb
|
|
58
58
|
- lib/cdc/sidekiq/version.rb
|
|
59
59
|
- lib/cdc_sidekiq.rb
|
|
60
|
+
- sig/cdc/sidekiq.rbs
|
|
61
|
+
- sig/cdc/sidekiq/configuration.rbs
|
|
62
|
+
- sig/cdc/sidekiq/errors.rbs
|
|
63
|
+
- sig/cdc/sidekiq/processor_job.rbs
|
|
64
|
+
- sig/cdc/sidekiq/runtime.rbs
|
|
65
|
+
- sig/cdc/sidekiq/version.rbs
|
|
66
|
+
- sig/cdc_sidekiq.rbs
|
|
60
67
|
homepage: https://github.com/kanutocd/cdc-sidekiq
|
|
61
68
|
licenses:
|
|
62
69
|
- MIT
|