wurk 1.7.0 → 1.7.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 4ce7dc086fc21fd3dd300206e2c9b15b2e8720a59e8186ea1459b66067bf6148
4
- data.tar.gz: 1a659643c694f6fadd7bd34313604df1093274c4e50597c00ff767db4376f3ed
3
+ metadata.gz: 60f518fc753861a0670a0922cff88670ed430890c8d7bad70bf2e9ff4bcd173e
4
+ data.tar.gz: c9ed06c696ba18a30658ccd30cdbc29217bed5b0a66e02af13ae831021a210a6
5
5
  SHA512:
6
- metadata.gz: e6dd4d7643349e6ed95496e5149f19f3ca832589f4475b5de8e03f438dc4e445a3871b1ccf6240754bb85708f78a8118962234a465f3afa150b9753368a6fe77
7
- data.tar.gz: 4a31cfe2170ab1d726db165d354edc00763e14b118b0219f15bfb0bc0de909c03a795829196e8a6b6e56c7f5e013a3dde4b7d94d0ebf49f2aec0c75d2c6a267e
6
+ metadata.gz: fa05f9a8433f9a5b475b7e4d075e8fe209aebffc6e286ac57f28c9dcbf3efdc18a36d4393fcc718e663df3b59dd508bf0effb709e59752a127ab8f58d409728b
7
+ data.tar.gz: 3a8fab767e83ae14c185a054be1d356a7161b9ded7771cafc4384db1e5f14292169dce1aafe5038113586379838a0053b4d4b67f551823e6bc2268a6cf3c1c75
data/README.md CHANGED
@@ -48,18 +48,44 @@ gem "wurk"
48
48
 
49
49
  ## Feature matrix
50
50
 
51
- Everything below is in the one free gem. The "Sidekiq tier" column maps each area onto Sidekiq's own lineup, so you can see at a glance what a migration covers.
52
-
53
- | Area | What you get | Sidekiq tier |
54
- |---|---|---|
55
- | **Runtime** | Fork-based real parallelism, reliable `BLMOVE` fetch, PID supervision, rolling restarts, graceful drain, scheduled/retry pollers | OSS + Pro |
56
- | **Batches** | `Sidekiq::Batch` with `on(:success/:complete/:death)` callbacks, nested batches, progress | Pro |
57
- | **Limiters** | Concurrent, bucket, window, leaky, and points rate limiters via `Sidekiq::Limiter` | Enterprise |
58
- | **Periodic** | Cron/periodic jobs, leader-elected so each tick fires exactly once across the cluster | Enterprise |
59
- | **Encryption** | Transparent AES-256-GCM job-argument encryption with zero-downtime key rotation | Enterprise |
60
- | **Dashboard** | Mountable Rails engine, precompiled SolidJS SPA (no Node needed), live SSE, charts, host-app auth hook | OSS + Pro/Ent |
61
-
62
- Plus the [Wurk extras](#wurk-extras) below: a worker topology DSL, a Kubernetes liveness/readiness listener, OpenTelemetry tracing, job status/progress/results, an HTTP producer+observe API, per-job timeouts/deadlines, debounce/throttle/collapse, global per-queue concurrency caps, and DAG flows.
51
+ Every capability Sidekiq splits across three tiers is in the one free gem. Columns are Sidekiq's own lineup, so you can see exactly what a migration covers.
52
+
53
+ | Capability | OSS | Pro | Ent | **Wurk** |
54
+ |---|:---:|:---:|:---:|:---:|
55
+ | Threaded workers, middleware, retries with backoff, dead set | | | | **✅** |
56
+ | Scheduled jobs (`perform_in` / `perform_at`), Active Job adapter | | ✅ | ✅ | **✅** |
57
+ | Web dashboard, Data API, testing modes | | | | **✅** |
58
+ | Reliable fetch atomic `BLMOVE`, survives `SIGKILL` | | | | **✅** |
59
+ | Batches: `on(:success/:complete/:death)`, nesting, progress | | | | **✅** |
60
+ | Reliable scheduler · reliable client (Redis-outage buffering) | | | | **✅** |
61
+ | Queue pause/resume · job expiration (`expires_in`) | — | ✅ | ✅ | **✅** |
62
+ | StatsD / DogStatsD metrics export | | | | **✅** |
63
+ | Rate limiting — concurrent, bucket, window, leaky, points | — | — | ✅ | **✅** |
64
+ | Periodic (cron) jobs, leader-elected so each tick fires once | — | — | ✅ | **✅** |
65
+ | Unique jobs, with custom lock context | — | — | ✅ | **✅** |
66
+ | Encryption — AES-256-GCM args, zero-downtime key rotation | — | — | ✅ | **✅** |
67
+ | Historical metrics retained in Redis | — | — | ✅ | **✅** |
68
+ | Multi-process fork parallelism (`swarm`) + rolling restarts | — | — | ✅ | **✅** |
69
+ | **Licence** | LGPL-3.0 | commercial | commercial | **MIT** |
70
+
71
+ ### Beyond Sidekiq
72
+
73
+ Same table, other direction — these have no Sidekiq equivalent at any tier. All opt-in, and free on the job path until you turn them on.
74
+
75
+ | Capability | OSS | Pro | Ent | **Wurk** |
76
+ |---|:---:|:---:|:---:|:---:|
77
+ | [Kubernetes `/live` + `/ready` probe listener](#kubernetes-metrics--tracing) | — | — | — | **✅** |
78
+ | [OpenTelemetry tracing](docs/telemetry.md) — W3C context, client → server | — | — | — | **✅** |
79
+ | [HTTP producer + observe API](docs/api-http.md) — enqueue/inspect over JSON | — | — | — | **✅** |
80
+ | [Job status, progress & results](docs/job-status.md) | — | — | — | **✅** |
81
+ | [Flows — DAG on batches](docs/flows.md) with piped results | — | — | — | **✅** |
82
+ | [Global per-queue concurrency caps](docs/rate-limiting.md) (cluster-wide) | — | — | — | **✅** |
83
+ | [Debounce, throttle-to-slot & collapse](docs/unique-jobs.md) | — | — | — | **✅** |
84
+ | [Per-job timeouts & deadlines](docs/retries.md) | — | — | — | **✅** |
85
+ | Worker topology DSL — fleet roles in code, not `-q` flags | — | — | — | **✅** |
86
+ | Dashboard theme, locale & 400-zone timezone picker | — | — | — | **✅** |
87
+
88
+ Details, and what you give up if you migrate back, in [Wurk extras](#wurk-extras).
63
89
 
64
90
  ## Wurk extras
65
91
 
@@ -75,11 +101,31 @@ Sidekiq has no equivalent for any of these — they aren't parity, they're new s
75
101
  | **[Per-job timeouts & deadlines](docs/retries.md)** | `timeout:` bounds one attempt, `deadline:` bounds the whole job from enqueue, enforced by a lightweight per-capsule watchdog thread (no thread-per-job) | Runaway/stuck jobs run unbounded except for `shutdown_timeout` |
76
102
  | **[Global per-queue concurrency caps](docs/rate-limiting.md)** | `config.global_concurrency = { critical: 20 }` caps in-flight jobs for a queue across the whole cluster, folded into the fetch pipeline | The cluster-wide cap; only per-key `Limiter`s remain |
77
103
  | **Worker topology DSL** | Declare which queues/classes a given fleet role runs, in code instead of ad hoc `-q` flags | The declarative topology; fall back to CLI queue flags |
78
- | **[Kubernetes probes](#kubernetes-probes)** | `config.health_check` opens a thin `/live`/`/ready` HTTP listener, self-electing across a swarm's children | The built-in probe listener; roll your own liveness check |
104
+ | **[Kubernetes probes](#kubernetes-metrics--tracing)** | `config.health_check` opens a thin `/live`/`/ready` HTTP listener, self-electing across a swarm's children | The built-in probe listener; roll your own liveness check |
79
105
  | **Dashboard theme, locale & timezone** | Light/dark/system theme, per-visitor locale override, and a 400-zone timezone picker for every timestamp in the SPA | Nothing server-side — this is dashboard-only |
80
106
 
81
107
  AI dashboard panes — anomaly detection, natural-language queries, error triage, and capacity forecasting — are **planned, not shipped**: they're [roadmap M5](docs/idea/13-roadmap.md#m5--ai-dashboard), after the M4.5 extras above.
82
108
 
109
+ ## Benchmarks
110
+
111
+ **Wurk is not faster than stock Sidekiq today.** Here is where it actually stands, measured rather than claimed — wurk 1.5.0 vs sidekiq 8.1.6, ruby 3.4.7, local Redis 7.4.10, 5000 jobs/run, 12 runs per topology, paired per-run ratios.
112
+
113
+ <picture>
114
+ <source media="(prefers-color-scheme: dark)" srcset="docs/assets/bench-throughput-dark.svg">
115
+ <img alt="Throughput relative to stock Sidekiq. Median of 12 paired runs with min–max range. 1 process × 5 threads: noop 0.87×, cpu 0.99×, io 0.99×. 4 processes × 5 threads: noop 0.95×, cpu 1.02×, io 0.97×." src="docs/assets/bench-throughput-light.svg" width="100%">
116
+ </picture>
117
+
118
+ Parity on `cpu` and `io`; still behind on `noop`, which is pure framework overhead. The spread is wide because the host carried background load — the paired-ratio median is the number to trust, not any single run.
119
+
120
+ <picture>
121
+ <source media="(prefers-color-scheme: dark)" srcset="docs/assets/bench-boot-dark.svg">
122
+ <img alt="Boot to first job, median seconds. 1p × 5t: Sidekiq 0.56s, Wurk 0.72s. 4p × 5t: Sidekiq 0.60s, Wurk 0.78s." src="docs/assets/bench-boot-light.svg" width="100%">
123
+ </picture>
124
+
125
+ Forking is not what closes the throughput gap — a stock Sidekiq user reaches multi-core by running N processes, which is the second topology above. The swarm buys copy-on-write memory and one supervisor, not raw speed.
126
+
127
+ Method, per-invocation records, workload definitions, and the separate `rake bench` regression gate (wurk vs its own past self, which says nothing about Sidekiq): **[docs/benchmarks.md](docs/benchmarks.md)**. Reproduce with `bin/rake bench:vs_sidekiq`.
128
+
83
129
  ## Documentation
84
130
 
85
131
  - **[Website](https://developerz-ai.github.io/wurk/)** · **[Wiki / full docs](https://github.com/developerz-ai/wurk/wiki)** — the pitch, install, and the complete guide.
@@ -103,7 +149,7 @@ AI dashboard panes — anomaly detection, natural-language queries, error triage
103
149
  - **API reference (parity specs):** [Sidekiq OSS](https://github.com/developerz-ai/wurk/blob/main/docs/target/sidekiq-free.md) · [Pro](https://github.com/developerz-ai/wurk/blob/main/docs/target/sidekiq-pro.md) · [Enterprise](https://github.com/developerz-ai/wurk/blob/main/docs/target/sidekiq-ent.md) — the authoritative surface Wurk matches exactly.
104
150
  - **[Authentication & authorization](https://github.com/developerz-ai/wurk/blob/main/docs/authentication.md)** — gate the dashboard behind Devise/Warden, Sorcery, Basic auth, or a token; role-based read/write; CSRF.
105
151
  - **[Securing the dashboard](https://github.com/developerz-ai/wurk/blob/main/docs/dashboard.md)** · **[Metrics history](https://github.com/developerz-ai/wurk/blob/main/docs/metrics-history.md)**
106
- - **[Compatibility & legal basis](https://github.com/developerz-ai/wurk/blob/main/docs/clean-room.md)** — clean-room implementation: Wurk copies the API, not the code (Google v. Oracle).
152
+ - **[Compatibility & legal basis](https://github.com/developerz-ai/wurk/blob/main/docs/compatibility.md)** — independent reimplementation: Wurk reproduces the API and wire format, not Sidekiq's implementation.
107
153
  - **Live demo:** [wurk.demo.developerz.ai](https://wurk.demo.developerz.ai)
108
154
 
109
155
  ## Requirements
@@ -190,9 +236,11 @@ end
190
236
 
191
237
  Keys rotate without downtime — keep every still-in-flight version resolvable so old jobs decrypt, then bump `active_version`. A job that can't be decrypted (key rotated away, corrupt ciphertext) goes **straight to the dead set in under a second** rather than crash-looping through 25 retries, with the still-encrypted payload preserved for replay. The dashboard renders encrypted args as `"<encrypted>"`; cleartext is never written to Redis.
192
238
 
193
- ## Kubernetes probes
239
+ ## Kubernetes, metrics & tracing
194
240
 
195
- Opt in to a thin HTTP listener for liveness/readiness:
241
+ Wurk is built to run as a fleet: one supervisor per pod forking N children across the cores you gave it, drained gracefully on `SIGTERM`, replaced one slot at a time on `SIGUSR1`, and answerable to your existing monitoring rather than a bespoke one.
242
+
243
+ **Probes.** Opt in to a thin HTTP listener for liveness/readiness:
196
244
 
197
245
  ```ruby
198
246
  Wurk.configure_server do |config|
@@ -205,7 +253,21 @@ end
205
253
  | `/live` | 200 while the Launcher is running; 503 once `stop`/`quiet` is called. |
206
254
  | `/ready` | 200 only when Redis is reachable **and** the heartbeat fired within `ready_window` (default 30s); 503 otherwise. |
207
255
 
208
- Knobs: `health_check(port:, bind: "0.0.0.0", ready_window: 30)`. In swarm mode one child owns the port; the others poll every 5s and take it over if the owner dies, so probes survive a child restart.
256
+ Knobs: `health_check(port:, bind: "0.0.0.0", ready_window: 30)`. In swarm mode one child owns the port; the others poll every 5s and take it over if the owner dies, so probes survive a child restart — a pod never fails a probe just because a worker recycled.
257
+
258
+ **Getting the numbers out.** Point these at whatever you already run:
259
+
260
+ | Signal | How it leaves the process | Docs |
261
+ |---|---|---|
262
+ | Job metrics (counts, latency, per-class timing) | StatsD / DogStatsD via `config.dogstatsd` — into Datadog directly, or into Grafana through your StatsD exporter | [metrics](docs/metrics.md) |
263
+ | Historical time series | Retained in Redis, queried by the dashboard or `Wurk::History` | [metrics-history](docs/metrics-history.md) |
264
+ | Distributed traces | OpenTelemetry — W3C `traceparent` propagated client → server, one span per attempt | [telemetry](docs/telemetry.md) |
265
+ | Queue/job/swarm state for external scrapers and autoscalers | Bearer-token `/v1` JSON API, mountable standalone or via `wurk api` | [api-http](docs/api-http.md) |
266
+ | Errors | Built-in Sentry reporting, terminal failures only, no job args | [sentry](docs/sentry.md) |
267
+
268
+ There is no native Prometheus `/metrics` endpoint — the StatsD export or the `/v1` API is the current path into a Prometheus/Grafana stack.
269
+
270
+ **Backpressure at fleet scale.** `config.global_concurrency = { critical: 20 }` caps in-flight jobs for a queue across every pod, folded into the fetch pipeline rather than bolted on as a middleware sleep — see [rate limiting](docs/rate-limiting.md).
209
271
 
210
272
  ## Why Wurk exists
211
273
 
@@ -214,7 +276,7 @@ Infrastructure this basic should be free software. A Rails app shouldn't need a
214
276
  What has made that hard is maintenance: someone has to be paid to do it. Sidekiq funds a decade of *human* maintenance through its paid tiers, which is an honest trade. Wurk makes a different one — it is maintained **AI-first**: implementation, parity suite, docs, and benchmarks are written and kept current by AI agents under human review. A fix, a doc update, or a version bump is no longer somebody's week, which is what makes it practical to:
215
277
 
216
278
  - ship the entire Pro + Enterprise surface with no tier, no flag gate, and no license check;
217
- - keep parity honest mechanically rather than by hand — Sidekiq's own tests run as an oracle suite, and third-party gems (sidekiq-cron, sidekiq-unique-jobs, sidekiq-scheduler, sidekiq-status, sidekiq-failures, sidekiq-throttled) run their upstream suites against Wurk on every push;
279
+ - keep parity honest mechanically rather than by hand — an independently written parity oracle suite, pinned to a documented Sidekiq revision, plus third-party gems (sidekiq-cron, sidekiq-unique-jobs, sidekiq-scheduler, sidekiq-status, sidekiq-failures, sidekiq-throttled) running their own upstream suites against Wurk on every push;
218
280
  - keep adding surface Sidekiq doesn't have — the [Wurk extras](#wurk-extras) above landed as one release;
219
281
  - hold ourselves to published numbers instead of adjectives — the suite runs against stock Sidekiq every release and ships the results [as measured](docs/benchmarks.md), including the unflattering ones.
220
282
 
@@ -241,9 +303,11 @@ Issues and pull requests are welcome — see **[CONTRIBUTING.md](https://github.
241
303
 
242
304
  MIT. See [LICENSE](https://github.com/developerz-ai/wurk/blob/main/LICENSE).
243
305
 
244
- Wurk is a clean-room reimplementation of the Sidekiq **API** — it copies the
245
- interface (so your jobs run unchanged), not Sidekiq's implementation code. This
246
- is the same basis the Supreme Court upheld for Google's reuse of the Java API in
247
- *Google v. Oracle* (2021). "Sidekiq" is a trademark of Contributed Systems, LLC;
248
- Wurk is independent and not affiliated with or endorsed by them. Full reasoning:
249
- **[docs/clean-room.md](https://github.com/developerz-ai/wurk/blob/main/docs/clean-room.md)**.
306
+ Wurk is an independent reimplementation of the Sidekiq **API** — it reproduces
307
+ the interface and wire format (so your jobs run unchanged), not Sidekiq's
308
+ implementation. Reusing an API for interoperability is what the Supreme Court
309
+ held to be fair use in *Google v. Oracle* (2021). Sidekiq itself is LGPL-3.0;
310
+ Wurk neither vendors nor links against it. "Sidekiq" is a trademark of
311
+ Contributed Systems, LLC; Wurk is independent and not affiliated with or
312
+ endorsed by them. Full reasoning:
313
+ **[docs/compatibility.md](https://github.com/developerz-ai/wurk/blob/main/docs/compatibility.md)**.
@@ -30,10 +30,9 @@ module Wurk
30
30
  end
31
31
 
32
32
  # `[records_batch, batch.first.id]` pairs. The size lambda is the record
33
- # count, NOT the batch count — byte-for-byte with upstream Sidekiq's
34
- # `ActiveRecordEnumerator#batches`, so `enum.size` returns the same value
35
- # a drop-in app gets from Sidekiq. (Only the lazy `#size` differs from
36
- # `relations`; the run loop never calls it.)
33
+ # count, NOT the batch count — that is what a drop-in app already gets
34
+ # from `enum.size`, so it is preserved deliberately. (Only the lazy
35
+ # `#size` differs from `relations`; the run loop never calls it.)
37
36
  def batches
38
37
  ::Enumerator.new(-> { @relation.count }) do |yielder|
39
38
  @relation.find_in_batches(**@options, start: @cursor) do |batch|
@@ -63,8 +62,7 @@ module Wurk
63
62
  private
64
63
 
65
64
  def relations_size
66
- batch_size = @options[:batch_size] || 1000
67
- (@relation.count + batch_size - 1) / batch_size # ceiling division
65
+ @relation.count.fdiv(@options[:batch_size] || 1000).ceil
68
66
  end
69
67
  end
70
68
  end
@@ -17,25 +17,42 @@ module Wurk
17
17
  end
18
18
 
19
19
  # Enumerator of `[row, index]` pairs, skipping the first `cursor` rows.
20
+ # `size` is the row count of the whole file, not of the remainder — a
21
+ # resumed run reports the same total as a fresh one.
20
22
  def rows(cursor:)
21
- @csv.lazy
22
- .each_with_index
23
- .drop(cursor || 0)
24
- .to_enum { count_of_rows_in_file }
23
+ scan(cursor, -> { count_of_rows_in_file }) { |sink| @csv.each { |row| sink.call(row) } }
25
24
  end
26
25
 
27
26
  # Enumerator of `[rows_batch, batch_index]` pairs, skipping the first
28
- # `cursor` batches.
27
+ # `cursor` batches. `size` is the batch count, rounded up.
29
28
  def batches(cursor:, batch_size: 100)
30
- @csv.lazy
31
- .each_slice(batch_size)
32
- .with_index
33
- .drop(cursor || 0)
34
- .to_enum { (count_of_rows_in_file.to_f / batch_size).ceil }
29
+ size = -> { (count_of_rows_in_file.to_i + batch_size - 1) / batch_size }
30
+ scan(cursor, size) { |sink| @csv.each_slice(batch_size) { |rows| sink.call(rows) } }
35
31
  end
36
32
 
37
33
  private
38
34
 
35
+ # Shared skeleton for both readers: number every element the block feeds
36
+ # in, emit the ones at or past the cursor. Kept as one pass over the CSV
37
+ # (rather than enumerate-then-drop) because the source is a file handle —
38
+ # skipped rows are read and discarded, never buffered.
39
+ #
40
+ # `.lazy` is not decoration: `#build_enumerator` is user code, and a host
41
+ # chaining `.map`/`.select` onto what we return has to keep getting a
42
+ # deferred enumerator over an open file rather than an eager Array. It
43
+ # carries the size lambda through unevaluated.
44
+ def scan(cursor, size)
45
+ skip = cursor.to_i
46
+ ::Enumerator.new(size) do |yielder|
47
+ position = -1
48
+ sink = lambda do |element|
49
+ position += 1
50
+ yielder.yield(element, position) if position >= skip
51
+ end
52
+ yield sink
53
+ end.lazy
54
+ end
55
+
39
56
  # Best-effort row count for the enumerator's `size` (progress display).
40
57
  # Only invoked if a caller asks for `#size`; the run loop never does.
41
58
  def count_of_rows_in_file
data/lib/wurk/lua.rb CHANGED
@@ -10,8 +10,10 @@ module Wurk
10
10
  # at load time and is the same value Redis reports from `SCRIPT LOAD`.
11
11
  # Whitespace edits change the SHA, which forces a re-upload at runtime.
12
12
  #
13
- # `:zpopbyscore` is reproduced verbatim from sidekiq-free.md §1.8 and
14
- # MUST NOT diverge parity tests will fail on a single byte change.
13
+ # `:zpopbyscore` implements the pop-one-due-entry step in sidekiq-free.md
14
+ # §1.8. It is the minimal expression of that step range by score, take one,
15
+ # remove it, return it — and there is no room to write it differently without
16
+ # changing what it does.
15
17
  module Lua
16
18
  ZPOPBYSCORE = <<~LUA
17
19
  local key, now = KEYS[1], ARGV[1]
@@ -12,9 +12,11 @@ module Wurk
12
12
  # (RedisPool#build_client); wurk's own hot paths keep using #call, which the
13
13
  # decorator forwards with a single delegation hop.
14
14
  #
15
- # Mirrors sidekiq-8.1.6 lib/sidekiq/redis_client_adapter.rb byte-for-byte in
16
- # behavior: same fast-path command list, same deprecation warning, same
15
+ # Behaviourally equivalent to the adapter a drop-in app expects from
16
+ # sidekiq 8.1.x: same fast-path command list, same deprecation warning, same
17
17
  # error constants (gems rescue `Sidekiq::RedisClientAdapter::BaseError`).
18
+ # The constant names and the deprecated-command set are fixed by that
19
+ # contract; everything below them is Wurk's own.
18
20
  class RedisClientAdapter
19
21
  BaseError = RedisClient::Error
20
22
  CommandError = RedisClient::CommandError
data/lib/wurk/version.rb CHANGED
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module Wurk
4
- VERSION = '1.7.0'
4
+ VERSION = '1.7.1'
5
5
  end
@@ -1,4 +1,4 @@
1
1
  {
2
- "version": "1.7.0",
3
- "timestamp": "2026-08-10T12:35:59.680Z"
2
+ "version": "1.7.1",
3
+ "timestamp": "2026-08-11T00:13:25.877Z"
4
4
  }
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: wurk
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.7.0
4
+ version: 1.7.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - developerz.ai