wurk 1.6.0 → 1.7.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cbdba1bc48d5cc6029c22b78376e6b6a0b93e4f519f6c717fd3f2ce84b5900d2
-  data.tar.gz: 0d0c5e36c75c898fa05d1b448e4730391cbcf883d39402db49ac879d3168aa91
+  metadata.gz: 60f518fc753861a0670a0922cff88670ed430890c8d7bad70bf2e9ff4bcd173e
+  data.tar.gz: c9ed06c696ba18a30658ccd30cdbc29217bed5b0a66e02af13ae831021a210a6
 SHA512:
-  metadata.gz: 29c1cb70a235bffed95cfa399328be393c4f653df388eff3f06a36e605f7f941ec21f1dc21b6fa832876f09635991c69087ff18dec9dc3a08cedb50f85945338
-  data.tar.gz: ca4aa23ed8658e6fc98cbe14501a1789cc38cc43a6a539c6a41fd3aebdcba97bb91fef58c84ae26f0d36cc42518b8279b348336efa93a3b4f0aaf37bf886c625
+  metadata.gz: fa05f9a8433f9a5b475b7e4d075e8fe209aebffc6e286ac57f28c9dcbf3efdc18a36d4393fcc718e663df3b59dd508bf0effb709e59752a127ab8f58d409728b
+  data.tar.gz: 3a8fab767e83ae14c185a054be1d356a7161b9ded7771cafc4384db1e5f14292169dce1aafe5038113586379838a0053b4d4b67f551823e6bc2268a6cf3c1c75

data/README.md

@@ -21,7 +21,11 @@
 
 Wurk is wire-compatible with Sidekiq — same Redis keys, same job JSON, same Ruby DSL. Swap one line in your `Gemfile` and your existing jobs, batches, limiters, cron entries, and live Redis data keep working untouched. The Pro and Enterprise feature sets ship in the same free gem, with no license check and no tiers.
 
-**On Sidekiq:** Sidekiq is the reason Ruby background processing works as well as it does. Mike Perham and Contributed Systems have maintained it for well over a decade and funded that work through Pro and Enterprise — a model that kept a critical piece of the ecosystem healthy, documented, and supported, and one the whole community has benefited from. Wurk implements Sidekiq's API because it is a genuinely good API, and exists to make a different bet about what maintenance costs now — see [Why Wurk exists](#why-wurk-exists).
+**In production:** Wurk runs the background work at [developerz.ai](https://developerz.ai) and at partner deployments — millions of jobs an hour, across many servers, on the fork-based swarm described below. It is not a preview.
+
+**At scale:** Wurk is built for fleets, not just for one box. Kubernetes `/live` + `/ready` probes are a config line, not a sidecar; a bearer-scoped [HTTP API](docs/api-http.md) lets non-Ruby services enqueue and inspect; [OpenTelemetry](docs/telemetry.md) traces propagate client → server; per-queue [global concurrency caps](docs/rate-limiting.md) hold cluster-wide limits; and monitoring is the dashboard you already mount — live SSE, charts, per-job progress, no separate stack to run. See [Wurk extras](#wurk-extras).
+
+**On Sidekiq:** Wurk implements Sidekiq's API because it is a genuinely good API. Sidekiq is human-maintained and funds that work through its paid tiers; Wurk is AI-maintained, which is what lets the same surface be free software. Wurk is independent and not affiliated with or endorsed by Sidekiq or its maintainers — see [Why Wurk exists](#why-wurk-exists).
 
 **On speed:** Wurk is not currently faster than stock Sidekiq — it runs at roughly 0.87×–1.02× depending on workload shape, with parity on CPU and I/O but still behind on framework overhead (noop) and boot time. Numbers, method, and the reproduction command are in [docs/benchmarks.md](docs/benchmarks.md); run them yourself with `rake bench:vs_sidekiq`.
 
@@ -44,18 +48,44 @@ gem "wurk"
 
 ## Feature matrix
 
-Everything below is in the one free gem. The "Sidekiq tier" column maps each area onto Sidekiq's own lineup, so you can see at a glance what a migration covers.
-
-| Area | What you get | Sidekiq tier |
-|---|---|---|
-| **Runtime** | Fork-based real parallelism, reliable `BLMOVE` fetch, PID supervision, rolling restarts, graceful drain, scheduled/retry pollers | OSS + Pro |
-| **Batches** | `Sidekiq::Batch` with `on(:success/:complete/:death)` callbacks, nested batches, progress | Pro |
-| **Limiters** | Concurrent, bucket, window, leaky, and points rate limiters via `Sidekiq::Limiter` | Enterprise |
-| **Periodic** | Cron/periodic jobs, leader-elected so each tick fires exactly once across the cluster | Enterprise |
-| **Encryption** | Transparent AES-256-GCM job-argument encryption with zero-downtime key rotation | Enterprise |
-| **Dashboard** | Mountable Rails engine, precompiled SolidJS SPA (no Node needed), live SSE, charts, host-app auth hook | OSS + Pro/Ent |
-
-Plus the [Wurk extras](#wurk-extras) below: a worker topology DSL, a Kubernetes liveness/readiness listener, OpenTelemetry tracing, job status/progress/results, an HTTP producer+observe API, per-job timeouts/deadlines, debounce/throttle/collapse, global per-queue concurrency caps, and DAG flows.
+Every capability Sidekiq splits across three tiers is in the one free gem. Columns are Sidekiq's own lineup, so you can see exactly what a migration covers.
+
+| Capability | OSS | Pro | Ent | **Wurk** |
+|---|:---:|:---:|:---:|:---:|
+| Threaded workers, middleware, retries with backoff, dead set | ✅ | ✅ | ✅ | **✅** |
+| Scheduled jobs (`perform_in` / `perform_at`), Active Job adapter | ✅ | ✅ | ✅ | **✅** |
+| Web dashboard, Data API, testing modes | ✅ | ✅ | ✅ | **✅** |
+| Reliable fetch — atomic `BLMOVE`, survives `SIGKILL` | — | ✅ | ✅ | **✅** |
+| Batches: `on(:success/:complete/:death)`, nesting, progress | — | ✅ | ✅ | **✅** |
+| Reliable scheduler · reliable client (Redis-outage buffering) | — | ✅ | ✅ | **✅** |
+| Queue pause/resume · job expiration (`expires_in`) | — | ✅ | ✅ | **✅** |
+| StatsD / DogStatsD metrics export | — | ✅ | ✅ | **✅** |
+| Rate limiting — concurrent, bucket, window, leaky, points | — | — | ✅ | **✅** |
+| Periodic (cron) jobs, leader-elected so each tick fires once | — | — | ✅ | **✅** |
+| Unique jobs, with custom lock context | — | — | ✅ | **✅** |
+| Encryption — AES-256-GCM args, zero-downtime key rotation | — | — | ✅ | **✅** |
+| Historical metrics retained in Redis | — | — | ✅ | **✅** |
+| Multi-process fork parallelism (`swarm`) + rolling restarts | — | — | ✅ | **✅** |
+| **Licence** | LGPL-3.0 | commercial | commercial | **MIT** |
+
+### Beyond Sidekiq
+
+Same table, other direction — these have no Sidekiq equivalent at any tier. All opt-in, and free on the job path until you turn them on.
+
+| Capability | OSS | Pro | Ent | **Wurk** |
+|---|:---:|:---:|:---:|:---:|
+| [Kubernetes `/live` + `/ready` probe listener](#kubernetes-metrics--tracing) | — | — | — | **✅** |
+| [OpenTelemetry tracing](docs/telemetry.md) — W3C context, client → server | — | — | — | **✅** |
+| [HTTP producer + observe API](docs/api-http.md) — enqueue/inspect over JSON | — | — | — | **✅** |
+| [Job status, progress & results](docs/job-status.md) | — | — | — | **✅** |
+| [Flows — DAG on batches](docs/flows.md) with piped results | — | — | — | **✅** |
+| [Global per-queue concurrency caps](docs/rate-limiting.md) (cluster-wide) | — | — | — | **✅** |
+| [Debounce, throttle-to-slot & collapse](docs/unique-jobs.md) | — | — | — | **✅** |
+| [Per-job timeouts & deadlines](docs/retries.md) | — | — | — | **✅** |
+| Worker topology DSL — fleet roles in code, not `-q` flags | — | — | — | **✅** |
+| Dashboard theme, locale & 400-zone timezone picker | — | — | — | **✅** |
+
+Details, and what you give up if you migrate back, in [Wurk extras](#wurk-extras).
 
 ## Wurk extras
 
@@ -71,11 +101,31 @@ Sidekiq has no equivalent for any of these — they aren't parity, they're new s
 | **[Per-job timeouts & deadlines](docs/retries.md)** | `timeout:` bounds one attempt, `deadline:` bounds the whole job from enqueue, enforced by a lightweight per-capsule watchdog thread (no thread-per-job) | Runaway/stuck jobs run unbounded except for `shutdown_timeout` |
 | **[Global per-queue concurrency caps](docs/rate-limiting.md)** | `config.global_concurrency = { critical: 20 }` caps in-flight jobs for a queue across the whole cluster, folded into the fetch pipeline | The cluster-wide cap; only per-key `Limiter`s remain |
 | **Worker topology DSL** | Declare which queues/classes a given fleet role runs, in code instead of ad hoc `-q` flags | The declarative topology; fall back to CLI queue flags |
-| **[Kubernetes probes](#kubernetes-probes)** | `config.health_check` opens a thin `/live`/`/ready` HTTP listener, self-electing across a swarm's children | The built-in probe listener; roll your own liveness check |
+| **[Kubernetes probes](#kubernetes-metrics--tracing)** | `config.health_check` opens a thin `/live`/`/ready` HTTP listener, self-electing across a swarm's children | The built-in probe listener; roll your own liveness check |
 | **Dashboard theme, locale & timezone** | Light/dark/system theme, per-visitor locale override, and a 400-zone timezone picker for every timestamp in the SPA | Nothing server-side — this is dashboard-only |
 
 AI dashboard panes — anomaly detection, natural-language queries, error triage, and capacity forecasting — are **planned, not shipped**: they're [roadmap M5](docs/idea/13-roadmap.md#m5--ai-dashboard), after the M4.5 extras above.
 
+## Benchmarks
+
+**Wurk is not faster than stock Sidekiq today.** Here is where it actually stands, measured rather than claimed — wurk 1.5.0 vs sidekiq 8.1.6, ruby 3.4.7, local Redis 7.4.10, 5000 jobs/run, 12 runs per topology, paired per-run ratios.
+
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="docs/assets/bench-throughput-dark.svg">
+  <img alt="Throughput relative to stock Sidekiq. Median of 12 paired runs with min–max range. 1 process × 5 threads: noop 0.87×, cpu 0.99×, io 0.99×. 4 processes × 5 threads: noop 0.95×, cpu 1.02×, io 0.97×." src="docs/assets/bench-throughput-light.svg" width="100%">
+</picture>
+
+Parity on `cpu` and `io`; still behind on `noop`, which is pure framework overhead. The spread is wide because the host carried background load — the paired-ratio median is the number to trust, not any single run.
+
+<picture>
+  <source media="(prefers-color-scheme: dark)" srcset="docs/assets/bench-boot-dark.svg">
+  <img alt="Boot to first job, median seconds. 1p × 5t: Sidekiq 0.56s, Wurk 0.72s. 4p × 5t: Sidekiq 0.60s, Wurk 0.78s." src="docs/assets/bench-boot-light.svg" width="100%">
+</picture>
+
+Forking is not what closes the throughput gap — a stock Sidekiq user reaches multi-core by running N processes, which is the second topology above. The swarm buys copy-on-write memory and one supervisor, not raw speed.
+
+Method, per-invocation records, workload definitions, and the separate `rake bench` regression gate (wurk vs its own past self, which says nothing about Sidekiq): **[docs/benchmarks.md](docs/benchmarks.md)**. Reproduce with `bin/rake bench:vs_sidekiq`.
+
 ## Documentation
 
 - **[Website](https://developerz-ai.github.io/wurk/)** · **[Wiki / full docs](https://github.com/developerz-ai/wurk/wiki)** — the pitch, install, and the complete guide.
@@ -99,7 +149,7 @@ AI dashboard panes — anomaly detection, natural-language queries, error triage
 - **API reference (parity specs):** [Sidekiq OSS](https://github.com/developerz-ai/wurk/blob/main/docs/target/sidekiq-free.md) · [Pro](https://github.com/developerz-ai/wurk/blob/main/docs/target/sidekiq-pro.md) · [Enterprise](https://github.com/developerz-ai/wurk/blob/main/docs/target/sidekiq-ent.md) — the authoritative surface Wurk matches exactly.
 - **[Authentication & authorization](https://github.com/developerz-ai/wurk/blob/main/docs/authentication.md)** — gate the dashboard behind Devise/Warden, Sorcery, Basic auth, or a token; role-based read/write; CSRF.
 - **[Securing the dashboard](https://github.com/developerz-ai/wurk/blob/main/docs/dashboard.md)** · **[Metrics history](https://github.com/developerz-ai/wurk/blob/main/docs/metrics-history.md)**
-- **[Compatibility & legal basis](https://github.com/developerz-ai/wurk/blob/main/docs/clean-room.md)** — clean-room implementation: Wurk copies the API, not the code (Google v. Oracle).
+- **[Compatibility & legal basis](https://github.com/developerz-ai/wurk/blob/main/docs/compatibility.md)** — independent reimplementation: Wurk reproduces the API and wire format, not Sidekiq's implementation.
 - **Live demo:** [wurk.demo.developerz.ai](https://wurk.demo.developerz.ai)
 
 ## Requirements
@@ -186,9 +236,11 @@ end
 
 Keys rotate without downtime — keep every still-in-flight version resolvable so old jobs decrypt, then bump `active_version`. A job that can't be decrypted (key rotated away, corrupt ciphertext) goes **straight to the dead set in under a second** rather than crash-looping through 25 retries, with the still-encrypted payload preserved for replay. The dashboard renders encrypted args as `"<encrypted>"`; cleartext is never written to Redis.
 
-## Kubernetes probes
+## Kubernetes, metrics & tracing
+
+Wurk is built to run as a fleet: one supervisor per pod forking N children across the cores you gave it, drained gracefully on `SIGTERM`, replaced one slot at a time on `SIGUSR1`, and answerable to your existing monitoring rather than a bespoke one.
 
-Opt in to a thin HTTP listener for liveness/readiness:
+**Probes.** Opt in to a thin HTTP listener for liveness/readiness:
 
 ```ruby
 Wurk.configure_server do |config|
@@ -201,22 +253,34 @@ end
 | `/live` | 200 while the Launcher is running; 503 once `stop`/`quiet` is called. |
 | `/ready` | 200 only when Redis is reachable **and** the heartbeat fired within `ready_window` (default 30s); 503 otherwise. |
 
-Knobs: `health_check(port:, bind: "0.0.0.0", ready_window: 30)`. In swarm mode one child owns the port; the others poll every 5s and take it over if the owner dies, so probes survive a child restart.
+Knobs: `health_check(port:, bind: "0.0.0.0", ready_window: 30)`. In swarm mode one child owns the port; the others poll every 5s and take it over if the owner dies, so probes survive a child restart — a pod never fails a probe just because a worker recycled.
+
+**Getting the numbers out.** Point these at whatever you already run:
+
+| Signal | How it leaves the process | Docs |
+|---|---|---|
+| Job metrics (counts, latency, per-class timing) | StatsD / DogStatsD via `config.dogstatsd` — into Datadog directly, or into Grafana through your StatsD exporter | [metrics](docs/metrics.md) |
+| Historical time series | Retained in Redis, queried by the dashboard or `Wurk::History` | [metrics-history](docs/metrics-history.md) |
+| Distributed traces | OpenTelemetry — W3C `traceparent` propagated client → server, one span per attempt | [telemetry](docs/telemetry.md) |
+| Queue/job/swarm state for external scrapers and autoscalers | Bearer-token `/v1` JSON API, mountable standalone or via `wurk api` | [api-http](docs/api-http.md) |
+| Errors | Built-in Sentry reporting, terminal failures only, no job args | [sentry](docs/sentry.md) |
+
+There is no native Prometheus `/metrics` endpoint — the StatsD export or the `/v1` API is the current path into a Prometheus/Grafana stack.
+
+**Backpressure at fleet scale.** `config.global_concurrency = { critical: 20 }` caps in-flight jobs for a queue across every pod, folded into the fetch pipeline rather than bolted on as a middleware sleep — see [rate limiting](docs/rate-limiting.md).
 
 ## Why Wurk exists
 
-Sidekiq's split into OSS, Pro, and Enterprise is how a decade of serious maintenance got funded, and it worked. Ruby got a background-job library that stayed maintained, documented, and answerable to its users for longer than most infrastructure gems survive at all — and the API in this README is the one that came out of it. Wurk is standing on that work.
+Infrastructure this basic should be free software. A Rails app shouldn't need a licence key to get reliable fetch, batches, rate limiting, or cron — those are table stakes, not a premium tier, and the free-software tradition is that the best tools belong to everyone who runs them.
 
-What Wurk bets on is that the economics underneath changed. Wurk is built and maintained **AI-first** — implementation, parity suite, docs, and benchmarks are written and kept current by AI agents working under human review. That is what makes it practical to:
+What has made that hard is maintenance: someone has to be paid to do it. Sidekiq funds a decade of *human* maintenance through its paid tiers, which is an honest trade. Wurk makes a different one — it is maintained **AI-first**: implementation, parity suite, docs, and benchmarks are written and kept current by AI agents under human review. A fix, a doc update, or a version bump is no longer somebody's week, which is what makes it practical to:
 
 - ship the entire Pro + Enterprise surface with no tier, no flag gate, and no license check;
-- keep parity honest mechanically rather than by hand — Sidekiq's own tests run as an oracle suite, and third-party gems (sidekiq-cron, sidekiq-unique-jobs, sidekiq-scheduler, sidekiq-status, sidekiq-failures, sidekiq-throttled) run their upstream suites against Wurk on every push;
+- keep parity honest mechanically rather than by hand — an independently written parity oracle suite, pinned to a documented Sidekiq revision, plus third-party gems (sidekiq-cron, sidekiq-unique-jobs, sidekiq-scheduler, sidekiq-status, sidekiq-failures, sidekiq-throttled) running their own upstream suites against Wurk on every push;
 - keep adding surface Sidekiq doesn't have — the [Wurk extras](#wurk-extras) above landed as one release;
-- sustain that over the long run, because the marginal cost of a fix, a doc update, or a version bump is no longer somebody's week.
-
-It also means Wurk holds itself to published numbers instead of adjectives: the benchmark suite runs against stock Sidekiq every release and the results ship [as measured](docs/benchmarks.md), including the unflattering ones.
+- hold ourselves to published numbers instead of adjectives — the suite runs against stock Sidekiq every release and ships the results [as measured](docs/benchmarks.md), including the unflattering ones.
 
-None of this makes Wurk the right call for everyone. Sidekiq Pro and Enterprise come with a commercial support contract, a decade of production track record, and a human on the other end of an email. If that is what your risk profile needs, buy it — it is worth the money, and it is the reason the API Wurk implements exists in the first place.
+Wurk is MIT and stays that way. If what you need is a commercial support contract and a human on the other end of an email, buying that is a perfectly good answer.
 
 ## Migrating from Sidekiq
 
@@ -239,9 +303,11 @@ Issues and pull requests are welcome — see **[CONTRIBUTING.md](https://github.
 
 MIT. See [LICENSE](https://github.com/developerz-ai/wurk/blob/main/LICENSE).
 
-Wurk is a clean-room reimplementation of the Sidekiq **API** — it copies the
-interface (so your jobs run unchanged), not Sidekiq's implementation code. This
-is the same basis the Supreme Court upheld for Google's reuse of the Java API in
-*Google v. Oracle* (2021). "Sidekiq" is a trademark of Contributed Systems, LLC;
-Wurk is independent and not affiliated with or endorsed by them. Full reasoning:
-**[docs/clean-room.md](https://github.com/developerz-ai/wurk/blob/main/docs/clean-room.md)**.
+Wurk is an independent reimplementation of the Sidekiq **API** — it reproduces
+the interface and wire format (so your jobs run unchanged), not Sidekiq's
+implementation. Reusing an API for interoperability is what the Supreme Court
+held to be fair use in *Google v. Oracle* (2021). Sidekiq itself is LGPL-3.0;
+Wurk neither vendors nor links against it. "Sidekiq" is a trademark of
+Contributed Systems, LLC; Wurk is independent and not affiliated with or
+endorsed by them. Full reasoning:
+**[docs/compatibility.md](https://github.com/developerz-ai/wurk/blob/main/docs/compatibility.md)**.

data/app/controllers/wurk/dashboard_controller.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require 'erb'
 require 'json'
 require 'net/http'
 require 'uri'

data/lib/wurk/cli.rb

@@ -1,9 +1,11 @@
 # frozen_string_literal: true
 
-require 'yaml'
-require 'optparse'
-require 'erb'
-
+# yaml, optparse and erb are required inside the two methods that use them
+# (#parse_config and #option_parser) rather than here. Together they cost ~45ms
+# of the ~290ms `require "wurk"` spends, and only `exe/wurk` ever reaches them —
+# a Rails app that enqueues and processes jobs pays that on every boot for a
+# command-line parser it never runs. `require` is idempotent, so the lazy call
+# is a $LOADED_FEATURES hash lookup on every invocation after the first.
 require_relative 'version'
 require_relative 'configuration'
 require_relative 'component'
@@ -410,6 +412,9 @@ module Wurk
     def parse_config(path)
       raise ArgumentError, "No such file #{path}" unless ::File.exist?(path)
 
+      require 'erb'
+      require 'yaml'
+
       erb = ::ERB.new(::File.read(path), trim_mode: '-')
       erb.filename = ::File.expand_path(path)
       opts = ::YAML.safe_load(erb.result, permitted_classes: [Symbol], aliases: true) || {}
@@ -444,6 +449,8 @@ module Wurk
     end
 
     def option_parser(opts)
+      require 'optparse'
+
       ::OptionParser.new do |o|
         o.banner = @command ? "wurk #{@command} [options]" : 'wurk [options]'
         define_value_flags(o, opts, OPTION_FLAGS)

data/lib/wurk/command_builder.rb

@@ -0,0 +1,49 @@
+# frozen_string_literal: true
+
+require 'redis-client'
+
+module Wurk
+  # Fast path for redis-client's command normalization.
+  #
+  # `RedisClient::CommandBuilder.generate` splices Hash arguments with a
+  # `flat_map` and then stringifies Symbols/Integers/Floats with a `map!` —
+  # two array allocations and two per-element type dispatches for a command
+  # whose arguments are already Strings. Every command Wurk sends per job is
+  # exactly that shape (LMOVE, LREM, DEL, LPUSH, SADD), and the fetch pipeline
+  # sends three of them. Measured ~4x faster on those commands (2.2µs → 0.6µs
+  # each) with `bench/command_builder.rb`.
+  #
+  # Anything else — a Hash to splice, a Symbol or a number to stringify,
+  # keyword arguments, an empty command — falls straight through to
+  # redis-client's own builder. This is a shortcut, never a second
+  # implementation of the semantics, so a host app calling
+  # `Sidekiq.redis { |c| c.call("HSET", key, hash) }` is unaffected.
+  #
+  # The array is duped rather than handed back: `call_v` and the pipelined
+  # forms pass the caller's own array through, and redis-client's builder
+  # always returns a fresh one that middleware is free to mutate in place.
+  module CommandBuilder
+    module_function
+
+    def generate(args, kwargs = nil)
+      return args.dup if fast?(args, kwargs)
+
+      ::RedisClient::CommandBuilder.generate(args, kwargs)
+    end
+
+    # Split out, and public, because the two branches produce byte-identical
+    # output by construction — which leaves allocation count as the only
+    # runtime difference between "the shortcut fired" and "it silently didn't",
+    # and that is not something a test can assert safely in a process running
+    # other suites. So the decision itself is what the suite asserts.
+    #
+    # `kwargs` is nil only from `call_v` and the scan helpers. Every `call`,
+    # `blocking_call` and pipelined `call` arrives with `**kwargs` already
+    # splatted into a Hash — EMPTY when the caller passed no keywords — so a
+    # guard that only tested `.nil?` sent the entire hot path down the slow
+    # branch while every direct-call test still passed.
+    def fast?(args, kwargs)
+      (kwargs.nil? || kwargs.empty?) && !args.empty? && args.all?(String)
+    end
+  end
+end

data/lib/wurk/component.rb

@@ -2,6 +2,7 @@
 
 require 'socket'
 require 'securerandom'
+require_relative 'pid_cache'
 
 module Wurk
   # Shared mixin for runtime components (Launcher, Manager, Processor, Fetcher,
@@ -55,7 +56,7 @@ module Wurk
     def self.tid
       thread = Thread.current
       memo = thread.thread_variable_get(:wurk_tid)
-      pid = ::Process.pid
+      pid = PidCache.pid
       return memo[1] if memo && memo[0] == pid
 
       id = (thread.object_id ^ pid).to_s(36).freeze

data/lib/wurk/fetcher/reliable.rb

@@ -2,6 +2,7 @@
 
 require 'socket'
 require_relative '../component'
+require_relative '../pid_cache'
 require_relative '../keys'
 require_relative '../lua'
 require_relative '../fetcher'
@@ -366,7 +367,7 @@ module Wurk
       # fetcher and read this without a lock. A racing writer's entry can be
       # lost, and the next fetch rebuilds it.
       def queue_keys(public_q)
-        pid = ::Process.pid
+        pid = PidCache.pid
         if @queue_keys_pid != pid
           @queue_keys = {}
           @queue_keys_pid = pid

data/lib/wurk/fetcher/unit_of_work.rb

@@ -32,6 +32,13 @@ module Wurk
       # admitted under, and are set only by Fetcher::Capped — nil is the whole
       # uncapped path, which is why the release below is a field test rather
       # than a question asked of Redis.
+      # LREM's count, pre-stringified. Every argument of every command on the
+      # ACK pipeline is then a String, which is what keeps it on
+      # {Wurk::CommandBuilder}'s allocation-free fast path; an Integer here
+      # would send the whole command back through redis-client's normalizer.
+      LREM_COUNT = '1'
+      private_constant :LREM_COUNT
+
       UnitOfWork = Struct.new(:queue, :queue_name, :private_queue, :job, :config, :jid, :fetcher,
                               :slot_key, :slot_token, keyword_init: true) do
         # Deferred, never skipped: the LREM goes back to the fetcher, which
@@ -55,7 +62,7 @@ module Wurk
         # wire-frozen, so the reaper cannot flag it). Reading the counter to
         # decide would spend the very round trip the DEL is riding for free.
         def write_ack(pipe)
-          pipe.call('LREM', private_queue, 1, job)
+          pipe.call('LREM', private_queue, LREM_COUNT, job)
           job_jid = jid.to_s
           Middleware::PoisonPill.clear_in(pipe, job_jid) unless job_jid.empty?
           release_slot_in(pipe) if slot_key

data/lib/wurk/iterable_job/active_record_enumerator.rb

@@ -30,10 +30,9 @@ module Wurk
       end
 
       # `[records_batch, batch.first.id]` pairs. The size lambda is the record
-      # count, NOT the batch count — byte-for-byte with upstream Sidekiq's
-      # `ActiveRecordEnumerator#batches`, so `enum.size` returns the same value
-      # a drop-in app gets from Sidekiq. (Only the lazy `#size` differs from
-      # `relations`; the run loop never calls it.)
+      # count, NOT the batch count — that is what a drop-in app already gets
+      # from `enum.size`, so it is preserved deliberately. (Only the lazy
+      # `#size` differs from `relations`; the run loop never calls it.)
       def batches
         ::Enumerator.new(-> { @relation.count }) do |yielder|
           @relation.find_in_batches(**@options, start: @cursor) do |batch|
@@ -63,8 +62,7 @@ module Wurk
       private
 
       def relations_size
-        batch_size = @options[:batch_size] || 1000
-        (@relation.count + batch_size - 1) / batch_size # ceiling division
+        @relation.count.fdiv(@options[:batch_size] || 1000).ceil
       end
     end
   end

data/lib/wurk/iterable_job/csv_enumerator.rb

@@ -17,25 +17,42 @@ module Wurk
       end
 
       # Enumerator of `[row, index]` pairs, skipping the first `cursor` rows.
+      # `size` is the row count of the whole file, not of the remainder — a
+      # resumed run reports the same total as a fresh one.
       def rows(cursor:)
-        @csv.lazy
-            .each_with_index
-            .drop(cursor || 0)
-            .to_enum { count_of_rows_in_file }
+        scan(cursor, -> { count_of_rows_in_file }) { |sink| @csv.each { |row| sink.call(row) } }
       end
 
       # Enumerator of `[rows_batch, batch_index]` pairs, skipping the first
-      # `cursor` batches.
+      # `cursor` batches. `size` is the batch count, rounded up.
       def batches(cursor:, batch_size: 100)
-        @csv.lazy
-            .each_slice(batch_size)
-            .with_index
-            .drop(cursor || 0)
-            .to_enum { (count_of_rows_in_file.to_f / batch_size).ceil }
+        size = -> { (count_of_rows_in_file.to_i + batch_size - 1) / batch_size }
+        scan(cursor, size) { |sink| @csv.each_slice(batch_size) { |rows| sink.call(rows) } }
       end
 
       private
 
+      # Shared skeleton for both readers: number every element the block feeds
+      # in, emit the ones at or past the cursor. Kept as one pass over the CSV
+      # (rather than enumerate-then-drop) because the source is a file handle —
+      # skipped rows are read and discarded, never buffered.
+      #
+      # `.lazy` is not decoration: `#build_enumerator` is user code, and a host
+      # chaining `.map`/`.select` onto what we return has to keep getting a
+      # deferred enumerator over an open file rather than an eager Array. It
+      # carries the size lambda through unevaluated.
+      def scan(cursor, size)
+        skip = cursor.to_i
+        ::Enumerator.new(size) do |yielder|
+          position = -1
+          sink = lambda do |element|
+            position += 1
+            yielder.yield(element, position) if position >= skip
+          end
+          yield sink
+        end.lazy
+      end
+
       # Best-effort row count for the enumerator's `size` (progress display).
       # Only invoked if a caller asks for `#size`; the run loop never does.
       def count_of_rows_in_file

data/lib/wurk/job_logger.rb

@@ -20,6 +20,10 @@ module Wurk
     def initialize(config)
       @config = config
       @logger = @config.logger
+      # Read once: `config[]` is a Hash lookup through a delegating Capsule, and
+      # #context_hash runs it per job only to walk a list that cannot change
+      # after boot (Configuration freezes at launch).
+      @logged_attributes = Array(@config[:logged_job_attributes]).map { |attr| [attr, attr.to_sym] }.freeze
       @skip = !!@config[:skip_default_job_logging]
     end
 
@@ -68,8 +72,8 @@ module Wurk
         class: job_hash['wrapped'] || job_hash['class']
       }
 
-      @config[:logged_job_attributes].each do |attr|
-        h[attr.to_sym] = job_hash[attr] if job_hash.key?(attr)
+      @logged_attributes.each do |attr, sym|
+        h[sym] = job_hash[attr] if job_hash.key?(attr)
       end
       h
     end

data/lib/wurk/lua.rb

@@ -10,8 +10,10 @@ module Wurk
   # at load time and is the same value Redis reports from `SCRIPT LOAD`.
   # Whitespace edits change the SHA, which forces a re-upload at runtime.
   #
-  # `:zpopbyscore` is reproduced verbatim from sidekiq-free.md §1.8 and
-  # MUST NOT diverge — parity tests will fail on a single byte change.
+  # `:zpopbyscore` implements the pop-one-due-entry step in sidekiq-free.md
+  # §1.8. It is the minimal expression of that step — range by score, take one,
+  # remove it, return it — and there is no room to write it differently without
+  # changing what it does.
   module Lua
     ZPOPBYSCORE = <<~LUA
       local key, now = KEYS[1], ARGV[1]

data/lib/wurk/pid_cache.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+module Wurk
+  # `Process.pid` is a real syscall on Linux — glibc dropped its getpid cache in
+  # 2.25 — and the fetch+execute path reads it twice per job ({Component.tid}
+  # and the fetcher's per-queue key cache, both of which read it only to notice
+  # a fork). Measured at ~640ns a call against ~110ns for this memo, which is
+  # about 1% of a job's CPU spent asking the kernel a question whose answer
+  # changes at most once in a process's life.
+  #
+  # Refreshed in the child half of every fork, so a swarm child never answers
+  # with the pid it inherited — the whole reason those two call sites read it.
+  #
+  # Same shape as redis-client's PIDCache, deliberately not borrowed from it:
+  # that is a private constant of a dependency, and this one is on our hot path.
+  module PidCache
+    if ::Process.respond_to?(:fork)
+      class << self
+        attr_reader :pid
+
+        def update!
+          @pid = ::Process.pid
+        end
+      end
+      update!
+
+      # Prepended, not aliased: `Process._fork` is a documented hook (Ruby 3.1+,
+      # and Wurk requires 3.2) that several gems wrap, and prepending composes
+      # with theirs instead of replacing it.
+      module CoreExt
+        def _fork
+          child_pid = super
+          PidCache.update! if child_pid.zero?
+          child_pid
+        end
+      end
+      ::Process.singleton_class.prepend(CoreExt)
+    else # JRuby / TruffleRuby / Windows — no fork, so nothing can invalidate it
+      @pid = ::Process.pid
+      singleton_class.attr_reader(:pid)
+    end
+  end
+end

data/lib/wurk/profiler.rb

@@ -3,7 +3,6 @@
 require 'securerandom'
 require 'zlib'
 require 'stringio'
-require 'tempfile'
 require_relative 'keys'
 require_relative 'pool_checkout'
 
@@ -109,7 +108,12 @@ module Wurk
         Wurk.configuration.handle_exception(e, context: 'Wurk::Profiler')
       end
 
+      # `tempfile` (and the `tmpdir` it drags in) is ~19ms of `require "wurk"`,
+      # spent only by an install that has vernier loaded AND profiling switched
+      # on for a job. Everyone else was paying it at boot.
       def profile_to_json(&)
+        require 'tempfile'
+
         Tempfile.create(['wurk-profile', '.json']) do |file|
           ::Vernier.profile(out: file.path, &)
           File.read(file.path)

data/lib/wurk/redis_client_adapter.rb

@@ -12,9 +12,11 @@ module Wurk
   # (RedisPool#build_client); wurk's own hot paths keep using #call, which the
   # decorator forwards with a single delegation hop.
   #
-  # Mirrors sidekiq-8.1.6 lib/sidekiq/redis_client_adapter.rb byte-for-byte in
-  # behavior: same fast-path command list, same deprecation warning, same
+  # Behaviourally equivalent to the adapter a drop-in app expects from
+  # sidekiq 8.1.x: same fast-path command list, same deprecation warning, same
   # error constants (gems rescue `Sidekiq::RedisClientAdapter::BaseError`).
+  # The constant names and the deprecated-command set are fixed by that
+  # contract; everything below them is Wurk's own.
   class RedisClientAdapter
     BaseError = RedisClient::Error
     CommandError = RedisClient::CommandError

data/lib/wurk/redis_pool.rb

@@ -2,6 +2,7 @@
 
 require 'redis-client'
 require 'connection_pool'
+require_relative 'command_builder'
 require_relative 'redis_client_adapter'
 require_relative 'redis_options'
 
@@ -52,13 +53,15 @@ module Wurk
     DEFAULT_WRITE_TIMEOUT      = 2.5
     DEFAULT_RECONNECT_ATTEMPTS = 1
 
-    # The floor every pool starts from; any key the host passed wins over it.
+    # The floor every pool starts from; any key the host passed wins over it —
+    # including `command_builder`, so an app that has its own keeps it.
     DEFAULT_CLIENT_CONFIG = {
       url: DEFAULT_URL,
       connect_timeout: DEFAULT_CONNECT_TIMEOUT,
       read_timeout: DEFAULT_READ_TIMEOUT,
       write_timeout: DEFAULT_WRITE_TIMEOUT,
-      reconnect_attempts: DEFAULT_RECONNECT_ATTEMPTS
+      reconnect_attempts: DEFAULT_RECONNECT_ATTEMPTS,
+      command_builder: CommandBuilder
     }.freeze
 
     # Server-side messages where the connection is closed and the block retried

data/lib/wurk/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Wurk
-  VERSION = '1.6.0'
+  VERSION = '1.7.1'
 end

data/lib/wurk/web/extension.rb

@@ -1,7 +1,12 @@
 # frozen_string_literal: true
 
-require 'erb'
-require 'cgi'
+# erb and cgi are required by #call, the single entry point every extension
+# request comes through, rather than here: together with the `cgi` erb drags in
+# they are ~40ms of the ~220ms `require "wurk"` costs, and they are reachable
+# only from a host that has actually registered a Web extension. A worker
+# process — which loads this file because `wurk.rb` loads the web layer — never
+# renders anything. `require` is idempotent, so this is a $LOADED_FEATURES
+# lookup per request after the first.
 require 'securerandom'
 
 module Wurk
@@ -230,6 +235,9 @@ module Wurk
             ext = registered_extension(name)
             return nil unless ext
 
+            require 'erb'
+            require 'cgi'
+
             verb = method.to_s.upcase
             route, block, route_params = match_route(ext, verb, subpath)
             return [404, html_headers, "No #{verb} route #{subpath} in extension #{name}"] unless route

data/lib/wurk.rb

@@ -8,6 +8,8 @@
 require_relative 'wurk/errors'
 require_relative 'wurk/version'
 require_relative 'wurk/keys'
+require_relative 'wurk/pid_cache'
+require_relative 'wurk/command_builder'
 require_relative 'wurk/redis_pool'
 require_relative 'wurk/pool_checkout'
 require_relative 'wurk/redis_connection'

data/vendor/assets/dashboard/wurk-manifest.json

@@ -1,4 +1,4 @@
 {
-  "version": "1.6.0",
-  "timestamp": "2026-08-09T17:57:48.347Z"
+  "version": "1.7.1",
+  "timestamp": "2026-08-11T00:13:25.877Z"
 }

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: wurk
 version: !ruby/object:Gem::Version
-  version: 1.6.0
+  version: 1.7.1
 platform: ruby
 authors:
 - developerz.ai
@@ -224,6 +224,7 @@ files:
 - lib/wurk/client.rb
 - lib/wurk/client/buffered.rb
 - lib/wurk/collapse.rb
+- lib/wurk/command_builder.rb
 - lib/wurk/compat.rb
 - lib/wurk/component.rb
 - lib/wurk/configuration.rb
@@ -317,6 +318,7 @@ files:
 - lib/wurk/middleware/poison_pill.rb
 - lib/wurk/middleware/status.rb
 - lib/wurk/middleware/timeout.rb
+- lib/wurk/pid_cache.rb
 - lib/wurk/pool_checkout.rb
 - lib/wurk/process_set.rb
 - lib/wurk/processor.rb