wurk 1.3.1 → 1.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2b1f8eac5cefbd9f645598b9a37dca543aeebb361967d820f369cc14420ad3f6
-  data.tar.gz: 813958b643c78cdd2ffa5063798bb3994f6510b1b0d50a0a54db1622f3bb325a
+  metadata.gz: '0995fd5827870e14584104a8998db58e9927339197460596567c05f2dacc7da0'
+  data.tar.gz: 8159d245826bf646a9c6025e4e06a49717f957aabe18dad5d2a5b58ed99818c0
 SHA512:
-  metadata.gz: 65ca6b60bf079d626f3887d9e1e12b7d1d1f91ab264c5fcba7c77f54667961159fb8a63c8be89d44e9583ef7c264669e546eb15ab4bc13e392ef85b35dfea6ad
-  data.tar.gz: b865d8cd28483589bad0f6d879d80a85c9f9d5cb9b79a494634c90ebaf0f5371169ed7012303dabdc736b32be3596dbffd2ac60b2bbeb3d24c1b40514e483b54
+  metadata.gz: d45c20c9a8c29695c7ea2e04066dda58454fea0ce56453ed866cd111e38977fb1739ae47429a60b68380fac2f4d7fe85f232b420f92f95cba330c389ee435b5b
+  data.tar.gz: 94c5599bc28408e1e5eb30967d1b0fad4806d415a17e515774c057120d0833660f3c2860d9cf0cb2ea320ae3a27bd9ce683385ce5ace1c347d46530a96cd9a1a

data/README.md

@@ -6,7 +6,7 @@
 
 <p align="center"><strong>Wurk, wurk.</strong> 🪓 <em>Ready to work. Zug zug.</em></p>
 
-<p align="center"><strong>A 100% drop-in replacement for Sidekiq + Sidekiq Pro + Sidekiq Enterprise. Free forever. Faster.</strong></p>
+<p align="center"><strong>A 100% drop-in replacement for Sidekiq + Sidekiq Pro + Sidekiq Enterprise. Free forever.</strong></p>
 
 <div align="center">
 
@@ -21,6 +21,8 @@
 
 Wurk is wire-compatible with Sidekiq — same Redis keys, same job JSON, same Ruby DSL. Swap one line in your `Gemfile` and your existing jobs, batches, limiters, cron entries, and live Redis data keep working untouched. The Pro and Enterprise feature sets ship in the same free gem, with no license check and no tiers.
 
+**On speed:** Wurk is not currently faster than stock Sidekiq — it runs at roughly 0.87×–1.02× depending on workload shape, with parity on CPU and I/O but still behind on framework overhead (noop) and boot time. Numbers, method, and the reproduction command are in [docs/benchmarks.md](docs/benchmarks.md); run them yourself with `rake bench:vs_sidekiq`.
+
 ## Install
 
 ```ruby
@@ -61,6 +63,7 @@ Plus Wurk extras: a worker topology DSL, a Kubernetes liveness/readiness listene
 - **[Starting the worker](https://github.com/developerz-ai/wurk/blob/main/docs/running.md)** — Rails auto-start, the `wurk`/`wurkswarm` runners, and running standalone without Rails.
 - **[Configuration reference](https://github.com/developerz-ai/wurk/blob/main/docs/configuration.md)** — every option, env var, YAML key, and CLI flag, with precedence and pool sizing.
 - **[Deploying](https://github.com/developerz-ai/wurk/blob/main/docs/deployment.md)** — systemd, Capistrano, Heroku, Docker, Kubernetes, rolling restarts, memory limits.
+- **[Secrets & credentials](https://github.com/developerz-ai/wurk/blob/main/docs/secrets.md)** — which values are secret vs config, how to supply them (ENV, Rails credentials, an init file), precedence, and what to never commit.
 - **[Active Job adapter](https://github.com/developerz-ai/wurk/blob/main/docs/active-job.md)** — run `ActiveJob`/`deliver_later` on Wurk with `queue_adapter = :wurk`.
 - **[Testing jobs](https://github.com/developerz-ai/wurk/blob/main/docs/testing.md)** — fake/inline modes, the jobs array, Minitest and RSpec setup.
 - **[Migrating from Sidekiq](#migrating-from-sidekiq)** — the one-line swap and what to expect.

data/app/controllers/concerns/wurk/stream_concurrency_guard.rb

@@ -7,28 +7,45 @@ module Wurk
   # we 503 with Retry-After — the SPA's EventSource reconnects (and its polling
   # fallback honors Retry-After) once a slot frees. Per-process is the right
   # scope: it's this process's own thread pool we're protecting.
+  #
+  # Slots are held as thread references rather than tallied in a counter so the
+  # cap can heal itself. A stream whose thread is killed mid-flight never
+  # reaches the `ensure` below (Puma hard-reaps worker threads past
+  # `force_shutdown_after`, and a thread killed inside an uninterruptible read
+  # can skip its ensure), which a counter would record as a slot held by nobody
+  # — ten of those and `/api/stream` 503s for the life of the process. A dead
+  # holder is instead evicted by the next acquire.
   module StreamConcurrencyGuard
     extend ActiveSupport::Concern
 
     MAX_CONCURRENT_STREAMS = 10
     RETRY_AFTER_SECONDS = 3
 
-    @open = 0
+    @holders = []
     @lock = Mutex.new
 
     class << self
-      # Reserve a stream slot; false when the cap is already reached.
+      # Reserve a stream slot for the calling thread; false when the cap is
+      # already reached by threads that are still alive.
       def acquire
         @lock.synchronize do
-          return false if @open >= MAX_CONCURRENT_STREAMS
+          @holders.keep_if(&:alive?)
+          return false if @holders.size >= MAX_CONCURRENT_STREAMS
 
-          @open += 1
+          @holders << Thread.current
           true
         end
       end
 
+      # Drops one slot held by the calling thread. Acquire and release always
+      # bracket a single block on one thread (`#with_stream_slot`), so a call
+      # from a thread holding nothing is a no-op rather than a slot taken away
+      # from whoever is actually streaming.
       def release
-        @lock.synchronize { @open -= 1 if @open.positive? }
+        @lock.synchronize do
+          index = @holders.rindex(Thread.current)
+          @holders.delete_at(index) if index
+        end
       end
     end
 

data/lib/wurk/batch/callbacks.rb

@@ -26,8 +26,9 @@ module Wurk
       # parent's *own* last job acks while a child batch is still running,
       # nothing fires here; the last child's propagate_to_parent re-invokes
       # this and fires then. The SREM in pkids_drained? happens before that
-      # re-invocation, so exactly one of the racing paths fires (dedup_set
-      # absorbs the overlap).
+      # re-invocation, so at most one of the racing paths reaches a fire and
+      # the callback markers absorb the rest (see `fire_complete` for the one
+      # window that can still duplicate).
       def maybe_fire(bid, pending:, live:)
         return unless live.zero?
         return unless kids_finished?(bid)
@@ -49,15 +50,38 @@ module Wurk
       # the dedup guard so it is restored on re-death; the callback enqueue
       # and parent cascade stay behind the guard so `:death` is enqueued at
       # most once per batch.
+      #
+      # That claim-before-enqueue ordering is kept deliberately, against the
+      # enqueue-before-mark rule `fire_complete` explains: everything that
+      # makes the batch *look* dead is already persisted above the guard, so a
+      # crash in the window costs the notification while `Status`, the
+      # dashboard and `subtree_dead?` all still see a dead batch. `:complete`
+      # and `:success` have no such fallback — the callback is their whole
+      # signal — and this claim additionally gates `cascade_death`, which
+      # would otherwise re-walk the ancestor chain on every re-invocation.
       def fire_death(bid)
         record_event(bid, 'death_at')
-        Wurk.redis { |conn| conn.call('ZADD', 'dead-batches', Time.now.to_f.to_s, bid) }
+        index_dead(bid)
         return unless dedup_set(bid, 'death')
 
         enqueue_callbacks(bid, 'death')
         cascade_death(bid)
       end
 
+      # Index the batch as dead and bound the set in the same round trip. The
+      # score stays `Time.now.to_f` (wire format, spec §2.8); `Batch.trim_index`
+      # reads it as the epoch seconds it is. See there for why the set needs a
+      # trim at all — only `Status#delete` and the death-recovery ZREM ever
+      # remove a member, and neither runs for a batch left to expire.
+      def index_dead(bid)
+        Wurk.redis do |conn|
+          conn.pipelined do |pipe|
+            pipe.call('ZADD', 'dead-batches', Time.now.to_f.to_s, bid)
+            Batch.trim_index(pipe, 'dead-batches')
+          end
+        end
+      end
+
       # A child's death means the parent — and every ancestor — can never
       # fully succeed, so `:death` propagates up the parent chain. The
       # recursion bottoms out at the root (empty parent_bid); fire_death's own
@@ -70,19 +94,45 @@ module Wurk
         fire_death(parent_bid)
       end
 
+      # `:complete` and `:success` mark their dedup key *after* the enqueue,
+      # never before (F16). Nothing re-drives a fire once the acking job's
+      # BATCH_ACK_SUCCESS has SREM'd its jid — that job's retry gets
+      # `pending == -1` and returns before maybe_fire — so a claim-then-enqueue
+      # ordering turns a crash in between into callbacks that are never
+      # enqueued by anyone, ever. Enqueuing first makes the durable side effect
+      # happen before the marker that suppresses it.
+      #
+      # The accepted direction is a duplicate over a lost callback: callback
+      # jobs retry like any other job and must already be idempotent (spec
+      # §2.4, §12 "Callback retries"), so firing one twice is a cost the app
+      # is required to absorb, while losing one silently strands the batch.
+      #
+      # `dedup_marked?` still collapses every *sequential* re-invocation — a
+      # reclaimed child re-running propagate_to_parent, a second DeathHandler
+      # pass — so the duplicate window is only two genuinely concurrent acks
+      # interleaving between each other's check and mark.
+      #
+      # `record_event` stays ahead of the enqueue: the callback job reads a
+      # Status snapshot and must see `complete_at`/`success_at` already set.
       def fire_complete(bid)
-        return unless dedup_set(bid, 'complete')
+        return if dedup_marked?(bid, 'complete')
 
         record_event(bid, 'complete_at')
         enqueue_callbacks(bid, 'complete')
+        dedup_set(bid, 'complete')
       end
 
+      # Same enqueue-then-mark ordering as fire_complete. `apply_linger` runs
+      # last of all: it EXPIREs `b-<bid>-success` down to the linger window,
+      # which only holds if the marker already exists — `dedup_set`'s 30d
+      # `EX` would otherwise re-create it outside that window.
       def fire_success(bid)
-        return unless dedup_set(bid, 'success')
+        return if dedup_marked?(bid, 'success')
 
         record_event(bid, 'success_at')
         emit_duration_metric(bid)
         enqueue_callbacks(bid, 'success')
+        dedup_set(bid, 'success')
         apply_linger(bid)
       end
 
@@ -90,10 +140,11 @@ module Wurk
       # full success. `created_at` shares the CLOCK_REALTIME epoch we record it
       # with. No-op without a dogstatsd client.
       #
-      # Strictly best-effort: `fire_success` has already burned the `success`
-      # dedup key by the time we run, so a raise here (e.g. a Redis hiccup on the
-      # HGET) would permanently strand the success callbacks and linger that
-      # follow — a retry can't re-fire them. Swallow and log instead.
+      # Strictly best-effort: this runs on the acking job's thread ahead of the
+      # enqueue, and that ack already removed the jid, so a raise here (e.g. a
+      # Redis hiccup on the HGET) would abort `fire_success` with nothing left
+      # to re-drive it — the success callbacks and linger would be stranded for
+      # good. Swallow and log instead.
       def emit_duration_metric(bid)
         created = Wurk.redis { |conn| conn.call('HGET', "b-#{bid}", 'created_at') }
         return if created.nil? || created.to_s.empty?
@@ -116,9 +167,22 @@ module Wurk
         end
       end
 
-      # Atomically marks `bid` as having fired `event`. Returns true the
-      # first time, false thereafter — caller skips the enqueue when false.
-      # SET NX makes this safe under racing acks.
+      # True once `b-<bid>-<event>` exists, i.e. an enqueue pass for `event`
+      # has completed. The read-side half of the enqueue-then-mark ordering in
+      # `fire_complete`/`fire_success`; `fire_death` needs no equivalent
+      # because its `dedup_set` still doubles as the claim.
+      def dedup_marked?(bid, event)
+        Wurk.redis { |conn| conn.call('EXISTS', "b-#{bid}-#{event}") }.to_i == 1
+      end
+
+      # Writes `b-<bid>-<event>`, the marker that `event`'s callbacks have been
+      # enqueued. Returns true when this call created it, false when it was
+      # already there.
+      #
+      # Two usages, deliberately different: `fire_death` calls it *before* its
+      # enqueue and treats the return as a claim (at most once); `fire_complete`
+      # and `fire_success` call it *after* theirs and ignore the return, gating
+      # on `dedup_marked?` instead. SET NX keeps both safe under racing acks.
       def dedup_set(bid, event)
         Wurk.redis do |conn|
           ok = conn.call('SET', "b-#{bid}-#{event}", '1', 'NX', 'EX', Batch::CALLBACK_NOTIFY_TTL)
@@ -126,11 +190,17 @@ module Wurk
         end
       end
 
+      # The HSETs resurrect the hash when a callback fires for a batch whose keys
+      # already expired (a child batch outliving its parent's 30d window), so the
+      # write is followed by an NX stamp — without it the resurrected hash would
+      # have no clock at all. NX leaves a live batch's expiry, and the shorter
+      # post-success `linger` window, untouched.
       def record_event(bid, field)
         now = ::Process.clock_gettime(::Process::CLOCK_REALTIME)
         Wurk.redis do |conn|
           conn.call('HSET', "b-#{bid}", field, now.to_s)
           conn.call('HSET', "b-#{bid}", field.to_s.sub('_at', ''), '1')
+          conn.call('EXPIRE', "b-#{bid}", Batch::DEFAULT_EXPIRY_SECONDS, 'NX')
         end
       end
 

data/lib/wurk/batch/death_handler.rb

@@ -21,7 +21,7 @@ module Wurk
             conn,
             :batch_ack_complete,
             keys: ["b-#{bid}", "b-#{bid}-jids", "b-#{bid}-died", "b-#{bid}-failed"],
-            argv: [job['jid']]
+            argv: [job['jid'], Batch::DEFAULT_EXPIRY_SECONDS]
           )
         end
         live, _died, first_death = Array(result).map(&:to_i)
@@ -38,9 +38,12 @@ module Wurk
         Wurk::Batch::Callbacks.maybe_fire(bid, pending: Wurk::Batch::Callbacks.pending_for(bid), live: 0)
       end
 
-      # A stale job can die AFTER its batch keys expired; the ack writes
-      # recreate them with no TTL — permanent key leakage. EXPIRE NX stamps
-      # only keys that lost their TTL, leaving live batches' clocks alone.
+      # BATCH_ACK_COMPLETE stamps the two keys it can itself resurrect; this
+      # sweeps the rest of the batch (`-jids`, `-failed`, `-kids`, `-pkids`,
+      # callback markers). A death is the one moment we know the batch is
+      # winding down, so it is worth a round trip to leave nothing without a
+      # clock. EXPIRE NX touches only keys that have none, so a live batch's
+      # clock and a post-success `linger` window both survive.
       def self.restamp_ttls(bid)
         Wurk.redis do |conn|
           conn.pipelined do |pipe|

data/lib/wurk/batch/server_middleware.rb

@@ -85,7 +85,7 @@ module Wurk
             conn,
             :batch_ack_failed,
             keys: ["b-#{bid}", "b-#{bid}-failed"],
-            argv: [jid]
+            argv: [jid, Batch::DEFAULT_EXPIRY_SECONDS]
           )
         end
       end

data/lib/wurk/batch.rb

@@ -41,6 +41,22 @@ module Wurk
     POST_SUCCESS_EXPIRY_SECONDS = 24 * 60 * 60
     CALLBACK_NOTIFY_TTL = 30 * 24 * 60 * 60
 
+    # Member ceiling for the two batch index ZSETs (`batches`, `dead-batches`).
+    # The score axis in `.trim_index` retires entries in step with the batch data
+    # itself, so this is only the backstop for a workload creating batches faster
+    # than that window retires them. Deliberately generous: a cap that bites drops
+    # batches whose data is still live out of `BatchSet`, and at this scale the
+    # per-batch hashes dwarf the index anyway.
+    INDEX_MAX = 1_000_000
+
+    # Ceiling on the `callbacks` array of one batch hash, enforced by
+    # BATCH_APPEND_CALLBACK. Every registration re-encodes the whole array,
+    # and every entry becomes a callback job when the event fires, so an
+    # unbounded array is both a hot-path cost and a fan-out. Far above any
+    # legitimate batch — real ones register a handful — so hitting it means a
+    # loop is registering callbacks it should have registered once.
+    CALLBACKS_MAX = 1_000
+
     # Bid is URL-safe base64 of 10 random bytes — matches Sidekiq Pro's BID
     # generator. Length matters: third-party gems that key off bid prefix
     # (sharded batches in Pro 8) inspect the first character.
@@ -48,10 +64,21 @@ module Wurk
 
     VALID_EVENTS = %i[success complete death].freeze
 
+    # Every key a batch owns, for the sweep paths: `Status#delete` (UNLINK),
+    # `Callbacks#apply_linger` and `DeathHandler.restamp_ttls` (EXPIRE).
+    #
     # The 'live' set tracks jobs that have not yet reached a terminal state.
     # When it's empty, every job has either succeeded or died → `:complete`
     # is allowed to fire.
-    KEY_SUFFIXES = %w[jids failed died notify cbsucc kids pkids tags].freeze
+    #
+    # `complete`/`success`/`death` are the callback dedup markers written by
+    # `Callbacks#dedup_set`; they belong to the batch and must die with it.
+    # `notify`/`cbsucc`/`tags` are Sidekiq Pro's own key layout (spec §2.8) that
+    # Wurk never writes — Wurk dedups on the three markers above and indexes
+    # tags at `tags:<tag>`. They stay listed so a Redis dataset carried over
+    # from Sidekiq Pro on the gem swap gets swept too; EXPIRE/UNLINK of a
+    # missing key is a no-op for batches Wurk created itself.
+    KEY_SUFFIXES = %w[jids failed died complete success death notify cbsucc kids pkids tags].freeze
 
     THREAD_KEY = :wurk_current_batch
 
@@ -68,6 +95,33 @@ module Wurk
       [base, *KEY_SUFFIXES.map { |s| "#{base}-#{s}" }]
     end
 
+    # Two-axis trim of a batch index ZSET (`batches`, `dead-batches`), in the
+    # shape of the morgue trim (`DeadSet#trim`): `ZREMRANGEBYSCORE` evicts
+    # entries older than `timeout`, `ZREMRANGEBYRANK 0 -max` caps the member
+    # count — and, like the morgue, that bound keeps `max - 1` of a full set.
+    # Appended to the caller's pipeline so bounding the index costs neither
+    # writer an extra round trip.
+    #
+    # Nothing else ever shrinks either set: `Status#delete` and the
+    # death-recovery `ZREM` are manual, so an index entry outlives the batch it
+    # points at and both sets grow for the life of the Redis without this.
+    #
+    # Both index in epoch seconds — `batches` from CLOCK_REALTIME,
+    # `dead-batches` from `Time.now.to_f` — so one cutoff serves both. The
+    # default window is the batch hash TTL: past it `b-<bid>` is gone and the
+    # entry only yields an empty Status. A batch that overrode `expires_in`
+    # beyond that window outlives its index entry — still reachable by bid,
+    # just no longer enumerated by `BatchSet`.
+    #
+    # `max:` / `timeout:` override the defaults for one call, so parallel tests
+    # can drive the trim on isolated limits without mutating the process-global
+    # `Wurk.configuration`.
+    def self.trim_index(pipe, key, max: nil, timeout: nil)
+      cutoff = ::Process.clock_gettime(::Process::CLOCK_REALTIME) - (timeout || DEFAULT_EXPIRY_SECONDS)
+      pipe.call('ZREMRANGEBYSCORE', key, '-inf', "(#{cutoff}")
+      pipe.call('ZREMRANGEBYRANK', key, 0, -(max || INDEX_MAX))
+    end
+
     def initialize(bid = nil)
       @bid              = bid || SecureRandom.urlsafe_base64(BID_BYTES)
       @existing         = !bid.nil?
@@ -79,6 +133,11 @@ module Wurk
       @linger           = nil
       @parent_bid       = nil
       @callbacks        = []
+      # Dedup index over `@callbacks`, keyed on the encoded entry. Only the
+      # pre-flush staging path feeds it — once flushed, Redis holds the array
+      # and BATCH_APPEND_CALLBACK does the deduping — so a batch reopened by
+      # bid never pays to build it.
+      @callback_index   = Set.new
       @expires_in       = DEFAULT_EXPIRY_SECONDS
       @mutable          = !@existing
       @flushed_once     = @existing
@@ -166,17 +225,22 @@ module Wurk
       self
     end
 
-    # Register a callback. Multiple callbacks of the same event are allowed.
-    # The callback target may be a Class, "Foo#bar" string spec, or anything
-    # responding to `name`. `options` must be JSON-serializable.
+    # Register a callback. Any number of *distinct* callbacks may be attached
+    # to one event; re-registering an identical `[event, target, options]`
+    # triple is a no-op, and past `CALLBACKS_MAX` entries the registration is
+    # dropped with a warning. The callback target may be a Class, "Foo#bar"
+    # string spec, or anything responding to `name`. `options` must be
+    # JSON-serializable.
     def on(event, callback, options = {})
       sym = event.to_sym
       raise ArgumentError, "invalid event #{event.inspect}" unless VALID_EVENTS.include?(sym)
       raise ArgumentError, 'callback options must be a Hash' unless options.is_a?(Hash)
 
       entry = [sym.to_s, callback_target(callback), options]
-      @callbacks << entry
-      persist_callback!(entry) if @flushed_once
+      # Before the first flush the array lives only in memory; after it, Redis
+      # is authoritative and `@callbacks` is a stale mirror nothing reads —
+      # appending to it there would just leak one entry per registration.
+      @flushed_once ? persist_callback!(entry) : stage_callback(entry)
       self
     end
 
@@ -246,22 +310,52 @@ module Wurk
       Wurk::Client.new.flush_batched(payloads) unless payloads.empty?
     end
 
+    # Pre-flush counterpart to `persist_callback!`. Entries registered before
+    # the first flush only exist in memory until `first_flush_hash` writes the
+    # whole array in one HSET, so BATCH_APPEND_CALLBACK never sees them — the
+    # same dedup and cap have to be applied here or the very first write can
+    # already ship duplicates and an unbounded array.
+    #
+    # Keyed on the encoded entry, which is what actually lands in the hash:
+    # `{a: 1}` and `{'a' => 1}` are one callback once persisted, so they must
+    # be one entry here too.
+    def stage_callback(entry)
+      json = entry.to_json
+      return if @callback_index.include?(json)
+
+      if @callbacks.size >= CALLBACKS_MAX
+        Wurk.logger.warn("batch #{@bid}: #{entry[0]} callback dropped — #{CALLBACKS_MAX} callback limit reached")
+        return
+      end
+
+      @callbacks << entry
+      @callback_index << json
+    end
+
     # Like `linger=`, anything registered after the first flush must reach
     # Redis — `Callbacks.enqueue_callbacks` reads specs from the hash, so an
     # in-memory-only append would silently never fire (#213). Covers both
     # `on` after `#jobs` and batches reopened by bid. The append runs
     # server-side (Lua) so concurrent registrations from different processes
-    # can't lose each other to a read-modify-write race.
+    # can't lose each other to a read-modify-write race, and it dedups
+    # identical triples so the reopen-per-job shape stops growing the array.
     def persist_callback!(entry)
       event = entry[0]
-      fired = Wurk.redis do |conn|
+      status = Wurk.redis do |conn|
         Wurk::Lua::Loader.eval_cached(conn, :batch_append_callback,
-                                      keys: ["b-#{@bid}"], argv: [entry.to_json, event])
+                                      keys: ["b-#{@bid}"], argv: [entry.to_json, event, CALLBACKS_MAX])
+      end
+      raise ArgumentError, "cannot register #{event} callback: batch #{@bid} no longer exists" if status == -1
+
+      # Sentinels are Integers, the fired flag is the String the `<event>`
+      # hash field holds — asymmetric on purpose, so a sentinel can never be
+      # read as a fired event.
+      case status
+      when -2
+        Wurk.logger.warn("batch #{@bid}: #{event} callback dropped — #{CALLBACKS_MAX} callback limit reached")
+      when '1'
+        Wurk.logger.warn("batch #{@bid}: #{event} callback registered after #{event} already fired — it will never run")
       end
-      raise ArgumentError, "cannot register #{event} callback: batch #{@bid} no longer exists" if fired == -1
-      return unless fired == '1'
-
-      Wurk.logger.warn("batch #{@bid}: #{event} callback registered after #{event} already fired — it will never run")
     end
 
     # First flush writes the core hash, registers in the global `batches`
@@ -278,10 +372,15 @@ module Wurk
       Wurk::Metrics::Statsd.increment('batch.created')
     end
 
+    # Only `b-#{@bid}` is stamped here — none of the sub-keys exist yet at first
+    # flush (BATCH_PUSH/BATCH_SCHEDULE create `-jids`, the acks create
+    # `-failed`/`-died`), and EXPIRE on a missing key is a no-op. Each key is
+    # stamped `NX` where it is created instead; see BATCH_PUSH in lua.rb.
     def pipelined_first_flush(pipe, now)
       pipe.call('HSET', "b-#{@bid}", *first_flush_hash(now).flatten)
       pipe.call('EXPIRE', "b-#{@bid}", @expires_in)
       pipe.call('ZADD', 'batches', now.to_s, @bid)
+      Batch.trim_index(pipe, 'batches')
       @tags.each { |t| pipe.call('SADD', "tags:#{t}", @bid) }
       link_to_parent(pipe) if current_parent_bid
     end
@@ -312,9 +411,16 @@ module Wurk
       outer.bid
     end
 
+    # The only place `-kids`/`-pkids` are created, so this is where they get
+    # their clock. TTL is the default, not this batch's `@expires_in`: the keys
+    # belong to the *parent*, and NX means the first link sets the retention for
+    # every sibling that follows.
     def link_to_parent(pipe)
-      pipe.call('SADD', "b-#{current_parent_bid}-kids", @bid)
-      pipe.call('SADD', "b-#{current_parent_bid}-pkids", @bid)
+      parent_key = "b-#{current_parent_bid}"
+      pipe.call('SADD', "#{parent_key}-kids", @bid)
+      pipe.call('SADD', "#{parent_key}-pkids", @bid)
+      pipe.call('EXPIRE', "#{parent_key}-kids", DEFAULT_EXPIRY_SECONDS, 'NX')
+      pipe.call('EXPIRE', "#{parent_key}-pkids", DEFAULT_EXPIRY_SECONDS, 'NX')
     end
 
     def job_count

data/lib/wurk/capsule.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require_relative 'redis_pool'
+require_relative 'pool_checkout'
 require_relative 'middleware/chain'
 require_relative 'fetcher/reliable'
 
@@ -16,6 +17,11 @@ module Wurk
     attr_reader :name, :queues, :mode, :weights, :config
     attr_accessor :concurrency, :fetcher
 
+    # Capsule-hosted components (Manager, Processor, Fetcher) hand their capsule
+    # to Component as `config`, and `safe_thread` reads the priority off it.
+    # Sidekiq delegates the same accessor (capsule.rb:30).
+    def thread_priority = @config.thread_priority
+
     def initialize(name, config)
       @name = name.to_s
       @config = config
@@ -24,8 +30,15 @@ module Wurk
       @mode = :strict
       @weights = { 'default' => 0 }
       @fetcher = nil
-      @redis_pool = nil
-      @fetch_redis_pool = nil
+      # One mutable Hash rather than two ivars: the pools are the only part of a
+      # capsule that legitimately changes after Configuration#freeze! — fork
+      # closes them, Launcher#stop releases them, an embedded host that boots
+      # again rebuilds them. `Object#freeze` is shallow, so the Hash stays
+      # writable and freezing a capsule keeps meaning "no more configuration"
+      # instead of "these sockets are yours forever". Before this, a reset on a
+      # frozen capsule disconnected the pool and then raised FrozenError on the
+      # memo, leaving `redis_pool` answering with a shut-down pool for good.
+      @pools = {}
       @client_chain = nil
       @server_chain = nil
     end
@@ -63,9 +76,26 @@ module Wurk
     # by hand; centralizing it here covers the standalone CLI and embedded
     # paths too (the bug behind a nil `fetcher` in `exe/wurk`). Idempotent.
     def prepare!
+      prepare_shared!
       @fetcher ||= build_fetcher
       redis_pool
       fetch_redis_pool
+      self
+    end
+
+    # The half of `prepare!` a forking parent can run on every child's behalf:
+    # the chains are a pure function of this capsule's identity, not of the slot
+    # (queues + concurrency) a swarm child is assigned later, and `copy_for`
+    # opens nothing. Run before the fork, the entries are allocated once and
+    # inherited copy-on-write instead of rebuilt in every child.
+    #
+    # The rest of `prepare!` deliberately stays post-fork: both pools are sized
+    # off the slot's concurrency and one built here would hand every child an
+    # inherited socket, and `build_fetcher` fires the host's
+    # `config[:fetch_setup]` hook, which is per-child — running it in the parent
+    # would let a custom fetcher snapshot the wrong queues, or leak whatever the
+    # hook opened across the fork. Idempotent.
+    def prepare_shared!
       client_middleware
       server_middleware
       self
@@ -98,7 +128,7 @@ module Wurk
     MIN_POOL_SIZE = 10
 
     def redis_pool
-      @redis_pool ||= build_pool(size: main_pool_size, name: "#{@name}-main")
+      @pools[:main] ||= build_pool(size: main_pool_size, name: "#{@name}-main")
     end
 
     # Dedicated pool for the reliable fetcher's blocking BLMOVE: one slot per
@@ -106,28 +136,27 @@ module Wurk
     # fetch at once. Keeping fetch off the main pool is what lets an idle worker
     # hold zero main-pool connections again.
     def fetch_redis_pool
-      @fetch_redis_pool ||= build_pool(size: @concurrency, name: "#{@name}-fetch")
+      @pools[:fetch] ||= build_pool(size: @concurrency, name: "#{@name}-fetch")
     end
 
-    # Disconnect and drop cached pools. Called by Wurk::Swarm just before
-    # fork (parent side: close inherited sockets) and just after fork
-    # (child side: rebuild lazily). Connection_pool#shutdown is terminal,
-    # so dropping the reference is required — `redis_pool` will rebuild.
+    # Disconnect and drop cached pools. Called by Wurk::Swarm just before fork
+    # (parent side: close inherited sockets), just after fork (child side:
+    # rebuild lazily), and by Launcher#stop (release what this process held).
+    # Connection_pool#shutdown is terminal, so dropping the reference is
+    # required — `redis_pool` will rebuild.
     def reset_redis_pools!
-      @redis_pool&.disconnect!
-      @redis_pool = nil
-      @fetch_redis_pool&.disconnect!
-      @fetch_redis_pool = nil
+      @pools.each_value(&:disconnect!)
+      @pools.clear
     end
 
-    def redis(&)
-      redis_pool.with(&)
+    def redis(idempotent: false, &)
+      PoolCheckout.with(redis_pool, idempotent, &)
     end
 
     # Checkout from the dedicated fetch pool. Only the reliable fetcher's
     # blocking BLMOVE uses this, so a parked fetch never holds a main-pool slot.
-    def fetch_redis(&)
-      fetch_redis_pool.with(&)
+    def fetch_redis(idempotent: false, &)
+      PoolCheckout.with(fetch_redis_pool, idempotent, &)
     end
 
     def lookup(name)