wurk 1.6.0 → 1.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cbdba1bc48d5cc6029c22b78376e6b6a0b93e4f519f6c717fd3f2ce84b5900d2
-  data.tar.gz: 0d0c5e36c75c898fa05d1b448e4730391cbcf883d39402db49ac879d3168aa91
+  metadata.gz: 4ce7dc086fc21fd3dd300206e2c9b15b2e8720a59e8186ea1459b66067bf6148
+  data.tar.gz: 1a659643c694f6fadd7bd34313604df1093274c4e50597c00ff767db4376f3ed
 SHA512:
-  metadata.gz: 29c1cb70a235bffed95cfa399328be393c4f653df388eff3f06a36e605f7f941ec21f1dc21b6fa832876f09635991c69087ff18dec9dc3a08cedb50f85945338
-  data.tar.gz: ca4aa23ed8658e6fc98cbe14501a1789cc38cc43a6a539c6a41fd3aebdcba97bb91fef58c84ae26f0d36cc42518b8279b348336efa93a3b4f0aaf37bf886c625
+  metadata.gz: e6dd4d7643349e6ed95496e5149f19f3ca832589f4475b5de8e03f438dc4e445a3871b1ccf6240754bb85708f78a8118962234a465f3afa150b9753368a6fe77
+  data.tar.gz: 4a31cfe2170ab1d726db165d354edc00763e14b118b0219f15bfb0bc0de909c03a795829196e8a6b6e56c7f5e013a3dde4b7d94d0ebf49f2aec0c75d2c6a267e

data/README.md

@@ -21,7 +21,11 @@
 
 Wurk is wire-compatible with Sidekiq — same Redis keys, same job JSON, same Ruby DSL. Swap one line in your `Gemfile` and your existing jobs, batches, limiters, cron entries, and live Redis data keep working untouched. The Pro and Enterprise feature sets ship in the same free gem, with no license check and no tiers.
 
-**On Sidekiq:** Sidekiq is the reason Ruby background processing works as well as it does. Mike Perham and Contributed Systems have maintained it for well over a decade and funded that work through Pro and Enterprise — a model that kept a critical piece of the ecosystem healthy, documented, and supported, and one the whole community has benefited from. Wurk implements Sidekiq's API because it is a genuinely good API, and exists to make a different bet about what maintenance costs now — see [Why Wurk exists](#why-wurk-exists).
+**In production:** Wurk runs the background work at [developerz.ai](https://developerz.ai) and at partner deployments — millions of jobs an hour, across many servers, on the fork-based swarm described below. It is not a preview.
+
+**At scale:** Wurk is built for fleets, not just for one box. Kubernetes `/live` + `/ready` probes are a config line, not a sidecar; a bearer-scoped [HTTP API](docs/api-http.md) lets non-Ruby services enqueue and inspect; [OpenTelemetry](docs/telemetry.md) traces propagate client → server; per-queue [global concurrency caps](docs/rate-limiting.md) hold cluster-wide limits; and monitoring is the dashboard you already mount — live SSE, charts, per-job progress, no separate stack to run. See [Wurk extras](#wurk-extras).
+
+**On Sidekiq:** Wurk implements Sidekiq's API because it is a genuinely good API. Sidekiq is human-maintained and funds that work through its paid tiers; Wurk is AI-maintained, which is what lets the same surface be free software. Wurk is independent and not affiliated with or endorsed by Sidekiq or its maintainers — see [Why Wurk exists](#why-wurk-exists).
 
 **On speed:** Wurk is not currently faster than stock Sidekiq — it runs at roughly 0.87×–1.02× depending on workload shape, with parity on CPU and I/O but still behind on framework overhead (noop) and boot time. Numbers, method, and the reproduction command are in [docs/benchmarks.md](docs/benchmarks.md); run them yourself with `rake bench:vs_sidekiq`.
 
@@ -205,18 +209,16 @@ Knobs: `health_check(port:, bind: "0.0.0.0", ready_window: 30)`. In swarm mode o
 
 ## Why Wurk exists
 
-Sidekiq's split into OSS, Pro, and Enterprise is how a decade of serious maintenance got funded, and it worked. Ruby got a background-job library that stayed maintained, documented, and answerable to its users for longer than most infrastructure gems survive at all — and the API in this README is the one that came out of it. Wurk is standing on that work.
+Infrastructure this basic should be free software. A Rails app shouldn't need a licence key to get reliable fetch, batches, rate limiting, or cron — those are table stakes, not a premium tier, and the free-software tradition is that the best tools belong to everyone who runs them.
 
-What Wurk bets on is that the economics underneath changed. Wurk is built and maintained **AI-first** — implementation, parity suite, docs, and benchmarks are written and kept current by AI agents working under human review. That is what makes it practical to:
+What has made that hard is maintenance: someone has to be paid to do it. Sidekiq funds a decade of *human* maintenance through its paid tiers, which is an honest trade. Wurk makes a different one — it is maintained **AI-first**: implementation, parity suite, docs, and benchmarks are written and kept current by AI agents under human review. A fix, a doc update, or a version bump is no longer somebody's week, which is what makes it practical to:
 
 - ship the entire Pro + Enterprise surface with no tier, no flag gate, and no license check;
 - keep parity honest mechanically rather than by hand — Sidekiq's own tests run as an oracle suite, and third-party gems (sidekiq-cron, sidekiq-unique-jobs, sidekiq-scheduler, sidekiq-status, sidekiq-failures, sidekiq-throttled) run their upstream suites against Wurk on every push;
 - keep adding surface Sidekiq doesn't have — the [Wurk extras](#wurk-extras) above landed as one release;
-- sustain that over the long run, because the marginal cost of a fix, a doc update, or a version bump is no longer somebody's week.
-
-It also means Wurk holds itself to published numbers instead of adjectives: the benchmark suite runs against stock Sidekiq every release and the results ship [as measured](docs/benchmarks.md), including the unflattering ones.
+- hold ourselves to published numbers instead of adjectives — the suite runs against stock Sidekiq every release and ships the results [as measured](docs/benchmarks.md), including the unflattering ones.
 
-None of this makes Wurk the right call for everyone. Sidekiq Pro and Enterprise come with a commercial support contract, a decade of production track record, and a human on the other end of an email. If that is what your risk profile needs, buy it — it is worth the money, and it is the reason the API Wurk implements exists in the first place.
+Wurk is MIT and stays that way. If what you need is a commercial support contract and a human on the other end of an email, buying that is a perfectly good answer.
 
 ## Migrating from Sidekiq
 

data/app/controllers/wurk/dashboard_controller.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require 'erb'
 require 'json'
 require 'net/http'
 require 'uri'

data/lib/wurk/cli.rb

@@ -1,9 +1,11 @@
 # frozen_string_literal: true
 
-require 'yaml'
-require 'optparse'
-require 'erb'
-
+# yaml, optparse and erb are required inside the two methods that use them
+# (#parse_config and #option_parser) rather than here. Together they cost ~45ms
+# of the ~290ms `require "wurk"` spends, and only `exe/wurk` ever reaches them —
+# a Rails app that enqueues and processes jobs pays that on every boot for a
+# command-line parser it never runs. `require` is idempotent, so the lazy call
+# is a $LOADED_FEATURES hash lookup on every invocation after the first.
 require_relative 'version'
 require_relative 'configuration'
 require_relative 'component'
@@ -410,6 +412,9 @@ module Wurk
     def parse_config(path)
       raise ArgumentError, "No such file #{path}" unless ::File.exist?(path)
 
+      require 'erb'
+      require 'yaml'
+
       erb = ::ERB.new(::File.read(path), trim_mode: '-')
       erb.filename = ::File.expand_path(path)
       opts = ::YAML.safe_load(erb.result, permitted_classes: [Symbol], aliases: true) || {}
@@ -444,6 +449,8 @@ module Wurk
     end
 
     def option_parser(opts)
+      require 'optparse'
+
       ::OptionParser.new do |o|
         o.banner = @command ? "wurk #{@command} [options]" : 'wurk [options]'
         define_value_flags(o, opts, OPTION_FLAGS)

data/lib/wurk/command_builder.rb

@@ -0,0 +1,49 @@
+# frozen_string_literal: true
+
+require 'redis-client'
+
+module Wurk
+  # Fast path for redis-client's command normalization.
+  #
+  # `RedisClient::CommandBuilder.generate` splices Hash arguments with a
+  # `flat_map` and then stringifies Symbols/Integers/Floats with a `map!` —
+  # two array allocations and two per-element type dispatches for a command
+  # whose arguments are already Strings. Every command Wurk sends per job is
+  # exactly that shape (LMOVE, LREM, DEL, LPUSH, SADD), and the fetch pipeline
+  # sends three of them. Measured ~4x faster on those commands (2.2µs → 0.6µs
+  # each) with `bench/command_builder.rb`.
+  #
+  # Anything else — a Hash to splice, a Symbol or a number to stringify,
+  # keyword arguments, an empty command — falls straight through to
+  # redis-client's own builder. This is a shortcut, never a second
+  # implementation of the semantics, so a host app calling
+  # `Sidekiq.redis { |c| c.call("HSET", key, hash) }` is unaffected.
+  #
+  # The array is duped rather than handed back: `call_v` and the pipelined
+  # forms pass the caller's own array through, and redis-client's builder
+  # always returns a fresh one that middleware is free to mutate in place.
+  module CommandBuilder
+    module_function
+
+    def generate(args, kwargs = nil)
+      return args.dup if fast?(args, kwargs)
+
+      ::RedisClient::CommandBuilder.generate(args, kwargs)
+    end
+
+    # Split out, and public, because the two branches produce byte-identical
+    # output by construction — which leaves allocation count as the only
+    # runtime difference between "the shortcut fired" and "it silently didn't",
+    # and that is not something a test can assert safely in a process running
+    # other suites. So the decision itself is what the suite asserts.
+    #
+    # `kwargs` is nil only from `call_v` and the scan helpers. Every `call`,
+    # `blocking_call` and pipelined `call` arrives with `**kwargs` already
+    # splatted into a Hash — EMPTY when the caller passed no keywords — so a
+    # guard that only tested `.nil?` sent the entire hot path down the slow
+    # branch while every direct-call test still passed.
+    def fast?(args, kwargs)
+      (kwargs.nil? || kwargs.empty?) && !args.empty? && args.all?(String)
+    end
+  end
+end

data/lib/wurk/component.rb

@@ -2,6 +2,7 @@
 
 require 'socket'
 require 'securerandom'
+require_relative 'pid_cache'
 
 module Wurk
   # Shared mixin for runtime components (Launcher, Manager, Processor, Fetcher,
@@ -55,7 +56,7 @@ module Wurk
     def self.tid
       thread = Thread.current
       memo = thread.thread_variable_get(:wurk_tid)
-      pid = ::Process.pid
+      pid = PidCache.pid
       return memo[1] if memo && memo[0] == pid
 
       id = (thread.object_id ^ pid).to_s(36).freeze

data/lib/wurk/fetcher/reliable.rb

@@ -2,6 +2,7 @@
 
 require 'socket'
 require_relative '../component'
+require_relative '../pid_cache'
 require_relative '../keys'
 require_relative '../lua'
 require_relative '../fetcher'
@@ -366,7 +367,7 @@ module Wurk
       # fetcher and read this without a lock. A racing writer's entry can be
       # lost, and the next fetch rebuilds it.
       def queue_keys(public_q)
-        pid = ::Process.pid
+        pid = PidCache.pid
         if @queue_keys_pid != pid
           @queue_keys = {}
           @queue_keys_pid = pid

data/lib/wurk/fetcher/unit_of_work.rb

@@ -32,6 +32,13 @@ module Wurk
       # admitted under, and are set only by Fetcher::Capped — nil is the whole
       # uncapped path, which is why the release below is a field test rather
       # than a question asked of Redis.
+      # LREM's count, pre-stringified. Every argument of every command on the
+      # ACK pipeline is then a String, which is what keeps it on
+      # {Wurk::CommandBuilder}'s allocation-free fast path; an Integer here
+      # would send the whole command back through redis-client's normalizer.
+      LREM_COUNT = '1'
+      private_constant :LREM_COUNT
+
       UnitOfWork = Struct.new(:queue, :queue_name, :private_queue, :job, :config, :jid, :fetcher,
                               :slot_key, :slot_token, keyword_init: true) do
         # Deferred, never skipped: the LREM goes back to the fetcher, which
@@ -55,7 +62,7 @@ module Wurk
         # wire-frozen, so the reaper cannot flag it). Reading the counter to
         # decide would spend the very round trip the DEL is riding for free.
         def write_ack(pipe)
-          pipe.call('LREM', private_queue, 1, job)
+          pipe.call('LREM', private_queue, LREM_COUNT, job)
           job_jid = jid.to_s
           Middleware::PoisonPill.clear_in(pipe, job_jid) unless job_jid.empty?
           release_slot_in(pipe) if slot_key

data/lib/wurk/job_logger.rb

@@ -20,6 +20,10 @@ module Wurk
     def initialize(config)
       @config = config
       @logger = @config.logger
+      # Read once: `config[]` is a Hash lookup through a delegating Capsule, and
+      # #context_hash runs it per job only to walk a list that cannot change
+      # after boot (Configuration freezes at launch).
+      @logged_attributes = Array(@config[:logged_job_attributes]).map { |attr| [attr, attr.to_sym] }.freeze
       @skip = !!@config[:skip_default_job_logging]
     end
 
@@ -68,8 +72,8 @@ module Wurk
         class: job_hash['wrapped'] || job_hash['class']
       }
 
-      @config[:logged_job_attributes].each do |attr|
-        h[attr.to_sym] = job_hash[attr] if job_hash.key?(attr)
+      @logged_attributes.each do |attr, sym|
+        h[sym] = job_hash[attr] if job_hash.key?(attr)
       end
       h
     end

data/lib/wurk/pid_cache.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+module Wurk
+  # `Process.pid` is a real syscall on Linux — glibc dropped its getpid cache in
+  # 2.25 — and the fetch+execute path reads it twice per job ({Component.tid}
+  # and the fetcher's per-queue key cache, both of which read it only to notice
+  # a fork). Measured at ~640ns a call against ~110ns for this memo, which is
+  # about 1% of a job's CPU spent asking the kernel a question whose answer
+  # changes at most once in a process's life.
+  #
+  # Refreshed in the child half of every fork, so a swarm child never answers
+  # with the pid it inherited — the whole reason those two call sites read it.
+  #
+  # Same shape as redis-client's PIDCache, deliberately not borrowed from it:
+  # that is a private constant of a dependency, and this one is on our hot path.
+  module PidCache
+    if ::Process.respond_to?(:fork)
+      class << self
+        attr_reader :pid
+
+        def update!
+          @pid = ::Process.pid
+        end
+      end
+      update!
+
+      # Prepended, not aliased: `Process._fork` is a documented hook (Ruby 3.1+,
+      # and Wurk requires 3.2) that several gems wrap, and prepending composes
+      # with theirs instead of replacing it.
+      module CoreExt
+        def _fork
+          child_pid = super
+          PidCache.update! if child_pid.zero?
+          child_pid
+        end
+      end
+      ::Process.singleton_class.prepend(CoreExt)
+    else # JRuby / TruffleRuby / Windows — no fork, so nothing can invalidate it
+      @pid = ::Process.pid
+      singleton_class.attr_reader(:pid)
+    end
+  end
+end

data/lib/wurk/profiler.rb

@@ -3,7 +3,6 @@
 require 'securerandom'
 require 'zlib'
 require 'stringio'
-require 'tempfile'
 require_relative 'keys'
 require_relative 'pool_checkout'
 
@@ -109,7 +108,12 @@ module Wurk
         Wurk.configuration.handle_exception(e, context: 'Wurk::Profiler')
       end
 
+      # `tempfile` (and the `tmpdir` it drags in) is ~19ms of `require "wurk"`,
+      # spent only by an install that has vernier loaded AND profiling switched
+      # on for a job. Everyone else was paying it at boot.
       def profile_to_json(&)
+        require 'tempfile'
+
         Tempfile.create(['wurk-profile', '.json']) do |file|
           ::Vernier.profile(out: file.path, &)
           File.read(file.path)

data/lib/wurk/redis_pool.rb

@@ -2,6 +2,7 @@
 
 require 'redis-client'
 require 'connection_pool'
+require_relative 'command_builder'
 require_relative 'redis_client_adapter'
 require_relative 'redis_options'
 
@@ -52,13 +53,15 @@ module Wurk
     DEFAULT_WRITE_TIMEOUT      = 2.5
     DEFAULT_RECONNECT_ATTEMPTS = 1
 
-    # The floor every pool starts from; any key the host passed wins over it.
+    # The floor every pool starts from; any key the host passed wins over it —
+    # including `command_builder`, so an app that has its own keeps it.
     DEFAULT_CLIENT_CONFIG = {
       url: DEFAULT_URL,
       connect_timeout: DEFAULT_CONNECT_TIMEOUT,
       read_timeout: DEFAULT_READ_TIMEOUT,
       write_timeout: DEFAULT_WRITE_TIMEOUT,
-      reconnect_attempts: DEFAULT_RECONNECT_ATTEMPTS
+      reconnect_attempts: DEFAULT_RECONNECT_ATTEMPTS,
+      command_builder: CommandBuilder
     }.freeze
 
     # Server-side messages where the connection is closed and the block retried

data/lib/wurk/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Wurk
-  VERSION = '1.6.0'
+  VERSION = '1.7.0'
 end

data/lib/wurk/web/extension.rb

@@ -1,7 +1,12 @@
 # frozen_string_literal: true
 
-require 'erb'
-require 'cgi'
+# erb and cgi are required by #call, the single entry point every extension
+# request comes through, rather than here: together with the `cgi` erb drags in
+# they are ~40ms of the ~220ms `require "wurk"` costs, and they are reachable
+# only from a host that has actually registered a Web extension. A worker
+# process — which loads this file because `wurk.rb` loads the web layer — never
+# renders anything. `require` is idempotent, so this is a $LOADED_FEATURES
+# lookup per request after the first.
 require 'securerandom'
 
 module Wurk
@@ -230,6 +235,9 @@ module Wurk
             ext = registered_extension(name)
             return nil unless ext
 
+            require 'erb'
+            require 'cgi'
+
             verb = method.to_s.upcase
             route, block, route_params = match_route(ext, verb, subpath)
             return [404, html_headers, "No #{verb} route #{subpath} in extension #{name}"] unless route

data/lib/wurk.rb

@@ -8,6 +8,8 @@
 require_relative 'wurk/errors'
 require_relative 'wurk/version'
 require_relative 'wurk/keys'
+require_relative 'wurk/pid_cache'
+require_relative 'wurk/command_builder'
 require_relative 'wurk/redis_pool'
 require_relative 'wurk/pool_checkout'
 require_relative 'wurk/redis_connection'

data/vendor/assets/dashboard/wurk-manifest.json

@@ -1,4 +1,4 @@
 {
-  "version": "1.6.0",
-  "timestamp": "2026-08-09T17:57:48.347Z"
+  "version": "1.7.0",
+  "timestamp": "2026-08-10T12:35:59.680Z"
 }

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: wurk
 version: !ruby/object:Gem::Version
-  version: 1.6.0
+  version: 1.7.0
 platform: ruby
 authors:
 - developerz.ai
@@ -224,6 +224,7 @@ files:
 - lib/wurk/client.rb
 - lib/wurk/client/buffered.rb
 - lib/wurk/collapse.rb
+- lib/wurk/command_builder.rb
 - lib/wurk/compat.rb
 - lib/wurk/component.rb
 - lib/wurk/configuration.rb
@@ -317,6 +318,7 @@ files:
 - lib/wurk/middleware/poison_pill.rb
 - lib/wurk/middleware/status.rb
 - lib/wurk/middleware/timeout.rb
+- lib/wurk/pid_cache.rb
 - lib/wurk/pool_checkout.rb
 - lib/wurk/process_set.rb
 - lib/wurk/processor.rb