solid-jobs 0.4.2 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7179d19681ddd3f238ba480d935c3a7e3829368ec011db4adfc571979d7174a9
-  data.tar.gz: e2669fd2af3814e7d1b2b6a58626dbbeee5ab3322067e99343b7722f186d7cd8
+  metadata.gz: 7c1b35b5001c3c0467635a9fb46e3cab6305700a629c23babb604fc026895ec4
+  data.tar.gz: 441859a1e6880c259a61a8e9bf1c7004caf9c25d967335a322558ec7227da3e2
 SHA512:
-  metadata.gz: 8905446cb4bfa04ab5c5341e018e69dac330dfef9c89f0240c4b1857c83b233da4b894421400a28381fd18079fa84ace67505bee530ac9c270d25ca23a68287b
-  data.tar.gz: 235d3165daca0d685385afaa38df3b75612d5a53d4992d5585a55efd30cca21656f5cb350dad2ca0c204dd2080cfd8067fef12c33f6da92fe51c5d472a8c02e8
+  metadata.gz: 0ece103959e94a949ad6c85e3591dbebcc3d82e495c9b4ab3d17f8729dd37bdaa591d12f5bdfe6adf8f4b7298078ef2c5326e1b36bffb3024d236a054980b999
+  data.tar.gz: 5f0e56b6900bec66f374f3446d4b553c0b84a934c06718abc77219286002b4b977de68072c3f524f295ceb0c0e81c5f4bd4b5b25a6b42776f791ddb2626614ab

data/CHANGELOG.md

@@ -2,6 +2,14 @@
 
 All notable changes to this project will be documented in this file.
 
+## [0.5.0] - 2026-10-12
+
+### Added
+
+- Warm worker routing: `enqueue` reroutes tasks holding non Ractor-safe state
+  (`Mutex`, `Thread`, `Fiber`, pools) to persistent Unix-socket workers
+  (`SolidJobsGenericExecutor`, `script/warm_worker.rb`).
+
 ## [0.4.2] - 2026-10-05
 
 ### Changed

data/README.md

@@ -3,6 +3,7 @@
 [![Build Status](https://github.com/nicolasva/solid-jobs/actions/workflows/ci.yml/badge.svg)](https://github.com/nicolasva/solid-jobs/actions/workflows/ci.yml)
 [![Gem Version](https://badge.fury.io/rb/solid-jobs.svg)](https://rubygems.org/gems/solid-jobs)
 [![Downloads](https://img.shields.io/gem/dt/solid-jobs?style=flat)](https://rubygems.org/gems/solid-jobs)
+[![Documentation Status](https://img.shields.io/badge/docs-RubyDoc.info-blue.svg)](https://www.rubydoc.info/gems/solid-jobs)
 
 SolidJobs is a Ractor-oriented Redis task runner for Ruby. It provides its own
 task API, Redis envelope, keyspace, interceptor model, failure policy, and
@@ -12,6 +13,39 @@ SolidJobs is **not compatible with Sidekiq**. It does not read Sidekiq queues,
 does not write Sidekiq payloads, and does not expose Sidekiq's job API. Existing
 applications and queued work must be migrated explicitly.
 
+## SolidTrace observability
+
+SolidJobs and SolidTrace have separate, complementary roles:
+
+- **SolidJobs** is the job runner. It enqueues and executes background tasks,
+  manages channels, retries, recovery, and worker Ractors.
+- **SolidTrace** is the observability dashboard. It consumes structured
+  lifecycle and operational telemetry emitted by SolidJobs and presents a
+  read-only view of job activity and system health.
+
+With SolidTrace, operators can inspect recent job executions and their
+observed lifecycle, monitor queues and worker Ractors, and review process
+resource usage, Redis health, telemetry freshness, and gaps that require
+resynchronization. SolidTrace does not enqueue, execute, retry, or control
+SolidJobs tasks; SolidJobs remains responsible for processing them. The
+dashboard is a separate Sinatra/Rack component that can be mounted in a Rails
+application, with host authentication and access protection supplied by that
+application.
+
+SolidTrace uses structured telemetry rather than job arguments or business
+payloads. Its trace views show retained lifecycle events and calculable
+durations, and identify missing or unavailable information instead of
+guessing. See the [SolidTrace project](https://github.com/nicolasva/SolidTrace)
+for more information.
+
+### SolidTrace dashboard
+
+![SolidTrace dashboard overview](https://github.com/nicolasva/SolidTrace/blob/main/file_00000000708882108cd8ae9a1106f601.jpg)
+
+![SolidTrace dashboard pages](https://github.com/nicolasva/SolidTrace/blob/main/SolidTrace%20Futuristic%20Dashboard%20Showcase.png)
+
+For access to SolidTrace, contact me at [nicolas.vandenbogaerde@gmail.com](mailto:nicolas.vandenbogaerde@gmail.com).
+
 ## Installation
 
 ```ruby
@@ -195,18 +229,282 @@ The Active Job adapter remains available as
 `ActiveJob::QueueAdapters::SolidJobsAdapter`, but it writes only SolidJobs
 envelopes and keys.
 
-## Monitoring with SolidTrace
+## Instrumentation
+
+SolidJobs exposes an optional Ractor-local instrumenter and defaults to a
+no-op. Any configured instrumenter must respond to `instrument(name, payload)`
+and be Ractor-shareable before a conductor starts. Ordinary instrumentation
+failures are logged when possible and never change publication, execution,
+retry, acknowledgement, recovery, or shutdown behavior. Process-control
+interruptions still propagate so shutdown can requeue in-flight work.
+
+SolidJobs emits the v1 lifecycle events `job.enqueued`, `job.journaled`,
+`job.reserved`, `job.started`, `job.completed`, `job.failed`,
+`job.retry_scheduled`, `job.dead`, `job.acknowledged`, and `job.recovered`.
+Ready jobs are not reservable until `job.enqueued` and `job.journaled` have
+been emitted; an expiring Redis publication barrier preserves this order
+without stranding work if a publisher exits.
+They carry correlation identifiers and structured errors, but never task
+arguments, business payloads, secrets, or textual logs. `node_id` is shared by
+all Ractors in a conductor; `ractor_id` and `worker_id` are integer processor
+identities. `reservation_id` is the reliable claim token.
+
+The heartbeat Ractor also samples and emits `process.observed`, one
+`ractor.observed` per logical processor, and one `redis.observed` per processor
+every five seconds. Process observations are limited to process CPU time, RSS,
+GC count/time, and total Ruby allocations. Sources are local Ruby/OS reads made
+outside worker Ractors. A source that has never succeeded is `unavailable`; a
+later failure preserves the last value as `stale` with its increasing age.
+SolidJobs never estimates CPU, memory, GC, or allocations per Ractor.
+
+Ractor observations expose only state, activity, integer
+`ractor_id`/`worker_id`/`executor_id`, and current job IDs. Quiet workers report
+`waiting`; shutdown reports `stopping` then `stopped` without inventing work.
+Redis connection state comes from worker operations already being performed:
+successful operations report `connected` and `SolidRedis::ConnectionError`
+reports `disconnected`. No monitoring-only Redis command is added. Redis
+latency, command, memory, and connection-count metrics remain explicitly
+`unavailable` until a reliable source exists.
+
+## Transparent Ractor-safe hybrid execution (warm worker routing)
+
+Ractors enforce a shared-nothing model: referencing mutable global state, legacy
+connection pools, or thread-based dependencies (ActiveRecord, ActionMailer, gems
+using `Mutex`, `Thread` or `Fiber`) inside a Ractor raises
+`ArgumentError: can't share mutable object`. `SolidJobs::Task#enqueue` detects
+such tasks and reroutes them to a persistent **warm worker** process, so you do
+not have to rewrite them.
+
+### How routing is decided
+
+`requires_warm_pool?(arguments)` returns `true` only when **all** of these hold:
+
+1. `defined?(Rails)` is truthy.
+2. At least one argument is not `Ractor.shareable?`.
+3. The task class holds, in a class-level instance variable, a `Mutex`,
+   `Thread` or `Fiber`, or an object whose class name contains `Pool`.
+
+Otherwise `enqueue` runs the normal Ractor-native path. When routed, the
+arguments are converted to plain JSON data (deep-copied and frozen) and enqueued
+as a `SolidJobsGenericExecutor` task, which forwards them over a Unix socket.
+
+> Detection is heuristic and inspects only class-level instance variables. It
+> does not detect constants such as `ActiveRecord::Base` referenced inside
+> `execute_task`. Arguments must be JSON-serializable, and a task with
+> shareable arguments always stays Ractor-native.
+
+### Components
+
+- `app/jobs/solid_jobs_generic_executor.rb` — `SolidJobsGenericExecutor`
+  connects to `/tmp/solid_jobs_rails_<worker_id>.sock`, sends
+  `{"target_class": ..., "arguments": [...]}` as one JSON line, and returns the
+  result or raises `SolidJobsGenericExecutor::RemoteError`. The worker is chosen
+  as `hash(task name) % SOLID_JOBS_WARM_WORKERS` (default `1`). Require this file
+  explicitly (it is loaded automatically on first routing).
+- `script/warm_worker.rb` — a standalone process, run outside any Ractor, that
+  loads `config/environment.rb` from its working directory once, listens on the
+  socket, resolves the class with `Object.const_get`, runs `execute_task` (or
+  `perform`), and replies with `{"status":"ok","result":...}` or
+  `{"status":"error","error_class":...,"message":...}`. Requests are handled one
+  at a time per worker. The socket is created with mode `0600` and removed on
+  `INT`/`TERM`.
+
+### Running warm workers
+
+Warm workers are **not** spawned by `solid-jobs`; start and supervise them
+yourself (systemd, foreman, Procfile, ...), one per worker id, from the Rails
+root:
+
+```bash
+SOLID_JOBS_WARM_WORKERS=2 bundle exec ruby script/warm_worker.rb 0 &
+SOLID_JOBS_WARM_WORKERS=2 bundle exec ruby script/warm_worker.rb 1 &
+bundle exec solid-jobs --require ./config/environment --concurrency 8 --channel critical
+```
+
+The worker count used by the runner must match the number of workers started.
+The boot cost is paid once per worker; the process then stays warm. Memory is
+reclaimed only when a worker is restarted, so supervise and recycle them as you
+would any long-lived Rails process.
+
+### Example
+
+```ruby
+class SyncAccounts
+  include SolidJobs::Task
+  @lock = Mutex.new # class-level non-shareable state => warm worker routing
+
+  def execute_task(account_ids)
+    Account.where(id: account_ids).find_each(&:sync!)
+  end
+end
+
+SyncAccounts.enqueue([1, 2, 3]) # routed to the warm worker
+```
+
+Tests live in `test/warm_pool_test.rb` and spawn a real worker process.
+
+## The Ractor concurrency challenge
+
+Core Rails frameworks like `ActiveRecord` and `ActionMailer` rely on global
+variables, shared connection pools, and mutable global state. Referencing these
+modules directly inside a strict, isolated `Ractor` raises
+`ArgumentError: can't share mutable object`.
+
+To keep SolidJobs' Ractor-native execution while preserving your existing Rails
+business logic, use a **hybrid execution bridge**: delegate mutable work to an
+isolated sub-process via `bin/rails runner`. The worker stays Ractor-native and
+the heavy Rails operations are offloaded to the operating system layer.
+
+### Hybrid execution examples
+
+> **Important:** These task classes must be loaded explicitly (e.g., inside
+> `config/boot.rb`) outside of Zeitwerk management, as standard Rails
+> autoloading is restricted inside isolated Ractors.
+
+#### Isolated ActionMailer delivery task
+
+Trigger an email workflow using your existing Rails mailers and templates
+without memory isolation crashes:
+
+```ruby
+# frozen_string_literal: true
+
+require "open3"
+
+class SendEmailTask
+  include SolidJobs::Task
+
+  # Define your processing channel and failure thresholds
+  task_options channel: "default", max_failures: 3
+
+  APP_ROOT = File.expand_path("../..", __dir__).freeze
+  SCRIPT = "WelcomeMailer.hello(ARGV[0]).deliver_now"
+
+  def execute_task(recipient = "nicolas.vandenbogaerde@gmail.com")
+    output, status = Open3.capture2e(
+      { "RAILS_ENV" => ENV.fetch("RAILS_ENV", "development") },
+      "bin/rails", "runner", SCRIPT, recipient,
+      chdir: APP_ROOT
+    )
+
+    unless status.success?
+      raise "Email delivery failed: #{output.lines.grep_v(/^\s+from /).first(5).join}"
+    end
+  end
+end
+```
+
+#### Isolated ActiveRecord data sync task
+
+Load database records and run model mutations safely outside of the Ractor:
+
+```ruby
+# frozen_string_literal: true
+
+require "open3"
+
+class ProcessUserDataTask
+  include SolidJobs::Task
+
+  # Dedicated data synchronization channel
+  task_options channel: "data_sync", max_failures: 3
+
+  APP_ROOT = File.expand_path("../..", __dir__).freeze
+  SCRIPT = "User.find(ARGV[0]).update_metrics!"
+
+  def execute_task(user_id)
+    output, status = Open3.capture2e(
+      { "RAILS_ENV" => ENV.fetch("RAILS_ENV", "development") },
+      "bin/rails", "runner", SCRIPT, user_id.to_s,
+      chdir: APP_ROOT
+    )
+
+    unless status.success?
+      raise "ActiveRecord task failed: #{output.lines.grep_v(/^\s+from /).first(5).join}"
+    end
+  end
+end
+```
+
+#### Complete business service object execution (zero-leak memory architecture)
+
+In complex production monoliths, a single service object often encapsulates
+multiple `ActiveRecord` updates, pricing mutations, external API calls, and
+email notifications. Running this orchestration layer directly inside a
+`Ractor` triggers memory isolation failures.
+
+By serializing the service input into a lightweight, Ractor-safe **JSON
+payload**, SolidJobs can hand the task details to an isolated `bin/rails runner`
+sub-process.
+
+A major benefit is **automatic memory recovery**: once the service has
+allocated thousands of transient Ruby objects, the sub-process terminates and
+the operating system reclaims all of its memory, keeping the worker immune to
+memory bloat and leaks.
+
+Example: processing a heavy order checkout service.
+
+```ruby
+# frozen_string_literal: true
+
+require "open3"
+require "json"
+
+# Loaded explicitly (e.g., in config/boot.rb) outside of Zeitwerk.
+# SolidJobs executors run within Ractors where standard Rails autoloading is restricted.
+class ExecuteServiceTask
+  include SolidJobs::Task
+
+  # Dedicated high-priority business services channel
+  task_options channel: "services", max_failures: 2
+
+  APP_ROOT = File.expand_path("../..", __dir__).freeze
+
+  # The runner script parses the input JSON from the command line and executes the service
+  SCRIPT = <<~RUBY
+    require 'json'
+    params = JSON.parse(ARGV[0])
+    OrderCheckoutService.call(user_id: params['user_id'], cart_id: params['cart_id'])
+  RUBY
+
+  def execute_task(params_hash)
+    # Serialize the pure Hash into a strict JSON string for safe Ractor transmission
+    json_payload = params_hash.to_json
+
+    output, status = Open3.capture2e(
+      { "RAILS_ENV" => ENV.fetch("RAILS_ENV", "development") },
+      "bin/rails", "runner", SCRIPT, json_payload,
+      chdir: APP_ROOT
+    )
+
+    unless status.success?
+      raise "Service object execution failed: #{output.lines.first(5).join}"
+    end
+  end
+end
+```
+
+Enqueuing from your Rails application:
+
+```ruby
+# Construct a clean, 100% Ractor-safe arguments structure
+task_payload = { user_id: 42, cart_id: 891 }
+
+# Push into the Redis pipeline
+ExecuteServiceTask.enqueue(task_payload)
+```
+
+## Commercial licensing & observability
 
-**SolidTrace** is a statistics and observability tool for SolidJobs. It shows
-real-time SolidJobs metrics while your application runs: jobs per second, CPU
-and memory usage per Ractor, queue depth, Redis latency, job flow, and recent
-job executions.
+`SolidJobs` is open source under the **LGPL-3.0-or-later** license.
 
-![SolidTrace dashboard](docs/images/solid-trace.jpg)
+Running multi-core clusters blind in staging or production makes monitoring
+hard. For visual insight into Ractor memory allocation, CPU thresholds, and live
+queue states, pair this gem with **SolidTrace** (commercial licenses available
+from $50 to $200 / month).
 
-SolidTrace is a commercial product. If you would like to acquire it, please
-contact me at [nicolas.vandenbogaerde@gmail.com](mailto:nicolas.vandenbogaerde@gmail.com)
-or via GitHub ([@nicolasva](https://github.com/nicolasva)).
+For commercial keys, custom architecture audits, or enterprise support, contact
+**licensing@solidtrace.io**.
 
 ## Performance: SolidJobs vs Sidekiq
 

data/docs/reliability.md

@@ -29,6 +29,11 @@ attempt      monotonically increasing execution count
 claimed_at   wall-clock claim time
 ```
 
+A ready payload has a short-lived publication barrier while its
+`job.enqueued` and `job.journaled` events are emitted. Executors leave marked
+payloads in READY until the publisher removes the marker. Markers expire after
+30 seconds, so a publisher crash can delay a task but cannot strand it.
+
 Completion and requeue are fenced by `claim_token`. Their Lua scripts verify
 that the executor still owns the current journal generation before changing
 state. A delayed or revived executor cannot complete or requeue a newer claim.

data/lib/solid_jobs/blueprint.rb

@@ -1,6 +1,8 @@
 # frozen_string_literal: true
 
 require "logger"
+require "securerandom"
+require "socket"
 
 module SolidJobs
   class Blueprint
@@ -11,9 +13,9 @@ module SolidJobs
 
     attr_accessor :concurrency, :discarded_limit, :discarded_retention, :logger,
       :on_complex_arguments, :poll_interval_average, :shutdown_timeout,
-      :reliable_fetch, :retry_base_delay, :retry_max_delay
+      :reliable_fetch, :retry_base_delay, :retry_max_delay, :instrumenter
     attr_reader :default_task_options, :error_handlers, :execute_interceptors,
-      :publish_interceptors, :redis_config
+      :publish_interceptors, :redis_config, :identity
 
     def initialize(redis: nil, concurrency: 5, channels: ["default"])
       @redis_config = redis || SolidRedis.config(url: ENV.fetch("REDIS_URL", "redis://127.0.0.1:6379/0"))
@@ -34,6 +36,8 @@ module SolidJobs
       @discarded_limit = 10_000
       @discarded_retention = 180 * 24 * 60 * 60
       @logger = Logger.new($stdout)
+      @identity = "#{Socket.gethostname}:#{::Process.pid}:#{SecureRandom.hex(6)}".freeze
+      @instrumenter = NullInstrumenter
       @redis_pool = nil
     end
 
@@ -115,6 +119,10 @@ module SolidJobs
     end
 
     def ractor_snapshot
+      unless Ractor.shareable?(instrumenter)
+        raise ArgumentError, "SolidJobs instrumenter must be Ractor-shareable"
+      end
+
       Utilities.shareable_copy(
         redis_config: redis_config,
         concurrency: concurrency,
@@ -131,6 +139,8 @@ module SolidJobs
         retry_max_delay: retry_max_delay,
         publish_interceptors: publish_interceptors.export,
         execute_interceptors: execute_interceptors.export,
+        identity: identity,
+        instrumenter: instrumenter,
       )
     end
 
@@ -152,6 +162,8 @@ module SolidJobs
       config.retry_max_delay = snapshot.fetch(:retry_max_delay)
       config.publish_interceptors.import(snapshot.fetch(:publish_interceptors))
       config.execute_interceptors.import(snapshot.fetch(:execute_interceptors))
+      config.instance_variable_set(:@identity, snapshot.fetch(:identity))
+      config.instrumenter = snapshot.fetch(:instrumenter)
       config
     end
 

data/lib/solid_jobs/claim.rb

@@ -12,9 +12,14 @@ module SolidJobs
       if not payload then
         return nil
       end
+      local job = cjson.decode(payload)
+      local publication = job["#{PublicationBarrier::FIELD}"]
+      if publication and redis.call("exists", ARGV[5] .. publication) == 1 then
+        redis.call("rpush", KEYS[1], payload)
+        return nil
+      end
 
       redis.call("lpush", KEYS[2], payload)
-      local job = cjson.decode(payload)
       local attempt = redis.call("hincrby", KEYS[4], job["id"], 1)
       local metadata = {
         task_id = job["id"],
@@ -28,6 +33,19 @@ module SolidJobs
       redis.call("hset", KEYS[3], ARGV[3], cjson.encode(metadata))
       return {payload, attempt}
     LUA
+    POP_READY = <<~LUA.freeze
+      local payload = redis.call("rpop", KEYS[1])
+      if not payload then
+        return nil
+      end
+      local job = cjson.decode(payload)
+      local publication = job["#{PublicationBarrier::FIELD}"]
+      if publication and redis.call("exists", ARGV[1] .. publication) == 1 then
+        redis.call("rpush", KEYS[1], payload)
+        return nil
+      end
+      return payload
+    LUA
     REGISTER = <<~LUA.freeze
       local attempt = redis.call("hincrby", KEYS[2], ARGV[1], 1)
       local metadata = cjson.decode(ARGV[3])
@@ -66,6 +84,7 @@ module SolidJobs
       return removed
     LUA
     RESERVE_SHA = Digest::SHA1.hexdigest(RESERVE).freeze
+    POP_READY_SHA = Digest::SHA1.hexdigest(POP_READY).freeze
     REGISTER_SHA = Digest::SHA1.hexdigest(REGISTER).freeze
     ACK_SHA = Digest::SHA1.hexdigest(ACK).freeze
     REQUEUE_SHA = Digest::SHA1.hexdigest(REQUEUE).freeze
@@ -131,6 +150,7 @@ module SolidJobs
     end
 
     def initialize(config, identity: nil, processor_id: nil)
+      @config = config
       @redis_pool = config.redis_pool
       @channel_order = config.channel_order
       @reliable = config.reliable_fetch
@@ -167,12 +187,7 @@ module SolidJobs
       if @claimed_key
         claim_reliable(channels)
       else
-        channels = channels.shuffle if @channel_order == :shuffle
-        result = @redis_pool.blocking_call(TIMEOUT, "BRPOP", *channels, TIMEOUT)
-        result && record(
-          result.fetch(1),
-          result.fetch(0).delete_prefix("#{Keyspace::PREFIX}:channel:"),
-        )
+        claim_unreliable(channels)
       end
     end
 
@@ -183,10 +198,10 @@ module SolidJobs
     private
 
     def record(payload, channel = nil, claim_token: nil, attempt: nil)
-      envelope = JSON.parse(payload)
+      envelope = PublicationBarrier.strip(JSON.parse(payload))
       channel ||= envelope.fetch("channel")
       claim_token, attempt = register(envelope, channel) unless claim_token
-      ClaimRecord.new(
+      claim = ClaimRecord.new(
         channel: channel,
         payload: payload,
         envelope: envelope,
@@ -197,13 +212,22 @@ module SolidJobs
         claim_token: claim_token,
         attempt: attempt,
       )
+      Instrumentation.emit(
+        @config,
+        :reserved,
+        envelope,
+        reservation_id: claim_token,
+        attempt: attempt,
+        **Instrumentation.worker_attributes(@processor_id),
+      )
+      claim
     end
 
     def existing_claim(payload)
       raw = @redis_pool.call("HGET", @claims_key, @processor_id)
       if raw
         metadata = JSON.parse(raw)
-        envelope = JSON.parse(payload)
+        envelope = PublicationBarrier.strip(JSON.parse(payload))
         if metadata["task_id"] == envelope["id"]
           return ClaimRecord.new(
             channel: envelope.fetch("channel"),
@@ -260,22 +284,33 @@ module SolidJobs
     end
 
     def claim_reliable(channels)
-      if @channel_order == :priority
-        channels.each do |source|
-          claim = claim_now(source)
-          return claim if claim
-        end
-        source = channels.first
-      else
-        source = next_channel(channels)
+      ordered_sources(channels).each do |source|
         claim = claim_now(source)
         return claim if claim
       end
-      payload = @redis_pool.blocking_call(
-        TIMEOUT,
-        "BLMOVE", source, @claimed_key, "RIGHT", "LEFT", TIMEOUT,
-      )
-      payload && record(payload, @channel_names.fetch(source))
+      sleep(TIMEOUT)
+      nil
+    end
+
+    def claim_unreliable(channels)
+      ordered_sources(channels).each do |source|
+        payload = pop_ready(source)
+        return record(payload, @channel_names.fetch(source)) if payload
+      end
+      sleep(TIMEOUT)
+      nil
+    end
+
+    def ordered_sources(channels)
+      case @channel_order
+      when :priority
+        channels.uniq
+      when :shuffle
+        channels.shuffle.uniq
+      else
+        preferred = next_channel(channels)
+        ([preferred] + channels).uniq
+      end
     end
 
     def claim_now(source)
@@ -285,7 +320,7 @@ module SolidJobs
         @redis_pool.call(
           "EVALSHA", RESERVE_SHA, 4,
           source, @claimed_key, @claims_key, Keyspace::ATTEMPTS,
-          claim_token, @identity, @processor_id, claimed_at,
+          claim_token, @identity, @processor_id, claimed_at, Keyspace::PUBLICATION_PREFIX,
         )
       rescue SolidRedis::CommandError => error
         raise unless error.message.include?("NOSCRIPT")
@@ -293,7 +328,7 @@ module SolidJobs
         @redis_pool.call(
           "EVAL", RESERVE, 4,
           source, @claimed_key, @claims_key, Keyspace::ATTEMPTS,
-          claim_token, @identity, @processor_id, claimed_at,
+          claim_token, @identity, @processor_id, claimed_at, Keyspace::PUBLICATION_PREFIX,
         )
       end
       return unless result
@@ -307,6 +342,20 @@ module SolidJobs
       )
     end
 
+    def pop_ready(source)
+      @redis_pool.call(
+        "EVALSHA", POP_READY_SHA, 1,
+        source, Keyspace::PUBLICATION_PREFIX,
+      )
+    rescue SolidRedis::CommandError => error
+      raise unless error.message.include?("NOSCRIPT")
+
+      @redis_pool.call(
+        "EVAL", POP_READY, 1,
+        source, Keyspace::PUBLICATION_PREFIX,
+      )
+    end
+
     def refresh_paused
       now = ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
       return if now < @paused_refresh_at