solid-jobs 0.5.0 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7c1b35b5001c3c0467635a9fb46e3cab6305700a629c23babb604fc026895ec4
-  data.tar.gz: 441859a1e6880c259a61a8e9bf1c7004caf9c25d967335a322558ec7227da3e2
+  metadata.gz: 926751135fceec06e8dc722a4e69fc2397b5abdbccc2b555ec9ffddd064e30ca
+  data.tar.gz: a5e48fa286ed78aa8d568de6946ae139c3d1a1f125e5090b17062cca61f46d28
 SHA512:
-  metadata.gz: 0ece103959e94a949ad6c85e3591dbebcc3d82e495c9b4ab3d17f8729dd37bdaa591d12f5bdfe6adf8f4b7298078ef2c5326e1b36bffb3024d236a054980b999
-  data.tar.gz: 5f0e56b6900bec66f374f3446d4b553c0b84a934c06718abc77219286002b4b977de68072c3f524f295ceb0c0e81c5f4bd4b5b25a6b42776f791ddb2626614ab
+  metadata.gz: 4290b8368db3b6ae96393dd9437fdf39b98b996b7363a9e9929cffd75cb9539e393ddbcdfa6d37d01dc4a36eaa1439054792ecaa032d41341735c5cb76fcff25
+  data.tar.gz: 706dc9eae83cfee673b202bfe7ce098837f5726ef65206c62aaba5bc3c1a6cdd15cd9461a4061a07c525397c6da41a23d9f512064ff9170a31f8ba75039cccc1

data/CHANGELOG.md

@@ -2,6 +2,23 @@
 
 All notable changes to this project will be documented in this file.
 
+## [Unreleased]
+
+## [0.6.0] - 2026-10-19
+
+### Added
+
+- Real-process reliability tests for SolidEngine (worker SIGKILL, lost ACK, server
+  restart, clean stop); `Backends::Worker` now survives connection errors while
+  settling a job and lets the lease expire for redelivery.
+- Backend adapters: `Backends::Base` contract, `Backends::Redis` and
+  `Backends::SolidEngine`; `config.backends`, `config.channel_backends` and
+  per-channel routing in `Publisher` and `Conductor`.
+- `config.execution_mode` (`:ractor` default, or `:thread`): non-Redis backend
+  workers run in Ractors, each with its own client, so the backend no longer
+  determines CPU parallelism. Ruby 4 is recommended: on 3.4.4 Ractor CPU cost per
+  job rises sharply beyond 4 workers.
+
 ## [0.5.0] - 2026-10-12
 
 ### Added

data/README.md

@@ -177,6 +177,139 @@ application-level deduplication when duplicate effects are unsafe.
 See [docs/reliability.md](docs/reliability.md) for the state machine, claim
 fencing, recovery rules, and Ruby Ractor caveats.
 
+## SolidEngine backend (SolidRedis or SolidEngine)
+
+SolidJobs is the job API; the storage and execution engine sits behind a
+backend adapter. Two adapters ship with the gem and your tasks do not change:
+
+```
+Application → SolidJobs → Backend adapter ─┬─ Backends::Redis       → SolidRedis → Redis
+                                           └─ Backends::SolidEngine → Rust server (Unix socket / TCP)
+```
+
+```ruby
+SolidJobs.configure do |config|
+  config.backend = :redis                    # default for every channel
+
+  config.backends = {
+    redis:  {adapter: :solid_redis, url: "redis://localhost:6379"},
+    engine: {adapter: :solid_engine, url: "unix:///var/run/solidengine.sock"},
+    # TCP also works: "tcp://127.0.0.1:7777"
+  }
+
+  # Per-channel routing: run both engines side by side and migrate gradually.
+  config.channel_backends = {"critical" => :redis, "default" => :engine, "bulk" => :engine}
+  config.channels = %w[critical default bulk]
+end
+```
+
+Setting only `config.backend = :solid_engine` and `config.url = "unix://..."`
+routes every channel to SolidEngine. Without these options nothing changes:
+the Redis path is the unchanged default.
+
+```ruby
+class SendEmail
+  include SolidJobs::Task
+  task_options channel: "critical"
+  def execute_task(user_id) = # ...
+end
+SendEmail.enqueue(42)   # Redis or SolidEngine, decided by the channel
+```
+
+`Conductor` starts Ractor processors for channels served by Redis and, in
+the same process, `concurrency` workers per non-Redis backend (one connection
+each, lease keep-alive while a task runs). If every channel is
+routed to SolidEngine, no Ractor, heartbeat or Redis scheduler is started and
+Redis is not required.
+
+### Backend contract
+
+`SolidJobs::Backends::Base` defines `enqueue`, `schedule(at:)`, `claim(channel)`,
+`ack(claim)`, `retry_job(claim, delay:)`, `fail_job`, `get_job`, `recover`,
+`capabilities` and `supports?`. A *claim* is the object returned by `claim`; it
+carries the reservation and fencing tokens, so `ack` and `retry_job` take the
+claim rather than a bare job id. Delivery is at-least-once on both adapters.
+
+Guarantees are not hidden. Operations a backend cannot honor raise
+`SolidJobs::Backends::NotSupported`; check `backend.supports?(:capability)`:
+
+| Capability | SolidRedis | SolidEngine |
+|---|---|---|
+| At-least-once, claim/ack, retries, scheduling | yes | yes |
+| Crash recovery | yes (node heartbeat scan) | yes (server-side lease expiry) |
+| Deduplication by job id | no | yes (`push` of an existing id is idempotent) |
+| Fencing tokens, lease extension | claim-token check only | yes (stale `ack`/`nack` is rejected) |
+| `get_job` by id, `cancel` | no | yes |
+| Durability | Redis persistence settings | `memory`, `balanced` or `strict` (server config) |
+
+Job ids must be UUIDs on SolidEngine (the default). Channel names map to
+engine channel ids by CRC32 unless `channel_ids:` is set in the backend spec:
+pin the mapping in production. Retry/discard follows the task options
+(`max_failures`, `retry_within`, `retry_delay`, `after_final_failure`);
+`discarded` archives stay Redis-only, SolidEngine keeps dead jobs itself.
+
+### Switching a channel
+
+Changing a channel's backend **does not migrate jobs already queued** in the
+old engine. Drain the old queue first (keep a worker on it until empty), then
+switch producers, then workers. To roll back, keep a worker on the SolidEngine
+channel until `stats` shows it is empty. Jobs must be idempotent in both
+directions.
+
+### Execution mode (independent of the backend)
+
+```ruby
+SolidJobs.configure do |config|
+  config.backend = :solid_engine
+  config.execution_mode = :ractor   # default; or :thread
+  config.concurrency = 4
+end
+```
+
+- `:ractor` (default): each worker is a Ractor that builds **its own**
+  SolidEngine client (own socket, no shared state, no central relay), so CPU
+  tasks run in parallel exactly as with Redis.
+- `:thread`: workers are threads of the main Ractor, bound by the GVL. Use it as
+  an explicit fallback when a task depends on code that is not Ractor-compatible
+  (non-shareable globals, unsupported native extensions, closures as callbacks).
+
+Redis channels always use Ractor processors. The mode is never changed silently
+by the backend choice.
+
+### Reliability of the SolidEngine backend
+
+Guarantees (verified by `test/solid_engine_reliability_test.rb`, which uses real
+`solid-engine-server` and worker processes):
+
+- **At-least-once.** A task is acknowledged only after it succeeds. A worker that
+  dies (SIGKILL) mid-job loses its lease; the job is claimed again after
+  `lease_ms` and may run more than once. Exactly-once is **not** provided.
+- **Fencing.** A late ACK from a reservation that was reclaimed is rejected
+  (`stale_reservation`) and ignored by the worker.
+- **Connection errors.** An error while claiming or settling does not stop the
+  worker; an unsettled job is redelivered once its lease expires.
+- **Clean stop.** `stop` lets the active job finish and claims nothing new.
+- **Durability is a server setting.** With `durability = "memory"` (default) a
+  server restart loses every job. With `"strict"` or `"balanced"` and a
+  `data_dir`, confirmed jobs survive a SIGKILL of the server. The 60 s
+  retention of completed jobs is not a persistence guarantee.
+- **Recovery delay** is roughly `lease_ms` (default 30 s) after a worker crash.
+
+### Limits of this release
+
+The Python (`solid-jobs-py`) and Node.js (`solid-jobs-node`) packages contain
+the SolidEngine client but not yet this routing layer. `Catalog`, `Observations`,
+`Console` and `IntegrityCheck` still read Redis only.
+
+### SolidEngine benchmark
+
+Run `bundle exec rake comparison` in `benchmark_sidekiq_solid-jobs` with
+`SOLID_ENGINE_BIN` pointing at `solid-engine-server`; the `process-*`
+scenarios then add a `solid-jobs-engine` client (jobs through SolidEngine in
+`memory` durability, matching Redis without persistence; Redis only holds the
+benchmark counters). Results: see [Performance](#performance-solidjobs-vs-sidekiq)
+and [SolidEngine A/B on CPU tasks](#solidengine-ab-on-cpu-tasks-process-cpu).
+
 ## Node control
 
 ```ruby
@@ -266,14 +399,40 @@ reports `disconnected`. No monitoring-only Redis command is added. Redis
 latency, command, memory, and connection-count metrics remain explicitly
 `unavailable` until a reliable source exists.
 
-## Transparent Ractor-safe hybrid execution (warm worker routing)
+## Transparent Ractor-Safe Hybrid Execution Architecture
+
+Ractors enforce a strict shared-nothing model. Referencing mutable global
+state, legacy connection pools, or thread-based dependencies (**ActiveRecord**,
+**ActionMailer**, or gems built on `Mutex`, `Thread` and `Fiber`) inside a
+Ractor raises `ArgumentError: can't share mutable object`.
+
+SolidJobs handles this with an **adaptive router** in `enqueue`: tasks that are
+safe stay Ractor-native, and tasks that hold non-shareable state are
+transparently forwarded to a persistent **warm worker** process. You do not
+have to rewrite your gem stack.
+
+### How it works
+
+![SolidJobs hybrid execution: heuristic Ractor-compatibility detection routes to a Ractor worker (native) or a warm worker (fallback), communicating over a Unix socket with JSON](docs/images/hybrid-execution.svg)
+
+```text
+Task.enqueue(*args)
+   │
+   ├─ requires_warm_pool? = false ─► Ractor-native path (Redis → worker Ractors)
+   │
+   └─ requires_warm_pool? = true ──► JSON payload ─► SolidJobsGenericExecutor
+                                      ─► UNIXSocket ─► script/warm_worker.rb
+                                         (booted Rails process, runs the task)
+```
+
+| Context | Path | Typical tasks |
+|---|---|---|
+| Pure / stateless | Ractor-native, runs on all cores | CPU work, stateless I/O |
+| Holds a `Mutex`, `Thread`, `Fiber` or pool in class-level state | Warm worker over a Unix socket | Legacy gems, shared connection pools |
 
-Ractors enforce a shared-nothing model: referencing mutable global state, legacy
-connection pools, or thread-based dependencies (ActiveRecord, ActionMailer, gems
-using `Mutex`, `Thread` or `Fiber`) inside a Ractor raises
-`ArgumentError: can't share mutable object`. `SolidJobs::Task#enqueue` detects
-such tasks and reroutes them to a persistent **warm worker** process, so you do
-not have to rewrite them.
+The warm worker boots your application once and stays resident, so a routed job
+pays no per-job boot cost, only a local socket round trip. The latency has not
+been benchmarked.
 
 ### How routing is decided
 
@@ -299,7 +458,7 @@ as a `SolidJobsGenericExecutor` task, which forwards them over a Unix socket.
   connects to `/tmp/solid_jobs_rails_<worker_id>.sock`, sends
   `{"target_class": ..., "arguments": [...]}` as one JSON line, and returns the
   result or raises `SolidJobsGenericExecutor::RemoteError`. The worker is chosen
-  as `hash(task name) % SOLID_JOBS_WARM_WORKERS` (default `1`). Require this file
+  as `crc32(task name) % SOLID_JOBS_WARM_WORKERS` (default `1`, stable across processes). Require this file
   explicitly (it is loaded automatically on first routing).
 - `script/warm_worker.rb` — a standalone process, run outside any Ractor, that
   loads `config/environment.rb` from its working directory once, listens on the
@@ -326,7 +485,11 @@ The boot cost is paid once per worker; the process then stays warm. Memory is
 reclaimed only when a worker is restarted, so supervise and recycle them as you
 would any long-lived Rails process.
 
-### Example
+### Usage
+
+Include the task module as usual; the only requirement is that the class-level
+state that makes the task unsafe is visible to the detector, and that your
+arguments are JSON-serializable:
 
 ```ruby
 class SyncAccounts
@@ -343,156 +506,52 @@ SyncAccounts.enqueue([1, 2, 3]) # routed to the warm worker
 
 Tests live in `test/warm_pool_test.rb` and spawn a real worker process.
 
-## The Ractor concurrency challenge
-
-Core Rails frameworks like `ActiveRecord` and `ActionMailer` rely on global
-variables, shared connection pools, and mutable global state. Referencing these
-modules directly inside a strict, isolated `Ractor` raises
-`ArgumentError: can't share mutable object`.
-
-To keep SolidJobs' Ractor-native execution while preserving your existing Rails
-business logic, use a **hybrid execution bridge**: delegate mutable work to an
-isolated sub-process via `bin/rails runner`. The worker stays Ractor-native and
-the heavy Rails operations are offloaded to the operating system layer.
-
-### Hybrid execution examples
-
-> **Important:** These task classes must be loaded explicitly (e.g., inside
-> `config/boot.rb`) outside of Zeitwerk management, as standard Rails
-> autoloading is restricted inside isolated Ractors.
-
-#### Isolated ActionMailer delivery task
-
-Trigger an email workflow using your existing Rails mailers and templates
-without memory isolation crashes:
-
-```ruby
-# frozen_string_literal: true
-
-require "open3"
-
-class SendEmailTask
-  include SolidJobs::Task
-
-  # Define your processing channel and failure thresholds
-  task_options channel: "default", max_failures: 3
-
-  APP_ROOT = File.expand_path("../..", __dir__).freeze
-  SCRIPT = "WelcomeMailer.hello(ARGV[0]).deliver_now"
-
-  def execute_task(recipient = "nicolas.vandenbogaerde@gmail.com")
-    output, status = Open3.capture2e(
-      { "RAILS_ENV" => ENV.fetch("RAILS_ENV", "development") },
-      "bin/rails", "runner", SCRIPT, recipient,
-      chdir: APP_ROOT
-    )
-
-    unless status.success?
-      raise "Email delivery failed: #{output.lines.grep_v(/^\s+from /).first(5).join}"
-    end
-  end
-end
-```
-
-#### Isolated ActiveRecord data sync task
-
-Load database records and run model mutations safely outside of the Ractor:
-
-```ruby
-# frozen_string_literal: true
-
-require "open3"
-
-class ProcessUserDataTask
-  include SolidJobs::Task
-
-  # Dedicated data synchronization channel
-  task_options channel: "data_sync", max_failures: 3
-
-  APP_ROOT = File.expand_path("../..", __dir__).freeze
-  SCRIPT = "User.find(ARGV[0]).update_metrics!"
-
-  def execute_task(user_id)
-    output, status = Open3.capture2e(
-      { "RAILS_ENV" => ENV.fetch("RAILS_ENV", "development") },
-      "bin/rails", "runner", SCRIPT, user_id.to_s,
-      chdir: APP_ROOT
-    )
-
-    unless status.success?
-      raise "ActiveRecord task failed: #{output.lines.grep_v(/^\s+from /).first(5).join}"
-    end
-  end
-end
-```
-
-#### Complete business service object execution (zero-leak memory architecture)
-
-In complex production monoliths, a single service object often encapsulates
-multiple `ActiveRecord` updates, pricing mutations, external API calls, and
-email notifications. Running this orchestration layer directly inside a
-`Ractor` triggers memory isolation failures.
-
-By serializing the service input into a lightweight, Ractor-safe **JSON
-payload**, SolidJobs can hand the task details to an isolated `bin/rails runner`
-sub-process.
-
-A major benefit is **automatic memory recovery**: once the service has
-allocated thousands of transient Ruby objects, the sub-process terminates and
-the operating system reclaims all of its memory, keeping the worker immune to
-memory bloat and leaks.
-
-Example: processing a heavy order checkout service.
-
-```ruby
-# frozen_string_literal: true
-
-require "open3"
-require "json"
-
-# Loaded explicitly (e.g., in config/boot.rb) outside of Zeitwerk.
-# SolidJobs executors run within Ractors where standard Rails autoloading is restricted.
-class ExecuteServiceTask
-  include SolidJobs::Task
-
-  # Dedicated high-priority business services channel
-  task_options channel: "services", max_failures: 2
-
-  APP_ROOT = File.expand_path("../..", __dir__).freeze
-
-  # The runner script parses the input JSON from the command line and executes the service
-  SCRIPT = <<~RUBY
-    require 'json'
-    params = JSON.parse(ARGV[0])
-    OrderCheckoutService.call(user_id: params['user_id'], cart_id: params['cart_id'])
-  RUBY
-
-  def execute_task(params_hash)
-    # Serialize the pure Hash into a strict JSON string for safe Ractor transmission
-    json_payload = params_hash.to_json
-
-    output, status = Open3.capture2e(
-      { "RAILS_ENV" => ENV.fetch("RAILS_ENV", "development") },
-      "bin/rails", "runner", SCRIPT, json_payload,
-      chdir: APP_ROOT
-    )
-
-    unless status.success?
-      raise "Service object execution failed: #{output.lines.first(5).join}"
-    end
-  end
-end
-```
-
-Enqueuing from your Rails application:
-
-```ruby
-# Construct a clean, 100% Ractor-safe arguments structure
-task_payload = { user_id: 42, cart_id: 891 }
-
-# Push into the Redis pipeline
-ExecuteServiceTask.enqueue(task_payload)
-```
+### Hybrid architecture benchmark: Ractor vs Unix-socket IPC
+
+Run with `bundle exec rake hybrid` in `benchmark_sidekiq_solid-jobs`
+(Ruby 4.0.7, 4 Conductor Ractors, 4 warm workers, 10,000 mutable tasks per
+mode, all routed to the warm pool). Each latency series is stored as its own
+microsecond array and histogram, in JSON and Markdown under `results/`.
+
+- **Ractor pure:** time inside a Conductor Ractor between the `job.reserved`
+  event (payload returned by Redis) and `job.started` (just before the task
+  runs). SolidJobs reserves with an atomic Lua `RPOP`, not `BLPOP`, so this is
+  the closest equivalent.
+- **IPC roundtrip:** `UNIXSocket` open → status line read, reported by the
+  `warm_pool.roundtrip` instrumentation event of `SolidJobsGenericExecutor`.
+- `transport` uses zero-work warm tasks to isolate the pure socket cost;
+  `sustained` paces the 10,000 tasks linearly over 30 s; `burst` enqueues them
+  at once. In `sustained` and `burst`, each warm task sleeps 5 ms to simulate
+  service time, so their IPC roundtrip is dominated by that work (macOS timer
+  granularity makes it ≈6.4 ms).
+
+| Mode | Series | p50 (µs) | p95 (µs) | p99 (µs) |
+|---|---|---:|---:|---:|
+| Transport | Ractor pure | 9 | 21 | 39 |
+| Transport | IPC roundtrip (no work) | 43 | 77 | 192 |
+| Sustained 30 s (≈333 jobs/s) | Ractor pure | 25 | 51 | 119 |
+| Sustained 30 s (≈333 jobs/s) | IPC roundtrip (5 ms work) | 6,355 | 11,713 | 12,680 |
+| Burst | Ractor pure | 23 | 58 | 249 |
+| Burst | IPC roundtrip (5 ms work) | 6,426 | 12,522 | 13,183 |
+
+All 30,000 tasks completed with zero failures and **zero warm pool exhaustion
+events**: the 4 Conductor Ractors never have more requests in flight than there
+are warm workers. The pure socket overhead is tens of microseconds (p50 43 µs,
+p99 192 µs); the rest of the IPC latency is the task's own work in the warm
+worker.
+
+**Saturation (oversubscribed pool).** With 8 Ractors on a single warm worker, a
+30 ms timeout and 1,000 burst tasks, the pool is exhausted: 432 timeouts and 563
+refused connections (the worker's Unix socket listen backlog is full), only 5
+tasks completed on the first attempt, and the rest failed and went through
+SolidJobs retries. Exhaustion is counted as `timeout` + `refused` + `ok`
+roundtrips above `HYBRID_EXHAUSTION_MS`. Size the number of warm workers at
+least equal to the Conductor concurrency, and set `SOLID_JOBS_WARM_TIMEOUT`
+(default 30 s) accordingly. A timed-out request is not cancelled in the worker,
+so make routed tasks idempotent.
+
+*Environment: Ruby 4.0.7 (arm64-darwin27); SolidJobs 0.5.0; local loopback,
+synthetic work, single run per mode.*
 
 ## Commercial licensing & observability
 
@@ -527,7 +586,151 @@ scales linearly across cores:
 | 4 | 303 | 1,094 | 96.9% | 100.3% | 378.2% |
 | 8 | 302 | **2,170** | **96.2%** | 100.1% | **761.0%** |
 
-*At 8 concurrency, SolidJobs delivers **7.2× the throughput** of Sidekiq on pure CPU workloads.*
+*At 8 concurrency, SolidJobs delivers **7.2× the throughput** of a single Sidekiq process on pure CPU workloads
+(one process, N threads; see the multi-process comparison below).
+This table comes from the Ruby 4.x campaign (see the environment line at the end
+of this section); it predates the A/B campaign below and is not directly comparable with it.*
+
+### SolidEngine A/B on CPU tasks (`process-cpu`)
+
+Same tasks and repetitions for every client; median of 6 runs (min–max), jobs/s.
+Sidekiq is one process with N threads. Machine: 12 cores (8 performance).
+
+**Ruby 4.0.7** (separate campaign: `ruby-4.0.7-process-cpu-procs.md`)
+
+"Sidekiq (N procs)" is N Sidekiq processes with one thread each, i.e. the same
+number of cores as SolidJobs' N Ractors.
+
+| Concurrency | Sidekiq (1 proc, N threads) | Sidekiq (N procs) | SolidJobs + Redis (Ractors) | SolidJobs + SolidEngine (Ractors) |
+|---:|---:|---:|---:|---:|
+| 1 | 287 (283–288) | 286 (278–288) | 269 (260–271) | 286 (284–289) |
+| 2 | 297 (296–298) | 563 (561–564) | 516 (510–517) | **568** (566–570) |
+| 4 | 298 (295–300) | **1,107** (1,104–1,127) | 953 (947–964) | 1,103 (1,100–1,117) |
+| 8 | 299 (298–301) | **2,156** (2,145–2,161) | 1,667 (1,644–1,691) | 2,058 (2,049–2,086) |
+
+Ruby-worker CPU-seconds per 1,000 jobs: 3.4–3.6 for every client at every concurrency.
+
+Whole-infrastructure memory and CPU (separate run of the same workload,
+`ruby-4.0.7-process-cpu-memory.md`; peak physical footprint measured with macOS
+`footprint` during the run, workers + Redis + SolidEngine server):
+
+| Concurrency | Client | jobs/s | Workers MiB | Redis MiB | SolidEngine MiB | Total MiB | MiB per 1,000 jobs/s | Total CPU-s/1,000 jobs |
+|---:|---|---:|---:|---:|---:|---:|---:|---:|
+| 4 | Sidekiq (N procs) | 1,090 | 138.0 | 5.8 | – | **143.9** | 132 | 3.55 |
+| 4 | SolidJobs + Redis | 954 | 32.3 | 5.8 | – | **38.1** | 40 | 3.55 |
+| 4 | SolidJobs + SolidEngine | 1,101 | 54.7 | 5.8 | 6.0 | **66.6** | 61 | 3.51 |
+| 8 | Sidekiq (N procs) | 2,025 | 243.7 | 6.1 | – | **249.8** | 123 | 3.83 |
+| 8 | SolidJobs + Redis | 1,663 | 32.4 | 6.1 | – | **38.6** | 23 | 3.65 |
+| 8 | SolidJobs + SolidEngine | 1,968 | 55.0 | 6.1 | 6.4 | **67.6** | 34 | 3.71 |
+
+At 8 workers SolidJobs + SolidEngine uses about 73% less physical memory than 8
+Sidekiq processes (68 vs 250 MiB) for 3% less throughput in this run, with a
+similar total CPU per job. Redis holds only the benchmark counters here (about
+2,000 queued jobs), so its footprint is small; a real backlog would grow it.
+
+**Ruby 3.4.4**
+
+| Concurrency | Sidekiq (threads) | SolidJobs + Redis (Ractors) | SolidJobs + SolidEngine (Ractors) | SolidJobs + SolidEngine (threads) |
+|---:|---:|---:|---:|---:|
+| 1 | 202 (200–207) | 196 (192–199) | 209 (196–210) | 206 (205–208) |
+| 2 | 211 (209–213) | 296 (277–319) | **327** (310–337) | 205 (202–209) |
+| 4 | 210 (204–213) | **464** (281–497) | 401 (376–446) | 208 (203–209) |
+| 8 | 214 (211–216) | **313** (290–317) | 303 (299–329) | 209 (207–211) |
+
+CPU-seconds per 1,000 jobs on Ruby 3.4.4: 4.7–4.8 at concurrency 1; at concurrency 8
+Redis Ractors 24.3, Engine Ractors 25.8, Engine threads 4.8, Sidekiq 4.7.
+
+CPU covers the Ruby worker process only, not Redis or the SolidEngine server.
+
+Separate conclusions:
+
+1. **Execution model** (same SolidEngine backend): Ractors reach 2,058 jobs/s at
+   concurrency 8 on Ruby 4.0.7 against about 290 for threads (about 7×). The earlier
+   plateau was the GVL, not the server.
+2. **Backend** (both in Ractors, Ruby 4.0.7): SolidEngine leads Redis at every
+   concurrency with non-overlapping ranges (+10% at 2, +16% at 4, +23% at 8). On
+   Ruby 3.4.4 the two are comparable and ranges overlap.
+3. **SolidJobs vs Sidekiq:** against one Sidekiq process with N threads SolidJobs is
+   about 6.9× faster at concurrency 8, but that comparison uses one core for Sidekiq.
+   Against N Sidekiq processes (same cores) throughput is **on par**: Sidekiq is
+   within ±5% of SolidJobs + SolidEngine and ahead of SolidJobs + Redis at
+   concurrency 4–8. SolidJobs' measured advantage on this workload is memory
+   (one process with Ractors instead of N processes), not throughput.
+4. **Ruby 3.4.4:** throughput drops and CPU per job rises sharply at concurrency 8 for
+   both backends (Ractor coordination cost in the runtime); use Ruby 4.
+
+Not covered: Linux PSS (macOS `footprint` is used instead), memory under large
+backlogs, more than 8 performance cores, Ruby 3.4.4 with multi-process Sidekiq.
+Reports: `benchmark_sidekiq_solid-jobs/results/ruby-3.4.4-process-cpu-ab.md` and
+`ruby-4.0.7-process-cpu-procs.md`.
+
+### SolidEngine re-run (Ruby 4.0.7, all processing scenarios)
+
+Re-run with the Rust `solid-engine-server` (release build, `memory` durability):
+`process-cpu`, `process-io` and `process-mixed`, median of 6 runs, jobs/s, 0 errors
+everywhere. Sidekiq is one process with N threads; "Engine (threads)" is SolidJobs +
+SolidEngine with threads instead of Ractors.
+
+| Scenario | Concurrency | Sidekiq | SolidJobs + Redis | SolidJobs + SolidEngine | Engine (threads) |
+|---|---:|---:|---:|---:|---:|
+| `process-cpu` | 1 | 280 | 268 | **285** | 283 |
+| `process-cpu` | 2 | 292 | 516 | **565** | 288 |
+| `process-cpu` | 4 | 296 | 956 | **1,109** | 291 |
+| `process-cpu` | 8 | 292 | 1,635 | **2,021** | 286 |
+| `process-io` | 1 | 156 | 160 | **166** | 153 |
+| `process-io` | 2 | 308 | 300 | **322** | 300 |
+| `process-io` | 4 | 625 | 594 | **653** | 616 |
+| `process-io` | 8 | 1,221 | 1,076 | **1,242** | 1,200 |
+| `process-mixed` | 1 | 189 | 194 | **204** | 192 |
+| `process-mixed` | 2 | 402 | 396 | **426** | 398 |
+| `process-mixed` | 4 | 568 | 752 | **874** | 554 |
+| `process-mixed` | 8 | 572 | 1,471 | **1,763** | 552 |
+
+- CPU and mixed work: Ractors scale with SolidEngine (about 6.9× Sidekiq on
+  `process-cpu` and 3.1× on `process-mixed` at concurrency 8), while threads stay
+  GVL-bound at the Sidekiq level. SolidEngine is +20% to +24% ahead of Redis at
+  concurrency 8.
+- I/O work: all clients are within about 15% of each other because the wait
+  dominates; SolidEngine is on par with or slightly ahead of Sidekiq.
+- Footprint at `process-cpu`, concurrency 8 (workers / Redis / SolidEngine MiB):
+  Sidekiq 28.8 / 6.0 / –; SolidJobs + Redis 32.4 / 6.0 / –; SolidJobs + SolidEngine
+  54.8 / 6.0 / 7.3. Single process, N Ractors: the Sidekiq column is one process, not N.
+- Reports: `benchmark_sidekiq_solid-jobs/results/ruby-4.0.7-solid-engine.md` and
+  `.csv`. Command: `RBENV_VERSION=4.0.7 BENCHMARK_SCENARIOS=process-cpu,process-io,process-mixed bundle exec rake comparison`.
+
+### SolidRedis backend re-run (solid-redis 1.0.13, Ruby 4.0.7)
+
+Second run with the local `solid-redis` 1.0.13 checkout wired into the benchmark
+bundle (`SOLID_REDIS_PATH`, default `../solid-redis`; the previous run resolved
+1.0.12 from rubygems). Only Sidekiq and SolidJobs + SolidRedis (Ractors) were run
+(144 cases, median of 6, 0 errors). The SolidEngine column is the run above, for
+comparison.
+
+| Scenario | Concurrency | Sidekiq | SolidJobs + SolidRedis | SolidJobs + SolidEngine |
+|---|---:|---:|---:|---:|
+| `process-cpu` | 1 | 282 | 266 | **285** |
+| `process-cpu` | 2 | 281 | 490 | **565** |
+| `process-cpu` | 4 | 292 | 946 | **1,109** |
+| `process-cpu` | 8 | 291 | 1,603 | **2,021** |
+| `process-io` | 1 | 154 | 160 | **166** |
+| `process-io` | 2 | 307 | 302 | **322** |
+| `process-io` | 4 | 604 | 566 | **653** |
+| `process-io` | 8 | 1,172 | 998 | **1,242** |
+| `process-mixed` | 1 | 185 | 192 | **204** |
+| `process-mixed` | 2 | 400 | 397 | **426** |
+| `process-mixed` | 4 | 566 | 746 | **874** |
+| `process-mixed` | 8 | 573 | 1,468 | **1,763** |
+
+- SolidRedis scales like SolidEngine but stays 10–20% below it on CPU and mixed work
+  (concurrency 8: 1,603 vs 2,021 on `process-cpu`, 1,468 vs 1,763 on `process-mixed`),
+  and is still 5.5× Sidekiq (threads) on `process-cpu` and 2.6× on `process-mixed`.
+- On `process-io` SolidRedis is within about 15% of Sidekiq and below it at
+  concurrency 4–8 (998 vs 1,172 at 8); SolidEngine matches or beats Sidekiq.
+- The SolidEngine column comes from a separate run on the same machine, so
+  differences of a few percent are within run-to-run noise.
+- Reports: `benchmark_sidekiq_solid-jobs/results/ruby-4.0.7-solid-redis-1.0.13.md`
+  and `.csv`. Command:
+  `RBENV_VERSION=4.0.7 BENCHMARK_CLIENTS=sidekiq,solid-jobs BENCHMARK_SCENARIOS=process-cpu,process-io,process-mixed bundle exec rake comparison`.
 
 ### Mixed Workload (`process-mixed` — CPU + I/O)
 
@@ -549,6 +752,35 @@ scales linearly across cores:
 | 4 | 593 | **604** | 50 |
 | 8 | 1,121 | **1,133** | 52 |
 
+### Stress test: concurrency 4, 8 and 16
+
+Run with `bundle exec rake stress` (5,000 pre-seeded jobs per case, medians of
+3 runs; `STRESS_CONCURRENCY`, `STRESS_REPETITIONS` and `STRESS_JOBS` override
+the defaults). It writes a Markdown report and a CSV for plotting curves.
+
+| Workload | Concurrency | Sidekiq (jobs/s) | SolidJobs (jobs/s) | Ratio | Sidekiq CPU | SolidJobs CPU |
+|---|---:|---:|---:|---:|---:|---:|
+| `process-cpu` | 4 | 295 | 1,021 | 3.5× | 100% | 358% |
+| `process-cpu` | 8 | 288 | 1,916 | 6.7× | 100% | 683% |
+| `process-cpu` | 16 | 293 | 2,117 | 7.2× | 100% | 875% |
+| `process-mixed` | 4 | 557 | 792 | 1.4× | 99% | 144% |
+| `process-mixed` | 8 | 567 | 1,656 | 2.9× | 101% | 305% |
+| `process-mixed` | 16 | 568 | 3,073 | 5.4× | 101% | 590% |
+| `process-io` | 4 | 595 | 574 | 0.96× | 21% | 28% |
+| `process-io` | 8 | 1,177 | 1,053 | 0.89× | 31% | 43% |
+| `process-io` | 16 | 2,295 | 1,932 | 0.84× | 48% | 75% |
+
+- Sidekiq stays flat near one core on CPU and mixed work at every level, while
+  SolidJobs keeps scaling; CPU gains flatten past 8 because the machine runs out
+  of cores.
+- On pure I/O (sleep) work, Sidekiq's threads are slightly faster at these
+  levels and use less CPU. SolidJobs is not faster on I/O-bound jobs.
+- Latencies in the report measure the drain of the pre-seeded backlog, not
+  steady-state latency. Per-job execution p99 stayed under 10 ms for both.
+- Zero errors in all 54 measurements.
+
+*Environment: Ruby 4.0.7 (arm64-darwin27); Sidekiq 8.1.7; SolidJobs 0.5.0.*
+
 ### Client Enqueue Throughput
 
 | Concurrency | Sidekiq Enqueue (jobs/s) | SolidJobs Enqueue (jobs/s) | Sidekiq Bulk (jobs/s) | SolidJobs Bulk (jobs/s) |