solid-jobs 0.4.2 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7179d19681ddd3f238ba480d935c3a7e3829368ec011db4adfc571979d7174a9
-  data.tar.gz: e2669fd2af3814e7d1b2b6a58626dbbeee5ab3322067e99343b7722f186d7cd8
+  metadata.gz: 926751135fceec06e8dc722a4e69fc2397b5abdbccc2b555ec9ffddd064e30ca
+  data.tar.gz: a5e48fa286ed78aa8d568de6946ae139c3d1a1f125e5090b17062cca61f46d28
 SHA512:
-  metadata.gz: 8905446cb4bfa04ab5c5341e018e69dac330dfef9c89f0240c4b1857c83b233da4b894421400a28381fd18079fa84ace67505bee530ac9c270d25ca23a68287b
-  data.tar.gz: 235d3165daca0d685385afaa38df3b75612d5a53d4992d5585a55efd30cca21656f5cb350dad2ca0c204dd2080cfd8067fef12c33f6da92fe51c5d472a8c02e8
+  metadata.gz: 4290b8368db3b6ae96393dd9437fdf39b98b996b7363a9e9929cffd75cb9539e393ddbcdfa6d37d01dc4a36eaa1439054792ecaa032d41341735c5cb76fcff25
+  data.tar.gz: 706dc9eae83cfee673b202bfe7ce098837f5726ef65206c62aaba5bc3c1a6cdd15cd9461a4061a07c525397c6da41a23d9f512064ff9170a31f8ba75039cccc1

data/CHANGELOG.md

@@ -2,6 +2,31 @@
 
 All notable changes to this project will be documented in this file.
 
+## [Unreleased]
+
+## [0.6.0] - 2026-10-19
+
+### Added
+
+- Real-process reliability tests for SolidEngine (worker SIGKILL, lost ACK, server
+  restart, clean stop); `Backends::Worker` now survives connection errors while
+  settling a job and lets the lease expire for redelivery.
+- Backend adapters: `Backends::Base` contract, `Backends::Redis` and
+  `Backends::SolidEngine`; `config.backends`, `config.channel_backends` and
+  per-channel routing in `Publisher` and `Conductor`.
+- `config.execution_mode` (`:ractor` default, or `:thread`): non-Redis backend
+  workers run in Ractors, each with its own client, so the backend no longer
+  determines CPU parallelism. Ruby 4 is recommended: on 3.4.4 Ractor CPU cost per
+  job rises sharply beyond 4 workers.
+
+## [0.5.0] - 2026-10-12
+
+### Added
+
+- Warm worker routing: `enqueue` reroutes tasks holding non Ractor-safe state
+  (`Mutex`, `Thread`, `Fiber`, pools) to persistent Unix-socket workers
+  (`SolidJobsGenericExecutor`, `script/warm_worker.rb`).
+
 ## [0.4.2] - 2026-10-05
 
 ### Changed

data/README.md

@@ -3,6 +3,7 @@
 [![Build Status](https://github.com/nicolasva/solid-jobs/actions/workflows/ci.yml/badge.svg)](https://github.com/nicolasva/solid-jobs/actions/workflows/ci.yml)
 [![Gem Version](https://badge.fury.io/rb/solid-jobs.svg)](https://rubygems.org/gems/solid-jobs)
 [![Downloads](https://img.shields.io/gem/dt/solid-jobs?style=flat)](https://rubygems.org/gems/solid-jobs)
+[![Documentation Status](https://img.shields.io/badge/docs-RubyDoc.info-blue.svg)](https://www.rubydoc.info/gems/solid-jobs)
 
 SolidJobs is a Ractor-oriented Redis task runner for Ruby. It provides its own
 task API, Redis envelope, keyspace, interceptor model, failure policy, and
@@ -12,6 +13,39 @@ SolidJobs is **not compatible with Sidekiq**. It does not read Sidekiq queues,
 does not write Sidekiq payloads, and does not expose Sidekiq's job API. Existing
 applications and queued work must be migrated explicitly.
 
+## SolidTrace observability
+
+SolidJobs and SolidTrace have separate, complementary roles:
+
+- **SolidJobs** is the job runner. It enqueues and executes background tasks,
+  manages channels, retries, recovery, and worker Ractors.
+- **SolidTrace** is the observability dashboard. It consumes structured
+  lifecycle and operational telemetry emitted by SolidJobs and presents a
+  read-only view of job activity and system health.
+
+With SolidTrace, operators can inspect recent job executions and their
+observed lifecycle, monitor queues and worker Ractors, and review process
+resource usage, Redis health, telemetry freshness, and gaps that require
+resynchronization. SolidTrace does not enqueue, execute, retry, or control
+SolidJobs tasks; SolidJobs remains responsible for processing them. The
+dashboard is a separate Sinatra/Rack component that can be mounted in a Rails
+application, with host authentication and access protection supplied by that
+application.
+
+SolidTrace uses structured telemetry rather than job arguments or business
+payloads. Its trace views show retained lifecycle events and calculable
+durations, and identify missing or unavailable information instead of
+guessing. See the [SolidTrace project](https://github.com/nicolasva/SolidTrace)
+for more information.
+
+### SolidTrace dashboard
+
+![SolidTrace dashboard overview](https://github.com/nicolasva/SolidTrace/blob/main/file_00000000708882108cd8ae9a1106f601.jpg)
+
+![SolidTrace dashboard pages](https://github.com/nicolasva/SolidTrace/blob/main/SolidTrace%20Futuristic%20Dashboard%20Showcase.png)
+
+For access to SolidTrace, contact me at [nicolas.vandenbogaerde@gmail.com](mailto:nicolas.vandenbogaerde@gmail.com).
+
 ## Installation
 
 ```ruby
@@ -143,6 +177,139 @@ application-level deduplication when duplicate effects are unsafe.
 See [docs/reliability.md](docs/reliability.md) for the state machine, claim
 fencing, recovery rules, and Ruby Ractor caveats.
 
+## SolidEngine backend (SolidRedis or SolidEngine)
+
+SolidJobs is the job API; the storage and execution engine sits behind a
+backend adapter. Two adapters ship with the gem and your tasks do not change:
+
+```
+Application → SolidJobs → Backend adapter ─┬─ Backends::Redis       → SolidRedis → Redis
+                                           └─ Backends::SolidEngine → Rust server (Unix socket / TCP)
+```
+
+```ruby
+SolidJobs.configure do |config|
+  config.backend = :redis                    # default for every channel
+
+  config.backends = {
+    redis:  {adapter: :solid_redis, url: "redis://localhost:6379"},
+    engine: {adapter: :solid_engine, url: "unix:///var/run/solidengine.sock"},
+    # TCP also works: "tcp://127.0.0.1:7777"
+  }
+
+  # Per-channel routing: run both engines side by side and migrate gradually.
+  config.channel_backends = {"critical" => :redis, "default" => :engine, "bulk" => :engine}
+  config.channels = %w[critical default bulk]
+end
+```
+
+Setting only `config.backend = :solid_engine` and `config.url = "unix://..."`
+routes every channel to SolidEngine. Without these options nothing changes:
+the Redis path is the unchanged default.
+
+```ruby
+class SendEmail
+  include SolidJobs::Task
+  task_options channel: "critical"
+  def execute_task(user_id) = # ...
+end
+SendEmail.enqueue(42)   # Redis or SolidEngine, decided by the channel
+```
+
+`Conductor` starts Ractor processors for channels served by Redis and, in
+the same process, `concurrency` workers per non-Redis backend (one connection
+each, lease keep-alive while a task runs). If every channel is
+routed to SolidEngine, no Ractor, heartbeat or Redis scheduler is started and
+Redis is not required.
+
+### Backend contract
+
+`SolidJobs::Backends::Base` defines `enqueue`, `schedule(at:)`, `claim(channel)`,
+`ack(claim)`, `retry_job(claim, delay:)`, `fail_job`, `get_job`, `recover`,
+`capabilities` and `supports?`. A *claim* is the object returned by `claim`; it
+carries the reservation and fencing tokens, so `ack` and `retry_job` take the
+claim rather than a bare job id. Delivery is at-least-once on both adapters.
+
+Guarantees are not hidden. Operations a backend cannot honor raise
+`SolidJobs::Backends::NotSupported`; check `backend.supports?(:capability)`:
+
+| Capability | SolidRedis | SolidEngine |
+|---|---|---|
+| At-least-once, claim/ack, retries, scheduling | yes | yes |
+| Crash recovery | yes (node heartbeat scan) | yes (server-side lease expiry) |
+| Deduplication by job id | no | yes (`push` of an existing id is idempotent) |
+| Fencing tokens, lease extension | claim-token check only | yes (stale `ack`/`nack` is rejected) |
+| `get_job` by id, `cancel` | no | yes |
+| Durability | Redis persistence settings | `memory`, `balanced` or `strict` (server config) |
+
+Job ids must be UUIDs on SolidEngine (the default). Channel names map to
+engine channel ids by CRC32 unless `channel_ids:` is set in the backend spec:
+pin the mapping in production. Retry/discard follows the task options
+(`max_failures`, `retry_within`, `retry_delay`, `after_final_failure`);
+`discarded` archives stay Redis-only, SolidEngine keeps dead jobs itself.
+
+### Switching a channel
+
+Changing a channel's backend **does not migrate jobs already queued** in the
+old engine. Drain the old queue first (keep a worker on it until empty), then
+switch producers, then workers. To roll back, keep a worker on the SolidEngine
+channel until `stats` shows it is empty. Jobs must be idempotent in both
+directions.
+
+### Execution mode (independent of the backend)
+
+```ruby
+SolidJobs.configure do |config|
+  config.backend = :solid_engine
+  config.execution_mode = :ractor   # default; or :thread
+  config.concurrency = 4
+end
+```
+
+- `:ractor` (default): each worker is a Ractor that builds **its own**
+  SolidEngine client (own socket, no shared state, no central relay), so CPU
+  tasks run in parallel exactly as with Redis.
+- `:thread`: workers are threads of the main Ractor, bound by the GVL. Use it as
+  an explicit fallback when a task depends on code that is not Ractor-compatible
+  (non-shareable globals, unsupported native extensions, closures as callbacks).
+
+Redis channels always use Ractor processors. The mode is never changed silently
+by the backend choice.
+
+### Reliability of the SolidEngine backend
+
+Guarantees (verified by `test/solid_engine_reliability_test.rb`, which uses real
+`solid-engine-server` and worker processes):
+
+- **At-least-once.** A task is acknowledged only after it succeeds. A worker that
+  dies (SIGKILL) mid-job loses its lease; the job is claimed again after
+  `lease_ms` and may run more than once. Exactly-once is **not** provided.
+- **Fencing.** A late ACK from a reservation that was reclaimed is rejected
+  (`stale_reservation`) and ignored by the worker.
+- **Connection errors.** An error while claiming or settling does not stop the
+  worker; an unsettled job is redelivered once its lease expires.
+- **Clean stop.** `stop` lets the active job finish and claims nothing new.
+- **Durability is a server setting.** With `durability = "memory"` (default) a
+  server restart loses every job. With `"strict"` or `"balanced"` and a
+  `data_dir`, confirmed jobs survive a SIGKILL of the server. The 60 s
+  retention of completed jobs is not a persistence guarantee.
+- **Recovery delay** is roughly `lease_ms` (default 30 s) after a worker crash.
+
+### Limits of this release
+
+The Python (`solid-jobs-py`) and Node.js (`solid-jobs-node`) packages contain
+the SolidEngine client but not yet this routing layer. `Catalog`, `Observations`,
+`Console` and `IntegrityCheck` still read Redis only.
+
+### SolidEngine benchmark
+
+Run `bundle exec rake comparison` in `benchmark_sidekiq_solid-jobs` with
+`SOLID_ENGINE_BIN` pointing at `solid-engine-server`; the `process-*`
+scenarios then add a `solid-jobs-engine` client (jobs through SolidEngine in
+`memory` durability, matching Redis without persistence; Redis only holds the
+benchmark counters). Results: see [Performance](#performance-solidjobs-vs-sidekiq)
+and [SolidEngine A/B on CPU tasks](#solidengine-ab-on-cpu-tasks-process-cpu).
+
 ## Node control
 
 ```ruby
@@ -195,18 +362,208 @@ The Active Job adapter remains available as
 `ActiveJob::QueueAdapters::SolidJobsAdapter`, but it writes only SolidJobs
 envelopes and keys.
 
-## Monitoring with SolidTrace
+## Instrumentation
+
+SolidJobs exposes an optional Ractor-local instrumenter and defaults to a
+no-op. Any configured instrumenter must respond to `instrument(name, payload)`
+and be Ractor-shareable before a conductor starts. Ordinary instrumentation
+failures are logged when possible and never change publication, execution,
+retry, acknowledgement, recovery, or shutdown behavior. Process-control
+interruptions still propagate so shutdown can requeue in-flight work.
+
+SolidJobs emits the v1 lifecycle events `job.enqueued`, `job.journaled`,
+`job.reserved`, `job.started`, `job.completed`, `job.failed`,
+`job.retry_scheduled`, `job.dead`, `job.acknowledged`, and `job.recovered`.
+Ready jobs are not reservable until `job.enqueued` and `job.journaled` have
+been emitted; an expiring Redis publication barrier preserves this order
+without stranding work if a publisher exits.
+They carry correlation identifiers and structured errors, but never task
+arguments, business payloads, secrets, or textual logs. `node_id` is shared by
+all Ractors in a conductor; `ractor_id` and `worker_id` are integer processor
+identities. `reservation_id` is the reliable claim token.
+
+The heartbeat Ractor also samples and emits `process.observed`, one
+`ractor.observed` per logical processor, and one `redis.observed` per processor
+every five seconds. Process observations are limited to process CPU time, RSS,
+GC count/time, and total Ruby allocations. Sources are local Ruby/OS reads made
+outside worker Ractors. A source that has never succeeded is `unavailable`; a
+later failure preserves the last value as `stale` with its increasing age.
+SolidJobs never estimates CPU, memory, GC, or allocations per Ractor.
+
+Ractor observations expose only state, activity, integer
+`ractor_id`/`worker_id`/`executor_id`, and current job IDs. Quiet workers report
+`waiting`; shutdown reports `stopping` then `stopped` without inventing work.
+Redis connection state comes from worker operations already being performed:
+successful operations report `connected` and `SolidRedis::ConnectionError`
+reports `disconnected`. No monitoring-only Redis command is added. Redis
+latency, command, memory, and connection-count metrics remain explicitly
+`unavailable` until a reliable source exists.
+
+## Transparent Ractor-Safe Hybrid Execution Architecture
+
+Ractors enforce a strict shared-nothing model. Referencing mutable global
+state, legacy connection pools, or thread-based dependencies (**ActiveRecord**,
+**ActionMailer**, or gems built on `Mutex`, `Thread` and `Fiber`) inside a
+Ractor raises `ArgumentError: can't share mutable object`.
+
+SolidJobs handles this with an **adaptive router** in `enqueue`: tasks that are
+safe stay Ractor-native, and tasks that hold non-shareable state are
+transparently forwarded to a persistent **warm worker** process. You do not
+have to rewrite your gem stack.
+
+### How it works
+
+![SolidJobs hybrid execution: heuristic Ractor-compatibility detection routes to a Ractor worker (native) or a warm worker (fallback), communicating over a Unix socket with JSON](docs/images/hybrid-execution.svg)
+
+```text
+Task.enqueue(*args)
+   │
+   ├─ requires_warm_pool? = false ─► Ractor-native path (Redis → worker Ractors)
+   │
+   └─ requires_warm_pool? = true ──► JSON payload ─► SolidJobsGenericExecutor
+                                      ─► UNIXSocket ─► script/warm_worker.rb
+                                         (booted Rails process, runs the task)
+```
+
+| Context | Path | Typical tasks |
+|---|---|---|
+| Pure / stateless | Ractor-native, runs on all cores | CPU work, stateless I/O |
+| Holds a `Mutex`, `Thread`, `Fiber` or pool in class-level state | Warm worker over a Unix socket | Legacy gems, shared connection pools |
+
+The warm worker boots your application once and stays resident, so a routed job
+pays no per-job boot cost, only a local socket round trip. The latency has not
+been benchmarked.
+
+### How routing is decided
+
+`requires_warm_pool?(arguments)` returns `true` only when **all** of these hold:
+
+1. `defined?(Rails)` is truthy.
+2. At least one argument is not `Ractor.shareable?`.
+3. The task class holds, in a class-level instance variable, a `Mutex`,
+   `Thread` or `Fiber`, or an object whose class name contains `Pool`.
+
+Otherwise `enqueue` runs the normal Ractor-native path. When routed, the
+arguments are converted to plain JSON data (deep-copied and frozen) and enqueued
+as a `SolidJobsGenericExecutor` task, which forwards them over a Unix socket.
+
+> Detection is heuristic and inspects only class-level instance variables. It
+> does not detect constants such as `ActiveRecord::Base` referenced inside
+> `execute_task`. Arguments must be JSON-serializable, and a task with
+> shareable arguments always stays Ractor-native.
+
+### Components
+
+- `app/jobs/solid_jobs_generic_executor.rb` — `SolidJobsGenericExecutor`
+  connects to `/tmp/solid_jobs_rails_<worker_id>.sock`, sends
+  `{"target_class": ..., "arguments": [...]}` as one JSON line, and returns the
+  result or raises `SolidJobsGenericExecutor::RemoteError`. The worker is chosen
+  as `crc32(task name) % SOLID_JOBS_WARM_WORKERS` (default `1`, stable across processes). Require this file
+  explicitly (it is loaded automatically on first routing).
+- `script/warm_worker.rb` — a standalone process, run outside any Ractor, that
+  loads `config/environment.rb` from its working directory once, listens on the
+  socket, resolves the class with `Object.const_get`, runs `execute_task` (or
+  `perform`), and replies with `{"status":"ok","result":...}` or
+  `{"status":"error","error_class":...,"message":...}`. Requests are handled one
+  at a time per worker. The socket is created with mode `0600` and removed on
+  `INT`/`TERM`.
+
+### Running warm workers
+
+Warm workers are **not** spawned by `solid-jobs`; start and supervise them
+yourself (systemd, foreman, Procfile, ...), one per worker id, from the Rails
+root:
+
+```bash
+SOLID_JOBS_WARM_WORKERS=2 bundle exec ruby script/warm_worker.rb 0 &
+SOLID_JOBS_WARM_WORKERS=2 bundle exec ruby script/warm_worker.rb 1 &
+bundle exec solid-jobs --require ./config/environment --concurrency 8 --channel critical
+```
 
-**SolidTrace** is a statistics and observability tool for SolidJobs. It shows
-real-time SolidJobs metrics while your application runs: jobs per second, CPU
-and memory usage per Ractor, queue depth, Redis latency, job flow, and recent
-job executions.
+The worker count used by the runner must match the number of workers started.
+The boot cost is paid once per worker; the process then stays warm. Memory is
+reclaimed only when a worker is restarted, so supervise and recycle them as you
+would any long-lived Rails process.
 
-![SolidTrace dashboard](docs/images/solid-trace.jpg)
+### Usage
 
-SolidTrace is a commercial product. If you would like to acquire it, please
-contact me at [nicolas.vandenbogaerde@gmail.com](mailto:nicolas.vandenbogaerde@gmail.com)
-or via GitHub ([@nicolasva](https://github.com/nicolasva)).
+Include the task module as usual; the only requirement is that the class-level
+state that makes the task unsafe is visible to the detector, and that your
+arguments are JSON-serializable:
+
+```ruby
+class SyncAccounts
+  include SolidJobs::Task
+  @lock = Mutex.new # class-level non-shareable state => warm worker routing
+
+  def execute_task(account_ids)
+    Account.where(id: account_ids).find_each(&:sync!)
+  end
+end
+
+SyncAccounts.enqueue([1, 2, 3]) # routed to the warm worker
+```
+
+Tests live in `test/warm_pool_test.rb` and spawn a real worker process.
+
+### Hybrid architecture benchmark: Ractor vs Unix-socket IPC
+
+Run with `bundle exec rake hybrid` in `benchmark_sidekiq_solid-jobs`
+(Ruby 4.0.7, 4 Conductor Ractors, 4 warm workers, 10,000 mutable tasks per
+mode, all routed to the warm pool). Each latency series is stored as its own
+microsecond array and histogram, in JSON and Markdown under `results/`.
+
+- **Ractor pure:** time inside a Conductor Ractor between the `job.reserved`
+  event (payload returned by Redis) and `job.started` (just before the task
+  runs). SolidJobs reserves with an atomic Lua `RPOP`, not `BLPOP`, so this is
+  the closest equivalent.
+- **IPC roundtrip:** `UNIXSocket` open → status line read, reported by the
+  `warm_pool.roundtrip` instrumentation event of `SolidJobsGenericExecutor`.
+- `transport` uses zero-work warm tasks to isolate the pure socket cost;
+  `sustained` paces the 10,000 tasks linearly over 30 s; `burst` enqueues them
+  at once. In `sustained` and `burst`, each warm task sleeps 5 ms to simulate
+  service time, so their IPC roundtrip is dominated by that work (macOS timer
+  granularity makes it ≈6.4 ms).
+
+| Mode | Series | p50 (µs) | p95 (µs) | p99 (µs) |
+|---|---|---:|---:|---:|
+| Transport | Ractor pure | 9 | 21 | 39 |
+| Transport | IPC roundtrip (no work) | 43 | 77 | 192 |
+| Sustained 30 s (≈333 jobs/s) | Ractor pure | 25 | 51 | 119 |
+| Sustained 30 s (≈333 jobs/s) | IPC roundtrip (5 ms work) | 6,355 | 11,713 | 12,680 |
+| Burst | Ractor pure | 23 | 58 | 249 |
+| Burst | IPC roundtrip (5 ms work) | 6,426 | 12,522 | 13,183 |
+
+All 30,000 tasks completed with zero failures and **zero warm pool exhaustion
+events**: the 4 Conductor Ractors never have more requests in flight than there
+are warm workers. The pure socket overhead is tens of microseconds (p50 43 µs,
+p99 192 µs); the rest of the IPC latency is the task's own work in the warm
+worker.
+
+**Saturation (oversubscribed pool).** With 8 Ractors on a single warm worker, a
+30 ms timeout and 1,000 burst tasks, the pool is exhausted: 432 timeouts and 563
+refused connections (the worker's Unix socket listen backlog is full), only 5
+tasks completed on the first attempt, and the rest failed and went through
+SolidJobs retries. Exhaustion is counted as `timeout` + `refused` + `ok`
+roundtrips above `HYBRID_EXHAUSTION_MS`. Size the number of warm workers at
+least equal to the Conductor concurrency, and set `SOLID_JOBS_WARM_TIMEOUT`
+(default 30 s) accordingly. A timed-out request is not cancelled in the worker,
+so make routed tasks idempotent.
+
+*Environment: Ruby 4.0.7 (arm64-darwin27); SolidJobs 0.5.0; local loopback,
+synthetic work, single run per mode.*
+
+## Commercial licensing & observability
+
+`SolidJobs` is open source under the **LGPL-3.0-or-later** license.
+
+Running multi-core clusters blind in staging or production makes monitoring
+hard. For visual insight into Ractor memory allocation, CPU thresholds, and live
+queue states, pair this gem with **SolidTrace** (commercial licenses available
+from $50 to $200 / month).
+
+For commercial keys, custom architecture audits, or enterprise support, contact
+**licensing@solidtrace.io**.
 
 ## Performance: SolidJobs vs Sidekiq
 
@@ -229,7 +586,151 @@ scales linearly across cores:
 | 4 | 303 | 1,094 | 96.9% | 100.3% | 378.2% |
 | 8 | 302 | **2,170** | **96.2%** | 100.1% | **761.0%** |
 
-*At 8 concurrency, SolidJobs delivers **7.2× the throughput** of Sidekiq on pure CPU workloads.*
+*At 8 concurrency, SolidJobs delivers **7.2× the throughput** of a single Sidekiq process on pure CPU workloads
+(one process, N threads; see the multi-process comparison below).
+This table comes from the Ruby 4.x campaign (see the environment line at the end
+of this section); it predates the A/B campaign below and is not directly comparable with it.*
+
+### SolidEngine A/B on CPU tasks (`process-cpu`)
+
+Same tasks and repetitions for every client; median of 6 runs (min–max), jobs/s.
+Sidekiq is one process with N threads. Machine: 12 cores (8 performance).
+
+**Ruby 4.0.7** (separate campaign: `ruby-4.0.7-process-cpu-procs.md`)
+
+"Sidekiq (N procs)" is N Sidekiq processes with one thread each, i.e. the same
+number of cores as SolidJobs' N Ractors.
+
+| Concurrency | Sidekiq (1 proc, N threads) | Sidekiq (N procs) | SolidJobs + Redis (Ractors) | SolidJobs + SolidEngine (Ractors) |
+|---:|---:|---:|---:|---:|
+| 1 | 287 (283–288) | 286 (278–288) | 269 (260–271) | 286 (284–289) |
+| 2 | 297 (296–298) | 563 (561–564) | 516 (510–517) | **568** (566–570) |
+| 4 | 298 (295–300) | **1,107** (1,104–1,127) | 953 (947–964) | 1,103 (1,100–1,117) |
+| 8 | 299 (298–301) | **2,156** (2,145–2,161) | 1,667 (1,644–1,691) | 2,058 (2,049–2,086) |
+
+Ruby-worker CPU-seconds per 1,000 jobs: 3.4–3.6 for every client at every concurrency.
+
+Whole-infrastructure memory and CPU (separate run of the same workload,
+`ruby-4.0.7-process-cpu-memory.md`; peak physical footprint measured with macOS
+`footprint` during the run, workers + Redis + SolidEngine server):
+
+| Concurrency | Client | jobs/s | Workers MiB | Redis MiB | SolidEngine MiB | Total MiB | MiB per 1,000 jobs/s | Total CPU-s/1,000 jobs |
+|---:|---|---:|---:|---:|---:|---:|---:|---:|
+| 4 | Sidekiq (N procs) | 1,090 | 138.0 | 5.8 | – | **143.9** | 132 | 3.55 |
+| 4 | SolidJobs + Redis | 954 | 32.3 | 5.8 | – | **38.1** | 40 | 3.55 |
+| 4 | SolidJobs + SolidEngine | 1,101 | 54.7 | 5.8 | 6.0 | **66.6** | 61 | 3.51 |
+| 8 | Sidekiq (N procs) | 2,025 | 243.7 | 6.1 | – | **249.8** | 123 | 3.83 |
+| 8 | SolidJobs + Redis | 1,663 | 32.4 | 6.1 | – | **38.6** | 23 | 3.65 |
+| 8 | SolidJobs + SolidEngine | 1,968 | 55.0 | 6.1 | 6.4 | **67.6** | 34 | 3.71 |
+
+At 8 workers SolidJobs + SolidEngine uses about 73% less physical memory than 8
+Sidekiq processes (68 vs 250 MiB) for 3% less throughput in this run, with a
+similar total CPU per job. Redis holds only the benchmark counters here (about
+2,000 queued jobs), so its footprint is small; a real backlog would grow it.
+
+**Ruby 3.4.4**
+
+| Concurrency | Sidekiq (threads) | SolidJobs + Redis (Ractors) | SolidJobs + SolidEngine (Ractors) | SolidJobs + SolidEngine (threads) |
+|---:|---:|---:|---:|---:|
+| 1 | 202 (200–207) | 196 (192–199) | 209 (196–210) | 206 (205–208) |
+| 2 | 211 (209–213) | 296 (277–319) | **327** (310–337) | 205 (202–209) |
+| 4 | 210 (204–213) | **464** (281–497) | 401 (376–446) | 208 (203–209) |
+| 8 | 214 (211–216) | **313** (290–317) | 303 (299–329) | 209 (207–211) |
+
+CPU-seconds per 1,000 jobs on Ruby 3.4.4: 4.7–4.8 at concurrency 1; at concurrency 8
+Redis Ractors 24.3, Engine Ractors 25.8, Engine threads 4.8, Sidekiq 4.7.
+
+CPU covers the Ruby worker process only, not Redis or the SolidEngine server.
+
+Separate conclusions:
+
+1. **Execution model** (same SolidEngine backend): Ractors reach 2,058 jobs/s at
+   concurrency 8 on Ruby 4.0.7 against about 290 for threads (about 7×). The earlier
+   plateau was the GVL, not the server.
+2. **Backend** (both in Ractors, Ruby 4.0.7): SolidEngine leads Redis at every
+   concurrency with non-overlapping ranges (+10% at 2, +16% at 4, +23% at 8). On
+   Ruby 3.4.4 the two are comparable and ranges overlap.
+3. **SolidJobs vs Sidekiq:** against one Sidekiq process with N threads SolidJobs is
+   about 6.9× faster at concurrency 8, but that comparison uses one core for Sidekiq.
+   Against N Sidekiq processes (same cores) throughput is **on par**: Sidekiq is
+   within ±5% of SolidJobs + SolidEngine and ahead of SolidJobs + Redis at
+   concurrency 4–8. SolidJobs' measured advantage on this workload is memory
+   (one process with Ractors instead of N processes), not throughput.
+4. **Ruby 3.4.4:** throughput drops and CPU per job rises sharply at concurrency 8 for
+   both backends (Ractor coordination cost in the runtime); use Ruby 4.
+
+Not covered: Linux PSS (macOS `footprint` is used instead), memory under large
+backlogs, more than 8 performance cores, Ruby 3.4.4 with multi-process Sidekiq.
+Reports: `benchmark_sidekiq_solid-jobs/results/ruby-3.4.4-process-cpu-ab.md` and
+`ruby-4.0.7-process-cpu-procs.md`.
+
+### SolidEngine re-run (Ruby 4.0.7, all processing scenarios)
+
+Re-run with the Rust `solid-engine-server` (release build, `memory` durability):
+`process-cpu`, `process-io` and `process-mixed`, median of 6 runs, jobs/s, 0 errors
+everywhere. Sidekiq is one process with N threads; "Engine (threads)" is SolidJobs +
+SolidEngine with threads instead of Ractors.
+
+| Scenario | Concurrency | Sidekiq | SolidJobs + Redis | SolidJobs + SolidEngine | Engine (threads) |
+|---|---:|---:|---:|---:|---:|
+| `process-cpu` | 1 | 280 | 268 | **285** | 283 |
+| `process-cpu` | 2 | 292 | 516 | **565** | 288 |
+| `process-cpu` | 4 | 296 | 956 | **1,109** | 291 |
+| `process-cpu` | 8 | 292 | 1,635 | **2,021** | 286 |
+| `process-io` | 1 | 156 | 160 | **166** | 153 |
+| `process-io` | 2 | 308 | 300 | **322** | 300 |
+| `process-io` | 4 | 625 | 594 | **653** | 616 |
+| `process-io` | 8 | 1,221 | 1,076 | **1,242** | 1,200 |
+| `process-mixed` | 1 | 189 | 194 | **204** | 192 |
+| `process-mixed` | 2 | 402 | 396 | **426** | 398 |
+| `process-mixed` | 4 | 568 | 752 | **874** | 554 |
+| `process-mixed` | 8 | 572 | 1,471 | **1,763** | 552 |
+
+- CPU and mixed work: Ractors scale with SolidEngine (about 6.9× Sidekiq on
+  `process-cpu` and 3.1× on `process-mixed` at concurrency 8), while threads stay
+  GVL-bound at the Sidekiq level. SolidEngine is +20% to +24% ahead of Redis at
+  concurrency 8.
+- I/O work: all clients are within about 15% of each other because the wait
+  dominates; SolidEngine is on par with or slightly ahead of Sidekiq.
+- Footprint at `process-cpu`, concurrency 8 (workers / Redis / SolidEngine MiB):
+  Sidekiq 28.8 / 6.0 / –; SolidJobs + Redis 32.4 / 6.0 / –; SolidJobs + SolidEngine
+  54.8 / 6.0 / 7.3. Single process, N Ractors: the Sidekiq column is one process, not N.
+- Reports: `benchmark_sidekiq_solid-jobs/results/ruby-4.0.7-solid-engine.md` and
+  `.csv`. Command: `RBENV_VERSION=4.0.7 BENCHMARK_SCENARIOS=process-cpu,process-io,process-mixed bundle exec rake comparison`.
+
+### SolidRedis backend re-run (solid-redis 1.0.13, Ruby 4.0.7)
+
+Second run with the local `solid-redis` 1.0.13 checkout wired into the benchmark
+bundle (`SOLID_REDIS_PATH`, default `../solid-redis`; the previous run resolved
+1.0.12 from rubygems). Only Sidekiq and SolidJobs + SolidRedis (Ractors) were run
+(144 cases, median of 6, 0 errors). The SolidEngine column is the run above, for
+comparison.
+
+| Scenario | Concurrency | Sidekiq | SolidJobs + SolidRedis | SolidJobs + SolidEngine |
+|---|---:|---:|---:|---:|
+| `process-cpu` | 1 | 282 | 266 | **285** |
+| `process-cpu` | 2 | 281 | 490 | **565** |
+| `process-cpu` | 4 | 292 | 946 | **1,109** |
+| `process-cpu` | 8 | 291 | 1,603 | **2,021** |
+| `process-io` | 1 | 154 | 160 | **166** |
+| `process-io` | 2 | 307 | 302 | **322** |
+| `process-io` | 4 | 604 | 566 | **653** |
+| `process-io` | 8 | 1,172 | 998 | **1,242** |
+| `process-mixed` | 1 | 185 | 192 | **204** |
+| `process-mixed` | 2 | 400 | 397 | **426** |
+| `process-mixed` | 4 | 566 | 746 | **874** |
+| `process-mixed` | 8 | 573 | 1,468 | **1,763** |
+
+- SolidRedis scales like SolidEngine but stays 10–20% below it on CPU and mixed work
+  (concurrency 8: 1,603 vs 2,021 on `process-cpu`, 1,468 vs 1,763 on `process-mixed`),
+  and is still 5.5× Sidekiq (threads) on `process-cpu` and 2.6× on `process-mixed`.
+- On `process-io` SolidRedis is within about 15% of Sidekiq and below it at
+  concurrency 4–8 (998 vs 1,172 at 8); SolidEngine matches or beats Sidekiq.
+- The SolidEngine column comes from a separate run on the same machine, so
+  differences of a few percent are within run-to-run noise.
+- Reports: `benchmark_sidekiq_solid-jobs/results/ruby-4.0.7-solid-redis-1.0.13.md`
+  and `.csv`. Command:
+  `RBENV_VERSION=4.0.7 BENCHMARK_CLIENTS=sidekiq,solid-jobs BENCHMARK_SCENARIOS=process-cpu,process-io,process-mixed bundle exec rake comparison`.
 
 ### Mixed Workload (`process-mixed` — CPU + I/O)
 
@@ -251,6 +752,35 @@ scales linearly across cores:
 | 4 | 593 | **604** | 50 |
 | 8 | 1,121 | **1,133** | 52 |
 
+### Stress test: concurrency 4, 8 and 16
+
+Run with `bundle exec rake stress` (5,000 pre-seeded jobs per case, medians of
+3 runs; `STRESS_CONCURRENCY`, `STRESS_REPETITIONS` and `STRESS_JOBS` override
+the defaults). It writes a Markdown report and a CSV for plotting curves.
+
+| Workload | Concurrency | Sidekiq (jobs/s) | SolidJobs (jobs/s) | Ratio | Sidekiq CPU | SolidJobs CPU |
+|---|---:|---:|---:|---:|---:|---:|
+| `process-cpu` | 4 | 295 | 1,021 | 3.5× | 100% | 358% |
+| `process-cpu` | 8 | 288 | 1,916 | 6.7× | 100% | 683% |
+| `process-cpu` | 16 | 293 | 2,117 | 7.2× | 100% | 875% |
+| `process-mixed` | 4 | 557 | 792 | 1.4× | 99% | 144% |
+| `process-mixed` | 8 | 567 | 1,656 | 2.9× | 101% | 305% |
+| `process-mixed` | 16 | 568 | 3,073 | 5.4× | 101% | 590% |
+| `process-io` | 4 | 595 | 574 | 0.96× | 21% | 28% |
+| `process-io` | 8 | 1,177 | 1,053 | 0.89× | 31% | 43% |
+| `process-io` | 16 | 2,295 | 1,932 | 0.84× | 48% | 75% |
+
+- Sidekiq stays flat near one core on CPU and mixed work at every level, while
+  SolidJobs keeps scaling; CPU gains flatten past 8 because the machine runs out
+  of cores.
+- On pure I/O (sleep) work, Sidekiq's threads are slightly faster at these
+  levels and use less CPU. SolidJobs is not faster on I/O-bound jobs.
+- Latencies in the report measure the drain of the pre-seeded backlog, not
+  steady-state latency. Per-job execution p99 stayed under 10 ms for both.
+- Zero errors in all 54 measurements.
+
+*Environment: Ruby 4.0.7 (arm64-darwin27); Sidekiq 8.1.7; SolidJobs 0.5.0.*
+
 ### Client Enqueue Throughput
 
 | Concurrency | Sidekiq Enqueue (jobs/s) | SolidJobs Enqueue (jobs/s) | Sidekiq Bulk (jobs/s) | SolidJobs Bulk (jobs/s) |

data/docs/reliability.md

@@ -29,6 +29,11 @@ attempt      monotonically increasing execution count
 claimed_at   wall-clock claim time
 ```
 
+A ready payload has a short-lived publication barrier while its
+`job.enqueued` and `job.journaled` events are emitted. Executors leave marked
+payloads in READY until the publisher removes the marker. Markers expire after
+30 seconds, so a publisher crash can delay a task but cannot strand it.
+
 Completion and requeue are fenced by `claim_token`. Their Lua scripts verify
 that the executor still owns the current journal generation before changing
 state. A delayed or revived executor cannot complete or requeue a newer claim.