solid-jobs 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 8bbfb02b4261c0c6164bae65b6a47ebceda70a1dd0b3b59835e1a48a9c58bc97
+  data.tar.gz: f4306875809f7d8c5287ba9de5022ee1ac29da163a3448f0ed7de0492c9beccd
+SHA512:
+  metadata.gz: 71c1383a610c4d230f7db3b02f09664de638ff5009b22f3b44fe3b7dedbfe0f2e0aad5fb791c6816adea711b0e3e20766dc406b40e21cfe109ae429e8d07d7fd
+  data.tar.gz: d5e9beff88328ba42cc3a2bdf3167fd12acd7391b8c61e373664bc408b4055bcdb9fdb0915308b0fde7c521aea7527be63ba8e18d332458938a95a9401897832

data/CHANGELOG.md

@@ -0,0 +1,27 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+## [Unreleased]
+
+## [0.1.0] - 2026-09-30
+
+### Added
+
+- Ractor-oriented client, Processor, Scheduler, Heartbeat, and Server runtime.
+- Sidekiq-compatible Redis queue keys and open-source job payload format.
+- Immediate, scheduled, retry, and dead-job handling.
+- At-least-once reservations with atomic journaling, generation-fenced ACK
+  and requeue operations, and crashed-process recovery.
+- `SolidJobs::IntegrityCheck` for lost jobs, orphaned references, invalid
+  reservations, dangling indexes, and impossible duplicate states.
+- `SolidJobs::StartupBarrier`, which preconnects every component before
+  releasing Processors into their fetch loops.
+- Graceful quiet and shutdown handling with requeue of interrupted work.
+- Minitest unit, Redis integration, bounded stress, torture, startup-torture,
+  and soak suites.
+- Reliable-hot-path and CPU-scaling profilers plus the independent Sidekiq
+  comparison benchmark.
+
+[Unreleased]: https://github.com/nicolasva/solid-jobs/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/nicolasva/solid-jobs/releases/tag/v0.1.0

data/LICENSE.txt

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2026 Nicolas Vandenbogaerde
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

data/README.md

@@ -0,0 +1,159 @@
+# SolidJobs
+
+[![Build Status](https://github.com/nicolasva/solid-jobs/actions/workflows/ci.yml/badge.svg)](https://github.com/nicolasva/solid-jobs/actions/workflows/ci.yml)
+[![Code Climate](https://codeclimate.com/github/nicolasva/solid-jobs/badges/gpa.svg)](https://codeclimate.com/github/nicolasva/solid-jobs)
+[![Gem Version](https://badge.fury.io/rb/solid-jobs.svg)](https://rubygems.org/gems/solid-jobs)
+[![Documentation Status](https://img.shields.io/badge/docs-rubydoc.info-blue.svg)](https://www.rubydoc.info/gems/solid-jobs)
+[![Downloads](https://img.shields.io/gem/dt/solid-jobs.svg)](https://rubygems.org/gems/solid-jobs)
+
+SolidJobs is a Ractor-oriented Redis background job system for Ruby. It uses
+`solid-redis` for Redis access and keeps mutable clients, pools, middleware,
+and runtime state local to their owning Ractor.
+
+SolidJobs is an independent implementation. It does not depend on or load the
+Sidekiq gem. Its Redis job payloads and queue keys are designed to be
+compatible with the Sidekiq 8 open-source data format.
+
+SolidJobs and its required gems use pure Ruby and do not require native
+extensions. Hot paths are designed around bounded buffers, reusable immutable
+configuration, and low-allocation command batches.
+
+## Installation
+
+Add SolidJobs 0.1 to your bundle:
+
+```ruby
+gem "solid-jobs", "~> 0.1.0"
+```
+
+Then run:
+
+```sh
+bundle install
+```
+
+## Delivery semantics
+
+SolidJobs provides **at-least-once** job delivery. A worker atomically moves a
+job from `queue:<name>` to a process reservation list before execution and
+removes it only after a successful ACK. Graceful shutdown requeues unfinished
+jobs; reservations owned by a crashed process are recovered into their
+original queues.
+
+A process can still crash after the application side effect and before the
+ACK. The recovered job will then run again. Jobs must therefore be idempotent
+or implement an application-level deduplication key when duplicate side
+effects are unsafe. SolidJobs does not claim exactly-once execution.
+
+```ruby
+class HardJob
+  include SolidJobs::Job
+
+  solid_jobs_options queue: "critical", retry: 10
+
+  def perform(account_id)
+    Account.find(account_id).recalculate!
+  end
+end
+
+HardJob.perform_async(42)
+HardJob.perform_in(30, 42)
+```
+
+Run workers:
+
+```sh
+bundle exec solid-jobs --require ./config/environment \
+  --concurrency 8 --queue critical,3 --queue default
+```
+
+## Tests
+
+SolidJobs uses Minitest exclusively:
+
+```sh
+# Unit and bounded Redis integration tests
+bundle exec rake test
+
+# Bounded concurrency and multi-process recovery stress
+STRESS_JOBS=10000 bundle exec rake stress
+
+# Reproducible random-fault campaign
+SOLID_JOBS_TORTURE=1 STRESS_JOBS=100000 bundle exec rake torture
+
+# Re-run an exact failure sequence
+SOLID_JOBS_TORTURE=1 STRESS_JOBS=100000 STRESS_SEED=123456 bundle exec rake torture
+
+# Repeated fresh-process RESP/Ractor startup torture
+STARTUP_TORTURE_CYCLES=1000 \
+STARTUP_TORTURE_READERS=100 \
+bundle exec rake startup_torture
+
+# Long-running stability; defaults to 24 hours
+SOLID_JOBS_SOAK=1 SOLID_JOBS_SOAK_SECONDS=86400 bundle exec rake soak
+```
+
+The torture report reconciles enqueued, uniquely completed, duplicate,
+dead, queued, and reserved jobs. Any non-zero `LOST` value fails the test.
+
+Profile the reliable execution path independently from application work:
+
+```sh
+REDIS_URL=redis://127.0.0.1:6379/0 \
+HOT_PATH_JOBS=10000 \
+bundle exec rake benchmark:hot_path
+```
+
+The report separates time and allocations for reservation, payload reuse,
+observability registration, dispatch/perform wrapping, observability cleanup,
+and fenced ACK. Fetch decodes the payload once for both reservation metadata
+and dispatch; its job body is intentionally empty.
+
+Diagnose CPU scaling independently from Redis:
+
+```sh
+CPU_SCALING_JOBS=1000 \
+CPU_SCALING_ITERATIONS=210000 \
+bundle exec rake benchmark:cpu_scaling
+```
+
+This runs the identical CPU loop through pure Ractors and through the
+SolidJobs in-memory dispatch path. Each 1/2/4/8 case uses fresh processes and
+reports execution latency, scaling efficiency, CPU-seconds per 1,000 jobs,
+RSS, allocations, GC time, heap slots, and malloc growth.
+
+## Sidekiq comparison
+
+The separate `benchmark_sidekiq_solid-jobs` bundle runs Sidekiq and SolidJobs
+against the same isolated Redis server. It covers enqueue, bulk enqueue,
+CPU-bound processing, I/O-bound processing, mixed processing, and long-running
+stability. Every measurement runs in a fresh Ruby process; client order
+alternates and the default report uses the median of six repetitions.
+
+The current homogeneous CPU-processing reference uses Ruby 4.0.1,
+Sidekiq 8.1.7, a 210,000-iteration integer workload, and 1,000 jobs per case:
+
+| Concurrency | Sidekiq jobs/s | SolidJobs jobs/s | SolidJobs scaling | Sidekiq CPU-s/1k | SolidJobs CPU-s/1k | Sidekiq RSS | SolidJobs RSS | Sidekiq alloc/job | SolidJobs alloc/job |
+|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| 1 | 142 | 138 | 100.0% | 6.95 | 7.11 | 41.5 MiB | 72.0 MiB | 129.8 | 89.9 |
+| 2 | 144 | 272 | 98.7% | 6.94 | 7.16 | 41.8 MiB | 77.6 MiB | 117.5 | 81.3 |
+| 4 | 143 | 539 | 97.8% | 6.99 | 7.18 | 41.9 MiB | 71.4 MiB | 111.9 | 77.2 |
+| 8 | 144 | 957 | 86.7% | 6.94 | 8.01 | 42.2 MiB | 69.5 MiB | 108.7 | 75.4 |
+
+At eight concurrency units, SolidJobs reaches 957 jobs/s versus 144 jobs/s
+for one Sidekiq process. This is Ractor parallelism rather than equal CPU
+efficiency: SolidJobs consumes 766% CPU and 8.01 CPU-seconds per 1,000 jobs,
+while Sidekiq consumes 100% CPU and 6.94 CPU-seconds per 1,000 jobs. SolidJobs
+also uses more RSS, but reaches 13.77 jobs/s/MiB versus 3.41 for Sidekiq.
+
+These are local synthetic measurements, not application-capacity claims.
+Queue p95/p99 values in this run use sparse sampling and are excluded from the
+summary until the final latency campaign increases the sample count. Ruby
+3.4.4 eight-Ractor results are also excluded: concurrent TCP/RESP
+initialization triggered a reproducible native crash on the tested Apple
+Silicon environment. Ruby 4.0.1 passed the equivalent reproducer 100/100
+times, and `StartupBarrier` serializes component initialization before
+releasing normal parallel processing.
+
+The project is under active development. The Web UI and commercial Sidekiq
+features are not part of the initial scope.

data/docs/reliability.md

@@ -0,0 +1,114 @@
+# Reliability model
+
+SolidJobs uses a Redis-backed at-least-once state machine:
+
+```text
+READY
+  |
+  | BLMOVE reservation
+  v
+IN_PROGRESS ---- ACK/LREM ----> removed
+  |
+  +---- application error ----> RETRY or DEAD, then ACK
+  +---- graceful timeout -----> READY
+  +---- process crash --------> reservation retained
+                                  |
+                                  +---- recovery ----> READY
+```
+
+The reservation list is named `<process-identity>:reserved:<processor-id>`.
+Each execution receives a distinct reservation journal entry:
+
+```text
+job_id          stable across replay
+reservation_id  unique per execution attempt
+process_id      owning process identity
+worker_id       owning Processor Ractor
+attempt         monotonically increasing execution count
+reserved_at     wall-clock reservation time
+```
+
+ACK and requeue are fenced by `reservation_id`. Their Lua scripts first
+verify that the worker still owns the current journal generation. A delayed
+or revived worker cannot remove or requeue a newer reservation, even when a
+supervisor reuses the same processor slot.
+
+The payload retains its canonical `queue` field, allowing another process to
+restore it after the owner is no longer alive. Recovery uses process liveness
+and heartbeat, never reservation age alone, so a legitimate long-running job
+is not stolen while its owner remains alive.
+
+## Failure boundaries
+
+- Before reservation: the job remains in `queue:<name>`.
+- After reservation or during `perform`: the job remains reserved.
+- After the application effect but before ACK: recovery replays the job.
+- Redis unavailable during ACK: the job remains reserved and is replayed.
+- Graceful shutdown: the active job may finish within the configured timeout;
+  otherwise it is interrupted and requeued.
+- `SIGKILL`: no handler runs; recovery relies exclusively on Redis state.
+
+This design intentionally favors no job loss over duplicate suppression.
+Exactly-once side effects require application-level idempotency.
+
+## Startup isolation
+
+The server does not reserve work while components are booting. Heartbeat,
+Processor, and Scheduler Ractors initialize their local configuration and
+Redis pool, report `READY` exactly once, and wait behind
+`SolidJobs::StartupBarrier`. Processing starts only after every component is
+ready:
+
+```text
+BOOTING -> ALL_READY -> RUNNING
+    \-------> BOOT_FAILED -> cleanup
+```
+
+Boot failure is terminal. Already-ready components receive `:abort`, close
+their local resources, and never enter their fetch loops. Cleanup is
+idempotent. Component startup is serialized, avoiding concurrent TCP/RESP
+initialization paths known to crash Ruby 3.4.4 on the tested Apple Silicon
+environment; normal processing remains parallel after `RUNNING`.
+
+## Integrity auditing
+
+Fault and chaos tests can reconcile a known set of job IDs against every
+Redis-backed state:
+
+```ruby
+report = SolidJobs::IntegrityCheck.call(
+  expected_job_ids: submitted_job_ids,
+  acked_key: "test:completed",
+)
+
+raise report.inspect unless report.ok?
+```
+
+The report separates lost jobs, unexpected/orphaned jobs, inconsistent
+reservation journals, dangling attempt indexes, duplicate active states, and
+malformed payloads. ACK removes the completed job's attempt index atomically;
+requeue retains it so a recovered reservation increments the same attempt
+sequence.
+
+## Backpressure and queues
+
+Each Processor owns at most one reservation and reserves only immediately
+before execution. A process with concurrency `N` therefore holds at most `N`
+active reservations, regardless of Redis queue depth.
+
+Queue mode defaults to `:weighted`:
+
+- `:weighted` uses bounded weighted round-robin and does not starve configured
+  queues;
+- `:strict` always checks queues in declaration order and may intentionally
+  starve lower-priority queues while a higher-priority queue remains busy;
+- `:random` samples the configured weighted queue list on each reservation.
+
+Paused queues are excluded before reservation.
+
+## Retry storms
+
+Retries use capped exponential backoff with equal jitter. For attempt `n`, the
+ceiling is `min(retry_base_delay * 2**n, retry_max_delay)` and the actual delay
+is distributed between half and all of that ceiling. Defaults are 15 seconds
+and one hour. This spreads recovery traffic after a shared external outage.

data/exe/solid-jobs

@@ -0,0 +1,8 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "solid_jobs"
+require "solid_jobs/cli"
+
+exit SolidJobs::CLI.new.run
+

data/lib/active_job/queue_adapters/solid_jobs_adapter.rb

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+require "solid_jobs"
+require "solid_jobs/active_job"
+
+module ActiveJob
+  module QueueAdapters
+    class SolidJobsAdapter
+      def enqueue(job)
+        job.provider_job_id = push(job)
+      end
+
+      def enqueue_at(job, timestamp)
+        job.provider_job_id = push(job, at: timestamp)
+      end
+
+      def enqueue_all(jobs)
+        jobs.count do |job|
+          job.provider_job_id = job.scheduled_at ? enqueue_at(job, job.scheduled_at.to_f) : enqueue(job)
+        end
+      end
+
+      def stopping?
+        false
+      end
+
+      private
+
+      def push(job, at: nil)
+        payload = {
+          "class" => SolidJobs::ActiveJob::Wrapper,
+          "wrapped" => job.class.name,
+          "queue" => job.queue_name,
+          "args" => [job.serialize],
+        }
+        payload["at"] = at if at
+        SolidJobs::Client.push(payload)
+      end
+    end
+  end
+end
+

data/lib/solid_jobs/active_job.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+module SolidJobs
+  module ActiveJob
+    class Wrapper
+      include SolidJobs::Job
+      solid_jobs_options retry: true
+
+      def perform(job_data)
+        ::ActiveJob::Base.execute(job_data.merge("provider_job_id" => jid))
+      end
+    end
+    JobWrapper = Wrapper
+  end
+end
+