solid-jobs 0.1.3 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 52c2c82a111ab574881c9efe6f5c692c145b50ad85542844e34de74a44860a8e
-  data.tar.gz: 7a23ba9ea19c901c7c888680351b018b8f554cac0192a17195682aaa3725bd82
+  metadata.gz: 7f95c0daf98dbfa77203eaa6d182f0b0a5ce4c22738a29f7d81b587c9f3abe3a
+  data.tar.gz: bf5524aa572a45f439732ec30e5ad12d5ad25b37a98462381d39bd90b5dd7c91
 SHA512:
-  metadata.gz: 63447e81affab81cac25ed04aed7fa8cc8b27d89e5fd6425db087ed0feadc4d950266c35c16df6e91f27f5a5562844610516048ccbc4120f5132f9ab83456970
-  data.tar.gz: 3da40a61ca01cd8f868ebc0dd734b925ad0d46852517f34ea73b23cb8e9762c05b09db55caf218729a734975c579cb97d27253666142dbc694e0c5a018df427a
+  metadata.gz: cecbfb4efd3abe1598e66b1764ca07da25e4d103c5e59b66c14f84f5f8f3af6b45e91710d535b0b15ee69636a9396629a0e980451202f390a8aebb8a6b41091b
+  data.tar.gz: cf40a09b495d1ea62e0789ea8688b530b26bfc7414e77cd2d4cc30f8c874c1a8f6c19b20f2eb2c493c8ca27bbeb2a319a62024dff639d6748d720b1ab13c04d6

data/CHANGELOG.md

@@ -2,6 +2,73 @@
 
 All notable changes to this project will be documented in this file.
 
+## [0.4.0] - 2026-10-03
+
+### Changed
+
+- Keep the independent SolidJobs runtime, envelope, `solid_jobs:` keyspace,
+  interceptors, claims, and renamed component vocabulary from 0.3.0.
+- Remaining shared names with other Ruby job libraries are limited to
+  `version.rb`, Rails Active Job adapter contracts (`ActiveJob`,
+  `QueueAdapters`, `enqueue_at`), and generic Ruby methods.
+
+## [0.3.0] - 2026-10-03
+
+### Breaking
+
+- Rename the runtime components to SolidJobs-specific names:
+  `Config` to `Blueprint`, `Server` to `Conductor`, `Processor` to `Engine`,
+  `CLI` to `Console`, `Testing` to `Lab`, and `Metrics` to `Counters`.
+- Rename the task execution callback from `perform` to `execute_task`.
+- Rename `Task.enqueue_at` to `Task.schedule_at`. The Active Job adapter keeps
+  its Rails-required `enqueue_at` method.
+- Rename the Active Job execution task from `ActiveJob::Wrapper` to
+  `RailsAdapter::AdapterTask`.
+- Rename generic implementation files to match the new component vocabulary.
+  `version.rb` remains unchanged as the standard Ruby gem version entry point.
+- Remove the remaining generic internal constants `ClassMethods`,
+  `EmptyQueueError`, and `Shutdown` in favor of `TaskMethods`,
+  `NoCapturedTask`, and `ExecutionHalt`.
+- Replace `Node#quiet`, `Node#stop`, and `Node#dump_threads` with
+  `Node#request_control(:pause)`, `:shutdown`, or `:backtraces`.
+
+### Changed
+
+- Update the executable, Rails adapter, documentation, benchmarks, and test
+  suite to use the new vocabulary consistently.
+- Keep the SolidJobs task API, envelope, `solid_jobs:` Redis keyspace, claims,
+  interceptors, and failure model introduced in 0.2.0.
+- Convert configuration keys using Ruby hash transformations, preserving
+  nested values and leaving the original input unchanged.
+
+## [0.2.0] - 2026-10-03
+
+### Breaking
+
+- Replace the previous job API with the independent `SolidJobs::Task` API:
+  `enqueue`, `enqueue_after`, `schedule_at`, `enqueue_many`, `execute`, and
+  `with_options`.
+- Replace the previous payload with a SolidJobs envelope using `id`, `task`,
+  `arguments`, `channel`, `run_at`, `created_ms`, and `queued_ms`.
+- Move all Redis data into the `solid_jobs:` keyspace. Existing queued data is
+  not read or migrated automatically.
+- Replace middleware chains with `publish_interceptors` and
+  `execute_interceptors`, whose interceptors implement `around(context)`.
+- Replace the administrative API with `Counters`, `Channel`, `StoredTask`,
+  `PlannedTasks`, `RetryingTasks`, `DiscardedTasks`, `Node`, `Nodes`,
+  `Execution`, and `Claims`.
+- Replace queue configuration with channels and the `:weighted`, `:priority`,
+  and `:shuffle` ordering modes.
+- Remove all Sidekiq compatibility aliases and wire-format compatibility.
+
+### Changed
+
+- Claims use UUID task IDs and claim tokens, with generation-fenced completion
+  and requeue operations.
+- Failure handling uses `max_failures`, `retry_within`, `retry_delay`, and
+  `after_final_failure`.
+- Lab modes are now `:capture` and `:execute`.
+
 ## [0.1.3] - 2026-10-01
 
 ### Changed
@@ -13,9 +80,9 @@ All notable changes to this project will be documented in this file.
 
 ### Added
 
-- `Server#start` logs a warning on Ruby < 4 when `concurrency > 1`, pointing
+- `Conductor#start` logs a warning on Ruby < 4 when `concurrency > 1`, pointing
   to the Ruby 3.4 Ractor GC-barrier deadlock and the recommended setups.
-- `Server#stop` is bounded: components get `shutdown_timeout + STOP_GRACE`
+- `Conductor#stop` is bounded: components get `shutdown_timeout + STOP_GRACE`
   to return, `:stop` is re-sent up to `STOP_RESENDS` times, then the
   component is abandoned with an error log instead of hanging shutdown.
 - `test/support/ractor_barrier_repro.rb`: standalone reproducer of the Ruby
@@ -23,7 +90,7 @@ All notable changes to this project will be documented in this file.
 
 ### Changed
 
-- Server-based stress tests are skipped on Ruby < 4: Ruby 3.4's Ractor
+- Conductor-based stress tests are skipped on Ruby < 4: Ruby 3.4's Ractor
   scheduler deadlocks the VM on a GC barrier under cross-Ractor `move:`
   traffic. Multi-Ractor servers are recommended on Ruby ≥ 4.0 (see
   `docs/reliability.md`). CI jobs now time out after 20 minutes.
@@ -46,7 +113,7 @@ All notable changes to this project will be documented in this file.
 
 ### Added
 
-- Ractor-oriented client, Processor, Scheduler, Heartbeat, and Server runtime.
+- Ractor-oriented client, Engine, Scheduler, Heartbeat, and Conductor runtime.
 - Sidekiq-compatible Redis queue keys and open-source job payload format.
 - Immediate, scheduled, retry, and dead-job handling.
 - At-least-once reservations with atomic journaling, generation-fenced ACK
@@ -61,6 +128,8 @@ All notable changes to this project will be documented in this file.
 - Reliable-hot-path and CPU-scaling profilers plus the independent Sidekiq
   comparison benchmark.
 
+[0.3.0]: https://github.com/nicolasva/solid-jobs/compare/v0.2.0...v0.3.0
+[0.2.0]: https://github.com/nicolasva/solid-jobs/compare/v0.1.3...v0.2.0
 [0.1.3]: https://github.com/nicolasva/solid-jobs/compare/v0.1.2...v0.1.3
 [0.1.2]: https://github.com/nicolasva/solid-jobs/compare/v0.1.1...v0.1.2
 [0.1.1]: https://github.com/nicolasva/solid-jobs/compare/v0.1.0...v0.1.1

data/README.md

@@ -1,164 +1,195 @@
 # SolidJobs
 
 [![Build Status](https://github.com/nicolasva/solid-jobs/actions/workflows/ci.yml/badge.svg)](https://github.com/nicolasva/solid-jobs/actions/workflows/ci.yml)
-[![Code Climate](https://codeclimate.com/github/nicolasva/solid-jobs/badges/gpa.svg)](https://codeclimate.com/github/nicolasva/solid-jobs)
 [![Gem Version](https://badge.fury.io/rb/solid-jobs.svg)](https://rubygems.org/gems/solid-jobs)
-[![Documentation Status](https://img.shields.io/badge/docs-rubydoc.info-blue.svg)](https://www.rubydoc.info/gems/solid-jobs)
-[![Downloads](https://img.shields.io/gem/dt/solid-jobs.svg)](https://rubygems.org/gems/solid-jobs)
 
-SolidJobs is a Ractor-oriented Redis background job system for Ruby. It uses
-`solid-redis` for Redis access and keeps mutable clients, pools, middleware,
-and runtime state local to their owning Ractor.
+SolidJobs is a Ractor-oriented Redis task runner for Ruby. It provides its own
+task API, Redis envelope, keyspace, interceptor model, failure policy, and
+inspection API.
 
-SolidJobs is an independent implementation. It does not depend on or load the
-Sidekiq gem. Its Redis job payloads and queue keys are designed to be
-compatible with the Sidekiq 8 open-source data format.
-
-SolidJobs and its required gems use pure Ruby and do not require native
-extensions. Hot paths are designed around bounded buffers, reusable immutable
-configuration, and low-allocation command batches.
+SolidJobs is **not compatible with Sidekiq**. It does not read Sidekiq queues,
+does not write Sidekiq payloads, and does not expose Sidekiq's job API. Existing
+applications and queued work must be migrated explicitly.
 
 ## Installation
 
-Add SolidJobs 0.1 to your bundle:
-
 ```ruby
-gem "solid-jobs", "~> 0.1.0"
+gem "solid-jobs"
 ```
 
-Then run:
-
-```sh
-bundle install
-```
-
-## Delivery semantics
-
-SolidJobs provides **at-least-once** job delivery. A worker atomically moves a
-job from `queue:<name>` to a process reservation list before execution and
-removes it only after a successful ACK. Graceful shutdown requeues unfinished
-jobs; reservations owned by a crashed process are recovered into their
-original queues.
+Then run `bundle install`.
 
-A process can still crash after the application side effect and before the
-ACK. The recovered job will then run again. Jobs must therefore be idempotent
-or implement an application-level deduplication key when duplicate side
-effects are unsafe. SolidJobs does not claim exactly-once execution.
+## Defining and submitting tasks
 
 ```ruby
-class HardJob
-  include SolidJobs::Job
+class RecalculateAccount
+  include SolidJobs::Task
 
-  solid_jobs_options queue: "critical", retry: 10
+  task_options channel: "critical", max_failures: 10
 
-  def perform(account_id)
+  def execute_task(account_id)
     Account.find(account_id).recalculate!
   end
 end
 
-HardJob.perform_async(42)
-HardJob.perform_in(30, 42)
+RecalculateAccount.enqueue(42)
+RecalculateAccount.enqueue_after(30, 42)
+RecalculateAccount.schedule_at(Time.now + 300, 42)
+RecalculateAccount.enqueue_many([[42], [43], [44]])
+```
+
+The persisted envelope is specific to SolidJobs:
+
+```json
+{
+  "id": "7ec33ed5-1b77-4bde-8960-bd77231f10ef",
+  "task": "RecalculateAccount",
+  "arguments": [42],
+  "channel": "critical",
+  "max_failures": 10,
+  "created_ms": 1790981000000,
+  "queued_ms": 1790981000001
+}
 ```
 
-Run workers:
+All internal Redis keys use the `solid_jobs:` namespace. Ready work is stored
+under `solid_jobs:channel:<name>`; planned, retrying, discarded, node, claim,
+attempt, and metric data use separate namespaced keys.
+
+Run executors:
 
 ```sh
 bundle exec solid-jobs --require ./config/environment \
-  --concurrency 8 --queue critical,3 --queue default
+  --concurrency 8 --channel critical,3 --channel default
+```
+
+## Configuration
+
+```ruby
+SolidJobs.configure do |config|
+  config.channels = [["critical", 3], "default"]
+  config.channel_order = :weighted
+  config.concurrency = 8
+  config.shutdown_timeout = 25
+  config.default_task_options = {
+    channel: "default",
+    max_failures: 25,
+  }
+end
 ```
 
-## Tests
+`channel_order` accepts `:weighted` for bounded weighted round-robin,
+`:priority` for declaration-order priority, and `:shuffle` for random
+selection from the weighted channel list.
 
-SolidJobs uses Minitest exclusively:
+## Interceptors
 
-```sh
-# Unit and bounded Redis integration tests
-bundle exec rake test
+Publication and execution use SolidJobs interceptors. An interceptor implements
+`around(context)` and yields to continue:
 
-# Bounded concurrency and multi-process recovery stress
-STRESS_JOBS=10000 bundle exec rake stress
+```ruby
+class TraceExecution
+  def around(context)
+    Telemetry.start(context.envelope.fetch("id"))
+    yield
+  ensure
+    Telemetry.finish
+  end
+end
 
-# Reproducible random-fault campaign
-SOLID_JOBS_TORTURE=1 STRESS_JOBS=100000 bundle exec rake torture
+SolidJobs.configure do |config|
+  config.execute_interceptors.use(TraceExecution)
+end
+```
 
-# Re-run an exact failure sequence
-SOLID_JOBS_TORTURE=1 STRESS_JOBS=100000 STRESS_SEED=123456 bundle exec rake torture
+Publication interceptors receive `SolidJobs::Publisher::Publication`; execution
+interceptors receive `SolidJobs::Executor::Execution`.
 
-# Repeated fresh-process RESP/Ractor startup torture
-STARTUP_TORTURE_CYCLES=1000 \
-STARTUP_TORTURE_READERS=100 \
-bundle exec rake startup_torture
+## Failure handling
 
-# Long-running stability; defaults to 24 hours
-SOLID_JOBS_SOAK=1 SOLID_JOBS_SOAK_SECONDS=86400 bundle exec rake soak
+Tasks default to 25 failures. A task can customize the delay or final action:
+
+```ruby
+class ImportCatalog
+  include SolidJobs::Task
+
+  task_options max_failures: 8, retry_within: 3_600
+
+  retry_delay do |failure_count, error, envelope|
+    :drop if error.is_a?(InvalidCatalog)
+  end
+
+  after_final_failure do |envelope, error|
+    Alerts.catalog_import_failed(envelope.fetch("id"), error)
+  end
+end
 ```
 
-The torture report reconciles enqueued, uniquely completed, duplicate,
-dead, queued, and reserved jobs. Any non-zero `LOST` value fails the test.
+`retry_delay` may return a delay in seconds, `:drop`, or `:archive`. Without an
+override, SolidJobs uses capped exponential backoff with equal jitter.
+
+## Delivery semantics
 
-Profile the reliable execution path independently from application work:
+SolidJobs provides **at-least-once** delivery. An executor atomically claims a
+task before execution and completes the claim only after `execute_task` returns.
+Graceful shutdown requeues unfinished tasks. Claims owned by a crashed node are
+recovered into their original channels.
 
-```sh
-REDIS_URL=redis://127.0.0.1:6379/0 \
-HOT_PATH_JOBS=10000 \
-bundle exec rake benchmark:hot_path
+A process can still crash after an application side effect and before claim
+completion. The recovered task then runs again. Tasks must be idempotent or use
+application-level deduplication when duplicate effects are unsafe.
+
+See [docs/reliability.md](docs/reliability.md) for the state machine, claim
+fencing, recovery rules, and Ruby Ractor caveats.
+
+## Node control
+
+```ruby
+node = SolidJobs::Nodes.new.first
+node.request_control(:pause)
+node.request_control(:backtraces)
+node.request_control(:shutdown)
 ```
 
-The report separates time and allocations for reservation, payload reuse,
-observability registration, dispatch/perform wrapping, observability cleanup,
-and fenced ACK. Fetch decodes the payload once for both reservation metadata
-and dispatch; its job body is intentionally empty.
+Control requests are asynchronous and use the selected node's namespaced Redis
+mailbox. Unknown actions raise `KeyError` without writing a request.
 
-Diagnose CPU scaling independently from Redis:
+## Lab
+
+```ruby
+SolidJobs.testing!(:capture) do
+  RecalculateAccount.enqueue(42)
+  RecalculateAccount.captured
+end
+
+SolidJobs.testing!(:execute) do
+  RecalculateAccount.enqueue(42)
+end
+```
+
+Run the project suites:
 
 ```sh
-CPU_SCALING_JOBS=1000 \
-CPU_SCALING_ITERATIONS=210000 \
-bundle exec rake benchmark:cpu_scaling
+bundle exec rake test
+STRESS_JOBS=10000 bundle exec rake stress
+SOLID_JOBS_TORTURE=1 STRESS_JOBS=100000 bundle exec rake torture
+SOLID_JOBS_SOAK=1 SOLID_JOBS_SOAK_SECONDS=86400 bundle exec rake soak
 ```
 
-This runs the identical CPU loop through pure Ractors and through the
-SolidJobs in-memory dispatch path. Each 1/2/4/8 case uses fresh processes and
-reports execution latency, scaling efficiency, CPU-seconds per 1,000 jobs,
-RSS, allocations, GC time, heap slots, and malloc growth.
-
-## Sidekiq comparison
-
-The separate `benchmark_sidekiq_solid-jobs` bundle runs Sidekiq and SolidJobs
-against the same isolated Redis server. It covers enqueue, bulk enqueue,
-CPU-bound processing, I/O-bound processing, mixed processing, and long-running
-stability. Every measurement runs in a fresh Ruby process; client order
-alternates and the default report uses the median of six repetitions.
-
-The current homogeneous CPU-processing reference uses Ruby 4.0.1,
-Sidekiq 8.1.7, SolidJobs 0.1.2, a 100,000-iteration integer workload, and
-2,000 jobs per case:
-
-| Concurrency | Sidekiq jobs/s | SolidJobs jobs/s | SolidJobs scaling | Sidekiq CPU-s/1k | SolidJobs CPU-s/1k | Sidekiq RSS | SolidJobs RSS | Sidekiq alloc/job | SolidJobs alloc/job |
-|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
-| 1 | 289 | 281 | 100.0% | 3.35 | 3.40 | 41.4 MiB | 81.8 MiB | 117.2 | 84.0 |
-| 2 | 296 | 549 | 97.7% | 3.38 | 3.44 | 41.7 MiB | 83.1 MiB | 116.8 | 79.7 |
-| 4 | 298 | 1,090 | 97.0% | 3.37 | 3.43 | 41.8 MiB | 79.5 MiB | 111.1 | 77.7 |
-| 8 | 298 | 2,053 | 91.4% | 3.36 | 3.67 | 42.4 MiB | 74.1 MiB | 108.1 | 74.3 |
-
-At eight concurrency units, SolidJobs reaches 2,053 jobs/s versus 298 jobs/s
-for Sidekiq. This is Ractor parallelism rather than equal CPU efficiency:
-SolidJobs consumes 752.6% CPU and 3.67 CPU-seconds per 1,000 jobs, while
-Sidekiq consumes 100.2% CPU and 3.36 CPU-seconds per 1,000 jobs. SolidJobs
-also uses more RSS, but reaches 27.71 jobs/s/MiB versus 7.03 for Sidekiq.
-
-These are local synthetic measurements, not application-capacity claims.
-Queue p95/p99 values in this run use sparse sampling and are excluded from the
-summary until the final latency campaign increases the sample count. Ruby
-3.4.4 eight-Ractor results are also excluded: Ruby 3.4's Ractor scheduler
-crashed (concurrent TCP/RESP initialization, reproduced on macOS arm64 and
-Linux x86_64) or deadlocked VM-wide on a GC barrier under cross-Ractor
-message traffic (`test/support/ractor_barrier_repro.rb` reproduces it without
-SolidJobs).
-Ruby 4.0.1 passed the equivalent reproducers, and `StartupBarrier` serializes
-component initialization before releasing normal parallel processing.
-**Multi-Ractor servers are recommended on Ruby ≥ 4.0**; see
-`docs/reliability.md`.
-
-The project is under active development. The Web UI and commercial Sidekiq
-features are not part of the initial scope.
+## Migrating from Sidekiq
+
+There is no transparent migration path because compatibility is intentionally
+absent:
+
+1. Replace `include Sidekiq::Job` with `include SolidJobs::Task`.
+2. Replace `sidekiq_options` with `task_options`.
+3. Replace `perform_async`, `perform_in`, and `perform_bulk` with `enqueue`,
+   `enqueue_after`, and `enqueue_many`.
+4. Replace middleware with SolidJobs interceptors.
+5. Drain or export existing Sidekiq queues before switching. SolidJobs will not
+   consume them.
+6. Start SolidJobs with `--channel`; `--queue` is not accepted.
+
+The Active Job adapter remains available as
+`ActiveJob::QueueAdapters::SolidJobsAdapter`, but it writes only SolidJobs
+envelopes and keys.