solid-jobs 0.1.1 → 0.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ce7c73e437bf4a78efbf9422e4a5cb7cf05635b5f3c9cbca4cd44a9baa0ee2d9
-  data.tar.gz: f0b4513d2c4d75bc9f871ba2c64c7848789747cf797e32044d56eefdab961647
+  metadata.gz: 52c2c82a111ab574881c9efe6f5c692c145b50ad85542844e34de74a44860a8e
+  data.tar.gz: 7a23ba9ea19c901c7c888680351b018b8f554cac0192a17195682aaa3725bd82
 SHA512:
-  metadata.gz: e6d245f323c4cdf117f2bb53584eed2e2c697a72fdc170792954d25feccb15b32d2cae7f3d8a337e7b29c3fedb071ab4ad9b773ac414fe893e389754532ca5af
-  data.tar.gz: 9abce08d1c4fb800a5f39e56fcebc266a7cc1dd52acd891fdaccf9fd1f2840ead4a3c4ce0a5649acc6c27a8f26ad5554d0599a5eb55f8e742afc568a307ddc09
+  metadata.gz: 63447e81affab81cac25ed04aed7fa8cc8b27d89e5fd6425db087ed0feadc4d950266c35c16df6e91f27f5a5562844610516048ccbc4120f5132f9ab83456970
+  data.tar.gz: 3da40a61ca01cd8f868ebc0dd734b925ad0d46852517f34ea73b23cb8e9762c05b09db55caf218729a734975c579cb97d27253666142dbc694e0c5a018df427a

data/CHANGELOG.md

@@ -2,7 +2,31 @@
 
 All notable changes to this project will be documented in this file.
 
-## [Unreleased]
+## [0.1.3] - 2026-10-01
+
+### Changed
+
+- Require `solid-redis` 1.0.11 and rely on its `solid-resp-ractor` dependency
+  instead of declaring and loading the RESP codec directly.
+
+## [0.1.2] - 2026-10-01
+
+### Added
+
+- `Server#start` logs a warning on Ruby < 4 when `concurrency > 1`, pointing
+  to the Ruby 3.4 Ractor GC-barrier deadlock and the recommended setups.
+- `Server#stop` is bounded: components get `shutdown_timeout + STOP_GRACE`
+  to return, `:stop` is re-sent up to `STOP_RESENDS` times, then the
+  component is abandoned with an error log instead of hanging shutdown.
+- `test/support/ractor_barrier_repro.rb`: standalone reproducer of the Ruby
+  3.4 Ractor GC-barrier deadlock (no SolidJobs, no Redis).
+
+### Changed
+
+- Server-based stress tests are skipped on Ruby < 4: Ruby 3.4's Ractor
+  scheduler deadlocks the VM on a GC barrier under cross-Ractor `move:`
+  traffic. Multi-Ractor servers are recommended on Ruby ≥ 4.0 (see
+  `docs/reliability.md`). CI jobs now time out after 20 minutes.
 
 ## [0.1.1] - 2026-10-01
 
@@ -37,5 +61,7 @@ All notable changes to this project will be documented in this file.
 - Reliable-hot-path and CPU-scaling profilers plus the independent Sidekiq
   comparison benchmark.
 
-[Unreleased]: https://github.com/nicolasva/solid-jobs/compare/v0.1.0...HEAD
+[0.1.3]: https://github.com/nicolasva/solid-jobs/compare/v0.1.2...v0.1.3
+[0.1.2]: https://github.com/nicolasva/solid-jobs/compare/v0.1.1...v0.1.2
+[0.1.1]: https://github.com/nicolasva/solid-jobs/compare/v0.1.0...v0.1.1
 [0.1.0]: https://github.com/nicolasva/solid-jobs/releases/tag/v0.1.0

data/README.md

@@ -131,29 +131,34 @@ stability. Every measurement runs in a fresh Ruby process; client order
 alternates and the default report uses the median of six repetitions.
 
 The current homogeneous CPU-processing reference uses Ruby 4.0.1,
-Sidekiq 8.1.7, a 210,000-iteration integer workload, and 1,000 jobs per case:
+Sidekiq 8.1.7, SolidJobs 0.1.2, a 100,000-iteration integer workload, and
+2,000 jobs per case:
 
 | Concurrency | Sidekiq jobs/s | SolidJobs jobs/s | SolidJobs scaling | Sidekiq CPU-s/1k | SolidJobs CPU-s/1k | Sidekiq RSS | SolidJobs RSS | Sidekiq alloc/job | SolidJobs alloc/job |
 |---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
-| 1 | 142 | 138 | 100.0% | 6.95 | 7.11 | 41.5 MiB | 72.0 MiB | 129.8 | 89.9 |
-| 2 | 144 | 272 | 98.7% | 6.94 | 7.16 | 41.8 MiB | 77.6 MiB | 117.5 | 81.3 |
-| 4 | 143 | 539 | 97.8% | 6.99 | 7.18 | 41.9 MiB | 71.4 MiB | 111.9 | 77.2 |
-| 8 | 144 | 957 | 86.7% | 6.94 | 8.01 | 42.2 MiB | 69.5 MiB | 108.7 | 75.4 |
+| 1 | 289 | 281 | 100.0% | 3.35 | 3.40 | 41.4 MiB | 81.8 MiB | 117.2 | 84.0 |
+| 2 | 296 | 549 | 97.7% | 3.38 | 3.44 | 41.7 MiB | 83.1 MiB | 116.8 | 79.7 |
+| 4 | 298 | 1,090 | 97.0% | 3.37 | 3.43 | 41.8 MiB | 79.5 MiB | 111.1 | 77.7 |
+| 8 | 298 | 2,053 | 91.4% | 3.36 | 3.67 | 42.4 MiB | 74.1 MiB | 108.1 | 74.3 |
 
-At eight concurrency units, SolidJobs reaches 957 jobs/s versus 144 jobs/s
-for one Sidekiq process. This is Ractor parallelism rather than equal CPU
-efficiency: SolidJobs consumes 766% CPU and 8.01 CPU-seconds per 1,000 jobs,
-while Sidekiq consumes 100% CPU and 6.94 CPU-seconds per 1,000 jobs. SolidJobs
-also uses more RSS, but reaches 13.77 jobs/s/MiB versus 3.41 for Sidekiq.
+At eight concurrency units, SolidJobs reaches 2,053 jobs/s versus 298 jobs/s
+for Sidekiq. This is Ractor parallelism rather than equal CPU efficiency:
+SolidJobs consumes 752.6% CPU and 3.67 CPU-seconds per 1,000 jobs, while
+Sidekiq consumes 100.2% CPU and 3.36 CPU-seconds per 1,000 jobs. SolidJobs
+also uses more RSS, but reaches 27.71 jobs/s/MiB versus 7.03 for Sidekiq.
 
 These are local synthetic measurements, not application-capacity claims.
 Queue p95/p99 values in this run use sparse sampling and are excluded from the
 summary until the final latency campaign increases the sample count. Ruby
-3.4.4 eight-Ractor results are also excluded: concurrent TCP/RESP
-initialization triggered a reproducible native crash on the tested Apple
-Silicon environment. Ruby 4.0.1 passed the equivalent reproducer 100/100
-times, and `StartupBarrier` serializes component initialization before
-releasing normal parallel processing.
+3.4.4 eight-Ractor results are also excluded: Ruby 3.4's Ractor scheduler
+crashed (concurrent TCP/RESP initialization, reproduced on macOS arm64 and
+Linux x86_64) or deadlocked VM-wide on a GC barrier under cross-Ractor
+message traffic (`test/support/ractor_barrier_repro.rb` reproduces it without
+SolidJobs).
+Ruby 4.0.1 passed the equivalent reproducers, and `StartupBarrier` serializes
+component initialization before releasing normal parallel processing.
+**Multi-Ractor servers are recommended on Ruby ≥ 4.0**; see
+`docs/reliability.md`.
 
 The project is under active development. The Web UI and commercial Sidekiq
 features are not part of the initial scope.

data/docs/reliability.md

@@ -71,6 +71,33 @@ initialization paths known to crash Ruby 3.4 (reproduced on 3.4.4 macOS arm64
 and 3.4.11 Linux x86_64; `rake startup_torture` is the reproducer); normal
 processing remains parallel after `RUNNING`.
 
+### Ruby 3.4 Ractor caveat
+
+Ruby 3.4's Ractor scheduler can deadlock the whole VM (main thread included)
+when a GC-triggered `rb_ractor_sched_barrier_start` runs while several
+Ractors exchange `move: true` messages: every thread parks in
+`ractor_sched_barrier_join_wait_locked` and the barrier never completes.
+`test/support/ractor_barrier_repro.rb` reproduces it **without SolidJobs or
+Redis** (one receiver, four senders, 24k moved messages per iteration):
+Ruby 3.4.4 freezes within the first iterations, Ruby 4.0.1 completes 30/30.
+Inside SolidJobs the same traffic pattern is the Processor → Heartbeat
+`:work`/`:done`/`:stats` channel, so any multi-Processor server on Ruby 3.4
+is exposed; once frozen, neither `Timeout` nor process exit
+(`rb_ractor_terminate_all`) can recover.
+
+Recommendation: **run multi-Ractor SolidJobs servers on Ruby ≥ 4.0**. On
+Ruby 3.4 use the client/API side freely, and prefer one process per
+Processor (`concurrency: 1`) for the server. Server-based stress tests are
+skipped on Ruby < 4 for this reason, and `Server#start` logs a warning when
+it detects `concurrency > 1` on Ruby < 4.
+
+### Bounded shutdown
+
+`Server#stop` never waits forever for a component. Each Ractor gets
+`shutdown_timeout + Server::STOP_GRACE` to return after `:stop`; the signal
+is re-sent up to `STOP_RESENDS` times, then the component is abandoned with
+an error log so the process can proceed with shutdown.
+
 ## Configuration scope
 
 `SolidJobs.config` and the testing mode are Ractor-local, not thread-local.

data/lib/solid_jobs/server.rb

@@ -2,13 +2,20 @@
 
 require "securerandom"
 require "socket"
+require "timeout"
 
 module SolidJobs
   class Server
+    # Extra time granted beyond shutdown_timeout before a component is
+    # considered stuck, and how many times :stop is re-sent before giving up.
+    STOP_GRACE = 5.0
+    STOP_RESENDS = 3
+
     attr_reader :config, :identity, :ractors
 
-    def initialize(config: SolidJobs.config)
+    def initialize(config: SolidJobs.config, stop_grace: STOP_GRACE)
       @config = config
+      @stop_grace = stop_grace
       @ractors = []
       @scheduler = nil
       @heartbeat = nil
@@ -20,6 +27,7 @@ module SolidJobs
     def start
       raise Error, "Server is already running" if @started
 
+      warn_ruby34_multi_ractor
       snapshot = Utilities.shareable_copy(
         config.ractor_snapshot.merge(identity: identity, started_at: @started_at),
       )
@@ -197,10 +205,12 @@ module SolidJobs
 
       @ractors.each { |ractor| ractor.send(:stop) }
       @scheduler&.send(:stop)
-      results = @ractors.map { |ractor| RactorSupport.value(ractor) }
-      RactorSupport.value(@scheduler) if @scheduler
+      results = @ractors.each_with_index.map do |ractor, index|
+        await_termination(ractor, "processor #{index}")
+      end
+      await_termination(@scheduler, "scheduler") if @scheduler
       @heartbeat&.send(:stop)
-      RactorSupport.value(@heartbeat) if @heartbeat
+      await_termination(@heartbeat, "heartbeat") if @heartbeat
       @ractors = []
       @scheduler = nil
       @heartbeat = nil
@@ -213,6 +223,47 @@ module SolidJobs
       @started
     end
 
+    private
+
+    RUBY34_MULTI_RACTOR_WARNING =
+      "SolidJobs: Ruby %s can deadlock the whole VM on a Ractor GC barrier " \
+      "under multi-Processor load (see docs/reliability.md). Run multi-Ractor " \
+      "servers on Ruby >= 4.0, or use concurrency: 1 (one process per Processor)."
+
+    def warn_ruby34_multi_ractor
+      return if RUBY_VERSION >= "4" || config.concurrency <= 1
+
+      config.logger.warn(format(RUBY34_MULTI_RACTOR_WARNING, RUBY_VERSION))
+    end
+
+    # Waits for a component Ractor to return after :stop. Ruby 3.4 can lose
+    # the wakeup of a Ractor.receive running in a secondary thread while the
+    # inbox is busy; a fresh :stop message re-triggers it. If the component
+    # still does not return, it is abandoned so shutdown never hangs forever.
+    def await_termination(ractor, name)
+      deadline = config.shutdown_timeout + @stop_grace
+      resends = 0
+      begin
+        Timeout.timeout(deadline) { RactorSupport.value(ractor) }
+      rescue Timeout::Error
+        if resends < STOP_RESENDS
+          resends += 1
+          config.logger.warn(
+            "SolidJobs #{name} did not stop within #{deadline}s, " \
+            "re-sending :stop (#{resends}/#{STOP_RESENDS})",
+          )
+          begin
+            ractor.send(:stop)
+          rescue Ractor::ClosedError
+            nil
+          end
+          retry
+        end
+        config.logger.error("SolidJobs #{name} did not stop; abandoning it")
+        nil
+      end
+    end
+
     def remote_signal
       config.redis_pool.call("RPOP", "#{identity}-signals")
     end

data/lib/solid_jobs/version.rb

@@ -1,6 +1,5 @@
 # frozen_string_literal: true
 
 module SolidJobs
-  VERSION = "0.1.1"
+  VERSION = "0.1.3"
 end
-

data/lib/solid_jobs.rb

@@ -3,7 +3,6 @@
 require "base_service"
 require "callback_collection"
 require "solid_redis"
-require "solid_resp_ractor"
 
 require_relative "solid_jobs/version"
 require_relative "solid_jobs/errors"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: solid-jobs
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.1.3
 platform: ruby
 authors:
 - Nicolas Vandenbogaerde
@@ -60,7 +60,7 @@ dependencies:
         version: '1.0'
     - - ">="
       - !ruby/object:Gem::Version
-        version: 1.0.10
+        version: 1.0.11
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -70,27 +70,7 @@ dependencies:
         version: '1.0'
     - - ">="
       - !ruby/object:Gem::Version
-        version: 1.0.10
-- !ruby/object:Gem::Dependency
-  name: solid-resp-ractor
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '0.1'
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: 0.1.3
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '0.1'
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: 0.1.3
+        version: 1.0.11
 - !ruby/object:Gem::Dependency
   name: minitest
   requirement: !ruby/object:Gem::Requirement