turbo_tests2 3.2.15 → 3.2.17

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b455a333f0fe9a3521173ff0e73b309b74baa6b37da905f5fcfa1a79a5eb075b
-  data.tar.gz: 8a2f7d15910049b3b8366fd1ed6e604581a6c7444a6e95babdcb2f344e73b054
+  metadata.gz: 4d0f84a37aeb74853501933d16cd9476a1f82800b3d33e9da544af8ef0e0c145
+  data.tar.gz: 581d982f38436235925c6bc8180a4ac29513c198599ae841949e47bce114186f
 SHA512:
-  metadata.gz: 3cee9299d9a75a189a763621873df30959b68999972212f05f4ea81ba04200c66d9cbb56e2cc402947b58689881b16cdbc25ac9579205c473c4f359a2cab72ca
-  data.tar.gz: 8ec3ac430a73305ae22bbc58c4ad9e82e0f04f41645853b8da4d50a24f868fe5a6611e5a222774a423df2db89e5176f4f69221ed32adcb85771fab482093b85b
+  metadata.gz: 9d0f7b4e2080fd36f4da47ff4bdebed7fce369995e9c1b6cb305bc2caf9ad935ec2d65eaa84483721b361c57a12eb24b36fd3d0c5fc0ce16a62638f14df16636
+  data.tar.gz: ffc015acc34a5ad4658a2a5aa53ceb02ec6621eea4e625bc1a7d2b6558c1cc728037ad984c32cc52e08108f8c322011dee41926dfdde3400c5c65ae7021c82dc

data/CHANGELOG.md

@@ -30,6 +30,38 @@ Please file a bug if you notice a violation of semantic versioning.
 
 ### Security
 
+## [3.2.17] - 2026-10-14
+
+- TAG: [v3.2.17][3.2.17t]
+- COVERAGE: 95.94% -- 993/1035 lines in 11 files
+- BRANCH COVERAGE: 83.86% -- 265/316 branches in 11 files
+- 39.42% documented
+
+### Changed
+
+- README: document the spawn-based (fork-free) worker architecture that makes parallel workers engine-portable to JRuby and TruffleRuby, and add a Resource-Constrained CI (JVM Engines) section with the canonical 2-worker + capped-heap + raised-watchdog setup, matching the kettle-jem CI templates
+
+- Stall diagnostics and in-code docs now explain silent worker death: the 900s silence warning and the watchdog stall path point at the OOM killer as the most common cause of total silence in a parallel lane (spawned workers, JVM engines defaulting to 1/4 RAM heap each), with the fix inline (TURBO_TESTS2_MAX_PROCESSES=2 + capped heap); start_subprocess and max_processes_from_environment document the spawn-based engine-portable architecture
+
+## [3.2.16] - 2026-10-14
+
+- TAG: [v3.2.16][3.2.16t]
+- COVERAGE: 95.94% -- 993/1035 lines in 11 files
+- BRANCH COVERAGE: 83.86% -- 265/316 branches in 11 files
+- 39.42% documented
+
+### Fixed
+
+- Watchdog kills now actually end the run: kill_remaining_workers takes pids from the live wait_threads (the parallel_tests pid file was unreliable — each track/untrack built a fresh Pids instance, so any normal worker exit wiped the file and the watchdog killed nothing), handle_messages stops at the watchdog sentinel instead of waiting for per-worker exits that can never arrive, and post-kill joins are bounded so a wedged worker can no longer hold the run past the watchdog (observed: CI job printed the stall report, then hung 14 more minutes until cancellation)
+
+### Changed
+
+- [kc] kettle-jem/template: updated 32 project files:
+  - code and tests (1)
+  - dependencies (3)
+  - other (1)
+  - workflows (27)
+
 ## [3.2.15] - 2026-10-13
 
 - TAG: [v3.2.15][3.2.15t]
@@ -742,7 +774,11 @@ Please file a bug if you notice a violation of semantic versioning.
 
 - Initial release
 
-[Unreleased]: https://github.com/galtzo-floss/turbo_tests2/compare/v3.2.15...HEAD
+[Unreleased]: https://github.com/galtzo-floss/turbo_tests2/compare/v3.2.17...HEAD
+[3.2.17]: https://github.com/galtzo-floss/turbo_tests2/compare/v3.2.16...v3.2.17
+[3.2.17t]: https://github.com/galtzo-floss/turbo_tests2/releases/tag/v3.2.17
+[3.2.16]: https://github.com/galtzo-floss/turbo_tests2/compare/v3.2.15...v3.2.16
+[3.2.16t]: https://github.com/galtzo-floss/turbo_tests2/releases/tag/v3.2.16
 [3.2.15]: https://github.com/galtzo-floss/turbo_tests2/compare/v3.2.14...v3.2.15
 [3.2.15t]: https://github.com/galtzo-floss/turbo_tests2/releases/tag/v3.2.15
 [3.2.14]: https://github.com/galtzo-floss/turbo_tests2/compare/v3.2.13...v3.2.14

data/README.md

@@ -384,6 +384,60 @@ count at the number of discovered test groups.
 Known coverage formatter chatter remains collapsed into the concise coverage
 summary.
 
+### Resource-Constrained CI (JVM Engines)
+
+Worker parallelism is spawn-based, not fork-based: each worker is started as a
+real subprocess (`Process.spawn` / `Open3.popen3`), never `Kernel#fork`. That
+makes parallel workers engine-portable — MRI, JRuby, and TruffleRuby all run
+the same spawn-based pool, and the default worker count is the machine's core
+count (`Etc.nprocessors`) on every engine. There is no serial-only mode for
+JVM engines unless you ask for one with `TURBO_TESTS2_MAX_PROCESSES=1`.
+
+The cost of that portability on JVM engines is memory: each JRuby or
+TruffleRuby worker is a full JVM subprocess, and JRuby defaults its heap to
+one quarter of machine RAM. On a CI runner, `Etc.nprocessors` workers each
+grabbing a quarter of RAM is a recipe for the kernel's OOM killer, which kills
+workers **silently** — every worker stops messaging at once, and the stall
+watchdog then reports total silence. This is a resource problem, not a
+parallelism problem.
+
+The fix is to run fewer, smaller workers. This is the canonical
+resource-constrained setup, and the `kettle-jem` CI templates bake it into
+every JRuby workflow they generate:
+
+```bash
+TURBO_TESTS2_MAX_PROCESSES=2 \
+JRUBY_OPTS="-J-Xmx1280m" \
+TURBO_TESTS2_WATCHDOG_SECONDS=1500 \
+bundle exec turbo_tests2
+```
+
+Or in a GitHub Actions workflow:
+
+```yaml
+env:
+  # Fewer, smaller JVM workers on constrained runners.
+  TURBO_TESTS2_MAX_PROCESSES: 2
+  # Cap each worker JVM's heap (JRuby default: 1/4 of runner RAM).
+  JRUBY_OPTS: "-J-Xmx1280m"
+  # JVM lanes are slow; give the watchdog room without hitting the job timeout.
+  TURBO_TESTS2_WATCHDOG_SECONDS: 1500
+```
+
+Guidance:
+
+- **Cap workers** (`TURBO_TESTS2_MAX_PROCESSES=2`) when each worker is a heavy
+  process: JVM engines, large test suites, or shared/undersized runners. Two
+  JVM workers still roughly halve wall-clock time versus one.
+- **Cap the heap** (`JRUBY_OPTS="-J-Xmx1280m"`, or your engine's equivalent)
+  so total memory is bounded regardless of runner size.
+- **Raise the watchdog** (1500s) on slow engines: JIT warmup and heavy GC make
+  silence windows longer, and you want the watchdog report, not the provider
+  timeout, when something genuinely hangs.
+- The same knobs apply to any engine where a worker process is expensive —
+  the principle is: bound `workers × per-worker memory` under the runner's
+  capacity.
+
 ### Stall watchdog
 
 If the worker message queue goes silent — every worker alive but none
@@ -859,7 +913,7 @@ Thanks for RTFM. ☺️
 [📌gitmoji]: https://gitmoji.dev
 [📌gitmoji-img]: https://img.shields.io/badge/gitmoji_commits-%20%F0%9F%98%9C%20%F0%9F%98%8D-34495e.svg?style=flat-square
 [🧮kloc]: https://www.youtube.com/watch?v=dQw4w9WgXcQ
-[🧮kloc-img]: https://img.shields.io/badge/KLOC-1.022-FFDD67.svg?style=for-the-badge&logo=YouTube&logoColor=blue
+[🧮kloc-img]: https://img.shields.io/badge/KLOC-1.035-FFDD67.svg?style=for-the-badge&logo=YouTube&logoColor=blue
 [🔐security]: https://github.com/galtzo-floss/turbo_tests2/blob/main/SECURITY.md
 [🔐security-img]: https://img.shields.io/badge/security-policy-259D6C.svg?style=flat
 [📄copyright-notice-explainer]: https://opensource.stackexchange.com/questions/5778/why-do-licenses-such-as-the-mit-license-specify-a-single-year

data/lib/turbo_tests/cli.rb

@@ -243,6 +243,16 @@ module TurboTests
       ]
     end
 
+    # Inclusive ceiling on the worker pool when neither --count nor --workers
+    # is passed. Explicit CLI counts always win. Two main uses:
+    #
+    # 1. Outer orchestrators (kettle-family) budgeting a machine-wide process
+    #    pool across concurrent family jobs.
+    # 2. Resource-constrained CI, where each worker is an expensive process —
+    #    JVM engines (JRuby, TruffleRuby) spawn a full JVM per worker, so cap
+    #    the pool (typically 2) AND the per-worker heap (e.g. JRUBY_OPTS=
+    #    "-J-Xmx1280m") to stay under runner RAM. See README
+    #    "Resource-Constrained CI (JVM Engines)".
     def max_processes_from_environment
       value = ENV.fetch("TURBO_TESTS2_MAX_PROCESSES", "").strip
       return if value.empty?

data/lib/turbo_tests/runner.rb

@@ -318,24 +318,41 @@ module TurboTests
 
           handle_messages
 
-          @threads.each(&:join)
+          if @watchdog_fired
+            # Workers were SIGKILLed by the watchdog; their reader threads end
+            # at EOF and waiters return as the OS reaps them — but if a kill
+            # failed (permissions, zombie, pid reuse) an unbounded join here
+            # would hang the run anyway, defeating the watchdog. Bound every
+            # join and treat any worker that still has not exited as failed.
+            @threads.each do |thread|
+              thread.join(FORCE_STOP_JOIN)
+              thread.kill if thread.alive?
+            end
+            @wait_threads.each do |wait_thr|
+              wait_thr.join(FORCE_STOP_JOIN)
+            end
+            flush_worker_output unless stream_worker_output?
+            exit_status = 1
+          else
+            @threads.each(&:join)
 
-          report_workers_without_results
+            report_workers_without_results
 
-          report_failed_group(tests_in_groups) if @print_failed_group
+            report_failed_group(tests_in_groups) if @print_failed_group
 
-          Signal.trap(:INT, old_signal)
+            Signal.trap(:INT, old_signal)
 
-          statuses = @wait_threads.map(&:value)
+            statuses = @wait_threads.map(&:value)
 
-          if @reporter.failed_examples.empty? && statuses.all?(&:success?)
-            flush_successful_worker_output
-            report_coverage = true
-            exit_status = 0
-          else
-            flush_worker_output unless stream_worker_output?
-            # From https://github.com/galtzo-floss/turbo_tests2/pull/20/
-            exit_status = statuses.map(&:exitstatus).max
+            if @reporter.failed_examples.empty? && statuses.all?(&:success?)
+              flush_successful_worker_output
+              report_coverage = true
+              exit_status = 0
+            else
+              flush_worker_output unless stream_worker_output?
+              # From https://github.com/galtzo-floss/turbo_tests2/pull/20/
+              exit_status = statuses.map(&:exitstatus).max
+            end
           end
         end
 
@@ -391,6 +408,14 @@ module TurboTests
       )
     end
 
+    # Spawns one worker as a real OS subprocess via Open3.popen3 — never
+    # Kernel#fork. This is what makes the worker pool engine-portable: MRI,
+    # JRuby, and TruffleRuby all spawn subprocesses the same way, and the
+    # default pool size is Etc.nprocessors on every engine. Remember each
+    # worker is a full interpreter/VM process: on JVM engines that is a whole
+    # JVM with its own heap, so bound the pool (TURBO_TESTS2_MAX_PROCESSES)
+    # and the per-worker heap (e.g. JRUBY_OPTS) on memory-constrained
+    # runners — see README "Resource-Constrained CI (JVM Engines)".
     def start_subprocess(env, extra_args, tests, process_id, record_runtime:)
       if tests.empty?
         @messages << {
@@ -739,6 +764,14 @@ module TurboTests
     # on the queue forever — CI jobs hung for hours with no diagnostics.
     # On a stall we kill the surviving worker pids, report which groups
     # never finished, and fail the run instead.
+    #
+    # Note the OOM killer is the most common silent worker death on CI: the
+    # kernel kills the process with no stderr, no exit message, and no
+    # partial output. Workers are spawn-based real processes (never forked),
+    # so on JVM engines (JRuby, TruffleRuby) each one is a full JVM whose
+    # default heap can be 1/4 of machine RAM — a core-count pool of those
+    # exhausts a CI runner. Bound `workers x per-worker memory` on
+    # constrained runners (see README "Resource-Constrained CI").
     def pop_message_with_watchdog
       # 0 (the default) disables the watchdog: block indefinitely, exactly the
       # pre-watchdog behavior.
@@ -759,6 +792,11 @@ module TurboTests
             "turbo_tests2: watchdog fired after #{@watchdog_seconds}s without worker messages; workers killed\n#{stall_report}"
           )
           @error = true
+          # The run is over: workers were just SIGKILLed. handle_messages must
+          # not keep waiting for per-worker exit messages that can never
+          # arrive, and the caller must not re-enter this watchdog (which would
+          # warn-and-kill in a 1-second spin).
+          @watchdog_fired = true
           return {type: "exit", process_id: :watchdog}
         end
       end
@@ -792,7 +830,19 @@ module TurboTests
     end
 
     def kill_remaining_workers
-      tracked_pids.each do |pid|
+      # The live waiter objects are authoritative: the parallel_tests pid file
+      # loses entries because every track/untrack call builds a fresh Pids
+      # instance whose in-memory list starts empty, so a save overwrites (or,
+      # for untrack, empties) the file. After any normal worker exit the file
+      # is guaranteed useless — exactly when the watchdog needs it most.
+      pids = (@wait_threads || []).map do |wait_thr|
+        begin
+          wait_thr.pid
+        rescue
+          nil
+        end
+      end.compact
+      (pids | tracked_pids).each do |pid|
         begin
           Process.kill("KILL", pid)
         rescue
@@ -822,7 +872,10 @@ module TurboTests
               "outer process watcher (a CI job timeout, a container orchestrator, or a local supervisor) will then " \
               "be what eventually ends the run, usually without a useful stall report. A single long test only " \
               "trips this in serial lanes (TURBO_TESTS2_MAX_PROCESSES=1); in parallel lanes other workers keep " \
-              "messages flowing."
+              "messages flowing. Total silence in a PARALLEL lane usually means workers died rather than slowed " \
+              "down — the most common cause is the kernel OOM killer (each worker is a real spawned process; on " \
+              "JVM engines each is a full JVM whose default heap is 1/4 of RAM). Cap workers x per-worker memory: " \
+              "TURBO_TESTS2_MAX_PROCESSES=2 with a capped heap (e.g. JRUBY_OPTS=\"-J-Xmx1280m\")."
             )
             warned_silence = true
           end
@@ -870,6 +923,10 @@ module TurboTests
           nil
         when "exit"
           process_id = message[:process_id]
+          # Watchdog exit: workers were just SIGKILLed and no further per-worker
+          # exit messages can be relied upon — stop the loop immediately
+          # instead of waiting for process_count exits that never arrive.
+          break if process_id == :watchdog
           next if exited_process_ids.key?(process_id)
 
           exited_process_ids[process_id] = true

data/lib/turbo_tests/version.rb

@@ -4,7 +4,7 @@ module TurboTests
   # Version namespace for this gem.
   module Version
     # Current gem version.
-    VERSION = "3.2.15"
+    VERSION = "3.2.17"
   end
   # Current gem version exposed at the traditional constant location.
   VERSION = Version::VERSION # Traditional Constant Location

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: turbo_tests2
 version: !ruby/object:Gem::Version
-  version: 3.2.15
+  version: 3.2.17
 platform: ruby
 authors:
 - Illia
@@ -111,7 +111,7 @@ dependencies:
         version: '3.1'
     - - ">="
       - !ruby/object:Gem::Version
-        version: 3.1.11
+        version: 3.1.12
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -121,7 +121,7 @@ dependencies:
         version: '3.1'
     - - ">="
       - !ruby/object:Gem::Version
-        version: 3.1.11
+        version: 3.1.12
 - !ruby/object:Gem::Dependency
   name: bundler-audit
   requirement: !ruby/object:Gem::Requirement
@@ -219,7 +219,7 @@ dependencies:
         version: '2.0'
     - - ">="
       - !ruby/object:Gem::Version
-        version: 2.0.24
+        version: 2.0.25
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -229,7 +229,7 @@ dependencies:
         version: '2.0'
     - - ">="
       - !ruby/object:Gem::Version
-        version: 2.0.24
+        version: 2.0.25
 - !ruby/object:Gem::Dependency
   name: ruby-progressbar
   requirement: !ruby/object:Gem::Requirement
@@ -316,10 +316,10 @@ licenses:
 - MIT
 metadata:
   homepage_uri: https://turbo-tests2.galtzo.com
-  source_code_uri: https://github.com/galtzo-floss/turbo_tests2/tree/v3.2.15
-  changelog_uri: https://github.com/galtzo-floss/turbo_tests2/blob/v3.2.15/CHANGELOG.md
+  source_code_uri: https://github.com/galtzo-floss/turbo_tests2/tree/v3.2.17
+  changelog_uri: https://github.com/galtzo-floss/turbo_tests2/blob/v3.2.17/CHANGELOG.md
   bug_tracker_uri: https://github.com/galtzo-floss/turbo_tests2/issues
-  documentation_uri: https://www.rubydoc.info/gems/turbo_tests2/3.2.15
+  documentation_uri: https://www.rubydoc.info/gems/turbo_tests2/3.2.17
   funding_uri: https://github.com/sponsors/pboling
   wiki_uri: https://github.com/galtzo-floss/turbo_tests2/wiki
   news_uri: https://www.railsbling.com/tags/turbo_tests2