kettle-family 1.2.89 → 1.2.90

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 82f0fe9d1ff1a9570736167dbad614ce55a2929c01da9cee1cfad7c8e9bae9d8
-  data.tar.gz: e3bbc4d7236b5ec4a7874e5a6304b9c9e02f1e3e726a37a361b5e97d8aed6d1e
+  metadata.gz: fd4de333d6aa3b2bc027ec582b2bcc0b890d61b20a0a57c70bcf25196b2f9c32
+  data.tar.gz: 67754f998efad13f47f00f0e07222a17e9413c890b2cee1097ffbccd1114801d
 SHA512:
-  metadata.gz: a8e8d571b682158386d7d4320941a5b77eb7b29d99f5653142126e5cb771c8deeb577ca36d0959b77f9de91471f9726a1c08ac95f8b525aa9ce6edef15ece787
-  data.tar.gz: 0d91b201b66f7c31ee41f80b0e09346f8d228d977c2730e0bbcbef7aa0ddb31d7b9f71032b248c24881f9bc987dd1967dcc0a85977a9d2a12eced9ab090aae7a
+  metadata.gz: 7238680a28fa74944856d3ebe514193ac070df02fa25b319e28090bbb18f8f6a0f989ccf0d73e84f485d5e7d446de692440304dd0e99cf79b9834f4ba1212143
+  data.tar.gz: 75c99992e6539a9aa9e37f6ad5ea45afa0692bd9214f10df75f9c604f56af1799099e4cb1fa80b47bd9e843fa8a130103dffae010b14eb8e8f64936834447c35

data/CHANGELOG.md

@@ -30,6 +30,25 @@ Please file a bug if you notice a violation of semantic versioning.
 
 ### Security
 
+## [1.2.90] - 2026-09-06
+
+- TAG: [v1.2.90][1.2.90t]
+- COVERAGE: 93.13% -- 6048/6494 lines in 35 files
+- BRANCH COVERAGE: 76.05% -- 2407/3165 branches in 35 files
+- 32.41% documented
+
+### Added
+
+- Run sibling test workflows concurrently with dynamic TurboTests2 process budgets.
+
+- Support aggregate and disposable-worktree monorepo test execution modes.
+
+### Fixed
+
+- Template ready monorepo sibling gems concurrently in isolated worktrees while materializing only member-scoped changes.
+
+- Derive family command concurrency from the executing CPU count and pass template wave width to nested tools.
+
 ## [1.2.89] - 2026-09-06
 
 - TAG: [v1.2.89][1.2.89t]
@@ -2573,7 +2592,9 @@ Please file a bug if you notice a violation of semantic versioning.
 - Fixed CI load failures on engines without compatible `pty` support by falling back to Open3 for interactive release commands.
 - Fixed Ruby 3.2 version-bump support by loading Prism lazily and wiring the Prism gem only for MRI versions that need it.
 
-[Unreleased]: https://github.com/kettle-dev/kettle-family/compare/v1.2.89...HEAD
+[Unreleased]: https://github.com/kettle-dev/kettle-family/compare/v1.2.90...HEAD
+[1.2.90]: https://github.com/kettle-dev/kettle-family/compare/v1.2.89...v1.2.90
+[1.2.90t]: https://github.com/kettle-dev/kettle-family/releases/tag/v1.2.90
 [1.2.89]: https://github.com/kettle-dev/kettle-family/compare/v1.2.88...v1.2.89
 [1.2.89t]: https://github.com/kettle-dev/kettle-family/releases/tag/v1.2.89
 [1.2.88]: https://github.com/kettle-dev/kettle-family/compare/v1.2.87...v1.2.88

data/README.md

@@ -281,6 +281,37 @@ canonical development lockfile and CI checkout share that path graph. In both
 cases, release normalization rejects every path outside the configured allow
 list.
 
+### Test concurrency
+
+`kettle-family test --execute` runs independent sibling repositories in
+parallel. Its default outer width is half of the CPUs detected on the current
+machine, capped by the number of selected members; `--jobs` and `test.jobs`
+override that width. The same budget is passed to member `kettle-release`
+commands, because their local test phase uses the same process pool.
+
+Family sets `TURBO_TESTS2_MAX_PROCESSES` for each active wave. This is an
+inclusive ceiling for TurboTests2: it includes the member's primary test
+process, not merely additional workers. Given `C` detected CPUs and `W` active
+Family wave slots, the ceiling is `min(floor(C / 2), 1 + floor((C - W) / W))`.
+An explicit `turbo_tests2 --count`/`--workers` value, or an explicitly supplied
+`TURBO_TESTS2_MAX_PROCESSES`, takes precedence.
+
+Monorepos default to `test.monorepo_mode: aggregate`: one selected member runs
+the shared root suite and the remaining selected subgems are reported as
+skipped. Set `test.aggregate_member` to choose that owner when it is selected.
+For monorepos with genuinely member-scoped suites, opt in to disposable
+worktrees instead:
+
+```yaml
+test:
+  monorepo_mode: worktrees
+  jobs: 4
+```
+
+Those worktrees isolate coverage, temporary files, and Bundler state. Their
+changes are always discarded; test execution never materializes or commits
+worktree artifacts.
+
 ### Release waves and dependency refresh
 
 Release waves are both a sequencing mechanism and a dependency propagation
@@ -533,9 +564,9 @@ stops before family changelog generation or coverage and reports the members
 that need a version bump. Run `kettle-family bump --execute patch` first, then
 rerun the release.
 
-State checks run concurrently by default, using up to four jobs or the number
-of available CPU cores, whichever is lower. Use `--jobs N` to set an explicit
-limit; results remain in family order.
+State checks run concurrently by default, using up to half of the available CPU
+cores and no more than the selected member count. Use `--jobs N` to set an
+explicit limit; results remain in family order.
 
 Most commands accept `--json` to print a machine-readable report instead of the
 text report, and `--report PATH` to write that JSON report while still printing
@@ -732,9 +763,15 @@ kettle-family bump --execute --only unreleased minor
 ```
 
 Run `kettle-jem` templating across the selected family members. Planning shows
-the commands that would run; `--execute` runs them. Executed templating defaults
-to a parallel job count based on CPU cores, capped for readability, and can be
-overridden with `--jobs`.
+the commands that would run; `--execute` runs them. Executed templating uses a
+rolling outer wave of up to half the available CPU cores, bounded by ready
+members; `--jobs` overrides that width.
+
+For a Kettle Jem member command, Family exports the active outer width as
+`KETTLE_FAMILY_WAVE_JOBS`. Kettle Jem retains the member's base process and
+uses an equal share of CPUs left by that wave for its recipe and file worker
+pools, capped at half the available CPUs. Explicit `KETTLE_JEM_*_WORKERS`
+settings take precedence over this derived budget.
 
 ```console
 kettle-family template
@@ -1084,7 +1121,7 @@ Thanks for RTFM. ☺️
 [📌gitmoji]: https://gitmoji.dev
 [📌gitmoji-img]: https://img.shields.io/badge/gitmoji_commits-%20%F0%9F%98%9C%20%F0%9F%98%8D-34495e.svg?style=flat-square
 [🧮kloc]: https://www.youtube.com/watch?v=dQw4w9WgXcQ
-[🧮kloc-img]: https://img.shields.io/badge/KLOC-6.247-FFDD67.svg?style=for-the-badge&logo=YouTube&logoColor=blue
+[🧮kloc-img]: https://img.shields.io/badge/KLOC-6.494-FFDD67.svg?style=for-the-badge&logo=YouTube&logoColor=blue
 [🔐security]: https://github.com/kettle-dev/kettle-family/blob/main/SECURITY.md
 [🔐security-img]: https://img.shields.io/badge/security-policy-259D6C.svg?style=flat
 [📄copyright-notice-explainer]: https://opensource.stackexchange.com/questions/5778/why-do-licenses-such-as-the-mit-license-specify-a-single-year

data/lib/kettle/family/cli.rb

@@ -910,7 +910,7 @@ module Kettle
           io: stdout,
           label: "release state",
           total: members.length,
-          jobs: [options[:jobs].to_i, 1].max,
+          jobs: Concurrency.wave_jobs(requested: options[:jobs], item_count: members.length),
           members: members,
           heading: "release state #{members.length} member#{"s" unless members.length == 1}:"
         )

data/lib/kettle/family/concurrency.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+require "etc"
+
+module Kettle
+  module Family
+    # Allocates CPU capacity between concurrent family members and optional
+    # member-internal workers. A member consumes one wave slot before any
+    # command-specific workers are considered.
+    module Concurrency
+      module_function
+
+      def default_wave_jobs(cpu_count: Etc.nprocessors)
+        [cpu_count / 2, 1].max
+      end
+
+      def wave_jobs(requested:, item_count:, cpu_count: Etc.nprocessors)
+        return 0 if item_count.zero?
+
+        count = requested.nil? ? default_wave_jobs(cpu_count: cpu_count) : requested.to_i
+        count.clamp(1, item_count)
+      end
+
+      def internal_worker_limit(wave_jobs:, cpu_count: Etc.nprocessors)
+        raise ArgumentError, "wave jobs must be positive" unless wave_jobs.to_i.positive?
+
+        half_cores = default_wave_jobs(cpu_count: cpu_count)
+        available_per_member = (cpu_count - wave_jobs.to_i) / wave_jobs.to_i
+        available_per_member.clamp(1, half_cores)
+      end
+
+      # Test runners count their primary worker as part of the requested
+      # process pool. Reserve one process for each wave member, then divide
+      # the remaining capacity across them. No member receives more than half
+      # of the detected CPUs, including that primary process.
+      def test_process_ceiling(wave_jobs:, cpu_count: Etc.nprocessors)
+        [default_wave_jobs(cpu_count: cpu_count), 1 + internal_worker_limit(wave_jobs: wave_jobs, cpu_count: cpu_count)].min
+      end
+    end
+  end
+end

data/lib/kettle/family/config.rb

@@ -208,6 +208,24 @@ module Kettle
         fetch_path("template", "jobs")
       end
 
+      # A monorepo commonly has one root suite that covers every subgem. The
+      # configured owner is optional; an --only invocation never expands its
+      # own selection solely to reach that owner.
+      def test_aggregate_member
+        fetch_path("test", "aggregate_member")
+      end
+
+      def test_monorepo_mode
+        mode = (fetch_path("test", "monorepo_mode") || "aggregate").to_s
+        return mode if %w[aggregate worktrees].include?(mode)
+
+        raise Error, "test.monorepo_mode must be aggregate or worktrees"
+      end
+
+      def test_jobs
+        fetch_path("test", "jobs")
+      end
+
       def readme_corporate_sponsors
         raw = fetch_path("readme", "corporate_sponsors") ||
           fetch_path("sponsorships", "corporate") ||

data/lib/kettle/family/local_install.rb

@@ -6,6 +6,8 @@ require "json"
 require "open3"
 require "time"
 
+require_relative "concurrency"
+
 module Kettle
   module Family
     class LocalInstall
@@ -113,8 +115,7 @@ module Kettle
       end
 
       def install_jobs(candidate_members)
-        count = jobs ? jobs.to_i : [Etc.nprocessors, 4].min
-        count.clamp(1, candidate_members.length)
+        Concurrency.wave_jobs(requested: jobs, item_count: candidate_members.length)
       end
 
       def member_from_path(path)

data/lib/kettle/family/release_state_check.rb

@@ -9,6 +9,8 @@ require "rubygems"
 require "securerandom"
 require "yaml"
 
+require_relative "concurrency"
+
 module Kettle
   module Family
     class ReleaseStateCheck
@@ -574,10 +576,7 @@ module Kettle
       end
 
       def state_jobs(items)
-        return 1 if items.length < 2
-
-        requested = jobs || [Etc.nprocessors, 4].min
-        requested.to_i.clamp(1, items.length)
+        Concurrency.wave_jobs(requested: jobs, item_count: items.length)
       end
 
       def parallel_map(items, &block)

data/lib/kettle/family/version.rb

@@ -5,7 +5,7 @@ module Kettle
     # Version namespace for this gem.
     module Version
       # Current gem version.
-      VERSION = "1.2.89"
+      VERSION = "1.2.90"
     end
     # Current gem version exposed at the traditional constant location.
     VERSION = Version::VERSION # Traditional Constant Location

data/lib/kettle/family/workflow.rb

@@ -11,6 +11,7 @@ require "shellwords"
 require "yaml"
 require "kettle/dev"
 
+require_relative "concurrency"
 require_relative "workflow_progress"
 
 module Kettle
@@ -569,6 +570,7 @@ module Kettle
 
       def member_workflow_results(workflow_members)
         return template_member_workflow_results(workflow_members) if command == "template" && execute
+        return test_member_workflow_results(workflow_members) if command == "test" && execute
 
         runner = CommandRunner.new(execute: execute, accept: accept)
         results = []
@@ -614,6 +616,215 @@ module Kettle
         results
       end
 
+      # Sibling repositories have isolated working trees and can run their
+      # tests concurrently. A monorepo's default kettle-test command normally
+      # executes the shared root suite, so it is represented by one selected
+      # owner instead of repeating that suite per subgem.
+      def test_member_workflow_results(workflow_members)
+        test_members, skipped_members = test_members_for(workflow_members)
+        return [] if test_members.empty?
+
+        return monorepo_test_worktree_results(test_members) if monorepo_test_worktrees?(test_members)
+
+        wave_jobs = test_jobs(test_members)
+        queue = Queue.new
+        test_members.each_with_index { |member, index| queue << [index, member] }
+        ordered_results = Array.new(test_members.length)
+        mutex = Mutex.new
+        stop = false
+
+        Array.new(wave_jobs) do
+          Thread.new do # rubocop:disable ThreadSafety/NewThread -- sibling test suites run in isolated project roots.
+            runner = CommandRunner.new(execute: execute, accept: accept)
+            loop do
+              break if mutex.synchronize { stop }
+
+              index, member = queue.pop(true)
+              result = runner.call(
+                member: member,
+                phase: command,
+                command: workflow_command(member),
+                env: test_command_env(wave_jobs: wave_jobs)
+              )
+              mutex.synchronize do
+                ordered_results[index] = result
+                stop = true unless result.ok?
+              end
+            rescue ThreadError
+              break
+            rescue => error
+              mutex.synchronize do
+                ordered_results[index] = test_worker_error_result(member, error)
+                stop = true
+              end
+              break
+            end
+          end
+        end.each(&:join)
+
+        results = ordered_results.compact
+        return results unless results.all?(&:ok?)
+
+        results + skipped_members.map { |member| aggregate_test_skip_result(member, owner: test_members.first) }
+      end
+
+      def test_members_for(workflow_members)
+        return [workflow_members, []] unless aggregate_monorepo_test?(workflow_members)
+
+        owner_name = config.test_aggregate_member.to_s.strip
+        owner = workflow_members.find { |member| member.name == owner_name } || workflow_members.first
+        [[owner], workflow_members - [owner]]
+      end
+
+      def aggregate_monorepo_test?(workflow_members)
+        config.family_mode == "monorepo" &&
+          config.test_monorepo_mode == "aggregate" &&
+          workflow_members.length > 1
+      end
+
+      def monorepo_test_worktrees?(test_members)
+        config.family_mode == "monorepo" &&
+          config.test_monorepo_mode == "worktrees" &&
+          test_members.length > 1 &&
+          test_jobs(test_members) > 1
+      end
+
+      # Member-scoped monorepo suites cannot share the primary checkout: test
+      # coverage, temporary files, and Bundler state are all mutable. Unlike
+      # template worktrees, test worktrees intentionally discard every change.
+      def monorepo_test_worktree_results(test_members)
+        entries, setup_results = monorepo_test_worktree_entries(test_members)
+        return setup_results unless setup_results.all?(&:ok?)
+
+        run_monorepo_test_worktree_entries(entries).flatten
+      ensure
+        cleanup_monorepo_test_worktrees(entries || [])
+      end
+
+      def monorepo_test_worktree_entries(test_members)
+        runner = command_runner
+        control_member = monorepo_template_control_member
+        entries = []
+        results = []
+        test_members.each do |member|
+          worktree_root = monorepo_test_worktree_path(member)
+          result = runner.call(
+            member: control_member,
+            phase: "test_member_worktree_add",
+            command: ["git", "worktree", "add", "--detach", worktree_root, "HEAD"]
+          )
+          result.member_name = member.name
+          results << result
+          break unless result.ok?
+
+          entries << {
+            member: relocate_member_to_worktree(member, worktree_root, source_root: config.root),
+            original_member: member,
+            worktree_root: worktree_root
+          }
+        end
+        [entries, results]
+      end
+
+      def monorepo_test_worktree_path(member)
+        safe_member = member.name.to_s.gsub(/[^A-Za-z0-9_.-]+/, "_")
+        File.join(config.root, "tmp", "kettle-family", "test-member-worktrees", safe_member, "#{Process.pid}-#{Time.now.to_i}")
+      end
+
+      def run_monorepo_test_worktree_entries(entries)
+        wave_jobs = test_jobs(entries)
+        queue = Queue.new
+        entries.each_with_index { |entry, index| queue << [index, entry] }
+        ordered_results = Array.new(entries.length)
+        mutex = Mutex.new
+        stop = false
+
+        Array.new(wave_jobs) do
+          Thread.new do # rubocop:disable ThreadSafety/NewThread -- detached worktrees isolate concurrent monorepo test suites.
+            loop do
+              break if mutex.synchronize { stop }
+
+              index, entry = queue.pop(true)
+              member = entry.fetch(:member)
+              result = CommandRunner.new(execute: execute, accept: accept).call(
+                member: member,
+                phase: command,
+                command: workflow_command(member),
+                env: test_command_env(wave_jobs: wave_jobs)
+              )
+              result.member_name = entry.fetch(:original_member).name
+              mutex.synchronize do
+                ordered_results[index] = [result]
+                stop = true unless result.ok?
+              end
+            rescue ThreadError
+              break
+            rescue => error
+              mutex.synchronize do
+                ordered_results[index] = [test_worker_error_result(entry.fetch(:original_member), error)]
+                stop = true
+              end
+              break
+            end
+          end
+        end.each(&:join)
+        ordered_results.compact
+      end
+
+      def cleanup_monorepo_test_worktrees(entries)
+        runner = command_runner
+        control_member = monorepo_template_control_member
+        entries.each do |entry|
+          root = entry.fetch(:worktree_root)
+          next unless Dir.exist?(root)
+
+          runner.call(member: control_member, phase: "test_member_worktree_remove", command: ["git", "worktree", "remove", "--force", root])
+        end
+      end
+
+      def aggregate_test_skip_result(member, owner:)
+        CommandResult.new(
+          member_name: member.name,
+          phase: command,
+          command: workflow_command(member),
+          workdir: member.root,
+          status: 0,
+          success: true,
+          stdout: "",
+          stderr: "",
+          elapsed_seconds: 0.0,
+          skipped: true,
+          reason: "shared monorepo suite owned by #{owner.name}"
+        )
+      end
+
+      def test_worker_error_result(member, error)
+        CommandResult.new(
+          member_name: member.name,
+          phase: command,
+          command: ["internal", "test-worker"],
+          workdir: member.root,
+          status: 1,
+          success: false,
+          stdout: "",
+          stderr: "#{error.class}: #{error.message}\n",
+          elapsed_seconds: 0.0,
+          skipped: false,
+          reason: error.message
+        )
+      end
+
+      def test_jobs(workflow_members)
+        requested = jobs || config.test_jobs
+        Concurrency.wave_jobs(requested: requested, item_count: workflow_members.length)
+      end
+
+      def test_command_env(wave_jobs:)
+        env = command_env
+        env["TURBO_TESTS2_MAX_PROCESSES"] ||= Concurrency.test_process_ceiling(wave_jobs: wave_jobs).to_s
+        env
+      end
+
       def review_gha_sha_pins(workflow_members, runner:, memo:, env: command_env)
         return true if workflow_members.empty?
 
@@ -719,17 +930,20 @@ module Kettle
       end
 
       def template_results_for_wave(wave, progress:)
+        return monorepo_template_worktree_wave_results(wave, progress: progress) if monorepo_template_worktrees?(wave)
+
+        wave_jobs = template_jobs(wave)
         queue = Queue.new
         wave.each_with_index { |member, index| queue << [index, member] }
         ordered_results = Array.new(wave.length)
         mutex = Mutex.new
         stop = false
-        Array.new([template_jobs(wave), wave.length].min) do
+        Array.new(wave_jobs) do
           Thread.new do # rubocop:disable ThreadSafety/NewThread -- family templating intentionally runs independent members concurrently.
             loop do
               break if mutex.synchronize { stop }
               index, member = queue.pop(true)
-              member_results = template_results_for_member(member, progress: progress)
+              member_results = template_results_for_member(member, progress: progress, wave_jobs: wave_jobs)
               mutex.synchronize do
                 ordered_results[index] = member_results
                 stop = true unless member_results.all?(&:ok?)
@@ -751,10 +965,15 @@ module Kettle
             ready = remaining.select do |member|
               (member.dependencies & by_name.keys).all? { |name| completed.include?(name) }
             end
-            # A Git checkout is a shared mutable transaction boundary. Nested
-            # gems in one checkout must not run kettle-jem concurrently, even
-            # when their dependency graph would otherwise place them together.
-            wave = ready.group_by { |member| git_root_for(member) }.values.map(&:first)
+            # A shared checkout remains a transaction boundary unless each
+            # ready monorepo member will receive an isolated worktree.  The
+            # latter keeps the normal dependency graph intact while allowing
+            # independent sibling gems to use the requested worker budget.
+            wave = if monorepo_template_worktrees_enabled?
+              ready
+            else
+              ready.group_by { |member| git_root_for(member) }.values.map(&:first)
+            end
             raise Error, "template dependency wave could not resolve: #{remaining.map(&:name).join(", ")}" if wave.empty?
 
             waves << wave
@@ -764,6 +983,218 @@ module Kettle
         end
       end
 
+      # A monorepo has one mutable primary checkout, but its ready sibling
+      # gems can template independently.  Run them in detached worktrees and
+      # materialize only changes beneath each member root before the existing
+      # serialized commit step runs in the primary checkout.
+      def monorepo_template_worktree_wave_results(wave, progress:)
+        entries, setup_results = monorepo_template_worktree_entries(wave)
+        return setup_results unless setup_results.all?(&:ok?)
+
+        worker_results = run_monorepo_template_worktree_entries(entries, progress: progress)
+        results = []
+        worker_results.each_with_index do |member_results, index|
+          next unless member_results
+
+          results.concat(member_results)
+          next unless member_results.all?(&:ok?)
+
+          entry = entries.fetch(index)
+          materialized = materialize_monorepo_template_changes(entry)
+          results << materialized
+          next unless materialized.ok?
+
+          commit_results = []
+          commit_template_changes(member: entry.fetch(:original_member), runner: command_runner, memo: commit_results)
+          results.concat(commit_results)
+        end
+        results
+      ensure
+        cleanup_monorepo_template_worktrees(entries || [])
+      end
+
+      def monorepo_template_worktrees?(wave)
+        monorepo_template_worktrees_enabled? && wave.length > 1
+      end
+
+      def monorepo_template_worktrees_enabled?
+        command_text = config.template_command || DEFAULT_COMMANDS.fetch("template")
+        execute && monorepo_template? && kettle_jem_template_command?(command_text) && members.length > 1 && template_jobs(members) > 1
+      end
+
+      def monorepo_template_worktree_entries(wave)
+        runner = command_runner
+        control_member = monorepo_template_control_member
+        entries = []
+        results = []
+        wave.each do |member|
+          worktree_root = monorepo_template_worktree_path(member)
+          result = runner.call(
+            member: control_member,
+            phase: "template_member_worktree_add",
+            command: ["git", "worktree", "add", "--detach", worktree_root, "HEAD"]
+          )
+          result.member_name = member.name
+          results << result
+          break unless result.ok?
+
+          entries << {
+            member: relocate_member_to_worktree(member, worktree_root, source_root: config.root),
+            original_member: member,
+            worktree_root: worktree_root
+          }
+        end
+        [entries, results]
+      end
+
+      def monorepo_template_worktree_path(member)
+        safe_member = member.name.to_s.gsub(/[^A-Za-z0-9_.-]+/, "_")
+        File.join(config.root, "tmp", "kettle-family", "template-member-worktrees", safe_member, "#{Process.pid}-#{Time.now.to_i}")
+      end
+
+      def monorepo_template_control_member
+        family_member.dup.tap { |member| member.root = config.root }
+      end
+
+      def run_monorepo_template_worktree_entries(entries, progress:)
+        wave_jobs = template_jobs(entries)
+        queue = Queue.new
+        entries.each_with_index { |entry, index| queue << [index, entry] }
+        ordered_results = Array.new(entries.length)
+        mutex = Mutex.new
+        stop = false
+        Array.new(wave_jobs) do
+          Thread.new do # rubocop:disable ThreadSafety/NewThread -- detached worktrees isolate concurrent monorepo template members.
+            loop do
+              break if mutex.synchronize { stop }
+
+              index, entry = queue.pop(true)
+              member_results = template_results_for_member(
+                entry.fetch(:member),
+                progress: progress,
+                commit_changes: false,
+                wave_jobs: wave_jobs
+              )
+              mutex.synchronize do
+                ordered_results[index] = member_results
+                stop = true unless member_results.all?(&:ok?)
+              end
+            rescue ThreadError
+              break
+            rescue => error
+              mutex.synchronize do
+                ordered_results[index] = [template_worktree_failure_result(entry.fetch(:original_member), error)]
+                stop = true
+              end
+              break
+            end
+          end
+        end.each(&:join)
+        ordered_results
+      end
+
+      def materialize_monorepo_template_changes(entry)
+        original_member = entry.fetch(:original_member)
+        worktree_root = entry.fetch(:worktree_root)
+        relative_root = Pathname.new(original_member.root).relative_path_from(Pathname.new(config.root)).to_s
+        paths = monorepo_template_worktree_changed_paths(worktree_root)
+        return template_worktree_failure_result(original_member, paths) if paths.is_a?(String)
+
+        invalid_paths = paths.reject { |path| path == relative_root || path.start_with?("#{relative_root}/") }
+        unless invalid_paths.empty?
+          return template_worktree_failure_result(
+            original_member,
+            "isolated template worker modified shared path(s): #{invalid_paths.sort.join(", ")}; only #{relative_root}/ may be materialized"
+          )
+        end
+
+        tracked_paths = monorepo_template_worktree_tracked_paths(worktree_root)
+        return template_worktree_failure_result(original_member, tracked_paths) if tracked_paths.is_a?(String)
+
+        runner = command_runner
+        control_member = monorepo_template_control_member.dup
+        control_member.name = original_member.name
+        command = if tracked_paths.empty?
+          ["true"]
+        else
+          source = Shellwords.escape(worktree_root)
+          scope = Shellwords.escape(relative_root)
+          ["sh", "-lc", "git -C #{source} diff --binary HEAD -- #{scope} | git apply --whitespace=nowarn"]
+        end
+        result = runner.call(member: control_member, phase: "template_worktree_materialize", command: command)
+        return result unless result.ok?
+
+        copy_monorepo_template_untracked_paths(worktree_root, config.root, relative_root)
+        result.stdout = "#{paths.length} path(s) materialized from isolated worktree"
+        result
+      rescue => error
+        template_worktree_failure_result(original_member, error)
+      end
+
+      def monorepo_template_worktree_changed_paths(worktree_root)
+        tracked_paths = monorepo_template_worktree_tracked_paths(worktree_root)
+        return tracked_paths if tracked_paths.is_a?(String)
+
+        untracked_paths = monorepo_template_worktree_untracked_paths(worktree_root)
+        return untracked_paths if untracked_paths.is_a?(String)
+
+        (tracked_paths + untracked_paths).uniq
+      end
+
+      def monorepo_template_worktree_tracked_paths(worktree_root)
+        git_worktree_paths(worktree_root, %w[diff --name-only -z HEAD])
+      end
+
+      def monorepo_template_worktree_untracked_paths(worktree_root)
+        git_worktree_paths(worktree_root, %w[ls-files --others --exclude-standard -z])
+      end
+
+      def git_worktree_paths(worktree_root, arguments)
+        stdout, stderr, status = Open3.capture3("git", *arguments, chdir: worktree_root)
+        return stdout.split("\0").reject(&:empty?) if status.success?
+
+        "git #{arguments.join(" ")} failed in #{worktree_root}: #{stderr.strip}"
+      end
+
+      def copy_monorepo_template_untracked_paths(worktree_root, primary_root, relative_root)
+        monorepo_template_worktree_untracked_paths(worktree_root).each do |path|
+          next unless path == relative_root || path.start_with?("#{relative_root}/")
+
+          source = File.join(worktree_root, path)
+          destination = File.join(primary_root, path)
+          FileUtils.mkdir_p(File.dirname(destination))
+          FileUtils.cp_r(source, destination, preserve: true)
+        end
+      end
+
+      def cleanup_monorepo_template_worktrees(entries)
+        runner = command_runner
+        control_member = monorepo_template_control_member
+        entries.each do |entry|
+          root = entry.fetch(:worktree_root)
+          next unless Dir.exist?(root)
+
+          runner.call(member: control_member, phase: "template_member_worktree_remove", command: ["git", "worktree", "remove", "--force", root])
+        end
+      end
+
+      def template_worktree_failure_result(member, error)
+        message = error.is_a?(Exception) ? "#{error.class}: #{error.message}" : error.to_s
+        CommandResult.new(
+          member_name: member.name,
+          phase: "template_member_worktree",
+          command: ["internal", "template-member-worktree"],
+          workdir: member.root,
+          status: 1,
+          success: false,
+          stdout: "",
+          stderr: message,
+          elapsed_seconds: 0.0,
+          skipped: false,
+          reason: "template member worktree failed"
+        )
+      end
+
       def template_bootstrap_dependency_results(workflow_members)
         return [] if workflow_members.empty?
 
@@ -856,17 +1287,23 @@ module Kettle
         )
       end
 
-      def template_results_for_member(member, progress: nil)
+      def template_results_for_member(member, progress: nil, commit_changes: true, wave_jobs: 1)
         progress&.start_member(member, total: template_phase_total(member), status: template_initial_status(member))
         runner = CommandRunner.new(execute: execute, accept: accept)
         [].tap do |memo|
           if config.normalize_lockfiles?
-            normalize_lockfiles(member: member, runner: runner, memo: memo, phase: "prepare_lockfiles")
+            normalize_lockfiles(member: member, runner: runner, memo: memo, phase: "prepare_lockfiles", commit_changes: commit_changes)
             emit_member_result_progress(member, memo.last, progress: progress)
             return memo unless memo.last.ok?
           end
 
-          prepared = prepare_template_dependencies(member: member, runner: runner, memo: memo)
+          prepared = prepare_template_dependencies(
+            member: member,
+            runner: runner,
+            memo: memo,
+            commit_changes: commit_changes,
+            wave_jobs: wave_jobs
+          )
           emit_member_result_progress(member, memo.last, progress: progress) if memo.last&.phase == "prepare_template_dependencies"
           return memo if prepared == false
 
@@ -874,15 +1311,15 @@ module Kettle
             member: member,
             phase: command,
             command: workflow_command(member),
-            env: template_command_env,
+            env: template_command_env(wave_jobs: wave_jobs),
             stdout_line_handler: template_event_line_handler(member, progress: progress)
           )
           emit_member_result_progress(member, memo.last, progress: progress)
           return memo unless memo.last.ok?
 
-          normalize_lockfiles(member: member, runner: runner, memo: memo, phase: "normalize_lockfiles")
+          normalize_lockfiles(member: member, runner: runner, memo: memo, phase: "normalize_lockfiles", commit_changes: commit_changes)
           emit_member_result_progress(member, memo.last, progress: progress)
-          commit_template_changes(member: member, runner: runner, memo: memo)
+          commit_template_changes(member: member, runner: runner, memo: memo) if commit_changes
           emit_member_result_progress(member, memo.last, progress: progress) if memo.last&.phase == "commit_template"
         ensure
           template_result = memo.find { |result| result.phase == "template" } || memo.last
@@ -898,8 +1335,7 @@ module Kettle
 
       def template_jobs(workflow_members)
         requested = jobs || config.template_jobs
-        count = requested ? requested.to_i : [Etc.nprocessors, 4].min
-        count.clamp(1, workflow_members.length)
+        Concurrency.wave_jobs(requested: requested, item_count: workflow_members.length)
       end
 
       def check_results(workflow_members)
@@ -1056,17 +1492,18 @@ module Kettle
       end
 
       def template_branch_worktree_entries_results(entries)
+        wave_jobs = template_jobs(entries)
         queue = Queue.new
         entries.each_with_index { |entry, index| queue << [index, entry] }
         ordered_results = Array.new(entries.length)
         mutex = Mutex.new
         stop = false
-        Array.new(template_jobs(entries)) do
+        Array.new(wave_jobs) do
           Thread.new do # rubocop:disable ThreadSafety/NewThread -- independent branch worktrees are safe to template concurrently.
             loop do
               break if mutex.synchronize { stop }
               index, entry = queue.pop(true)
-              branch_results = template_branch_worktree_entry_results(entry)
+              branch_results = template_branch_worktree_entry_results(entry, wave_jobs: wave_jobs)
               mutex.synchronize do
                 ordered_results[index] = branch_results
                 stop = true unless branch_results.all?(&:ok?)
@@ -1079,7 +1516,7 @@ module Kettle
         ordered_results.compact.flatten
       end
 
-      def template_branch_worktree_entry_results(entry)
+      def template_branch_worktree_entry_results(entry, wave_jobs: 1)
         member = entry.fetch(:member)
         branch = entry.fetch(:branch)
         runner = command_runner
@@ -1087,7 +1524,7 @@ module Kettle
         if git_upstream_for(member)
           results << runner.call(member: member, phase: "template_branch_sync", command: ["git", "rebase", "@{upstream}"])
         end
-        results.concat(template_results_for_member(member)) if results.all?(&:ok?)
+        results.concat(template_results_for_member(member, wave_jobs: wave_jobs)) if results.all?(&:ok?)
         tag_branch_results(results, branch)
         emit_template_event_line(member, results.all?(&:ok?) ? "." : "F", "branch #{branch}")
         results
@@ -1452,7 +1889,7 @@ module Kettle
           waves.each_with_index do |wave, index|
             results << release_wave_result(wave, index: index, total: waves.length, jobs: 1) if show_wave_markers
             wave.each do |member|
-              results.concat(release_results_for_member(member, runner: runner))
+              results.concat(release_results_for_member(member, runner: runner, wave_jobs: 1))
               break unless results.last.ok?
 
               remaining_members = ordered_members.drop(ordered_members.index(member) + 1)
@@ -1481,7 +1918,7 @@ module Kettle
         begin
           waves.each_with_index do |wave, index|
             results << release_wave_result(wave, index: index, total: waves.length)
-            wave_results = run_release_wave(wave)
+            wave_results = run_release_wave(wave, wave_jobs: release_jobs(wave))
             results.concat(wave_results.flatten)
             break unless wave_results.all? { |member_results| member_results.all?(&:ok?) }
 
@@ -1498,11 +1935,10 @@ module Kettle
         results
       end
 
-      def run_release_wave(wave)
+      def run_release_wave(wave, wave_jobs: release_jobs(wave))
         queue = Queue.new
         wave.each_with_index { |member, index| queue << [index, member] }
         ordered_results = Array.new(wave.length)
-        wave_jobs = release_jobs(wave)
         mutex = Mutex.new
         stop = false
         Array.new(wave_jobs) do
@@ -1512,7 +1948,7 @@ module Kettle
               break if mutex.synchronize { stop }
 
               index, member = queue.pop(true)
-              member_results = release_results_for_member(member, runner: runner)
+              member_results = release_results_for_member(member, runner: runner, wave_jobs: wave_jobs)
               mutex.synchronize do
                 ordered_results[index] = member_results
                 stop = true unless member_results.all?(&:ok?)
@@ -1543,7 +1979,7 @@ module Kettle
             aggregate_members, independent_members = wave.partition { |member| aggregate_release_member?(member) }
 
             unless aggregate_members.empty?
-              worker_results = run_monorepo_release_wave(aggregate_members)
+              worker_results = run_monorepo_release_wave(aggregate_members, wave_jobs: release_jobs(aggregate_members))
               results.concat(worker_results.flatten)
               break unless worker_results.all? { |member_results| member_results.all?(&:ok?) }
 
@@ -1553,7 +1989,7 @@ module Kettle
             end
 
             independent_members.each do |member|
-              results.concat(release_results_for_member(member, runner: release_command_runner))
+              results.concat(release_results_for_member(member, runner: release_command_runner, wave_jobs: 1))
               break unless results.last.ok?
             end
             break unless results.last&.ok?
@@ -1571,13 +2007,13 @@ module Kettle
         results
       end
 
-      def run_monorepo_release_wave(members)
+      def run_monorepo_release_wave(members, wave_jobs: release_jobs(members))
         entries, setup_results = monorepo_release_worktree_entries(members)
         unless setup_results.all?(&:ok?)
           return setup_results.zip
         end
 
-        worker_results = run_monorepo_release_worktree_entries(entries)
+        worker_results = run_monorepo_release_worktree_entries(entries, wave_jobs: wave_jobs)
         materialization_results = materialize_monorepo_release_artifacts(entries, worker_results)
         worker_results.each_with_index.map do |member_results, index|
           artifact_result = materialization_results[index]
@@ -1616,19 +2052,19 @@ module Kettle
         family_member.dup.tap { |member| member.root = config.root }
       end
 
-      def run_monorepo_release_worktree_entries(entries)
+      def run_monorepo_release_worktree_entries(entries, wave_jobs: release_jobs(entries))
         queue = Queue.new
         entries.each_with_index { |entry, index| queue << [index, entry] }
         ordered_results = Array.new(entries.length)
         mutex = Mutex.new
         stop = false
-        Array.new(release_jobs(entries)) do
+        Array.new(wave_jobs) do
           Thread.new do # rubocop:disable ThreadSafety/NewThread -- each member has an isolated detached monorepo worktree.
             loop do
               break if mutex.synchronize { stop }
 
               index, entry = queue.pop(true)
-              member_results = release_results_for_monorepo_worktree(entry)
+              member_results = release_results_for_monorepo_worktree(entry, wave_jobs: wave_jobs)
               mutex.synchronize do
                 ordered_results[index] = member_results
                 stop = true unless member_results.all?(&:ok?)
@@ -1647,7 +2083,7 @@ module Kettle
         ordered_results.compact
       end
 
-      def release_results_for_monorepo_worktree(entry)
+      def release_results_for_monorepo_worktree(entry, wave_jobs: 1)
         member = entry.fetch(:member)
         original_member = entry.fetch(:original_member)
         progress = @release_progress
@@ -1668,7 +2104,7 @@ module Kettle
             member: member,
             phase: release_phase,
             command: release_command_for(member),
-            env: release_env_for_member(member, family_root: entry.fetch(:worktree_root)).merge("KETTLE_RELEASE_FAMILY_MEMBER_PUBLISH" => "true"),
+            env: release_env_for_member(member, family_root: entry.fetch(:worktree_root), wave_jobs: wave_jobs).merge("KETTLE_RELEASE_FAMILY_MEMBER_PUBLISH" => "true"),
             interactive: release_command_interactive?,
             stdout_line_handler: release_event_line_handler(original_member, progress: progress, state: release_event_state),
             log_path: release_command_log_path(original_member, release_phase),
@@ -1805,7 +2241,7 @@ module Kettle
         )
       end
 
-      def release_results_for_member(member, runner:)
+      def release_results_for_member(member, runner:, wave_jobs: 1)
         progress = @release_progress
         progress&.start_member(member, total: release_phase_total(member), status: "check")
         [].tap do |memo|
@@ -1840,7 +2276,7 @@ module Kettle
             member: member,
             phase: release_phase,
             command: release_command_for(member),
-            env: release_env_for_member(member),
+            env: release_env_for_member(member, wave_jobs: wave_jobs),
             interactive: release_command_interactive?,
             stdout_line_handler: release_event_line_handler(member, progress: progress, state: release_event_state),
             log_path: release_command_log_path(member, release_phase),
@@ -1917,8 +2353,7 @@ module Kettle
         return 1 if truffleruby?
 
         requested = jobs || config.release_jobs
-        count = requested ? requested.to_i : [Etc.nprocessors, 4].min
-        count.clamp(1, release_members.length)
+        Concurrency.wave_jobs(requested: requested, item_count: release_members.length)
       end
 
       def truffleruby?
@@ -2963,7 +3398,7 @@ module Kettle
       # A monorepo member runs kettle-release from its own checkout, while its
       # changelog may live at the family root. Pass the shared paths to the
       # member release phase as well as to the separate family phase.
-      def release_env_for_member(member, family_root: config.root)
+      def release_env_for_member(member, family_root: config.root, wave_jobs: 1)
         env = release_env
         env.merge!(release_wave_local_path_env_for(member, family_root: family_root))
         env.merge!(release_local_path_policy_env(family_root: family_root))
@@ -2997,6 +3432,10 @@ module Kettle
         if aggregate_validation_member?(member)
           env["KETTLE_RELEASE_FAMILY_CI_MODE"] = aggregate_validation_member_name?(member) ? "validation" : "member"
         end
+        # kettle-release invokes kettle-test as part of its member lifecycle.
+        # Preserve an operator-supplied ceiling, otherwise budget the complete
+        # TurboTests2 process pool for this active Family wave.
+        env["TURBO_TESTS2_MAX_PROCESSES"] ||= Concurrency.test_process_ceiling(wave_jobs: wave_jobs).to_s
         return env unless config.shared_changelog?
         if config.member_local_changelog?(member)
           return env.merge(
@@ -3592,7 +4031,6 @@ module Kettle
             env["BUNDLE_DISABLE_CHECKSUM_VALIDATION"] = "true"
             env["KETTLE_JEM_TEMPLATE_PROFILE"] = config.template_profile if config.template_profile
             env["KJ_REPOSITORY_TOPOLOGY"] = config.template_repository_topology if config.template_repository_topology
-            env["KETTLE_JEM_THREAD_WORKERS"] ||= template_thread_worker_budget.to_s
             if monorepo_template?
               template_git_lock_path = template_git_commit_lock_path
               env["KETTLE_JEM_GIT_LOCK"] = template_git_lock_path
@@ -4348,7 +4786,7 @@ module Kettle
         "#{outcomes.fetch(:checksum_hits)} checksum hits#{protected_label}, #{outcomes.fetch(:unchanged)} unchanged, #{changed_label}"
       end
 
-      def normalize_lockfiles(member:, runner:, memo:, phase:)
+      def normalize_lockfiles(member:, runner:, memo:, phase:, commit_changes: true)
         return unless config.normalize_lockfiles?
 
         env = template_lockfile_phase?(phase) ? template_prepare_env : workflow_env
@@ -4384,7 +4822,7 @@ module Kettle
           )
         end
         if template_lockfile_phase?(phase) && recoverable_bundle_failure?(result)
-          recover_template_lockfiles(member: member, runner: runner, memo: memo, phase: "#{phase}_recovery")
+          recover_template_lockfiles(member: member, runner: runner, memo: memo, phase: "#{phase}_recovery", commit_changes: commit_changes)
           return unless memo.last&.ok?
 
           result = runner.call(
@@ -4397,7 +4835,7 @@ module Kettle
         memo << result
       end
 
-      def recover_template_lockfiles(member:, runner:, memo:, phase:)
+      def recover_template_lockfiles(member:, runner:, memo:, phase:, commit_changes: true)
         result = runner.call(
           member: member,
           phase: phase,
@@ -4407,13 +4845,9 @@ module Kettle
         memo << result
         return unless memo.last&.ok?
 
-        commit_normalized_lockfiles(
-          branch_members: [member],
-          runner: runner,
-          memo: memo,
-          reason: "reset",
-          force: true
-        )
+        return unless commit_changes
+
+        commit_normalized_lockfiles(branch_members: [member], runner: runner, memo: memo, reason: "reset", force: true)
       end
 
       def recoverable_bundle_failure?(result)
@@ -4555,7 +4989,7 @@ module Kettle
         output.to_s.scan(/from the lockfile CHECKSUMS at Gemfile\.lock:(\d+):\d+/).flatten.map(&:to_i).uniq
       end
 
-      def prepare_template_dependencies(member:, runner:, memo:)
+      def prepare_template_dependencies(member:, runner:, memo:, commit_changes: true, wave_jobs: 1)
         command_text = config.template_command || default_template_command(member)
         return true unless kettle_jem_template_command?(command_text)
 
@@ -4563,14 +4997,15 @@ module Kettle
           member: member,
           phase: "prepare_template_dependencies",
           command: template_prepare_command(member),
-          env: template_prepare_env
+          env: template_prepare_env(wave_jobs: wave_jobs)
         )
         if recoverable_bundle_failure?(result)
           recover_template_lockfiles(
             member: member,
             runner: runner,
             memo: memo,
-            phase: "prepare_template_dependencies_recovery"
+            phase: "prepare_template_dependencies_recovery",
+            commit_changes: commit_changes
           )
           if no_release_lockfiles_failure?(memo.last)
             memo.pop
@@ -4583,7 +5018,7 @@ module Kettle
             member: member,
             phase: "prepare_template_dependencies",
             command: template_prepare_command(member),
-            env: template_prepare_env
+            env: template_prepare_env(wave_jobs: wave_jobs)
           )
         end
         memo << result
@@ -4635,15 +5070,17 @@ module Kettle
         end
       end
 
-      def template_prepare_env
+      def template_prepare_env(wave_jobs: 1)
         # Templating is development work.  It must resolve the configured
         # family graph exactly as template application does; release-only
         # lockfile cleanup owns disabling local path sources.
-        execution_profile(template_execution_profile, workflow_env)
+        execution_profile(template_execution_profile, workflow_env).merge(
+          "KETTLE_FAMILY_WAVE_JOBS" => wave_jobs.to_s
+        )
       end
 
-      def template_command_env
-        (command == "template") ? template_prepare_env : command_env
+      def template_command_env(wave_jobs: 1)
+        (command == "template") ? template_prepare_env(wave_jobs: wave_jobs) : command_env
       end
 
       def template_execution_profile
@@ -4680,11 +5117,6 @@ module Kettle
         env_overrides.key?(name) || ENV.key?(name)
       end
 
-      def template_thread_worker_budget
-        member_jobs = template_jobs(members)
-        [1, (Etc.nprocessors / member_jobs) - 1].max
-      end
-
       def normalize_release_lockfiles(member:, runner:, memo:)
         result = runner.call(
           member: member,

data/lib/kettle/family.rb

@@ -13,6 +13,7 @@ require_relative "family/secrets"
 require_relative "family/secrets_broker"
 require_relative "family/branch_target_config"
 require_relative "family/release_state_event_tape"
+require_relative "family/concurrency"
 require_relative "family/release_state_check"
 require_relative "family/release_reconciler"
 require_relative "family/unreleased_gem_cleanup"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: kettle-family
 version: !ruby/object:Gem::Version
-  version: 1.2.89
+  version: 1.2.90
 platform: ruby
 authors:
 - Peter H. Boling
@@ -410,6 +410,7 @@ files:
 - lib/kettle/family/cli.rb
 - lib/kettle/family/command_result.rb
 - lib/kettle/family/command_runner.rb
+- lib/kettle/family/concurrency.rb
 - lib/kettle/family/config.rb
 - lib/kettle/family/debugger_bootstrap.rb
 - lib/kettle/family/dependency_floor.rb
@@ -442,10 +443,10 @@ licenses:
 - AGPL-3.0-only
 metadata:
   homepage_uri: https://kettle-family.galtzo.com
-  source_code_uri: https://github.com/kettle-dev/kettle-family/tree/v1.2.89
-  changelog_uri: https://github.com/kettle-dev/kettle-family/blob/v1.2.89/CHANGELOG.md
+  source_code_uri: https://github.com/kettle-dev/kettle-family/tree/v1.2.90
+  changelog_uri: https://github.com/kettle-dev/kettle-family/blob/v1.2.90/CHANGELOG.md
   bug_tracker_uri: https://github.com/kettle-dev/kettle-family/issues
-  documentation_uri: https://www.rubydoc.info/gems/kettle-family/1.2.89
+  documentation_uri: https://www.rubydoc.info/gems/kettle-family/1.2.90
   funding_uri: https://github.com/sponsors/pboling
   wiki_uri: https://github.com/kettle-dev/kettle-family/wiki
   news_uri: https://www.railsbling.com/tags/kettle-family