kettle-family 1.2.89 → 1.2.91

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 82f0fe9d1ff1a9570736167dbad614ce55a2929c01da9cee1cfad7c8e9bae9d8
4
- data.tar.gz: e3bbc4d7236b5ec4a7874e5a6304b9c9e02f1e3e726a37a361b5e97d8aed6d1e
3
+ metadata.gz: e63a81da6b871ee1c27d813b23f7840e7c0e0a577e3a992076922ff2750574b3
4
+ data.tar.gz: 859158cd1aa7b293126edb628f02eb8bc641a0fcd4797a6f097af9fe46a9ef27
5
5
  SHA512:
6
- metadata.gz: a8e8d571b682158386d7d4320941a5b77eb7b29d99f5653142126e5cb771c8deeb577ca36d0959b77f9de91471f9726a1c08ac95f8b525aa9ce6edef15ece787
7
- data.tar.gz: 0d91b201b66f7c31ee41f80b0e09346f8d228d977c2730e0bbcbef7aa0ddb31d7b9f71032b248c24881f9bc987dd1967dcc0a85977a9d2a12eced9ab090aae7a
6
+ metadata.gz: ee75302308856eb21755bfdc3a598fe9e0d3992b4a493c84825e891dbd76f514f976549fde093de80bfe55d822567d36cba2375a666c1ca2fc6b4a9844f8ae5c
7
+ data.tar.gz: 3845dff951350adc5177e8ea3bb4ce9b80a155a4a6fbb2b05618acb2363e77e9e8ce158129521f03ca8d02816b8bdfd1f1c5ec80507c982a0546e95314815068
checksums.yaml.gz.sig CHANGED
Binary file
data/CHANGELOG.md CHANGED
@@ -30,6 +30,38 @@ Please file a bug if you notice a violation of semantic versioning.
30
30
 
31
31
  ### Security
32
32
 
33
+ ## [1.2.91] - 2026-09-06
34
+
35
+ - TAG: [v1.2.91][1.2.91t]
36
+ - COVERAGE: 93.58% -- 6109/6528 lines in 35 files
37
+ - BRANCH COVERAGE: 78.06% -- 2483/3181 branches in 35 files
38
+ - 32.41% documented
39
+
40
+ ### Fixed
41
+
42
+ - Trust Mise configurations in disposable worktrees before member commands run.
43
+
44
+ - Remove retired appraisal declarations cleanly from files without a trailing newline during template bootstrap.
45
+
46
+ ## [1.2.90] - 2026-09-06
47
+
48
+ - TAG: [v1.2.90][1.2.90t]
49
+ - COVERAGE: 93.13% -- 6048/6494 lines in 35 files
50
+ - BRANCH COVERAGE: 76.05% -- 2407/3165 branches in 35 files
51
+ - 32.41% documented
52
+
53
+ ### Added
54
+
55
+ - Run sibling test workflows concurrently with dynamic TurboTests2 process budgets.
56
+
57
+ - Support aggregate and disposable-worktree monorepo test execution modes.
58
+
59
+ ### Fixed
60
+
61
+ - Template ready monorepo sibling gems concurrently in isolated worktrees while materializing only member-scoped changes.
62
+
63
+ - Derive family command concurrency from the executing CPU count and pass template wave width to nested tools.
64
+
33
65
  ## [1.2.89] - 2026-09-06
34
66
 
35
67
  - TAG: [v1.2.89][1.2.89t]
@@ -2573,7 +2605,11 @@ Please file a bug if you notice a violation of semantic versioning.
2573
2605
  - Fixed CI load failures on engines without compatible `pty` support by falling back to Open3 for interactive release commands.
2574
2606
  - Fixed Ruby 3.2 version-bump support by loading Prism lazily and wiring the Prism gem only for MRI versions that need it.
2575
2607
 
2576
- [Unreleased]: https://github.com/kettle-dev/kettle-family/compare/v1.2.89...HEAD
2608
+ [Unreleased]: https://github.com/kettle-dev/kettle-family/compare/v1.2.91...HEAD
2609
+ [1.2.91]: https://github.com/kettle-dev/kettle-family/compare/v1.2.90...v1.2.91
2610
+ [1.2.91t]: https://github.com/kettle-dev/kettle-family/releases/tag/v1.2.91
2611
+ [1.2.90]: https://github.com/kettle-dev/kettle-family/compare/v1.2.89...v1.2.90
2612
+ [1.2.90t]: https://github.com/kettle-dev/kettle-family/releases/tag/v1.2.90
2577
2613
  [1.2.89]: https://github.com/kettle-dev/kettle-family/compare/v1.2.88...v1.2.89
2578
2614
  [1.2.89t]: https://github.com/kettle-dev/kettle-family/releases/tag/v1.2.89
2579
2615
  [1.2.88]: https://github.com/kettle-dev/kettle-family/compare/v1.2.87...v1.2.88
data/README.md CHANGED
@@ -281,6 +281,37 @@ canonical development lockfile and CI checkout share that path graph. In both
281
281
  cases, release normalization rejects every path outside the configured allow
282
282
  list.
283
283
 
284
+ ### Test concurrency
285
+
286
+ `kettle-family test --execute` runs independent sibling repositories in
287
+ parallel. Its default outer width is half of the CPUs detected on the current
288
+ machine, capped by the number of selected members; `--jobs` and `test.jobs`
289
+ override that width. The same budget is passed to member `kettle-release`
290
+ commands, because their local test phase uses the same process pool.
291
+
292
+ Family sets `TURBO_TESTS2_MAX_PROCESSES` for each active wave. This is an
293
+ inclusive ceiling for TurboTests2: it includes the member's primary test
294
+ process, not merely additional workers. Given `C` detected CPUs and `W` active
295
+ Family wave slots, the ceiling is `min(floor(C / 2), 1 + floor((C - W) / W))`.
296
+ An explicit `turbo_tests2 --count`/`--workers` value, or an explicitly supplied
297
+ `TURBO_TESTS2_MAX_PROCESSES`, takes precedence.
298
+
299
+ Monorepos default to `test.monorepo_mode: aggregate`: one selected member runs
300
+ the shared root suite and the remaining selected subgems are reported as
301
+ skipped. Set `test.aggregate_member` to choose that owner when it is selected.
302
+ For monorepos with genuinely member-scoped suites, opt in to disposable
303
+ worktrees instead:
304
+
305
+ ```yaml
306
+ test:
307
+ monorepo_mode: worktrees
308
+ jobs: 4
309
+ ```
310
+
311
+ Those worktrees isolate coverage, temporary files, and Bundler state. Their
312
+ changes are always discarded; test execution never materializes or commits
313
+ worktree artifacts.
314
+
284
315
  ### Release waves and dependency refresh
285
316
 
286
317
  Release waves are both a sequencing mechanism and a dependency propagation
@@ -533,9 +564,9 @@ stops before family changelog generation or coverage and reports the members
533
564
  that need a version bump. Run `kettle-family bump --execute patch` first, then
534
565
  rerun the release.
535
566
 
536
- State checks run concurrently by default, using up to four jobs or the number
537
- of available CPU cores, whichever is lower. Use `--jobs N` to set an explicit
538
- limit; results remain in family order.
567
+ State checks run concurrently by default, using up to half of the available CPU
568
+ cores and no more than the selected member count. Use `--jobs N` to set an
569
+ explicit limit; results remain in family order.
539
570
 
540
571
  Most commands accept `--json` to print a machine-readable report instead of the
541
572
  text report, and `--report PATH` to write that JSON report while still printing
@@ -732,9 +763,15 @@ kettle-family bump --execute --only unreleased minor
732
763
  ```
733
764
 
734
765
  Run `kettle-jem` templating across the selected family members. Planning shows
735
- the commands that would run; `--execute` runs them. Executed templating defaults
736
- to a parallel job count based on CPU cores, capped for readability, and can be
737
- overridden with `--jobs`.
766
+ the commands that would run; `--execute` runs them. Executed templating uses a
767
+ rolling outer wave of up to half the available CPU cores, bounded by ready
768
+ members; `--jobs` overrides that width.
769
+
770
+ For a Kettle Jem member command, Family exports the active outer width as
771
+ `KETTLE_FAMILY_WAVE_JOBS`. Kettle Jem retains the member's base process and
772
+ uses an equal share of CPUs left by that wave for its recipe and file worker
773
+ pools, capped at half the available CPUs. Explicit `KETTLE_JEM_*_WORKERS`
774
+ settings take precedence over this derived budget.
738
775
 
739
776
  ```console
740
777
  kettle-family template
@@ -1084,7 +1121,7 @@ Thanks for RTFM. ☺️
1084
1121
  [📌gitmoji]: https://gitmoji.dev
1085
1122
  [📌gitmoji-img]: https://img.shields.io/badge/gitmoji_commits-%20%F0%9F%98%9C%20%F0%9F%98%8D-34495e.svg?style=flat-square
1086
1123
  [🧮kloc]: https://www.youtube.com/watch?v=dQw4w9WgXcQ
1087
- [🧮kloc-img]: https://img.shields.io/badge/KLOC-6.247-FFDD67.svg?style=for-the-badge&logo=YouTube&logoColor=blue
1124
+ [🧮kloc-img]: https://img.shields.io/badge/KLOC-6.528-FFDD67.svg?style=for-the-badge&logo=YouTube&logoColor=blue
1088
1125
  [🔐security]: https://github.com/kettle-dev/kettle-family/blob/main/SECURITY.md
1089
1126
  [🔐security-img]: https://img.shields.io/badge/security-policy-259D6C.svg?style=flat
1090
1127
  [📄copyright-notice-explainer]: https://opensource.stackexchange.com/questions/5778/why-do-licenses-such-as-the-mit-license-specify-a-single-year
@@ -66,7 +66,8 @@ module Kettle
66
66
  def obsolete_appraisal_edit(path, source, node)
67
67
  return unless legacy_appraisal_call?(node)
68
68
 
69
- line_start = source.rindex("\n", node.location.start_offset - 1).to_i + 1
69
+ preceding_newline = source.rindex("\n", node.location.start_offset - 1)
70
+ line_start = preceding_newline ? preceding_newline + 1 : 0
70
71
  line_end = source.index("\n", node.location.end_offset) || source.length
71
72
  line_end += 1 if line_end < source.length
72
73
  if final_standalone_call?(source, node, line_start, line_end)
@@ -910,7 +910,7 @@ module Kettle
910
910
  io: stdout,
911
911
  label: "release state",
912
912
  total: members.length,
913
- jobs: [options[:jobs].to_i, 1].max,
913
+ jobs: Concurrency.wave_jobs(requested: options[:jobs], item_count: members.length),
914
914
  members: members,
915
915
  heading: "release state #{members.length} member#{"s" unless members.length == 1}:"
916
916
  )
@@ -230,8 +230,8 @@ module Kettle
230
230
  @otp_coordinator = otp_coordinator
231
231
  end
232
232
 
233
- def call(member:, phase:, command:, env: {}, interactive: false, stdout_line_handler: nil, log_path: nil, passthrough_output: true)
234
- argv = command_argv(member: member, command: command, env: env)
233
+ def call(member:, phase:, command:, env: {}, interactive: false, raw: false, stdout_line_handler: nil, log_path: nil, passthrough_output: true)
234
+ argv = command_argv(member: member, command: command, env: env, raw: raw)
235
235
  process_env = process_env(member: member, env: env)
236
236
  spawn_options = process_options
237
237
  return skipped_result(member: member, phase: phase, argv: argv) unless execute
@@ -598,9 +598,9 @@ module Kettle
598
598
  nil
599
599
  end
600
600
 
601
- def command_argv(member:, command:, env: {})
601
+ def command_argv(member:, command:, env: {}, raw: false)
602
602
  argv = normalize_command(command)
603
- return argv unless mise_configured?(member)
603
+ return argv if raw || !mise_configured?(member)
604
604
 
605
605
  command_env = nonsensitive_env(env)
606
606
  unset_env, set_env = command_env.partition { |_key, value| value.nil? }
@@ -0,0 +1,41 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "etc"
4
+
5
+ module Kettle
6
+ module Family
7
+ # Allocates CPU capacity between concurrent family members and optional
8
+ # member-internal workers. A member consumes one wave slot before any
9
+ # command-specific workers are considered.
10
+ module Concurrency
11
+ module_function
12
+
13
+ def default_wave_jobs(cpu_count: Etc.nprocessors)
14
+ [cpu_count / 2, 1].max
15
+ end
16
+
17
+ def wave_jobs(requested:, item_count:, cpu_count: Etc.nprocessors)
18
+ return 0 if item_count.zero?
19
+
20
+ count = requested.nil? ? default_wave_jobs(cpu_count: cpu_count) : requested.to_i
21
+ count.clamp(1, item_count)
22
+ end
23
+
24
+ def internal_worker_limit(wave_jobs:, cpu_count: Etc.nprocessors)
25
+ raise ArgumentError, "wave jobs must be positive" unless wave_jobs.to_i.positive?
26
+
27
+ half_cores = default_wave_jobs(cpu_count: cpu_count)
28
+ available_per_member = (cpu_count - wave_jobs.to_i) / wave_jobs.to_i
29
+ available_per_member.clamp(1, half_cores)
30
+ end
31
+
32
+ # Test runners count their primary worker as part of the requested
33
+ # process pool. Reserve one process for each wave member, then divide
34
+ # the remaining capacity across them. No member receives more than half
35
+ # of the detected CPUs, including that primary process.
36
+ def test_process_ceiling(wave_jobs:, cpu_count: Etc.nprocessors)
37
+ [default_wave_jobs(cpu_count: cpu_count), 1 + internal_worker_limit(wave_jobs: wave_jobs, cpu_count: cpu_count)].min
38
+ end
39
+ end
40
+ end
41
+ end
@@ -208,6 +208,24 @@ module Kettle
208
208
  fetch_path("template", "jobs")
209
209
  end
210
210
 
211
+ # A monorepo commonly has one root suite that covers every subgem. The
212
+ # configured owner is optional; an --only invocation never expands its
213
+ # own selection solely to reach that owner.
214
+ def test_aggregate_member
215
+ fetch_path("test", "aggregate_member")
216
+ end
217
+
218
+ def test_monorepo_mode
219
+ mode = (fetch_path("test", "monorepo_mode") || "aggregate").to_s
220
+ return mode if %w[aggregate worktrees].include?(mode)
221
+
222
+ raise Error, "test.monorepo_mode must be aggregate or worktrees"
223
+ end
224
+
225
+ def test_jobs
226
+ fetch_path("test", "jobs")
227
+ end
228
+
211
229
  def readme_corporate_sponsors
212
230
  raw = fetch_path("readme", "corporate_sponsors") ||
213
231
  fetch_path("sponsorships", "corporate") ||
@@ -6,6 +6,8 @@ require "json"
6
6
  require "open3"
7
7
  require "time"
8
8
 
9
+ require_relative "concurrency"
10
+
9
11
  module Kettle
10
12
  module Family
11
13
  class LocalInstall
@@ -113,8 +115,7 @@ module Kettle
113
115
  end
114
116
 
115
117
  def install_jobs(candidate_members)
116
- count = jobs ? jobs.to_i : [Etc.nprocessors, 4].min
117
- count.clamp(1, candidate_members.length)
118
+ Concurrency.wave_jobs(requested: jobs, item_count: candidate_members.length)
118
119
  end
119
120
 
120
121
  def member_from_path(path)
@@ -9,6 +9,8 @@ require "rubygems"
9
9
  require "securerandom"
10
10
  require "yaml"
11
11
 
12
+ require_relative "concurrency"
13
+
12
14
  module Kettle
13
15
  module Family
14
16
  class ReleaseStateCheck
@@ -574,10 +576,7 @@ module Kettle
574
576
  end
575
577
 
576
578
  def state_jobs(items)
577
- return 1 if items.length < 2
578
-
579
- requested = jobs || [Etc.nprocessors, 4].min
580
- requested.to_i.clamp(1, items.length)
579
+ Concurrency.wave_jobs(requested: jobs, item_count: items.length)
581
580
  end
582
581
 
583
582
  def parallel_map(items, &block)
@@ -5,7 +5,7 @@ module Kettle
5
5
  # Version namespace for this gem.
6
6
  module Version
7
7
  # Current gem version.
8
- VERSION = "1.2.89"
8
+ VERSION = "1.2.91"
9
9
  end
10
10
  # Current gem version exposed at the traditional constant location.
11
11
  VERSION = Version::VERSION # Traditional Constant Location
@@ -11,6 +11,7 @@ require "shellwords"
11
11
  require "yaml"
12
12
  require "kettle/dev"
13
13
 
14
+ require_relative "concurrency"
14
15
  require_relative "workflow_progress"
15
16
 
16
17
  module Kettle
@@ -569,6 +570,7 @@ module Kettle
569
570
 
570
571
  def member_workflow_results(workflow_members)
571
572
  return template_member_workflow_results(workflow_members) if command == "template" && execute
573
+ return test_member_workflow_results(workflow_members) if command == "test" && execute
572
574
 
573
575
  runner = CommandRunner.new(execute: execute, accept: accept)
574
576
  results = []
@@ -614,6 +616,218 @@ module Kettle
614
616
  results
615
617
  end
616
618
 
619
+ # Sibling repositories have isolated working trees and can run their
620
+ # tests concurrently. A monorepo's default kettle-test command normally
621
+ # executes the shared root suite, so it is represented by one selected
622
+ # owner instead of repeating that suite per subgem.
623
+ def test_member_workflow_results(workflow_members)
624
+ test_members, skipped_members = test_members_for(workflow_members)
625
+ return [] if test_members.empty?
626
+
627
+ return monorepo_test_worktree_results(test_members) if monorepo_test_worktrees?(test_members)
628
+
629
+ wave_jobs = test_jobs(test_members)
630
+ queue = Queue.new
631
+ test_members.each_with_index { |member, index| queue << [index, member] }
632
+ ordered_results = Array.new(test_members.length)
633
+ mutex = Mutex.new
634
+ stop = false
635
+
636
+ Array.new(wave_jobs) do
637
+ Thread.new do # rubocop:disable ThreadSafety/NewThread -- sibling test suites run in isolated project roots.
638
+ runner = CommandRunner.new(execute: execute, accept: accept)
639
+ loop do
640
+ break if mutex.synchronize { stop }
641
+
642
+ index, member = queue.pop(true)
643
+ result = runner.call(
644
+ member: member,
645
+ phase: command,
646
+ command: workflow_command(member),
647
+ env: test_command_env(wave_jobs: wave_jobs)
648
+ )
649
+ mutex.synchronize do
650
+ ordered_results[index] = result
651
+ stop = true unless result.ok?
652
+ end
653
+ rescue ThreadError
654
+ break
655
+ rescue => error
656
+ mutex.synchronize do
657
+ ordered_results[index] = test_worker_error_result(member, error)
658
+ stop = true
659
+ end
660
+ break
661
+ end
662
+ end
663
+ end.each(&:join)
664
+
665
+ results = ordered_results.compact
666
+ return results unless results.all?(&:ok?)
667
+
668
+ results + skipped_members.map { |member| aggregate_test_skip_result(member, owner: test_members.first) }
669
+ end
670
+
671
+ def test_members_for(workflow_members)
672
+ return [workflow_members, []] unless aggregate_monorepo_test?(workflow_members)
673
+
674
+ owner_name = config.test_aggregate_member.to_s.strip
675
+ owner = workflow_members.find { |member| member.name == owner_name } || workflow_members.first
676
+ [[owner], workflow_members - [owner]]
677
+ end
678
+
679
+ def aggregate_monorepo_test?(workflow_members)
680
+ config.family_mode == "monorepo" &&
681
+ config.test_monorepo_mode == "aggregate" &&
682
+ workflow_members.length > 1
683
+ end
684
+
685
+ def monorepo_test_worktrees?(test_members)
686
+ config.family_mode == "monorepo" &&
687
+ config.test_monorepo_mode == "worktrees" &&
688
+ test_members.length > 1 &&
689
+ test_jobs(test_members) > 1
690
+ end
691
+
692
+ # Member-scoped monorepo suites cannot share the primary checkout: test
693
+ # coverage, temporary files, and Bundler state are all mutable. Unlike
694
+ # template worktrees, test worktrees intentionally discard every change.
695
+ def monorepo_test_worktree_results(test_members)
696
+ entries, setup_results = monorepo_test_worktree_entries(test_members)
697
+ return setup_results unless setup_results.all?(&:ok?)
698
+
699
+ run_monorepo_test_worktree_entries(entries).flatten
700
+ ensure
701
+ cleanup_monorepo_test_worktrees(entries || [])
702
+ end
703
+
704
+ def monorepo_test_worktree_entries(test_members)
705
+ runner = command_runner
706
+ control_member = monorepo_template_control_member
707
+ entries = []
708
+ results = []
709
+ test_members.each do |member|
710
+ worktree_root = monorepo_test_worktree_path(member)
711
+ result = runner.call(
712
+ member: control_member,
713
+ phase: "test_member_worktree_add",
714
+ command: ["git", "worktree", "add", "--detach", worktree_root, "HEAD"]
715
+ )
716
+ result.member_name = member.name
717
+ results << result
718
+ break unless result.ok?
719
+
720
+ entry = {
721
+ member: relocate_member_to_worktree(member, worktree_root, source_root: config.root),
722
+ original_member: member,
723
+ worktree_root: worktree_root
724
+ }
725
+ entries << entry
726
+ results.concat(worktree_mise_trust_results(entry, runner: runner, phase: "test_member_worktree_mise_trust"))
727
+ break unless results.last.ok?
728
+ end
729
+ [entries, results]
730
+ end
731
+
732
+ def monorepo_test_worktree_path(member)
733
+ safe_member = member.name.to_s.gsub(/[^A-Za-z0-9_.-]+/, "_")
734
+ File.join(config.root, "tmp", "kettle-family", "test-member-worktrees", safe_member, "#{Process.pid}-#{Time.now.to_i}")
735
+ end
736
+
737
+ def run_monorepo_test_worktree_entries(entries)
738
+ wave_jobs = test_jobs(entries)
739
+ queue = Queue.new
740
+ entries.each_with_index { |entry, index| queue << [index, entry] }
741
+ ordered_results = Array.new(entries.length)
742
+ mutex = Mutex.new
743
+ stop = false
744
+
745
+ Array.new(wave_jobs) do
746
+ Thread.new do # rubocop:disable ThreadSafety/NewThread -- detached worktrees isolate concurrent monorepo test suites.
747
+ loop do
748
+ break if mutex.synchronize { stop }
749
+
750
+ index, entry = queue.pop(true)
751
+ member = entry.fetch(:member)
752
+ result = CommandRunner.new(execute: execute, accept: accept).call(
753
+ member: member,
754
+ phase: command,
755
+ command: workflow_command(member),
756
+ env: test_command_env(wave_jobs: wave_jobs)
757
+ )
758
+ result.member_name = entry.fetch(:original_member).name
759
+ mutex.synchronize do
760
+ ordered_results[index] = [result]
761
+ stop = true unless result.ok?
762
+ end
763
+ rescue ThreadError
764
+ break
765
+ rescue => error
766
+ mutex.synchronize do
767
+ ordered_results[index] = [test_worker_error_result(entry.fetch(:original_member), error)]
768
+ stop = true
769
+ end
770
+ break
771
+ end
772
+ end
773
+ end.each(&:join)
774
+ ordered_results.compact
775
+ end
776
+
777
+ def cleanup_monorepo_test_worktrees(entries)
778
+ runner = command_runner
779
+ control_member = monorepo_template_control_member
780
+ entries.each do |entry|
781
+ root = entry.fetch(:worktree_root)
782
+ next unless Dir.exist?(root)
783
+
784
+ runner.call(member: control_member, phase: "test_member_worktree_remove", command: ["git", "worktree", "remove", "--force", root])
785
+ end
786
+ end
787
+
788
+ def aggregate_test_skip_result(member, owner:)
789
+ CommandResult.new(
790
+ member_name: member.name,
791
+ phase: command,
792
+ command: workflow_command(member),
793
+ workdir: member.root,
794
+ status: 0,
795
+ success: true,
796
+ stdout: "",
797
+ stderr: "",
798
+ elapsed_seconds: 0.0,
799
+ skipped: true,
800
+ reason: "shared monorepo suite owned by #{owner.name}"
801
+ )
802
+ end
803
+
804
+ def test_worker_error_result(member, error)
805
+ CommandResult.new(
806
+ member_name: member.name,
807
+ phase: command,
808
+ command: ["internal", "test-worker"],
809
+ workdir: member.root,
810
+ status: 1,
811
+ success: false,
812
+ stdout: "",
813
+ stderr: "#{error.class}: #{error.message}\n",
814
+ elapsed_seconds: 0.0,
815
+ skipped: false,
816
+ reason: error.message
817
+ )
818
+ end
819
+
820
+ def test_jobs(workflow_members)
821
+ requested = jobs || config.test_jobs
822
+ Concurrency.wave_jobs(requested: requested, item_count: workflow_members.length)
823
+ end
824
+
825
+ def test_command_env(wave_jobs:)
826
+ env = command_env
827
+ env["TURBO_TESTS2_MAX_PROCESSES"] ||= Concurrency.test_process_ceiling(wave_jobs: wave_jobs).to_s
828
+ env
829
+ end
830
+
617
831
  def review_gha_sha_pins(workflow_members, runner:, memo:, env: command_env)
618
832
  return true if workflow_members.empty?
619
833
 
@@ -719,17 +933,20 @@ module Kettle
719
933
  end
720
934
 
721
935
  def template_results_for_wave(wave, progress:)
936
+ return monorepo_template_worktree_wave_results(wave, progress: progress) if monorepo_template_worktrees?(wave)
937
+
938
+ wave_jobs = template_jobs(wave)
722
939
  queue = Queue.new
723
940
  wave.each_with_index { |member, index| queue << [index, member] }
724
941
  ordered_results = Array.new(wave.length)
725
942
  mutex = Mutex.new
726
943
  stop = false
727
- Array.new([template_jobs(wave), wave.length].min) do
944
+ Array.new(wave_jobs) do
728
945
  Thread.new do # rubocop:disable ThreadSafety/NewThread -- family templating intentionally runs independent members concurrently.
729
946
  loop do
730
947
  break if mutex.synchronize { stop }
731
948
  index, member = queue.pop(true)
732
- member_results = template_results_for_member(member, progress: progress)
949
+ member_results = template_results_for_member(member, progress: progress, wave_jobs: wave_jobs)
733
950
  mutex.synchronize do
734
951
  ordered_results[index] = member_results
735
952
  stop = true unless member_results.all?(&:ok?)
@@ -751,10 +968,15 @@ module Kettle
751
968
  ready = remaining.select do |member|
752
969
  (member.dependencies & by_name.keys).all? { |name| completed.include?(name) }
753
970
  end
754
- # A Git checkout is a shared mutable transaction boundary. Nested
755
- # gems in one checkout must not run kettle-jem concurrently, even
756
- # when their dependency graph would otherwise place them together.
757
- wave = ready.group_by { |member| git_root_for(member) }.values.map(&:first)
971
+ # A shared checkout remains a transaction boundary unless each
972
+ # ready monorepo member will receive an isolated worktree. The
973
+ # latter keeps the normal dependency graph intact while allowing
974
+ # independent sibling gems to use the requested worker budget.
975
+ wave = if monorepo_template_worktrees_enabled?
976
+ ready
977
+ else
978
+ ready.group_by { |member| git_root_for(member) }.values.map(&:first)
979
+ end
758
980
  raise Error, "template dependency wave could not resolve: #{remaining.map(&:name).join(", ")}" if wave.empty?
759
981
 
760
982
  waves << wave
@@ -764,6 +986,221 @@ module Kettle
764
986
  end
765
987
  end
766
988
 
989
+ # A monorepo has one mutable primary checkout, but its ready sibling
990
+ # gems can template independently. Run them in detached worktrees and
991
+ # materialize only changes beneath each member root before the existing
992
+ # serialized commit step runs in the primary checkout.
993
+ def monorepo_template_worktree_wave_results(wave, progress:)
994
+ entries, setup_results = monorepo_template_worktree_entries(wave)
995
+ return setup_results unless setup_results.all?(&:ok?)
996
+
997
+ worker_results = run_monorepo_template_worktree_entries(entries, progress: progress)
998
+ results = []
999
+ worker_results.each_with_index do |member_results, index|
1000
+ next unless member_results
1001
+
1002
+ results.concat(member_results)
1003
+ next unless member_results.all?(&:ok?)
1004
+
1005
+ entry = entries.fetch(index)
1006
+ materialized = materialize_monorepo_template_changes(entry)
1007
+ results << materialized
1008
+ next unless materialized.ok?
1009
+
1010
+ commit_results = []
1011
+ commit_template_changes(member: entry.fetch(:original_member), runner: command_runner, memo: commit_results)
1012
+ results.concat(commit_results)
1013
+ end
1014
+ results
1015
+ ensure
1016
+ cleanup_monorepo_template_worktrees(entries || [])
1017
+ end
1018
+
1019
+ def monorepo_template_worktrees?(wave)
1020
+ monorepo_template_worktrees_enabled? && wave.length > 1
1021
+ end
1022
+
1023
+ def monorepo_template_worktrees_enabled?
1024
+ command_text = config.template_command || DEFAULT_COMMANDS.fetch("template")
1025
+ execute && monorepo_template? && kettle_jem_template_command?(command_text) && members.length > 1 && template_jobs(members) > 1
1026
+ end
1027
+
1028
+ def monorepo_template_worktree_entries(wave)
1029
+ runner = command_runner
1030
+ control_member = monorepo_template_control_member
1031
+ entries = []
1032
+ results = []
1033
+ wave.each do |member|
1034
+ worktree_root = monorepo_template_worktree_path(member)
1035
+ result = runner.call(
1036
+ member: control_member,
1037
+ phase: "template_member_worktree_add",
1038
+ command: ["git", "worktree", "add", "--detach", worktree_root, "HEAD"]
1039
+ )
1040
+ result.member_name = member.name
1041
+ results << result
1042
+ break unless result.ok?
1043
+
1044
+ entry = {
1045
+ member: relocate_member_to_worktree(member, worktree_root, source_root: config.root),
1046
+ original_member: member,
1047
+ worktree_root: worktree_root
1048
+ }
1049
+ entries << entry
1050
+ results.concat(worktree_mise_trust_results(entry, runner: runner, phase: "template_member_worktree_mise_trust"))
1051
+ break unless results.last.ok?
1052
+ end
1053
+ [entries, results]
1054
+ end
1055
+
1056
+ def monorepo_template_worktree_path(member)
1057
+ safe_member = member.name.to_s.gsub(/[^A-Za-z0-9_.-]+/, "_")
1058
+ File.join(config.root, "tmp", "kettle-family", "template-member-worktrees", safe_member, "#{Process.pid}-#{Time.now.to_i}")
1059
+ end
1060
+
1061
+ def monorepo_template_control_member
1062
+ family_member.dup.tap { |member| member.root = config.root }
1063
+ end
1064
+
1065
+ def run_monorepo_template_worktree_entries(entries, progress:)
1066
+ wave_jobs = template_jobs(entries)
1067
+ queue = Queue.new
1068
+ entries.each_with_index { |entry, index| queue << [index, entry] }
1069
+ ordered_results = Array.new(entries.length)
1070
+ mutex = Mutex.new
1071
+ stop = false
1072
+ Array.new(wave_jobs) do
1073
+ Thread.new do # rubocop:disable ThreadSafety/NewThread -- detached worktrees isolate concurrent monorepo template members.
1074
+ loop do
1075
+ break if mutex.synchronize { stop }
1076
+
1077
+ index, entry = queue.pop(true)
1078
+ member_results = template_results_for_member(
1079
+ entry.fetch(:member),
1080
+ progress: progress,
1081
+ commit_changes: false,
1082
+ wave_jobs: wave_jobs
1083
+ )
1084
+ mutex.synchronize do
1085
+ ordered_results[index] = member_results
1086
+ stop = true unless member_results.all?(&:ok?)
1087
+ end
1088
+ rescue ThreadError
1089
+ break
1090
+ rescue => error
1091
+ mutex.synchronize do
1092
+ ordered_results[index] = [template_worktree_failure_result(entry.fetch(:original_member), error)]
1093
+ stop = true
1094
+ end
1095
+ break
1096
+ end
1097
+ end
1098
+ end.each(&:join)
1099
+ ordered_results
1100
+ end
1101
+
1102
+ def materialize_monorepo_template_changes(entry)
1103
+ original_member = entry.fetch(:original_member)
1104
+ worktree_root = entry.fetch(:worktree_root)
1105
+ relative_root = Pathname.new(original_member.root).relative_path_from(Pathname.new(config.root)).to_s
1106
+ paths = monorepo_template_worktree_changed_paths(worktree_root)
1107
+ return template_worktree_failure_result(original_member, paths) if paths.is_a?(String)
1108
+
1109
+ invalid_paths = paths.reject { |path| path == relative_root || path.start_with?("#{relative_root}/") }
1110
+ unless invalid_paths.empty?
1111
+ return template_worktree_failure_result(
1112
+ original_member,
1113
+ "isolated template worker modified shared path(s): #{invalid_paths.sort.join(", ")}; only #{relative_root}/ may be materialized"
1114
+ )
1115
+ end
1116
+
1117
+ tracked_paths = monorepo_template_worktree_tracked_paths(worktree_root)
1118
+ return template_worktree_failure_result(original_member, tracked_paths) if tracked_paths.is_a?(String)
1119
+
1120
+ runner = command_runner
1121
+ control_member = monorepo_template_control_member.dup
1122
+ control_member.name = original_member.name
1123
+ command = if tracked_paths.empty?
1124
+ ["true"]
1125
+ else
1126
+ source = Shellwords.escape(worktree_root)
1127
+ scope = Shellwords.escape(relative_root)
1128
+ ["sh", "-lc", "git -C #{source} diff --binary HEAD -- #{scope} | git apply --whitespace=nowarn"]
1129
+ end
1130
+ result = runner.call(member: control_member, phase: "template_worktree_materialize", command: command)
1131
+ return result unless result.ok?
1132
+
1133
+ copy_monorepo_template_untracked_paths(worktree_root, config.root, relative_root)
1134
+ result.stdout = "#{paths.length} path(s) materialized from isolated worktree"
1135
+ result
1136
+ rescue => error
1137
+ template_worktree_failure_result(original_member, error)
1138
+ end
1139
+
1140
+ def monorepo_template_worktree_changed_paths(worktree_root)
1141
+ tracked_paths = monorepo_template_worktree_tracked_paths(worktree_root)
1142
+ return tracked_paths if tracked_paths.is_a?(String)
1143
+
1144
+ untracked_paths = monorepo_template_worktree_untracked_paths(worktree_root)
1145
+ return untracked_paths if untracked_paths.is_a?(String)
1146
+
1147
+ (tracked_paths + untracked_paths).uniq
1148
+ end
1149
+
1150
+ def monorepo_template_worktree_tracked_paths(worktree_root)
1151
+ git_worktree_paths(worktree_root, %w[diff --name-only -z HEAD])
1152
+ end
1153
+
1154
+ def monorepo_template_worktree_untracked_paths(worktree_root)
1155
+ git_worktree_paths(worktree_root, %w[ls-files --others --exclude-standard -z])
1156
+ end
1157
+
1158
+ def git_worktree_paths(worktree_root, arguments)
1159
+ stdout, stderr, status = Open3.capture3("git", *arguments, chdir: worktree_root)
1160
+ return stdout.split("\0").reject(&:empty?) if status.success?
1161
+
1162
+ "git #{arguments.join(" ")} failed in #{worktree_root}: #{stderr.strip}"
1163
+ end
1164
+
1165
+ def copy_monorepo_template_untracked_paths(worktree_root, primary_root, relative_root)
1166
+ monorepo_template_worktree_untracked_paths(worktree_root).each do |path|
1167
+ next unless path == relative_root || path.start_with?("#{relative_root}/")
1168
+
1169
+ source = File.join(worktree_root, path)
1170
+ destination = File.join(primary_root, path)
1171
+ FileUtils.mkdir_p(File.dirname(destination))
1172
+ FileUtils.cp_r(source, destination, preserve: true)
1173
+ end
1174
+ end
1175
+
1176
+ def cleanup_monorepo_template_worktrees(entries)
1177
+ runner = command_runner
1178
+ control_member = monorepo_template_control_member
1179
+ entries.each do |entry|
1180
+ root = entry.fetch(:worktree_root)
1181
+ next unless Dir.exist?(root)
1182
+
1183
+ runner.call(member: control_member, phase: "template_member_worktree_remove", command: ["git", "worktree", "remove", "--force", root])
1184
+ end
1185
+ end
1186
+
1187
+ def template_worktree_failure_result(member, error)
1188
+ message = error.is_a?(Exception) ? "#{error.class}: #{error.message}" : error.to_s
1189
+ CommandResult.new(
1190
+ member_name: member.name,
1191
+ phase: "template_member_worktree",
1192
+ command: ["internal", "template-member-worktree"],
1193
+ workdir: member.root,
1194
+ status: 1,
1195
+ success: false,
1196
+ stdout: "",
1197
+ stderr: message,
1198
+ elapsed_seconds: 0.0,
1199
+ skipped: false,
1200
+ reason: "template member worktree failed"
1201
+ )
1202
+ end
1203
+
767
1204
  def template_bootstrap_dependency_results(workflow_members)
768
1205
  return [] if workflow_members.empty?
769
1206
 
@@ -856,17 +1293,23 @@ module Kettle
856
1293
  )
857
1294
  end
858
1295
 
859
- def template_results_for_member(member, progress: nil)
1296
+ def template_results_for_member(member, progress: nil, commit_changes: true, wave_jobs: 1)
860
1297
  progress&.start_member(member, total: template_phase_total(member), status: template_initial_status(member))
861
1298
  runner = CommandRunner.new(execute: execute, accept: accept)
862
1299
  [].tap do |memo|
863
1300
  if config.normalize_lockfiles?
864
- normalize_lockfiles(member: member, runner: runner, memo: memo, phase: "prepare_lockfiles")
1301
+ normalize_lockfiles(member: member, runner: runner, memo: memo, phase: "prepare_lockfiles", commit_changes: commit_changes)
865
1302
  emit_member_result_progress(member, memo.last, progress: progress)
866
1303
  return memo unless memo.last.ok?
867
1304
  end
868
1305
 
869
- prepared = prepare_template_dependencies(member: member, runner: runner, memo: memo)
1306
+ prepared = prepare_template_dependencies(
1307
+ member: member,
1308
+ runner: runner,
1309
+ memo: memo,
1310
+ commit_changes: commit_changes,
1311
+ wave_jobs: wave_jobs
1312
+ )
870
1313
  emit_member_result_progress(member, memo.last, progress: progress) if memo.last&.phase == "prepare_template_dependencies"
871
1314
  return memo if prepared == false
872
1315
 
@@ -874,15 +1317,15 @@ module Kettle
874
1317
  member: member,
875
1318
  phase: command,
876
1319
  command: workflow_command(member),
877
- env: template_command_env,
1320
+ env: template_command_env(wave_jobs: wave_jobs),
878
1321
  stdout_line_handler: template_event_line_handler(member, progress: progress)
879
1322
  )
880
1323
  emit_member_result_progress(member, memo.last, progress: progress)
881
1324
  return memo unless memo.last.ok?
882
1325
 
883
- normalize_lockfiles(member: member, runner: runner, memo: memo, phase: "normalize_lockfiles")
1326
+ normalize_lockfiles(member: member, runner: runner, memo: memo, phase: "normalize_lockfiles", commit_changes: commit_changes)
884
1327
  emit_member_result_progress(member, memo.last, progress: progress)
885
- commit_template_changes(member: member, runner: runner, memo: memo)
1328
+ commit_template_changes(member: member, runner: runner, memo: memo) if commit_changes
886
1329
  emit_member_result_progress(member, memo.last, progress: progress) if memo.last&.phase == "commit_template"
887
1330
  ensure
888
1331
  template_result = memo.find { |result| result.phase == "template" } || memo.last
@@ -898,8 +1341,7 @@ module Kettle
898
1341
 
899
1342
  def template_jobs(workflow_members)
900
1343
  requested = jobs || config.template_jobs
901
- count = requested ? requested.to_i : [Etc.nprocessors, 4].min
902
- count.clamp(1, workflow_members.length)
1344
+ Concurrency.wave_jobs(requested: requested, item_count: workflow_members.length)
903
1345
  end
904
1346
 
905
1347
  def check_results(workflow_members)
@@ -999,7 +1441,11 @@ module Kettle
999
1441
  outcomes << add
1000
1442
  break unless add.ok?
1001
1443
 
1002
- entries << {branch: branch, member: relocate_member_to_worktree(member, worktree_root), worktree_root: worktree_root}
1444
+ entry = {branch: branch, member: relocate_member_to_worktree(member, worktree_root), worktree_root: worktree_root}
1445
+ entries << entry
1446
+ trust_results = worktree_mise_trust_results(entry.merge(original_member: member), runner: runner, phase: "template_worktree_mise_trust")
1447
+ outcomes.concat(trust_results)
1448
+ break unless trust_results.all?(&:ok?)
1003
1449
  end
1004
1450
 
1005
1451
  if outcomes.all?(&:ok?)
@@ -1055,18 +1501,58 @@ module Kettle
1055
1501
  end
1056
1502
  end
1057
1503
 
1504
+ # A git worktree has a distinct absolute path, and Mise deliberately
1505
+ # tracks trust by config path. Trust every active config between the
1506
+ # worktree root and the member command directory before invoking a
1507
+ # command there. This bootstrap must run raw: wrapping it in `mise exec`
1508
+ # would require the trust it is establishing.
1509
+ def worktree_mise_trust_results(entry, runner:, phase:)
1510
+ member = entry.fetch(:member)
1511
+ original_member = entry.fetch(:original_member, member)
1512
+ worktree_root = entry.fetch(:worktree_root)
1513
+ mise_config_roots_in_worktree(worktree_root, member.root).map do |root|
1514
+ result = runner.call(
1515
+ member: member,
1516
+ phase: phase,
1517
+ command: ["mise", "trust", "-C", root],
1518
+ raw: true
1519
+ )
1520
+ result.member_name = original_member.name
1521
+ result
1522
+ end
1523
+ end
1524
+
1525
+ def mise_config_roots_in_worktree(worktree_root, member_root)
1526
+ root_path = Pathname.new(worktree_root).realpath
1527
+ member_path = Pathname.new(member_root).realpath
1528
+ return [] unless member_path.to_s.start_with?("#{root_path}#{File::SEPARATOR}") || member_path == root_path
1529
+
1530
+ current = member_path
1531
+ roots = []
1532
+ loop do
1533
+ roots << current.to_s if %w[mise.toml .mise.toml].any? { |name| File.file?(current.join(name)) }
1534
+ break if current == root_path
1535
+
1536
+ current = current.parent
1537
+ end
1538
+ roots.reverse
1539
+ rescue Errno::ENOENT
1540
+ []
1541
+ end
1542
+
1058
1543
  def template_branch_worktree_entries_results(entries)
1544
+ wave_jobs = template_jobs(entries)
1059
1545
  queue = Queue.new
1060
1546
  entries.each_with_index { |entry, index| queue << [index, entry] }
1061
1547
  ordered_results = Array.new(entries.length)
1062
1548
  mutex = Mutex.new
1063
1549
  stop = false
1064
- Array.new(template_jobs(entries)) do
1550
+ Array.new(wave_jobs) do
1065
1551
  Thread.new do # rubocop:disable ThreadSafety/NewThread -- independent branch worktrees are safe to template concurrently.
1066
1552
  loop do
1067
1553
  break if mutex.synchronize { stop }
1068
1554
  index, entry = queue.pop(true)
1069
- branch_results = template_branch_worktree_entry_results(entry)
1555
+ branch_results = template_branch_worktree_entry_results(entry, wave_jobs: wave_jobs)
1070
1556
  mutex.synchronize do
1071
1557
  ordered_results[index] = branch_results
1072
1558
  stop = true unless branch_results.all?(&:ok?)
@@ -1079,7 +1565,7 @@ module Kettle
1079
1565
  ordered_results.compact.flatten
1080
1566
  end
1081
1567
 
1082
- def template_branch_worktree_entry_results(entry)
1568
+ def template_branch_worktree_entry_results(entry, wave_jobs: 1)
1083
1569
  member = entry.fetch(:member)
1084
1570
  branch = entry.fetch(:branch)
1085
1571
  runner = command_runner
@@ -1087,7 +1573,7 @@ module Kettle
1087
1573
  if git_upstream_for(member)
1088
1574
  results << runner.call(member: member, phase: "template_branch_sync", command: ["git", "rebase", "@{upstream}"])
1089
1575
  end
1090
- results.concat(template_results_for_member(member)) if results.all?(&:ok?)
1576
+ results.concat(template_results_for_member(member, wave_jobs: wave_jobs)) if results.all?(&:ok?)
1091
1577
  tag_branch_results(results, branch)
1092
1578
  emit_template_event_line(member, results.all?(&:ok?) ? "." : "F", "branch #{branch}")
1093
1579
  results
@@ -1452,7 +1938,7 @@ module Kettle
1452
1938
  waves.each_with_index do |wave, index|
1453
1939
  results << release_wave_result(wave, index: index, total: waves.length, jobs: 1) if show_wave_markers
1454
1940
  wave.each do |member|
1455
- results.concat(release_results_for_member(member, runner: runner))
1941
+ results.concat(release_results_for_member(member, runner: runner, wave_jobs: 1))
1456
1942
  break unless results.last.ok?
1457
1943
 
1458
1944
  remaining_members = ordered_members.drop(ordered_members.index(member) + 1)
@@ -1481,7 +1967,7 @@ module Kettle
1481
1967
  begin
1482
1968
  waves.each_with_index do |wave, index|
1483
1969
  results << release_wave_result(wave, index: index, total: waves.length)
1484
- wave_results = run_release_wave(wave)
1970
+ wave_results = run_release_wave(wave, wave_jobs: release_jobs(wave))
1485
1971
  results.concat(wave_results.flatten)
1486
1972
  break unless wave_results.all? { |member_results| member_results.all?(&:ok?) }
1487
1973
 
@@ -1498,11 +1984,10 @@ module Kettle
1498
1984
  results
1499
1985
  end
1500
1986
 
1501
- def run_release_wave(wave)
1987
+ def run_release_wave(wave, wave_jobs: release_jobs(wave))
1502
1988
  queue = Queue.new
1503
1989
  wave.each_with_index { |member, index| queue << [index, member] }
1504
1990
  ordered_results = Array.new(wave.length)
1505
- wave_jobs = release_jobs(wave)
1506
1991
  mutex = Mutex.new
1507
1992
  stop = false
1508
1993
  Array.new(wave_jobs) do
@@ -1512,7 +1997,7 @@ module Kettle
1512
1997
  break if mutex.synchronize { stop }
1513
1998
 
1514
1999
  index, member = queue.pop(true)
1515
- member_results = release_results_for_member(member, runner: runner)
2000
+ member_results = release_results_for_member(member, runner: runner, wave_jobs: wave_jobs)
1516
2001
  mutex.synchronize do
1517
2002
  ordered_results[index] = member_results
1518
2003
  stop = true unless member_results.all?(&:ok?)
@@ -1543,7 +2028,7 @@ module Kettle
1543
2028
  aggregate_members, independent_members = wave.partition { |member| aggregate_release_member?(member) }
1544
2029
 
1545
2030
  unless aggregate_members.empty?
1546
- worker_results = run_monorepo_release_wave(aggregate_members)
2031
+ worker_results = run_monorepo_release_wave(aggregate_members, wave_jobs: release_jobs(aggregate_members))
1547
2032
  results.concat(worker_results.flatten)
1548
2033
  break unless worker_results.all? { |member_results| member_results.all?(&:ok?) }
1549
2034
 
@@ -1553,7 +2038,7 @@ module Kettle
1553
2038
  end
1554
2039
 
1555
2040
  independent_members.each do |member|
1556
- results.concat(release_results_for_member(member, runner: release_command_runner))
2041
+ results.concat(release_results_for_member(member, runner: release_command_runner, wave_jobs: 1))
1557
2042
  break unless results.last.ok?
1558
2043
  end
1559
2044
  break unless results.last&.ok?
@@ -1571,13 +2056,13 @@ module Kettle
1571
2056
  results
1572
2057
  end
1573
2058
 
1574
- def run_monorepo_release_wave(members)
2059
+ def run_monorepo_release_wave(members, wave_jobs: release_jobs(members))
1575
2060
  entries, setup_results = monorepo_release_worktree_entries(members)
1576
2061
  unless setup_results.all?(&:ok?)
1577
2062
  return setup_results.zip
1578
2063
  end
1579
2064
 
1580
- worker_results = run_monorepo_release_worktree_entries(entries)
2065
+ worker_results = run_monorepo_release_worktree_entries(entries, wave_jobs: wave_jobs)
1581
2066
  materialization_results = materialize_monorepo_release_artifacts(entries, worker_results)
1582
2067
  worker_results.each_with_index.map do |member_results, index|
1583
2068
  artifact_result = materialization_results[index]
@@ -1602,7 +2087,10 @@ module Kettle
1602
2087
  results << result
1603
2088
  break unless result.ok?
1604
2089
 
1605
- entries << {member: relocate_member_to_worktree(member, root, source_root: config.root), original_member: member, worktree_root: root}
2090
+ entry = {member: relocate_member_to_worktree(member, root, source_root: config.root), original_member: member, worktree_root: root}
2091
+ entries << entry
2092
+ results.concat(worktree_mise_trust_results(entry, runner: runner, phase: "release_worktree_mise_trust"))
2093
+ break unless results.last.ok?
1606
2094
  end
1607
2095
  [entries, results]
1608
2096
  end
@@ -1616,19 +2104,19 @@ module Kettle
1616
2104
  family_member.dup.tap { |member| member.root = config.root }
1617
2105
  end
1618
2106
 
1619
- def run_monorepo_release_worktree_entries(entries)
2107
+ def run_monorepo_release_worktree_entries(entries, wave_jobs: release_jobs(entries))
1620
2108
  queue = Queue.new
1621
2109
  entries.each_with_index { |entry, index| queue << [index, entry] }
1622
2110
  ordered_results = Array.new(entries.length)
1623
2111
  mutex = Mutex.new
1624
2112
  stop = false
1625
- Array.new(release_jobs(entries)) do
2113
+ Array.new(wave_jobs) do
1626
2114
  Thread.new do # rubocop:disable ThreadSafety/NewThread -- each member has an isolated detached monorepo worktree.
1627
2115
  loop do
1628
2116
  break if mutex.synchronize { stop }
1629
2117
 
1630
2118
  index, entry = queue.pop(true)
1631
- member_results = release_results_for_monorepo_worktree(entry)
2119
+ member_results = release_results_for_monorepo_worktree(entry, wave_jobs: wave_jobs)
1632
2120
  mutex.synchronize do
1633
2121
  ordered_results[index] = member_results
1634
2122
  stop = true unless member_results.all?(&:ok?)
@@ -1647,7 +2135,7 @@ module Kettle
1647
2135
  ordered_results.compact
1648
2136
  end
1649
2137
 
1650
- def release_results_for_monorepo_worktree(entry)
2138
+ def release_results_for_monorepo_worktree(entry, wave_jobs: 1)
1651
2139
  member = entry.fetch(:member)
1652
2140
  original_member = entry.fetch(:original_member)
1653
2141
  progress = @release_progress
@@ -1668,7 +2156,7 @@ module Kettle
1668
2156
  member: member,
1669
2157
  phase: release_phase,
1670
2158
  command: release_command_for(member),
1671
- env: release_env_for_member(member, family_root: entry.fetch(:worktree_root)).merge("KETTLE_RELEASE_FAMILY_MEMBER_PUBLISH" => "true"),
2159
+ env: release_env_for_member(member, family_root: entry.fetch(:worktree_root), wave_jobs: wave_jobs).merge("KETTLE_RELEASE_FAMILY_MEMBER_PUBLISH" => "true"),
1672
2160
  interactive: release_command_interactive?,
1673
2161
  stdout_line_handler: release_event_line_handler(original_member, progress: progress, state: release_event_state),
1674
2162
  log_path: release_command_log_path(original_member, release_phase),
@@ -1805,7 +2293,7 @@ module Kettle
1805
2293
  )
1806
2294
  end
1807
2295
 
1808
- def release_results_for_member(member, runner:)
2296
+ def release_results_for_member(member, runner:, wave_jobs: 1)
1809
2297
  progress = @release_progress
1810
2298
  progress&.start_member(member, total: release_phase_total(member), status: "check")
1811
2299
  [].tap do |memo|
@@ -1840,7 +2328,7 @@ module Kettle
1840
2328
  member: member,
1841
2329
  phase: release_phase,
1842
2330
  command: release_command_for(member),
1843
- env: release_env_for_member(member),
2331
+ env: release_env_for_member(member, wave_jobs: wave_jobs),
1844
2332
  interactive: release_command_interactive?,
1845
2333
  stdout_line_handler: release_event_line_handler(member, progress: progress, state: release_event_state),
1846
2334
  log_path: release_command_log_path(member, release_phase),
@@ -1917,8 +2405,7 @@ module Kettle
1917
2405
  return 1 if truffleruby?
1918
2406
 
1919
2407
  requested = jobs || config.release_jobs
1920
- count = requested ? requested.to_i : [Etc.nprocessors, 4].min
1921
- count.clamp(1, release_members.length)
2408
+ Concurrency.wave_jobs(requested: requested, item_count: release_members.length)
1922
2409
  end
1923
2410
 
1924
2411
  def truffleruby?
@@ -2963,7 +3450,7 @@ module Kettle
2963
3450
  # A monorepo member runs kettle-release from its own checkout, while its
2964
3451
  # changelog may live at the family root. Pass the shared paths to the
2965
3452
  # member release phase as well as to the separate family phase.
2966
- def release_env_for_member(member, family_root: config.root)
3453
+ def release_env_for_member(member, family_root: config.root, wave_jobs: 1)
2967
3454
  env = release_env
2968
3455
  env.merge!(release_wave_local_path_env_for(member, family_root: family_root))
2969
3456
  env.merge!(release_local_path_policy_env(family_root: family_root))
@@ -2997,6 +3484,10 @@ module Kettle
2997
3484
  if aggregate_validation_member?(member)
2998
3485
  env["KETTLE_RELEASE_FAMILY_CI_MODE"] = aggregate_validation_member_name?(member) ? "validation" : "member"
2999
3486
  end
3487
+ # kettle-release invokes kettle-test as part of its member lifecycle.
3488
+ # Preserve an operator-supplied ceiling, otherwise budget the complete
3489
+ # TurboTests2 process pool for this active Family wave.
3490
+ env["TURBO_TESTS2_MAX_PROCESSES"] ||= Concurrency.test_process_ceiling(wave_jobs: wave_jobs).to_s
3000
3491
  return env unless config.shared_changelog?
3001
3492
  if config.member_local_changelog?(member)
3002
3493
  return env.merge(
@@ -3592,7 +4083,6 @@ module Kettle
3592
4083
  env["BUNDLE_DISABLE_CHECKSUM_VALIDATION"] = "true"
3593
4084
  env["KETTLE_JEM_TEMPLATE_PROFILE"] = config.template_profile if config.template_profile
3594
4085
  env["KJ_REPOSITORY_TOPOLOGY"] = config.template_repository_topology if config.template_repository_topology
3595
- env["KETTLE_JEM_THREAD_WORKERS"] ||= template_thread_worker_budget.to_s
3596
4086
  if monorepo_template?
3597
4087
  template_git_lock_path = template_git_commit_lock_path
3598
4088
  env["KETTLE_JEM_GIT_LOCK"] = template_git_lock_path
@@ -4348,7 +4838,7 @@ module Kettle
4348
4838
  "#{outcomes.fetch(:checksum_hits)} checksum hits#{protected_label}, #{outcomes.fetch(:unchanged)} unchanged, #{changed_label}"
4349
4839
  end
4350
4840
 
4351
- def normalize_lockfiles(member:, runner:, memo:, phase:)
4841
+ def normalize_lockfiles(member:, runner:, memo:, phase:, commit_changes: true)
4352
4842
  return unless config.normalize_lockfiles?
4353
4843
 
4354
4844
  env = template_lockfile_phase?(phase) ? template_prepare_env : workflow_env
@@ -4384,7 +4874,7 @@ module Kettle
4384
4874
  )
4385
4875
  end
4386
4876
  if template_lockfile_phase?(phase) && recoverable_bundle_failure?(result)
4387
- recover_template_lockfiles(member: member, runner: runner, memo: memo, phase: "#{phase}_recovery")
4877
+ recover_template_lockfiles(member: member, runner: runner, memo: memo, phase: "#{phase}_recovery", commit_changes: commit_changes)
4388
4878
  return unless memo.last&.ok?
4389
4879
 
4390
4880
  result = runner.call(
@@ -4397,7 +4887,7 @@ module Kettle
4397
4887
  memo << result
4398
4888
  end
4399
4889
 
4400
- def recover_template_lockfiles(member:, runner:, memo:, phase:)
4890
+ def recover_template_lockfiles(member:, runner:, memo:, phase:, commit_changes: true)
4401
4891
  result = runner.call(
4402
4892
  member: member,
4403
4893
  phase: phase,
@@ -4407,13 +4897,9 @@ module Kettle
4407
4897
  memo << result
4408
4898
  return unless memo.last&.ok?
4409
4899
 
4410
- commit_normalized_lockfiles(
4411
- branch_members: [member],
4412
- runner: runner,
4413
- memo: memo,
4414
- reason: "reset",
4415
- force: true
4416
- )
4900
+ return unless commit_changes
4901
+
4902
+ commit_normalized_lockfiles(branch_members: [member], runner: runner, memo: memo, reason: "reset", force: true)
4417
4903
  end
4418
4904
 
4419
4905
  def recoverable_bundle_failure?(result)
@@ -4555,7 +5041,7 @@ module Kettle
4555
5041
  output.to_s.scan(/from the lockfile CHECKSUMS at Gemfile\.lock:(\d+):\d+/).flatten.map(&:to_i).uniq
4556
5042
  end
4557
5043
 
4558
- def prepare_template_dependencies(member:, runner:, memo:)
5044
+ def prepare_template_dependencies(member:, runner:, memo:, commit_changes: true, wave_jobs: 1)
4559
5045
  command_text = config.template_command || default_template_command(member)
4560
5046
  return true unless kettle_jem_template_command?(command_text)
4561
5047
 
@@ -4563,14 +5049,15 @@ module Kettle
4563
5049
  member: member,
4564
5050
  phase: "prepare_template_dependencies",
4565
5051
  command: template_prepare_command(member),
4566
- env: template_prepare_env
5052
+ env: template_prepare_env(wave_jobs: wave_jobs)
4567
5053
  )
4568
5054
  if recoverable_bundle_failure?(result)
4569
5055
  recover_template_lockfiles(
4570
5056
  member: member,
4571
5057
  runner: runner,
4572
5058
  memo: memo,
4573
- phase: "prepare_template_dependencies_recovery"
5059
+ phase: "prepare_template_dependencies_recovery",
5060
+ commit_changes: commit_changes
4574
5061
  )
4575
5062
  if no_release_lockfiles_failure?(memo.last)
4576
5063
  memo.pop
@@ -4583,7 +5070,7 @@ module Kettle
4583
5070
  member: member,
4584
5071
  phase: "prepare_template_dependencies",
4585
5072
  command: template_prepare_command(member),
4586
- env: template_prepare_env
5073
+ env: template_prepare_env(wave_jobs: wave_jobs)
4587
5074
  )
4588
5075
  end
4589
5076
  memo << result
@@ -4635,15 +5122,17 @@ module Kettle
4635
5122
  end
4636
5123
  end
4637
5124
 
4638
- def template_prepare_env
5125
+ def template_prepare_env(wave_jobs: 1)
4639
5126
  # Templating is development work. It must resolve the configured
4640
5127
  # family graph exactly as template application does; release-only
4641
5128
  # lockfile cleanup owns disabling local path sources.
4642
- execution_profile(template_execution_profile, workflow_env)
5129
+ execution_profile(template_execution_profile, workflow_env).merge(
5130
+ "KETTLE_FAMILY_WAVE_JOBS" => wave_jobs.to_s
5131
+ )
4643
5132
  end
4644
5133
 
4645
- def template_command_env
4646
- (command == "template") ? template_prepare_env : command_env
5134
+ def template_command_env(wave_jobs: 1)
5135
+ (command == "template") ? template_prepare_env(wave_jobs: wave_jobs) : command_env
4647
5136
  end
4648
5137
 
4649
5138
  def template_execution_profile
@@ -4680,11 +5169,6 @@ module Kettle
4680
5169
  env_overrides.key?(name) || ENV.key?(name)
4681
5170
  end
4682
5171
 
4683
- def template_thread_worker_budget
4684
- member_jobs = template_jobs(members)
4685
- [1, (Etc.nprocessors / member_jobs) - 1].max
4686
- end
4687
-
4688
5172
  def normalize_release_lockfiles(member:, runner:, memo:)
4689
5173
  result = runner.call(
4690
5174
  member: member,
data/lib/kettle/family.rb CHANGED
@@ -13,6 +13,7 @@ require_relative "family/secrets"
13
13
  require_relative "family/secrets_broker"
14
14
  require_relative "family/branch_target_config"
15
15
  require_relative "family/release_state_event_tape"
16
+ require_relative "family/concurrency"
16
17
  require_relative "family/release_state_check"
17
18
  require_relative "family/release_reconciler"
18
19
  require_relative "family/unreleased_gem_cleanup"
data.tar.gz.sig CHANGED
Binary file
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: kettle-family
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.2.89
4
+ version: 1.2.91
5
5
  platform: ruby
6
6
  authors:
7
7
  - Peter H. Boling
@@ -410,6 +410,7 @@ files:
410
410
  - lib/kettle/family/cli.rb
411
411
  - lib/kettle/family/command_result.rb
412
412
  - lib/kettle/family/command_runner.rb
413
+ - lib/kettle/family/concurrency.rb
413
414
  - lib/kettle/family/config.rb
414
415
  - lib/kettle/family/debugger_bootstrap.rb
415
416
  - lib/kettle/family/dependency_floor.rb
@@ -442,10 +443,10 @@ licenses:
442
443
  - AGPL-3.0-only
443
444
  metadata:
444
445
  homepage_uri: https://kettle-family.galtzo.com
445
- source_code_uri: https://github.com/kettle-dev/kettle-family/tree/v1.2.89
446
- changelog_uri: https://github.com/kettle-dev/kettle-family/blob/v1.2.89/CHANGELOG.md
446
+ source_code_uri: https://github.com/kettle-dev/kettle-family/tree/v1.2.91
447
+ changelog_uri: https://github.com/kettle-dev/kettle-family/blob/v1.2.91/CHANGELOG.md
447
448
  bug_tracker_uri: https://github.com/kettle-dev/kettle-family/issues
448
- documentation_uri: https://www.rubydoc.info/gems/kettle-family/1.2.89
449
+ documentation_uri: https://www.rubydoc.info/gems/kettle-family/1.2.91
449
450
  funding_uri: https://github.com/sponsors/pboling
450
451
  wiki_uri: https://github.com/kettle-dev/kettle-family/wiki
451
452
  news_uri: https://www.railsbling.com/tags/kettle-family
metadata.gz.sig CHANGED
Binary file