polyrun 2.1.0 → 2.1.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 4ab4d8c8553db3e8e62e83967c9090fc7e788c994d450febb0dc64d39d946f3b
4
- data.tar.gz: 3edbc34c324880653c8fd6d7675a31193adfebb0f0a4439d33f74e4ab9d1c327
3
+ metadata.gz: 4a5cd4e5af5195cb25ca3185b3e828556746f21a6911f8a60f1b43b5b6a316d0
4
+ data.tar.gz: 67362c68f463339cfd8a66e665b9fb6a5f75cf8ed6276e1a846c68147a27e848
5
5
  SHA512:
6
- metadata.gz: 1e372f737bf9d8cf76d39979f87dec6acd33e325ef1b5355acdbdb588d9542bc94d0fa0236eecb6632152c0b148560e489a9e26132bede885ddbb73628fa3830
7
- data.tar.gz: c61bedba0ea3f6399ffee3d217d719182de6c3e945fbbb14aea9c811f0b75e7e968b7e30a30162fb944635a949629e0102976d0f01e2851411b7694649ac8c5f
6
+ metadata.gz: b7f7ff8592774bfb1bd173753f5c3cca77d99a392a9dbabdd0bf11c85b510a3dc4cf0c2a66912cf5d48384445e74039a2cfa296d6aeb40b5d008aa575700ef0b
7
+ data.tar.gz: 05aad7cb17b505112548d072895d56c88f885f07b31e3052ab12e5b4ab0979a555bdf9872380107a60211a8bd54dbd176b98a8970eaba7756fa2e87f8a475a15
data/CHANGELOG.md CHANGED
@@ -2,6 +2,21 @@
2
2
 
3
3
  ## Unreleased
4
4
 
5
+ ## 2.1.3 (2026-07-10)
6
+
7
+ - Add `Polyrun::WorkerOutput` for per-shard worker stdout/stderr routing (`POLYRUN_WORKER_OUTPUT_ROUTING=1` or `POLYRUN_WORKER_LOG_DIR`); optional prefixed TTY echo (`POLYRUN_WORKER_OUTPUT_PREFIX=0` for log-only).
8
+ - Add `partition.paths_build.exclude_prefixes` to omit path prefixes from `build-paths` / `start` while keeping files runnable outside polyrun.
9
+ - Add `Polyrun::RSpec.install_example_debug!`, `install_example_timeout!`, `install_example_rails_logging!`, `install_example_prosopite!` (`POLYRUN_EXAMPLE_DEBUG=1` and related env vars).
10
+ - Add `Polyrun::RSpec.install_sharded_formatter_compat!` for progress formatters (e.g. Fuubar) under `POLYRUN_SHARD_*`.
11
+
12
+ ## 2.1.2 (2026-07-05)
13
+
14
+ - Fix per-example timing JSON when RSpec supplies `absolute_file_path` metadata; fix formatter registration when `install_example_timing!` uses a custom output path.
15
+
16
+ ## 2.1.1 (2026-07-05)
17
+
18
+ - Fix HTML coverage report `Encoding::CompatibilityError` when source files contain UTF-8; read templates, assets, and source lines as UTF-8 in `formatters_html.rb`.
19
+
5
20
  ## 2.1.0 (2026-07-03)
6
21
 
7
22
  - Add experimental per-example spec quality (`Polyrun::SpecQuality`): `POLYRUN_SPEC_QUALITY=1`, worker JSONL fragments, `merge-spec-quality`, `report-spec-quality`, and `run-shards --merge-spec-quality`.
data/CONTRIBUTING.md CHANGED
@@ -27,6 +27,7 @@ Run the suite under alternate Ruby constraints (see `Appraisals` and `gemfiles/`
27
27
  ```bash
28
28
  bundle exec appraisal ruby32 rspec
29
29
  bundle exec appraisal ruby34 rspec
30
+ bundle exec appraisal ruby40 rspec
30
31
  ```
31
32
 
32
33
  ## Linting
@@ -44,11 +44,40 @@ Rule: anything expensive (compile, `yarn`, Playwright download) belongs in prepa
44
44
  |--------|---------|
45
45
  | Plain glob | `partition.paths_build.all_glob: spec/**/*_spec.rb` and empty or minimal `stages` |
46
46
  | Ordered stages | `partition.paths_build.stages`: regex (e.g. slow integration first) or `sort_by_substring_order` for stable ordering |
47
+ | Exclude prefixes | `partition.paths_build.exclude_prefixes`: omit paths (e.g. `spec/benchmark/`) from `paths_file` while keeping them runnable outside polyrun |
47
48
 
48
49
  `paths_build` controls **membership** in `partition.paths_file`, not shard assignment order. Default `round_robin` sorts paths alphabetically before mod assignment. Use `strategy: preserve_order_round_robin` to honor paths-file line order. Set `partition.timing_file` without `strategy` to auto-select `cost_binpack`; use `lazy_robin` for round-robin assignment with timing diagnostics.
49
50
 
50
51
  Refresh list: `polyrun -c polyrun.yml build-paths` (also runs automatically before `plan` / `run-shards` when configured).
51
52
 
53
+ ## 5b. Parallel worker logs (opt-in)
54
+
55
+ When `run-shards` fans out multiple workers, stdout/stderr interleave on the parent TTY by default.
56
+
57
+ | Env | Effect |
58
+ |--------|--------|
59
+ | `POLYRUN_WORKER_OUTPUT_ROUTING=1` | Route each worker through per-shard log files (default dir `tmp/polyrun/workers`) |
60
+ | `POLYRUN_WORKER_LOG_DIR` | Custom log directory (also enables routing when set) |
61
+ | `POLYRUN_WORKER_OUTPUT_ROUTING=0` | Force inherited stdio even when log dir is set |
62
+ | `POLYRUN_WORKER_OUTPUT_PREFIX=0` | Write logs only; no live prefixed TTY echo |
63
+
64
+ Carriage-return progress chunks (Fuubar-style) pass through unprefixed on the TTY and in logs.
65
+
66
+ ## 5c. RSpec example debug (local investigation)
67
+
68
+ Orchestration tracing stays on `DEBUG=1` / `POLYRUN_DEBUG=1`. Per-example tooling is separate:
69
+
70
+ | Env | Effect |
71
+ |--------|--------|
72
+ | `POLYRUN_EXAMPLE_DEBUG=1` | Enable example debug installers |
73
+ | `POLYRUN_DEBUG_SQL=1` or `DEBUG_SQL=1` | Log mutating SQL |
74
+ | `POLYRUN_DEBUG_TRACE=1` or `DEBUG_TRACE=1` | TracePoint under app root |
75
+ | `DEBUG_PROSOPITE=1` | Prosopite N+1 scan when gem is loaded |
76
+ | `DEBUG_LOG_LEVEL` | Rails / ActiveRecord logger severity: `debug`, `info`, `warn`, `error`, `fatal`, or Ruby Logger integers (`0` debug … `4` fatal); default `debug` |
77
+ | `RSPEC_EXAMPLE_TIMEOUT_SEC` | Per-example timeout (disabled while example debug is on) |
78
+
79
+ In `spec_helper` / `spec/support`: `require "polyrun/rspec"` then `Polyrun::RSpec.install_example_debug!`, `install_example_rails_logging!`, `install_example_timeout!`, and `install_sharded_formatter_compat!` when using Fuubar under `POLYRUN_SHARD_*`. Pair with `install_worker_ping!` for idle timeouts.
80
+
52
81
  ## 6. Coverage and CI reports
53
82
 
54
83
  | Field | Options |
@@ -92,7 +92,7 @@ module Polyrun
92
92
  def merge_coverage_after_shards(output:, format_list:, config_path:)
93
93
  files = merge_coverage_fragment_json_files
94
94
  if files.empty?
95
- Polyrun::Log.warn "polyrun run-shards: --merge-coverage: no coverage/polyrun-fragment-*.json found (enable Polyrun coverage in spec_helper?)"
95
+ Polyrun::Log.warn "polyrun run-shards: --merge-coverage: no coverage fragments found under coverage (enable coverage collection in your test setup)"
96
96
  return 0
97
97
  end
98
98
 
@@ -57,7 +57,7 @@ module Polyrun
57
57
  pattern = Polyrun::Reporting::FailureMerge.default_fragment_glob
58
58
  files = Dir.glob(pattern).sort
59
59
  if files.empty?
60
- Polyrun::Log.warn "polyrun run-shards: --merge-failures: no #{Polyrun::Reporting::FailureMerge::FRAGMENT_GLOB} under fragment dir (enable Polyrun::RSpec.install_failure_fragments! in spec_helper?)"
60
+ Polyrun::Log.warn "polyrun run-shards: --merge-failures: no failure fragments found under tmp/polyrun_failures (enable failure fragments in your test setup)"
61
61
  return nil
62
62
  end
63
63
 
@@ -13,21 +13,22 @@ module Polyrun
13
13
  -h, --help
14
14
 
15
15
  Trace timing (stderr): DEBUG=1 or POLYRUN_DEBUG=1
16
- Branch coverage in JSON fragments: POLYRUN_COVERAGE_BRANCHES=1 (stdlib Coverage; merge-coverage merges branches)
17
- polyrun quick coverage: POLYRUN_COVERAGE=1 or (config/polyrun_coverage.yml + POLYRUN_QUICK_COVERAGE=1); POLYRUN_COVERAGE_DISABLE=1 skips
18
- Merge wall time (stderr): POLYRUN_PROFILE_MERGE=1 (or verbose / DEBUG)
16
+ Coverage: POLYRUN_COVERAGE=1 (or config/polyrun_coverage.yml + POLYRUN_QUICK_COVERAGE=1); POLYRUN_COVERAGE_DISABLE=1 skips; POLYRUN_COVERAGE_BRANCHES=1 for branch data in fragments
17
+ Merge profiling (stderr): POLYRUN_PROFILE_MERGE=1 (or verbose / DEBUG)
19
18
  Post-merge formats (run-shards): POLYRUN_MERGE_FORMATS (default: json,lcov,cobertura,console,html)
20
- Skip optional script/build_spec_paths.rb before start: POLYRUN_SKIP_BUILD_SPEC_PATHS=1
21
- Skip start auto-prepare / auto DB provision: POLYRUN_START_SKIP_PREPARE=1, POLYRUN_START_SKIP_DATABASES=1
22
- Skip writing paths_file from partition.paths_build: POLYRUN_SKIP_PATHS_BUILD=1
23
- Warn if merge-coverage wall time exceeds N seconds (default 10): POLYRUN_MERGE_SLOW_WARN_SECONDS (0 disables)
24
- Failure fragments (run-shards --merge-failures): POLYRUN_MERGE_FAILURES=1; parent sets POLYRUN_FAILURE_FRAGMENTS=1 in workers; POLYRUN_FAILURE_FRAGMENT_DIR, POLYRUN_MERGED_FAILURES_OUT, POLYRUN_MERGED_FAILURES_FORMAT; after_suite sets POLYRUN_MERGED_FAILURES_PATH when merge ran
25
- Parallel RSpec workers: POLYRUN_WORKERS default 5, max 10 (run-shards / parallel-rspec / start); distinct from POLYRUN_SHARD_PROCESSES / ci-shard --shard-processes (local processes per CI matrix job)
26
- Per-worker wall timeout: run-shards --worker-timeout SEC or POLYRUN_WORKER_TIMEOUT_SEC (max time since each worker spawn). Parent polls all live workers together. Exit 124; remaining workers stopped.
27
- Per-worker idle timeout: --worker-idle-timeout SEC or POLYRUN_WORKER_IDLE_TIMEOUT_SEC counts only after a successful ping timestamp (positive float in POLYRUN_WORKER_PING_FILE); empty or unreadable pings do not satisfy idle enforcement—use wall timeout until the first ping. RSpec/Minitest/Quick installers call Polyrun::WorkerPing.ping! per example/suite. Ping files live under tmp/polyrun/ (gitignored via tmp/); parent unlinks each after its worker exits. Exit 125. Optional outer cap: --worker-timeout (exit 124). Optional periodic pings: POLYRUN_WORKER_PING_THREAD=1 (POLYRUN_WORKER_PING_INTERVAL_SEC); WorkerPing.ensure_interval_ping_thread! (installers invoke it—call yourself if wiring workers without install_worker_ping!).
28
- If Polyrun::Log.stderr is null or redirected away, set POLYRUN_ORCHESTRATION_STDERR=1 to also print timeout/SIGINT summary lines to process stderr.
29
- Spec quality (opt-in): POLYRUN_SPEC_QUALITY=1; per-example JSONL fragments; POLYRUN_SPEC_QUALITY_SAMPLE=0.0-1.0 (default 1.0); POLYRUN_SPEC_QUALITY_STRICT=1; run-shards --merge-spec-quality sets POLYRUN_SPEC_QUALITY_FRAGMENTS=1 in workers; merge-spec-quality / report-spec-quality
30
- Partition timing granularity (default file): POLYRUN_TIMING_GRANULARITY=file|example (experimental per-example; see partition.timing_granularity)
19
+ Start skips: POLYRUN_SKIP_BUILD_SPEC_PATHS=1, POLYRUN_START_SKIP_PREPARE=1, POLYRUN_START_SKIP_DATABASES=1
20
+ Paths build skip: POLYRUN_SKIP_PATHS_BUILD=1
21
+ Slow merge warning (seconds, default 10; 0 disables): POLYRUN_MERGE_SLOW_WARN_SECONDS
22
+ Failure merge: run-shards --merge-failures (enable failure fragments in test setup); POLYRUN_MERGE_FAILURES=1, POLYRUN_FAILURE_FRAGMENT_DIR, POLYRUN_MERGED_FAILURES_OUT
23
+ Parallel workers: POLYRUN_WORKERS default 5, max 10 (run-shards / parallel-rspec / start). CI local processes per job: POLYRUN_SHARD_PROCESSES or ci-shard --shard-processes (not POLYRUN_WORKERS)
24
+ Per-worker wall timeout: --worker-timeout SEC or POLYRUN_WORKER_TIMEOUT_SEC. Exit 124; parent stops remaining workers.
25
+ Per-worker idle timeout: --worker-idle-timeout SEC or POLYRUN_WORKER_IDLE_TIMEOUT_SEC after a progress ping (POLYRUN_WORKER_PING_FILE). Enable pings in test setup. Exit 125. Optional periodic pings: POLYRUN_WORKER_PING_THREAD=1 (POLYRUN_WORKER_PING_INTERVAL_SEC).
26
+ Worker output routing (opt-in): POLYRUN_WORKER_OUTPUT_ROUTING=1 or POLYRUN_WORKER_LOG_DIR; per-shard logs under tmp/polyrun/workers (POLYRUN_WORKER_OUTPUT_PREFIX=0 for log-only)
27
+ Example debug (RSpec, opt-in): POLYRUN_EXAMPLE_DEBUG=1; POLYRUN_DEBUG_SQL / POLYRUN_DEBUG_TRACE
28
+ Sharded formatter compat: progress formatters (e.g. Fuubar) under POLYRUN_SHARD_* (see docs/SETUP_PROFILE.md)
29
+ Orchestration warnings on process stderr: POLYRUN_ORCHESTRATION_STDERR=1
30
+ Spec quality (opt-in): POLYRUN_SPEC_QUALITY=1; run-shards --merge-spec-quality; merge-spec-quality / report-spec-quality
31
+ Partition timing granularity (default file): POLYRUN_TIMING_GRANULARITY=file|example (experimental; see partition.timing_granularity)
31
32
  Partition strategies: round_robin (default, sorted), preserve_order_round_robin (paths-file order), lazy_robin (sorted RR + timing diagnostics), cost_binpack (LPT), hrw. partition.timing_file without strategy implies cost_binpack.
32
33
 
33
34
  commands:
@@ -45,7 +46,7 @@ module Polyrun
45
46
  init write a starter polyrun.yml or POLYRUN.md from built-in templates (see docs/SETUP_PROFILE.md)
46
47
  queue file-backed batch queue: init (optional --shard/--total etc. as plan, then claim/ack/reclaim/status --json)
47
48
  run-queue init queue and run N workers that claim batches until drained
48
- quick run Polyrun::Quick (describe/it, before/after, let, expect…to, assert_*; optional capybara!)
49
+ quick quick test runner (describe/it, before/after, let, expect…to, assert_*; optional capybara!)
49
50
  hook run <phase> run one shell hook from polyrun.yml hooks: (e.g. before_suite); optional --shard/--total
50
51
  report-coverage write all coverage formats from one JSON file
51
52
  report-junit RSpec JSON or Polyrun testcase JSON → JUnit XML (CI)
@@ -53,7 +54,7 @@ module Polyrun
53
54
  merge-timing merge polyrun_timing_*.json shards
54
55
  merge-spec-quality merge polyrun-spec-quality-fragment-*.jsonl shards
55
56
  report-spec-quality spec quality report from merged JSON (zero-hit, hot lines, churn)
56
- config print effective config by dotted path (see Polyrun::Config::Effective; same tree as YAML plus merged prepare.env, resolved partition shard fields, workers)
57
+ config print effective config by dotted path (loaded YAML plus merged prepare.env, resolved partition shard fields, workers)
57
58
  env print shard + database env (see polyrun.yml databases)
58
59
  db:setup-template migrate template DB (PostgreSQL)
59
60
  db:setup-shard CREATE DATABASE shard FROM template (one POLYRUN_SHARD_INDEX)
@@ -3,6 +3,7 @@ require "optparse"
3
3
  require "shellwords"
4
4
 
5
5
  require_relative "../queue/duration"
6
+ require_relative "../queue/worker_loop"
6
7
 
7
8
  module Polyrun
8
9
  class CLI
@@ -83,44 +84,16 @@ module Polyrun
83
84
  workers.times do |i|
84
85
  wid = "worker-#{i}"
85
86
  pid = Process.fork do
86
- run_queue_worker_loop(store: store, worker_id: wid, batch: batch, cmd: cmd, on_failure: on_failure)
87
+ result = Polyrun::Queue::WorkerLoop.run(
88
+ store: store, worker_id: wid, batch: batch, cmd: cmd, on_failure: on_failure
89
+ )
90
+ exit result[:exit_code]
87
91
  end
88
92
  pids << {pid: pid, worker_id: wid}
89
93
  end
90
94
  pids
91
95
  end
92
96
 
93
- def run_queue_worker_loop(store:, worker_id:, batch:, cmd:, on_failure:)
94
- batches_ok = 0
95
- batches_fail = 0
96
- loop do
97
- claim = store.claim!(worker_id: worker_id, batch_size: batch)
98
- paths = claim["paths"] || []
99
- break if paths.empty?
100
-
101
- code = run_queue_run_batch(cmd, paths)
102
- if code == 0
103
- store.ack!(lease_id: claim["lease_id"], worker_id: worker_id)
104
- batches_ok += 1
105
- elsif on_failure.to_s == "requeue"
106
- store.reclaim_lease!(claim["lease_id"])
107
- batches_fail += 1
108
- exit 1
109
- else
110
- batches_fail += 1
111
- exit code.zero? ? 1 : code
112
- end
113
- end
114
- exit 0
115
- rescue Polyrun::Error => e
116
- Polyrun::Log.warn "polyrun run-queue worker #{worker_id}: #{e.message}"
117
- exit 2
118
- end
119
-
120
- def run_queue_run_batch(cmd, paths)
121
- system(*cmd, *paths) ? 0 : ($?.exitstatus || 1)
122
- end
123
-
124
97
  def run_queue_wait_workers(pids, store:, on_failure:)
125
98
  ok = 0
126
99
  fail = 0
@@ -1,5 +1,6 @@
1
1
  require "fileutils"
2
2
 
3
+ require_relative "../worker_output"
3
4
  require_relative "run_shards_parallel_wait"
4
5
 
5
6
  module Polyrun
@@ -22,6 +23,7 @@ module Polyrun
22
23
  mt = ctx[:matrix_shard_total]
23
24
 
24
25
  pids = []
26
+ Polyrun::WorkerOutput.prepare_log_dir! if Polyrun::WorkerOutput.routing_enabled? && parallel
25
27
  workers.times do |shard|
26
28
  paths = plan.shard(shard)
27
29
  if paths.empty?
@@ -60,6 +62,9 @@ module Polyrun
60
62
  pids << {pid: pid, shard: shard, spawned_at: spawned_at, ping_path: ping_path}
61
63
  Polyrun::Debug.log("[parent pid=#{$$}] run-shards: Process.spawn shard=#{shard} child_pid=#{pid} spec_files=#{paths.size}")
62
64
  Polyrun::Log.warn "polyrun run-shards: started shard #{shard} pid=#{pid} (#{paths.size} file(s))" if parallel
65
+ if Polyrun::WorkerOutput.routing_enabled? && parallel
66
+ Polyrun::Log.warn "polyrun run-shards: shard #{shard} log → #{Polyrun::WorkerOutput.log_path_for(shard)}"
67
+ end
63
68
  end
64
69
  [pids, nil]
65
70
  end
@@ -81,6 +86,8 @@ module Polyrun
81
86
  end
82
87
 
83
88
  def run_shards_spawn_one_worker(child_env, cmd, paths, hook_cfg)
89
+ return Polyrun::WorkerOutput.spawn_worker(child_env, cmd, paths, hook_cfg) if Polyrun::WorkerOutput.routing_enabled?
90
+
84
91
  if hook_cfg.worker_hooks? && !Polyrun::Hooks.disabled?
85
92
  Process.spawn(child_env, "sh", "-c", hook_cfg.build_worker_shell_script(cmd, paths))
86
93
  else
@@ -2,6 +2,10 @@
2
2
  module Polyrun
3
3
  class CLI
4
4
  # Wait, wall/idle timeout, and +after_shard+ hooks for parallel workers (+run-shards+ / +ci-shard-*+).
5
+ #
6
+ # Per-worker wait states: running → normal_exit | wall_timeout (124) | idle_timeout (125).
7
+ # When wall or idle caps are set, the multiplex loop polls every live PID each tick so timeouts
8
+ # apply fairly across shards (not only the worker currently in Process.wait).
5
9
  module RunShardsParallelWait
6
10
  WORKER_TIMEOUT_EXIT_STATUS = 124
7
11
  WORKER_IDLE_TIMEOUT_EXIT_STATUS = 125
@@ -107,6 +111,7 @@ module Polyrun
107
111
  end
108
112
 
109
113
  def run_shards_finalize_reaped_worker!(h, hook_cfg, workers, st, shard_results, after_hook_err)
114
+ Polyrun::WorkerOutput.finish_worker(h[:pid])
110
115
  exitstatus = st.exitstatus
111
116
  ok = st.success?
112
117
  Polyrun::Debug.log("[parent pid=#{$$}] run-shards: Process.wait child_pid=#{h[:pid]} shard=#{h[:shard]} exit=#{exitstatus} success=#{ok}")
@@ -214,8 +219,9 @@ module Polyrun
214
219
  _t, loc = run_shards_read_worker_ping_payload(h[:ping_path])
215
220
  ping_suffix = (loc && !loc.to_s.strip.empty?) ? "; last ping #{loc.to_s.strip}" : ""
216
221
  Polyrun::Log.orchestration_warn "polyrun run-shards: WORKER IDLE TIMEOUT after #{idle_sec}s since last per-example progress ping — shard #{h[:shard]} pid #{h[:pid]}#{ping_suffix}."
222
+ Polyrun::WorkerOutput.warn_shard_log(h[:shard])
217
223
  Polyrun::Log.warn "polyrun run-shards: idle shard file sample: #{sample}#{suffix}"
218
- Polyrun::Log.warn "polyrun run-shards: use Polyrun::RSpec.install_worker_ping! / Polyrun::Minitest.install_worker_ping! (Polyrun Quick calls ping! each example); exit #{WORKER_IDLE_TIMEOUT_EXIT_STATUS}."
224
+ Polyrun::Log.warn "polyrun run-shards: enable per-example worker progress pings in your test setup so idle timeouts reflect real work; exit #{WORKER_IDLE_TIMEOUT_EXIT_STATUS}."
219
225
  end
220
226
 
221
227
  def run_shards_wait_or_force_stop_status(pid)
@@ -258,6 +264,7 @@ module Polyrun
258
264
  ""
259
265
  end
260
266
  Polyrun::Log.orchestration_warn "polyrun run-shards: WORKER TIMEOUT after #{timeout_sec}s (wall time since worker spawn) — shard #{h[:shard]} pid #{h[:pid]}."
267
+ Polyrun::WorkerOutput.warn_shard_log(h[:shard])
261
268
  Polyrun::Log.warn "polyrun run-shards: timeout shard includes: #{sample}#{suffix}"
262
269
  Polyrun::Log.warn "polyrun run-shards: override with --worker-timeout SEC or POLYRUN_WORKER_TIMEOUT_SEC; recorded exit #{WORKER_TIMEOUT_EXIT_STATUS} for this worker."
263
270
  end
@@ -48,7 +48,7 @@ module Polyrun
48
48
  opts.banner = "usage: polyrun run-shards [--workers N] [--worker-timeout SEC] [--worker-idle-timeout SEC] [--strategy NAME] [--paths-file P] [--timing P] [--timing-granularity VAL] [--constraints P] [--seed S] [--merge-coverage] [--merge-output P] [--merge-format LIST] [--merge-failures] [--merge-failures-output P] [--merge-failures-format jsonl|json] [--merge-spec-quality] [--merge-spec-quality-output P] [--no-report-spec-quality] [--] <command> [args...]"
49
49
  opts.on("--workers N", Integer) { |v| st[:workers] = v }
50
50
  opts.on("--worker-timeout SEC", Float, "Max seconds per worker since spawn (also POLYRUN_WORKER_TIMEOUT_SEC); kills stuck workers (exit 124)") { |v| st[:worker_timeout_sec] = v }
51
- opts.on("--worker-idle-timeout SEC", Float, "Max seconds since last valid WorkerPing timestamp in POLYRUN_WORKER_PING_FILE (needs prior ping); RSpec/Minitest: install_worker_ping!; Quick: automatic; exit 125") { |v| st[:worker_idle_timeout_sec] = v }
51
+ opts.on("--worker-idle-timeout SEC", Float, "Max seconds since last worker progress ping (POLYRUN_WORKER_PING_FILE); enable pings in test setup; exit 125") { |v| st[:worker_idle_timeout_sec] = v }
52
52
  opts.on("--strategy NAME", String) do |v|
53
53
  st[:strategy] = v
54
54
  st[:strategy_explicit] = true
@@ -61,10 +61,10 @@ module Polyrun
61
61
  opts.on("--merge-coverage", "After success, merge coverage/polyrun-fragment-*.json (Polyrun coverage must be enabled)") { st[:merge_coverage] = true }
62
62
  opts.on("--merge-output PATH", String) { |v| st[:merge_output] = v }
63
63
  opts.on("--merge-format LIST", String) { |v| st[:merge_format] = v }
64
- opts.on("--merge-failures", "After all workers exit, merge tmp/polyrun_failures/polyrun-failure-fragment-*.jsonl (use Polyrun::RSpec.install_failure_fragments!)") { st[:merge_failures] = true }
64
+ opts.on("--merge-failures", "After all workers exit, merge failure fragments from tmp/polyrun_failures (requires failure fragments in test setup)") { st[:merge_failures] = true }
65
65
  opts.on("--merge-failures-output PATH", String) { |v| st[:merge_failures_output] = v }
66
66
  opts.on("--merge-failures-format VAL", "jsonl (default) or json") { |v| st[:merge_failures_format] = v }
67
- opts.on("--merge-spec-quality", "After workers exit, merge coverage/polyrun-spec-quality-fragment-*.jsonl (POLYRUN_SPEC_QUALITY in workers)") { st[:merge_spec_quality] = true }
67
+ opts.on("--merge-spec-quality", "After workers exit, merge spec-quality fragments from coverage (enable spec-quality collection in test setup)") { st[:merge_spec_quality] = true }
68
68
  opts.on("--merge-spec-quality-output PATH", String) { |v| st[:merge_spec_quality_output] = v }
69
69
  opts.on("--no-report-spec-quality", "Skip printing spec-quality report after merge") { st[:report_spec_quality] = false }
70
70
  end
@@ -124,6 +124,7 @@ module Polyrun
124
124
 
125
125
  def run_shards_warn_interleaved(parallel, pid_count)
126
126
  return unless parallel && pid_count > 1
127
+ return if Polyrun::WorkerOutput.routing_enabled?
127
128
 
128
129
  Polyrun::Log.warn "polyrun run-shards: #{pid_count} children running; RSpec output below may be interleaved."
129
130
  Polyrun::Log.warn "polyrun run-shards: each worker prints its own summary line; the last \"N examples\" line is not a total across shards."
@@ -162,7 +163,7 @@ module Polyrun
162
163
  Polyrun::Log.warn "polyrun run-shards: shard #{s} re-run (same spec list, no interleave): #{rerun}"
163
164
  end
164
165
  unless merge_failures
165
- Polyrun::Log.warn "polyrun run-shards: one merged failure report use run-shards --merge-failures with Polyrun::RSpec.install_failure_fragments!; POLYRUN_MERGED_FAILURES_PATH is set on after_suite when merge runs."
166
+ Polyrun::Log.warn "polyrun run-shards: for one combined failure report, add --merge-failures (requires failure fragments enabled in your test setup)"
166
167
  end
167
168
  end
168
169
  end
@@ -1,3 +1,5 @@
1
+ require_relative "../worker_output"
2
+
1
3
  module Polyrun
2
4
  class CLI
3
5
  # SIGINT/SIGTERM handling and non-blocking reap for parallel worker PIDs (used by run-shards / ci-shard fan-out).
@@ -7,12 +9,17 @@ module Polyrun
7
9
  def run_shards_log_interrupt_workers(pids, _ctx)
8
10
  parts = pids.map { |h| "shard=#{h[:shard]} pid=#{h[:pid]}" }
9
11
  Polyrun::Log.orchestration_warn "polyrun run-shards: SIGINT/SIGTERM while waiting on workers — stopping: #{parts.join(", ")}"
10
- Polyrun::Log.warn "polyrun run-shards: search this log for each shard's started … pid= line and RSpec output; repeat SIGINT during cleanup escalates to SIGKILL"
12
+ if Polyrun::WorkerOutput.routing_enabled?
13
+ Polyrun::Log.warn "polyrun run-shards: per-shard worker logs under #{Polyrun::WorkerOutput.worker_log_directory_label}"
14
+ else
15
+ Polyrun::Log.warn "polyrun run-shards: search this log for each shard's started … pid= line and RSpec output; repeat SIGINT during cleanup escalates to SIGKILL"
16
+ end
11
17
  end
12
18
 
13
19
  # Best-effort worker teardown then exit. Does not return.
14
20
  def run_shards_shutdown_on_signal!(pids, code)
15
21
  run_shards_log_interrupt_workers(pids, nil)
22
+ Polyrun::WorkerOutput.shutdown_all!
16
23
  run_shards_terminate_children!(pids)
17
24
  exit(code)
18
25
  rescue Interrupt
@@ -23,6 +30,7 @@ module Polyrun
23
30
 
24
31
  # Send SIGTERM to each worker PID and wait so Ctrl+C / SIGTERM does not leave orphans.
25
32
  def run_shards_terminate_children!(pids)
33
+ Polyrun::WorkerOutput.shutdown_all!
26
34
  run_shards_signal_workers_term(pids)
27
35
  run_shards_reap_worker_pids_interruptible(pids.map { |h| h[:pid] })
28
36
  end
@@ -96,7 +96,7 @@ module Polyrun
96
96
  def merge_spec_quality_after_shards(ctx)
97
97
  files = merge_spec_quality_fragment_files
98
98
  if files.empty?
99
- Polyrun::Log.warn "polyrun run-shards: --merge-spec-quality: no coverage/polyrun-spec-quality-fragment-*.jsonl found (enable POLYRUN_SPEC_QUALITY in spec_helper?)"
99
+ Polyrun::Log.warn "polyrun run-shards: --merge-spec-quality: no spec-quality fragments found under coverage (enable spec-quality collection in your test setup)"
100
100
  return nil
101
101
  end
102
102
 
@@ -23,14 +23,14 @@ module Polyrun
23
23
  )
24
24
  file_list_html = render_html_partial("file_list", file_rows_html: files.map { |file| html_file_list_row(file) }.join("\n"))
25
25
  file_sections_html = files.map { |file| render_html_partial("file_section", file: file) }.join("\n")
26
- ERB.new(File.read(html_template_path), trim_mode: "-").result_with_hash(
26
+ ERB.new(File.read(html_template_path, encoding: Encoding::UTF_8), trim_mode: "-").result_with_hash(
27
27
  title: CGI.escapeHTML(title.to_s),
28
28
  generated_label: html_generated_label(generated_at),
29
29
  overview_html: overview_html,
30
30
  file_list_html: file_list_html,
31
31
  file_sections_html: file_sections_html,
32
- stylesheet: File.read(html_stylesheet_path),
33
- javascript: File.read(html_javascript_path)
32
+ stylesheet: File.read(html_stylesheet_path, encoding: Encoding::UTF_8),
33
+ javascript: File.read(html_javascript_path, encoding: Encoding::UTF_8)
34
34
  )
35
35
  end
36
36
  # rubocop:enable Metrics/AbcSize
@@ -56,7 +56,7 @@ module Polyrun
56
56
  end
57
57
 
58
58
  def render_html_partial(name, locals = {})
59
- ERB.new(File.read(html_partial_path(name)), trim_mode: "-").result_with_hash(locals)
59
+ ERB.new(File.read(html_partial_path(name), encoding: Encoding::UTF_8), trim_mode: "-").result_with_hash(locals)
60
60
  end
61
61
 
62
62
  def html_file_payload(path, file, root)
@@ -152,7 +152,7 @@ module Polyrun
152
152
  def html_source_lines(path, fallback_length)
153
153
  return Array.new(fallback_length, "") unless File.file?(path.to_s)
154
154
 
155
- File.readlines(path.to_s, chomp: true)
155
+ File.readlines(path.to_s, chomp: true, encoding: Encoding::UTF_8)
156
156
  rescue Errno::ENOENT, Errno::EACCES, ArgumentError
157
157
  Array.new(fallback_length, "")
158
158
  end
@@ -40,12 +40,20 @@ module Polyrun
40
40
 
41
41
  pool = glob_under_cwd(all_glob, cwd)
42
42
  pool.uniq!
43
+ pool = reject_exclude_prefixes(pool, pb)
43
44
  stages = Array(pb["stages"])
44
45
  return sort_paths(pool) if stages.empty?
45
46
 
46
47
  apply_stages_to_pool(stages, pool, cwd)
47
48
  end
48
49
 
50
+ def reject_exclude_prefixes(pool, pb)
51
+ prefixes = Array(pb["exclude_prefixes"]).map(&:to_s).reject(&:empty?)
52
+ return pool if prefixes.empty?
53
+
54
+ pool.reject { |path| prefixes.any? { |prefix| path.start_with?(prefix) } }
55
+ end
56
+
49
57
  def apply_stages_to_pool(stages, pool, cwd)
50
58
  remaining = Set.new(pool)
51
59
  out = []
@@ -6,6 +6,13 @@ require "time"
6
6
  module Polyrun
7
7
  module Queue
8
8
  # File-backed queue (spec_queue.md): +queue.json+, +pending/*.json+ chunks, +done.jsonl+, +leases.json+ (OS flock).
9
+ #
10
+ # Path lifecycle (lease transitions):
11
+ # init! → paths in pending chunks only
12
+ # claim! → pending −batch → active lease in leases.json
13
+ # ack! → lease removed; paths appended to done.jsonl
14
+ # reclaim! → stale or matching lease → paths returned to pending
15
+ # reclaim_lease! → one lease by id → paths returned to pending
9
16
  class FileStore
10
17
  CHUNK_SIZE = 500
11
18
 
@@ -0,0 +1,39 @@
1
+ module Polyrun
2
+ module Queue
3
+ # Claim / run / ack loop for +polyrun run-queue+ workers (forked or direct).
4
+ module WorkerLoop
5
+ module_function
6
+
7
+ def run(store:, worker_id:, batch:, cmd:, on_failure:)
8
+ batches_ok = 0
9
+ batches_fail = 0
10
+ loop do
11
+ claim = store.claim!(worker_id: worker_id, batch_size: batch)
12
+ paths = claim["paths"] || []
13
+ break if paths.empty?
14
+
15
+ code = run_batch(cmd, paths)
16
+ if code == 0
17
+ store.ack!(lease_id: claim["lease_id"], worker_id: worker_id)
18
+ batches_ok += 1
19
+ elsif on_failure.to_s == "requeue"
20
+ store.reclaim_lease!(claim["lease_id"])
21
+ batches_fail += 1
22
+ return {ok: batches_ok, fail: batches_fail, exit_code: 1}
23
+ else
24
+ batches_fail += 1
25
+ return {ok: batches_ok, fail: batches_fail, exit_code: code.zero? ? 1 : code}
26
+ end
27
+ end
28
+ {ok: batches_ok, fail: batches_fail, exit_code: 0}
29
+ rescue Polyrun::Error => e
30
+ Polyrun::Log.warn "polyrun run-queue worker #{worker_id}: #{e.message}"
31
+ {ok: batches_ok, fail: batches_fail, exit_code: 2}
32
+ end
33
+
34
+ def run_batch(cmd, paths)
35
+ system(*cmd, *paths) ? 0 : ($?.exitstatus || 1)
36
+ end
37
+ end
38
+ end
39
+ end
@@ -0,0 +1,158 @@
1
+ require "logger"
2
+ require "timeout"
3
+ require_relative "example_debug_instrumentation"
4
+
5
+ module Polyrun
6
+ module RSpec
7
+ # Per-example RSpec debugging for local investigation (SQL, TracePoint, timeouts).
8
+ #
9
+ # +POLYRUN_EXAMPLE_DEBUG=1+ enables installers; +DEBUG=1+ / +POLYRUN_DEBUG=1+ trace orchestration only.
10
+ # +POLYRUN_DEBUG_SQL=1+ or legacy +DEBUG_SQL=1+ logs mutating SQL.
11
+ # +POLYRUN_DEBUG_TRACE=1+ or legacy +DEBUG_TRACE=1+ traces :call/:raise under the app root.
12
+ # +DEBUG_LOG_LEVEL+ accepts Ruby Logger severities as integers (0 debug … 4 fatal) or names.
13
+ module ExampleDebug
14
+ LOG_LEVEL_BY_NAME = {
15
+ "debug" => Logger::DEBUG,
16
+ "info" => Logger::INFO,
17
+ "warn" => Logger::WARN,
18
+ "error" => Logger::ERROR,
19
+ "fatal" => Logger::FATAL
20
+ }.freeze
21
+
22
+ module_function
23
+
24
+ def enabled?
25
+ truthy?(ENV["POLYRUN_EXAMPLE_DEBUG"])
26
+ end
27
+
28
+ def sql_enabled?
29
+ enabled? && (truthy?(ENV["POLYRUN_DEBUG_SQL"]) || truthy?(ENV["DEBUG_SQL"]))
30
+ end
31
+
32
+ def trace_enabled?
33
+ enabled? && (truthy?(ENV["POLYRUN_DEBUG_TRACE"]) || truthy?(ENV["DEBUG_TRACE"]))
34
+ end
35
+
36
+ def prosopite_enabled?
37
+ enabled? && truthy?(ENV["DEBUG_PROSOPITE"])
38
+ end
39
+
40
+ def print_spec_enabled?
41
+ enabled? && truthy?(ENV["DEBUG_PRINT_SPEC"])
42
+ end
43
+
44
+ def example_timeout_disabled?
45
+ enabled?
46
+ end
47
+
48
+ def log_level
49
+ parse_log_level(ENV.fetch("DEBUG_LOG_LEVEL", "debug"))
50
+ end
51
+
52
+ def rails_log_level
53
+ case log_level
54
+ when Logger::DEBUG then :debug
55
+ when Logger::INFO then :info
56
+ when Logger::WARN then :warn
57
+ when Logger::ERROR then :error
58
+ else :fatal
59
+ end
60
+ end
61
+
62
+ def parse_log_level(raw)
63
+ normalized = raw.to_s.strip.downcase
64
+ return LOG_LEVEL_BY_NAME.fetch(normalized) if LOG_LEVEL_BY_NAME.key?(normalized)
65
+
66
+ Integer(normalized)
67
+ rescue ArgumentError
68
+ Logger::DEBUG
69
+ end
70
+
71
+ def install!(rspec_config: fetch_rspec_configuration!)
72
+ install_spec_path_helpers!(rspec_config)
73
+ install_sql_debug!(rspec_config) if sql_enabled?
74
+ install_trace_debug!(rspec_config) if trace_enabled?
75
+ end
76
+
77
+ def install_rails_logging!(rspec_config: fetch_rspec_configuration!)
78
+ return unless enabled?
79
+ return unless defined?(Rails)
80
+
81
+ rspec_config.before do |example|
82
+ group = example.metadata[:example_group]
83
+ Polyrun::Log.puts "\n\nRunning #{group[:file_path]}:#{group[:line_number]}"
84
+
85
+ level = log_level
86
+ Rails.logger.level = level
87
+ if defined?(ActiveRecord::Base)
88
+ ar_logger = Logger.new(Polyrun::Log.stdout)
89
+ ar_logger.level = level
90
+ ActiveRecord::Base.logger = ar_logger
91
+ end
92
+ end
93
+ end
94
+
95
+ def install_example_timeout!(
96
+ rspec_config,
97
+ seconds: ENV.fetch("RSPEC_EXAMPLE_TIMEOUT_SEC", "30").to_f
98
+ )
99
+ return if seconds <= 0
100
+ return if example_timeout_disabled?
101
+
102
+ rspec_config.around(:each) do |example|
103
+ if example.metadata[:benchmark] || example.metadata[:slow]
104
+ example.run
105
+ else
106
+ Timeout.timeout(seconds) { example.run }
107
+ end
108
+ rescue Timeout::Error
109
+ raise "Example timed out after #{seconds}s (#{example.location})"
110
+ end
111
+ end
112
+
113
+ def install_prosopite!(rspec_config: fetch_rspec_configuration!)
114
+ return unless prosopite_enabled?
115
+ return unless defined?(Prosopite) && defined?(Rails)
116
+
117
+ log_path = Rails.root.join("tmp", "prosopite_#{Time.current.strftime("%Y%m%d_%H%M%S")}.log")
118
+ Prosopite.custom_logger = Logger.new(log_path)
119
+
120
+ rspec_config.before { Prosopite.scan }
121
+ rspec_config.after { Prosopite.finish }
122
+ end
123
+
124
+ def install_spec_path_helpers!(rspec_config)
125
+ rspec_config.before do |example|
126
+ group = example.metadata[:example_group]
127
+ spec_file_path = group[:file_path]
128
+
129
+ define_singleton_method(:spec_dirname) { File.dirname(spec_file_path) }
130
+ define_singleton_method(:spec_basename) { File.basename(spec_file_path) }
131
+
132
+ if ExampleDebug.print_spec_enabled?
133
+ Polyrun::Log.puts "\nRunning #{spec_file_path}:#{group[:line_number]}\n"
134
+ end
135
+ end
136
+
137
+ rspec_config.after do |example|
138
+ next unless ExampleDebug.print_spec_enabled?
139
+
140
+ group = example.metadata[:example_group]
141
+ Polyrun::Log.puts "\nFinished #{group[:file_path]}:#{group[:line_number]}\n"
142
+ end
143
+ end
144
+
145
+ def fetch_rspec_configuration!
146
+ require "rspec/core"
147
+ ::RSpec.configuration
148
+ end
149
+
150
+ def truthy?(value)
151
+ return false if value.nil?
152
+
153
+ %w[1 true yes on].include?(value.to_s.strip.downcase)
154
+ end
155
+ private_class_method :truthy?, :parse_log_level, :fetch_rspec_configuration!
156
+ end
157
+ end
158
+ end
@@ -0,0 +1,73 @@
1
+ module Polyrun
2
+ module RSpec
3
+ module ExampleDebug
4
+ SKIPPED_SQL_PREFIX = /\A(?:SELECT|SET|SHOW|BEGIN|COMMIT|ROLLBACK|RELEASE|SAVEPOINT)/
5
+
6
+ module_function
7
+
8
+ def install_sql_debug!(rspec_config, io: Polyrun::Log.stdout)
9
+ return unless defined?(ActiveSupport::Notifications)
10
+
11
+ rspec_config.around do |example|
12
+ subscriber = ActiveSupport::Notifications.subscribe("sql.active_record") do |event|
13
+ payload = event.payload[:sql]
14
+ next unless loggable_sql?(payload)
15
+
16
+ line = sql_with_interpolated_binds(payload, event.payload[:type_casted_binds])
17
+ io.puts "+ #{line}"
18
+ end
19
+
20
+ example.run
21
+ ensure
22
+ ActiveSupport::Notifications.unsubscribe(subscriber) if subscriber
23
+ end
24
+ end
25
+
26
+ def install_trace_debug!(rspec_config, root: trace_root, io: Polyrun::Log.stdout)
27
+ require "pp"
28
+ root_path = File.expand_path(root.to_s)
29
+ trace = TracePoint.new do |trace_point|
30
+ if trace_point.event == :call && trace_point.path.to_s.start_with?(root_path)
31
+ io.puts PP.pp({event: :call, path: "#{trace_point.path}:#{trace_point.lineno}"}, +"")
32
+ elsif trace_point.event == :raise
33
+ io.puts PP.pp(
34
+ {
35
+ event: :raise,
36
+ raised_exception: trace_point.raised_exception,
37
+ path: "#{trace_point.path}:#{trace_point.lineno}",
38
+ method_id: trace_point.method_id
39
+ },
40
+ +""
41
+ )
42
+ end
43
+ end
44
+
45
+ rspec_config.around do |example|
46
+ trace.enable
47
+ example.run
48
+ ensure
49
+ trace.disable
50
+ end
51
+ end
52
+
53
+ def loggable_sql?(sql)
54
+ !sql.to_s.match?(SKIPPED_SQL_PREFIX)
55
+ end
56
+
57
+ def sql_with_interpolated_binds(sql, binds)
58
+ output = sql.to_s.dup
59
+ Array(binds).each_with_index do |bind, index|
60
+ output = output.gsub("$#{index + 1}", "'#{bind}'")
61
+ end
62
+ output
63
+ end
64
+
65
+ def trace_root
66
+ return Rails.root.to_s if defined?(Rails)
67
+
68
+ Dir.pwd
69
+ end
70
+ private_class_method :trace_root
71
+ end
72
+ end
73
+ end
@@ -0,0 +1,34 @@
1
+ module Polyrun
2
+ module RSpec
3
+ # Formatter tweaks for progress bars (e.g. Fuubar) under +POLYRUN_SHARD_*+ workers.
4
+ module ShardedFormatterCompat
5
+ module_function
6
+
7
+ def install!(rspec_config: fetch_rspec_configuration!)
8
+ return unless sharded_worker?
9
+
10
+ rspec_config.silence_filter_announcements = true
11
+ noop_fuubar_seed! if defined?(Fuubar)
12
+ end
13
+
14
+ def sharded_worker?
15
+ !ENV["POLYRUN_SHARD_TOTAL"].to_s.strip.empty?
16
+ end
17
+
18
+ def noop_fuubar_seed!
19
+ return unless defined?(Fuubar)
20
+
21
+ Fuubar.class_eval do
22
+ def seed(_notification)
23
+ end
24
+ end
25
+ end
26
+
27
+ def fetch_rspec_configuration!
28
+ require "rspec/core"
29
+ ::RSpec.configuration
30
+ end
31
+ private_class_method :fetch_rspec_configuration!
32
+ end
33
+ end
34
+ end
data/lib/polyrun/rspec.rb CHANGED
@@ -21,6 +21,8 @@ module Polyrun
21
21
  if output_path
22
22
  op = output_path
23
23
  Class.new(Polyrun::Timing::RSpecExampleFormatter) do
24
+ ::RSpec::Core::Formatters.register self, :example_finished, :close
25
+
24
26
  define_method(:timing_output_path) { op }
25
27
  end
26
28
  else
@@ -70,5 +72,39 @@ module Polyrun
70
72
 
71
73
  Polyrun::WorkerPing.ensure_interval_ping_thread!
72
74
  end
75
+
76
+ def install_example_debug!(rspec_config: nil)
77
+ require_relative "rspec/example_debug"
78
+ Polyrun::RSpec::ExampleDebug.install!(rspec_config: rspec_config || fetch_rspec_configuration!)
79
+ end
80
+
81
+ def install_example_timeout!(rspec_config: nil, seconds: ENV.fetch("RSPEC_EXAMPLE_TIMEOUT_SEC", "30").to_f)
82
+ require_relative "rspec/example_debug"
83
+ Polyrun::RSpec::ExampleDebug.install_example_timeout!(
84
+ rspec_config || fetch_rspec_configuration!,
85
+ seconds: seconds
86
+ )
87
+ end
88
+
89
+ def install_example_rails_logging!(rspec_config: nil)
90
+ require_relative "rspec/example_debug"
91
+ Polyrun::RSpec::ExampleDebug.install_rails_logging!(rspec_config: rspec_config || fetch_rspec_configuration!)
92
+ end
93
+
94
+ def install_example_prosopite!(rspec_config: nil)
95
+ require_relative "rspec/example_debug"
96
+ Polyrun::RSpec::ExampleDebug.install_prosopite!(rspec_config: rspec_config || fetch_rspec_configuration!)
97
+ end
98
+
99
+ def install_sharded_formatter_compat!(rspec_config: nil)
100
+ require_relative "rspec/sharded_formatter_compat"
101
+ Polyrun::RSpec::ShardedFormatterCompat.install!(rspec_config: rspec_config || fetch_rspec_configuration!)
102
+ end
103
+
104
+ def fetch_rspec_configuration!
105
+ require "rspec/core"
106
+ ::RSpec.configuration
107
+ end
108
+ private_class_method :fetch_rspec_configuration!
73
109
  end
74
110
  end
@@ -1,6 +1,6 @@
1
1
  require "json"
2
2
 
3
- require "rspec/core/formatters/base_formatter"
3
+ require "rspec/core/formatters"
4
4
 
5
5
  module Polyrun
6
6
  module Timing
@@ -13,23 +13,23 @@ module Polyrun
13
13
  # Or {Polyrun::RSpec.install_example_timing!} (+output_path:+ avoids touching +ENV+).
14
14
  #
15
15
  # Default output path: +ENV["POLYRUN_EXAMPLE_TIMING_OUT"]+ if set, else +polyrun_timing_examples.json+.
16
- class RSpecExampleFormatter < RSpec::Core::Formatters::BaseFormatter
17
- RSpec::Core::Formatters.register self, :example_finished, :close
16
+ class RSpecExampleFormatter
17
+ ::RSpec::Core::Formatters.register self, :example_finished, :close
18
18
 
19
19
  def initialize(output)
20
- super
20
+ @output = output
21
21
  @times = {}
22
22
  end
23
23
 
24
24
  def example_finished(notification)
25
25
  ex = notification.example
26
- result = ex.execution_result
27
- return if result.pending?
26
+ return if ex.pending?
28
27
 
28
+ result = ex.execution_result
29
29
  t = result.run_time
30
30
  return unless t
31
31
 
32
- path = ex.metadata[:absolute_path]
32
+ path = example_absolute_path(ex)
33
33
  return unless path
34
34
 
35
35
  line = ex.metadata[:line_number]
@@ -48,6 +48,13 @@ module Polyrun
48
48
  def timing_output_path
49
49
  ENV["POLYRUN_EXAMPLE_TIMING_OUT"] || "polyrun_timing_examples.json"
50
50
  end
51
+
52
+ private
53
+
54
+ def example_absolute_path(example)
55
+ metadata = example.metadata
56
+ metadata[:absolute_file_path] || metadata[:file_path] || metadata[:absolute_path]
57
+ end
51
58
  end
52
59
  end
53
60
  end
@@ -1,3 +1,3 @@
1
1
  module Polyrun
2
- VERSION = "2.1.0"
2
+ VERSION = "2.1.3"
3
3
  end
@@ -0,0 +1,159 @@
1
+ require "fileutils"
2
+ require_relative "worker_output_forwarders"
3
+
4
+ module Polyrun
5
+ # Routes parallel worker stdout/stderr through per-shard log files with optional prefixed TTY echo.
6
+ #
7
+ # Opt-in: +POLYRUN_WORKER_OUTPUT_ROUTING=1+ (or set +POLYRUN_WORKER_LOG_DIR+).
8
+ # +POLYRUN_WORKER_OUTPUT_ROUTING=0+ keeps inherited stdio (default polyrun behavior).
9
+ # +POLYRUN_WORKER_OUTPUT_PREFIX=0+ writes logs only (no live prefixed echo).
10
+ module WorkerOutput
11
+ module_function
12
+
13
+ def routing_enabled?
14
+ return false if disabled_by_env?
15
+
16
+ truthy?(ENV["POLYRUN_WORKER_OUTPUT_ROUTING"]) || !log_directory.nil?
17
+ end
18
+
19
+ def log_directory
20
+ value = ENV["POLYRUN_WORKER_LOG_DIR"]
21
+ return nil if value.nil?
22
+
23
+ stripped = value.to_s.strip
24
+ stripped.empty? ? nil : stripped
25
+ end
26
+
27
+ def prefix_live?
28
+ return true if ENV["POLYRUN_WORKER_OUTPUT_PREFIX"].nil?
29
+
30
+ truthy?(ENV["POLYRUN_WORKER_OUTPUT_PREFIX"])
31
+ end
32
+
33
+ def log_path_for(shard)
34
+ File.expand_path("shard-#{shard}.log", log_directory || default_log_directory)
35
+ end
36
+
37
+ def worker_log_directory_label
38
+ log_directory || default_log_directory
39
+ end
40
+
41
+ def prepare_log_dir!
42
+ directory = log_directory || default_log_directory
43
+ FileUtils.mkdir_p(directory)
44
+ Dir.glob(File.join(directory, "shard-*.log")).each { |path| File.delete(path) }
45
+ end
46
+
47
+ def warn_shard_log(shard)
48
+ return unless routing_enabled?
49
+
50
+ Polyrun::Log.warn "polyrun run-shards: shard #{shard} worker log → #{log_path_for(shard)}"
51
+ end
52
+
53
+ def spawn_worker(child_env, cmd, paths, hook_cfg)
54
+ shard = child_env.fetch("POLYRUN_SHARD_INDEX", "?")
55
+ out_read, out_write = IO.pipe
56
+ err_read, err_write = IO.pipe
57
+
58
+ pid =
59
+ if hook_cfg.worker_hooks? && !Polyrun::Hooks.disabled?
60
+ Process.spawn(
61
+ child_env,
62
+ "sh",
63
+ "-c",
64
+ hook_cfg.build_worker_shell_script(cmd, paths),
65
+ out: out_write,
66
+ err: err_write
67
+ )
68
+ else
69
+ Process.spawn(child_env, *cmd, *paths, out: out_write, err: err_write)
70
+ end
71
+
72
+ out_write.close
73
+ err_write.close
74
+
75
+ start_forwarders(
76
+ pid: pid,
77
+ shard: shard,
78
+ stdout_io: out_read,
79
+ stderr_io: err_read,
80
+ log_path: log_path_for(shard)
81
+ )
82
+ pid
83
+ end
84
+
85
+ def start_forwarders(pid:, shard:, stdout_io:, stderr_io:, log_path:)
86
+ forwarder = WorkerForwarder.new(
87
+ shard: shard,
88
+ pid: pid,
89
+ log_path: log_path,
90
+ prefix_live: prefix_live?
91
+ )
92
+ threads = [
93
+ forwarder_thread(stdout_io, forwarder, :stdout),
94
+ forwarder_thread(stderr_io, forwarder, :stderr)
95
+ ]
96
+ registry[pid] = {threads: threads, ios: [stdout_io, stderr_io], forwarder: forwarder}
97
+ end
98
+
99
+ def finish_worker(pid)
100
+ entry = registry.delete(pid)
101
+ return unless entry
102
+
103
+ drain_forwarders(entry)
104
+ end
105
+
106
+ def shutdown_all!
107
+ registry.each_value { |entry| drain_forwarders(entry) }
108
+ registry.clear
109
+ end
110
+
111
+ def default_log_directory
112
+ "tmp/polyrun/workers"
113
+ end
114
+
115
+ def disabled_by_env?
116
+ %w[0 false no off].include?(ENV["POLYRUN_WORKER_OUTPUT_ROUTING"].to_s.strip.downcase)
117
+ end
118
+
119
+ def truthy?(value)
120
+ return false if value.nil?
121
+
122
+ %w[1 true yes on].include?(value.to_s.strip.downcase)
123
+ end
124
+ private_class_method :truthy?
125
+
126
+ # rubocop:disable ThreadSafety/ClassInstanceVariable -- per-parent-process worker forwarder registry
127
+ def registry
128
+ @registry ||= {}
129
+ end
130
+ private_class_method :registry
131
+ # rubocop:enable ThreadSafety/ClassInstanceVariable
132
+
133
+ def forwarder_thread(io, forwarder, stream)
134
+ # rubocop:disable ThreadSafety/NewThread -- one reader thread per worker pipe
135
+ Thread.new do
136
+ loop do
137
+ forwarder.consume(stream, io.readpartial(4096))
138
+ end
139
+ rescue IOError, Errno::EPIPE
140
+ # worker closed the pipe
141
+ ensure
142
+ io.close unless io.closed?
143
+ end
144
+ # rubocop:enable ThreadSafety/NewThread
145
+ end
146
+ private_class_method :forwarder_thread
147
+
148
+ def drain_forwarders(entry)
149
+ entry[:ios].each do |io|
150
+ io.close unless io.closed?
151
+ rescue IOError
152
+ nil
153
+ end
154
+ entry[:threads].each(&:join)
155
+ entry[:forwarder].close
156
+ end
157
+ private_class_method :drain_forwarders
158
+ end
159
+ end
@@ -0,0 +1,90 @@
1
+ require "time"
2
+
3
+ module Polyrun
4
+ module WorkerOutput
5
+ class LineForwarder
6
+ def initialize(shard:, pid:, log_io:, prefix_live:, tty_io:)
7
+ @shard = shard
8
+ @pid = pid
9
+ @log_io = log_io
10
+ @prefix_live = prefix_live
11
+ @tty_io = tty_io
12
+ @buffer = +""
13
+ @mutex = Mutex.new
14
+ end
15
+
16
+ def consume(chunk)
17
+ return if chunk.nil? || chunk.empty?
18
+
19
+ @mutex.synchronize do
20
+ if progress_chunk?(chunk)
21
+ write_tty(chunk)
22
+ @log_io&.write(chunk)
23
+ return
24
+ end
25
+
26
+ @buffer << chunk
27
+ while (newline_index = @buffer.index("\n"))
28
+ emit_line(@buffer.slice!(0, newline_index + 1))
29
+ end
30
+ end
31
+ end
32
+
33
+ def flush
34
+ @mutex.synchronize do
35
+ next if @buffer.empty?
36
+
37
+ emit_line(@buffer)
38
+ @buffer.clear
39
+ end
40
+ end
41
+
42
+ private
43
+
44
+ def progress_chunk?(chunk)
45
+ chunk.include?("\r") && !chunk.include?("\n")
46
+ end
47
+
48
+ def emit_line(line)
49
+ prefixed = "#{prefix}#{line}"
50
+ @log_io&.write(prefixed)
51
+ write_tty(prefixed) if @prefix_live
52
+ end
53
+
54
+ def prefix
55
+ "[#{Time.now.strftime("%Y-%m-%dT%H:%M:%S")} shard=#{@shard} pid=#{@pid}] "
56
+ end
57
+
58
+ def write_tty(text)
59
+ @tty_io.write(text)
60
+ @tty_io.flush
61
+ end
62
+ end
63
+
64
+ class WorkerForwarder
65
+ def initialize(shard:, pid:, log_path:, prefix_live:)
66
+ @shard = shard
67
+ @pid = pid
68
+ @log_io = File.open(log_path, "wb")
69
+ @stdout = LineForwarder.new(shard: shard, pid: pid, log_io: @log_io, prefix_live: prefix_live, tty_io: Polyrun::Log.stdout)
70
+ @stderr = LineForwarder.new(shard: shard, pid: pid, log_io: @log_io, prefix_live: prefix_live, tty_io: Polyrun::Log.stderr)
71
+ @write_mutex = Mutex.new
72
+ end
73
+
74
+ def consume(stream, chunk)
75
+ @write_mutex.synchronize do
76
+ case stream
77
+ when :stdout then @stdout.consume(chunk)
78
+ when :stderr then @stderr.consume(chunk)
79
+ end
80
+ end
81
+ end
82
+
83
+ def close
84
+ @stdout.flush
85
+ @stderr.flush
86
+ @log_io.close
87
+ end
88
+ end
89
+ end
90
+ end
@@ -7,5 +7,15 @@ module Polyrun
7
7
  def self.install_failure_fragments!: (?only_if: untyped?) -> void
8
8
 
9
9
  def self.install_worker_ping!: () -> void
10
+
11
+ def self.install_example_debug!: (?rspec_config: untyped?) -> void
12
+
13
+ def self.install_example_timeout!: (?rspec_config: untyped?, ?seconds: Float) -> void
14
+
15
+ def self.install_example_rails_logging!: (?rspec_config: untyped?) -> void
16
+
17
+ def self.install_example_prosopite!: (?rspec_config: untyped?) -> void
18
+
19
+ def self.install_sharded_formatter_compat!: (?rspec_config: untyped?) -> void
10
20
  end
11
21
  end
@@ -0,0 +1,11 @@
1
+ module Polyrun
2
+ module WorkerOutput
3
+ def self.routing_enabled?: () -> bool
4
+ def self.log_path_for: (Integer | String shard) -> String
5
+ def self.prepare_log_dir!: () -> void
6
+ def self.warn_shard_log: (Integer | String shard) -> void
7
+ def self.spawn_worker: (Hash[Symbol, untyped] child_env, Array[String] cmd, Array[String] paths, untyped hook_cfg) -> Integer
8
+ def self.finish_worker: (Integer pid) -> void
9
+ def self.shutdown_all!: () -> void
10
+ end
11
+ end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: polyrun
3
3
  version: !ruby/object:Gem::Version
4
- version: 2.1.0
4
+ version: 2.1.3
5
5
  platform: ruby
6
6
  authors:
7
7
  - Andrei Makarov
@@ -266,6 +266,7 @@ files:
266
266
  - lib/polyrun/queue/duration.rb
267
267
  - lib/polyrun/queue/file_store.rb
268
268
  - lib/polyrun/queue/file_store_pending.rb
269
+ - lib/polyrun/queue/worker_loop.rb
269
270
  - lib/polyrun/quick.rb
270
271
  - lib/polyrun/quick/assertions.rb
271
272
  - lib/polyrun/quick/errors.rb
@@ -281,6 +282,9 @@ files:
281
282
  - lib/polyrun/reporting/rspec_failure_fragment_formatter.rb
282
283
  - lib/polyrun/reporting/rspec_junit.rb
283
284
  - lib/polyrun/rspec.rb
285
+ - lib/polyrun/rspec/example_debug.rb
286
+ - lib/polyrun/rspec/example_debug_instrumentation.rb
287
+ - lib/polyrun/rspec/sharded_formatter_compat.rb
284
288
  - lib/polyrun/spec_quality.rb
285
289
  - lib/polyrun/spec_quality/config.rb
286
290
  - lib/polyrun/spec_quality/fragment.rb
@@ -303,6 +307,8 @@ files:
303
307
  - lib/polyrun/timing/summary.rb
304
308
  - lib/polyrun/timing/variance_report.rb
305
309
  - lib/polyrun/version.rb
310
+ - lib/polyrun/worker_output.rb
311
+ - lib/polyrun/worker_output_forwarders.rb
306
312
  - lib/polyrun/worker_ping.rb
307
313
  - polyrun.gemspec
308
314
  - sig/polyrun.rbs
@@ -313,6 +319,7 @@ files:
313
319
  - sig/polyrun/minitest.rbs
314
320
  - sig/polyrun/quick.rbs
315
321
  - sig/polyrun/rspec.rbs
322
+ - sig/polyrun/worker_output.rbs
316
323
  - sig/polyrun/worker_ping.rbs
317
324
  homepage: https://github.com/amkisko/polyrun.rb
318
325
  licenses: