polyrun 1.3.0 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +8 -0
- data/README.md +58 -0
- data/lib/polyrun/cli/ci_shard_hooks.rb +121 -0
- data/lib/polyrun/cli/ci_shard_run_command.rb +6 -22
- data/lib/polyrun/cli/help.rb +1 -0
- data/lib/polyrun/cli/hooks_command.rb +97 -0
- data/lib/polyrun/cli/run_shards_parallel_children.rb +92 -0
- data/lib/polyrun/cli/run_shards_run.rb +54 -64
- data/lib/polyrun/cli.rb +5 -1
- data/lib/polyrun/config.rb +5 -0
- data/lib/polyrun/hooks/dsl.rb +128 -0
- data/lib/polyrun/hooks/worker_runner.rb +27 -0
- data/lib/polyrun/hooks/worker_shell.rb +50 -0
- data/lib/polyrun/hooks.rb +185 -0
- data/lib/polyrun/version.rb +1 -1
- data/lib/polyrun.rb +1 -0
- metadata +8 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 04bfc3ed1a2c01072864dd179decce4d13c68221bba4da0964f4ea600401b57c
|
|
4
|
+
data.tar.gz: df02fc828fb8ac8c9cf3792ae08420ee87d328a89d79a7eeec26c9e239506ee4
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 378921ebc46b80562c5ae4bb529a3035eed7366cbfbcffd9a5c4bcb1d18f1c1fe6b737dfcc580c9b7e3fb1ce4360ac9ad04c7a7edbc50e4763a2e5430a61c873
|
|
7
|
+
data.tar.gz: 552545582ab8c34e1411834d5f5ef6d1b6f1d022ead68746059b709cd04d733c6d778b13621c003ec2f4525367bdb5e849577d109e2e04835fcf9b12407bf2ef
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,13 @@
|
|
|
1
1
|
# CHANGELOG
|
|
2
2
|
|
|
3
|
+
## 1.4.0 (2026-04-16)
|
|
4
|
+
|
|
5
|
+
- Add `hooks:` in `polyrun.yml` — shell commands for `before_suite` / `after_suite`, `before_shard` / `after_shard`, `before_worker` / `after_worker` (RSpec-style YAML keys `before(:suite)`, `before(:all)`, `before(:each)` accepted). Wire hooks into `run-shards`, `parallel-rspec`, and `ci-shard-*`.
|
|
6
|
+
- Add `hooks.ruby` / `hooks.ruby_file` and `Polyrun::Hooks::Dsl` (`before(:suite)` … `after(:each)` blocks); worker Ruby hooks run in the child via `ruby -e` + `POLYRUN_HOOKS_RUBY_FILE`.
|
|
7
|
+
- Add `polyrun hook run <phase>` (`--shard` / `--total` optional). Set `POLYRUN_HOOKS_DISABLE=1` to skip hooks during orchestration only; `hook run` still executes.
|
|
8
|
+
- On `ci-shard-run` / `ci-shard-rspec`, skip automatic `before_suite` / `after_suite` when `POLYRUN_SHARD_TOTAL` > 1 (matrix); run suite hooks once via `polyrun hook run` or set `POLYRUN_HOOKS_SUITE_PER_MATRIX_JOB=1` to run them on every matrix job.
|
|
9
|
+
- Document hook phases, matrix vs suite, and `after_shard` ordering in `README.md`; list `Polyrun::Hooks` and `Polyrun::Hooks::Dsl` in the library section.
|
|
10
|
+
|
|
3
11
|
## 1.3.0 (2026-04-15)
|
|
4
12
|
|
|
5
13
|
- Add safe parsing for `ci-shard-run` / `ci-shard-rspec` `--shard-processes` and `--workers` (warn + exit 2 on missing or non-integer values).
|
data/README.md
CHANGED
|
@@ -25,6 +25,61 @@ Capybara and Playwright stay in your application; Polyrun does not replace brows
|
|
|
25
25
|
4. Run workers with `bin/polyrun run-shards --workers N -- bundle exec rspec`: N separate OS processes, each running RSpec with its own file list from `partition.paths_file`, or `spec/spec_paths.txt`, or else `spec/**/*_spec.rb`. Stderr shows where paths came from; after a successful multi-worker run it reminds you to run merge-coverage unless you use `parallel-rspec` or `run-shards --merge-coverage`.
|
|
26
26
|
5. Merge artifacts with `bin/polyrun merge-coverage` on `coverage/polyrun-fragment-*.json` (one fragment per `POLYRUN_SHARD_INDEX` when coverage is on), or use `bin/polyrun parallel-rspec` or `run-shards --merge-coverage` so Polyrun runs merge for you. Optional: `merge-timing`, `report-timing`, `report-junit`.
|
|
27
27
|
|
|
28
|
+
### Hooks (`hooks:` in `polyrun.yml`)
|
|
29
|
+
|
|
30
|
+
Optional **shell** commands and/or a **Ruby DSL** file for instrumentation (telemetry, Slack, logging, manual debugging). Names mirror RSpec’s API (`before(:suite)`, `before(:all)`, `before(:each)`), but **Polyrun hooks are about process orchestration**, not RSpec example groups. Below, **suite / shard / worker** mean Polyrun’s model unless stated otherwise.
|
|
31
|
+
|
|
32
|
+
#### What “suite”, “shard”, and “worker” mean
|
|
33
|
+
|
|
34
|
+
| Term | Process | Meaning |
|
|
35
|
+
|------|---------|---------|
|
|
36
|
+
| **Suite** | **Parent** only | One **orchestration run** on a single machine: a single `polyrun run-shards` / `parallel-rspec` / `ci-shard-run` (with **one** global shard, see below). `before_suite` runs once before any worker is started; `after_suite` runs once after all workers have exited and (when used) merge-coverage has finished. This is **not** the same as “the whole RSpec suite in one process”—with `--workers N`, RSpec runs in **N separate processes**, each with its own examples. |
|
|
37
|
+
| **Shard** | **Parent** only | One **partition** of the path list for this run, identified by `POLYRUN_SHARD_INDEX` / `POLYRUN_SHARD_TOTAL` **for that parallel layout** (0 … N−1 for `run-shards` with N workers). `before_shard` runs in the parent **immediately before** `Process.spawn` for that index; `after_shard` runs **after** that child has exited. The parent **waits workers in shard index order** (0, then 1, …), so `after_shard` runs in that order—not in “who finished first” order if workers overlap in time. Empty partitions are skipped (no spawn, no hooks). |
|
|
38
|
+
| **Worker** | **Child** OS process | The process that runs your command after `--` (e.g. `bundle exec rspec …`). `before_worker` / `after_worker` run **in that child**, directly before and after the test command. **Individual examples run only after `before_worker` completes** (inside the same process as RSpec/Minitest). |
|
|
39
|
+
|
|
40
|
+
**CI matrix** (`POLYRUN_SHARD_TOTAL` > 1, one job per index): each job is a **global shard**, not a full “suite” in the pipeline sense. **`ci-shard-run` / `ci-shard-rspec` with one process per job do not run `before_suite` / `after_suite` automatically**—otherwise they would run once per matrix cell. Put pipeline-wide setup/teardown in a **separate CI step** (e.g. `bin/polyrun hook run before_suite` and `hook run after_suite` once), or set `POLYRUN_HOOKS_SUITE_PER_MATRIX_JOB=1` to restore the old behaviour (suite hooks on every matrix job). **`before_shard` / `after_shard` / worker hooks still run** per job, with `POLYRUN_SHARD_INDEX` / `POLYRUN_SHARD_TOTAL` set from the matrix. A **single** non-matrix `ci-shard-run` (`POLYRUN_SHARD_TOTAL` is 1) still runs suite hooks like `run-shards --workers 1`. Fan-out on one host (`--shard-processes` > 1) still runs **`before_suite` / `after_suite` once** for that job, around the local workers.
|
|
41
|
+
|
|
42
|
+
#### Lifecycle (typical `run-shards` with multiple workers)
|
|
43
|
+
|
|
44
|
+
```text
|
|
45
|
+
[Parent] before_suite
|
|
46
|
+
[Parent] for each shard index i that has paths:
|
|
47
|
+
before_shard(i) → spawn worker i
|
|
48
|
+
[Child i] before_worker → test runner starts → examples run → runner exits
|
|
49
|
+
[Parent] after_shard(i) (parent waits children in shard index order 0…N−1, not global finish order)
|
|
50
|
+
[Parent] merge-coverage (if requested)
|
|
51
|
+
[Parent] after_suite
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
`after_worker` runs in the child after the test command exits, before the parent’s `after_shard`.
|
|
55
|
+
|
|
56
|
+
#### Order and priority within one phase
|
|
57
|
+
|
|
58
|
+
1. **Ruby DSL, then shell (YAML)** — For the same phase (e.g. `before_suite`), all **Ruby** blocks from `hooks.ruby` run first, then every **shell** command from YAML for that phase.
|
|
59
|
+
2. **Multiple Ruby blocks** — In one DSL file, registrations run in **source order** (e.g. two `before(:suite)` blocks run top to bottom).
|
|
60
|
+
3. **Multiple shell commands** — Use a **YAML list**; entries run in list order:
|
|
61
|
+
```yaml
|
|
62
|
+
before_suite:
|
|
63
|
+
- echo first
|
|
64
|
+
- echo second
|
|
65
|
+
```
|
|
66
|
+
4. **Duplicate YAML keys** — Do **not** repeat the same key (e.g. two `before_suite:` lines). Parsers may keep only one value; behaviour is undefined. Prefer a **list** under a single key.
|
|
67
|
+
5. **Failure** — If any step in a phase fails (non-zero exit from shell; uncaught error in Ruby), orchestration stops or marks failure per existing `run-shards` rules; `after_worker` shell steps use `|| true` so a failing teardown does not mask the test exit code (Ruby `after_worker` is wrapped similarly in the worker script).
|
|
68
|
+
|
|
69
|
+
Environment includes `POLYRUN_HOOK_PHASE`, `POLYRUN_HOOK=1`, `POLYRUN_HOOK_ORCHESTRATOR` (`1` in parent, `0` in workers), `POLYRUN_SHARD_*`, and `POLYRUN_SUITE_EXIT_STATUS` on `after_suite`. Worker children get `POLYRUN_HOOKS_RUBY_FILE` when using the Ruby DSL. Set `POLYRUN_HOOKS_DISABLE=1` to skip hooks during `run-shards` / `parallel-rspec` / `ci-shard-*` (orchestration only); `polyrun hook run` still executes hooks. **`POLYRUN_HOOKS_SUITE_PER_MATRIX_JOB=1`** — when set, run `before_suite` / `after_suite` on every CI matrix job (not recommended for expensive global setup).
|
|
70
|
+
|
|
71
|
+
**YAML keys (shell) and RSpec-style names in YAML:**
|
|
72
|
+
|
|
73
|
+
| YAML key | DSL in `hooks.ruby` | When |
|
|
74
|
+
|----------|----------------------|------|
|
|
75
|
+
| `before_suite` / `after_suite` | `before(:suite)` / `after(:suite)` | Parent: once per orchestration on one host; **skipped** for `ci-shard-run` when `POLYRUN_SHARD_TOTAL` > 1 unless `POLYRUN_HOOKS_SUITE_PER_MATRIX_JOB=1` (see matrix paragraph above) |
|
|
76
|
+
| `before_shard` / `after_shard` | `before(:all)` / `after(:all)` | Parent: per shard index (spawn / after exit) |
|
|
77
|
+
| `before_worker` / `after_worker` | `before(:each)` / `after(:each)` | Child: around the test command |
|
|
78
|
+
|
|
79
|
+
**Ruby DSL (`hooks.ruby` or `hooks.ruby_file`):** path to a `.rb` file (relative to the project root). Blocks receive a `Hash` with **string keys** (same env as shell hooks). Worker-phase Ruby hooks run in the child via `ruby -e 'require "polyrun"; …'`.
|
|
80
|
+
|
|
81
|
+
Run one phase by hand: `bin/polyrun hook run before_suite` (optional `--shard N --total M`). YAML may use quoted keys such as `"before(:suite)"` instead of `before_suite`.
|
|
82
|
+
|
|
28
83
|
Quick CLI samples:
|
|
29
84
|
|
|
30
85
|
If the current directory already has `polyrun.yml` or `config/polyrun.yml`, you can omit `-c` (same as `Config.load` default discovery). Pass `-c PATH` or set `POLYRUN_CONFIG` when the file lives elsewhere or uses another name.
|
|
@@ -41,6 +96,7 @@ bin/polyrun env --shard 0 --total 4 # print DATABASE_URL exports from polyrun.
|
|
|
41
96
|
bin/polyrun init --list
|
|
42
97
|
bin/polyrun init --profile gem -o polyrun.yml # starter YAML; see docs/SETUP_PROFILE.md
|
|
43
98
|
bin/polyrun quick # Polyrun::Quick examples under spec/polyrun_quick/ or test/polyrun_quick/
|
|
99
|
+
bin/polyrun hook run before_suite # run hooks.before_suite from polyrun.yml (manual / CI)
|
|
44
100
|
```
|
|
45
101
|
|
|
46
102
|
### Matrix shards and timing
|
|
@@ -87,6 +143,8 @@ That single require loads the CLI and core library **without** loading RSpec or
|
|
|
87
143
|
| `Polyrun::Prepare::Assets` | Digest trees, marker file, `assets:precompile`. |
|
|
88
144
|
| `Polyrun::Database::Shard` | Shard env map, `%{shard}` DB names, URL path suffix for `postgres://`, `mysql2://`, `mongodb://`, etc. |
|
|
89
145
|
| `Polyrun::Database::UrlBuilder` | URLs from `polyrun.yml` `databases:` — nested blocks or `adapter:` for common Rails stacks (`postgresql`, `mysql`/`mysql2`, `trilogy`, `sqlserver`/`mssql`, `sqlite3`/`sqlite`, `mongodb`/`mongo`). |
|
|
146
|
+
| `Polyrun::Hooks` | Load from `Config#hooks`; `run_phase` / `run_phase_if_enabled`; `build_worker_shell_script` wraps the worker command. |
|
|
147
|
+
| `Polyrun::Hooks::Dsl` | Ruby hook file (`hooks.ruby`); `before(:suite)` / `after(:each)` etc. in `config/polyrun_hooks.rb` (see README). |
|
|
90
148
|
|
|
91
149
|
## Development
|
|
92
150
|
|
|
@@ -0,0 +1,121 @@
|
|
|
1
|
+
module Polyrun
|
|
2
|
+
class CLI
|
|
3
|
+
# Suite / shard / worker shell hooks for +ci-shard-run+ / +ci-shard-rspec+.
|
|
4
|
+
module CiShardHooks
|
|
5
|
+
private
|
|
6
|
+
|
|
7
|
+
# rubocop:disable Metrics/AbcSize -- suite hooks + spawn/wait + failure paths
|
|
8
|
+
def ci_shard_run_fanout!(ctx)
|
|
9
|
+
hook_cfg = Polyrun::Hooks.from_config(ctx[:cfg])
|
|
10
|
+
suite_started = false
|
|
11
|
+
exit_code = 1
|
|
12
|
+
|
|
13
|
+
begin
|
|
14
|
+
env_suite = ENV.to_h.merge(
|
|
15
|
+
"POLYRUN_HOOK_ORCHESTRATOR" => "1",
|
|
16
|
+
"POLYRUN_SHARD_TOTAL" => ctx[:workers].to_s
|
|
17
|
+
)
|
|
18
|
+
code = hook_cfg.run_phase_if_enabled(:before_suite, env_suite)
|
|
19
|
+
return code if code != 0
|
|
20
|
+
|
|
21
|
+
suite_started = true
|
|
22
|
+
|
|
23
|
+
pids, spawn_err = run_shards_spawn_workers(ctx, hook_cfg)
|
|
24
|
+
if spawn_err
|
|
25
|
+
exit_code = spawn_err
|
|
26
|
+
return spawn_err
|
|
27
|
+
end
|
|
28
|
+
return 1 if pids.empty?
|
|
29
|
+
|
|
30
|
+
run_shards_warn_interleaved(ctx[:parallel], pids.size)
|
|
31
|
+
shard_results, wait_hook_err = run_shards_wait_all_children(pids, hook_cfg, ctx)
|
|
32
|
+
failed = shard_results.reject { |r| r[:success] }.map { |r| r[:shard] }
|
|
33
|
+
|
|
34
|
+
if failed.any?
|
|
35
|
+
Polyrun::Log.warn "polyrun ci-shard: finished #{pids.size} worker(s) (some failed)"
|
|
36
|
+
run_shards_log_failed_reruns(failed, shard_results, ctx[:plan], ctx[:parallel], ctx[:workers], ctx[:cmd])
|
|
37
|
+
exit_code = 1
|
|
38
|
+
exit_code = 1 if wait_hook_err != 0
|
|
39
|
+
return exit_code
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
exit_code = (wait_hook_err == 0) ? 0 : 1
|
|
43
|
+
Polyrun::Log.warn "polyrun ci-shard: finished #{pids.size} worker(s) (exit 0)" if exit_code == 0
|
|
44
|
+
exit_code
|
|
45
|
+
ensure
|
|
46
|
+
if suite_started
|
|
47
|
+
env_after = ENV.to_h.merge(
|
|
48
|
+
"POLYRUN_HOOK_ORCHESTRATOR" => "1",
|
|
49
|
+
"POLYRUN_SHARD_TOTAL" => ctx[:workers].to_s,
|
|
50
|
+
"POLYRUN_SUITE_EXIT_STATUS" => exit_code.to_s
|
|
51
|
+
)
|
|
52
|
+
hook_cfg.run_phase_if_enabled(:after_suite, env_after)
|
|
53
|
+
end
|
|
54
|
+
end
|
|
55
|
+
end
|
|
56
|
+
# rubocop:enable Metrics/AbcSize
|
|
57
|
+
|
|
58
|
+
# One matrix shard, one OS process: same hook phases as +run-shards+ with +--workers 1+ (no +exec+ when hooks exist).
|
|
59
|
+
# rubocop:disable Metrics/AbcSize -- suite / shard / worker lifecycle
|
|
60
|
+
def ci_shard_run_single!(cmd, paths, cfg, pc, _config_path)
|
|
61
|
+
hook_cfg = Polyrun::Hooks.from_config(cfg)
|
|
62
|
+
if hook_cfg.empty? || Polyrun::Hooks.disabled?
|
|
63
|
+
exec(*cmd, *paths)
|
|
64
|
+
end
|
|
65
|
+
|
|
66
|
+
si = Polyrun::Config::Resolver.resolve_shard_index(pc)
|
|
67
|
+
st = Polyrun::Config::Resolver.resolve_shard_total(pc)
|
|
68
|
+
suite_started = false
|
|
69
|
+
exit_code = 1
|
|
70
|
+
# Distributed CI matrix (N > 1 global shards): each job is one shard; suite hooks are pipeline-wide.
|
|
71
|
+
# Run them once via +polyrun hook run before_suite+ / +after_suite+ (e.g. dedicated job), or set
|
|
72
|
+
# +POLYRUN_HOOKS_SUITE_PER_MATRIX_JOB=1+ to run suite hooks on every matrix job.
|
|
73
|
+
matrix_shards = st > 1
|
|
74
|
+
run_suite_hooks = !matrix_shards || Polyrun::Hooks.suite_per_matrix_job?
|
|
75
|
+
|
|
76
|
+
begin
|
|
77
|
+
env_orch = ENV.to_h.merge(
|
|
78
|
+
"POLYRUN_HOOK_ORCHESTRATOR" => "1",
|
|
79
|
+
"POLYRUN_SHARD_INDEX" => si.to_s,
|
|
80
|
+
"POLYRUN_SHARD_TOTAL" => st.to_s
|
|
81
|
+
)
|
|
82
|
+
if run_suite_hooks
|
|
83
|
+
code = hook_cfg.run_phase_if_enabled(:before_suite, env_orch)
|
|
84
|
+
return code if code != 0
|
|
85
|
+
|
|
86
|
+
suite_started = true
|
|
87
|
+
end
|
|
88
|
+
|
|
89
|
+
code = hook_cfg.run_phase_if_enabled(:before_shard, env_orch)
|
|
90
|
+
return code if code != 0
|
|
91
|
+
|
|
92
|
+
mx, mt = ci_shard_matrix_context(pc, 1)
|
|
93
|
+
child_env = shard_child_env(cfg: cfg, workers: 1, shard: 0, matrix_index: mx, matrix_total: mt)
|
|
94
|
+
child_env = child_env.merge("POLYRUN_HOOK_ORCHESTRATOR" => "0")
|
|
95
|
+
child_env = hook_cfg.merge_worker_ruby_env(child_env)
|
|
96
|
+
|
|
97
|
+
if hook_cfg.worker_hooks? && !Polyrun::Hooks.disabled?
|
|
98
|
+
system(child_env, "sh", "-c", hook_cfg.build_worker_shell_script(cmd, paths))
|
|
99
|
+
else
|
|
100
|
+
system(child_env, *cmd, *paths)
|
|
101
|
+
end
|
|
102
|
+
exit_code = $?.exitstatus
|
|
103
|
+
|
|
104
|
+
rc = hook_cfg.run_phase_if_enabled(:after_shard, env_orch.merge(
|
|
105
|
+
"POLYRUN_WORKER_EXIT_STATUS" => exit_code.to_s
|
|
106
|
+
))
|
|
107
|
+
exit_code = rc if rc != 0
|
|
108
|
+
|
|
109
|
+
exit_code
|
|
110
|
+
ensure
|
|
111
|
+
if suite_started
|
|
112
|
+
hook_cfg.run_phase_if_enabled(:after_suite, env_orch.merge(
|
|
113
|
+
"POLYRUN_SUITE_EXIT_STATUS" => exit_code.to_s
|
|
114
|
+
))
|
|
115
|
+
end
|
|
116
|
+
end
|
|
117
|
+
end
|
|
118
|
+
# rubocop:enable Metrics/AbcSize
|
|
119
|
+
end
|
|
120
|
+
end
|
|
121
|
+
end
|
|
@@ -1,5 +1,7 @@
|
|
|
1
1
|
require "shellwords"
|
|
2
2
|
|
|
3
|
+
require_relative "ci_shard_hooks"
|
|
4
|
+
|
|
3
5
|
module Polyrun
|
|
4
6
|
class CLI
|
|
5
7
|
# One CI matrix job = one global shard (POLYRUN_SHARD_INDEX / POLYRUN_SHARD_TOTAL), not +run-shards+
|
|
@@ -14,6 +16,8 @@ module Polyrun
|
|
|
14
16
|
# After +--+, prefer **multiple argv tokens** (+bundle+, +exec+, +rspec+, …). A single token that
|
|
15
17
|
# contains spaces is split with +Shellwords+ (not a full shell); exotic quoting differs from +sh -c+.
|
|
16
18
|
module CiShardRunCommand
|
|
19
|
+
include CiShardHooks
|
|
20
|
+
|
|
17
21
|
private
|
|
18
22
|
|
|
19
23
|
# @return [Array(Array<String>, Integer)] [paths, 0] on success, or [nil, exit_code] on failure
|
|
@@ -47,24 +51,6 @@ module Polyrun
|
|
|
47
51
|
[resolve_shard_index(pc), n]
|
|
48
52
|
end
|
|
49
53
|
|
|
50
|
-
def ci_shard_run_fanout!(ctx)
|
|
51
|
-
pids = run_shards_spawn_workers(ctx)
|
|
52
|
-
return 1 if pids.empty?
|
|
53
|
-
|
|
54
|
-
run_shards_warn_interleaved(ctx[:parallel], pids.size)
|
|
55
|
-
shard_results = run_shards_wait_all_children(pids)
|
|
56
|
-
failed = shard_results.reject { |r| r[:success] }.map { |r| r[:shard] }
|
|
57
|
-
|
|
58
|
-
if failed.any?
|
|
59
|
-
Polyrun::Log.warn "polyrun ci-shard: finished #{pids.size} worker(s) (some failed)"
|
|
60
|
-
run_shards_log_failed_reruns(failed, shard_results, ctx[:plan], ctx[:parallel], ctx[:workers], ctx[:cmd])
|
|
61
|
-
return 1
|
|
62
|
-
end
|
|
63
|
-
|
|
64
|
-
Polyrun::Log.warn "polyrun ci-shard: finished #{pids.size} worker(s) (exit 0)"
|
|
65
|
-
0
|
|
66
|
-
end
|
|
67
|
-
|
|
68
54
|
def ci_shard_fanout_context(cfg:, pc:, paths:, shard_processes:, cmd:, config_path:)
|
|
69
55
|
plan = ci_shard_local_plan!(paths, shard_processes)
|
|
70
56
|
mx, mt = ci_shard_matrix_context(pc, shard_processes)
|
|
@@ -113,8 +99,7 @@ module Polyrun
|
|
|
113
99
|
return code if code != 0
|
|
114
100
|
|
|
115
101
|
if shard_processes <= 1
|
|
116
|
-
|
|
117
|
-
return 0
|
|
102
|
+
return ci_shard_run_single!(cmd, paths, cfg, pc, config_path)
|
|
118
103
|
end
|
|
119
104
|
|
|
120
105
|
ctx = ci_shard_fanout_context(
|
|
@@ -145,8 +130,7 @@ module Polyrun
|
|
|
145
130
|
cmd = ["bundle", "exec", "rspec", *rspec_argv]
|
|
146
131
|
|
|
147
132
|
if shard_processes <= 1
|
|
148
|
-
|
|
149
|
-
return 0
|
|
133
|
+
return ci_shard_run_single!(cmd, paths, cfg, pc, config_path)
|
|
150
134
|
end
|
|
151
135
|
|
|
152
136
|
ctx = ci_shard_fanout_context(
|
data/lib/polyrun/cli/help.rb
CHANGED
|
@@ -38,6 +38,7 @@ module Polyrun
|
|
|
38
38
|
init write a starter polyrun.yml or POLYRUN.md from built-in templates (see docs/SETUP_PROFILE.md)
|
|
39
39
|
queue file-backed batch queue: init (optional --shard/--total etc. as plan, then claim/ack); M workers share one dir; no duplicate paths across claims
|
|
40
40
|
quick run Polyrun::Quick (describe/it, before/after, let, expect…to, assert_*; optional capybara!)
|
|
41
|
+
hook run <phase> run one shell hook from polyrun.yml hooks: (e.g. before_suite); optional --shard/--total
|
|
41
42
|
report-coverage write all coverage formats from one JSON file
|
|
42
43
|
report-junit RSpec JSON or Polyrun testcase JSON → JUnit XML (CI)
|
|
43
44
|
report-timing print slow-file summary from merged timing JSON
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
module Polyrun
|
|
2
|
+
class CLI
|
|
3
|
+
# +polyrun hook run <phase>+ — run one lifecycle phase from +polyrun.yml+ +hooks:+ (manual debugging / CI).
|
|
4
|
+
module HooksCommand
|
|
5
|
+
private
|
|
6
|
+
|
|
7
|
+
def cmd_hook(argv, config_path)
|
|
8
|
+
sub = argv.shift
|
|
9
|
+
case sub
|
|
10
|
+
when "run"
|
|
11
|
+
cmd_hook_run(argv, config_path)
|
|
12
|
+
when nil, "help", "-h", "--help"
|
|
13
|
+
print_hook_help
|
|
14
|
+
0
|
|
15
|
+
else
|
|
16
|
+
Polyrun::Log.warn "polyrun hook: unknown subcommand #{sub.inspect} (try: polyrun hook run <phase>)"
|
|
17
|
+
print_hook_help
|
|
18
|
+
2
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
def cmd_hook_run(argv, config_path)
|
|
23
|
+
phase = argv.shift
|
|
24
|
+
if phase.nil? || phase == "-h" || phase == "--help"
|
|
25
|
+
print_hook_help
|
|
26
|
+
return 2
|
|
27
|
+
end
|
|
28
|
+
|
|
29
|
+
shard, total = hook_run_parse_shard_flags!(argv)
|
|
30
|
+
|
|
31
|
+
unless argv.empty?
|
|
32
|
+
Polyrun::Log.warn "polyrun hook run: unexpected arguments: #{argv.inspect}"
|
|
33
|
+
return 2
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
phase_sym = Polyrun::Hooks.parse_phase(phase)
|
|
37
|
+
unless phase_sym && Polyrun::Hooks::PHASES.include?(phase_sym)
|
|
38
|
+
Polyrun::Log.warn "polyrun hook run: unknown phase #{phase.inspect} (expected: #{Polyrun::Hooks::PHASES.join(", ")})"
|
|
39
|
+
return 2
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
cfg = Polyrun::Config.load(path: config_path || ENV["POLYRUN_CONFIG"])
|
|
43
|
+
hook_cfg = Polyrun::Hooks.from_config(cfg)
|
|
44
|
+
env = hook_run_env(shard, total)
|
|
45
|
+
|
|
46
|
+
hook_cfg.run_phase(phase_sym, env)
|
|
47
|
+
rescue ArgumentError => e
|
|
48
|
+
Polyrun::Log.warn "polyrun hook run: #{e.message}"
|
|
49
|
+
2
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
def hook_run_parse_shard_flags!(argv)
|
|
53
|
+
shard = nil
|
|
54
|
+
total = nil
|
|
55
|
+
while (a = argv.first)
|
|
56
|
+
case a
|
|
57
|
+
when "--shard"
|
|
58
|
+
argv.shift
|
|
59
|
+
shard = Integer(argv.shift || (raise ArgumentError, "--shard needs a value"))
|
|
60
|
+
when "--total"
|
|
61
|
+
argv.shift
|
|
62
|
+
total = Integer(argv.shift || (raise ArgumentError, "--total needs a value"))
|
|
63
|
+
else
|
|
64
|
+
break
|
|
65
|
+
end
|
|
66
|
+
end
|
|
67
|
+
[shard, total]
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
def hook_run_env(shard, total)
|
|
71
|
+
env = ENV.to_h.merge(
|
|
72
|
+
"POLYRUN_HOOK_ORCHESTRATOR" => "1",
|
|
73
|
+
"POLYRUN_HOOK_CLI" => "1"
|
|
74
|
+
)
|
|
75
|
+
env["POLYRUN_SHARD_INDEX"] = shard.to_s unless shard.nil?
|
|
76
|
+
env["POLYRUN_SHARD_TOTAL"] = total.to_s unless total.nil?
|
|
77
|
+
env
|
|
78
|
+
end
|
|
79
|
+
|
|
80
|
+
def print_hook_help
|
|
81
|
+
Polyrun::Log.puts <<~HELP
|
|
82
|
+
usage: polyrun hook run <phase> [--shard N] [--total M]
|
|
83
|
+
|
|
84
|
+
Runs hook(s) from polyrun.yml: Ruby DSL (+hooks.ruby+) then shell strings for <phase> (same names as RSpec lifecycle:
|
|
85
|
+
before_suite / after_suite as before(:suite) / after(:suite); before_shard / after_shard as
|
|
86
|
+
before(:all) / after(:all); before_worker / after_worker as before(:each) / after(:each)).
|
|
87
|
+
|
|
88
|
+
Phases: #{Polyrun::Hooks::PHASES.join(", ")}
|
|
89
|
+
|
|
90
|
+
Optional --shard / --total set POLYRUN_SHARD_INDEX / POLYRUN_SHARD_TOTAL for the hook process.
|
|
91
|
+
POLYRUN_HOOKS_DISABLE=1 skips hooks during run-shards / ci-shard only; polyrun hook run still executes.
|
|
92
|
+
For CI matrix (POLYRUN_SHARD_TOTAL > 1), ci-shard-run skips before_suite / after_suite unless POLYRUN_HOOKS_SUITE_PER_MATRIX_JOB=1; run those phases here or in a dedicated CI job.
|
|
93
|
+
HELP
|
|
94
|
+
end
|
|
95
|
+
end
|
|
96
|
+
end
|
|
97
|
+
end
|
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
module Polyrun
|
|
2
|
+
class CLI
|
|
3
|
+
# Spawns and waits on worker processes for +run-shards+ / +ci-shard-*+ fan-out.
|
|
4
|
+
module RunShardsParallelChildren
|
|
5
|
+
private
|
|
6
|
+
|
|
7
|
+
# @return [Array(Array, Integer, nil)] +[pids, spawn_error_code]+; +spawn_error_code+ is +nil+ when all spawns succeeded
|
|
8
|
+
# rubocop:disable Metrics/AbcSize -- shard loop: spawn + shard hooks + env
|
|
9
|
+
def run_shards_spawn_workers(ctx, hook_cfg)
|
|
10
|
+
workers = ctx[:workers]
|
|
11
|
+
cmd = ctx[:cmd]
|
|
12
|
+
cfg = ctx[:cfg]
|
|
13
|
+
plan = ctx[:plan]
|
|
14
|
+
parallel = ctx[:parallel]
|
|
15
|
+
mx = ctx[:matrix_shard_index]
|
|
16
|
+
mt = ctx[:matrix_shard_total]
|
|
17
|
+
|
|
18
|
+
pids = []
|
|
19
|
+
workers.times do |shard|
|
|
20
|
+
paths = plan.shard(shard)
|
|
21
|
+
if paths.empty?
|
|
22
|
+
Polyrun::Log.warn "polyrun run-shards: shard #{shard} skipped (no paths)" if @verbose || parallel
|
|
23
|
+
next
|
|
24
|
+
end
|
|
25
|
+
|
|
26
|
+
env_shard = ENV.to_h.merge(
|
|
27
|
+
"POLYRUN_HOOK_ORCHESTRATOR" => "1",
|
|
28
|
+
"POLYRUN_SHARD_INDEX" => shard.to_s,
|
|
29
|
+
"POLYRUN_SHARD_TOTAL" => workers.to_s
|
|
30
|
+
)
|
|
31
|
+
code = hook_cfg.run_phase_if_enabled(:before_shard, env_shard)
|
|
32
|
+
if code != 0
|
|
33
|
+
run_shards_terminate_children!(pids)
|
|
34
|
+
return [pids, code]
|
|
35
|
+
end
|
|
36
|
+
|
|
37
|
+
child_env = shard_child_env(cfg: cfg, workers: workers, shard: shard, matrix_index: mx, matrix_total: mt)
|
|
38
|
+
child_env = child_env.merge("POLYRUN_HOOK_ORCHESTRATOR" => "0")
|
|
39
|
+
child_env = hook_cfg.merge_worker_ruby_env(child_env)
|
|
40
|
+
|
|
41
|
+
Polyrun::Log.warn "polyrun run-shards: shard #{shard} → #{paths.size} file(s)" if @verbose
|
|
42
|
+
pid = run_shards_spawn_one_worker(child_env, cmd, paths, hook_cfg)
|
|
43
|
+
pids << {pid: pid, shard: shard}
|
|
44
|
+
Polyrun::Debug.log("[parent pid=#{$$}] run-shards: Process.spawn shard=#{shard} child_pid=#{pid} spec_files=#{paths.size}")
|
|
45
|
+
Polyrun::Log.warn "polyrun run-shards: started shard #{shard} pid=#{pid} (#{paths.size} file(s))" if parallel
|
|
46
|
+
end
|
|
47
|
+
[pids, nil]
|
|
48
|
+
end
|
|
49
|
+
# rubocop:enable Metrics/AbcSize
|
|
50
|
+
|
|
51
|
+
def run_shards_spawn_one_worker(child_env, cmd, paths, hook_cfg)
|
|
52
|
+
if hook_cfg.worker_hooks? && !Polyrun::Hooks.disabled?
|
|
53
|
+
Process.spawn(child_env, "sh", "-c", hook_cfg.build_worker_shell_script(cmd, paths))
|
|
54
|
+
else
|
|
55
|
+
Process.spawn(child_env, *cmd, *paths)
|
|
56
|
+
end
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
# @return [Array(Array, Integer)] +[shard_results, after_shard_hook_error_code]+ (0 when all +after_shard+ hooks passed)
|
|
60
|
+
def run_shards_wait_all_children(pids, hook_cfg, ctx)
|
|
61
|
+
workers = ctx[:workers]
|
|
62
|
+
shard_results = []
|
|
63
|
+
after_hook_err = 0
|
|
64
|
+
Polyrun::Debug.time("Process.wait (#{pids.size} worker process(es))") do
|
|
65
|
+
pids.each do |h|
|
|
66
|
+
Process.wait(h[:pid])
|
|
67
|
+
exitstatus = $?.exitstatus
|
|
68
|
+
ok = $?.success?
|
|
69
|
+
Polyrun::Debug.log("[parent pid=#{$$}] run-shards: Process.wait child_pid=#{h[:pid]} shard=#{h[:shard]} exit=#{exitstatus} success=#{ok}")
|
|
70
|
+
env_after = ENV.to_h.merge(
|
|
71
|
+
"POLYRUN_HOOK_ORCHESTRATOR" => "1",
|
|
72
|
+
"POLYRUN_SHARD_INDEX" => h[:shard].to_s,
|
|
73
|
+
"POLYRUN_SHARD_TOTAL" => workers.to_s,
|
|
74
|
+
"POLYRUN_WORKER_EXIT_STATUS" => exitstatus.to_s
|
|
75
|
+
)
|
|
76
|
+
rc = hook_cfg.run_phase_if_enabled(:after_shard, env_after)
|
|
77
|
+
after_hook_err = rc if rc != 0 && after_hook_err == 0
|
|
78
|
+
shard_results << {shard: h[:shard], exitstatus: exitstatus, success: ok}
|
|
79
|
+
end
|
|
80
|
+
rescue Interrupt
|
|
81
|
+
# Do not trap SIGINT: Process.wait raises Interrupt; a trap races and prints Interrupt + SystemExit traces.
|
|
82
|
+
run_shards_shutdown_on_signal!(pids, 130)
|
|
83
|
+
rescue SignalException => e
|
|
84
|
+
raise unless e.signm == "SIGTERM"
|
|
85
|
+
|
|
86
|
+
run_shards_shutdown_on_signal!(pids, 143)
|
|
87
|
+
end
|
|
88
|
+
[shard_results, after_hook_err]
|
|
89
|
+
end
|
|
90
|
+
end
|
|
91
|
+
end
|
|
92
|
+
end
|
|
@@ -2,12 +2,14 @@ require "shellwords"
|
|
|
2
2
|
require "rbconfig"
|
|
3
3
|
|
|
4
4
|
require_relative "run_shards_planning"
|
|
5
|
+
require_relative "run_shards_parallel_children"
|
|
5
6
|
|
|
6
7
|
module Polyrun
|
|
7
8
|
class CLI
|
|
8
9
|
# Partition + spawn workers for `polyrun run-shards` (keeps {RunShardsCommand} file small).
|
|
9
10
|
module RunShardsRun
|
|
10
11
|
include RunShardsPlanning
|
|
12
|
+
include RunShardsParallelChildren
|
|
11
13
|
|
|
12
14
|
private
|
|
13
15
|
|
|
@@ -18,59 +20,68 @@ module Polyrun
|
|
|
18
20
|
run_shards_workers_and_merge(ctx)
|
|
19
21
|
end
|
|
20
22
|
|
|
23
|
+
# rubocop:disable Metrics/AbcSize -- orchestration: hooks, merge, worker failures
|
|
21
24
|
def run_shards_workers_and_merge(ctx)
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
25
|
+
hook_cfg = Polyrun::Hooks.from_config(ctx[:cfg])
|
|
26
|
+
suite_started = false
|
|
27
|
+
exit_code = 1
|
|
28
|
+
|
|
29
|
+
begin
|
|
30
|
+
env_suite = ENV.to_h.merge(
|
|
31
|
+
"POLYRUN_HOOK_ORCHESTRATOR" => "1",
|
|
32
|
+
"POLYRUN_SHARD_TOTAL" => ctx[:workers].to_s
|
|
33
|
+
)
|
|
34
|
+
code = hook_cfg.run_phase_if_enabled(:before_suite, env_suite)
|
|
35
|
+
return code if code != 0
|
|
36
|
+
|
|
37
|
+
suite_started = true
|
|
38
|
+
|
|
39
|
+
pids, spawn_err = run_shards_spawn_workers(ctx, hook_cfg)
|
|
40
|
+
if spawn_err
|
|
41
|
+
exit_code = spawn_err
|
|
42
|
+
return spawn_err
|
|
43
|
+
end
|
|
44
|
+
if pids.empty?
|
|
45
|
+
exit_code = 1
|
|
46
|
+
return 1
|
|
47
|
+
end
|
|
34
48
|
|
|
35
|
-
|
|
36
|
-
Polyrun::Log.warn "polyrun run-shards: finished #{pids.size} worker(s)" + (failed.any? ? " (some failed)" : " (exit 0)")
|
|
37
|
-
end
|
|
49
|
+
run_shards_warn_interleaved(ctx[:parallel], pids.size)
|
|
38
50
|
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
return 1
|
|
42
|
-
end
|
|
51
|
+
shard_results, wait_hook_err = run_shards_wait_all_children(pids, hook_cfg, ctx)
|
|
52
|
+
failed = shard_results.reject { |r| r[:success] }.map { |r| r[:shard] }
|
|
43
53
|
|
|
44
|
-
|
|
45
|
-
|
|
54
|
+
Polyrun::Debug.log(format(
|
|
55
|
+
"run-shards: workers wall time since start: %.3fs",
|
|
56
|
+
Process.clock_gettime(Process::CLOCK_MONOTONIC) - ctx[:run_t0]
|
|
57
|
+
))
|
|
46
58
|
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
cmd = ctx[:cmd]
|
|
50
|
-
cfg = ctx[:cfg]
|
|
51
|
-
plan = ctx[:plan]
|
|
52
|
-
parallel = ctx[:parallel]
|
|
53
|
-
mx = ctx[:matrix_shard_index]
|
|
54
|
-
mt = ctx[:matrix_shard_total]
|
|
55
|
-
|
|
56
|
-
pids = []
|
|
57
|
-
workers.times do |shard|
|
|
58
|
-
paths = plan.shard(shard)
|
|
59
|
-
if paths.empty?
|
|
60
|
-
Polyrun::Log.warn "polyrun run-shards: shard #{shard} skipped (no paths)" if @verbose || parallel
|
|
61
|
-
next
|
|
59
|
+
if ctx[:parallel]
|
|
60
|
+
Polyrun::Log.warn "polyrun run-shards: finished #{pids.size} worker(s)" + (failed.any? ? " (some failed)" : " (exit 0)")
|
|
62
61
|
end
|
|
63
62
|
|
|
64
|
-
|
|
63
|
+
if failed.any?
|
|
64
|
+
run_shards_log_failed_reruns(failed, shard_results, ctx[:plan], ctx[:parallel], ctx[:workers], ctx[:cmd])
|
|
65
|
+
exit_code = 1
|
|
66
|
+
exit_code = 1 if wait_hook_err != 0
|
|
67
|
+
return exit_code
|
|
68
|
+
end
|
|
65
69
|
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
70
|
+
exit_code = run_shards_merge_or_hint_coverage(ctx)
|
|
71
|
+
exit_code = 1 if wait_hook_err != 0 && exit_code == 0
|
|
72
|
+
exit_code
|
|
73
|
+
ensure
|
|
74
|
+
if suite_started
|
|
75
|
+
env_after = ENV.to_h.merge(
|
|
76
|
+
"POLYRUN_HOOK_ORCHESTRATOR" => "1",
|
|
77
|
+
"POLYRUN_SHARD_TOTAL" => ctx[:workers].to_s,
|
|
78
|
+
"POLYRUN_SUITE_EXIT_STATUS" => exit_code.to_s
|
|
79
|
+
)
|
|
80
|
+
hook_cfg.run_phase_if_enabled(:after_suite, env_after)
|
|
81
|
+
end
|
|
71
82
|
end
|
|
72
|
-
pids
|
|
73
83
|
end
|
|
84
|
+
# rubocop:enable Metrics/AbcSize
|
|
74
85
|
|
|
75
86
|
def run_shards_warn_interleaved(parallel, pid_count)
|
|
76
87
|
return unless parallel && pid_count > 1
|
|
@@ -79,27 +90,6 @@ module Polyrun
|
|
|
79
90
|
Polyrun::Log.warn "polyrun run-shards: each worker prints its own summary line; the last \"N examples\" line is not a total across shards."
|
|
80
91
|
end
|
|
81
92
|
|
|
82
|
-
def run_shards_wait_all_children(pids)
|
|
83
|
-
shard_results = []
|
|
84
|
-
Polyrun::Debug.time("Process.wait (#{pids.size} worker process(es))") do
|
|
85
|
-
pids.each do |h|
|
|
86
|
-
Process.wait(h[:pid])
|
|
87
|
-
exitstatus = $?.exitstatus
|
|
88
|
-
ok = $?.success?
|
|
89
|
-
Polyrun::Debug.log("[parent pid=#{$$}] run-shards: Process.wait child_pid=#{h[:pid]} shard=#{h[:shard]} exit=#{exitstatus} success=#{ok}")
|
|
90
|
-
shard_results << {shard: h[:shard], exitstatus: exitstatus, success: ok}
|
|
91
|
-
end
|
|
92
|
-
rescue Interrupt
|
|
93
|
-
# Do not trap SIGINT: Process.wait raises Interrupt; a trap races and prints Interrupt + SystemExit traces.
|
|
94
|
-
run_shards_shutdown_on_signal!(pids, 130)
|
|
95
|
-
rescue SignalException => e
|
|
96
|
-
raise unless e.signm == "SIGTERM"
|
|
97
|
-
|
|
98
|
-
run_shards_shutdown_on_signal!(pids, 143)
|
|
99
|
-
end
|
|
100
|
-
shard_results
|
|
101
|
-
end
|
|
102
|
-
|
|
103
93
|
# Best-effort worker teardown then exit. Does not return.
|
|
104
94
|
def run_shards_shutdown_on_signal!(pids, code)
|
|
105
95
|
run_shards_terminate_children!(pids)
|
data/lib/polyrun/cli.rb
CHANGED
|
@@ -16,6 +16,7 @@ require_relative "cli/ci_shard_run_parse"
|
|
|
16
16
|
require_relative "cli/ci_shard_run_command"
|
|
17
17
|
require_relative "cli/config_command"
|
|
18
18
|
require_relative "cli/default_run"
|
|
19
|
+
require_relative "cli/hooks_command"
|
|
19
20
|
require_relative "cli/help"
|
|
20
21
|
|
|
21
22
|
module Polyrun
|
|
@@ -29,7 +30,7 @@ module Polyrun
|
|
|
29
30
|
DISPATCH_SUBCOMMAND_NAMES = %w[
|
|
30
31
|
plan prepare merge-coverage report-coverage report-junit report-timing
|
|
31
32
|
env config merge-timing db:setup-template db:setup-shard db:clone-shards
|
|
32
|
-
run-shards parallel-rspec start build-paths init queue quick
|
|
33
|
+
run-shards parallel-rspec start build-paths init queue quick hook
|
|
33
34
|
].freeze
|
|
34
35
|
|
|
35
36
|
# First argv token that is a normal subcommand (not a path); if argv[0] is not here but looks like paths, run implicit parallel.
|
|
@@ -53,6 +54,7 @@ module Polyrun
|
|
|
53
54
|
include CiShardRunCommand
|
|
54
55
|
include ConfigCommand
|
|
55
56
|
include DefaultRun
|
|
57
|
+
include HooksCommand
|
|
56
58
|
include Help
|
|
57
59
|
|
|
58
60
|
def self.run(argv = ARGV)
|
|
@@ -177,6 +179,8 @@ module Polyrun
|
|
|
177
179
|
cmd_queue(argv)
|
|
178
180
|
when "quick"
|
|
179
181
|
cmd_quick(argv)
|
|
182
|
+
when "hook"
|
|
183
|
+
cmd_hook(argv, config_path)
|
|
180
184
|
else
|
|
181
185
|
Polyrun::Log.warn "unknown command: #{command}"
|
|
182
186
|
2
|
data/lib/polyrun/config.rb
CHANGED
|
@@ -0,0 +1,128 @@
|
|
|
1
|
+
require "monitor"
|
|
2
|
+
|
|
3
|
+
require_relative "../log"
|
|
4
|
+
|
|
5
|
+
module Polyrun
|
|
6
|
+
class Hooks
|
|
7
|
+
# Ruby DSL for +hooks.ruby+ / +hooks.ruby_file+ in +polyrun.yml+ (see README).
|
|
8
|
+
#
|
|
9
|
+
# Example file (+config/polyrun_hooks.rb+):
|
|
10
|
+
#
|
|
11
|
+
# before(:suite) { |env| puts env["POLYRUN_HOOK_PHASE"] }
|
|
12
|
+
# after(:each) { |env| }
|
|
13
|
+
#
|
|
14
|
+
# Blocks receive a hash with string keys (same env as shell hooks).
|
|
15
|
+
module Dsl
|
|
16
|
+
class Registry
|
|
17
|
+
def initialize
|
|
18
|
+
@phases = Hash.new { |h, k| h[k] = [] }
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
def add(phase, proc)
|
|
22
|
+
@phases[phase.to_sym] << proc
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
# @param env [Hash] string-keyed env
|
|
26
|
+
def run(phase, env)
|
|
27
|
+
@phases[phase.to_sym].each { |pr| pr.call(env) }
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
def any?(phase)
|
|
31
|
+
@phases[phase.to_sym].any?
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
def empty?
|
|
35
|
+
@phases.values.all?(&:empty?)
|
|
36
|
+
end
|
|
37
|
+
|
|
38
|
+
def worker_hooks?
|
|
39
|
+
any?(:before_worker) || any?(:after_worker)
|
|
40
|
+
end
|
|
41
|
+
end
|
|
42
|
+
|
|
43
|
+
# Evaluates a hook file with +before+ / +after+ (+before(:suite)+, etc.).
|
|
44
|
+
class FileContext
|
|
45
|
+
def initialize(path)
|
|
46
|
+
@path = path
|
|
47
|
+
@registry = Registry.new
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
attr_reader :registry
|
|
51
|
+
|
|
52
|
+
def load_file
|
|
53
|
+
instance_eval(File.read(@path), @path)
|
|
54
|
+
@registry
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
def before(sym, &block)
|
|
58
|
+
@registry.add(map_before(sym), block)
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
def after(sym, &block)
|
|
62
|
+
@registry.add(map_after(sym), block)
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
private
|
|
66
|
+
|
|
67
|
+
def map_before(sym)
|
|
68
|
+
case sym.to_sym
|
|
69
|
+
when :suite then :before_suite
|
|
70
|
+
when :all then :before_shard
|
|
71
|
+
when :each then :before_worker
|
|
72
|
+
else
|
|
73
|
+
raise ArgumentError, "hooks DSL: before(#{sym.inspect}) — use :suite, :all, or :each"
|
|
74
|
+
end
|
|
75
|
+
end
|
|
76
|
+
|
|
77
|
+
def map_after(sym)
|
|
78
|
+
case sym.to_sym
|
|
79
|
+
when :suite then :after_suite
|
|
80
|
+
when :all then :after_shard
|
|
81
|
+
when :each then :after_worker
|
|
82
|
+
else
|
|
83
|
+
raise ArgumentError, "hooks DSL: after(#{sym.inspect}) — use :suite, :all, or :each"
|
|
84
|
+
end
|
|
85
|
+
end
|
|
86
|
+
end
|
|
87
|
+
|
|
88
|
+
class << self
|
|
89
|
+
# @return [Registry, nil]
|
|
90
|
+
def load_registry(path)
|
|
91
|
+
return nil if path.nil? || path.to_s.strip.empty?
|
|
92
|
+
|
|
93
|
+
full = File.expand_path(path.to_s, Dir.pwd)
|
|
94
|
+
return nil unless File.file?(full)
|
|
95
|
+
|
|
96
|
+
mtime = File.mtime(full)
|
|
97
|
+
cache_mu.synchronize do
|
|
98
|
+
hit = registry_cache[full]
|
|
99
|
+
return hit[:registry] if hit && hit[:mtime] == mtime
|
|
100
|
+
|
|
101
|
+
reg = FileContext.new(full).load_file
|
|
102
|
+
registry_cache[full] = {mtime: mtime, registry: reg}
|
|
103
|
+
reg
|
|
104
|
+
end
|
|
105
|
+
rescue => e
|
|
106
|
+
Polyrun::Log.warn "polyrun hooks: failed to load #{full}: #{e.class}: #{e.message}"
|
|
107
|
+
nil
|
|
108
|
+
end
|
|
109
|
+
|
|
110
|
+
def clear_cache!
|
|
111
|
+
cache_mu.synchronize { registry_cache.clear }
|
|
112
|
+
end
|
|
113
|
+
|
|
114
|
+
private
|
|
115
|
+
|
|
116
|
+
# rubocop:disable ThreadSafety/ClassInstanceVariable -- single-threaded hook load; Monitor protects cache
|
|
117
|
+
def cache_mu
|
|
118
|
+
@cache_mu ||= Monitor.new
|
|
119
|
+
end
|
|
120
|
+
|
|
121
|
+
def registry_cache
|
|
122
|
+
@registry_cache ||= {}
|
|
123
|
+
end
|
|
124
|
+
# rubocop:enable ThreadSafety/ClassInstanceVariable
|
|
125
|
+
end
|
|
126
|
+
end
|
|
127
|
+
end
|
|
128
|
+
end
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
require_relative "../log"
|
|
2
|
+
|
|
3
|
+
module Polyrun
|
|
4
|
+
class Hooks
|
|
5
|
+
# Invoked by worker shell wrapper (+ruby -e+). Requires +POLYRUN_HOOKS_RUBY_FILE+.
|
|
6
|
+
module WorkerRunner
|
|
7
|
+
module_function
|
|
8
|
+
|
|
9
|
+
# @return [Integer] exit code (0 on success)
|
|
10
|
+
def run!(phase)
|
|
11
|
+
phase = phase.to_sym
|
|
12
|
+
path = ENV["POLYRUN_HOOKS_RUBY_FILE"]
|
|
13
|
+
return 0 if path.nil? || path.empty?
|
|
14
|
+
|
|
15
|
+
registry = Dsl.load_registry(path)
|
|
16
|
+
return 0 if registry.nil? || !registry.any?(phase)
|
|
17
|
+
|
|
18
|
+
env = ENV.to_h
|
|
19
|
+
registry.run(phase, env)
|
|
20
|
+
0
|
|
21
|
+
rescue => e
|
|
22
|
+
Polyrun::Log.warn "polyrun hooks worker #{phase}: #{e.class}: #{e.message}"
|
|
23
|
+
1
|
|
24
|
+
end
|
|
25
|
+
end
|
|
26
|
+
end
|
|
27
|
+
end
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
require "shellwords"
|
|
2
|
+
require "rbconfig"
|
|
3
|
+
|
|
4
|
+
module Polyrun
|
|
5
|
+
class Hooks
|
|
6
|
+
# Builds +sh -c+ script for worker processes (shell + Ruby +before_worker+ / +after_worker+).
|
|
7
|
+
module WorkerShell
|
|
8
|
+
# @param cmd [Array<String>] argv before paths
|
|
9
|
+
# @param paths [Array<String>]
|
|
10
|
+
# @return [String] shell script body for +sh -c+ (worker process)
|
|
11
|
+
# rubocop:disable Metrics/AbcSize -- shell + ruby worker hook branches
|
|
12
|
+
def build_worker_shell_script(cmd, paths)
|
|
13
|
+
main = Shellwords.join(cmd + paths)
|
|
14
|
+
rb = RbConfig.ruby
|
|
15
|
+
bw_shell = commands_for(:before_worker)
|
|
16
|
+
aw_shell = commands_for(:after_worker)
|
|
17
|
+
bw_ruby = ruby_registry&.any?(:before_worker)
|
|
18
|
+
aw_ruby = ruby_registry&.any?(:after_worker)
|
|
19
|
+
|
|
20
|
+
lines = []
|
|
21
|
+
lines << "export POLYRUN_HOOK_PHASE=before_worker"
|
|
22
|
+
if bw_ruby || bw_shell.any?
|
|
23
|
+
lines << "set -e"
|
|
24
|
+
lines << worker_ruby_line(rb, :before_worker) if bw_ruby
|
|
25
|
+
bw_shell.each { |c| lines << c }
|
|
26
|
+
end
|
|
27
|
+
lines << "set +e"
|
|
28
|
+
lines << main
|
|
29
|
+
lines << "ec=$?"
|
|
30
|
+
lines << "export POLYRUN_HOOK_PHASE=after_worker"
|
|
31
|
+
if aw_ruby || aw_shell.any?
|
|
32
|
+
lines << "set +e"
|
|
33
|
+
aw_shell.each { |c| lines << "( #{c} ) || true" }
|
|
34
|
+
lines << worker_ruby_line(rb, :after_worker, wrap_allow_fail: true) if aw_ruby
|
|
35
|
+
end
|
|
36
|
+
lines << "exit $ec"
|
|
37
|
+
lines.join("\n")
|
|
38
|
+
end
|
|
39
|
+
# rubocop:enable Metrics/AbcSize
|
|
40
|
+
|
|
41
|
+
private
|
|
42
|
+
|
|
43
|
+
def worker_ruby_line(rb_exe, phase, wrap_allow_fail: false)
|
|
44
|
+
code = %(require "polyrun"; Polyrun::Hooks::WorkerRunner.run!(:#{phase}))
|
|
45
|
+
line = "#{rb_exe} -e #{Shellwords.escape(code)}"
|
|
46
|
+
wrap_allow_fail ? "( #{line} ) || true" : line
|
|
47
|
+
end
|
|
48
|
+
end
|
|
49
|
+
end
|
|
50
|
+
end
|
|
@@ -0,0 +1,185 @@
|
|
|
1
|
+
require "shellwords"
|
|
2
|
+
|
|
3
|
+
require_relative "log"
|
|
4
|
+
require_relative "hooks/dsl"
|
|
5
|
+
require_relative "hooks/worker_runner"
|
|
6
|
+
require_relative "hooks/worker_shell"
|
|
7
|
+
|
|
8
|
+
module Polyrun
|
|
9
|
+
# Shell and Ruby DSL hooks around parallel orchestration, named like RSpec lifecycle callbacks:
|
|
10
|
+
# +before_suite+ / +after_suite+ (+before(:suite)+ / +after(:suite)+),
|
|
11
|
+
# +before_shard+ / +after_shard+ (parent process, per partition index),
|
|
12
|
+
# +before_worker+ / +after_worker+ (inside each worker process, around the test command).
|
|
13
|
+
#
|
|
14
|
+
# Configure under +hooks:+ in +polyrun.yml+: shell strings, and/or +ruby:+ path to a Ruby DSL file.
|
|
15
|
+
# Run manually: +polyrun hook run <phase>+ (see {CLI}).
|
|
16
|
+
#
|
|
17
|
+
# Orchestration respects +POLYRUN_HOOKS_DISABLE=1+ (+run_phase_if_enabled+); +polyrun hook run+ always runs {#run_phase}.
|
|
18
|
+
class Hooks
|
|
19
|
+
include WorkerShell
|
|
20
|
+
|
|
21
|
+
PHASES = %i[
|
|
22
|
+
before_suite after_suite
|
|
23
|
+
before_shard after_shard
|
|
24
|
+
before_worker after_worker
|
|
25
|
+
].freeze
|
|
26
|
+
|
|
27
|
+
attr_reader :ruby_file
|
|
28
|
+
|
|
29
|
+
def self.disabled?
|
|
30
|
+
v = ENV["POLYRUN_HOOKS_DISABLE"].to_s.downcase
|
|
31
|
+
%w[1 true yes].include?(v)
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
# When +POLYRUN_SHARD_TOTAL+ is greater than 1 (+ci-shard-run+ matrix), suite hooks are skipped by default; set
|
|
35
|
+
# +POLYRUN_HOOKS_SUITE_PER_MATRIX_JOB=1+ to run +before_suite+ / +after_suite+ on every matrix job (legacy).
|
|
36
|
+
def self.suite_per_matrix_job?
|
|
37
|
+
v = ENV["POLYRUN_HOOKS_SUITE_PER_MATRIX_JOB"].to_s.downcase
|
|
38
|
+
%w[1 true yes].include?(v)
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
def self.from_config(cfg)
|
|
42
|
+
raw = cfg.respond_to?(:hooks) ? cfg.hooks : {}
|
|
43
|
+
new(raw.is_a?(Hash) ? raw : {})
|
|
44
|
+
end
|
|
45
|
+
|
|
46
|
+
# Maps CLI or YAML-style names (+before_suite+, +"before(:suite)"+) to a phase symbol or +nil+.
|
|
47
|
+
def self.parse_phase(str)
|
|
48
|
+
new({}).send(:canonical_key, str)
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
# @param raw [Hash] +hooks+ block from YAML
|
|
52
|
+
def initialize(raw)
|
|
53
|
+
@ruby_file = extract_ruby_file(raw)
|
|
54
|
+
h = {}
|
|
55
|
+
raw.each do |k, v|
|
|
56
|
+
next if %w[ruby ruby_file].include?(k.to_s)
|
|
57
|
+
|
|
58
|
+
ck = canonical_key(k)
|
|
59
|
+
next if ck.nil?
|
|
60
|
+
|
|
61
|
+
h[ck] = v
|
|
62
|
+
end
|
|
63
|
+
@raw = h.freeze
|
|
64
|
+
@ruby_registry_loaded = false
|
|
65
|
+
@ruby_registry = nil
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
def empty?
|
|
69
|
+
no_shell = PHASES.all? { |p| commands_for(p).empty? }
|
|
70
|
+
return false unless no_shell
|
|
71
|
+
|
|
72
|
+
return true if @ruby_file.nil? || @ruby_file.to_s.strip.empty?
|
|
73
|
+
return true unless File.file?(File.expand_path(@ruby_file, Dir.pwd))
|
|
74
|
+
|
|
75
|
+
reg = ruby_registry
|
|
76
|
+
reg.nil? || reg.empty?
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
def worker_hooks?
|
|
80
|
+
return true if commands_for(:before_worker).any? || commands_for(:after_worker).any?
|
|
81
|
+
|
|
82
|
+
!!ruby_registry&.worker_hooks?
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
# Merges +POLYRUN_HOOKS_RUBY_FILE+ when a DSL file is configured (for worker +ruby -e+).
|
|
86
|
+
def merge_worker_ruby_env(env)
|
|
87
|
+
return env unless @ruby_file
|
|
88
|
+
abs = File.expand_path(@ruby_file, Dir.pwd)
|
|
89
|
+
return env unless File.file?(abs)
|
|
90
|
+
|
|
91
|
+
env.merge("POLYRUN_HOOKS_RUBY_FILE" => abs)
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
# @param phase [Symbol]
|
|
95
|
+
# @return [Array<String>]
|
|
96
|
+
def commands_for(phase)
|
|
97
|
+
v = @raw[phase.to_sym]
|
|
98
|
+
case v
|
|
99
|
+
when nil then []
|
|
100
|
+
when Array then v.map(&:to_s).map(&:strip).reject(&:empty?)
|
|
101
|
+
else
|
|
102
|
+
s = v.to_s.strip
|
|
103
|
+
s.empty? ? [] : [s]
|
|
104
|
+
end
|
|
105
|
+
end
|
|
106
|
+
|
|
107
|
+
# Runs Ruby DSL blocks (if any), then shell commands for +phase+.
|
|
108
|
+
# @return [Integer] exit code (0 if no commands)
|
|
109
|
+
def run_phase(phase, env)
|
|
110
|
+
return 0 unless PHASES.include?(phase.to_sym)
|
|
111
|
+
|
|
112
|
+
merged = stringify_env_for_hook(env).merge(
|
|
113
|
+
"POLYRUN_HOOK_PHASE" => phase.to_s,
|
|
114
|
+
"POLYRUN_HOOK" => "1"
|
|
115
|
+
)
|
|
116
|
+
|
|
117
|
+
reg = ruby_registry
|
|
118
|
+
if reg&.any?(phase)
|
|
119
|
+
begin
|
|
120
|
+
reg.run(phase, merged)
|
|
121
|
+
rescue => e
|
|
122
|
+
Polyrun::Log.warn "polyrun hooks: #{phase} ruby hook failed: #{e.class}: #{e.message}"
|
|
123
|
+
return 1
|
|
124
|
+
end
|
|
125
|
+
end
|
|
126
|
+
|
|
127
|
+
commands_for(phase).each do |cmd|
|
|
128
|
+
ok = system(merged, "sh", "-c", cmd)
|
|
129
|
+
return $?.exitstatus unless ok
|
|
130
|
+
end
|
|
131
|
+
0
|
|
132
|
+
end
|
|
133
|
+
|
|
134
|
+
# Like {#run_phase}, but no-ops when {disabled?} (+POLYRUN_HOOKS_DISABLE=1+). Used by run-shards / ci-shard orchestration.
|
|
135
|
+
def run_phase_if_enabled(phase, env)
|
|
136
|
+
return 0 if self.class.disabled?
|
|
137
|
+
|
|
138
|
+
run_phase(phase, env)
|
|
139
|
+
end
|
|
140
|
+
|
|
141
|
+
private
|
|
142
|
+
|
|
143
|
+
def extract_ruby_file(raw)
|
|
144
|
+
v = raw["ruby"] || raw[:ruby] || raw["ruby_file"] || raw[:ruby_file]
|
|
145
|
+
return nil if v.nil?
|
|
146
|
+
|
|
147
|
+
s = v.to_s.strip
|
|
148
|
+
s.empty? ? nil : s
|
|
149
|
+
end
|
|
150
|
+
|
|
151
|
+
def ruby_registry
|
|
152
|
+
return @ruby_registry if @ruby_registry_loaded
|
|
153
|
+
|
|
154
|
+
@ruby_registry_loaded = true
|
|
155
|
+
@ruby_registry = Dsl.load_registry(@ruby_file)
|
|
156
|
+
end
|
|
157
|
+
|
|
158
|
+
def stringify_env_for_hook(env)
|
|
159
|
+
h = {}
|
|
160
|
+
env.each { |k, v| h[k.to_s] = v }
|
|
161
|
+
h
|
|
162
|
+
end
|
|
163
|
+
|
|
164
|
+
# Accept RSpec-style quoted keys from YAML, e.g. +"before(:suite)"+.
|
|
165
|
+
def canonical_key(k)
|
|
166
|
+
s = k.to_s.strip
|
|
167
|
+
sym = if s.match?(/\Abefore\(\s*:suite\s*\)\z/i)
|
|
168
|
+
:before_suite
|
|
169
|
+
elsif s.match?(/\Aafter\(\s*:suite\s*\)\z/i)
|
|
170
|
+
:after_suite
|
|
171
|
+
elsif s.match?(/\Abefore\(\s*:all\s*\)\z/i)
|
|
172
|
+
:before_shard
|
|
173
|
+
elsif s.match?(/\Aafter\(\s*:all\s*\)\z/i)
|
|
174
|
+
:after_shard
|
|
175
|
+
elsif s.match?(/\Abefore\(\s*:each\s*\)\z/i)
|
|
176
|
+
:before_worker
|
|
177
|
+
elsif s.match?(/\Aafter\(\s*:each\s*\)\z/i)
|
|
178
|
+
:after_worker
|
|
179
|
+
else
|
|
180
|
+
s.downcase.tr("-", "_").to_sym
|
|
181
|
+
end
|
|
182
|
+
PHASES.include?(sym) ? sym : nil
|
|
183
|
+
end
|
|
184
|
+
end
|
|
185
|
+
end
|
data/lib/polyrun/version.rb
CHANGED
data/lib/polyrun.rb
CHANGED
|
@@ -25,6 +25,7 @@ require_relative "polyrun/database/shard"
|
|
|
25
25
|
require_relative "polyrun/database/url_builder"
|
|
26
26
|
require_relative "polyrun/database/provision"
|
|
27
27
|
require_relative "polyrun/database/clone_shards"
|
|
28
|
+
require_relative "polyrun/hooks"
|
|
28
29
|
require_relative "polyrun/env/ci"
|
|
29
30
|
require_relative "polyrun/timing/merge"
|
|
30
31
|
require_relative "polyrun/timing/summary"
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: polyrun
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.4.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Andrei Makarov
|
|
@@ -166,6 +166,7 @@ files:
|
|
|
166
166
|
- docs/SETUP_PROFILE.md
|
|
167
167
|
- lib/polyrun.rb
|
|
168
168
|
- lib/polyrun/cli.rb
|
|
169
|
+
- lib/polyrun/cli/ci_shard_hooks.rb
|
|
169
170
|
- lib/polyrun/cli/ci_shard_run_command.rb
|
|
170
171
|
- lib/polyrun/cli/ci_shard_run_parse.rb
|
|
171
172
|
- lib/polyrun/cli/config_command.rb
|
|
@@ -176,6 +177,7 @@ files:
|
|
|
176
177
|
- lib/polyrun/cli/env_commands.rb
|
|
177
178
|
- lib/polyrun/cli/help.rb
|
|
178
179
|
- lib/polyrun/cli/helpers.rb
|
|
180
|
+
- lib/polyrun/cli/hooks_command.rb
|
|
179
181
|
- lib/polyrun/cli/init_command.rb
|
|
180
182
|
- lib/polyrun/cli/plan_command.rb
|
|
181
183
|
- lib/polyrun/cli/prepare_command.rb
|
|
@@ -184,6 +186,7 @@ files:
|
|
|
184
186
|
- lib/polyrun/cli/quick_command.rb
|
|
185
187
|
- lib/polyrun/cli/report_commands.rb
|
|
186
188
|
- lib/polyrun/cli/run_shards_command.rb
|
|
189
|
+
- lib/polyrun/cli/run_shards_parallel_children.rb
|
|
187
190
|
- lib/polyrun/cli/run_shards_plan_boot_phases.rb
|
|
188
191
|
- lib/polyrun/cli/run_shards_plan_options.rb
|
|
189
192
|
- lib/polyrun/cli/run_shards_planning.rb
|
|
@@ -225,6 +228,10 @@ files:
|
|
|
225
228
|
- lib/polyrun/database/url_builder/template_prepare.rb
|
|
226
229
|
- lib/polyrun/debug.rb
|
|
227
230
|
- lib/polyrun/env/ci.rb
|
|
231
|
+
- lib/polyrun/hooks.rb
|
|
232
|
+
- lib/polyrun/hooks/dsl.rb
|
|
233
|
+
- lib/polyrun/hooks/worker_runner.rb
|
|
234
|
+
- lib/polyrun/hooks/worker_shell.rb
|
|
228
235
|
- lib/polyrun/log.rb
|
|
229
236
|
- lib/polyrun/minitest.rb
|
|
230
237
|
- lib/polyrun/partition/constraints.rb
|