polyrun 1.1.0 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +12 -0
- data/README.md +1 -1
- data/lib/polyrun/cli/config_command.rb +42 -0
- data/lib/polyrun/cli/default_run.rb +115 -0
- data/lib/polyrun/cli/help.rb +54 -0
- data/lib/polyrun/cli/helpers.rb +4 -31
- data/lib/polyrun/cli/prepare_command.rb +2 -2
- data/lib/polyrun/cli/run_shards_command.rb +49 -3
- data/lib/polyrun/cli/run_shards_plan_boot_phases.rb +1 -1
- data/lib/polyrun/cli/run_shards_plan_options.rb +1 -1
- data/lib/polyrun/cli/run_shards_planning.rb +8 -8
- data/lib/polyrun/cli/start_bootstrap.rb +2 -6
- data/lib/polyrun/cli.rb +44 -50
- data/lib/polyrun/config/dotted_path.rb +21 -0
- data/lib/polyrun/config/effective.rb +71 -0
- data/lib/polyrun/config/resolver.rb +70 -0
- data/lib/polyrun/config.rb +7 -0
- data/lib/polyrun/partition/paths.rb +83 -2
- data/lib/polyrun/quick/runner.rb +26 -17
- data/lib/polyrun/version.rb +1 -1
- metadata +7 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 5dd8b178e07e0cf6648284d59d5b8f58a5feceeb2c7570edfd381aafaed207cb
|
|
4
|
+
data.tar.gz: cd2846dc77b56bccf0ac411fcc4f6a2a7aaf2e106609172e84299c2a1d253f64
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: c9a71f317d28ce3dcdf25d9c8e08221e71eadb2cf044217170ca0ba08cdc22cb702e6adaf02383d4e6089fa2fdddc94198cc420f031f53186715b67c14c92567
|
|
7
|
+
data.tar.gz: 81469ea975e78befbf6c66e3482e5c6006c11bb8b1806a079545a00ac057f250d49323aeab6685e8eaec22ff65e5aa72fb28aaca688365048483650d3c78de00
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,17 @@
|
|
|
1
1
|
# CHANGELOG
|
|
2
2
|
|
|
3
|
+
## 1.2.0 (2026-04-15)
|
|
4
|
+
|
|
5
|
+
- Add `polyrun config <dotted.path>` to print values from `Polyrun::Config::Effective` (same effective tree as runtime: arbitrary YAML paths, merged `prepare.env.<KEY>` as for `polyrun prepare`, resolved `partition.shard_index`, `partition.shard_total`, `partition.timing_granularity`, and `workers`).
|
|
6
|
+
- Memoize `Polyrun::Config::Effective.build` per thread (keyed by config path, object id, and env fingerprint) so repeated `dig` calls do not rebuild the merged tree.
|
|
7
|
+
- Add `DISPATCH_SUBCOMMAND_NAMES` and `IMPLICIT_PATH_EXCLUSION_TOKENS`; route implicit path-only argv against one list (includes `ci-shard-*`, `help`, `version`); add spec that dispatch names match `when` branches in `lib/polyrun/cli.rb`.
|
|
8
|
+
- Run `polyrun` with no subcommand to fan out parallel tests: pick RSpec (`start`), Minitest (`bundle exec rails test` or `bundle exec ruby -I test`), or Polyrun Quick (`bundle exec polyrun quick`) from `spec/**/*_spec.rb` vs `test/**/*_test.rb` vs Quick globs.
|
|
9
|
+
- Accept path-only argv (and optional `run-shards` options before paths, e.g. `--workers`) to shard those files without naming a subcommand; infer suite from `_spec.rb` / `_test.rb` vs other `.rb` files.
|
|
10
|
+
- Add optional `partition.suite` (`auto`, `rspec`, `minitest`, `quick`) when resolving globbed paths for `run-shards` / `parallel-rspec` / default runs.
|
|
11
|
+
- Document implicit argv (known subcommand first vs path-like implicit parallel) and parallel Quick `bundle exec` from app root in `polyrun help` and `examples/README.md`.
|
|
12
|
+
- Comment `detect_auto_suite` glob order in `lib/polyrun/partition/paths.rb` (RSpec/Minitest globs before Quick discovery).
|
|
13
|
+
- Remove redundant `OptionParser` from `polyrun config` (no options; banner only).
|
|
14
|
+
|
|
3
15
|
## 1.1.0 (2026-04-15)
|
|
4
16
|
|
|
5
17
|
- Add `ci-shard-run` / `ci-shard-rspec` for matrix-style sharding (one job per `POLYRUN_SHARD_INDEX` / `POLYRUN_SHARD_TOTAL`): resolve paths via the same plan as `polyrun plan`, then `exec` the given command with this shard’s paths (unlike `run-shards`, which fans out multiple workers on one host).
|
data/README.md
CHANGED
|
@@ -19,7 +19,7 @@ Capybara and Playwright stay in your application; Polyrun does not replace brows
|
|
|
19
19
|
|
|
20
20
|
## How?
|
|
21
21
|
|
|
22
|
-
1. Add the gem (path or RubyGems) and `require "polyrun"` where you integrate—for example coverage merge in CI or prepare hooks.
|
|
22
|
+
1. Add the gem (path or RubyGems) and `require "polyrun"` where you integrate—for example coverage merge in CI or prepare hooks. To pin the executable in your app, run `bundle binstubs polyrun` (writes `bin/polyrun`; ensure `bin/` is on `PATH` or invoke `./bin/polyrun`).
|
|
23
23
|
2. Add a `polyrun.yml` beside the app, or pass `-c` to point at one. Configure `partition` (paths, shard index and total, strategy), and optionally `databases` (Postgres template and `shard_db_pattern`), `prepare`, and `coverage`. If you use `partition.paths_build`, Polyrun can write `partition.paths_file` (for example `spec/spec_paths.txt`) from globs and ordered stages—substring priorities for integration specs, or a regex stage for “Rails-heavy files first”—without a per-project Ruby script. That step runs before `plan` and `run-shards`. Use `bin/polyrun build-paths` to refresh the paths file only.
|
|
24
24
|
3. Run prepare once before fan-out—for example `script/ci_prepare` for Vite or webpack builds, and `Polyrun::Prepare::Assets` digest markers. See `examples/TESTING_REQUIREMENTS.md`.
|
|
25
25
|
4. Run workers with `bin/polyrun run-shards --workers N -- bundle exec rspec`: N separate OS processes, each running RSpec with its own file list from `partition.paths_file`, or `spec/spec_paths.txt`, or else `spec/**/*_spec.rb`. Stderr shows where paths came from; after a successful multi-worker run it reminds you to run merge-coverage unless you use `parallel-rspec` or `run-shards --merge-coverage`.
|
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
require "json"
|
|
2
|
+
|
|
3
|
+
require_relative "../config/effective"
|
|
4
|
+
|
|
5
|
+
module Polyrun
|
|
6
|
+
class CLI
|
|
7
|
+
module ConfigCommand
|
|
8
|
+
private
|
|
9
|
+
|
|
10
|
+
def cmd_config(argv, config_path)
|
|
11
|
+
dotted = argv.shift
|
|
12
|
+
if dotted.nil? || dotted.strip.empty?
|
|
13
|
+
Polyrun::Log.warn "polyrun config: need a dotted path (e.g. prepare.env.PLAYWRIGHT_ENV, partition.paths_file, workers)"
|
|
14
|
+
return 2
|
|
15
|
+
end
|
|
16
|
+
unless argv.empty?
|
|
17
|
+
Polyrun::Log.warn "polyrun config: unexpected arguments: #{argv.join(" ")}"
|
|
18
|
+
return 2
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
cfg = Polyrun::Config.load(path: config_path || ENV["POLYRUN_CONFIG"])
|
|
22
|
+
val = Polyrun::Config::Effective.dig(cfg, dotted)
|
|
23
|
+
if val.nil?
|
|
24
|
+
Polyrun::Log.warn "polyrun config: no value for #{dotted}"
|
|
25
|
+
return 1
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
Polyrun::Log.puts format_config_value(val)
|
|
29
|
+
0
|
|
30
|
+
end
|
|
31
|
+
|
|
32
|
+
def format_config_value(val)
|
|
33
|
+
case val
|
|
34
|
+
when Hash, Array
|
|
35
|
+
JSON.generate(val)
|
|
36
|
+
else
|
|
37
|
+
val.to_s
|
|
38
|
+
end
|
|
39
|
+
end
|
|
40
|
+
end
|
|
41
|
+
end
|
|
42
|
+
end
|
|
@@ -0,0 +1,115 @@
|
|
|
1
|
+
require "tempfile"
|
|
2
|
+
|
|
3
|
+
module Polyrun
|
|
4
|
+
class CLI
|
|
5
|
+
# No-subcommand default (`polyrun`) and path-only argv (implicit parallel run).
|
|
6
|
+
module DefaultRun
|
|
7
|
+
private
|
|
8
|
+
|
|
9
|
+
def dispatch_default_parallel!(config_path)
|
|
10
|
+
suite = Polyrun::Partition::Paths.detect_auto_suite(Dir.pwd)
|
|
11
|
+
unless suite
|
|
12
|
+
Polyrun::Log.warn "polyrun: no tests found (spec/**/*_spec.rb, test/**/*_test.rb, or Polyrun quick files). See polyrun help."
|
|
13
|
+
return 2
|
|
14
|
+
end
|
|
15
|
+
|
|
16
|
+
Polyrun::Log.warn "polyrun: default → parallel #{suite} (use `polyrun help` for subcommands)" if @verbose
|
|
17
|
+
|
|
18
|
+
case suite
|
|
19
|
+
when :rspec
|
|
20
|
+
cmd_start([], config_path)
|
|
21
|
+
when :minitest
|
|
22
|
+
cmd_parallel_minitest([], config_path)
|
|
23
|
+
when :quick
|
|
24
|
+
cmd_parallel_quick([], config_path)
|
|
25
|
+
else
|
|
26
|
+
2
|
|
27
|
+
end
|
|
28
|
+
end
|
|
29
|
+
|
|
30
|
+
# If +argv[0]+ is in {IMPLICIT_PATH_EXCLUSION_TOKENS}, treat as a normal subcommand. Otherwise, path-like
|
|
31
|
+
# tokens may trigger implicit parallel sharding (see +print_help+).
|
|
32
|
+
def implicit_parallel_run?(argv)
|
|
33
|
+
return false if argv.empty?
|
|
34
|
+
return false if Polyrun::CLI::IMPLICIT_PATH_EXCLUSION_TOKENS.include?(argv[0])
|
|
35
|
+
|
|
36
|
+
argv.any? { |a| cli_implicit_path_token?(a) }
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
def cli_implicit_path_token?(s)
|
|
40
|
+
return false if s.start_with?("-") && s != "-"
|
|
41
|
+
return true if s == "-"
|
|
42
|
+
return true if s.start_with?("./", "../", "/")
|
|
43
|
+
return true if s.end_with?(".rb")
|
|
44
|
+
return true if File.exist?(File.expand_path(s))
|
|
45
|
+
return true if /[*?\[]/.match?(s)
|
|
46
|
+
|
|
47
|
+
false
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
def dispatch_implicit_parallel_targets!(argv, config_path)
|
|
51
|
+
path_tokens = argv.select { |a| cli_implicit_path_token?(a) }
|
|
52
|
+
head = argv.reject { |a| cli_implicit_path_token?(a) }
|
|
53
|
+
expanded = expand_implicit_target_paths(path_tokens)
|
|
54
|
+
if expanded.empty?
|
|
55
|
+
Polyrun::Log.warn "polyrun: no files matched path arguments"
|
|
56
|
+
return 2
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
suite = Polyrun::Partition::Paths.infer_suite_from_paths(expanded)
|
|
60
|
+
if suite == :invalid
|
|
61
|
+
Polyrun::Log.warn "polyrun: mixing _spec.rb and _test.rb paths in one run is not supported"
|
|
62
|
+
return 2
|
|
63
|
+
end
|
|
64
|
+
if suite.nil?
|
|
65
|
+
Polyrun::Log.warn "polyrun: could not infer suite from paths"
|
|
66
|
+
return 2
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
tmp = Tempfile.new(["polyrun-paths-", ".txt"])
|
|
70
|
+
begin
|
|
71
|
+
tmp.write(expanded.join("\n") + "\n")
|
|
72
|
+
tmp.close
|
|
73
|
+
combined = head + ["--paths-file", tmp.path]
|
|
74
|
+
case suite
|
|
75
|
+
when :rspec
|
|
76
|
+
cmd_start(combined, config_path)
|
|
77
|
+
when :minitest
|
|
78
|
+
cmd_parallel_minitest(combined, config_path)
|
|
79
|
+
when :quick
|
|
80
|
+
cmd_parallel_quick(combined, config_path)
|
|
81
|
+
else
|
|
82
|
+
2
|
|
83
|
+
end
|
|
84
|
+
ensure
|
|
85
|
+
tmp.close! unless tmp.closed?
|
|
86
|
+
begin
|
|
87
|
+
File.unlink(tmp.path)
|
|
88
|
+
rescue Errno::ENOENT
|
|
89
|
+
# already removed
|
|
90
|
+
end
|
|
91
|
+
end
|
|
92
|
+
end
|
|
93
|
+
|
|
94
|
+
def expand_implicit_target_paths(path_tokens)
|
|
95
|
+
path_tokens.flat_map do |p|
|
|
96
|
+
abs = File.expand_path(p)
|
|
97
|
+
if File.directory?(abs)
|
|
98
|
+
spec = Dir.glob(File.join(abs, "**", "*_spec.rb")).sort
|
|
99
|
+
test = Dir.glob(File.join(abs, "**", "*_test.rb")).sort
|
|
100
|
+
quick = Dir.glob(File.join(abs, "**", "*.rb")).sort.reject do |f|
|
|
101
|
+
File.basename(f).end_with?("_spec.rb", "_test.rb")
|
|
102
|
+
end
|
|
103
|
+
spec + test + quick
|
|
104
|
+
elsif /[*?\[]/.match?(p)
|
|
105
|
+
Dir.glob(abs).sort
|
|
106
|
+
elsif File.file?(abs)
|
|
107
|
+
[abs]
|
|
108
|
+
else
|
|
109
|
+
[]
|
|
110
|
+
end
|
|
111
|
+
end.uniq
|
|
112
|
+
end
|
|
113
|
+
end
|
|
114
|
+
end
|
|
115
|
+
end
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
module Polyrun
|
|
2
|
+
class CLI
|
|
3
|
+
module Help
|
|
4
|
+
def print_help
|
|
5
|
+
Polyrun::Log.puts <<~HELP
|
|
6
|
+
usage: polyrun [global options] [<command> | <paths...>]
|
|
7
|
+
|
|
8
|
+
With no command, runs parallel tests for the detected suite: RSpec under spec/, Minitest under test/, or Polyrun Quick (same discovery as polyrun quick). If the first argument is a known subcommand name, it is dispatched. Otherwise, path-like tokens (optionally with run-shards flags such as --workers) shard those files in parallel; see commands below.
|
|
9
|
+
|
|
10
|
+
global:
|
|
11
|
+
-c, --config PATH polyrun.yml path (or POLYRUN_CONFIG)
|
|
12
|
+
-v, --verbose
|
|
13
|
+
-h, --help
|
|
14
|
+
|
|
15
|
+
Trace timing (stderr): DEBUG=1 or POLYRUN_DEBUG=1
|
|
16
|
+
Branch coverage in JSON fragments: POLYRUN_COVERAGE_BRANCHES=1 (stdlib Coverage; merge-coverage merges branches)
|
|
17
|
+
polyrun quick coverage: POLYRUN_COVERAGE=1 or (config/polyrun_coverage.yml + POLYRUN_QUICK_COVERAGE=1); POLYRUN_COVERAGE_DISABLE=1 skips
|
|
18
|
+
Merge wall time (stderr): POLYRUN_PROFILE_MERGE=1 (or verbose / DEBUG)
|
|
19
|
+
Post-merge formats (run-shards): POLYRUN_MERGE_FORMATS (default: json,lcov,cobertura,console,html)
|
|
20
|
+
Skip optional script/build_spec_paths.rb before start: POLYRUN_SKIP_BUILD_SPEC_PATHS=1
|
|
21
|
+
Skip start auto-prepare / auto DB provision: POLYRUN_START_SKIP_PREPARE=1, POLYRUN_START_SKIP_DATABASES=1
|
|
22
|
+
Skip writing paths_file from partition.paths_build: POLYRUN_SKIP_PATHS_BUILD=1
|
|
23
|
+
Warn if merge-coverage wall time exceeds N seconds (default 10): POLYRUN_MERGE_SLOW_WARN_SECONDS (0 disables)
|
|
24
|
+
Parallel RSpec workers: POLYRUN_WORKERS default 5, max 10 (run-shards / parallel-rspec / start)
|
|
25
|
+
Partition timing granularity (default file): POLYRUN_TIMING_GRANULARITY=file|example (experimental per-example; see partition.timing_granularity)
|
|
26
|
+
|
|
27
|
+
commands:
|
|
28
|
+
version print version
|
|
29
|
+
plan emit partition manifest JSON
|
|
30
|
+
prepare run prepare recipe: default | assets (optional prepare.command overrides bin/rails assets:precompile) | shell (prepare.command required)
|
|
31
|
+
merge-coverage merge SimpleCov JSON fragments (json/lcov/cobertura/console)
|
|
32
|
+
run-shards fan out N parallel OS processes (POLYRUN_SHARD_*; not Ruby threads); optional --merge-coverage
|
|
33
|
+
parallel-rspec run-shards + merge-coverage (defaults to: bundle exec rspec after --)
|
|
34
|
+
start parallel-rspec; auto-runs prepare (shell/assets) and db:setup-* when polyrun.yml configures them; legacy script/build_spec_paths.rb if paths_build absent
|
|
35
|
+
ci-shard-run CI matrix: build-paths + plan for POLYRUN_SHARD_INDEX / POLYRUN_SHARD_TOTAL (or config), then run your command with that shard's paths after -- (like run-shards; not multi-worker)
|
|
36
|
+
ci-shard-rspec same as ci-shard-run -- bundle exec rspec; optional -- [rspec-only flags]
|
|
37
|
+
build-paths write partition.paths_file from partition.paths_build (same as auto step before plan/run-shards)
|
|
38
|
+
init write a starter polyrun.yml or POLYRUN.md from built-in templates (see docs/SETUP_PROFILE.md)
|
|
39
|
+
queue file-backed batch queue (init / claim / ack / status)
|
|
40
|
+
quick run Polyrun::Quick (describe/it, before/after, let, expect…to, assert_*; optional capybara!)
|
|
41
|
+
report-coverage write all coverage formats from one JSON file
|
|
42
|
+
report-junit RSpec JSON or Polyrun testcase JSON → JUnit XML (CI)
|
|
43
|
+
report-timing print slow-file summary from merged timing JSON
|
|
44
|
+
merge-timing merge polyrun_timing_*.json shards
|
|
45
|
+
config print effective config by dotted path (see Polyrun::Config::Effective; same tree as YAML plus merged prepare.env, resolved partition shard fields, workers)
|
|
46
|
+
env print shard + database env (see polyrun.yml databases)
|
|
47
|
+
db:setup-template migrate template DB (PostgreSQL)
|
|
48
|
+
db:setup-shard CREATE DATABASE shard FROM template (one POLYRUN_SHARD_INDEX)
|
|
49
|
+
db:clone-shards migrate templates + DROP/CREATE all shard DBs (replaces clone_shard shell scripts)
|
|
50
|
+
HELP
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
end
|
|
54
|
+
end
|
data/lib/polyrun/cli/helpers.rb
CHANGED
|
@@ -7,40 +7,16 @@ module Polyrun
|
|
|
7
7
|
module Helpers
|
|
8
8
|
private
|
|
9
9
|
|
|
10
|
-
def partition_int(pc, keys, default)
|
|
11
|
-
keys.each do |k|
|
|
12
|
-
v = pc[k] || pc[k.to_sym]
|
|
13
|
-
next if v.nil? || v.to_s.empty?
|
|
14
|
-
|
|
15
|
-
i = Integer(v, exception: false)
|
|
16
|
-
return i unless i.nil?
|
|
17
|
-
end
|
|
18
|
-
default
|
|
19
|
-
end
|
|
20
|
-
|
|
21
10
|
def env_int(name, fallback)
|
|
22
|
-
|
|
23
|
-
return fallback if s.nil? || s.empty?
|
|
24
|
-
|
|
25
|
-
Integer(s, exception: false) || fallback
|
|
11
|
+
Polyrun::Config::Resolver.env_int(name, fallback)
|
|
26
12
|
end
|
|
27
13
|
|
|
28
14
|
def resolve_shard_index(pc)
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
ci = Polyrun::Env::Ci.detect_shard_index
|
|
32
|
-
return ci unless ci.nil?
|
|
33
|
-
|
|
34
|
-
partition_int(pc, %w[shard_index shard], 0)
|
|
15
|
+
Polyrun::Config::Resolver.resolve_shard_index(pc)
|
|
35
16
|
end
|
|
36
17
|
|
|
37
18
|
def resolve_shard_total(pc)
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
ci = Polyrun::Env::Ci.detect_shard_total
|
|
41
|
-
return ci unless ci.nil?
|
|
42
|
-
|
|
43
|
-
partition_int(pc, %w[shard_total total], 1)
|
|
19
|
+
Polyrun::Config::Resolver.resolve_shard_total(pc)
|
|
44
20
|
end
|
|
45
21
|
|
|
46
22
|
def expand_merge_input_pattern(path)
|
|
@@ -119,10 +95,7 @@ module Polyrun
|
|
|
119
95
|
|
|
120
96
|
# CLI + polyrun.yml + POLYRUN_TIMING_GRANULARITY; default +:file+.
|
|
121
97
|
def resolve_partition_timing_granularity(pc, cli_val)
|
|
122
|
-
|
|
123
|
-
raw ||= pc && (pc["timing_granularity"] || pc[:timing_granularity])
|
|
124
|
-
raw ||= ENV["POLYRUN_TIMING_GRANULARITY"]
|
|
125
|
-
Polyrun::Partition::TimingKeys.normalize_granularity(raw || "file")
|
|
98
|
+
Polyrun::Config::Resolver.resolve_partition_timing_granularity(pc, cli_val)
|
|
126
99
|
end
|
|
127
100
|
end
|
|
128
101
|
end
|
|
@@ -19,8 +19,8 @@ module Polyrun
|
|
|
19
19
|
cfg = Polyrun::Config.load(path: config_path || ENV["POLYRUN_CONFIG"])
|
|
20
20
|
prep = cfg.prepare
|
|
21
21
|
recipe = prep["recipe"] || prep[:recipe] || "default"
|
|
22
|
-
prep_env = (prep
|
|
23
|
-
child_env = prep_env.empty? ? nil :
|
|
22
|
+
prep_env = Polyrun::Config::Resolver.prepare_env_yaml_string_map(prep)
|
|
23
|
+
child_env = prep_env.empty? ? nil : Polyrun::Config::Resolver.merged_prepare_env(prep)
|
|
24
24
|
manifest = prepare_build_manifest(recipe, dry, prep_env)
|
|
25
25
|
|
|
26
26
|
exit_code = prepare_dispatch_recipe(manifest, prep, recipe, dry, child_env)
|
|
@@ -12,9 +12,7 @@ module Polyrun
|
|
|
12
12
|
|
|
13
13
|
private
|
|
14
14
|
|
|
15
|
-
# Default and upper bound for parallel OS processes (POLYRUN_WORKERS / --workers).
|
|
16
|
-
DEFAULT_PARALLEL_WORKERS = 5
|
|
17
|
-
MAX_PARALLEL_WORKERS = 10
|
|
15
|
+
# Default and upper bound for parallel OS processes (POLYRUN_WORKERS / --workers); see {Polyrun::Config}.
|
|
18
16
|
|
|
19
17
|
# Spawns N OS processes (not Ruby threads) with POLYRUN_SHARD_INDEX / POLYRUN_SHARD_TOTAL so
|
|
20
18
|
# {Coverage::Collector} writes coverage/polyrun-fragment-<shard>.json. Merge with merge-coverage.
|
|
@@ -37,6 +35,54 @@ module Polyrun
|
|
|
37
35
|
cmd_run_shards(combined, config_path)
|
|
38
36
|
end
|
|
39
37
|
|
|
38
|
+
# Same as parallel-rspec but runs +bundle exec rails test+ or +bundle exec ruby -I test+ after +--+.
|
|
39
|
+
def cmd_parallel_minitest(argv, config_path)
|
|
40
|
+
cfg = Polyrun::Config.load(path: config_path || ENV["POLYRUN_CONFIG"])
|
|
41
|
+
code = start_bootstrap!(cfg, argv, config_path)
|
|
42
|
+
return code if code != 0
|
|
43
|
+
|
|
44
|
+
sep = argv.index("--")
|
|
45
|
+
combined =
|
|
46
|
+
if sep
|
|
47
|
+
head = argv[0...sep]
|
|
48
|
+
tail = argv[sep..]
|
|
49
|
+
head + ["--merge-coverage"] + tail
|
|
50
|
+
else
|
|
51
|
+
argv + ["--merge-coverage", "--"] + minitest_parallel_cmd
|
|
52
|
+
end
|
|
53
|
+
Polyrun::Debug.log_kv(parallel_minitest: "combined argv", argv: combined)
|
|
54
|
+
cmd_run_shards(combined, config_path)
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
# Same as parallel-rspec but runs +bundle exec polyrun quick+ after +--+ (one Quick process per shard).
|
|
58
|
+
# Run from the app root with +bundle exec+ so workers resolve the same gem as the parent (same concern as +bundle exec rspec+).
|
|
59
|
+
def cmd_parallel_quick(argv, config_path)
|
|
60
|
+
cfg = Polyrun::Config.load(path: config_path || ENV["POLYRUN_CONFIG"])
|
|
61
|
+
code = start_bootstrap!(cfg, argv, config_path)
|
|
62
|
+
return code if code != 0
|
|
63
|
+
|
|
64
|
+
sep = argv.index("--")
|
|
65
|
+
combined =
|
|
66
|
+
if sep
|
|
67
|
+
head = argv[0...sep]
|
|
68
|
+
tail = argv[sep..]
|
|
69
|
+
head + ["--merge-coverage"] + tail
|
|
70
|
+
else
|
|
71
|
+
argv + ["--merge-coverage", "--", "bundle", "exec", "polyrun", "quick"]
|
|
72
|
+
end
|
|
73
|
+
Polyrun::Debug.log_kv(parallel_quick: "combined argv", argv: combined)
|
|
74
|
+
cmd_run_shards(combined, config_path)
|
|
75
|
+
end
|
|
76
|
+
|
|
77
|
+
def minitest_parallel_cmd
|
|
78
|
+
rails_bin = File.expand_path("bin/rails", Dir.pwd)
|
|
79
|
+
if File.file?(rails_bin)
|
|
80
|
+
["bundle", "exec", "rails", "test"]
|
|
81
|
+
else
|
|
82
|
+
["bundle", "exec", "ruby", "-I", "test"]
|
|
83
|
+
end
|
|
84
|
+
end
|
|
85
|
+
|
|
40
86
|
# Convenience alias: optional legacy script/build_spec_paths.rb (if present and partition.paths_build unset), then parallel-rspec.
|
|
41
87
|
def cmd_start(argv, config_path)
|
|
42
88
|
cfg = Polyrun::Config.load(path: config_path || ENV["POLYRUN_CONFIG"])
|
|
@@ -25,7 +25,7 @@ module Polyrun
|
|
|
25
25
|
end
|
|
26
26
|
|
|
27
27
|
def run_shards_plan_phase_b(o, cmd, cfg, pc, run_t0, config_path)
|
|
28
|
-
items, paths_source, err = run_shards_resolve_items(o[:paths_file])
|
|
28
|
+
items, paths_source, err = run_shards_resolve_items(o[:paths_file], pc)
|
|
29
29
|
return [err, nil] if err
|
|
30
30
|
|
|
31
31
|
costs, strategy, err = run_shards_resolve_costs(o[:timing_path], o[:strategy], o[:timing_granularity])
|
|
@@ -15,7 +15,7 @@ module Polyrun
|
|
|
15
15
|
|
|
16
16
|
def run_shards_plan_options_state(pc)
|
|
17
17
|
{
|
|
18
|
-
workers: env_int("POLYRUN_WORKERS",
|
|
18
|
+
workers: env_int("POLYRUN_WORKERS", Polyrun::Config::DEFAULT_PARALLEL_WORKERS),
|
|
19
19
|
paths_file: nil,
|
|
20
20
|
strategy: (pc["strategy"] || pc[:strategy] || "round_robin").to_s,
|
|
21
21
|
seed: pc["seed"] || pc[:seed],
|
|
@@ -38,9 +38,9 @@ module Polyrun
|
|
|
38
38
|
Polyrun::Log.warn "polyrun run-shards: --workers must be >= 1"
|
|
39
39
|
return 2
|
|
40
40
|
end
|
|
41
|
-
if w >
|
|
42
|
-
Polyrun::Log.warn "polyrun run-shards: capping --workers / POLYRUN_WORKERS from #{w} to #{
|
|
43
|
-
o[:workers] =
|
|
41
|
+
if w > Polyrun::Config::MAX_PARALLEL_WORKERS
|
|
42
|
+
Polyrun::Log.warn "polyrun run-shards: capping --workers / POLYRUN_WORKERS from #{w} to #{Polyrun::Config::MAX_PARALLEL_WORKERS}"
|
|
43
|
+
o[:workers] = Polyrun::Config::MAX_PARALLEL_WORKERS
|
|
44
44
|
end
|
|
45
45
|
nil
|
|
46
46
|
end
|
|
@@ -53,18 +53,18 @@ module Polyrun
|
|
|
53
53
|
nil
|
|
54
54
|
end
|
|
55
55
|
|
|
56
|
-
def run_shards_resolve_items(paths_file)
|
|
57
|
-
resolved = Polyrun::Partition::Paths.resolve_run_shard_items(paths_file: paths_file)
|
|
56
|
+
def run_shards_resolve_items(paths_file, partition)
|
|
57
|
+
resolved = Polyrun::Partition::Paths.resolve_run_shard_items(paths_file: paths_file, partition: partition)
|
|
58
58
|
if resolved[:error]
|
|
59
59
|
Polyrun::Log.warn "polyrun run-shards: #{resolved[:error]}"
|
|
60
60
|
return [nil, nil, 2]
|
|
61
61
|
end
|
|
62
62
|
items = resolved[:items]
|
|
63
63
|
paths_source = resolved[:source]
|
|
64
|
-
Polyrun::Log.warn "polyrun run-shards: #{items.size}
|
|
64
|
+
Polyrun::Log.warn "polyrun run-shards: #{items.size} path(s) from #{paths_source}"
|
|
65
65
|
|
|
66
66
|
if items.empty?
|
|
67
|
-
Polyrun::Log.warn "polyrun run-shards: no
|
|
67
|
+
Polyrun::Log.warn "polyrun run-shards: no paths (empty paths file or list)"
|
|
68
68
|
return [nil, nil, 2]
|
|
69
69
|
end
|
|
70
70
|
[items, paths_source, nil]
|
|
@@ -119,7 +119,7 @@ module Polyrun
|
|
|
119
119
|
|
|
120
120
|
def run_shards_warn_parallel_banner(item_count, workers, strategy)
|
|
121
121
|
Polyrun::Log.warn <<~MSG
|
|
122
|
-
polyrun run-shards: #{item_count}
|
|
122
|
+
polyrun run-shards: #{item_count} path(s) -> #{workers} parallel worker processes (not Ruby threads); strategy=#{strategy}
|
|
123
123
|
(plain `bundle exec rspec` is one process; this command fans out.)
|
|
124
124
|
MSG
|
|
125
125
|
end
|
|
@@ -4,10 +4,6 @@ module Polyrun
|
|
|
4
4
|
module StartBootstrap
|
|
5
5
|
private
|
|
6
6
|
|
|
7
|
-
# Keep in sync with {RunShardsCommand} worker defaults.
|
|
8
|
-
START_ARG_WORKERS_DEFAULT = 5
|
|
9
|
-
START_ARG_WORKERS_MAX = 10
|
|
10
|
-
|
|
11
7
|
def start_bootstrap!(cfg, argv, config_path)
|
|
12
8
|
if start_run_prepare?(cfg) && !truthy_env?("POLYRUN_START_SKIP_PREPARE")
|
|
13
9
|
recipe = cfg.prepare["recipe"] || cfg.prepare[:recipe] || "default"
|
|
@@ -76,7 +72,7 @@ module Polyrun
|
|
|
76
72
|
def parse_workers_from_start_argv(argv)
|
|
77
73
|
sep = argv.index("--")
|
|
78
74
|
head = sep ? argv[0...sep] : argv
|
|
79
|
-
workers = env_int("POLYRUN_WORKERS",
|
|
75
|
+
workers = env_int("POLYRUN_WORKERS", Polyrun::Config::DEFAULT_PARALLEL_WORKERS)
|
|
80
76
|
i = 0
|
|
81
77
|
while i < head.size
|
|
82
78
|
if head[i] == "--workers" && head[i + 1]
|
|
@@ -87,7 +83,7 @@ module Polyrun
|
|
|
87
83
|
i += 1
|
|
88
84
|
end
|
|
89
85
|
end
|
|
90
|
-
workers.clamp(1,
|
|
86
|
+
workers.clamp(1, Polyrun::Config::MAX_PARALLEL_WORKERS)
|
|
91
87
|
end
|
|
92
88
|
|
|
93
89
|
def truthy_env?(name)
|
data/lib/polyrun/cli.rb
CHANGED
|
@@ -13,6 +13,9 @@ require_relative "cli/timing_command"
|
|
|
13
13
|
require_relative "cli/init_command"
|
|
14
14
|
require_relative "cli/quick_command"
|
|
15
15
|
require_relative "cli/ci_shard_run_command"
|
|
16
|
+
require_relative "cli/config_command"
|
|
17
|
+
require_relative "cli/default_run"
|
|
18
|
+
require_relative "cli/help"
|
|
16
19
|
|
|
17
20
|
module Polyrun
|
|
18
21
|
class CLI
|
|
@@ -21,6 +24,18 @@ module Polyrun
|
|
|
21
24
|
"ci-shard-rspec" => :cmd_ci_shard_rspec
|
|
22
25
|
}.freeze
|
|
23
26
|
|
|
27
|
+
# Keep in sync with +dispatch_cli_command_subcommands+ (+when+ branches). Used for implicit path routing.
|
|
28
|
+
DISPATCH_SUBCOMMAND_NAMES = %w[
|
|
29
|
+
plan prepare merge-coverage report-coverage report-junit report-timing
|
|
30
|
+
env config merge-timing db:setup-template db:setup-shard db:clone-shards
|
|
31
|
+
run-shards parallel-rspec start build-paths init queue quick
|
|
32
|
+
].freeze
|
|
33
|
+
|
|
34
|
+
# First argv token that is a normal subcommand (not a path); if argv[0] is not here but looks like paths, run implicit parallel.
|
|
35
|
+
IMPLICIT_PATH_EXCLUSION_TOKENS = (
|
|
36
|
+
DISPATCH_SUBCOMMAND_NAMES + CI_SHARD_COMMANDS.keys + %w[help version]
|
|
37
|
+
).freeze
|
|
38
|
+
|
|
24
39
|
include Helpers
|
|
25
40
|
include PlanCommand
|
|
26
41
|
include PrepareCommand
|
|
@@ -34,6 +49,9 @@ module Polyrun
|
|
|
34
49
|
include InitCommand
|
|
35
50
|
include QuickCommand
|
|
36
51
|
include CiShardRunCommand
|
|
52
|
+
include ConfigCommand
|
|
53
|
+
include DefaultRun
|
|
54
|
+
include Help
|
|
37
55
|
|
|
38
56
|
def self.run(argv = ARGV)
|
|
39
57
|
new.run(argv)
|
|
@@ -44,12 +62,30 @@ module Polyrun
|
|
|
44
62
|
config_path = parse_global_cli!(argv)
|
|
45
63
|
return config_path if config_path.is_a?(Integer)
|
|
46
64
|
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
65
|
+
if argv.empty?
|
|
66
|
+
Polyrun::Debug.log_kv(
|
|
67
|
+
command: "(default)",
|
|
68
|
+
cwd: Dir.pwd,
|
|
69
|
+
polyrun_config: config_path,
|
|
70
|
+
argv_rest: [],
|
|
71
|
+
verbose: @verbose
|
|
72
|
+
)
|
|
73
|
+
return dispatch_default_parallel!(config_path)
|
|
74
|
+
end
|
|
75
|
+
|
|
76
|
+
if implicit_parallel_run?(argv)
|
|
77
|
+
Polyrun::Debug.log_kv(
|
|
78
|
+
command: "(paths)",
|
|
79
|
+
cwd: Dir.pwd,
|
|
80
|
+
polyrun_config: config_path,
|
|
81
|
+
argv_rest: argv.dup,
|
|
82
|
+
verbose: @verbose
|
|
83
|
+
)
|
|
84
|
+
return dispatch_implicit_parallel_targets!(argv, config_path)
|
|
51
85
|
end
|
|
52
86
|
|
|
87
|
+
command = argv.shift
|
|
88
|
+
|
|
53
89
|
Polyrun::Debug.log_kv(
|
|
54
90
|
command: command,
|
|
55
91
|
cwd: Dir.pwd,
|
|
@@ -96,6 +132,7 @@ module Polyrun
|
|
|
96
132
|
end
|
|
97
133
|
end
|
|
98
134
|
|
|
135
|
+
# rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity -- explicit dispatch table
|
|
99
136
|
def dispatch_cli_command_subcommands(command, argv, config_path)
|
|
100
137
|
case command
|
|
101
138
|
when "plan"
|
|
@@ -112,6 +149,8 @@ module Polyrun
|
|
|
112
149
|
cmd_report_timing(argv)
|
|
113
150
|
when "env"
|
|
114
151
|
cmd_env(argv, config_path)
|
|
152
|
+
when "config"
|
|
153
|
+
cmd_config(argv, config_path)
|
|
115
154
|
when "merge-timing"
|
|
116
155
|
cmd_merge_timing(argv)
|
|
117
156
|
when "db:setup-template"
|
|
@@ -141,52 +180,7 @@ module Polyrun
|
|
|
141
180
|
2
|
|
142
181
|
end
|
|
143
182
|
end
|
|
144
|
-
|
|
145
|
-
def print_help
|
|
146
|
-
Polyrun::Log.puts <<~HELP
|
|
147
|
-
usage: polyrun [global options] <command> [options]
|
|
148
|
-
|
|
149
|
-
global:
|
|
150
|
-
-c, --config PATH polyrun.yml path (or POLYRUN_CONFIG)
|
|
151
|
-
-v, --verbose
|
|
152
|
-
-h, --help
|
|
153
|
-
|
|
154
|
-
Trace timing (stderr): DEBUG=1 or POLYRUN_DEBUG=1
|
|
155
|
-
Branch coverage in JSON fragments: POLYRUN_COVERAGE_BRANCHES=1 (stdlib Coverage; merge-coverage merges branches)
|
|
156
|
-
polyrun quick coverage: POLYRUN_COVERAGE=1 or (config/polyrun_coverage.yml + POLYRUN_QUICK_COVERAGE=1); POLYRUN_COVERAGE_DISABLE=1 skips
|
|
157
|
-
Merge wall time (stderr): POLYRUN_PROFILE_MERGE=1 (or verbose / DEBUG)
|
|
158
|
-
Post-merge formats (run-shards): POLYRUN_MERGE_FORMATS (default: json,lcov,cobertura,console,html)
|
|
159
|
-
Skip optional script/build_spec_paths.rb before start: POLYRUN_SKIP_BUILD_SPEC_PATHS=1
|
|
160
|
-
Skip start auto-prepare / auto DB provision: POLYRUN_START_SKIP_PREPARE=1, POLYRUN_START_SKIP_DATABASES=1
|
|
161
|
-
Skip writing paths_file from partition.paths_build: POLYRUN_SKIP_PATHS_BUILD=1
|
|
162
|
-
Warn if merge-coverage wall time exceeds N seconds (default 10): POLYRUN_MERGE_SLOW_WARN_SECONDS (0 disables)
|
|
163
|
-
Parallel RSpec workers: POLYRUN_WORKERS default 5, max 10 (run-shards / parallel-rspec / start)
|
|
164
|
-
Partition timing granularity (default file): POLYRUN_TIMING_GRANULARITY=file|example (experimental per-example; see partition.timing_granularity)
|
|
165
|
-
|
|
166
|
-
commands:
|
|
167
|
-
version print version
|
|
168
|
-
plan emit partition manifest JSON
|
|
169
|
-
prepare run prepare recipe: default | assets (optional prepare.command overrides bin/rails assets:precompile) | shell (prepare.command required)
|
|
170
|
-
merge-coverage merge SimpleCov JSON fragments (json/lcov/cobertura/console)
|
|
171
|
-
run-shards fan out N parallel OS processes (POLYRUN_SHARD_*; not Ruby threads); optional --merge-coverage
|
|
172
|
-
parallel-rspec run-shards + merge-coverage (defaults to: bundle exec rspec after --)
|
|
173
|
-
start parallel-rspec; auto-runs prepare (shell/assets) and db:setup-* when polyrun.yml configures them; legacy script/build_spec_paths.rb if paths_build absent
|
|
174
|
-
ci-shard-run CI matrix: build-paths + plan for POLYRUN_SHARD_INDEX / POLYRUN_SHARD_TOTAL (or config), then run your command with that shard's paths after -- (like run-shards; not multi-worker)
|
|
175
|
-
ci-shard-rspec same as ci-shard-run -- bundle exec rspec; optional -- [rspec-only flags]
|
|
176
|
-
build-paths write partition.paths_file from partition.paths_build (same as auto step before plan/run-shards)
|
|
177
|
-
init write a starter polyrun.yml or POLYRUN.md from built-in templates (see docs/SETUP_PROFILE.md)
|
|
178
|
-
queue file-backed batch queue (init / claim / ack / status)
|
|
179
|
-
quick run Polyrun::Quick (describe/it, before/after, let, expect…to, assert_*; optional capybara!)
|
|
180
|
-
report-coverage write all coverage formats from one JSON file
|
|
181
|
-
report-junit RSpec JSON or Polyrun testcase JSON → JUnit XML (CI)
|
|
182
|
-
report-timing print slow-file summary from merged timing JSON
|
|
183
|
-
merge-timing merge polyrun_timing_*.json shards
|
|
184
|
-
env print shard + database env (see polyrun.yml databases)
|
|
185
|
-
db:setup-template migrate template DB (PostgreSQL)
|
|
186
|
-
db:setup-shard CREATE DATABASE shard FROM template (one POLYRUN_SHARD_INDEX)
|
|
187
|
-
db:clone-shards migrate templates + DROP/CREATE all shard DBs (replaces clone_shard shell scripts)
|
|
188
|
-
HELP
|
|
189
|
-
end
|
|
183
|
+
# rubocop:enable Metrics/AbcSize, Metrics/CyclomaticComplexity
|
|
190
184
|
|
|
191
185
|
def cmd_version
|
|
192
186
|
Polyrun::Log.puts "polyrun #{Polyrun::VERSION}"
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
module Polyrun
|
|
2
|
+
class Config
|
|
3
|
+
# Read nested keys from loaded YAML (+String+ / +Symbol+ indifferent at each step).
|
|
4
|
+
module DottedPath
|
|
5
|
+
module_function
|
|
6
|
+
|
|
7
|
+
def dig(raw, dotted)
|
|
8
|
+
segments = dotted.split(".")
|
|
9
|
+
return nil if segments.empty?
|
|
10
|
+
return nil if segments.any?(&:empty?)
|
|
11
|
+
|
|
12
|
+
segments.reduce(raw) do |m, seg|
|
|
13
|
+
break nil if m.nil?
|
|
14
|
+
break nil unless m.is_a?(Hash)
|
|
15
|
+
|
|
16
|
+
m[seg] || m[seg.to_sym]
|
|
17
|
+
end
|
|
18
|
+
end
|
|
19
|
+
end
|
|
20
|
+
end
|
|
21
|
+
end
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
require_relative "dotted_path"
|
|
2
|
+
require_relative "resolver"
|
|
3
|
+
|
|
4
|
+
module Polyrun
|
|
5
|
+
class Config
|
|
6
|
+
# Nested hash of values Polyrun uses: loaded YAML (string keys) with overlays for
|
|
7
|
+
# merged +prepare.env+, resolved +partition.shard_index+ / +shard_total+ / +timing_granularity+,
|
|
8
|
+
# and top-level +workers+ (+POLYRUN_WORKERS+ default).
|
|
9
|
+
#
|
|
10
|
+
# +build+ memoizes the last (cfg, env) in-process so repeated +dig+ calls on the same load do not
|
|
11
|
+
# rebuild the tree (single-threaded CLI).
|
|
12
|
+
module Effective
|
|
13
|
+
class << self
|
|
14
|
+
# Per-thread cache avoids rebuilding the effective tree on repeated +dig+; no class ivars (RuboCop ThreadSafety).
|
|
15
|
+
def build(cfg, env: ENV)
|
|
16
|
+
key = cache_key(cfg, env)
|
|
17
|
+
per_thread = (Thread.current[:polyrun_effective_build] ||= {})
|
|
18
|
+
per_thread[key] ||= build_uncached(cfg, env: env)
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
def dig(cfg, dotted_path, env: ENV)
|
|
22
|
+
Polyrun::Config::DottedPath.dig(build(cfg, env: env), dotted_path)
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
private
|
|
26
|
+
|
|
27
|
+
def cache_key(cfg, env)
|
|
28
|
+
[cfg.path, cfg.object_id, env_fingerprint(env)]
|
|
29
|
+
end
|
|
30
|
+
|
|
31
|
+
def env_fingerprint(env)
|
|
32
|
+
env.to_h.keys.sort.map { |k| [k, env[k]] }.hash
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
def build_uncached(cfg, env:)
|
|
36
|
+
r = Polyrun::Config::Resolver
|
|
37
|
+
base = deep_stringify_keys(cfg.raw)
|
|
38
|
+
|
|
39
|
+
prep = cfg.prepare
|
|
40
|
+
base["prepare"] = deep_stringify_keys(prep)
|
|
41
|
+
base["prepare"]["env"] = r.merged_prepare_env(prep, env)
|
|
42
|
+
|
|
43
|
+
pc = cfg.partition
|
|
44
|
+
part = deep_stringify_keys(pc).merge(
|
|
45
|
+
"shard_index" => r.resolve_shard_index(pc, env),
|
|
46
|
+
"shard_total" => r.resolve_shard_total(pc, env),
|
|
47
|
+
"timing_granularity" => r.resolve_partition_timing_granularity(pc, nil, env).to_s
|
|
48
|
+
)
|
|
49
|
+
base["partition"] = part
|
|
50
|
+
|
|
51
|
+
base["workers"] = r.parallel_worker_count_default(env)
|
|
52
|
+
|
|
53
|
+
base
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
def deep_stringify_keys(obj)
|
|
57
|
+
case obj
|
|
58
|
+
when Hash
|
|
59
|
+
obj.each_with_object({}) do |(k, v), m|
|
|
60
|
+
m[k.to_s] = deep_stringify_keys(v)
|
|
61
|
+
end
|
|
62
|
+
when Array
|
|
63
|
+
obj.map { |e| deep_stringify_keys(e) }
|
|
64
|
+
else
|
|
65
|
+
obj
|
|
66
|
+
end
|
|
67
|
+
end
|
|
68
|
+
end
|
|
69
|
+
end
|
|
70
|
+
end
|
|
71
|
+
end
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
require_relative "../env/ci"
|
|
2
|
+
require_relative "../partition/timing_keys"
|
|
3
|
+
|
|
4
|
+
module Polyrun
|
|
5
|
+
class Config
|
|
6
|
+
# Single source for values derived from +polyrun.yml+, +ENV+, and CI detection.
|
|
7
|
+
# Used by {Effective}, CLI helpers, and prepare.
|
|
8
|
+
module Resolver
|
|
9
|
+
module_function
|
|
10
|
+
|
|
11
|
+
def env_int(name, fallback, env = ENV)
|
|
12
|
+
s = env[name]
|
|
13
|
+
return fallback if s.nil? || s.empty?
|
|
14
|
+
|
|
15
|
+
Integer(s, exception: false) || fallback
|
|
16
|
+
end
|
|
17
|
+
|
|
18
|
+
def prepare_env_yaml_string_map(prep)
|
|
19
|
+
(prep["env"] || prep[:env] || {}).transform_keys(&:to_s).transform_values(&:to_s)
|
|
20
|
+
end
|
|
21
|
+
|
|
22
|
+
# Same merge order as +polyrun prepare+: YAML +prepare.env+ overrides process +ENV+ for overlapping keys.
|
|
23
|
+
def merged_prepare_env(prep, env = ENV)
|
|
24
|
+
prep_env = prepare_env_yaml_string_map(prep)
|
|
25
|
+
env.to_h.merge(prep_env)
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
def partition_int(pc, keys, default)
|
|
29
|
+
keys.each do |k|
|
|
30
|
+
v = pc[k] || pc[k.to_sym]
|
|
31
|
+
next if v.nil? || v.to_s.empty?
|
|
32
|
+
|
|
33
|
+
i = Integer(v, exception: false)
|
|
34
|
+
return i unless i.nil?
|
|
35
|
+
end
|
|
36
|
+
default
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
def resolve_shard_index(pc, env = ENV)
|
|
40
|
+
return Integer(env["POLYRUN_SHARD_INDEX"]) if env["POLYRUN_SHARD_INDEX"] && !env["POLYRUN_SHARD_INDEX"].empty?
|
|
41
|
+
|
|
42
|
+
ci = Polyrun::Env::Ci.detect_shard_index
|
|
43
|
+
return ci unless ci.nil?
|
|
44
|
+
|
|
45
|
+
partition_int(pc, %w[shard_index shard], 0)
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
def resolve_shard_total(pc, env = ENV)
|
|
49
|
+
return Integer(env["POLYRUN_SHARD_TOTAL"]) if env["POLYRUN_SHARD_TOTAL"] && !env["POLYRUN_SHARD_TOTAL"].empty?
|
|
50
|
+
|
|
51
|
+
ci = Polyrun::Env::Ci.detect_shard_total
|
|
52
|
+
return ci unless ci.nil?
|
|
53
|
+
|
|
54
|
+
partition_int(pc, %w[shard_total total], 1)
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
# +cli_val+ is an override (e.g. +run-shards --timing-granularity+); +nil+ uses YAML then +POLYRUN_TIMING_GRANULARITY+.
|
|
58
|
+
def resolve_partition_timing_granularity(pc, cli_val, env = ENV)
|
|
59
|
+
raw = cli_val
|
|
60
|
+
raw ||= pc && (pc["timing_granularity"] || pc[:timing_granularity])
|
|
61
|
+
raw ||= env["POLYRUN_TIMING_GRANULARITY"]
|
|
62
|
+
Polyrun::Partition::TimingKeys.normalize_granularity(raw || "file")
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
def parallel_worker_count_default(env = ENV)
|
|
66
|
+
env_int("POLYRUN_WORKERS", Polyrun::Config::DEFAULT_PARALLEL_WORKERS, env)
|
|
67
|
+
end
|
|
68
|
+
end
|
|
69
|
+
end
|
|
70
|
+
end
|
data/lib/polyrun/config.rb
CHANGED
|
@@ -5,6 +5,10 @@ module Polyrun
|
|
|
5
5
|
class Config
|
|
6
6
|
DEFAULT_FILENAMES = %w[polyrun.yml config/polyrun.yml].freeze
|
|
7
7
|
|
|
8
|
+
# Parallel worker defaults (+run-shards+, +POLYRUN_WORKERS+); single source with {Resolver} and {Effective}.
|
|
9
|
+
DEFAULT_PARALLEL_WORKERS = 5
|
|
10
|
+
MAX_PARALLEL_WORKERS = 10
|
|
11
|
+
|
|
8
12
|
attr_reader :path, :raw
|
|
9
13
|
|
|
10
14
|
def self.load(path: nil)
|
|
@@ -59,3 +63,6 @@ module Polyrun
|
|
|
59
63
|
end
|
|
60
64
|
end
|
|
61
65
|
end
|
|
66
|
+
|
|
67
|
+
require_relative "config/resolver"
|
|
68
|
+
require_relative "config/effective"
|
|
@@ -8,9 +8,45 @@ module Polyrun
|
|
|
8
8
|
File.read(File.expand_path(path.to_s, Dir.pwd)).split("\n").map(&:strip).reject(&:empty?)
|
|
9
9
|
end
|
|
10
10
|
|
|
11
|
+
# Prefer +spec/+ RSpec files, then +test/+ Minitest, then Polyrun Quick files (same globs as +polyrun quick+).
|
|
12
|
+
# Order avoids running the broader Quick glob when RSpec or Minitest files already exist.
|
|
13
|
+
def detect_auto_suite(cwd = Dir.pwd)
|
|
14
|
+
base = File.expand_path(cwd)
|
|
15
|
+
return :rspec if Dir.glob(File.join(base, "spec/**/*_spec.rb")).any?
|
|
16
|
+
|
|
17
|
+
return :minitest if Dir.glob(File.join(base, "test/**/*_test.rb")).any?
|
|
18
|
+
|
|
19
|
+
quick = quick_parallel_default_paths(base)
|
|
20
|
+
return :quick if quick.any?
|
|
21
|
+
|
|
22
|
+
nil
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
# Infer parallel suite from explicit paths (+_spec.rb+ vs +_test.rb+ vs Polyrun quick-style +.rb+).
|
|
26
|
+
# Returns +:rspec+, +:minitest+, +:quick+, +:invalid+ (mixed spec and test), or +nil+ (empty).
|
|
27
|
+
def infer_suite_from_paths(paths)
|
|
28
|
+
paths = paths.map { |p| File.expand_path(p) }
|
|
29
|
+
return nil if paths.empty?
|
|
30
|
+
|
|
31
|
+
specs = paths.count { |p| File.basename(p).end_with?("_spec.rb") }
|
|
32
|
+
tests = paths.count { |p| File.basename(p).end_with?("_test.rb") }
|
|
33
|
+
return :invalid if specs.positive? && tests.positive?
|
|
34
|
+
|
|
35
|
+
return :rspec if specs.positive?
|
|
36
|
+
return :minitest if tests.positive?
|
|
37
|
+
|
|
38
|
+
others = paths.size - specs - tests
|
|
39
|
+
return :quick if others.positive?
|
|
40
|
+
|
|
41
|
+
nil
|
|
42
|
+
end
|
|
43
|
+
|
|
11
44
|
# When +paths_file+ is set but missing, returns +{ error: "..." }+.
|
|
12
45
|
# Otherwise returns +{ items:, source: }+ (human-readable source label).
|
|
13
|
-
|
|
46
|
+
#
|
|
47
|
+
# +partition.suite+ (optional): +auto+ (default), +rspec+, +minitest+, +quick+ — used only when resolving
|
|
48
|
+
# from globs (no explicit +paths_file+ and no +spec/spec_paths.txt+).
|
|
49
|
+
def resolve_run_shard_items(paths_file: nil, cwd: Dir.pwd, partition: {})
|
|
14
50
|
if paths_file
|
|
15
51
|
abs = File.expand_path(paths_file.to_s, cwd)
|
|
16
52
|
unless File.file?(abs)
|
|
@@ -20,9 +56,54 @@ module Polyrun
|
|
|
20
56
|
elsif File.file?(File.join(cwd, "spec", "spec_paths.txt"))
|
|
21
57
|
{items: read_lines(File.join(cwd, "spec", "spec_paths.txt")), source: "spec/spec_paths.txt"}
|
|
22
58
|
else
|
|
23
|
-
|
|
59
|
+
resolve_run_shard_items_glob(cwd: cwd, partition: partition)
|
|
24
60
|
end
|
|
25
61
|
end
|
|
62
|
+
|
|
63
|
+
def resolve_run_shard_items_glob(cwd:, partition: {})
|
|
64
|
+
suite = (partition["suite"] || partition[:suite] || "auto").to_s.downcase
|
|
65
|
+
suite = "auto" if suite.empty?
|
|
66
|
+
|
|
67
|
+
base = File.expand_path(cwd)
|
|
68
|
+
spec = Dir.glob(File.join(base, "spec/**/*_spec.rb")).sort
|
|
69
|
+
test = Dir.glob(File.join(base, "test/**/*_test.rb")).sort
|
|
70
|
+
quick = quick_parallel_default_paths(base)
|
|
71
|
+
|
|
72
|
+
case suite
|
|
73
|
+
when "rspec"
|
|
74
|
+
return {error: "partition.suite is rspec but no spec/**/*_spec.rb files"} if spec.empty?
|
|
75
|
+
|
|
76
|
+
{items: spec, source: "spec/**/*_spec.rb glob"}
|
|
77
|
+
when "minitest"
|
|
78
|
+
return {error: "partition.suite is minitest but no test/**/*_test.rb files"} if test.empty?
|
|
79
|
+
|
|
80
|
+
{items: test, source: "test/**/*_test.rb glob"}
|
|
81
|
+
when "quick"
|
|
82
|
+
return {error: "partition.suite is quick but no Polyrun quick files under spec/ or test/"} if quick.empty?
|
|
83
|
+
|
|
84
|
+
{items: quick, source: "Polyrun quick glob"}
|
|
85
|
+
when "auto"
|
|
86
|
+
if spec.any?
|
|
87
|
+
{items: spec, source: "spec/**/*_spec.rb glob"}
|
|
88
|
+
elsif test.any?
|
|
89
|
+
{items: test, source: "test/**/*_test.rb glob"}
|
|
90
|
+
elsif quick.any?
|
|
91
|
+
{items: quick, source: "Polyrun quick glob"}
|
|
92
|
+
else
|
|
93
|
+
{
|
|
94
|
+
error: "no spec paths (spec/spec_paths.txt, partition.paths_file, or spec/**/*_spec.rb); " \
|
|
95
|
+
"no test/**/*_test.rb; no Polyrun quick files"
|
|
96
|
+
}
|
|
97
|
+
end
|
|
98
|
+
else
|
|
99
|
+
{error: "unknown partition.suite: #{suite.inspect} (expected auto, rspec, minitest, quick)"}
|
|
100
|
+
end
|
|
101
|
+
end
|
|
102
|
+
|
|
103
|
+
def quick_parallel_default_paths(base)
|
|
104
|
+
require_relative "../quick/runner"
|
|
105
|
+
Polyrun::Quick::Runner.parallel_default_paths(base)
|
|
106
|
+
end
|
|
26
107
|
end
|
|
27
108
|
end
|
|
28
109
|
end
|
data/lib/polyrun/quick/runner.rb
CHANGED
|
@@ -62,6 +62,30 @@ module Polyrun
|
|
|
62
62
|
new(out: out, err: err, verbose: verbose).run(paths)
|
|
63
63
|
end
|
|
64
64
|
|
|
65
|
+
# Files Polyrun::Quick would run with no explicit paths (excludes normal RSpec/Minitest files).
|
|
66
|
+
def self.parallel_default_paths(cwd = Dir.pwd)
|
|
67
|
+
base = File.expand_path(cwd)
|
|
68
|
+
globs = [
|
|
69
|
+
File.join(base, "spec", "polyrun_quick", "**", "*.rb"),
|
|
70
|
+
File.join(base, "test", "polyrun_quick", "**", "*.rb"),
|
|
71
|
+
File.join(base, "spec", "**", "*.rb"),
|
|
72
|
+
File.join(base, "test", "**", "*.rb")
|
|
73
|
+
]
|
|
74
|
+
globs.flat_map { |g| Dir.glob(g) }.uniq.reject { |p| quick_path_excluded?(p, base) }.sort
|
|
75
|
+
end
|
|
76
|
+
|
|
77
|
+
def self.quick_path_excluded?(path, base)
|
|
78
|
+
rel = Pathname.new(path).relative_path_from(Pathname.new(base)).to_s
|
|
79
|
+
parts = rel.split(File::SEPARATOR)
|
|
80
|
+
bn = File.basename(path)
|
|
81
|
+
return true if bn.end_with?("_spec.rb", "_test.rb")
|
|
82
|
+
return true if %w[spec_helper.rb rails_helper.rb test_helper.rb].include?(bn)
|
|
83
|
+
return true if parts[0] == "spec" && %w[support fixtures factories].include?(parts[1])
|
|
84
|
+
return true if parts[0] == "test" && %w[support fixtures].include?(parts[1])
|
|
85
|
+
|
|
86
|
+
false
|
|
87
|
+
end
|
|
88
|
+
|
|
65
89
|
def initialize(out: $stdout, err: $stderr, verbose: false)
|
|
66
90
|
@out = out
|
|
67
91
|
@err = err
|
|
@@ -153,27 +177,12 @@ module Polyrun
|
|
|
153
177
|
end
|
|
154
178
|
|
|
155
179
|
def default_globs
|
|
156
|
-
|
|
157
|
-
globs = [
|
|
158
|
-
File.join(base, "spec", "polyrun_quick", "**", "*.rb"),
|
|
159
|
-
File.join(base, "test", "polyrun_quick", "**", "*.rb"),
|
|
160
|
-
File.join(base, "spec", "**", "*.rb"),
|
|
161
|
-
File.join(base, "test", "**", "*.rb")
|
|
162
|
-
]
|
|
163
|
-
globs.flat_map { |g| Dir.glob(g) }.uniq.reject { |p| default_quick_exclude?(p, base) }.sort
|
|
180
|
+
Runner.parallel_default_paths(Dir.pwd)
|
|
164
181
|
end
|
|
165
182
|
|
|
166
183
|
# Omit RSpec/Minitest files and common helpers so +polyrun quick+ with no args does not load normal suites.
|
|
167
184
|
def default_quick_exclude?(path, base)
|
|
168
|
-
|
|
169
|
-
parts = rel.split(File::SEPARATOR)
|
|
170
|
-
bn = File.basename(path)
|
|
171
|
-
return true if bn.end_with?("_spec.rb", "_test.rb")
|
|
172
|
-
return true if %w[spec_helper.rb rails_helper.rb test_helper.rb].include?(bn)
|
|
173
|
-
return true if parts[0] == "spec" && %w[support fixtures factories].include?(parts[1])
|
|
174
|
-
return true if parts[0] == "test" && %w[support fixtures].include?(parts[1])
|
|
175
|
-
|
|
176
|
-
false
|
|
185
|
+
Runner.quick_path_excluded?(path, base)
|
|
177
186
|
end
|
|
178
187
|
end
|
|
179
188
|
end
|
data/lib/polyrun/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: polyrun
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.2.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Andrei Makarov
|
|
@@ -167,10 +167,13 @@ files:
|
|
|
167
167
|
- lib/polyrun.rb
|
|
168
168
|
- lib/polyrun/cli.rb
|
|
169
169
|
- lib/polyrun/cli/ci_shard_run_command.rb
|
|
170
|
+
- lib/polyrun/cli/config_command.rb
|
|
170
171
|
- lib/polyrun/cli/coverage_commands.rb
|
|
171
172
|
- lib/polyrun/cli/coverage_merge_io.rb
|
|
172
173
|
- lib/polyrun/cli/database_commands.rb
|
|
174
|
+
- lib/polyrun/cli/default_run.rb
|
|
173
175
|
- lib/polyrun/cli/env_commands.rb
|
|
176
|
+
- lib/polyrun/cli/help.rb
|
|
174
177
|
- lib/polyrun/cli/helpers.rb
|
|
175
178
|
- lib/polyrun/cli/init_command.rb
|
|
176
179
|
- lib/polyrun/cli/plan_command.rb
|
|
@@ -187,6 +190,9 @@ files:
|
|
|
187
190
|
- lib/polyrun/cli/start_bootstrap.rb
|
|
188
191
|
- lib/polyrun/cli/timing_command.rb
|
|
189
192
|
- lib/polyrun/config.rb
|
|
193
|
+
- lib/polyrun/config/dotted_path.rb
|
|
194
|
+
- lib/polyrun/config/effective.rb
|
|
195
|
+
- lib/polyrun/config/resolver.rb
|
|
190
196
|
- lib/polyrun/coverage/cobertura_zero_lines.rb
|
|
191
197
|
- lib/polyrun/coverage/collector.rb
|
|
192
198
|
- lib/polyrun/coverage/collector_finish.rb
|