nukadoko 0.5.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +12 -0
- data/README.ja.md +2 -1
- data/README.md +1 -1
- data/lib/nukadoko/cli.rb +9 -3
- data/lib/nukadoko/judgment_file.rb +21 -0
- data/lib/nukadoko/judgments.rb +30 -0
- data/lib/nukadoko/run.rb +8 -0
- data/lib/nukadoko/summary.rb +4 -1
- data/lib/nukadoko/verdict.rb +18 -1
- data/lib/nukadoko/version.rb +1 -1
- data/llms.txt +1 -1
- data/sig/generated/nukadoko/judgment_file.rbs +6 -0
- data/sig/generated/nukadoko/judgments.rbs +10 -0
- data/sig/generated/nukadoko/verdict.rbs +8 -0
- data/skills/nukadoko/references/done.md +22 -0
- data/skills/nukadoko/references/judgments.md +6 -0
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 8e53cd82516de0c5338bbe7bf875e534dbdffec55bef983e17e0b4e5c42e7e5b
|
|
4
|
+
data.tar.gz: ee731ea548a72f78147c3f9a81b81728eb99b69040467c2ec00397c02137b799
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 4724272cb7bd2529912b710f5b0ce67e161a8cee08aafd82f849ef8d3e7b8d57e6ec9b715659889ca5dae623a85a7065b0c9c129dd0630c33654b620a25b5498
|
|
7
|
+
data.tar.gz: 1bddca443dbabb8b0d4b0f69b5e99245c54b5c2278323a7ccde3d5420ddcd1032ba67d789f571e81d4d164a6cd1e9220dfa608ec2f8c97675876d33bd15dd086
|
data/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,18 @@ notes come from the section below whose heading starts with `## [<version>]`.
|
|
|
6
6
|
|
|
7
7
|
## [Unreleased]
|
|
8
8
|
|
|
9
|
+
## [0.6.0] - 2026-11-17
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- `nuka verdict --trailer` prints one git trailer line,
|
|
14
|
+
`Nukadoko-Verdict: <passed|failed|stale|none|unknown> run <id> tree <id>`,
|
|
15
|
+
to stamp a commit, and exits 0 whatever it says. `--json` carries the
|
|
16
|
+
same line as `trailer`. The same as the TypeScript engine.
|
|
17
|
+
- Under `--base`, each judgment a read cites carries `in_base`, and a row
|
|
18
|
+
that stands on a judgment the change added says so in its `judgment`
|
|
19
|
+
limit. The same as the TypeScript engine.
|
|
20
|
+
|
|
9
21
|
## [0.5.0] - 2026-11-16
|
|
10
22
|
|
|
11
23
|
### Added
|
data/README.ja.md
CHANGED
|
@@ -68,8 +68,9 @@ seed が同じなら、TypeScript 版と同じ順番です。
|
|
|
68
68
|
`--concurrency` は、feature ファイルを丸ごと、その数の worker プロセスに分けて実行します。
|
|
69
69
|
Feature の行に `@nukadoko:serial` があるファイルは、ほかのファイルのあとに 1 つずつ実行します。
|
|
70
70
|
`--repeat` は選んだシナリオをその回数だけ実行し、`--proof-runs` は control を何回実行するかを決め、`--records` はシナリオの記録を終わるたびに stdout に書きます。
|
|
71
|
-
- `nuka verdict [--json]` は、いちばん新しい run が今の作業ツリーを判定し、赤を残さなかったかを答えます。
|
|
71
|
+
- `nuka verdict [--json] [--trailer]` は、いちばん新しい run が今の作業ツリーを判定し、赤を残さなかったかを答えます。
|
|
72
72
|
run は始めません。
|
|
73
|
+
`--trailer` は、その答えでコミットに印を付ける git の trailer を 1 行出し、終了コード 0 で終わります。
|
|
73
74
|
- `nuka judge <feature>:<line> [--run <id>] [--json]` は、Then が判断を求める基準(`judged-as` の claim)について、run が読んだ値を見せます。
|
|
74
75
|
`--meets` か `--misses` に `--by <who>` と `--because <一文>` を付けると、その値の判断を `judgments/<feature>.json` に記録します。
|
|
75
76
|
次の run は、この記録と照らして基準を読みます。
|
data/README.md
CHANGED
|
@@ -50,7 +50,7 @@ end
|
|
|
50
50
|
- `nuka init` writes `nukadoko.config.rb`, the empty `features/steps/`, and the `.gitignore` entries for `.nukadoko/` and `.env` files. A Rakefile with no `scenarios` task gets one that runs `bundle exec nuka run`, so people and CI run `rake scenarios`. It refuses when the config file exists.
|
|
51
51
|
- `nuka check [--json]` reports what is wrong before anything runs.
|
|
52
52
|
- `nuka run [<feature[:line]|dir>...] [--tag <expression>]... [--env <name>]... [--order file|shuffled[:<seed>]] [--repeat <n>] [--concurrency <n>] [--proof-runs <n>] [--step] [--records] [--json] [--no-check]` starts the program in each selected world, runs the selected scenarios, and writes a record per step under `.nukadoko/runs/`. With no target it runs `features/`. `--order shuffled` plays the files, then each file's scenarios, in an order a seed names, the same order the TypeScript engine plays for that seed. `--repeat` plays the selection that many times, `--concurrency` spreads whole feature files over that many worker processes (a file tagged `@nukadoko:serial` on its Feature line runs alone, after them), `--proof-runs` sets how many times each control runs, and `--records` writes each scenario record to stdout as it ends.
|
|
53
|
-
- `nuka verdict [--json]` says whether the newest run judged the working tree as it is now and left nothing red. It never starts a run.
|
|
53
|
+
- `nuka verdict [--json] [--trailer]` says whether the newest run judged the working tree as it is now and left nothing red. It never starts a run. `--trailer` prints one git trailer line that stamps a commit with that answer, and exits 0.
|
|
54
54
|
- `nuka judge <feature>:<line> [--run <id>] [--json]` shows the values a run read for a criterion whose Then asks for a judgment (a `judged-as` claim). With `--meets` or `--misses`, `--by <who>`, and `--because <one sentence>` it records a judgment of them in `judgments/<feature>.json`, which the next run reads the criterion against.
|
|
55
55
|
- `nuka report [run] --format markdown|messages|allure [--out <path>] [--json]` renders a run again from its run directory alone, the newest run when none is named. `markdown` is its `summary.md`. `messages` is a cucumber-messages stream, one envelope per line, so a formatter or a JUnit step that reads cucumber's protocol keeps working: a claim step's claims and each criterion's verdict ride as attachments. `allure` is a directory of Allure results, `.nukadoko/export/allure-results` unless `--out` names another: one result per execution, each claim a child step of its Then, with the history id the TypeScript engine's results of the same scenario carry.
|
|
56
56
|
- `nuka run <feature>:<line> --step` walks one scenario a step at a time: after each step it waits until `nuka step` answers. `nuka step wait [--walk <n>] [--timeout <s>] [--json]` prints where the walk stands: the step it paused after, its record, and a screenshot and an ARIA snapshot of the page it left open. `nuka step next <pause> --walk <n>` runs the next step, and `nuka step stop <pause> --walk <n>` skips the rest, which leaves the scenario stopped and its row red. Either engine's `nuka step` answers either engine's walk.
|
data/lib/nukadoko/cli.rb
CHANGED
|
@@ -5,7 +5,7 @@
|
|
|
5
5
|
# TypeScript engine prints, so an agent reads both engines the same way.
|
|
6
6
|
module Nukadoko
|
|
7
7
|
class CLI
|
|
8
|
-
FLAGS = %w[--json --no-check --dry-run --then --replace --allure --records --meets --misses --step].freeze
|
|
8
|
+
FLAGS = %w[--json --no-check --dry-run --then --replace --allure --records --meets --misses --step --trailer].freeze
|
|
9
9
|
REPEATED = %w[--tag --env --use].freeze
|
|
10
10
|
VALUED = %w[--format --out --base --repeat --concurrency --order --proof-runs --args --pattern --description --feature --command --request --from --cwd --body --argv
|
|
11
11
|
--run --by --because --walk --timeout].freeze
|
|
@@ -44,7 +44,8 @@ module Nukadoko
|
|
|
44
44
|
alone
|
|
45
45
|
nuka verdict say whether the newest run judged the working tree as
|
|
46
46
|
it is now and left nothing red; exits 1 when it did
|
|
47
|
-
not, and never starts a run
|
|
47
|
+
not, and never starts a run; --trailer prints one git
|
|
48
|
+
trailer line to stamp a commit, and exits 0
|
|
48
49
|
nuka judge <target> show the values a run read for a criterion whose Then
|
|
49
50
|
asks for a judgment (<feature file>:<scenario line>),
|
|
50
51
|
or record a judgment of them with --meets or --misses
|
|
@@ -761,13 +762,18 @@ module Nukadoko
|
|
|
761
762
|
#: () -> Integer
|
|
762
763
|
def verdict
|
|
763
764
|
answer = Engine::Verdict.new(@project)
|
|
765
|
+
# --trailer observes: a stamp never blocks, so only the exit code
|
|
766
|
+
# moves, and the envelope still says what the verdict was.
|
|
767
|
+
trailer = @options.key?("--trailer")
|
|
764
768
|
if @json
|
|
765
769
|
emit({ **head("verdict", answer.open? ? "failed" : "passed", answer.findings), **answer.envelope_fields }, answer.findings, "verdict")
|
|
770
|
+
elsif trailer
|
|
771
|
+
@out.puts answer.trailer
|
|
766
772
|
else
|
|
767
773
|
@out.puts answer.headline
|
|
768
774
|
emit({}, answer.findings, "verdict")
|
|
769
775
|
end
|
|
770
|
-
answer.open? ? 1 : 0
|
|
776
|
+
answer.open? && !trailer ? 1 : 0
|
|
771
777
|
end
|
|
772
778
|
|
|
773
779
|
# What check says about the env files; a file it cannot read, or a key
|
|
@@ -73,6 +73,27 @@ module Nukadoko
|
|
|
73
73
|
name.delete_prefix(prefix.empty? ? "" : "#{prefix}/") if now != before&.b
|
|
74
74
|
end.sort
|
|
75
75
|
end
|
|
76
|
+
|
|
77
|
+
# What `commit` holds as the judgments of a feature file: its
|
|
78
|
+
# entries, none when it holds no such file, or why the file did not
|
|
79
|
+
# parse. nil when git does not answer.
|
|
80
|
+
#: (String, String, String) -> Hash[Symbol, untyped]?
|
|
81
|
+
def at_commit(project, commit, feature)
|
|
82
|
+
prefix, ok = Base.git(project, "rev-parse", "--show-prefix")
|
|
83
|
+
return nil unless ok
|
|
84
|
+
|
|
85
|
+
prefix = prefix.delete_suffix("/")
|
|
86
|
+
name = prefix.empty? ? of(feature) : "#{prefix}/#{of(feature)}"
|
|
87
|
+
top, ok = Base.git(project, "rev-parse", "--show-toplevel")
|
|
88
|
+
return nil unless ok
|
|
89
|
+
|
|
90
|
+
listed, ok = Base.git(top, "ls-tree", "-z", "--name-only", commit, "--", name, strip: false)
|
|
91
|
+
return nil unless ok
|
|
92
|
+
return { entries: [] } unless listed.split("\0").include?(name)
|
|
93
|
+
|
|
94
|
+
text, ok = Base.git(top, "show", "#{commit}:#{name}", strip: false)
|
|
95
|
+
ok ? Judgments.parse_file(text.dup.force_encoding("UTF-8")) : nil
|
|
96
|
+
end
|
|
76
97
|
end
|
|
77
98
|
end
|
|
78
99
|
end
|
data/lib/nukadoko/judgments.rb
CHANGED
|
@@ -184,6 +184,36 @@ module Nukadoko
|
|
|
184
184
|
end
|
|
185
185
|
end
|
|
186
186
|
|
|
187
|
+
# Under --base, each judgment a read cites gains `in_base`: true when
|
|
188
|
+
# the base's file already held that very judgment (the same claim,
|
|
189
|
+
# value, verdict, judge, reason, and time), false when the change
|
|
190
|
+
# brought it. A change cannot be trusted to judge its own output any
|
|
191
|
+
# more than to rewrite its own Thens, and the old world covers only
|
|
192
|
+
# the Thens. `base_entries_of` answers nil for a feature whose base
|
|
193
|
+
# file could not be read, and its reads stay unmarked.
|
|
194
|
+
#: (Array[Hash[Symbol, untyped]], ^(String) -> Array[Hash[Symbol, untyped]]?) -> Array[Hash[Symbol, untyped]]
|
|
195
|
+
def with_base_judgments(rows, base_entries_of)
|
|
196
|
+
rows.map do |row|
|
|
197
|
+
judgment = row[:judgment]
|
|
198
|
+
next row unless judgment && judgment[:reads].any? { _1[:judgment] }
|
|
199
|
+
|
|
200
|
+
entries = base_entries_of.(row[:feature])
|
|
201
|
+
next row unless entries
|
|
202
|
+
|
|
203
|
+
reads = judgment[:reads].map do |read|
|
|
204
|
+
cited = read[:judgment]
|
|
205
|
+
next read unless cited
|
|
206
|
+
|
|
207
|
+
held = entries.any? do |entry|
|
|
208
|
+
entry[:criterion] == row[:name] && entry[:then] == read[:step] && entry[:claim] == read[:claim] &&
|
|
209
|
+
%i[value_digest verdict by because at].all? { entry[_1] == cited[_1] }
|
|
210
|
+
end
|
|
211
|
+
{ **read, judgment: { **cited, in_base: held } }
|
|
212
|
+
end
|
|
213
|
+
{ **row, judgment: { **judgment, reads: reads } }
|
|
214
|
+
end
|
|
215
|
+
end
|
|
216
|
+
|
|
187
217
|
#: (Array[Hash[Symbol, untyped]]) -> bool
|
|
188
218
|
def open?(rows) = rows.any? { _1[:judgment] && _1[:judgment][:state] != "judged" }
|
|
189
219
|
|
data/lib/nukadoko/run.rb
CHANGED
|
@@ -360,6 +360,14 @@ module Nukadoko
|
|
|
360
360
|
if @base && changed.nil?
|
|
361
361
|
warnings << { severity: "warning", code: "judgments-not-compared", message: "the judgment files were not compared with the base: git did not answer" }
|
|
362
362
|
end
|
|
363
|
+
# And per row, which judgment the change brought: a row that stands
|
|
364
|
+
# on a judgment the change made of its own output.
|
|
365
|
+
if (base = @base) && changed
|
|
366
|
+
at_base = {} #: Hash[String, Array[Hash[Symbol, untyped]]?]
|
|
367
|
+
rows = Judgments.with_base_judgments(rows, lambda do |feature|
|
|
368
|
+
at_base.fetch(feature) { at_base[feature] = JudgmentFile.at_commit(@project, base.commit, feature)&.dig(:entries) }
|
|
369
|
+
end)
|
|
370
|
+
end
|
|
363
371
|
judged += Judgments.findings(rows, @id) + problems
|
|
364
372
|
{ run_id: @id, run_dir: File.join(".nukadoko/runs", @id), started_at: @started_at, finished_at: now, commit: @commit,
|
|
365
373
|
environments: @environments.map { environment_entry(_1) }, worlds: @world_entries, order: @order,
|
data/lib/nukadoko/summary.rb
CHANGED
|
@@ -451,8 +451,11 @@ module Nukadoko
|
|
|
451
451
|
|
|
452
452
|
by = judgment[:reads].map { _1.dig(:judgment, :by) || "" }.uniq.map { code_span(_1) }.join(", ")
|
|
453
453
|
aria = judgment[:reads].any? { _1[:measured] == "aria-snapshot" } ? "; it is bound to the ARIA snapshot, so a change only to how the page looks keeps it current" : ""
|
|
454
|
+
# Under --base, a judgment the change itself brought is the change
|
|
455
|
+
# judging its own output, which the reviewer reads first.
|
|
456
|
+
added = judgment[:reads].any? { _1.dig(:judgment, :in_base) == false } ? " that this change added" : ""
|
|
454
457
|
{ kind: "judgment", feature: row[:feature], line: row[:line], text: row[:name],
|
|
455
|
-
detail: "stands on a judgment by #{by}: nukadoko checked that the value is the one judged, not the judgment#{aria}" }
|
|
458
|
+
detail: "stands on a judgment by #{by}#{added}: nukadoko checked that the value is the one judged, not the judgment#{aria}" }
|
|
456
459
|
end
|
|
457
460
|
end
|
|
458
461
|
|
data/lib/nukadoko/verdict.rb
CHANGED
|
@@ -17,6 +17,8 @@ module Nukadoko
|
|
|
17
17
|
"working-tree-unknown" => "No change is necessary. In a git repository with a commit, nuka verdict compares the tree with the one the run judged."
|
|
18
18
|
}.freeze
|
|
19
19
|
|
|
20
|
+
TRAILER_KEY = "Nukadoko-Verdict" #: String
|
|
21
|
+
|
|
20
22
|
attr_reader :judged #: String
|
|
21
23
|
attr_reader :findings #: Array[Hash[Symbol, untyped]]
|
|
22
24
|
attr_reader :newest #: Hash[Symbol, untyped]?
|
|
@@ -51,7 +53,22 @@ module Nukadoko
|
|
|
51
53
|
#: () -> Hash[Symbol, untyped]
|
|
52
54
|
def envelope_fields
|
|
53
55
|
{ judged: judged, working_tree: working_tree, run_id: newest&.fetch(:run_id), run_dir: newest&.fetch(:run_dir, nil),
|
|
54
|
-
run_working_tree: newest && recorded, base: newest&.dig(:table, :base) }
|
|
56
|
+
run_working_tree: newest && recorded, base: newest&.dig(:table, :base), trailer: trailer }
|
|
57
|
+
end
|
|
58
|
+
|
|
59
|
+
# One git trailer line that stamps a commit with this answer. The tree
|
|
60
|
+
# named is the one the run recorded, so a reader compares it with the
|
|
61
|
+
# commit's own tree later and knows whether the stamp speaks for it.
|
|
62
|
+
#: () -> String
|
|
63
|
+
def trailer
|
|
64
|
+
run = newest or return "#{TRAILER_KEY}: none"
|
|
65
|
+
word = case judged
|
|
66
|
+
when "stale" then "stale"
|
|
67
|
+
when "unknown" then "unknown"
|
|
68
|
+
else open? ? "failed" : "passed"
|
|
69
|
+
end
|
|
70
|
+
tree = recorded
|
|
71
|
+
"#{TRAILER_KEY}: #{word} run #{run[:run_id]}#{" tree #{tree}" if tree}"
|
|
55
72
|
end
|
|
56
73
|
|
|
57
74
|
#: () -> String
|
data/lib/nukadoko/version.rb
CHANGED
data/llms.txt
CHANGED
|
@@ -51,7 +51,7 @@ Run them in this order. Exit 0 is the only pass.
|
|
|
51
51
|
- `nuka check [feature..] [--json] [--fail-on <code>] [--env <name>]...`: static checks for the files `nuka run` would select (a file, several, a directory, or every feature with no target), and the program's placeholders for the worlds a `nuka run` with the same `--env` flags would start. Findings are errors, warnings, and notes; a note is rot rather than breakage (an unbound pattern, an unused `from` or fixture, a field with no `.describe()`), and only a check with no target reports notes, because they are facts about every feature the project scans. Exit 1 on any error, or on a code `--fail-on` names (one value per flag, repeatable, or comma-separated; the finding keeps its severity, and `--json` lists the hits in `fail_on_hits`; a note code with targets, which compute no notes, is refused with `fail-on-note-with-targets`); warnings and notes alone exit 0. `nuka check --codes` prints every code `check`, `run`, and a refusal can carry; the table under "Findings" is that list. Condition tags are read against the environments `--env` names, as `nuka run --env` does, or without it the ones a run without `--env` selects: `condition-tag-malformed`, `condition-unknown` (a condition no environment declares), and `condition-unmet` (a scenario no selected environment meets) are errors.
|
|
52
52
|
- `nuka run [feature..] [--no-check] [--env <name>]... [--tag <expression>]... [--concurrency <n>] [--repeat <n>] [--order file|shuffled[:<seed>]] [--proof-runs <n>] [--base <git ref>] [--step] [--records] [--json]`: runs `check` first for the selected scenarios and refuses on an error (`--no-check` runs anyway). `<file>:<line>` runs one scenario. Progress, failures, and the paths it wrote go to stderr (none of them under `--json`); each step's line says `passed`, `failed`, or `skipped`, the word its step record uses. stdout stays empty unless asked: `--records` writes each scenario's `run.json` entry there as one JSON line when it finishes (a step's `program_log_late` is only in `run.json`, written when the world stops), and `--json` prints `run.json` itself, which is the envelope: `run_dir` (the run directory, relative to the project root, the argument `nuka report` takes), `commit` (`sha`, `dirty`, and `working_tree`, the working tree as one git tree id: tracked files as on disk and untracked files `.gitignore` does not leave out, the state directory left out; absent when git gives none), `environments` (each `name`, `target_version`, `base_url`, its world's URL or `null`, and `conditions` when it declares them), `worlds` (each world the run started, with `name`, `tree`, `url`, `commands`, `timings`, `log`, and `error`; empty with no program), `selection` (`partial` when a `:line` target left out a scenario of its file), `order` (`{ kind: "file" }`, or `{ kind: "shuffled", seed }` under `--order shuffled`), `counts` (`scenarios`, `passed`, `failed`, and `stopped` (a walk stopped them, `--step`) of the executions of required cells, `not_run` required cells, `controls` of the executions of controls, and `controls_not_run` control cells), `table` (the verdict table: `base` (`{ ref, commit }` under `--base`, else `null`), `mapping_changed` and `mapping_changes` (`files`, `config`, `conditions`, `default_environments`, `unseen`; `null` without `--base`), `worlds`, its columns `{ world, environment }` with `world` of `head`, `base`, or `old`, `left_out`, the selected criteria `--env` left out, `proof_runs`, the `--proof-runs` a control's `runs` is measured against, and `rows`, one per selected criterion and then one per removed one, with `feature`, `line`, `name`, `change` (`null`, or under `--base` `unchanged`, `changed`, `new`, or `removed`), `renamed_from` (`{ feature, name }` at the base), `retagged` (`true` for an unchanged row whose tags differ from the base's), `conditions` (its condition tags and `@held`), `cells` (each `world`, `environment`, `role` of `required` or `control`, `conditions` (each condition its row's tags name, with the value its environment declares, `null` where none), `outcome` of `pass`, `fail`, `broke`, `stopped`, `flaky`, or `not-run`, the `line` of every one but `pass` and `not-run`, `runs`, for `fail` the `claim` every run broke and `actuals`, each run's actual value, and a `reason` for `not-run`, for a `broke` a hook caused, and for a control `broke` whose proof round could not finish), `verdict` of `held`, `proven`, `removed`, `failed`, `unproven`, or `moved`, `reason`, one sentence, and for a removed row `lines`, its sentences at the base), `flags` (only when a head step's execution events name a flag or a head world's `lists.flags` list declares one: one row per `name` and `side`, a declared side no criterion evaluated included with no `criteria`, each with `listed_in` (each environment whose list declares the side, empty for one no list declares) whenever a flags list was read, and `only_side_evaluated: true` on a flag's one side when the criteria evaluated no other, the side as its lines name it and never as an environment's conditions declare it, with `criteria`, each criterion that evaluated it there with its row's `verdict` and its `cells` on that side (`environment`, `role`, `outcome`, `line`, and `gates`, each gate its lines named or `null`), and `counts` by verdict, where a criterion seen on the side only in controls counts as `controls`; evidence to join with the program's own record of the flag by `name` and `side`, not a verdict on the flag; the summary gives each row one line under "Flags"), `routes` (only when a head world read a `lists.routes` list: one entry per `method` and `route` a list or a head step's `request` line names, `null` for the requests the program matched to no route, ordered by route and then method, with `handlers`, `listed_in` (each environment whose list holds it; empty for a route no list names), `requests` (every `request` line of a head step naming it, a late one included, each line in exactly one entry; 0 for one no step requested), `statuses`, and `criteria` and `counts` by the rules of `flags`, each cell with its `requests` and `statuses`; a request counts the same in a Given as in a Then and says the program matched the route, never that a claim read its answer; the summary's "Routes" names the listed routes no step requested, apart from those whose handler a request reached through another route (PUT and PATCH of one resource), the requests no list names, and each world whose list was not read), `paths` (only when a pair of sides read execution events: under `--base` each environment's base world against its head world, `between: { kind: "base" }`, base pairs first, for the rows the base ran there, which, when both programs name `{events}`, is every row whose head cell there is required and ran (head's `{events}` alone builds the base world for this even when no criterion is new or changed, so a refactor is compared), and two environments whose `conditions` differ in exactly one, `between: { kind: "condition", name, a, b }`; one entry per pair, with the sides `a` and `b` (`world`, `environment`), side `a` being the base world, or the environment where the condition is `false` or not declared, `unread` naming why nothing was compared or `null`, `counts` of `changed`, `open`, `same_so_far` (open, with every open count one range on both sides under the same gap words), `marked_only` (the only differences are on tables marked as bookkeeping or infrastructure; a criterion's `result` reads every other difference), `same`, and `not_compared`, `criteria` with each criterion's `result`, under a base pair `explained_by` on a `changed` one (`criterion_change` when it is new or changed, `null` when it is unchanged and nothing in the feature files asked for the difference), its side executions `a` and `b` (`scenario_record_id`, the cell's `outcome` and `line`), and `steps`, the steps with a difference aligned by line: `ran: "both"` with `changed` and `open` entries, each carrying the range of its count on side `a` and side `b`, and `holds` on a marked table, and `gaps`, or `ran: "a"` or `"b"` for a step only that side ran; a count a side's missing lines could still reach is `open`, never `changed`, a flag's side is the premise of a pair of environments and never compared there, while against the base a flag whose side changed is a difference, and `not_compared` names each criterion left out with a `reason`, such as `did not run at the base`; `unread` says when the base's program names no `{events}`; evidence, not a verdict; the summary lists up to 20 criteria per heading under "Changed paths", every base pair under one `base → head` heading that lists a criterion with the same lines in several environments once, counts the unchanged criteria compared (`Each of the 2 unchanged criteria compared did the same at the base and at head.` when each did), and lists the unchanged criteria that changed first, then the new or changed ones, then those that may differ, each group under a line of its own, and then, under `Bookkeeping and infrastructure writes that differ, which no sentence is about, though another reader of those tables may be:`, the differences with `holds`, which never rank a criterion), `limits` (what the run does not show, computed from the other fields of run.json: `left-out`, `check-skipped`, `control-not-proving` for a held row where no control failed at a Then, `no-claim`, `unmeasured`, `after-record`, `unrecorded-log`, and `unrecorded-events`, each with a `detail` and, where one exists, a `do` or `fix`; the summary renders it under `What this run does not show` right after the table, and stderr ends with one line of counts by kind, `limits: <n> <kind>, ... (run.json limits)`), and per scenario execution its `environment`, `role`, under `--base` the `world` and `row` of a base or old execution, and `for_paths` (`outcome`, and `line` when it did not pass) on a base execution run for `paths` alone, which is no cell, gets no proof runs, never changes the table, and whose stderr line ends `(for paths)`, and steps (each with `line`, `text`, `step`, `status`, `step_record_id`, `record`, `record_unreadable` with a `step-record-unreadable` warning when its step record could not be read back, a claim step's `claims`, `observation`, and `attempts`, and for a failure `error` with `kind` and `message` and the `used` results it read, a result over 4096 bytes of JSON cut to a preview with `truncated` and `result_bytes`), a failed scenario's `do` (a control has none), and every finding the run raised. `run.json` is written to `.nukadoko/runs/<run_id>/run.json` either way, and every path inside it (`record`, `evidence.dir`) is relative to that run directory; there is no scenario record file. The Markdown run summary, rendered from `run.json` alone, is always written beside it and appended to `$GITHUB_STEP_SUMMARY` when that is set; the run writes no Allure results or messages stream. Each failed scenario is a `scenario-failed` finding on stderr whose ` do:` line is the `nuka run <file>:<line>` that reruns it (under `--order shuffled`, which runs the feature files and then each file's scenarios in the order a seed names, with the seed on stderr before the first scenario, in `order`, and in the summary, the line names the scenarios that ran before it in its pass, then it, with `--order shuffled:<seed>`, or is the run itself under `--repeat` or `--concurrency` above 1), and a scenario a walk stopped is a `walk-stopped` one. `--step` walks one scenario with a person: see `nuka step` below. `--env` repeats: `--env a --env b` is one run with one `run_id` that runs every selected scenario in `a`, then in `b`; each environment gets its own env files, secrets, version probe, world, BeforeAll and AfterAll, and process-scope fixtures. A run without `--env` runs the environments the program's `defaultEnvironments` names, or else every environment that declares `conditions`, in the order the file lists them, and `default` when none does; any other, such as a deployment target, runs when `--env` names it. Under `--env`, a scenario no named environment meets is not run: its row is left out of the table, counted in `table.left_out`, in one line on stderr (`<n> criteria not run: their conditions hold in no selected environment`), and as the `left-out` entry of `limits`, and `nuka check` reports `condition-unmet` for the default selection only. `--env default` names the top-level environment (as run.json names it), whether or not the program defines it, so it combines with named ones. The tags `@with:<condition>`, `@with:<condition>=<value>` (exact text; `=true` and `=false` are malformed, write `@with:` or `@without:`), and `@without:<condition>` (off: `false` or not declared) on a Feature, Rule, Scenario, or Examples line make a scenario's cell `required` in an environment that meets them all and a `control` elsewhere; a scenario with none is required everywhere, and `@held` says the base already has the behaviour. `@expects:status=<500-599>`, `@expects:page-error=<name>`, and `@expects:error=<class>` on the same lines accept one failure underneath the scenario: a passed execution whose step got a 5xx response, whose page threw an error nothing caught, or whose trace holds an `error` with `handled: false`, is `broke@<that step's line>` unless a tag names that failure (`unexpected-failure` names it and spells the tag; `expects-unseen` notes a tag that matched nothing); only a pass moves, and like condition tags they are content under `--base`. A cell is `pass`, `fail@<line>` (each execution broke a claim at that line in Then position), `broke@<line>` (each stopped there before a claim was judged: a Given or When error, an undefined line, an unjudged claim; or, at the scenario's line, a hook, which the cell's `reason` names with when it ran), `flaky@<line>` (its executions disagree), `stopped@<line>` (a walk was stopped after the step at that line with `nuka step stop`), or `·` (none ran, with the `reason`). A required cell runs once, or `--repeat` times; a control that failed at a Then line runs again in the same world until it has broken the same claim there `--proof-runs` times (default 2) or a run disagrees (runs agree on the claim, by its position in the step, not on its value, which may differ), each a `proof run <k>/<n>` line on stderr; under `--concurrency` a proof round runs in a worker of its own, which runs BeforeAll and AfterAll again, and a hook that fails there, or a worker that records nothing, makes the round's control cells `broke` with the reason and a `proof-run-incomplete` warning, and the row reads as one no control proves (under `--base`, the base world runs for a new or changed row). A row is `failed` when a required cell is not `pass` or it has no required cell (only under `--no-check` without `--env`, with a `criterion-without-required-cell` finding whose `do:` runs the scenario where it is met, or `nuka check` on its file), `proven` when every required cell passed and a control failed in `--proof-runs` runs, and `held` otherwise. The exit code is 1 when a row is `failed`, `unproven`, or `moved`, or when the run failed outside every scenario: a BeforeAll or AfterAll of a pass failed (at any `--concurrency`), a head world did not start (`program-start-failed`), or a `--concurrency` worker exited before each of its scenarios had a record (`run-incomplete`); a control never makes it red. A `held` row whose control passed says the Then does not see what the switch changes, or the switch does not reach it: read it before calling the criterion covered. The stderr count line counts required cells alone, and a run with a control adds a `<n> control runs, <k> not run` line and a `criteria: <held> held, <proven> proven, <failed> failed` line under it, which every run under `--base` has, with `<unproven> unproven, <moved> moved, <removed> removed` after them. The summary opens with the table, one row per criterion and one `head <environment>` column per environment, headed with the conditions it declares (and `base <environment>` and `old <environment>` under `--base`), then one line per red row with its reason and where it stopped; a failure's `do:` names its environment. Naming one environment twice is refused. `--base <git ref>` compares each criterion with the commit the ref names: its identity is its feature file and name, and its content its steps with their effective keyword, docstrings, and tables, its Examples row, and its condition tags and `@held`, so a rename, a move, or another tag leaves it `unchanged` (with `renamed_from` when its name or file differs, and `retagged` when its tags differ) and an edit makes it `changed`; the rest are `new` or `removed`. The mapping (files under the features directory other than `.feature` files, the whole config but `stateDir` and `retention`, by its value and by the config file's own bytes, so a constant a config function closes over counts and an edit to those two fields alone does too, and each environment's `conditions` and `defaultEnvironments`) is compared as one unit. A `new` or `changed` row that passed at head also runs in the `base` world (the base's code under head's criteria and mapping, built from the base's program in `trees/base` under the run directory), where a base cell proves like a control after `--proof-runs` failures at one Then line; it is `proven` then, `unproven` (red) when nothing proves it (a compat Then never can, and the reason says so), and a row tagged `@held` is `held` when the base passes it. An `unchanged` row that passed at head runs in the `old` world (head's code under the base's feature files and mapping, in `trees/old`) when the mapping changed or the row is retagged or renamed, and is `moved` (red) when it does not pass there, or else `proven` when a control proves it, as without `--base`, and `held` when none does. Each `unproven` or `moved` row is a `criterion-unproven` or `criterion-moved` finding whose `do:` is `nuka run <file>:<line> --base <commit>`, naming the full hash the ref resolved to, which judges it again against the same commit after the ref moves. `--base` does not support yet a symbolic link at, above, or under the features directory or at the config file, a feature file outside the features directory, a submodule or a nested repository, or, for the base world alone, a workspace package linked into `node_modules` from inside the repository: the world such a shape reaches is not built, its cells are `·` with the reason, and a red row's finding has a `do:` that runs the row without `--base` and a `fix:` when an edit lets `--base` build the world; a shape that hides part of the mapping is named in `mapping_changes.unseen` and makes `mapping_changed` true. A criterion only the base holds is a `removed` row, quoted under the summary's "Removed criteria". The trees install nothing and resolve packages from head's project; a `url` function receives `tree` (`"base"` or `"head"`) and may return nothing for the base, and one that returns head's URL for the base leaves the base world not run, so it has to branch on `tree`. A tree whose cells do not run says why on stderr, `<world> <environment> not run: <reason>`. The summary opens with a line naming the base and what changed. A project outside git is refused with `base-outside-git`, a ref that names no commit with `base-ref-unknown`, a shallow clone without it with `base-shallow` (its `do` is the `git fetch`), a git failure with `base-git-failed`, and a `featuresDir` at the project root with `base-features-at-root`; `--since` is refused with `removed-flag`.
|
|
53
53
|
- `nuka report [<run dir>] --format markdown|allure|messages [--out <path>] [--json]`: renders one finished run from its run directory alone, so a run directory copied off a CI machine renders too: `markdown` reads `run.json` alone, and `allure` and `messages` read `run.json`, the step records, and `sources.json`, the feature text the run parsed. With no run dir it renders the newest run. `markdown` is the run summary, on stdout or to `--out` (the file a CI step posts as a pull request comment); `allure` writes Allure 2 format results to `.nukadoko/export/allure-results/` or the directory `--out` names; `messages` is a cucumber-messages NDJSON stream on stdout or to `--out`, where a run over several environments gives each environment its own pickle, named `<scenario> (<environment>)`, and every execution of a control is left out, so a JUnit report of a green run is green; Allure keeps each control with the status it ran to, named `<scenario> (control)` with a `role` label, so a control that failed is a failed result there by design, and a CI gates on `nuka run`'s exit code. A file written under `.nukadoko/export/` is noted in the run's exports manifest, so retention removes it with the run, and rendering the same run there again replaces it; a file written anywhere else is not noted, and stays unless it sits inside a run directory that is removed. `--json` prints the envelope with `run_id`, `format`, and `written`, and needs `--out` for `markdown` and `messages`. A directory with no `run.json` is refused with `run-not-found`, and one whose `run.json` this release does not read with `run-invalid`; for `allure` and `messages`, one with no `sources.json` (a run from an earlier release) is refused with `run-sources-missing`, and one whose `sources.json` is corrupt or of another version with `run-sources-invalid`. A missing directory on the way to `--out` is created; anything else in the way (a directory where a file goes, a file where the Allure directory goes, a FIFO, a dangling symbolic link, a path under a regular file, or `allure-history.jsonl` in the export directory in any letter case) is refused with `report-out-invalid` before the run is read, and so is a markdown or messages file the file system refuses to write, with its error code; an Allure directory the file system refuses to write into ends as `report-incomplete`. A render that left out part of the run (a scenario, a step record, or an attachment it could not read) writes the rest and exits 1 with `report-incomplete`; a step whose step record is gone is never shown passed.
|
|
54
|
-
- `nuka verdict [--json]`: before you say a change is done. It reads the newest run that ended and never starts one, and exits 1 when no run ended (`run-missing`), the newest run judged another working tree (`run-stale`: a file git counts changed after it started), it ran part of its files or was a walk (`run-partial`), or it left something red (its own failed findings, relayed as it printed them). A pass says that run judged exactly this tree and left nothing red; it does not say the change has a criterion of its own. Notes: `run-without-base`, `criteria-unchanged` (under `--base`, no criterion is new or changed), `working-tree-unknown` (outside git or before the first commit, where the verdict is the run's own). `--json` adds `judged` (`missing`, `stale`, `judged`, `unknown`), `working_tree`, `run_id`, `run_dir`, `run_working_tree`, and `
|
|
54
|
+
- `nuka verdict [--json] [--trailer]`: before you say a change is done. It reads the newest run that ended and never starts one, and exits 1 when no run ended (`run-missing`), the newest run judged another working tree (`run-stale`: a file git counts changed after it started), it ran part of its files or was a walk (`run-partial`), or it left something red (its own failed findings, relayed as it printed them). A pass says that run judged exactly this tree and left nothing red; it does not say the change has a criterion of its own. Notes: `run-without-base`, `criteria-unchanged` (under `--base`, no criterion is new or changed), `working-tree-unknown` (outside git or before the first commit, where the verdict is the run's own). `--json` adds `judged` (`missing`, `stale`, `judged`, `unknown`), `working_tree`, `run_id`, `run_dir`, `run_working_tree`, `base`, and `trailer`. `--trailer` prints only that trailer, one git trailer line that stamps a commit (`Nukadoko-Verdict: passed run <run_id> tree <tree id>`; `failed`, `stale`, `none` when no run ended, or `unknown` when git gave no tree id; the tree is the one the run recorded, so compare it with `git rev-parse <commit>^{tree}`), and exits 0 whatever it says: it observes, and the plain command blocks. `skills/nukadoko/scripts/claude-code-verdict-hook.mjs` is an opt-in Claude Code hook that asks for it at the end of a turn that changed the tree.
|
|
55
55
|
- `nuka judge <feature:line> [--run <run-id>] [--meets|--misses --by <name> --because <sentence>] [--json]`: when a run leaves a judgment `pending` or `stale`. A Then with a `judged-as` claim asks for a judgment no compare can give (whether a text reads as friendly); the run reads the value and judges nothing. Without a verdict flag it shows each value, the quality the sentence names, its digest, its step record, and the judged view. With `--meets` or `--misses`, `--by`, and `--because` it records one judgment per value in `judgments/<feature path>.json`, which you commit. The judge is not the agent that wrote the change. A row whose judgment is not `judged` fails the run (`judgment-pending`, `judgment-stale`, `judgment-misses`, `judgment-uncitable`); the verdict column is never changed by a judgment.
|
|
56
56
|
|
|
57
57
|
Exploring one step at a time, when the path is not known yet:
|
|
@@ -34,6 +34,12 @@ module Nukadoko
|
|
|
34
34
|
# root and sorted. nil when git does not answer.
|
|
35
35
|
# : (String, String) -> Array[String]?
|
|
36
36
|
def self?.changed: (String, String) -> Array[String]?
|
|
37
|
+
|
|
38
|
+
# What `commit` holds as the judgments of a feature file: its
|
|
39
|
+
# entries, none when it holds no such file, or why the file did not
|
|
40
|
+
# parse. nil when git does not answer.
|
|
41
|
+
# : (String, String, String) -> Hash[Symbol, untyped]?
|
|
42
|
+
def self?.at_commit: (String, String, String) -> Hash[Symbol, untyped]?
|
|
37
43
|
end
|
|
38
44
|
end
|
|
39
45
|
end
|
|
@@ -89,6 +89,16 @@ module Nukadoko
|
|
|
89
89
|
# : (Array[Hash[Symbol, untyped]], Array[Hash[Symbol, untyped]], ^(String) -> Array[Hash[Symbol, untyped]]) -> Array[Hash[Symbol, untyped]]
|
|
90
90
|
def self?.with_judgments: (Array[Hash[Symbol, untyped]], Array[Hash[Symbol, untyped]], ^(String) -> Array[Hash[Symbol, untyped]]) -> Array[Hash[Symbol, untyped]]
|
|
91
91
|
|
|
92
|
+
# Under --base, each judgment a read cites gains `in_base`: true when
|
|
93
|
+
# the base's file already held that very judgment (the same claim,
|
|
94
|
+
# value, verdict, judge, reason, and time), false when the change
|
|
95
|
+
# brought it. A change cannot be trusted to judge its own output any
|
|
96
|
+
# more than to rewrite its own Thens, and the old world covers only
|
|
97
|
+
# the Thens. `base_entries_of` answers nil for a feature whose base
|
|
98
|
+
# file could not be read, and its reads stay unmarked.
|
|
99
|
+
# : (Array[Hash[Symbol, untyped]], ^(String) -> Array[Hash[Symbol, untyped]]?) -> Array[Hash[Symbol, untyped]]
|
|
100
|
+
def self?.with_base_judgments: (Array[Hash[Symbol, untyped]], ^(String) -> Array[Hash[Symbol, untyped]]?) -> Array[Hash[Symbol, untyped]]
|
|
101
|
+
|
|
92
102
|
# : (Array[Hash[Symbol, untyped]]) -> bool
|
|
93
103
|
def self?.open?: (Array[Hash[Symbol, untyped]]) -> bool
|
|
94
104
|
|
|
@@ -10,6 +10,8 @@ module Nukadoko
|
|
|
10
10
|
class Verdict
|
|
11
11
|
FIXES: untyped
|
|
12
12
|
|
|
13
|
+
TRAILER_KEY: String
|
|
14
|
+
|
|
13
15
|
attr_reader judged: String
|
|
14
16
|
|
|
15
17
|
attr_reader findings: Array[Hash[Symbol, untyped]]
|
|
@@ -32,6 +34,12 @@ module Nukadoko
|
|
|
32
34
|
# : () -> Hash[Symbol, untyped]
|
|
33
35
|
def envelope_fields: () -> Hash[Symbol, untyped]
|
|
34
36
|
|
|
37
|
+
# One git trailer line that stamps a commit with this answer. The tree
|
|
38
|
+
# named is the one the run recorded, so a reader compares it with the
|
|
39
|
+
# commit's own tree later and knows whether the stamp speaks for it.
|
|
40
|
+
# : () -> String
|
|
41
|
+
def trailer: () -> String
|
|
42
|
+
|
|
35
43
|
# : () -> String
|
|
36
44
|
def headline: () -> String
|
|
37
45
|
|
|
@@ -91,6 +91,28 @@ Code's hook fields as of 2.1.317 (`hook_event_name`, `session_id`, `cwd`,
|
|
|
91
91
|
`stop_hook_active`; `decision: "block"` with `reason`); check them against
|
|
92
92
|
the version you run.
|
|
93
93
|
|
|
94
|
+
## Stamping each commit, before the hook blocks
|
|
95
|
+
|
|
96
|
+
A gate that blocks from its first day blocks on the cases nobody
|
|
97
|
+
measured. `nuka verdict --trailer` is the observing form: it prints one
|
|
98
|
+
git trailer line and exits 0 whatever it says, so it never stops a
|
|
99
|
+
commit. Add it in a `prepare-commit-msg` hook:
|
|
100
|
+
|
|
101
|
+
```sh
|
|
102
|
+
git interpret-trailers --in-place --trailer "$(npx nuka verdict --trailer)" "$1"
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
Git runs the hook at the repository's top. When the project is below it,
|
|
106
|
+
`cd` inside the `$(...)`, as `"$(cd app && npx nuka verdict --trailer)"`:
|
|
107
|
+
`$1` is a path from the top.
|
|
108
|
+
|
|
109
|
+
Every commit then carries `Nukadoko-Verdict: passed`, `failed`, `stale`,
|
|
110
|
+
`none` (no run ended), or `unknown` (no tree id), with the run id and the
|
|
111
|
+
tree the run recorded. The stamp speaks for the commit only when
|
|
112
|
+
`git rev-parse <commit>^{tree}` is that tree; a commit of part of the
|
|
113
|
+
working tree carries another. Count the commits that were not `passed`
|
|
114
|
+
for a while, then turn on the Stop hook above, which blocks.
|
|
115
|
+
|
|
94
116
|
## When the reviewer asks how risky it is
|
|
95
117
|
|
|
96
118
|
A pass and a clean intent still leave the reviewer's own question: what
|
|
@@ -93,3 +93,9 @@ never `unproven` (no base can break a claim that compares nothing), and
|
|
|
93
93
|
its judgment decides it. The summary names the judgment files the change
|
|
94
94
|
brings ("A judgment file changed since the base"), so the person reviews
|
|
95
95
|
each judgment with the feature.
|
|
96
|
+
|
|
97
|
+
A judgment the change itself added is the change judging its own output.
|
|
98
|
+
Such a row's `judgment` limit says `stands on a judgment by <name> that
|
|
99
|
+
this change added`, and each cited judgment in run.json carries `in_base:
|
|
100
|
+
false`. When the agent that wrote the change also recorded the judgment,
|
|
101
|
+
tell the person so, and ask a second agent or the person to judge it.
|