@mrciphersmith/keryx 0.3.84 → 0.3.86
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +3 -1
- package/dist/cli.js +2552 -2309
- package/dist/core.js +1145 -1034
- package/package.json +1 -1
- package/src/gdskills/bundled/rules/core/model-selection.mdc +6 -0
- package/src/gdskills/bundled/skills/orchestration/flow-orchestrator/SKILL.md +22 -22
- package/src/gdskills/bundled/skills/orchestration/job-orchestrator/SKILL.md +28 -36
- package/src/gdskills/bundled/skills/planning/autodoc-orchestrator/SKILL.md +1 -2
- package/src/gdskills/bundled/skills/review/review-orchestrator/SKILL.detail.md +289 -0
- package/src/gdskills/bundled/skills/review/review-orchestrator/SKILL.md +116 -213
- package/src/gdskills/bundled/skills/review/review-orchestrator/review-context.schema.json +57 -0
- package/src/gdskills/bundled/skills/review/review-orchestrator/templates/review-report.md +1 -1
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@mrciphersmith/keryx",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.86",
|
|
4
4
|
"description": "Version-controlled project context for AI coding agents: code graph, architecture wiki, project memory, relevant tests, quality signals, and task flows.",
|
|
5
5
|
"private": false,
|
|
6
6
|
"publishConfig": {
|
|
@@ -326,6 +326,12 @@ recording them is one copy rather than four decisions.
|
|
|
326
326
|
remove.
|
|
327
327
|
- **Do not ask the operator which model to use per dispatch.** Asking every time
|
|
328
328
|
is what made adaptive selection impossible.
|
|
329
|
+
- **An interactive review round shows the computed plan once.** It is the third
|
|
330
|
+
start question of `review-orchestrator` (`model_strategy: ask`, the
|
|
331
|
+
interactive default): the complexity the round's signals give, each role's tier
|
|
332
|
+
and the model it resolves to here. The operator accepts it, takes every role to
|
|
333
|
+
`deep`, or sets it — for that round. An unattended round dispatches the
|
|
334
|
+
computed plan without asking (`adaptive`).
|
|
329
335
|
- Never write a concrete model name into a skill, a rule, a dispatch template, or
|
|
330
336
|
the resolution code.
|
|
331
337
|
- Take the candidate set from runtime detection. Never from a literal list of
|
|
@@ -21,6 +21,10 @@ license: "MIT"
|
|
|
21
21
|
|
|
22
22
|
# Flow Orchestrator
|
|
23
23
|
|
|
24
|
+
## Preflight
|
|
25
|
+
|
|
26
|
+
Run in the main session: this skill dispatches workers. Without the `Agent` tool, or with the depth budget spent (Claude Code nests at most 3 deep), the whole output is this line, with no tool calls and no work of your own: `STATUS: BLOCKED nested_dispatch_unavailable — run this skill in the main session`
|
|
27
|
+
|
|
24
28
|
## Purpose
|
|
25
29
|
|
|
26
30
|
Flow Orchestrator is the Task Manager-aware implementation orchestrator. Plan bridge: publish the tasks with `plan_set` under the ids `T1`…`Tn`, mirror each `keryx flow task done` with `plan_update`, and publish Phase 4's completion choice as `proposed` — see the `session-plan-bridge` rule.
|
|
@@ -28,8 +32,6 @@ It wraps the existing gdskills pipeline with `keryx flow` state.
|
|
|
28
32
|
|
|
29
33
|
Use this skill instead of `job-orchestrator` when the user wants a managed story/issue lifecycle with frozen acceptance criteria, task state, an explicit completion choice, Code Health, and a durable flow package in `.metaproject/flows/`.
|
|
30
34
|
|
|
31
|
-
Do not modify `job-orchestrator` or `task-implementer` behavior. They remain usable without Task Manager. This skill coordinates them through flow state.
|
|
32
|
-
|
|
33
35
|
## Hard Preconditions
|
|
34
36
|
|
|
35
37
|
1. Read `.metaproject/index.md` and `.metaproject/metaproject.json`.
|
|
@@ -122,9 +124,8 @@ it already tried. The flow package does.
|
|
|
122
124
|
keryx flow next <id> --json
|
|
123
125
|
```
|
|
124
126
|
|
|
125
|
-
This is the first task
|
|
126
|
-
`
|
|
127
|
-
instead of re-derived by you from prose.
|
|
127
|
+
This is the first task not `done` whose declared `dependsOn` are all
|
|
128
|
+
`done` — computed from the package, not re-derived from prose.
|
|
128
129
|
|
|
129
130
|
4. **Read the `resume` field before dispatching anything.** It has three
|
|
130
131
|
answers and they are not interchangeable:
|
|
@@ -141,10 +142,9 @@ it already tried. The flow package does.
|
|
|
141
142
|
or close the task (`keryx flow task done <id> <Tn>`). Re-dispatching
|
|
142
143
|
over an unresolved attempt is how the same work gets done twice.
|
|
143
144
|
|
|
144
|
-
`keryx flow next` also lists every OTHER not-done task
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
idle.
|
|
145
|
+
`keryx flow next` also lists every OTHER not-done task with an unresolved
|
|
146
|
+
attempt, under `unresolved`: parallel dispatches that never reported back.
|
|
147
|
+
Resolve them the same way before assuming the flow is idle.
|
|
148
148
|
|
|
149
149
|
5. Before dispatching a worker for that task, record the attempt:
|
|
150
150
|
|
|
@@ -433,22 +433,20 @@ Before accepting implementation:
|
|
|
433
433
|
1. Run focused tests for touched scope.
|
|
434
434
|
2. Run `code-verifier`.
|
|
435
435
|
3. Run `keryx health run` when Code Health is enabled.
|
|
436
|
-
4. Check the bounds, then run `review-orchestrator` with relevant domains
|
|
437
|
-
|
|
436
|
+
4. Check the bounds, then run `review-orchestrator` with relevant domains — in this session,
|
|
437
|
+
never handed whole to one subagent: one without a dispatch tool returns `STATUS: BLOCKED`.
|
|
438
438
|
```bash
|
|
439
|
-
keryx review budget --spent <usd-so-far> --outstanding <subagents
|
|
439
|
+
keryx review budget --spent <usd-so-far> --full --outstanding <subagents still running, 0 if none>
|
|
440
440
|
```
|
|
441
|
-
|
|
442
441
|
A non-zero exit means the spend ceiling (3 USD by default) has been reached:
|
|
443
|
-
**stop and ask the user** rather than dispatching another fan-out.
|
|
442
|
+
**stop and ask the user** rather than dispatching another fan-out — never shrink the fan-out to fit.
|
|
443
|
+
Dispatch prompts point at sources (files, SHAs, the PR) and never paraphrase a contract.
|
|
444
444
|
|
|
445
|
-
`--outstanding` is the part that matters here. `review-orchestrator`
|
|
446
|
-
dispatches reviewers in parallel
|
|
447
|
-
|
|
448
|
-
|
|
449
|
-
|
|
450
|
-
alone, which the review record then states plainly rather than implying
|
|
451
|
-
otherwise.
|
|
445
|
+
`--outstanding` is the part that matters here. `review-orchestrator` runs inline
|
|
446
|
+
in this session and dispatches reviewers in parallel; keryx cannot observe other
|
|
447
|
+
subagents. Passing the count still running is the only thing that makes the
|
|
448
|
+
concurrency cap mean anything; omit it and the cap bounds the reviewer fan-out
|
|
449
|
+
alone, which the review record then states plainly. Add `--full` for `--all`.
|
|
452
450
|
|
|
453
451
|
5. If findings require code changes, dispatch fix work through `task-implementer`
|
|
454
452
|
and record the fix task in the flow.
|
|
@@ -586,7 +584,9 @@ the bound plus an escalation — never an unbounded loop.
|
|
|
586
584
|
findings, affected graph, relevant wiki, and health/testing artifacts;
|
|
587
585
|
identify the likely cycle cause; choose a materially different fix strategy
|
|
588
586
|
or split the work into narrower tasks; record the decision in `journal.md`;
|
|
589
|
-
then continue with the enriched context.
|
|
587
|
+
then continue with the enriched context. The review round count is not reset:
|
|
588
|
+
past `REVIEW_ROUND_CAP` rounds on one merge-base..HEAD, report to the operator
|
|
589
|
+
instead of running another round.
|
|
590
590
|
6. Never merge while findings or required checks remain unresolved. If the
|
|
591
591
|
re-planned approach still cannot produce a mergeable PR, leave the flow
|
|
592
592
|
`in-progress` and report the blocker instead of forcing completion.
|
|
@@ -22,14 +22,12 @@ metadata:
|
|
|
22
22
|
license: "MIT"
|
|
23
23
|
---
|
|
24
24
|
|
|
25
|
-
<SUBAGENT-STOP>
|
|
26
|
-
If you were dispatched as a subagent to execute a specific task, skip this skill entirely.
|
|
27
|
-
This skill is for orchestrators and interactive session-level routing only.
|
|
28
|
-
Proceed directly with your assigned task.
|
|
29
|
-
</SUBAGENT-STOP>
|
|
30
|
-
|
|
31
25
|
# Job Orchestrator
|
|
32
26
|
|
|
27
|
+
## Preflight
|
|
28
|
+
|
|
29
|
+
Run in the main session: this skill dispatches sub-agents. Without the `Agent` tool, or with the depth budget spent (Claude Code nests at most 3 deep), the whole output is this line, with no tool calls and no work of your own: `STATUS: BLOCKED nested_dispatch_unavailable — run this skill in the main session`
|
|
30
|
+
|
|
33
31
|
## Purpose
|
|
34
32
|
|
|
35
33
|
Dynamic orchestrator that builds execution plans based on user intent. Unlike a fixed pipeline, the orchestrator adapts its workflow to what the user actually needs — from "just analyze this issue" to "implement, review, and create a PR". It dispatches sub-agents (`issue-analyzer`, `context-collector`, `tests-creator`, `task-implementer`, `code-verifier`, `review-orchestrator`) and persists every step, document and retry through `keryx job`, which writes `.metaproject/jobs/<job-name>/`. Plan bridge: publish these steps with `plan_set` under the ids `keryx job status` reports, mirror each `keryx job step` with `plan_update`, and mark them `proposed` while the approval above is open — see the `session-plan-bridge` rule.
|
|
@@ -1059,16 +1057,14 @@ carried in the dispatch prompt and reported in 2.9.
|
|
|
1059
1057
|
**Step 1 — check the budget before dispatching, while stopping is still possible.**
|
|
1060
1058
|
|
|
1061
1059
|
```bash
|
|
1062
|
-
keryx review budget --spent <usd-so-far> --outstanding <subagents
|
|
1060
|
+
keryx review budget --spent <usd-so-far> --full --outstanding <subagents still running, 0 if none>
|
|
1063
1061
|
```
|
|
1064
1062
|
|
|
1065
|
-
`--outstanding` is
|
|
1066
|
-
|
|
1067
|
-
|
|
1068
|
-
|
|
1069
|
-
|
|
1070
|
-
total only when the parent declares its own in-flight count.** Omit `--outstanding`
|
|
1071
|
-
and the cap bounds the reviewer fan-out alone, which the record then states plainly.
|
|
1063
|
+
`--outstanding` is the number of subagents this session still has running while the
|
|
1064
|
+
round starts (0 when none). All three orchestrators run inline in the main session, so the
|
|
1065
|
+
reviewers are the only fan-out; keryx cannot observe other subagents, so **the wave size
|
|
1066
|
+
(at most 10 for `--all` or a full review; add `--full`) binds the total only when you
|
|
1067
|
+
declare that count.** Omit it and the record states that the cap bounds the reviewers alone.
|
|
1072
1068
|
|
|
1073
1069
|
A non-zero exit means the spend ceiling (3 USD by default) is reached: stop and ask
|
|
1074
1070
|
the user rather than dispatching another fan-out.
|
|
@@ -1122,30 +1118,26 @@ replaced. The command names no model — it ranks what the provider reports at r
|
|
|
1122
1118
|
and, when it cannot rank anything, prints `inherit: true`, which means the dispatch
|
|
1123
1119
|
runs on the session model. That is a correct answer, not a failure.
|
|
1124
1120
|
|
|
1125
|
-
**Step 6 —
|
|
1121
|
+
**Step 6 — run `review-orchestrator` inline, in this main session.** Load
|
|
1122
|
+
`skills/gdskills/review/review-orchestrator/SKILL.md` yourself; do **not** wrap it in a
|
|
1123
|
+
`Task` — a subagent cannot spawn the reviewers and the skill answers
|
|
1124
|
+
`STATUS: BLOCKED nested_dispatch_unavailable`. Give it these inputs:
|
|
1126
1125
|
|
|
1127
1126
|
```
|
|
1128
|
-
|
|
1129
|
-
|
|
1130
|
-
|
|
1131
|
-
|
|
1132
|
-
|
|
1133
|
-
|
|
1134
|
-
|
|
1135
|
-
|
|
1136
|
-
issue_url: <issue URL, when the job has one — enables the Stage 1 spec gate>
|
|
1137
|
-
context_doc: <JOBS_ROOT>/<job-name>/ai/context.md
|
|
1138
|
-
verification_mode: annotate
|
|
1139
|
-
managed_review: { mode: "review-flow", target: "branch", target_ref: "<feature-branch>" }
|
|
1140
|
-
is_fix_round: <true on any round after the first>
|
|
1141
|
-
pr_comments: { enabled: <true when a PR exists> }
|
|
1142
|
-
|
|
1143
|
-
Emit the unified report AND the fenced ```json keryx:findings``` block.
|
|
1144
|
-
Dispatch review-verifier (Wave C) over the consolidated findings and return
|
|
1145
|
-
its verification claims as a file path.
|
|
1146
|
-
})
|
|
1127
|
+
flags: <selected flags, e.g. --backend --security --testing-practices>
|
|
1128
|
+
commit_range: <BASE_SHA>..HEAD
|
|
1129
|
+
issue_url: <issue URL, when the job has one — enables the Stage 1 spec gate>
|
|
1130
|
+
context_doc: <JOBS_ROOT>/<job-name>/ai/context.md
|
|
1131
|
+
verification_mode: annotate
|
|
1132
|
+
managed_review: { mode: "review-flow", target: "branch", target_ref: "<feature-branch>" }
|
|
1133
|
+
is_fix_round: <true on any round after the first>
|
|
1134
|
+
pr_comments: { enabled: <true when a PR exists> }
|
|
1147
1135
|
```
|
|
1148
1136
|
|
|
1137
|
+
It emits the unified report AND the fenced ```json keryx:findings``` block, and runs
|
|
1138
|
+
`review-verifier` (Wave C) over the consolidated findings, returning its verification
|
|
1139
|
+
claims as a file path.
|
|
1140
|
+
|
|
1149
1141
|
**Step 7 — verification is part of the round, not an extra.** `review-orchestrator`
|
|
1150
1142
|
dispatches `review-verifier` in Wave C over the consolidated findings. The verifier
|
|
1151
1143
|
**runs something** and can only delete — it never raises a severity, adds a finding,
|
|
@@ -1310,7 +1302,7 @@ FOR iteration in [1, 2, 3]:
|
|
|
1310
1302
|
keryx review comments collect --repo <r> --pr <n> --sha <new-head> --round <N+1> --out <file>
|
|
1311
1303
|
keryx review scope --ref "$BASE_SHA" --json > scope.json
|
|
1312
1304
|
keryx review blast-radius --ref "$BASE_SHA" --previous blast-radius.json --json > blast-radius.json
|
|
1313
|
-
<
|
|
1305
|
+
<run review-orchestrator inline, in this session, with is_fix_round: true>
|
|
1314
1306
|
keryx review ingest --report <new-report> --ref <feature-branch> --head <new-head> \
|
|
1315
1307
|
--scope scope.json --blast-radius blast-radius.json \
|
|
1316
1308
|
--verifications <file> --refuted <file> --outstanding <n>
|
|
@@ -1985,7 +1977,7 @@ What actually bounds this orchestrator:
|
|
|
1985
1977
|
| review → fix rounds | 3 | 2.7, and `max_review_iterations` (`maximum: 3`) in the input contract |
|
|
1986
1978
|
| repetition, whatever the count says | first repeat | the STUCK CHECK in 2.7, and `keryx review loop` against the durable record |
|
|
1987
1979
|
| retries per step | recorded, not guessed | `metrics.steps[].retries`, incremented by `keryx job step --status in-progress` and read back with `keryx job status --json` |
|
|
1988
|
-
| reviewer fan-out |
|
|
1980
|
+
| reviewer fan-out | ≤10 per wave | `keryx review budget --outstanding <n>` before every dispatch (2.6.1) |
|
|
1989
1981
|
| spend | 3 USD by default | `keryx review budget --spent <usd>` — a non-zero exit means stop and ask |
|
|
1990
1982
|
|
|
1991
1983
|
Each of these is a number some command reads or writes. A guard no command can
|
|
@@ -20,8 +20,7 @@ metadata:
|
|
|
20
20
|
version: 1.0.0
|
|
21
21
|
---
|
|
22
22
|
|
|
23
|
-
|
|
24
|
-
Return STATUS: BLOCKED — autodoc-orchestrator must run as top-level agent only. -->
|
|
23
|
+
**Preflight:** run in the main session. Without the `Agent` tool or depth budget, output only `STATUS: BLOCKED nested_dispatch_unavailable — run this skill in the main session`, with no tool calls.
|
|
25
24
|
|
|
26
25
|
# autodoc-orchestrator
|
|
27
26
|
|
|
@@ -368,3 +368,292 @@ inside) the findings they annotate — same separation the finding schema
|
|
|
368
368
|
itself draws between a reviewer's claim (`severity`, `problem`, …) and what
|
|
369
369
|
became of it (`disposition`), which a reviewer never states and this pass
|
|
370
370
|
does not either.
|
|
371
|
+
|
|
372
|
+
## Start questions (Step 5) — once per round, after the context pack
|
|
373
|
+
|
|
374
|
+
The round asks its operator three questions, once, in one interaction, after
|
|
375
|
+
Step 1 (context pack) and Step 3 (scope) — never before them: the counterpart
|
|
376
|
+
question cannot be answered well until the context pack has looked for the
|
|
377
|
+
other side. On Claude Code that is one `AskUserQuestion` call carrying the
|
|
378
|
+
three questions; on any other host, the lettered prompt below. An unattended
|
|
379
|
+
round (job pipeline, trigger, schedule) asks nothing, applies the defaults and
|
|
380
|
+
records them under `### How this review was run`.
|
|
381
|
+
|
|
382
|
+
1. **Reviewers** — default `--all`: every reviewer the engine detects plus every
|
|
383
|
+
project and convention reviewer its gates admit. Narrowing is an explicit
|
|
384
|
+
choice (tests only, one domain, a custom list). Flags the caller passed
|
|
385
|
+
answer this question and skip it. Convention reviewers are part of this
|
|
386
|
+
question; the job setting `convention_reviewers` still answers it in a
|
|
387
|
+
`job-orchestrator` pipeline.
|
|
388
|
+
2. **Counterpart** — the other side of every contract the diff consumes or
|
|
389
|
+
mirrors: the backend for a frontend change, the frontend and other consumers
|
|
390
|
+
for a backend change. Before asking, the context pack looks for it:
|
|
391
|
+
endpoints, DTOs and rules the diff calls or mirrors, a paired PR
|
|
392
|
+
(`gh pr list --repo <counterpart> --search <branch-or-feature>`), the
|
|
393
|
+
producer at a freshly fetched SHA. The question shows what was found —
|
|
394
|
+
"no link found" included — and one checkbox, **verify the counterpart**, ON
|
|
395
|
+
by default even when nothing was found. Verification is not limited to a
|
|
396
|
+
paired PR: the other side's contracts and logic are re-read at a pinned SHA
|
|
397
|
+
every round. When the round cannot locate that code itself (repository,
|
|
398
|
+
handler, validator), it asks the operator where it is; it never records
|
|
399
|
+
`state: unavailable` on its own. Skipping verification takes a reason, and
|
|
400
|
+
the reason goes in the report.
|
|
401
|
+
3. **Models** — the complexity the round already holds signals for (changed
|
|
402
|
+
files and lines, the risk class from Depth floor below, domains, scope B),
|
|
403
|
+
the per-role tier `keryx review tier` prints for each planned dispatch, and
|
|
404
|
+
the model each tier resolves to on this host. Choices: accept the plan (the
|
|
405
|
+
default), all `deep`, or custom. The plan is shown once per round, never per
|
|
406
|
+
dispatch (`rules/core/model-selection.mdc`). Skills, rules and this file name
|
|
407
|
+
tiers only; model ids appear in the rendered question and the report.
|
|
408
|
+
|
|
409
|
+
```text
|
|
410
|
+
Context: <PR or branch>, <N> files / <N> lines, <domains>; risk: <class>
|
|
411
|
+
|
|
412
|
+
1) Reviewers A) --all (default) B) tests only C) <domain> only D) choose
|
|
413
|
+
2) Counterpart found: <endpoints, rules, paired PR — or "no link found">
|
|
414
|
+
A) verify <backend|frontend> (default) B) skip — give a reason
|
|
415
|
+
3) Models complexity: <low|medium|high>; <role>: <tier> → <model>, …
|
|
416
|
+
A) accept (default) B) all deep C) custom
|
|
417
|
+
|
|
418
|
+
> answers (default: 1A 2A 3A)
|
|
419
|
+
```
|
|
420
|
+
|
|
421
|
+
## Depth floor — a gating round does not degrade
|
|
422
|
+
|
|
423
|
+
A round is **gating** when its verdict lets a change move: the review step of a
|
|
424
|
+
flow (`flow-orchestrator` Phase 3), a pre-merge or pre-ship review, a
|
|
425
|
+
re-request after fixes. A gating round runs the composition the start questions
|
|
426
|
+
selected.
|
|
427
|
+
|
|
428
|
+
- **No dispatch tool, no review.** A host that cannot dispatch subagents — this
|
|
429
|
+
engine running inside a subagent, say — returns
|
|
430
|
+
`STATUS: BLOCKED` with reason `nested_dispatch_unavailable` and no verdict.
|
|
431
|
+
It does not review the diff alone and call that a round. Measured (flow 405):
|
|
432
|
+
a single-agent round on a 150-line frontend PR wrote "did not dispatch any
|
|
433
|
+
sub-reviewers", approved it with two cosmetic minors, and the next human round
|
|
434
|
+
filed six real findings, every one reachable from the diff and the backend at
|
|
435
|
+
a pinned SHA.
|
|
436
|
+
- **Size never lowers depth below the risk class.** The budget table sizes the
|
|
437
|
+
context, not the depth.
|
|
438
|
+
- **High-risk includes a client-side mirror of a server rule** — validation,
|
|
439
|
+
limits, permissions, uniqueness, reserved names — beside the triggers the main
|
|
440
|
+
file lists.
|
|
441
|
+
|
|
442
|
+
## Counterpart verification — the contract is enumerated, not summarised
|
|
443
|
+
|
|
444
|
+
For an **input rule** the diff mirrors or depends on (what the server accepts
|
|
445
|
+
for a name, a value, a role), the `cross_repo` entry records the producer's
|
|
446
|
+
complete check set in `checks` and how that set was found in
|
|
447
|
+
`enumeration_method` (`review-context.schema.json`):
|
|
448
|
+
|
|
449
|
+
- every check the producer runs on that input: validators, restriction
|
|
450
|
+
utilities, uniqueness queries, reserved-name registries, and the storage bound
|
|
451
|
+
(column width, index, a downstream identifier);
|
|
452
|
+
- for each check: its `source` (`file:line` at the pinned SHA), whether it is
|
|
453
|
+
case-sensitive, and the `paths` it runs on (create, update, rename, copy,
|
|
454
|
+
import, move, …).
|
|
455
|
+
|
|
456
|
+
Put these rules in every reviewer dispatch:
|
|
457
|
+
|
|
458
|
+
- **The facts are a starting point, not a boundary.** A reviewer may read the
|
|
459
|
+
producer itself and file what the entry missed.
|
|
460
|
+
- **"The producer has no limit" is a question, not a fact.** Find what bounds
|
|
461
|
+
the value — that is where an over-long or unexpected value fails, usually as a
|
|
462
|
+
server error.
|
|
463
|
+
- **Build the surfaces × checks matrix**: every surface that sends the input
|
|
464
|
+
against every check, each cell `same`, `missing`, `stricter` or `looser`.
|
|
465
|
+
`missing` is a refusal the user meets late; `stricter` blocks legal input; a
|
|
466
|
+
check the producer lacks but storage enforces is a producer defect — filed as
|
|
467
|
+
pre-existing, or fixed by a paired change.
|
|
468
|
+
- **Every refusal the producer can return is shown where the user can act on
|
|
469
|
+
it** — next to the field, not as a generic toast.
|
|
470
|
+
|
|
471
|
+
## Round obligations — always on, never asked
|
|
472
|
+
|
|
473
|
+
These apply to every round without a question:
|
|
474
|
+
|
|
475
|
+
1. **Enumerate before concluding.** Every caller and surface of a changed value
|
|
476
|
+
is found with `keryx ctx rg` or gdgraph, and the query is recorded — never
|
|
477
|
+
taken from the diff's own list.
|
|
478
|
+
2. **Quantified claims are checked.** "Every", "all", "only" and "never" in the
|
|
479
|
+
PR body or the commits are compared with that enumeration; an over-claim is a
|
|
480
|
+
`minor` finding, the same class as the description-versus-diff rule.
|
|
481
|
+
3. **A verified-clean line names its sites.** Each line of the `Verified clean`
|
|
482
|
+
lane names the files, sites or commands it rests on; "the frontend does the
|
|
483
|
+
same" without them is not verified.
|
|
484
|
+
4. **A pre-existing defect gets a disposition.** Every pre-existing defect the
|
|
485
|
+
round finds in the contract the change touches is recorded with one: fixed in
|
|
486
|
+
the change (small, same contract), named out of scope in the PR body, or a
|
|
487
|
+
recommended issue. Found and dropped is the failure this rule exists for.
|
|
488
|
+
5. **Red on the parent, per test.** Every added test fails with the parent
|
|
489
|
+
commit's production files restored; the run is the evidence.
|
|
490
|
+
6. **A threaded value is pinned per site.** A parameter, key or prop passed at
|
|
491
|
+
several render sites is asserted at each site, or centralised so there is
|
|
492
|
+
one.
|
|
493
|
+
7. **One rule, one implementation.** The same check implemented twice is a
|
|
494
|
+
finding, even while both copies agree.
|
|
495
|
+
8. **"Do not re-raise" needs a record.** A dispatch may exclude an item only by
|
|
496
|
+
citing the PR comment, issue or decision that settled it — never a whole
|
|
497
|
+
surface ("X has no test; covered elsewhere").
|
|
498
|
+
|
|
499
|
+
### Runtime research-completion ledger
|
|
500
|
+
|
|
501
|
+
New managed packages carry `research.json`, initially pending. Before closing a round, enumerate unresolved observations from raw reviewers and the changed-file/consumer census, assign stable obligation IDs, and investigate them. Run the probe or inspect the relevant source and assertions: merely proposing a check does not resolve it. An unexecuted probe is not a verified claim.
|
|
502
|
+
|
|
503
|
+
Supply the ledger using `keryx review ingest --research <ledger.json>` alongside the report and findings. Format: `version: 1`, `scopeReviewed: true`, `rawReconciled: true`, `obligations: [...]`. Each obligation requires `id`, `source`, `question`, `status`, `evidence`, `reason`. Status `finding` additionally requires `finding` pointing to exactly one canonical finding ID or global ID. Other terminal statuses are `refuted` and `out-of-scope`, both requiring evidence and reason. Preserve distinct scenarios when deduplicating. Assert census/reconciliation flags only after performing those checks; an empty obligations array is valid only if the actual census found no open observations.
|
|
504
|
+
|
|
505
|
+
Open, deferred and unverifiable obligations block completion. If context, runtime or budget prevents resolution, retain the obligation and report incomplete; do not relabel it refuted or out-of-scope. Completion and the flow gate re-read the ledger even when the manifest says closed. Legacy packages without this artifact remain compatible, not proven complete. This gate validates the declared ledger, not the truth or exhaustiveness of its census; independent source/raw audits remain necessary.
|
|
506
|
+
|
|
507
|
+
## Why Wave C verifies instead of re-scoring
|
|
508
|
+
|
|
509
|
+
Wave C used to run `review-strict`: a meta-pass that re-read the consolidated
|
|
510
|
+
findings and **adjusted their severity with no new evidence**, under an elevation
|
|
511
|
+
table biased 3:1 toward escalation. It was **removed, not improved**, and the
|
|
512
|
+
reason is measured rather than stylistic:
|
|
513
|
+
|
|
514
|
+
- **GPT-4 on GSM8K across self-correction rounds: 95.5 → 91.5 → 89.0.**
|
|
515
|
+
**GPT-3.5 on CommonSenseQA: 75.8 → 38.1.** Among the answers that changed,
|
|
516
|
+
correct → incorrect exceeded incorrect → correct (Huang et al., *Large Language
|
|
517
|
+
Models Cannot Self-Correct Reasoning Yet*, ICLR 2024, arXiv:2310.01798).
|
|
518
|
+
- **Self-Refine (arXiv:2303.17651): +49.2 on dialogue response generation, +0.2
|
|
519
|
+
on maths.** Self-refinement gains are on subjective tasks and vanish on
|
|
520
|
+
verifiable reasoning. Judging whether a null-guard is missing is verifiable
|
|
521
|
+
reasoning.
|
|
522
|
+
|
|
523
|
+
Re-scoring a finding by re-reading it is therefore not a rigour pass; it is a
|
|
524
|
+
coin flip weighted toward more findings. **Do not restore it because it looks
|
|
525
|
+
obviously useful — it looked obviously useful the first time.**
|
|
526
|
+
|
|
527
|
+
`review-verifier` occupies the slot and differs in exactly one way that matters:
|
|
528
|
+
**it runs something.** Verification that executes rejects 85–96% of false reports
|
|
529
|
+
against 4–15% unaided while finding 30–44% more true bugs (AnyPoC,
|
|
530
|
+
arXiv:2604.11950); Meta's TestGen-LLM funnel discards 75% of its own output
|
|
531
|
+
(75% build → 57% build and pass → 25% improve coverage) and the surviving quarter
|
|
532
|
+
reaches 73% human acceptance (arXiv:2402.09171).
|
|
533
|
+
|
|
534
|
+
## Scope B — bounds, rejections and recompute
|
|
535
|
+
|
|
536
|
+
It walks `gdgraph affected` outward from every changed file, ranks by edge
|
|
537
|
+
distance, keeps distance ≤ 2, cuts at 40 files closest-first, and adds a changed
|
|
538
|
+
file's naming-related tests when the graph did not already reach them. Requires a
|
|
539
|
+
built graph — run `keryx gdgraph build` if it refuses.
|
|
540
|
+
|
|
541
|
+
**Do not pick the files yourself, and do not widen it.** "Review the
|
|
542
|
+
functionality so nothing breaks" naively means "review the whole repository every
|
|
543
|
+
round", which is unaffordable *and* actively harmful: review quality decays as
|
|
544
|
+
context grows — measured F1 0.65 at round 2 falling to 0.29 at round 10. An
|
|
545
|
+
unbounded scope B makes later rounds worse than earlier ones.
|
|
546
|
+
|
|
547
|
+
The bounds are measured on this repository, not guessed: at depth 2 the set is a
|
|
548
|
+
median of 19 files (p90 65); depth 3 buys eight more in the median and doubles
|
|
549
|
+
the p90. The 40-file cap fires on 25% of commits and removes only hop-2 entries
|
|
550
|
+
on all but 2 of 80, so it almost never costs a direct dependent — and when it
|
|
551
|
+
does, it says so.
|
|
552
|
+
|
|
553
|
+
**Record the whole thing.** `--out "<review-package>/blast-radius.md"` writes the
|
|
554
|
+
set, the depth, and **every file the cap removed**. A truncation nobody can see
|
|
555
|
+
reads afterwards as "we checked everything", which is the claim this pipeline
|
|
556
|
+
exists to stop making. An empty radius is reported as `unresolved`, not as clean:
|
|
557
|
+
the graph indexes code, so a change to a skill, a rule or a schema has no blast
|
|
558
|
+
radius at all and that is a different fact from "nothing depends on it".
|
|
559
|
+
|
|
560
|
+
### The scope-B question, and what is rejected
|
|
561
|
+
|
|
562
|
+
> Does this change break an existing behaviour **at these sites**?
|
|
563
|
+
|
|
564
|
+
Nothing else. The blast-radius set is **under regression check, not under
|
|
565
|
+
review**. A finding about style, naming or architecture in code the change did
|
|
566
|
+
not touch is refused **by the orchestrator in code** — not discouraged here —
|
|
567
|
+
under three rules, every one of them a fact about the claim rather than about who
|
|
568
|
+
made it:
|
|
569
|
+
|
|
570
|
+
| Rule | Refused because |
|
|
571
|
+
|---|---|
|
|
572
|
+
| `outside-set` | the file is neither in the computed set nor in the changed set; the reviewer went browsing |
|
|
573
|
+
| `non-regression-severity` | below `major`. Under the canonical rubric `minor` states the code behaves correctly and `info` names neither trigger nor outcome; neither can be a claim that something broke |
|
|
574
|
+
| `no-link-to-change` | nothing in the finding names a changed file, module or symbol. A regression claim says THE CHANGE broke this site |
|
|
575
|
+
|
|
576
|
+
Rejections are **recorded, not deleted** — raise the observation under scope A or
|
|
577
|
+
as a separate review. Pass `--brief` output verbatim into the scope-B dispatch:
|
|
578
|
+
the code rejection is the enforcement, but a reviewer told afterwards has already
|
|
579
|
+
spent the round producing findings that will all be refused.
|
|
580
|
+
|
|
581
|
+
`class_scope` on a scope-B finding names the **caller that breaks**, not the
|
|
582
|
+
changed line, because that is the site a human has to look at.
|
|
583
|
+
|
|
584
|
+
### When it is recomputed
|
|
585
|
+
|
|
586
|
+
| Round | Scope A | Scope B |
|
|
587
|
+
|---|---|---|
|
|
588
|
+
| 1 (first after the draft PR) | yes | yes |
|
|
589
|
+
| 2..N | yes | recomputed only if the changed-file set moved |
|
|
590
|
+
| final | yes | **yes, always** |
|
|
591
|
+
|
|
592
|
+
Do not decide this by memory:
|
|
593
|
+
|
|
594
|
+
```bash
|
|
595
|
+
keryx review blast-radius --ref "${BASE_SHA}" --previous blast-radius.json [--final]
|
|
596
|
+
```
|
|
597
|
+
|
|
598
|
+
It prints the decision and the reason, and reuses the previous record when
|
|
599
|
+
nothing moved. The final round recomputes whatever the file set did — otherwise a
|
|
600
|
+
fix introduced in round 3 gets no regression check at all, and the round that
|
|
601
|
+
certifies the flow is the one that checked the least.
|
|
602
|
+
|
|
603
|
+
## Fix rounds — dispositions, regressions, withdrawal
|
|
604
|
+
|
|
605
|
+
### Every prior finding leaves the round with a disposition
|
|
606
|
+
|
|
607
|
+
A fix round that reports only new findings is unreadable: the author cannot tell
|
|
608
|
+
which of their fixes landed. Close the loop explicitly — one line per prior
|
|
609
|
+
finding, in the report, before the new findings:
|
|
610
|
+
|
|
611
|
+
| Disposition | Meaning |
|
|
612
|
+
|---|---|
|
|
613
|
+
| `closed` | Checked against the code, not against the commit message, and the defect is gone |
|
|
614
|
+
| `open` | The fix does not reach the defect; say what is still true |
|
|
615
|
+
| `partial` | One site of the class was fixed and the enumeration named others |
|
|
616
|
+
| `regressed` | The fix removed this defect and introduced another — file the new one separately |
|
|
617
|
+
| `withdrawn` | The finding was wrong. See below |
|
|
618
|
+
|
|
619
|
+
**Check the code, not the commit message.** A commit titled *"report an abandoned
|
|
620
|
+
sync as abandoned"* is a claim; the disposition is whether the branch it renamed
|
|
621
|
+
is reachable and pinned. On a recorded round, two such commits asserted behaviour
|
|
622
|
+
on lines no test could reach.
|
|
623
|
+
|
|
624
|
+
### A fix is a change, and changes get reviewed
|
|
625
|
+
|
|
626
|
+
The most expensive class in a multi-round review is **the defect the fix
|
|
627
|
+
introduced**. It is systematically under-found, for a structural reason: the fix
|
|
628
|
+
arrives framed as the answer to a finding, so it is read as an answer rather than
|
|
629
|
+
as new code. It is new code.
|
|
630
|
+
|
|
631
|
+
So scope A of a fix round includes the fix, reviewed on its own merits, and the
|
|
632
|
+
report carries its own section:
|
|
633
|
+
|
|
634
|
+
```markdown
|
|
635
|
+
## Regressions the fixes introduced
|
|
636
|
+
<[F-NNN] — the finding it was answering, and the new defect it created>
|
|
637
|
+
```
|
|
638
|
+
|
|
639
|
+
Recorded shapes, all from fixes that correctly closed the finding they answered:
|
|
640
|
+
an early return added to stop a fall-through, which then skipped the work the
|
|
641
|
+
caller needed; a persistence call added to save expanded state, which then
|
|
642
|
+
persisted the broken state on the error path too. Both were closed correctly and
|
|
643
|
+
both shipped a new bug in the same commit.
|
|
644
|
+
|
|
645
|
+
### Withdrawing your own earlier finding
|
|
646
|
+
|
|
647
|
+
A finding from a previous round that this round disproves is **withdrawn**,
|
|
648
|
+
explicitly, at the top of the report, with the evidence — before any new finding.
|
|
649
|
+
|
|
650
|
+
This is not a courtesy. An uncorrected wrong finding costs the author a fix they
|
|
651
|
+
did not need, and it stays in `prior_findings` steering later rounds. Withdrawal
|
|
652
|
+
is also the one self-correction this pipeline permits, and it is permitted because
|
|
653
|
+
it is asymmetric: it *deletes* a claim, so it cannot inflate the finding count,
|
|
654
|
+
which is the failure mode that removed the re-scoring pass from Wave C.
|
|
655
|
+
|
|
656
|
+
State what made the original claim wrong, in one sentence, and if the same
|
|
657
|
+
reasoning error has now happened twice in one review, say that too. A reviewer
|
|
658
|
+
that names its own recurring error is calibrating; one that quietly drops a
|
|
659
|
+
finding is hiding a result.
|