tinybeeai 0.0.0-stage → 0.16.1

package/docs/PR-REVIEW.md

@@ -0,0 +1,357 @@
+# PR review
+
+The PR review reads a change the way a QA engineer would before it ships: what could break for users, which tests to run, and which tests the change puts at risk. It runs on every PR check (opened, new commits, reopened) and every post-merge check of a dot that watches a repo. The design, prompts, schemas, and evals are in [PR-REVIEW-QA.md](PR-REVIEW-QA.md). This page covers what you see and what you can configure.
+
+## What the review contains
+
+The review appears in three places: one sticky comment on the PR, the check result, and the run page in the app.
+
+### Sticky comment
+
+localdost keeps one comment per PR and edits it on each new commit. The comment is written for a manual tester, in plain words (KAT-376). It reports results and never gives merge advice: the team decides the merge. From the top:
+
+1. **The first line**, in bold, says what happened and whether anything blocks: `localdost: nothing blocks this PR. 2 things need a person to try.` A failed test says `the check failed. 1 test failed.`; a failure the verdict blames on the test or its setup adds that localdost reads it that way and not as a bug in the PR.
+2. **One line of counts**, in plain words: `1 automated test passed (the home page loads). No automated test covers Search yet.` The counts appear here once. The words in brackets are what the test checked, left out when the test's name holds a tool word such as smoke or spec. A skipped test did not run, so it is never counted as ran: `85 automated tests ran and passed. 53 were skipped (they do not apply to this project).` With a failure the line is `3 automated tests ran: 2 passed, 1 failed.` followed by the skipped sentence, and a run where every test was skipped says `All 53 automated tests were skipped (they do not apply to this project).` The line of a Tests repo keeps the rule: `Ran 85 tests from e2e: 85 passed. 53 were skipped (they do not apply to this project).`, and the shared build note shows only when a test ran. Details under What I tested repeats it (`Ran 85 selected tests: 85 passed, 0 failed. 53 were skipped (...)`). The consistency check that runs before a report is drafted reads these numbers back: a text that says more tests ran than passed, failed and flaky tests add up to, a different skipped count, or a line that lists skipped tests under a ran count holds the draft, a sentence in the dot's own notes that gives a test count is left out of the report because the counts come from the run, and the facts the safety reviewer reads say that a skipped test did not run (KAT-402). Tests of the PR's own CI show as `The build ran 12 tests: 12 passed.`
+3. **Areas**, only when the change touches two or more: **What this change does** (2 or 3 sentences, no file names) and a table with each area's size and risk. A change in one area explains itself in Details. See [Change summary](#change-summary).
+4. **Bugs found**: each bug in one line, then what was expected, what happened, how often it failed (`Failed 3 of 3 times.`), and what became of its bug report. A bug is a test that failed or a risk that a failing check reproduced. Each bug is also drafted as a defect, see [Defects](#defects). Steps and evidence are in Details.
+5. **Questions for the team** and **Behavior changes to confirm**, when a finding has no concrete expected result or the PR description, a linked ticket or the docs say the change does it on purpose. A question that goes to a linked ticket as a draft comment is asked once per PR and set of questions, so a push that changes nothing about them draws no second draft; a PR is its repo plus its number, so PR 12 of one repo of a dot and PR 12 of another repo each ask (KAT-400).
+6. **Please try by hand**: a task list. Each item is something a person tries, with its steps, its test data and its expected result under it. A risk the review could not confirm, or one that no check with steps covers, is an item too, worded from the PR description, a linked ticket or the docs when they say the change does it on purpose (`If users should see something else here, this is a bug.`). A check with no steps and no expected result, and an automated test that did not run, are no tasks for a tester: they are listed in Details.
+7. **Tests to update**: specs that a ticket of the PR changes and the PR leaves alone, as tick items for the author. An item names the test and links it when the repo is known; the file path is in Details.
+8. **Who ticks** (the PR author or QA, anyone with write access), then the footer: `Open the full check` when the server has a public address, and how to check again. `Reply /localdost review to check again` shows only on a server that GitHub has sent a signed event to; a server that only polls says `Push a new commit to check again.`
+9. **Details for developers**, collapsed: bugs with steps and evidence, risks, what ran (the selected run and the `Tests in CI: 12 passed (job unit) on 7c6d5e4.` line, see [CI results](#ci-results)), open checks that are no task for a tester, suggestions (test gaps to close and tests the change may break), the changelog note, the test notes from the model, the change summary, risk ids with why and user impact, check ids with steps, expected results, test data and where to start, impact areas and test gaps, the regression scope, the head commit, coverage, what the verifier did, the engines and timings, confidence, and release-note impact.
+
+   When a repo has a `CHANGELOG.md` and the change touches `src/` or `client/` (tests excluded) without adding a line under Unreleased, Details carries a **Note** that says so. It is a note, not a task item and not a bug, and it does not count as an untested check.
+
+A comment for a run with no QA review has the same layout with the verdict's test ideas as the things to try by hand. Each idea is a tick item with a hidden id, so its tick carries over, and the comment says who ticks. A failure found only by reading the code says `the check failed. 1 bug found by reading the code.`
+
+When a new push arrives, the comment becomes `This report is out of date. localdost is checking your latest push now.`, then what to do if no new report appears within 15 minutes, and the old report folds under `Earlier report, for the previous push`. Ticks in the old report carry over to the new one.
+
+Sentences stay under 20 words in active voice, and ids, shas, file paths, scope and verifier counts stay inside Details. Risks below the display threshold are counted in Details. When the comment would pass GitHub's size limit, Details parts drop from the end first.
+
+A review is **incomplete** when the reviewer could not see every changed range, for example when GitHub sent no patch for a file. Details names the reason.
+
+The Inbox, the run page, and report pages render the comment as GitHub does: task lists, tables, details and summary, code, and links. When the dot's GitHub connection works, the preview is GitHub's own HTML (`POST /markdown`, mode `gfm`, the repo as context), cached by content hash. Otherwise a local sanitised renderer shows it. **Show Markdown** shows the raw text. localdost markers (`<!-- localdost:... -->`) never show.
+
+### Change summary
+
+A QA engineer reads the change summary instead of every diff (KAT-144). The reviewer writes **What this change does** under ASD-STE100 rules (Simplified Technical English), which the review prompt carries as a rule block:
+
+- 2 or 3 sentences. A description has at most 25 words, and an instruction has at most 20 words and starts with the verb.
+- Active voice, one word for one meaning, and everyday words. A list of banned words with their plain replacements (`utilize`, `leverage`, `refactor`, `functionality` and others) lives in `src/review/ste.ts`.
+- No file names, paths, function names, ids or line numbers.
+
+The answer check sends a broken rule back to the reviewer. If the text still breaks a rule, localdost keeps only the sentences that pass. When fewer than two pass, or no model ran, a template writes the text from the areas, so the summary is never empty.
+
+The areas and their numbers do not come from the model. Every changed file goes into exactly one area: the review's strongest impact area that lists it (owner map areas from `.localdost/review.yml` included), else a path group (Documentation, Automated tests, Build and setup, Other code).
+
+| Field | Rule |
+|---|---|
+| Size | Small: under 50 changed lines and at most 3 files. Large: 400 lines or more, or 15 files or more. Medium: the rest. Lock files and build output do not count. |
+| Risk | High when the area has a critical or high risk the verifier did not refute; medium for a medium risk; low otherwise. A large area with no risk found is medium. |
+
+The same summary leads the run page and the PR comment. When a new commit makes the comment stale, the stale notice moves to the top with the earlier test result.
+
+### Defects
+
+Every bug becomes one `file_defect` action: a Bug in the dot's Jira project, an issue in its Linear team, or a GitHub issue on the repo when the dot has no tracker. The default rule for **File defects** is **Ask**, so each draft waits in the Inbox with its steps, expected and actual results, and evidence. A bug is drafted once per PR: later pushes link the same draft or the filed issue, and a rejected draft is not drafted again. A test that failed on every retry in a product failure drafts an `issue_create` instead (rule **Create issues**, default Ask) with a signature marker, and a repeat of the same failure becomes a comment on the open issue. See [TRACKERS.md](TRACKERS.md#defects-filed-as-issues).
+
+A risk that no test reproduced is not a bug, but it is still filed when the reviewer can point at the code: one cited file and line, and one input that breaks it, such as "a cart with two items and the coupon SAVE10 entered twice". That draft is labelled **found by reading the code**. The label shows on the Inbox card, in the report ("Found by reading the code, not reproduced (lower confidence)"), in the draft body next to the file and line and the failing input, and as a label on the tracker issue. Confidence is lower than for a failing test. The risk stays a "risk to test" in the headline and the counts. A vague risk (no input, or "some edge cases") is never filed: it stays a note under **Risks to test**. A decision the PR states on purpose, or an Expected that is a design question, still wins over both.
+
+Every defect carries the label `localdost`, and a Linear defect goes to the project named by the tracker's `projectName` when the dot has one. [TRACKERS.md](TRACKERS.md#defects-the-dot-files) has the details.
+
+Before it drafts a bug that has no draft yet, the dot looks for an open issue about the same bug in the tracker: the open issues labeled `localdost` in Jira and GitHub, the open issues of the team (and project) in Linear. An issue matches when it names the same PR or the same repo, and its title shares at least 60 percent of the words with the new title, or it cites the same file and line, or the same file in the same PR. On a match the dot drafts no second defect. It adds the new evidence and the head commit as a comment on the existing issue, once per head commit, and the report links that issue as filed. The comment is a `ticket_comment` action, so its rule applies: with the default **Ask**, it waits in the Inbox. If the tracker cannot be read, or the dot runs in demo mode, the dot drafts the defect as usual.
+
+### CI results
+
+The dot's own run often covers only the smoke tests, so the unit tests of the PR would show as not run. localdost also reads the PR's own CI. It lists the check runs of the reviewed head commit (`GET /commits/{sha}/check-runs`), keeps those whose head sha is the reviewed head, and reads the log of each finished GitHub Actions job (5 MB at most per log). A job whose conclusion is success, neutral, or skipped counts no failed test, and linter or compiler summary lines such as `✖ 31 problems (0 errors, 31 warnings)` are not tests. Passing and failing test lines are parsed from vitest and jest check marks, `ok N - title` and `not ok N - title` TAP lines, node:test marks, Playwright list lines, and `x title` lines.
+
+A checklist item with an existing test that the dot's run left not run is matched against those lines by test title (case-insensitive substring, taken from the item's `existingTitle`) and, when the log prints a file, by file path. A match becomes `Passed` or `Failed` with the note `passed in CI job <name>`. A job that is still queued or running settles nothing: its items stay not run, and the reason says the CI job is still running. The comment also gets the line `Tests in CI: N passed (job X) on <sha7>.` (only jobs that printed a test line are listed), and Details lists each job with its conclusion and the counts parsed.
+
+A missing permission (the check runs need the Checks read permission), an API error, or a log that cannot be read means no CI evidence for that part. It never fails a review. A manual tick does not replace a CI result.
+
+### Check result
+
+In App mode, the check run named "localdost" carries the same report as its text and adds a line annotation for each shown risk and each test at risk, at the first line the item cites. In PAT mode, GitHub allows only a commit status, so you get the conclusion and a title without annotations. See [GITHUB.md](GITHUB.md).
+
+The title is the result in under 60 characters, which is where GitHub cuts a longer one: `1 test failed: discount applied at checkout`, `Passed. 2 things to try by hand`, `Passed`, `Tests did not run`. The body opens with one sentence on what users may see and whether the check blocks merging (a failed check "blocks merging only if your repo requires it"; any other says it does not block), then the counts, each bug with its areas, what was expected and what happened, then the same sections as the comment: Questions for the team, Behavior changes to confirm, what is left to try by hand and Tests to update, each cut at a few lines with an `And N more` line. The developer details are in the check text under `Details for developers`.
+
+The conclusion follows results: `failure` when a selected test failed. A bug that only the review found (no failing test) is `failure` only when `blockOn` is set and the bug is at or above that severity, and `neutral` otherwise. The conclusion is also `neutral` when the tests could not run, and `success` when every selected test passed and no bug was found.
+
+### Run page
+
+On PR and merge runs the page opens with the change summary: what the change does, the size of the whole change, one chip per area with its size and risk, and the test result line from the PR comment. A chip filters **Changed files** to that area and opens its group. **Changed files** groups the files by area, collapsed, with a file count and +/- per group, and **Raw diff** shows the unified diff of the files in view in one click. Screenshots in the Artifacts tab and the Browser view carry the test title in plain words, such as `Chat: a suggested question streams a reply with a tool card`, with the file name below it. Runs from before the change summary get one built from the stored diff with the same rules.
+
+The run page also shows the same sections as the comment with live test results. Each risk has a **Not a risk** button. A mark hides nothing on the PR. It saves the risk and its diff hunks under `<LOCALDOST_DATA_DIR>/eval-candidates/`, so a maintainer can turn a false positive into an eval fixture.
+
+## How the review works
+
+1. **Scope.** localdost collects the changed lines from the PR files, the raw diff, or `git diff` in a checkout. Common secret shapes in the diff are masked before anything reaches a prompt.
+2. **Impact.** Changed files map to impact areas through `.localdost/review.yml`, `CODEOWNERS`, routes, the import graph, and spec links.
+3. **Signals and lenses.** Deterministic checks look for untouched tests, uncovered lines, disabled tests, removed assertions, changed test config, selector drift, changed routes, and more. The QA lenses below match changed lines and add their checklists to the prompt.
+4. **Security and robustness pass.** See [below](#security-and-robustness-pass).
+5. **Reviewer pass.** A model writes risks, What to test items, tests at risk, test gaps, the regression scope, and release-note impact. Every risk must cite a changed line.
+6. **Verifier pass.** A second model pass checks each risk against the code and marks it confirmed, uncertain, or refuted. Turn it off with `verify: false`.
+7. **Script checks and merge.** Code checks reject citations of lines the change does not contain, ask the model to correct them, apply confidence floors, and merge the passes.
+
+The review uses the dot's model backend. With `LOCALDOST_LLM=mock`, it returns canned output.
+
+### Rule-based fallback
+
+Without a model, or when the model pass fails, a rule-based review stands in. It only reports what the changed lines show. Each risk must cite a changed line of a changed file, and the text of that line must match a pattern for the risk kind: money (`amount`, `price`, `total`, `round`, `currency`, `tax`), auth (`auth`, `token`, `permission`, `role`, `session`, `acl`, `can[A-Z]`, `isAdmin`, `login`), dates (`new Date`, `Date.`, `moment`, `dayjs`, `toISOString`, `timezone`), concurrency (`await`, `Promise`, `setTimeout`, `debounce`, `async`, `lock`, `queue`), migration (`ALTER`, `CREATE`, `DROP`, `migrat`), feature flags (`flag`, `feature`, `toggle`, `enabled`), and a few more. A risk that fails is dropped, together with its checklist items and test gaps, and its signal is listed as dropped with the reason `Rule-based review: the line does not show the pattern`.
+
+A risk that stays is a guess. Its severity is `medium` at most, `low` when only a comment or a string shows the pattern, and its text starts with `Rule-based guess, not verified:`. The one-letter-search guard (`<x>.length >= N`) fires only when `<x>` names text a user types (`q`, `query`, `search`, `term`, `text`, `input`, `value`, `keyword`, `filter`, `name`, `title`, `str`, `s`, or a name made from `trim()`, `toLowerCase()`, or a debounce hook of one of those) and the line is source code. The summary names at most three product areas and never lists test files.
+
+## Repo files
+
+Both files are read from the **base** branch of the PR. A PR cannot change the rules of its own review. When a PR edits one of them, the review uses the base copy and notes it.
+
+### .localdost/review.yml
+
+An owner map that names your product areas, the paths that belong to each area, the user flows they carry, and the specs that test them:
+
+```yaml
+areas:
+  - name: Checkout
+    paths:
+      - src/checkout/**
+      - src/payments/**
+    flows:
+      - Pay with a saved card
+      - Apply a coupon
+    specs:
+      - tests/checkout*.spec.ts
+  - name: Search
+    paths: [src/search/**]
+    flows: [Search by product name]
+    specs: [tests/search.spec.ts]
+```
+
+| Key | Required | Meaning |
+|---|---|---|
+| `areas[].name` | yes | Area name shown in the review |
+| `areas[].paths` | yes | Globs of files in the area |
+| `areas[].flows` | no | User flows the area carries. The reviewer uses them in risks and What to test items |
+| `areas[].specs` | no | Globs of specs that cover the area. They run when the area changes |
+
+A file in the wrong shape is skipped, and the review details say why. Without the file, the review falls back to `CODEOWNERS` teams, routes, and paths.
+
+### .localdost/review.md
+
+Free-text instructions from the repo owners, for example:
+
+```markdown
+- Prices are integers in cents. Flag any float arithmetic on amounts.
+- The admin app under src/admin is internal. Rate its risks one level lower.
+- Do not flag copy changes in src/i18n unless a placeholder changed.
+```
+
+The reviewer reads the file as quoted data that widens or narrows what it flags. It cannot change the output format or the rules of the review.
+
+## Security and robustness pass
+
+The pass starts when the changed lines touch process execution, git clone or checkout, file system paths or copies, tokens or auth, webhooks or event intake, deletes, temp folders, the clock in a test, an approval or status path, or the result of an exported function. It adds a fixed checklist to the reviewer prompt, with only the questions that apply:
+
+| Question id | Kind | Question |
+|---|---|---|
+| `attacker_choice` | security | Can an attacker choose what runs or which commit is checked out (forged event fields, unsigned webhooks)? |
+| `path_safety` | security | Path traversal, a symlink on the way, a swap between the check and the use (TOCTOU), a special file such as a named pipe? |
+| `delete_in_flight` | robustness | What happens to in-flight runs, jobs and approvals when their object is deleted? |
+| `caller_access` | security | Who may call this: no token, a phone token, the owner token? |
+| `changed_contract` | robustness | Which callers of a function whose result changed still expect the old one? |
+| `temp_cleanup` | robustness | New temp folders or files without cleanup on every path? |
+| `time_dependent_test` | robustness | A test that depends on "today", a local time, or a date near midnight? |
+| `open_status` | robustness | An approval, check or status with no close path when it is rejected, expires or fails? |
+
+Script detectors in `src/review/security.ts` add leads for five questions: a git checkout of an event commit with no head or signature check in the file, a copy by name after path checks with no file descriptor or `O_NOFOLLOW`, an exported `delete<Thing>` whose body names runs, jobs or approvals, a changed return or throw with every unchanged caller listed from the import graph, and a clock-bound assertion in a test that uses the real clock. Each lead cites a changed line and a failure scenario. It goes to the reviewer as a signal, and the rule-based review turns it into a risk. The other questions go to the reviewer model with no lead.
+
+A risk from the pass has kind `security` or `robustness` and source `security:<question id>`. It is a risk to test, not a bug. Even when the verifier confirms it, it becomes a bug only when a test reproduces it, for example a failed test in the run.
+
+## Lenses
+
+A lens is a checklist for one kind of change. When a lens matches a changed line, its text goes into the prompt, at most four lenses per review, most specific first. Each lens has a confidence floor: a risk that the lens raised needs at least that confidence to show.
+
+| Lens | Fires on | Floor | Severity hint |
+|---|---|---|---|
+| `qa-auth` | Login, session, role, or permission logic | 0.65 | critical |
+| `qa-money` | Money arithmetic, rounding, currency, or sign | 0.65 | critical |
+| `qa-migration` | A schema or data migration | 0.65 | critical |
+| `qa-check-then-act` | A read followed by a write on shared state, or parallel work on it | 0.7 | high |
+| `qa-response-shape-break` | A response, serializer, or returned object that changed its fields | 0.7 | high |
+| `qa-dates` | Date, time zone, or calendar arithmetic | 0.6 | high |
+| `qa-double-submit` | An async UI action, timer, debounce, or throttle | 0.65 | high |
+| `qa-feature-flag` | A feature flag, an experiment, or its default | 0.6 | high |
+| `qa-control-has-no-effect` | UI state that a control sets | 0.65 | medium |
+| `qa-validation` | Input validation rules | 0.6 | medium |
+
+The lenses live in `src/agent/qa-lenses/` of the localdost checkout. To add one, add a Markdown file there with YAML frontmatter and restart the server:
+
+```markdown
+---
+name: qa-tax
+kind: money
+description: Tax rate or tax rounding changed on a changed line
+triggers:
+  files: ["src/**/*.ts"]
+  line_regex: 'taxRate|vatRate|roundTax\('
+confidence_floor: 0.7
+severity_hint: high
+---
+
+A change touches how tax is computed. Check ...
+
+## What to test
+
+- ...
+```
+
+| Frontmatter key | Meaning |
+|---|---|
+| `name` | Kebab-case name |
+| `kind` | One of the signal kinds, for example `money`, `auth`, `dates`, `concurrency`, `migration`, `feature_flag`, `validation_changed`, `ui_state`, `response_shape` |
+| `description` | One line shown in the review |
+| `triggers.files` | Globs of files the lens reads. Empty means every file |
+| `triggers.line_regex` | Matched against added lines |
+| `triggers.removed_regex` | Matched against removed lines |
+| `triggers.classes` | File classes where the lens fires once per file |
+| `triggers.line_flags` | Regex flags, from `i`, `m`, `s`, `u` |
+| `confidence_floor` | 0 to 1, default 0.6 |
+| `severity_hint` | `critical`, `high`, `medium`, or `low` |
+| `skip_comments` | `false` to also match comment-only lines |
+
+A lens needs at least one of `line_regex`, `removed_regex`, or `classes`. A lens file with an error breaks every review until you fix it, so run `npm test` after you add one.
+
+## Options
+
+Each dot has PR review options. Change them in the app under the Change reviewer stage of the dot dialog (Settings on the dot page, then Settings under Change reviewer), or with `PUT /api/dots/:id/review`:
+
+| Option | Default | Meaning |
+|---|---|---|
+| `enabled` | `true` when the dot has a repo | Runs the review |
+| `verify` | `true` | Runs the verifier pass |
+| `minConfidence` | `0.6` | Lowest confidence a risk needs to show. A lens floor can raise it |
+| `showFrom` | `medium` | Lowest severity listed in the comment and annotated |
+| `blockOn` | `null` | A severity. A confirmed risk at or above it is recorded as blocking on the run and fails the check. Without it, a bug found only by the review leaves the check `neutral`; a failed test still fails it |
+| `maxPromptKb` | `160` | Prompt size budget, 16 to 1024 |
+| `toolMode` | `none` | `none` or `read`. Stored for a later version; the review does not read it yet |
+
+```bash
+curl -X PUT http://localhost:4319/api/dots/<dot id>/review \
+  -H 'content-type: application/json' \
+  -d '{"blockOn":"high","showFrom":"low"}'
+```
+
+## Ticking the checklist
+
+Each item under Please try by hand and Tests to update in the sticky comment is a checkbox, and every tick stays when the dot later rewrites the comment. When a tester runs a check from the review by hand (an item with its steps and expected result) and ticks it on GitHub, localdost also records a manual pass on the run. A tick on a possible bug, on a Tests to update item, or on a thing to try in a comment without a review stays in the comment and records no result on the run. localdost counts a tick only when:
+
+- The GitHub Issue comment webhook reaches localdost. Polling does not see comment edits.
+- The person who edited has write, maintain, or admin access to the repo. localdost asks GitHub for the editor's permission and ignores the edit when GitHub cannot answer.
+- The editor is not a bot.
+
+You can also tick an item on the run page in the app, which calls `POST /api/runs/:id/review/tested`. An item that the test run already covered keeps the runner's result and cannot be ticked.
+
+## Evals
+
+`test/fixtures/qa-review/` holds 17 eval fixtures: a diff, the head files, and the expected review for each. `test/fixtures/qa-review/dogfood/` holds 5 of them: golden cases from the real misses in `docs/DOGFOOD-LEDGER.md` on the `dogfood/fp-fn-ledger` branch, each the actual diff at that head. A golden case passes when the review names the risk. `--only dogfood` runs the group. `npm test` runs them through the mock backend to check the plumbing. `npm run eval:qa-review -- --backend claude-cli --runs 3` (or `LOCALDOST_LLM=claude-cli npm run eval:qa-review`) runs them against a live model and writes the median metrics to `test-results/qa-review-eval.json`. `--compare <old.json>` shows what changed against an earlier run, `--only <name>` picks fixtures, and `--gate` exits 1 when a target fails.
+
+### Eval results
+
+2026-10-08, Claude CLI backend (`sonnet` for reviewer and verifier), 12 fixtures, verifier on.
+
+| Metric | Target | Before | After |
+|---|---|---|---|
+| Precision after verifier | 0.8 or more | 0.75 | 1.0 |
+| Recall | 0.7 or more | 0.89 (8 of 9) | 1.0 (9 of 9) |
+| Negative-control false positives | 0 | 0 | 0 |
+| Verifier recall loss | 1 or less | 1 | 0 |
+| Must-not-flag hits | 0 | 1 | 0.5 |
+| Coverage complete | 12 | 12 | 12 |
+| Precision before verifier | | 0.77 | 0.83 |
+
+Before is one run on the prompts at `b4a9956`; after is the median of two runs. The misses were a refuted prompt-injection risk and duplicate risks for one root cause (a caller, a type or a skipped test raised as its own risk). The fixes: the merge stage folds risks that share a cited line and test-only risks into the product risk, the verifier may not drop a prompt-injection risk, and the reviewer prompt asks for one risk per root cause and no contrast citations. The remaining must-not-flag hit is `auth-guard-removed` citing the intact sibling guard as further evidence in one run of two.
+
+The same build reviewed the demo shop PR (duonghaijoe/localdost-demo-shop#1) from a fresh clone. It flagged the double-subtracted total, the additive member and promo stacking, and the free-shipping check on the pre-discount subtotal, and asked for a Gold plus TRAIL10 checkout test.
+
+### Readability
+
+`npm run eval:qa-review -- --readability` also renders each fixture review in the old comment format (frozen in `src/review/baseline/render-v1.ts`) and in the current one, and scores both. `--db <copy of localdost.db> --pr 3 --pr 4` adds the latest reviewed run of those PRs; open a copy, never the live database. `--readability-only` skips the fixture metrics, and `--gate` fails when the gate fails.
+
+The script rubric checks the part a reader sees without opening Details:
+
+| Check | Passes when |
+|---|---|
+| Summary first | The first line says what happened and whether anything blocks (an older comment: what was tested, the bugs found, and what is not tested yet) |
+| No merge advice | Nowhere does the comment say safe to merge, hold, approve, or merge readiness |
+| Bugs reproducible | Every bug says what was expected and what happened (an older comment: also steps and evidence) |
+| Checklist actionable | Every Please try by hand item opens with the action, carries its steps or expected result under it, and stays under 45 words (an older Not tested yet item says why and how) |
+| Plain words | At most 1 in 10 sentences is over 20 words |
+| No jargon | No internal ids, regression scope, verifier, coverage, confidence, or shas |
+| Length | 350 words or fewer, not counting the change summary block |
+| STE summary | The opening lines are short, active, free of banned words and file names; a change in two or more areas also shows What this change does with its areas table (an older comment opens with it in 2 or 3 sentences) |
+
+Then a judge compares the two comments for each sample, once with the new comment in slot A and once in slot B, and gives weights to A and B instead of a bare winner. `--judge codex` (default) runs `codex exec -m gpt-5.5 --sandbox read-only --ephemeral -c "mcp_servers={}"`; `--judge claude` runs `claude -p` when Codex is unavailable. The gate passes when every new comment passes all eight checks and the mean weight for the new format is above 0.5.
+
+2026-10-08, 12 fixtures (mock reviewer) and the latest reviewed runs of PR #3 and PR #4 from a copy of the dogfood database:
+
+| | Old format | New format |
+|---|---|---|
+| Script rubric, mean | 0.43 (3 of 7) | 1.0 (7 of 7 on all 14) |
+| Codex weight, both orders, mean | 0.08 | 0.92 |
+| Samples where Codex prefers it | 0 of 14 | 14 of 14, no order flips |
+
+The lowest Codex weight for the new format was 0.735, on `docs-only`, where both comments are short. A fresh-context Claude Opus judge on 6 of the samples, run while Codex was over its spend cap, gave the new format 0.86 to 0.94.
+
+Every PR check also scores its own comment and records a Readability step on the run, so each dogfood round shows the score.
+
+#### STE check (KAT-144)
+
+`src/review/ste.ts` holds the ASD-STE100 checks a script can run. A sentence that starts with a QA verb (test, check, open, run, press) is an instruction with a 20-word limit; any other sentence is a description with a 25-word limit. Passive voice is a be-verb or "get" followed by a past participle, minus a short list of adjectives such as "based". The check is a heuristic: it misses a passive without a be-verb and can flag a rare adjective. The banned words and their replacements are the same list the prompt carries. "One meaning per word" stays a prompt rule, as no script can check it.
+
+The eval reports, for the old and the new comment, the sentences over their STE limit, the passive phrases and the banned words in the visible text, and gates the change summary block on all of them.
+
+2026-10-08, after the change summary. Codex was over its spend cap, so a fresh-context `claude -p` Opus judge ran both slot orders on 4 fixtures (mock reviewer) and the latest reviewed runs of PR #12 and PR #4 from a copy of the dogfood database. Those two runs predate KAT-144, so their summary comes from the stored diff and the review summary, as on the run page.
+
+| Sample | Rubric old | Rubric new | STE old: long, passive, banned | STE new: long, passive, banned | Judge weight for new |
+|---|---|---|---|---|---|
+| docs-only | 4 of 8 | 8 of 8 | 0 of 7, 0, 0 | 0 of 5, 0, 0 | 0.90 |
+| money-rounding | 3 of 8 | 8 of 8 | 0 of 14, 0, 0 | 0 of 19, 0, 0 | 0.865 |
+| pure-refactor | 4 of 8 | 8 of 8 | 0 of 10, 0, 1 | 0 of 10, 0, 0 | 0.85 |
+| selector-drift | 3 of 8 | 8 of 8 | 0 of 14, 0, 0 | 0 of 21, 0, 0 | 0.85 |
+| PR #12 | 1 of 8 | 7 of 8 | 7 of 30, 2, 0 | 3 of 49, 0, 0 | 0.835 |
+| PR #4 | 2 of 8 | 7 of 8 | 1 of 41, 0, 0 | 5 of 57, 0, 0 | 0.865 |
+
+All 12 fixtures pass 8 of 8 with the script rubric alone. The judge preferred the new comment on every sample, with no order flips. PR #12 and PR #4 fail only Length (467 and 514 words outside Details, limit 350). They failed it on `main` before this change too (465 and 512 words): the long part is their Not tested yet list, and the change summary block does not count toward the limit.
+
+### Prompt comparison
+
+`npm run eval:qa-review -- --backend claude-cli --variant <name>` runs the 12 fixtures through the shipped reviewer prompt and through a variant from `src/review/prompt-variants.ts`, renders both reviews as the sticky comment, and has a judge compare the two comments for each fixture. `--baseline <name>` picks the other side (default `current`). `--reviews <file>` caches each generated review, so a second judge pass does not rerun the reviewer.
+
+The judge follows the ranking rules for comparing candidates:
+
+- It sees the diff and both comments together, never one comment alone.
+- Each pair runs twice, with the variant in slot A and then in slot B. A judge that always prefers one slot scores 0.5 and the run counts it as an order flip.
+- For readability, correctness and overall, the judge spreads a weight of 1 over five grades, from A much better to B much better. The score is the expectation over the grades, from 0 (baseline much better) to 1 (variant much better).
+
+The readability rubric is the reader check from the table above: testing summary first, actionable Not tested yet items, plain words, no jargon outside Details, and length. The correctness rubric asks whether each bug cites a changed line that shows the problem, whether any finding has no support in the diff, whether the comment catches the regressions the diff introduces, and whether the suggested tests would catch them. Beside the judge, the script reports the readability rubric for both sides, how many shown risks cite a changed line, and the section 7.2 recall and precision.
+
+`--judge codex` (default) asks Codex and falls back to `claude -p` for any call where Codex fails. `--judge claude` asks only Claude. The variant wins clearly when its mean overall score is 0.6 or more, its correctness score is at least 0.5, its script recall, precision and must-not-flag hits are no worse, and at most a quarter of the fixtures flip with the slot order. A clear winner is then gated on the section 7.2 targets, which stay pass or fail; `--gate` exits 1 when the variant fails one.
+
+2026-10-08, Claude CLI reviewer and verifier (`sonnet`), Codex judge (`gpt-5.5`, all 24 calls answered by Codex). Baseline `current` against the `actual-result` variant. The variant asks for the observed wrong result in userImpact, which the comment prints as the bug's Actual line, 3 to 5 short steps, a concrete expected value, the common term for the trigger condition, and a before-and-after statement in why.
+
+| | Baseline | Variant |
+|---|---|---|
+| Readability rubric, mean | 0.964 | 0.964 |
+| Shown risks on a changed line | 11 of 11 | 11 of 11 |
+| Recall | 9 of 9 | 9 of 9 |
+| Precision after verifier | 0.909 | 0.909 |
+| Section 7.2 gate | pass | pass |
+
+| Judge expectation for the variant | Score |
+|---|---|
+| Readability | 0.518 |
+| Correctness | 0.500 |
+| Overall | 0.509 |
+| Fixtures where the slot order flips the winner | 5 of 12 |
+
+No clear winner, so `QA_REVIEW_SYSTEM` stays as shipped and the variant stays in `prompt-variants.ts` as a recorded candidate. The variant won `flag-default-flip` (0.78) and `date-timezone` (0.69), where the Actual line now names what the tester sees. It lost `auth-guard-removed` (0.32) and `money-rounding` (0.35): Codex found a delete-route test gap that the diff does not support, and example totals that the diff does not support. `false-positive-bait` (0.38) went to the baseline for a sharper visible summary. An earlier judge pass on the same reviews gave 0.528 overall with 7 flips, so one pass moves the mean by about 0.02. Both prompts already pass the script rubric on 11 or 12 of 12 fixtures, which leaves little room for a prompt change to separate them.
+