dsh-vet 0.3.0 → 0.4.0

package/README.md

@@ -34,12 +34,32 @@ npx dsh-vet --json <specifier>   # dsh-vet/v1 report on stdout
 npx dsh-vet --strict <specifier> # exit 1 on findings >= high (confidence >= medium)
 npx dsh-vet --rules dep.install-scripts <specifier>
 npx dsh-vet validate <report.json> # check a report against the contract
+npx dsh-vet diff base.report.json head.report.json --json
 ```
 
 Any completed report exits `0` — grades describe findings, they do not gate.
 Scanner failures exit non-zero. The scanner runs locally, reads the npm
 registry for dependency metadata only, and never transmits audited code.
 
+Every report records what the scan actually covered (`x-dsh-vet` scan
+context: rule profile, coverage, content digests, stable finding
+identities); human output and badges show coverage alongside the grade,
+and a missing context reads as `unknown`, never complete.
+
+### Comparing two reports (release review)
+
+`dsh-vet diff` compares two reports produced by the same scanner version
+and rule profile over complete scans, and reports what changed:
+[`dsh-vet/diff/v1`](docs/report-diff-v1.md) — added/removed/changed
+findings by stable identity, behavior-observation deltas, both sides of
+every transition. Exit `0` for a comparable result regardless of risk
+changes, `1` when the pair cannot be trusted to describe the same
+subject under the same checks (with explicit reasons), `2` on usage or
+invalid reports. Two local-directory scans additionally need
+`--subject <label>`. An incomparable pair is a prompt to rescan both
+artifacts with the same configuration — never a claim that nothing
+changed.
+
 Shipped rules (each with a public rationale under
 [`docs/rules/`](docs/rules)):
 
@@ -64,13 +84,19 @@ from your repo, so its value is auditable through git history and no badge
 service is involved:
 
 ```yaml
-- uses: rogerdigital/dsh-vet/action@v0.2.0
+- uses: rogerdigital/dsh-vet/action@v0.4.0
   with:
     specifier: '.'
     commit-report: true
 ```
 
 Every run uploads the full report as an artifact; PRs get a single
+comment, edited in place. Set `baseline-report` to a report scanned from
+the merge base and PRs additionally show what changed since it — see
+[comparing releases](action/README.md#comparing-releases) for the
+trusted-baseline recipe, and the
+[pilot record](docs/release-risk-pilot.md) for how the comparison
+behaved on real release pairs.
 edited-in-place findings comment. Badge snippet and all inputs:
 [`action/README.md`](action/README.md). The `dsh-vet badge <report.json>`
 command renders the shields endpoint JSON if you wire CI yourself.

package/docs/adopt-marketplace.md

@@ -48,9 +48,17 @@ grade from the findings, so an emitter cannot assert a grade its evidence
 does not support:
 
 ```sh
-npx dsh-vet validate report.json && echo trustworthy-shape
+npx dsh-vet validate report.json && echo structurally-conformant
 ```
 
+`validate` proves structural conformance only. It does not prove
+completeness (that the emitter ran every check and omitted nothing),
+artifact identity (that the report describes the plugin version you are
+serving), or provenance (who produced the report). For those guarantees,
+fetch reports through channels you control — the Action's report branch in
+the author's own repo, or a scan you run yourself — and treat reports from
+unverified emitters as unreviewed claims.
+
 ## Consumer rules (from the spec)
 
 - **Ignore unknown fields** — emitters may add `x-`-prefixed extras; tolerate

package/docs/dsh-vet-v1.md

@@ -101,13 +101,43 @@ initially broke vendor-prefixed check ids — do not repeat that.
 
 | Grade | Condition (over findings with confidence ≥ `medium`) |
 |---|---|
-| `A` | none, or only `info`/`low`-severity findings |
+| `A` | no graded findings — the report is empty, or every finding is `info`-severity or `low`-confidence |
 | `B` | worst graded finding is `low` |
 | `C` | worst graded finding is `medium` |
 | `D` | worst graded finding is `high` |
 | `F` | at least one `critical` |
 | `X` | scan incomplete or errored — never presented as the plugin's grade |
 
+## What validation proves — and what it does not
+
+`dsh-vet validate` checks **structural conformance** with this contract:
+field types and enums, rule-id shape, evidence presence, the deterministic
+sort, and — the load-bearing check — that `summary` is exactly what the
+`findings` derive. A conformant report cannot assert a grade its evidence
+does not support.
+
+It does not, and cannot, verify:
+
+- **Emitter honesty** — whether the emitter omitted findings or fabricated
+  evidence. Structural consistency is not evidence that the audit was
+  thorough.
+- **Completeness** — v1 records nothing about which files or rules a scan
+  covered. A subset scan's report is structurally indistinguishable from a
+  full one.
+- **Artifact identity** — nothing ties a report to the artifact you are
+  holding. `target.resolved.integrity` is recorded by the emitter, not
+  checked against your copy.
+- **Provenance** — who actually produced the report, and whether it changed
+  in transit.
+
+Those guarantees come from delivery channels, not from the report's shape:
+see [emitters.md](emitters.md) for how emitters are verified and
+[adopt-marketplace.md](adopt-marketplace.md) for consumer guidance. The
+optional scan-context extension ([scan-context-v1.md](scan-context-v1.md))
+records coverage and content identity; the reference scanner begins
+emitting it in a later release, and until a report carries it, absent
+coverage data means **unknown**, never complete.
+
 ## CLI recommendations (non-normative)
 
 Emitters that ship a CLI should exit `0` whenever a report was produced —

package/docs/emitters.md

@@ -17,7 +17,9 @@ honesty.
    `dsh-vet validate` (`npx dsh-vet validate <report.json>`). Either path
    guarantees the derived summary, the deterministic sort, and well-formed
    rule ids; an emitter that hand-assembles reports and skips validation is
-   not verified.
+   not verified. Validation proves structure, not honesty — it cannot
+   detect omitted findings, which is why the remaining items on this list
+   exist.
 2. **Determinism.** Two runs over the same artifact with the same emitter
    version produce identical reports, `scanner.ranAt` aside.
 3. **Conservative severity.** Findings follow the severity ladder in the

package/docs/outreach/README.md

@@ -1,12 +1,15 @@
 # Outreach drafts
 
-Ready-to-send drafts for the v0.3 announcement round. Each file names its
-destination. None are sent yet — send from the maintainer's account, then
-record the thread link in ROADMAP.md v0.3 so feedback has a traceable home.
+Ready-to-send drafts. Each file names its destination. The v0.3
+announcement round went live on 2026-09-02 (thread links in ROADMAP.md);
+the release-risk round below is pending. Send from the maintainer's
+account, then record the thread link so feedback has a traceable home.
 
 | File | Destination | Depends on |
 |---|---|---|
-| [deepseek-harness-1115-reply.md](deepseek-harness-1115-reply.md) | reply in [deepseek-harness#1115](https://github.com/deepseek-ai/deepseek-harness/discussions/1115) | npm 0.3.0 published |
-| [show-your-plugins-post.md](show-your-plugins-post.md) | community "Show Your Plugins" thread | 0.3.0 published |
-| [awesome-list-pr.md](awesome-list-pr.md) | PR against a dsh/agent awesome-list | — |
-| [dsh-plugin-audit-collab.md](dsh-plugin-audit-collab.md) | issue/DM to dsh-plugin-audit maintainers | — |
+| [deepseek-harness-1115-reply.md](deepseek-harness-1115-reply.md) | reply in [deepseek-harness#1115](https://github.com/deepseek-ai/deepseek-harness/discussions/1115) | npm 0.3.0 published — **sent 2026-09-02** |
+| [show-your-plugins-post.md](show-your-plugins-post.md) | community "Show Your Plugins" thread | 0.3.0 published — **sent 2026-09-02** |
+| [awesome-list-pr.md](awesome-list-pr.md) | PR against a dsh/agent awesome-list | — **merged 2026-09-03** |
+| [dsh-plugin-audit-collab.md](dsh-plugin-audit-collab.md) | issue/DM to dsh-plugin-audit maintainers | — **sent 2026-09-02** |
+| [release-risk-1115-followup.md](release-risk-1115-followup.md) | follow-up in [deepseek-harness#1115](https://github.com/deepseek-ai/deepseek-harness/discussions/1115) | next npm release ships `dsh-vet diff` |
+| [release-risk-pilot-ask.md](release-risk-pilot-ask.md) | issue/DM per target plugin author (template with pre-run results) | — |

package/docs/outreach/release-risk-1115-followup.md

@@ -0,0 +1,37 @@
+<!-- Destination: follow-up comment in deepseek-ai/deepseek-harness discussion #1115
+     (marketplace standards / review mechanisms), under our 2026-09-02 reply.
+     Depends on: an npm release that ships `dsh-vet diff` (next version) —
+     send from the maintainer's account, then record the thread link in ROADMAP. -->
+
+Following up with the piece the contract was missing: **what changed since
+the last release**.
+
+"Should I install this plugin?" got a machine-readable answer with
+`dsh-vet/v1`. "Should I *upgrade*?" now has one too:
+
+- Every report records **what the scan actually covered** — rule profile,
+  coverage, content digests, stable finding identities — as an optional
+  `x-dsh-vet` extension ([spec](https://github.com/rogerdigital/dsh-vet/blob/main/docs/scan-context-v1.md)).
+  A grade no longer silently means "whatever subset happened to run":
+  partial coverage says so, in the CLI, the PR comment, and the badge.
+- `dsh-vet diff` compares two reports from the same scanner and profile
+  and reports added / removed / changed risk by **stable identity** —
+  line moves and reformatting are not risk additions, a second endpoint
+  is ([diff spec](https://github.com/rogerdigital/dsh-vet/blob/main/docs/report-diff-v1.md)).
+  Two reports that can't be honestly compared say so with explicit
+  reasons instead of a fake "no changes".
+- The GitHub Action takes an optional `baseline-report` scanned from the
+  merge base, so PRs show exactly what risk-relevant behavior the PR
+  introduces — with a recipe that keeps the baseline out of the PR's
+  control ([action docs](https://github.com/rogerdigital/dsh-vet/blob/main/action/README.md#comparing-releases)).
+
+We piloted it on real releases —
+[left-pad and chalk, every reported addition reconciled against the
+code](https://github.com/rogerdigital/dsh-vet/blob/main/docs/release-risk-pilot.md).
+The chalk case is the interesting one: 5.x vendored its dependencies and
+the diff flagged exactly the two vendored files unreachable through the
+`#`-imports map — nothing else moved.
+
+If you maintain a plugin and want the same read on your last two
+releases, reply here or open an issue — we'll run the comparison and
+post the result for you to check against what you actually changed.

package/docs/outreach/release-risk-pilot-ask.md

@@ -0,0 +1,71 @@
+<!-- Destination: issue (or DM) to the author of each target plugin, one
+     thread per plugin. The template below is filled with real pre-run
+     results — send as-is; the recipient installs nothing and answers from
+     their own knowledge of the release. Send from the maintainer's account,
+     then record each thread link in docs/release-risk-pilot.md. -->
+
+# Release-risk pilot ask (template)
+
+Subject: `Did your last release change what your plugin can do? (2-minute check)`
+
+Hi — I maintain [`dsh-vet`](https://github.com/rogerdigital/dsh-vet), the
+static audit tool for DSH plugins. It just learned to **compare two
+releases of the same plugin and report exactly what risk-relevant
+behavior changed** — new endpoints, new capabilities, new install
+scripts, changed confidence — while ignoring line moves and reformatting.
+
+I ran it on your last two releases so you don't have to install anything.
+The results are below; three questions at the end.
+
+---
+
+## Filled example — dsh-doctor 0.4.2 → 0.4.3
+
+Both scans: same scanner, full rule profile, complete coverage. Grades
+stayed **D → D**. What moved:
+
+- **New network client call** in `lib/client.js` — the destination is
+  computed at runtime, so the scanner can't see where it sends
+  (`perm.network-client`, subject `runtime-target`).
+- **Subprocess spawns went from 1 to 2** in `lib/index.js`
+  (`child_process.spawn`).
+
+Nothing else changed (13 identities unchanged).
+
+## Filled example — dsh-searxng 0.2.1 → 0.3.0
+
+Grades stayed **A → A**. One transition: **write/delete calls with
+runtime-computed targets went from 1 to 17** in `lib/cli.mjs`
+(`perm.undeclared-fs-write`, dynamic variant). These are low-severity /
+low-confidence — they never affect the grade — but the count jump is the
+kind of thing worth a look during a minor bump.
+
+## Filled example — dsh-wechat 0.9.5 → 0.9.6
+
+**Zero risk-relevant changes.** All 7 audited behaviors identical, grade
+C → C. If that matches your intent for the release, the diff just saved
+you the re-review.
+
+---
+
+## The three questions
+
+1. Does the reported delta match what you intended to change in that
+   release? Anything you'd flag that the scanner missed or got wrong?
+2. Any entries that felt like noise (didn't help you understand the
+   release)?
+3. If this ran automatically on your PRs against the base revision —
+   [one Action input](https://github.com/rogerdigital/dsh-vet/blob/main/action/README.md#comparing-releases) —
+   would you read it before merging?
+
+Answers in any form are useful; I'll record anonymized takeaways in the
+[pilot record](https://github.com/rogerdigital/dsh-vet/blob/main/docs/release-risk-pilot.md).
+This is the last feedback loop before deciding whether to build CI
+gating on top (opt-in "fail on new risk"), so negative answers are as
+valuable as positive ones.
+
+<!-- Per-send checklist:
+     - re-run the pair with the latest main build the day of sending
+     - paste the actual result block for THIS author's package
+     - keep grades/identities verbatim from the diff, don't paraphrase
+     - one package per thread; don't batch multiple authors -->

package/docs/release-risk-pilot.md

@@ -0,0 +1,109 @@
+# Release-risk review pilot (M1)
+
+Status: **internal pilot complete; external feedback pending.** This page
+records how the first milestone was exercised and what the comparison
+showed on real release pairs. It is not an adoption claim — no external
+maintainer has used the workflow yet, and the M2 entry gate (repeated
+pilot use showing review friction) is not met.
+
+## What was exercised
+
+| Exercise | Result |
+|---|---|
+| Controlled pair (committed fixtures, one added host) | Comparable; exactly the added-host identities and observation; grades equal |
+| Built-output smoke (`node bin/dsh-vet.mjs`, section 8 of the plan) | Help, scan ×2, validate ×2, diff, badge all pass; same-profile comparison of two runs reports zero risk changes despite distinct `ranAt`; badge unqualified for complete coverage |
+| `left-pad@1.2.0` → `1.3.0` | Comparable, zero added/removed/changed — reconciled below |
+| `chalk@4.1.2` → `5.3.0` | Comparable, two added `perm.unreachable-files` identities — reconciled below |
+
+All pairs were scanned with the same scanner build and full rule profile;
+both sides validated with `dsh-vet validate` before comparison.
+
+## Reconciliation: left-pad 1.2.0 → 1.3.0
+
+Both versions: identical audited behavior. The five unchanged identities
+cover the benchmark/test files that ship unreachably (`perf/*`, `test.js`)
+and the hex table in `perf/perf.js` (`obf.encoded-payload`, matched by
+content hash) — all present in both versions, none moved. **Zero
+additions to reconcile; the release changed padding internals without
+touching any audited behavior.** Effort: under a minute; the diff
+answered "anything new to review?" with a definite no.
+
+## Reconciliation: chalk 4.1.2 → 5.3.0
+
+Reported additions, each verified against the shipped tarballs:
+
+1. `perm.unreachable-files` · `source/vendor/supports-color/index.js`
+2. `perm.unreachable-files` · `source/vendor/supports-color/browser.js`
+
+Code evidence:
+
+- 4.1.2 has no `source/vendor/` directory at all; 5.3.0 vendors its
+  dependencies there.
+- 5.3.0's `source/index.js` imports `#supports-color`, which resolves
+  through the package.json `imports` map to exactly those two files
+  (`node` → `index.js`, `default` → `browser.js`).
+- The reference analyzer resolves relative imports but not `#`-prefixed
+  subpath imports, so neither file is statically reachable from the
+  declared entry — the finding text ("no static import path from any
+  entry point reaches this file") is accurate for this analyzer.
+- The sibling vendored `ansi-styles` **is** imported relatively
+  (`'./vendor/ansi-styles/index.js'`), is reachable, and is correctly
+  **not** reported — the delta is per-subject, not per-directory.
+
+Both grades stay `A` (unreachable files are informational); the delta is
+visible precisely because observations and identities are reported even
+when grades do not move.
+
+Honest limitation observed: `#`-subpath `imports` maps are not resolved,
+so code reached only through them reads as unreachable. That is a
+documented analyzer gap to close in a later revision — visible here
+because the delta surfaces it, which is the point of the workflow.
+
+## Pilot observations
+
+- The two real pairs took minutes to reconcile; every reported addition
+  mapped to a code fact, no formatting-driven false additions appeared
+  (line movement is excluded from identity by design and it held).
+- The interesting noise class is informational unreachable-file churn on
+  refactors that vendor or move code — visible, cheap to dismiss, and
+  exactly what M2's exception mechanism would absorb if it repeats.
+- One pilot round is not enough to justify policy work. The M2 entry
+  gate stays closed until a maintainer uses the comparison on real
+  releases repeatedly and reports the friction.
+
+## External feedback (pending)
+
+No external maintainer has used the comparison yet — this section stays
+empty until one does, and a friendly reply is not adoption. The ask and
+its per-plugin pre-run results live in
+[`docs/outreach/release-risk-pilot-ask.md`](outreach/release-risk-pilot-ask.md);
+record each thread here as it happens.
+
+Intake questions (mirroring the plan's §10 usefulness criteria):
+
+1. Does the reported delta match what the author intended to change?
+2. Anything the scanner missed or got wrong?
+3. Any entries that felt like noise?
+4. Would they read it on their PRs — and did they run it on a *second*
+   release? (The repeated-use answer is the M2 entry gate.)
+
+| Date | Plugin | Thread | Verdict (used / not / second release) |
+|---|---|---|---|
+| — | — | — | — |
+
+## Reproduction
+
+```sh
+pnpm build
+node bin/dsh-vet.mjs --json left-pad@1.2.0 > /tmp/lp-base.json
+node bin/dsh-vet.mjs --json left-pad@1.3.0 > /tmp/lp-head.json
+node bin/dsh-vet.mjs validate /tmp/lp-base.json /tmp/lp-head.json
+node bin/dsh-vet.mjs diff --json /tmp/lp-base.json /tmp/lp-head.json
+
+node bin/dsh-vet.mjs --json chalk@4.1.2 > /tmp/ch-base.json
+node bin/dsh-vet.mjs --json chalk@5.3.0 > /tmp/ch-head.json
+node bin/dsh-vet.mjs diff --json /tmp/ch-base.json /tmp/ch-head.json
+```
+
+The controlled pair lives in `test/fixtures/release-risk/` and is
+asserted by `test/golden.test.ts` on every run.

package/docs/report-diff-v1.md

@@ -0,0 +1,80 @@
+# The `dsh-vet/diff/v1` comparison result
+
+Status: **defined; the reference CLI exposes it from the next release.**
+This document is normative for consumers of `compareReports()` and the
+`dsh-vet diff` output. See [dsh-vet-v1.md](dsh-vet-v1.md) for the base
+report and [scan-context-v1.md](scan-context-v1.md) for the extension the
+comparator reads.
+
+`compareReports(base, head)` is pure data in, pure data out: it never
+fetches packages, never mutates either report, and never recalculates a
+report under new rules. It compares what two validated reports actually
+recorded.
+
+## Comparability
+
+A pair is **comparable** only when every gate passes:
+
+| Gate | Reason code when it fails |
+|---|---|
+| Both are structurally valid `dsh-vet/v1` reports | `report-invalid-base` / `report-invalid-head` |
+| Both carry a validated version-1 `x-dsh-vet` context | `context-absent-*`, `context-unsupported-*`, `context-invalid-*` |
+| Both carry finding identities | `identities-absent-base` / `identities-absent-head` |
+| Same scanner name and version | `scanner-mismatch` |
+| Same profile digest (analyzer, catalog, rules, options) | `profile-mismatch` |
+| Same package identity, or an explicit shared subject label | `subject-mismatch` / `subject-unlabeled` |
+| Neither side graded `X` | `grade-x-base` / `grade-x-head` |
+| Complete coverage on both sides | `coverage-partial-base` / `coverage-partial-head` |
+
+An incomparable result carries the reason codes and both report
+references, and **no change collections** — an absent diff must never
+read as "nothing changed". There is no force-comparison switch in M1;
+when practical, rescan both artifacts with the same scanner and profile
+instead. Two local-directory scans without package identity compare only
+under an explicit `subjectLabel` option; identity is never inferred from
+temporary absolute paths. Different analysis-input or archive digests
+are expected and never block comparison.
+
+## Result shape
+
+```jsonc
+{
+  "schema": "dsh-vet/diff/v1",
+  "base": { "scanner": { /* name, version, ranAt */ }, "grade": "C", "subject": { /* … */ } },
+  "head":  { "scanner": { /* … */ }, "grade": "C", "subject": { /* … */ } },
+  "comparability": "comparable",
+  "reasons": [],
+  "findings": {
+    "added":    [ /* identities present only in head */ ],
+    "removed":  [ /* identities present only in base */ ],
+    "changed":  [ /* matched identities that moved; both sides recorded */ ],
+    "unchanged": [ /* matched and identical */ ]
+  },
+  "observations": {
+    "added": [], "removed": [], "countChanged": [], "unchanged": []
+  }
+}
+```
+
+Identity entries and observations are the extension's own objects
+(`rule`/`variant`/`file`/`subject`/`finding`/`count?` and
+`kind`/`file`/`subject`/`count?`). Every collection is sorted and
+deterministic: two comparisons of the same pair produce byte-identical
+results.
+
+A **changed** finding records both sides' severity, confidence, and
+count. Severity and confidence transitions are classified separately
+from additions on purpose: moving from `low` to `medium` confidence can
+introduce a graded risk at unchanged severity.
+
+## Reading the result honestly
+
+- `added` means newly observed under the same checks — not proof of new
+  malice, and not a grade change by itself.
+- `removed` means no longer observed under comparable checks — never a
+  claim that anything was fixed.
+- Observation deltas are visible even when grades are identical or the
+  observations are informational; an unchanged grade never hides new
+  behavior.
+- The comparator reports line-position changes as nothing: identities
+  exclude line numbers, so a line insertion is not a risk addition.