@intentsolutions/audit-harness 1.1.5 → 1.1.7

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -2,6 +2,77 @@
2
2
 
3
3
  All notable changes are recorded here. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
4
4
 
5
+ ## [Unreleased]
6
+
7
+ ### Added — CI-only signed evidence emit for the intent-eval-dashboard (nr75.12)
8
+
9
+ The dashboard reports hub (labs.intentsolutions.io) ingests a signed `report-manifest.json` of kernel `gate-result/v1` rows per repo. This adds audit-harness's own emit, lighting up its row.
10
+
11
+ - **`ci/emit-evidence.ts` + `ci/assemble-manifest.ts`** — run the real deterministic self-gate (`harness-hash --verify`), shape it into a kernel `gate-result/v1` + `EvidenceBundle` (fail-closed against `@intentsolutions/core`), cosign-sign the canonical bytes (Fulcio OIDC + Rekor), and assemble the manifest the dashboard re-verifies at ingest.
12
+ - **Zero-dep guarantee preserved.** The emitter lives in `ci/` (excluded from `package.json#files`) and the kernel is installed CI-only via `npm i --no-save` — `dependencies` + `devDependencies` stay empty and the published tarball is unchanged (verified via `npm pack --dry-run`).
13
+ - **`.github/workflows/release.yml`** — adds a GitHub Release on tag push + an `emit-evidence` job (tag-only) that publishes the manifest as a Release asset.
14
+
15
+ ### Added — `currency` advisory upstream-currency report (PP-PLAN-040 Phase 5 / E7)
16
+
17
+ The fifth verb, and deliberately the weakest: an advisory report with no exit-code authority.
18
+
19
+ - **`audit-harness currency`** (`scripts/currency.py`, stdlib): reads the per-upstream-identity pin relation (`schemas/currency/pins.v1.json`) and reports which pins are themselves **stale** — `checked_at` older than the pin's staleness window. Each upstream (mcp-spec, skill-md-schema, claude-code, gate-result-predicate, anthropic-sdk, agentskills-spec) carries its own `pinned_version` + `checked_at` + window, so the *pin's own staleness* is detectable (not one opaque scalar).
20
+ - **No exit-code authority (always exit 0), no live-fetch, no auto-fix.** Currency depends on upstream state — non-deterministic and network-bound — so it only reports. `/sync-testing-harness` consumes the report to open advisory bump PRs; it never reddens a build. `--today YYYY-MM-DD` makes reports reproducible.
21
+ - **`tests/currency/`**: golden suite (3 checks) — stale/current/unknown classification, the no-exit-authority guarantee (exit 0 even when all pins are stale), and the shipped relation reporting.
22
+
23
+ ### Added — `scan` security/hygiene/skill-quality gate-runner (PP-PLAN-040 Phase 4 / E6)
24
+
25
+ The fourth read-only verb: security + hygiene + skill-quality, by orchestrating standard tools (never reimplementing them).
26
+
27
+ - **`audit-harness scan [repo]`** (`scripts/scan.py`, stdlib): for every `dimension: security | hygiene | skill-quality` gate in the profile, emits a `gate-result/v1` row. Three strategies: **local** (`hygiene-readme` README presence — deterministic), **shell-out** (every gate carrying a `tool` — gitleaks / osv-scanner / semgrep / syft / markdownlint / lychee — clean exit → PASS, findings → ADVISORY(error), tool absent → ADVISORY indeterminate), **consume** (`skill-behavioral` ingests a j-rig Evidence Bundle verdict via `--jrig-verdict`; the harness never runs behavioral judgment itself — no verdict → indeterminate).
28
+ - Advisory-first; `--strict` (or a blocking gate) turns a finding/gap into `FAIL`. Kill-switch → `[]`. Each row records `metadata.method` (`local-presence` / `shell-out` / `consume-j-rig`).
29
+ - **`tests/scan/`**: golden suite (10 checks) with pinned-profile isolation so shell-out tool availability never makes the suite flaky.
30
+
31
+ **Security note:** on first run this gate caught — and this release redacts from HEAD — a PyPI publish token that had been pasted as a literal value in `python/PUBLISH.md`. The value remains in git history; it must be rotated at the registry (tracked separately). The doc now carries a placeholder.
32
+
33
+ ### Added — `audit` testing-depth gate-runner (PP-PLAN-040 Phase 3 / E5)
34
+
35
+ The third read-only verb: the "finish the pyramid" testing-depth diagnostic.
36
+
37
+ - **`audit-harness audit [repo]`** (`scripts/audit.py`, stdlib): for every `dimension: testing-depth` gate in the profile, assesses the gate and emits a `gate-result/v1` row. Two read-only strategies: `crap-score` runs the bundled `crap` scorer (static complexity×coverage); every pyramid layer (unit/integration/e2e/smoke/perf/a11y/contract/migration/property-based/fuzz/sanitizers) gets a per-layer **presence heuristic** (test dirs, framework configs, dependency markers). Layer present → `PASS`; absent → `ADVISORY(warn)` testing-depth gap; not statically assessable → `ADVISORY` indeterminate.
38
+ - **`--fast` (default)** presence heuristics only (<10s); **`--deep`** adds `crap-score`; **`--strict`** turns a gap on a blocking gate into `FAIL`. Kill-switch → `[]`. Each row records `metadata.method` (`crap-static` / `presence-heuristic` / `delegated`) for provenance.
39
+ - **Deliberately does NOT execute the repo's test suite.** Running arbitrary untrusted suites is the repo's own CI's job; the harness reports coverage *presence* and the repo's CI test step produces the execution verdict. `audit` is the diagnostic, not the test runner.
40
+ - **`tests/audit/`**: golden suite (7 checks) + `has-tests`/`no-tests` fixtures — asserts unit→PASS / gap→ADVISORY(default) / gap→FAIL(`--strict`), crap deep-only-in-fast, kill-switch, and gate-result/v1 validity. CI `audit` job.
41
+
42
+ ### Added — registry projection + FP-rate harness (PP-PLAN-040 Phase 0 completion: c2b + c2e)
43
+
44
+ Closes the data/safety-spine epic (E2): the registry becomes the single canonical datum and gate promotion gets a measured bar.
45
+
46
+ - **`audit-harness gen-layer-applicability`** (`scripts/gen-layer-applicability.py`): projects `schemas/audit-profile/registry.v1.json` into `schemas/audit-profile/layer-applicability.md`. `--write` regenerates; `--check` fails on drift. The doc is now a **projection** of the registry datum, not a hand-maintained parallel source — CI gate `layer-applicability-drift` enforces it (c2b).
47
+ - **`audit-harness fp-rate`** (`scripts/fp-rate.py`): measures each gate's false-positive / false-negative rate over a labeled corpus (`tests/fixtures/conform/{valid,malformed}/`). This is the metric that gates advisory→blocking promotion. `--max-fp-rate X` exits 1 if any gate exceeds the bar; CI runs it advisory at the 5% default bar (c2e).
48
+ - **`docs/gate-promotion.md`**: the dedicated advisory→blocking promotion rule — FP-rate ≤ 5% bar, engineer-pinned in `tests/TESTING.md`, re-pinned manifest. Documents *why* FP-rate (not FN-rate) is the gate and how demotion/kill-switch works. `docs/` now ships in the npm package (`files`).
49
+
50
+ ### Added — `conform` verb + bundled content-addressed schemas (PP-PLAN-040 Phase 2)
51
+
52
+ The second piece of the read-only brain: deterministic conformance, emitting Evidence Bundle rows.
53
+
54
+ - **`audit-harness conform [repo]`** (`scripts/conform.py`, stdlib + PyYAML): read-only conformance gate-runner. For every `dimension: conformance` gate in the repo's `audit-profile/v1`, it locates the artifact(s) and emits a `gate-result/v1` row (JSON array, stdout). **Never writes, never live-fetches.**
55
+ - **Bundled content-addressed schemas** (`schemas/conform/v1/`): `skillmd-frontmatter`, `mcp-config`, `plugin-manifest`, `agent-frontmatter` — the deterministic *structural floor* (parses + required keys + types), distinct from the IS 100-point rubric / SAK authoring kernel (judgment, stays in `/validate-*`). conform records each schema's sha256 in the row's `policy_hash`, so a row re-verifies against the exact schema version that produced it.
56
+ - **Reproducible-by-design engine.** Bundled JSON-Schemas are checked by an embedded subset validator (complete for the closed bundled schemas) rather than ajv — deliberately, because ajv's availability/version varies per machine and would make signed evidence non-reproducible. Same commit + same harness version produce an identical verdict.
57
+ - **Genuinely-external formats shell out**: OpenAPI to `spectral`, GitHub Action to `yamllint`. Missing tool produces an `ADVISORY` indeterminate (never a false `FAIL`).
58
+ - **Advisory-first.** A conformance violation on an `enforcement: advisory` gate is `ADVISORY` (severity `error`), exit 0 — logged, not blocking. `--strict` (or an engineer-promoted `enforcement: blocking` gate) turns a violation into `FAIL` (exit 1). Missing artifact produces `NOT_APPLICABLE`. Kill-switch (`AUDIT_HARNESS_DISABLE=1` / `.audit-harness.yml`) produces an empty `[]`, exit 0.
59
+ - **`tests/conform/`**: golden suite (31 checks) + pass/fail fixtures (valid + malformed SKILL.md, .mcp.json, plugin manifest, agent) — asserts valid to PASS, malformed to ADVISORY (default) / FAIL (`--strict`), every row validates against `gate-result/v1`, the NOT_APPLICABLE + indeterminate paths, and `policy_hash` == bundled-schema sha256 + reproducible. Wired into CI (`conform` job).
60
+
61
+ Scope boundary: conformance kinds without a bundled schema (`marketplace`, `hook`) resolve to `ADVISORY` indeterminate — drop a schema into `schemas/conform/v1/` to light them up, no code change. No gate *execution* for testing-depth/security yet (Phase 3+).
62
+
63
+ ### Added — `classify` verb + `audit-profile/v1` (PP-PLAN-040 Phase 0 + Phase 1)
64
+
65
+ The first piece of the "comprehensive audit, on any repo" build: the read-only brain.
66
+
67
+ - **`audit-profile/v1` schema** (`schemas/audit-profile/v1.schema.json`): closed, versioned, hash-bearing value mirroring `gate-result/v1`. Four invariants: classifications are a UNION (not a winner), `unresolved[]` is the only Claude-refinable surface, `waived ⇒ disabled` (allOf-enforced), `registry_hash` makes a profile reproducible.
68
+ - **Canonical dimension→gate registry** (`schemas/audit-profile/registry.v1.json`): the single datum that answers "which gates apply to repo-type X, in which dimension, at what applicability" — `layer-applicability.md` and `TESTING.md` become projections of it.
69
+ - **`audit-harness classify [repo]`** (`scripts/classify.py`, stdlib-only): read-only repository classifier. Detects the UNION of repo-type + Claude-artifact classifications, resolves the gate set against the registry, records `registry_hash`, and emits an `audit-profile/v1` value to stdout. **Never writes to the repo.**
70
+ - **Safety levers**: `INDETERMINATE` result class (infra failure ≠ policy failure); dispatcher per-command supervision via `AUDIT_HARNESS_TIMEOUT` (kill a hung gate, exit 124); `AUDIT_HARNESS_DISABLE=1` kill-switch (gate commands no-op; classify emits an all-disabled profile); engineer-owned `.audit-harness.yml` override (`classify_pins`, `advisory`, `disable_gates`, `disable`) — see `.audit-harness.example.yml`.
71
+ - **`tests/classify/`**: golden fixture corpus (6 fixtures, authored before the classifier) + suite — golden-matches classifications, schema-validates every profile, exercises the kill-switch, the unknown/unresolved path, and override honoring. Wired into CI (`classify` job).
72
+ - **`schemas/` now ships in the npm package** (`files`) so the registry + schema are available to consumers on any repo.
73
+
74
+ Scope boundary: no `conform` verb, no gate execution yet (Phase 2+). `classify` is read-only and emits a profile only.
75
+
5
76
  ## [v1.1.5] - 2026-06-03
6
77
 
7
78
  ### Added — npm release pipeline (closes the publish-pipeline gap)
@@ -22,8 +22,22 @@ const COMMANDS = {
22
22
  'gherkin-lint': { script: 'gherkin-lint.sh', args: [] },
23
23
  'crap': { script: 'crap-score.py', args: [] },
24
24
  'emit-evidence': { script: 'emit-evidence.sh', args: [] },
25
+ 'classify': { script: 'classify.py', args: [] },
26
+ 'conform': { script: 'conform.py', args: [] },
27
+ 'audit': { script: 'audit.py', args: [] },
28
+ 'scan': { script: 'scan.py', args: [] },
29
+ 'fp-rate': { script: 'fp-rate.py', args: [] },
30
+ 'currency': { script: 'currency.py', args: [] },
31
+ 'gen-layer-applicability': { script: 'gen-layer-applicability.py', args: [] },
25
32
  };
26
33
 
34
+ // Gate commands that may be no-op'd by the AUDIT_HARNESS_DISABLE kill-switch.
35
+ // classify is intentionally NOT here: it emits a meaningful kill-switched profile
36
+ // itself (every gate enforcement=disabled). verify/init/list always run.
37
+ const KILLABLE_GATES = new Set([
38
+ 'escape-scan', 'arch', 'bias', 'gherkin-lint', 'crap', 'emit-evidence',
39
+ ]);
40
+
27
41
  function usage() {
28
42
  console.log(`audit-harness — deterministic test-enforcement toolkit
29
43
 
@@ -40,6 +54,46 @@ Commands:
40
54
  bias Count test-bias patterns (tautology, smoke-only, etc.)
41
55
  gherkin-lint Advisory Gherkin quality check
42
56
  crap [args...] CRAP complexity × coverage scorer (multi-language)
57
+ classify [repo] Read-only repository classifier. Emits an audit-profile/v1
58
+ value (JSON, stdout) describing the UNION of detected
59
+ classifications + the resolved gate set. Never writes.
60
+ conform [repo] Read-only conformance gate-runner. Validates each repo
61
+ artifact (SKILL.md, .mcp.json, plugin/agent ...) against a
62
+ content-addressed schema BUNDLED in this harness version
63
+ and emits gate-result/v1 rows (JSON array, stdout). Never
64
+ writes, never live-fetches. Advisory by default; --strict
65
+ turns any conformance violation into FAIL (exit 1).
66
+ OpenAPI -> spectral, Action -> yamllint (missing tool =
67
+ INDETERMINATE advisory).
68
+ audit [repo] Read-only testing-depth gate-runner. For each
69
+ testing-depth gate in the profile, reports coverage
70
+ PRESENCE per pyramid layer (unit/integration/e2e/perf/
71
+ fuzz/...) + runs crap-score. Emits gate-result/v1 rows.
72
+ --fast (default) presence only; --deep adds crap-score;
73
+ --strict turns a gap into FAIL. Does NOT execute the
74
+ repo's test suite — execution stays in the repo's CI.
75
+ scan [repo] Read-only security/hygiene/skill-quality gate-runner.
76
+ hygiene-readme is a local presence check; every tool-
77
+ backed gate (gitleaks/osv-scanner/semgrep/syft/
78
+ markdownlint/lychee) shells out (clean->PASS, findings->
79
+ ADVISORY, tool absent->INDETERMINATE); skill-behavioral
80
+ CONSUMES a j-rig verdict (--jrig-verdict PATH), never
81
+ reimplementing judgment. Emits gate-result/v1 rows.
82
+ Advisory by default; --strict turns findings into FAIL.
83
+ fp-rate Measure each gate's false-positive / false-negative rate
84
+ over a labeled corpus (valid/ should be clean, malformed/
85
+ should flag). The metric that gates advisory->blocking
86
+ promotion. --max-fp-rate X exits 1 if any gate exceeds X.
87
+ See docs/gate-promotion.md.
88
+ currency Advisory upstream-currency report. Reads the per-upstream
89
+ pin relation (schemas/currency/pins.v1.json) and flags
90
+ pins whose checked_at is past their staleness window.
91
+ NO exit-code authority (always exit 0), no live-fetch,
92
+ no auto-fix — it reports; /sync-testing-harness acts.
93
+ gen-layer-applicability Project schemas/audit-profile/registry.v1.json into
94
+ schemas/audit-profile/layer-applicability.md. --write to
95
+ regenerate, --check to fail on drift (CI gate). The doc
96
+ is a PROJECTION of the canonical registry datum.
43
97
  emit-evidence Wrap a gate-result JSON envelope in an in-toto
44
98
  Statement v1 (predicate https://evals.intentsolutions.io/gate-result/v1)
45
99
  Read JSON on stdin: <gate> --json | audit-harness emit-evidence
@@ -50,6 +104,14 @@ Evidence Bundle (v0.3.0+):
50
104
  and intent-eval-lab/specs/evidence-bundle/v0.1.0-draft/SPEC.md for the
51
105
  envelope schema.
52
106
 
107
+ Safety levers:
108
+ AUDIT_HARNESS_DISABLE=1 Kill-switch. Gate commands no-op (exit 0, banner);
109
+ classify still emits a profile with every gate disabled.
110
+ AUDIT_HARNESS_TIMEOUT=N Per-command supervision: kill the gate after N seconds
111
+ (exit 124) so a hung gate never blocks the pipeline.
112
+ .audit-harness.yml Engineer-owned per-repo override (classify_pins, advisory,
113
+ disable_gates, disable) honored by classify.
114
+
53
115
  Options:
54
116
  --version, -v Print version
55
117
  --help, -h Print this help
@@ -87,12 +149,39 @@ if (!existsSync(scriptPath)) {
87
149
  process.exit(2);
88
150
  }
89
151
 
152
+ // Kill-switch: gate commands no-op; classify/verify/init/list still run.
153
+ if (process.env.AUDIT_HARNESS_DISABLE === '1' && KILLABLE_GATES.has(cmd)) {
154
+ console.error(`audit-harness: KILL-SWITCH active (AUDIT_HARNESS_DISABLE=1) — '${cmd}' skipped`);
155
+ process.exit(0);
156
+ }
157
+
90
158
  const isPython = entry.script.endsWith('.py');
91
159
  const interpreter = isPython ? 'python3' : 'bash';
92
160
  const finalArgs = [scriptPath, ...entry.args, ...rest];
93
161
 
162
+ // Per-command supervision: a hung gate hits its timeout and is killed (exit 124)
163
+ // rather than blocking the pipeline. 0/unset = no timeout.
164
+ const timeoutSec = Number(process.env.AUDIT_HARNESS_TIMEOUT) || 0;
165
+
94
166
  const child = spawn(interpreter, finalArgs, { stdio: 'inherit' });
167
+
168
+ let timedOut = false;
169
+ let timer = null;
170
+ if (timeoutSec > 0) {
171
+ timer = setTimeout(() => {
172
+ timedOut = true;
173
+ child.kill('SIGTERM');
174
+ setTimeout(() => child.kill('SIGKILL'), 2000).unref();
175
+ }, timeoutSec * 1000);
176
+ timer.unref();
177
+ }
178
+
95
179
  child.on('exit', (code, signal) => {
180
+ if (timer) clearTimeout(timer);
181
+ if (timedOut) {
182
+ console.error(`audit-harness: ${entry.script} exceeded AUDIT_HARNESS_TIMEOUT=${timeoutSec}s — killed (INDETERMINATE)`);
183
+ process.exit(124);
184
+ }
96
185
  if (signal) {
97
186
  console.error(`audit-harness: ${entry.script} killed by ${signal}`);
98
187
  process.exit(128);
@@ -100,6 +189,7 @@ child.on('exit', (code, signal) => {
100
189
  process.exit(code ?? 0);
101
190
  });
102
191
  child.on('error', (err) => {
192
+ if (timer) clearTimeout(timer);
103
193
  console.error(`audit-harness: failed to spawn ${interpreter}: ${err.message}`);
104
194
  process.exit(2);
105
195
  });
@@ -0,0 +1,45 @@
1
+ # Gate promotion — advisory → blocking
2
+
3
+ Every gate in the [dimension→gate registry](../schemas/audit-profile/registry.v1.json) ships `enforcement: advisory`. Advisory means: the gate runs, emits its `gate-result/v1` row, logs any finding — and **exits 0**. It never reddens a build. Blocking (`enforcement: blocking`, exit 1 on violation) is **earned**, not default.
4
+
5
+ This is deliberate. A gate that blocks before its false-positive rate is known erodes trust, and trust-erosion ends one way: someone appends `|| true` and the gate is dead. Advisory-first + earned-promotion keeps every blocking gate credible.
6
+
7
+ ## The rule
8
+
9
+ A gate may be promoted to `blocking` for a repo when **all** hold:
10
+
11
+ 1. **Measured FP-rate below the bar.** Run the [FP-rate harness](../scripts/fp-rate.py) over a labeled corpus and confirm the gate's false-positive rate is **≤ 5%** (the default bar). A false positive is a *clean* input the gate wrongly flags — the failure mode that destroys trust.
12
+
13
+ ```bash
14
+ audit-harness fp-rate # human-readable report
15
+ audit-harness fp-rate --json # machine-readable
16
+ audit-harness fp-rate --max-fp-rate 0.05 # exit 1 if any gate exceeds the bar
17
+ ```
18
+
19
+ The labeled corpus is `tests/fixtures/conform/{valid,malformed}/` (extend it: a gate is only as trustworthy as the corpus it was measured on — `valid/` inputs must stay clean, `malformed/` inputs must get flagged).
20
+
21
+ 2. **Engineer-pinned in the repo's policy.** Promotion is an engineer decision recorded in the target repo's hash-pinned `tests/TESTING.md` — never inferred by a tool, never proposed by an AI. The classifier reads the per-gate `enforcement` override from the engineer-owned `.audit-harness.yml` (`advisory:` / `disable_gates:` lists) and the policy in `tests/TESTING.md`; the registry default stays `advisory`.
22
+
23
+ 3. **Re-pinned manifest.** After editing the policy, re-init the hash manifest so the change travels with the code and `escape-scan` won't REFUSE a later legitimate edit:
24
+
25
+ ```bash
26
+ audit-harness init # re-pin after an engineer-reviewed policy edit
27
+ git add .harness-hash tests/TESTING.md # commit policy + manifest together
28
+ ```
29
+
30
+ ## Why FP-rate, not FN-rate, is the promotion gate
31
+
32
+ A false **negative** (a real problem the gate misses) is a coverage gap — annoying, but advisory output still surfaces it elsewhere and the gate can tighten over time. A false **positive** on a blocking gate halts a correct build, and the human response is to route around the gate permanently. So promotion is gated on FP-rate; FN-rate is reported for visibility but does not block promotion.
33
+
34
+ ## Demotion / kill-switch
35
+
36
+ Promotion is reversible without ceremony:
37
+
38
+ - Per-gate, per-repo: list the `gate_id` under `advisory:` or `disable_gates:` in `.audit-harness.yml`.
39
+ - Whole-repo break-glass: `AUDIT_HARNESS_DISABLE=1` (gates no-op; `classify` emits an all-disabled profile; `conform` emits `[]`).
40
+
41
+ A blocking gate that starts throwing false positives in the field should be demoted to advisory immediately, then re-measured before re-promotion.
42
+
43
+ ## Provenance
44
+
45
+ Each emitted `gate-result/v1` row records the `policy_hash` (sha256 of the policy/schema the gate evaluated against), so any promotion decision is auditable back to the exact policy version that produced the measurement.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@intentsolutions/audit-harness",
3
- "version": "1.1.5",
3
+ "version": "1.1.7",
4
4
  "description": "Deterministic test-enforcement harness — escape-scan, hash-pinning, CRAP, architecture checks, bias detection, Gherkin lint. Companion to the audit-tests and implement-tests Claude Code skills.",
5
5
  "license": "Apache-2.0",
6
6
  "author": "Jeremy Longshore <jeremy@intentsolutions.io>",
@@ -31,6 +31,8 @@
31
31
  "files": [
32
32
  "bin",
33
33
  "scripts",
34
+ "schemas",
35
+ "docs",
34
36
  "README.md",
35
37
  "LICENSE",
36
38
  "NOTICE",
@@ -0,0 +1,146 @@
1
+ # Layer Applicability — GENERATED from `registry.v1.json`
2
+
3
+ > ⚠️ **GENERATED FILE — do not edit by hand.**
4
+ > Source of truth: [`registry.v1.json`](registry.v1.json) (the canonical dimension→gate datum; `classify` resolves against it).
5
+ > Regenerate: `audit-harness gen-layer-applicability --write` (or `python3 scripts/gen-layer-applicability.py --write`).
6
+ > CI gate `layer-applicability-drift` fails the build if this file drifts from the registry.
7
+ >
8
+ > registry `sha256:ffbc75700fb5eb501cb47f1e4038f47ab95ae1fba534b38095e1fe7820c80ed1`
9
+
10
+ THE canonical dimension-to-gate registry: the single datum that answers 'which gates apply to repo-type X, in which dimension, at what applicability'. layer-applicability.md and each repo's TESTING.md are PROJECTIONS of this datum. `classify` resolves the UNION of a repo's detected classifications against this registry and records its sha256 as the audit-profile's registry_hash. Every gate defaults to enforcement=advisory; blocking is earned (engineer-pinned in TESTING.md, FP-rate-gated). applicability mirrors the matrix glyphs: required(✅) recommended(⭕) conditional(⚠) waived(❌).
11
+
12
+ **Legend (applicability):** ✅ required · ⭕ recommended · ⚠ conditional · ❌ waived
13
+
14
+ Every gate defaults to `enforcement: advisory`. Blocking is **earned** — engineer-pinned in the target repo's `tests/TESTING.md`, FP-rate-gated (see [`gate-promotion.md`](../../docs/gate-promotion.md)).
15
+
16
+ ## Base gates (apply to every repo)
17
+
18
+ | Gate | Dimension | Applicability | Enforcement | Tool |
19
+ |---|---|---|---|---|
20
+ | `audit-harness:local:hygiene-links` | hygiene | ⭕ recommended | advisory | `lychee` |
21
+ | `audit-harness:local:hygiene-markdown` | hygiene | ⭕ recommended | advisory | `markdownlint` |
22
+ | `audit-harness:local:hygiene-readme` | hygiene | ⭕ recommended | advisory | — |
23
+ | `audit-harness:ci:cve-osv` | security | ⭕ recommended | advisory | `osv-scanner` |
24
+ | `audit-harness:ci:secrets-gitleaks` | security | ⭕ recommended | advisory | `gitleaks` |
25
+
26
+ ## By classification
27
+
28
+ A repo carries the **UNION** of every classification it matches (`classify` never picks a single winner). Gates dedup by `gate_id`, keeping the highest applicability.
29
+
30
+ ### `action`
31
+
32
+ | Gate | Dimension | Applicability | Enforcement | Tool |
33
+ |---|---|---|---|---|
34
+ | `audit-harness:local:conform-action` | conformance | ✅ required | advisory | `yamllint` |
35
+ | `audit-harness:ci:smoke` | testing-depth | ⭕ recommended | advisory | — |
36
+
37
+ ### `agent`
38
+
39
+ | Gate | Dimension | Applicability | Enforcement | Tool |
40
+ |---|---|---|---|---|
41
+ | `audit-harness:local:conform-agent` | conformance | ✅ required | advisory | `validate-agent` |
42
+
43
+ ### `api`
44
+
45
+ | Gate | Dimension | Applicability | Enforcement | Tool |
46
+ |---|---|---|---|---|
47
+ | `audit-harness:local:conform-openapi` | conformance | ✅ required | advisory | `spectral` |
48
+ | `audit-harness:ci:sast` | security | ✅ required | advisory | `semgrep` |
49
+ | `audit-harness:ci:contract` | testing-depth | ✅ required | advisory | — |
50
+ | `audit-harness:ci:crap-score` | testing-depth | ✅ required | advisory | — |
51
+ | `audit-harness:ci:integration` | testing-depth | ✅ required | advisory | — |
52
+ | `audit-harness:ci:unit` | testing-depth | ✅ required | advisory | — |
53
+
54
+ ### `cli`
55
+
56
+ | Gate | Dimension | Applicability | Enforcement | Tool |
57
+ |---|---|---|---|---|
58
+ | `audit-harness:ci:crap-score` | testing-depth | ✅ required | advisory | — |
59
+ | `audit-harness:ci:smoke` | testing-depth | ✅ required | advisory | — |
60
+ | `audit-harness:ci:unit` | testing-depth | ✅ required | advisory | — |
61
+
62
+ ### `embedded`
63
+
64
+ | Gate | Dimension | Applicability | Enforcement | Tool |
65
+ |---|---|---|---|---|
66
+ | `audit-harness:ci:fuzz` | testing-depth | ✅ required | advisory | — |
67
+ | `audit-harness:ci:sanitizers` | testing-depth | ✅ required | advisory | — |
68
+ | `audit-harness:ci:unit` | testing-depth | ✅ required | advisory | — |
69
+
70
+ ### `frontend`
71
+
72
+ | Gate | Dimension | Applicability | Enforcement | Tool |
73
+ |---|---|---|---|---|
74
+ | `audit-harness:ci:a11y` | testing-depth | ✅ required | advisory | `axe` |
75
+ | `audit-harness:ci:contract` | testing-depth | ⚠ conditional | advisory | — |
76
+ | `audit-harness:ci:crap-score` | testing-depth | ✅ required | advisory | — |
77
+ | `audit-harness:ci:e2e` | testing-depth | ✅ required | advisory | — |
78
+ | `audit-harness:ci:unit` | testing-depth | ✅ required | advisory | — |
79
+
80
+ ### `hook`
81
+
82
+ | Gate | Dimension | Applicability | Enforcement | Tool |
83
+ |---|---|---|---|---|
84
+ | `audit-harness:local:conform-hook` | conformance | ✅ required | advisory | `validate-hook` |
85
+
86
+ ### `library`
87
+
88
+ | Gate | Dimension | Applicability | Enforcement | Tool |
89
+ |---|---|---|---|---|
90
+ | `audit-harness:ci:cve-osv` | security | ✅ required | advisory | `osv-scanner` |
91
+ | `audit-harness:ci:crap-score` | testing-depth | ✅ required | advisory | — |
92
+ | `audit-harness:ci:property-based` | testing-depth | ⭕ recommended | advisory | — |
93
+ | `audit-harness:ci:unit` | testing-depth | ✅ required | advisory | — |
94
+
95
+ ### `marketplace`
96
+
97
+ | Gate | Dimension | Applicability | Enforcement | Tool |
98
+ |---|---|---|---|---|
99
+ | `audit-harness:local:conform-marketplace` | conformance | ✅ required | advisory | `validate-marketplace` |
100
+
101
+ ### `mcp`
102
+
103
+ | Gate | Dimension | Applicability | Enforcement | Tool |
104
+ |---|---|---|---|---|
105
+ | `audit-harness:local:conform-mcp` | conformance | ✅ required | advisory | `validate-mcp` |
106
+
107
+ ### `monorepo`
108
+
109
+ | Gate | Dimension | Applicability | Enforcement | Tool |
110
+ |---|---|---|---|---|
111
+ | `audit-harness:local:per-package-classify` | testing-depth | ✅ required | advisory | — |
112
+
113
+ ### `plugin`
114
+
115
+ | Gate | Dimension | Applicability | Enforcement | Tool |
116
+ |---|---|---|---|---|
117
+ | `audit-harness:local:conform-plugin` | conformance | ✅ required | advisory | `validate-plugin` |
118
+ | `audit-harness:server:skill-behavioral` | skill-quality | ⭕ recommended | advisory | `j-rig` |
119
+
120
+ ### `service`
121
+
122
+ | Gate | Dimension | Applicability | Enforcement | Tool |
123
+ |---|---|---|---|---|
124
+ | `audit-harness:ci:sast` | security | ✅ required | advisory | `semgrep` |
125
+ | `audit-harness:ci:sbom-syft` | security | ⭕ recommended | advisory | `syft` |
126
+ | `audit-harness:ci:contract` | testing-depth | ✅ required | advisory | — |
127
+ | `audit-harness:ci:crap-score` | testing-depth | ✅ required | advisory | — |
128
+ | `audit-harness:ci:integration` | testing-depth | ✅ required | advisory | — |
129
+ | `audit-harness:ci:migration` | testing-depth | ✅ required | advisory | — |
130
+ | `audit-harness:ci:perf` | testing-depth | ⭕ recommended | advisory | — |
131
+ | `audit-harness:ci:unit` | testing-depth | ✅ required | advisory | — |
132
+
133
+ ### `skill`
134
+
135
+ | Gate | Dimension | Applicability | Enforcement | Tool |
136
+ |---|---|---|---|---|
137
+ | `audit-harness:local:conform-skillmd` | conformance | ✅ required | advisory | `validate-skillmd` |
138
+ | `audit-harness:server:skill-behavioral` | skill-quality | ⭕ recommended | advisory | `j-rig` |
139
+
140
+ ## Overlays
141
+
142
+ ### `regulated`
143
+
144
+ Compliance overlay (HIPAA/SOX/PCI-DSS/SOC2/GDPR/FedRAMP markers). Promotes recommended security + conformance gates to required and escalates uncovered SHOULD requirements.
145
+
146
+ Promotes to **required**: `security`, `conformance`.
@@ -0,0 +1,87 @@
1
+ {
2
+ "registry_version": "audit-profile-registry/v1",
3
+ "description": "THE canonical dimension-to-gate registry: the single datum that answers 'which gates apply to repo-type X, in which dimension, at what applicability'. layer-applicability.md and each repo's TESTING.md are PROJECTIONS of this datum. `classify` resolves the UNION of a repo's detected classifications against this registry and records its sha256 as the audit-profile's registry_hash. Every gate defaults to enforcement=advisory; blocking is earned (engineer-pinned in TESTING.md, FP-rate-gated). applicability mirrors the matrix glyphs: required(✅) recommended(⭕) conditional(⚠) waived(❌).",
4
+ "base": [
5
+ { "gate_id": "audit-harness:local:hygiene-readme", "dimension": "hygiene", "applicability": "recommended", "enforcement": "advisory", "result_class_default": "PASS" },
6
+ { "gate_id": "audit-harness:local:hygiene-links", "dimension": "hygiene", "applicability": "recommended", "enforcement": "advisory", "tool": "lychee", "result_class_default": "INDETERMINATE" },
7
+ { "gate_id": "audit-harness:local:hygiene-markdown", "dimension": "hygiene", "applicability": "recommended", "enforcement": "advisory", "tool": "markdownlint" },
8
+ { "gate_id": "audit-harness:ci:secrets-gitleaks", "dimension": "security", "applicability": "recommended", "enforcement": "advisory", "tool": "gitleaks", "result_class_default": "INDETERMINATE" },
9
+ { "gate_id": "audit-harness:ci:cve-osv", "dimension": "security", "applicability": "recommended", "enforcement": "advisory", "tool": "osv-scanner", "result_class_default": "INDETERMINATE" }
10
+ ],
11
+ "classifications": {
12
+ "service": [
13
+ { "gate_id": "audit-harness:ci:unit", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
14
+ { "gate_id": "audit-harness:ci:crap-score", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
15
+ { "gate_id": "audit-harness:ci:integration", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
16
+ { "gate_id": "audit-harness:ci:contract", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
17
+ { "gate_id": "audit-harness:ci:migration", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
18
+ { "gate_id": "audit-harness:ci:sast", "dimension": "security", "applicability": "required", "enforcement": "advisory", "tool": "semgrep" },
19
+ { "gate_id": "audit-harness:ci:sbom-syft", "dimension": "security", "applicability": "recommended", "enforcement": "advisory", "tool": "syft" },
20
+ { "gate_id": "audit-harness:ci:perf", "dimension": "testing-depth", "applicability": "recommended", "enforcement": "advisory" }
21
+ ],
22
+ "api": [
23
+ { "gate_id": "audit-harness:ci:unit", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
24
+ { "gate_id": "audit-harness:ci:crap-score", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
25
+ { "gate_id": "audit-harness:ci:integration", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
26
+ { "gate_id": "audit-harness:ci:contract", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
27
+ { "gate_id": "audit-harness:local:conform-openapi", "dimension": "conformance", "applicability": "required", "enforcement": "advisory", "tool": "spectral" },
28
+ { "gate_id": "audit-harness:ci:sast", "dimension": "security", "applicability": "required", "enforcement": "advisory", "tool": "semgrep" }
29
+ ],
30
+ "frontend": [
31
+ { "gate_id": "audit-harness:ci:unit", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
32
+ { "gate_id": "audit-harness:ci:crap-score", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
33
+ { "gate_id": "audit-harness:ci:e2e", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
34
+ { "gate_id": "audit-harness:ci:a11y", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory", "tool": "axe" },
35
+ { "gate_id": "audit-harness:ci:contract", "dimension": "testing-depth", "applicability": "conditional", "enforcement": "advisory" }
36
+ ],
37
+ "cli": [
38
+ { "gate_id": "audit-harness:ci:unit", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
39
+ { "gate_id": "audit-harness:ci:crap-score", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
40
+ { "gate_id": "audit-harness:ci:smoke", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" }
41
+ ],
42
+ "library": [
43
+ { "gate_id": "audit-harness:ci:unit", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
44
+ { "gate_id": "audit-harness:ci:crap-score", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
45
+ { "gate_id": "audit-harness:ci:property-based", "dimension": "testing-depth", "applicability": "recommended", "enforcement": "advisory" },
46
+ { "gate_id": "audit-harness:ci:cve-osv", "dimension": "security", "applicability": "required", "enforcement": "advisory", "tool": "osv-scanner", "result_class_default": "INDETERMINATE" }
47
+ ],
48
+ "embedded": [
49
+ { "gate_id": "audit-harness:ci:unit", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
50
+ { "gate_id": "audit-harness:ci:fuzz", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
51
+ { "gate_id": "audit-harness:ci:sanitizers", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" }
52
+ ],
53
+ "monorepo": [
54
+ { "gate_id": "audit-harness:local:per-package-classify", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" }
55
+ ],
56
+ "skill": [
57
+ { "gate_id": "audit-harness:local:conform-skillmd", "dimension": "conformance", "applicability": "required", "enforcement": "advisory", "tool": "validate-skillmd" },
58
+ { "gate_id": "audit-harness:server:skill-behavioral", "dimension": "skill-quality", "applicability": "recommended", "enforcement": "advisory", "tool": "j-rig" }
59
+ ],
60
+ "agent": [
61
+ { "gate_id": "audit-harness:local:conform-agent", "dimension": "conformance", "applicability": "required", "enforcement": "advisory", "tool": "validate-agent" }
62
+ ],
63
+ "hook": [
64
+ { "gate_id": "audit-harness:local:conform-hook", "dimension": "conformance", "applicability": "required", "enforcement": "advisory", "tool": "validate-hook" }
65
+ ],
66
+ "mcp": [
67
+ { "gate_id": "audit-harness:local:conform-mcp", "dimension": "conformance", "applicability": "required", "enforcement": "advisory", "tool": "validate-mcp" }
68
+ ],
69
+ "plugin": [
70
+ { "gate_id": "audit-harness:local:conform-plugin", "dimension": "conformance", "applicability": "required", "enforcement": "advisory", "tool": "validate-plugin" },
71
+ { "gate_id": "audit-harness:server:skill-behavioral", "dimension": "skill-quality", "applicability": "recommended", "enforcement": "advisory", "tool": "j-rig" }
72
+ ],
73
+ "marketplace": [
74
+ { "gate_id": "audit-harness:local:conform-marketplace", "dimension": "conformance", "applicability": "required", "enforcement": "advisory", "tool": "validate-marketplace" }
75
+ ],
76
+ "action": [
77
+ { "gate_id": "audit-harness:local:conform-action", "dimension": "conformance", "applicability": "required", "enforcement": "advisory", "tool": "yamllint" },
78
+ { "gate_id": "audit-harness:ci:smoke", "dimension": "testing-depth", "applicability": "recommended", "enforcement": "advisory" }
79
+ ]
80
+ },
81
+ "overlays": {
82
+ "regulated": {
83
+ "description": "Compliance overlay (HIPAA/SOX/PCI-DSS/SOC2/GDPR/FedRAMP markers). Promotes recommended security + conformance gates to required and escalates uncovered SHOULD requirements.",
84
+ "promote_to_required": ["security", "conformance"]
85
+ }
86
+ }
87
+ }