@intentsolutions/audit-harness 1.1.6 → 1.1.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -4,6 +4,22 @@ All notable changes are recorded here. Format follows [Keep a Changelog](https:/
4
4
 
5
5
  ## [Unreleased]
6
6
 
7
+ ### Added — golden-master suite for gherkin-lint + crap-score stdout shapes (iah-golden-master)
8
+
9
+ A fitness function that pins the raw stdout of the two scorers whose output is a downstream contract.
10
+
11
+ - **`tests/golden/run-golden.sh`** captures `gherkin-lint.sh` (text rubric) and `crap-score.py --json` (gate-result envelope) stdout against a `tests/fixtures/deliberate-failure/` corpus and diffs each against a checked-in golden, failing on any drift. Environment-volatile bytes are normalized out (gherkin-lint's installed-vs-awk-fallback first line; crap-score's absolute `summary_path`) so the golden is byte-stable across machines. CI installs no complexity provider, so the crap golden captures the deterministic no-provider envelope shape.
12
+ - **Why this and not the per-row schema gate:** the schema gate validates the *augmented* predicate that `emit-evidence` produces, not the raw scorer stdout. A silent reshape of the scorer stdout — a renamed field, a dropped WARN line, changed summary wording — is a backward-compat break the schema gate cannot see. This suite is that missing guard.
13
+ - Regenerate intentional changes with `bash tests/golden/run-golden.sh --update` and review the golden diff in the PR. Wired into `.github/workflows/ci.yml` as the `golden` job.
14
+
15
+ ### Changed — `install.sh` vendors NOTICE + the Node dispatcher (iah-install-sh-completeness)
16
+
17
+ The vendored-install path (non-Node repos) now ships a complete, traceable copy.
18
+
19
+ - **`NOTICE`** is copied into `.audit-harness/` — Apache-2.0 §4(d) requires the NOTICE file to travel with any distribution, and vendoring is a distribution.
20
+ - **`bin/audit-harness.js`** (the Node CLI dispatcher) and **`package.json`** are copied into `.audit-harness/bin/` + `.audit-harness/` so the canonical dispatcher surface is present and its `--version` (which reads `../package.json`) resolves in the vendored tree.
21
+ - A **`PROVENANCE`** file records the source repo, version, tarball URL, and install timestamp so a vendored tree is traceable back to the exact release it came from.
22
+
7
23
  ### Added — CI-only signed evidence emit for the intent-eval-dashboard (nr75.12)
8
24
 
9
25
  The dashboard reports hub (labs.intentsolutions.io) ingests a signed `report-manifest.json` of kernel `gate-result/v1` rows per repo. This adds audit-harness's own emit, lighting up its row.
@@ -73,6 +89,30 @@ The first piece of the "comprehensive audit, on any repo" build: the read-only b
73
89
 
74
90
  Scope boundary: no `conform` verb, no gate execution yet (Phase 2+). `classify` is read-only and emits a profile only.
75
91
 
92
+ ## [1.1.8] - 2026-06-18
93
+
94
+ Ships the iah-E06 production-signing pre-flight gate to downstream consumers.
95
+
96
+ ### Added — DNSSEC + CAA production-signing pre-flight (iah-E06)
97
+
98
+ Before a production-mode `emit-evidence` run signs canonical bytes, two deterministic pre-flight scripts assert the signing domain is cryptographically sound. Both fail closed: any error, missing record, or unreachable resolver blocks the signing path rather than emitting an unverifiable attestation.
99
+
100
+ - **`scripts/dnssec-check.sh`** — verifies the signing domain's DNSSEC chain is present and validates.
101
+ - **`scripts/caa-check.sh`** — verifies the domain's CAA records authorize the signing certificate authority.
102
+ - The `emit-evidence` production path gates on both before signing; staging/draft emit is unaffected.
103
+
104
+ ### Fixed — query a trusted validating resolver in the DNSSEC + CAA pre-flight (PR #75)
105
+
106
+ The pre-flight previously trusted the ambient resolver, which may not validate DNSSEC. Both scripts now query known validating resolvers (`1.1.1.1`, `8.8.8.8`) and require the authenticated-data (AD) flag plus an `RRSIG` on the answer. A resolver that does not set AD, or an answer with no RRSIG, is treated as a validation failure (fail-closed) rather than a pass.
107
+
108
+ ### Changed — Version bumped to 1.1.8 across all manifests
109
+
110
+ Per the `version-canonical-check` CI gate. `package.json` (canonical), `version.txt`, `python/pyproject.toml`, `python/src/intent_audit_harness/__init__.py`, and `rust/Cargo.toml` all report `1.1.8`.
111
+
112
+ ### Why patch, not minor
113
+
114
+ The pre-flight scripts shipped to the repo in earlier PRs (#70, #75); this patch propagates them to npm consumers via a version bump. No new public CLI commands or flag changes in this release boundary.
115
+
76
116
  ## [v1.1.5] - 2026-06-03
77
117
 
78
118
  ### Added — npm release pipeline (closes the publish-pipeline gap)
@@ -7,7 +7,7 @@
7
7
  * and language-portable. The CLI just adds discoverability + cross-platform-ish shell resolution.
8
8
  */
9
9
  const { spawn } = require('node:child_process');
10
- const { resolve, dirname } = require('node:path');
10
+ const { resolve } = require('node:path');
11
11
  const { existsSync } = require('node:fs');
12
12
 
13
13
  const SCRIPTS = resolve(__dirname, '..', 'scripts');
@@ -85,11 +85,12 @@ Commands:
85
85
  should flag). The metric that gates advisory->blocking
86
86
  promotion. --max-fp-rate X exits 1 if any gate exceeds X.
87
87
  See docs/gate-promotion.md.
88
- currency Advisory upstream-currency report. Reads the per-upstream
88
+ currency Advisory poll-freshness report. Reads the per-upstream
89
89
  pin relation (schemas/currency/pins.v1.json) and flags
90
- pins whose checked_at is past their staleness window.
91
- NO exit-code authority (always exit 0), no live-fetch,
92
- no auto-fix it reports; /sync-testing-harness acts.
90
+ pins whose checked_at is past their poll-freshness SLA
91
+ (the SLA gates nothing but human attention). NO exit-code
92
+ authority (always exit 0), no live-fetch, no auto-fix
93
+ it reports; /sync-testing-harness acts.
93
94
  gen-layer-applicability Project schemas/audit-profile/registry.v1.json into
94
95
  schemas/audit-profile/layer-applicability.md. --write to
95
96
  regenerate, --check to fail on drift (CI gate). The doc
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@intentsolutions/audit-harness",
3
- "version": "1.1.6",
3
+ "version": "1.1.8",
4
4
  "description": "Deterministic test-enforcement harness — escape-scan, hash-pinning, CRAP, architecture checks, bias detection, Gherkin lint. Companion to the audit-tests and implement-tests Claude Code skills.",
5
5
  "license": "Apache-2.0",
6
6
  "author": "Jeremy Longshore <jeremy@intentsolutions.io>",
@@ -40,8 +40,15 @@
40
40
  ],
41
41
  "scripts": {
42
42
  "test": "bash scripts/escape-scan.sh --staged || true",
43
+ "lint": "eslint \"bin/**/*.js\"",
44
+ "lint:fix": "eslint --fix \"bin/**/*.js\"",
43
45
  "prepublishOnly": "node bin/audit-harness.js --version"
44
46
  },
47
+ "devDependencies": {
48
+ "@eslint/js": "^9.39.4",
49
+ "eslint": "^9.39.4",
50
+ "lefthook": "^1.13.6"
51
+ },
45
52
  "publishConfig": {
46
53
  "access": "public"
47
54
  },
@@ -1,34 +1,183 @@
1
1
  {
2
2
  "pins_version": "currency-pins/v1",
3
- "description": "Per-upstream-identity pin relation. Each upstream the harness/skills depend on carries ITS OWN pinned version + the date it was last verified against upstream (checked_at) + a staleness window. The `currency` advisory report reads this datum and flags pins whose checked_at is older than their window — i.e. it makes the PIN'S OWN STALENESS detectable, without ever live-fetching. Currency is advisory-only: it reports + (in /sync-testing-harness) opens PRs; it has no exit-code authority and never auto-fixes. Updating a pin (after a human re-verifies against upstream) is an engineer edit to this file + a fresh checked_at.",
3
+ "description": "Per-upstream-identity pin relation. Each upstream the harness/skills depend on carries ITS OWN pinned version + the date it was last verified against upstream (checked_at) + an advisory poll-freshness SLA (staleness_window_days, resolvable from the pin's class). The `currency` advisory report reads this datum and flags pins whose checked_at is older than their SLA — i.e. it makes the PIN'S OWN STALENESS detectable, without ever live-fetching. The SLA gates NOTHING except human attention: currency is advisory-only it reports + (in /sync-testing-harness) opens PRs; it has no exit-code authority and never auto-fixes. Updating a pin (after a human re-verifies against upstream) is an engineer edit to this file + a fresh checked_at. Pins whose identity matches a surface in the intent-eval-lab upstream-surface registry (specs/upstream-surface-registry.v1.json, 16 monitored surfaces) use the registry surface id as their identity; where the registry declares no version, pinned_version is the sha256 prefix of the lab's vendored snapshot baseline (intent-eval-lab specs/snapshots/.sha/<surface>.sha).",
4
+ "$comment": "2026-06-12 [9k5h.10]: terminology — the former 'bounded-staleness window' framing is now the advisory 'poll-freshness SLA' (same datum, honest name: it is a polling-attention SLA, not a consistency bound, and it gates nothing). Renamed pins to lab-registry surface ids: 'mcp-spec' -> 'mcp-spec-docs', 'claude-code' -> 'claude-code-changelog' ('agentskills-spec' already matched). Added one pin per remaining registry surface (13 new; 16 registry pins total) + kept the 3 internal-contract pins (skill-md-schema, gate-result-predicate, anthropic-sdk). The per-pin 'class' field is an additive, backward-compatible v1 extension (window resolution: explicit staleness_window_days > class SLA > default_staleness_window_days), so no pins.v2.json bump is needed.",
4
5
  "default_staleness_window_days": 90,
6
+ "staleness_classes": {
7
+ "spec-page": {
8
+ "staleness_window_days": 7,
9
+ "description": "Human-readable spec / reference doc pages (agentskills.io, modelcontextprotocol.io, code.claude.com + platform.claude.com .md shims). Re-verify weekly."
10
+ },
11
+ "schema-file": {
12
+ "staleness_window_days": 7,
13
+ "description": "Machine-readable schema files (e.g. the MCP schema.ts) — exact field-level diff possible upstream. Re-verify weekly."
14
+ },
15
+ "release-feed": {
16
+ "staleness_window_days": 3,
17
+ "description": "Release/version signals (GH releases.atom, commits.atom, npm version probes, changelogs, engineering-blog index) — earliest material-change signal, so the tightest SLA. Re-verify every 3 days."
18
+ },
19
+ "internal-contract": {
20
+ "staleness_window_days": null,
21
+ "description": "Intent Solutions internal contracts (not lab-registry surfaces). No shared class SLA; each pin carries its own staleness_window_days."
22
+ }
23
+ },
5
24
  "pins": [
6
25
  {
7
- "identity": "mcp-spec",
26
+ "identity": "agentskills-spec",
27
+ "class": "spec-page",
28
+ "pinned_version": "1.0.0",
29
+ "source": "https://agentskills.io/specification.md",
30
+ "checked_at": "2026-06-12",
31
+ "staleness_window_days": 7,
32
+ "notes": "Open SKILL.md standard (compatibility/metadata/license fields). Lab-registry surface (wave 0, official-spec)."
33
+ },
34
+ {
35
+ "identity": "platform-skills-overview",
36
+ "class": "spec-page",
37
+ "pinned_version": "sha256:0bd9758afca5",
38
+ "source": "https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview.md",
39
+ "checked_at": "2026-06-12",
40
+ "staleness_window_days": 7,
41
+ "notes": "Anthropic doc page about agent skills. Lab-registry surface (wave 0, anthropic-doc); version = vendored snapshot sha prefix."
42
+ },
43
+ {
44
+ "identity": "mcp-spec-docs",
45
+ "class": "spec-page",
8
46
  "pinned_version": "2025-06-18",
9
- "source": "https://spec.modelcontextprotocol.io/ (protocol revision)",
10
- "checked_at": "2026-06-06",
11
- "staleness_window_days": 90,
12
- "notes": "MCP protocol spec revision the .mcp.json conform schema targets."
47
+ "source": "https://modelcontextprotocol.io/specification/draft",
48
+ "checked_at": "2026-06-12",
49
+ "staleness_window_days": 7,
50
+ "notes": "MCP protocol spec revision the .mcp.json conform schema targets. Lab-registry surface (wave 1, official-spec); renamed from 'mcp-spec' 2026-06-12."
51
+ },
52
+ {
53
+ "identity": "claude-hooks",
54
+ "class": "spec-page",
55
+ "pinned_version": "sha256:0b644e2208f8",
56
+ "source": "https://code.claude.com/docs/en/hooks.md",
57
+ "checked_at": "2026-06-12",
58
+ "staleness_window_days": 7,
59
+ "notes": "Claude Code hooks reference (hook-config contract). Lab-registry surface (wave 1, reference); version = vendored snapshot sha prefix."
60
+ },
61
+ {
62
+ "identity": "claude-settings",
63
+ "class": "spec-page",
64
+ "pinned_version": "sha256:491b623ae274",
65
+ "source": "https://code.claude.com/docs/en/settings.md",
66
+ "checked_at": "2026-06-12",
67
+ "staleness_window_days": 7,
68
+ "notes": "Claude Code settings reference (hook-config contract). Lab-registry surface (wave 1, reference); version = vendored snapshot sha prefix."
69
+ },
70
+ {
71
+ "identity": "claude-slash-commands",
72
+ "class": "spec-page",
73
+ "pinned_version": "sha256:d7d367c7d004",
74
+ "source": "https://code.claude.com/docs/en/slash-commands.md",
75
+ "checked_at": "2026-06-12",
76
+ "staleness_window_days": 7,
77
+ "notes": "Claude Code slash-commands reference. Lab-registry surface (wave 1, reference); version = vendored snapshot sha prefix."
78
+ },
79
+ {
80
+ "identity": "plugins-reference",
81
+ "class": "spec-page",
82
+ "pinned_version": "sha256:bbb4618ec8b1",
83
+ "source": "https://code.claude.com/docs/en/plugins-reference.md",
84
+ "checked_at": "2026-06-12",
85
+ "staleness_window_days": 7,
86
+ "notes": "Claude Code plugin-manifest reference. Lab-registry surface (wave 2, reference); version = vendored snapshot sha prefix."
87
+ },
88
+ {
89
+ "identity": "sub-agents",
90
+ "class": "spec-page",
91
+ "pinned_version": "sha256:824162201ae4",
92
+ "source": "https://code.claude.com/docs/en/sub-agents.md",
93
+ "checked_at": "2026-06-12",
94
+ "staleness_window_days": 7,
95
+ "notes": "Claude Code sub-agents reference (agent-definition contract). Lab-registry surface (wave 2, reference); version = vendored snapshot sha prefix."
96
+ },
97
+ {
98
+ "identity": "plugin-marketplaces",
99
+ "class": "spec-page",
100
+ "pinned_version": "sha256:1f37e87ff344",
101
+ "source": "https://code.claude.com/docs/en/plugin-marketplaces.md",
102
+ "checked_at": "2026-06-12",
103
+ "staleness_window_days": 7,
104
+ "notes": "Claude Code marketplace-catalog reference. Lab-registry surface (wave 2, reference); version = vendored snapshot sha prefix."
105
+ },
106
+ {
107
+ "identity": "mcp-schema-ts",
108
+ "class": "schema-file",
109
+ "pinned_version": "sha256:1bf94a601817",
110
+ "source": "https://raw.githubusercontent.com/modelcontextprotocol/modelcontextprotocol/main/schema/draft/schema.ts",
111
+ "checked_at": "2026-06-12",
112
+ "staleness_window_days": 7,
113
+ "notes": "MCP machine-readable schema (mcp-config contract; exact field-level diff possible). Lab-registry surface (wave 1, official-spec-machine-readable); version = vendored snapshot sha prefix."
114
+ },
115
+ {
116
+ "identity": "skills-releases",
117
+ "class": "release-feed",
118
+ "pinned_version": "sha256:8ab0fc2a54fa",
119
+ "source": "https://github.com/anthropics/skills/releases.atom + commits/main.atom",
120
+ "checked_at": "2026-06-12",
121
+ "staleness_window_days": 3,
122
+ "notes": "anthropics/skills release + commit feeds (skill-frontmatter contract). Lab-registry surface (wave 0, release-feed); version = vendored snapshot sha prefix."
123
+ },
124
+ {
125
+ "identity": "mcp-releases",
126
+ "class": "release-feed",
127
+ "pinned_version": "sha256:1b180712c47f",
128
+ "source": "https://github.com/modelcontextprotocol/modelcontextprotocol/releases.atom",
129
+ "checked_at": "2026-06-12",
130
+ "staleness_window_days": 3,
131
+ "notes": "MCP spec-repo release feed (mcp-config contract). Lab-registry surface (wave 2, release-feed); version = vendored snapshot sha prefix."
132
+ },
133
+ {
134
+ "identity": "claude-code-changelog",
135
+ "class": "release-feed",
136
+ "pinned_version": "2.1.152",
137
+ "source": "https://raw.githubusercontent.com/anthropics/claude-code/main/CHANGELOG.md",
138
+ "checked_at": "2026-06-12",
139
+ "staleness_window_days": 3,
140
+ "notes": "Claude Code release changelog (version-signal contract; 2.1.152 added disallowed-tools frontmatter). Lab-registry surface (wave 0, changelog); renamed from 'claude-code' 2026-06-12."
141
+ },
142
+ {
143
+ "identity": "claude-code-npm",
144
+ "class": "release-feed",
145
+ "pinned_version": "2.1.152",
146
+ "source": "npm view @anthropic-ai/claude-code version",
147
+ "checked_at": "2026-06-12",
148
+ "staleness_window_days": 3,
149
+ "notes": "Claude Code npm version probe (version-signal contract). Lab-registry surface (wave 0, changelog); version mirrors the last verified npm version."
150
+ },
151
+ {
152
+ "identity": "claude-code-releases",
153
+ "class": "release-feed",
154
+ "pinned_version": "sha256:556b4faba702",
155
+ "source": "https://github.com/anthropics/claude-code/releases.atom",
156
+ "checked_at": "2026-06-12",
157
+ "staleness_window_days": 3,
158
+ "notes": "Claude Code GH release feed (version-signal contract). Lab-registry surface (wave 2, release-feed); version = vendored snapshot sha prefix."
159
+ },
160
+ {
161
+ "identity": "anthropic-engineering",
162
+ "class": "release-feed",
163
+ "pinned_version": "sha256:f99507064aeb",
164
+ "source": "https://www.anthropic.com/engineering",
165
+ "checked_at": "2026-06-12",
166
+ "staleness_window_days": 3,
167
+ "notes": "Anthropic engineering-blog index (cross-cutting-signal contract). Lab-registry surface (wave 0, release-feed); version = vendored snapshot sha prefix."
13
168
  },
14
169
  {
15
170
  "identity": "skill-md-schema",
171
+ "class": "internal-contract",
16
172
  "pinned_version": "3.7.0",
17
173
  "source": "claude-code-plugins 000-docs/SCHEMA_CHANGELOG.md",
18
174
  "checked_at": "2026-06-06",
19
175
  "staleness_window_days": 90,
20
176
  "notes": "IS SKILL.md schema the conform skillmd-frontmatter floor tracks (full rubric stays in /validate-skillmd)."
21
177
  },
22
- {
23
- "identity": "claude-code",
24
- "pinned_version": "2.1.152",
25
- "source": "https://code.claude.com/docs/en/changelog",
26
- "checked_at": "2026-06-06",
27
- "staleness_window_days": 60,
28
- "notes": "Claude Code release (added disallowed-tools frontmatter at 2.1.152)."
29
- },
30
178
  {
31
179
  "identity": "gate-result-predicate",
180
+ "class": "internal-contract",
32
181
  "pinned_version": "v1",
33
182
  "source": "@intentsolutions/core gate-result/v1 (https://evals.intentsolutions.io/gate-result/v1)",
34
183
  "checked_at": "2026-06-06",
@@ -37,19 +186,12 @@
37
186
  },
38
187
  {
39
188
  "identity": "anthropic-sdk",
189
+ "class": "internal-contract",
40
190
  "pinned_version": "unverified",
41
191
  "source": "https://github.com/anthropics/anthropic-sdk-python (+ -typescript)",
42
192
  "checked_at": "2026-06-06",
43
193
  "staleness_window_days": 90,
44
194
  "notes": "Anthropic SDK surface referenced by downstream skills; pinned_version=unverified until first deliberate verification."
45
- },
46
- {
47
- "identity": "agentskills-spec",
48
- "pinned_version": "1.0.0",
49
- "source": "https://agentskills.io/specification",
50
- "checked_at": "2026-06-06",
51
- "staleness_window_days": 90,
52
- "notes": "Open SKILL.md standard (compatibility/metadata/license fields)."
53
195
  }
54
196
  ]
55
197
  }
@@ -17,6 +17,24 @@
17
17
 
18
18
  set -euo pipefail
19
19
 
20
+ # Bash version floor: these gates rely on bash 4+ features. Refuse early with a
21
+ # clear message on bash 3.x (e.g. macOS system bash) instead of failing later
22
+ # with a cryptic syntax error (jcgw).
23
+ [ "${BASH_VERSINFO:-0}" -ge 4 ] || { echo 'audit-harness requires bash >= 4' >&2; exit 3; }
24
+
25
+ # Cross-platform SHA-256: `sha256sum` ships with GNU coreutils (Linux);
26
+ # macOS only has `shasum -a 256`. Both produce identical `<hash> <file>`
27
+ # output, so downstream awk parsing is unchanged. Same pattern as
28
+ # harness-hash.sh / escape-scan.sh / bias-count.sh.
29
+ if command -v sha256sum >/dev/null 2>&1; then
30
+ SHA256_CMD=(sha256sum)
31
+ elif command -v shasum >/dev/null 2>&1; then
32
+ SHA256_CMD=(shasum -a 256)
33
+ else
34
+ echo "arch-check: neither sha256sum nor shasum found in PATH" >&2
35
+ exit 2
36
+ fi
37
+
20
38
  ROOT="${ROOT:-$(pwd)}"
21
39
  JSON_OUT=0
22
40
  REPORT_DIR="${ROOT}/reports/arch"
@@ -51,12 +69,12 @@ emit_result() {
51
69
  local policy_hash="sha256:0000000000000000000000000000000000000000000000000000000000000000"
52
70
  # Best-effort: input_hash is the source tree fingerprint when running against ROOT/src
53
71
  if [[ -d "${ROOT}/src" ]]; then
54
- input_hash=$(find "${ROOT}/src" -type f \( -name "*.ts" -o -name "*.tsx" -o -name "*.js" -o -name "*.py" -o -name "*.go" -o -name "*.rs" -o -name "*.java" -o -name "*.kt" -o -name "*.cs" -o -name "*.php" \) -exec sha256sum {} \; 2>/dev/null | sort | sha256sum | awk '{print "sha256:"$1}')
72
+ input_hash=$(find "${ROOT}/src" -type f \( -name "*.ts" -o -name "*.tsx" -o -name "*.js" -o -name "*.py" -o -name "*.go" -o -name "*.rs" -o -name "*.java" -o -name "*.kt" -o -name "*.cs" -o -name "*.php" \) -exec "${SHA256_CMD[@]}" {} \; 2>/dev/null | sort | "${SHA256_CMD[@]}" | awk '{print "sha256:"$1}')
55
73
  fi
56
74
  # Hash the architecture rule config (whichever tool's config was used)
57
75
  for cfg in .dependency-cruiser.js .dependency-cruiser.cjs .importlinter deptrac.yaml arch-go.yml; do
58
76
  if [[ -f "${ROOT}/${cfg}" ]]; then
59
- policy_hash=$(sha256sum "${ROOT}/${cfg}" | awk '{print "sha256:"$1}')
77
+ policy_hash=$("${SHA256_CMD[@]}" "${ROOT}/${cfg}" | awk '{print "sha256:"$1}')
60
78
  break
61
79
  fi
62
80
  done
@@ -12,6 +12,23 @@
12
12
 
13
13
  set -euo pipefail
14
14
 
15
+ # Bash version floor: these gates rely on bash 4+ features. Refuse early with a
16
+ # clear message on bash 3.x (e.g. macOS system bash) instead of failing later
17
+ # with a cryptic syntax error (jcgw).
18
+ [ "${BASH_VERSINFO:-0}" -ge 4 ] || { echo 'audit-harness requires bash >= 4' >&2; exit 3; }
19
+
20
+ # Cross-platform SHA-256: `sha256sum` ships with GNU coreutils (Linux);
21
+ # macOS only has `shasum -a 256`. Both produce identical `<hash> <file>`
22
+ # output, so downstream awk parsing is unchanged. Mirrors harness-hash.sh.
23
+ if command -v sha256sum >/dev/null 2>&1; then
24
+ SHA256_CMD=(sha256sum)
25
+ elif command -v shasum >/dev/null 2>&1; then
26
+ SHA256_CMD=(shasum -a 256)
27
+ else
28
+ echo "bias-count: neither sha256sum nor shasum found in PATH" >&2
29
+ exit 2
30
+ fi
31
+
15
32
  JSON_OUT=0
16
33
  TEST_DIR="tests"
17
34
 
@@ -36,7 +53,7 @@ if [ ! -d "$TEST_DIR" ]; then
36
53
  fi
37
54
 
38
55
  # Hash the test directory tree as the "input"
39
- INPUT_HASH=$(find "$TEST_DIR" -type f \( -name "*.py" -o -name "*.ts" -o -name "*.js" -o -name "*.tsx" -o -name "*.jsx" -o -name "*.go" -o -name "*.rs" -o -name "*.java" -o -name "*.kt" -o -name "*.cs" -o -name "*.php" -o -name "*.rb" \) -exec sha256sum {} + 2>/dev/null | sort | sha256sum | awk '{print "sha256:"$1}')
56
+ INPUT_HASH=$(find "$TEST_DIR" -type f \( -name "*.py" -o -name "*.ts" -o -name "*.js" -o -name "*.tsx" -o -name "*.jsx" -o -name "*.go" -o -name "*.rs" -o -name "*.java" -o -name "*.kt" -o -name "*.cs" -o -name "*.php" -o -name "*.rb" \) -exec "${SHA256_CMD[@]}" {} + 2>/dev/null | sort | "${SHA256_CMD[@]}" | awk '{print "sha256:"$1}')
40
57
 
41
58
  if [[ "$JSON_OUT" -eq 1 ]]; then
42
59
  exec 3>&1 # save stdout for the JSON object
@@ -0,0 +1,143 @@
1
+ #!/usr/bin/env bash
2
+ # caa-check.sh — verify a namespace publishes CAA records (and, when configured,
3
+ # pins the EXPECTED certificate authority) before a production signed attestation
4
+ # is anchored against it.
5
+ #
6
+ # WHY THIS EXISTS (CISO binding, DR-010 Q5 / ISEDC v1 Q1 2026-05-10):
7
+ # CAA (RFC 8659) records constrain which CAs may issue certificates for a
8
+ # namespace. Pinning the CA on evals.intentsolutions.io closes the mis-issuance
9
+ # path an attacker could otherwise use to obtain a look-alike cert and present
10
+ # forged attestation infrastructure. This must be verified BEFORE the first
11
+ # production attestation. This script is that gate — read-only, fail-closed.
12
+ #
13
+ # WHY IT QUERIES AN EXPLICIT RESOLVER (the bug this version fixes):
14
+ # Querying the LOCAL STUB RESOLVER (plain `dig`, no `@server`) FALSE-NEGATIVES
15
+ # on hosts whose stub resolver lags CAA propagation or strips the record type
16
+ # (systemd-resolved, many CI runners, dev boxes). On such a host a correctly
17
+ # CAA-pinned zone looks like it has no CAA, and the gate refuses a legitimate
18
+ # production sign. The fix is to query a TRUSTED PUBLIC resolver. The gate
19
+ # stays fail-closed: PASS only on a positive matching CAA record from a trusted
20
+ # resolver; absence / mismatch / unreachable => non-zero.
21
+ #
22
+ # Usage:
23
+ # bash scripts/caa-check.sh [DOMAIN]
24
+ # EXPECTED_CAA_ISSUER=letsencrypt.org bash scripts/caa-check.sh evals.intentsolutions.io
25
+ #
26
+ # Resolution order for the domain:
27
+ # 1. $1 (positional)
28
+ # 2. $CAA_CHECK_DOMAIN
29
+ # 3. default: evals.intentsolutions.io
30
+ #
31
+ # Issuer policy:
32
+ # - EXPECTED_CAA_ISSUER (env) — when set, at least one CAA `issue` (or
33
+ # `issuewild`) record MUST name this CA, else the check FAILS (exit 1).
34
+ # Default: letsencrypt.org (the CA the IS public-namespace certs are issued
35
+ # by). Override per-deployment.
36
+ # - EXPECTED_CAA_ISSUER=ANY (case-insensitive) — relax to "any CAA record is
37
+ # acceptable"; presence of ANY CAA record passes, absence fails, and a
38
+ # warning is emitted that no specific CA is being pinned.
39
+ #
40
+ # Exit codes:
41
+ # 0 — CAA verified (present at a trusted resolver, and matches
42
+ # EXPECTED_CAA_ISSUER when a specific issuer is required)
43
+ # 1 — CAA NOT verified (no CAA records, or expected issuer not present, from
44
+ # any trusted resolver)
45
+ # 2 — UNKNOWN/UNREACHABLE (no resolver tool installed)
46
+ #
47
+ # Override knobs:
48
+ # CAA_CHECK_RESOLVERS — space-separated list of trusted public resolvers to
49
+ # query in order (default: "1.1.1.1 8.8.8.8").
50
+ # CAA_CHECK_DIG_CMD — command used in place of `dig` (default: dig)
51
+
52
+ set -euo pipefail
53
+
54
+ DOMAIN="${1:-${CAA_CHECK_DOMAIN:-evals.intentsolutions.io}}"
55
+ EXPECTED_CAA_ISSUER="${EXPECTED_CAA_ISSUER:-letsencrypt.org}"
56
+ DIG_CMD="${CAA_CHECK_DIG_CMD:-dig}"
57
+ # Trusted public resolvers, queried in order, until one returns a CAA record.
58
+ RESOLVERS="${CAA_CHECK_RESOLVERS:-1.1.1.1 8.8.8.8}"
59
+
60
+ log() { printf 'caa-check: %s\n' "$1" >&2; }
61
+
62
+ if [[ "$DOMAIN" == "-h" || "$DOMAIN" == "--help" ]]; then
63
+ sed -n '2,60p' "$0"
64
+ exit 0
65
+ fi
66
+
67
+ have() { command -v "$1" >/dev/null 2>&1; }
68
+
69
+ if ! have "$DIG_CMD"; then
70
+ log "UNKNOWN/UNREACHABLE — '$DIG_CMD' is not installed; cannot look up CAA for '$DOMAIN'"
71
+ log " failing closed (production must not sign on UNKNOWN)"
72
+ log " remediation: install bind9-dnsutils (provides dig) on the signing host"
73
+ exit 2
74
+ fi
75
+
76
+ # issuer_matches CAA_TEXT -> 0 if a matching issue/issuewild record is present.
77
+ # Match any `issue` or `issuewild` property whose value contains the expected
78
+ # CA. CAA values are quoted; we match case-insensitively on the issuer substring.
79
+ issuer_matches() {
80
+ printf '%s\n' "$1" \
81
+ | grep -iE '[[:space:]]issue(wild)?[[:space:]]' \
82
+ | grep -iqF "$EXPECTED_CAA_ISSUER"
83
+ }
84
+
85
+ # is_blank CAA_TEXT -> 0 if the text is empty after stripping whitespace.
86
+ is_blank() {
87
+ [[ -z "${1//[$' \t\r\n']/}" ]]
88
+ }
89
+
90
+ last_caa_out="" # records from the last resolver that returned ANY CAA records
91
+ saw_records=0 # at least one trusted resolver returned CAA records
92
+
93
+ shopt -s nocasematch
94
+ relax_any=0
95
+ [[ "$EXPECTED_CAA_ISSUER" == "ANY" ]] && relax_any=1
96
+ shopt -u nocasematch
97
+
98
+ for resolver in $RESOLVERS; do
99
+ log "looking up CAA records for '$DOMAIN' via $DIG_CMD @$resolver"
100
+ # `dig @resolver +short CAA` prints one line per record, e.g.:
101
+ # 0 issue "letsencrypt.org"
102
+ # 0 issuewild ";"
103
+ caa_out="$("$DIG_CMD" "@$resolver" +short CAA "$DOMAIN" 2>/dev/null || true)"
104
+
105
+ if is_blank "$caa_out"; then
106
+ log " no CAA records returned by @$resolver"
107
+ continue
108
+ fi
109
+
110
+ saw_records=1
111
+ last_caa_out="$caa_out"
112
+
113
+ # --- ANY-issuer relaxation: any CAA record present passes ---
114
+ if [[ "$relax_any" -eq 1 ]]; then
115
+ log "VERIFIED (presence only) — CAA records exist for '$DOMAIN' (via @$resolver)"
116
+ log " WARNING: EXPECTED_CAA_ISSUER=ANY — no specific CA is being pinned."
117
+ log " Records found:"
118
+ printf '%s\n' "$caa_out" | sed 's/^/ /' >&2
119
+ exit 0
120
+ fi
121
+
122
+ # --- Specific-issuer pinning ---
123
+ if issuer_matches "$caa_out"; then
124
+ log "VERIFIED — '$DOMAIN' pins issuance to '$EXPECTED_CAA_ISSUER' (via @$resolver)"
125
+ exit 0
126
+ fi
127
+
128
+ log " CAA records exist at @$resolver but none pin '$EXPECTED_CAA_ISSUER'; trying next resolver"
129
+ done
130
+
131
+ # No trusted resolver yielded a matching CAA record -> fail-closed (exit 1).
132
+ if [[ "$saw_records" -eq 1 ]]; then
133
+ log "NOT VERIFIED — CAA records exist for '$DOMAIN' but none pin '$EXPECTED_CAA_ISSUER'"
134
+ log " Records found:"
135
+ printf '%s\n' "$last_caa_out" | sed 's/^/ /' >&2
136
+ log " remediation: add a CAA record pinning the expected CA, or set"
137
+ log " EXPECTED_CAA_ISSUER to the CA actually published (or ANY to accept any CAA)."
138
+ else
139
+ log "NOT VERIFIED — no CAA records found for '$DOMAIN' (resolvers tried: $RESOLVERS)"
140
+ log " remediation: publish a CAA record pinning the issuing CA, e.g.:"
141
+ log " $DOMAIN. CAA 0 issue \"$EXPECTED_CAA_ISSUER\""
142
+ fi
143
+ exit 1