@intentsolutions/audit-harness 1.1.6 → 1.1.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +40 -0
- package/bin/audit-harness.js +6 -5
- package/package.json +8 -1
- package/schemas/currency/pins.v1.json +164 -22
- package/scripts/arch-check.sh +20 -2
- package/scripts/bias-count.sh +18 -1
- package/scripts/caa-check.sh +143 -0
- package/scripts/crap-score.py +57 -6
- package/scripts/currency.py +70 -25
- package/scripts/dnssec-check.sh +158 -0
- package/scripts/emit-evidence.sh +79 -9
- package/scripts/escape-scan.sh +28 -3
- package/scripts/gherkin-lint.sh +5 -0
- package/scripts/harness-hash.sh +5 -0
- package/scripts/kernel-shadow-check.sh +132 -0
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,22 @@ All notable changes are recorded here. Format follows [Keep a Changelog](https:/
|
|
|
4
4
|
|
|
5
5
|
## [Unreleased]
|
|
6
6
|
|
|
7
|
+
### Added — golden-master suite for gherkin-lint + crap-score stdout shapes (iah-golden-master)
|
|
8
|
+
|
|
9
|
+
A fitness function that pins the raw stdout of the two scorers whose output is a downstream contract.
|
|
10
|
+
|
|
11
|
+
- **`tests/golden/run-golden.sh`** captures `gherkin-lint.sh` (text rubric) and `crap-score.py --json` (gate-result envelope) stdout against a `tests/fixtures/deliberate-failure/` corpus and diffs each against a checked-in golden, failing on any drift. Environment-volatile bytes are normalized out (gherkin-lint's installed-vs-awk-fallback first line; crap-score's absolute `summary_path`) so the golden is byte-stable across machines. CI installs no complexity provider, so the crap golden captures the deterministic no-provider envelope shape.
|
|
12
|
+
- **Why this and not the per-row schema gate:** the schema gate validates the *augmented* predicate that `emit-evidence` produces, not the raw scorer stdout. A silent reshape of the scorer stdout — a renamed field, a dropped WARN line, changed summary wording — is a backward-compat break the schema gate cannot see. This suite is that missing guard.
|
|
13
|
+
- Regenerate intentional changes with `bash tests/golden/run-golden.sh --update` and review the golden diff in the PR. Wired into `.github/workflows/ci.yml` as the `golden` job.
|
|
14
|
+
|
|
15
|
+
### Changed — `install.sh` vendors NOTICE + the Node dispatcher (iah-install-sh-completeness)
|
|
16
|
+
|
|
17
|
+
The vendored-install path (non-Node repos) now ships a complete, traceable copy.
|
|
18
|
+
|
|
19
|
+
- **`NOTICE`** is copied into `.audit-harness/` — Apache-2.0 §4(d) requires the NOTICE file to travel with any distribution, and vendoring is a distribution.
|
|
20
|
+
- **`bin/audit-harness.js`** (the Node CLI dispatcher) and **`package.json`** are copied into `.audit-harness/bin/` + `.audit-harness/` so the canonical dispatcher surface is present and its `--version` (which reads `../package.json`) resolves in the vendored tree.
|
|
21
|
+
- A **`PROVENANCE`** file records the source repo, version, tarball URL, and install timestamp so a vendored tree is traceable back to the exact release it came from.
|
|
22
|
+
|
|
7
23
|
### Added — CI-only signed evidence emit for the intent-eval-dashboard (nr75.12)
|
|
8
24
|
|
|
9
25
|
The dashboard reports hub (labs.intentsolutions.io) ingests a signed `report-manifest.json` of kernel `gate-result/v1` rows per repo. This adds audit-harness's own emit, lighting up its row.
|
|
@@ -73,6 +89,30 @@ The first piece of the "comprehensive audit, on any repo" build: the read-only b
|
|
|
73
89
|
|
|
74
90
|
Scope boundary: no `conform` verb, no gate execution yet (Phase 2+). `classify` is read-only and emits a profile only.
|
|
75
91
|
|
|
92
|
+
## [1.1.8] - 2026-06-18
|
|
93
|
+
|
|
94
|
+
Ships the iah-E06 production-signing pre-flight gate to downstream consumers.
|
|
95
|
+
|
|
96
|
+
### Added — DNSSEC + CAA production-signing pre-flight (iah-E06)
|
|
97
|
+
|
|
98
|
+
Before a production-mode `emit-evidence` run signs canonical bytes, two deterministic pre-flight scripts assert the signing domain is cryptographically sound. Both fail closed: any error, missing record, or unreachable resolver blocks the signing path rather than emitting an unverifiable attestation.
|
|
99
|
+
|
|
100
|
+
- **`scripts/dnssec-check.sh`** — verifies the signing domain's DNSSEC chain is present and validates.
|
|
101
|
+
- **`scripts/caa-check.sh`** — verifies the domain's CAA records authorize the signing certificate authority.
|
|
102
|
+
- The `emit-evidence` production path gates on both before signing; staging/draft emit is unaffected.
|
|
103
|
+
|
|
104
|
+
### Fixed — query a trusted validating resolver in the DNSSEC + CAA pre-flight (PR #75)
|
|
105
|
+
|
|
106
|
+
The pre-flight previously trusted the ambient resolver, which may not validate DNSSEC. Both scripts now query known validating resolvers (`1.1.1.1`, `8.8.8.8`) and require the authenticated-data (AD) flag plus an `RRSIG` on the answer. A resolver that does not set AD, or an answer with no RRSIG, is treated as a validation failure (fail-closed) rather than a pass.
|
|
107
|
+
|
|
108
|
+
### Changed — Version bumped to 1.1.8 across all manifests
|
|
109
|
+
|
|
110
|
+
Per the `version-canonical-check` CI gate. `package.json` (canonical), `version.txt`, `python/pyproject.toml`, `python/src/intent_audit_harness/__init__.py`, and `rust/Cargo.toml` all report `1.1.8`.
|
|
111
|
+
|
|
112
|
+
### Why patch, not minor
|
|
113
|
+
|
|
114
|
+
The pre-flight scripts shipped to the repo in earlier PRs (#70, #75); this patch propagates them to npm consumers via a version bump. No new public CLI commands or flag changes in this release boundary.
|
|
115
|
+
|
|
76
116
|
## [v1.1.5] - 2026-06-03
|
|
77
117
|
|
|
78
118
|
### Added — npm release pipeline (closes the publish-pipeline gap)
|
package/bin/audit-harness.js
CHANGED
|
@@ -7,7 +7,7 @@
|
|
|
7
7
|
* and language-portable. The CLI just adds discoverability + cross-platform-ish shell resolution.
|
|
8
8
|
*/
|
|
9
9
|
const { spawn } = require('node:child_process');
|
|
10
|
-
const { resolve
|
|
10
|
+
const { resolve } = require('node:path');
|
|
11
11
|
const { existsSync } = require('node:fs');
|
|
12
12
|
|
|
13
13
|
const SCRIPTS = resolve(__dirname, '..', 'scripts');
|
|
@@ -85,11 +85,12 @@ Commands:
|
|
|
85
85
|
should flag). The metric that gates advisory->blocking
|
|
86
86
|
promotion. --max-fp-rate X exits 1 if any gate exceeds X.
|
|
87
87
|
See docs/gate-promotion.md.
|
|
88
|
-
currency Advisory
|
|
88
|
+
currency Advisory poll-freshness report. Reads the per-upstream
|
|
89
89
|
pin relation (schemas/currency/pins.v1.json) and flags
|
|
90
|
-
pins whose checked_at is past their
|
|
91
|
-
|
|
92
|
-
|
|
90
|
+
pins whose checked_at is past their poll-freshness SLA
|
|
91
|
+
(the SLA gates nothing but human attention). NO exit-code
|
|
92
|
+
authority (always exit 0), no live-fetch, no auto-fix —
|
|
93
|
+
it reports; /sync-testing-harness acts.
|
|
93
94
|
gen-layer-applicability Project schemas/audit-profile/registry.v1.json into
|
|
94
95
|
schemas/audit-profile/layer-applicability.md. --write to
|
|
95
96
|
regenerate, --check to fail on drift (CI gate). The doc
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@intentsolutions/audit-harness",
|
|
3
|
-
"version": "1.1.
|
|
3
|
+
"version": "1.1.8",
|
|
4
4
|
"description": "Deterministic test-enforcement harness — escape-scan, hash-pinning, CRAP, architecture checks, bias detection, Gherkin lint. Companion to the audit-tests and implement-tests Claude Code skills.",
|
|
5
5
|
"license": "Apache-2.0",
|
|
6
6
|
"author": "Jeremy Longshore <jeremy@intentsolutions.io>",
|
|
@@ -40,8 +40,15 @@
|
|
|
40
40
|
],
|
|
41
41
|
"scripts": {
|
|
42
42
|
"test": "bash scripts/escape-scan.sh --staged || true",
|
|
43
|
+
"lint": "eslint \"bin/**/*.js\"",
|
|
44
|
+
"lint:fix": "eslint --fix \"bin/**/*.js\"",
|
|
43
45
|
"prepublishOnly": "node bin/audit-harness.js --version"
|
|
44
46
|
},
|
|
47
|
+
"devDependencies": {
|
|
48
|
+
"@eslint/js": "^9.39.4",
|
|
49
|
+
"eslint": "^9.39.4",
|
|
50
|
+
"lefthook": "^1.13.6"
|
|
51
|
+
},
|
|
45
52
|
"publishConfig": {
|
|
46
53
|
"access": "public"
|
|
47
54
|
},
|
|
@@ -1,34 +1,183 @@
|
|
|
1
1
|
{
|
|
2
2
|
"pins_version": "currency-pins/v1",
|
|
3
|
-
"description": "Per-upstream-identity pin relation. Each upstream the harness/skills depend on carries ITS OWN pinned version + the date it was last verified against upstream (checked_at) +
|
|
3
|
+
"description": "Per-upstream-identity pin relation. Each upstream the harness/skills depend on carries ITS OWN pinned version + the date it was last verified against upstream (checked_at) + an advisory poll-freshness SLA (staleness_window_days, resolvable from the pin's class). The `currency` advisory report reads this datum and flags pins whose checked_at is older than their SLA — i.e. it makes the PIN'S OWN STALENESS detectable, without ever live-fetching. The SLA gates NOTHING except human attention: currency is advisory-only — it reports + (in /sync-testing-harness) opens PRs; it has no exit-code authority and never auto-fixes. Updating a pin (after a human re-verifies against upstream) is an engineer edit to this file + a fresh checked_at. Pins whose identity matches a surface in the intent-eval-lab upstream-surface registry (specs/upstream-surface-registry.v1.json, 16 monitored surfaces) use the registry surface id as their identity; where the registry declares no version, pinned_version is the sha256 prefix of the lab's vendored snapshot baseline (intent-eval-lab specs/snapshots/.sha/<surface>.sha).",
|
|
4
|
+
"$comment": "2026-06-12 [9k5h.10]: terminology — the former 'bounded-staleness window' framing is now the advisory 'poll-freshness SLA' (same datum, honest name: it is a polling-attention SLA, not a consistency bound, and it gates nothing). Renamed pins to lab-registry surface ids: 'mcp-spec' -> 'mcp-spec-docs', 'claude-code' -> 'claude-code-changelog' ('agentskills-spec' already matched). Added one pin per remaining registry surface (13 new; 16 registry pins total) + kept the 3 internal-contract pins (skill-md-schema, gate-result-predicate, anthropic-sdk). The per-pin 'class' field is an additive, backward-compatible v1 extension (window resolution: explicit staleness_window_days > class SLA > default_staleness_window_days), so no pins.v2.json bump is needed.",
|
|
4
5
|
"default_staleness_window_days": 90,
|
|
6
|
+
"staleness_classes": {
|
|
7
|
+
"spec-page": {
|
|
8
|
+
"staleness_window_days": 7,
|
|
9
|
+
"description": "Human-readable spec / reference doc pages (agentskills.io, modelcontextprotocol.io, code.claude.com + platform.claude.com .md shims). Re-verify weekly."
|
|
10
|
+
},
|
|
11
|
+
"schema-file": {
|
|
12
|
+
"staleness_window_days": 7,
|
|
13
|
+
"description": "Machine-readable schema files (e.g. the MCP schema.ts) — exact field-level diff possible upstream. Re-verify weekly."
|
|
14
|
+
},
|
|
15
|
+
"release-feed": {
|
|
16
|
+
"staleness_window_days": 3,
|
|
17
|
+
"description": "Release/version signals (GH releases.atom, commits.atom, npm version probes, changelogs, engineering-blog index) — earliest material-change signal, so the tightest SLA. Re-verify every 3 days."
|
|
18
|
+
},
|
|
19
|
+
"internal-contract": {
|
|
20
|
+
"staleness_window_days": null,
|
|
21
|
+
"description": "Intent Solutions internal contracts (not lab-registry surfaces). No shared class SLA; each pin carries its own staleness_window_days."
|
|
22
|
+
}
|
|
23
|
+
},
|
|
5
24
|
"pins": [
|
|
6
25
|
{
|
|
7
|
-
"identity": "
|
|
26
|
+
"identity": "agentskills-spec",
|
|
27
|
+
"class": "spec-page",
|
|
28
|
+
"pinned_version": "1.0.0",
|
|
29
|
+
"source": "https://agentskills.io/specification.md",
|
|
30
|
+
"checked_at": "2026-06-12",
|
|
31
|
+
"staleness_window_days": 7,
|
|
32
|
+
"notes": "Open SKILL.md standard (compatibility/metadata/license fields). Lab-registry surface (wave 0, official-spec)."
|
|
33
|
+
},
|
|
34
|
+
{
|
|
35
|
+
"identity": "platform-skills-overview",
|
|
36
|
+
"class": "spec-page",
|
|
37
|
+
"pinned_version": "sha256:0bd9758afca5",
|
|
38
|
+
"source": "https://platform.claude.com/docs/en/agents-and-tools/agent-skills/overview.md",
|
|
39
|
+
"checked_at": "2026-06-12",
|
|
40
|
+
"staleness_window_days": 7,
|
|
41
|
+
"notes": "Anthropic doc page about agent skills. Lab-registry surface (wave 0, anthropic-doc); version = vendored snapshot sha prefix."
|
|
42
|
+
},
|
|
43
|
+
{
|
|
44
|
+
"identity": "mcp-spec-docs",
|
|
45
|
+
"class": "spec-page",
|
|
8
46
|
"pinned_version": "2025-06-18",
|
|
9
|
-
"source": "https://
|
|
10
|
-
"checked_at": "2026-06-
|
|
11
|
-
"staleness_window_days":
|
|
12
|
-
"notes": "MCP protocol spec revision the .mcp.json conform schema targets."
|
|
47
|
+
"source": "https://modelcontextprotocol.io/specification/draft",
|
|
48
|
+
"checked_at": "2026-06-12",
|
|
49
|
+
"staleness_window_days": 7,
|
|
50
|
+
"notes": "MCP protocol spec revision the .mcp.json conform schema targets. Lab-registry surface (wave 1, official-spec); renamed from 'mcp-spec' 2026-06-12."
|
|
51
|
+
},
|
|
52
|
+
{
|
|
53
|
+
"identity": "claude-hooks",
|
|
54
|
+
"class": "spec-page",
|
|
55
|
+
"pinned_version": "sha256:0b644e2208f8",
|
|
56
|
+
"source": "https://code.claude.com/docs/en/hooks.md",
|
|
57
|
+
"checked_at": "2026-06-12",
|
|
58
|
+
"staleness_window_days": 7,
|
|
59
|
+
"notes": "Claude Code hooks reference (hook-config contract). Lab-registry surface (wave 1, reference); version = vendored snapshot sha prefix."
|
|
60
|
+
},
|
|
61
|
+
{
|
|
62
|
+
"identity": "claude-settings",
|
|
63
|
+
"class": "spec-page",
|
|
64
|
+
"pinned_version": "sha256:491b623ae274",
|
|
65
|
+
"source": "https://code.claude.com/docs/en/settings.md",
|
|
66
|
+
"checked_at": "2026-06-12",
|
|
67
|
+
"staleness_window_days": 7,
|
|
68
|
+
"notes": "Claude Code settings reference (hook-config contract). Lab-registry surface (wave 1, reference); version = vendored snapshot sha prefix."
|
|
69
|
+
},
|
|
70
|
+
{
|
|
71
|
+
"identity": "claude-slash-commands",
|
|
72
|
+
"class": "spec-page",
|
|
73
|
+
"pinned_version": "sha256:d7d367c7d004",
|
|
74
|
+
"source": "https://code.claude.com/docs/en/slash-commands.md",
|
|
75
|
+
"checked_at": "2026-06-12",
|
|
76
|
+
"staleness_window_days": 7,
|
|
77
|
+
"notes": "Claude Code slash-commands reference. Lab-registry surface (wave 1, reference); version = vendored snapshot sha prefix."
|
|
78
|
+
},
|
|
79
|
+
{
|
|
80
|
+
"identity": "plugins-reference",
|
|
81
|
+
"class": "spec-page",
|
|
82
|
+
"pinned_version": "sha256:bbb4618ec8b1",
|
|
83
|
+
"source": "https://code.claude.com/docs/en/plugins-reference.md",
|
|
84
|
+
"checked_at": "2026-06-12",
|
|
85
|
+
"staleness_window_days": 7,
|
|
86
|
+
"notes": "Claude Code plugin-manifest reference. Lab-registry surface (wave 2, reference); version = vendored snapshot sha prefix."
|
|
87
|
+
},
|
|
88
|
+
{
|
|
89
|
+
"identity": "sub-agents",
|
|
90
|
+
"class": "spec-page",
|
|
91
|
+
"pinned_version": "sha256:824162201ae4",
|
|
92
|
+
"source": "https://code.claude.com/docs/en/sub-agents.md",
|
|
93
|
+
"checked_at": "2026-06-12",
|
|
94
|
+
"staleness_window_days": 7,
|
|
95
|
+
"notes": "Claude Code sub-agents reference (agent-definition contract). Lab-registry surface (wave 2, reference); version = vendored snapshot sha prefix."
|
|
96
|
+
},
|
|
97
|
+
{
|
|
98
|
+
"identity": "plugin-marketplaces",
|
|
99
|
+
"class": "spec-page",
|
|
100
|
+
"pinned_version": "sha256:1f37e87ff344",
|
|
101
|
+
"source": "https://code.claude.com/docs/en/plugin-marketplaces.md",
|
|
102
|
+
"checked_at": "2026-06-12",
|
|
103
|
+
"staleness_window_days": 7,
|
|
104
|
+
"notes": "Claude Code marketplace-catalog reference. Lab-registry surface (wave 2, reference); version = vendored snapshot sha prefix."
|
|
105
|
+
},
|
|
106
|
+
{
|
|
107
|
+
"identity": "mcp-schema-ts",
|
|
108
|
+
"class": "schema-file",
|
|
109
|
+
"pinned_version": "sha256:1bf94a601817",
|
|
110
|
+
"source": "https://raw.githubusercontent.com/modelcontextprotocol/modelcontextprotocol/main/schema/draft/schema.ts",
|
|
111
|
+
"checked_at": "2026-06-12",
|
|
112
|
+
"staleness_window_days": 7,
|
|
113
|
+
"notes": "MCP machine-readable schema (mcp-config contract; exact field-level diff possible). Lab-registry surface (wave 1, official-spec-machine-readable); version = vendored snapshot sha prefix."
|
|
114
|
+
},
|
|
115
|
+
{
|
|
116
|
+
"identity": "skills-releases",
|
|
117
|
+
"class": "release-feed",
|
|
118
|
+
"pinned_version": "sha256:8ab0fc2a54fa",
|
|
119
|
+
"source": "https://github.com/anthropics/skills/releases.atom + commits/main.atom",
|
|
120
|
+
"checked_at": "2026-06-12",
|
|
121
|
+
"staleness_window_days": 3,
|
|
122
|
+
"notes": "anthropics/skills release + commit feeds (skill-frontmatter contract). Lab-registry surface (wave 0, release-feed); version = vendored snapshot sha prefix."
|
|
123
|
+
},
|
|
124
|
+
{
|
|
125
|
+
"identity": "mcp-releases",
|
|
126
|
+
"class": "release-feed",
|
|
127
|
+
"pinned_version": "sha256:1b180712c47f",
|
|
128
|
+
"source": "https://github.com/modelcontextprotocol/modelcontextprotocol/releases.atom",
|
|
129
|
+
"checked_at": "2026-06-12",
|
|
130
|
+
"staleness_window_days": 3,
|
|
131
|
+
"notes": "MCP spec-repo release feed (mcp-config contract). Lab-registry surface (wave 2, release-feed); version = vendored snapshot sha prefix."
|
|
132
|
+
},
|
|
133
|
+
{
|
|
134
|
+
"identity": "claude-code-changelog",
|
|
135
|
+
"class": "release-feed",
|
|
136
|
+
"pinned_version": "2.1.152",
|
|
137
|
+
"source": "https://raw.githubusercontent.com/anthropics/claude-code/main/CHANGELOG.md",
|
|
138
|
+
"checked_at": "2026-06-12",
|
|
139
|
+
"staleness_window_days": 3,
|
|
140
|
+
"notes": "Claude Code release changelog (version-signal contract; 2.1.152 added disallowed-tools frontmatter). Lab-registry surface (wave 0, changelog); renamed from 'claude-code' 2026-06-12."
|
|
141
|
+
},
|
|
142
|
+
{
|
|
143
|
+
"identity": "claude-code-npm",
|
|
144
|
+
"class": "release-feed",
|
|
145
|
+
"pinned_version": "2.1.152",
|
|
146
|
+
"source": "npm view @anthropic-ai/claude-code version",
|
|
147
|
+
"checked_at": "2026-06-12",
|
|
148
|
+
"staleness_window_days": 3,
|
|
149
|
+
"notes": "Claude Code npm version probe (version-signal contract). Lab-registry surface (wave 0, changelog); version mirrors the last verified npm version."
|
|
150
|
+
},
|
|
151
|
+
{
|
|
152
|
+
"identity": "claude-code-releases",
|
|
153
|
+
"class": "release-feed",
|
|
154
|
+
"pinned_version": "sha256:556b4faba702",
|
|
155
|
+
"source": "https://github.com/anthropics/claude-code/releases.atom",
|
|
156
|
+
"checked_at": "2026-06-12",
|
|
157
|
+
"staleness_window_days": 3,
|
|
158
|
+
"notes": "Claude Code GH release feed (version-signal contract). Lab-registry surface (wave 2, release-feed); version = vendored snapshot sha prefix."
|
|
159
|
+
},
|
|
160
|
+
{
|
|
161
|
+
"identity": "anthropic-engineering",
|
|
162
|
+
"class": "release-feed",
|
|
163
|
+
"pinned_version": "sha256:f99507064aeb",
|
|
164
|
+
"source": "https://www.anthropic.com/engineering",
|
|
165
|
+
"checked_at": "2026-06-12",
|
|
166
|
+
"staleness_window_days": 3,
|
|
167
|
+
"notes": "Anthropic engineering-blog index (cross-cutting-signal contract). Lab-registry surface (wave 0, release-feed); version = vendored snapshot sha prefix."
|
|
13
168
|
},
|
|
14
169
|
{
|
|
15
170
|
"identity": "skill-md-schema",
|
|
171
|
+
"class": "internal-contract",
|
|
16
172
|
"pinned_version": "3.7.0",
|
|
17
173
|
"source": "claude-code-plugins 000-docs/SCHEMA_CHANGELOG.md",
|
|
18
174
|
"checked_at": "2026-06-06",
|
|
19
175
|
"staleness_window_days": 90,
|
|
20
176
|
"notes": "IS SKILL.md schema the conform skillmd-frontmatter floor tracks (full rubric stays in /validate-skillmd)."
|
|
21
177
|
},
|
|
22
|
-
{
|
|
23
|
-
"identity": "claude-code",
|
|
24
|
-
"pinned_version": "2.1.152",
|
|
25
|
-
"source": "https://code.claude.com/docs/en/changelog",
|
|
26
|
-
"checked_at": "2026-06-06",
|
|
27
|
-
"staleness_window_days": 60,
|
|
28
|
-
"notes": "Claude Code release (added disallowed-tools frontmatter at 2.1.152)."
|
|
29
|
-
},
|
|
30
178
|
{
|
|
31
179
|
"identity": "gate-result-predicate",
|
|
180
|
+
"class": "internal-contract",
|
|
32
181
|
"pinned_version": "v1",
|
|
33
182
|
"source": "@intentsolutions/core gate-result/v1 (https://evals.intentsolutions.io/gate-result/v1)",
|
|
34
183
|
"checked_at": "2026-06-06",
|
|
@@ -37,19 +186,12 @@
|
|
|
37
186
|
},
|
|
38
187
|
{
|
|
39
188
|
"identity": "anthropic-sdk",
|
|
189
|
+
"class": "internal-contract",
|
|
40
190
|
"pinned_version": "unverified",
|
|
41
191
|
"source": "https://github.com/anthropics/anthropic-sdk-python (+ -typescript)",
|
|
42
192
|
"checked_at": "2026-06-06",
|
|
43
193
|
"staleness_window_days": 90,
|
|
44
194
|
"notes": "Anthropic SDK surface referenced by downstream skills; pinned_version=unverified until first deliberate verification."
|
|
45
|
-
},
|
|
46
|
-
{
|
|
47
|
-
"identity": "agentskills-spec",
|
|
48
|
-
"pinned_version": "1.0.0",
|
|
49
|
-
"source": "https://agentskills.io/specification",
|
|
50
|
-
"checked_at": "2026-06-06",
|
|
51
|
-
"staleness_window_days": 90,
|
|
52
|
-
"notes": "Open SKILL.md standard (compatibility/metadata/license fields)."
|
|
53
195
|
}
|
|
54
196
|
]
|
|
55
197
|
}
|
package/scripts/arch-check.sh
CHANGED
|
@@ -17,6 +17,24 @@
|
|
|
17
17
|
|
|
18
18
|
set -euo pipefail
|
|
19
19
|
|
|
20
|
+
# Bash version floor: these gates rely on bash 4+ features. Refuse early with a
|
|
21
|
+
# clear message on bash 3.x (e.g. macOS system bash) instead of failing later
|
|
22
|
+
# with a cryptic syntax error (jcgw).
|
|
23
|
+
[ "${BASH_VERSINFO:-0}" -ge 4 ] || { echo 'audit-harness requires bash >= 4' >&2; exit 3; }
|
|
24
|
+
|
|
25
|
+
# Cross-platform SHA-256: `sha256sum` ships with GNU coreutils (Linux);
|
|
26
|
+
# macOS only has `shasum -a 256`. Both produce identical `<hash> <file>`
|
|
27
|
+
# output, so downstream awk parsing is unchanged. Same pattern as
|
|
28
|
+
# harness-hash.sh / escape-scan.sh / bias-count.sh.
|
|
29
|
+
if command -v sha256sum >/dev/null 2>&1; then
|
|
30
|
+
SHA256_CMD=(sha256sum)
|
|
31
|
+
elif command -v shasum >/dev/null 2>&1; then
|
|
32
|
+
SHA256_CMD=(shasum -a 256)
|
|
33
|
+
else
|
|
34
|
+
echo "arch-check: neither sha256sum nor shasum found in PATH" >&2
|
|
35
|
+
exit 2
|
|
36
|
+
fi
|
|
37
|
+
|
|
20
38
|
ROOT="${ROOT:-$(pwd)}"
|
|
21
39
|
JSON_OUT=0
|
|
22
40
|
REPORT_DIR="${ROOT}/reports/arch"
|
|
@@ -51,12 +69,12 @@ emit_result() {
|
|
|
51
69
|
local policy_hash="sha256:0000000000000000000000000000000000000000000000000000000000000000"
|
|
52
70
|
# Best-effort: input_hash is the source tree fingerprint when running against ROOT/src
|
|
53
71
|
if [[ -d "${ROOT}/src" ]]; then
|
|
54
|
-
input_hash=$(find "${ROOT}/src" -type f \( -name "*.ts" -o -name "*.tsx" -o -name "*.js" -o -name "*.py" -o -name "*.go" -o -name "*.rs" -o -name "*.java" -o -name "*.kt" -o -name "*.cs" -o -name "*.php" \) -exec
|
|
72
|
+
input_hash=$(find "${ROOT}/src" -type f \( -name "*.ts" -o -name "*.tsx" -o -name "*.js" -o -name "*.py" -o -name "*.go" -o -name "*.rs" -o -name "*.java" -o -name "*.kt" -o -name "*.cs" -o -name "*.php" \) -exec "${SHA256_CMD[@]}" {} \; 2>/dev/null | sort | "${SHA256_CMD[@]}" | awk '{print "sha256:"$1}')
|
|
55
73
|
fi
|
|
56
74
|
# Hash the architecture rule config (whichever tool's config was used)
|
|
57
75
|
for cfg in .dependency-cruiser.js .dependency-cruiser.cjs .importlinter deptrac.yaml arch-go.yml; do
|
|
58
76
|
if [[ -f "${ROOT}/${cfg}" ]]; then
|
|
59
|
-
policy_hash=$(
|
|
77
|
+
policy_hash=$("${SHA256_CMD[@]}" "${ROOT}/${cfg}" | awk '{print "sha256:"$1}')
|
|
60
78
|
break
|
|
61
79
|
fi
|
|
62
80
|
done
|
package/scripts/bias-count.sh
CHANGED
|
@@ -12,6 +12,23 @@
|
|
|
12
12
|
|
|
13
13
|
set -euo pipefail
|
|
14
14
|
|
|
15
|
+
# Bash version floor: these gates rely on bash 4+ features. Refuse early with a
|
|
16
|
+
# clear message on bash 3.x (e.g. macOS system bash) instead of failing later
|
|
17
|
+
# with a cryptic syntax error (jcgw).
|
|
18
|
+
[ "${BASH_VERSINFO:-0}" -ge 4 ] || { echo 'audit-harness requires bash >= 4' >&2; exit 3; }
|
|
19
|
+
|
|
20
|
+
# Cross-platform SHA-256: `sha256sum` ships with GNU coreutils (Linux);
|
|
21
|
+
# macOS only has `shasum -a 256`. Both produce identical `<hash> <file>`
|
|
22
|
+
# output, so downstream awk parsing is unchanged. Mirrors harness-hash.sh.
|
|
23
|
+
if command -v sha256sum >/dev/null 2>&1; then
|
|
24
|
+
SHA256_CMD=(sha256sum)
|
|
25
|
+
elif command -v shasum >/dev/null 2>&1; then
|
|
26
|
+
SHA256_CMD=(shasum -a 256)
|
|
27
|
+
else
|
|
28
|
+
echo "bias-count: neither sha256sum nor shasum found in PATH" >&2
|
|
29
|
+
exit 2
|
|
30
|
+
fi
|
|
31
|
+
|
|
15
32
|
JSON_OUT=0
|
|
16
33
|
TEST_DIR="tests"
|
|
17
34
|
|
|
@@ -36,7 +53,7 @@ if [ ! -d "$TEST_DIR" ]; then
|
|
|
36
53
|
fi
|
|
37
54
|
|
|
38
55
|
# Hash the test directory tree as the "input"
|
|
39
|
-
INPUT_HASH=$(find "$TEST_DIR" -type f \( -name "*.py" -o -name "*.ts" -o -name "*.js" -o -name "*.tsx" -o -name "*.jsx" -o -name "*.go" -o -name "*.rs" -o -name "*.java" -o -name "*.kt" -o -name "*.cs" -o -name "*.php" -o -name "*.rb" \) -exec
|
|
56
|
+
INPUT_HASH=$(find "$TEST_DIR" -type f \( -name "*.py" -o -name "*.ts" -o -name "*.js" -o -name "*.tsx" -o -name "*.jsx" -o -name "*.go" -o -name "*.rs" -o -name "*.java" -o -name "*.kt" -o -name "*.cs" -o -name "*.php" -o -name "*.rb" \) -exec "${SHA256_CMD[@]}" {} + 2>/dev/null | sort | "${SHA256_CMD[@]}" | awk '{print "sha256:"$1}')
|
|
40
57
|
|
|
41
58
|
if [[ "$JSON_OUT" -eq 1 ]]; then
|
|
42
59
|
exec 3>&1 # save stdout for the JSON object
|
|
@@ -0,0 +1,143 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# caa-check.sh — verify a namespace publishes CAA records (and, when configured,
|
|
3
|
+
# pins the EXPECTED certificate authority) before a production signed attestation
|
|
4
|
+
# is anchored against it.
|
|
5
|
+
#
|
|
6
|
+
# WHY THIS EXISTS (CISO binding, DR-010 Q5 / ISEDC v1 Q1 2026-05-10):
|
|
7
|
+
# CAA (RFC 8659) records constrain which CAs may issue certificates for a
|
|
8
|
+
# namespace. Pinning the CA on evals.intentsolutions.io closes the mis-issuance
|
|
9
|
+
# path an attacker could otherwise use to obtain a look-alike cert and present
|
|
10
|
+
# forged attestation infrastructure. This must be verified BEFORE the first
|
|
11
|
+
# production attestation. This script is that gate — read-only, fail-closed.
|
|
12
|
+
#
|
|
13
|
+
# WHY IT QUERIES AN EXPLICIT RESOLVER (the bug this version fixes):
|
|
14
|
+
# Querying the LOCAL STUB RESOLVER (plain `dig`, no `@server`) FALSE-NEGATIVES
|
|
15
|
+
# on hosts whose stub resolver lags CAA propagation or strips the record type
|
|
16
|
+
# (systemd-resolved, many CI runners, dev boxes). On such a host a correctly
|
|
17
|
+
# CAA-pinned zone looks like it has no CAA, and the gate refuses a legitimate
|
|
18
|
+
# production sign. The fix is to query a TRUSTED PUBLIC resolver. The gate
|
|
19
|
+
# stays fail-closed: PASS only on a positive matching CAA record from a trusted
|
|
20
|
+
# resolver; absence / mismatch / unreachable => non-zero.
|
|
21
|
+
#
|
|
22
|
+
# Usage:
|
|
23
|
+
# bash scripts/caa-check.sh [DOMAIN]
|
|
24
|
+
# EXPECTED_CAA_ISSUER=letsencrypt.org bash scripts/caa-check.sh evals.intentsolutions.io
|
|
25
|
+
#
|
|
26
|
+
# Resolution order for the domain:
|
|
27
|
+
# 1. $1 (positional)
|
|
28
|
+
# 2. $CAA_CHECK_DOMAIN
|
|
29
|
+
# 3. default: evals.intentsolutions.io
|
|
30
|
+
#
|
|
31
|
+
# Issuer policy:
|
|
32
|
+
# - EXPECTED_CAA_ISSUER (env) — when set, at least one CAA `issue` (or
|
|
33
|
+
# `issuewild`) record MUST name this CA, else the check FAILS (exit 1).
|
|
34
|
+
# Default: letsencrypt.org (the CA the IS public-namespace certs are issued
|
|
35
|
+
# by). Override per-deployment.
|
|
36
|
+
# - EXPECTED_CAA_ISSUER=ANY (case-insensitive) — relax to "any CAA record is
|
|
37
|
+
# acceptable"; presence of ANY CAA record passes, absence fails, and a
|
|
38
|
+
# warning is emitted that no specific CA is being pinned.
|
|
39
|
+
#
|
|
40
|
+
# Exit codes:
|
|
41
|
+
# 0 — CAA verified (present at a trusted resolver, and matches
|
|
42
|
+
# EXPECTED_CAA_ISSUER when a specific issuer is required)
|
|
43
|
+
# 1 — CAA NOT verified (no CAA records, or expected issuer not present, from
|
|
44
|
+
# any trusted resolver)
|
|
45
|
+
# 2 — UNKNOWN/UNREACHABLE (no resolver tool installed)
|
|
46
|
+
#
|
|
47
|
+
# Override knobs:
|
|
48
|
+
# CAA_CHECK_RESOLVERS — space-separated list of trusted public resolvers to
|
|
49
|
+
# query in order (default: "1.1.1.1 8.8.8.8").
|
|
50
|
+
# CAA_CHECK_DIG_CMD — command used in place of `dig` (default: dig)
|
|
51
|
+
|
|
52
|
+
set -euo pipefail
|
|
53
|
+
|
|
54
|
+
DOMAIN="${1:-${CAA_CHECK_DOMAIN:-evals.intentsolutions.io}}"
|
|
55
|
+
EXPECTED_CAA_ISSUER="${EXPECTED_CAA_ISSUER:-letsencrypt.org}"
|
|
56
|
+
DIG_CMD="${CAA_CHECK_DIG_CMD:-dig}"
|
|
57
|
+
# Trusted public resolvers, queried in order, until one returns a CAA record.
|
|
58
|
+
RESOLVERS="${CAA_CHECK_RESOLVERS:-1.1.1.1 8.8.8.8}"
|
|
59
|
+
|
|
60
|
+
log() { printf 'caa-check: %s\n' "$1" >&2; }
|
|
61
|
+
|
|
62
|
+
if [[ "$DOMAIN" == "-h" || "$DOMAIN" == "--help" ]]; then
|
|
63
|
+
sed -n '2,60p' "$0"
|
|
64
|
+
exit 0
|
|
65
|
+
fi
|
|
66
|
+
|
|
67
|
+
have() { command -v "$1" >/dev/null 2>&1; }
|
|
68
|
+
|
|
69
|
+
if ! have "$DIG_CMD"; then
|
|
70
|
+
log "UNKNOWN/UNREACHABLE — '$DIG_CMD' is not installed; cannot look up CAA for '$DOMAIN'"
|
|
71
|
+
log " failing closed (production must not sign on UNKNOWN)"
|
|
72
|
+
log " remediation: install bind9-dnsutils (provides dig) on the signing host"
|
|
73
|
+
exit 2
|
|
74
|
+
fi
|
|
75
|
+
|
|
76
|
+
# issuer_matches CAA_TEXT -> 0 if a matching issue/issuewild record is present.
|
|
77
|
+
# Match any `issue` or `issuewild` property whose value contains the expected
|
|
78
|
+
# CA. CAA values are quoted; we match case-insensitively on the issuer substring.
|
|
79
|
+
issuer_matches() {
|
|
80
|
+
printf '%s\n' "$1" \
|
|
81
|
+
| grep -iE '[[:space:]]issue(wild)?[[:space:]]' \
|
|
82
|
+
| grep -iqF "$EXPECTED_CAA_ISSUER"
|
|
83
|
+
}
|
|
84
|
+
|
|
85
|
+
# is_blank CAA_TEXT -> 0 if the text is empty after stripping whitespace.
|
|
86
|
+
is_blank() {
|
|
87
|
+
[[ -z "${1//[$' \t\r\n']/}" ]]
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
last_caa_out="" # records from the last resolver that returned ANY CAA records
|
|
91
|
+
saw_records=0 # at least one trusted resolver returned CAA records
|
|
92
|
+
|
|
93
|
+
shopt -s nocasematch
|
|
94
|
+
relax_any=0
|
|
95
|
+
[[ "$EXPECTED_CAA_ISSUER" == "ANY" ]] && relax_any=1
|
|
96
|
+
shopt -u nocasematch
|
|
97
|
+
|
|
98
|
+
for resolver in $RESOLVERS; do
|
|
99
|
+
log "looking up CAA records for '$DOMAIN' via $DIG_CMD @$resolver"
|
|
100
|
+
# `dig @resolver +short CAA` prints one line per record, e.g.:
|
|
101
|
+
# 0 issue "letsencrypt.org"
|
|
102
|
+
# 0 issuewild ";"
|
|
103
|
+
caa_out="$("$DIG_CMD" "@$resolver" +short CAA "$DOMAIN" 2>/dev/null || true)"
|
|
104
|
+
|
|
105
|
+
if is_blank "$caa_out"; then
|
|
106
|
+
log " no CAA records returned by @$resolver"
|
|
107
|
+
continue
|
|
108
|
+
fi
|
|
109
|
+
|
|
110
|
+
saw_records=1
|
|
111
|
+
last_caa_out="$caa_out"
|
|
112
|
+
|
|
113
|
+
# --- ANY-issuer relaxation: any CAA record present passes ---
|
|
114
|
+
if [[ "$relax_any" -eq 1 ]]; then
|
|
115
|
+
log "VERIFIED (presence only) — CAA records exist for '$DOMAIN' (via @$resolver)"
|
|
116
|
+
log " WARNING: EXPECTED_CAA_ISSUER=ANY — no specific CA is being pinned."
|
|
117
|
+
log " Records found:"
|
|
118
|
+
printf '%s\n' "$caa_out" | sed 's/^/ /' >&2
|
|
119
|
+
exit 0
|
|
120
|
+
fi
|
|
121
|
+
|
|
122
|
+
# --- Specific-issuer pinning ---
|
|
123
|
+
if issuer_matches "$caa_out"; then
|
|
124
|
+
log "VERIFIED — '$DOMAIN' pins issuance to '$EXPECTED_CAA_ISSUER' (via @$resolver)"
|
|
125
|
+
exit 0
|
|
126
|
+
fi
|
|
127
|
+
|
|
128
|
+
log " CAA records exist at @$resolver but none pin '$EXPECTED_CAA_ISSUER'; trying next resolver"
|
|
129
|
+
done
|
|
130
|
+
|
|
131
|
+
# No trusted resolver yielded a matching CAA record -> fail-closed (exit 1).
|
|
132
|
+
if [[ "$saw_records" -eq 1 ]]; then
|
|
133
|
+
log "NOT VERIFIED — CAA records exist for '$DOMAIN' but none pin '$EXPECTED_CAA_ISSUER'"
|
|
134
|
+
log " Records found:"
|
|
135
|
+
printf '%s\n' "$last_caa_out" | sed 's/^/ /' >&2
|
|
136
|
+
log " remediation: add a CAA record pinning the expected CA, or set"
|
|
137
|
+
log " EXPECTED_CAA_ISSUER to the CA actually published (or ANY to accept any CAA)."
|
|
138
|
+
else
|
|
139
|
+
log "NOT VERIFIED — no CAA records found for '$DOMAIN' (resolvers tried: $RESOLVERS)"
|
|
140
|
+
log " remediation: publish a CAA record pinning the issuing CA, e.g.:"
|
|
141
|
+
log " $DOMAIN. CAA 0 issue \"$EXPECTED_CAA_ISSUER\""
|
|
142
|
+
fi
|
|
143
|
+
exit 1
|