@intentsolutions/audit-harness 0.1.0 → 1.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +435 -0
- package/LICENSE +202 -21
- package/NOTICE +15 -0
- package/README.md +36 -4
- package/bin/audit-harness.js +108 -8
- package/docs/gate-promotion.md +45 -0
- package/package.json +13 -9
- package/schemas/audit-profile/layer-applicability.md +146 -0
- package/schemas/audit-profile/registry.v1.json +87 -0
- package/schemas/audit-profile/v1.schema.json +294 -0
- package/schemas/conform/v1/agent-frontmatter.schema.json +24 -0
- package/schemas/conform/v1/mcp-config.schema.json +31 -0
- package/schemas/conform/v1/plugin-manifest.schema.json +26 -0
- package/schemas/conform/v1/skillmd-frontmatter.schema.json +40 -0
- package/schemas/currency/pins.v1.json +55 -0
- package/scripts/arch-check.sh +25 -1
- package/scripts/audit.py +386 -0
- package/scripts/bias-count.sh +50 -4
- package/scripts/classify.py +403 -0
- package/scripts/conform.py +481 -0
- package/scripts/crap-score.py +65 -5
- package/scripts/currency.py +118 -0
- package/scripts/emit-evidence.sh +256 -0
- package/scripts/escape-scan.sh +58 -4
- package/scripts/fp-rate.py +145 -0
- package/scripts/gen-layer-applicability.py +157 -0
- package/scripts/gherkin-lint.sh +53 -9
- package/scripts/harness-hash.sh +78 -5
- package/scripts/scan.py +228 -0
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
# Layer Applicability — GENERATED from `registry.v1.json`
|
|
2
|
+
|
|
3
|
+
> ⚠️ **GENERATED FILE — do not edit by hand.**
|
|
4
|
+
> Source of truth: [`registry.v1.json`](registry.v1.json) (the canonical dimension→gate datum; `classify` resolves against it).
|
|
5
|
+
> Regenerate: `audit-harness gen-layer-applicability --write` (or `python3 scripts/gen-layer-applicability.py --write`).
|
|
6
|
+
> CI gate `layer-applicability-drift` fails the build if this file drifts from the registry.
|
|
7
|
+
>
|
|
8
|
+
> registry `sha256:ffbc75700fb5eb501cb47f1e4038f47ab95ae1fba534b38095e1fe7820c80ed1`
|
|
9
|
+
|
|
10
|
+
THE canonical dimension-to-gate registry: the single datum that answers 'which gates apply to repo-type X, in which dimension, at what applicability'. layer-applicability.md and each repo's TESTING.md are PROJECTIONS of this datum. `classify` resolves the UNION of a repo's detected classifications against this registry and records its sha256 as the audit-profile's registry_hash. Every gate defaults to enforcement=advisory; blocking is earned (engineer-pinned in TESTING.md, FP-rate-gated). applicability mirrors the matrix glyphs: required(✅) recommended(⭕) conditional(⚠) waived(❌).
|
|
11
|
+
|
|
12
|
+
**Legend (applicability):** ✅ required · ⭕ recommended · ⚠ conditional · ❌ waived
|
|
13
|
+
|
|
14
|
+
Every gate defaults to `enforcement: advisory`. Blocking is **earned** — engineer-pinned in the target repo's `tests/TESTING.md`, FP-rate-gated (see [`gate-promotion.md`](../../docs/gate-promotion.md)).
|
|
15
|
+
|
|
16
|
+
## Base gates (apply to every repo)
|
|
17
|
+
|
|
18
|
+
| Gate | Dimension | Applicability | Enforcement | Tool |
|
|
19
|
+
|---|---|---|---|---|
|
|
20
|
+
| `audit-harness:local:hygiene-links` | hygiene | ⭕ recommended | advisory | `lychee` |
|
|
21
|
+
| `audit-harness:local:hygiene-markdown` | hygiene | ⭕ recommended | advisory | `markdownlint` |
|
|
22
|
+
| `audit-harness:local:hygiene-readme` | hygiene | ⭕ recommended | advisory | — |
|
|
23
|
+
| `audit-harness:ci:cve-osv` | security | ⭕ recommended | advisory | `osv-scanner` |
|
|
24
|
+
| `audit-harness:ci:secrets-gitleaks` | security | ⭕ recommended | advisory | `gitleaks` |
|
|
25
|
+
|
|
26
|
+
## By classification
|
|
27
|
+
|
|
28
|
+
A repo carries the **UNION** of every classification it matches (`classify` never picks a single winner). Gates dedup by `gate_id`, keeping the highest applicability.
|
|
29
|
+
|
|
30
|
+
### `action`
|
|
31
|
+
|
|
32
|
+
| Gate | Dimension | Applicability | Enforcement | Tool |
|
|
33
|
+
|---|---|---|---|---|
|
|
34
|
+
| `audit-harness:local:conform-action` | conformance | ✅ required | advisory | `yamllint` |
|
|
35
|
+
| `audit-harness:ci:smoke` | testing-depth | ⭕ recommended | advisory | — |
|
|
36
|
+
|
|
37
|
+
### `agent`
|
|
38
|
+
|
|
39
|
+
| Gate | Dimension | Applicability | Enforcement | Tool |
|
|
40
|
+
|---|---|---|---|---|
|
|
41
|
+
| `audit-harness:local:conform-agent` | conformance | ✅ required | advisory | `validate-agent` |
|
|
42
|
+
|
|
43
|
+
### `api`
|
|
44
|
+
|
|
45
|
+
| Gate | Dimension | Applicability | Enforcement | Tool |
|
|
46
|
+
|---|---|---|---|---|
|
|
47
|
+
| `audit-harness:local:conform-openapi` | conformance | ✅ required | advisory | `spectral` |
|
|
48
|
+
| `audit-harness:ci:sast` | security | ✅ required | advisory | `semgrep` |
|
|
49
|
+
| `audit-harness:ci:contract` | testing-depth | ✅ required | advisory | — |
|
|
50
|
+
| `audit-harness:ci:crap-score` | testing-depth | ✅ required | advisory | — |
|
|
51
|
+
| `audit-harness:ci:integration` | testing-depth | ✅ required | advisory | — |
|
|
52
|
+
| `audit-harness:ci:unit` | testing-depth | ✅ required | advisory | — |
|
|
53
|
+
|
|
54
|
+
### `cli`
|
|
55
|
+
|
|
56
|
+
| Gate | Dimension | Applicability | Enforcement | Tool |
|
|
57
|
+
|---|---|---|---|---|
|
|
58
|
+
| `audit-harness:ci:crap-score` | testing-depth | ✅ required | advisory | — |
|
|
59
|
+
| `audit-harness:ci:smoke` | testing-depth | ✅ required | advisory | — |
|
|
60
|
+
| `audit-harness:ci:unit` | testing-depth | ✅ required | advisory | — |
|
|
61
|
+
|
|
62
|
+
### `embedded`
|
|
63
|
+
|
|
64
|
+
| Gate | Dimension | Applicability | Enforcement | Tool |
|
|
65
|
+
|---|---|---|---|---|
|
|
66
|
+
| `audit-harness:ci:fuzz` | testing-depth | ✅ required | advisory | — |
|
|
67
|
+
| `audit-harness:ci:sanitizers` | testing-depth | ✅ required | advisory | — |
|
|
68
|
+
| `audit-harness:ci:unit` | testing-depth | ✅ required | advisory | — |
|
|
69
|
+
|
|
70
|
+
### `frontend`
|
|
71
|
+
|
|
72
|
+
| Gate | Dimension | Applicability | Enforcement | Tool |
|
|
73
|
+
|---|---|---|---|---|
|
|
74
|
+
| `audit-harness:ci:a11y` | testing-depth | ✅ required | advisory | `axe` |
|
|
75
|
+
| `audit-harness:ci:contract` | testing-depth | ⚠ conditional | advisory | — |
|
|
76
|
+
| `audit-harness:ci:crap-score` | testing-depth | ✅ required | advisory | — |
|
|
77
|
+
| `audit-harness:ci:e2e` | testing-depth | ✅ required | advisory | — |
|
|
78
|
+
| `audit-harness:ci:unit` | testing-depth | ✅ required | advisory | — |
|
|
79
|
+
|
|
80
|
+
### `hook`
|
|
81
|
+
|
|
82
|
+
| Gate | Dimension | Applicability | Enforcement | Tool |
|
|
83
|
+
|---|---|---|---|---|
|
|
84
|
+
| `audit-harness:local:conform-hook` | conformance | ✅ required | advisory | `validate-hook` |
|
|
85
|
+
|
|
86
|
+
### `library`
|
|
87
|
+
|
|
88
|
+
| Gate | Dimension | Applicability | Enforcement | Tool |
|
|
89
|
+
|---|---|---|---|---|
|
|
90
|
+
| `audit-harness:ci:cve-osv` | security | ✅ required | advisory | `osv-scanner` |
|
|
91
|
+
| `audit-harness:ci:crap-score` | testing-depth | ✅ required | advisory | — |
|
|
92
|
+
| `audit-harness:ci:property-based` | testing-depth | ⭕ recommended | advisory | — |
|
|
93
|
+
| `audit-harness:ci:unit` | testing-depth | ✅ required | advisory | — |
|
|
94
|
+
|
|
95
|
+
### `marketplace`
|
|
96
|
+
|
|
97
|
+
| Gate | Dimension | Applicability | Enforcement | Tool |
|
|
98
|
+
|---|---|---|---|---|
|
|
99
|
+
| `audit-harness:local:conform-marketplace` | conformance | ✅ required | advisory | `validate-marketplace` |
|
|
100
|
+
|
|
101
|
+
### `mcp`
|
|
102
|
+
|
|
103
|
+
| Gate | Dimension | Applicability | Enforcement | Tool |
|
|
104
|
+
|---|---|---|---|---|
|
|
105
|
+
| `audit-harness:local:conform-mcp` | conformance | ✅ required | advisory | `validate-mcp` |
|
|
106
|
+
|
|
107
|
+
### `monorepo`
|
|
108
|
+
|
|
109
|
+
| Gate | Dimension | Applicability | Enforcement | Tool |
|
|
110
|
+
|---|---|---|---|---|
|
|
111
|
+
| `audit-harness:local:per-package-classify` | testing-depth | ✅ required | advisory | — |
|
|
112
|
+
|
|
113
|
+
### `plugin`
|
|
114
|
+
|
|
115
|
+
| Gate | Dimension | Applicability | Enforcement | Tool |
|
|
116
|
+
|---|---|---|---|---|
|
|
117
|
+
| `audit-harness:local:conform-plugin` | conformance | ✅ required | advisory | `validate-plugin` |
|
|
118
|
+
| `audit-harness:server:skill-behavioral` | skill-quality | ⭕ recommended | advisory | `j-rig` |
|
|
119
|
+
|
|
120
|
+
### `service`
|
|
121
|
+
|
|
122
|
+
| Gate | Dimension | Applicability | Enforcement | Tool |
|
|
123
|
+
|---|---|---|---|---|
|
|
124
|
+
| `audit-harness:ci:sast` | security | ✅ required | advisory | `semgrep` |
|
|
125
|
+
| `audit-harness:ci:sbom-syft` | security | ⭕ recommended | advisory | `syft` |
|
|
126
|
+
| `audit-harness:ci:contract` | testing-depth | ✅ required | advisory | — |
|
|
127
|
+
| `audit-harness:ci:crap-score` | testing-depth | ✅ required | advisory | — |
|
|
128
|
+
| `audit-harness:ci:integration` | testing-depth | ✅ required | advisory | — |
|
|
129
|
+
| `audit-harness:ci:migration` | testing-depth | ✅ required | advisory | — |
|
|
130
|
+
| `audit-harness:ci:perf` | testing-depth | ⭕ recommended | advisory | — |
|
|
131
|
+
| `audit-harness:ci:unit` | testing-depth | ✅ required | advisory | — |
|
|
132
|
+
|
|
133
|
+
### `skill`
|
|
134
|
+
|
|
135
|
+
| Gate | Dimension | Applicability | Enforcement | Tool |
|
|
136
|
+
|---|---|---|---|---|
|
|
137
|
+
| `audit-harness:local:conform-skillmd` | conformance | ✅ required | advisory | `validate-skillmd` |
|
|
138
|
+
| `audit-harness:server:skill-behavioral` | skill-quality | ⭕ recommended | advisory | `j-rig` |
|
|
139
|
+
|
|
140
|
+
## Overlays
|
|
141
|
+
|
|
142
|
+
### `regulated`
|
|
143
|
+
|
|
144
|
+
Compliance overlay (HIPAA/SOX/PCI-DSS/SOC2/GDPR/FedRAMP markers). Promotes recommended security + conformance gates to required and escalates uncovered SHOULD requirements.
|
|
145
|
+
|
|
146
|
+
Promotes to **required**: `security`, `conformance`.
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
{
|
|
2
|
+
"registry_version": "audit-profile-registry/v1",
|
|
3
|
+
"description": "THE canonical dimension-to-gate registry: the single datum that answers 'which gates apply to repo-type X, in which dimension, at what applicability'. layer-applicability.md and each repo's TESTING.md are PROJECTIONS of this datum. `classify` resolves the UNION of a repo's detected classifications against this registry and records its sha256 as the audit-profile's registry_hash. Every gate defaults to enforcement=advisory; blocking is earned (engineer-pinned in TESTING.md, FP-rate-gated). applicability mirrors the matrix glyphs: required(✅) recommended(⭕) conditional(⚠) waived(❌).",
|
|
4
|
+
"base": [
|
|
5
|
+
{ "gate_id": "audit-harness:local:hygiene-readme", "dimension": "hygiene", "applicability": "recommended", "enforcement": "advisory", "result_class_default": "PASS" },
|
|
6
|
+
{ "gate_id": "audit-harness:local:hygiene-links", "dimension": "hygiene", "applicability": "recommended", "enforcement": "advisory", "tool": "lychee", "result_class_default": "INDETERMINATE" },
|
|
7
|
+
{ "gate_id": "audit-harness:local:hygiene-markdown", "dimension": "hygiene", "applicability": "recommended", "enforcement": "advisory", "tool": "markdownlint" },
|
|
8
|
+
{ "gate_id": "audit-harness:ci:secrets-gitleaks", "dimension": "security", "applicability": "recommended", "enforcement": "advisory", "tool": "gitleaks", "result_class_default": "INDETERMINATE" },
|
|
9
|
+
{ "gate_id": "audit-harness:ci:cve-osv", "dimension": "security", "applicability": "recommended", "enforcement": "advisory", "tool": "osv-scanner", "result_class_default": "INDETERMINATE" }
|
|
10
|
+
],
|
|
11
|
+
"classifications": {
|
|
12
|
+
"service": [
|
|
13
|
+
{ "gate_id": "audit-harness:ci:unit", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
|
|
14
|
+
{ "gate_id": "audit-harness:ci:crap-score", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
|
|
15
|
+
{ "gate_id": "audit-harness:ci:integration", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
|
|
16
|
+
{ "gate_id": "audit-harness:ci:contract", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
|
|
17
|
+
{ "gate_id": "audit-harness:ci:migration", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
|
|
18
|
+
{ "gate_id": "audit-harness:ci:sast", "dimension": "security", "applicability": "required", "enforcement": "advisory", "tool": "semgrep" },
|
|
19
|
+
{ "gate_id": "audit-harness:ci:sbom-syft", "dimension": "security", "applicability": "recommended", "enforcement": "advisory", "tool": "syft" },
|
|
20
|
+
{ "gate_id": "audit-harness:ci:perf", "dimension": "testing-depth", "applicability": "recommended", "enforcement": "advisory" }
|
|
21
|
+
],
|
|
22
|
+
"api": [
|
|
23
|
+
{ "gate_id": "audit-harness:ci:unit", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
|
|
24
|
+
{ "gate_id": "audit-harness:ci:crap-score", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
|
|
25
|
+
{ "gate_id": "audit-harness:ci:integration", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
|
|
26
|
+
{ "gate_id": "audit-harness:ci:contract", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
|
|
27
|
+
{ "gate_id": "audit-harness:local:conform-openapi", "dimension": "conformance", "applicability": "required", "enforcement": "advisory", "tool": "spectral" },
|
|
28
|
+
{ "gate_id": "audit-harness:ci:sast", "dimension": "security", "applicability": "required", "enforcement": "advisory", "tool": "semgrep" }
|
|
29
|
+
],
|
|
30
|
+
"frontend": [
|
|
31
|
+
{ "gate_id": "audit-harness:ci:unit", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
|
|
32
|
+
{ "gate_id": "audit-harness:ci:crap-score", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
|
|
33
|
+
{ "gate_id": "audit-harness:ci:e2e", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
|
|
34
|
+
{ "gate_id": "audit-harness:ci:a11y", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory", "tool": "axe" },
|
|
35
|
+
{ "gate_id": "audit-harness:ci:contract", "dimension": "testing-depth", "applicability": "conditional", "enforcement": "advisory" }
|
|
36
|
+
],
|
|
37
|
+
"cli": [
|
|
38
|
+
{ "gate_id": "audit-harness:ci:unit", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
|
|
39
|
+
{ "gate_id": "audit-harness:ci:crap-score", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
|
|
40
|
+
{ "gate_id": "audit-harness:ci:smoke", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" }
|
|
41
|
+
],
|
|
42
|
+
"library": [
|
|
43
|
+
{ "gate_id": "audit-harness:ci:unit", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
|
|
44
|
+
{ "gate_id": "audit-harness:ci:crap-score", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
|
|
45
|
+
{ "gate_id": "audit-harness:ci:property-based", "dimension": "testing-depth", "applicability": "recommended", "enforcement": "advisory" },
|
|
46
|
+
{ "gate_id": "audit-harness:ci:cve-osv", "dimension": "security", "applicability": "required", "enforcement": "advisory", "tool": "osv-scanner", "result_class_default": "INDETERMINATE" }
|
|
47
|
+
],
|
|
48
|
+
"embedded": [
|
|
49
|
+
{ "gate_id": "audit-harness:ci:unit", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
|
|
50
|
+
{ "gate_id": "audit-harness:ci:fuzz", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" },
|
|
51
|
+
{ "gate_id": "audit-harness:ci:sanitizers", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" }
|
|
52
|
+
],
|
|
53
|
+
"monorepo": [
|
|
54
|
+
{ "gate_id": "audit-harness:local:per-package-classify", "dimension": "testing-depth", "applicability": "required", "enforcement": "advisory" }
|
|
55
|
+
],
|
|
56
|
+
"skill": [
|
|
57
|
+
{ "gate_id": "audit-harness:local:conform-skillmd", "dimension": "conformance", "applicability": "required", "enforcement": "advisory", "tool": "validate-skillmd" },
|
|
58
|
+
{ "gate_id": "audit-harness:server:skill-behavioral", "dimension": "skill-quality", "applicability": "recommended", "enforcement": "advisory", "tool": "j-rig" }
|
|
59
|
+
],
|
|
60
|
+
"agent": [
|
|
61
|
+
{ "gate_id": "audit-harness:local:conform-agent", "dimension": "conformance", "applicability": "required", "enforcement": "advisory", "tool": "validate-agent" }
|
|
62
|
+
],
|
|
63
|
+
"hook": [
|
|
64
|
+
{ "gate_id": "audit-harness:local:conform-hook", "dimension": "conformance", "applicability": "required", "enforcement": "advisory", "tool": "validate-hook" }
|
|
65
|
+
],
|
|
66
|
+
"mcp": [
|
|
67
|
+
{ "gate_id": "audit-harness:local:conform-mcp", "dimension": "conformance", "applicability": "required", "enforcement": "advisory", "tool": "validate-mcp" }
|
|
68
|
+
],
|
|
69
|
+
"plugin": [
|
|
70
|
+
{ "gate_id": "audit-harness:local:conform-plugin", "dimension": "conformance", "applicability": "required", "enforcement": "advisory", "tool": "validate-plugin" },
|
|
71
|
+
{ "gate_id": "audit-harness:server:skill-behavioral", "dimension": "skill-quality", "applicability": "recommended", "enforcement": "advisory", "tool": "j-rig" }
|
|
72
|
+
],
|
|
73
|
+
"marketplace": [
|
|
74
|
+
{ "gate_id": "audit-harness:local:conform-marketplace", "dimension": "conformance", "applicability": "required", "enforcement": "advisory", "tool": "validate-marketplace" }
|
|
75
|
+
],
|
|
76
|
+
"action": [
|
|
77
|
+
{ "gate_id": "audit-harness:local:conform-action", "dimension": "conformance", "applicability": "required", "enforcement": "advisory", "tool": "yamllint" },
|
|
78
|
+
{ "gate_id": "audit-harness:ci:smoke", "dimension": "testing-depth", "applicability": "recommended", "enforcement": "advisory" }
|
|
79
|
+
]
|
|
80
|
+
},
|
|
81
|
+
"overlays": {
|
|
82
|
+
"regulated": {
|
|
83
|
+
"description": "Compliance overlay (HIPAA/SOX/PCI-DSS/SOC2/GDPR/FedRAMP markers). Promotes recommended security + conformance gates to required and escalates uncovered SHOULD requirements.",
|
|
84
|
+
"promote_to_required": ["security", "conformance"]
|
|
85
|
+
}
|
|
86
|
+
}
|
|
87
|
+
}
|
|
@@ -0,0 +1,294 @@
|
|
|
1
|
+
{
|
|
2
|
+
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
3
|
+
"$id": "https://evals.intentsolutions.io/audit-profile/v1.schema.json",
|
|
4
|
+
"title": "Audit Profile (v1)",
|
|
5
|
+
"description": "Closed, versioned, hash-bearing value describing the audit profile of a repository: the UNION of detected classifications, the resolved gate set (each with applicability + enforcement + dimension), and the explicit unresolved residue. Produced by `audit-harness classify` as read-only stdout JSON; the classifier NEVER mutates the repo. A profile is a deterministic function of (repo contents at commit_sha, the canonical dimension-to-gate registry pinned by registry_hash, and any engineer .audit-harness.yml overrides). Mirrors the gate-result/v1 closed-value conventions (additionalProperties:false, version-in-$id) so the two predicates compose in one Evidence Bundle.",
|
|
6
|
+
"type": "object",
|
|
7
|
+
"required": [
|
|
8
|
+
"schema_version",
|
|
9
|
+
"subject",
|
|
10
|
+
"classifier",
|
|
11
|
+
"registry_hash",
|
|
12
|
+
"timestamp",
|
|
13
|
+
"classifications",
|
|
14
|
+
"gates",
|
|
15
|
+
"unresolved"
|
|
16
|
+
],
|
|
17
|
+
"additionalProperties": false,
|
|
18
|
+
"properties": {
|
|
19
|
+
"schema_version": {
|
|
20
|
+
"description": "Self-describing schema identity. MUST equal the version segment of $id.",
|
|
21
|
+
"const": "audit-profile/v1"
|
|
22
|
+
},
|
|
23
|
+
"subject": {
|
|
24
|
+
"description": "Identity of the repository (or monorepo package) the profile was computed for.",
|
|
25
|
+
"type": "object",
|
|
26
|
+
"required": [
|
|
27
|
+
"name",
|
|
28
|
+
"commit_sha"
|
|
29
|
+
],
|
|
30
|
+
"additionalProperties": false,
|
|
31
|
+
"properties": {
|
|
32
|
+
"name": {
|
|
33
|
+
"type": "string",
|
|
34
|
+
"description": "Repository or package name (e.g. 'audit-harness', '@j-rig/core').",
|
|
35
|
+
"minLength": 1
|
|
36
|
+
},
|
|
37
|
+
"commit_sha": {
|
|
38
|
+
"type": "string",
|
|
39
|
+
"description": "Git commit SHA the classification was computed against. Full 40-hex or short 7-hex.",
|
|
40
|
+
"pattern": "^[a-f0-9]{7,40}$"
|
|
41
|
+
},
|
|
42
|
+
"root": {
|
|
43
|
+
"type": "string",
|
|
44
|
+
"description": "Repo-relative path of the classified root. '.' for the repo root; a package path (e.g. 'packages/core') for a monorepo member classified independently.",
|
|
45
|
+
"default": "."
|
|
46
|
+
}
|
|
47
|
+
}
|
|
48
|
+
},
|
|
49
|
+
"classifier": {
|
|
50
|
+
"type": "string",
|
|
51
|
+
"description": "Tool + semver identifier that produced the profile. Same shape as gate-result/v1 'runner'.",
|
|
52
|
+
"pattern": "^[a-z0-9][a-z0-9-]*@[0-9]+\\.[0-9]+\\.[0-9]+(-[A-Za-z0-9.-]+)?(\\+[A-Za-z0-9.-]+)?$",
|
|
53
|
+
"examples": [
|
|
54
|
+
"audit-harness@1.2.0"
|
|
55
|
+
]
|
|
56
|
+
},
|
|
57
|
+
"registry_hash": {
|
|
58
|
+
"type": "string",
|
|
59
|
+
"description": "SHA-256 of the canonical dimension-to-gate registry datum the profile was resolved against, prefixed sha256:. The registry is the single canonical source of 'which gates apply to repo-type X'; layer-applicability.md and TESTING.md are projections of it. Recording its hash makes a profile reproducible against the exact registry version.",
|
|
60
|
+
"pattern": "^sha256:[a-f0-9]{64}$"
|
|
61
|
+
},
|
|
62
|
+
"timestamp": {
|
|
63
|
+
"type": "string",
|
|
64
|
+
"format": "date-time",
|
|
65
|
+
"description": "RFC 3339 UTC timestamp. Moment the classifier emitted the profile."
|
|
66
|
+
},
|
|
67
|
+
"classifications": {
|
|
68
|
+
"type": "array",
|
|
69
|
+
"description": "The UNION of all repo-type classifications detected (NOT a single winner). A repo that is a TS monorepo AND ships SKILL.md AND an MCP server carries all three entries; picking one silently drops the others' gates (a false-negative, worse than a false-positive).",
|
|
70
|
+
"minItems": 1,
|
|
71
|
+
"items": {
|
|
72
|
+
"$ref": "#/$defs/classification"
|
|
73
|
+
}
|
|
74
|
+
},
|
|
75
|
+
"dimensions": {
|
|
76
|
+
"type": "array",
|
|
77
|
+
"description": "Convenience projection: the distinct audit dimensions covered by the resolved gates. Derivable from gates[].dimension; consumers MUST treat gates[] as authoritative.",
|
|
78
|
+
"items": {
|
|
79
|
+
"$ref": "#/$defs/dimension"
|
|
80
|
+
},
|
|
81
|
+
"uniqueItems": true
|
|
82
|
+
},
|
|
83
|
+
"gates": {
|
|
84
|
+
"type": "array",
|
|
85
|
+
"description": "The resolved gate set for this repo: every gate the registry maps to the detected classifications, each tagged with applicability (from the registry) and enforcement (advisory by default; blocking only when engineer-pinned in TESTING.md and FP-rate-gated).",
|
|
86
|
+
"items": {
|
|
87
|
+
"$ref": "#/$defs/gate"
|
|
88
|
+
}
|
|
89
|
+
},
|
|
90
|
+
"unresolved": {
|
|
91
|
+
"type": "array",
|
|
92
|
+
"description": "The explicit residue the DETERMINISTIC classifier could not resolve. This is the ONLY surface a Claude inspector (/audit-tests) may refine — it never co-authors the deterministic classification value, so CI stays reproducible. An empty array means a fully deterministic profile.",
|
|
93
|
+
"items": {
|
|
94
|
+
"$ref": "#/$defs/unresolved_item"
|
|
95
|
+
}
|
|
96
|
+
},
|
|
97
|
+
"overrides": {
|
|
98
|
+
"type": "object",
|
|
99
|
+
"description": "Engineer-owned .audit-harness.yml directives that influenced this profile (classification pins, per-gate advisory/disable toggles, kill-switch). Informative provenance; the effect is already baked into classifications[] and gates[].",
|
|
100
|
+
"additionalProperties": false,
|
|
101
|
+
"properties": {
|
|
102
|
+
"source": {
|
|
103
|
+
"type": "string",
|
|
104
|
+
"description": "Path of the override file applied, repo-relative.",
|
|
105
|
+
"examples": [
|
|
106
|
+
".audit-harness.yml"
|
|
107
|
+
]
|
|
108
|
+
},
|
|
109
|
+
"override_hash": {
|
|
110
|
+
"type": "string",
|
|
111
|
+
"description": "SHA-256 of the applied override file, prefixed sha256:.",
|
|
112
|
+
"pattern": "^sha256:[a-f0-9]{64}$"
|
|
113
|
+
},
|
|
114
|
+
"kill_switch": {
|
|
115
|
+
"type": "boolean",
|
|
116
|
+
"description": "True when AUDIT_HARNESS_DISABLE=1 or the override file disabled all gates. A killed profile still emits classifications for visibility but marks every gate enforcement=disabled.",
|
|
117
|
+
"default": false
|
|
118
|
+
}
|
|
119
|
+
}
|
|
120
|
+
}
|
|
121
|
+
},
|
|
122
|
+
"$defs": {
|
|
123
|
+
"dimension": {
|
|
124
|
+
"type": "string",
|
|
125
|
+
"description": "The comprehensive audit dimensions. testing-depth = the 7-layer pyramid; conformance = artifact/schema/protocol validation; currency = upstream-drift (advisory only); security = supply-chain; hygiene = repo docs/links/changelog; skill-quality = behavioral verdicts consumed from j-rig.",
|
|
126
|
+
"enum": [
|
|
127
|
+
"testing-depth",
|
|
128
|
+
"conformance",
|
|
129
|
+
"currency",
|
|
130
|
+
"security",
|
|
131
|
+
"hygiene",
|
|
132
|
+
"skill-quality"
|
|
133
|
+
]
|
|
134
|
+
},
|
|
135
|
+
"classification": {
|
|
136
|
+
"type": "object",
|
|
137
|
+
"required": [
|
|
138
|
+
"kind",
|
|
139
|
+
"confidence"
|
|
140
|
+
],
|
|
141
|
+
"additionalProperties": false,
|
|
142
|
+
"properties": {
|
|
143
|
+
"kind": {
|
|
144
|
+
"type": "string",
|
|
145
|
+
"description": "Repo-type or artifact-kind classification. The first block mirrors layer-applicability.md repo types; the second covers Claude-ecosystem artifact kinds; 'regulated' is an overlay that escalates gate severity.",
|
|
146
|
+
"enum": [
|
|
147
|
+
"service",
|
|
148
|
+
"api",
|
|
149
|
+
"frontend",
|
|
150
|
+
"cli",
|
|
151
|
+
"library",
|
|
152
|
+
"embedded",
|
|
153
|
+
"monorepo",
|
|
154
|
+
"regulated",
|
|
155
|
+
"skill",
|
|
156
|
+
"agent",
|
|
157
|
+
"hook",
|
|
158
|
+
"mcp",
|
|
159
|
+
"plugin",
|
|
160
|
+
"marketplace",
|
|
161
|
+
"action",
|
|
162
|
+
"unknown"
|
|
163
|
+
]
|
|
164
|
+
},
|
|
165
|
+
"confidence": {
|
|
166
|
+
"type": "string",
|
|
167
|
+
"description": "How the classification was reached. 'declared' = an explicit engineer override/marker; 'detected' = an unambiguous deterministic signal; 'inferred' = a weaker heuristic; 'unresolved' = the classifier could not decide (a matching entry MUST appear in unresolved[]).",
|
|
168
|
+
"enum": [
|
|
169
|
+
"declared",
|
|
170
|
+
"detected",
|
|
171
|
+
"inferred",
|
|
172
|
+
"unresolved"
|
|
173
|
+
]
|
|
174
|
+
},
|
|
175
|
+
"signals": {
|
|
176
|
+
"type": "array",
|
|
177
|
+
"description": "Deterministic evidence for the classification (e.g. 'package.json:react', 'SKILL.md present', 'pnpm-workspace.yaml').",
|
|
178
|
+
"items": {
|
|
179
|
+
"type": "string"
|
|
180
|
+
}
|
|
181
|
+
}
|
|
182
|
+
}
|
|
183
|
+
},
|
|
184
|
+
"gate": {
|
|
185
|
+
"type": "object",
|
|
186
|
+
"required": [
|
|
187
|
+
"gate_id",
|
|
188
|
+
"dimension",
|
|
189
|
+
"applicability",
|
|
190
|
+
"enforcement"
|
|
191
|
+
],
|
|
192
|
+
"additionalProperties": false,
|
|
193
|
+
"properties": {
|
|
194
|
+
"gate_id": {
|
|
195
|
+
"type": "string",
|
|
196
|
+
"description": "Pipeline-hop-qualified gate identifier, same regex as gate-result/v1 gate_id (tool:side:gate-id). The gate-result row a run later emits for this gate MUST carry the same gate_id.",
|
|
197
|
+
"pattern": "^[a-z0-9][a-z0-9-]*:(client|server|ci|sandbox|local):[a-zA-Z0-9][a-zA-Z0-9.-]*$",
|
|
198
|
+
"examples": [
|
|
199
|
+
"audit-harness:ci:crap-score",
|
|
200
|
+
"audit-harness:local:conform-skillmd",
|
|
201
|
+
"audit-harness:ci:secrets-gitleaks"
|
|
202
|
+
]
|
|
203
|
+
},
|
|
204
|
+
"dimension": {
|
|
205
|
+
"$ref": "#/$defs/dimension"
|
|
206
|
+
},
|
|
207
|
+
"applicability": {
|
|
208
|
+
"type": "string",
|
|
209
|
+
"description": "From the registry, given the detected classifications. required = P0 (✅); recommended = P1 advisory (⭕); conditional = fires P0/P1 only if a sibling signal is present (⚠); waived = not run for this repo type (❌).",
|
|
210
|
+
"enum": [
|
|
211
|
+
"required",
|
|
212
|
+
"recommended",
|
|
213
|
+
"conditional",
|
|
214
|
+
"waived"
|
|
215
|
+
]
|
|
216
|
+
},
|
|
217
|
+
"enforcement": {
|
|
218
|
+
"type": "string",
|
|
219
|
+
"description": "What a FAIL does. New gates ship 'advisory' (exit 0, finding logged). 'blocking' is earned: an engineer pins it in the hash-pinned TESTING.md after the gate shows a measured false-positive rate below the stated bar on the IEP corpus. 'disabled' = waived or kill-switched.",
|
|
220
|
+
"enum": [
|
|
221
|
+
"blocking",
|
|
222
|
+
"advisory",
|
|
223
|
+
"disabled"
|
|
224
|
+
]
|
|
225
|
+
},
|
|
226
|
+
"result_class_default": {
|
|
227
|
+
"type": "string",
|
|
228
|
+
"description": "The result class a successful no-finding run would emit (informative). INDETERMINATE is reserved for infra failures (≠ policy failure): pure-local policy gates fail closed (FAIL); network-touching checks fail open/advisory (INDETERMINATE).",
|
|
229
|
+
"enum": [
|
|
230
|
+
"PASS",
|
|
231
|
+
"FAIL",
|
|
232
|
+
"ADVISORY",
|
|
233
|
+
"NOT_APPLICABLE",
|
|
234
|
+
"INDETERMINATE"
|
|
235
|
+
]
|
|
236
|
+
},
|
|
237
|
+
"tool": {
|
|
238
|
+
"type": "string",
|
|
239
|
+
"description": "External tool this gate shells out to, if any (e.g. 'gitleaks', 'osv-scanner', 'spectral', 'ajv', 'markdownlint', 'lychee'). Absent for native/policy gates. A missing tool at run-time yields an 'unmeasured' gate-result, not a FAIL.",
|
|
240
|
+
"examples": [
|
|
241
|
+
"gitleaks",
|
|
242
|
+
"osv-scanner",
|
|
243
|
+
"spectral"
|
|
244
|
+
]
|
|
245
|
+
}
|
|
246
|
+
},
|
|
247
|
+
"allOf": [
|
|
248
|
+
{
|
|
249
|
+
"if": {
|
|
250
|
+
"properties": {
|
|
251
|
+
"applicability": {
|
|
252
|
+
"const": "waived"
|
|
253
|
+
}
|
|
254
|
+
}
|
|
255
|
+
},
|
|
256
|
+
"then": {
|
|
257
|
+
"properties": {
|
|
258
|
+
"enforcement": {
|
|
259
|
+
"const": "disabled"
|
|
260
|
+
}
|
|
261
|
+
}
|
|
262
|
+
}
|
|
263
|
+
}
|
|
264
|
+
]
|
|
265
|
+
},
|
|
266
|
+
"unresolved_item": {
|
|
267
|
+
"type": "object",
|
|
268
|
+
"required": [
|
|
269
|
+
"kind",
|
|
270
|
+
"reason"
|
|
271
|
+
],
|
|
272
|
+
"additionalProperties": false,
|
|
273
|
+
"properties": {
|
|
274
|
+
"kind": {
|
|
275
|
+
"type": "string",
|
|
276
|
+
"description": "What could not be resolved (e.g. 'repo-type', 'monorepo-package-boundary', 'is-published-library').",
|
|
277
|
+
"minLength": 1
|
|
278
|
+
},
|
|
279
|
+
"reason": {
|
|
280
|
+
"type": "string",
|
|
281
|
+
"description": "Why the deterministic classifier abstained — the question a human (or /audit-tests) must answer.",
|
|
282
|
+
"minLength": 1
|
|
283
|
+
},
|
|
284
|
+
"candidates": {
|
|
285
|
+
"type": "array",
|
|
286
|
+
"description": "The candidate classifications the classifier was torn between, if any.",
|
|
287
|
+
"items": {
|
|
288
|
+
"type": "string"
|
|
289
|
+
}
|
|
290
|
+
}
|
|
291
|
+
}
|
|
292
|
+
}
|
|
293
|
+
}
|
|
294
|
+
}
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
{
|
|
2
|
+
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
3
|
+
"$id": "https://evals.intentsolutions.io/conform/v1/agent-frontmatter.schema.json",
|
|
4
|
+
"title": "Subagent definition frontmatter — deterministic structural floor (conform/v1)",
|
|
5
|
+
"description": "The DETERMINISTIC STRUCTURAL FLOOR for a Claude subagent definition (.md with YAML frontmatter): it parses and carries name + description. tools/model are type-checked when present. Trigger-precision + tool-allowlist-tightness judgment stays in /validate-agent. additionalProperties:true for forward-compat.",
|
|
6
|
+
"type": "object",
|
|
7
|
+
"required": ["name", "description"],
|
|
8
|
+
"additionalProperties": true,
|
|
9
|
+
"properties": {
|
|
10
|
+
"name": {
|
|
11
|
+
"type": "string",
|
|
12
|
+
"minLength": 1,
|
|
13
|
+
"pattern": "^[a-zA-Z0-9][a-zA-Z0-9_-]*$"
|
|
14
|
+
},
|
|
15
|
+
"description": { "type": "string", "minLength": 1 },
|
|
16
|
+
"tools": {
|
|
17
|
+
"anyOf": [
|
|
18
|
+
{ "type": "string" },
|
|
19
|
+
{ "type": "array", "items": { "type": "string" } }
|
|
20
|
+
]
|
|
21
|
+
},
|
|
22
|
+
"model": { "type": "string" }
|
|
23
|
+
}
|
|
24
|
+
}
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
{
|
|
2
|
+
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
3
|
+
"$id": "https://evals.intentsolutions.io/conform/v1/mcp-config.schema.json",
|
|
4
|
+
"title": ".mcp.json MCP server config — deterministic structural floor (conform/v1)",
|
|
5
|
+
"description": "The DETERMINISTIC STRUCTURAL FLOOR for a Claude .mcp.json: it parses, declares an mcpServers map, and each declared server is launchable (carries either a `command` for a stdio server or a `url` for a remote sse/http server). Server entries keep additionalProperties:true for forward-compat with newer transport fields; the floor only asserts the launch contract + types of known keys.",
|
|
6
|
+
"type": "object",
|
|
7
|
+
"required": ["mcpServers"],
|
|
8
|
+
"additionalProperties": true,
|
|
9
|
+
"properties": {
|
|
10
|
+
"mcpServers": {
|
|
11
|
+
"type": "object",
|
|
12
|
+
"additionalProperties": {
|
|
13
|
+
"type": "object",
|
|
14
|
+
"additionalProperties": true,
|
|
15
|
+
"anyOf": [
|
|
16
|
+
{ "required": ["command"] },
|
|
17
|
+
{ "required": ["url"] }
|
|
18
|
+
],
|
|
19
|
+
"properties": {
|
|
20
|
+
"command": { "type": "string", "minLength": 1 },
|
|
21
|
+
"args": { "type": "array", "items": { "type": "string" } },
|
|
22
|
+
"env": { "type": "object" },
|
|
23
|
+
"url": { "type": "string", "minLength": 1 },
|
|
24
|
+
"type": { "type": "string", "enum": ["stdio", "sse", "http"] },
|
|
25
|
+
"headers": { "type": "object" },
|
|
26
|
+
"cwd": { "type": "string" }
|
|
27
|
+
}
|
|
28
|
+
}
|
|
29
|
+
}
|
|
30
|
+
}
|
|
31
|
+
}
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
{
|
|
2
|
+
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
3
|
+
"$id": "https://evals.intentsolutions.io/conform/v1/plugin-manifest.schema.json",
|
|
4
|
+
"title": ".claude-plugin/plugin.json manifest — deterministic structural floor (conform/v1)",
|
|
5
|
+
"description": "The DETERMINISTIC STRUCTURAL FLOOR for a Claude plugin manifest: it parses and carries a name. Version/description/author are type-checked when present. The richer catalog-invariant + cross-artifact checks (INV-PLUGIN-SKILL etc.) live in /validate-plugin + the SAK kernel — conform asserts only the manifest floor. additionalProperties:true for forward-compat with the evolving plugin manifest surface.",
|
|
6
|
+
"type": "object",
|
|
7
|
+
"required": ["name"],
|
|
8
|
+
"additionalProperties": true,
|
|
9
|
+
"properties": {
|
|
10
|
+
"name": {
|
|
11
|
+
"type": "string",
|
|
12
|
+
"minLength": 1,
|
|
13
|
+
"pattern": "^[a-zA-Z0-9][a-zA-Z0-9_-]*$"
|
|
14
|
+
},
|
|
15
|
+
"version": { "type": "string" },
|
|
16
|
+
"description": { "type": "string" },
|
|
17
|
+
"author": {
|
|
18
|
+
"anyOf": [
|
|
19
|
+
{ "type": "string" },
|
|
20
|
+
{ "type": "object" }
|
|
21
|
+
]
|
|
22
|
+
},
|
|
23
|
+
"license": { "type": "string" },
|
|
24
|
+
"homepage": { "type": "string" }
|
|
25
|
+
}
|
|
26
|
+
}
|