agentic-scorecard 0.3.1 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (31) hide show
  1. package/AGENT_PROMPT.md +11 -11
  2. package/README.md +35 -29
  3. package/benchmark/v0.4/adapter-contract.md +129 -0
  4. package/benchmark/v0.4/adapters/known-agent-harnesses.yaml +266 -0
  5. package/benchmark/v0.4/adapters/known-ci-providers.yaml +57 -0
  6. package/benchmark/v0.4/adapters/known-command-wrappers.yaml +37 -0
  7. package/benchmark/v0.4/adapters/known-environment-tooling.yaml +17 -0
  8. package/benchmark/v0.4/adapters/known-github-conventions.yaml +27 -0
  9. package/benchmark/v0.4/adapters/known-python-tooling.yaml +15 -0
  10. package/benchmark/v0.4/adapters/known-security-tools.yaml +45 -0
  11. package/benchmark/v0.4/adapters/known-verification-tools.yaml +65 -0
  12. package/benchmark/v0.4/agent-evidence-schema.json +74 -0
  13. package/benchmark/v0.4/attestation-schema.json +35 -0
  14. package/benchmark/v0.4/benchmark.yaml +127 -0
  15. package/benchmark/v0.4/calibration-matrix.md +60 -0
  16. package/benchmark/v0.4/controls/context.yaml +75 -0
  17. package/benchmark/v0.4/controls/environment.yaml +93 -0
  18. package/benchmark/v0.4/controls/governance.yaml +132 -0
  19. package/benchmark/v0.4/controls/learning.yaml +78 -0
  20. package/benchmark/v0.4/controls/observability.yaml +91 -0
  21. package/benchmark/v0.4/controls/resilience.yaml +170 -0
  22. package/benchmark/v0.4/controls/security.yaml +143 -0
  23. package/benchmark/v0.4/controls/specification.yaml +65 -0
  24. package/benchmark/v0.4/controls/testing.yaml +82 -0
  25. package/benchmark/v0.4/controls/tooling.yaml +69 -0
  26. package/benchmark/v0.4/report-schema.json +89 -0
  27. package/benchmark/v0.4/scoring-policy.md +175 -0
  28. package/dist/cli.js +2287 -162
  29. package/package.json +3 -2
  30. package/templates/agent-evidence.yaml +2 -2
  31. package/templates/attestations.yaml +3 -1
package/AGENT_PROMPT.md CHANGED
@@ -1,4 +1,4 @@
1
- # Ask an AI coding agent to run ADRB v0.3
1
+ # Ask an AI coding agent to run ADRB v0.4
2
2
 
3
3
  These prompts are vendor-neutral. The agent needs terminal access to the repository and permission
4
4
  to download the published npm package. The default benchmark runs locally, reads only tracked paths,
@@ -9,7 +9,7 @@ invokes no model, uploads no source, and makes no network calls after installati
9
9
  Copy and paste:
10
10
 
11
11
  ```text
12
- Perform the complete ADRB v0.3 assessment of this repository for AI-agent pull-request work.
12
+ Perform the complete ADRB v0.4 assessment of this repository for AI-agent pull-request work.
13
13
 
14
14
  Do not change product source, tests, configuration, policies, Git history, or the user's current
15
15
  checkout. First inspect the repository's commit, tracked-file status, upstream tracking branch, and
@@ -22,14 +22,14 @@ I explicitly asked for upstream or another commit. State the exact commit you se
22
22
 
23
23
  Before running the scorecard, choose and state an absolute durable artifact directory outside any
24
24
  temporary worktree. Do not delete that directory during worktree cleanup. In the commands below,
25
- replace `<artifact-dir>` with that path. The npm package version is 0.3.1, while report filenames use
26
- the immutable ADRB benchmark version 0.3.0.
25
+ replace `<artifact-dir>` with that path. The npm package and immutable ADRB benchmark are both
26
+ version 0.4.0, so report filenames also use 0.4.0.
27
27
 
28
28
  From that clean target, run the repository-only baseline:
29
- npx --yes agentic-scorecard@0.3.1 assess . --profile pr-creation --scope tracked --format markdown --output <artifact-dir>/agentic-readiness-v0.3.0-baseline.md
29
+ npx --yes agentic-scorecard@0.4.0 assess . --profile pr-creation --scope tracked --format markdown --output <artifact-dir>/agentic-readiness-v0.4.0-baseline.md
30
30
 
31
31
  Then prepare unresolved evidence:
32
- npx --yes agentic-scorecard@0.3.1 init-evidence . --output <artifact-dir>/agent-evidence.yaml --request-output <artifact-dir>/evidence-request.md
32
+ npx --yes agentic-scorecard@0.4.0 init-evidence . --output <artifact-dir>/agent-evidence.yaml --request-output <artifact-dir>/evidence-request.md
33
33
 
34
34
  Read the baseline, `<artifact-dir>/evidence-request.md`, and the empty target-bound
35
35
  `<artifact-dir>/agent-evidence.yaml`. Investigate nuanced repository-scoped requests only from
@@ -48,7 +48,7 @@ Do not place credentials, prompts, source excerpts, raw logs, personal data, cus
48
48
  URLs that reveal secrets, or sensitive dashboard contents in generated files.
49
49
 
50
50
  When authorized evidence collection is complete, run:
51
- npx --yes agentic-scorecard@0.3.1 assess . --profile pr-creation --scope tracked --agent-evidence <artifact-dir>/agent-evidence.yaml --format markdown --output <artifact-dir>/agentic-readiness-v0.3.0-assisted.md
51
+ npx --yes agentic-scorecard@0.4.0 assess . --profile pr-creation --scope tracked --agent-evidence <artifact-dir>/agent-evidence.yaml --format markdown --output <artifact-dir>/agentic-readiness-v0.4.0-assisted.md
52
52
 
53
53
  Tell me the repository-detected progress and normative score, the highest passed profile, whether
54
54
  pr-creation passes, every target blocker, and the five highest-value improvements. Compare baseline
@@ -69,9 +69,9 @@ Use this when you want an offline first look and accept that platform, organizat
69
69
  controls will remain unresolved:
70
70
 
71
71
  ```text
72
- Run a fast repository-only ADRB v0.3 baseline for AI-agent pull-request work. Verify and state the
72
+ Run a fast repository-only ADRB v0.4 baseline for AI-agent pull-request work. Verify and state the
73
73
  selected commit and whether tracked files match HEAD, then run:
74
- npx --yes agentic-scorecard@0.3.1 assess . --profile pr-creation --scope tracked --format markdown --output .agentic/reports/agentic-readiness-v0.3.0-baseline.md
74
+ npx --yes agentic-scorecard@0.4.0 assess . --profile pr-creation --scope tracked --format markdown --output .agentic/reports/agentic-readiness-v0.4.0-baseline.md
75
75
 
76
76
  Do not change repository files or invent evidence. Summarize repository-detected progress before
77
77
  the normative score, target-profile blockers, and the five highest-value repository improvements.
@@ -95,7 +95,7 @@ No benchmark profile grants production deployment authority.
95
95
  ## Guided remediation
96
96
 
97
97
  ```text
98
- Read the latest ADRB v0.3 report and create a remediation proposal for the failed target-profile
98
+ Read the latest ADRB v0.4 report and create a remediation proposal for the failed target-profile
99
99
  controls. Do not implement changes yet. Group recommendations into repository changes, platform or
100
100
  organizational controls, and outcome measurement. For each include the control ID, risk reduced,
101
101
  systems affected, accountable role, verification method, dependencies, and effort estimate. Never
@@ -106,7 +106,7 @@ authorize.
106
106
  ## Reassessment
107
107
 
108
108
  ```text
109
- Re-run the pinned ADRB v0.3.0 benchmark with agentic-scorecard@0.3.1 for the same profile and tracked scope. Compare reports
109
+ Re-run the pinned ADRB v0.4.0 benchmark with agentic-scorecard@0.4.0 for the same profile and tracked scope. Compare reports
110
110
  control by control. Explain changes using evidence and identify changed commit state, expired
111
111
  claims, scope differences, regressions, or newly established controls. Compare normative and
112
112
  repository-detected progress separately. Write a new dated report; do not overwrite the previous
package/README.md CHANGED
@@ -16,8 +16,8 @@ The benchmark measures the harness, not the model brand. It assesses ten dimensi
16
16
  maturity levels and applies non-compensating floors to five autonomy profiles. A high total score
17
17
  cannot hide a critical security, testing, governance, or recovery gap.
18
18
 
19
- This repository is a **v0.3 reference implementation** intended for public review and piloting. It
20
- is not a certification standard. The immutable v0.1 and v0.2 benchmarks remain available for
19
+ This repository is a **v0.4 reference implementation** intended for public review and piloting. It
20
+ is not a certification standard. The immutable v0.1, v0.2, and v0.3 benchmarks remain available for
21
21
  historical reproduction; scores from different benchmark versions are not directly comparable.
22
22
 
23
23
  ## Ask your coding agent
@@ -25,8 +25,8 @@ historical reproduction; scores from different benchmark versions are not direct
25
25
  You do not need to install or learn the CLI yourself. Paste this instruction into Codex, Claude Code,
26
26
  GitHub Copilot, Cursor, or another coding agent that has terminal access to your repository:
27
27
 
28
- > Perform the complete ADRB v0.3 assessment of this repository for AI-agent pull-request work using
29
- > `agentic-scorecard@0.3.1`. First select and state the commit without modifying my checkout; preserve
28
+ > Perform the complete ADRB v0.4 assessment of this repository for AI-agent pull-request work using
29
+ > `agentic-scorecard@0.4.0`. First select and state the commit without modifying my checkout; preserve
30
30
  > HEAD when the checkout is dirty, and use upstream only if I requested the latest upstream state.
31
31
  > Assess it from a clean worktree and keep all generated artifacts in a durable directory outside any
32
32
  > temporary worktree. Run the tracked repository baseline, then run `init-evidence`, read its evidence
@@ -41,8 +41,8 @@ The complete copy-and-paste prompt, including privacy limits and exact commands,
41
41
  fast filesystem scan remains available, but it is explicitly a **repository-only baseline** and
42
42
  cannot establish platform, organization, or outcome controls by itself.
43
43
 
44
- The reference CLI/package is version 0.3.1. Generated report names use benchmark version 0.3.0
45
- because the patch release changed workflow guidance, not the immutable scoring benchmark.
44
+ The reference CLI/package and immutable benchmark are both version 0.4.0. Generated report and
45
+ evidence filenames use the benchmark version.
46
46
 
47
47
  ## Feedback and community
48
48
 
@@ -77,7 +77,7 @@ A supported Node.js LTS release (20.19+, 22.13+, or 24+) is required. Assessment
77
77
  read-only, and offline by default.
78
78
 
79
79
  ```bash
80
- npx agentic-scorecard@0.3.1 assess /path/to/repository \
80
+ npx agentic-scorecard@0.4.0 assess /path/to/repository \
81
81
  --profile pr-creation \
82
82
  --scope tracked \
83
83
  --format markdown \
@@ -87,7 +87,7 @@ npx agentic-scorecard@0.3.1 assess /path/to/repository \
87
87
  To record controls that repository inspection cannot prove:
88
88
 
89
89
  ```bash
90
- npx agentic-scorecard@0.3.1 init /path/to/repository
90
+ npx agentic-scorecard@0.4.0 init /path/to/repository
91
91
  ```
92
92
 
93
93
  Complete `.agentic/attestations.yaml` with owners and durable evidence links, then assess again.
@@ -98,7 +98,7 @@ external systems, first generate a target-bound template. The repository must be
98
98
  with at least one commit so the bundle can bind to the exact assessed state:
99
99
 
100
100
  ```bash
101
- npx agentic-scorecard@0.3.1 init-evidence /path/to/repository
101
+ npx agentic-scorecard@0.4.0 init-evidence /path/to/repository
102
102
  ```
103
103
 
104
104
  Ask the agent to review `.agentic/evidence-request.md`, obtain approval before using
@@ -106,17 +106,19 @@ least-privileged read-only connectors, add attempted claims to `.agentic/agent-e
106
106
  rerun with `--agent-evidence`. The default bundle path is loaded automatically. See
107
107
  [AGENT_PROMPT.md](AGENT_PROMPT.md) for the complete copy-and-paste workflow.
108
108
 
109
- ### Migrating from v0.1 or v0.2
109
+ ### Migrating from v0.1, v0.2, or v0.3
110
110
 
111
111
  Run a new tracked-scope baseline and retain the old report as historical evidence. Do not present the
112
- score change as improvement or regression because v0.3 changes evidence and control semantics.
113
- Re-review prior attestations and agent evidence before recreating them for v0.3. All claims must
114
- expire, and repository-scoped semantic claims require tracked files to match the commit-bound target
115
- plus `repo:<path>[#Lx-Ly]` references whose optional line ranges exist in the cited tracked file.
116
- Untracked generated reports do not block this workflow.
112
+ score change as improvement or regression because v0.4 changes evidence and control semantics.
113
+ Re-review prior attestations and agent evidence before recreating them for v0.4. All claims must
114
+ expire. Human-attestation files created by `init` are bound to the assessed repository; they are not
115
+ commit-bound because accountable platform, organization, and outcome facts may span commits.
116
+ Repository-scoped semantic agent claims require tracked files to match the commit-bound target plus
117
+ `repo:<path>[#Lx-Ly]` references whose optional line ranges exist in the cited tracked file. Untracked
118
+ generated reports do not block this workflow.
117
119
 
118
120
  The default `.agentic/attestations.yaml` and `.agentic/agent-evidence.yaml` paths are migration-safe:
119
- if either contains an older benchmark version, v0.3 ignores that auto-loaded file and places a
121
+ if either contains an older benchmark version, v0.4 ignores that auto-loaded file and places a
120
122
  prominent warning in the report. Regenerate it with `init --force` or `init-evidence --force` before
121
123
  relying on those claims. An explicitly supplied `--attestations` or `--agent-evidence` file still
122
124
  fails closed on a version mismatch.
@@ -126,7 +128,7 @@ For development from this checkout:
126
128
  ```bash
127
129
  npm ci
128
130
  npm run check
129
- npm run dev -- assess tests/fixtures/mature --profile pr-creation --attestations tests/fixtures/mature/.agentic/attestations-v0.3.yaml
131
+ npx vitest run tests/v04-calibration.test.ts -t "retains the mature level-three conformance result"
130
132
  ```
131
133
 
132
134
  ## What it assesses
@@ -172,8 +174,8 @@ No profile grants production deployment authority. Organizations should evaluate
172
174
  through a separate, system-specific safety case.
173
175
 
174
176
  The exact floors live in
175
- [`benchmark/v0.3/benchmark.yaml`](benchmark/v0.3/benchmark.yaml) and their rationale in
176
- [`benchmark/v0.3/scoring-policy.md`](benchmark/v0.3/scoring-policy.md).
177
+ [`benchmark/v0.4/benchmark.yaml`](benchmark/v0.4/benchmark.yaml) and their rationale in
178
+ [`benchmark/v0.4/scoring-policy.md`](benchmark/v0.4/scoring-policy.md).
177
179
 
178
180
  ## Evidence scopes and trust labels
179
181
 
@@ -189,11 +191,11 @@ Controls also identify whether their evidence belongs in the repository, hosting
189
191
  organization, or outcome systems. This prevents expected external unknowns from masquerading as
190
192
  missing files. Declarative detector adapters recognize common harness layouts without changing
191
193
  portable control outcomes; future independently conformant external adapters require a separate
192
- protocol. v0.3 does not issue certification or independent-verification claims.
194
+ protocol. v0.4 does not issue certification or independent-verification claims.
193
195
 
194
196
  ### Interpreting the two score views
195
197
 
196
- The normative score remains `N/40` and drives readiness profiles. v0.3 also reports
198
+ The normative score remains `N/40` and drives readiness profiles. v0.4 also reports
197
199
  `repository-detected progress: A/C`, where `C` is the maximum consecutive maturity the offline
198
200
  collector can establish without platform, organization, outcome, agent-collected, or human-attested
199
201
  evidence. This second view explains how complete the visible repository harness is; it does not
@@ -225,7 +227,7 @@ The default collector:
225
227
  - reads only Git-tracked paths and records commit and dirty-worktree metadata;
226
228
  - ignores `.git`, dependencies, build output, coverage, generated reports, attestations, and imported
227
229
  evidence bundles;
228
- - caps content scanning at 512 KB per file, 5 MB total, and 250 candidates per check, with v0.3
230
+ - caps content scanning at 512 KB per file, 5 MB total, and 250 candidates per check, with v0.4
229
231
  per-pattern balancing where configured;
230
232
  - reports file paths and match counts, never matching source snippets;
231
233
  - writes nothing unless `--output`, `init`, or `init-evidence` is explicitly requested.
@@ -238,7 +240,8 @@ access-controlled evidence instead. Report suspected vulnerabilities through [SE
238
240
  ```text
239
241
  benchmark/v0.1/ immutable historical v0.1 definition
240
242
  benchmark/v0.2/ immutable historical v0.2 definition
241
- benchmark/v0.3/ current normative benchmark, schemas, and controls
243
+ benchmark/v0.3/ immutable historical v0.3 definition
244
+ benchmark/v0.4/ current normative benchmark, schemas, and controls
242
245
  adapters/ declarative path/term aliases for recognized agent harness layouts
243
246
  benchmark/mappings/ informative mappings to external frameworks
244
247
  src/ reference CLI and local evidence collectors
@@ -268,12 +271,15 @@ system, and keep exceptions narrow, owned, expiring, and visible.
268
271
 
269
272
  ## Status and roadmap
270
273
 
271
- v0.3 adds portable harness discovery with declarative detector adapters, proximity-bounded content evidence, prioritized candidate
272
- selection for large repositories, source-backed semantic repository claims, and an explanatory
273
- repository-only score and ceiling. Candidate next steps
274
- include conformant signed adapters, SARIF/HTML reports, organization-level aggregation,
275
- statistically designed benchmark tasks, and an independent-review protocol. These require public
276
- design review before becoming normative.
274
+ v0.4 adds semantic component evidence for containment guidance, partial-check reporting, structural
275
+ ownership maps, enabled integration-triggered CI command detection for guidance integrity,
276
+ verification, and secret scanning, plus a separate Level 3 control for untrusted agent input. The
277
+ public [v0.4 calibration matrix](benchmark/v0.4/calibration-matrix.md) records expected positive and
278
+ adversarial outcomes. It retains portable harness adapters, source-backed
279
+ semantic claims, and the explanatory repository score and ceiling. Candidate next steps include
280
+ conformant signed adapters, broader structural CI provider coverage, SARIF/HTML reports,
281
+ organization-level aggregation, statistically designed benchmark tasks, and an independent-review
282
+ protocol. These require public design review before becoming normative.
277
283
 
278
284
  Apache-2.0 licensed. The benchmark is a community engineering tool, not legal, compliance, or
279
285
  security advice.
@@ -0,0 +1,129 @@
1
+ # v0.4 adapter contracts
2
+
3
+ ## Repository detector adapters
4
+
5
+ Portable controls define outcomes and generic repository conventions. Files in `adapters/*.yaml`
6
+ may extend one control evidence check with recognized harness-specific `patterns`, `files`, `terms`,
7
+ `required_any_terms`, `ci_providers`, or `ci_tools`. An extension names the control and zero-based
8
+ evidence index and adds exactly one of those fields. Repository-host aliases, CI-provider entries,
9
+ and CI-tool entries bind
10
+ provider-specific configuration paths to the applicable structural or integration-trigger semantics.
11
+ An adapter may also declare catalog-wide `ci_invocation_grammar`; this supplies recognized wrapper
12
+ patterns, command prefixes, and value-taking options to every `ci_command` check without putting
13
+ those identities in scanner core.
14
+ An adapter cannot remove or weaken a threshold, change a control level, alter a readiness floor, add
15
+ an attestation path, or execute code.
16
+
17
+ Portable v0.4 collectors may use semantic `content_groups`, structural `ownership_map`, and
18
+ command-bearing `ci_command` evidence. Adapters may add harness paths to those collectors plus
19
+ provider discovery rules and recognized command definitions to `ci_command`. Ownership evidence
20
+ uses structural mappings outside former, inactive, past, or retired sections; descendant headings
21
+ remain inactive until a same-or-higher-level section resumes current ownership. A command class uses
22
+ full owner/repository action identities, explicit executable signatures with independently required
23
+ argument groups and prohibited non-blocking modes, or semantically specific standalone executables.
24
+ Signatures may cap accepted arguments when any explicit target would leave the assessed checkout.
25
+ Executables and arguments from different signatures never combine. GitHub action invocations also
26
+ require an explicit non-empty `@ref`. Names alone never turn comments, version/help commands,
27
+ display commands, disabled or non-integration steps, manual-only workflows, ineffective shell branches, or documentation prose
28
+ into enforced evidence. Push-only, post-close, or path-gated execution does not establish the
29
+ repository-wide before-integration outcome; the external platform alternative remains available
30
+ when enforcement lives outside pull-request CI. Unsupported provider conditions and any configured
31
+ non-literal blocking override fail closed. Executable GitHub step jobs require a runner; job-level reusable workflow references
32
+ are not step actions, and invalid steps that combine `uses` with `run` are rejected. Provider path
33
+ filters and GitLab rules with unparsed gating fields fail closed. GitLab jobs with unresolved
34
+ inheritance, `except` conditions, non-blocking defaults, disabled source checkout, or failure-only
35
+ job disposition also fail closed. Repository-bound commands qualify only after the provider has
36
+ made the assessed repository available: GitHub requires a preceding `actions/checkout` step,
37
+ GitLab must retain its clone/fetch checkout, and Azure must retain its default checkout or execute
38
+ `checkout: self` before the command. GitHub checkout inputs that select another repository, ref,
39
+ sparse subset, or object filter cannot establish this state, and a later checkout that replaces the
40
+ workspace clears earlier target proof for the affected event. A secondary checkout with an explicit
41
+ non-root `path` preserves an already established primary checkout but cannot establish it by itself.
42
+ A replacing checkout with an unsupported condition clears prior proof because its effect cannot be
43
+ excluded. Workflow-level `defaults.run` settings are inherited by jobs, whose defaults override
44
+ them field by field. GitHub run working directories and explicit directory-changing shell commands
45
+ must remain at the assessed checkout root. Multi-statement shell forms qualify only when each
46
+ contributing command's failure determines the step result: supported GitHub fail-fast shells and
47
+ explicit Azure fail-fast blocks may contribute earlier statements, while other multiline forms
48
+ contribute only their final effective foreground statement. Agent-guidance integrity requires a recognized validation command bound to a tracked
49
+ source file with adapter-declared executable patterns that bind guidance inspection to a blocking
50
+ failure path; comments, inert string literals, and non-empty no-op scripts are insufficient.
51
+ Pattern groups are evaluated only over top-level code and function bodies reachable from top-level
52
+ execution. Constant-false branches and unreachable validation functions fail closed; separate
53
+ unreachable functions cannot combine their inert patterns into evidence.
54
+ Scanner modes that require a source target must bind that target to the assessed checkout rather
55
+ than an arbitrary path; explicit source overrides fail closed unless the adapter can prove that
56
+ binding. Repository-relative executables do not inherit a trusted tool identity from their basename.
57
+ Verification requires both a test command class and a static-analysis command class. `ADRB-TST-003`
58
+ may aggregate them across fail-fast multiline steps and auto-loaded CI entry points;
59
+ `ADRB-ENV-003` additionally requires its locked-install, test, and static-analysis classes in one CI
60
+ execution group. Adapters may declare tool-specific listing/discovery arguments that cannot establish
61
+ execution. Scanner tools may require the final effective command to carry the step exit status
62
+ without imposing that restriction on ordinary fail-fast verification steps. Repository-specific command
63
+ surfaces that are not recognized may use eligible source-backed agent evidence instead of keyword
64
+ inference. Package-manager task invocations are resolved through the tracked root `package.json` and
65
+ must bind to a recognized executable command even through supported wrappers and global options;
66
+ context-changing package/workspace flags fail closed unless their selected manifest is resolved,
67
+ while recognized nested package-exec targets are evaluated at their executable position. Package
68
+ manager built-ins are not treated as same-named scripts, and path-qualified local executables never
69
+ inherit package-manager identity: npm resolves arbitrary task names only
70
+ through explicit `run` or `run-script`, while its documented lifecycle aliases remain eligible.
71
+ Command-wrapper identities, command prefixes, and option arity are versioned adapter data. The
72
+ scanner core applies that grammar conservatively and retains generic package-task resolution
73
+ semantics; recognized wrapper, tool, and action identities and command signatures do not live in
74
+ the portable control catalog or core wrapper parser.
75
+ Wrapper and package-manager options with separate values are consumed before executable or task
76
+ identity is evaluated; option values never inherit tool identity.
77
+ Arguments forwarded to a package task fail closed unless their effect is structurally resolved.
78
+ Package task lookup preserves the manifest's case-sensitive script identity. Backslash/backtick
79
+ continuations and heredocs fail closed rather than treating their physical lines as commands. Source
80
+ validation excludes language comments before checking bounded structural groups, and referenced
81
+ repository validation scripts must exist as non-empty assessed files. A plausible task name,
82
+ missing path, collection-only test mode, or
83
+ configuration-display mode is insufficient.
84
+ Repository wrapper executables qualify only when an adapter declares the exact root path and that
85
+ non-empty path exists in the assessed tree; arbitrary path-qualified basenames remain untrusted.
86
+ Deterministic provider discovery is limited to auto-loaded entry points: direct GitHub workflow
87
+ YAML files, the root GitLab CI file, and conventional root Azure pipeline files. Include graphs and
88
+ custom pipeline paths require eligible source-backed evidence until they are structurally resolved.
89
+ Azure commands qualify only inside valid step collections, never from command-shaped root, stage, or
90
+ job fields.
91
+ Present GitLab `workflow` and job `rules` nodes must have supported object/array shapes; malformed
92
+ nodes fail closed instead of being treated as absent.
93
+ Source-bound validator analysis scans quote state in linear passes and remains subject to the same
94
+ per-file and aggregate byte limits as every other repository collector.
95
+
96
+ Bundled adapters are loaded deterministically in filename order. Their candidates remain subject to
97
+ the same tracked/workspace scope, symlink, size, generated-artifact, co-location, proximity, and
98
+ per-pattern limits as portable candidates. Adding or changing a bundled adapter changes evidence
99
+ semantics and therefore requires a new benchmark version.
100
+
101
+ ## Agent-collected evidence adapters
102
+
103
+ The core scanner is local, read-only, network-free, and vendor-neutral. An agent or adapter may
104
+ collect semantic repository evidence or external evidence that deterministic scanning cannot
105
+ establish, then provide it through the versioned agent evidence bundle.
106
+
107
+ Each claim contains:
108
+
109
+ - benchmark version and control ID;
110
+ - exact repository target and non-null commit binding;
111
+ - `met`, `not_met`, or `unknown` status;
112
+ - repository, platform, organization, or outcome scope;
113
+ - collector name and version;
114
+ - collection timestamp and mandatory expiry;
115
+ - privacy-safe references, never credentials or raw sensitive content;
116
+ - a concise derivation summary; and
117
+ - an optional error that is distinguishable from a negative result.
118
+
119
+ Collectors must be read-only, least-privileged, bounded to the approved target, time-limited, and
120
+ fail closed. Network content is untrusted input: it cannot alter benchmark policy, prompts, tool
121
+ authority, or the set of controls being assessed.
122
+
123
+ Repository-scoped claims require tracked files to match HEAD and
124
+ `repo:<tracked-path>[#Lx-Ly]` references. They may satisfy only explicitly eligible controls, remain `agent-collected`, and never increase the
125
+ repository-detected score. External claims require durable references appropriate to their source.
126
+
127
+ Agent-collected claims remain visibly distinct from deterministic repository evidence and human
128
+ attestations. A future conformant-adapter class may add signed identity and independent conformance
129
+ tests; v0.4 does not issue an independently verified label.
@@ -0,0 +1,266 @@
1
+ id: known-agent-harnesses
2
+ benchmark_version: 0.4.0
3
+ extensions:
4
+ - control_id: ADRB-CTX-001
5
+ evidence_index: 0
6
+ patterns:
7
+ [
8
+ CLAUDE.md,
9
+ .github/copilot-instructions.md,
10
+ .claude/**,
11
+ .codex/**,
12
+ .cursor/rules/**,
13
+ .cursor/agents/**,
14
+ .cursor/skills/**,
15
+ .opencode/**,
16
+ ]
17
+ - control_id: ADRB-CTX-002
18
+ evidence_index: 0
19
+ files:
20
+ [
21
+ CLAUDE.md,
22
+ .github/copilot-instructions.md,
23
+ .claude/**/*.md,
24
+ .codex/**/*.md,
25
+ .cursor/rules/**/*.md,
26
+ .cursor/agents/**/*.md,
27
+ .opencode/**/*.md,
28
+ ]
29
+ - control_id: ADRB-CTX-003
30
+ evidence_index: 0
31
+ ci_tools:
32
+ - id: agent-guidance-validation
33
+ commands:
34
+ - executables: [node, tsx]
35
+ argument_groups:
36
+ - [
37
+ scripts/check-agent-docs.js,
38
+ scripts/check-agent-docs.ts,
39
+ scripts/validate-agent-docs.js,
40
+ scripts/validate-agent-docs.ts,
41
+ ]
42
+ source_pattern_groups:
43
+ - [
44
+ '\b(?:readfilesync|readfile)\s*\([^\n]*(?:agents\.md|agent[-_ ](?:guidance|instructions))',
45
+ ]
46
+ - [
47
+ '\bthrow\s+new\s+error\s*\(',
48
+ '\bprocess\.exit\s*\(\s*[1-9]',
49
+ '\bprocess\.exitcode\s*=\s*[1-9]',
50
+ ]
51
+ - executables: [bash, sh]
52
+ argument_groups:
53
+ - [scripts/check-agent-docs.sh, scripts/validate-agent-docs.sh]
54
+ source_pattern_groups:
55
+ - ['\b(?:grep|test)\b[^\n]*(?:agents\.md|agent[-_ ](?:guidance|instructions))']
56
+ - ['\b(?:exit|return)\s+[1-9]']
57
+ - executables: [python, python3]
58
+ argument_groups:
59
+ - [scripts/check_agent_docs.py, scripts/validate_agent_docs.py]
60
+ source_pattern_groups:
61
+ - [
62
+ '\bopen\s*\([^\n]*(?:agents\.md|agent[-_ ](?:guidance|instructions))',
63
+ '\bpath\s*\([^\n]*(?:agents\.md|agent[-_ ](?:guidance|instructions))[^\n]*\)\s*\.\s*read_text\s*\(',
64
+ '\bread_text\s*\([^\n]*(?:agents\.md|agent[-_ ](?:guidance|instructions))',
65
+ ]
66
+ - ['\braise\b', '\bsys\.exit\s*\(\s*[1-9]']
67
+ - control_id: ADRB-SPC-003
68
+ evidence_index: 0
69
+ ci_tools:
70
+ - id: specification-traceability
71
+ commands:
72
+ - executables: [node, tsx]
73
+ argument_groups:
74
+ - [
75
+ scripts/check-specs.js,
76
+ scripts/check-specs.ts,
77
+ scripts/validate-specs.js,
78
+ scripts/validate-specs.ts,
79
+ scripts/check-traceability.js,
80
+ scripts/check-traceability.ts,
81
+ ]
82
+ source_pattern_groups:
83
+ - ['\b(?:readfilesync|readfile)\s*\([^\n]*(?:specs?/|requirements?/|\.ai/specs?)']
84
+ - ['\b(?:task|work\s*item|issue)[^\n]*\.(?:exec|match)\s*\(']
85
+ - ['\b(?:implementation|change|source|code)[^\n]*\.(?:exec|match)\s*\(']
86
+ - ['\b(?:verification|tests?)[^\n]*\.(?:exec|match)\s*\(']
87
+ - ['\b(?:readfilesync|readfile)\s*\([^\n]*(?:implementation|change|source|code)\w*']
88
+ - ['\b(?:readfilesync|readfile)\s*\([^\n]*(?:verification|tests?)\w*']
89
+ - [
90
+ '\b(?:implementation|change|source|code)\w*\.includes\s*\([^\n]*(?:work|task|issue)',
91
+ ]
92
+ - ['\b(?:verification|tests?)\w*\.includes\s*\([^\n]*(?:work|task|issue)']
93
+ - [
94
+ '\bthrow\s+new\s+error\s*\(',
95
+ '\bprocess\.exit\s*\(\s*[1-9]',
96
+ '\bprocess\.exitcode\s*=\s*[1-9]',
97
+ ]
98
+ - control_id: ADRB-LRN-003
99
+ evidence_index: 0
100
+ ci_tools:
101
+ - id: knowledge-curation
102
+ commands:
103
+ - executables: [node, tsx]
104
+ argument_groups:
105
+ - [
106
+ scripts/check-knowledge.js,
107
+ scripts/check-knowledge.ts,
108
+ scripts/validate-knowledge.js,
109
+ scripts/validate-knowledge.ts,
110
+ scripts/curate-knowledge.js,
111
+ scripts/curate-knowledge.ts,
112
+ ]
113
+ source_pattern_groups:
114
+ - ['\b(?:readfilesync|readfile)\s*\([^\n]*(?:lessons|memory|agents\.md|knowledge)']
115
+ - ['\bif\s*\([^\n]*(?:duplicate|stale|contradict|unpromoted)']
116
+ - [
117
+ '\bthrow\s+new\s+error\s*\(',
118
+ '\bprocess\.exit\s*\(\s*[1-9]',
119
+ '\bprocess\.exitcode\s*=\s*[1-9]',
120
+ ]
121
+ - control_id: ADRB-ENV-001
122
+ evidence_index: 0
123
+ files:
124
+ [
125
+ CLAUDE.md,
126
+ .claude/**/*.md,
127
+ .codex/**/*.md,
128
+ .cursor/agents/**/*.md,
129
+ .cursor/rules/**/*.md,
130
+ .cursor/skills/**/*.md,
131
+ .opencode/**/*.md,
132
+ ]
133
+ - control_id: ADRB-SPC-001
134
+ evidence_index: 0
135
+ patterns: [.kiro/specs/**, .cursor/plans/**, .codex/plans/**]
136
+ - control_id: ADRB-SPC-002
137
+ evidence_index: 0
138
+ files: [.kiro/specs/**, .cursor/plans/**, .codex/plans/**]
139
+ - control_id: ADRB-TOL-001
140
+ evidence_index: 0
141
+ patterns:
142
+ [
143
+ .claude/**,
144
+ .codex/**,
145
+ .cursor/**,
146
+ .opencode/**,
147
+ .github/copilot-instructions.md,
148
+ opencode.json,
149
+ ]
150
+ - control_id: ADRB-TOL-002
151
+ evidence_index: 0
152
+ files:
153
+ [
154
+ CLAUDE.md,
155
+ .claude/**/*.md,
156
+ .codex/**/*.md,
157
+ .cursor/agents/**/*.md,
158
+ .cursor/rules/**/*.md,
159
+ .cursor/skills/**/*.md,
160
+ .opencode/**/*.md,
161
+ ]
162
+ - control_id: ADRB-SEC-001
163
+ evidence_index: 0
164
+ files:
165
+ [
166
+ CLAUDE.md,
167
+ .claude/**/*.md,
168
+ .codex/**/*.md,
169
+ .cursor/agents/**/*.md,
170
+ .cursor/rules/**/*.md,
171
+ .cursor/skills/**/*.md,
172
+ .opencode/**/*.md,
173
+ ]
174
+ - control_id: ADRB-SEC-002
175
+ evidence_index: 0
176
+ files:
177
+ [
178
+ CLAUDE.md,
179
+ .claude/**/*.md,
180
+ .codex/**/*.md,
181
+ .cursor/agents/**/*.md,
182
+ .cursor/rules/**/*.md,
183
+ .cursor/skills/**/*.md,
184
+ .opencode/**/*.md,
185
+ ]
186
+ - control_id: ADRB-TST-002
187
+ evidence_index: 0
188
+ files:
189
+ [
190
+ CLAUDE.md,
191
+ .claude/**/*.md,
192
+ .codex/**/*.md,
193
+ .cursor/agents/**/*.md,
194
+ .cursor/rules/**/*.md,
195
+ .cursor/skills/**/*.md,
196
+ .opencode/**/*.md,
197
+ ]
198
+ - control_id: ADRB-GOV-001
199
+ evidence_index: 0
200
+ files:
201
+ [
202
+ CLAUDE.md,
203
+ .claude/**/*.md,
204
+ .codex/**/*.md,
205
+ .cursor/agents/**/*.md,
206
+ .cursor/rules/**/*.md,
207
+ .cursor/skills/**/*.md,
208
+ .opencode/**/*.md,
209
+ ]
210
+ - control_id: ADRB-GOV-002
211
+ evidence_index: 1
212
+ files:
213
+ [
214
+ CLAUDE.md,
215
+ .claude/**/*.md,
216
+ .codex/**/*.md,
217
+ .cursor/agents/**/*.md,
218
+ .cursor/rules/**/*.md,
219
+ .cursor/skills/**/*.md,
220
+ .opencode/**/*.md,
221
+ ]
222
+ - control_id: ADRB-LRN-001
223
+ evidence_index: 0
224
+ patterns: [.claude/memory/**, .codex/memory/**, .cursor/memory/**, .opencode/memory/**]
225
+ - control_id: ADRB-LRN-002
226
+ evidence_index: 0
227
+ files:
228
+ [
229
+ CLAUDE.md,
230
+ .claude/**/*.md,
231
+ .codex/**/*.md,
232
+ .cursor/agents/**/*.md,
233
+ .cursor/rules/**/*.md,
234
+ .cursor/skills/**/*.md,
235
+ .opencode/**/*.md,
236
+ ]
237
+ - control_id: ADRB-OBS-001
238
+ evidence_index: 0
239
+ files: [.cursor/runs/README*, .codex/runs/README*, .claude/runs/README*]
240
+ - control_id: ADRB-OBS-002
241
+ evidence_index: 0
242
+ files: [.cursor/runs/README*, .codex/runs/README*, .claude/runs/README*]
243
+ - control_id: ADRB-RES-001
244
+ evidence_index: 0
245
+ files:
246
+ [
247
+ CLAUDE.md,
248
+ .claude/**/*.md,
249
+ .codex/**/*.md,
250
+ .cursor/agents/**/*.md,
251
+ .cursor/rules/**/*.md,
252
+ .cursor/skills/**/*.md,
253
+ .opencode/**/*.md,
254
+ ]
255
+ - control_id: ADRB-RES-002
256
+ evidence_index: 0
257
+ files:
258
+ [
259
+ CLAUDE.md,
260
+ .claude/**/*.md,
261
+ .codex/**/*.md,
262
+ .cursor/agents/**/*.md,
263
+ .cursor/rules/**/*.md,
264
+ .cursor/skills/**/*.md,
265
+ .opencode/**/*.md,
266
+ ]