agentic-scorecard 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENT_PROMPT.md +60 -31
- package/FEEDBACK.md +53 -0
- package/README.md +96 -37
- package/benchmark/v0.2/adapter-contract.md +25 -0
- package/benchmark/v0.2/agent-evidence-schema.json +52 -0
- package/benchmark/v0.2/attestation-schema.json +26 -0
- package/benchmark/v0.2/benchmark.yaml +127 -0
- package/benchmark/v0.2/controls/context.yaml +51 -0
- package/benchmark/v0.2/controls/environment.yaml +70 -0
- package/benchmark/v0.2/controls/governance.yaml +60 -0
- package/benchmark/v0.2/controls/learning.yaml +48 -0
- package/benchmark/v0.2/controls/observability.yaml +76 -0
- package/benchmark/v0.2/controls/resilience.yaml +60 -0
- package/benchmark/v0.2/controls/security.yaml +71 -0
- package/benchmark/v0.2/controls/specification.yaml +62 -0
- package/benchmark/v0.2/controls/testing.yaml +60 -0
- package/benchmark/v0.2/controls/tooling.yaml +58 -0
- package/benchmark/v0.2/report-schema.json +47 -0
- package/benchmark/v0.2/scoring-policy.md +71 -0
- package/dist/cli.js +607 -102
- package/package.json +5 -3
- package/templates/agent-evidence.yaml +11 -0
- package/templates/attestations.yaml +4 -4
- package/templates/baseline.csv +1 -1
package/AGENT_PROMPT.md
CHANGED
|
@@ -1,10 +1,10 @@
|
|
|
1
|
-
# Ask an AI coding agent to run
|
|
1
|
+
# Ask an AI coding agent to run ADRB v0.2
|
|
2
2
|
|
|
3
|
-
These prompts are
|
|
4
|
-
|
|
5
|
-
model, uploads no source, and makes no network calls after
|
|
3
|
+
These prompts are vendor-neutral. The agent needs terminal access to the repository and permission
|
|
4
|
+
to download the published npm package. The default benchmark runs locally, reads only tracked paths,
|
|
5
|
+
invokes no model, uploads no source, and makes no network calls after installation.
|
|
6
6
|
|
|
7
|
-
## Recommended:
|
|
7
|
+
## Recommended: repository baseline
|
|
8
8
|
|
|
9
9
|
Copy and paste:
|
|
10
10
|
|
|
@@ -12,25 +12,57 @@ Copy and paste:
|
|
|
12
12
|
Assess this repository's readiness for AI-agent pull-request work.
|
|
13
13
|
|
|
14
14
|
From the repository root, run:
|
|
15
|
-
npx --yes agentic-scorecard@0.
|
|
15
|
+
npx --yes agentic-scorecard@0.2.0 assess . --profile pr-creation --scope tracked --format markdown --output .agentic/reports/agentic-readiness-v0.2.0.md
|
|
16
16
|
|
|
17
|
-
Do not change product source
|
|
18
|
-
|
|
17
|
+
Do not change product source, configuration, policies, or tests. Do not invent evidence or mark an
|
|
18
|
+
attestation as met. Read the report and tell me:
|
|
19
19
|
1. the score and highest passed readiness profile;
|
|
20
20
|
2. whether the pr-creation target passes;
|
|
21
|
-
3. each target-profile blocker and
|
|
22
|
-
4. the five highest-value improvements in dependency order;
|
|
23
|
-
5.
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
Keep
|
|
27
|
-
uncommitted unless I explicitly ask
|
|
28
|
-
blocker without
|
|
21
|
+
3. each target-profile blocker and its evidence scope;
|
|
22
|
+
4. the five highest-value improvements in dependency order;
|
|
23
|
+
5. which findings are repository gaps versus external or outcome evidence not yet established; and
|
|
24
|
+
6. any likely false positive or false negative.
|
|
25
|
+
|
|
26
|
+
Keep repository-detected, agent-collected, and human-attested evidence separate. Leave generated
|
|
27
|
+
artifacts uncommitted unless I explicitly ask otherwise. If the command cannot run, explain the
|
|
28
|
+
exact blocker without modifying the repository to work around it.
|
|
29
29
|
```
|
|
30
30
|
|
|
31
|
+
## Agent-assisted external evidence
|
|
32
|
+
|
|
33
|
+
Use this only after reviewing the repository baseline:
|
|
34
|
+
|
|
35
|
+
```text
|
|
36
|
+
Collect the external evidence that ADRB v0.2 could not establish from tracked repository files.
|
|
37
|
+
|
|
38
|
+
First run:
|
|
39
|
+
npx --yes agentic-scorecard@0.2.0 init-evidence .
|
|
40
|
+
|
|
41
|
+
Read `.agentic/evidence-request.md` and the empty target-bound
|
|
42
|
+
`.agentic/agent-evidence.yaml` bundle. Before accessing any connected system, tell me which
|
|
43
|
+
read-only tools, accounts, repositories, branches, dashboards, and time ranges you need. Wait for
|
|
44
|
+
my authorization.
|
|
45
|
+
|
|
46
|
+
After authorization, use only least-privileged read-only operations. For each eligible control,
|
|
47
|
+
record the source-backed status, scope, concise derivation, collection time, 30-day-or-shorter
|
|
48
|
+
expiry, and durable privacy-safe references. Add only claims you actually attempted. Do not paste
|
|
49
|
+
credentials, prompts, source excerpts, raw logs, personal data, or sensitive dashboard contents. A
|
|
50
|
+
permission error or inconclusive result must remain unknown and include the error; never infer a
|
|
51
|
+
pass.
|
|
52
|
+
|
|
53
|
+
Then run:
|
|
54
|
+
npx --yes agentic-scorecard@0.2.0 assess . --profile pr-creation --scope tracked --agent-evidence .agentic/agent-evidence.yaml --format markdown --output .agentic/reports/agentic-readiness-v0.2.0-assisted.md
|
|
55
|
+
|
|
56
|
+
Summarize every score change and keep agent-collected evidence distinct from repository evidence and
|
|
57
|
+
human attestations. Do not describe the result as certified, compliant, safe, secure, or
|
|
58
|
+
independently verified.
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
The agent is an evidence investigator. The scorecard remains the scoring authority.
|
|
62
|
+
|
|
31
63
|
## Choose a different authority level
|
|
32
64
|
|
|
33
|
-
Replace `pr-creation` with
|
|
65
|
+
Replace `pr-creation` with one of:
|
|
34
66
|
|
|
35
67
|
- `read-only-analysis` — inspect approved source and return advice;
|
|
36
68
|
- `planning` — draft plans and specifications;
|
|
@@ -42,23 +74,20 @@ No benchmark profile grants production deployment authority.
|
|
|
42
74
|
|
|
43
75
|
## Guided remediation
|
|
44
76
|
|
|
45
|
-
Use this only after reviewing the first report:
|
|
46
|
-
|
|
47
77
|
```text
|
|
48
|
-
Read
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
implement.
|
|
78
|
+
Read the latest ADRB v0.2 report and create a remediation proposal for the failed target-profile
|
|
79
|
+
controls. Do not implement changes yet. Group recommendations into repository changes, platform or
|
|
80
|
+
organizational controls, and outcome measurement. For each include the control ID, risk reduced,
|
|
81
|
+
systems affected, accountable role, verification method, dependencies, and effort estimate. Never
|
|
82
|
+
invent an attestation or convert unavailable evidence into a failure. End by asking which changes I
|
|
83
|
+
authorize.
|
|
55
84
|
```
|
|
56
85
|
|
|
57
|
-
## Reassessment
|
|
86
|
+
## Reassessment
|
|
58
87
|
|
|
59
88
|
```text
|
|
60
|
-
Re-run the pinned
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
89
|
+
Re-run the pinned ADRB v0.2.0 benchmark for the same profile and tracked scope. Compare reports
|
|
90
|
+
control by control. Explain changes using evidence and identify changed commit state, expired claims,
|
|
91
|
+
scope differences, regressions, or newly established controls. Write a new dated report; do not
|
|
92
|
+
overwrite the previous one.
|
|
64
93
|
```
|
package/FEEDBACK.md
ADDED
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# Feedback guide
|
|
2
|
+
|
|
3
|
+
Feedback from real assessments is essential to improving the Agentic Development Readiness
|
|
4
|
+
Benchmark without weakening its evidence standards or vendor neutrality.
|
|
5
|
+
|
|
6
|
+
## Choose the right channel
|
|
7
|
+
|
|
8
|
+
| Feedback | Channel |
|
|
9
|
+
| ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ |
|
|
10
|
+
| False positive, false negative, unclear result, usability problem, or adoption experience | [Assessment feedback issue](https://github.com/Planet-B2B/agentic-readiness/issues/new?template=assessment-feedback.yml) |
|
|
11
|
+
| Proposed control, scoring, evidence, or readiness-profile change | [Benchmark change issue](https://github.com/Planet-B2B/agentic-readiness/issues/new?template=benchmark-change.yml) |
|
|
12
|
+
| Private adoption question or feedback that cannot be public | [benchmark@planetb2b.com](mailto:benchmark@planetb2b.com) |
|
|
13
|
+
| Suspected security vulnerability | Follow [SECURITY.md](SECURITY.md); do not open a public issue or email the benchmark contact |
|
|
14
|
+
|
|
15
|
+
Public issues are preferred when the information can be safely shared because they let other users
|
|
16
|
+
compare experiences and participate in the design review.
|
|
17
|
+
|
|
18
|
+
## What useful feedback includes
|
|
19
|
+
|
|
20
|
+
- benchmark and package version;
|
|
21
|
+
- requested readiness profile and assessment scope;
|
|
22
|
+
- relevant control IDs;
|
|
23
|
+
- the observed result and the expected result;
|
|
24
|
+
- whether the concern is scanner behavior, control wording, evidence semantics, scoring policy, or
|
|
25
|
+
documentation;
|
|
26
|
+
- a minimal sanitized reproduction or description; and
|
|
27
|
+
- the material impact on adoption or decision-making.
|
|
28
|
+
|
|
29
|
+
Do not attach complete private reports. Never include credentials, proprietary source, private
|
|
30
|
+
URLs, personal data, customer data, or confidential dashboard content. A path name or summarized
|
|
31
|
+
evidence characteristic is usually enough to begin triage.
|
|
32
|
+
|
|
33
|
+
## Ask a coding agent to prepare feedback
|
|
34
|
+
|
|
35
|
+
Paste this prompt into an agent that can read your assessment report:
|
|
36
|
+
|
|
37
|
+
> Review my Agentic Development Readiness report and draft feedback for the
|
|
38
|
+
> `Planet-B2B/agentic-readiness` repository. Include the benchmark version, target profile,
|
|
39
|
+
> assessment scope, relevant control IDs, observed result, expected result, and a minimal sanitized
|
|
40
|
+
> explanation. Classify the feedback as a likely scanner defect, documentation problem, adoption
|
|
41
|
+
> experience, or proposed normative benchmark change. Do not include credentials, proprietary
|
|
42
|
+
> source, private URLs, personal data, customer data, or full report contents. Show me the complete
|
|
43
|
+
> draft and ask for my approval before opening a GitHub issue. If I approve and you have GitHub
|
|
44
|
+
> access, use the assessment feedback issue form; otherwise give me the final text to submit.
|
|
45
|
+
|
|
46
|
+
Agents must treat opening an issue as an external write and obtain accountable human approval first.
|
|
47
|
+
Security concerns must follow `SECURITY.md` rather than this workflow.
|
|
48
|
+
|
|
49
|
+
## Help others discover the benchmark
|
|
50
|
+
|
|
51
|
+
If this project helps your team, [star the repository](https://github.com/Planet-B2B/agentic-readiness).
|
|
52
|
+
Stars make the public benchmark easier for other engineering teams to discover; substantive feedback
|
|
53
|
+
and reproducible examples are even more valuable.
|
package/README.md
CHANGED
|
@@ -1,3 +1,9 @@
|
|
|
1
|
+

|
|
2
|
+
|
|
3
|
+
<p align="center">
|
|
4
|
+
<img src="assets/brand/avatar.png" width="96" height="96" alt="Agentic Development Readiness project mark">
|
|
5
|
+
</p>
|
|
6
|
+
|
|
1
7
|
# Agentic Development Readiness Benchmark
|
|
2
8
|
|
|
3
9
|
A vendor-neutral, evidence-backed benchmark for the engineering harness around AI coding agents.
|
|
@@ -10,8 +16,9 @@ The benchmark measures the harness, not the model brand. It assesses ten dimensi
|
|
|
10
16
|
maturity levels and applies non-compensating floors to five autonomy profiles. A high total score
|
|
11
17
|
cannot hide a critical security, testing, governance, or recovery gap.
|
|
12
18
|
|
|
13
|
-
This repository is
|
|
14
|
-
|
|
19
|
+
This repository is a **v0.2 reference implementation** intended for public review and piloting. It
|
|
20
|
+
is not a certification standard. The immutable v0.1 benchmark remains available for historical
|
|
21
|
+
reproduction; v0.1 and v0.2 scores are not directly comparable.
|
|
15
22
|
|
|
16
23
|
## Ask your coding agent
|
|
17
24
|
|
|
@@ -19,17 +26,42 @@ You do not need to install or learn the CLI yourself. Paste this prompt into Cod
|
|
|
19
26
|
GitHub Copilot, Cursor, or another coding agent that has terminal access to your repository:
|
|
20
27
|
|
|
21
28
|
> Assess this repository's readiness for AI-agent pull-request work. From the repository root, run
|
|
22
|
-
> `npx --yes agentic-scorecard@0.
|
|
29
|
+
> `npx --yes agentic-scorecard@0.2.0 assess . --profile pr-creation --scope tracked --format markdown --output .agentic/reports/agentic-readiness-v0.2.0.md`.
|
|
23
30
|
> Do not change product source code or invent attestations. Read the resulting report and summarize
|
|
24
31
|
> the score, highest passed profile, whether `pr-creation` passes, target-profile blockers, and the
|
|
25
|
-
> five highest-value improvements.
|
|
26
|
-
>
|
|
27
|
-
> uncommitted unless I ask you to commit
|
|
32
|
+
> five highest-value improvements. Separate repository gaps from external and outcome evidence, and
|
|
33
|
+
> keep repository-detected, agent-collected, and human-attested evidence distinct. Flag likely false
|
|
34
|
+
> positives or negatives. Leave generated artifacts uncommitted unless I ask you to commit them.
|
|
28
35
|
|
|
29
36
|
That is the recommended first assessment. The agent downloads the pinned benchmark package, runs the
|
|
30
37
|
read-only local collector, and explains the results. For alternate authority levels or a guided
|
|
31
38
|
remediation session, use the prompts in [AGENT_PROMPT.md](AGENT_PROMPT.md).
|
|
32
39
|
|
|
40
|
+
## Feedback and community
|
|
41
|
+
|
|
42
|
+
Real-world assessment feedback helps make the benchmark more accurate and useful across different
|
|
43
|
+
engineering organizations and agent harnesses.
|
|
44
|
+
|
|
45
|
+
- Share false positives, false negatives, unclear controls, and adoption experience through the
|
|
46
|
+
[assessment feedback form](https://github.com/Planet-B2B/agentic-readiness/issues/new?template=assessment-feedback.yml).
|
|
47
|
+
- Propose scoring, control, evidence, or readiness-profile changes through the
|
|
48
|
+
[benchmark change form](https://github.com/Planet-B2B/agentic-readiness/issues/new?template=benchmark-change.yml).
|
|
49
|
+
- Send private adoption questions or feedback that cannot be shared publicly to
|
|
50
|
+
[benchmark@planetb2b.com](mailto:benchmark@planetb2b.com). Do not email vulnerability reports;
|
|
51
|
+
follow [SECURITY.md](SECURITY.md) instead.
|
|
52
|
+
|
|
53
|
+
Ask a coding agent to prepare privacy-safe feedback with this prompt:
|
|
54
|
+
|
|
55
|
+
> Review my Agentic Development Readiness report and draft feedback for the
|
|
56
|
+
> `Planet-B2B/agentic-readiness` repository. Include the benchmark version, target profile, relevant
|
|
57
|
+
> control IDs, observed result, expected result, and a minimal sanitized explanation. Distinguish a
|
|
58
|
+
> likely scanner defect from a proposed benchmark-policy change. Do not include credentials,
|
|
59
|
+
> proprietary source, private URLs, personal data, customer data, or full report contents. Show me
|
|
60
|
+
> the draft and ask for my approval before opening a GitHub issue.
|
|
61
|
+
|
|
62
|
+
See [FEEDBACK.md](FEEDBACK.md) for the full feedback guide. If the project is useful to you, please
|
|
63
|
+
[star the repository](https://github.com/Planet-B2B/agentic-readiness) so others can discover it.
|
|
64
|
+
|
|
33
65
|
## Quick start
|
|
34
66
|
|
|
35
67
|
### Installation and prerequisites
|
|
@@ -38,8 +70,9 @@ A supported Node.js LTS release (20.19+, 22.13+, or 24+) is required. Assessment
|
|
|
38
70
|
read-only, and offline by default.
|
|
39
71
|
|
|
40
72
|
```bash
|
|
41
|
-
npx agentic-scorecard@0.
|
|
73
|
+
npx agentic-scorecard@0.2.0 assess /path/to/repository \
|
|
42
74
|
--profile pr-creation \
|
|
75
|
+
--scope tracked \
|
|
43
76
|
--format markdown \
|
|
44
77
|
--output agentic-readiness.md
|
|
45
78
|
```
|
|
@@ -47,11 +80,31 @@ npx agentic-scorecard@0.1.0 assess /path/to/repository \
|
|
|
47
80
|
To record controls that repository inspection cannot prove:
|
|
48
81
|
|
|
49
82
|
```bash
|
|
50
|
-
npx agentic-scorecard@0.
|
|
83
|
+
npx agentic-scorecard@0.2.0 init /path/to/repository
|
|
51
84
|
```
|
|
52
85
|
|
|
53
|
-
Complete `.agentic/attestations.yaml` with owners and
|
|
54
|
-
|
|
86
|
+
Complete `.agentic/attestations.yaml` with owners and durable evidence links, then assess again.
|
|
87
|
+
Human-attested evidence remains visibly distinct in every report.
|
|
88
|
+
|
|
89
|
+
To let an authorized coding agent collect evidence from Git hosting, CI, dashboards, or other
|
|
90
|
+
external systems, first generate a target-bound template. The repository must be a Git worktree
|
|
91
|
+
with at least one commit so the bundle can bind to the exact assessed state:
|
|
92
|
+
|
|
93
|
+
```bash
|
|
94
|
+
npx agentic-scorecard@0.2.0 init-evidence /path/to/repository
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
Ask the agent to review `.agentic/evidence-request.md`, obtain approval before using
|
|
98
|
+
least-privileged read-only connectors, add attempted claims to `.agentic/agent-evidence.yaml`, and
|
|
99
|
+
rerun with `--agent-evidence`. The default bundle path is loaded automatically. See
|
|
100
|
+
[AGENT_PROMPT.md](AGENT_PROMPT.md) for the complete copy-and-paste workflow.
|
|
101
|
+
|
|
102
|
+
### Migrating from v0.1
|
|
103
|
+
|
|
104
|
+
Run a new tracked-scope baseline and retain the old report as historical evidence. Do not present the
|
|
105
|
+
score change as improvement or regression because v0.2 changes evidence semantics. Re-review v0.1
|
|
106
|
+
attestations before recreating them for v0.2; v0.2 repository-artifact controls cannot be overridden
|
|
107
|
+
by declaration, and every human or agent-collected external claim must expire.
|
|
55
108
|
|
|
56
109
|
For development from this checkout:
|
|
57
110
|
|
|
@@ -104,30 +157,32 @@ No profile grants production deployment authority. Organizations should evaluate
|
|
|
104
157
|
through a separate, system-specific safety case.
|
|
105
158
|
|
|
106
159
|
The exact floors live in
|
|
107
|
-
[`benchmark/v0.
|
|
108
|
-
[`benchmark/v0.
|
|
160
|
+
[`benchmark/v0.2/benchmark.yaml`](benchmark/v0.2/benchmark.yaml) and their rationale in
|
|
161
|
+
[`benchmark/v0.2/scoring-policy.md`](benchmark/v0.2/scoring-policy.md).
|
|
109
162
|
|
|
110
|
-
## Evidence and trust labels
|
|
163
|
+
## Evidence scopes and trust labels
|
|
111
164
|
|
|
112
|
-
- **
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
- **
|
|
117
|
-
|
|
165
|
+
- **Repository-detected:** the local collector found qualifying evidence in the selected path scope.
|
|
166
|
+
This proves an artifact match, not consistent practice or external enforcement.
|
|
167
|
+
- **Agent-collected:** an authorized agent supplied a target-bound, expiring, source-backed external
|
|
168
|
+
claim. It is not independently verified.
|
|
169
|
+
- **Human-attested:** an accountable owner supplied a dated evidence link or explanation.
|
|
170
|
+
- **Unknown:** evidence is unavailable, expired, unauthorized, mismatched, or inconclusive.
|
|
118
171
|
|
|
119
|
-
|
|
120
|
-
or
|
|
121
|
-
|
|
122
|
-
|
|
172
|
+
Controls also identify whether their evidence belongs in the repository, hosting platform,
|
|
173
|
+
organization, or outcome systems. This prevents expected external unknowns from masquerading as
|
|
174
|
+
missing files. Future independently conformant adapters require a separate protocol; v0.2 does not
|
|
175
|
+
issue certification or independent-verification claims.
|
|
123
176
|
|
|
124
177
|
## Reports and CI
|
|
125
178
|
|
|
126
179
|
The CLI emits Markdown for people and stable JSON for automation:
|
|
127
180
|
|
|
128
181
|
```bash
|
|
129
|
-
agentic-scorecard assess . --format json --output .agentic/report.json
|
|
182
|
+
agentic-scorecard assess . --scope tracked --format json --output .agentic/report.json
|
|
183
|
+
agentic-scorecard assess . --profile pr-creation --agent-evidence .agentic/agent-evidence.yaml
|
|
130
184
|
agentic-scorecard assess . --profile pr-creation --enforce
|
|
185
|
+
agentic-scorecard init-evidence .
|
|
131
186
|
agentic-scorecard explain ADRB-SEC-003
|
|
132
187
|
agentic-scorecard validate
|
|
133
188
|
```
|
|
@@ -142,11 +197,12 @@ funded the remediation plan. See
|
|
|
142
197
|
The default collector:
|
|
143
198
|
|
|
144
199
|
- makes no network calls and invokes no model;
|
|
145
|
-
- reads only
|
|
146
|
-
- ignores `.git`,
|
|
200
|
+
- reads only Git-tracked paths and records commit and dirty-worktree metadata;
|
|
201
|
+
- ignores `.git`, dependencies, build output, coverage, generated reports, attestations, and imported
|
|
202
|
+
evidence bundles;
|
|
147
203
|
- caps content-scanned files at 512 KB;
|
|
148
204
|
- reports file paths and match counts, never matching source snippets;
|
|
149
|
-
- writes nothing unless `--output` or `init` is explicitly requested.
|
|
205
|
+
- writes nothing unless `--output`, `init`, or `init-evidence` is explicitly requested.
|
|
150
206
|
|
|
151
207
|
Do not place credentials, private prompts, source excerpts, or personal data in attestations. Link to
|
|
152
208
|
access-controlled evidence instead. Report suspected vulnerabilities through [SECURITY.md](SECURITY.md).
|
|
@@ -154,7 +210,8 @@ access-controlled evidence instead. Report suspected vulnerabilities through [SE
|
|
|
154
210
|
## Repository structure
|
|
155
211
|
|
|
156
212
|
```text
|
|
157
|
-
benchmark/v0.1/ immutable
|
|
213
|
+
benchmark/v0.1/ immutable historical v0.1 definition
|
|
214
|
+
benchmark/v0.2/ current normative benchmark, schemas, and controls
|
|
158
215
|
benchmark/mappings/ informative mappings to external frameworks
|
|
159
216
|
src/ reference CLI and local evidence collectors
|
|
160
217
|
templates/ adoption, preflight, attestation, and remediation templates
|
|
@@ -171,21 +228,23 @@ OIDC-provenance release process.
|
|
|
171
228
|
|
|
172
229
|
1. Run a local baseline for the least-authoritative profile you actually need.
|
|
173
230
|
2. Review every result with security, platform, and representative delivery teams.
|
|
174
|
-
3.
|
|
175
|
-
4.
|
|
176
|
-
5.
|
|
177
|
-
6.
|
|
178
|
-
7.
|
|
231
|
+
3. Use an authorized agent to collect source-backed external evidence where appropriate.
|
|
232
|
+
4. Supply owned human attestations only where deterministic collection is unavailable.
|
|
233
|
+
5. Publish the report internally with explicit limitations, scope, commit, and benchmark version.
|
|
234
|
+
6. Fund the smallest improvements that close target-profile blockers.
|
|
235
|
+
7. Reassess on material harness changes and at least quarterly.
|
|
236
|
+
8. Add a non-blocking CI report; enforce only the agreed target profile after a pilot.
|
|
179
237
|
|
|
180
238
|
Never optimize to the number alone. Use the control evidence and outcome metrics to improve the
|
|
181
239
|
system, and keep exceptions narrow, owned, expiring, and visible.
|
|
182
240
|
|
|
183
241
|
## Status and roadmap
|
|
184
242
|
|
|
185
|
-
v0.
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
independent-review protocol. These require public
|
|
243
|
+
v0.2 adds integrity-safe tracked-path collection, evidence scopes, precise co-located content
|
|
244
|
+
matching, agent-collected external evidence, and transparent report grouping. Candidate next steps
|
|
245
|
+
include conformant signed adapters, SARIF/HTML reports, organization-level aggregation,
|
|
246
|
+
statistically designed benchmark tasks, and an independent-review protocol. These require public
|
|
247
|
+
design review before becoming normative.
|
|
189
248
|
|
|
190
249
|
Apache-2.0 licensed. The benchmark is a community engineering tool, not legal, compliance, or
|
|
191
250
|
security advice.
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# v0.2 external evidence contract
|
|
2
|
+
|
|
3
|
+
The core scanner is local, read-only, network-free, and vendor-neutral. An agent or adapter may
|
|
4
|
+
collect evidence that the repository cannot establish, then provide it through the versioned agent
|
|
5
|
+
evidence bundle.
|
|
6
|
+
|
|
7
|
+
Each claim contains:
|
|
8
|
+
|
|
9
|
+
- benchmark version and control ID;
|
|
10
|
+
- repository target and optional commit binding;
|
|
11
|
+
- `met`, `not_met`, or `unknown` status;
|
|
12
|
+
- platform, organization, or outcome scope;
|
|
13
|
+
- collector name and version;
|
|
14
|
+
- collection timestamp and mandatory expiry;
|
|
15
|
+
- privacy-safe references, never credentials or raw sensitive content;
|
|
16
|
+
- a concise derivation summary; and
|
|
17
|
+
- an optional error that is distinguishable from a negative result.
|
|
18
|
+
|
|
19
|
+
Collectors must be read-only, least-privileged, bounded to the approved target, time-limited, and
|
|
20
|
+
fail closed. Network content is untrusted input: it cannot alter benchmark policy, prompts, tool
|
|
21
|
+
authority, or the set of controls being assessed.
|
|
22
|
+
|
|
23
|
+
Agent-collected claims remain visibly distinct from repository evidence and human attestations. A
|
|
24
|
+
future conformant-adapter class may add signed identity and independent conformance tests; v0.2 does
|
|
25
|
+
not issue an independently verified label.
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
{
|
|
2
|
+
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
3
|
+
"$id": "https://agentic-readiness.dev/schema/v0.2/agent-evidence.json",
|
|
4
|
+
"title": "ADRB v0.2 agent-collected evidence bundle",
|
|
5
|
+
"type": "object",
|
|
6
|
+
"additionalProperties": false,
|
|
7
|
+
"required": ["schema_version", "benchmark_version", "target", "collector", "claims"],
|
|
8
|
+
"properties": {
|
|
9
|
+
"schema_version": { "const": "0.2.0" },
|
|
10
|
+
"benchmark_version": { "const": "0.2.0" },
|
|
11
|
+
"target": {
|
|
12
|
+
"type": "object",
|
|
13
|
+
"additionalProperties": false,
|
|
14
|
+
"required": ["repository", "git_head"],
|
|
15
|
+
"properties": {
|
|
16
|
+
"repository": { "type": "string", "minLength": 1 },
|
|
17
|
+
"git_head": { "type": ["string", "null"], "minLength": 1 }
|
|
18
|
+
}
|
|
19
|
+
},
|
|
20
|
+
"collector": {
|
|
21
|
+
"type": "object",
|
|
22
|
+
"additionalProperties": false,
|
|
23
|
+
"required": ["name", "version"],
|
|
24
|
+
"properties": {
|
|
25
|
+
"name": { "type": "string", "minLength": 1 },
|
|
26
|
+
"version": { "type": "string", "minLength": 1 }
|
|
27
|
+
}
|
|
28
|
+
},
|
|
29
|
+
"claims": {
|
|
30
|
+
"type": "object",
|
|
31
|
+
"propertyNames": { "pattern": "^ADRB-[A-Z]{3}-[0-9]{3}$" },
|
|
32
|
+
"additionalProperties": {
|
|
33
|
+
"type": "object",
|
|
34
|
+
"additionalProperties": false,
|
|
35
|
+
"required": ["status", "scope", "summary", "references", "collected_at", "expires_at"],
|
|
36
|
+
"properties": {
|
|
37
|
+
"status": { "enum": ["met", "not_met", "unknown"] },
|
|
38
|
+
"scope": { "enum": ["platform", "organization", "outcome"] },
|
|
39
|
+
"summary": { "type": "string", "minLength": 1 },
|
|
40
|
+
"references": {
|
|
41
|
+
"type": "array",
|
|
42
|
+
"minItems": 1,
|
|
43
|
+
"items": { "type": "string", "minLength": 1 }
|
|
44
|
+
},
|
|
45
|
+
"collected_at": { "type": "string", "format": "date-time" },
|
|
46
|
+
"expires_at": { "type": "string", "format": "date-time" },
|
|
47
|
+
"error": { "type": ["string", "null"], "minLength": 1 }
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
}
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
{
|
|
2
|
+
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
3
|
+
"$id": "https://agentic-readiness.dev/schema/v0.2/attestations.json",
|
|
4
|
+
"title": "ADRB v0.2 human attestation file",
|
|
5
|
+
"type": "object",
|
|
6
|
+
"additionalProperties": false,
|
|
7
|
+
"required": ["benchmark_version", "attestations"],
|
|
8
|
+
"properties": {
|
|
9
|
+
"benchmark_version": { "const": "0.2.0" },
|
|
10
|
+
"attestations": {
|
|
11
|
+
"type": "object",
|
|
12
|
+
"additionalProperties": {
|
|
13
|
+
"type": "object",
|
|
14
|
+
"additionalProperties": false,
|
|
15
|
+
"required": ["status", "evidence", "owner", "reviewed_at", "expires_at"],
|
|
16
|
+
"properties": {
|
|
17
|
+
"status": { "enum": ["met", "not_met", "not_applicable", "unknown"] },
|
|
18
|
+
"evidence": { "type": "string", "minLength": 1 },
|
|
19
|
+
"owner": { "type": "string", "minLength": 1 },
|
|
20
|
+
"reviewed_at": { "type": "string", "format": "date" },
|
|
21
|
+
"expires_at": { "type": "string", "format": "date" }
|
|
22
|
+
}
|
|
23
|
+
}
|
|
24
|
+
}
|
|
25
|
+
}
|
|
26
|
+
}
|
|
@@ -0,0 +1,127 @@
|
|
|
1
|
+
id: agentic-development-readiness
|
|
2
|
+
version: 0.2.0
|
|
3
|
+
title: Agentic Development Readiness Benchmark
|
|
4
|
+
description: >-
|
|
5
|
+
A vendor-neutral, evidence-backed assessment of the engineering harness that surrounds AI agents.
|
|
6
|
+
It measures whether a repository and its operating controls can safely support increasing levels
|
|
7
|
+
of delegated software work.
|
|
8
|
+
|
|
9
|
+
maturity_levels:
|
|
10
|
+
0: Absent
|
|
11
|
+
1: Ad hoc
|
|
12
|
+
2: Documented
|
|
13
|
+
3: Enforced
|
|
14
|
+
4: Measured and improving
|
|
15
|
+
|
|
16
|
+
dimensions:
|
|
17
|
+
- id: context
|
|
18
|
+
title: Context and knowledge
|
|
19
|
+
description: Agents can discover authoritative, scoped, current instructions without guesswork.
|
|
20
|
+
- id: environment
|
|
21
|
+
title: Reproducible environment
|
|
22
|
+
description: Agents can establish and verify a faithful development environment.
|
|
23
|
+
- id: specification
|
|
24
|
+
title: Specification and planning
|
|
25
|
+
description: Work starts from testable outcomes, constraints, and explicit acceptance criteria.
|
|
26
|
+
- id: tooling
|
|
27
|
+
title: Tooling and interfaces
|
|
28
|
+
description: Agent tools are discoverable, least-privileged, observable, and fail safely.
|
|
29
|
+
- id: security
|
|
30
|
+
title: Security and data governance
|
|
31
|
+
description: Credentials, data, autonomy, and untrusted content have explicit boundaries.
|
|
32
|
+
- id: testing
|
|
33
|
+
title: Verification and testing
|
|
34
|
+
description: Changes are checked at the right layers with trustworthy, reproducible signals.
|
|
35
|
+
- id: governance
|
|
36
|
+
title: Review and change governance
|
|
37
|
+
description: Human accountability, change scope, and merge authority remain explicit.
|
|
38
|
+
- id: learning
|
|
39
|
+
title: Learning and knowledge maintenance
|
|
40
|
+
description: Corrections compound into maintained instructions instead of being rediscovered.
|
|
41
|
+
- id: observability
|
|
42
|
+
title: Outcome observability
|
|
43
|
+
description: Runs and outcomes can be evaluated by task class, risk, cost, and quality.
|
|
44
|
+
- id: resilience
|
|
45
|
+
title: Failure containment and recovery
|
|
46
|
+
description: Agent failures are bounded, reversible, diagnosed, and rehearsed.
|
|
47
|
+
|
|
48
|
+
readiness_profiles:
|
|
49
|
+
- id: read-only-analysis
|
|
50
|
+
title: Read-only analysis
|
|
51
|
+
description: Agent may inspect approved repository content and return advice, without mutations.
|
|
52
|
+
floors:
|
|
53
|
+
context: 1
|
|
54
|
+
environment: 0
|
|
55
|
+
specification: 0
|
|
56
|
+
tooling: 0
|
|
57
|
+
security: 1
|
|
58
|
+
testing: 0
|
|
59
|
+
governance: 0
|
|
60
|
+
learning: 0
|
|
61
|
+
observability: 0
|
|
62
|
+
resilience: 0
|
|
63
|
+
- id: planning
|
|
64
|
+
title: Planning and specification
|
|
65
|
+
description: Agent may draft plans, specifications, and proposed changes for human review.
|
|
66
|
+
floors:
|
|
67
|
+
context: 2
|
|
68
|
+
environment: 0
|
|
69
|
+
specification: 1
|
|
70
|
+
tooling: 1
|
|
71
|
+
security: 1
|
|
72
|
+
testing: 0
|
|
73
|
+
governance: 1
|
|
74
|
+
learning: 0
|
|
75
|
+
observability: 0
|
|
76
|
+
resilience: 0
|
|
77
|
+
- id: local-implementation
|
|
78
|
+
title: Local implementation
|
|
79
|
+
description: Agent may edit an isolated checkout and run approved local verification.
|
|
80
|
+
floors:
|
|
81
|
+
context: 2
|
|
82
|
+
environment: 2
|
|
83
|
+
specification: 2
|
|
84
|
+
tooling: 2
|
|
85
|
+
security: 2
|
|
86
|
+
testing: 2
|
|
87
|
+
governance: 1
|
|
88
|
+
learning: 1
|
|
89
|
+
observability: 1
|
|
90
|
+
resilience: 1
|
|
91
|
+
- id: pr-creation
|
|
92
|
+
title: Pull request creation
|
|
93
|
+
description: Agent may create a branch and propose a pull request; a human retains merge authority.
|
|
94
|
+
floors:
|
|
95
|
+
context: 2
|
|
96
|
+
environment: 2
|
|
97
|
+
specification: 2
|
|
98
|
+
tooling: 2
|
|
99
|
+
security: 2
|
|
100
|
+
testing: 2
|
|
101
|
+
governance: 2
|
|
102
|
+
learning: 1
|
|
103
|
+
observability: 1
|
|
104
|
+
resilience: 2
|
|
105
|
+
- id: limited-autonomous-maintenance
|
|
106
|
+
title: Limited autonomous maintenance
|
|
107
|
+
description: >-
|
|
108
|
+
Agent may complete pre-approved, low-risk maintenance within strict scope, budget, and stop
|
|
109
|
+
conditions. This profile never grants production deployment authority.
|
|
110
|
+
floors:
|
|
111
|
+
context: 3
|
|
112
|
+
environment: 3
|
|
113
|
+
specification: 3
|
|
114
|
+
tooling: 3
|
|
115
|
+
security: 3
|
|
116
|
+
testing: 3
|
|
117
|
+
governance: 3
|
|
118
|
+
learning: 3
|
|
119
|
+
observability: 3
|
|
120
|
+
resilience: 3
|
|
121
|
+
|
|
122
|
+
scoring:
|
|
123
|
+
dimension_method: consecutive-levels
|
|
124
|
+
overall_method: sum
|
|
125
|
+
maximum_score: 40
|
|
126
|
+
readiness_method: non-compensating-floors
|
|
127
|
+
attestation_counts_as_verified: false
|