agentic-scorecard 0.1.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENT_PROMPT.md +68 -34
- package/FEEDBACK.md +53 -0
- package/README.md +120 -39
- package/benchmark/v0.2/adapter-contract.md +25 -0
- package/benchmark/v0.2/agent-evidence-schema.json +52 -0
- package/benchmark/v0.2/attestation-schema.json +26 -0
- package/benchmark/v0.2/benchmark.yaml +127 -0
- package/benchmark/v0.2/controls/context.yaml +51 -0
- package/benchmark/v0.2/controls/environment.yaml +70 -0
- package/benchmark/v0.2/controls/governance.yaml +60 -0
- package/benchmark/v0.2/controls/learning.yaml +48 -0
- package/benchmark/v0.2/controls/observability.yaml +76 -0
- package/benchmark/v0.2/controls/resilience.yaml +60 -0
- package/benchmark/v0.2/controls/security.yaml +71 -0
- package/benchmark/v0.2/controls/specification.yaml +62 -0
- package/benchmark/v0.2/controls/testing.yaml +60 -0
- package/benchmark/v0.2/controls/tooling.yaml +58 -0
- package/benchmark/v0.2/report-schema.json +47 -0
- package/benchmark/v0.2/scoring-policy.md +71 -0
- package/benchmark/v0.3/adapter-contract.md +44 -0
- package/benchmark/v0.3/adapters/known-agent-harnesses.yaml +177 -0
- package/benchmark/v0.3/adapters/known-python-tooling.yaml +18 -0
- package/benchmark/v0.3/adapters/known-security-tools.yaml +6 -0
- package/benchmark/v0.3/agent-evidence-schema.json +67 -0
- package/benchmark/v0.3/attestation-schema.json +26 -0
- package/benchmark/v0.3/benchmark.yaml +127 -0
- package/benchmark/v0.3/controls/context.yaml +79 -0
- package/benchmark/v0.3/controls/environment.yaml +93 -0
- package/benchmark/v0.3/controls/governance.yaml +97 -0
- package/benchmark/v0.3/controls/learning.yaml +80 -0
- package/benchmark/v0.3/controls/observability.yaml +91 -0
- package/benchmark/v0.3/controls/resilience.yaml +107 -0
- package/benchmark/v0.3/controls/security.yaml +129 -0
- package/benchmark/v0.3/controls/specification.yaml +83 -0
- package/benchmark/v0.3/controls/testing.yaml +85 -0
- package/benchmark/v0.3/controls/tooling.yaml +69 -0
- package/benchmark/v0.3/report-schema.json +89 -0
- package/benchmark/v0.3/scoring-policy.md +100 -0
- package/dist/cli.js +1043 -111
- package/package.json +5 -3
- package/templates/agent-evidence.yaml +11 -0
- package/templates/attestations.yaml +4 -4
- package/templates/baseline.csv +1 -1
|
@@ -0,0 +1,127 @@
|
|
|
1
|
+
id: agentic-development-readiness
|
|
2
|
+
version: 0.2.0
|
|
3
|
+
title: Agentic Development Readiness Benchmark
|
|
4
|
+
description: >-
|
|
5
|
+
A vendor-neutral, evidence-backed assessment of the engineering harness that surrounds AI agents.
|
|
6
|
+
It measures whether a repository and its operating controls can safely support increasing levels
|
|
7
|
+
of delegated software work.
|
|
8
|
+
|
|
9
|
+
maturity_levels:
|
|
10
|
+
0: Absent
|
|
11
|
+
1: Ad hoc
|
|
12
|
+
2: Documented
|
|
13
|
+
3: Enforced
|
|
14
|
+
4: Measured and improving
|
|
15
|
+
|
|
16
|
+
dimensions:
|
|
17
|
+
- id: context
|
|
18
|
+
title: Context and knowledge
|
|
19
|
+
description: Agents can discover authoritative, scoped, current instructions without guesswork.
|
|
20
|
+
- id: environment
|
|
21
|
+
title: Reproducible environment
|
|
22
|
+
description: Agents can establish and verify a faithful development environment.
|
|
23
|
+
- id: specification
|
|
24
|
+
title: Specification and planning
|
|
25
|
+
description: Work starts from testable outcomes, constraints, and explicit acceptance criteria.
|
|
26
|
+
- id: tooling
|
|
27
|
+
title: Tooling and interfaces
|
|
28
|
+
description: Agent tools are discoverable, least-privileged, observable, and fail safely.
|
|
29
|
+
- id: security
|
|
30
|
+
title: Security and data governance
|
|
31
|
+
description: Credentials, data, autonomy, and untrusted content have explicit boundaries.
|
|
32
|
+
- id: testing
|
|
33
|
+
title: Verification and testing
|
|
34
|
+
description: Changes are checked at the right layers with trustworthy, reproducible signals.
|
|
35
|
+
- id: governance
|
|
36
|
+
title: Review and change governance
|
|
37
|
+
description: Human accountability, change scope, and merge authority remain explicit.
|
|
38
|
+
- id: learning
|
|
39
|
+
title: Learning and knowledge maintenance
|
|
40
|
+
description: Corrections compound into maintained instructions instead of being rediscovered.
|
|
41
|
+
- id: observability
|
|
42
|
+
title: Outcome observability
|
|
43
|
+
description: Runs and outcomes can be evaluated by task class, risk, cost, and quality.
|
|
44
|
+
- id: resilience
|
|
45
|
+
title: Failure containment and recovery
|
|
46
|
+
description: Agent failures are bounded, reversible, diagnosed, and rehearsed.
|
|
47
|
+
|
|
48
|
+
readiness_profiles:
|
|
49
|
+
- id: read-only-analysis
|
|
50
|
+
title: Read-only analysis
|
|
51
|
+
description: Agent may inspect approved repository content and return advice, without mutations.
|
|
52
|
+
floors:
|
|
53
|
+
context: 1
|
|
54
|
+
environment: 0
|
|
55
|
+
specification: 0
|
|
56
|
+
tooling: 0
|
|
57
|
+
security: 1
|
|
58
|
+
testing: 0
|
|
59
|
+
governance: 0
|
|
60
|
+
learning: 0
|
|
61
|
+
observability: 0
|
|
62
|
+
resilience: 0
|
|
63
|
+
- id: planning
|
|
64
|
+
title: Planning and specification
|
|
65
|
+
description: Agent may draft plans, specifications, and proposed changes for human review.
|
|
66
|
+
floors:
|
|
67
|
+
context: 2
|
|
68
|
+
environment: 0
|
|
69
|
+
specification: 1
|
|
70
|
+
tooling: 1
|
|
71
|
+
security: 1
|
|
72
|
+
testing: 0
|
|
73
|
+
governance: 1
|
|
74
|
+
learning: 0
|
|
75
|
+
observability: 0
|
|
76
|
+
resilience: 0
|
|
77
|
+
- id: local-implementation
|
|
78
|
+
title: Local implementation
|
|
79
|
+
description: Agent may edit an isolated checkout and run approved local verification.
|
|
80
|
+
floors:
|
|
81
|
+
context: 2
|
|
82
|
+
environment: 2
|
|
83
|
+
specification: 2
|
|
84
|
+
tooling: 2
|
|
85
|
+
security: 2
|
|
86
|
+
testing: 2
|
|
87
|
+
governance: 1
|
|
88
|
+
learning: 1
|
|
89
|
+
observability: 1
|
|
90
|
+
resilience: 1
|
|
91
|
+
- id: pr-creation
|
|
92
|
+
title: Pull request creation
|
|
93
|
+
description: Agent may create a branch and propose a pull request; a human retains merge authority.
|
|
94
|
+
floors:
|
|
95
|
+
context: 2
|
|
96
|
+
environment: 2
|
|
97
|
+
specification: 2
|
|
98
|
+
tooling: 2
|
|
99
|
+
security: 2
|
|
100
|
+
testing: 2
|
|
101
|
+
governance: 2
|
|
102
|
+
learning: 1
|
|
103
|
+
observability: 1
|
|
104
|
+
resilience: 2
|
|
105
|
+
- id: limited-autonomous-maintenance
|
|
106
|
+
title: Limited autonomous maintenance
|
|
107
|
+
description: >-
|
|
108
|
+
Agent may complete pre-approved, low-risk maintenance within strict scope, budget, and stop
|
|
109
|
+
conditions. This profile never grants production deployment authority.
|
|
110
|
+
floors:
|
|
111
|
+
context: 3
|
|
112
|
+
environment: 3
|
|
113
|
+
specification: 3
|
|
114
|
+
tooling: 3
|
|
115
|
+
security: 3
|
|
116
|
+
testing: 3
|
|
117
|
+
governance: 3
|
|
118
|
+
learning: 3
|
|
119
|
+
observability: 3
|
|
120
|
+
resilience: 3
|
|
121
|
+
|
|
122
|
+
scoring:
|
|
123
|
+
dimension_method: consecutive-levels
|
|
124
|
+
overall_method: sum
|
|
125
|
+
maximum_score: 40
|
|
126
|
+
readiness_method: non-compensating-floors
|
|
127
|
+
attestation_counts_as_verified: false
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
dimension: context
|
|
2
|
+
controls:
|
|
3
|
+
- id: ADRB-CTX-001
|
|
4
|
+
level: 1
|
|
5
|
+
title: Agent guidance is discoverable
|
|
6
|
+
outcome: A contributor or agent can locate the repository's authoritative operating guidance.
|
|
7
|
+
risk: Agents infer repository rules from generic practice or stale neighboring files.
|
|
8
|
+
evidence:
|
|
9
|
+
- type: path_any
|
|
10
|
+
scope: repository
|
|
11
|
+
patterns: [AGENTS.md, CLAUDE.md, .github/copilot-instructions.md, .agents/**, .claude/**]
|
|
12
|
+
remediation: Add a tracked, non-empty agent entry point that names the authoritative guidance.
|
|
13
|
+
- id: ADRB-CTX-002
|
|
14
|
+
level: 2
|
|
15
|
+
title: Guidance scope and precedence are explicit
|
|
16
|
+
outcome: Instructions state their scope, precedence, important constraints, and architecture boundaries.
|
|
17
|
+
risk: Conflicting or nested guidance produces plausible changes that violate local invariants.
|
|
18
|
+
evidence:
|
|
19
|
+
- type: content_terms
|
|
20
|
+
scope: repository
|
|
21
|
+
files:
|
|
22
|
+
[AGENTS.md, CLAUDE.md, .github/copilot-instructions.md, .agents/**/*.md, .claude/**/*.md]
|
|
23
|
+
terms: [scope, precedence, architecture, must, constraint]
|
|
24
|
+
min_terms: 2
|
|
25
|
+
remediation: Document instruction scope, precedence, non-negotiable constraints, and where deeper guidance lives.
|
|
26
|
+
- id: ADRB-CTX-003
|
|
27
|
+
level: 3
|
|
28
|
+
title: Agent guidance integrity is mechanically checked
|
|
29
|
+
outcome: Automated validation detects missing, broken, stale, or structurally invalid agent guidance.
|
|
30
|
+
risk: Critical instructions silently drift while agents continue treating them as authoritative.
|
|
31
|
+
evidence:
|
|
32
|
+
- type: content_terms
|
|
33
|
+
scope: repository
|
|
34
|
+
files: [.github/workflows/**, scripts/**, package.json, Makefile]
|
|
35
|
+
terms: [validate, lint, stale, broken link, integrity, check]
|
|
36
|
+
min_terms: 1
|
|
37
|
+
required_any_terms:
|
|
38
|
+
[agent docs, agent instructions, AGENTS.md, CLAUDE.md, copilot instructions]
|
|
39
|
+
remediation: Add a CI check for required agent files, links, ownership, and structural invariants.
|
|
40
|
+
- id: ADRB-CTX-004
|
|
41
|
+
level: 4
|
|
42
|
+
title: Guidance effectiveness drives improvement
|
|
43
|
+
outcome: Retrieval failures, repeated instruction errors, and correction latency are trended and improve guidance.
|
|
44
|
+
risk: Documentation volume grows without evidence that agents find and apply the right context.
|
|
45
|
+
allow_attestation: true
|
|
46
|
+
allow_agent_evidence: true
|
|
47
|
+
evidence:
|
|
48
|
+
- type: manual
|
|
49
|
+
scope: outcome
|
|
50
|
+
prompt: Provide time-series guidance-quality outcomes and a recent instruction change driven by them.
|
|
51
|
+
remediation: Measure context retrieval and recurring instruction failures, then update guidance from the results.
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
dimension: environment
|
|
2
|
+
controls:
|
|
3
|
+
- id: ADRB-ENV-001
|
|
4
|
+
level: 1
|
|
5
|
+
title: Development setup is documented
|
|
6
|
+
outcome: Contributors can identify supported runtime prerequisites and the canonical setup path.
|
|
7
|
+
risk: Agents improvise dependencies or run commands in an incompatible environment.
|
|
8
|
+
evidence:
|
|
9
|
+
- type: content_terms
|
|
10
|
+
scope: repository
|
|
11
|
+
files: [README.md, CONTRIBUTING.md, docs/setup/**, docs/development/**]
|
|
12
|
+
terms: [install, setup, runtime, prerequisites, dependencies, development]
|
|
13
|
+
min_terms: 2
|
|
14
|
+
remediation: Document supported runtimes, prerequisites, installation, and the first verification command.
|
|
15
|
+
- id: ADRB-ENV-002
|
|
16
|
+
level: 2
|
|
17
|
+
title: Dependencies and toolchain are reproducibly pinned
|
|
18
|
+
outcome: The repository records deterministic dependency resolution and a supported runtime or container definition.
|
|
19
|
+
risk: Local and CI behavior diverge as dependency or runtime versions float.
|
|
20
|
+
evidence:
|
|
21
|
+
- type: path_any
|
|
22
|
+
scope: repository
|
|
23
|
+
patterns:
|
|
24
|
+
[
|
|
25
|
+
package-lock.json,
|
|
26
|
+
pnpm-lock.yaml,
|
|
27
|
+
yarn.lock,
|
|
28
|
+
uv.lock,
|
|
29
|
+
poetry.lock,
|
|
30
|
+
Pipfile.lock,
|
|
31
|
+
Cargo.lock,
|
|
32
|
+
go.sum,
|
|
33
|
+
]
|
|
34
|
+
- type: path_any
|
|
35
|
+
scope: repository
|
|
36
|
+
patterns:
|
|
37
|
+
[
|
|
38
|
+
.node-version,
|
|
39
|
+
.nvmrc,
|
|
40
|
+
.python-version,
|
|
41
|
+
mise.toml,
|
|
42
|
+
.tool-versions,
|
|
43
|
+
Dockerfile,
|
|
44
|
+
.devcontainer/**,
|
|
45
|
+
]
|
|
46
|
+
remediation: Commit a lockfile and a supported runtime, toolchain, or development-container declaration.
|
|
47
|
+
- id: ADRB-ENV-003
|
|
48
|
+
level: 3
|
|
49
|
+
title: CI recreates the canonical environment
|
|
50
|
+
outcome: Continuous integration installs from locked inputs and runs canonical verification in a clean environment.
|
|
51
|
+
risk: A local-only setup passes while clean automation fails or uses materially different dependencies.
|
|
52
|
+
evidence:
|
|
53
|
+
- type: content_terms
|
|
54
|
+
scope: repository
|
|
55
|
+
files: [.github/workflows/**, .gitlab-ci.yml, Jenkinsfile, azure-pipelines.yml]
|
|
56
|
+
terms: [npm ci, frozen-lockfile, uv sync, lockfile, checkout, test, typecheck]
|
|
57
|
+
min_terms: 2
|
|
58
|
+
remediation: Make CI install locked dependencies from a clean checkout and run the documented verification path.
|
|
59
|
+
- id: ADRB-ENV-004
|
|
60
|
+
level: 4
|
|
61
|
+
title: Environment reliability drives improvement
|
|
62
|
+
outcome: Setup failures, cache misses, drift, and reproducibility incidents are trended and reduced.
|
|
63
|
+
risk: Environment friction and nondeterminism remain anecdotal and repeatedly consume agent time.
|
|
64
|
+
allow_attestation: true
|
|
65
|
+
allow_agent_evidence: true
|
|
66
|
+
evidence:
|
|
67
|
+
- type: manual
|
|
68
|
+
scope: outcome
|
|
69
|
+
prompt: Provide environment reliability trends and a recent setup or CI improvement driven by them.
|
|
70
|
+
remediation: Track clean-setup and CI-environment outcomes, then prioritize the largest sources of drift.
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
dimension: governance
|
|
2
|
+
controls:
|
|
3
|
+
- id: ADRB-GOV-001
|
|
4
|
+
level: 1
|
|
5
|
+
title: Contribution and review expectations are documented
|
|
6
|
+
outcome: Contributors can find non-empty guidance describing how changes are proposed and reviewed.
|
|
7
|
+
risk: Agents submit changes without a shared review or accountability process.
|
|
8
|
+
evidence:
|
|
9
|
+
- type: content_terms
|
|
10
|
+
scope: repository
|
|
11
|
+
files:
|
|
12
|
+
[
|
|
13
|
+
CONTRIBUTING.md,
|
|
14
|
+
.github/pull_request_template.md,
|
|
15
|
+
.github/PULL_REQUEST_TEMPLATE/**,
|
|
16
|
+
docs/contributing/**,
|
|
17
|
+
]
|
|
18
|
+
terms: [pull request, review, approval, testing, contribution]
|
|
19
|
+
min_terms: 2
|
|
20
|
+
remediation: Publish contribution guidance covering change proposals, verification, reviewers, and approvals.
|
|
21
|
+
- id: ADRB-GOV-002
|
|
22
|
+
level: 2
|
|
23
|
+
title: Ownership and merge authority are declared
|
|
24
|
+
outcome: The tracked repository assigns review ownership and states who may approve or merge changes.
|
|
25
|
+
risk: Sensitive or cross-cutting changes reach reviewers without accountable ownership.
|
|
26
|
+
evidence:
|
|
27
|
+
- type: path_any
|
|
28
|
+
scope: repository
|
|
29
|
+
patterns: [.github/CODEOWNERS, CODEOWNERS]
|
|
30
|
+
- type: content_terms
|
|
31
|
+
scope: repository
|
|
32
|
+
files:
|
|
33
|
+
[CONTRIBUTING.md, GOVERNANCE.md, .github/pull_request_template.md, docs/governance/**]
|
|
34
|
+
terms: [owner, reviewer, approval, merge authority, required review]
|
|
35
|
+
min_terms: 2
|
|
36
|
+
remediation: Commit ownership rules and document reviewer, approval, merge, and exception authority.
|
|
37
|
+
- id: ADRB-GOV-003
|
|
38
|
+
level: 3
|
|
39
|
+
title: Protected merge governance is enforced
|
|
40
|
+
outcome: Platform rules enforce reviews, required checks, protected branches, and controlled exceptions.
|
|
41
|
+
risk: Repository-side declarations can be bypassed because platform enforcement is absent or too broad.
|
|
42
|
+
allow_attestation: true
|
|
43
|
+
allow_agent_evidence: true
|
|
44
|
+
evidence:
|
|
45
|
+
- type: manual
|
|
46
|
+
scope: platform
|
|
47
|
+
prompt: Show protected branch or ruleset coverage, required reviews and checks, merge authority, bypass actors, and exception records.
|
|
48
|
+
remediation: Enforce branch or merge rules for relevant branches and restrict, record, and review bypasses.
|
|
49
|
+
- id: ADRB-GOV-004
|
|
50
|
+
level: 4
|
|
51
|
+
title: Governance outcomes drive improvement
|
|
52
|
+
outcome: Review escapes, bypasses, rollback causes, and approval latency are trended and improve governance.
|
|
53
|
+
risk: Governance becomes ceremony without evidence that it catches risk efficiently.
|
|
54
|
+
allow_attestation: true
|
|
55
|
+
allow_agent_evidence: true
|
|
56
|
+
evidence:
|
|
57
|
+
- type: manual
|
|
58
|
+
scope: outcome
|
|
59
|
+
prompt: Provide governance outcome trends and a recent review or merge-policy improvement driven by them.
|
|
60
|
+
remediation: Trend governance outcomes by risk class and tune ownership, review depth, and exceptions.
|
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
dimension: learning
|
|
2
|
+
controls:
|
|
3
|
+
- id: ADRB-LRN-001
|
|
4
|
+
level: 1
|
|
5
|
+
title: Agent corrections have a durable home
|
|
6
|
+
outcome: Review corrections and post-merge lessons are recorded in a tracked repository artifact.
|
|
7
|
+
risk: Every agent session repeats the same repository-specific mistakes.
|
|
8
|
+
evidence:
|
|
9
|
+
- type: path_any
|
|
10
|
+
scope: repository
|
|
11
|
+
patterns: [.ai/lessons.md, docs/lessons/**, docs/decisions/**, ADR/**, docs/adr/**]
|
|
12
|
+
remediation: Create a concise tracked correction log with context, rule, owner, and date.
|
|
13
|
+
- id: ADRB-LRN-002
|
|
14
|
+
level: 2
|
|
15
|
+
title: Knowledge promotion and ownership are documented
|
|
16
|
+
outcome: Guidance defines when corrections become instructions, skills, decisions, or maintained references.
|
|
17
|
+
risk: The lesson store becomes a stale archive rather than usable agent knowledge.
|
|
18
|
+
evidence:
|
|
19
|
+
- type: content_terms
|
|
20
|
+
scope: repository
|
|
21
|
+
files: [AGENTS.md, .ai/lessons.md, docs/knowledge/**, .agents/skills/**, .claude/skills/**]
|
|
22
|
+
terms: [curation, promote, lesson, correction, stale, owner, deduplication]
|
|
23
|
+
min_terms: 3
|
|
24
|
+
remediation: Define curation cadence, promotion rules, ownership, deduplication, and stale-content handling.
|
|
25
|
+
- id: ADRB-LRN-003
|
|
26
|
+
level: 3
|
|
27
|
+
title: Knowledge curation is mechanically enforced
|
|
28
|
+
outcome: Automation detects stale, duplicate, contradictory, or unpromoted agent knowledge.
|
|
29
|
+
risk: Knowledge quality relies on memory and quietly deteriorates as the repository evolves.
|
|
30
|
+
evidence:
|
|
31
|
+
- type: content_terms
|
|
32
|
+
scope: repository
|
|
33
|
+
files: [.github/workflows/**, scripts/**, package.json, Makefile]
|
|
34
|
+
terms: [curator, curation, knowledge lint, stale, contradiction, duplicate]
|
|
35
|
+
min_terms: 2
|
|
36
|
+
remediation: Automate mechanical checks and require periodic judgement over knowledge quality.
|
|
37
|
+
- id: ADRB-LRN-004
|
|
38
|
+
level: 4
|
|
39
|
+
title: Knowledge effectiveness drives improvement
|
|
40
|
+
outcome: Recurring mistakes, stale guidance, retrieval success, and correction latency are trended and improve curation.
|
|
41
|
+
risk: Knowledge grows without evidence that it changes agent behavior.
|
|
42
|
+
allow_attestation: true
|
|
43
|
+
allow_agent_evidence: true
|
|
44
|
+
evidence:
|
|
45
|
+
- type: manual
|
|
46
|
+
scope: outcome
|
|
47
|
+
prompt: Provide recurring-error and knowledge-quality trends plus a recent curation improvement driven by them.
|
|
48
|
+
remediation: Measure whether promoted knowledge prevents repeated mistakes and prioritize weak guidance.
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
dimension: observability
|
|
2
|
+
controls:
|
|
3
|
+
- id: ADRB-OBS-001
|
|
4
|
+
level: 1
|
|
5
|
+
title: Agent run records are defined
|
|
6
|
+
outcome: A tracked schema or guide defines how material agent activity can be reconstructed.
|
|
7
|
+
risk: Teams cannot explain what happened after a poor change or security concern.
|
|
8
|
+
evidence:
|
|
9
|
+
- type: content_terms
|
|
10
|
+
scope: repository
|
|
11
|
+
files:
|
|
12
|
+
[
|
|
13
|
+
docs/observability/**,
|
|
14
|
+
docs/agentic/**,
|
|
15
|
+
evals/**,
|
|
16
|
+
evaluations/**,
|
|
17
|
+
observability/**,
|
|
18
|
+
.agentic/run-envelope.*,
|
|
19
|
+
]
|
|
20
|
+
terms: [task, actor, harness, result, timestamp, evidence]
|
|
21
|
+
min_terms: 3
|
|
22
|
+
remediation: Define a privacy-safe run record containing task, harness, result, timing, and evidence references.
|
|
23
|
+
- id: ADRB-OBS-002
|
|
24
|
+
level: 2
|
|
25
|
+
title: Run envelope and outcome taxonomy are documented
|
|
26
|
+
outcome: Run guidance covers authority, tools, budget, stop reason, verification, and outcome classification.
|
|
27
|
+
risk: Comparisons confound model, tooling, permissions, task risk, and checkout state.
|
|
28
|
+
evidence:
|
|
29
|
+
- type: content_terms
|
|
30
|
+
scope: repository
|
|
31
|
+
files:
|
|
32
|
+
[
|
|
33
|
+
docs/observability/**,
|
|
34
|
+
docs/agentic/**,
|
|
35
|
+
evals/**,
|
|
36
|
+
evaluations/**,
|
|
37
|
+
observability/**,
|
|
38
|
+
.agentic/run-envelope.*,
|
|
39
|
+
]
|
|
40
|
+
terms:
|
|
41
|
+
[
|
|
42
|
+
task class,
|
|
43
|
+
tool,
|
|
44
|
+
permission,
|
|
45
|
+
budget,
|
|
46
|
+
stop reason,
|
|
47
|
+
outcome,
|
|
48
|
+
harness version,
|
|
49
|
+
verification,
|
|
50
|
+
]
|
|
51
|
+
min_terms: 4
|
|
52
|
+
remediation: Adopt a common run envelope and outcome taxonomy with privacy and retention rules.
|
|
53
|
+
- id: ADRB-OBS-003
|
|
54
|
+
level: 3
|
|
55
|
+
title: Outcome evidence is captured consistently
|
|
56
|
+
outcome: Material agent runs automatically emit retained metadata and verification outcomes.
|
|
57
|
+
risk: Only memorable successes and failures shape policy, creating selection bias.
|
|
58
|
+
allow_attestation: true
|
|
59
|
+
allow_agent_evidence: true
|
|
60
|
+
evidence:
|
|
61
|
+
- type: manual
|
|
62
|
+
scope: platform
|
|
63
|
+
prompt: Show automated run-envelope capture, completeness monitoring, retention controls, and access controls.
|
|
64
|
+
remediation: Emit structured run and outcome records automatically while minimizing sensitive content.
|
|
65
|
+
- id: ADRB-OBS-004
|
|
66
|
+
level: 4
|
|
67
|
+
title: Agent outcomes drive autonomy decisions
|
|
68
|
+
outcome: Quality, escape rate, cycle time, cost, intervention, and rollback trends change autonomy or harness policy.
|
|
69
|
+
risk: Token use or completion rate becomes a misleading proxy for engineering value and safety.
|
|
70
|
+
allow_attestation: true
|
|
71
|
+
allow_agent_evidence: true
|
|
72
|
+
evidence:
|
|
73
|
+
- type: manual
|
|
74
|
+
scope: outcome
|
|
75
|
+
prompt: Provide outcome trends segmented by task and risk class and an autonomy or harness decision they changed.
|
|
76
|
+
remediation: Evaluate quality and total effort by task class and use results to narrow or expand autonomy.
|
|
@@ -0,0 +1,60 @@
|
|
|
1
|
+
dimension: resilience
|
|
2
|
+
controls:
|
|
3
|
+
- id: ADRB-RES-001
|
|
4
|
+
level: 1
|
|
5
|
+
title: Changes have a documented recovery path
|
|
6
|
+
outcome: Contributors can locate actionable rollback, revert, recovery, or incident guidance.
|
|
7
|
+
risk: A harmful agent change persists because recovery is improvised under pressure.
|
|
8
|
+
evidence:
|
|
9
|
+
- type: content_terms
|
|
10
|
+
scope: repository
|
|
11
|
+
files:
|
|
12
|
+
[
|
|
13
|
+
README.md,
|
|
14
|
+
CONTRIBUTING.md,
|
|
15
|
+
docs/recovery/**,
|
|
16
|
+
docs/operations/**,
|
|
17
|
+
runbooks/**,
|
|
18
|
+
.github/**,
|
|
19
|
+
]
|
|
20
|
+
terms: [rollback, revert, recovery, incident, restore]
|
|
21
|
+
min_terms: 2
|
|
22
|
+
remediation: Document how to stop work, revert code, recover affected state, and escalate an incident.
|
|
23
|
+
- id: ADRB-RES-002
|
|
24
|
+
level: 2
|
|
25
|
+
title: Failure containment and stop conditions are documented
|
|
26
|
+
outcome: Agent guidance defines mutation limits, budgets, stop triggers, escalation, and recovery ownership.
|
|
27
|
+
risk: A looping or confused agent continues spending or mutating beyond the intended scope.
|
|
28
|
+
evidence:
|
|
29
|
+
- type: content_terms
|
|
30
|
+
scope: repository
|
|
31
|
+
files:
|
|
32
|
+
[AGENTS.md, SECURITY.md, docs/agentic/**, docs/recovery/**, runbooks/**, templates/**]
|
|
33
|
+
terms:
|
|
34
|
+
[stop condition, rollback, isolation, budget, kill switch, escalation, recovery owner]
|
|
35
|
+
min_terms: 3
|
|
36
|
+
remediation: Add bounded scope, budget, retry, stop, escalation, and recovery requirements to task preflight.
|
|
37
|
+
- id: ADRB-RES-003
|
|
38
|
+
level: 3
|
|
39
|
+
title: Containment and recovery controls are exercised
|
|
40
|
+
outcome: Cancellation, credential revocation, rollback, timeout, and partial-failure recovery are tested.
|
|
41
|
+
risk: Recovery mechanisms exist on paper but fail when the harness behaves unexpectedly.
|
|
42
|
+
allow_attestation: true
|
|
43
|
+
allow_agent_evidence: true
|
|
44
|
+
evidence:
|
|
45
|
+
- type: manual
|
|
46
|
+
scope: organization
|
|
47
|
+
prompt: Show recent automated tests or exercises for stop, timeout, revocation, rollback, and partial failure.
|
|
48
|
+
remediation: Add failure-injection tests and periodic exercises for containment and reversible recovery.
|
|
49
|
+
- id: ADRB-RES-004
|
|
50
|
+
level: 4
|
|
51
|
+
title: Recovery outcomes drive improvement
|
|
52
|
+
outcome: Time-to-detect, time-to-stop, rollback success, impact, and recurrence trends improve containment.
|
|
53
|
+
risk: The same failure patterns repeat because recovery performance and residual impact are unknown.
|
|
54
|
+
allow_attestation: true
|
|
55
|
+
allow_agent_evidence: true
|
|
56
|
+
evidence:
|
|
57
|
+
- type: manual
|
|
58
|
+
scope: outcome
|
|
59
|
+
prompt: Provide recovery outcome trends and a recent containment or rollback improvement driven by them.
|
|
60
|
+
remediation: Trend containment and recovery outcomes and feed them into autonomy and tool design.
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
dimension: security
|
|
2
|
+
controls:
|
|
3
|
+
- id: ADRB-SEC-001
|
|
4
|
+
level: 1
|
|
5
|
+
title: Security reporting and secret handling are documented
|
|
6
|
+
outcome: Contributors can find security contacts plus rules for credentials and sensitive data.
|
|
7
|
+
risk: Agents expose secrets or sensitive data and responders lack a defined reporting path.
|
|
8
|
+
evidence:
|
|
9
|
+
- type: content_terms
|
|
10
|
+
scope: repository
|
|
11
|
+
files: [SECURITY.md, .github/SECURITY.md, docs/security/**]
|
|
12
|
+
terms: [report, vulnerability, secret, credential, sensitive]
|
|
13
|
+
min_terms: 2
|
|
14
|
+
remediation: Publish a security policy covering reporting, secrets, sensitive data, and agent-specific risks.
|
|
15
|
+
- id: ADRB-SEC-002
|
|
16
|
+
level: 2
|
|
17
|
+
title: Agent data and autonomy boundaries are explicit
|
|
18
|
+
outcome: Agent preflight guidance binds permitted data, tools, mutations, approvals, and stop conditions.
|
|
19
|
+
risk: Operators grant more data or authority than the task requires.
|
|
20
|
+
evidence:
|
|
21
|
+
- type: content_terms
|
|
22
|
+
scope: repository
|
|
23
|
+
files: [SECURITY.md, AGENTS.md, docs/security/**, docs/agentic/**, templates/**]
|
|
24
|
+
terms:
|
|
25
|
+
[
|
|
26
|
+
data classification,
|
|
27
|
+
allowed tools,
|
|
28
|
+
mutation,
|
|
29
|
+
budget,
|
|
30
|
+
human approval,
|
|
31
|
+
autonomy,
|
|
32
|
+
stop condition,
|
|
33
|
+
]
|
|
34
|
+
min_terms: 3
|
|
35
|
+
remediation: Add a task preflight that binds data, tools, write scope, budget, approvals, and stop conditions.
|
|
36
|
+
- id: ADRB-SEC-003
|
|
37
|
+
level: 3
|
|
38
|
+
title: Automated agent security safeguards are configured
|
|
39
|
+
outcome: Repository automation checks secrets and defines handling for untrusted agent input.
|
|
40
|
+
risk: Credential exposure or prompt-borne instructions can pass through an otherwise healthy pipeline.
|
|
41
|
+
evidence:
|
|
42
|
+
- type: content_terms
|
|
43
|
+
scope: repository
|
|
44
|
+
files: [.github/workflows/**, .gitlab-ci.yml, scripts/**, SECURITY.md, docs/security/**]
|
|
45
|
+
terms: [secret scan, gitleaks, trufflehog, prompt injection, untrusted content, sandbox]
|
|
46
|
+
min_terms: 2
|
|
47
|
+
remediation: Configure secret scanning and explicit untrusted-content or prompt-injection safeguards with tests.
|
|
48
|
+
- id: ADRB-SEC-005
|
|
49
|
+
level: 3
|
|
50
|
+
title: High-impact agent actions are technically gated
|
|
51
|
+
outcome: External authority controls prevent unapproved high-impact actions and record controlled exceptions.
|
|
52
|
+
risk: Repository instructions alone cannot prevent credential misuse or irreversible external actions.
|
|
53
|
+
allow_attestation: true
|
|
54
|
+
allow_agent_evidence: true
|
|
55
|
+
evidence:
|
|
56
|
+
- type: manual
|
|
57
|
+
scope: platform
|
|
58
|
+
prompt: Show authorization gates, scoped credentials, approval enforcement, audit records, and controlled bypasses for high-impact agent actions.
|
|
59
|
+
remediation: Enforce least-privileged credentials, approval gates, auditability, and restricted exception handling.
|
|
60
|
+
- id: ADRB-SEC-004
|
|
61
|
+
level: 4
|
|
62
|
+
title: Agent security outcomes drive improvement
|
|
63
|
+
outcome: Denials, near misses, incidents, exceptions, and adversarial exercises are trended and improve controls.
|
|
64
|
+
risk: Controls appear complete while effectiveness against real failure modes remains unknown.
|
|
65
|
+
allow_attestation: true
|
|
66
|
+
allow_agent_evidence: true
|
|
67
|
+
evidence:
|
|
68
|
+
- type: manual
|
|
69
|
+
scope: outcome
|
|
70
|
+
prompt: Provide agent-security outcome trends or exercise results and a recent control change driven by them.
|
|
71
|
+
remediation: Run adversarial exercises, trend security outcomes, and update boundaries from the results.
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
dimension: specification
|
|
2
|
+
controls:
|
|
3
|
+
- id: ADRB-SPC-001
|
|
4
|
+
level: 1
|
|
5
|
+
title: Work has a durable objective
|
|
6
|
+
outcome: Engineering work starts from a tracked issue, request, or specification artifact.
|
|
7
|
+
risk: Agents optimize an inferred problem and completion cannot be judged consistently.
|
|
8
|
+
evidence:
|
|
9
|
+
- type: path_any
|
|
10
|
+
scope: repository
|
|
11
|
+
patterns:
|
|
12
|
+
[.github/ISSUE_TEMPLATE/**, docs/requirements/**, specs/**, .ai/specs/**, .kiro/specs/**]
|
|
13
|
+
remediation: Adopt a tracked work-request or specification template with objective, owner, and outcome.
|
|
14
|
+
- id: ADRB-SPC-002
|
|
15
|
+
level: 2
|
|
16
|
+
title: Acceptance criteria and constraints are documented
|
|
17
|
+
outcome: Work artifacts state testable acceptance criteria plus material constraints or exclusions.
|
|
18
|
+
risk: An implementation can appear plausible without satisfying a shared definition of done.
|
|
19
|
+
evidence:
|
|
20
|
+
- type: content_terms
|
|
21
|
+
scope: repository
|
|
22
|
+
files:
|
|
23
|
+
[.github/ISSUE_TEMPLATE/**, specs/**, .ai/specs/**, .kiro/specs/**, docs/requirements/**]
|
|
24
|
+
terms: [acceptance criteria, requirements, constraints, non-goals, verification]
|
|
25
|
+
min_terms: 2
|
|
26
|
+
remediation: Add testable outcomes, constraints, exclusions, and verification expectations to work artifacts.
|
|
27
|
+
- id: ADRB-SPC-003
|
|
28
|
+
level: 3
|
|
29
|
+
title: Specification traceability is mechanically checked
|
|
30
|
+
outcome: Automation checks that applicable changes and verification trace to an approved work artifact.
|
|
31
|
+
risk: Specifications become optional prose disconnected from implementation and tests.
|
|
32
|
+
evidence:
|
|
33
|
+
- type: content_terms
|
|
34
|
+
scope: repository
|
|
35
|
+
files: [.github/workflows/**, scripts/**, package.json, Makefile]
|
|
36
|
+
terms: [acceptance criteria, traceability, requirements coverage, spec check, task id]
|
|
37
|
+
min_terms: 2
|
|
38
|
+
remediation: Add a deterministic traceability check from change to work artifact and verification record.
|
|
39
|
+
- id: ADRB-SPC-005
|
|
40
|
+
level: 3
|
|
41
|
+
title: Specification traceability is required before merge
|
|
42
|
+
outcome: Repository merge policy requires the applicable traceability check or equivalent review evidence.
|
|
43
|
+
risk: A configured check can be skipped or bypassed when delivery pressure rises.
|
|
44
|
+
allow_attestation: true
|
|
45
|
+
allow_agent_evidence: true
|
|
46
|
+
evidence:
|
|
47
|
+
- type: manual
|
|
48
|
+
scope: platform
|
|
49
|
+
prompt: Show that the traceability check or equivalent review requirement is an enforced merge gate with controlled exceptions.
|
|
50
|
+
remediation: Require the traceability check in merge policy and restrict, record, and review bypasses.
|
|
51
|
+
- id: ADRB-SPC-004
|
|
52
|
+
level: 4
|
|
53
|
+
title: Specification effectiveness drives improvement
|
|
54
|
+
outcome: Rework and escaped ambiguity attributable to specifications are trended and improve the process.
|
|
55
|
+
risk: Specification ceremony grows without reducing misunderstanding or rework.
|
|
56
|
+
allow_attestation: true
|
|
57
|
+
allow_agent_evidence: true
|
|
58
|
+
evidence:
|
|
59
|
+
- type: manual
|
|
60
|
+
scope: outcome
|
|
61
|
+
prompt: Provide specification-related rework or ambiguity trends and a recent process change driven by them.
|
|
62
|
+
remediation: Classify rework causes and evolve specification practices using outcome evidence.
|