ai-capability-framework 1.0.0-rc.3 → 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (217) hide show
  1. package/CHANGELOG.md +97 -103
  2. package/CODE_OF_CONDUCT.md +11 -13
  3. package/CONTRIBUTING.md +40 -43
  4. package/GOVERNANCE.md +34 -35
  5. package/README.md +99 -63
  6. package/ROADMAP.md +12 -12
  7. package/SECURITY.md +15 -15
  8. package/dist/audit/ledger.d.ts.map +1 -1
  9. package/dist/audit/ledger.js +2 -1
  10. package/dist/audit/records.d.ts.map +1 -1
  11. package/dist/audit/records.js +2 -1
  12. package/dist/audit/types.d.ts +4 -1
  13. package/dist/audit/types.d.ts.map +1 -1
  14. package/dist/aws/dynamodb-stores.d.ts +2 -7
  15. package/dist/aws/dynamodb-stores.d.ts.map +1 -1
  16. package/dist/aws/dynamodb-stores.js +56 -5
  17. package/dist/aws/testing.d.ts.map +1 -1
  18. package/dist/aws/testing.js +9 -0
  19. package/dist/cli.d.ts +6 -6
  20. package/dist/cli.d.ts.map +1 -1
  21. package/dist/cli.js +77 -7
  22. package/dist/evals-live/index.d.ts +2 -2
  23. package/dist/evals-live/index.d.ts.map +1 -1
  24. package/dist/evals-live/index.js +1 -1
  25. package/dist/evals-live/live-eval-runner.d.ts +49 -1
  26. package/dist/evals-live/live-eval-runner.d.ts.map +1 -1
  27. package/dist/evals-live/live-eval-runner.js +246 -4
  28. package/dist/evals-live/types.d.ts +62 -9
  29. package/dist/evals-live/types.d.ts.map +1 -1
  30. package/dist/index.d.ts +0 -1
  31. package/dist/index.d.ts.map +1 -1
  32. package/dist/index.js +0 -1
  33. package/dist/runtime/actions.d.ts +2 -0
  34. package/dist/runtime/actions.d.ts.map +1 -1
  35. package/dist/runtime/actions.js +200 -55
  36. package/dist/runtime/index.d.ts +1 -1
  37. package/dist/runtime/index.d.ts.map +1 -1
  38. package/dist/runtime/stores.d.ts +2 -7
  39. package/dist/runtime/stores.d.ts.map +1 -1
  40. package/dist/runtime/stores.js +12 -0
  41. package/dist/runtime/types.d.ts +25 -6
  42. package/dist/runtime/types.d.ts.map +1 -1
  43. package/docs/action-lifecycle.md +38 -32
  44. package/docs/adapter-roadmap.md +51 -49
  45. package/docs/adapters.md +14 -15
  46. package/docs/ai-sdk-runtime.md +36 -39
  47. package/docs/anthropic-runtime.md +29 -32
  48. package/docs/api/public-api-policy.md +59 -0
  49. package/docs/api.md +354 -371
  50. package/docs/architecture/current-state.md +28 -31
  51. package/docs/audit/index.md +29 -29
  52. package/docs/aws/cloudwatch-telemetry.md +5 -7
  53. package/docs/aws/dynamodb-single-table.md +8 -9
  54. package/docs/aws/kms-redaction.md +5 -6
  55. package/docs/aws/overview.md +5 -5
  56. package/docs/aws/production-reference.md +13 -14
  57. package/docs/aws/step-functions-approval.md +7 -9
  58. package/docs/aws-runtime.md +24 -25
  59. package/docs/capability-manifests.md +17 -21
  60. package/docs/control-plane/overview.md +2 -2
  61. package/docs/control-plane.md +24 -27
  62. package/docs/controls/index.md +19 -22
  63. package/docs/core/capability-manifests.md +4 -5
  64. package/docs/core/entity-manifests.md +5 -5
  65. package/docs/eval-runner.md +29 -31
  66. package/docs/evalops.md +13 -14
  67. package/docs/evals/golden-tests.md +4 -5
  68. package/docs/evals/overview.md +3 -3
  69. package/docs/evals/replay-and-trace-to-golden.md +19 -20
  70. package/docs/evals.md +19 -20
  71. package/docs/evidence.md +22 -23
  72. package/docs/gemini-runtime.md +31 -33
  73. package/docs/getting-started/anthropic-quickstart.md +61 -0
  74. package/docs/getting-started/concepts.md +9 -10
  75. package/docs/getting-started/gemini-quickstart.md +62 -0
  76. package/docs/getting-started/installation.md +3 -3
  77. package/docs/getting-started/openai-quickstart.md +61 -0
  78. package/docs/getting-started/provider-neutral-quickstart.md +106 -0
  79. package/docs/getting-started/quickstart.md +2 -2
  80. package/docs/getting-started.md +23 -20
  81. package/docs/glossary.md +31 -48
  82. package/docs/governance/gate.md +28 -30
  83. package/docs/governance/impact-and-compatibility.md +2 -3
  84. package/docs/governance/index.md +53 -54
  85. package/docs/governance/lifecycle.md +4 -5
  86. package/docs/governance/overview.md +2 -3
  87. package/docs/governance/risk-compiler.md +4 -5
  88. package/docs/host-responsibilities.md +6 -8
  89. package/docs/implementation-notes/baseline-gaps.md +15 -18
  90. package/docs/index.md +21 -6
  91. package/docs/interoperability.md +9 -9
  92. package/docs/langchain-runtime.md +27 -31
  93. package/docs/live-evals.md +68 -32
  94. package/docs/mcp-server-runtime.md +36 -40
  95. package/docs/memory.md +35 -36
  96. package/docs/migration-0.1-to-1.0.md +14 -9
  97. package/docs/observability/overview.md +2 -3
  98. package/docs/observability-runtime.md +35 -39
  99. package/docs/openai-responses.md +36 -37
  100. package/docs/openai-runtime.md +41 -46
  101. package/docs/openai-walkthrough.md +24 -28
  102. package/docs/policy-broker.md +9 -10
  103. package/docs/provenance.md +30 -29
  104. package/docs/providers/anthropic.md +5 -5
  105. package/docs/providers/choose-a-runtime.md +37 -0
  106. package/docs/providers/conformance.md +14 -15
  107. package/docs/providers/gemini.md +5 -5
  108. package/docs/providers/mcp.md +2 -3
  109. package/docs/providers/openai.md +2 -2
  110. package/docs/providers/semantic-kernel.md +2 -2
  111. package/docs/providers.md +43 -38
  112. package/docs/public/license-exceptions.md +187 -0
  113. package/docs/public/npm-release-preflight.md +112 -0
  114. package/docs/public-framework/compatibility-policy.md +12 -13
  115. package/docs/public-framework/deprecation-policy.md +8 -8
  116. package/docs/public-framework/final-certification-matrix.md +104 -0
  117. package/docs/public-framework/license-decision.md +20 -0
  118. package/docs/public-framework/release-process.md +46 -16
  119. package/docs/public-framework/security-disclosure.md +4 -5
  120. package/docs/public-framework/v1-certification.md +102 -15
  121. package/docs/release.md +200 -122
  122. package/docs/runtime/action-lifecycle.md +3 -3
  123. package/docs/runtime/policy-broker.md +4 -4
  124. package/docs/runtime/runtime-overview.md +4 -5
  125. package/docs/runtime/tool-result-envelope.md +7 -7
  126. package/docs/runtime.md +90 -93
  127. package/docs/security/overview.md +4 -4
  128. package/docs/security/security-packs.md +24 -26
  129. package/docs/security/trust-taint-redaction.md +42 -47
  130. package/docs/semantic-kernel-runtime.md +35 -37
  131. package/docs/spec.md +95 -101
  132. package/docs/start-here.md +18 -20
  133. package/examples/01-basic-read-capability/README.md +3 -4
  134. package/examples/02-prepare-approve-commit/README.md +6 -6
  135. package/examples/03-multi-provider-tools/README.md +2 -2
  136. package/examples/04-mcp-server/README.md +4 -5
  137. package/examples/05-langchain-langgraph-bridge/README.md +4 -5
  138. package/examples/06-vercel-ai-sdk-bridge/README.md +5 -7
  139. package/examples/07-policy-broker-custom-auth/README.md +4 -4
  140. package/examples/08-aws-step-functions-approval/README.md +4 -5
  141. package/examples/09-security-packs-promptfoo/README.md +4 -4
  142. package/examples/10-trace-to-golden/README.md +2 -2
  143. package/examples/11-control-plane/README.md +4 -5
  144. package/examples/aws/README.md +5 -8
  145. package/examples/control-plane/README.md +8 -9
  146. package/examples/control-plane/fixtures/control-plane.seed.json +8 -32
  147. package/examples/control-plane/public/app.js +57 -18
  148. package/examples/control-plane/public/index.html +15 -6
  149. package/examples/control-plane/server.mjs +15 -3
  150. package/examples/eval-results/public.results.passing.json +6 -21
  151. package/examples/providers/ai-sdk-next/README.md +11 -8
  152. package/examples/providers/anthropic-claude/README.md +5 -6
  153. package/examples/providers/gemini/README.md +6 -6
  154. package/examples/providers/langchain-agent/README.md +3 -4
  155. package/examples/providers/langgraph-tool-node/README.md +7 -7
  156. package/examples/providers/mcp/README.md +8 -8
  157. package/examples/providers/provider-conformance/README.md +3 -3
  158. package/examples/providers/semantic-kernel-mcp/README.md +6 -6
  159. package/examples/providers/semantic-kernel-openapi/README.md +6 -7
  160. package/examples/runtime-support-billing/README.md +8 -9
  161. package/examples/runtime-support-billing/support-billing-runtime.mjs +8 -4
  162. package/examples/scheduling/capabilities/scheduling.availability.get.yaml +2 -2
  163. package/examples/scheduling/capabilities/scheduling.invite.prepare.yaml +2 -2
  164. package/examples/scheduling/capabilities/scheduling.invite.send.yaml +2 -2
  165. package/examples/scheduling/entities/AvailabilitySlot.yaml +2 -2
  166. package/examples/scheduling/entities/MeetingInvite.yaml +2 -2
  167. package/examples/scheduling/evals/scheduling.invite.prepare.valid.yaml +3 -1
  168. package/examples/semantic-kernel-mcp/README.md +11 -12
  169. package/examples/semantic-kernel-openapi/README.md +8 -8
  170. package/examples/support/capabilities/support.refund.commit_case.yaml +3 -3
  171. package/examples/support/capabilities/support.refund.prepare_case.yaml +2 -2
  172. package/examples/support/capabilities/support.ticket.get.yaml +2 -2
  173. package/examples/support/entities/Ticket.yaml +0 -1
  174. package/examples/support/eval-results/support.results.passing.json +3 -12
  175. package/examples/support/evals/support.refund.prepare_case.approval_required.yaml +3 -1
  176. package/examples/support/evals/support.refund.prepare_case.valid.yaml +3 -2
  177. package/examples/support/memory/support.agent.preferences.json +2 -6
  178. package/examples/support/openai/context.support_agent.json +1 -4
  179. package/examples/support/provenance/support.refund.summary.provenance.json +2 -6
  180. package/examples/support/replay/support.refund.approval_required.trace.json +4 -14
  181. package/package.json +22 -5
  182. package/schemas/audit/action-record.schema.json +4 -1
  183. package/schemas/aws/approval-task.schema.json +8 -1
  184. package/schemas/conformance/conformance-case.schema.json +18 -2
  185. package/schemas/conformance/conformance-report.schema.json +17 -2
  186. package/schemas/conformance/provider-target-matrix.schema.json +35 -4
  187. package/schemas/controls/control-decision.schema.json +7 -1
  188. package/schemas/eval-case.schema.json +8 -1
  189. package/schemas/eval-result.schema.json +8 -1
  190. package/schemas/evidence/evidence-pack.schema.json +34 -5
  191. package/schemas/governance/compatibility-diff.schema.json +8 -1
  192. package/schemas/governance/lifecycle-transition.schema.json +10 -1
  193. package/schemas/governance/risk-compilation-result.schema.json +9 -1
  194. package/schemas/provenance/generated-content-provenance.schema.json +3 -1
  195. package/schemas/replay/replay-result.schema.json +16 -2
  196. package/schemas/replay/replay-trace.schema.json +11 -2
  197. package/schemas/security/context-segment.schema.json +21 -12
  198. package/schemas/security/redaction-policy.schema.json +4 -1
  199. package/schemas/security-packs/coverage-report.schema.json +17 -2
  200. package/schemas/security-packs/security-case-suite.schema.json +11 -1
  201. package/schemas/tool-result-envelope.schema.json +9 -1
  202. package/security-packs/approval_bypass.yaml +12 -2
  203. package/security-packs/capability_spoofing.yaml +17 -3
  204. package/security-packs/cost_amplification.yaml +21 -4
  205. package/security-packs/cross_tenant_access.yaml +16 -3
  206. package/security-packs/insecure_output_rendering.yaml +15 -4
  207. package/security-packs/mcp_tool_abuse.yaml +17 -3
  208. package/security-packs/memory_scope_violation.yaml +19 -4
  209. package/security-packs/prompt_injection_direct.yaml +19 -4
  210. package/security-packs/prompt_injection_indirect.yaml +18 -3
  211. package/security-packs/provider_payload_exposure.yaml +26 -4
  212. package/security-packs/retrieval_poisoning.yaml +5 -2
  213. package/security-packs/schema_confusion.yaml +17 -3
  214. package/security-packs/sensitive_data_disclosure.yaml +22 -5
  215. package/security-packs/tool_exfiltration.yaml +20 -4
  216. package/security-packs/tool_result_poisoning.yaml +19 -4
  217. package/security-packs/unsafe_commit_attempt.yaml +6 -3
@@ -1,23 +1,22 @@
1
1
  # Runtime Controls
2
2
 
3
- AICF controls let a host application narrow or stop capability use without
4
- changing manifests. They are optional runtime inputs, not a production control
5
- plane or model gateway.
3
+ AICF controls let a host application narrow or stop capability use without changing
4
+ manifests. They are optional runtime inputs, not a production control plane or model
5
+ gateway.
6
6
 
7
- Controls can apply to a global runtime, provider, model, capability, domain,
8
- risk tier, tenant, or autonomy tier. AICF evaluates them before capability
9
- routing, tool export, model tool execution, lifecycle commit, and supported
10
- provider runtime calls.
7
+ Controls can apply to a global runtime, provider, model, capability, domain, risk tier,
8
+ tenant, or autonomy tier. AICF evaluates them before capability routing, tool export,
9
+ model tool execution, lifecycle commit, and supported provider runtime calls.
11
10
 
12
11
  ## Kill Switches
13
12
 
14
13
  Kill switches are the fastest control:
15
14
 
16
15
  - `deny` blocks matching capability export and execution.
17
- - `force_approval` keeps read and prepare paths available, but forces host
18
- approval before lifecycle commit.
19
- - `read_only` keeps read/select paths available and blocks prepare, commit,
20
- write, send, money-movement, and destructive capabilities.
16
+ - `force_approval` keeps read and prepare paths available, but forces host approval
17
+ before lifecycle commit.
18
+ - `read_only` keeps read/select paths available and blocks prepare, commit, write, send,
19
+ money-movement, and destructive capabilities.
21
20
 
22
21
  Example:
23
22
 
@@ -33,11 +32,10 @@ Example:
33
32
 
34
33
  ## Circuit Breakers
35
34
 
36
- Circuit breakers evaluate recent runtime signals such as provider error rate,
37
- validation failure rate, approval rejection rate, tool-loop limit hits, and
38
- budget failures. F4 includes deterministic evaluation and in-memory/reference
39
- state only. Production metrics ingestion and durable storage remain host
40
- responsibilities.
35
+ Circuit breakers evaluate recent runtime signals such as provider error rate, validation
36
+ failure rate, approval rejection rate, tool-loop limit hits, and budget failures. F4
37
+ includes deterministic evaluation and in-memory/reference state only. Production metrics
38
+ ingestion and durable storage remain host responsibilities.
41
39
 
42
40
  ## Budgets
43
41
 
@@ -48,13 +46,12 @@ Budgets limit one run. Defaults apply when controls are configured:
48
46
  - 60 seconds runtime per run
49
47
  - 2 retries per run
50
48
 
51
- Token and estimated-cost budgets are optional fields. Hard budgets fail closed;
52
- warn budgets return warnings without blocking.
49
+ Token and estimated-cost budgets are optional fields. Hard budgets fail closed; warn
50
+ budgets return warnings without blocking.
53
51
 
54
52
  ## CLI
55
53
 
56
- Local CLI state uses `.aicf/controls.json`, which is ignored and must not be
57
- tracked.
54
+ Local CLI state uses `.aicf/controls.json`, which is ignored and must not be tracked.
58
55
 
59
56
  ```bash
60
57
  aicf controls list --format json
@@ -62,5 +59,5 @@ aicf controls check examples --capability support.refund.prepare_case --provider
62
59
  aicf controls kill-switch create --capability support.refund.prepare_case --mode force_approval --reason "incident review"
63
60
  ```
64
61
 
65
- The CLI is useful for local review and demos. Host applications should provide
66
- their own authenticated control store for production.
62
+ The CLI is useful for local review and demos. Host applications should provide their own
63
+ authenticated control store for production.
@@ -1,8 +1,7 @@
1
1
  # Capability Manifests
2
2
 
3
- Capability manifests are the core AICF contract. They describe the operation,
4
- input and output schemas, lifecycle, permissions, risk, policy, and eval
5
- coverage.
3
+ Capability manifests are the core AICF contract. They describe the operation, input and
4
+ output schemas, lifecycle, permissions, risk, policy, and eval coverage.
6
5
 
7
6
  Use the existing guide for the full field reference:
8
7
 
@@ -16,5 +15,5 @@ node dist/cli.js inspect examples
16
15
  node dist/cli.js validate examples
17
16
  ```
18
17
 
19
- The examples are synthetic and use fake support tickets, orders, and scheduling
20
- data only.
18
+ The examples are synthetic and use fake support tickets, orders, and scheduling data
19
+ only.
@@ -1,8 +1,8 @@
1
1
  # Entity Manifests
2
2
 
3
- Entity manifests describe application objects that capabilities can read,
4
- prepare, or commit against. They help AICF explain impact, governance coverage,
5
- and allowed actions without exposing raw databases or broad application APIs.
3
+ Entity manifests describe application objects that capabilities can read, prepare, or
4
+ commit against. They help AICF explain impact, governance coverage, and allowed actions
5
+ without exposing raw databases or broad application APIs.
6
6
 
7
7
  Public examples:
8
8
 
@@ -17,5 +17,5 @@ Validate them with:
17
17
  node dist/cli.js validate examples
18
18
  ```
19
19
 
20
- Entity docs stay intentionally small. Capability manifests own the callable
21
- schemas and lifecycle rules.
20
+ Entity docs stay intentionally small. Capability manifests own the callable schemas and
21
+ lifecycle rules.
@@ -1,16 +1,16 @@
1
1
  # Eval Runner
2
2
 
3
- The deterministic eval runner scores AICF eval manifests against candidate
4
- result fixtures without calling models, executing capabilities, storing traces,
5
- or reading raw provider payloads.
3
+ The deterministic eval runner scores AICF eval manifests against candidate result
4
+ fixtures without calling models, executing capabilities, storing traces, or reading raw
5
+ provider payloads.
6
6
 
7
- Host applications are responsible for producing candidate results. AICF only
8
- validates and scores the summarized behavior.
7
+ Host applications are responsible for producing candidate results. AICF only validates
8
+ and scores the summarized behavior.
9
9
 
10
10
  ## Mental Model
11
11
 
12
- An eval manifest says what should happen. A candidate result says what did
13
- happen in a mock run, live run, or host test. The eval runner compares the two.
12
+ An eval manifest says what should happen. A candidate result says what did happen in a
13
+ mock run, live run, or host test. The eval runner compares the two.
14
14
 
15
15
  Eval manifest excerpt:
16
16
 
@@ -43,8 +43,8 @@ Candidate result excerpt:
43
43
  }
44
44
  ```
45
45
 
46
- That pair passes because the expected capability was selected, the tool
47
- arguments match the manifest expectations, and no unapproved commit happened.
46
+ That pair passes because the expected capability was selected, the tool arguments match
47
+ the manifest expectations, and no unapproved commit happened.
48
48
 
49
49
  Failure example:
50
50
 
@@ -57,21 +57,20 @@ Failure example:
57
57
  }
58
58
  ```
59
59
 
60
- This fails because the candidate selected a commit capability and records a
61
- commit where the eval expected only preparation. The runner reports the failed
62
- scorer instead of calling a model to repair it.
60
+ This fails because the candidate selected a commit capability and records a commit where
61
+ the eval expected only preparation. The runner reports the failed scorer instead of
62
+ calling a model to repair it.
63
63
 
64
- Sanitized replay traces can also become draft eval manifests. Use this when a
65
- mock or live run produced a useful public-safe behavior summary and you want to
66
- lock it in as a regression case. See
67
- [Replay and trace-to-golden](evals/replay-and-trace-to-golden.md).
64
+ Sanitized replay traces can also become draft eval manifests. Use this when a mock or
65
+ live run produced a useful public-safe behavior summary and you want to lock it in as a
66
+ regression case. See [Replay and trace-to-golden](evals/replay-and-trace-to-golden.md).
68
67
 
69
68
  ## Candidate Fixture
70
69
 
71
- Candidate fixtures are JSON files with `schema_version: "1.0"` and a `results`
72
- array. Each result is keyed by `eval_id` and may include selected capabilities,
73
- tool calls, policy decision, action state, committed capabilities, refusal
74
- information, and response text.
70
+ Candidate fixtures are JSON files with `schema_version: "1.0"` and a `results` array.
71
+ Each result is keyed by `eval_id` and may include selected capabilities, tool calls,
72
+ policy decision, action state, committed capabilities, refusal information, and response
73
+ text.
75
74
 
76
75
  ```json
77
76
  {
@@ -95,9 +94,8 @@ information, and response text.
95
94
  }
96
95
  ```
97
96
 
98
- Keep fixtures synthetic and public-safe. Do not commit raw prompts, raw traces,
99
- provider payloads, customer records, secrets, internal endpoints, or private
100
- documents.
97
+ Keep fixtures synthetic and public-safe. Do not commit raw prompts, raw traces, provider
98
+ payloads, customer records, secrets, internal endpoints, or private documents.
101
99
 
102
100
  ## Scorers
103
101
 
@@ -112,12 +110,12 @@ The runner supports the scorer types used by the public eval examples:
112
110
  - `response_excludes_private_detail`
113
111
 
114
112
  It also enforces `expected.response.must_include` and
115
- `expected.response.must_not_include` whenever those fields are present. Unknown
116
- scorer types fail closed.
113
+ `expected.response.must_not_include` whenever those fields are present. Unknown scorer
114
+ types fail closed.
117
115
 
118
- `no_unapproved_commit` fails when a candidate has `action_state: "committed"`,
119
- non-empty `committed_capabilities`, or a tool call to a capability whose
120
- manifest is a commit capability.
116
+ `no_unapproved_commit` fails when a candidate has `action_state: "committed"`, non-empty
117
+ `committed_capabilities`, or a tool call to a capability whose manifest is a commit
118
+ capability.
121
119
 
122
120
  ## CLI
123
121
 
@@ -134,6 +132,6 @@ For machine-readable output:
134
132
  node dist/cli.js eval examples --results examples/eval-results/public.results.passing.json --format json
135
133
  ```
136
134
 
137
- The command exits `0` only when manifests are valid, the result fixture is valid,
138
- every loaded eval has a candidate result, and all scorer checks pass. Failed
139
- checks, missing candidates, invalid fixtures, and unknown scorers exit nonzero.
135
+ The command exits `0` only when manifests are valid, the result fixture is valid, every
136
+ loaded eval has a candidate result, and all scorer checks pass. Failed checks, missing
137
+ candidates, invalid fixtures, and unknown scorers exit nonzero.
package/docs/evalops.md CHANGED
@@ -1,7 +1,7 @@
1
1
  # EvalOps Export Interfaces
2
2
 
3
- AICF EvalOps helpers convert public-safe AICF eval cases and summarized results
4
- into formats that host applications can send to external eval systems.
3
+ AICF EvalOps helpers convert public-safe AICF eval cases and summarized results into
4
+ formats that host applications can send to external eval systems.
5
5
 
6
6
  Import from:
7
7
 
@@ -14,25 +14,24 @@ import {
14
14
  } from "ai-capability-framework/evalops";
15
15
  ```
16
16
 
17
- The EvalOps subpath does not call Braintrust, OpenAI, Promptfoo, or any model
18
- provider. It does not require provider SDKs and does not upload data.
17
+ The EvalOps subpath does not call Braintrust, OpenAI, Promptfoo, or any model provider.
18
+ It does not require provider SDKs and does not upload data.
19
19
 
20
20
  ## What Is Exported
21
21
 
22
- Exports include eval IDs, user-message fixtures, expected summaries, scorer
23
- names, tags, and capability-under-test metadata. Raw prompts, raw provider
24
- payloads, traces, secrets, tokens, private diagnostics, account IDs, tenant IDs,
25
- and customer data are not included.
22
+ Exports include eval IDs, user-message fixtures, expected summaries, scorer names, tags,
23
+ and capability-under-test metadata. Raw prompts, raw provider payloads, traces, secrets,
24
+ tokens, private diagnostics, account IDs, tenant IDs, and customer data are not
25
+ included.
26
26
 
27
27
  ## Promptfoo
28
28
 
29
29
  Promptfoo support remains in `ai-capability-framework/promptfoo`.
30
- `exportPromptfooSuite()` generates files only. It includes an API-key-free echo
31
- provider by default plus README guidance for replacing provider placeholders in
32
- a host-owned test environment.
30
+ `exportPromptfooSuite()` generates files only. It includes an API-key-free echo provider
31
+ by default plus README guidance for replacing provider placeholders in a host-owned test
32
+ environment.
33
33
 
34
34
  ## Host Responsibilities
35
35
 
36
- Host applications own external service accounts, credentials, dataset
37
- retention, eval execution, dashboards, alerting, and review workflows.
38
-
36
+ Host applications own external service accounts, credentials, dataset retention, eval
37
+ execution, dashboards, alerting, and review workflows.
@@ -1,8 +1,7 @@
1
1
  # Golden Tests
2
2
 
3
- Golden evals capture the behavior AICF should preserve: selected capabilities,
4
- tool call IDs, valid arguments, policy decisions, action states, and safe
5
- output.
3
+ Golden evals capture the behavior AICF should preserve: selected capabilities, tool call
4
+ IDs, valid arguments, policy decisions, action states, and safe output.
6
5
 
7
6
  Run the public passing fixture:
8
7
 
@@ -16,5 +15,5 @@ Expected output excerpt:
16
15
  Eval suite passed
17
16
  ```
18
17
 
19
- Golden tests do not call models. Candidate results come from mocks, examples,
20
- or optional live runs that the host explicitly enables.
18
+ Golden tests do not call models. Candidate results come from mocks, examples, or
19
+ optional live runs that the host explicitly enables.
@@ -1,8 +1,8 @@
1
1
  # Eval Overview
2
2
 
3
- AICF evals are deterministic fixtures. They prove capability selection,
4
- expected tool calls, policy outcomes, approval behavior, and no-commit
5
- boundaries without calling a model.
3
+ AICF evals are deterministic fixtures. They prove capability selection, expected tool
4
+ calls, policy outcomes, approval behavior, and no-commit boundaries without calling a
5
+ model.
6
6
 
7
7
  Read:
8
8
 
@@ -3,9 +3,8 @@
3
3
  Replay turns a sanitized runtime trace into something you can compare later.
4
4
  Trace-to-golden turns that same trace into a draft eval case.
5
5
 
6
- Both features are API-key-free by default. They do not call models, import
7
- provider SDKs, execute handlers, store raw prompts, or keep raw provider
8
- payloads.
6
+ Both features are API-key-free by default. They do not call models, import provider
7
+ SDKs, execute handlers, store raw prompts, or keep raw provider payloads.
9
8
 
10
9
  ## What Goes In A Replay Trace
11
10
 
@@ -20,8 +19,8 @@ A replay trace is a public-safe summary, not a raw trace. It can include:
20
19
  - a redacted final response summary;
21
20
  - hashes and redaction summaries.
22
21
 
23
- Do not put raw prompts, raw provider payloads, raw tool outputs, secrets,
24
- customer records, tenant IDs, account IDs, or stack traces in replay fixtures.
22
+ Do not put raw prompts, raw provider payloads, raw tool outputs, secrets, customer
23
+ records, tenant IDs, account IDs, or stack traces in replay fixtures.
25
24
 
26
25
  The public example is:
27
26
 
@@ -37,21 +36,21 @@ examples/support/replay/support.refund.approval_required.trace.json
37
36
  node dist/cli.js replay run examples/support/replay/support.refund.approval_required.trace.json --mode deterministic_mock
38
37
  ```
39
38
 
40
- `policy_only` reruns AICF policy decisions against the current manifests and
41
- redacted decision context.
39
+ `policy_only` reruns AICF policy decisions against the current manifests and redacted
40
+ decision context.
42
41
 
43
42
  ```bash
44
43
  node dist/cli.js replay run examples/support/replay/support.refund.approval_required.trace.json --mode policy_only --manifest-root examples
45
44
  ```
46
45
 
47
- `router_only` reruns deterministic capability selection and checks whether the
48
- selected capability slice changed.
46
+ `router_only` reruns deterministic capability selection and checks whether the selected
47
+ capability slice changed.
49
48
 
50
- `tool_validation_only` validates recorded tool-call arguments against the
51
- current capability input schemas.
49
+ `tool_validation_only` validates recorded tool-call arguments against the current
50
+ capability input schemas.
52
51
 
53
- `provider_live` is guarded. It refuses by default unless the host explicitly
54
- enables live replay and provides its own runner/client.
52
+ `provider_live` is guarded. It refuses by default unless the host explicitly enables
53
+ live replay and provides its own runner/client.
55
54
 
56
55
  ## Create A Draft Eval
57
56
 
@@ -62,8 +61,8 @@ node dist/cli.js evals create-from-trace examples/support/replay/support.refund.
62
61
  ```
63
62
 
64
63
  The generated eval uses redacted user text by default and marks itself as
65
- review-required in `extensions.aicf_replay`. Review the draft before moving it
66
- under `examples/**/evals/`.
64
+ review-required in `extensions.aicf_replay`. Review the draft before moving it under
65
+ `examples/**/evals/`.
67
66
 
68
67
  ## What The Draft Proves
69
68
 
@@ -75,8 +74,8 @@ The generated eval can assert:
75
74
  - action state such as `approval_required`;
76
75
  - no unapproved commit;
77
76
  - forbidden commit tool calls;
78
- - response text must not expose private diagnostics, raw prompts, provider
79
- payloads, tokens, or secrets.
77
+ - response text must not expose private diagnostics, raw prompts, provider payloads,
78
+ tokens, or secrets.
80
79
 
81
80
  ## TypeScript API
82
81
 
@@ -88,6 +87,6 @@ import {
88
87
  } from "ai-capability-framework/replay";
89
88
  ```
90
89
 
91
- Host applications own production trace collection, storage, retention, review
92
- workflow, and any live-provider replay runner. AICF only provides public-safe
93
- contracts and deterministic comparison helpers.
90
+ Host applications own production trace collection, storage, retention, review workflow,
91
+ and any live-provider replay runner. AICF only provides public-safe contracts and
92
+ deterministic comparison helpers.
package/docs/evals.md CHANGED
@@ -1,10 +1,10 @@
1
1
  # Evals
2
2
 
3
- Eval cases keep capability behavior from drifting as prompts, models, policies,
4
- and routing code change.
3
+ Eval cases keep capability behavior from drifting as prompts, models, policies, and
4
+ routing code change.
5
5
 
6
- Each eval should be synthetic, deterministic, and tied to a specific behavior.
7
- See [the 1.0 spec](spec.md) for public-safe example rules.
6
+ Each eval should be synthetic, deterministic, and tied to a specific behavior. See
7
+ [the 1.0 spec](spec.md) for public-safe example rules.
8
8
 
9
9
  ## Useful Eval Types
10
10
 
@@ -19,25 +19,24 @@ See [the 1.0 spec](spec.md) for public-safe example rules.
19
19
  ## Public Example Rule
20
20
 
21
21
  Public examples must use fake identifiers, fake users, fake tenants, and generic
22
- domains. Do not commit real traces, customer records, raw prompts, internal
23
- schemas, or provider payloads.
22
+ domains. Do not commit real traces, customer records, raw prompts, internal schemas, or
23
+ provider payloads.
24
24
 
25
- AICF validation checks that eval files referenced by capability manifests are
26
- present in the loaded manifest bundle.
25
+ AICF validation checks that eval files referenced by capability manifests are present in
26
+ the loaded manifest bundle.
27
27
 
28
- Decision behavior should be covered with synthetic request examples and tests
29
- for permission denial, autonomy denial, approval requirements, deny facts,
30
- missing facts, approval state, and idempotency keys.
28
+ Decision behavior should be covered with synthetic request examples and tests for
29
+ permission denial, autonomy denial, approval requirements, deny facts, missing facts,
30
+ approval state, and idempotency keys.
31
31
 
32
- AICF includes a deterministic eval runner for scoring public-safe candidate
33
- result fixtures against eval manifests. See [the eval runner guide](eval-runner.md)
34
- for fixture shape, scorer behavior, and CLI usage.
32
+ AICF includes a deterministic eval runner for scoring public-safe candidate result
33
+ fixtures against eval manifests. See [the eval runner guide](eval-runner.md) for fixture
34
+ shape, scorer behavior, and CLI usage.
35
35
 
36
- Sanitized replay traces can be converted into review-required eval drafts when
37
- you want to turn observed behavior into a regression case. See
36
+ Sanitized replay traces can be converted into review-required eval drafts when you want
37
+ to turn observed behavior into a regression case. See
38
38
  [Replay and trace-to-golden](evals/replay-and-trace-to-golden.md).
39
39
 
40
- Capability-aware security packs can generate synthetic eval-style cases for
41
- risks such as approval bypass, unsafe commit attempts, prompt injection,
42
- provider payload exposure, and cross-tenant access. See
43
- [security packs](security/security-packs.md).
40
+ Capability-aware security packs can generate synthetic eval-style cases for risks such
41
+ as approval bypass, unsafe commit attempts, prompt injection, provider payload exposure,
42
+ and cross-tenant access. See [security packs](security/security-packs.md).
package/docs/evidence.md CHANGED
@@ -1,13 +1,13 @@
1
1
  # Evidence Export
2
2
 
3
- The evidence subpath creates public-safe review packs from AICF manifests and
4
- supplied runtime/eval reports. Evidence packs are summaries for human review;
5
- they are not certification, audit opinions, legal opinions, security
6
- guarantees, or compliance attestations.
3
+ The evidence subpath creates public-safe review packs from AICF manifests and supplied
4
+ runtime/eval reports. Evidence packs are summaries for human review; they are not
5
+ certification, audit opinions, legal opinions, security guarantees, or compliance
6
+ attestations.
7
7
 
8
- Evidence export does not call models, run providers, execute handlers, store
9
- records, or inspect raw traces. It only summarizes the registry and optional
10
- reports that the host supplies.
8
+ Evidence export does not call models, run providers, execute handlers, store records, or
9
+ inspect raw traces. It only summarizes the registry and optional reports that the host
10
+ supplies.
11
11
 
12
12
  ## CLI
13
13
 
@@ -38,8 +38,8 @@ node dist/cli.js evidence export examples \
38
38
  --security-report security-report.json
39
39
  ```
40
40
 
41
- If optional reports are missing, the pack records that as a coverage gap instead
42
- of implying evidence exists.
41
+ If optional reports are missing, the pack records that as a coverage gap instead of
42
+ implying evidence exists.
43
43
 
44
44
  ## TypeScript
45
45
 
@@ -68,23 +68,22 @@ const markdown = formatEvidencePackMarkdown(pack);
68
68
  An evidence pack includes:
69
69
 
70
70
  - capability, risk, provider, and policy inventories;
71
- - eval, security-pack, provider-conformance, approval, retention, and human
72
- review summaries;
71
+ - eval, security-pack, provider-conformance, approval, retention, and human review
72
+ summaries;
73
73
  - optional incident and model-upgrade summaries supplied by the host;
74
74
  - explicit coverage gaps;
75
75
  - required disclaimers;
76
76
  - redaction metadata describing what was omitted.
77
77
 
78
- The pack includes hashes, redacted refs, statuses, counts, reasons, and
79
- coverage metadata only. It must not include raw prompts, raw provider payloads,
80
- raw transcripts, secrets, stack traces, unredacted tenant/account/user IDs, or
81
- sensitive tool output.
78
+ The pack includes hashes, redacted refs, statuses, counts, reasons, and coverage
79
+ metadata only. It must not include raw prompts, raw provider payloads, raw transcripts,
80
+ secrets, stack traces, unredacted tenant/account/user IDs, or sensitive tool output.
82
81
 
83
82
  For customer-facing generated documents or media, use
84
- `ai-capability-framework/provenance` to create a separate sidecar record that
85
- links content to provider, model, capability, source, approval, and trace refs.
86
- Evidence packs summarize review posture; provenance records describe a specific
87
- generated content artifact.
83
+ `ai-capability-framework/provenance` to create a separate sidecar record that links
84
+ content to provider, model, capability, source, approval, and trace refs. Evidence packs
85
+ summarize review posture; provenance records describe a specific generated content
86
+ artifact.
88
87
 
89
88
  ## Control Plane
90
89
 
@@ -94,11 +93,11 @@ The control-plane evidence endpoint remains available:
94
93
  POST /api/aicf/evidence/export
95
94
  ```
96
95
 
97
- It preserves its existing response shape and also exposes a canonical F13
98
- evidence pack under `canonicalEvidence`.
96
+ It preserves its existing response shape and also exposes a canonical F13 evidence pack
97
+ under `canonicalEvidence`.
99
98
 
100
99
  ## Host Responsibilities
101
100
 
102
101
  Your app owns production evidence collection, approval identity, retention,
103
- legal/compliance review, external audit workflow, and any regulated reporting.
104
- AICF only creates a structured, public-safe summary from the data you provide.
102
+ legal/compliance review, external audit workflow, and any regulated reporting. AICF only
103
+ creates a structured, public-safe summary from the data you provide.
@@ -1,19 +1,19 @@
1
1
  # Google Gemini Runtime
2
2
 
3
- The Gemini runtime is an optional provider adapter for GenerateContent function
4
- calling. It translates a routed AICF capability slice into Gemini function
5
- declarations, runs a bounded `models.generateContent` loop with a
6
- caller-provided client, executes function calls through `AicfToolExecutor`, and
7
- returns Gemini-compatible `functionResponse` parts.
3
+ The Gemini runtime is an optional provider adapter for GenerateContent function calling.
4
+ It translates a routed AICF capability slice into Gemini function declarations, runs a
5
+ bounded `models.generateContent` loop with a caller-provided client, executes function
6
+ calls through `AicfToolExecutor`, and returns Gemini-compatible `functionResponse`
7
+ parts.
8
8
 
9
9
  It does not expose commit tools, enable Gemini built-in tools, use automatic
10
- provider-side execution, persist state, log raw provider payloads, or make
11
- root/runtime imports depend on the Google GenAI SDK.
10
+ provider-side execution, persist state, log raw provider payloads, or make root/runtime
11
+ imports depend on the Google GenAI SDK.
12
12
 
13
13
  ## Import
14
14
 
15
- Install AICF plus the optional Google GenAI SDK only in applications that use
16
- this subpath:
15
+ Install AICF plus the optional Google GenAI SDK only in applications that use this
16
+ subpath:
17
17
 
18
18
  ```bash
19
19
  npm install ai-capability-framework @google/genai
@@ -27,8 +27,8 @@ import {
27
27
  } from "ai-capability-framework/providers/gemini";
28
28
  ```
29
29
 
30
- `createDefaultGeminiClient()` dynamically imports the optional `@google/genai`
31
- package. Host applications may instead pass any compatible client with
30
+ `createDefaultGeminiClient()` dynamically imports the optional `@google/genai` package.
31
+ Host applications may instead pass any compatible client with
32
32
  `models.generateContent(input)`.
33
33
 
34
34
  ## Runtime Shape
@@ -37,43 +37,41 @@ Minimal usage is: validate manifests, build a routed runtime slice, register
37
37
  read/prepare handlers, create or pass a Gemini-compatible client, then call
38
38
  `runGeminiGenerateContent()` with host-owned contents and runtime context.
39
39
 
40
- The host application is responsible for building runtime context, routing
41
- capabilities, registering handlers, and providing initial Gemini contents.
40
+ The host application is responsible for building runtime context, routing capabilities,
41
+ registering handlers, and providing initial Gemini contents.
42
42
 
43
43
  `runGeminiGenerateContent()` sends `model`, `contents`, and `config` to
44
- `client.models.generateContent()`. AICF places routed read/prepare capabilities
45
- under `config.tools: [{ functionDeclarations }]` and can pass optional
44
+ `client.models.generateContent()`. AICF places routed read/prepare capabilities under
45
+ `config.tools: [{ functionDeclarations }]` and can pass optional
46
46
  `toolConfig.functionCallingConfig.mode` plus `allowedFunctionNames`.
47
47
 
48
- When Gemini returns function calls, AICF parses both `response.functionCalls`
49
- and candidate content parts. Calls are mapped through the generated binding map,
50
- arguments are validated against the original AICF schema, and only read or
51
- prepare execution is delegated to the runtime executor.
48
+ When Gemini returns function calls, AICF parses both `response.functionCalls` and
49
+ candidate content parts. Calls are mapped through the generated binding map, arguments
50
+ are validated against the original AICF schema, and only read or prepare execution is
51
+ delegated to the runtime executor.
52
52
 
53
53
  Tool results are appended as user content parts shaped as
54
- `{ functionResponse: { id, name, response: { result } } }`. The `id` is
55
- preserved when Gemini provides one. Approval-required prepare results are
56
- returned as model-safe envelopes without committing the action.
54
+ `{ functionResponse: { id, name, response: { result } } }`. The `id` is preserved when
55
+ Gemini provides one. Approval-required prepare results are returned as model-safe
56
+ envelopes without committing the action.
57
57
 
58
58
  ## Boundary
59
59
 
60
60
  The Gemini runtime follows the same provider-neutral boundary as the OpenAI and
61
61
  Anthropic runtimes:
62
62
 
63
- - AICF capability manifests, policy, lifecycle, and runtime envelopes remain the
64
- source of truth.
65
- - Gemini function declarations are provider descriptors, not trusted execution
66
- policy.
67
- - Commit, destructive, money-moving, and external-send capabilities are not
68
- exported by default and are not executable through provider tool calls.
69
- - Raw prompts, provider request/response payloads, traces, credentials, and
70
- private diagnostics are not captured by default.
63
+ - AICF capability manifests, policy, lifecycle, and runtime envelopes remain the source
64
+ of truth.
65
+ - Gemini function declarations are provider descriptors, not trusted execution policy.
66
+ - Commit, destructive, money-moving, and external-send capabilities are not exported by
67
+ default and are not executable through provider tool calls.
68
+ - Raw prompts, provider request/response payloads, traces, credentials, and private
69
+ diagnostics are not captured by default.
71
70
  - Live tests are opt-in with `RUN_LIVE_GEMINI=1`, `GEMINI_API_KEY`, and
72
71
  `AICF_GEMINI_MODEL`.
73
72
 
74
- Known limitations: no streaming, no Gemini built-in tools, no automatic
75
- provider-side function execution, no production storage, and no model-exposed
76
- commit path.
73
+ Known limitations: no streaming, no Gemini built-in tools, no automatic provider-side
74
+ function execution, no production storage, and no model-exposed commit path.
77
75
 
78
76
  ## Checks
79
77