ai-capability-framework 1.0.0-rc.3 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +97 -103
- package/CODE_OF_CONDUCT.md +11 -13
- package/CONTRIBUTING.md +40 -43
- package/GOVERNANCE.md +34 -35
- package/README.md +99 -63
- package/ROADMAP.md +12 -12
- package/SECURITY.md +15 -15
- package/dist/audit/ledger.d.ts.map +1 -1
- package/dist/audit/ledger.js +2 -1
- package/dist/audit/records.d.ts.map +1 -1
- package/dist/audit/records.js +2 -1
- package/dist/audit/types.d.ts +4 -1
- package/dist/audit/types.d.ts.map +1 -1
- package/dist/aws/dynamodb-stores.d.ts +2 -7
- package/dist/aws/dynamodb-stores.d.ts.map +1 -1
- package/dist/aws/dynamodb-stores.js +56 -5
- package/dist/aws/testing.d.ts.map +1 -1
- package/dist/aws/testing.js +9 -0
- package/dist/cli.d.ts +6 -6
- package/dist/cli.d.ts.map +1 -1
- package/dist/cli.js +77 -7
- package/dist/evals-live/index.d.ts +2 -2
- package/dist/evals-live/index.d.ts.map +1 -1
- package/dist/evals-live/index.js +1 -1
- package/dist/evals-live/live-eval-runner.d.ts +49 -1
- package/dist/evals-live/live-eval-runner.d.ts.map +1 -1
- package/dist/evals-live/live-eval-runner.js +246 -4
- package/dist/evals-live/types.d.ts +62 -9
- package/dist/evals-live/types.d.ts.map +1 -1
- package/dist/index.d.ts +0 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +0 -1
- package/dist/runtime/actions.d.ts +2 -0
- package/dist/runtime/actions.d.ts.map +1 -1
- package/dist/runtime/actions.js +200 -55
- package/dist/runtime/index.d.ts +1 -1
- package/dist/runtime/index.d.ts.map +1 -1
- package/dist/runtime/stores.d.ts +2 -7
- package/dist/runtime/stores.d.ts.map +1 -1
- package/dist/runtime/stores.js +12 -0
- package/dist/runtime/types.d.ts +25 -6
- package/dist/runtime/types.d.ts.map +1 -1
- package/docs/action-lifecycle.md +38 -32
- package/docs/adapter-roadmap.md +51 -49
- package/docs/adapters.md +14 -15
- package/docs/ai-sdk-runtime.md +36 -39
- package/docs/anthropic-runtime.md +29 -32
- package/docs/api/public-api-policy.md +59 -0
- package/docs/api.md +354 -371
- package/docs/architecture/current-state.md +28 -31
- package/docs/audit/index.md +29 -29
- package/docs/aws/cloudwatch-telemetry.md +5 -7
- package/docs/aws/dynamodb-single-table.md +8 -9
- package/docs/aws/kms-redaction.md +5 -6
- package/docs/aws/overview.md +5 -5
- package/docs/aws/production-reference.md +13 -14
- package/docs/aws/step-functions-approval.md +7 -9
- package/docs/aws-runtime.md +24 -25
- package/docs/capability-manifests.md +17 -21
- package/docs/control-plane/overview.md +2 -2
- package/docs/control-plane.md +24 -27
- package/docs/controls/index.md +19 -22
- package/docs/core/capability-manifests.md +4 -5
- package/docs/core/entity-manifests.md +5 -5
- package/docs/eval-runner.md +29 -31
- package/docs/evalops.md +13 -14
- package/docs/evals/golden-tests.md +4 -5
- package/docs/evals/overview.md +3 -3
- package/docs/evals/replay-and-trace-to-golden.md +19 -20
- package/docs/evals.md +19 -20
- package/docs/evidence.md +22 -23
- package/docs/gemini-runtime.md +31 -33
- package/docs/getting-started/anthropic-quickstart.md +61 -0
- package/docs/getting-started/concepts.md +9 -10
- package/docs/getting-started/gemini-quickstart.md +62 -0
- package/docs/getting-started/installation.md +3 -3
- package/docs/getting-started/openai-quickstart.md +61 -0
- package/docs/getting-started/provider-neutral-quickstart.md +106 -0
- package/docs/getting-started/quickstart.md +2 -2
- package/docs/getting-started.md +23 -20
- package/docs/glossary.md +31 -48
- package/docs/governance/gate.md +28 -30
- package/docs/governance/impact-and-compatibility.md +2 -3
- package/docs/governance/index.md +53 -54
- package/docs/governance/lifecycle.md +4 -5
- package/docs/governance/overview.md +2 -3
- package/docs/governance/risk-compiler.md +4 -5
- package/docs/host-responsibilities.md +6 -8
- package/docs/implementation-notes/baseline-gaps.md +15 -18
- package/docs/index.md +21 -6
- package/docs/interoperability.md +9 -9
- package/docs/langchain-runtime.md +27 -31
- package/docs/live-evals.md +68 -32
- package/docs/mcp-server-runtime.md +36 -40
- package/docs/memory.md +35 -36
- package/docs/migration-0.1-to-1.0.md +14 -9
- package/docs/observability/overview.md +2 -3
- package/docs/observability-runtime.md +35 -39
- package/docs/openai-responses.md +36 -37
- package/docs/openai-runtime.md +41 -46
- package/docs/openai-walkthrough.md +24 -28
- package/docs/policy-broker.md +9 -10
- package/docs/provenance.md +30 -29
- package/docs/providers/anthropic.md +5 -5
- package/docs/providers/choose-a-runtime.md +37 -0
- package/docs/providers/conformance.md +14 -15
- package/docs/providers/gemini.md +5 -5
- package/docs/providers/mcp.md +2 -3
- package/docs/providers/openai.md +2 -2
- package/docs/providers/semantic-kernel.md +2 -2
- package/docs/providers.md +43 -38
- package/docs/public/license-exceptions.md +187 -0
- package/docs/public/npm-release-preflight.md +112 -0
- package/docs/public-framework/compatibility-policy.md +12 -13
- package/docs/public-framework/deprecation-policy.md +8 -8
- package/docs/public-framework/final-certification-matrix.md +104 -0
- package/docs/public-framework/license-decision.md +20 -0
- package/docs/public-framework/release-process.md +46 -16
- package/docs/public-framework/security-disclosure.md +4 -5
- package/docs/public-framework/v1-certification.md +102 -15
- package/docs/release.md +200 -122
- package/docs/runtime/action-lifecycle.md +3 -3
- package/docs/runtime/policy-broker.md +4 -4
- package/docs/runtime/runtime-overview.md +4 -5
- package/docs/runtime/tool-result-envelope.md +7 -7
- package/docs/runtime.md +90 -93
- package/docs/security/overview.md +4 -4
- package/docs/security/security-packs.md +24 -26
- package/docs/security/trust-taint-redaction.md +42 -47
- package/docs/semantic-kernel-runtime.md +35 -37
- package/docs/spec.md +95 -101
- package/docs/start-here.md +18 -20
- package/examples/01-basic-read-capability/README.md +3 -4
- package/examples/02-prepare-approve-commit/README.md +6 -6
- package/examples/03-multi-provider-tools/README.md +2 -2
- package/examples/04-mcp-server/README.md +4 -5
- package/examples/05-langchain-langgraph-bridge/README.md +4 -5
- package/examples/06-vercel-ai-sdk-bridge/README.md +5 -7
- package/examples/07-policy-broker-custom-auth/README.md +4 -4
- package/examples/08-aws-step-functions-approval/README.md +4 -5
- package/examples/09-security-packs-promptfoo/README.md +4 -4
- package/examples/10-trace-to-golden/README.md +2 -2
- package/examples/11-control-plane/README.md +4 -5
- package/examples/aws/README.md +5 -8
- package/examples/control-plane/README.md +8 -9
- package/examples/control-plane/fixtures/control-plane.seed.json +8 -32
- package/examples/control-plane/public/app.js +57 -18
- package/examples/control-plane/public/index.html +15 -6
- package/examples/control-plane/server.mjs +15 -3
- package/examples/eval-results/public.results.passing.json +6 -21
- package/examples/providers/ai-sdk-next/README.md +11 -8
- package/examples/providers/anthropic-claude/README.md +5 -6
- package/examples/providers/gemini/README.md +6 -6
- package/examples/providers/langchain-agent/README.md +3 -4
- package/examples/providers/langgraph-tool-node/README.md +7 -7
- package/examples/providers/mcp/README.md +8 -8
- package/examples/providers/provider-conformance/README.md +3 -3
- package/examples/providers/semantic-kernel-mcp/README.md +6 -6
- package/examples/providers/semantic-kernel-openapi/README.md +6 -7
- package/examples/runtime-support-billing/README.md +8 -9
- package/examples/runtime-support-billing/support-billing-runtime.mjs +8 -4
- package/examples/scheduling/capabilities/scheduling.availability.get.yaml +2 -2
- package/examples/scheduling/capabilities/scheduling.invite.prepare.yaml +2 -2
- package/examples/scheduling/capabilities/scheduling.invite.send.yaml +2 -2
- package/examples/scheduling/entities/AvailabilitySlot.yaml +2 -2
- package/examples/scheduling/entities/MeetingInvite.yaml +2 -2
- package/examples/scheduling/evals/scheduling.invite.prepare.valid.yaml +3 -1
- package/examples/semantic-kernel-mcp/README.md +11 -12
- package/examples/semantic-kernel-openapi/README.md +8 -8
- package/examples/support/capabilities/support.refund.commit_case.yaml +3 -3
- package/examples/support/capabilities/support.refund.prepare_case.yaml +2 -2
- package/examples/support/capabilities/support.ticket.get.yaml +2 -2
- package/examples/support/entities/Ticket.yaml +0 -1
- package/examples/support/eval-results/support.results.passing.json +3 -12
- package/examples/support/evals/support.refund.prepare_case.approval_required.yaml +3 -1
- package/examples/support/evals/support.refund.prepare_case.valid.yaml +3 -2
- package/examples/support/memory/support.agent.preferences.json +2 -6
- package/examples/support/openai/context.support_agent.json +1 -4
- package/examples/support/provenance/support.refund.summary.provenance.json +2 -6
- package/examples/support/replay/support.refund.approval_required.trace.json +4 -14
- package/package.json +22 -5
- package/schemas/audit/action-record.schema.json +4 -1
- package/schemas/aws/approval-task.schema.json +8 -1
- package/schemas/conformance/conformance-case.schema.json +18 -2
- package/schemas/conformance/conformance-report.schema.json +17 -2
- package/schemas/conformance/provider-target-matrix.schema.json +35 -4
- package/schemas/controls/control-decision.schema.json +7 -1
- package/schemas/eval-case.schema.json +8 -1
- package/schemas/eval-result.schema.json +8 -1
- package/schemas/evidence/evidence-pack.schema.json +34 -5
- package/schemas/governance/compatibility-diff.schema.json +8 -1
- package/schemas/governance/lifecycle-transition.schema.json +10 -1
- package/schemas/governance/risk-compilation-result.schema.json +9 -1
- package/schemas/provenance/generated-content-provenance.schema.json +3 -1
- package/schemas/replay/replay-result.schema.json +16 -2
- package/schemas/replay/replay-trace.schema.json +11 -2
- package/schemas/security/context-segment.schema.json +21 -12
- package/schemas/security/redaction-policy.schema.json +4 -1
- package/schemas/security-packs/coverage-report.schema.json +17 -2
- package/schemas/security-packs/security-case-suite.schema.json +11 -1
- package/schemas/tool-result-envelope.schema.json +9 -1
- package/security-packs/approval_bypass.yaml +12 -2
- package/security-packs/capability_spoofing.yaml +17 -3
- package/security-packs/cost_amplification.yaml +21 -4
- package/security-packs/cross_tenant_access.yaml +16 -3
- package/security-packs/insecure_output_rendering.yaml +15 -4
- package/security-packs/mcp_tool_abuse.yaml +17 -3
- package/security-packs/memory_scope_violation.yaml +19 -4
- package/security-packs/prompt_injection_direct.yaml +19 -4
- package/security-packs/prompt_injection_indirect.yaml +18 -3
- package/security-packs/provider_payload_exposure.yaml +26 -4
- package/security-packs/retrieval_poisoning.yaml +5 -2
- package/security-packs/schema_confusion.yaml +17 -3
- package/security-packs/sensitive_data_disclosure.yaml +22 -5
- package/security-packs/tool_exfiltration.yaml +20 -4
- package/security-packs/tool_result_poisoning.yaml +19 -4
- package/security-packs/unsafe_commit_attempt.yaml +6 -3
package/docs/controls/index.md
CHANGED
|
@@ -1,23 +1,22 @@
|
|
|
1
1
|
# Runtime Controls
|
|
2
2
|
|
|
3
|
-
AICF controls let a host application narrow or stop capability use without
|
|
4
|
-
|
|
5
|
-
|
|
3
|
+
AICF controls let a host application narrow or stop capability use without changing
|
|
4
|
+
manifests. They are optional runtime inputs, not a production control plane or model
|
|
5
|
+
gateway.
|
|
6
6
|
|
|
7
|
-
Controls can apply to a global runtime, provider, model, capability, domain,
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
provider runtime calls.
|
|
7
|
+
Controls can apply to a global runtime, provider, model, capability, domain, risk tier,
|
|
8
|
+
tenant, or autonomy tier. AICF evaluates them before capability routing, tool export,
|
|
9
|
+
model tool execution, lifecycle commit, and supported provider runtime calls.
|
|
11
10
|
|
|
12
11
|
## Kill Switches
|
|
13
12
|
|
|
14
13
|
Kill switches are the fastest control:
|
|
15
14
|
|
|
16
15
|
- `deny` blocks matching capability export and execution.
|
|
17
|
-
- `force_approval` keeps read and prepare paths available, but forces host
|
|
18
|
-
|
|
19
|
-
- `read_only` keeps read/select paths available and blocks prepare, commit,
|
|
20
|
-
|
|
16
|
+
- `force_approval` keeps read and prepare paths available, but forces host approval
|
|
17
|
+
before lifecycle commit.
|
|
18
|
+
- `read_only` keeps read/select paths available and blocks prepare, commit, write, send,
|
|
19
|
+
money-movement, and destructive capabilities.
|
|
21
20
|
|
|
22
21
|
Example:
|
|
23
22
|
|
|
@@ -33,11 +32,10 @@ Example:
|
|
|
33
32
|
|
|
34
33
|
## Circuit Breakers
|
|
35
34
|
|
|
36
|
-
Circuit breakers evaluate recent runtime signals such as provider error rate,
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
responsibilities.
|
|
35
|
+
Circuit breakers evaluate recent runtime signals such as provider error rate, validation
|
|
36
|
+
failure rate, approval rejection rate, tool-loop limit hits, and budget failures. F4
|
|
37
|
+
includes deterministic evaluation and in-memory/reference state only. Production metrics
|
|
38
|
+
ingestion and durable storage remain host responsibilities.
|
|
41
39
|
|
|
42
40
|
## Budgets
|
|
43
41
|
|
|
@@ -48,13 +46,12 @@ Budgets limit one run. Defaults apply when controls are configured:
|
|
|
48
46
|
- 60 seconds runtime per run
|
|
49
47
|
- 2 retries per run
|
|
50
48
|
|
|
51
|
-
Token and estimated-cost budgets are optional fields. Hard budgets fail closed;
|
|
52
|
-
|
|
49
|
+
Token and estimated-cost budgets are optional fields. Hard budgets fail closed; warn
|
|
50
|
+
budgets return warnings without blocking.
|
|
53
51
|
|
|
54
52
|
## CLI
|
|
55
53
|
|
|
56
|
-
Local CLI state uses `.aicf/controls.json`, which is ignored and must not be
|
|
57
|
-
tracked.
|
|
54
|
+
Local CLI state uses `.aicf/controls.json`, which is ignored and must not be tracked.
|
|
58
55
|
|
|
59
56
|
```bash
|
|
60
57
|
aicf controls list --format json
|
|
@@ -62,5 +59,5 @@ aicf controls check examples --capability support.refund.prepare_case --provider
|
|
|
62
59
|
aicf controls kill-switch create --capability support.refund.prepare_case --mode force_approval --reason "incident review"
|
|
63
60
|
```
|
|
64
61
|
|
|
65
|
-
The CLI is useful for local review and demos. Host applications should provide
|
|
66
|
-
|
|
62
|
+
The CLI is useful for local review and demos. Host applications should provide their own
|
|
63
|
+
authenticated control store for production.
|
|
@@ -1,8 +1,7 @@
|
|
|
1
1
|
# Capability Manifests
|
|
2
2
|
|
|
3
|
-
Capability manifests are the core AICF contract. They describe the operation,
|
|
4
|
-
|
|
5
|
-
coverage.
|
|
3
|
+
Capability manifests are the core AICF contract. They describe the operation, input and
|
|
4
|
+
output schemas, lifecycle, permissions, risk, policy, and eval coverage.
|
|
6
5
|
|
|
7
6
|
Use the existing guide for the full field reference:
|
|
8
7
|
|
|
@@ -16,5 +15,5 @@ node dist/cli.js inspect examples
|
|
|
16
15
|
node dist/cli.js validate examples
|
|
17
16
|
```
|
|
18
17
|
|
|
19
|
-
The examples are synthetic and use fake support tickets, orders, and scheduling
|
|
20
|
-
|
|
18
|
+
The examples are synthetic and use fake support tickets, orders, and scheduling data
|
|
19
|
+
only.
|
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
# Entity Manifests
|
|
2
2
|
|
|
3
|
-
Entity manifests describe application objects that capabilities can read,
|
|
4
|
-
|
|
5
|
-
|
|
3
|
+
Entity manifests describe application objects that capabilities can read, prepare, or
|
|
4
|
+
commit against. They help AICF explain impact, governance coverage, and allowed actions
|
|
5
|
+
without exposing raw databases or broad application APIs.
|
|
6
6
|
|
|
7
7
|
Public examples:
|
|
8
8
|
|
|
@@ -17,5 +17,5 @@ Validate them with:
|
|
|
17
17
|
node dist/cli.js validate examples
|
|
18
18
|
```
|
|
19
19
|
|
|
20
|
-
Entity docs stay intentionally small. Capability manifests own the callable
|
|
21
|
-
|
|
20
|
+
Entity docs stay intentionally small. Capability manifests own the callable schemas and
|
|
21
|
+
lifecycle rules.
|
package/docs/eval-runner.md
CHANGED
|
@@ -1,16 +1,16 @@
|
|
|
1
1
|
# Eval Runner
|
|
2
2
|
|
|
3
|
-
The deterministic eval runner scores AICF eval manifests against candidate
|
|
4
|
-
|
|
5
|
-
|
|
3
|
+
The deterministic eval runner scores AICF eval manifests against candidate result
|
|
4
|
+
fixtures without calling models, executing capabilities, storing traces, or reading raw
|
|
5
|
+
provider payloads.
|
|
6
6
|
|
|
7
|
-
Host applications are responsible for producing candidate results. AICF only
|
|
8
|
-
|
|
7
|
+
Host applications are responsible for producing candidate results. AICF only validates
|
|
8
|
+
and scores the summarized behavior.
|
|
9
9
|
|
|
10
10
|
## Mental Model
|
|
11
11
|
|
|
12
|
-
An eval manifest says what should happen. A candidate result says what did
|
|
13
|
-
|
|
12
|
+
An eval manifest says what should happen. A candidate result says what did happen in a
|
|
13
|
+
mock run, live run, or host test. The eval runner compares the two.
|
|
14
14
|
|
|
15
15
|
Eval manifest excerpt:
|
|
16
16
|
|
|
@@ -43,8 +43,8 @@ Candidate result excerpt:
|
|
|
43
43
|
}
|
|
44
44
|
```
|
|
45
45
|
|
|
46
|
-
That pair passes because the expected capability was selected, the tool
|
|
47
|
-
|
|
46
|
+
That pair passes because the expected capability was selected, the tool arguments match
|
|
47
|
+
the manifest expectations, and no unapproved commit happened.
|
|
48
48
|
|
|
49
49
|
Failure example:
|
|
50
50
|
|
|
@@ -57,21 +57,20 @@ Failure example:
|
|
|
57
57
|
}
|
|
58
58
|
```
|
|
59
59
|
|
|
60
|
-
This fails because the candidate selected a commit capability and records a
|
|
61
|
-
|
|
62
|
-
|
|
60
|
+
This fails because the candidate selected a commit capability and records a commit where
|
|
61
|
+
the eval expected only preparation. The runner reports the failed scorer instead of
|
|
62
|
+
calling a model to repair it.
|
|
63
63
|
|
|
64
|
-
Sanitized replay traces can also become draft eval manifests. Use this when a
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
[Replay and trace-to-golden](evals/replay-and-trace-to-golden.md).
|
|
64
|
+
Sanitized replay traces can also become draft eval manifests. Use this when a mock or
|
|
65
|
+
live run produced a useful public-safe behavior summary and you want to lock it in as a
|
|
66
|
+
regression case. See [Replay and trace-to-golden](evals/replay-and-trace-to-golden.md).
|
|
68
67
|
|
|
69
68
|
## Candidate Fixture
|
|
70
69
|
|
|
71
|
-
Candidate fixtures are JSON files with `schema_version: "1.0"` and a `results`
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
70
|
+
Candidate fixtures are JSON files with `schema_version: "1.0"` and a `results` array.
|
|
71
|
+
Each result is keyed by `eval_id` and may include selected capabilities, tool calls,
|
|
72
|
+
policy decision, action state, committed capabilities, refusal information, and response
|
|
73
|
+
text.
|
|
75
74
|
|
|
76
75
|
```json
|
|
77
76
|
{
|
|
@@ -95,9 +94,8 @@ information, and response text.
|
|
|
95
94
|
}
|
|
96
95
|
```
|
|
97
96
|
|
|
98
|
-
Keep fixtures synthetic and public-safe. Do not commit raw prompts, raw traces,
|
|
99
|
-
|
|
100
|
-
documents.
|
|
97
|
+
Keep fixtures synthetic and public-safe. Do not commit raw prompts, raw traces, provider
|
|
98
|
+
payloads, customer records, secrets, internal endpoints, or private documents.
|
|
101
99
|
|
|
102
100
|
## Scorers
|
|
103
101
|
|
|
@@ -112,12 +110,12 @@ The runner supports the scorer types used by the public eval examples:
|
|
|
112
110
|
- `response_excludes_private_detail`
|
|
113
111
|
|
|
114
112
|
It also enforces `expected.response.must_include` and
|
|
115
|
-
`expected.response.must_not_include` whenever those fields are present. Unknown
|
|
116
|
-
|
|
113
|
+
`expected.response.must_not_include` whenever those fields are present. Unknown scorer
|
|
114
|
+
types fail closed.
|
|
117
115
|
|
|
118
|
-
`no_unapproved_commit` fails when a candidate has `action_state: "committed"`,
|
|
119
|
-
|
|
120
|
-
|
|
116
|
+
`no_unapproved_commit` fails when a candidate has `action_state: "committed"`, non-empty
|
|
117
|
+
`committed_capabilities`, or a tool call to a capability whose manifest is a commit
|
|
118
|
+
capability.
|
|
121
119
|
|
|
122
120
|
## CLI
|
|
123
121
|
|
|
@@ -134,6 +132,6 @@ For machine-readable output:
|
|
|
134
132
|
node dist/cli.js eval examples --results examples/eval-results/public.results.passing.json --format json
|
|
135
133
|
```
|
|
136
134
|
|
|
137
|
-
The command exits `0` only when manifests are valid, the result fixture is valid,
|
|
138
|
-
|
|
139
|
-
|
|
135
|
+
The command exits `0` only when manifests are valid, the result fixture is valid, every
|
|
136
|
+
loaded eval has a candidate result, and all scorer checks pass. Failed checks, missing
|
|
137
|
+
candidates, invalid fixtures, and unknown scorers exit nonzero.
|
package/docs/evalops.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# EvalOps Export Interfaces
|
|
2
2
|
|
|
3
|
-
AICF EvalOps helpers convert public-safe AICF eval cases and summarized results
|
|
4
|
-
|
|
3
|
+
AICF EvalOps helpers convert public-safe AICF eval cases and summarized results into
|
|
4
|
+
formats that host applications can send to external eval systems.
|
|
5
5
|
|
|
6
6
|
Import from:
|
|
7
7
|
|
|
@@ -14,25 +14,24 @@ import {
|
|
|
14
14
|
} from "ai-capability-framework/evalops";
|
|
15
15
|
```
|
|
16
16
|
|
|
17
|
-
The EvalOps subpath does not call Braintrust, OpenAI, Promptfoo, or any model
|
|
18
|
-
|
|
17
|
+
The EvalOps subpath does not call Braintrust, OpenAI, Promptfoo, or any model provider.
|
|
18
|
+
It does not require provider SDKs and does not upload data.
|
|
19
19
|
|
|
20
20
|
## What Is Exported
|
|
21
21
|
|
|
22
|
-
Exports include eval IDs, user-message fixtures, expected summaries, scorer
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
22
|
+
Exports include eval IDs, user-message fixtures, expected summaries, scorer names, tags,
|
|
23
|
+
and capability-under-test metadata. Raw prompts, raw provider payloads, traces, secrets,
|
|
24
|
+
tokens, private diagnostics, account IDs, tenant IDs, and customer data are not
|
|
25
|
+
included.
|
|
26
26
|
|
|
27
27
|
## Promptfoo
|
|
28
28
|
|
|
29
29
|
Promptfoo support remains in `ai-capability-framework/promptfoo`.
|
|
30
|
-
`exportPromptfooSuite()` generates files only. It includes an API-key-free echo
|
|
31
|
-
|
|
32
|
-
|
|
30
|
+
`exportPromptfooSuite()` generates files only. It includes an API-key-free echo provider
|
|
31
|
+
by default plus README guidance for replacing provider placeholders in a host-owned test
|
|
32
|
+
environment.
|
|
33
33
|
|
|
34
34
|
## Host Responsibilities
|
|
35
35
|
|
|
36
|
-
Host applications own external service accounts, credentials, dataset
|
|
37
|
-
|
|
38
|
-
|
|
36
|
+
Host applications own external service accounts, credentials, dataset retention, eval
|
|
37
|
+
execution, dashboards, alerting, and review workflows.
|
|
@@ -1,8 +1,7 @@
|
|
|
1
1
|
# Golden Tests
|
|
2
2
|
|
|
3
|
-
Golden evals capture the behavior AICF should preserve: selected capabilities,
|
|
4
|
-
|
|
5
|
-
output.
|
|
3
|
+
Golden evals capture the behavior AICF should preserve: selected capabilities, tool call
|
|
4
|
+
IDs, valid arguments, policy decisions, action states, and safe output.
|
|
6
5
|
|
|
7
6
|
Run the public passing fixture:
|
|
8
7
|
|
|
@@ -16,5 +15,5 @@ Expected output excerpt:
|
|
|
16
15
|
Eval suite passed
|
|
17
16
|
```
|
|
18
17
|
|
|
19
|
-
Golden tests do not call models. Candidate results come from mocks, examples,
|
|
20
|
-
|
|
18
|
+
Golden tests do not call models. Candidate results come from mocks, examples, or
|
|
19
|
+
optional live runs that the host explicitly enables.
|
package/docs/evals/overview.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
# Eval Overview
|
|
2
2
|
|
|
3
|
-
AICF evals are deterministic fixtures. They prove capability selection,
|
|
4
|
-
|
|
5
|
-
|
|
3
|
+
AICF evals are deterministic fixtures. They prove capability selection, expected tool
|
|
4
|
+
calls, policy outcomes, approval behavior, and no-commit boundaries without calling a
|
|
5
|
+
model.
|
|
6
6
|
|
|
7
7
|
Read:
|
|
8
8
|
|
|
@@ -3,9 +3,8 @@
|
|
|
3
3
|
Replay turns a sanitized runtime trace into something you can compare later.
|
|
4
4
|
Trace-to-golden turns that same trace into a draft eval case.
|
|
5
5
|
|
|
6
|
-
Both features are API-key-free by default. They do not call models, import
|
|
7
|
-
|
|
8
|
-
payloads.
|
|
6
|
+
Both features are API-key-free by default. They do not call models, import provider
|
|
7
|
+
SDKs, execute handlers, store raw prompts, or keep raw provider payloads.
|
|
9
8
|
|
|
10
9
|
## What Goes In A Replay Trace
|
|
11
10
|
|
|
@@ -20,8 +19,8 @@ A replay trace is a public-safe summary, not a raw trace. It can include:
|
|
|
20
19
|
- a redacted final response summary;
|
|
21
20
|
- hashes and redaction summaries.
|
|
22
21
|
|
|
23
|
-
Do not put raw prompts, raw provider payloads, raw tool outputs, secrets,
|
|
24
|
-
|
|
22
|
+
Do not put raw prompts, raw provider payloads, raw tool outputs, secrets, customer
|
|
23
|
+
records, tenant IDs, account IDs, or stack traces in replay fixtures.
|
|
25
24
|
|
|
26
25
|
The public example is:
|
|
27
26
|
|
|
@@ -37,21 +36,21 @@ examples/support/replay/support.refund.approval_required.trace.json
|
|
|
37
36
|
node dist/cli.js replay run examples/support/replay/support.refund.approval_required.trace.json --mode deterministic_mock
|
|
38
37
|
```
|
|
39
38
|
|
|
40
|
-
`policy_only` reruns AICF policy decisions against the current manifests and
|
|
41
|
-
|
|
39
|
+
`policy_only` reruns AICF policy decisions against the current manifests and redacted
|
|
40
|
+
decision context.
|
|
42
41
|
|
|
43
42
|
```bash
|
|
44
43
|
node dist/cli.js replay run examples/support/replay/support.refund.approval_required.trace.json --mode policy_only --manifest-root examples
|
|
45
44
|
```
|
|
46
45
|
|
|
47
|
-
`router_only` reruns deterministic capability selection and checks whether the
|
|
48
|
-
|
|
46
|
+
`router_only` reruns deterministic capability selection and checks whether the selected
|
|
47
|
+
capability slice changed.
|
|
49
48
|
|
|
50
|
-
`tool_validation_only` validates recorded tool-call arguments against the
|
|
51
|
-
|
|
49
|
+
`tool_validation_only` validates recorded tool-call arguments against the current
|
|
50
|
+
capability input schemas.
|
|
52
51
|
|
|
53
|
-
`provider_live` is guarded. It refuses by default unless the host explicitly
|
|
54
|
-
|
|
52
|
+
`provider_live` is guarded. It refuses by default unless the host explicitly enables
|
|
53
|
+
live replay and provides its own runner/client.
|
|
55
54
|
|
|
56
55
|
## Create A Draft Eval
|
|
57
56
|
|
|
@@ -62,8 +61,8 @@ node dist/cli.js evals create-from-trace examples/support/replay/support.refund.
|
|
|
62
61
|
```
|
|
63
62
|
|
|
64
63
|
The generated eval uses redacted user text by default and marks itself as
|
|
65
|
-
review-required in `extensions.aicf_replay`. Review the draft before moving it
|
|
66
|
-
|
|
64
|
+
review-required in `extensions.aicf_replay`. Review the draft before moving it under
|
|
65
|
+
`examples/**/evals/`.
|
|
67
66
|
|
|
68
67
|
## What The Draft Proves
|
|
69
68
|
|
|
@@ -75,8 +74,8 @@ The generated eval can assert:
|
|
|
75
74
|
- action state such as `approval_required`;
|
|
76
75
|
- no unapproved commit;
|
|
77
76
|
- forbidden commit tool calls;
|
|
78
|
-
- response text must not expose private diagnostics, raw prompts, provider
|
|
79
|
-
|
|
77
|
+
- response text must not expose private diagnostics, raw prompts, provider payloads,
|
|
78
|
+
tokens, or secrets.
|
|
80
79
|
|
|
81
80
|
## TypeScript API
|
|
82
81
|
|
|
@@ -88,6 +87,6 @@ import {
|
|
|
88
87
|
} from "ai-capability-framework/replay";
|
|
89
88
|
```
|
|
90
89
|
|
|
91
|
-
Host applications own production trace collection, storage, retention, review
|
|
92
|
-
|
|
93
|
-
|
|
90
|
+
Host applications own production trace collection, storage, retention, review workflow,
|
|
91
|
+
and any live-provider replay runner. AICF only provides public-safe contracts and
|
|
92
|
+
deterministic comparison helpers.
|
package/docs/evals.md
CHANGED
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
# Evals
|
|
2
2
|
|
|
3
|
-
Eval cases keep capability behavior from drifting as prompts, models, policies,
|
|
4
|
-
|
|
3
|
+
Eval cases keep capability behavior from drifting as prompts, models, policies, and
|
|
4
|
+
routing code change.
|
|
5
5
|
|
|
6
|
-
Each eval should be synthetic, deterministic, and tied to a specific behavior.
|
|
7
|
-
|
|
6
|
+
Each eval should be synthetic, deterministic, and tied to a specific behavior. See
|
|
7
|
+
[the 1.0 spec](spec.md) for public-safe example rules.
|
|
8
8
|
|
|
9
9
|
## Useful Eval Types
|
|
10
10
|
|
|
@@ -19,25 +19,24 @@ See [the 1.0 spec](spec.md) for public-safe example rules.
|
|
|
19
19
|
## Public Example Rule
|
|
20
20
|
|
|
21
21
|
Public examples must use fake identifiers, fake users, fake tenants, and generic
|
|
22
|
-
domains. Do not commit real traces, customer records, raw prompts, internal
|
|
23
|
-
|
|
22
|
+
domains. Do not commit real traces, customer records, raw prompts, internal schemas, or
|
|
23
|
+
provider payloads.
|
|
24
24
|
|
|
25
|
-
AICF validation checks that eval files referenced by capability manifests are
|
|
26
|
-
|
|
25
|
+
AICF validation checks that eval files referenced by capability manifests are present in
|
|
26
|
+
the loaded manifest bundle.
|
|
27
27
|
|
|
28
|
-
Decision behavior should be covered with synthetic request examples and tests
|
|
29
|
-
|
|
30
|
-
|
|
28
|
+
Decision behavior should be covered with synthetic request examples and tests for
|
|
29
|
+
permission denial, autonomy denial, approval requirements, deny facts, missing facts,
|
|
30
|
+
approval state, and idempotency keys.
|
|
31
31
|
|
|
32
|
-
AICF includes a deterministic eval runner for scoring public-safe candidate
|
|
33
|
-
|
|
34
|
-
|
|
32
|
+
AICF includes a deterministic eval runner for scoring public-safe candidate result
|
|
33
|
+
fixtures against eval manifests. See [the eval runner guide](eval-runner.md) for fixture
|
|
34
|
+
shape, scorer behavior, and CLI usage.
|
|
35
35
|
|
|
36
|
-
Sanitized replay traces can be converted into review-required eval drafts when
|
|
37
|
-
|
|
36
|
+
Sanitized replay traces can be converted into review-required eval drafts when you want
|
|
37
|
+
to turn observed behavior into a regression case. See
|
|
38
38
|
[Replay and trace-to-golden](evals/replay-and-trace-to-golden.md).
|
|
39
39
|
|
|
40
|
-
Capability-aware security packs can generate synthetic eval-style cases for
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
[security packs](security/security-packs.md).
|
|
40
|
+
Capability-aware security packs can generate synthetic eval-style cases for risks such
|
|
41
|
+
as approval bypass, unsafe commit attempts, prompt injection, provider payload exposure,
|
|
42
|
+
and cross-tenant access. See [security packs](security/security-packs.md).
|
package/docs/evidence.md
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
# Evidence Export
|
|
2
2
|
|
|
3
|
-
The evidence subpath creates public-safe review packs from AICF manifests and
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
3
|
+
The evidence subpath creates public-safe review packs from AICF manifests and supplied
|
|
4
|
+
runtime/eval reports. Evidence packs are summaries for human review; they are not
|
|
5
|
+
certification, audit opinions, legal opinions, security guarantees, or compliance
|
|
6
|
+
attestations.
|
|
7
7
|
|
|
8
|
-
Evidence export does not call models, run providers, execute handlers, store
|
|
9
|
-
|
|
10
|
-
|
|
8
|
+
Evidence export does not call models, run providers, execute handlers, store records, or
|
|
9
|
+
inspect raw traces. It only summarizes the registry and optional reports that the host
|
|
10
|
+
supplies.
|
|
11
11
|
|
|
12
12
|
## CLI
|
|
13
13
|
|
|
@@ -38,8 +38,8 @@ node dist/cli.js evidence export examples \
|
|
|
38
38
|
--security-report security-report.json
|
|
39
39
|
```
|
|
40
40
|
|
|
41
|
-
If optional reports are missing, the pack records that as a coverage gap instead
|
|
42
|
-
|
|
41
|
+
If optional reports are missing, the pack records that as a coverage gap instead of
|
|
42
|
+
implying evidence exists.
|
|
43
43
|
|
|
44
44
|
## TypeScript
|
|
45
45
|
|
|
@@ -68,23 +68,22 @@ const markdown = formatEvidencePackMarkdown(pack);
|
|
|
68
68
|
An evidence pack includes:
|
|
69
69
|
|
|
70
70
|
- capability, risk, provider, and policy inventories;
|
|
71
|
-
- eval, security-pack, provider-conformance, approval, retention, and human
|
|
72
|
-
|
|
71
|
+
- eval, security-pack, provider-conformance, approval, retention, and human review
|
|
72
|
+
summaries;
|
|
73
73
|
- optional incident and model-upgrade summaries supplied by the host;
|
|
74
74
|
- explicit coverage gaps;
|
|
75
75
|
- required disclaimers;
|
|
76
76
|
- redaction metadata describing what was omitted.
|
|
77
77
|
|
|
78
|
-
The pack includes hashes, redacted refs, statuses, counts, reasons, and
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
sensitive tool output.
|
|
78
|
+
The pack includes hashes, redacted refs, statuses, counts, reasons, and coverage
|
|
79
|
+
metadata only. It must not include raw prompts, raw provider payloads, raw transcripts,
|
|
80
|
+
secrets, stack traces, unredacted tenant/account/user IDs, or sensitive tool output.
|
|
82
81
|
|
|
83
82
|
For customer-facing generated documents or media, use
|
|
84
|
-
`ai-capability-framework/provenance` to create a separate sidecar record that
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
83
|
+
`ai-capability-framework/provenance` to create a separate sidecar record that links
|
|
84
|
+
content to provider, model, capability, source, approval, and trace refs. Evidence packs
|
|
85
|
+
summarize review posture; provenance records describe a specific generated content
|
|
86
|
+
artifact.
|
|
88
87
|
|
|
89
88
|
## Control Plane
|
|
90
89
|
|
|
@@ -94,11 +93,11 @@ The control-plane evidence endpoint remains available:
|
|
|
94
93
|
POST /api/aicf/evidence/export
|
|
95
94
|
```
|
|
96
95
|
|
|
97
|
-
It preserves its existing response shape and also exposes a canonical F13
|
|
98
|
-
|
|
96
|
+
It preserves its existing response shape and also exposes a canonical F13 evidence pack
|
|
97
|
+
under `canonicalEvidence`.
|
|
99
98
|
|
|
100
99
|
## Host Responsibilities
|
|
101
100
|
|
|
102
101
|
Your app owns production evidence collection, approval identity, retention,
|
|
103
|
-
legal/compliance review, external audit workflow, and any regulated reporting.
|
|
104
|
-
|
|
102
|
+
legal/compliance review, external audit workflow, and any regulated reporting. AICF only
|
|
103
|
+
creates a structured, public-safe summary from the data you provide.
|
package/docs/gemini-runtime.md
CHANGED
|
@@ -1,19 +1,19 @@
|
|
|
1
1
|
# Google Gemini Runtime
|
|
2
2
|
|
|
3
|
-
The Gemini runtime is an optional provider adapter for GenerateContent function
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
3
|
+
The Gemini runtime is an optional provider adapter for GenerateContent function calling.
|
|
4
|
+
It translates a routed AICF capability slice into Gemini function declarations, runs a
|
|
5
|
+
bounded `models.generateContent` loop with a caller-provided client, executes function
|
|
6
|
+
calls through `AicfToolExecutor`, and returns Gemini-compatible `functionResponse`
|
|
7
|
+
parts.
|
|
8
8
|
|
|
9
9
|
It does not expose commit tools, enable Gemini built-in tools, use automatic
|
|
10
|
-
provider-side execution, persist state, log raw provider payloads, or make
|
|
11
|
-
|
|
10
|
+
provider-side execution, persist state, log raw provider payloads, or make root/runtime
|
|
11
|
+
imports depend on the Google GenAI SDK.
|
|
12
12
|
|
|
13
13
|
## Import
|
|
14
14
|
|
|
15
|
-
Install AICF plus the optional Google GenAI SDK only in applications that use
|
|
16
|
-
|
|
15
|
+
Install AICF plus the optional Google GenAI SDK only in applications that use this
|
|
16
|
+
subpath:
|
|
17
17
|
|
|
18
18
|
```bash
|
|
19
19
|
npm install ai-capability-framework @google/genai
|
|
@@ -27,8 +27,8 @@ import {
|
|
|
27
27
|
} from "ai-capability-framework/providers/gemini";
|
|
28
28
|
```
|
|
29
29
|
|
|
30
|
-
`createDefaultGeminiClient()` dynamically imports the optional `@google/genai`
|
|
31
|
-
|
|
30
|
+
`createDefaultGeminiClient()` dynamically imports the optional `@google/genai` package.
|
|
31
|
+
Host applications may instead pass any compatible client with
|
|
32
32
|
`models.generateContent(input)`.
|
|
33
33
|
|
|
34
34
|
## Runtime Shape
|
|
@@ -37,43 +37,41 @@ Minimal usage is: validate manifests, build a routed runtime slice, register
|
|
|
37
37
|
read/prepare handlers, create or pass a Gemini-compatible client, then call
|
|
38
38
|
`runGeminiGenerateContent()` with host-owned contents and runtime context.
|
|
39
39
|
|
|
40
|
-
The host application is responsible for building runtime context, routing
|
|
41
|
-
|
|
40
|
+
The host application is responsible for building runtime context, routing capabilities,
|
|
41
|
+
registering handlers, and providing initial Gemini contents.
|
|
42
42
|
|
|
43
43
|
`runGeminiGenerateContent()` sends `model`, `contents`, and `config` to
|
|
44
|
-
`client.models.generateContent()`. AICF places routed read/prepare capabilities
|
|
45
|
-
|
|
44
|
+
`client.models.generateContent()`. AICF places routed read/prepare capabilities under
|
|
45
|
+
`config.tools: [{ functionDeclarations }]` and can pass optional
|
|
46
46
|
`toolConfig.functionCallingConfig.mode` plus `allowedFunctionNames`.
|
|
47
47
|
|
|
48
|
-
When Gemini returns function calls, AICF parses both `response.functionCalls`
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
48
|
+
When Gemini returns function calls, AICF parses both `response.functionCalls` and
|
|
49
|
+
candidate content parts. Calls are mapped through the generated binding map, arguments
|
|
50
|
+
are validated against the original AICF schema, and only read or prepare execution is
|
|
51
|
+
delegated to the runtime executor.
|
|
52
52
|
|
|
53
53
|
Tool results are appended as user content parts shaped as
|
|
54
|
-
`{ functionResponse: { id, name, response: { result } } }`. The `id` is
|
|
55
|
-
|
|
56
|
-
|
|
54
|
+
`{ functionResponse: { id, name, response: { result } } }`. The `id` is preserved when
|
|
55
|
+
Gemini provides one. Approval-required prepare results are returned as model-safe
|
|
56
|
+
envelopes without committing the action.
|
|
57
57
|
|
|
58
58
|
## Boundary
|
|
59
59
|
|
|
60
60
|
The Gemini runtime follows the same provider-neutral boundary as the OpenAI and
|
|
61
61
|
Anthropic runtimes:
|
|
62
62
|
|
|
63
|
-
- AICF capability manifests, policy, lifecycle, and runtime envelopes remain the
|
|
64
|
-
|
|
65
|
-
- Gemini function declarations are provider descriptors, not trusted execution
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
private diagnostics are not captured by default.
|
|
63
|
+
- AICF capability manifests, policy, lifecycle, and runtime envelopes remain the source
|
|
64
|
+
of truth.
|
|
65
|
+
- Gemini function declarations are provider descriptors, not trusted execution policy.
|
|
66
|
+
- Commit, destructive, money-moving, and external-send capabilities are not exported by
|
|
67
|
+
default and are not executable through provider tool calls.
|
|
68
|
+
- Raw prompts, provider request/response payloads, traces, credentials, and private
|
|
69
|
+
diagnostics are not captured by default.
|
|
71
70
|
- Live tests are opt-in with `RUN_LIVE_GEMINI=1`, `GEMINI_API_KEY`, and
|
|
72
71
|
`AICF_GEMINI_MODEL`.
|
|
73
72
|
|
|
74
|
-
Known limitations: no streaming, no Gemini built-in tools, no automatic
|
|
75
|
-
|
|
76
|
-
commit path.
|
|
73
|
+
Known limitations: no streaming, no Gemini built-in tools, no automatic provider-side
|
|
74
|
+
function execution, no production storage, and no model-exposed commit path.
|
|
77
75
|
|
|
78
76
|
## Checks
|
|
79
77
|
|