ai-capability-framework 1.0.0-rc.3 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +97 -103
- package/CODE_OF_CONDUCT.md +11 -13
- package/CONTRIBUTING.md +40 -43
- package/GOVERNANCE.md +34 -35
- package/README.md +99 -63
- package/ROADMAP.md +12 -12
- package/SECURITY.md +15 -15
- package/dist/audit/ledger.d.ts.map +1 -1
- package/dist/audit/ledger.js +2 -1
- package/dist/audit/records.d.ts.map +1 -1
- package/dist/audit/records.js +2 -1
- package/dist/audit/types.d.ts +4 -1
- package/dist/audit/types.d.ts.map +1 -1
- package/dist/aws/dynamodb-stores.d.ts +2 -7
- package/dist/aws/dynamodb-stores.d.ts.map +1 -1
- package/dist/aws/dynamodb-stores.js +56 -5
- package/dist/aws/testing.d.ts.map +1 -1
- package/dist/aws/testing.js +9 -0
- package/dist/cli.d.ts +6 -6
- package/dist/cli.d.ts.map +1 -1
- package/dist/cli.js +77 -7
- package/dist/evals-live/index.d.ts +2 -2
- package/dist/evals-live/index.d.ts.map +1 -1
- package/dist/evals-live/index.js +1 -1
- package/dist/evals-live/live-eval-runner.d.ts +49 -1
- package/dist/evals-live/live-eval-runner.d.ts.map +1 -1
- package/dist/evals-live/live-eval-runner.js +246 -4
- package/dist/evals-live/types.d.ts +62 -9
- package/dist/evals-live/types.d.ts.map +1 -1
- package/dist/index.d.ts +0 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +0 -1
- package/dist/runtime/actions.d.ts +2 -0
- package/dist/runtime/actions.d.ts.map +1 -1
- package/dist/runtime/actions.js +200 -55
- package/dist/runtime/index.d.ts +1 -1
- package/dist/runtime/index.d.ts.map +1 -1
- package/dist/runtime/stores.d.ts +2 -7
- package/dist/runtime/stores.d.ts.map +1 -1
- package/dist/runtime/stores.js +12 -0
- package/dist/runtime/types.d.ts +25 -6
- package/dist/runtime/types.d.ts.map +1 -1
- package/docs/action-lifecycle.md +38 -32
- package/docs/adapter-roadmap.md +51 -49
- package/docs/adapters.md +14 -15
- package/docs/ai-sdk-runtime.md +36 -39
- package/docs/anthropic-runtime.md +29 -32
- package/docs/api/public-api-policy.md +59 -0
- package/docs/api.md +354 -371
- package/docs/architecture/current-state.md +28 -31
- package/docs/audit/index.md +29 -29
- package/docs/aws/cloudwatch-telemetry.md +5 -7
- package/docs/aws/dynamodb-single-table.md +8 -9
- package/docs/aws/kms-redaction.md +5 -6
- package/docs/aws/overview.md +5 -5
- package/docs/aws/production-reference.md +13 -14
- package/docs/aws/step-functions-approval.md +7 -9
- package/docs/aws-runtime.md +24 -25
- package/docs/capability-manifests.md +17 -21
- package/docs/control-plane/overview.md +2 -2
- package/docs/control-plane.md +24 -27
- package/docs/controls/index.md +19 -22
- package/docs/core/capability-manifests.md +4 -5
- package/docs/core/entity-manifests.md +5 -5
- package/docs/eval-runner.md +29 -31
- package/docs/evalops.md +13 -14
- package/docs/evals/golden-tests.md +4 -5
- package/docs/evals/overview.md +3 -3
- package/docs/evals/replay-and-trace-to-golden.md +19 -20
- package/docs/evals.md +19 -20
- package/docs/evidence.md +22 -23
- package/docs/gemini-runtime.md +31 -33
- package/docs/getting-started/anthropic-quickstart.md +61 -0
- package/docs/getting-started/concepts.md +9 -10
- package/docs/getting-started/gemini-quickstart.md +62 -0
- package/docs/getting-started/installation.md +3 -3
- package/docs/getting-started/openai-quickstart.md +61 -0
- package/docs/getting-started/provider-neutral-quickstart.md +106 -0
- package/docs/getting-started/quickstart.md +2 -2
- package/docs/getting-started.md +23 -20
- package/docs/glossary.md +31 -48
- package/docs/governance/gate.md +28 -30
- package/docs/governance/impact-and-compatibility.md +2 -3
- package/docs/governance/index.md +53 -54
- package/docs/governance/lifecycle.md +4 -5
- package/docs/governance/overview.md +2 -3
- package/docs/governance/risk-compiler.md +4 -5
- package/docs/host-responsibilities.md +6 -8
- package/docs/implementation-notes/baseline-gaps.md +15 -18
- package/docs/index.md +21 -6
- package/docs/interoperability.md +9 -9
- package/docs/langchain-runtime.md +27 -31
- package/docs/live-evals.md +68 -32
- package/docs/mcp-server-runtime.md +36 -40
- package/docs/memory.md +35 -36
- package/docs/migration-0.1-to-1.0.md +14 -9
- package/docs/observability/overview.md +2 -3
- package/docs/observability-runtime.md +35 -39
- package/docs/openai-responses.md +36 -37
- package/docs/openai-runtime.md +41 -46
- package/docs/openai-walkthrough.md +24 -28
- package/docs/policy-broker.md +9 -10
- package/docs/provenance.md +30 -29
- package/docs/providers/anthropic.md +5 -5
- package/docs/providers/choose-a-runtime.md +37 -0
- package/docs/providers/conformance.md +14 -15
- package/docs/providers/gemini.md +5 -5
- package/docs/providers/mcp.md +2 -3
- package/docs/providers/openai.md +2 -2
- package/docs/providers/semantic-kernel.md +2 -2
- package/docs/providers.md +43 -38
- package/docs/public/license-exceptions.md +187 -0
- package/docs/public/npm-release-preflight.md +112 -0
- package/docs/public-framework/compatibility-policy.md +12 -13
- package/docs/public-framework/deprecation-policy.md +8 -8
- package/docs/public-framework/final-certification-matrix.md +104 -0
- package/docs/public-framework/license-decision.md +20 -0
- package/docs/public-framework/release-process.md +46 -16
- package/docs/public-framework/security-disclosure.md +4 -5
- package/docs/public-framework/v1-certification.md +102 -15
- package/docs/release.md +200 -122
- package/docs/runtime/action-lifecycle.md +3 -3
- package/docs/runtime/policy-broker.md +4 -4
- package/docs/runtime/runtime-overview.md +4 -5
- package/docs/runtime/tool-result-envelope.md +7 -7
- package/docs/runtime.md +90 -93
- package/docs/security/overview.md +4 -4
- package/docs/security/security-packs.md +24 -26
- package/docs/security/trust-taint-redaction.md +42 -47
- package/docs/semantic-kernel-runtime.md +35 -37
- package/docs/spec.md +95 -101
- package/docs/start-here.md +18 -20
- package/examples/01-basic-read-capability/README.md +3 -4
- package/examples/02-prepare-approve-commit/README.md +6 -6
- package/examples/03-multi-provider-tools/README.md +2 -2
- package/examples/04-mcp-server/README.md +4 -5
- package/examples/05-langchain-langgraph-bridge/README.md +4 -5
- package/examples/06-vercel-ai-sdk-bridge/README.md +5 -7
- package/examples/07-policy-broker-custom-auth/README.md +4 -4
- package/examples/08-aws-step-functions-approval/README.md +4 -5
- package/examples/09-security-packs-promptfoo/README.md +4 -4
- package/examples/10-trace-to-golden/README.md +2 -2
- package/examples/11-control-plane/README.md +4 -5
- package/examples/aws/README.md +5 -8
- package/examples/control-plane/README.md +8 -9
- package/examples/control-plane/fixtures/control-plane.seed.json +8 -32
- package/examples/control-plane/public/app.js +57 -18
- package/examples/control-plane/public/index.html +15 -6
- package/examples/control-plane/server.mjs +15 -3
- package/examples/eval-results/public.results.passing.json +6 -21
- package/examples/providers/ai-sdk-next/README.md +11 -8
- package/examples/providers/anthropic-claude/README.md +5 -6
- package/examples/providers/gemini/README.md +6 -6
- package/examples/providers/langchain-agent/README.md +3 -4
- package/examples/providers/langgraph-tool-node/README.md +7 -7
- package/examples/providers/mcp/README.md +8 -8
- package/examples/providers/provider-conformance/README.md +3 -3
- package/examples/providers/semantic-kernel-mcp/README.md +6 -6
- package/examples/providers/semantic-kernel-openapi/README.md +6 -7
- package/examples/runtime-support-billing/README.md +8 -9
- package/examples/runtime-support-billing/support-billing-runtime.mjs +8 -4
- package/examples/scheduling/capabilities/scheduling.availability.get.yaml +2 -2
- package/examples/scheduling/capabilities/scheduling.invite.prepare.yaml +2 -2
- package/examples/scheduling/capabilities/scheduling.invite.send.yaml +2 -2
- package/examples/scheduling/entities/AvailabilitySlot.yaml +2 -2
- package/examples/scheduling/entities/MeetingInvite.yaml +2 -2
- package/examples/scheduling/evals/scheduling.invite.prepare.valid.yaml +3 -1
- package/examples/semantic-kernel-mcp/README.md +11 -12
- package/examples/semantic-kernel-openapi/README.md +8 -8
- package/examples/support/capabilities/support.refund.commit_case.yaml +3 -3
- package/examples/support/capabilities/support.refund.prepare_case.yaml +2 -2
- package/examples/support/capabilities/support.ticket.get.yaml +2 -2
- package/examples/support/entities/Ticket.yaml +0 -1
- package/examples/support/eval-results/support.results.passing.json +3 -12
- package/examples/support/evals/support.refund.prepare_case.approval_required.yaml +3 -1
- package/examples/support/evals/support.refund.prepare_case.valid.yaml +3 -2
- package/examples/support/memory/support.agent.preferences.json +2 -6
- package/examples/support/openai/context.support_agent.json +1 -4
- package/examples/support/provenance/support.refund.summary.provenance.json +2 -6
- package/examples/support/replay/support.refund.approval_required.trace.json +4 -14
- package/package.json +22 -5
- package/schemas/audit/action-record.schema.json +4 -1
- package/schemas/aws/approval-task.schema.json +8 -1
- package/schemas/conformance/conformance-case.schema.json +18 -2
- package/schemas/conformance/conformance-report.schema.json +17 -2
- package/schemas/conformance/provider-target-matrix.schema.json +35 -4
- package/schemas/controls/control-decision.schema.json +7 -1
- package/schemas/eval-case.schema.json +8 -1
- package/schemas/eval-result.schema.json +8 -1
- package/schemas/evidence/evidence-pack.schema.json +34 -5
- package/schemas/governance/compatibility-diff.schema.json +8 -1
- package/schemas/governance/lifecycle-transition.schema.json +10 -1
- package/schemas/governance/risk-compilation-result.schema.json +9 -1
- package/schemas/provenance/generated-content-provenance.schema.json +3 -1
- package/schemas/replay/replay-result.schema.json +16 -2
- package/schemas/replay/replay-trace.schema.json +11 -2
- package/schemas/security/context-segment.schema.json +21 -12
- package/schemas/security/redaction-policy.schema.json +4 -1
- package/schemas/security-packs/coverage-report.schema.json +17 -2
- package/schemas/security-packs/security-case-suite.schema.json +11 -1
- package/schemas/tool-result-envelope.schema.json +9 -1
- package/security-packs/approval_bypass.yaml +12 -2
- package/security-packs/capability_spoofing.yaml +17 -3
- package/security-packs/cost_amplification.yaml +21 -4
- package/security-packs/cross_tenant_access.yaml +16 -3
- package/security-packs/insecure_output_rendering.yaml +15 -4
- package/security-packs/mcp_tool_abuse.yaml +17 -3
- package/security-packs/memory_scope_violation.yaml +19 -4
- package/security-packs/prompt_injection_direct.yaml +19 -4
- package/security-packs/prompt_injection_indirect.yaml +18 -3
- package/security-packs/provider_payload_exposure.yaml +26 -4
- package/security-packs/retrieval_poisoning.yaml +5 -2
- package/security-packs/schema_confusion.yaml +17 -3
- package/security-packs/sensitive_data_disclosure.yaml +22 -5
- package/security-packs/tool_exfiltration.yaml +20 -4
- package/security-packs/tool_result_poisoning.yaml +19 -4
- package/security-packs/unsafe_commit_attempt.yaml +6 -3
package/docs/spec.md
CHANGED
|
@@ -1,52 +1,52 @@
|
|
|
1
1
|
# AICF 1.0 Spec
|
|
2
2
|
|
|
3
|
-
This page defines the normative public contract for AI Capability Framework
|
|
4
|
-
Core
|
|
3
|
+
This page defines the normative public contract for AI Capability Framework Core (AICF
|
|
4
|
+
Core) 1.0 manifests and deterministic framework behavior.
|
|
5
5
|
|
|
6
6
|
AICF Core is a no-execution framework. It describes capabilities, validates public
|
|
7
7
|
contracts, builds registries, evaluates deterministic policy/lifecycle decisions,
|
|
8
|
-
exports adapter metadata, and scores deterministic eval fixtures. It does not
|
|
9
|
-
|
|
10
|
-
|
|
8
|
+
exports adapter metadata, and scores deterministic eval fixtures. It does not call
|
|
9
|
+
models, execute handlers, verify real host authorization, collect approvals, persist
|
|
10
|
+
actions, write durable audit logs, or perform side effects.
|
|
11
11
|
|
|
12
12
|
## Manifest Versioning
|
|
13
13
|
|
|
14
14
|
- Every 1.0 manifest MUST use `schema_version: "1.0"`.
|
|
15
|
-
- AICF 1.0 is a clean cut from the pre-release `0.1` schema. Validators are not
|
|
16
|
-
|
|
17
|
-
- Capability manifests MUST include a semantic `version` for the application
|
|
18
|
-
|
|
19
|
-
- Breaking changes to manifest fields, validation, or deterministic semantics
|
|
20
|
-
|
|
15
|
+
- AICF 1.0 is a clean cut from the pre-release `0.1` schema. Validators are not required
|
|
16
|
+
to accept `0.1` manifests.
|
|
17
|
+
- Capability manifests MUST include a semantic `version` for the application capability
|
|
18
|
+
contract.
|
|
19
|
+
- Breaking changes to manifest fields, validation, or deterministic semantics MUST use a
|
|
20
|
+
new schema version.
|
|
21
21
|
- Capability behavior changes that affect input, output, policy, side effects,
|
|
22
|
-
lifecycle, model-facing meaning, or eval expectations MUST update the
|
|
23
|
-
|
|
22
|
+
lifecycle, model-facing meaning, or eval expectations MUST update the capability
|
|
23
|
+
`version`.
|
|
24
24
|
- Vendor or application-specific fields MUST live under `extensions`.
|
|
25
25
|
|
|
26
26
|
## ID Naming
|
|
27
27
|
|
|
28
28
|
- Capability IDs MUST use at least three dotted lowercase segments, such as
|
|
29
29
|
`support.refund.prepare_case`.
|
|
30
|
-
- Capability ID segments MAY contain lowercase letters, digits, and underscores,
|
|
31
|
-
|
|
30
|
+
- Capability ID segments MAY contain lowercase letters, digits, and underscores, and
|
|
31
|
+
MUST start with a lowercase letter.
|
|
32
32
|
- Entity IDs MUST use PascalCase, such as `Ticket` or `MeetingInvite`.
|
|
33
|
-
- Eval IDs MUST use lowercase dotted segments and SHOULD include the capability
|
|
34
|
-
|
|
33
|
+
- Eval IDs MUST use lowercase dotted segments and SHOULD include the capability or
|
|
34
|
+
behavior under test.
|
|
35
35
|
- Public examples MUST use synthetic IDs and `example.com` contacts only.
|
|
36
36
|
|
|
37
37
|
## Autonomy Tiers
|
|
38
38
|
|
|
39
|
-
- `A0`: Application-only execution. The model MUST NOT directly select or commit
|
|
40
|
-
|
|
39
|
+
- `A0`: Application-only execution. The model MUST NOT directly select or commit the
|
|
40
|
+
capability in a normal tool set.
|
|
41
41
|
- `A1`: Read or low-risk assistive behavior.
|
|
42
42
|
- `A2`: Prepare-only behavior that produces a reviewable preview.
|
|
43
43
|
- `A3`: Low-risk commits that policy explicitly allows without approval.
|
|
44
44
|
- `A4`: Higher-risk commits requiring strong policy controls and audit.
|
|
45
|
-
- `A5`: Critical actions. These SHOULD require explicit human approval and
|
|
46
|
-
|
|
45
|
+
- `A5`: Critical actions. These SHOULD require explicit human approval and separate
|
|
46
|
+
commit paths.
|
|
47
47
|
|
|
48
|
-
The control plane MUST deny a decision when the request autonomy tier exceeds
|
|
49
|
-
|
|
48
|
+
The control plane MUST deny a decision when the request autonomy tier exceeds the
|
|
49
|
+
capability autonomy tier or the capability policy max autonomy tier.
|
|
50
50
|
|
|
51
51
|
## Risk Tiers
|
|
52
52
|
|
|
@@ -58,16 +58,16 @@ the capability autonomy tier or the capability policy max autonomy tier.
|
|
|
58
58
|
consequential state.
|
|
59
59
|
- `critical`: Irreversible, regulated, privileged, or safety-critical behavior.
|
|
60
60
|
|
|
61
|
-
Risk tiers are manifest metadata used by AICF Core filters and diagnostics.
|
|
62
|
-
|
|
61
|
+
Risk tiers are manifest metadata used by AICF Core filters and diagnostics. Host
|
|
62
|
+
applications remain responsible for real enforcement.
|
|
63
63
|
|
|
64
64
|
## Capability Types
|
|
65
65
|
|
|
66
66
|
- `read_data`: Reads application data.
|
|
67
67
|
- `retrieve_documents`: Retrieves knowledge or source documents.
|
|
68
68
|
- `compute`: Produces deterministic calculations or transformations.
|
|
69
|
-
- `write_prepare_only`: Creates a proposed action or preview without committing
|
|
70
|
-
|
|
69
|
+
- `write_prepare_only`: Creates a proposed action or preview without committing the side
|
|
70
|
+
effect.
|
|
71
71
|
- `write_commit`: Commits a previously allowed and verified side effect.
|
|
72
72
|
- `external_message_prepare`: Prepares a message for review.
|
|
73
73
|
- `external_message_send`: Sends a message outside the application.
|
|
@@ -75,86 +75,81 @@ Host applications remain responsible for real enforcement.
|
|
|
75
75
|
- `workflow_step`: Advances a workflow.
|
|
76
76
|
- `human_handoff`: Transfers work to a human queue or reviewer.
|
|
77
77
|
|
|
78
|
-
Restricted side-effect capabilities SHOULD be excluded from broad model tool
|
|
79
|
-
|
|
78
|
+
Restricted side-effect capabilities SHOULD be excluded from broad model tool sets unless
|
|
79
|
+
the host has narrowed context and verified the required preconditions.
|
|
80
80
|
|
|
81
81
|
## Capability Manifests
|
|
82
82
|
|
|
83
|
-
A capability manifest MUST define model-selectable behavior, inputs, outputs,
|
|
84
|
-
|
|
85
|
-
|
|
83
|
+
A capability manifest MUST define model-selectable behavior, inputs, outputs, risk,
|
|
84
|
+
autonomy, side effects, authorization requirements, policy gates, lifecycle flags,
|
|
85
|
+
observability expectations, and optional eval links.
|
|
86
86
|
|
|
87
87
|
Schemas are strict by default with `additionalProperties: false`. The embedded
|
|
88
88
|
`input_schema` and `output_schema` fields are JSON Schema objects owned by the
|
|
89
|
-
capability author. AICF Core validators MUST compile these embedded schemas and
|
|
90
|
-
|
|
89
|
+
capability author. AICF Core validators MUST compile these embedded schemas and SHOULD
|
|
90
|
+
fail object-incompatible callable input schemas.
|
|
91
91
|
|
|
92
92
|
Capability `status` controls default selection:
|
|
93
93
|
|
|
94
|
-
- `active`: selectable when policy, lifecycle, risk, context, and adapter
|
|
95
|
-
|
|
94
|
+
- `active`: selectable when policy, lifecycle, risk, context, and adapter filters allow
|
|
95
|
+
it.
|
|
96
96
|
- `disabled`: denied and never exported by default.
|
|
97
|
-
- `deprecated`, `draft`, and `experimental`: excluded by default and selectable
|
|
98
|
-
|
|
97
|
+
- `deprecated`, `draft`, and `experimental`: excluded by default and selectable only
|
|
98
|
+
with explicit compatibility, development, or test options.
|
|
99
99
|
|
|
100
|
-
`authorization.permissions` declares deterministic permission strings required
|
|
101
|
-
|
|
102
|
-
|
|
100
|
+
`authorization.permissions` declares deterministic permission strings required by the
|
|
101
|
+
control plane. AICF checks that the request context includes them; host applications
|
|
102
|
+
MUST enforce real identity, tenant, and permission state.
|
|
103
103
|
|
|
104
104
|
## Entity Manifests
|
|
105
105
|
|
|
106
|
-
An entity manifest MUST describe a model-relevant application entity, canonical
|
|
107
|
-
|
|
106
|
+
An entity manifest MUST describe a model-relevant application entity, canonical ID
|
|
107
|
+
field, data classification, lookup path, allowed actions, and model guidance.
|
|
108
108
|
|
|
109
|
-
Entity manifests are descriptive. They do not grant access to records and do not
|
|
110
|
-
|
|
109
|
+
Entity manifests are descriptive. They do not grant access to records and do not replace
|
|
110
|
+
application data models or authorization.
|
|
111
111
|
|
|
112
112
|
## Eval Case Manifests
|
|
113
113
|
|
|
114
|
-
An eval case manifest MUST describe deterministic expected behavior for a
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
constraints.
|
|
114
|
+
An eval case manifest MUST describe deterministic expected behavior for a synthetic user
|
|
115
|
+
input. Evals MAY cover tool selection, argument extraction, policy decisions, approval
|
|
116
|
+
boundaries, refusals, response text, and no-commit constraints.
|
|
118
117
|
|
|
119
|
-
Eval cases MUST be public-safe when committed to this repository. They MUST NOT
|
|
120
|
-
|
|
121
|
-
|
|
118
|
+
Eval cases MUST be public-safe when committed to this repository. They MUST NOT contain
|
|
119
|
+
raw prompts, raw traces, provider payloads, customer records, secrets, internal
|
|
120
|
+
endpoints, or private documents.
|
|
122
121
|
|
|
123
122
|
## Eval Result Fixtures
|
|
124
123
|
|
|
125
|
-
An eval result fixture MUST use `schema_version: "1.0"` and contain `results`.
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
124
|
+
An eval result fixture MUST use `schema_version: "1.0"` and contain `results`. Each
|
|
125
|
+
result is keyed by `eval_id` and MAY include selected capabilities, tool calls, policy
|
|
126
|
+
decision, action state, committed capabilities, refusal metadata, response text, and
|
|
127
|
+
`extensions`.
|
|
129
128
|
|
|
130
|
-
The eval runner scores summarized candidate behavior only. It MUST NOT call
|
|
131
|
-
|
|
129
|
+
The eval runner scores summarized candidate behavior only. It MUST NOT call models,
|
|
130
|
+
ingest raw provider payloads, execute capabilities, or store traces.
|
|
132
131
|
|
|
133
132
|
## Policy Semantics
|
|
134
133
|
|
|
135
|
-
The deterministic control plane evaluates policy from manifest metadata plus a
|
|
136
|
-
|
|
134
|
+
The deterministic control plane evaluates policy from manifest metadata plus a request
|
|
135
|
+
context.
|
|
137
136
|
|
|
138
137
|
- Missing required permissions MUST deny.
|
|
139
|
-
- If `authorization.requires_user_context` is true, missing `context.userId`
|
|
140
|
-
|
|
141
|
-
- If `authorization.tenant_scoped` is true, missing `context.tenantId` MUST
|
|
142
|
-
deny.
|
|
138
|
+
- If `authorization.requires_user_context` is true, missing `context.userId` MUST deny.
|
|
139
|
+
- If `authorization.tenant_scoped` is true, missing `context.tenantId` MUST deny.
|
|
143
140
|
- Requested autonomy above capability or policy max MUST deny.
|
|
144
|
-
- Context `riskCeiling` and `allowedRiskTiers` MUST deny capabilities outside
|
|
145
|
-
|
|
146
|
-
- `prepare` and `commit` requests MUST include args that validate against the
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
- `policy.approval_required` MAY produce an approval-required prepare decision,
|
|
157
|
-
and MUST block commit unless approval is present.
|
|
141
|
+
- Context `riskCeiling` and `allowedRiskTiers` MUST deny capabilities outside the
|
|
142
|
+
requested risk boundary.
|
|
143
|
+
- `prepare` and `commit` requests MUST include args that validate against the capability
|
|
144
|
+
`input_schema`. `select` MAY omit args, but supplied select args MUST validate.
|
|
145
|
+
- `deny_if` checks `facts[rule]`. A true fact MUST deny, a false fact allows evaluation
|
|
146
|
+
to continue, and a missing fact MUST fail closed by denying unless a rule explicitly
|
|
147
|
+
sets `missing_behavior: ignore` or `missing_behavior: approval_required`.
|
|
148
|
+
- `approval_required_if` evaluates field paths against request args. Missing fields on
|
|
149
|
+
medium, high, or critical capabilities MUST fail closed to approval required unless
|
|
150
|
+
the rule explicitly overrides `missing_behavior`.
|
|
151
|
+
- `policy.approval_required` MAY produce an approval-required prepare decision, and MUST
|
|
152
|
+
block commit unless approval is present.
|
|
158
153
|
- Required idempotency MUST block commit unless `idempotencyKey` is present.
|
|
159
154
|
|
|
160
155
|
Host applications own producing facts, verifying approvals, and enforcing real
|
|
@@ -168,8 +163,8 @@ Side-effecting capabilities SHOULD make their lifecycle explicit:
|
|
|
168
163
|
- `preview`: Return a reviewable summary of the proposed action.
|
|
169
164
|
- `approve`: Record a policy or human approval.
|
|
170
165
|
- `commit`: Execute the approved side effect.
|
|
171
|
-
- `commit_capability_id`: Optional link from a prepare capability to the
|
|
172
|
-
|
|
166
|
+
- `commit_capability_id`: Optional link from a prepare capability to the specific commit
|
|
167
|
+
capability that may commit its prepared actions.
|
|
173
168
|
- `verify`: Confirm the side effect reached the expected state.
|
|
174
169
|
- `audit`: Record durable evidence of the decision and execution path.
|
|
175
170
|
|
|
@@ -179,8 +174,8 @@ AICF evaluates lifecycle flags only:
|
|
|
179
174
|
- `prepare` requires `lifecycle.prepare: true`.
|
|
180
175
|
- `commit` requires `lifecycle.commit: true`, a valid prepared-action to commit
|
|
181
176
|
capability link, valid approval when required, and idempotency when required.
|
|
182
|
-
- Decision results include an audit preview. The preview MUST NOT be treated as
|
|
183
|
-
|
|
177
|
+
- Decision results include an audit preview. The preview MUST NOT be treated as a
|
|
178
|
+
persisted audit record.
|
|
184
179
|
|
|
185
180
|
## Idempotency
|
|
186
181
|
|
|
@@ -194,42 +189,41 @@ idempotency storage, replay protection, and conflict handling.
|
|
|
194
189
|
|
|
195
190
|
## OpenAI Responses Adapter
|
|
196
191
|
|
|
197
|
-
The OpenAI Responses adapter converts validated capability manifests into
|
|
198
|
-
|
|
199
|
-
|
|
192
|
+
The OpenAI Responses adapter converts validated capability manifests into function tool
|
|
193
|
+
definitions and parses model tool calls back into AICF capability requests. The adapter
|
|
194
|
+
MUST NOT call OpenAI or execute tool handlers.
|
|
200
195
|
|
|
201
196
|
By default, the adapter MUST include only capabilities whose `select` decision is
|
|
202
|
-
allowed for the supplied context and SHOULD exclude restricted side-effect
|
|
203
|
-
|
|
204
|
-
|
|
197
|
+
allowed for the supplied context and SHOULD exclude restricted side-effect capabilities.
|
|
198
|
+
Restricted export requires an explicit option and is still gated by deterministic
|
|
199
|
+
selection.
|
|
205
200
|
|
|
206
|
-
Provider adapters MUST validate tool-call args against the provider-normalized
|
|
207
|
-
|
|
208
|
-
`input_schema`.
|
|
201
|
+
Provider adapters MUST validate tool-call args against the provider-normalized strict
|
|
202
|
+
schema and return args denormalized to the original capability `input_schema`.
|
|
209
203
|
|
|
210
204
|
## Public Fixture Validation
|
|
211
205
|
|
|
212
|
-
Public validation MUST parse every `.json`, `.yaml`, and `.yml` file under the
|
|
213
|
-
|
|
214
|
-
|
|
206
|
+
Public validation MUST parse every `.json`, `.yaml`, and `.yml` file under the validated
|
|
207
|
+
examples or conformance path. Known fixture directories MUST validate against public
|
|
208
|
+
schemas:
|
|
215
209
|
|
|
216
210
|
- `decisions/`: decision request schema.
|
|
217
211
|
- `eval-results/`: eval result schema.
|
|
218
212
|
- `openai/context*.json`: adapter context schema.
|
|
219
213
|
|
|
220
|
-
Unknown public structured fixture paths MUST fail validation unless they are
|
|
221
|
-
|
|
214
|
+
Unknown public structured fixture paths MUST fail validation unless they are explicitly
|
|
215
|
+
added to the public contract.
|
|
222
216
|
|
|
223
217
|
## Extensions
|
|
224
218
|
|
|
225
|
-
`extensions` is the only stable location for vendor, host, or experimental
|
|
226
|
-
|
|
227
|
-
|
|
219
|
+
`extensions` is the only stable location for vendor, host, or experimental fields. Core
|
|
220
|
+
validators MUST allow extension objects without interpreting them. Portable examples
|
|
221
|
+
SHOULD avoid relying on extension semantics.
|
|
228
222
|
|
|
229
223
|
## Public-Safe Examples
|
|
230
224
|
|
|
231
225
|
- Use fake users, tenants, tickets, orders, meeting IDs, approvals, and amounts.
|
|
232
226
|
- Use `example.com` for contacts.
|
|
233
|
-
- Do not publish raw prompts, traces, provider payloads, credentials, customer
|
|
234
|
-
|
|
227
|
+
- Do not publish raw prompts, traces, provider payloads, credentials, customer records,
|
|
228
|
+
account IDs, internal project names, or private source documents.
|
|
235
229
|
- Keep examples small enough that readers can inspect the full contract.
|
package/docs/start-here.md
CHANGED
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
# Start Here
|
|
2
2
|
|
|
3
|
-
This guide shows the smallest useful AICF path with the public support/refund
|
|
4
|
-
|
|
3
|
+
This guide shows the smallest useful AICF path with the public support/refund example.
|
|
4
|
+
It is API-key-free and uses synthetic data only.
|
|
5
5
|
|
|
6
|
-
You will validate manifests, inspect the registry, export OpenAI tools, run the
|
|
7
|
-
|
|
6
|
+
You will validate manifests, inspect the registry, export OpenAI tools, run the mock
|
|
7
|
+
runtime flow, and score evals.
|
|
8
8
|
|
|
9
9
|
## 1. Install
|
|
10
10
|
|
|
@@ -12,8 +12,8 @@ mock runtime flow, and score evals.
|
|
|
12
12
|
npm install
|
|
13
13
|
```
|
|
14
14
|
|
|
15
|
-
This installs the local package dependencies. It does not install provider SDKs
|
|
16
|
-
|
|
15
|
+
This installs the local package dependencies. It does not install provider SDKs or call
|
|
16
|
+
any model.
|
|
17
17
|
|
|
18
18
|
## 2. Validate The Public Examples
|
|
19
19
|
|
|
@@ -50,8 +50,8 @@ Capabilities by type:
|
|
|
50
50
|
- write_commit: scheduling.invite.send, support.refund.commit_case
|
|
51
51
|
```
|
|
52
52
|
|
|
53
|
-
This shows the validated registry AICF uses for routing, provider export,
|
|
54
|
-
|
|
53
|
+
This shows the validated registry AICF uses for routing, provider export, policy
|
|
54
|
+
decisions, and eval scoring.
|
|
55
55
|
|
|
56
56
|
## 4. Export OpenAI Tools
|
|
57
57
|
|
|
@@ -83,8 +83,8 @@ Expected excerpt:
|
|
|
83
83
|
}
|
|
84
84
|
```
|
|
85
85
|
|
|
86
|
-
This proves safe tool export. The model can see read and prepare tools, but the
|
|
87
|
-
|
|
86
|
+
This proves safe tool export. The model can see read and prepare tools, but the commit
|
|
87
|
+
capability is not exposed as a normal model tool.
|
|
88
88
|
|
|
89
89
|
## 5. Run The Mock Runtime Flow
|
|
90
90
|
|
|
@@ -102,10 +102,9 @@ Expected excerpt:
|
|
|
102
102
|
}
|
|
103
103
|
```
|
|
104
104
|
|
|
105
|
-
This runs route, read, prepare, approval, and lifecycle commit with mock
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
lifecycle manager.
|
|
105
|
+
This runs route, read, prepare, approval, and lifecycle commit with mock handlers and
|
|
106
|
+
in-memory reference stores. The model-facing path still cannot commit. Commit happens
|
|
107
|
+
only after the host records approval and calls the lifecycle manager.
|
|
109
108
|
|
|
110
109
|
## 6. Run Deterministic Evals
|
|
111
110
|
|
|
@@ -119,9 +118,9 @@ Expected excerpt:
|
|
|
119
118
|
Eval suite passed: 6/6 passed.
|
|
120
119
|
```
|
|
121
120
|
|
|
122
|
-
This proves the candidate fixture matches the eval manifests for tool
|
|
123
|
-
|
|
124
|
-
|
|
121
|
+
This proves the candidate fixture matches the eval manifests for tool selection,
|
|
122
|
+
arguments, policy decisions, refusals, response text, and no unapproved commit
|
|
123
|
+
boundaries.
|
|
125
124
|
|
|
126
125
|
## What You Just Proved
|
|
127
126
|
|
|
@@ -133,6 +132,5 @@ unapproved commit boundaries.
|
|
|
133
132
|
- Host-controlled lifecycle commit works after approval and idempotency checks.
|
|
134
133
|
- Evals score behavior without calling models.
|
|
135
134
|
|
|
136
|
-
Next, read the [OpenAI walkthrough](openai-walkthrough.md) for the same flow
|
|
137
|
-
|
|
138
|
-
the main terms.
|
|
135
|
+
Next, read the [OpenAI walkthrough](openai-walkthrough.md) for the same flow from the
|
|
136
|
+
model/tool-call perspective, then use the [glossary](glossary.md) for the main terms.
|
|
@@ -2,8 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Fake data: synthetic support tickets in `examples/support/`.
|
|
4
4
|
|
|
5
|
-
Goal: validate manifests and inspect the read capability
|
|
6
|
-
`support.ticket.get`.
|
|
5
|
+
Goal: validate manifests and inspect the read capability `support.ticket.get`.
|
|
7
6
|
|
|
8
7
|
Commands:
|
|
9
8
|
|
|
@@ -20,5 +19,5 @@ Validated 16 manifest(s) and 18 fixture(s).
|
|
|
20
19
|
support.ticket.get
|
|
21
20
|
```
|
|
22
21
|
|
|
23
|
-
No secrets are required. No live provider calls run by default. This example
|
|
24
|
-
|
|
22
|
+
No secrets are required. No live provider calls run by default. This example uses only
|
|
23
|
+
public synthetic support data.
|
|
@@ -1,10 +1,10 @@
|
|
|
1
1
|
# 02 Prepare, Approve, Commit
|
|
2
2
|
|
|
3
|
-
Fake data: synthetic support tickets, orders, refund previews, and host
|
|
4
|
-
|
|
3
|
+
Fake data: synthetic support tickets, orders, refund previews, and host approval records
|
|
4
|
+
in `examples/runtime-support-billing/`.
|
|
5
5
|
|
|
6
|
-
Goal: run the no-model runtime flow: route, read, prepare, approval required,
|
|
7
|
-
|
|
6
|
+
Goal: run the no-model runtime flow: route, read, prepare, approval required, host
|
|
7
|
+
approval, and lifecycle commit.
|
|
8
8
|
|
|
9
9
|
Command:
|
|
10
10
|
|
|
@@ -19,5 +19,5 @@ approval_required
|
|
|
19
19
|
committed
|
|
20
20
|
```
|
|
21
21
|
|
|
22
|
-
No secrets are required. No live provider calls run by default. Commit is never
|
|
23
|
-
|
|
22
|
+
No secrets are required. No live provider calls run by default. Commit is never exposed
|
|
23
|
+
as a model tool; the host lifecycle manager owns commit.
|
|
@@ -22,5 +22,5 @@ Expected output:
|
|
|
22
22
|
"excluded"
|
|
23
23
|
```
|
|
24
24
|
|
|
25
|
-
No secrets are required. No live provider calls run by default. The commands
|
|
26
|
-
|
|
25
|
+
No secrets are required. No live provider calls run by default. The commands only export
|
|
26
|
+
descriptors and binding maps.
|
|
@@ -2,8 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
Fake data: synthetic support manifests and mock runtime handlers.
|
|
4
4
|
|
|
5
|
-
Goal: understand how AICF can expose routed read/prepare capabilities as MCP
|
|
6
|
-
|
|
5
|
+
Goal: understand how AICF can expose routed read/prepare capabilities as MCP tools while
|
|
6
|
+
keeping commit unavailable to model clients.
|
|
7
7
|
|
|
8
8
|
Commands:
|
|
9
9
|
|
|
@@ -19,6 +19,5 @@ Test Files
|
|
|
19
19
|
passed
|
|
20
20
|
```
|
|
21
21
|
|
|
22
|
-
No secrets are required. No live provider calls run by default. R6 does not add
|
|
23
|
-
|
|
24
|
-
transport.
|
|
22
|
+
No secrets are required. No live provider calls run by default. R6 does not add a
|
|
23
|
+
standalone HTTP or stdio server; hosts wire `AicfMcpServer` to their own MCP transport.
|
|
@@ -2,8 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
Fake data: synthetic support manifests and mock executor calls.
|
|
4
4
|
|
|
5
|
-
Goal: build AICF-backed LangChain tools and a host-supplied LangGraph ToolNode
|
|
6
|
-
|
|
5
|
+
Goal: build AICF-backed LangChain tools and a host-supplied LangGraph ToolNode without
|
|
6
|
+
adding model provider packages.
|
|
7
7
|
|
|
8
8
|
Command:
|
|
9
9
|
|
|
@@ -18,6 +18,5 @@ Test Files
|
|
|
18
18
|
passed
|
|
19
19
|
```
|
|
20
20
|
|
|
21
|
-
No secrets are required. No live provider calls run by default. Optional live
|
|
22
|
-
|
|
23
|
-
host.
|
|
21
|
+
No secrets are required. No live provider calls run by default. Optional live LangChain
|
|
22
|
+
scaffolding is skipped unless `RUN_LIVE_LANGCHAIN=1` is set by the host.
|
|
@@ -1,10 +1,9 @@
|
|
|
1
1
|
# 06 Vercel AI SDK Bridge
|
|
2
2
|
|
|
3
|
-
Fake data: synthetic support manifests and mock `generateText`/`streamText`
|
|
4
|
-
functions.
|
|
3
|
+
Fake data: synthetic support manifests and mock `generateText`/`streamText` functions.
|
|
5
4
|
|
|
6
|
-
Goal: build AICF-backed AI SDK tools while keeping AICF validation, policy,
|
|
7
|
-
|
|
5
|
+
Goal: build AICF-backed AI SDK tools while keeping AICF validation, policy, lifecycle,
|
|
6
|
+
and envelopes authoritative.
|
|
8
7
|
|
|
9
8
|
Command:
|
|
10
9
|
|
|
@@ -19,6 +18,5 @@ Test Files
|
|
|
19
18
|
passed
|
|
20
19
|
```
|
|
21
20
|
|
|
22
|
-
No secrets are required. No live provider calls run by default. Optional live
|
|
23
|
-
|
|
24
|
-
test environment.
|
|
21
|
+
No secrets are required. No live provider calls run by default. Optional live AI SDK use
|
|
22
|
+
requires the host to supply a model configuration and explicit live test environment.
|
|
@@ -2,8 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
Fake data: synthetic support permissions and runtime context.
|
|
4
4
|
|
|
5
|
-
Goal: see how host auth and account systems can make AICF decisions stricter
|
|
6
|
-
|
|
5
|
+
Goal: see how host auth and account systems can make AICF decisions stricter without
|
|
6
|
+
replacing AICF validation.
|
|
7
7
|
|
|
8
8
|
Commands:
|
|
9
9
|
|
|
@@ -20,5 +20,5 @@ denied
|
|
|
20
20
|
passed
|
|
21
21
|
```
|
|
22
22
|
|
|
23
|
-
No secrets are required. No live provider calls run by default. Production auth,
|
|
24
|
-
|
|
23
|
+
No secrets are required. No live provider calls run by default. Production auth, tenant
|
|
24
|
+
boundaries, account state, and entitlements remain host responsibilities.
|
|
@@ -2,8 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
Fake data: synthetic approval payloads and fake AWS SDK clients.
|
|
4
4
|
|
|
5
|
-
Goal: inspect the optional AWS approval handoff and DynamoDB reference adapters
|
|
6
|
-
|
|
5
|
+
Goal: inspect the optional AWS approval handoff and DynamoDB reference adapters without
|
|
6
|
+
credentials or network calls.
|
|
7
7
|
|
|
8
8
|
Command:
|
|
9
9
|
|
|
@@ -18,6 +18,5 @@ Test Files
|
|
|
18
18
|
passed
|
|
19
19
|
```
|
|
20
20
|
|
|
21
|
-
No secrets are required. No live provider calls run by default. Optional live AWS
|
|
22
|
-
|
|
23
|
-
are configured.
|
|
21
|
+
No secrets are required. No live provider calls run by default. Optional live AWS tests
|
|
22
|
+
are skipped unless `RUN_AWS_INTEGRATION=1` and host-owned test resources are configured.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# 09 Security Packs And Promptfoo
|
|
2
2
|
|
|
3
|
-
Fake data: synthetic support and scheduling capabilities plus public security
|
|
4
|
-
|
|
3
|
+
Fake data: synthetic support and scheduling capabilities plus public security pack
|
|
4
|
+
templates.
|
|
5
5
|
|
|
6
6
|
Goal: generate public-safe security cases and API-key-free Promptfoo red-team
|
|
7
7
|
configuration.
|
|
@@ -21,5 +21,5 @@ unsafe_commit_attempt
|
|
|
21
21
|
provider_payload_exposure
|
|
22
22
|
```
|
|
23
23
|
|
|
24
|
-
No secrets are required. No live provider calls run by default. AICF generates
|
|
25
|
-
|
|
24
|
+
No secrets are required. No live provider calls run by default. AICF generates Promptfoo
|
|
25
|
+
config files; it does not run Promptfoo.
|
|
@@ -19,5 +19,5 @@ replay
|
|
|
19
19
|
passed
|
|
20
20
|
```
|
|
21
21
|
|
|
22
|
-
No secrets are required. No live provider calls run by default. Replay traces
|
|
23
|
-
|
|
22
|
+
No secrets are required. No live provider calls run by default. Replay traces contain
|
|
23
|
+
redacted refs, hashes, summaries, and model-safe envelopes only.
|
|
@@ -3,8 +3,8 @@
|
|
|
3
3
|
Fake data: synthetic control-plane seed state in
|
|
4
4
|
`examples/control-plane/fixtures/control-plane.seed.json`.
|
|
5
5
|
|
|
6
|
-
Goal: run the self-hostable local review surface for capability catalogue,
|
|
7
|
-
|
|
6
|
+
Goal: run the self-hostable local review surface for capability catalogue, approvals,
|
|
7
|
+
controls, coverage, replay metadata, and evidence export.
|
|
8
8
|
|
|
9
9
|
Commands:
|
|
10
10
|
|
|
@@ -19,6 +19,5 @@ Expected output:
|
|
|
19
19
|
AICF control plane listening
|
|
20
20
|
```
|
|
21
21
|
|
|
22
|
-
No secrets are required. No live provider calls run by default. The reference
|
|
23
|
-
|
|
24
|
-
and durable storage.
|
|
22
|
+
No secrets are required. No live provider calls run by default. The reference app uses a
|
|
23
|
+
fake local dev user; production deployments must enforce real auth and durable storage.
|
package/examples/aws/README.md
CHANGED
|
@@ -1,8 +1,7 @@
|
|
|
1
1
|
# AWS Reference Adapter Example
|
|
2
2
|
|
|
3
|
-
This directory is README-only and credential-free. It shows how a host
|
|
4
|
-
|
|
5
|
-
the package.
|
|
3
|
+
This directory is README-only and credential-free. It shows how a host application can
|
|
4
|
+
wire AICF AWS reference adapters without making AWS required for the package.
|
|
6
5
|
|
|
7
6
|
```ts
|
|
8
7
|
import {
|
|
@@ -18,10 +17,8 @@ const controlPlaneStore = new DynamoDbControlPlaneStore({
|
|
|
18
17
|
});
|
|
19
18
|
```
|
|
20
19
|
|
|
21
|
-
Use `docs/aws/production-reference.md` for the full guide. Production hosts must
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
cleanup.
|
|
20
|
+
Use `docs/aws/production-reference.md` for the full guide. Production hosts must provide
|
|
21
|
+
credentials, IAM, table provisioning, tenant/account authorization, approval identity,
|
|
22
|
+
workflow definitions, retention policy, monitoring, and cleanup.
|
|
25
23
|
|
|
26
24
|
Local package tests use fake AWS clients only and do not call AWS.
|
|
27
|
-
|