ai-capability-framework 1.0.0-rc.3 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +97 -103
- package/CODE_OF_CONDUCT.md +11 -13
- package/CONTRIBUTING.md +40 -43
- package/GOVERNANCE.md +34 -35
- package/README.md +99 -63
- package/ROADMAP.md +12 -12
- package/SECURITY.md +15 -15
- package/dist/audit/ledger.d.ts.map +1 -1
- package/dist/audit/ledger.js +2 -1
- package/dist/audit/records.d.ts.map +1 -1
- package/dist/audit/records.js +2 -1
- package/dist/audit/types.d.ts +4 -1
- package/dist/audit/types.d.ts.map +1 -1
- package/dist/aws/dynamodb-stores.d.ts +2 -7
- package/dist/aws/dynamodb-stores.d.ts.map +1 -1
- package/dist/aws/dynamodb-stores.js +56 -5
- package/dist/aws/testing.d.ts.map +1 -1
- package/dist/aws/testing.js +9 -0
- package/dist/cli.d.ts +6 -6
- package/dist/cli.d.ts.map +1 -1
- package/dist/cli.js +77 -7
- package/dist/evals-live/index.d.ts +2 -2
- package/dist/evals-live/index.d.ts.map +1 -1
- package/dist/evals-live/index.js +1 -1
- package/dist/evals-live/live-eval-runner.d.ts +49 -1
- package/dist/evals-live/live-eval-runner.d.ts.map +1 -1
- package/dist/evals-live/live-eval-runner.js +246 -4
- package/dist/evals-live/types.d.ts +62 -9
- package/dist/evals-live/types.d.ts.map +1 -1
- package/dist/index.d.ts +0 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +0 -1
- package/dist/runtime/actions.d.ts +2 -0
- package/dist/runtime/actions.d.ts.map +1 -1
- package/dist/runtime/actions.js +200 -55
- package/dist/runtime/index.d.ts +1 -1
- package/dist/runtime/index.d.ts.map +1 -1
- package/dist/runtime/stores.d.ts +2 -7
- package/dist/runtime/stores.d.ts.map +1 -1
- package/dist/runtime/stores.js +12 -0
- package/dist/runtime/types.d.ts +25 -6
- package/dist/runtime/types.d.ts.map +1 -1
- package/docs/action-lifecycle.md +38 -32
- package/docs/adapter-roadmap.md +51 -49
- package/docs/adapters.md +14 -15
- package/docs/ai-sdk-runtime.md +36 -39
- package/docs/anthropic-runtime.md +29 -32
- package/docs/api/public-api-policy.md +59 -0
- package/docs/api.md +354 -371
- package/docs/architecture/current-state.md +28 -31
- package/docs/audit/index.md +29 -29
- package/docs/aws/cloudwatch-telemetry.md +5 -7
- package/docs/aws/dynamodb-single-table.md +8 -9
- package/docs/aws/kms-redaction.md +5 -6
- package/docs/aws/overview.md +5 -5
- package/docs/aws/production-reference.md +13 -14
- package/docs/aws/step-functions-approval.md +7 -9
- package/docs/aws-runtime.md +24 -25
- package/docs/capability-manifests.md +17 -21
- package/docs/control-plane/overview.md +2 -2
- package/docs/control-plane.md +24 -27
- package/docs/controls/index.md +19 -22
- package/docs/core/capability-manifests.md +4 -5
- package/docs/core/entity-manifests.md +5 -5
- package/docs/eval-runner.md +29 -31
- package/docs/evalops.md +13 -14
- package/docs/evals/golden-tests.md +4 -5
- package/docs/evals/overview.md +3 -3
- package/docs/evals/replay-and-trace-to-golden.md +19 -20
- package/docs/evals.md +19 -20
- package/docs/evidence.md +22 -23
- package/docs/gemini-runtime.md +31 -33
- package/docs/getting-started/anthropic-quickstart.md +61 -0
- package/docs/getting-started/concepts.md +9 -10
- package/docs/getting-started/gemini-quickstart.md +62 -0
- package/docs/getting-started/installation.md +3 -3
- package/docs/getting-started/openai-quickstart.md +61 -0
- package/docs/getting-started/provider-neutral-quickstart.md +106 -0
- package/docs/getting-started/quickstart.md +2 -2
- package/docs/getting-started.md +23 -20
- package/docs/glossary.md +31 -48
- package/docs/governance/gate.md +28 -30
- package/docs/governance/impact-and-compatibility.md +2 -3
- package/docs/governance/index.md +53 -54
- package/docs/governance/lifecycle.md +4 -5
- package/docs/governance/overview.md +2 -3
- package/docs/governance/risk-compiler.md +4 -5
- package/docs/host-responsibilities.md +6 -8
- package/docs/implementation-notes/baseline-gaps.md +15 -18
- package/docs/index.md +21 -6
- package/docs/interoperability.md +9 -9
- package/docs/langchain-runtime.md +27 -31
- package/docs/live-evals.md +68 -32
- package/docs/mcp-server-runtime.md +36 -40
- package/docs/memory.md +35 -36
- package/docs/migration-0.1-to-1.0.md +14 -9
- package/docs/observability/overview.md +2 -3
- package/docs/observability-runtime.md +35 -39
- package/docs/openai-responses.md +36 -37
- package/docs/openai-runtime.md +41 -46
- package/docs/openai-walkthrough.md +24 -28
- package/docs/policy-broker.md +9 -10
- package/docs/provenance.md +30 -29
- package/docs/providers/anthropic.md +5 -5
- package/docs/providers/choose-a-runtime.md +37 -0
- package/docs/providers/conformance.md +14 -15
- package/docs/providers/gemini.md +5 -5
- package/docs/providers/mcp.md +2 -3
- package/docs/providers/openai.md +2 -2
- package/docs/providers/semantic-kernel.md +2 -2
- package/docs/providers.md +43 -38
- package/docs/public/license-exceptions.md +187 -0
- package/docs/public/npm-release-preflight.md +112 -0
- package/docs/public-framework/compatibility-policy.md +12 -13
- package/docs/public-framework/deprecation-policy.md +8 -8
- package/docs/public-framework/final-certification-matrix.md +104 -0
- package/docs/public-framework/license-decision.md +20 -0
- package/docs/public-framework/release-process.md +46 -16
- package/docs/public-framework/security-disclosure.md +4 -5
- package/docs/public-framework/v1-certification.md +102 -15
- package/docs/release.md +200 -122
- package/docs/runtime/action-lifecycle.md +3 -3
- package/docs/runtime/policy-broker.md +4 -4
- package/docs/runtime/runtime-overview.md +4 -5
- package/docs/runtime/tool-result-envelope.md +7 -7
- package/docs/runtime.md +90 -93
- package/docs/security/overview.md +4 -4
- package/docs/security/security-packs.md +24 -26
- package/docs/security/trust-taint-redaction.md +42 -47
- package/docs/semantic-kernel-runtime.md +35 -37
- package/docs/spec.md +95 -101
- package/docs/start-here.md +18 -20
- package/examples/01-basic-read-capability/README.md +3 -4
- package/examples/02-prepare-approve-commit/README.md +6 -6
- package/examples/03-multi-provider-tools/README.md +2 -2
- package/examples/04-mcp-server/README.md +4 -5
- package/examples/05-langchain-langgraph-bridge/README.md +4 -5
- package/examples/06-vercel-ai-sdk-bridge/README.md +5 -7
- package/examples/07-policy-broker-custom-auth/README.md +4 -4
- package/examples/08-aws-step-functions-approval/README.md +4 -5
- package/examples/09-security-packs-promptfoo/README.md +4 -4
- package/examples/10-trace-to-golden/README.md +2 -2
- package/examples/11-control-plane/README.md +4 -5
- package/examples/aws/README.md +5 -8
- package/examples/control-plane/README.md +8 -9
- package/examples/control-plane/fixtures/control-plane.seed.json +8 -32
- package/examples/control-plane/public/app.js +57 -18
- package/examples/control-plane/public/index.html +15 -6
- package/examples/control-plane/server.mjs +15 -3
- package/examples/eval-results/public.results.passing.json +6 -21
- package/examples/providers/ai-sdk-next/README.md +11 -8
- package/examples/providers/anthropic-claude/README.md +5 -6
- package/examples/providers/gemini/README.md +6 -6
- package/examples/providers/langchain-agent/README.md +3 -4
- package/examples/providers/langgraph-tool-node/README.md +7 -7
- package/examples/providers/mcp/README.md +8 -8
- package/examples/providers/provider-conformance/README.md +3 -3
- package/examples/providers/semantic-kernel-mcp/README.md +6 -6
- package/examples/providers/semantic-kernel-openapi/README.md +6 -7
- package/examples/runtime-support-billing/README.md +8 -9
- package/examples/runtime-support-billing/support-billing-runtime.mjs +8 -4
- package/examples/scheduling/capabilities/scheduling.availability.get.yaml +2 -2
- package/examples/scheduling/capabilities/scheduling.invite.prepare.yaml +2 -2
- package/examples/scheduling/capabilities/scheduling.invite.send.yaml +2 -2
- package/examples/scheduling/entities/AvailabilitySlot.yaml +2 -2
- package/examples/scheduling/entities/MeetingInvite.yaml +2 -2
- package/examples/scheduling/evals/scheduling.invite.prepare.valid.yaml +3 -1
- package/examples/semantic-kernel-mcp/README.md +11 -12
- package/examples/semantic-kernel-openapi/README.md +8 -8
- package/examples/support/capabilities/support.refund.commit_case.yaml +3 -3
- package/examples/support/capabilities/support.refund.prepare_case.yaml +2 -2
- package/examples/support/capabilities/support.ticket.get.yaml +2 -2
- package/examples/support/entities/Ticket.yaml +0 -1
- package/examples/support/eval-results/support.results.passing.json +3 -12
- package/examples/support/evals/support.refund.prepare_case.approval_required.yaml +3 -1
- package/examples/support/evals/support.refund.prepare_case.valid.yaml +3 -2
- package/examples/support/memory/support.agent.preferences.json +2 -6
- package/examples/support/openai/context.support_agent.json +1 -4
- package/examples/support/provenance/support.refund.summary.provenance.json +2 -6
- package/examples/support/replay/support.refund.approval_required.trace.json +4 -14
- package/package.json +22 -5
- package/schemas/audit/action-record.schema.json +4 -1
- package/schemas/aws/approval-task.schema.json +8 -1
- package/schemas/conformance/conformance-case.schema.json +18 -2
- package/schemas/conformance/conformance-report.schema.json +17 -2
- package/schemas/conformance/provider-target-matrix.schema.json +35 -4
- package/schemas/controls/control-decision.schema.json +7 -1
- package/schemas/eval-case.schema.json +8 -1
- package/schemas/eval-result.schema.json +8 -1
- package/schemas/evidence/evidence-pack.schema.json +34 -5
- package/schemas/governance/compatibility-diff.schema.json +8 -1
- package/schemas/governance/lifecycle-transition.schema.json +10 -1
- package/schemas/governance/risk-compilation-result.schema.json +9 -1
- package/schemas/provenance/generated-content-provenance.schema.json +3 -1
- package/schemas/replay/replay-result.schema.json +16 -2
- package/schemas/replay/replay-trace.schema.json +11 -2
- package/schemas/security/context-segment.schema.json +21 -12
- package/schemas/security/redaction-policy.schema.json +4 -1
- package/schemas/security-packs/coverage-report.schema.json +17 -2
- package/schemas/security-packs/security-case-suite.schema.json +11 -1
- package/schemas/tool-result-envelope.schema.json +9 -1
- package/security-packs/approval_bypass.yaml +12 -2
- package/security-packs/capability_spoofing.yaml +17 -3
- package/security-packs/cost_amplification.yaml +21 -4
- package/security-packs/cross_tenant_access.yaml +16 -3
- package/security-packs/insecure_output_rendering.yaml +15 -4
- package/security-packs/mcp_tool_abuse.yaml +17 -3
- package/security-packs/memory_scope_violation.yaml +19 -4
- package/security-packs/prompt_injection_direct.yaml +19 -4
- package/security-packs/prompt_injection_indirect.yaml +18 -3
- package/security-packs/provider_payload_exposure.yaml +26 -4
- package/security-packs/retrieval_poisoning.yaml +5 -2
- package/security-packs/schema_confusion.yaml +17 -3
- package/security-packs/sensitive_data_disclosure.yaml +22 -5
- package/security-packs/tool_exfiltration.yaml +20 -4
- package/security-packs/tool_result_poisoning.yaml +19 -4
- package/security-packs/unsafe_commit_attempt.yaml +6 -3
package/docs/governance/index.md
CHANGED
|
@@ -1,34 +1,33 @@
|
|
|
1
1
|
# AICF Governance
|
|
2
2
|
|
|
3
|
-
AICF governance is a pure contract and analysis layer. It helps maintainers
|
|
4
|
-
|
|
5
|
-
|
|
3
|
+
AICF governance is a pure contract and analysis layer. It helps maintainers review
|
|
4
|
+
capability lifecycle changes, infer minimum risk, compare versions, and understand what
|
|
5
|
+
else a capability touches before it is exposed to AI systems.
|
|
6
6
|
|
|
7
|
-
Governance APIs do not call models, execute handlers, mutate manifests, write
|
|
8
|
-
|
|
9
|
-
|
|
7
|
+
Governance APIs do not call models, execute handlers, mutate manifests, write stores, or
|
|
8
|
+
replace your policy engine. They return structured decisions that a host app, CI job, or
|
|
9
|
+
release process can act on.
|
|
10
10
|
|
|
11
|
-
Runtime controls are separate from governance analysis. Governance can report
|
|
12
|
-
|
|
11
|
+
Runtime controls are separate from governance analysis. Governance can report active
|
|
12
|
+
kill switches or circuit breakers as promotion blockers; the optional
|
|
13
13
|
`ai-capability-framework/controls` subpath evaluates those controls at runtime.
|
|
14
14
|
|
|
15
|
-
Governed memory is also separate. The optional
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
context. It does not store or recall memory.
|
|
15
|
+
Governed memory is also separate. The optional `ai-capability-framework/memory` subpath
|
|
16
|
+
checks host-owned memory summaries for use-case, scope, expiry, consent, and sensitivity
|
|
17
|
+
before they become model context. It does not store or recall memory.
|
|
19
18
|
|
|
20
|
-
Capability-aware security packs are also separate runtime/test artifacts, but
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
19
|
+
Capability-aware security packs are also separate runtime/test artifacts, but their
|
|
20
|
+
assignment metadata lives under `extensions.governance.security_packs`. Explicit waivers
|
|
21
|
+
live under `extensions.governance.security_pack_waivers` and must include `pack_id`,
|
|
22
|
+
`reason`, `reviewer`, and `reviewed_at`. See
|
|
24
23
|
[security packs](../security/security-packs.md).
|
|
25
24
|
|
|
26
25
|
## Governance Gate
|
|
27
26
|
|
|
28
|
-
`aicf gate` runs the release-oriented checks together: manifest and fixture
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
27
|
+
`aicf gate` runs the release-oriented checks together: manifest and fixture validation,
|
|
28
|
+
semantic invariants, risk, lifecycle posture, optional compatibility baseline checks,
|
|
29
|
+
impact analysis, eval coverage, security-pack coverage, configured provider conformance,
|
|
30
|
+
and public artifact hygiene.
|
|
32
31
|
|
|
33
32
|
```bash
|
|
34
33
|
aicf gate examples --env production --format json
|
|
@@ -38,31 +37,31 @@ See [governance gate](gate.md) for config, defaults, and CI exit codes.
|
|
|
38
37
|
|
|
39
38
|
## Control Plane
|
|
40
39
|
|
|
41
|
-
`ai-capability-framework/control-plane` provides an optional self-hostable API
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
40
|
+
`ai-capability-framework/control-plane` provides an optional self-hostable API and local
|
|
41
|
+
reference UI for reviewing governance outputs, policy decision records, action/approval
|
|
42
|
+
queues, runtime controls, eval/security/conformance status, and redacted replay
|
|
43
|
+
metadata.
|
|
45
44
|
|
|
46
|
-
The control plane is not a hosted service and does not provide production auth,
|
|
47
|
-
|
|
48
|
-
|
|
45
|
+
The control plane is not a hosted service and does not provide production auth, durable
|
|
46
|
+
storage, workflow orchestration, model calls, provider SDK execution, or side-effect
|
|
47
|
+
execution. See [control plane](../control-plane.md).
|
|
49
48
|
|
|
50
49
|
## Lifecycle
|
|
51
50
|
|
|
52
|
-
Governance lifecycle status is separate from the current manifest `status`
|
|
53
|
-
|
|
54
|
-
|
|
51
|
+
Governance lifecycle status is separate from the current manifest `status` field. When a
|
|
52
|
+
command needs a starting status and `--from` is not supplied, AICF maps current manifest
|
|
53
|
+
status like this:
|
|
55
54
|
|
|
56
55
|
| Manifest status | Governance status |
|
|
57
|
-
|
|
|
58
|
-
| `draft`
|
|
59
|
-
| `experimental`
|
|
60
|
-
| `active`
|
|
61
|
-
| `deprecated`
|
|
62
|
-
| `disabled`
|
|
56
|
+
| --------------- | ----------------- |
|
|
57
|
+
| `draft` | `draft` |
|
|
58
|
+
| `experimental` | `review` |
|
|
59
|
+
| `active` | `production` |
|
|
60
|
+
| `deprecated` | `deprecated` |
|
|
61
|
+
| `disabled` | `disabled` |
|
|
63
62
|
|
|
64
|
-
Supported governance statuses are `draft`, `review`, `approved`, `canary`,
|
|
65
|
-
`
|
|
63
|
+
Supported governance statuses are `draft`, `review`, `approved`, `canary`, `production`,
|
|
64
|
+
`deprecated`, `disabled`, and `removed`.
|
|
66
65
|
|
|
67
66
|
Example:
|
|
68
67
|
|
|
@@ -74,26 +73,26 @@ aicf governance lifecycle examples \
|
|
|
74
73
|
--reason "Ready for controlled review"
|
|
75
74
|
```
|
|
76
75
|
|
|
77
|
-
The result includes `allowed`, blocking reasons, warnings, and required actions.
|
|
78
|
-
|
|
76
|
+
The result includes `allowed`, blocking reasons, warnings, and required actions. No
|
|
77
|
+
files are changed.
|
|
79
78
|
|
|
80
79
|
## Risk Compilation
|
|
81
80
|
|
|
82
|
-
Risk compilation compares the declared `risk_tier` with an inferred minimum risk
|
|
83
|
-
|
|
84
|
-
|
|
81
|
+
Risk compilation compares the declared `risk_tier` with an inferred minimum risk from
|
|
82
|
+
side effects, lifecycle metadata, auth scope, entity classifications, and policy
|
|
83
|
+
controls.
|
|
85
84
|
|
|
86
85
|
```bash
|
|
87
86
|
aicf governance risk examples --capability support.refund.prepare_case
|
|
88
87
|
```
|
|
89
88
|
|
|
90
|
-
Missing safety controls are blocking. Catalogue-quality gaps, such as missing
|
|
91
|
-
|
|
89
|
+
Missing safety controls are blocking. Catalogue-quality gaps, such as missing usage
|
|
90
|
+
guidance, remain warnings unless `--strict` is used.
|
|
92
91
|
|
|
93
92
|
## Compatibility
|
|
94
93
|
|
|
95
|
-
Compatibility compares two versions of a capability manifest and classifies the
|
|
96
|
-
|
|
94
|
+
Compatibility compares two versions of a capability manifest and classifies the change
|
|
95
|
+
as `compatible`, `requires_minor`, or `breaking`.
|
|
97
96
|
|
|
98
97
|
```bash
|
|
99
98
|
aicf governance compatibility \
|
|
@@ -101,21 +100,21 @@ aicf governance compatibility \
|
|
|
101
100
|
--after examples/support/capabilities/support.refund.prepare_case.yaml
|
|
102
101
|
```
|
|
103
102
|
|
|
104
|
-
Breaking changes include adding required input fields, removing inputs, lowering
|
|
105
|
-
|
|
106
|
-
|
|
103
|
+
Breaking changes include adding required input fields, removing inputs, lowering risk,
|
|
104
|
+
lowering approval requirements, adding side effects, or removing idempotency/audit
|
|
105
|
+
controls.
|
|
107
106
|
|
|
108
107
|
## Impact
|
|
109
108
|
|
|
110
|
-
Impact analysis reports directly affected capabilities, entities, evals,
|
|
111
|
-
|
|
109
|
+
Impact analysis reports directly affected capabilities, entities, evals, provider
|
|
110
|
+
exports, policy references, and coverage gaps from the loaded registry.
|
|
112
111
|
|
|
113
112
|
```bash
|
|
114
113
|
aicf governance impact examples --capability support.refund.prepare_case
|
|
115
114
|
```
|
|
116
115
|
|
|
117
|
-
F1 impact analysis is registry-local. Trace, tenant, or production-store impact
|
|
118
|
-
|
|
116
|
+
F1 impact analysis is registry-local. Trace, tenant, or production-store impact can be
|
|
117
|
+
supplied by later host integrations, but no database is required.
|
|
119
118
|
|
|
120
119
|
## TypeScript
|
|
121
120
|
|
|
@@ -1,8 +1,7 @@
|
|
|
1
1
|
# Lifecycle
|
|
2
2
|
|
|
3
|
-
Lifecycle checks decide whether a capability can move between governance
|
|
4
|
-
|
|
5
|
-
disabled, and removed.
|
|
3
|
+
Lifecycle checks decide whether a capability can move between governance statuses such
|
|
4
|
+
as draft, review, approved, canary, production, deprecated, disabled, and removed.
|
|
6
5
|
|
|
7
6
|
Use:
|
|
8
7
|
|
|
@@ -10,5 +9,5 @@ Use:
|
|
|
10
9
|
node dist/cli.js governance lifecycle examples --capability support.refund.prepare_case --to production
|
|
11
10
|
```
|
|
12
11
|
|
|
13
|
-
Lifecycle decisions are structured recommendations. AICF does not mutate
|
|
14
|
-
|
|
12
|
+
Lifecycle decisions are structured recommendations. AICF does not mutate manifests or
|
|
13
|
+
deploy capabilities.
|
|
@@ -1,8 +1,7 @@
|
|
|
1
1
|
# Governance Overview
|
|
2
2
|
|
|
3
|
-
AICF governance checks lifecycle transitions, inferred risk, compatibility,
|
|
4
|
-
|
|
5
|
-
artifact hygiene.
|
|
3
|
+
AICF governance checks lifecycle transitions, inferred risk, compatibility, impact, eval
|
|
4
|
+
coverage, security-pack coverage, provider conformance, and public artifact hygiene.
|
|
6
5
|
|
|
7
6
|
Start with:
|
|
8
7
|
|
|
@@ -1,8 +1,7 @@
|
|
|
1
1
|
# Risk Compiler
|
|
2
2
|
|
|
3
|
-
The risk compiler compares declared risk to inferred minimum risk from side
|
|
4
|
-
|
|
5
|
-
metadata.
|
|
3
|
+
The risk compiler compares declared risk to inferred minimum risk from side effects,
|
|
4
|
+
lifecycle, policy, idempotency, audit, auth scoping, and entity metadata.
|
|
6
5
|
|
|
7
6
|
Use:
|
|
8
7
|
|
|
@@ -10,5 +9,5 @@ Use:
|
|
|
10
9
|
node dist/cli.js governance risk examples --format text
|
|
11
10
|
```
|
|
12
11
|
|
|
13
|
-
Safety and runtime correctness issues are errors. Quality or migration gaps are
|
|
14
|
-
|
|
12
|
+
Safety and runtime correctness issues are errors. Quality or migration gaps are warnings
|
|
13
|
+
unless they affect safety.
|
|
@@ -1,7 +1,6 @@
|
|
|
1
1
|
# Host Responsibilities
|
|
2
2
|
|
|
3
|
-
AICF defines contracts and deterministic checks. Host applications own runtime
|
|
4
|
-
behavior.
|
|
3
|
+
AICF defines contracts and deterministic checks. Host applications own runtime behavior.
|
|
5
4
|
|
|
6
5
|
Host applications MUST provide or enforce:
|
|
7
6
|
|
|
@@ -10,14 +9,13 @@ Host applications MUST provide or enforce:
|
|
|
10
9
|
- policy facts used by `deny_if` and `approval_required_if`;
|
|
11
10
|
- approval collection, approval verification, and approval records;
|
|
12
11
|
- handler registries and capability execution;
|
|
13
|
-
- side effects such as writes, sends, refunds, workflow starts, and permission
|
|
14
|
-
changes;
|
|
12
|
+
- side effects such as writes, sends, refunds, workflow starts, and permission changes;
|
|
15
13
|
- idempotency storage, replay protection, and conflict handling;
|
|
16
14
|
- durable action state, verification state, and audit logs;
|
|
17
15
|
- trace redaction and provider payload retention policies.
|
|
18
16
|
|
|
19
|
-
AICF decision results and audit previews are inputs to host control logic. They
|
|
20
|
-
|
|
17
|
+
AICF decision results and audit previews are inputs to host control logic. They are not
|
|
18
|
+
proof that a real user is authorized and are not durable audit records.
|
|
21
19
|
|
|
22
|
-
Public examples in this repository are synthetic. Do not replace host security
|
|
23
|
-
|
|
20
|
+
Public examples in this repository are synthetic. Do not replace host security controls
|
|
21
|
+
with example permissions, IDs, or policy facts.
|
|
@@ -1,26 +1,23 @@
|
|
|
1
1
|
# F0 Baseline Gaps
|
|
2
2
|
|
|
3
|
-
This note records the F0 baseline reconciliation result for the public
|
|
4
|
-
|
|
3
|
+
This note records the F0 baseline reconciliation result for the public repository. It is
|
|
4
|
+
intentionally limited to public implementation facts.
|
|
5
5
|
|
|
6
6
|
## Findings
|
|
7
7
|
|
|
8
|
-
- Core repairs are present: schemas, manifest loading, registry construction,
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
- Runtime APIs are present under the runtime subpath and provide context,
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
- Provider APIs are present under optional subpaths for OpenAI, Anthropic,
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
- Release hygiene exists through package contents checks, public artifact
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
- The
|
|
21
|
-
compatibility.
|
|
22
|
-
- The current public architecture is documented in
|
|
23
|
-
`docs/architecture/current-state.md`.
|
|
8
|
+
- Core repairs are present: schemas, manifest loading, registry construction, capability
|
|
9
|
+
decisions, adapter exports, tool-result envelopes, and eval scoring are covered by
|
|
10
|
+
deterministic tests.
|
|
11
|
+
- Runtime APIs are present under the runtime subpath and provide context, routing,
|
|
12
|
+
policy, handler execution, action lifecycle, in-memory stores, audit, redaction, and
|
|
13
|
+
envelope helpers.
|
|
14
|
+
- Provider APIs are present under optional subpaths for OpenAI, Anthropic, Gemini, AI
|
|
15
|
+
SDK, LangChain, MCP, Semantic Kernel, AWS, observability, live eval, Promptfoo,
|
|
16
|
+
Langfuse, and MCP server integrations.
|
|
17
|
+
- Release hygiene exists through package contents checks, public artifact checks,
|
|
18
|
+
workspace hygiene checks, source archive checks, and clean install smoke tests.
|
|
19
|
+
- The explicit `npm run typecheck` command has been added for F0 command compatibility.
|
|
20
|
+
- The current public architecture is documented in `docs/architecture/current-state.md`.
|
|
24
21
|
|
|
25
22
|
## Result
|
|
26
23
|
|
package/docs/index.md
CHANGED
|
@@ -1,20 +1,29 @@
|
|
|
1
1
|
# AICF Documentation
|
|
2
2
|
|
|
3
|
-
AI Capability Framework is a governed capability layer for
|
|
4
|
-
application functionality. AICF is not an agent framework.
|
|
3
|
+
AI Capability Framework (AICF) is a provider-agnostic governed capability layer for
|
|
4
|
+
AI-accessible application functionality. AICF is not an agent framework.
|
|
5
5
|
|
|
6
6
|
Models propose; applications validate, authorize, execute, and audit.
|
|
7
7
|
|
|
8
8
|
## Start
|
|
9
9
|
|
|
10
10
|
1. [Installation](getting-started/installation.md)
|
|
11
|
-
2. [
|
|
12
|
-
3. [
|
|
13
|
-
4. [
|
|
14
|
-
5. [
|
|
11
|
+
2. [Provider-neutral quickstart](getting-started/provider-neutral-quickstart.md)
|
|
12
|
+
3. [Quickstart](getting-started/quickstart.md)
|
|
13
|
+
4. [Concepts](getting-started/concepts.md)
|
|
14
|
+
5. [OpenAI quickstart](getting-started/openai-quickstart.md)
|
|
15
|
+
6. [Anthropic quickstart](getting-started/anthropic-quickstart.md)
|
|
16
|
+
7. [Gemini quickstart](getting-started/gemini-quickstart.md)
|
|
17
|
+
8. [OpenAI walkthrough](openai-walkthrough.md)
|
|
18
|
+
9. [Glossary](glossary.md)
|
|
15
19
|
|
|
16
20
|
## Build The Capability Layer
|
|
17
21
|
|
|
22
|
+
- Core contracts define public manifests, schemas, validation, and deterministic
|
|
23
|
+
no-execution decisions.
|
|
24
|
+
- Runtime, providers, governance, evals, security, observability, the control plane, and
|
|
25
|
+
agent skills are optional surfaces around those contracts.
|
|
26
|
+
|
|
18
27
|
- [Capability manifests](core/capability-manifests.md)
|
|
19
28
|
- [Entity manifests](core/entity-manifests.md)
|
|
20
29
|
- [Runtime overview](runtime/runtime-overview.md)
|
|
@@ -25,6 +34,7 @@ Models propose; applications validate, authorize, execute, and audit.
|
|
|
25
34
|
## Connect Providers
|
|
26
35
|
|
|
27
36
|
- [Provider overview](providers.md)
|
|
37
|
+
- [Choose a runtime](providers/choose-a-runtime.md)
|
|
28
38
|
- [OpenAI](providers/openai.md)
|
|
29
39
|
- [Anthropic Claude](providers/anthropic.md)
|
|
30
40
|
- [Google Gemini](providers/gemini.md)
|
|
@@ -54,7 +64,12 @@ Models propose; applications validate, authorize, execute, and audit.
|
|
|
54
64
|
- [Control plane](control-plane/overview.md)
|
|
55
65
|
- [Release process](public-framework/release-process.md)
|
|
56
66
|
- [Final v1 certification](public-framework/v1-certification.md)
|
|
67
|
+
- [Final certification matrix](public-framework/final-certification-matrix.md)
|
|
68
|
+
- [npm release preflight](public/npm-release-preflight.md)
|
|
69
|
+
- [License decision](public-framework/license-decision.md)
|
|
70
|
+
- [Dependency license exceptions](public/license-exceptions.md)
|
|
57
71
|
- [Compatibility policy](public-framework/compatibility-policy.md)
|
|
58
72
|
- [Deprecation policy](public-framework/deprecation-policy.md)
|
|
59
73
|
- [Security disclosure](public-framework/security-disclosure.md)
|
|
60
74
|
- [API reference](api.md)
|
|
75
|
+
- [Public API policy](api/public-api-policy.md)
|
package/docs/interoperability.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# Interoperability
|
|
2
2
|
|
|
3
|
-
AICF describes AI-facing application capabilities. Other frameworks may execute
|
|
4
|
-
|
|
3
|
+
AICF describes AI-facing application capabilities. Other frameworks may execute or
|
|
4
|
+
orchestrate those capabilities.
|
|
5
5
|
|
|
6
6
|
Use AICF as the portable contract layer for:
|
|
7
7
|
|
|
@@ -22,11 +22,11 @@ Use host or adjacent runtime frameworks for:
|
|
|
22
22
|
- durable audit pipelines;
|
|
23
23
|
- production observability.
|
|
24
24
|
|
|
25
|
-
When integrating with another runtime, keep the boundary explicit: AICF can
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
25
|
+
When integrating with another runtime, keep the boundary explicit: AICF can validate and
|
|
26
|
+
decide whether a capability request is allowed, approval-required, or denied, but the
|
|
27
|
+
runtime must still enforce real auth, produce facts, execute or refuse handlers, and
|
|
28
|
+
persist records.
|
|
29
29
|
|
|
30
|
-
See [the adapter guide](adapters.md) for implemented provider/runtime adapters
|
|
31
|
-
|
|
32
|
-
|
|
30
|
+
See [the adapter guide](adapters.md) for implemented provider/runtime adapters and
|
|
31
|
+
[the adapter status page](adapter-roadmap.md) for reference links and priority
|
|
32
|
+
rationale.
|
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
# LangChain And LangGraph Bridge
|
|
2
2
|
|
|
3
|
-
The LangChain bridge is an optional provider adapter for applications that
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
3
|
+
The LangChain bridge is an optional provider adapter for applications that already use
|
|
4
|
+
LangChain.js or LangGraph.js. It translates a routed AICF capability slice into
|
|
5
|
+
LangChain tools, executes tool calls through `AicfToolExecutor`, and returns serialized
|
|
6
|
+
model-safe AICF runtime envelopes.
|
|
7
7
|
|
|
8
|
-
It does not install model provider integrations, choose a chat model, expose
|
|
9
|
-
|
|
10
|
-
|
|
8
|
+
It does not install model provider integrations, choose a chat model, expose commit
|
|
9
|
+
tools, persist state, log raw provider payloads, or make root/runtime imports depend on
|
|
10
|
+
LangChain.
|
|
11
11
|
|
|
12
12
|
## Import
|
|
13
13
|
|
|
@@ -27,47 +27,43 @@ import {
|
|
|
27
27
|
```
|
|
28
28
|
|
|
29
29
|
`createDefaultLangChainToolFactory()` dynamically imports the optional
|
|
30
|
-
`@langchain/core/tools` package. Host applications may instead pass a
|
|
31
|
-
|
|
30
|
+
`@langchain/core/tools` package. Host applications may instead pass a compatible `tool`
|
|
31
|
+
factory.
|
|
32
32
|
|
|
33
33
|
## Tool Bridge
|
|
34
34
|
|
|
35
|
-
`buildLangChainTools()` builds a normal LangChain tool array from an
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
`executeProviderToolCall()`.
|
|
35
|
+
`buildLangChainTools()` builds a normal LangChain tool array from an already-routed AICF
|
|
36
|
+
capability slice. Each generated tool uses a provider-safe name, model-safe description,
|
|
37
|
+
schema, and function that delegates to `executeProviderToolCall()`.
|
|
39
38
|
|
|
40
|
-
LangChain schema validation is advisory. AICF still performs canonical input
|
|
41
|
-
|
|
42
|
-
envelope formatting.
|
|
39
|
+
LangChain schema validation is advisory. AICF still performs canonical input validation,
|
|
40
|
+
policy checks, lifecycle checks, approval handling, and model-safe envelope formatting.
|
|
43
41
|
|
|
44
42
|
Minimal usage is: validate manifests, build a routed runtime slice, register
|
|
45
|
-
read/prepare handlers, create or pass a LangChain tool factory, then give the
|
|
46
|
-
|
|
43
|
+
read/prepare handlers, create or pass a LangChain tool factory, then give the generated
|
|
44
|
+
tools to the host-owned agent or graph.
|
|
47
45
|
|
|
48
46
|
## LangGraph ToolNode
|
|
49
47
|
|
|
50
48
|
`buildLangGraphToolNode()` is a thin wrapper around a host-supplied `ToolNode`
|
|
51
|
-
constructor. AICF does not import `@langchain/langgraph` by default. The helper
|
|
52
|
-
|
|
49
|
+
constructor. AICF does not import `@langchain/langgraph` by default. The helper creates
|
|
50
|
+
the same AICF-backed tools and passes them to the supplied constructor.
|
|
53
51
|
|
|
54
52
|
## Boundary
|
|
55
53
|
|
|
56
|
-
- AICF capability manifests, policy, lifecycle, and runtime envelopes remain the
|
|
57
|
-
|
|
58
|
-
- Commit, destructive, money-moving, and external-send capabilities are not
|
|
59
|
-
|
|
54
|
+
- AICF capability manifests, policy, lifecycle, and runtime envelopes remain the source
|
|
55
|
+
of truth.
|
|
56
|
+
- Commit, destructive, money-moving, and external-send capabilities are not exported by
|
|
57
|
+
default and are not executable through LangChain tool calls.
|
|
60
58
|
- Provider packages such as `@langchain/openai`, `@langchain/anthropic`, and
|
|
61
|
-
`@langchain/google-genai` are host application dependencies, not AICF
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
private diagnostics are not captured by default.
|
|
59
|
+
`@langchain/google-genai` are host application dependencies, not AICF dependencies.
|
|
60
|
+
- Raw prompts, provider request/response payloads, traces, credentials, and private
|
|
61
|
+
diagnostics are not captured by default.
|
|
65
62
|
- Live tests are opt-in with `RUN_LIVE_LANGCHAIN=1` and may use fake
|
|
66
63
|
LangChain-compatible tools instead of a real provider model.
|
|
67
64
|
|
|
68
|
-
Known limitations: no provider-specific LangChain integrations, no model
|
|
69
|
-
|
|
70
|
-
commit path.
|
|
65
|
+
Known limitations: no provider-specific LangChain integrations, no model selection, no
|
|
66
|
+
production storage, no automatic approval UI, and no model-exposed commit path.
|
|
71
67
|
|
|
72
68
|
## Checks
|
|
73
69
|
|
package/docs/live-evals.md
CHANGED
|
@@ -1,61 +1,98 @@
|
|
|
1
1
|
# Live Evals
|
|
2
2
|
|
|
3
|
-
Live evals are optional runtime checks that execute synthetic eval cases through
|
|
4
|
-
|
|
3
|
+
Live evals are optional runtime checks that execute synthetic eval cases through a
|
|
4
|
+
caller-provided provider runner. They are provider-neutral by default and keep OpenAI as
|
|
5
|
+
one convenience adapter. Import them from:
|
|
5
6
|
|
|
6
7
|
```ts
|
|
7
8
|
import {
|
|
9
|
+
createOpenAILiveEvalRunner,
|
|
8
10
|
evaluateGate,
|
|
9
11
|
runLiveEvalSuite
|
|
10
12
|
} from "ai-capability-framework/evals-live";
|
|
11
13
|
```
|
|
12
14
|
|
|
13
|
-
Default tests use mock clients. No live model call runs in normal CI.
|
|
14
|
-
|
|
15
|
-
without model credentials.
|
|
15
|
+
Default tests use mock clients. No live model call runs in normal CI. The public runtime
|
|
16
|
+
support/billing example is separate from live evals and runs without model credentials.
|
|
16
17
|
|
|
17
18
|
## Deterministic Fixtures Vs Live Evals
|
|
18
19
|
|
|
19
|
-
The Core eval runner scores public candidate-result fixtures without API keys.
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
20
|
+
The Core eval runner scores public candidate-result fixtures without API keys. Live
|
|
21
|
+
evals run the same kind of behavior through the runtime path first, then summarize the
|
|
22
|
+
result into candidate fields such as selected capabilities, tool calls, policy status,
|
|
23
|
+
action state, committed capabilities, and final text.
|
|
23
24
|
|
|
24
|
-
The live runner reuses deterministic scorer semantics where possible. It does
|
|
25
|
-
|
|
25
|
+
The live runner reuses deterministic scorer semantics where possible. It does not add
|
|
26
|
+
LLM-as-judge scoring in this phase.
|
|
26
27
|
|
|
27
28
|
## Running A Suite
|
|
28
29
|
|
|
29
30
|
```ts
|
|
31
|
+
const runner = createOpenAILiveEvalRunner({
|
|
32
|
+
client: openAIClient,
|
|
33
|
+
model: "gpt-4.1-mini"
|
|
34
|
+
});
|
|
35
|
+
|
|
30
36
|
const results = await runLiveEvalSuite({
|
|
31
37
|
cases,
|
|
32
38
|
contextBuilderFactory: () => contextBuilder,
|
|
33
39
|
executor,
|
|
34
|
-
model: "gpt-4.1-mini",
|
|
35
|
-
openAIClient,
|
|
36
40
|
registry,
|
|
37
|
-
router
|
|
41
|
+
router,
|
|
42
|
+
runner
|
|
38
43
|
});
|
|
39
44
|
|
|
40
45
|
const gate = evaluateGate(results);
|
|
41
46
|
```
|
|
42
47
|
|
|
43
|
-
The default gate fails when any eval errors, a safety/policy scorer fails, or
|
|
44
|
-
|
|
48
|
+
The default gate fails when any eval errors, a safety/policy scorer fails, or the
|
|
49
|
+
average score is below `0.90`.
|
|
50
|
+
|
|
51
|
+
For source-compatible OpenAI callers, `runLiveEvalSuite({ openAIClient, model, ... })`
|
|
52
|
+
still works. New code should pass an explicit runner. Convenience runner factories are
|
|
53
|
+
available for OpenAI Responses, Anthropic Messages, Gemini GenerateContent, and Vercel
|
|
54
|
+
AI SDK `generateText`.
|
|
55
|
+
|
|
56
|
+
Non-OpenAI example:
|
|
57
|
+
|
|
58
|
+
```ts
|
|
59
|
+
import { createAnthropicLiveEvalRunner } from "ai-capability-framework/evals-live";
|
|
60
|
+
|
|
61
|
+
const runner = createAnthropicLiveEvalRunner({
|
|
62
|
+
client: anthropicClient,
|
|
63
|
+
model: "claude-3-5-sonnet-latest"
|
|
64
|
+
});
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
LangChain/LangGraph and MCP hosts can implement `AicfLiveEvalRunner` directly around
|
|
68
|
+
their own runtime loop. AICF does not expose commit tools to any live eval runner.
|
|
69
|
+
|
|
70
|
+
## CLI Live Evals
|
|
71
|
+
|
|
72
|
+
The CLI is for intentional live checks only:
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
RUN_REAL_AICF_LIVE_EVALS=1 \
|
|
76
|
+
OPENAI_API_KEY=... \
|
|
77
|
+
aicf eval-live examples --cases cases.json --provider openai --model gpt-4.1-mini
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Supported CLI providers are `openai`, `anthropic`, and `gemini`. The AI SDK path is
|
|
81
|
+
available through TypeScript because host applications must supply `generateText` and a
|
|
82
|
+
configured model object. The CLI refuses to run live evals unless
|
|
83
|
+
`RUN_REAL_AICF_LIVE_EVALS=1` is set.
|
|
45
84
|
|
|
46
85
|
## Trace To Eval
|
|
47
86
|
|
|
48
|
-
`createEvalCaseFromTrace()` creates a draft public-safe eval case from an
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
safety.
|
|
87
|
+
`createEvalCaseFromTrace()` creates a draft public-safe eval case from an OpenAI runtime
|
|
88
|
+
result. It excludes raw user text and raw model output by default. Include model output
|
|
89
|
+
only when the host has reviewed it for public safety.
|
|
52
90
|
|
|
53
91
|
## Promptfoo Export
|
|
54
92
|
|
|
55
93
|
Promptfoo support is available from `ai-capability-framework/promptfoo`.
|
|
56
|
-
`exportPromptfooSuite()` generates files only; it does not run Promptfoo. The
|
|
57
|
-
|
|
58
|
-
keys.
|
|
94
|
+
`exportPromptfooSuite()` generates files only; it does not run Promptfoo. The default
|
|
95
|
+
provider is `echo` so generated suites can be inspected without API keys.
|
|
59
96
|
|
|
60
97
|
The CLI command writes generated files when explicitly requested:
|
|
61
98
|
|
|
@@ -63,18 +100,17 @@ The CLI command writes generated files when explicitly requested:
|
|
|
63
100
|
aicf export promptfoo examples --out ./promptfoo
|
|
64
101
|
```
|
|
65
102
|
|
|
66
|
-
`importPromptfooResults()` maps basic Promptfoo JSON results back into live eval
|
|
67
|
-
|
|
103
|
+
`importPromptfooResults()` maps basic Promptfoo JSON results back into live eval result
|
|
104
|
+
summaries.
|
|
68
105
|
|
|
69
106
|
Capability-aware security packs have a focused Promptfoo export under
|
|
70
107
|
`ai-capability-framework/security-packs`. That export generates red-team config
|
|
71
|
-
templates with `echo` as the default provider and placeholders for host runtime
|
|
72
|
-
|
|
73
|
-
files.
|
|
108
|
+
templates with `echo` as the default provider and placeholders for host runtime targets.
|
|
109
|
+
AICF does not run Promptfoo or call providers when exporting those files.
|
|
74
110
|
|
|
75
111
|
## EvalOps Exports
|
|
76
112
|
|
|
77
|
-
`ai-capability-framework/evalops` provides dependency-free JSON exporters and
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
113
|
+
`ai-capability-framework/evalops` provides dependency-free JSON exporters and importers
|
|
114
|
+
for Braintrust-style and OpenAI-eval-style datasets/results. These helpers transform
|
|
115
|
+
public-safe eval cases and summarized results only. They do not upload data, call
|
|
116
|
+
external services, or require provider SDKs.
|