ai-capability-framework 1.0.0-rc.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +37 -0
- package/CONTRIBUTING.md +56 -0
- package/LICENSE +21 -0
- package/README.md +198 -0
- package/SECURITY.md +29 -0
- package/conformance/invalid/duplicate/capabilities/conformance.duplicate.one.yaml +55 -0
- package/conformance/invalid/duplicate/capabilities/conformance.duplicate.two.yaml +55 -0
- package/conformance/invalid/eval-results/conformance.results.failing.json +15 -0
- package/conformance/invalid/missing-reference/capabilities/conformance.missing_reference.prepare.yaml +58 -0
- package/conformance/invalid/schema/capabilities/conformance.invalid.missing_required.yaml +3 -0
- package/conformance/valid/capabilities/conformance.note.get.yaml +82 -0
- package/conformance/valid/decisions/conformance.note.get.denied_missing_permission.json +13 -0
- package/conformance/valid/entities/Note.yaml +18 -0
- package/conformance/valid/eval-results/conformance.results.passing.json +19 -0
- package/conformance/valid/evals/conformance.note.get.valid.yaml +33 -0
- package/conformance/valid/openai/context.no_permissions.json +6 -0
- package/dist/adapter-common.d.ts +62 -0
- package/dist/adapter-common.d.ts.map +1 -0
- package/dist/adapter-common.js +453 -0
- package/dist/ai-sdk.d.ts +5 -0
- package/dist/ai-sdk.d.ts.map +1 -0
- package/dist/ai-sdk.js +71 -0
- package/dist/anthropic-claude.d.ts +5 -0
- package/dist/anthropic-claude.d.ts.map +1 -0
- package/dist/anthropic-claude.js +62 -0
- package/dist/cli.d.ts +11 -0
- package/dist/cli.d.ts.map +1 -0
- package/dist/cli.js +360 -0
- package/dist/decision.d.ts +6 -0
- package/dist/decision.d.ts.map +1 -0
- package/dist/decision.js +237 -0
- package/dist/eval-runner.d.ts +6 -0
- package/dist/eval-runner.d.ts.map +1 -0
- package/dist/eval-runner.js +372 -0
- package/dist/gemini.d.ts +5 -0
- package/dist/gemini.d.ts.map +1 -0
- package/dist/gemini.js +64 -0
- package/dist/generated/manifest-types.d.ts +208 -0
- package/dist/generated/manifest-types.d.ts.map +1 -0
- package/dist/generated/manifest-types.js +1 -0
- package/dist/index.d.ts +15 -0
- package/dist/index.d.ts.map +1 -0
- package/dist/index.js +13 -0
- package/dist/langchain.d.ts +5 -0
- package/dist/langchain.d.ts.map +1 -0
- package/dist/langchain.js +65 -0
- package/dist/loader.d.ts +4 -0
- package/dist/loader.d.ts.map +1 -0
- package/dist/loader.js +94 -0
- package/dist/mcp.d.ts +5 -0
- package/dist/mcp.d.ts.map +1 -0
- package/dist/mcp.js +82 -0
- package/dist/openai-responses.d.ts +6 -0
- package/dist/openai-responses.d.ts.map +1 -0
- package/dist/openai-responses.js +65 -0
- package/dist/registry.d.ts +6 -0
- package/dist/registry.d.ts.map +1 -0
- package/dist/registry.js +134 -0
- package/dist/semantic-kernel.d.ts +5 -0
- package/dist/semantic-kernel.d.ts.map +1 -0
- package/dist/semantic-kernel.js +79 -0
- package/dist/types.d.ts +400 -0
- package/dist/types.d.ts.map +1 -0
- package/dist/types.js +1 -0
- package/dist/validator.d.ts +3 -0
- package/dist/validator.d.ts.map +1 -0
- package/dist/validator.js +105 -0
- package/docs/adapter-roadmap.md +179 -0
- package/docs/adapters.md +112 -0
- package/docs/api.md +115 -0
- package/docs/capability-manifests.md +47 -0
- package/docs/control-plane.md +59 -0
- package/docs/eval-runner.md +80 -0
- package/docs/evals.md +34 -0
- package/docs/getting-started.md +51 -0
- package/docs/host-responsibilities.md +23 -0
- package/docs/interoperability.md +32 -0
- package/docs/migration-0.1-to-1.0.md +22 -0
- package/docs/openai-responses.md +103 -0
- package/docs/release.md +75 -0
- package/docs/spec.md +194 -0
- package/examples/eval-results/public.results.passing.json +128 -0
- package/examples/scheduling/capabilities/scheduling.availability.get.yaml +91 -0
- package/examples/scheduling/capabilities/scheduling.invite.prepare.yaml +114 -0
- package/examples/scheduling/capabilities/scheduling.invite.send.yaml +97 -0
- package/examples/scheduling/decisions/scheduling.availability.get.select.json +14 -0
- package/examples/scheduling/decisions/scheduling.invite.prepare.allowed.json +20 -0
- package/examples/scheduling/decisions/scheduling.invite.send.allowed.json +22 -0
- package/examples/scheduling/decisions/scheduling.invite.send.denied_missing_approval.json +18 -0
- package/examples/scheduling/entities/AvailabilitySlot.yaml +26 -0
- package/examples/scheduling/entities/MeetingInvite.yaml +25 -0
- package/examples/scheduling/eval-results/scheduling.results.passing.json +59 -0
- package/examples/scheduling/evals/scheduling.availability.get.valid.yaml +38 -0
- package/examples/scheduling/evals/scheduling.invite.prepare.valid.yaml +44 -0
- package/examples/scheduling/evals/scheduling.invite.send.requires_approval.yaml +43 -0
- package/examples/scheduling/openai/context.scheduler.json +6 -0
- package/examples/support/capabilities/support.refund.commit_case.yaml +98 -0
- package/examples/support/capabilities/support.refund.prepare_case.yaml +116 -0
- package/examples/support/capabilities/support.ticket.get.yaml +85 -0
- package/examples/support/decisions/support.refund.commit_case.allowed.json +22 -0
- package/examples/support/decisions/support.refund.commit_case.denied_missing_approval.json +18 -0
- package/examples/support/decisions/support.refund.prepare_case.approval_required.json +19 -0
- package/examples/support/decisions/support.ticket.get.select.json +0 -0
- package/examples/support/entities/Order.yaml +23 -0
- package/examples/support/entities/Ticket.yaml +25 -0
- package/examples/support/eval-results/support.results.passing.json +68 -0
- package/examples/support/evals/support.refund.commit_case.requires_approval.yaml +42 -0
- package/examples/support/evals/support.refund.prepare_case.approval_required.yaml +45 -0
- package/examples/support/evals/support.refund.prepare_case.valid.yaml +43 -0
- package/examples/support/openai/context.support_agent.json +9 -0
- package/package.json +74 -0
- package/schemas/capability-manifest.schema.json +250 -0
- package/schemas/entity-manifest.schema.json +97 -0
- package/schemas/eval-case.schema.json +134 -0
- package/schemas/eval-result.schema.json +77 -0
|
@@ -0,0 +1,179 @@
|
|
|
1
|
+
# Adapter Status And Roadmap
|
|
2
|
+
|
|
3
|
+
AICF is provider-neutral. Adapters export tool metadata, bindings, parsers, and
|
|
4
|
+
diagnostics, but do not call models, execute handlers, persist actions, collect
|
|
5
|
+
approvals, or perform side effects.
|
|
6
|
+
|
|
7
|
+
## Priority Criteria
|
|
8
|
+
|
|
9
|
+
Adapter priority is based on:
|
|
10
|
+
|
|
11
|
+
- fit with AICF capability manifests and JSON Schema inputs;
|
|
12
|
+
- adoption and usefulness for public developers;
|
|
13
|
+
- ability to preserve the no-execution boundary;
|
|
14
|
+
- deterministic name binding and argument parsing support;
|
|
15
|
+
- implementation size in the TypeScript package.
|
|
16
|
+
|
|
17
|
+
Every adapter should expose the same high-level contract:
|
|
18
|
+
|
|
19
|
+
- `buildXTools(registry, options)` or equivalent export function;
|
|
20
|
+
- `parseXToolCall(toolset, call)` or equivalent parser;
|
|
21
|
+
- deterministic tool-name mapping back to AICF capability IDs;
|
|
22
|
+
- `bindings`, `excluded`, and `diagnostics`;
|
|
23
|
+
- selection gated by `decideCapability(..., operation: "select")`;
|
|
24
|
+
- restricted side-effect tools excluded by default;
|
|
25
|
+
- schema normalization that fails closed when the target cannot represent an
|
|
26
|
+
AICF input schema safely.
|
|
27
|
+
|
|
28
|
+
## Priority Order
|
|
29
|
+
|
|
30
|
+
| Priority | Adapter | Why It Comes Here | Primary Output |
|
|
31
|
+
| --- | --- | --- | --- |
|
|
32
|
+
| Priority | Adapter | Status | Primary Output |
|
|
33
|
+
| --- | --- | --- | --- |
|
|
34
|
+
| P1 | Anthropic Claude Messages tools | Implemented | Claude client tool definitions plus parser for `tool_use` blocks. |
|
|
35
|
+
| P2 | Google Gemini function calling | Implemented | Gemini `functionDeclarations` plus parser for `functionCall` objects. |
|
|
36
|
+
| P3 | Vercel AI SDK tools | Implemented | AI SDK tool map with `inputSchema`, `description`, `strict`, and no `execute`. |
|
|
37
|
+
| P4 | Model Context Protocol tool descriptors | Implemented | MCP `Tool` descriptors and parser for `tools/call` requests. |
|
|
38
|
+
| P5 | LangChain/LangGraph tools | Implemented | Structured tool parameter specs and bindings for host-supplied functions. |
|
|
39
|
+
| P6 | Semantic Kernel functions | Implemented | Function/plugin metadata and parser guidance for manual invocation flows. |
|
|
40
|
+
|
|
41
|
+
## P1: Anthropic Claude Messages Tools
|
|
42
|
+
|
|
43
|
+
Target a no-model-call adapter that exports Claude client tool definitions:
|
|
44
|
+
|
|
45
|
+
- `name`: sanitized AICF capability ID, capped at the provider limit.
|
|
46
|
+
- `description`: composed from `model_description`, use guidance, risk,
|
|
47
|
+
autonomy, lifecycle, and side-effect metadata.
|
|
48
|
+
- `input_schema`: normalized AICF `input_schema`.
|
|
49
|
+
- `strict: true` when the schema can be represented safely.
|
|
50
|
+
- optional `input_examples` only when public examples are available and validated.
|
|
51
|
+
|
|
52
|
+
Parser target:
|
|
53
|
+
|
|
54
|
+
- map Claude `tool_use.name` to capability ID;
|
|
55
|
+
- read `tool_use.id` as call ID;
|
|
56
|
+
- validate `tool_use.input` against the original AICF input schema;
|
|
57
|
+
- return structured diagnostics for unknown names, malformed input, and schema
|
|
58
|
+
failures.
|
|
59
|
+
|
|
60
|
+
Reference basis: Claude client tools include `name`, `description`, and
|
|
61
|
+
`input_schema`, can use `strict: true`, and return `tool_use` blocks that the
|
|
62
|
+
host application executes.
|
|
63
|
+
|
|
64
|
+
## P2: Google Gemini Function Calling
|
|
65
|
+
|
|
66
|
+
Target a no-model-call adapter that exports Gemini function declarations:
|
|
67
|
+
|
|
68
|
+
- `name`: sanitized capability ID using underscores or camelCase-safe names.
|
|
69
|
+
- `description`: composed from AICF model-facing and governance metadata.
|
|
70
|
+
- `parameters`: AICF input schema converted to Gemini's OpenAPI-compatible
|
|
71
|
+
schema subset.
|
|
72
|
+
|
|
73
|
+
Parser target:
|
|
74
|
+
|
|
75
|
+
- map `functionCall.name` to capability ID;
|
|
76
|
+
- preserve `functionCall.id` when present;
|
|
77
|
+
- validate `functionCall.args` against the original AICF input schema;
|
|
78
|
+
- report schema keywords that cannot be represented safely in Gemini parameters.
|
|
79
|
+
|
|
80
|
+
The hard part is schema normalization. Gemini function declarations use an
|
|
81
|
+
OpenAPI-compatible subset, so this adapter should fail closed for unsupported
|
|
82
|
+
JSON Schema 2020-12 features rather than silently weakening validation.
|
|
83
|
+
|
|
84
|
+
## P3: Vercel AI SDK Tools
|
|
85
|
+
|
|
86
|
+
Target a TypeScript-first adapter that exports an AI SDK-compatible tool map
|
|
87
|
+
without execution functions:
|
|
88
|
+
|
|
89
|
+
- object keys are deterministic tool names;
|
|
90
|
+
- values include `description`, `inputSchema`, and `strict` when supported;
|
|
91
|
+
- no `execute` function is generated by AICF;
|
|
92
|
+
- bindings preserve capability IDs and original schemas.
|
|
93
|
+
|
|
94
|
+
Parser target:
|
|
95
|
+
|
|
96
|
+
- map AI SDK tool call names to AICF capability IDs;
|
|
97
|
+
- validate tool call input against the original AICF input schema;
|
|
98
|
+
- leave tool result production and execution to the host app.
|
|
99
|
+
|
|
100
|
+
This adapter is attractive because the AI SDK accepts JSON schemas and `execute`
|
|
101
|
+
is optional, allowing host apps to forward calls to a queue, client, or approval
|
|
102
|
+
path instead of executing in the same process.
|
|
103
|
+
|
|
104
|
+
## P4: Model Context Protocol Tool Descriptors
|
|
105
|
+
|
|
106
|
+
Target MCP descriptor export, not an AICF-owned executable MCP server:
|
|
107
|
+
|
|
108
|
+
- `name`: deterministic MCP-safe tool name;
|
|
109
|
+
- `title`: human-readable capability name;
|
|
110
|
+
- `description`: AICF model description plus lifecycle/risk guidance;
|
|
111
|
+
- `inputSchema`: AICF input schema, preserving JSON Schema 2020-12 where valid;
|
|
112
|
+
- `outputSchema`: AICF output schema when representable;
|
|
113
|
+
- `annotations`: conservative hints derived from side effects, such as read-only
|
|
114
|
+
and idempotent hints.
|
|
115
|
+
|
|
116
|
+
Parser target:
|
|
117
|
+
|
|
118
|
+
- map `tools/call.params.name` to capability ID;
|
|
119
|
+
- validate `tools/call.params.arguments`;
|
|
120
|
+
- return an AICF capability request object for the host MCP server or runtime to
|
|
121
|
+
decide and execute.
|
|
122
|
+
|
|
123
|
+
Do not add an MCP server that executes capabilities inside AICF. Host apps may
|
|
124
|
+
use AICF descriptors inside their own MCP server implementation.
|
|
125
|
+
|
|
126
|
+
## P5: LangChain And LangGraph Tools
|
|
127
|
+
|
|
128
|
+
Target descriptor export for host-created tools:
|
|
129
|
+
|
|
130
|
+
- deterministic name;
|
|
131
|
+
- description;
|
|
132
|
+
- schema suitable for LangChain structured tools;
|
|
133
|
+
- binding metadata for capability ID, risk, autonomy, and restriction status.
|
|
134
|
+
|
|
135
|
+
Do not generate callable implementations. LangChain and LangGraph tools are
|
|
136
|
+
runtime functions that can update state or return commands, so the host must
|
|
137
|
+
supply execution, approval, and state handling.
|
|
138
|
+
|
|
139
|
+
This adapter should probably be implemented after the AI SDK adapter, because
|
|
140
|
+
both are TypeScript-oriented but AI SDK has a cleaner descriptor-only path.
|
|
141
|
+
|
|
142
|
+
## P6: Semantic Kernel Functions
|
|
143
|
+
|
|
144
|
+
Target metadata and manual-invocation compatibility:
|
|
145
|
+
|
|
146
|
+
- function/plugin names derived from AICF capability IDs;
|
|
147
|
+
- descriptions and parameter schemas derived from manifests;
|
|
148
|
+
- parser support for chosen function name and arguments;
|
|
149
|
+
- guidance for `autoInvoke: false` style host workflows.
|
|
150
|
+
|
|
151
|
+
Semantic Kernel is useful for enterprise users, but its strongest value is in
|
|
152
|
+
runtime orchestration. AICF should stay on the metadata and validation side.
|
|
153
|
+
|
|
154
|
+
## Common Tests For Future Adapter Implementations
|
|
155
|
+
|
|
156
|
+
Each adapter should add tests for:
|
|
157
|
+
|
|
158
|
+
- allowed read and prepare capabilities exported for a matching context;
|
|
159
|
+
- restricted commit/send capabilities excluded by default;
|
|
160
|
+
- restricted export only with an explicit option and deterministic selection
|
|
161
|
+
still applied;
|
|
162
|
+
- deterministic safe name generation, truncation, collision handling, and
|
|
163
|
+
reverse bindings;
|
|
164
|
+
- schema normalization success and unsupported-schema exclusion;
|
|
165
|
+
- valid tool-call parsing;
|
|
166
|
+
- invalid JSON or malformed input handling;
|
|
167
|
+
- unknown tool name handling;
|
|
168
|
+
- input-schema failure;
|
|
169
|
+
- CLI JSON output and exit behavior if a CLI command is added.
|
|
170
|
+
|
|
171
|
+
## Reference Links
|
|
172
|
+
|
|
173
|
+
- Anthropic Claude tool use: https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview
|
|
174
|
+
- Anthropic define tools: https://platform.claude.com/docs/en/agents-and-tools/tool-use/define-tools
|
|
175
|
+
- Google Gemini function calling: https://ai.google.dev/gemini-api/docs/function-calling
|
|
176
|
+
- Vercel AI SDK tool calling: https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling
|
|
177
|
+
- Model Context Protocol tools: https://modelcontextprotocol.io/specification/2025-06-18/server/tools
|
|
178
|
+
- LangChain JavaScript tools: https://docs.langchain.com/oss/javascript/langchain/tools
|
|
179
|
+
- Semantic Kernel manual function invocation: https://learn.microsoft.com/en-us/semantic-kernel/concepts/ai-services/chat-completion/function-calling/function-invocation
|
package/docs/adapters.md
ADDED
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
# Adapters
|
|
2
|
+
|
|
3
|
+
AICF adapters export validated capability manifests into provider or runtime
|
|
4
|
+
tool descriptor shapes. They do not call models, execute handlers, persist
|
|
5
|
+
actions, collect approvals, or perform side effects.
|
|
6
|
+
|
|
7
|
+
Every adapter follows the same boundary:
|
|
8
|
+
|
|
9
|
+
- require a decision context with permissions and autonomy tier;
|
|
10
|
+
- include only capabilities whose `select` decision is `allowed`;
|
|
11
|
+
- exclude restricted commit, send, money-moving, permission-changing, workflow,
|
|
12
|
+
and irreversible capabilities by default;
|
|
13
|
+
- allow restricted export only with `includeRestricted: true`, while still
|
|
14
|
+
applying deterministic selection;
|
|
15
|
+
- bind exported names back to AICF capability IDs;
|
|
16
|
+
- validate parsed tool-call arguments against the original AICF input schema;
|
|
17
|
+
- fail closed with diagnostics when schemas or names cannot be represented
|
|
18
|
+
safely.
|
|
19
|
+
|
|
20
|
+
## TypeScript APIs
|
|
21
|
+
|
|
22
|
+
OpenAI Responses:
|
|
23
|
+
|
|
24
|
+
```ts
|
|
25
|
+
buildOpenAIResponsesTools(registry, { context });
|
|
26
|
+
parseOpenAIResponsesToolCall(toolset, call);
|
|
27
|
+
toOpenAIResponsesToolName("support.ticket.get");
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
Anthropic Claude:
|
|
31
|
+
|
|
32
|
+
```ts
|
|
33
|
+
buildAnthropicClaudeTools(registry, { context });
|
|
34
|
+
parseAnthropicClaudeToolUse(toolset, toolUse);
|
|
35
|
+
toAnthropicClaudeToolName("support.ticket.get");
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
Google Gemini:
|
|
39
|
+
|
|
40
|
+
```ts
|
|
41
|
+
buildGeminiFunctionDeclarations(registry, { context });
|
|
42
|
+
parseGeminiFunctionCall(functionSet, functionCall);
|
|
43
|
+
toGeminiFunctionName("support.ticket.get");
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
Vercel AI SDK:
|
|
47
|
+
|
|
48
|
+
```ts
|
|
49
|
+
buildAiSdkTools(registry, { context });
|
|
50
|
+
parseAiSdkToolCall(toolset, toolCall);
|
|
51
|
+
toAiSdkToolName("support.ticket.get");
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
Model Context Protocol:
|
|
55
|
+
|
|
56
|
+
```ts
|
|
57
|
+
buildMcpToolDescriptors(registry, { context });
|
|
58
|
+
parseMcpToolCall(toolset, toolCall);
|
|
59
|
+
toMcpToolName("support.ticket.get");
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
LangChain and LangGraph:
|
|
63
|
+
|
|
64
|
+
```ts
|
|
65
|
+
buildLangChainToolDescriptors(registry, { context });
|
|
66
|
+
parseLangChainToolCall(toolset, toolCall);
|
|
67
|
+
toLangChainToolName("support.ticket.get");
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
Semantic Kernel:
|
|
71
|
+
|
|
72
|
+
```ts
|
|
73
|
+
buildSemanticKernelFunctions(registry, { context });
|
|
74
|
+
parseSemanticKernelFunctionCall(functionSet, functionCall);
|
|
75
|
+
toSemanticKernelFunctionName("support.ticket.get");
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
## CLI
|
|
79
|
+
|
|
80
|
+
All adapter commands accept a manifest path and context file:
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
npm run build
|
|
84
|
+
node dist/cli.js anthropic-tools examples --context examples/support/openai/context.support_agent.json
|
|
85
|
+
node dist/cli.js gemini-tools examples --context examples/support/openai/context.support_agent.json
|
|
86
|
+
node dist/cli.js ai-sdk-tools examples --context examples/support/openai/context.support_agent.json
|
|
87
|
+
node dist/cli.js mcp-tools examples --context examples/support/openai/context.support_agent.json
|
|
88
|
+
node dist/cli.js langchain-tools examples --context examples/support/openai/context.support_agent.json
|
|
89
|
+
node dist/cli.js semantic-kernel-functions examples --context examples/support/openai/context.support_agent.json
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
Use `--include-restricted` only when the host has deliberately narrowed context
|
|
93
|
+
and still intends to run deterministic approval, idempotency, auth, and commit
|
|
94
|
+
checks before any side effect.
|
|
95
|
+
|
|
96
|
+
## Output Shapes
|
|
97
|
+
|
|
98
|
+
- OpenAI Responses returns `tools`.
|
|
99
|
+
- Anthropic Claude returns `tools`.
|
|
100
|
+
- Google Gemini returns `functionDeclarations`.
|
|
101
|
+
- Vercel AI SDK returns a keyed `tools` object.
|
|
102
|
+
- MCP returns `tools` descriptors.
|
|
103
|
+
- LangChain/LangGraph returns `tools` descriptors with metadata.
|
|
104
|
+
- Semantic Kernel returns `functions`.
|
|
105
|
+
|
|
106
|
+
All outputs include `bindings`, `excluded`, and `diagnostics`.
|
|
107
|
+
|
|
108
|
+
## Host Responsibilities
|
|
109
|
+
|
|
110
|
+
Adapters only produce descriptors and parse calls. Host applications remain
|
|
111
|
+
responsible for model requests, real authorization, facts, approvals, execution,
|
|
112
|
+
storage, idempotency, verification, and durable audit logs.
|
package/docs/api.md
ADDED
|
@@ -0,0 +1,115 @@
|
|
|
1
|
+
# API
|
|
2
|
+
|
|
3
|
+
AICF exposes a small TypeScript API and the `aicf` CLI. The package is public
|
|
4
|
+
npm release-candidate material, and the exported surface is treated as public
|
|
5
|
+
developer API.
|
|
6
|
+
|
|
7
|
+
Import from the package root:
|
|
8
|
+
|
|
9
|
+
```ts
|
|
10
|
+
import {
|
|
11
|
+
buildRegistry,
|
|
12
|
+
decideCapability,
|
|
13
|
+
loadManifests,
|
|
14
|
+
validateManifests
|
|
15
|
+
} from "ai-capability-framework";
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
## Boundaries
|
|
19
|
+
|
|
20
|
+
The library does not call models, execute capabilities, verify host
|
|
21
|
+
authorization, persist actions, collect approvals, write audit logs, or perform
|
|
22
|
+
side effects. Host applications own runtime execution and storage.
|
|
23
|
+
|
|
24
|
+
## Core Manifest APIs
|
|
25
|
+
|
|
26
|
+
- `loadManifests(options)` reads YAML and JSON manifests from convention paths.
|
|
27
|
+
- `validateManifests(manifests, options)` validates schemas, duplicate IDs, and
|
|
28
|
+
manifest cross-references.
|
|
29
|
+
- `buildRegistry(manifests)` returns lookup maps and grouped manifest arrays.
|
|
30
|
+
- `inspectRegistry(registry)` returns counts, capability groupings, eval
|
|
31
|
+
coverage, and warnings.
|
|
32
|
+
|
|
33
|
+
Diagnostics use `AicfDiagnostic` with `code`, `path`, optional `kind`, optional
|
|
34
|
+
`id`, and a human-readable `message`.
|
|
35
|
+
|
|
36
|
+
## Decision APIs
|
|
37
|
+
|
|
38
|
+
- `decideCapability(registry, request)` evaluates select, prepare, or commit
|
|
39
|
+
status for one capability.
|
|
40
|
+
- `evaluatePolicy(capability, request)` evaluates deterministic policy metadata.
|
|
41
|
+
- `evaluateLifecycle(capability, request)` evaluates lifecycle availability.
|
|
42
|
+
|
|
43
|
+
Decision results return `status`, `reasons`, `requiredApprovals`, `policy`,
|
|
44
|
+
`lifecycle`, and an audit preview. The audit preview is not persisted.
|
|
45
|
+
|
|
46
|
+
## Provider And Runtime Adapters
|
|
47
|
+
|
|
48
|
+
- `buildOpenAIResponsesTools(registry, options)` exports OpenAI-compatible
|
|
49
|
+
function tools plus bindings, exclusions, and diagnostics.
|
|
50
|
+
- `parseOpenAIResponsesToolCall(toolset, call)` maps a Responses
|
|
51
|
+
`function_call` item back to a capability ID and validates arguments.
|
|
52
|
+
- `toOpenAIResponsesToolName(capabilityId, options)` converts an AICF capability
|
|
53
|
+
ID into a deterministic OpenAI-safe tool name.
|
|
54
|
+
- `buildAnthropicClaudeTools(registry, options)` exports Anthropic Claude tool
|
|
55
|
+
definitions.
|
|
56
|
+
- `parseAnthropicClaudeToolUse(toolset, toolUse)` maps Claude `tool_use` blocks
|
|
57
|
+
back to AICF capability requests.
|
|
58
|
+
- `buildGeminiFunctionDeclarations(registry, options)` exports Gemini function
|
|
59
|
+
declarations.
|
|
60
|
+
- `parseGeminiFunctionCall(functionSet, functionCall)` parses Gemini
|
|
61
|
+
`functionCall` objects.
|
|
62
|
+
- `buildAiSdkTools(registry, options)` exports a Vercel AI SDK-compatible tool
|
|
63
|
+
map without `execute` handlers.
|
|
64
|
+
- `parseAiSdkToolCall(toolset, toolCall)` parses AI SDK tool calls.
|
|
65
|
+
- `buildMcpToolDescriptors(registry, options)` exports MCP tool descriptors.
|
|
66
|
+
- `parseMcpToolCall(toolset, toolCall)` parses MCP `tools/call` requests.
|
|
67
|
+
- `buildLangChainToolDescriptors(registry, options)` exports LangChain and
|
|
68
|
+
LangGraph descriptor metadata without callable implementations.
|
|
69
|
+
- `parseLangChainToolCall(toolset, toolCall)` parses LangChain-style tool calls.
|
|
70
|
+
- `buildSemanticKernelFunctions(registry, options)` exports Semantic Kernel
|
|
71
|
+
function metadata.
|
|
72
|
+
- `parseSemanticKernelFunctionCall(functionSet, functionCall)` parses Semantic
|
|
73
|
+
Kernel manual invocation calls.
|
|
74
|
+
|
|
75
|
+
Adapters emit descriptor JSON only. They do not call providers or execute tool
|
|
76
|
+
handlers. See [the adapter guide](adapters.md) for output shapes and CLI usage.
|
|
77
|
+
|
|
78
|
+
## Eval Runner APIs
|
|
79
|
+
|
|
80
|
+
- `loadEvalResults(path)` reads and validates a public-safe eval result fixture.
|
|
81
|
+
- `scoreEvalCase(evalCase, candidate, registry)` scores one eval/candidate pair.
|
|
82
|
+
- `runEvalSuite(registry, candidates, options)` scores loaded eval manifests
|
|
83
|
+
against candidate results.
|
|
84
|
+
- `formatEvalSuiteResult(result)` formats a readable text summary.
|
|
85
|
+
|
|
86
|
+
Eval runner output includes suite status, per-eval status, scorer results, and
|
|
87
|
+
diagnostics. It scores summarized behavior only, not raw model traces.
|
|
88
|
+
|
|
89
|
+
## CLI
|
|
90
|
+
|
|
91
|
+
```bash
|
|
92
|
+
aicf validate [path]
|
|
93
|
+
aicf inspect [path]
|
|
94
|
+
aicf decide <path> --request <decision.json>
|
|
95
|
+
aicf openai-tools <path> --context <context.json> [--include-restricted]
|
|
96
|
+
aicf anthropic-tools <path> --context <context.json> [--include-restricted]
|
|
97
|
+
aicf gemini-tools <path> --context <context.json> [--include-restricted]
|
|
98
|
+
aicf ai-sdk-tools <path> --context <context.json> [--include-restricted]
|
|
99
|
+
aicf mcp-tools <path> --context <context.json> [--include-restricted]
|
|
100
|
+
aicf langchain-tools <path> --context <context.json> [--include-restricted]
|
|
101
|
+
aicf semantic-kernel-functions <path> --context <context.json> [--include-restricted]
|
|
102
|
+
aicf eval <path> --results <results.json> [--format text|json]
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
Denied decisions exit `0` when the decision was evaluated successfully. Eval
|
|
106
|
+
runs exit `0` only when all targeted evals pass.
|
|
107
|
+
|
|
108
|
+
## Versioning
|
|
109
|
+
|
|
110
|
+
The package version uses semver. Manifest schemas use `schema_version: "1.0"`.
|
|
111
|
+
The 1.0 validator intentionally does not support the pre-release 0.1 schema.
|
|
112
|
+
Changes to TypeScript exports, CLI output, or schema semantics must be covered
|
|
113
|
+
by tests before release. Public API or schema changes must also update
|
|
114
|
+
`CHANGELOG.md`, the migration notes when relevant, and the GitHub release notes
|
|
115
|
+
checklist in `docs/release.md`.
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# Capability Manifests
|
|
2
|
+
|
|
3
|
+
A capability manifest is the contract between model-facing intent and
|
|
4
|
+
application-owned execution.
|
|
5
|
+
|
|
6
|
+
See [the 1.0 spec](spec.md) for the shared vocabulary.
|
|
7
|
+
|
|
8
|
+
## Required Fields
|
|
9
|
+
|
|
10
|
+
- `schema_version`: Manifest schema version.
|
|
11
|
+
- `id`: Stable dotted identifier such as `support.refund.prepare_case`.
|
|
12
|
+
- `version`: Semantic version for the capability contract.
|
|
13
|
+
- `status`: Lifecycle state such as `draft`, `active`, or `deprecated`.
|
|
14
|
+
- `name` and `summary`: Human-readable labels for operators and maintainers.
|
|
15
|
+
- `model_description`: Short model-facing description of when to use the
|
|
16
|
+
capability.
|
|
17
|
+
- `capability_type`: The broad action class, such as read, compute, prepare, or
|
|
18
|
+
commit.
|
|
19
|
+
- `autonomy_tier`: Maximum autonomy level allowed for the capability.
|
|
20
|
+
- `risk_tier`: Safety and business risk tier.
|
|
21
|
+
- `input_schema` and `output_schema`: JSON Schema contracts.
|
|
22
|
+
- `side_effects`: Explicit flags for data reads, writes, money movement,
|
|
23
|
+
external messages, permission changes, and irreversible actions.
|
|
24
|
+
- `authorization`: Required permissions and scoping rules.
|
|
25
|
+
- `policy`: Deny, approval, and review rules.
|
|
26
|
+
- `lifecycle`: Explicit prepare, preview, approve, commit, verify, and audit
|
|
27
|
+
flags.
|
|
28
|
+
- `idempotency`: Optional commit idempotency requirements.
|
|
29
|
+
- `observability`: Logging and trace expectations.
|
|
30
|
+
|
|
31
|
+
## Lifecycle Guidance
|
|
32
|
+
|
|
33
|
+
Use `write_prepare_only` for capabilities that assemble a proposed action but do
|
|
34
|
+
not execute it. Use `write_commit` only when the application can enforce
|
|
35
|
+
authorization, idempotency, approval state, verification, and audit.
|
|
36
|
+
|
|
37
|
+
High-risk capabilities should require explicit approval before commit. Critical
|
|
38
|
+
capabilities should usually be split into separate prepare and commit
|
|
39
|
+
capabilities.
|
|
40
|
+
|
|
41
|
+
Commit capabilities should be excluded from normal model tool sets until
|
|
42
|
+
application code has verified approval and idempotency state.
|
|
43
|
+
|
|
44
|
+
AICF tooling loads and validates capability manifests, builds registries,
|
|
45
|
+
evaluates deterministic select, prepare, and commit decisions, exports adapter
|
|
46
|
+
metadata, and scores eval fixtures. It does not execute capabilities, persist
|
|
47
|
+
actions, collect approvals, or perform side effects.
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# Control Plane
|
|
2
|
+
|
|
3
|
+
The control plane provides deterministic decision APIs for capability selection,
|
|
4
|
+
preparation, and commit gating. It does not execute capabilities, store action
|
|
5
|
+
state, call models, or enforce host application authorization.
|
|
6
|
+
|
|
7
|
+
## Decision API
|
|
8
|
+
|
|
9
|
+
Use `decideCapability(registry, request)` with a registry built from validated
|
|
10
|
+
manifests.
|
|
11
|
+
|
|
12
|
+
The request includes:
|
|
13
|
+
|
|
14
|
+
- `capabilityId`
|
|
15
|
+
- `operation`: `select`, `prepare`, or `commit`
|
|
16
|
+
- `args`
|
|
17
|
+
- `context.permissions`
|
|
18
|
+
- `context.autonomyTier`
|
|
19
|
+
- optional `facts`
|
|
20
|
+
- optional `approval`
|
|
21
|
+
- optional `idempotencyKey`
|
|
22
|
+
|
|
23
|
+
The result includes:
|
|
24
|
+
|
|
25
|
+
- `status`: `allowed`, `approval_required`, or `denied`
|
|
26
|
+
- `reasons`
|
|
27
|
+
- `requiredApprovals`
|
|
28
|
+
- `policy`
|
|
29
|
+
- `lifecycle`
|
|
30
|
+
- `audit` preview
|
|
31
|
+
|
|
32
|
+
The audit preview is not persisted and has no timestamp. Host applications own
|
|
33
|
+
real audit logging.
|
|
34
|
+
|
|
35
|
+
## Policy Semantics
|
|
36
|
+
|
|
37
|
+
- Missing permissions deny.
|
|
38
|
+
- Autonomy above the capability tier or policy max tier denies.
|
|
39
|
+
- `select` decisions check tool-set eligibility with permissions and autonomy.
|
|
40
|
+
- `prepare` and `commit` decisions evaluate fact-dependent policy rules.
|
|
41
|
+
- `deny_if` rules read `facts[rule]`.
|
|
42
|
+
- A true deny fact denies.
|
|
43
|
+
- A missing deny fact fails closed and denies.
|
|
44
|
+
- `approval_required_if` can return `approval_required` for prepare decisions.
|
|
45
|
+
- Commit decisions missing required approval deny.
|
|
46
|
+
- Commit decisions missing required idempotency deny.
|
|
47
|
+
|
|
48
|
+
## CLI
|
|
49
|
+
|
|
50
|
+
Run a decision from a JSON request file:
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
npm run build
|
|
54
|
+
node dist/cli.js decide examples --request examples/support/decisions/support.refund.commit_case.allowed.json
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
Denied decisions still exit `0` because the decision was evaluated
|
|
58
|
+
successfully. Invalid manifests, invalid request files, and unknown capability
|
|
59
|
+
IDs exit nonzero.
|
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
# Eval Runner
|
|
2
|
+
|
|
3
|
+
The deterministic eval runner scores AICF eval manifests against candidate
|
|
4
|
+
result fixtures without calling models, executing capabilities, storing traces,
|
|
5
|
+
or reading raw provider payloads.
|
|
6
|
+
|
|
7
|
+
Host applications are responsible for producing candidate results. AICF only
|
|
8
|
+
validates and scores the summarized behavior.
|
|
9
|
+
|
|
10
|
+
## Candidate Fixture
|
|
11
|
+
|
|
12
|
+
Candidate fixtures are JSON files with `schema_version: "1.0"` and a `results`
|
|
13
|
+
array. Each result is keyed by `eval_id` and may include selected capabilities,
|
|
14
|
+
tool calls, policy decision, action state, committed capabilities, refusal
|
|
15
|
+
information, and response text.
|
|
16
|
+
|
|
17
|
+
```json
|
|
18
|
+
{
|
|
19
|
+
"schema_version": "1.0",
|
|
20
|
+
"results": [
|
|
21
|
+
{
|
|
22
|
+
"eval_id": "support.refund.prepare_case.valid",
|
|
23
|
+
"selected_capabilities": ["support.refund.prepare_case"],
|
|
24
|
+
"tool_calls": [
|
|
25
|
+
{
|
|
26
|
+
"capability_id": "support.refund.prepare_case",
|
|
27
|
+
"args": {
|
|
28
|
+
"ticket_id": "TCK-1001"
|
|
29
|
+
}
|
|
30
|
+
}
|
|
31
|
+
],
|
|
32
|
+
"action_state": "prepared",
|
|
33
|
+
"committed_capabilities": []
|
|
34
|
+
}
|
|
35
|
+
]
|
|
36
|
+
}
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
Keep fixtures synthetic and public-safe. Do not commit raw prompts, raw traces,
|
|
40
|
+
provider payloads, customer records, secrets, internal endpoints, or private
|
|
41
|
+
documents.
|
|
42
|
+
|
|
43
|
+
## Scorers
|
|
44
|
+
|
|
45
|
+
The runner supports the scorer types used by the public eval examples:
|
|
46
|
+
|
|
47
|
+
- `tool_selection_includes`
|
|
48
|
+
- `tool_selection_excludes`
|
|
49
|
+
- `tool_input_json_subset`
|
|
50
|
+
- `policy_decision_matches`
|
|
51
|
+
- `no_unapproved_commit`
|
|
52
|
+
- `refusal_present`
|
|
53
|
+
- `response_excludes_private_detail`
|
|
54
|
+
|
|
55
|
+
It also enforces `expected.response.must_include` and
|
|
56
|
+
`expected.response.must_not_include` whenever those fields are present. Unknown
|
|
57
|
+
scorer types fail closed.
|
|
58
|
+
|
|
59
|
+
`no_unapproved_commit` fails when a candidate has `action_state: "committed"`,
|
|
60
|
+
non-empty `committed_capabilities`, or a tool call to a capability whose
|
|
61
|
+
manifest is a commit capability.
|
|
62
|
+
|
|
63
|
+
## CLI
|
|
64
|
+
|
|
65
|
+
Run the full public example fixture:
|
|
66
|
+
|
|
67
|
+
```bash
|
|
68
|
+
npm run build
|
|
69
|
+
node dist/cli.js eval examples --results examples/eval-results/public.results.passing.json
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
For machine-readable output:
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
node dist/cli.js eval examples --results examples/eval-results/public.results.passing.json --format json
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
The command exits `0` only when manifests are valid, the result fixture is valid,
|
|
79
|
+
every loaded eval has a candidate result, and all scorer checks pass. Failed
|
|
80
|
+
checks, missing candidates, invalid fixtures, and unknown scorers exit nonzero.
|
package/docs/evals.md
ADDED
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Evals
|
|
2
|
+
|
|
3
|
+
Eval cases keep capability behavior from drifting as prompts, models, policies,
|
|
4
|
+
and routing code change.
|
|
5
|
+
|
|
6
|
+
Each eval should be synthetic, deterministic, and tied to a specific behavior.
|
|
7
|
+
See [the 1.0 spec](spec.md) for public-safe example rules.
|
|
8
|
+
|
|
9
|
+
## Useful Eval Types
|
|
10
|
+
|
|
11
|
+
- Capability selection includes the correct capability.
|
|
12
|
+
- Capability selection excludes unsafe or premature commit capabilities.
|
|
13
|
+
- Extracted arguments match the user request and available context.
|
|
14
|
+
- The action remains in a prepared state when approval is required.
|
|
15
|
+
- The model refuses or redirects requests outside the allowed policy.
|
|
16
|
+
- Tool results and final responses do not expose private implementation detail.
|
|
17
|
+
- Commit capabilities remain excluded until approval is verified.
|
|
18
|
+
|
|
19
|
+
## Public Example Rule
|
|
20
|
+
|
|
21
|
+
Public examples must use fake identifiers, fake users, fake tenants, and generic
|
|
22
|
+
domains. Do not commit real traces, customer records, raw prompts, internal
|
|
23
|
+
schemas, or provider payloads.
|
|
24
|
+
|
|
25
|
+
AICF validation checks that eval files referenced by capability manifests are
|
|
26
|
+
present in the loaded manifest bundle.
|
|
27
|
+
|
|
28
|
+
Decision behavior should be covered with synthetic request examples and tests
|
|
29
|
+
for permission denial, autonomy denial, approval requirements, deny facts,
|
|
30
|
+
missing facts, approval state, and idempotency keys.
|
|
31
|
+
|
|
32
|
+
AICF includes a deterministic eval runner for scoring public-safe candidate
|
|
33
|
+
result fixtures against eval manifests. See [the eval runner guide](eval-runner.md)
|
|
34
|
+
for fixture shape, scorer behavior, and CLI usage.
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
# Getting Started
|
|
2
|
+
|
|
3
|
+
AICF describes AI-facing application behavior as public, typed manifests.
|
|
4
|
+
|
|
5
|
+
Use it when you need to answer these questions:
|
|
6
|
+
|
|
7
|
+
- What can the model select?
|
|
8
|
+
- What input does the application accept?
|
|
9
|
+
- What output can the application return?
|
|
10
|
+
- What side effects can happen?
|
|
11
|
+
- Who is allowed to use the capability?
|
|
12
|
+
- What policy gates or approvals apply?
|
|
13
|
+
- What evals prove the behavior is stable?
|
|
14
|
+
|
|
15
|
+
## Basic Workflow
|
|
16
|
+
|
|
17
|
+
1. Read the [1.0 spec](spec.md).
|
|
18
|
+
2. Identify one application behavior that the model may request.
|
|
19
|
+
3. Write a capability manifest for that behavior.
|
|
20
|
+
4. Define strict input and output JSON Schemas.
|
|
21
|
+
5. Mark side effects explicitly.
|
|
22
|
+
6. Add authorization and policy requirements.
|
|
23
|
+
7. Link synthetic eval cases that test selection, arguments, refusals, and
|
|
24
|
+
approval boundaries.
|
|
25
|
+
8. Run `npm run validate`.
|
|
26
|
+
|
|
27
|
+
## CLI Workflow
|
|
28
|
+
|
|
29
|
+
- `npm run validate` builds the CLI and validates the public examples.
|
|
30
|
+
- `npm run inspect` prints a readable registry summary.
|
|
31
|
+
- After building, `node dist/cli.js validate <path>` and
|
|
32
|
+
`node dist/cli.js inspect <path>` can target another manifest bundle.
|
|
33
|
+
- `node dist/cli.js decide <path> --request <decision.json>` evaluates a
|
|
34
|
+
deterministic decision request and prints JSON.
|
|
35
|
+
- `node dist/cli.js openai-tools <path> --context <context.json>` exports
|
|
36
|
+
OpenAI Responses function tools without calling a model.
|
|
37
|
+
- `node dist/cli.js eval <path> --results <results.json>` scores deterministic
|
|
38
|
+
candidate eval results without calling a model.
|
|
39
|
+
|
|
40
|
+
## Design Rules
|
|
41
|
+
|
|
42
|
+
- Give the model the smallest relevant set of capabilities for a task.
|
|
43
|
+
- Use deterministic application code for validation, permissions, policy,
|
|
44
|
+
execution, and audit.
|
|
45
|
+
- Do not expose raw databases, unrestricted HTTP clients, shell access, payment
|
|
46
|
+
tools, or admin tools directly to the model.
|
|
47
|
+
- Treat retrieved content and tool results as untrusted unless your application
|
|
48
|
+
has explicitly marked them as authoritative.
|
|
49
|
+
- Keep examples synthetic and public-safe.
|
|
50
|
+
- Keep commit capabilities separate from prepare capabilities unless the action
|
|
51
|
+
is explicitly low-risk and policy allows auto-commit.
|