ai-capability-framework 1.0.0-rc.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (115) hide show
  1. package/CHANGELOG.md +37 -0
  2. package/CONTRIBUTING.md +56 -0
  3. package/LICENSE +21 -0
  4. package/README.md +198 -0
  5. package/SECURITY.md +29 -0
  6. package/conformance/invalid/duplicate/capabilities/conformance.duplicate.one.yaml +55 -0
  7. package/conformance/invalid/duplicate/capabilities/conformance.duplicate.two.yaml +55 -0
  8. package/conformance/invalid/eval-results/conformance.results.failing.json +15 -0
  9. package/conformance/invalid/missing-reference/capabilities/conformance.missing_reference.prepare.yaml +58 -0
  10. package/conformance/invalid/schema/capabilities/conformance.invalid.missing_required.yaml +3 -0
  11. package/conformance/valid/capabilities/conformance.note.get.yaml +82 -0
  12. package/conformance/valid/decisions/conformance.note.get.denied_missing_permission.json +13 -0
  13. package/conformance/valid/entities/Note.yaml +18 -0
  14. package/conformance/valid/eval-results/conformance.results.passing.json +19 -0
  15. package/conformance/valid/evals/conformance.note.get.valid.yaml +33 -0
  16. package/conformance/valid/openai/context.no_permissions.json +6 -0
  17. package/dist/adapter-common.d.ts +62 -0
  18. package/dist/adapter-common.d.ts.map +1 -0
  19. package/dist/adapter-common.js +453 -0
  20. package/dist/ai-sdk.d.ts +5 -0
  21. package/dist/ai-sdk.d.ts.map +1 -0
  22. package/dist/ai-sdk.js +71 -0
  23. package/dist/anthropic-claude.d.ts +5 -0
  24. package/dist/anthropic-claude.d.ts.map +1 -0
  25. package/dist/anthropic-claude.js +62 -0
  26. package/dist/cli.d.ts +11 -0
  27. package/dist/cli.d.ts.map +1 -0
  28. package/dist/cli.js +360 -0
  29. package/dist/decision.d.ts +6 -0
  30. package/dist/decision.d.ts.map +1 -0
  31. package/dist/decision.js +237 -0
  32. package/dist/eval-runner.d.ts +6 -0
  33. package/dist/eval-runner.d.ts.map +1 -0
  34. package/dist/eval-runner.js +372 -0
  35. package/dist/gemini.d.ts +5 -0
  36. package/dist/gemini.d.ts.map +1 -0
  37. package/dist/gemini.js +64 -0
  38. package/dist/generated/manifest-types.d.ts +208 -0
  39. package/dist/generated/manifest-types.d.ts.map +1 -0
  40. package/dist/generated/manifest-types.js +1 -0
  41. package/dist/index.d.ts +15 -0
  42. package/dist/index.d.ts.map +1 -0
  43. package/dist/index.js +13 -0
  44. package/dist/langchain.d.ts +5 -0
  45. package/dist/langchain.d.ts.map +1 -0
  46. package/dist/langchain.js +65 -0
  47. package/dist/loader.d.ts +4 -0
  48. package/dist/loader.d.ts.map +1 -0
  49. package/dist/loader.js +94 -0
  50. package/dist/mcp.d.ts +5 -0
  51. package/dist/mcp.d.ts.map +1 -0
  52. package/dist/mcp.js +82 -0
  53. package/dist/openai-responses.d.ts +6 -0
  54. package/dist/openai-responses.d.ts.map +1 -0
  55. package/dist/openai-responses.js +65 -0
  56. package/dist/registry.d.ts +6 -0
  57. package/dist/registry.d.ts.map +1 -0
  58. package/dist/registry.js +134 -0
  59. package/dist/semantic-kernel.d.ts +5 -0
  60. package/dist/semantic-kernel.d.ts.map +1 -0
  61. package/dist/semantic-kernel.js +79 -0
  62. package/dist/types.d.ts +400 -0
  63. package/dist/types.d.ts.map +1 -0
  64. package/dist/types.js +1 -0
  65. package/dist/validator.d.ts +3 -0
  66. package/dist/validator.d.ts.map +1 -0
  67. package/dist/validator.js +105 -0
  68. package/docs/adapter-roadmap.md +179 -0
  69. package/docs/adapters.md +112 -0
  70. package/docs/api.md +115 -0
  71. package/docs/capability-manifests.md +47 -0
  72. package/docs/control-plane.md +59 -0
  73. package/docs/eval-runner.md +80 -0
  74. package/docs/evals.md +34 -0
  75. package/docs/getting-started.md +51 -0
  76. package/docs/host-responsibilities.md +23 -0
  77. package/docs/interoperability.md +32 -0
  78. package/docs/migration-0.1-to-1.0.md +22 -0
  79. package/docs/openai-responses.md +103 -0
  80. package/docs/release.md +75 -0
  81. package/docs/spec.md +194 -0
  82. package/examples/eval-results/public.results.passing.json +128 -0
  83. package/examples/scheduling/capabilities/scheduling.availability.get.yaml +91 -0
  84. package/examples/scheduling/capabilities/scheduling.invite.prepare.yaml +114 -0
  85. package/examples/scheduling/capabilities/scheduling.invite.send.yaml +97 -0
  86. package/examples/scheduling/decisions/scheduling.availability.get.select.json +14 -0
  87. package/examples/scheduling/decisions/scheduling.invite.prepare.allowed.json +20 -0
  88. package/examples/scheduling/decisions/scheduling.invite.send.allowed.json +22 -0
  89. package/examples/scheduling/decisions/scheduling.invite.send.denied_missing_approval.json +18 -0
  90. package/examples/scheduling/entities/AvailabilitySlot.yaml +26 -0
  91. package/examples/scheduling/entities/MeetingInvite.yaml +25 -0
  92. package/examples/scheduling/eval-results/scheduling.results.passing.json +59 -0
  93. package/examples/scheduling/evals/scheduling.availability.get.valid.yaml +38 -0
  94. package/examples/scheduling/evals/scheduling.invite.prepare.valid.yaml +44 -0
  95. package/examples/scheduling/evals/scheduling.invite.send.requires_approval.yaml +43 -0
  96. package/examples/scheduling/openai/context.scheduler.json +6 -0
  97. package/examples/support/capabilities/support.refund.commit_case.yaml +98 -0
  98. package/examples/support/capabilities/support.refund.prepare_case.yaml +116 -0
  99. package/examples/support/capabilities/support.ticket.get.yaml +85 -0
  100. package/examples/support/decisions/support.refund.commit_case.allowed.json +22 -0
  101. package/examples/support/decisions/support.refund.commit_case.denied_missing_approval.json +18 -0
  102. package/examples/support/decisions/support.refund.prepare_case.approval_required.json +19 -0
  103. package/examples/support/decisions/support.ticket.get.select.json +0 -0
  104. package/examples/support/entities/Order.yaml +23 -0
  105. package/examples/support/entities/Ticket.yaml +25 -0
  106. package/examples/support/eval-results/support.results.passing.json +68 -0
  107. package/examples/support/evals/support.refund.commit_case.requires_approval.yaml +42 -0
  108. package/examples/support/evals/support.refund.prepare_case.approval_required.yaml +45 -0
  109. package/examples/support/evals/support.refund.prepare_case.valid.yaml +43 -0
  110. package/examples/support/openai/context.support_agent.json +9 -0
  111. package/package.json +74 -0
  112. package/schemas/capability-manifest.schema.json +250 -0
  113. package/schemas/entity-manifest.schema.json +97 -0
  114. package/schemas/eval-case.schema.json +134 -0
  115. package/schemas/eval-result.schema.json +77 -0
@@ -0,0 +1,179 @@
1
+ # Adapter Status And Roadmap
2
+
3
+ AICF is provider-neutral. Adapters export tool metadata, bindings, parsers, and
4
+ diagnostics, but do not call models, execute handlers, persist actions, collect
5
+ approvals, or perform side effects.
6
+
7
+ ## Priority Criteria
8
+
9
+ Adapter priority is based on:
10
+
11
+ - fit with AICF capability manifests and JSON Schema inputs;
12
+ - adoption and usefulness for public developers;
13
+ - ability to preserve the no-execution boundary;
14
+ - deterministic name binding and argument parsing support;
15
+ - implementation size in the TypeScript package.
16
+
17
+ Every adapter should expose the same high-level contract:
18
+
19
+ - `buildXTools(registry, options)` or equivalent export function;
20
+ - `parseXToolCall(toolset, call)` or equivalent parser;
21
+ - deterministic tool-name mapping back to AICF capability IDs;
22
+ - `bindings`, `excluded`, and `diagnostics`;
23
+ - selection gated by `decideCapability(..., operation: "select")`;
24
+ - restricted side-effect tools excluded by default;
25
+ - schema normalization that fails closed when the target cannot represent an
26
+ AICF input schema safely.
27
+
28
+ ## Priority Order
29
+
30
+ | Priority | Adapter | Why It Comes Here | Primary Output |
31
+ | --- | --- | --- | --- |
32
+ | Priority | Adapter | Status | Primary Output |
33
+ | --- | --- | --- | --- |
34
+ | P1 | Anthropic Claude Messages tools | Implemented | Claude client tool definitions plus parser for `tool_use` blocks. |
35
+ | P2 | Google Gemini function calling | Implemented | Gemini `functionDeclarations` plus parser for `functionCall` objects. |
36
+ | P3 | Vercel AI SDK tools | Implemented | AI SDK tool map with `inputSchema`, `description`, `strict`, and no `execute`. |
37
+ | P4 | Model Context Protocol tool descriptors | Implemented | MCP `Tool` descriptors and parser for `tools/call` requests. |
38
+ | P5 | LangChain/LangGraph tools | Implemented | Structured tool parameter specs and bindings for host-supplied functions. |
39
+ | P6 | Semantic Kernel functions | Implemented | Function/plugin metadata and parser guidance for manual invocation flows. |
40
+
41
+ ## P1: Anthropic Claude Messages Tools
42
+
43
+ Target a no-model-call adapter that exports Claude client tool definitions:
44
+
45
+ - `name`: sanitized AICF capability ID, capped at the provider limit.
46
+ - `description`: composed from `model_description`, use guidance, risk,
47
+ autonomy, lifecycle, and side-effect metadata.
48
+ - `input_schema`: normalized AICF `input_schema`.
49
+ - `strict: true` when the schema can be represented safely.
50
+ - optional `input_examples` only when public examples are available and validated.
51
+
52
+ Parser target:
53
+
54
+ - map Claude `tool_use.name` to capability ID;
55
+ - read `tool_use.id` as call ID;
56
+ - validate `tool_use.input` against the original AICF input schema;
57
+ - return structured diagnostics for unknown names, malformed input, and schema
58
+ failures.
59
+
60
+ Reference basis: Claude client tools include `name`, `description`, and
61
+ `input_schema`, can use `strict: true`, and return `tool_use` blocks that the
62
+ host application executes.
63
+
64
+ ## P2: Google Gemini Function Calling
65
+
66
+ Target a no-model-call adapter that exports Gemini function declarations:
67
+
68
+ - `name`: sanitized capability ID using underscores or camelCase-safe names.
69
+ - `description`: composed from AICF model-facing and governance metadata.
70
+ - `parameters`: AICF input schema converted to Gemini's OpenAPI-compatible
71
+ schema subset.
72
+
73
+ Parser target:
74
+
75
+ - map `functionCall.name` to capability ID;
76
+ - preserve `functionCall.id` when present;
77
+ - validate `functionCall.args` against the original AICF input schema;
78
+ - report schema keywords that cannot be represented safely in Gemini parameters.
79
+
80
+ The hard part is schema normalization. Gemini function declarations use an
81
+ OpenAPI-compatible subset, so this adapter should fail closed for unsupported
82
+ JSON Schema 2020-12 features rather than silently weakening validation.
83
+
84
+ ## P3: Vercel AI SDK Tools
85
+
86
+ Target a TypeScript-first adapter that exports an AI SDK-compatible tool map
87
+ without execution functions:
88
+
89
+ - object keys are deterministic tool names;
90
+ - values include `description`, `inputSchema`, and `strict` when supported;
91
+ - no `execute` function is generated by AICF;
92
+ - bindings preserve capability IDs and original schemas.
93
+
94
+ Parser target:
95
+
96
+ - map AI SDK tool call names to AICF capability IDs;
97
+ - validate tool call input against the original AICF input schema;
98
+ - leave tool result production and execution to the host app.
99
+
100
+ This adapter is attractive because the AI SDK accepts JSON schemas and `execute`
101
+ is optional, allowing host apps to forward calls to a queue, client, or approval
102
+ path instead of executing in the same process.
103
+
104
+ ## P4: Model Context Protocol Tool Descriptors
105
+
106
+ Target MCP descriptor export, not an AICF-owned executable MCP server:
107
+
108
+ - `name`: deterministic MCP-safe tool name;
109
+ - `title`: human-readable capability name;
110
+ - `description`: AICF model description plus lifecycle/risk guidance;
111
+ - `inputSchema`: AICF input schema, preserving JSON Schema 2020-12 where valid;
112
+ - `outputSchema`: AICF output schema when representable;
113
+ - `annotations`: conservative hints derived from side effects, such as read-only
114
+ and idempotent hints.
115
+
116
+ Parser target:
117
+
118
+ - map `tools/call.params.name` to capability ID;
119
+ - validate `tools/call.params.arguments`;
120
+ - return an AICF capability request object for the host MCP server or runtime to
121
+ decide and execute.
122
+
123
+ Do not add an MCP server that executes capabilities inside AICF. Host apps may
124
+ use AICF descriptors inside their own MCP server implementation.
125
+
126
+ ## P5: LangChain And LangGraph Tools
127
+
128
+ Target descriptor export for host-created tools:
129
+
130
+ - deterministic name;
131
+ - description;
132
+ - schema suitable for LangChain structured tools;
133
+ - binding metadata for capability ID, risk, autonomy, and restriction status.
134
+
135
+ Do not generate callable implementations. LangChain and LangGraph tools are
136
+ runtime functions that can update state or return commands, so the host must
137
+ supply execution, approval, and state handling.
138
+
139
+ This adapter should probably be implemented after the AI SDK adapter, because
140
+ both are TypeScript-oriented but AI SDK has a cleaner descriptor-only path.
141
+
142
+ ## P6: Semantic Kernel Functions
143
+
144
+ Target metadata and manual-invocation compatibility:
145
+
146
+ - function/plugin names derived from AICF capability IDs;
147
+ - descriptions and parameter schemas derived from manifests;
148
+ - parser support for chosen function name and arguments;
149
+ - guidance for `autoInvoke: false` style host workflows.
150
+
151
+ Semantic Kernel is useful for enterprise users, but its strongest value is in
152
+ runtime orchestration. AICF should stay on the metadata and validation side.
153
+
154
+ ## Common Tests For Future Adapter Implementations
155
+
156
+ Each adapter should add tests for:
157
+
158
+ - allowed read and prepare capabilities exported for a matching context;
159
+ - restricted commit/send capabilities excluded by default;
160
+ - restricted export only with an explicit option and deterministic selection
161
+ still applied;
162
+ - deterministic safe name generation, truncation, collision handling, and
163
+ reverse bindings;
164
+ - schema normalization success and unsupported-schema exclusion;
165
+ - valid tool-call parsing;
166
+ - invalid JSON or malformed input handling;
167
+ - unknown tool name handling;
168
+ - input-schema failure;
169
+ - CLI JSON output and exit behavior if a CLI command is added.
170
+
171
+ ## Reference Links
172
+
173
+ - Anthropic Claude tool use: https://platform.claude.com/docs/en/agents-and-tools/tool-use/overview
174
+ - Anthropic define tools: https://platform.claude.com/docs/en/agents-and-tools/tool-use/define-tools
175
+ - Google Gemini function calling: https://ai.google.dev/gemini-api/docs/function-calling
176
+ - Vercel AI SDK tool calling: https://ai-sdk.dev/docs/ai-sdk-core/tools-and-tool-calling
177
+ - Model Context Protocol tools: https://modelcontextprotocol.io/specification/2025-06-18/server/tools
178
+ - LangChain JavaScript tools: https://docs.langchain.com/oss/javascript/langchain/tools
179
+ - Semantic Kernel manual function invocation: https://learn.microsoft.com/en-us/semantic-kernel/concepts/ai-services/chat-completion/function-calling/function-invocation
@@ -0,0 +1,112 @@
1
+ # Adapters
2
+
3
+ AICF adapters export validated capability manifests into provider or runtime
4
+ tool descriptor shapes. They do not call models, execute handlers, persist
5
+ actions, collect approvals, or perform side effects.
6
+
7
+ Every adapter follows the same boundary:
8
+
9
+ - require a decision context with permissions and autonomy tier;
10
+ - include only capabilities whose `select` decision is `allowed`;
11
+ - exclude restricted commit, send, money-moving, permission-changing, workflow,
12
+ and irreversible capabilities by default;
13
+ - allow restricted export only with `includeRestricted: true`, while still
14
+ applying deterministic selection;
15
+ - bind exported names back to AICF capability IDs;
16
+ - validate parsed tool-call arguments against the original AICF input schema;
17
+ - fail closed with diagnostics when schemas or names cannot be represented
18
+ safely.
19
+
20
+ ## TypeScript APIs
21
+
22
+ OpenAI Responses:
23
+
24
+ ```ts
25
+ buildOpenAIResponsesTools(registry, { context });
26
+ parseOpenAIResponsesToolCall(toolset, call);
27
+ toOpenAIResponsesToolName("support.ticket.get");
28
+ ```
29
+
30
+ Anthropic Claude:
31
+
32
+ ```ts
33
+ buildAnthropicClaudeTools(registry, { context });
34
+ parseAnthropicClaudeToolUse(toolset, toolUse);
35
+ toAnthropicClaudeToolName("support.ticket.get");
36
+ ```
37
+
38
+ Google Gemini:
39
+
40
+ ```ts
41
+ buildGeminiFunctionDeclarations(registry, { context });
42
+ parseGeminiFunctionCall(functionSet, functionCall);
43
+ toGeminiFunctionName("support.ticket.get");
44
+ ```
45
+
46
+ Vercel AI SDK:
47
+
48
+ ```ts
49
+ buildAiSdkTools(registry, { context });
50
+ parseAiSdkToolCall(toolset, toolCall);
51
+ toAiSdkToolName("support.ticket.get");
52
+ ```
53
+
54
+ Model Context Protocol:
55
+
56
+ ```ts
57
+ buildMcpToolDescriptors(registry, { context });
58
+ parseMcpToolCall(toolset, toolCall);
59
+ toMcpToolName("support.ticket.get");
60
+ ```
61
+
62
+ LangChain and LangGraph:
63
+
64
+ ```ts
65
+ buildLangChainToolDescriptors(registry, { context });
66
+ parseLangChainToolCall(toolset, toolCall);
67
+ toLangChainToolName("support.ticket.get");
68
+ ```
69
+
70
+ Semantic Kernel:
71
+
72
+ ```ts
73
+ buildSemanticKernelFunctions(registry, { context });
74
+ parseSemanticKernelFunctionCall(functionSet, functionCall);
75
+ toSemanticKernelFunctionName("support.ticket.get");
76
+ ```
77
+
78
+ ## CLI
79
+
80
+ All adapter commands accept a manifest path and context file:
81
+
82
+ ```bash
83
+ npm run build
84
+ node dist/cli.js anthropic-tools examples --context examples/support/openai/context.support_agent.json
85
+ node dist/cli.js gemini-tools examples --context examples/support/openai/context.support_agent.json
86
+ node dist/cli.js ai-sdk-tools examples --context examples/support/openai/context.support_agent.json
87
+ node dist/cli.js mcp-tools examples --context examples/support/openai/context.support_agent.json
88
+ node dist/cli.js langchain-tools examples --context examples/support/openai/context.support_agent.json
89
+ node dist/cli.js semantic-kernel-functions examples --context examples/support/openai/context.support_agent.json
90
+ ```
91
+
92
+ Use `--include-restricted` only when the host has deliberately narrowed context
93
+ and still intends to run deterministic approval, idempotency, auth, and commit
94
+ checks before any side effect.
95
+
96
+ ## Output Shapes
97
+
98
+ - OpenAI Responses returns `tools`.
99
+ - Anthropic Claude returns `tools`.
100
+ - Google Gemini returns `functionDeclarations`.
101
+ - Vercel AI SDK returns a keyed `tools` object.
102
+ - MCP returns `tools` descriptors.
103
+ - LangChain/LangGraph returns `tools` descriptors with metadata.
104
+ - Semantic Kernel returns `functions`.
105
+
106
+ All outputs include `bindings`, `excluded`, and `diagnostics`.
107
+
108
+ ## Host Responsibilities
109
+
110
+ Adapters only produce descriptors and parse calls. Host applications remain
111
+ responsible for model requests, real authorization, facts, approvals, execution,
112
+ storage, idempotency, verification, and durable audit logs.
package/docs/api.md ADDED
@@ -0,0 +1,115 @@
1
+ # API
2
+
3
+ AICF exposes a small TypeScript API and the `aicf` CLI. The package is public
4
+ npm release-candidate material, and the exported surface is treated as public
5
+ developer API.
6
+
7
+ Import from the package root:
8
+
9
+ ```ts
10
+ import {
11
+ buildRegistry,
12
+ decideCapability,
13
+ loadManifests,
14
+ validateManifests
15
+ } from "ai-capability-framework";
16
+ ```
17
+
18
+ ## Boundaries
19
+
20
+ The library does not call models, execute capabilities, verify host
21
+ authorization, persist actions, collect approvals, write audit logs, or perform
22
+ side effects. Host applications own runtime execution and storage.
23
+
24
+ ## Core Manifest APIs
25
+
26
+ - `loadManifests(options)` reads YAML and JSON manifests from convention paths.
27
+ - `validateManifests(manifests, options)` validates schemas, duplicate IDs, and
28
+ manifest cross-references.
29
+ - `buildRegistry(manifests)` returns lookup maps and grouped manifest arrays.
30
+ - `inspectRegistry(registry)` returns counts, capability groupings, eval
31
+ coverage, and warnings.
32
+
33
+ Diagnostics use `AicfDiagnostic` with `code`, `path`, optional `kind`, optional
34
+ `id`, and a human-readable `message`.
35
+
36
+ ## Decision APIs
37
+
38
+ - `decideCapability(registry, request)` evaluates select, prepare, or commit
39
+ status for one capability.
40
+ - `evaluatePolicy(capability, request)` evaluates deterministic policy metadata.
41
+ - `evaluateLifecycle(capability, request)` evaluates lifecycle availability.
42
+
43
+ Decision results return `status`, `reasons`, `requiredApprovals`, `policy`,
44
+ `lifecycle`, and an audit preview. The audit preview is not persisted.
45
+
46
+ ## Provider And Runtime Adapters
47
+
48
+ - `buildOpenAIResponsesTools(registry, options)` exports OpenAI-compatible
49
+ function tools plus bindings, exclusions, and diagnostics.
50
+ - `parseOpenAIResponsesToolCall(toolset, call)` maps a Responses
51
+ `function_call` item back to a capability ID and validates arguments.
52
+ - `toOpenAIResponsesToolName(capabilityId, options)` converts an AICF capability
53
+ ID into a deterministic OpenAI-safe tool name.
54
+ - `buildAnthropicClaudeTools(registry, options)` exports Anthropic Claude tool
55
+ definitions.
56
+ - `parseAnthropicClaudeToolUse(toolset, toolUse)` maps Claude `tool_use` blocks
57
+ back to AICF capability requests.
58
+ - `buildGeminiFunctionDeclarations(registry, options)` exports Gemini function
59
+ declarations.
60
+ - `parseGeminiFunctionCall(functionSet, functionCall)` parses Gemini
61
+ `functionCall` objects.
62
+ - `buildAiSdkTools(registry, options)` exports a Vercel AI SDK-compatible tool
63
+ map without `execute` handlers.
64
+ - `parseAiSdkToolCall(toolset, toolCall)` parses AI SDK tool calls.
65
+ - `buildMcpToolDescriptors(registry, options)` exports MCP tool descriptors.
66
+ - `parseMcpToolCall(toolset, toolCall)` parses MCP `tools/call` requests.
67
+ - `buildLangChainToolDescriptors(registry, options)` exports LangChain and
68
+ LangGraph descriptor metadata without callable implementations.
69
+ - `parseLangChainToolCall(toolset, toolCall)` parses LangChain-style tool calls.
70
+ - `buildSemanticKernelFunctions(registry, options)` exports Semantic Kernel
71
+ function metadata.
72
+ - `parseSemanticKernelFunctionCall(functionSet, functionCall)` parses Semantic
73
+ Kernel manual invocation calls.
74
+
75
+ Adapters emit descriptor JSON only. They do not call providers or execute tool
76
+ handlers. See [the adapter guide](adapters.md) for output shapes and CLI usage.
77
+
78
+ ## Eval Runner APIs
79
+
80
+ - `loadEvalResults(path)` reads and validates a public-safe eval result fixture.
81
+ - `scoreEvalCase(evalCase, candidate, registry)` scores one eval/candidate pair.
82
+ - `runEvalSuite(registry, candidates, options)` scores loaded eval manifests
83
+ against candidate results.
84
+ - `formatEvalSuiteResult(result)` formats a readable text summary.
85
+
86
+ Eval runner output includes suite status, per-eval status, scorer results, and
87
+ diagnostics. It scores summarized behavior only, not raw model traces.
88
+
89
+ ## CLI
90
+
91
+ ```bash
92
+ aicf validate [path]
93
+ aicf inspect [path]
94
+ aicf decide <path> --request <decision.json>
95
+ aicf openai-tools <path> --context <context.json> [--include-restricted]
96
+ aicf anthropic-tools <path> --context <context.json> [--include-restricted]
97
+ aicf gemini-tools <path> --context <context.json> [--include-restricted]
98
+ aicf ai-sdk-tools <path> --context <context.json> [--include-restricted]
99
+ aicf mcp-tools <path> --context <context.json> [--include-restricted]
100
+ aicf langchain-tools <path> --context <context.json> [--include-restricted]
101
+ aicf semantic-kernel-functions <path> --context <context.json> [--include-restricted]
102
+ aicf eval <path> --results <results.json> [--format text|json]
103
+ ```
104
+
105
+ Denied decisions exit `0` when the decision was evaluated successfully. Eval
106
+ runs exit `0` only when all targeted evals pass.
107
+
108
+ ## Versioning
109
+
110
+ The package version uses semver. Manifest schemas use `schema_version: "1.0"`.
111
+ The 1.0 validator intentionally does not support the pre-release 0.1 schema.
112
+ Changes to TypeScript exports, CLI output, or schema semantics must be covered
113
+ by tests before release. Public API or schema changes must also update
114
+ `CHANGELOG.md`, the migration notes when relevant, and the GitHub release notes
115
+ checklist in `docs/release.md`.
@@ -0,0 +1,47 @@
1
+ # Capability Manifests
2
+
3
+ A capability manifest is the contract between model-facing intent and
4
+ application-owned execution.
5
+
6
+ See [the 1.0 spec](spec.md) for the shared vocabulary.
7
+
8
+ ## Required Fields
9
+
10
+ - `schema_version`: Manifest schema version.
11
+ - `id`: Stable dotted identifier such as `support.refund.prepare_case`.
12
+ - `version`: Semantic version for the capability contract.
13
+ - `status`: Lifecycle state such as `draft`, `active`, or `deprecated`.
14
+ - `name` and `summary`: Human-readable labels for operators and maintainers.
15
+ - `model_description`: Short model-facing description of when to use the
16
+ capability.
17
+ - `capability_type`: The broad action class, such as read, compute, prepare, or
18
+ commit.
19
+ - `autonomy_tier`: Maximum autonomy level allowed for the capability.
20
+ - `risk_tier`: Safety and business risk tier.
21
+ - `input_schema` and `output_schema`: JSON Schema contracts.
22
+ - `side_effects`: Explicit flags for data reads, writes, money movement,
23
+ external messages, permission changes, and irreversible actions.
24
+ - `authorization`: Required permissions and scoping rules.
25
+ - `policy`: Deny, approval, and review rules.
26
+ - `lifecycle`: Explicit prepare, preview, approve, commit, verify, and audit
27
+ flags.
28
+ - `idempotency`: Optional commit idempotency requirements.
29
+ - `observability`: Logging and trace expectations.
30
+
31
+ ## Lifecycle Guidance
32
+
33
+ Use `write_prepare_only` for capabilities that assemble a proposed action but do
34
+ not execute it. Use `write_commit` only when the application can enforce
35
+ authorization, idempotency, approval state, verification, and audit.
36
+
37
+ High-risk capabilities should require explicit approval before commit. Critical
38
+ capabilities should usually be split into separate prepare and commit
39
+ capabilities.
40
+
41
+ Commit capabilities should be excluded from normal model tool sets until
42
+ application code has verified approval and idempotency state.
43
+
44
+ AICF tooling loads and validates capability manifests, builds registries,
45
+ evaluates deterministic select, prepare, and commit decisions, exports adapter
46
+ metadata, and scores eval fixtures. It does not execute capabilities, persist
47
+ actions, collect approvals, or perform side effects.
@@ -0,0 +1,59 @@
1
+ # Control Plane
2
+
3
+ The control plane provides deterministic decision APIs for capability selection,
4
+ preparation, and commit gating. It does not execute capabilities, store action
5
+ state, call models, or enforce host application authorization.
6
+
7
+ ## Decision API
8
+
9
+ Use `decideCapability(registry, request)` with a registry built from validated
10
+ manifests.
11
+
12
+ The request includes:
13
+
14
+ - `capabilityId`
15
+ - `operation`: `select`, `prepare`, or `commit`
16
+ - `args`
17
+ - `context.permissions`
18
+ - `context.autonomyTier`
19
+ - optional `facts`
20
+ - optional `approval`
21
+ - optional `idempotencyKey`
22
+
23
+ The result includes:
24
+
25
+ - `status`: `allowed`, `approval_required`, or `denied`
26
+ - `reasons`
27
+ - `requiredApprovals`
28
+ - `policy`
29
+ - `lifecycle`
30
+ - `audit` preview
31
+
32
+ The audit preview is not persisted and has no timestamp. Host applications own
33
+ real audit logging.
34
+
35
+ ## Policy Semantics
36
+
37
+ - Missing permissions deny.
38
+ - Autonomy above the capability tier or policy max tier denies.
39
+ - `select` decisions check tool-set eligibility with permissions and autonomy.
40
+ - `prepare` and `commit` decisions evaluate fact-dependent policy rules.
41
+ - `deny_if` rules read `facts[rule]`.
42
+ - A true deny fact denies.
43
+ - A missing deny fact fails closed and denies.
44
+ - `approval_required_if` can return `approval_required` for prepare decisions.
45
+ - Commit decisions missing required approval deny.
46
+ - Commit decisions missing required idempotency deny.
47
+
48
+ ## CLI
49
+
50
+ Run a decision from a JSON request file:
51
+
52
+ ```bash
53
+ npm run build
54
+ node dist/cli.js decide examples --request examples/support/decisions/support.refund.commit_case.allowed.json
55
+ ```
56
+
57
+ Denied decisions still exit `0` because the decision was evaluated
58
+ successfully. Invalid manifests, invalid request files, and unknown capability
59
+ IDs exit nonzero.
@@ -0,0 +1,80 @@
1
+ # Eval Runner
2
+
3
+ The deterministic eval runner scores AICF eval manifests against candidate
4
+ result fixtures without calling models, executing capabilities, storing traces,
5
+ or reading raw provider payloads.
6
+
7
+ Host applications are responsible for producing candidate results. AICF only
8
+ validates and scores the summarized behavior.
9
+
10
+ ## Candidate Fixture
11
+
12
+ Candidate fixtures are JSON files with `schema_version: "1.0"` and a `results`
13
+ array. Each result is keyed by `eval_id` and may include selected capabilities,
14
+ tool calls, policy decision, action state, committed capabilities, refusal
15
+ information, and response text.
16
+
17
+ ```json
18
+ {
19
+ "schema_version": "1.0",
20
+ "results": [
21
+ {
22
+ "eval_id": "support.refund.prepare_case.valid",
23
+ "selected_capabilities": ["support.refund.prepare_case"],
24
+ "tool_calls": [
25
+ {
26
+ "capability_id": "support.refund.prepare_case",
27
+ "args": {
28
+ "ticket_id": "TCK-1001"
29
+ }
30
+ }
31
+ ],
32
+ "action_state": "prepared",
33
+ "committed_capabilities": []
34
+ }
35
+ ]
36
+ }
37
+ ```
38
+
39
+ Keep fixtures synthetic and public-safe. Do not commit raw prompts, raw traces,
40
+ provider payloads, customer records, secrets, internal endpoints, or private
41
+ documents.
42
+
43
+ ## Scorers
44
+
45
+ The runner supports the scorer types used by the public eval examples:
46
+
47
+ - `tool_selection_includes`
48
+ - `tool_selection_excludes`
49
+ - `tool_input_json_subset`
50
+ - `policy_decision_matches`
51
+ - `no_unapproved_commit`
52
+ - `refusal_present`
53
+ - `response_excludes_private_detail`
54
+
55
+ It also enforces `expected.response.must_include` and
56
+ `expected.response.must_not_include` whenever those fields are present. Unknown
57
+ scorer types fail closed.
58
+
59
+ `no_unapproved_commit` fails when a candidate has `action_state: "committed"`,
60
+ non-empty `committed_capabilities`, or a tool call to a capability whose
61
+ manifest is a commit capability.
62
+
63
+ ## CLI
64
+
65
+ Run the full public example fixture:
66
+
67
+ ```bash
68
+ npm run build
69
+ node dist/cli.js eval examples --results examples/eval-results/public.results.passing.json
70
+ ```
71
+
72
+ For machine-readable output:
73
+
74
+ ```bash
75
+ node dist/cli.js eval examples --results examples/eval-results/public.results.passing.json --format json
76
+ ```
77
+
78
+ The command exits `0` only when manifests are valid, the result fixture is valid,
79
+ every loaded eval has a candidate result, and all scorer checks pass. Failed
80
+ checks, missing candidates, invalid fixtures, and unknown scorers exit nonzero.
package/docs/evals.md ADDED
@@ -0,0 +1,34 @@
1
+ # Evals
2
+
3
+ Eval cases keep capability behavior from drifting as prompts, models, policies,
4
+ and routing code change.
5
+
6
+ Each eval should be synthetic, deterministic, and tied to a specific behavior.
7
+ See [the 1.0 spec](spec.md) for public-safe example rules.
8
+
9
+ ## Useful Eval Types
10
+
11
+ - Capability selection includes the correct capability.
12
+ - Capability selection excludes unsafe or premature commit capabilities.
13
+ - Extracted arguments match the user request and available context.
14
+ - The action remains in a prepared state when approval is required.
15
+ - The model refuses or redirects requests outside the allowed policy.
16
+ - Tool results and final responses do not expose private implementation detail.
17
+ - Commit capabilities remain excluded until approval is verified.
18
+
19
+ ## Public Example Rule
20
+
21
+ Public examples must use fake identifiers, fake users, fake tenants, and generic
22
+ domains. Do not commit real traces, customer records, raw prompts, internal
23
+ schemas, or provider payloads.
24
+
25
+ AICF validation checks that eval files referenced by capability manifests are
26
+ present in the loaded manifest bundle.
27
+
28
+ Decision behavior should be covered with synthetic request examples and tests
29
+ for permission denial, autonomy denial, approval requirements, deny facts,
30
+ missing facts, approval state, and idempotency keys.
31
+
32
+ AICF includes a deterministic eval runner for scoring public-safe candidate
33
+ result fixtures against eval manifests. See [the eval runner guide](eval-runner.md)
34
+ for fixture shape, scorer behavior, and CLI usage.
@@ -0,0 +1,51 @@
1
+ # Getting Started
2
+
3
+ AICF describes AI-facing application behavior as public, typed manifests.
4
+
5
+ Use it when you need to answer these questions:
6
+
7
+ - What can the model select?
8
+ - What input does the application accept?
9
+ - What output can the application return?
10
+ - What side effects can happen?
11
+ - Who is allowed to use the capability?
12
+ - What policy gates or approvals apply?
13
+ - What evals prove the behavior is stable?
14
+
15
+ ## Basic Workflow
16
+
17
+ 1. Read the [1.0 spec](spec.md).
18
+ 2. Identify one application behavior that the model may request.
19
+ 3. Write a capability manifest for that behavior.
20
+ 4. Define strict input and output JSON Schemas.
21
+ 5. Mark side effects explicitly.
22
+ 6. Add authorization and policy requirements.
23
+ 7. Link synthetic eval cases that test selection, arguments, refusals, and
24
+ approval boundaries.
25
+ 8. Run `npm run validate`.
26
+
27
+ ## CLI Workflow
28
+
29
+ - `npm run validate` builds the CLI and validates the public examples.
30
+ - `npm run inspect` prints a readable registry summary.
31
+ - After building, `node dist/cli.js validate <path>` and
32
+ `node dist/cli.js inspect <path>` can target another manifest bundle.
33
+ - `node dist/cli.js decide <path> --request <decision.json>` evaluates a
34
+ deterministic decision request and prints JSON.
35
+ - `node dist/cli.js openai-tools <path> --context <context.json>` exports
36
+ OpenAI Responses function tools without calling a model.
37
+ - `node dist/cli.js eval <path> --results <results.json>` scores deterministic
38
+ candidate eval results without calling a model.
39
+
40
+ ## Design Rules
41
+
42
+ - Give the model the smallest relevant set of capabilities for a task.
43
+ - Use deterministic application code for validation, permissions, policy,
44
+ execution, and audit.
45
+ - Do not expose raw databases, unrestricted HTTP clients, shell access, payment
46
+ tools, or admin tools directly to the model.
47
+ - Treat retrieved content and tool results as untrusted unless your application
48
+ has explicitly marked them as authoritative.
49
+ - Keep examples synthetic and public-safe.
50
+ - Keep commit capabilities separate from prepare capabilities unless the action
51
+ is explicitly low-risk and policy allows auto-commit.