@aicf/agent-skills 1.0.0-rc.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.codex-plugin/plugin.json +40 -0
- package/CHANGELOG.md +28 -0
- package/LICENSE +21 -0
- package/README.md +114 -0
- package/assets/aicf-agent-skills-icon.svg +7 -0
- package/assets/aicf-agent-skills-logo.svg +9 -0
- package/docs/codex-plugin-distribution.md +50 -0
- package/docs/installation.md +96 -0
- package/docs/public-release-checklist.md +57 -0
- package/docs/skill-authoring-standard.md +55 -0
- package/docs/skill-index.md +157 -0
- package/docs/validation.md +50 -0
- package/package.json +61 -0
- package/scripts/README.md +6 -0
- package/scripts/aicf-skills.mjs +51 -0
- package/scripts/check-release-readiness.mjs +197 -0
- package/scripts/check-skills-public.mjs +12 -0
- package/scripts/check-trigger-coverage.mjs +20 -0
- package/scripts/generate-skill-index.mjs +30 -0
- package/scripts/install-skills.mjs +31 -0
- package/scripts/lib/skill-common.mjs +511 -0
- package/scripts/validate-skills.mjs +13 -0
- package/skills/README.md +12 -0
- package/skills/aicf-action-lifecycle/SKILL.md +87 -0
- package/skills/aicf-action-lifecycle/assets/action-handler.template.ts +24 -0
- package/skills/aicf-action-lifecycle/assets/approval-record.json +9 -0
- package/skills/aicf-action-lifecycle/assets/prepared-action-record.json +11 -0
- package/skills/aicf-action-lifecycle/references/action-state-machine.md +17 -0
- package/skills/aicf-action-lifecycle/references/approval-patterns.md +9 -0
- package/skills/aicf-action-lifecycle/references/idempotency-and-audit.md +8 -0
- package/skills/aicf-action-lifecycle/references/verification-patterns.md +7 -0
- package/skills/aicf-capability-authoring/SKILL.md +100 -0
- package/skills/aicf-capability-authoring/assets/capability.manifest.yaml +31 -0
- package/skills/aicf-capability-authoring/assets/entity.manifest.yaml +12 -0
- package/skills/aicf-capability-authoring/assets/eval-case.yaml +12 -0
- package/skills/aicf-capability-authoring/references/checklist.md +13 -0
- package/skills/aicf-capability-authoring/references/examples.md +12 -0
- package/skills/aicf-capability-authoring/references/field-guide.md +13 -0
- package/skills/aicf-capability-authoring/references/manifest-decision-tree.md +10 -0
- package/skills/aicf-control-plane-ui/SKILL.md +90 -0
- package/skills/aicf-control-plane-ui/assets/control-plane-wireframe.md +29 -0
- package/skills/aicf-control-plane-ui/assets/ui-acceptance-checklist.md +11 -0
- package/skills/aicf-control-plane-ui/references/api-contracts.md +14 -0
- package/skills/aicf-control-plane-ui/references/approval-console.md +12 -0
- package/skills/aicf-control-plane-ui/references/control-plane-screens.md +14 -0
- package/skills/aicf-control-plane-ui/references/operational-ux.md +7 -0
- package/skills/aicf-controls-and-budgets/SKILL.md +88 -0
- package/skills/aicf-controls-and-budgets/assets/budget-fixture.json +15 -0
- package/skills/aicf-controls-and-budgets/assets/controls-policy.yaml +14 -0
- package/skills/aicf-controls-and-budgets/references/budget-policy.md +12 -0
- package/skills/aicf-controls-and-budgets/references/circuit-breakers.md +13 -0
- package/skills/aicf-controls-and-budgets/references/control-types.md +10 -0
- package/skills/aicf-docs-and-examples/SKILL.md +87 -0
- package/skills/aicf-docs-and-examples/assets/example-readme-template.md +21 -0
- package/skills/aicf-docs-and-examples/assets/provider-guide-template.md +17 -0
- package/skills/aicf-docs-and-examples/references/docs-style-guide.md +9 -0
- package/skills/aicf-docs-and-examples/references/example-app-checklist.md +9 -0
- package/skills/aicf-docs-and-examples/references/public-api-docs.md +11 -0
- package/skills/aicf-docs-and-examples/references/quickstart-template.md +17 -0
- package/skills/aicf-eval-authoring/SKILL.md +86 -0
- package/skills/aicf-eval-authoring/assets/eval-case.yaml +17 -0
- package/skills/aicf-eval-authoring/assets/eval-result.json +15 -0
- package/skills/aicf-eval-authoring/assets/promptfoo.config.yaml +12 -0
- package/skills/aicf-eval-authoring/references/deterministic-scorers.md +13 -0
- package/skills/aicf-eval-authoring/references/eval-rubric.md +10 -0
- package/skills/aicf-eval-authoring/references/golden-test-patterns.md +8 -0
- package/skills/aicf-eval-authoring/references/real-provider-evals.md +8 -0
- package/skills/aicf-governance-lifecycle/SKILL.md +91 -0
- package/skills/aicf-governance-lifecycle/assets/lifecycle-change-request.md +21 -0
- package/skills/aicf-governance-lifecycle/assets/riskwaiver.md +17 -0
- package/skills/aicf-governance-lifecycle/references/compatibility-checks.md +21 -0
- package/skills/aicf-governance-lifecycle/references/impact-analysis.md +13 -0
- package/skills/aicf-governance-lifecycle/references/lifecycle-state-machine.md +15 -0
- package/skills/aicf-governance-lifecycle/references/riskcompiler-rules.md +13 -0
- package/skills/aicf-migration/SKILL.md +87 -0
- package/skills/aicf-migration/assets/legacy-tool-inventory.csv +3 -0
- package/skills/aicf-migration/assets/migration-plan.md +19 -0
- package/skills/aicf-migration/references/conversion-playbook.md +9 -0
- package/skills/aicf-migration/references/legacy-tool-riskmap.md +14 -0
- package/skills/aicf-migration/references/migration-inventory.md +12 -0
- package/skills/aicf-migration/references/rollout-plan.md +12 -0
- package/skills/aicf-observability-replay/SKILL.md +88 -0
- package/skills/aicf-observability-replay/assets/replay-fixture.json +23 -0
- package/skills/aicf-observability-replay/assets/trace-to-golden-template.yaml +14 -0
- package/skills/aicf-observability-replay/references/redacted-telemetry.md +7 -0
- package/skills/aicf-observability-replay/references/replay-simulation.md +10 -0
- package/skills/aicf-observability-replay/references/trace-span-map.md +17 -0
- package/skills/aicf-observability-replay/references/trace-to-golden.md +12 -0
- package/skills/aicf-policy-and-risk/SKILL.md +88 -0
- package/skills/aicf-policy-and-risk/assets/policy-broker.template.ts +29 -0
- package/skills/aicf-policy-and-risk/assets/policy-decision-fixture.json +15 -0
- package/skills/aicf-policy-and-risk/references/deny-approval-allow-matrix.md +12 -0
- package/skills/aicf-policy-and-risk/references/policy-broker-contract.md +13 -0
- package/skills/aicf-policy-and-risk/references/risktier-rules.md +10 -0
- package/skills/aicf-policy-and-risk/references/semantic-invariants.md +9 -0
- package/skills/aicf-provider-conformance/SKILL.md +88 -0
- package/skills/aicf-provider-conformance/assets/provider-conformance-case.yaml +18 -0
- package/skills/aicf-provider-conformance/assets/provider-matrix.json +19 -0
- package/skills/aicf-provider-conformance/references/provider-normalization-matrix.md +14 -0
- package/skills/aicf-provider-conformance/references/provider-runtime-boundaries.md +10 -0
- package/skills/aicf-provider-conformance/references/schema-compatibility.md +8 -0
- package/skills/aicf-provider-conformance/references/tool-call-mapping.md +8 -0
- package/skills/aicf-release-hygiene/SKILL.md +85 -0
- package/skills/aicf-release-hygiene/assets/release-report-template.md +19 -0
- package/skills/aicf-release-hygiene/references/forbidden-artifacts.md +12 -0
- package/skills/aicf-release-hygiene/references/package-smoke-test.md +10 -0
- package/skills/aicf-release-hygiene/references/release-checklist.md +10 -0
- package/skills/aicf-release-hygiene/references/source-archive-rules.md +8 -0
- package/skills/aicf-runtime-integration/SKILL.md +87 -0
- package/skills/aicf-runtime-integration/assets/context-builder.template.ts +22 -0
- package/skills/aicf-runtime-integration/assets/handler-registry.template.ts +20 -0
- package/skills/aicf-runtime-integration/assets/runtime-integration-plan.md +23 -0
- package/skills/aicf-runtime-integration/references/capability-router-patterns.md +8 -0
- package/skills/aicf-runtime-integration/references/context-builder-patterns.md +8 -0
- package/skills/aicf-runtime-integration/references/handler-registry-patterns.md +8 -0
- package/skills/aicf-runtime-integration/references/policy-broker-adapter.md +8 -0
- package/skills/aicf-runtime-integration/references/runtime-integration-checklist.md +10 -0
- package/skills/aicf-security-redteam/SKILL.md +89 -0
- package/skills/aicf-security-redteam/assets/promptfoo-redteam.config.yaml +13 -0
- package/skills/aicf-security-redteam/assets/redteam-case.yaml +15 -0
- package/skills/aicf-security-redteam/references/prompt-injection-patterns.md +12 -0
- package/skills/aicf-security-redteam/references/redteam-acceptance.md +11 -0
- package/skills/aicf-security-redteam/references/security-test-pack-map.md +11 -0
- package/skills/aicf-security-redteam/references/tool-misuse-patterns.md +11 -0
- package/skills/aicf-skill-pack-maintenance/SKILL.md +88 -0
- package/skills/aicf-skill-pack-maintenance/assets/skill-template.md +52 -0
- package/skills/aicf-skill-pack-maintenance/references/overlap-prevention.md +9 -0
- package/skills/aicf-skill-pack-maintenance/references/plugin-packaging.md +8 -0
- package/skills/aicf-skill-pack-maintenance/references/skill-authoring-rules.md +10 -0
- package/skills/aicf-skill-pack-maintenance/references/trigger-evaluation.md +9 -0
- package/skills/aicf-storage-and-aws/SKILL.md +86 -0
- package/skills/aicf-storage-and-aws/assets/dynamodb-store.template.ts +21 -0
- package/skills/aicf-storage-and-aws/assets/postgres-store.template.ts +22 -0
- package/skills/aicf-storage-and-aws/assets/step-functions-approval.template.ts +15 -0
- package/skills/aicf-storage-and-aws/references/aws-adapter-boundaries.md +8 -0
- package/skills/aicf-storage-and-aws/references/least-privilege-iam.md +11 -0
- package/skills/aicf-storage-and-aws/references/step-functions-approval.md +10 -0
- package/skills/aicf-storage-and-aws/references/store-interface-map.md +10 -0
- package/skills/aicf-trust-redaction-retention/SKILL.md +88 -0
- package/skills/aicf-trust-redaction-retention/assets/redaction-policy.yaml +15 -0
- package/skills/aicf-trust-redaction-retention/assets/trust-labelled-context.json +18 -0
- package/skills/aicf-trust-redaction-retention/references/redaction-rules.md +8 -0
- package/skills/aicf-trust-redaction-retention/references/retention-rules.md +11 -0
- package/skills/aicf-trust-redaction-retention/references/taint-rules.md +9 -0
- package/skills/aicf-trust-redaction-retention/references/trust-labels.md +16 -0
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: aicf-docs-and-examples
|
|
3
|
+
description:
|
|
4
|
+
Update AICF public docs, quickstarts, examples, README files, API references, provider
|
|
5
|
+
guides, and sample apps so they match current public contracts and implementation.
|
|
6
|
+
license: MIT
|
|
7
|
+
compatibility:
|
|
8
|
+
Codex and Agent Skills-compatible coding agents. Works in repositories using AICF or
|
|
9
|
+
migrating to AICF.
|
|
10
|
+
metadata:
|
|
11
|
+
aicf.skill.version: "1.0.0-rc.4"
|
|
12
|
+
aicf.skill.package: "@aicf/agent-skills"
|
|
13
|
+
aicf.skill.category: "docs-examples"
|
|
14
|
+
aicf.skill.scope: "builder"
|
|
15
|
+
aicf.skill.strictness: "fail-closed-runtime"
|
|
16
|
+
---
|
|
17
|
+
|
|
18
|
+
# AICF Docs And Examples
|
|
19
|
+
|
|
20
|
+
## Purpose
|
|
21
|
+
|
|
22
|
+
Guide updates to public AICF docs and examples so they are accurate, simple,
|
|
23
|
+
implementation-aligned, and public-safe.
|
|
24
|
+
|
|
25
|
+
## Use this skill when
|
|
26
|
+
|
|
27
|
+
- Updating README, quickstarts, API docs, provider guides, release docs, or example
|
|
28
|
+
apps.
|
|
29
|
+
- Adding docs for changed schemas, commands, exports, providers, runtime behavior,
|
|
30
|
+
governance, security, or evals.
|
|
31
|
+
- Reviewing docs for public safety and current implementation accuracy.
|
|
32
|
+
|
|
33
|
+
## Do not use this skill when
|
|
34
|
+
|
|
35
|
+
- Implementing runtime/provider behavior without a docs task.
|
|
36
|
+
- Copying private specs or local notes into public docs.
|
|
37
|
+
- Adding examples that require credentials or live calls by default.
|
|
38
|
+
|
|
39
|
+
## Inputs to inspect first
|
|
40
|
+
|
|
41
|
+
- Changed public contracts, README, docs index, examples, API docs, package scripts,
|
|
42
|
+
changelog, and validation commands.
|
|
43
|
+
- References: [docs style guide](references/docs-style-guide.md),
|
|
44
|
+
[example app checklist](references/example-app-checklist.md),
|
|
45
|
+
[public API docs](references/public-api-docs.md), and
|
|
46
|
+
[quickstart template](references/quickstart-template.md).
|
|
47
|
+
|
|
48
|
+
## Workflow
|
|
49
|
+
|
|
50
|
+
1. Identify the public contract, command, export, schema, provider, CLI, or example that
|
|
51
|
+
changed.
|
|
52
|
+
2. Update the smallest set of public docs and examples needed to keep the learning path
|
|
53
|
+
accurate.
|
|
54
|
+
3. Keep provider-neutral docs neutral and label provider-specific docs clearly.
|
|
55
|
+
4. Include exact commands and short expected output excerpts when useful.
|
|
56
|
+
5. Keep examples synthetic, credential-free by default, and runnable or explicitly
|
|
57
|
+
documented as docs-only.
|
|
58
|
+
6. Run docs, example, validation, and public-safety checks available in the repo.
|
|
59
|
+
7. Update changelog or release notes when public behavior changed.
|
|
60
|
+
|
|
61
|
+
Use [example README template](assets/example-readme-template.md) and
|
|
62
|
+
[provider guide template](assets/provider-guide-template.md) for new docs.
|
|
63
|
+
|
|
64
|
+
## Required outputs
|
|
65
|
+
|
|
66
|
+
- Updated public docs/examples aligned with implementation.
|
|
67
|
+
- Commands run and any blockers for examples that cannot be executed.
|
|
68
|
+
- Public-safety notes for synthetic data and optional live steps.
|
|
69
|
+
|
|
70
|
+
## Validation
|
|
71
|
+
|
|
72
|
+
- Run docs checks, package checks, validation, and example tests relevant to the change.
|
|
73
|
+
- Confirm Markdown links resolve when link checking is available.
|
|
74
|
+
- Confirm no private or local-only material is included.
|
|
75
|
+
|
|
76
|
+
## Hard rules
|
|
77
|
+
|
|
78
|
+
- Do not copy private material verbatim.
|
|
79
|
+
- Do not include credentials, account-specific IDs, private traces, or provider
|
|
80
|
+
transport bodies.
|
|
81
|
+
- Do not present optional live integrations as default behavior.
|
|
82
|
+
- Do not make docs broader than the implemented public contract.
|
|
83
|
+
|
|
84
|
+
## Handoff format
|
|
85
|
+
|
|
86
|
+
Report docs/examples changed, commands run, expected output added, public-safety checks,
|
|
87
|
+
and remaining docs gaps.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# Example Name
|
|
2
|
+
|
|
3
|
+
## What this demonstrates
|
|
4
|
+
|
|
5
|
+
## Fake data used
|
|
6
|
+
|
|
7
|
+
## Commands
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
npm run validate
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
## Expected output excerpt
|
|
14
|
+
|
|
15
|
+
```text
|
|
16
|
+
Validation passed.
|
|
17
|
+
```
|
|
18
|
+
|
|
19
|
+
## Safety notes
|
|
20
|
+
|
|
21
|
+
This example is synthetic and does not require credentials by default.
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
# Docs Style Guide
|
|
2
|
+
|
|
3
|
+
- Lead with what the developer can do.
|
|
4
|
+
- Keep concepts short and link to reference docs.
|
|
5
|
+
- Use exact commands and public-safe expected output.
|
|
6
|
+
- Separate default mock paths from optional live paths.
|
|
7
|
+
- State host responsibilities clearly.
|
|
8
|
+
- Keep AICF boundaries explicit: models propose; apps validate, authorize, execute, and
|
|
9
|
+
audit.
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
# Example App Checklist
|
|
2
|
+
|
|
3
|
+
- Uses synthetic data and `example.com` when URLs are needed.
|
|
4
|
+
- Runs without credentials by default.
|
|
5
|
+
- Documents optional live setup separately.
|
|
6
|
+
- Shows expected output excerpts.
|
|
7
|
+
- Avoids private traces, provider transport details, and customer-like identifiers.
|
|
8
|
+
- Has validation or test instructions.
|
|
9
|
+
- Explains what the example proves.
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
# Public API Docs
|
|
2
|
+
|
|
3
|
+
API docs should include:
|
|
4
|
+
|
|
5
|
+
- import path.
|
|
6
|
+
- main types and helpers.
|
|
7
|
+
- minimal usage example.
|
|
8
|
+
- boundary and host responsibilities.
|
|
9
|
+
- validation behavior.
|
|
10
|
+
- optional dependency behavior where relevant.
|
|
11
|
+
- related CLI commands or examples.
|
|
@@ -0,0 +1,86 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: aicf-eval-authoring
|
|
3
|
+
description:
|
|
4
|
+
Create AICF capability-aware evals, golden tests, rubrics, deterministic fixtures,
|
|
5
|
+
real-provider test cases, and regression suites. Use for AI quality, tool-choice
|
|
6
|
+
accuracy, action correctness, or model upgrades.
|
|
7
|
+
license: MIT
|
|
8
|
+
compatibility:
|
|
9
|
+
Codex and Agent Skills-compatible coding agents. Works in repositories using AICF or
|
|
10
|
+
migrating to AICF.
|
|
11
|
+
metadata:
|
|
12
|
+
aicf.skill.version: "1.0.0-rc.4"
|
|
13
|
+
aicf.skill.package: "@aicf/agent-skills"
|
|
14
|
+
aicf.skill.category: "evals"
|
|
15
|
+
aicf.skill.scope: "builder"
|
|
16
|
+
aicf.skill.strictness: "fail-closed-runtime"
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
# AICF Eval Authoring
|
|
20
|
+
|
|
21
|
+
## Purpose
|
|
22
|
+
|
|
23
|
+
Guide creation of deterministic, capability-aware evals that prove tool selection, input
|
|
24
|
+
validation, policy behavior, lifecycle outcomes, and provider-safe outputs.
|
|
25
|
+
|
|
26
|
+
## Use this skill when
|
|
27
|
+
|
|
28
|
+
- Adding eval cases for a new or changed capability.
|
|
29
|
+
- Creating golden tests for tool choice, args, policy, or action state.
|
|
30
|
+
- Building regression coverage for model upgrades or provider adapter changes.
|
|
31
|
+
- Exporting public-safe Promptfoo or live-eval fixtures.
|
|
32
|
+
|
|
33
|
+
## Do not use this skill when
|
|
34
|
+
|
|
35
|
+
- Running live provider calls by default.
|
|
36
|
+
- Storing transcripts, private prompts, provider transport details, or credentials.
|
|
37
|
+
- Replacing AICF validation with provider or eval-tool validation.
|
|
38
|
+
|
|
39
|
+
## Inputs to inspect first
|
|
40
|
+
|
|
41
|
+
- Capability manifests, example contexts, existing eval manifests, candidate result
|
|
42
|
+
fixtures, and scorer docs.
|
|
43
|
+
- References for [rubrics](references/eval-rubric.md),
|
|
44
|
+
[golden patterns](references/golden-test-patterns.md),
|
|
45
|
+
[deterministic scorers](references/deterministic-scorers.md), and
|
|
46
|
+
[live-provider evals](references/real-provider-evals.md).
|
|
47
|
+
|
|
48
|
+
## Workflow
|
|
49
|
+
|
|
50
|
+
1. Identify the capability, risk tier, operation, and behavior being proven.
|
|
51
|
+
2. Prefer deterministic fixtures before live-provider cases.
|
|
52
|
+
3. Add positive cases for correct selection and valid tool input.
|
|
53
|
+
4. Add negative cases for forbidden tools, invalid args, missing context, unsafe
|
|
54
|
+
commits, and private-detail leakage.
|
|
55
|
+
5. Add expected action state and policy decision assertions where lifecycle behavior
|
|
56
|
+
matters.
|
|
57
|
+
6. Keep live-provider evals opt-in and separated from normal checks.
|
|
58
|
+
7. Update docs or indexes only when the public eval surface changes.
|
|
59
|
+
|
|
60
|
+
Use [eval case](assets/eval-case.yaml), [eval result](assets/eval-result.json), and
|
|
61
|
+
[Promptfoo config](assets/promptfoo.config.yaml) templates where useful.
|
|
62
|
+
|
|
63
|
+
## Required outputs
|
|
64
|
+
|
|
65
|
+
- Eval cases with clear expected behavior and scorer coverage.
|
|
66
|
+
- Candidate result fixtures when deterministic CLI tests need them.
|
|
67
|
+
- A short explanation of what behavior the eval proves.
|
|
68
|
+
|
|
69
|
+
## Validation
|
|
70
|
+
|
|
71
|
+
- Run the eval command with public candidate results.
|
|
72
|
+
- Run manifest validation when fixtures or examples change.
|
|
73
|
+
- Confirm evals fail for unsafe or incorrect tool calls.
|
|
74
|
+
|
|
75
|
+
## Hard rules
|
|
76
|
+
|
|
77
|
+
- Evals do not call models unless explicitly configured as live tests.
|
|
78
|
+
- Do not include private input text, provider transport bodies, credentials, or real
|
|
79
|
+
customer data.
|
|
80
|
+
- Do not make eval pass criteria vague for high-risk behavior.
|
|
81
|
+
- Do not ignore unknown capability IDs or duplicate candidate results.
|
|
82
|
+
|
|
83
|
+
## Handoff format
|
|
84
|
+
|
|
85
|
+
Report eval files added, scorers used, behavior proven, commands run, and known coverage
|
|
86
|
+
gaps.
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
id: support.refund.prepare.requires_approval
|
|
2
|
+
description:
|
|
3
|
+
"Refund preparation should produce approval-required behavior for the synthetic
|
|
4
|
+
threshold case."
|
|
5
|
+
capability_under_test: support.refund.prepare_case
|
|
6
|
+
input:
|
|
7
|
+
user_message: "Prepare a refund review for ticket TICKET-100."
|
|
8
|
+
expected:
|
|
9
|
+
selected_capabilities:
|
|
10
|
+
- support.refund.prepare_case
|
|
11
|
+
action_state: approval_required
|
|
12
|
+
forbidden_tool_calls:
|
|
13
|
+
- support.refund.commit_case
|
|
14
|
+
scorers:
|
|
15
|
+
- selected_capability_exact
|
|
16
|
+
- no_forbidden_tool_call
|
|
17
|
+
- action_state_matches
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
{
|
|
2
|
+
"eval_id": "support.refund.prepare.requires_approval",
|
|
3
|
+
"candidate_id": "synthetic-candidate",
|
|
4
|
+
"selected_capabilities": ["support.refund.prepare_case"],
|
|
5
|
+
"tool_calls": [
|
|
6
|
+
{
|
|
7
|
+
"capability_id": "support.refund.prepare_case",
|
|
8
|
+
"args": {
|
|
9
|
+
"ticket_id": "TICKET-100",
|
|
10
|
+
"amount": 25
|
|
11
|
+
}
|
|
12
|
+
}
|
|
13
|
+
],
|
|
14
|
+
"action_state": "approval_required"
|
|
15
|
+
}
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
description: "AICF synthetic eval export template"
|
|
2
|
+
providers:
|
|
3
|
+
- echo
|
|
4
|
+
prompts:
|
|
5
|
+
- "Use only the provided AICF capability descriptions."
|
|
6
|
+
tests:
|
|
7
|
+
- vars:
|
|
8
|
+
user_message: "Prepare a synthetic support refund review."
|
|
9
|
+
assert:
|
|
10
|
+
- type: contains-json
|
|
11
|
+
value:
|
|
12
|
+
expected_policy: approval_required
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
# Deterministic Scorers
|
|
2
|
+
|
|
3
|
+
Useful scorer categories:
|
|
4
|
+
|
|
5
|
+
- selected capability exact match.
|
|
6
|
+
- tool input exact JSON.
|
|
7
|
+
- tool input allowed fields.
|
|
8
|
+
- no forbidden tool call.
|
|
9
|
+
- tool call sequence match.
|
|
10
|
+
- action state match.
|
|
11
|
+
- private detail exclusion.
|
|
12
|
+
|
|
13
|
+
Prefer deterministic scorers for release gates.
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
# Eval Rubric
|
|
2
|
+
|
|
3
|
+
Score these dimensions when relevant:
|
|
4
|
+
|
|
5
|
+
- Correct capability selected.
|
|
6
|
+
- Forbidden capabilities not selected.
|
|
7
|
+
- Tool args exactly match required fields or allowed field subset.
|
|
8
|
+
- Policy decision matches expected allow, deny, or approval-required behavior.
|
|
9
|
+
- Action state matches expected lifecycle.
|
|
10
|
+
- Model-facing output omits private diagnostics and internal details.
|
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
# Golden Test Patterns
|
|
2
|
+
|
|
3
|
+
- One narrow behavior per eval case.
|
|
4
|
+
- Synthetic context and stable identifiers.
|
|
5
|
+
- Explicit expected selected capabilities.
|
|
6
|
+
- Explicit forbidden tool calls for safety cases.
|
|
7
|
+
- Expected action state for prepare, approval, commit, and failure paths.
|
|
8
|
+
- Candidate results checked into public examples only when safe.
|
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
# Live Provider Eval Guidance
|
|
2
|
+
|
|
3
|
+
Live-provider evals are optional and must be skipped unless explicit environment flags
|
|
4
|
+
and caller-provided clients are present.
|
|
5
|
+
|
|
6
|
+
Keep live cases synthetic, small, and read/prepare only. Do not require live tests for
|
|
7
|
+
normal package checks. Convert useful live findings into deterministic fixtures before
|
|
8
|
+
relying on them for release gates.
|
|
@@ -0,0 +1,91 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: aicf-governance-lifecycle
|
|
3
|
+
description:
|
|
4
|
+
Manage AICF capability lifecycle, risk compiler rules, compatibility checks, and
|
|
5
|
+
impact analysis. Use when moving capabilities through draft, review, approved, canary,
|
|
6
|
+
production, deprecated, disabled, or removed states.
|
|
7
|
+
license: MIT
|
|
8
|
+
compatibility:
|
|
9
|
+
Codex and Agent Skills-compatible coding agents. Works in repositories using AICF or
|
|
10
|
+
migrating to AICF.
|
|
11
|
+
metadata:
|
|
12
|
+
aicf.skill.version: "1.0.0-rc.4"
|
|
13
|
+
aicf.skill.package: "@aicf/agent-skills"
|
|
14
|
+
aicf.skill.category: "governance"
|
|
15
|
+
aicf.skill.scope: "builder"
|
|
16
|
+
aicf.skill.strictness: "fail-closed-runtime"
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
# AICF Governance Lifecycle
|
|
20
|
+
|
|
21
|
+
## Purpose
|
|
22
|
+
|
|
23
|
+
Guide lifecycle, risk, compatibility, and impact work for AICF capabilities without
|
|
24
|
+
mutating production behavior or bypassing review gates.
|
|
25
|
+
|
|
26
|
+
## Use this skill when
|
|
27
|
+
|
|
28
|
+
- Promoting, deprecating, disabling, or removing a capability.
|
|
29
|
+
- Reviewing inferred risk, required controls, or security/eval coverage.
|
|
30
|
+
- Comparing manifest versions for compatibility.
|
|
31
|
+
- Assessing impact across entities, evals, policies, providers, and linked
|
|
32
|
+
prepare/commit paths.
|
|
33
|
+
|
|
34
|
+
## Do not use this skill when
|
|
35
|
+
|
|
36
|
+
- Implementing runtime handlers, provider adapters, storage, UI workflows, or model
|
|
37
|
+
loops.
|
|
38
|
+
- Changing manifest status without review evidence.
|
|
39
|
+
- Treating warnings as permission to ignore safety or runtime correctness errors.
|
|
40
|
+
|
|
41
|
+
## Inputs to inspect first
|
|
42
|
+
|
|
43
|
+
- Current and proposed capability manifests.
|
|
44
|
+
- Governance docs, eval/security coverage, conformance results, and release rules.
|
|
45
|
+
- References for [lifecycle states](references/lifecycle-state-machine.md),
|
|
46
|
+
[risk compiler rules](references/riskcompiler-rules.md),
|
|
47
|
+
[compatibility checks](references/compatibility-checks.md), and
|
|
48
|
+
[impact analysis](references/impact-analysis.md).
|
|
49
|
+
|
|
50
|
+
## Workflow
|
|
51
|
+
|
|
52
|
+
1. Determine the current lifecycle state from request context or manifest status
|
|
53
|
+
mapping.
|
|
54
|
+
2. Validate manifests and identify safety/runtime errors before considering promotion.
|
|
55
|
+
3. Compile inferred risk and compare it with declared risk.
|
|
56
|
+
4. Check required evals, security packs, policy metadata, owner evidence, and control
|
|
57
|
+
states.
|
|
58
|
+
5. Compare old and new manifest versions when changing a public contract.
|
|
59
|
+
6. Analyze direct impact on entities, evals, policy references, provider exports, and
|
|
60
|
+
linked lifecycle capabilities.
|
|
61
|
+
7. Produce an allowed/blocked lifecycle decision with required actions.
|
|
62
|
+
|
|
63
|
+
Use [lifecycle request](assets/lifecycle-change-request.md) and
|
|
64
|
+
[risk waiver](assets/riskwaiver.md) templates when documenting review evidence.
|
|
65
|
+
|
|
66
|
+
## Required outputs
|
|
67
|
+
|
|
68
|
+
- Lifecycle decision with reasons, warnings, blockers, and required actions.
|
|
69
|
+
- Risk control summary and compatibility classification.
|
|
70
|
+
- Impact report with missing coverage gaps.
|
|
71
|
+
|
|
72
|
+
## Validation
|
|
73
|
+
|
|
74
|
+
- Run manifest validation and the governance command or tests relevant to the change.
|
|
75
|
+
- Confirm high/critical capabilities have required controls or explicit public-safe
|
|
76
|
+
waivers.
|
|
77
|
+
- Confirm deprecated or removed capabilities include replacement and migration notes.
|
|
78
|
+
|
|
79
|
+
## Hard rules
|
|
80
|
+
|
|
81
|
+
- Do not mutate manifests while evaluating lifecycle unless the user explicitly requests
|
|
82
|
+
an implementation step.
|
|
83
|
+
- Do not promote a capability with missing required context, unsafe risk metadata, or
|
|
84
|
+
failing eval/security gates.
|
|
85
|
+
- Do not turn terminal removed state back into an active state.
|
|
86
|
+
- Do not copy private review notes into public governance evidence.
|
|
87
|
+
|
|
88
|
+
## Handoff format
|
|
89
|
+
|
|
90
|
+
Summarize transition requested, decision, blockers, required evidence, commands run, and
|
|
91
|
+
exact follow-up changes needed.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# Lifecycle Change Request
|
|
2
|
+
|
|
3
|
+
## Capability
|
|
4
|
+
|
|
5
|
+
## From state
|
|
6
|
+
|
|
7
|
+
## To state
|
|
8
|
+
|
|
9
|
+
## Reason
|
|
10
|
+
|
|
11
|
+
## Owner
|
|
12
|
+
|
|
13
|
+
## Validation evidence
|
|
14
|
+
|
|
15
|
+
## Eval/security evidence
|
|
16
|
+
|
|
17
|
+
## Risk and control summary
|
|
18
|
+
|
|
19
|
+
## Rollout or migration notes
|
|
20
|
+
|
|
21
|
+
## Decision
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# Compatibility Checks
|
|
2
|
+
|
|
3
|
+
Compatible changes:
|
|
4
|
+
|
|
5
|
+
- Description improvements.
|
|
6
|
+
- Optional input additions.
|
|
7
|
+
- Stricter approval or audit requirements.
|
|
8
|
+
|
|
9
|
+
Minor-review changes:
|
|
10
|
+
|
|
11
|
+
- New output fields.
|
|
12
|
+
- New enum values when consumers tolerate them.
|
|
13
|
+
- More precise model-facing guidance.
|
|
14
|
+
|
|
15
|
+
Breaking changes:
|
|
16
|
+
|
|
17
|
+
- Added required inputs.
|
|
18
|
+
- Removed or renamed inputs/outputs.
|
|
19
|
+
- Weaker risk, approval, audit, idempotency, or tenant requirements.
|
|
20
|
+
- Added side effects.
|
|
21
|
+
- Lifecycle changes that expose new commit behavior.
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
# Impact Analysis
|
|
2
|
+
|
|
3
|
+
Check direct impact on:
|
|
4
|
+
|
|
5
|
+
- Entity manifests and allowed actions.
|
|
6
|
+
- Eval cases and candidate results.
|
|
7
|
+
- Provider export descriptors and conformance cases.
|
|
8
|
+
- Prepare-to-commit links.
|
|
9
|
+
- Policy rules, permission names, and autonomy limits.
|
|
10
|
+
- Security packs, redaction, retention, and audit evidence.
|
|
11
|
+
- Docs and public examples that mention the capability.
|
|
12
|
+
|
|
13
|
+
Report missing coverage as an explicit gap.
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# Lifecycle State Machine
|
|
2
|
+
|
|
3
|
+
Allowed lifecycle states:
|
|
4
|
+
|
|
5
|
+
- `draft`: early design, not ready for model exposure.
|
|
6
|
+
- `review`: ready for structured review.
|
|
7
|
+
- `approved`: accepted for controlled rollout.
|
|
8
|
+
- `canary`: limited exposure with monitoring.
|
|
9
|
+
- `production`: generally available under configured policy.
|
|
10
|
+
- `deprecated`: still present but replacement is expected.
|
|
11
|
+
- `disabled`: blocked from routing/execution.
|
|
12
|
+
- `removed`: terminal state for retired contracts.
|
|
13
|
+
|
|
14
|
+
Promotion should require owner, validation, eval/security coverage, risk controls, and
|
|
15
|
+
no active kill switch.
|
|
@@ -0,0 +1,13 @@
|
|
|
1
|
+
# Risk Compiler Rules
|
|
2
|
+
|
|
3
|
+
Infer at least medium risk for tenant-scoped data, external communication, or important
|
|
4
|
+
workflow state.
|
|
5
|
+
|
|
6
|
+
Infer high risk for money movement, permission changes, broad data access, personal
|
|
7
|
+
data, destructive operations, or approval-sensitive work.
|
|
8
|
+
|
|
9
|
+
Infer critical risk for irreversible destructive action, privileged account control,
|
|
10
|
+
regulated data exposure, or large-scale cross-tenant impact.
|
|
11
|
+
|
|
12
|
+
Declared risk lower than inferred risk is a blocker. Missing quality metadata can be a
|
|
13
|
+
warning only when it does not affect safety or runtime correctness.
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: aicf-migration
|
|
3
|
+
description:
|
|
4
|
+
Migrate existing AI tools, prompts, agents, raw function calls, RAG flows, or app
|
|
5
|
+
integrations into AICF capabilities with policy, schemas, evals, runtime handlers, and
|
|
6
|
+
safe rollout.
|
|
7
|
+
license: MIT
|
|
8
|
+
compatibility:
|
|
9
|
+
Codex and Agent Skills-compatible coding agents. Works in repositories using AICF or
|
|
10
|
+
migrating to AICF.
|
|
11
|
+
metadata:
|
|
12
|
+
aicf.skill.version: "1.0.0-rc.4"
|
|
13
|
+
aicf.skill.package: "@aicf/agent-skills"
|
|
14
|
+
aicf.skill.category: "migration"
|
|
15
|
+
aicf.skill.scope: "builder"
|
|
16
|
+
aicf.skill.strictness: "fail-closed-runtime"
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
# AICF Migration
|
|
20
|
+
|
|
21
|
+
## Purpose
|
|
22
|
+
|
|
23
|
+
Guide migration from existing AI tools, agent integrations, RAG flows, or function-call
|
|
24
|
+
code into governed AICF capabilities with tests and safe rollout.
|
|
25
|
+
|
|
26
|
+
## Use this skill when
|
|
27
|
+
|
|
28
|
+
- Inventorying existing model calls, tool definitions, prompts, app integrations, evals,
|
|
29
|
+
traces, or failure handling.
|
|
30
|
+
- Converting broad tools into AICF capability manifests, runtime handlers, policy
|
|
31
|
+
checks, and evals.
|
|
32
|
+
- Planning shadow, read-only, canary, production, or rollback migration steps.
|
|
33
|
+
|
|
34
|
+
## Do not use this skill when
|
|
35
|
+
|
|
36
|
+
- Creating a brand-new AICF capability with no existing AI surface.
|
|
37
|
+
- Replacing app architecture without a migration plan.
|
|
38
|
+
- Running live provider calls or moving private transcripts into public examples.
|
|
39
|
+
|
|
40
|
+
## Inputs to inspect first
|
|
41
|
+
|
|
42
|
+
- Current AI integration code, tool definitions, app service APIs, data access paths,
|
|
43
|
+
existing evals, docs, and operational constraints.
|
|
44
|
+
- References: [migration inventory](references/migration-inventory.md),
|
|
45
|
+
[conversion playbook](references/conversion-playbook.md),
|
|
46
|
+
[rollout plan](references/rollout-plan.md), and
|
|
47
|
+
[legacy tool risk map](references/legacy-tool-riskmap.md).
|
|
48
|
+
|
|
49
|
+
## Workflow
|
|
50
|
+
|
|
51
|
+
1. Inventory current model calls, tools, prompts, data access, app integrations, evals,
|
|
52
|
+
traces, and failure handling.
|
|
53
|
+
2. Identify broad tools, unsafe data exposure, missing policy, and missing validation.
|
|
54
|
+
3. Group operations into capability domains and entity manifests.
|
|
55
|
+
4. Create or update capability manifests with schemas, policy, lifecycle, and evals.
|
|
56
|
+
5. Replace direct tool exposure with routed AICF capability slices.
|
|
57
|
+
6. Wire runtime handlers and policy broker adapters to existing app services.
|
|
58
|
+
7. Add deterministic evals and red-team tests before rollout.
|
|
59
|
+
8. Plan shadow, read-only, canary, production, and rollback steps.
|
|
60
|
+
|
|
61
|
+
Use [migration plan](assets/migration-plan.md) and
|
|
62
|
+
[legacy tool inventory](assets/legacy-tool-inventory.csv) templates.
|
|
63
|
+
|
|
64
|
+
## Required outputs
|
|
65
|
+
|
|
66
|
+
- Migration inventory and target AICF capability plan.
|
|
67
|
+
- Manifests, runtime tasks, eval/security coverage, and rollout notes.
|
|
68
|
+
- Explicit unsupported behavior or breaking-change list.
|
|
69
|
+
|
|
70
|
+
## Validation
|
|
71
|
+
|
|
72
|
+
- Run manifest validation, evals, runtime tests, migration-specific tests, and
|
|
73
|
+
public-safety checks.
|
|
74
|
+
- Confirm unsafe broad access is removed or explicitly scoped.
|
|
75
|
+
- Confirm rollback and fallback behavior is documented.
|
|
76
|
+
|
|
77
|
+
## Hard rules
|
|
78
|
+
|
|
79
|
+
- Do not preserve unsafe broad tool access for convenience.
|
|
80
|
+
- Do not publish private prompts, transcripts, traces, or provider transport details.
|
|
81
|
+
- Do not weaken policy or validation to match legacy behavior.
|
|
82
|
+
- Do not consider migration complete before tests/evals cover core behavior.
|
|
83
|
+
|
|
84
|
+
## Handoff format
|
|
85
|
+
|
|
86
|
+
Report current architecture, target capabilities, files changed, tests/evals added,
|
|
87
|
+
rollout stage, blockers, and rollback path.
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
# Migration Plan
|
|
2
|
+
|
|
3
|
+
## Current AI surface
|
|
4
|
+
|
|
5
|
+
## Unsafe or broad access
|
|
6
|
+
|
|
7
|
+
## Target capability domains
|
|
8
|
+
|
|
9
|
+
## Capability and entity manifests
|
|
10
|
+
|
|
11
|
+
## Runtime handlers
|
|
12
|
+
|
|
13
|
+
## Policy and lifecycle
|
|
14
|
+
|
|
15
|
+
## Evals and red-team coverage
|
|
16
|
+
|
|
17
|
+
## Rollout stages
|
|
18
|
+
|
|
19
|
+
## Rollback path
|