@vegastack/skills 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +46 -0
- package/dist/index.js +512 -0
- package/package.json +35 -0
- package/skill/vegastack-arch-guardian/SKILL.md +96 -0
- package/skill/vegastack-arch-guardian/agents/openai.yaml +4 -0
- package/skill/vegastack-arch-guardian/assets/adr-template.md +40 -0
- package/skill/vegastack-arch-guardian/assets/answers-example.json +20 -0
- package/skill/vegastack-arch-guardian/assets/architecture-profile.json +25 -0
- package/skill/vegastack-arch-guardian/assets/architecture-profile.schema.json +213 -0
- package/skill/vegastack-arch-guardian/assets/deployment-review-template.md +24 -0
- package/skill/vegastack-arch-guardian/assets/service-design-template.md +33 -0
- package/skill/vegastack-arch-guardian/assets/threat-model-template.md +34 -0
- package/skill/vegastack-arch-guardian/references/architecture/agent-product.md +22 -0
- package/skill/vegastack-arch-guardian/references/architecture/ai-cost.md +22 -0
- package/skill/vegastack-arch-guardian/references/architecture/ai-data-boundaries.md +19 -0
- package/skill/vegastack-arch-guardian/references/architecture/ai-evals.md +26 -0
- package/skill/vegastack-arch-guardian/references/architecture/connectors-sandbox.md +39 -0
- package/skill/vegastack-arch-guardian/references/architecture/data-memory.md +25 -0
- package/skill/vegastack-arch-guardian/references/architecture/delivery-operations.md +34 -0
- package/skill/vegastack-arch-guardian/references/architecture/durable-execution.md +43 -0
- package/skill/vegastack-arch-guardian/references/architecture/flutter.md +26 -0
- package/skill/vegastack-arch-guardian/references/architecture/foundation.md +33 -0
- package/skill/vegastack-arch-guardian/references/architecture/hosting-reliability.md +37 -0
- package/skill/vegastack-arch-guardian/references/architecture/identity-tenancy.md +37 -0
- package/skill/vegastack-arch-guardian/references/architecture/model-lifecycle.md +18 -0
- package/skill/vegastack-arch-guardian/references/architecture/models-observability.md +23 -0
- package/skill/vegastack-arch-guardian/references/architecture/realtime-channels.md +16 -0
- package/skill/vegastack-arch-guardian/references/architecture/security-privacy.md +23 -0
- package/skill/vegastack-arch-guardian/references/architecture/topology-monorepo.md +47 -0
- package/skill/vegastack-arch-guardian/references/architecture/web.md +29 -0
- package/skill/vegastack-arch-guardian/references/control-catalog.json +55 -0
- package/skill/vegastack-arch-guardian/references/foundation-compatibility.json +44 -0
- package/skill/vegastack-arch-guardian/references/golden-patterns.md +43 -0
- package/skill/vegastack-arch-guardian/references/profile-governance.md +54 -0
- package/skill/vegastack-arch-guardian/references/rule-model.json +36 -0
- package/skill/vegastack-arch-guardian/references/workflows.md +45 -0
- package/skill/vegastack-arch-guardian/refresh/REFRESH.md +40 -0
- package/skill/vegastack-arch-guardian/refresh/sources.json +1159 -0
- package/skill/vegastack-arch-guardian/scripts/architecture-check.mjs +323 -0
- package/skill/vegastack-arch-guardian/scripts/lib.mjs +57 -0
- package/skill/vegastack-arch-guardian/scripts/profile-tool.mjs +223 -0
- package/skill/vegastack-arch-guardian/scripts/refresh-evidence.mjs +325 -0
- package/skill/vegastack-arch-guardian/scripts/schema-validate.mjs +63 -0
- package/skill/vegastack-arch-guardian/scripts/validate-profile.mjs +241 -0
- package/skill/vegastack-arch-guardian/scripts/verify-corpus.mjs +148 -0
- package/skill-integrity.json +48 -0
|
@@ -0,0 +1,96 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: vegastack-arch-guardian
|
|
3
|
+
description: Architecture advisor for VegaStack projects. Use when designing a new service or feature, reviewing architecture or a risky change (auth, tenancy, agents, jobs, connectors, data lifecycle, hosting), deciding between architectural options, writing or validating an ADR or .vegastack/architecture.json profile, threat modeling, planning a migration, or checking dependency and source drift. Gives one scoped recommendation with an explicit verdict; activates only the capabilities a project has actually enabled. Covers web-only, Flutter, agentic and non-agentic, single- and multi-tenant, internal, public, platform-service, and shared-package projects.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# VegaStack Architecture Guardian
|
|
7
|
+
|
|
8
|
+
Act as the project's decisive senior architect. Apply only activated capabilities. Treat the foundation as an advisory recommendation: state `GUARDIAN VERDICT: REJECT` when warranted, while allowing a project owner to accept any scoped risk through a valid ADR. Never invent compliance or project facts.
|
|
9
|
+
|
|
10
|
+
## Start every task
|
|
11
|
+
|
|
12
|
+
1. Identify the actual decision, its scope, and how expensive it is to reverse.
|
|
13
|
+
2. Inspect repository instructions and existing facts before asking questions. Keep read-only work read-only; never create a profile or artifact merely because it is absent.
|
|
14
|
+
3. Separate **facts**, **constraints**, **assumptions**, **preferences**, and **accepted project decisions**. Distinguish current, target, and migration state.
|
|
15
|
+
4. Ask at most three material questions at a time. If unanswered, proceed with bounded assumptions and the smallest architecture that meets stated requirements.
|
|
16
|
+
5. Read only the directly relevant references below. Use committed profiles, designs, and ADRs as architecture memory; never rely on hidden chat state.
|
|
17
|
+
6. Prioritize security/correctness, recovery, ownership, contracts, operability, delivery, then optional optimization. Make one primary recommendation, identify the rejected alternative, and surface irreversible or expensive choices first.
|
|
18
|
+
7. Evidence labels (**OBSERVED**, **DOCUMENTED**, **REPRODUCED**, **INFERRED**, **RECOMMENDED**, **NOT VERIFIED**) are optional; reserve them for review and drift reports where provenance matters. A static sentinel or declared control is never a reproduced runtime test.
|
|
19
|
+
|
|
20
|
+
Script invocations below write `<skill-dir>` as a placeholder: replace it with the absolute path of the directory containing this SKILL.md before running (no environment variable is set for you).
|
|
21
|
+
|
|
22
|
+
## Answer at the right size
|
|
23
|
+
|
|
24
|
+
- **Questions and explanations:** answer directly — the verdict (if one applies), the recommendation, and at most one material risk. No section headers, no full contract.
|
|
25
|
+
- **Design reviews, ADRs, migration plans:** use the full response contract: (1) Verdict, (2) Recommended architecture/decision, (3) Why it fits, (4) Material assumptions, (5) Rejected alternative and reason, (6) Implementation/migration sequence, (7) Risks and required evidence, (8) Artifacts/ADRs to update.
|
|
26
|
+
|
|
27
|
+
## Route progressively
|
|
28
|
+
|
|
29
|
+
| Need | Read |
|
|
30
|
+
|---|---|
|
|
31
|
+
| operating model, project types, profiles, exceptions | [foundation](references/architecture/foundation.md), [profile and governance](references/profile-governance.md) |
|
|
32
|
+
| greenfield, brownfield, migration, continuous work | [adaptive workflows](references/workflows.md) |
|
|
33
|
+
| topology, deployables, packages, contracts | [topology and monorepo](references/architecture/topology-monorepo.md) |
|
|
34
|
+
| web/Next/OpenAPI/cache | [web](references/architecture/web.md) |
|
|
35
|
+
| Flutter/mobile | [Flutter](references/architecture/flutter.md) |
|
|
36
|
+
| identity, organizations, tenancy, RLS | [identity and tenancy](references/architecture/identity-tenancy.md) |
|
|
37
|
+
| agent product and durable execution/jobs | [agent product](references/architecture/agent-product.md), [durable execution](references/architecture/durable-execution.md) |
|
|
38
|
+
| connectors, MCP, webhooks, sandbox, egress | [connectors and sandbox](references/architecture/connectors-sandbox.md) |
|
|
39
|
+
| data, knowledge, memory, objects | [data and memory](references/architecture/data-memory.md) |
|
|
40
|
+
| PII, prompt injection, output moderation | [AI data boundaries](references/architecture/ai-data-boundaries.md) |
|
|
41
|
+
| evals, prompt/model regression gates | [AI evals](references/architecture/ai-evals.md) |
|
|
42
|
+
| model pinning, deprecations, canary, backpressure | [model lifecycle](references/architecture/model-lifecycle.md) |
|
|
43
|
+
| model spend, budgets, cost attribution | [AI cost](references/architecture/ai-cost.md) |
|
|
44
|
+
| realtime, notifications, channels | [realtime and channels](references/architecture/realtime-channels.md) |
|
|
45
|
+
| models, BYOK, telemetry, audit | [models and observability](references/architecture/models-observability.md) |
|
|
46
|
+
| security, privacy, threat model | [security and privacy](references/architecture/security-privacy.md) |
|
|
47
|
+
| hosting, Cloudflare/OpenNext, SLO/recovery | [hosting and reliability](references/architecture/hosting-reliability.md) |
|
|
48
|
+
| delivery, migration, verification | [delivery and operations](references/architecture/delivery-operations.md) |
|
|
49
|
+
| fragile boundary implementation | [golden patterns](references/golden-patterns.md) |
|
|
50
|
+
|
|
51
|
+
Consult [refresh/sources.json](refresh/sources.json) only for affected unstable claims and [foundation compatibility](references/foundation-compatibility.json) for version adoption. Before advice that leans on a **critical** source (`critical: true` in the registry), check freshness: run `node <skill-dir>/scripts/refresh-evidence.mjs --topics <affected-topics>` when online, or treat the claim as `NOT VERIFIED` when offline and the registry snapshot is older than its `thresholdDays`. Source drift requests review; it never silently expires an ADR.
|
|
52
|
+
|
|
53
|
+
## Execute the task-specific workflow
|
|
54
|
+
|
|
55
|
+
- **Greenfield:** Follow the adaptive intake in [workflows](references/workflows.md). Recommend capability set, topology, ownership, boundaries, immediate/deferred decisions, risks, implementation order, and qualification. Offer scaffolding only after confirmation.
|
|
56
|
+
- **Brownfield/review:** Inspect instructions, manifests/locks, deployables, packages, schemas/migrations, auth, APIs/clients, jobs/workflows, infra, ADRs, observability, and runbooks before interviewing. Derive an observed draft without mutation; compare current, intended, and target states; prefer controlled migration over rewrites.
|
|
57
|
+
- **ADR/design/threat/deploy:** Use the relevant asset only after write authorization. Every action includes owner, risk, evidence, rollback/migration, and review trigger.
|
|
58
|
+
- **Continuous work:** Load only affected references and checks. Do not force absent capabilities into scope.
|
|
59
|
+
- **Source drift:** Follow [refresh/REFRESH.md](refresh/REFRESH.md). Refresh only affected topics; map changes to rules/capabilities and preserve unrelated fast paths.
|
|
60
|
+
|
|
61
|
+
## Profiles, checks, and mutation safety
|
|
62
|
+
|
|
63
|
+
The committed profile lives at `.vegastack/architecture.json` (legacy `.yaml`-named JSON is still discovered, with a deprecation notice). Inspect or draft without mutation:
|
|
64
|
+
|
|
65
|
+
```sh
|
|
66
|
+
node <skill-dir>/scripts/profile-tool.mjs inspect .
|
|
67
|
+
node <skill-dir>/scripts/profile-tool.mjs scaffold answers.json --dir .
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
`inspect` prints a compact summary; add `--json` for the full draft. The answers format is documented by example in `<skill-dir>/assets/answers-example.json`.
|
|
71
|
+
|
|
72
|
+
Write only after explicit authorization; writes are atomic, refuse symlinks, stay inside `--dir`, and require `--force` to replace differing content:
|
|
73
|
+
|
|
74
|
+
```sh
|
|
75
|
+
node <skill-dir>/scripts/profile-tool.mjs scaffold answers.json --dir . --write
|
|
76
|
+
node <skill-dir>/scripts/profile-tool.mjs migrate-v2 .vegastack/architecture.yaml --dir .
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
Validate and review (prefer `--summary` in conversation; `--json` is for CI):
|
|
80
|
+
|
|
81
|
+
```sh
|
|
82
|
+
node <skill-dir>/scripts/validate-profile.mjs .vegastack/architecture.json
|
|
83
|
+
node <skill-dir>/scripts/architecture-check.mjs . --summary
|
|
84
|
+
```
|
|
85
|
+
|
|
86
|
+
`architecture-check` exit codes: 0 = no FAIL findings, 1 = FAIL findings present, 2 = tool/usage error. A non-zero exit is a review result, not a crash. Paths can be excluded deliberately via a `.guardianignore` file (path prefixes, one per line). Symlinks, agent-skill trees, and generated/bundled files (very large or very long-line files) are skipped automatically and reported in one `NOT VERIFIED` finding.
|
|
87
|
+
|
|
88
|
+
Interpret outcomes exactly: `PASS` satisfies the recommendation; `FAIL` violates it without a valid exception; `EXCEPTED` is active project-owner accepted risk and remains visibly noncompliant although CI may pass; `NOT VERIFIED` records reason, risk, owner, and next action. Invalid, expired, or mismatched exceptions fail. Continue to recommend rejection when accepted risk remains unsafe.
|
|
89
|
+
|
|
90
|
+
## Guardrails
|
|
91
|
+
|
|
92
|
+
- Product-owned enabled capabilities are the default. Shared/external management is rare and requires the complete declared contract in [profile and governance](references/profile-governance.md).
|
|
93
|
+
- Challenge project proposals and foundation defaults when evidence warrants. Choose the smallest architecture meeting current requirements and measured objectives.
|
|
94
|
+
- Never require SQL/RLS, Better Auth, Flutter, EVE, pg-boss, sandbox, connectors, enterprise identity, realtime, notifications, knowledge, or OpenBao when their activation condition is absent.
|
|
95
|
+
- Never mutate a repository for an explanation or read-only review. Draft first and ask before writing.
|
|
96
|
+
- Never create paid/cloud resources or claim live recovery, isolation, failover, credential-backed, or provider tests ran unless they actually ran.
|
|
@@ -0,0 +1,4 @@
|
|
|
1
|
+
interface:
|
|
2
|
+
display_name: "VegaStack Architecture Guardian"
|
|
3
|
+
short_description: "Design and guard adaptive VegaStack architecture"
|
|
4
|
+
default_prompt: "Use $vegastack-arch-guardian to make one scoped architecture recommendation for this project without assuming absent capabilities."
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
# ADR-NNN: Decision title
|
|
2
|
+
|
|
3
|
+
- Status: proposed | accepted | superseded | rejected
|
|
4
|
+
- Date: YYYY-MM-DD
|
|
5
|
+
- Project-Owner: accountable person
|
|
6
|
+
- Scope-Paths: comma-separated exact repository-relative paths
|
|
7
|
+
- Rule-ID: canonical rule identifier or `none`
|
|
8
|
+
- Control-IDs: comma-separated exact checker/manual control IDs, or `all` for a rule-level exception that omits the profile controls list
|
|
9
|
+
- Exception-ID: profile exception ID or `none`
|
|
10
|
+
- Foundation-Deviation-Acknowledged: true | false
|
|
11
|
+
- Review-Date: YYYY-MM-DD | none
|
|
12
|
+
- Review-Event: concrete event | none
|
|
13
|
+
|
|
14
|
+
## Context and facts
|
|
15
|
+
|
|
16
|
+
Separate observed facts, constraints, assumptions, preferences and prior accepted decisions. Identify current, target and migration states.
|
|
17
|
+
|
|
18
|
+
## Decision and rationale
|
|
19
|
+
|
|
20
|
+
State the primary decision, ownership, boundaries and why it fits. State the rejected alternative and reason.
|
|
21
|
+
|
|
22
|
+
## Risks and accepted deviation
|
|
23
|
+
|
|
24
|
+
Describe each risk, who accepts it, affected paths/tenants/data, and why the foundation recommendation remains unmet.
|
|
25
|
+
|
|
26
|
+
## Compensating controls
|
|
27
|
+
|
|
28
|
+
Map preventive, detective and recovery controls to each risk. Do not claim an untested control works.
|
|
29
|
+
|
|
30
|
+
## Verification
|
|
31
|
+
|
|
32
|
+
List static sentinels, declared controls, reproduced tests, and `NOT VERIFIED` environment behavior with reason, risk, owner and next action.
|
|
33
|
+
|
|
34
|
+
## Rollback or migration
|
|
35
|
+
|
|
36
|
+
Describe rollback, migration/exit behavior, compatibility window, irreversible steps and cleanup.
|
|
37
|
+
|
|
38
|
+
## Review trigger
|
|
39
|
+
|
|
40
|
+
Give an expiry date or concrete event. Missing, expired or malformed exception ADRs fail validation.
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
{
|
|
2
|
+
"profileStatus": "draft",
|
|
3
|
+
"project": { "name": "example-product", "kind": "saas-product", "lifecycle": "greenfield", "access": "authenticated", "tenancy": "multi-tenant-shared-schema" },
|
|
4
|
+
"environments": { "production": { "hosting": "cloudflare-opennext" }, "localDevelopment": { "trusted": true, "allowances": ["local-secrets"] } },
|
|
5
|
+
"capabilities": {
|
|
6
|
+
"webControlPlane": { "status": "enabled", "ownership": "owned", "versions": { "next": "16.3.0", "node": "24.18.0", "bun": "1.3.14", "openNext": "1.20.2", "betterAuth": "1.6.26" }, "placement": "open-next-worker", "sourceRoots": ["apps/web"], "controls": { "canonicalApi": "rest-openapi", "openapiGenerated": true, "secureCookies": true } },
|
|
7
|
+
"flutter": { "status": "disabled", "ownership": "not-applicable" },
|
|
8
|
+
"agents": { "status": "disabled", "ownership": "not-applicable" },
|
|
9
|
+
"jobs": { "status": "disabled", "ownership": "not-applicable" },
|
|
10
|
+
"sandbox": { "status": "disabled", "ownership": "not-applicable" },
|
|
11
|
+
"connectors": { "status": "disabled", "ownership": "not-applicable" },
|
|
12
|
+
"knowledge": { "status": "disabled", "ownership": "not-applicable" },
|
|
13
|
+
"modelRouting": { "status": "disabled", "ownership": "not-applicable" },
|
|
14
|
+
"enterpriseIdentity": { "status": "disabled", "ownership": "not-applicable" },
|
|
15
|
+
"realtime": { "status": "disabled", "ownership": "not-applicable" },
|
|
16
|
+
"notifications": { "status": "disabled", "ownership": "not-applicable" },
|
|
17
|
+
"secrets": { "status": "enabled", "ownership": "owned", "versions": { "openbao": "2.3.2" }, "placement": "external-service", "sourceRoots": ["infra/secrets"], "controls": { "provider": "openbao" } }
|
|
18
|
+
},
|
|
19
|
+
"exceptions": []
|
|
20
|
+
}
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
{
|
|
2
|
+
"schemaVersion": 3,
|
|
3
|
+
"profileStatus": "draft",
|
|
4
|
+
"foundation": { "version": "0.3.0", "baseline": "vs-2026-08-07", "adoption": "supported" },
|
|
5
|
+
"project": { "name": "REQUIRED-CONFIRMED-PROJECT-NAME", "kind": "REQUIRED-CONFIRMED-PROJECT-KIND", "lifecycle": "REQUIRED-CONFIRMED-LIFECYCLE", "access": "REQUIRED-CONFIRMED-ACCESS", "tenancy": "REQUIRED-CONFIRMED-TENANCY" },
|
|
6
|
+
"environments": {
|
|
7
|
+
"production": { "hosting": "REQUIRED-CONFIRMED-PRODUCTION-TARGET" },
|
|
8
|
+
"localDevelopment": { "trusted": true, "allowances": [] }
|
|
9
|
+
},
|
|
10
|
+
"capabilities": {
|
|
11
|
+
"webControlPlane": { "status": "disabled", "ownership": "not-applicable" },
|
|
12
|
+
"flutter": { "status": "disabled", "ownership": "not-applicable" },
|
|
13
|
+
"agents": { "status": "disabled", "ownership": "not-applicable" },
|
|
14
|
+
"jobs": { "status": "disabled", "ownership": "not-applicable" },
|
|
15
|
+
"sandbox": { "status": "disabled", "ownership": "not-applicable" },
|
|
16
|
+
"connectors": { "status": "disabled", "ownership": "not-applicable" },
|
|
17
|
+
"knowledge": { "status": "disabled", "ownership": "not-applicable" },
|
|
18
|
+
"modelRouting": { "status": "disabled", "ownership": "not-applicable" },
|
|
19
|
+
"enterpriseIdentity": { "status": "disabled", "ownership": "not-applicable" },
|
|
20
|
+
"realtime": { "status": "disabled", "ownership": "not-applicable" },
|
|
21
|
+
"notifications": { "status": "disabled", "ownership": "not-applicable" },
|
|
22
|
+
"secrets": { "status": "disabled", "ownership": "not-applicable" }
|
|
23
|
+
},
|
|
24
|
+
"exceptions": []
|
|
25
|
+
}
|
|
@@ -0,0 +1,213 @@
|
|
|
1
|
+
{
|
|
2
|
+
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
3
|
+
"$id": "https://raw.githubusercontent.com/vegastack/vegastack-skills/main/skills/vegastack-arch-guardian/assets/architecture-profile.schema.json",
|
|
4
|
+
"title": "VegaStack Architecture Profile v3",
|
|
5
|
+
"type": "object",
|
|
6
|
+
"additionalProperties": false,
|
|
7
|
+
"required": ["schemaVersion", "profileStatus", "foundation", "project", "environments", "capabilities", "exceptions"],
|
|
8
|
+
"$defs": {
|
|
9
|
+
"exactVersion": { "type": "string", "pattern": "^[0-9]+\\.[0-9]+(?:\\.[0-9]+)?(?:-[0-9A-Za-z.-]+)?(?:\\+[0-9A-Za-z.-]+)?$" },
|
|
10
|
+
"relativePath": { "type": "string", "minLength": 1, "pattern": "^(?!/)(?!.*(?:^|/)\\.\\.(?:/|$))(?!.*\\\\).+$" },
|
|
11
|
+
"nonEmptyStrings": { "type": "array", "minItems": 1, "uniqueItems": true, "items": { "type": "string", "minLength": 1 } },
|
|
12
|
+
"serviceContract": {
|
|
13
|
+
"type": "object",
|
|
14
|
+
"additionalProperties": false,
|
|
15
|
+
"required": ["ownerService", "contract", "version", "tenantSecurityBoundary", "identityAudience", "dataResidency", "sloRecovery", "incidentOwner", "compatibility", "migrationExit"],
|
|
16
|
+
"properties": {
|
|
17
|
+
"ownerService": { "type": "string", "minLength": 1 },
|
|
18
|
+
"contract": { "type": "string", "minLength": 1 },
|
|
19
|
+
"version": { "$ref": "#/$defs/exactVersion" },
|
|
20
|
+
"tenantSecurityBoundary": { "type": "string", "minLength": 1 },
|
|
21
|
+
"identityAudience": { "type": "string", "minLength": 1 },
|
|
22
|
+
"dataResidency": { "type": "string", "minLength": 1 },
|
|
23
|
+
"sloRecovery": { "type": "string", "minLength": 1 },
|
|
24
|
+
"incidentOwner": { "type": "string", "minLength": 1 },
|
|
25
|
+
"compatibility": { "type": "string", "minLength": 1 },
|
|
26
|
+
"migrationExit": { "type": "string", "minLength": 1 }
|
|
27
|
+
}
|
|
28
|
+
},
|
|
29
|
+
"removal": {
|
|
30
|
+
"type": "object",
|
|
31
|
+
"additionalProperties": false,
|
|
32
|
+
"required": ["durableData", "credentials", "queues", "contracts", "owner", "verification", "rollback"],
|
|
33
|
+
"properties": {
|
|
34
|
+
"durableData": { "type": "string", "minLength": 1 },
|
|
35
|
+
"credentials": { "type": "string", "minLength": 1 },
|
|
36
|
+
"queues": { "type": "string", "minLength": 1 },
|
|
37
|
+
"contracts": { "type": "string", "minLength": 1 },
|
|
38
|
+
"owner": { "type": "string", "minLength": 1 },
|
|
39
|
+
"verification": { "type": "string", "minLength": 1 },
|
|
40
|
+
"rollback": { "type": "string", "minLength": 1 }
|
|
41
|
+
}
|
|
42
|
+
},
|
|
43
|
+
"capability": {
|
|
44
|
+
"type": "object",
|
|
45
|
+
"additionalProperties": false,
|
|
46
|
+
"required": ["status", "ownership"],
|
|
47
|
+
"properties": {
|
|
48
|
+
"status": { "enum": ["enabled", "disabled"] },
|
|
49
|
+
"ownership": { "enum": ["owned", "shared-managed", "external-managed", "not-applicable"] },
|
|
50
|
+
"versions": {
|
|
51
|
+
"type": "object",
|
|
52
|
+
"minProperties": 1,
|
|
53
|
+
"propertyNames": { "pattern": "^[A-Za-z][A-Za-z0-9]*$" },
|
|
54
|
+
"additionalProperties": { "$ref": "#/$defs/exactVersion" }
|
|
55
|
+
},
|
|
56
|
+
"placement": { "type": "string", "minLength": 1 },
|
|
57
|
+
"sourceRoots": { "type": "array", "minItems": 1, "uniqueItems": true, "items": { "$ref": "#/$defs/relativePath" } },
|
|
58
|
+
"contract": { "$ref": "#/$defs/serviceContract" },
|
|
59
|
+
"controls": { "type": "object", "additionalProperties": { "type": ["string", "boolean", "number", "array"], "items": { "type": "string" } } },
|
|
60
|
+
"transitionFrom": { "enum": ["enabled", "disabled"] },
|
|
61
|
+
"removal": { "$ref": "#/$defs/removal" }
|
|
62
|
+
},
|
|
63
|
+
"allOf": [
|
|
64
|
+
{
|
|
65
|
+
"if": { "properties": { "status": { "const": "enabled" } }, "required": ["status"] },
|
|
66
|
+
"then": { "required": ["versions", "placement"], "properties": { "ownership": { "enum": ["owned", "shared-managed", "external-managed"] } } }
|
|
67
|
+
},
|
|
68
|
+
{
|
|
69
|
+
"if": { "properties": { "status": { "const": "disabled" } }, "required": ["status"] },
|
|
70
|
+
"then": { "properties": { "ownership": { "const": "not-applicable" } } }
|
|
71
|
+
},
|
|
72
|
+
{
|
|
73
|
+
"if": { "properties": { "status": { "const": "enabled" }, "ownership": { "const": "owned" } }, "required": ["status", "ownership"] },
|
|
74
|
+
"then": { "required": ["sourceRoots"] }
|
|
75
|
+
},
|
|
76
|
+
{
|
|
77
|
+
"if": { "properties": { "ownership": { "enum": ["shared-managed", "external-managed"] } }, "required": ["ownership"] },
|
|
78
|
+
"then": { "required": ["contract"], "not": { "required": ["sourceRoots"] } }
|
|
79
|
+
},
|
|
80
|
+
{
|
|
81
|
+
"if": { "properties": { "status": { "const": "disabled" }, "transitionFrom": { "const": "enabled" } }, "required": ["status", "transitionFrom"] },
|
|
82
|
+
"then": { "required": ["removal"] }
|
|
83
|
+
}
|
|
84
|
+
]
|
|
85
|
+
},
|
|
86
|
+
"exception": {
|
|
87
|
+
"type": "object",
|
|
88
|
+
"additionalProperties": false,
|
|
89
|
+
"required": ["id", "ruleId", "paths", "adr", "projectOwner", "status", "verificationType", "rationale", "decision", "risks", "compensatingControls", "verification", "rollbackOrMigration", "review", "foundationDeviationAcknowledged"],
|
|
90
|
+
"properties": {
|
|
91
|
+
"id": { "type": "string", "pattern": "^EXC-[0-9]{3,}$" },
|
|
92
|
+
"ruleId": { "type": "string", "pattern": "^[A-Z]+-[0-9]{3}$" },
|
|
93
|
+
"controls": { "$ref": "#/$defs/nonEmptyStrings" },
|
|
94
|
+
"paths": { "type": "array", "minItems": 1, "uniqueItems": true, "items": { "$ref": "#/$defs/relativePath" } },
|
|
95
|
+
"adr": { "$ref": "#/$defs/relativePath" },
|
|
96
|
+
"projectOwner": { "type": "string", "minLength": 1 },
|
|
97
|
+
"status": { "enum": ["active", "withdrawn"] },
|
|
98
|
+
"verificationType": { "enum": ["static-sentinel", "manual-qualification"] },
|
|
99
|
+
"rationale": { "type": "string", "minLength": 1 },
|
|
100
|
+
"decision": { "type": "string", "minLength": 1 },
|
|
101
|
+
"risks": { "$ref": "#/$defs/nonEmptyStrings" },
|
|
102
|
+
"compensatingControls": { "$ref": "#/$defs/nonEmptyStrings" },
|
|
103
|
+
"verification": { "$ref": "#/$defs/nonEmptyStrings" },
|
|
104
|
+
"rollbackOrMigration": { "type": "string", "minLength": 1 },
|
|
105
|
+
"review": {
|
|
106
|
+
"oneOf": [
|
|
107
|
+
{ "type": "object", "additionalProperties": false, "required": ["date"], "properties": { "date": { "type": "string", "format": "date" } } },
|
|
108
|
+
{ "type": "object", "additionalProperties": false, "required": ["event"], "properties": { "event": { "type": "string", "minLength": 1 } } }
|
|
109
|
+
]
|
|
110
|
+
},
|
|
111
|
+
"foundationDeviationAcknowledged": { "const": true }
|
|
112
|
+
}
|
|
113
|
+
}
|
|
114
|
+
},
|
|
115
|
+
"properties": {
|
|
116
|
+
"schemaVersion": { "const": 3 },
|
|
117
|
+
"profileStatus": { "enum": ["draft", "confirmed"] },
|
|
118
|
+
"foundation": {
|
|
119
|
+
"type": "object",
|
|
120
|
+
"additionalProperties": false,
|
|
121
|
+
"required": ["version", "baseline", "adoption"],
|
|
122
|
+
"properties": {
|
|
123
|
+
"version": { "const": "0.3.0" },
|
|
124
|
+
"baseline": { "type": "string", "minLength": 1 },
|
|
125
|
+
"adoption": { "enum": ["supported", "candidate", "deprecated"] }
|
|
126
|
+
}
|
|
127
|
+
},
|
|
128
|
+
"project": {
|
|
129
|
+
"type": "object",
|
|
130
|
+
"additionalProperties": false,
|
|
131
|
+
"required": ["name", "kind", "lifecycle", "access", "tenancy"],
|
|
132
|
+
"properties": {
|
|
133
|
+
"name": { "type": "string", "minLength": 1 },
|
|
134
|
+
"kind": { "enum": ["saas-product", "internal-product", "public-product", "platform-service", "shared-package"] },
|
|
135
|
+
"lifecycle": { "enum": ["greenfield", "brownfield", "migration", "maintenance"] },
|
|
136
|
+
"access": { "enum": ["public", "authenticated", "internal", "mixed"] },
|
|
137
|
+
"tenancy": { "enum": ["none", "single-tenant", "multi-tenant-shared-schema", "multi-tenant-isolated"] }
|
|
138
|
+
}
|
|
139
|
+
},
|
|
140
|
+
"environments": {
|
|
141
|
+
"type": "object",
|
|
142
|
+
"additionalProperties": false,
|
|
143
|
+
"required": ["production", "localDevelopment"],
|
|
144
|
+
"properties": {
|
|
145
|
+
"production": {
|
|
146
|
+
"type": "object",
|
|
147
|
+
"additionalProperties": false,
|
|
148
|
+
"required": ["hosting"],
|
|
149
|
+
"properties": { "hosting": { "enum": ["self-hosted", "vercel", "cloudflare-opennext", "external", "none"] }, "region": { "type": "string", "minLength": 1 } }
|
|
150
|
+
},
|
|
151
|
+
"localDevelopment": {
|
|
152
|
+
"type": "object",
|
|
153
|
+
"additionalProperties": false,
|
|
154
|
+
"required": ["trusted", "allowances"],
|
|
155
|
+
"properties": { "trusted": { "const": true }, "allowances": { "type": "array", "uniqueItems": true, "items": { "enum": ["local-workflow-world", "local-secrets", "local-unsandboxed-code", "single-process"] } } }
|
|
156
|
+
}
|
|
157
|
+
}
|
|
158
|
+
},
|
|
159
|
+
"capabilities": {
|
|
160
|
+
"type": "object",
|
|
161
|
+
"additionalProperties": false,
|
|
162
|
+
"required": ["webControlPlane", "flutter", "agents", "jobs", "sandbox", "connectors", "knowledge", "modelRouting", "enterpriseIdentity", "realtime", "notifications", "secrets"],
|
|
163
|
+
"properties": {
|
|
164
|
+
"webControlPlane": { "$ref": "#/$defs/capability" },
|
|
165
|
+
"flutter": { "$ref": "#/$defs/capability" },
|
|
166
|
+
"agents": { "$ref": "#/$defs/capability" },
|
|
167
|
+
"jobs": { "$ref": "#/$defs/capability" },
|
|
168
|
+
"sandbox": { "$ref": "#/$defs/capability" },
|
|
169
|
+
"connectors": { "$ref": "#/$defs/capability" },
|
|
170
|
+
"knowledge": { "$ref": "#/$defs/capability" },
|
|
171
|
+
"modelRouting": { "$ref": "#/$defs/capability" },
|
|
172
|
+
"enterpriseIdentity": { "$ref": "#/$defs/capability" },
|
|
173
|
+
"realtime": { "$ref": "#/$defs/capability" },
|
|
174
|
+
"notifications": { "$ref": "#/$defs/capability" },
|
|
175
|
+
"secrets": { "$ref": "#/$defs/capability" }
|
|
176
|
+
}
|
|
177
|
+
},
|
|
178
|
+
"data": {
|
|
179
|
+
"type": "object",
|
|
180
|
+
"additionalProperties": false,
|
|
181
|
+
"required": ["classes"],
|
|
182
|
+
"properties": {
|
|
183
|
+
"classes": {
|
|
184
|
+
"type": "array",
|
|
185
|
+
"minItems": 1,
|
|
186
|
+
"items": {
|
|
187
|
+
"type": "object",
|
|
188
|
+
"additionalProperties": false,
|
|
189
|
+
"required": ["name"],
|
|
190
|
+
"properties": {
|
|
191
|
+
"name": { "enum": ["public", "internal", "confidential", "restricted"] },
|
|
192
|
+
"residency": { "type": "string", "minLength": 1 },
|
|
193
|
+
"retention": { "type": "string", "minLength": 1 },
|
|
194
|
+
"owner": { "type": "string", "minLength": 1 }
|
|
195
|
+
}
|
|
196
|
+
}
|
|
197
|
+
}
|
|
198
|
+
}
|
|
199
|
+
},
|
|
200
|
+
"objectives": {
|
|
201
|
+
"type": "object",
|
|
202
|
+
"additionalProperties": false,
|
|
203
|
+
"properties": {
|
|
204
|
+
"availability": { "type": "string", "minLength": 1 },
|
|
205
|
+
"latency": { "type": "string", "minLength": 1 },
|
|
206
|
+
"rpoMinutes": { "type": "number", "minimum": 0 },
|
|
207
|
+
"rtoMinutes": { "type": "number", "minimum": 0 }
|
|
208
|
+
},
|
|
209
|
+
"minProperties": 1
|
|
210
|
+
},
|
|
211
|
+
"exceptions": { "type": "array", "uniqueItems": true, "items": { "$ref": "#/$defs/exception" } }
|
|
212
|
+
}
|
|
213
|
+
}
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# Deployment review: release/profile
|
|
2
|
+
|
|
3
|
+
Delete checklist items for absent capabilities; absence is not a failed control.
|
|
4
|
+
|
|
5
|
+
## Artifact identity
|
|
6
|
+
|
|
7
|
+
- Commit:
|
|
8
|
+
- OCI/image/package digests:
|
|
9
|
+
- SBOM/provenance/signatures:
|
|
10
|
+
- Database migration:
|
|
11
|
+
- Architecture profile digest:
|
|
12
|
+
|
|
13
|
+
## Placement
|
|
14
|
+
|
|
15
|
+
Verify each enabled owned deployable and each shared/external contract against the selected profile. For Cloudflare/OpenNext, verify owned EVE/jobs stay in external long-running Node/OCI placement.
|
|
16
|
+
|
|
17
|
+
## Safety gates
|
|
18
|
+
|
|
19
|
+
- [ ] Backward-compatible API/events and expand/contract migration
|
|
20
|
+
- [ ] Applicable runtime/protocol families are exact and atomically qualified
|
|
21
|
+
- [ ] Applicable tenancy/authz paths are cache-safe and negatively tested
|
|
22
|
+
- [ ] Applicable sandbox/connector/secret boundaries are qualified
|
|
23
|
+
- [ ] Applicable backup/restore evidence is within confirmed RPO/RTO
|
|
24
|
+
- [ ] Rollout, health, rollback, and incident ownership are explicit
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Service design: name
|
|
2
|
+
|
|
3
|
+
Keep only sections and fields activated by the service. Mark unknown material facts `NOT VERIFIED`; do not fill owners, data classes or objectives with placeholders presented as compliance.
|
|
4
|
+
|
|
5
|
+
## Ownership and purpose
|
|
6
|
+
|
|
7
|
+
- Owner:
|
|
8
|
+
- Runtime:
|
|
9
|
+
- Hosting profiles:
|
|
10
|
+
- Data classes:
|
|
11
|
+
- SLO/RPO/RTO:
|
|
12
|
+
|
|
13
|
+
## Contracts
|
|
14
|
+
|
|
15
|
+
List only applicable REST/OpenAPI, events, queue jobs, EVE hooks/streams, shared-service contracts and generated clients. Identify idempotency and compatibility rules.
|
|
16
|
+
|
|
17
|
+
## Tenancy and identity
|
|
18
|
+
|
|
19
|
+
For shared-schema multi-tenancy, describe workspace derivation, composite keys, every tenant table's `ENABLE/FORCE RLS`, symmetric `USING/WITH CHECK`, transaction-local context, owner/BYPASSRLS separation and pool-reuse evidence. Otherwise state the actual access/tenancy boundary without adding RLS.
|
|
20
|
+
|
|
21
|
+
For identity-facing services, record secure-cookie/session-cache policy, sensitive database validation, OAuth issuer/audience/code+PKCE/redirect allowlist/refresh rotation/revocation, SCIM organization mapping/deprovision/last-owner behavior, API-key scope, and internal short-lived identity.
|
|
22
|
+
|
|
23
|
+
## State and failure recovery
|
|
24
|
+
|
|
25
|
+
Distinguish only state classes the service owns or consumes. Provide applicable retry, dedupe, DLQ, replay, recovery and reconciliation behavior.
|
|
26
|
+
|
|
27
|
+
## Security and observability
|
|
28
|
+
|
|
29
|
+
Define secret references/broker lifetime, credential-bearing env/mount prohibitions, egress hosts/CIDRs/ports/DNS/redirect/header policy, logging-sink credential review, redaction, audit events, OTel signals, alerts, and abuse limits.
|
|
30
|
+
|
|
31
|
+
## Verification
|
|
32
|
+
|
|
33
|
+
List tests only for enabled boundaries. Mark environment-bound qualification as NOT RUN with reason, risk, owner and next action.
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Threat model: system or change
|
|
2
|
+
|
|
3
|
+
Keep only enabled/exposed actors, assets and boundaries. Do not add agent, connector, sandbox, tenant or enterprise-identity threats to a project that lacks those surfaces.
|
|
4
|
+
|
|
5
|
+
## Scope and assets
|
|
6
|
+
|
|
7
|
+
List trust zones, tenant data, credentials, durable state, and excluded systems.
|
|
8
|
+
|
|
9
|
+
## Actors and entry points
|
|
10
|
+
|
|
11
|
+
List users, support operators, services, agents, connectors, webhooks, MCP servers, sandboxes, and supply-chain inputs.
|
|
12
|
+
|
|
13
|
+
## Data flows and boundaries
|
|
14
|
+
|
|
15
|
+
Reference the system-context, identity, sandbox, and evidence diagrams.
|
|
16
|
+
|
|
17
|
+
Record OAuth/session/SCIM trust transitions, database role and pool boundaries, support elevation, secret-broker exchange, sandbox env/mount/stdout/stderr paths, and every network redirect/DNS/header hop.
|
|
18
|
+
|
|
19
|
+
## Threats and controls
|
|
20
|
+
|
|
21
|
+
| ID | Threat | Boundary | Impact | Prevent | Detect | Recover | Owner | Test |
|
|
22
|
+
|---|---|---|---|---|---|---|---|---|
|
|
23
|
+
| T-001 | Cross-tenant object reference | REST/RLS | Restricted data disclosure | Composite keys + authz + RLS | Audit anomaly | Revoke/export/notify | Security | Negative tenant test |
|
|
24
|
+
|
|
25
|
+
## Residual risk and review triggers
|
|
26
|
+
|
|
27
|
+
Record accepted risks in linked ADRs and define review events.
|
|
28
|
+
|
|
29
|
+
## Required adversarial evidence
|
|
30
|
+
|
|
31
|
+
- Cross-tenant object/join/bulk/export/background access and pool-role reuse
|
|
32
|
+
- OAuth state/PKCE/replay/redirect, session revocation, SCIM deprovision/last-owner, and support expiry
|
|
33
|
+
- Credential injection/logging, webhook replay, SSRF redirect/DNS rebinding/IPv6/link-local, and sandbox proxy/header exfiltration
|
|
34
|
+
- Workflow lost acknowledgement, replayed effects, cancellation race, duplicate admission, and projector cursor rollback
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
# Agent product
|
|
2
|
+
|
|
3
|
+
Apply this reference only when the `agents` capability is enabled or agent authoring/execution is observed.
|
|
4
|
+
|
|
5
|
+
## Authoring and publication
|
|
6
|
+
|
|
7
|
+
- **AGENT-001 — Immutable publication.** Instructions, model policy, tools, skills, connections, channels, schedules, memory, knowledge, structured output, approvals, and evals MAY change in a draft. Publishing **MUST** compile a normalized immutable version.
|
|
8
|
+
- **AGENT-002 — Version-pinned execution.** Every run and conversation **MUST** pin the published agent version, policy version, tool schemas, model policy, knowledge policy, and EVE/Workflow protocol family. Draft edits **MUST NOT** mutate in-flight work.
|
|
9
|
+
- **AGENT-003 — Deterministic compiler.** Compilation **MUST** use canonical ordering, explicit defaults, a schema version, and source digest; identical inputs must produce byte-identical output. Compilation **MUST NOT** resolve plaintext secrets.
|
|
10
|
+
- **AGENT-004 — Risk-based publication.** Publishing **MUST** gate schema compatibility, secret references, permissions, capability escalation, external tools, EVE compatibility, eval regressions, output schemas, approval coverage, egress, quota, and migration impact.
|
|
11
|
+
|
|
12
|
+
| Draft concern | Published output | Required gate |
|
|
13
|
+
|---|---|---|
|
|
14
|
+
| instructions and model policy | normalized prompt and route | schema and data-class policy |
|
|
15
|
+
| tools, skills, connections | typed capabilities and scopes | risk, secret reference, approval |
|
|
16
|
+
| schedules and channels | admission triggers | tenant, dedupe, timezone, owner |
|
|
17
|
+
| output contract | versioned JSON Schema | conformance evaluation |
|
|
18
|
+
| memory and knowledge | provenance and retention policy | tenant and data-class validation |
|
|
19
|
+
|
|
20
|
+
Use the lifecycle `draft → preview → publish → production → rollback`. There is no separate staging state. Risk-increasing changes require review; a low-risk text change MAY use policy-approved automation. Rollback activates an earlier immutable version and never rewrites history.
|
|
21
|
+
|
|
22
|
+
Support private and workspace sharing plus versioned templates. Template copies retain provenance but become independent drafts. Public marketplace moderation, discovery, and billing remain out of scope.
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
# AI cost
|
|
2
|
+
|
|
3
|
+
Apply this reference only when metered AI resources exist — model calls, sandbox runtime, embedding or knowledge processing. Cost here is an architecture concern: unattributed spend is an ownership gap, and an unbounded token path is a reliability defect. Billing, pricing, and Stripe remain out of foundation scope.
|
|
4
|
+
|
|
5
|
+
- **COST-001 — Workspace attribution.** Every model, embedding, sandbox, and knowledge-processing call **MUST** record per-workspace attribution: tokens in/out, cached versus uncached, provider and model pin, feature or agent identity, and run correlation. Aggregate provider invoices are not attribution; the platform must be able to state what any workspace cost yesterday.
|
|
6
|
+
- **COST-002 — Budgets and alerts.** Production AI features **MUST** define token or spend budgets per workspace and per feature, with alert thresholds and an explicit overrun behavior — shed, degrade, or pause visibly. Silent unlimited spend and silent hard-stop are both forbidden; the overrun behavior is a declared product decision.
|
|
7
|
+
- **COST-003 — Cache economics.** Prompt/context caching **SHOULD** be a measured decision: record hit rate and cost delta per route, and structure prompts so stable prefixes actually cache. Caching **MUST NOT** weaken region, retention, or tenant policy (`MODEL-003`); a cache that saves money by crossing a policy boundary fails.
|
|
8
|
+
- **COST-004 — Cost regression gate.** AI feature deploys **MUST** compare cost per interaction (or per run) against the current baseline, like a performance budget. A regression beyond the declared threshold blocks promotion until explained — an intended trade recorded with the change, or a defect fixed. Model swaps, prompt growth, retry amplification, and cache misses are the usual suspects.
|
|
9
|
+
|
|
10
|
+
## Measuring
|
|
11
|
+
|
|
12
|
+
| Quantity | Source | Gate use |
|
|
13
|
+
|---|---|---|
|
|
14
|
+
| tokens per run | adapter attribution record | regression baseline |
|
|
15
|
+
| cached share | adapter cache metadata | cache economics |
|
|
16
|
+
| cost per interaction | tokens × pinned model price class | `COST-004` threshold |
|
|
17
|
+
| workspace spend rate | attribution rollup | budget alerts |
|
|
18
|
+
| shed/deferred work | admission records | overrun behavior audit |
|
|
19
|
+
|
|
20
|
+
Estimate with the provider's published price class for the exact pin; reconcile estimates against provider usage reports on a fixed cadence and alarm on divergence, which usually means unattributed traffic.
|
|
21
|
+
|
|
22
|
+
Budgets, quotas, and shedding compose with `MLIFE-004`: the same admission edge that protects the provider protects the budget. Retry and fallback paths carry their own attribution so a degraded week is explainable. Cost telemetry is metadata and follows `OBS-002` — amounts and counters, never prompt content.
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
# AI data boundaries
|
|
2
|
+
|
|
3
|
+
Apply this reference when personal data, knowledge ingestion, external model calls, or public-facing AI output exists. It concretizes `SEC-003` and `DATA-003` at the AI trust boundary: personal data moving outward, untrusted content moving inward.
|
|
4
|
+
|
|
5
|
+
- **PII-001 — Ingestion redaction.** PII detection and classification **MUST** run before content enters embedding, chunking, or knowledge storage; detected PII is redacted, tokenized, or explicitly admitted under the data class policy of the target store. Record the classifier version and decision as provenance so re-scanning after a classifier upgrade is possible. Embeddings of unredacted PII are copies of it and inherit deletion propagation (`DATA-004`).
|
|
6
|
+
- **PII-002 — Prompt boundary.** Before a prompt crosses the trust boundary to an external model provider, PII **MUST** be minimized to what the feature needs and stripped or masked where policy requires; the route's retention and region policy (`MODEL-001`) governs what may cross at all. Memory writes and eval/telemetry sampling are prompt-boundary crossings too — the same redaction applies.
|
|
7
|
+
- **PII-003 — Output moderation.** Public-facing products **MUST** pass model output through a moderation policy before delivery: unsafe-content classes, PII leakage from context or memory, and impersonation of the platform. Internal tools MAY relax categories by declared policy, never by omission. Moderation outcomes are audited metadata; blocked output fails visibly, not silently.
|
|
8
|
+
- **PII-004 — Injection mitigations.** Prompt injection is mitigated by structure, not instructions. Tool results, retrieved chunks, and connector content **MUST** be quarantined as data — spotlighted/delimited with an explicit untrusted marking the prompt template preserves — and **MUST NOT** be able to elevate capabilities: high-impact tool calls (irreversible, external-effect, privileged, or capability-escalating) require the `CONN-001` capability envelope and an approval gate regardless of what any content said. Instructions found inside untrusted spans are never followed into tool selection.
|
|
9
|
+
|
|
10
|
+
## Boundary placement
|
|
11
|
+
|
|
12
|
+
| Crossing | Control | Failure mode prevented |
|
|
13
|
+
|---|---|---|
|
|
14
|
+
| document → embedding/knowledge store | `PII-001` classify/redact | PII copies in vectors and chunks |
|
|
15
|
+
| context → external model | `PII-002` minimize/mask | retention outside policy |
|
|
16
|
+
| model → public user | `PII-003` moderation | unsafe or leaking output |
|
|
17
|
+
| tool/retrieval → prompt | `PII-004` quarantine + spotlight | injected instruction execution |
|
|
18
|
+
|
|
19
|
+
Detection is preventive, not perfect: pair classifiers with data-class defaults (unknown source means restricted), negative tests that plant markers and verify they never reach a provider or a public response, and deletion reconciliation that includes vector stores and cached prompts. Treat retrieved text as untrusted data everywhere, matching `data-memory.md`; an approval gate that a model can talk its way around is prompt-enforced security and fails `SEC-003`.
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# AI evaluation
|
|
2
|
+
|
|
3
|
+
Apply this reference only when agents, prompts, or other model-backed behavior ship to users. A project with no model-backed behavior needs no eval infrastructure.
|
|
4
|
+
|
|
5
|
+
An eval is a versioned golden dataset plus a scoring method plus an explicit pass threshold, run against a pinned model and prompt. A demo transcript, a vibe check, or an unpinned notebook run is not an eval and produces no evidence.
|
|
6
|
+
|
|
7
|
+
- **EVAL-001 — Eval definition.** Every evaluated behavior **MUST** define a golden dataset with provenance and tenant-safe sourcing, a deterministic scoring method or declared judge configuration, and numeric pass/regression thresholds recorded before the run, not chosen after it.
|
|
8
|
+
- **EVAL-002 — Promotion gate.** Changes to agent instructions, prompts, model policy, routed model, tool schemas, or knowledge policy **MUST** pass offline regression evals against the golden datasets before publish or promote. A threshold regression blocks promotion; overriding it is a project-owner accepted risk, not a pass.
|
|
9
|
+
- **EVAL-003 — Versioned datasets.** Eval datasets, scoring configuration, and thresholds **MUST** be versioned alongside the prompts and agent versions they gate, so any historical eval result can be reproduced from its exact inputs. Dataset edits that change pass rates are behavior changes and go through the same review as prompt changes.
|
|
10
|
+
- **EVAL-004 — Online sampling.** Production eval sampling **MUST** respect workspace data policy, retention, and redaction before any live interaction enters a dataset or judge prompt. Sampled cases feed dataset growth through review, never automatically.
|
|
11
|
+
|
|
12
|
+
## Judge models
|
|
13
|
+
|
|
14
|
+
Model-as-judge scoring is permitted with declared caveats: pin the judge model and prompt version, measure judge agreement against a human-labeled slice before trusting it, and re-baseline whenever the judge model changes. Judge scores are relative evidence — they compare candidates under one fixed judge; they do not certify absolute quality. Do not use the model under test as its own judge, and do not let judge drift silently move a threshold.
|
|
15
|
+
|
|
16
|
+
## Operating the gate
|
|
17
|
+
|
|
18
|
+
| Change | Required eval evidence |
|
|
19
|
+
|---|---|
|
|
20
|
+
| prompt or instruction edit | offline regression pass on affected datasets |
|
|
21
|
+
| model or route change | full regression pass plus cost/latency comparison |
|
|
22
|
+
| tool schema change | tool-selection and output-conformance pass |
|
|
23
|
+
| knowledge policy change | retrieval-grounded answer pass |
|
|
24
|
+
| dataset edit | reviewed diff and re-baselined thresholds |
|
|
25
|
+
|
|
26
|
+
Keep datasets small enough to run in CI and grow them from triaged production failures, not synthetic bulk. Record every gate run with dataset version, model, prompt version, scores, and outcome so `AGENT-004` publication gates can cite it. Offline pass plus bounded online sampling is the evidence pair; neither alone qualifies a behavior change for tenants.
|