system-design-primer-skill 0.0.0-stage → 2.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ATTRIBUTION.md +10 -0
- package/LICENSE +7 -0
- package/README.md +180 -2
- package/SKILL.md +447 -0
- package/VERSION +1 -0
- package/package.json +78 -4
- package/references/00-primer-map-and-attribution.md +29 -0
- package/references/01-requirements-and-capacity.md +107 -0
- package/references/02-architecture-and-networking.md +94 -0
- package/references/03-data-and-storage.md +88 -0
- package/references/04-caching.md +78 -0
- package/references/05-async-and-messaging.md +77 -0
- package/references/06-distributed-systems.md +59 -0
- package/references/07-reliability-and-resilience.md +71 -0
- package/references/08-api-security-observability.md +62 -0
- package/references/09-modern-engineering.md +58 -0
- package/references/10-workflows-and-checklists.md +97 -0
- package/references/11-case-studies.md +74 -0
- package/references/12-ai-systems-architecture.md +139 -0
- package/references/13-architecture-intelligence.md +91 -0
- package/references/14-api-and-data-contracts.md +80 -0
- package/references/15-failure-analysis.md +84 -0
- package/references/16-multi-agent-architecture.md +76 -0
- package/references/17-agent-harness-and-runtime.md +87 -0
- package/references/18-mcp-architecture.md +67 -0
- package/references/19-agent-security-and-permissions.md +74 -0
- package/references/20-agent-state-and-memory.md +72 -0
- package/references/21-agent-evaluation-and-observability.md +90 -0
- package/references/22-kubernetes-and-container-platforms.md +29 -0
- package/references/23-service-mesh-and-traffic-management.md +21 -0
- package/references/24-multi-region-and-disaster-recovery.md +24 -0
- package/references/25-platform-engineering-and-developer-experience.md +26 -0
- package/references/26-finops-and-cloud-cost.md +29 -0
- package/references/27-production-observability-and-slo.md +29 -0
- package/references/28-event-streaming-and-kafka-pulsar.md +27 -0
- package/references/29-event-sourcing-and-cqrs.md +23 -0
- package/references/30-change-data-capture-and-debezium.md +27 -0
- package/references/31-data-lakehouse-and-stream-batch.md +24 -0
- package/references/32-workflow-orchestration-and-temporal.md +24 -0
- package/references/33-schema-registry-and-event-governance.md +22 -0
- package/references/34-postgresql-internals-and-performance.md +43 -0
- package/references/35-mysql-innodb-and-performance.md +27 -0
- package/references/36-sharding-and-distributed-sql.md +22 -0
- package/references/37-consensus-and-distributed-transactions.md +25 -0
- package/references/38-search-architecture-elasticsearch-opensearch.md +24 -0
- package/references/39-vector-databases-and-hybrid-retrieval.md +31 -0
- package/references/40-zero-downtime-database-migrations.md +27 -0
- package/references/41-api-gateway-and-edge-architecture.md +23 -0
- package/references/42-rest-api-design-and-lifecycle.md +23 -0
- package/references/43-graphql-architecture-and-governance.md +16 -0
- package/references/44-grpc-and-service-contracts.md +16 -0
- package/references/45-websocket-sse-and-realtime-systems.md +16 -0
- package/references/46-realtime-collaboration-ot-crdt.md +16 -0
- package/references/47-api-security-rate-limiting-and-abuse.md +16 -0
- package/references/48-oauth-oidc-and-passkeys.md +20 -0
- package/references/49-iam-rbac-abac-rebac.md +16 -0
- package/references/50-zero-trust-and-workload-identity.md +16 -0
- package/references/51-secrets-kms-pki-and-key-management.md +16 -0
- package/references/52-supply-chain-security-sbom-slsa.md +16 -0
- package/references/53-threat-modeling-stride-and-attack-trees.md +16 -0
- package/references/54-privacy-engineering-and-data-governance.md +16 -0
- package/references/55-opentelemetry-and-instrumentation.md +50 -0
- package/references/56-distributed-tracing-and-context-propagation.md +43 -0
- package/references/57-continuous-profiling-and-performance-diagnostics.md +45 -0
- package/references/58-load-stress-soak-testing-and-workload-modeling.md +44 -0
- package/references/59-capacity-planning-and-performance-modeling.md +34 -0
- package/references/60-chaos-engineering-and-resilience-experiments.md +41 -0
- package/references/61-incident-management-and-postmortems.md +34 -0
- package/references/62-dns-and-service-discovery.md +35 -0
- package/references/63-load-balancing-and-global-traffic-management.md +31 -0
- package/references/64-cdn-and-edge-caching.md +34 -0
- package/references/65-cloud-networking-and-connectivity.md +34 -0
- package/references/66-edge-computing-architecture.md +33 -0
- package/references/67-storage-infrastructure-and-storage-systems.md +30 -0
- package/references/68-tcp-quic-and-network-performance.md +27 -0
- package/references/69-task-routing-and-evidence.md +58 -0
- package/scripts/install.ps1 +124 -0
- package/scripts/install.sh +105 -0
- package/scripts/validate.py +376 -0
- package/templates/adr-template.md +32 -0
- package/templates/agent-evaluation-plan-template.md +57 -0
- package/templates/agent-security-review-template.md +46 -0
- package/templates/agent-system-design-template.md +74 -0
- package/templates/api-architecture-design-template.md +33 -0
- package/templates/api-security-and-rate-limit-review-template.md +29 -0
- package/templates/architecture-decision-matrix-template.md +48 -0
- package/templates/architecture-design-template.md +95 -0
- package/templates/architecture-review-scorecard-template.md +36 -0
- package/templates/architecture-review-template.md +52 -0
- package/templates/cdc-data-pipeline-template.md +41 -0
- package/templates/cdn-and-edge-strategy-template.md +36 -0
- package/templates/chaos-experiment-template.md +39 -0
- package/templates/cloud-cost-model-template.md +37 -0
- package/templates/database-architecture-review-template.md +33 -0
- package/templates/edge-computing-readiness-template.md +36 -0
- package/templates/event-sourcing-cqrs-template.md +41 -0
- package/templates/event-streaming-design-template.md +47 -0
- package/templates/failure-analysis-template.md +36 -0
- package/templates/identity-and-access-architecture-template.md +29 -0
- package/templates/incident-postmortem-template.md +39 -0
- package/templates/multi-region-disaster-recovery-template.md +43 -0
- package/templates/network-architecture-design-template.md +38 -0
- package/templates/observability-architecture-template.md +52 -0
- package/templates/performance-test-plan-template.md +52 -0
- package/templates/privacy-data-governance-template.md +24 -0
- package/templates/production-readiness-checklist-template.md +52 -0
- package/templates/realtime-system-design-template.md +28 -0
- package/templates/security-threat-model-template.md +26 -0
- package/templates/sharding-distributed-sql-design-template.md +34 -0
- package/templates/storage-architecture-design-template.md +35 -0
- package/templates/supply-chain-security-review-template.md +20 -0
- package/templates/zero-downtime-database-migration-template.md +32 -0
|
@@ -0,0 +1,376 @@
|
|
|
1
|
+
#!/usr/bin/env python3
|
|
2
|
+
"""Validate the Agent Skill package without third-party dependencies."""
|
|
3
|
+
from __future__ import annotations
|
|
4
|
+
|
|
5
|
+
import argparse
|
|
6
|
+
import json
|
|
7
|
+
import re
|
|
8
|
+
import sys
|
|
9
|
+
from pathlib import Path
|
|
10
|
+
|
|
11
|
+
REQUIRED_REFERENCES = {
|
|
12
|
+
"00-primer-map-and-attribution.md",
|
|
13
|
+
"01-requirements-and-capacity.md",
|
|
14
|
+
"02-architecture-and-networking.md",
|
|
15
|
+
"03-data-and-storage.md",
|
|
16
|
+
"04-caching.md",
|
|
17
|
+
"05-async-and-messaging.md",
|
|
18
|
+
"06-distributed-systems.md",
|
|
19
|
+
"07-reliability-and-resilience.md",
|
|
20
|
+
"08-api-security-observability.md",
|
|
21
|
+
"09-modern-engineering.md",
|
|
22
|
+
"10-workflows-and-checklists.md",
|
|
23
|
+
"11-case-studies.md",
|
|
24
|
+
"12-ai-systems-architecture.md",
|
|
25
|
+
"13-architecture-intelligence.md",
|
|
26
|
+
"14-api-and-data-contracts.md",
|
|
27
|
+
"15-failure-analysis.md",
|
|
28
|
+
"16-multi-agent-architecture.md",
|
|
29
|
+
"17-agent-harness-and-runtime.md",
|
|
30
|
+
"18-mcp-architecture.md",
|
|
31
|
+
"19-agent-security-and-permissions.md",
|
|
32
|
+
"20-agent-state-and-memory.md",
|
|
33
|
+
"21-agent-evaluation-and-observability.md",
|
|
34
|
+
"22-kubernetes-and-container-platforms.md",
|
|
35
|
+
"23-service-mesh-and-traffic-management.md",
|
|
36
|
+
"24-multi-region-and-disaster-recovery.md",
|
|
37
|
+
"25-platform-engineering-and-developer-experience.md",
|
|
38
|
+
"26-finops-and-cloud-cost.md",
|
|
39
|
+
"27-production-observability-and-slo.md",
|
|
40
|
+
"28-event-streaming-and-kafka-pulsar.md",
|
|
41
|
+
"29-event-sourcing-and-cqrs.md",
|
|
42
|
+
"30-change-data-capture-and-debezium.md",
|
|
43
|
+
"31-data-lakehouse-and-stream-batch.md",
|
|
44
|
+
"32-workflow-orchestration-and-temporal.md",
|
|
45
|
+
"33-schema-registry-and-event-governance.md",
|
|
46
|
+
"34-postgresql-internals-and-performance.md",
|
|
47
|
+
"35-mysql-innodb-and-performance.md",
|
|
48
|
+
"36-sharding-and-distributed-sql.md",
|
|
49
|
+
"37-consensus-and-distributed-transactions.md",
|
|
50
|
+
"38-search-architecture-elasticsearch-opensearch.md",
|
|
51
|
+
"39-vector-databases-and-hybrid-retrieval.md",
|
|
52
|
+
"40-zero-downtime-database-migrations.md",
|
|
53
|
+
"41-api-gateway-and-edge-architecture.md",
|
|
54
|
+
"42-rest-api-design-and-lifecycle.md",
|
|
55
|
+
"43-graphql-architecture-and-governance.md",
|
|
56
|
+
"44-grpc-and-service-contracts.md",
|
|
57
|
+
"45-websocket-sse-and-realtime-systems.md",
|
|
58
|
+
"46-realtime-collaboration-ot-crdt.md",
|
|
59
|
+
"47-api-security-rate-limiting-and-abuse.md",
|
|
60
|
+
"48-oauth-oidc-and-passkeys.md",
|
|
61
|
+
"49-iam-rbac-abac-rebac.md",
|
|
62
|
+
"50-zero-trust-and-workload-identity.md",
|
|
63
|
+
"51-secrets-kms-pki-and-key-management.md",
|
|
64
|
+
"52-supply-chain-security-sbom-slsa.md",
|
|
65
|
+
"53-threat-modeling-stride-and-attack-trees.md",
|
|
66
|
+
"54-privacy-engineering-and-data-governance.md",
|
|
67
|
+
"55-opentelemetry-and-instrumentation.md",
|
|
68
|
+
"56-distributed-tracing-and-context-propagation.md",
|
|
69
|
+
"57-continuous-profiling-and-performance-diagnostics.md",
|
|
70
|
+
"58-load-stress-soak-testing-and-workload-modeling.md",
|
|
71
|
+
"59-capacity-planning-and-performance-modeling.md",
|
|
72
|
+
"60-chaos-engineering-and-resilience-experiments.md",
|
|
73
|
+
"61-incident-management-and-postmortems.md",
|
|
74
|
+
"62-dns-and-service-discovery.md",
|
|
75
|
+
"63-load-balancing-and-global-traffic-management.md",
|
|
76
|
+
"64-cdn-and-edge-caching.md",
|
|
77
|
+
"65-cloud-networking-and-connectivity.md",
|
|
78
|
+
"66-edge-computing-architecture.md",
|
|
79
|
+
"67-storage-infrastructure-and-storage-systems.md",
|
|
80
|
+
"68-tcp-quic-and-network-performance.md",
|
|
81
|
+
"69-task-routing-and-evidence.md",
|
|
82
|
+
}
|
|
83
|
+
REQUIRED_TEMPLATES = {
|
|
84
|
+
"adr-template.md",
|
|
85
|
+
"architecture-design-template.md",
|
|
86
|
+
"architecture-review-template.md",
|
|
87
|
+
"architecture-decision-matrix-template.md",
|
|
88
|
+
"architecture-review-scorecard-template.md",
|
|
89
|
+
"failure-analysis-template.md",
|
|
90
|
+
"agent-system-design-template.md",
|
|
91
|
+
"agent-security-review-template.md",
|
|
92
|
+
"agent-evaluation-plan-template.md",
|
|
93
|
+
"production-readiness-checklist-template.md",
|
|
94
|
+
"multi-region-disaster-recovery-template.md",
|
|
95
|
+
"cloud-cost-model-template.md",
|
|
96
|
+
"event-streaming-design-template.md",
|
|
97
|
+
"event-sourcing-cqrs-template.md",
|
|
98
|
+
"cdc-data-pipeline-template.md",
|
|
99
|
+
"database-architecture-review-template.md",
|
|
100
|
+
"sharding-distributed-sql-design-template.md",
|
|
101
|
+
"zero-downtime-database-migration-template.md",
|
|
102
|
+
"api-architecture-design-template.md",
|
|
103
|
+
"realtime-system-design-template.md",
|
|
104
|
+
"api-security-and-rate-limit-review-template.md",
|
|
105
|
+
"identity-and-access-architecture-template.md",
|
|
106
|
+
"security-threat-model-template.md",
|
|
107
|
+
"supply-chain-security-review-template.md",
|
|
108
|
+
"privacy-data-governance-template.md",
|
|
109
|
+
"observability-architecture-template.md",
|
|
110
|
+
"performance-test-plan-template.md",
|
|
111
|
+
"chaos-experiment-template.md",
|
|
112
|
+
"incident-postmortem-template.md",
|
|
113
|
+
"network-architecture-design-template.md",
|
|
114
|
+
"cdn-and-edge-strategy-template.md",
|
|
115
|
+
"storage-architecture-design-template.md",
|
|
116
|
+
"edge-computing-readiness-template.md",
|
|
117
|
+
}
|
|
118
|
+
PACKAGE_ENTRIES = (
|
|
119
|
+
"SKILL.md", "references", "templates", "scripts",
|
|
120
|
+
"ATTRIBUTION.md", "LICENSE", "VERSION",
|
|
121
|
+
)
|
|
122
|
+
LOCAL_LINK_RE = re.compile(r"\]\(([^)]+)\)")
|
|
123
|
+
SEMVER_RE = re.compile(r"\d+\.\d+\.\d+(?:-[0-9A-Za-z.-]+)?")
|
|
124
|
+
|
|
125
|
+
|
|
126
|
+
def parse_frontmatter(content: str) -> tuple[dict[str, str], str]:
|
|
127
|
+
lines = content.splitlines()
|
|
128
|
+
if not lines or lines[0].strip() != "---":
|
|
129
|
+
raise ValueError("SKILL.md must start with a YAML frontmatter delimiter")
|
|
130
|
+
try:
|
|
131
|
+
end = next(i for i in range(1, len(lines)) if lines[i].strip() == "---")
|
|
132
|
+
except StopIteration as exc:
|
|
133
|
+
raise ValueError("YAML frontmatter closing delimiter is missing") from exc
|
|
134
|
+
|
|
135
|
+
fields: dict[str, str] = {}
|
|
136
|
+
for line in lines[1:end]:
|
|
137
|
+
if not line.strip() or line.lstrip().startswith("#") or line[:1].isspace():
|
|
138
|
+
continue
|
|
139
|
+
key, sep, value = line.partition(":")
|
|
140
|
+
if not sep:
|
|
141
|
+
raise ValueError(f"invalid frontmatter line: {line}")
|
|
142
|
+
key = key.strip()
|
|
143
|
+
value = value.strip()
|
|
144
|
+
if key in fields:
|
|
145
|
+
raise ValueError(f"duplicate frontmatter field: {key}")
|
|
146
|
+
if len(value) >= 2 and value[0] == value[-1] and value[0] in ("'", '"'):
|
|
147
|
+
value = value[1:-1]
|
|
148
|
+
fields[key] = value
|
|
149
|
+
return fields, "\n".join(lines[end + 1:])
|
|
150
|
+
|
|
151
|
+
|
|
152
|
+
def _check_package_symlinks(root: Path) -> list[str]:
|
|
153
|
+
errors: list[str] = []
|
|
154
|
+
for entry in PACKAGE_ENTRIES:
|
|
155
|
+
path = root / entry
|
|
156
|
+
if path.is_symlink():
|
|
157
|
+
errors.append(f"distributable package entry must not be a symlink: {entry}")
|
|
158
|
+
continue
|
|
159
|
+
if path.is_dir():
|
|
160
|
+
for child in path.rglob("*"):
|
|
161
|
+
if child.is_symlink():
|
|
162
|
+
errors.append(
|
|
163
|
+
f"distributable package path must not contain symlinks: "
|
|
164
|
+
f"{child.relative_to(root)}"
|
|
165
|
+
)
|
|
166
|
+
return errors
|
|
167
|
+
|
|
168
|
+
|
|
169
|
+
def validate(root: Path) -> list[str]:
|
|
170
|
+
errors: list[str] = []
|
|
171
|
+
root = root.resolve()
|
|
172
|
+
skill_file = root / "SKILL.md"
|
|
173
|
+
if not skill_file.is_file():
|
|
174
|
+
return ["SKILL.md is missing"]
|
|
175
|
+
|
|
176
|
+
content = skill_file.read_text(encoding="utf-8")
|
|
177
|
+
try:
|
|
178
|
+
fields, body = parse_frontmatter(content)
|
|
179
|
+
except ValueError as exc:
|
|
180
|
+
return [str(exc)]
|
|
181
|
+
|
|
182
|
+
name = fields.get("name", "")
|
|
183
|
+
description = fields.get("description", "")
|
|
184
|
+
if not name:
|
|
185
|
+
errors.append("frontmatter field 'name' is required")
|
|
186
|
+
elif len(name) > 64 or not re.fullmatch(r"[a-z0-9]+(?:-[a-z0-9]+)*", name):
|
|
187
|
+
errors.append("name must be <=64 chars and contain lowercase letters, digits, and hyphens only")
|
|
188
|
+
if name in {"anthropic", "claude"}:
|
|
189
|
+
errors.append(f"reserved skill name: {name}")
|
|
190
|
+
if not description:
|
|
191
|
+
errors.append("frontmatter field 'description' is required and must be non-empty")
|
|
192
|
+
elif len(description) > 1024:
|
|
193
|
+
errors.append(f"description exceeds 1024 characters ({len(description)})")
|
|
194
|
+
if re.search(r"<[^>]+>", description):
|
|
195
|
+
errors.append("description must not contain XML/HTML tags")
|
|
196
|
+
if len(content.splitlines()) > 500:
|
|
197
|
+
errors.append(f"SKILL.md exceeds 500 lines ({len(content.splitlines())})")
|
|
198
|
+
if not body.strip():
|
|
199
|
+
errors.append("SKILL.md body is empty")
|
|
200
|
+
|
|
201
|
+
for entry in PACKAGE_ENTRIES:
|
|
202
|
+
if not (root / entry).exists():
|
|
203
|
+
errors.append(f"required package entry is missing: {entry}")
|
|
204
|
+
errors.extend(_check_package_symlinks(root))
|
|
205
|
+
|
|
206
|
+
refs_dir = root / "references"
|
|
207
|
+
if refs_dir.is_dir():
|
|
208
|
+
actual_refs = {p.name for p in refs_dir.glob("*.md")}
|
|
209
|
+
for filename in sorted(REQUIRED_REFERENCES - actual_refs):
|
|
210
|
+
errors.append(f"required reference is missing: references/{filename}")
|
|
211
|
+
for filename in sorted(actual_refs):
|
|
212
|
+
if (refs_dir / filename).stat().st_size == 0:
|
|
213
|
+
errors.append(f"empty reference file: references/{filename}")
|
|
214
|
+
|
|
215
|
+
templates_dir = root / "templates"
|
|
216
|
+
if templates_dir.is_dir():
|
|
217
|
+
actual_templates = {p.name for p in templates_dir.glob("*.md")}
|
|
218
|
+
for filename in sorted(REQUIRED_TEMPLATES - actual_templates):
|
|
219
|
+
errors.append(f"required template is missing: templates/{filename}")
|
|
220
|
+
|
|
221
|
+
for markdown in root.rglob("*.md"):
|
|
222
|
+
text = markdown.read_text(encoding="utf-8")
|
|
223
|
+
for target in LOCAL_LINK_RE.findall(text):
|
|
224
|
+
if target.startswith(("http://", "https://", "mailto:", "#")):
|
|
225
|
+
continue
|
|
226
|
+
path_text = target.split("#", 1)[0].split("?", 1)[0]
|
|
227
|
+
if not path_text:
|
|
228
|
+
continue
|
|
229
|
+
resolved = (markdown.parent / path_text).resolve()
|
|
230
|
+
try:
|
|
231
|
+
resolved.relative_to(root)
|
|
232
|
+
except ValueError:
|
|
233
|
+
errors.append(
|
|
234
|
+
f"local link escapes repository root in "
|
|
235
|
+
f"{markdown.relative_to(root)}: {target}"
|
|
236
|
+
)
|
|
237
|
+
continue
|
|
238
|
+
if not resolved.exists():
|
|
239
|
+
errors.append(
|
|
240
|
+
f"broken local link in {markdown.relative_to(root)}: {target}"
|
|
241
|
+
)
|
|
242
|
+
|
|
243
|
+
linked_paths = set(
|
|
244
|
+
re.findall(r"`((?:references|templates|scripts)/[^`]+)`", content)
|
|
245
|
+
)
|
|
246
|
+
for linked in sorted(linked_paths):
|
|
247
|
+
if not (root / linked).is_file():
|
|
248
|
+
errors.append(f"SKILL.md references a missing file: {linked}")
|
|
249
|
+
|
|
250
|
+
version_file = root / "VERSION"
|
|
251
|
+
version = ""
|
|
252
|
+
if version_file.is_file():
|
|
253
|
+
version = version_file.read_text(encoding="utf-8").strip()
|
|
254
|
+
if not SEMVER_RE.fullmatch(version):
|
|
255
|
+
errors.append(f"VERSION is not valid semantic version text: {version!r}")
|
|
256
|
+
metadata_version = re.search(
|
|
257
|
+
r"(?m)^metadata:\s*\n[ \t]+version:\s*['\"]?([^\s'\"]+)",
|
|
258
|
+
content,
|
|
259
|
+
)
|
|
260
|
+
if metadata_version and metadata_version.group(1) != version:
|
|
261
|
+
errors.append(
|
|
262
|
+
f"SKILL.md metadata version {metadata_version.group(1)!r} "
|
|
263
|
+
f"does not match VERSION {version!r}"
|
|
264
|
+
)
|
|
265
|
+
|
|
266
|
+
eval_file = root / "evals" / "system-design-cases.json"
|
|
267
|
+
if not eval_file.is_file():
|
|
268
|
+
errors.append("evaluation case file is missing: evals/system-design-cases.json")
|
|
269
|
+
else:
|
|
270
|
+
try:
|
|
271
|
+
evaluation = json.loads(eval_file.read_text(encoding="utf-8"))
|
|
272
|
+
except (OSError, json.JSONDecodeError) as exc:
|
|
273
|
+
errors.append(f"evaluation case file is invalid JSON: {exc}")
|
|
274
|
+
evaluation = {}
|
|
275
|
+
cases = evaluation.get("cases", []) if isinstance(evaluation, dict) else []
|
|
276
|
+
if not isinstance(cases, list) or not cases:
|
|
277
|
+
errors.append("evaluation case file must contain a non-empty 'cases' array")
|
|
278
|
+
else:
|
|
279
|
+
seen_case_ids: set[str] = set()
|
|
280
|
+
for index, case in enumerate(cases):
|
|
281
|
+
prefix = f"evaluation case {index + 1}"
|
|
282
|
+
if not isinstance(case, dict):
|
|
283
|
+
errors.append(f"{prefix} must be an object")
|
|
284
|
+
continue
|
|
285
|
+
case_id = case.get("id")
|
|
286
|
+
if not isinstance(case_id, str) or not case_id.strip():
|
|
287
|
+
errors.append(f"{prefix} must have a non-empty string id")
|
|
288
|
+
elif case_id in seen_case_ids:
|
|
289
|
+
errors.append(f"duplicate evaluation case id: {case_id}")
|
|
290
|
+
else:
|
|
291
|
+
seen_case_ids.add(case_id)
|
|
292
|
+
for field in ("task", "expected_mode", "required_outputs", "guardrails"):
|
|
293
|
+
if not case.get(field):
|
|
294
|
+
errors.append(f"{prefix} is missing required field: {field}")
|
|
295
|
+
references = case.get("references", [])
|
|
296
|
+
if not isinstance(references, list):
|
|
297
|
+
errors.append(f"{prefix} references must be an array")
|
|
298
|
+
else:
|
|
299
|
+
for reference in references:
|
|
300
|
+
if not isinstance(reference, str) or not (refs_dir / reference).is_file():
|
|
301
|
+
errors.append(f"{prefix} references a missing reference: {reference!r}")
|
|
302
|
+
|
|
303
|
+
package_file = root / "package.json"
|
|
304
|
+
if not package_file.is_file():
|
|
305
|
+
errors.append("package.json is missing")
|
|
306
|
+
else:
|
|
307
|
+
try:
|
|
308
|
+
package = json.loads(package_file.read_text(encoding="utf-8"))
|
|
309
|
+
except (OSError, json.JSONDecodeError) as exc:
|
|
310
|
+
errors.append(f"package.json is invalid: {exc}")
|
|
311
|
+
package = {}
|
|
312
|
+
package_name = package.get("name", "")
|
|
313
|
+
if not isinstance(package_name, str) or not re.fullmatch(
|
|
314
|
+
r"(?:@[a-z0-9][a-z0-9._-]*/)?[a-z0-9][a-z0-9._-]*",
|
|
315
|
+
package_name,
|
|
316
|
+
):
|
|
317
|
+
errors.append("package.json name must be a valid lowercase npm package name")
|
|
318
|
+
package_version = package.get("version", "")
|
|
319
|
+
if package_version != version:
|
|
320
|
+
errors.append(
|
|
321
|
+
f"package.json version {package_version!r} does not match VERSION {version!r}"
|
|
322
|
+
)
|
|
323
|
+
if version_file.is_file() and (root / "README.md").is_file():
|
|
324
|
+
readme_text = (root / "README.md").read_text(encoding="utf-8")
|
|
325
|
+
if not re.search(r"(?m)^- \*\*Version:\*\* " + re.escape(version) + r"$", readme_text):
|
|
326
|
+
errors.append(f"README version must match VERSION {version!r}")
|
|
327
|
+
changelog_file = root / "CHANGELOG.md"
|
|
328
|
+
if changelog_file.is_file():
|
|
329
|
+
first_version = re.search(r"(?m)^##\s*([^ —]+)", changelog_file.read_text(encoding="utf-8"))
|
|
330
|
+
if not first_version or first_version.group(1) != version:
|
|
331
|
+
errors.append(f"latest CHANGELOG version must match VERSION {version!r}")
|
|
332
|
+
repository = package.get("repository", {})
|
|
333
|
+
if not isinstance(repository, dict) or repository.get("url") != (
|
|
334
|
+
"https://github.com/jack-001-a/system-design-primer-skill.git"
|
|
335
|
+
):
|
|
336
|
+
errors.append("package.json repository.url must match this GitHub repository")
|
|
337
|
+
files = package.get("files", [])
|
|
338
|
+
required_npm_files = {
|
|
339
|
+
"SKILL.md", "README.md", "ATTRIBUTION.md", "LICENSE", "VERSION",
|
|
340
|
+
"references/", "templates/", "scripts/",
|
|
341
|
+
}
|
|
342
|
+
if not isinstance(files, list) or not required_npm_files.issubset(set(files)):
|
|
343
|
+
errors.append("package.json files must include the Skill, references, templates, scripts, and license files")
|
|
344
|
+
publish_config = package.get("publishConfig", {})
|
|
345
|
+
if not isinstance(publish_config, dict) or publish_config.get("access") != "public":
|
|
346
|
+
errors.append("package.json publishConfig.access must be public")
|
|
347
|
+
if package.get("private") is True:
|
|
348
|
+
errors.append("package.json private must not be true for an npm release")
|
|
349
|
+
|
|
350
|
+
return errors
|
|
351
|
+
|
|
352
|
+
|
|
353
|
+
def main() -> int:
|
|
354
|
+
parser = argparse.ArgumentParser(description=__doc__)
|
|
355
|
+
parser.add_argument(
|
|
356
|
+
"--root",
|
|
357
|
+
type=Path,
|
|
358
|
+
default=Path(__file__).resolve().parent.parent,
|
|
359
|
+
help="Skill package root (defaults to repository root)",
|
|
360
|
+
)
|
|
361
|
+
args = parser.parse_args()
|
|
362
|
+
errors = validate(args.root)
|
|
363
|
+
if errors:
|
|
364
|
+
for error in errors:
|
|
365
|
+
print(f"ERROR: {error}", file=sys.stderr)
|
|
366
|
+
return 1
|
|
367
|
+
fields, _ = parse_frontmatter((args.root / "SKILL.md").read_text(encoding="utf-8"))
|
|
368
|
+
version = (args.root / "VERSION").read_text(encoding="utf-8").strip()
|
|
369
|
+
print(f"OK: {fields['name']} ({version})")
|
|
370
|
+
print(f"OK: {len(REQUIRED_REFERENCES)} required references and {len(REQUIRED_TEMPLATES)} templates")
|
|
371
|
+
print("OK: frontmatter, symlink boundaries, local links, package entries, and semantic version")
|
|
372
|
+
return 0
|
|
373
|
+
|
|
374
|
+
|
|
375
|
+
if __name__ == "__main__":
|
|
376
|
+
raise SystemExit(main())
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# ADR-<number>: <decision title>
|
|
2
|
+
|
|
3
|
+
- Status: Proposed / Accepted / Superseded / Deprecated
|
|
4
|
+
- Date: YYYY-MM-DD
|
|
5
|
+
- Owners: <team/person>
|
|
6
|
+
|
|
7
|
+
## Context
|
|
8
|
+
<Problem, constraints, workload, relevant invariants.>
|
|
9
|
+
|
|
10
|
+
## Decision
|
|
11
|
+
<Chosen approach and scope.>
|
|
12
|
+
|
|
13
|
+
## Alternatives considered
|
|
14
|
+
| Alternative | Benefits | Costs / risks | Why not chosen |
|
|
15
|
+
|---|---|---|---|
|
|
16
|
+
| | | | |
|
|
17
|
+
|
|
18
|
+
## Consequences
|
|
19
|
+
### Positive
|
|
20
|
+
-
|
|
21
|
+
|
|
22
|
+
### Negative / risks
|
|
23
|
+
-
|
|
24
|
+
|
|
25
|
+
## Operational impact
|
|
26
|
+
<Observability, deployment, failure handling, ownership, cost.>
|
|
27
|
+
|
|
28
|
+
## Migration / rollback
|
|
29
|
+
<How to adopt, validate, and back out safely.>
|
|
30
|
+
|
|
31
|
+
## Revisit triggers
|
|
32
|
+
<Measured conditions that should cause this decision to be reconsidered.>
|
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
# Agent Evaluation Plan: <system / workflow>
|
|
2
|
+
|
|
3
|
+
## Evaluation objective
|
|
4
|
+
- Task classes and intended users:
|
|
5
|
+
- Business success criteria:
|
|
6
|
+
- Prohibited actions / safety invariants:
|
|
7
|
+
- Risk tiers and release scope:
|
|
8
|
+
|
|
9
|
+
## Versioned test assets
|
|
10
|
+
| Asset | Version / source | Owner | Change trigger |
|
|
11
|
+
|---|---|---|---|
|
|
12
|
+
| Model / provider | | | |
|
|
13
|
+
| Prompt / context policy | | | |
|
|
14
|
+
| Tool schemas / MCP servers | | | |
|
|
15
|
+
| Workflow / policy | | | |
|
|
16
|
+
| Dataset / graders | | | |
|
|
17
|
+
|
|
18
|
+
## Dataset coverage
|
|
19
|
+
| Scenario | Expected behavior | Risk class | Deterministic assertion / rubric |
|
|
20
|
+
|---|---|---|---|
|
|
21
|
+
| Normal task | | | |
|
|
22
|
+
| Ambiguous request | | | |
|
|
23
|
+
| Prompt injection | | | |
|
|
24
|
+
| Authorization denial | | | |
|
|
25
|
+
| Invalid / unavailable tool | | | |
|
|
26
|
+
| Duplicate / timeout / partial side effect | | | |
|
|
27
|
+
| Resume / cancellation | | | |
|
|
28
|
+
| Multi-agent disagreement | | | |
|
|
29
|
+
|
|
30
|
+
## Metrics and release gates
|
|
31
|
+
| Dimension | Metric / definition | Baseline | Threshold | Sample size / confidence |
|
|
32
|
+
|---|---|---|---|---|
|
|
33
|
+
| Task completion / correctness | | | | |
|
|
34
|
+
| Groundedness / provenance | | | | |
|
|
35
|
+
| Tool selection / argument validity | | | | |
|
|
36
|
+
| Policy compliance / unauthorized action | | | | |
|
|
37
|
+
| Recovery / duplicate side effects | | | | |
|
|
38
|
+
| Latency / tokens / cost | | | | |
|
|
39
|
+
| Human escalation / override | | | | |
|
|
40
|
+
|
|
41
|
+
Safety-critical failures are independent blockers; do not average them into a composite score.
|
|
42
|
+
|
|
43
|
+
## Test methods
|
|
44
|
+
- Unit / policy / schema tests:
|
|
45
|
+
- Tool and provider contract tests:
|
|
46
|
+
- End-to-end scenario suite:
|
|
47
|
+
- Adversarial / red-team suite:
|
|
48
|
+
- Fault injection / recovery tests:
|
|
49
|
+
- Human review and grader calibration:
|
|
50
|
+
- Staged online / canary evaluation:
|
|
51
|
+
|
|
52
|
+
## Observability and response
|
|
53
|
+
- Correlation IDs and trace fields:
|
|
54
|
+
- Redaction, access, and retention:
|
|
55
|
+
- Alerts and SLOs:
|
|
56
|
+
- Rollback thresholds:
|
|
57
|
+
- Incident runbooks and owners:
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# Agent Security Review: <system / agent / tool>
|
|
2
|
+
|
|
3
|
+
## Scope and trust boundaries
|
|
4
|
+
- Host / harness / agent:
|
|
5
|
+
- Tools, MCP servers, providers:
|
|
6
|
+
- Tenants, data classes, and external destinations:
|
|
7
|
+
- Reviewers and date:
|
|
8
|
+
|
|
9
|
+
## Threat register
|
|
10
|
+
| Threat / abuse case | Entry point / precondition | Impact | Existing control | Evidence / test | Gap / remediation |
|
|
11
|
+
|---|---|---|---|---|---|
|
|
12
|
+
| Prompt injection / indirect instruction | | | | | |
|
|
13
|
+
| Confused deputy / privilege escalation | | | | | |
|
|
14
|
+
| Cross-tenant access / data exfiltration | | | | | |
|
|
15
|
+
| SSRF / path traversal / unsafe execution | | | | | |
|
|
16
|
+
| Secret leakage / unsafe telemetry | | | | | |
|
|
17
|
+
| Tool or dependency compromise | | | | | |
|
|
18
|
+
| Resource exhaustion / runaway loops | | | | | |
|
|
19
|
+
|
|
20
|
+
## Identity and permissions
|
|
21
|
+
- User / tenant / agent / tool identities:
|
|
22
|
+
- Per-action authorization enforcement:
|
|
23
|
+
- Credential scope, audience, lifetime, and revocation:
|
|
24
|
+
- Read / draft / commit separation:
|
|
25
|
+
- Approval-bound action parameters and expiry:
|
|
26
|
+
|
|
27
|
+
## Isolation and data protection
|
|
28
|
+
- Worker/sandbox boundary:
|
|
29
|
+
- Filesystem, process, CPU, memory, and network egress:
|
|
30
|
+
- Context minimization and retrieval authorization:
|
|
31
|
+
- Encryption, retention, deletion, and redaction:
|
|
32
|
+
- Audit integrity and access:
|
|
33
|
+
|
|
34
|
+
## Findings
|
|
35
|
+
### [Critical / High / Medium / Low] <finding>
|
|
36
|
+
- Status: Confirmed / Likely / Unknown
|
|
37
|
+
- Evidence:
|
|
38
|
+
- Impact and trigger:
|
|
39
|
+
- Recommendation:
|
|
40
|
+
- Validation / owner:
|
|
41
|
+
|
|
42
|
+
## Release decision
|
|
43
|
+
- Blocking findings:
|
|
44
|
+
- Residual risks accepted by:
|
|
45
|
+
- Required tests:
|
|
46
|
+
- Kill switch / rollback / credential revocation:
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
# AI Agent System Design: <system / workflow>
|
|
2
|
+
|
|
3
|
+
## 1. Goals and constraints
|
|
4
|
+
- Business outcome / user journey:
|
|
5
|
+
- Task classes and risk tiers:
|
|
6
|
+
- Functional requirements:
|
|
7
|
+
- Quality, latency, availability, and cost objectives:
|
|
8
|
+
- Data classification / tenant boundaries:
|
|
9
|
+
- Non-goals and hard constraints:
|
|
10
|
+
|
|
11
|
+
## 2. Baseline and architecture choice
|
|
12
|
+
- Deterministic or single-agent baseline:
|
|
13
|
+
- Why multiple agents are/are not justified:
|
|
14
|
+
- Chosen topology: Sequential / Router / Supervisor-worker / Parallel fan-out / Peer / Hierarchical
|
|
15
|
+
- Alternatives and trade-offs:
|
|
16
|
+
|
|
17
|
+
## 3. Agent inventory and contracts
|
|
18
|
+
| Agent | Responsibility / non-goals | Inputs / outputs | Tools / data scopes | Budgets / stop conditions | Owner |
|
|
19
|
+
|---|---|---|---|---|---|
|
|
20
|
+
| | | | | | |
|
|
21
|
+
|
|
22
|
+
## 4. Architecture and execution flow
|
|
23
|
+
```mermaid
|
|
24
|
+
flowchart TD
|
|
25
|
+
User --> Host
|
|
26
|
+
Host --> Orchestrator
|
|
27
|
+
Orchestrator --> AgentA
|
|
28
|
+
AgentA --> ToolGateway
|
|
29
|
+
ToolGateway --> Service
|
|
30
|
+
Orchestrator --> StateStore
|
|
31
|
+
```
|
|
32
|
+
Replace with the actual design. Mark trust boundaries, synchronous/asynchronous edges, and human approval paths.
|
|
33
|
+
|
|
34
|
+
## 5. Harness and tools
|
|
35
|
+
- Model/provider routing and fallback:
|
|
36
|
+
- Context assembly and provenance:
|
|
37
|
+
- Tool registry and schema versions:
|
|
38
|
+
- Authorization and policy decision point:
|
|
39
|
+
- Sandbox, network, filesystem, and resource limits:
|
|
40
|
+
- Deadlines, cancellation, retries, and budgets:
|
|
41
|
+
|
|
42
|
+
## 6. MCP and external integrations
|
|
43
|
+
- Host/client/server inventory:
|
|
44
|
+
- Capabilities (Tools / Resources / Prompts):
|
|
45
|
+
- Transport and identity:
|
|
46
|
+
- Per-operation authorization:
|
|
47
|
+
- Version compatibility and lifecycle:
|
|
48
|
+
|
|
49
|
+
## 7. State, memory, and recovery
|
|
50
|
+
- Ephemeral context vs history vs memory vs workflow state:
|
|
51
|
+
- Source of truth and ownership:
|
|
52
|
+
- Checkpoint and resume protocol:
|
|
53
|
+
- Idempotency / reconciliation:
|
|
54
|
+
- Retention, deletion, and tenant isolation:
|
|
55
|
+
|
|
56
|
+
## 8. Security and human oversight
|
|
57
|
+
- Threat model:
|
|
58
|
+
- Least-privilege scopes:
|
|
59
|
+
- High-impact approval gates:
|
|
60
|
+
- Audit and redaction:
|
|
61
|
+
- Incident revocation / kill switch:
|
|
62
|
+
|
|
63
|
+
## 9. Evaluation and observability
|
|
64
|
+
- Versioned evaluation dataset:
|
|
65
|
+
- Task success and correctness:
|
|
66
|
+
- Safety/policy release gates:
|
|
67
|
+
- Latency, tokens, cost, and human intervention:
|
|
68
|
+
- Tracing and alerting:
|
|
69
|
+
|
|
70
|
+
## 10. Failure modes and rollout
|
|
71
|
+
- Provider/tool/worker failures:
|
|
72
|
+
- Partial completion and ambiguous side effects:
|
|
73
|
+
- Canary and rollback:
|
|
74
|
+
- Operational owner and runbooks:
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# API Architecture Design
|
|
2
|
+
|
|
3
|
+
## Context and goals
|
|
4
|
+
- Product journeys:
|
|
5
|
+
- Consumers and trust boundaries:
|
|
6
|
+
- Functional and non-functional requirements:
|
|
7
|
+
- Explicit non-goals:
|
|
8
|
+
|
|
9
|
+
## Contract and topology
|
|
10
|
+
- Protocols (REST / GraphQL / gRPC / async):
|
|
11
|
+
- Gateway, ingress, BFF, and service ownership:
|
|
12
|
+
- OpenAPI / GraphQL schema / Protobuf links:
|
|
13
|
+
- Authentication and authorization model:
|
|
14
|
+
- Versioning and deprecation policy:
|
|
15
|
+
|
|
16
|
+
## Capacity and resilience
|
|
17
|
+
- Average/peak RPS, concurrency, payload sizes:
|
|
18
|
+
- P50/P95/P99 targets and SLO:
|
|
19
|
+
- Deadlines, retry budget, idempotency:
|
|
20
|
+
- Rate limits, quotas, admission control:
|
|
21
|
+
- Failure modes and degraded behavior:
|
|
22
|
+
|
|
23
|
+
## Security and observability
|
|
24
|
+
- Threat model and tenant isolation:
|
|
25
|
+
- Input, output, and data classification:
|
|
26
|
+
- Metrics, logs, traces, audit events:
|
|
27
|
+
- Abuse detection and incident response:
|
|
28
|
+
|
|
29
|
+
## Rollout and validation
|
|
30
|
+
- Compatibility and contract tests:
|
|
31
|
+
- Canary, rollback, and migration plan:
|
|
32
|
+
- Load, fault-injection, and security tests:
|
|
33
|
+
- Owners, open risks, and decision record:
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# API Security and Rate-Limit Review
|
|
2
|
+
|
|
3
|
+
## Scope and assets
|
|
4
|
+
- APIs, routes, methods, and data classes:
|
|
5
|
+
- Principals, tenants, and trust boundaries:
|
|
6
|
+
- Critical business actions and abuse impact:
|
|
7
|
+
|
|
8
|
+
## Authorization
|
|
9
|
+
- Authentication and token validation:
|
|
10
|
+
- Object-level and function-level authorization:
|
|
11
|
+
- Tenant scoping and cache-key isolation:
|
|
12
|
+
- Service identity and secret lifecycle:
|
|
13
|
+
|
|
14
|
+
## Abuse controls
|
|
15
|
+
- Rate-limit keys and dimensions:
|
|
16
|
+
- Algorithm, burst, concurrency, and cost units:
|
|
17
|
+
- Distributed enforcement consistency:
|
|
18
|
+
- 429/Retry-After and client retry policy:
|
|
19
|
+
- Limiter outage and emergency fallback:
|
|
20
|
+
|
|
21
|
+
## Detection and response
|
|
22
|
+
- Metrics, alerts, audit, and investigation:
|
|
23
|
+
- False-positive handling and overrides:
|
|
24
|
+
- Load shedding and noisy-neighbor controls:
|
|
25
|
+
|
|
26
|
+
## Verification
|
|
27
|
+
- BOLA/BFLA, replay, enumeration, and bypass tests:
|
|
28
|
+
- Burst, hot-key, clock-skew, and dependency-failure tests:
|
|
29
|
+
- Findings, severity, owners, due dates:
|