system-design-primer-skill 0.0.0-stage → 2.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (112) hide show
  1. package/ATTRIBUTION.md +10 -0
  2. package/LICENSE +7 -0
  3. package/README.md +180 -2
  4. package/SKILL.md +447 -0
  5. package/VERSION +1 -0
  6. package/package.json +78 -4
  7. package/references/00-primer-map-and-attribution.md +29 -0
  8. package/references/01-requirements-and-capacity.md +107 -0
  9. package/references/02-architecture-and-networking.md +94 -0
  10. package/references/03-data-and-storage.md +88 -0
  11. package/references/04-caching.md +78 -0
  12. package/references/05-async-and-messaging.md +77 -0
  13. package/references/06-distributed-systems.md +59 -0
  14. package/references/07-reliability-and-resilience.md +71 -0
  15. package/references/08-api-security-observability.md +62 -0
  16. package/references/09-modern-engineering.md +58 -0
  17. package/references/10-workflows-and-checklists.md +97 -0
  18. package/references/11-case-studies.md +74 -0
  19. package/references/12-ai-systems-architecture.md +139 -0
  20. package/references/13-architecture-intelligence.md +91 -0
  21. package/references/14-api-and-data-contracts.md +80 -0
  22. package/references/15-failure-analysis.md +84 -0
  23. package/references/16-multi-agent-architecture.md +76 -0
  24. package/references/17-agent-harness-and-runtime.md +87 -0
  25. package/references/18-mcp-architecture.md +67 -0
  26. package/references/19-agent-security-and-permissions.md +74 -0
  27. package/references/20-agent-state-and-memory.md +72 -0
  28. package/references/21-agent-evaluation-and-observability.md +90 -0
  29. package/references/22-kubernetes-and-container-platforms.md +29 -0
  30. package/references/23-service-mesh-and-traffic-management.md +21 -0
  31. package/references/24-multi-region-and-disaster-recovery.md +24 -0
  32. package/references/25-platform-engineering-and-developer-experience.md +26 -0
  33. package/references/26-finops-and-cloud-cost.md +29 -0
  34. package/references/27-production-observability-and-slo.md +29 -0
  35. package/references/28-event-streaming-and-kafka-pulsar.md +27 -0
  36. package/references/29-event-sourcing-and-cqrs.md +23 -0
  37. package/references/30-change-data-capture-and-debezium.md +27 -0
  38. package/references/31-data-lakehouse-and-stream-batch.md +24 -0
  39. package/references/32-workflow-orchestration-and-temporal.md +24 -0
  40. package/references/33-schema-registry-and-event-governance.md +22 -0
  41. package/references/34-postgresql-internals-and-performance.md +43 -0
  42. package/references/35-mysql-innodb-and-performance.md +27 -0
  43. package/references/36-sharding-and-distributed-sql.md +22 -0
  44. package/references/37-consensus-and-distributed-transactions.md +25 -0
  45. package/references/38-search-architecture-elasticsearch-opensearch.md +24 -0
  46. package/references/39-vector-databases-and-hybrid-retrieval.md +31 -0
  47. package/references/40-zero-downtime-database-migrations.md +27 -0
  48. package/references/41-api-gateway-and-edge-architecture.md +23 -0
  49. package/references/42-rest-api-design-and-lifecycle.md +23 -0
  50. package/references/43-graphql-architecture-and-governance.md +16 -0
  51. package/references/44-grpc-and-service-contracts.md +16 -0
  52. package/references/45-websocket-sse-and-realtime-systems.md +16 -0
  53. package/references/46-realtime-collaboration-ot-crdt.md +16 -0
  54. package/references/47-api-security-rate-limiting-and-abuse.md +16 -0
  55. package/references/48-oauth-oidc-and-passkeys.md +20 -0
  56. package/references/49-iam-rbac-abac-rebac.md +16 -0
  57. package/references/50-zero-trust-and-workload-identity.md +16 -0
  58. package/references/51-secrets-kms-pki-and-key-management.md +16 -0
  59. package/references/52-supply-chain-security-sbom-slsa.md +16 -0
  60. package/references/53-threat-modeling-stride-and-attack-trees.md +16 -0
  61. package/references/54-privacy-engineering-and-data-governance.md +16 -0
  62. package/references/55-opentelemetry-and-instrumentation.md +50 -0
  63. package/references/56-distributed-tracing-and-context-propagation.md +43 -0
  64. package/references/57-continuous-profiling-and-performance-diagnostics.md +45 -0
  65. package/references/58-load-stress-soak-testing-and-workload-modeling.md +44 -0
  66. package/references/59-capacity-planning-and-performance-modeling.md +34 -0
  67. package/references/60-chaos-engineering-and-resilience-experiments.md +41 -0
  68. package/references/61-incident-management-and-postmortems.md +34 -0
  69. package/references/62-dns-and-service-discovery.md +35 -0
  70. package/references/63-load-balancing-and-global-traffic-management.md +31 -0
  71. package/references/64-cdn-and-edge-caching.md +34 -0
  72. package/references/65-cloud-networking-and-connectivity.md +34 -0
  73. package/references/66-edge-computing-architecture.md +33 -0
  74. package/references/67-storage-infrastructure-and-storage-systems.md +30 -0
  75. package/references/68-tcp-quic-and-network-performance.md +27 -0
  76. package/references/69-task-routing-and-evidence.md +58 -0
  77. package/scripts/install.ps1 +124 -0
  78. package/scripts/install.sh +105 -0
  79. package/scripts/validate.py +376 -0
  80. package/templates/adr-template.md +32 -0
  81. package/templates/agent-evaluation-plan-template.md +57 -0
  82. package/templates/agent-security-review-template.md +46 -0
  83. package/templates/agent-system-design-template.md +74 -0
  84. package/templates/api-architecture-design-template.md +33 -0
  85. package/templates/api-security-and-rate-limit-review-template.md +29 -0
  86. package/templates/architecture-decision-matrix-template.md +48 -0
  87. package/templates/architecture-design-template.md +95 -0
  88. package/templates/architecture-review-scorecard-template.md +36 -0
  89. package/templates/architecture-review-template.md +52 -0
  90. package/templates/cdc-data-pipeline-template.md +41 -0
  91. package/templates/cdn-and-edge-strategy-template.md +36 -0
  92. package/templates/chaos-experiment-template.md +39 -0
  93. package/templates/cloud-cost-model-template.md +37 -0
  94. package/templates/database-architecture-review-template.md +33 -0
  95. package/templates/edge-computing-readiness-template.md +36 -0
  96. package/templates/event-sourcing-cqrs-template.md +41 -0
  97. package/templates/event-streaming-design-template.md +47 -0
  98. package/templates/failure-analysis-template.md +36 -0
  99. package/templates/identity-and-access-architecture-template.md +29 -0
  100. package/templates/incident-postmortem-template.md +39 -0
  101. package/templates/multi-region-disaster-recovery-template.md +43 -0
  102. package/templates/network-architecture-design-template.md +38 -0
  103. package/templates/observability-architecture-template.md +52 -0
  104. package/templates/performance-test-plan-template.md +52 -0
  105. package/templates/privacy-data-governance-template.md +24 -0
  106. package/templates/production-readiness-checklist-template.md +52 -0
  107. package/templates/realtime-system-design-template.md +28 -0
  108. package/templates/security-threat-model-template.md +26 -0
  109. package/templates/sharding-distributed-sql-design-template.md +34 -0
  110. package/templates/storage-architecture-design-template.md +35 -0
  111. package/templates/supply-chain-security-review-template.md +20 -0
  112. package/templates/zero-downtime-database-migration-template.md +32 -0
@@ -0,0 +1,58 @@
1
+ # Modern engineering practices beyond the Primer
2
+
3
+ This reference extends the Primer's foundational patterns into day-to-day production engineering. Verify vendor-specific and version-specific details against official documentation.
4
+
5
+ ## 1. Architecture boundaries
6
+
7
+ - Prefer a modular monolith until independent deployment, ownership, scaling, security, or fault-isolation needs justify distributed services.
8
+ - Define module APIs and data ownership; avoid cross-module writes into private tables.
9
+ - Use dependency inversion at meaningful boundaries, not abstractions around every function.
10
+ - Keep domain logic separate from transport, persistence, and framework adapters where it improves testability.
11
+
12
+ ## 2. Microservice extraction criteria
13
+
14
+ A service extraction is justified when one or more are demonstrably important:
15
+
16
+ - independent scaling profile;
17
+ - different reliability/security boundary;
18
+ - different deployment cadence or team ownership;
19
+ - domain boundary is stable and data ownership can be isolated;
20
+ - blast-radius reduction is worth operational overhead.
21
+
22
+ Before extraction, define API contracts, data migration, event ownership, consistency, observability, deployment, on-call ownership, and rollback. Avoid distributed monoliths with synchronous call chains and shared database writes.
23
+
24
+ ## 3. Cloud and container deployment
25
+
26
+ Containers package process/runtime dependencies; they do not automatically provide resilience. Orchestration adds scheduling and rollout capabilities but also control-plane and configuration complexity. For each service define:
27
+
28
+ - CPU/memory requests and limits or equivalent resource reservations;
29
+ - readiness/liveness/startup checks;
30
+ - graceful shutdown and connection draining;
31
+ - replica/zone placement and disruption budget where relevant;
32
+ - autoscaling signals (often queue lag or concurrency in addition to CPU);
33
+ - configuration and secret separation;
34
+ - rolling/blue-green/canary strategy and rollback criteria.
35
+
36
+ Serverless can reduce operational work for bursty/event-driven workloads, but review cold starts, execution limits, concurrency quotas, vendor coupling, local testing, and cost under sustained load.
37
+
38
+ ## 4. Progressive delivery
39
+
40
+ Use feature flags, canary, blue/green, or phased tenant rollout when risk warrants. Define success and abort thresholds before rollout. Feature flags need owners, expiry/review dates, and safe defaults; stale flags become complexity.
41
+
42
+ ## 5. Database and event contract evolution
43
+
44
+ Treat API schemas, event schemas, database schema, and cached representations as versioned contracts. Plan forward/backward compatibility and the window in which old and new versions coexist. Use expand-migrate-contract for schema evolution and compatibility tests in CI.
45
+
46
+ ## 6. AI application architecture
47
+
48
+ For LLM/agent features, system design also includes model/provider routing, token and GPU capacity, prompt/tool versioning, prompt-injection boundaries, least-privilege tool authorization, output validation, retrieval freshness and tenant isolation, evaluation, and safe cancellation/recovery. Treat model output as untrusted data. A prompt or generated plan must not bypass application authorization, validation, or transaction controls.
49
+
50
+ Use `references/12-ai-systems-architecture.md` for detailed workload modeling, inference/GPU planning, model gateways, RAG pipelines, agent tool boundaries, evaluation, cost and failure handling.
51
+
52
+ ## 7. Operational ownership
53
+
54
+ Every production component needs an owner, runbook, alert routing, capacity plan, backup/recovery plan if stateful, patch/upgrade policy, and decommission path. A diagram without ownership and operations is incomplete architecture.
55
+
56
+ ## 8. Architecture decision records (ADR)
57
+
58
+ Record decisions that are expensive to reverse. An ADR should contain context, decision, alternatives, consequences, risks, and status. Do not write an ADR for every small implementation detail.
@@ -0,0 +1,97 @@
1
+ # Workflows, design review checklists, and decision process
2
+
3
+ ## 1. Design review sequence
4
+
5
+ 1. Problem statement and users.
6
+ 2. Functional requirements and critical journeys.
7
+ 3. NFRs with measurable targets.
8
+ 4. Workload/capacity model.
9
+ 5. Data model and ownership.
10
+ 6. High-level architecture and trust boundaries.
11
+ 7. Critical request and event flows.
12
+ 8. Consistency/concurrency invariants.
13
+ 9. Failure modes and recovery.
14
+ 10. Security/privacy/compliance.
15
+ 11. Observability and operations.
16
+ 12. Cost, alternatives, migration and rollback.
17
+
18
+ ## 2. Architecture review checklist
19
+
20
+ ### Product and requirements
21
+ - Is the problem scoped and tied to measurable outcomes?
22
+ - Are assumptions, unknowns, and exclusions explicit?
23
+ - Is the critical path identified?
24
+
25
+ ### Architecture
26
+ - Are module/service responsibilities and ownership clear?
27
+ - Are synchronous and asynchronous boundaries intentional?
28
+ - Is the design simpler than necessary, or unnecessarily complex?
29
+ - Are single points of failure and common-mode failures understood?
30
+
31
+ ### Data
32
+ - Is source of truth explicit?
33
+ - Are transaction boundaries and invariants enforced?
34
+ - Are indexes and queries aligned to access patterns?
35
+ - Are replication lag, duplicate events, and migration paths addressed?
36
+
37
+ ### Performance and capacity
38
+ - Are peak and burst loads modeled?
39
+ - Are tail latencies and downstream fan-out measured?
40
+ - Are connection pools, queues, caches and hot partitions bounded?
41
+
42
+ ### Reliability
43
+ - Are timeouts/deadlines, retries, idempotency and overload behavior specified?
44
+ - Can the system degrade safely?
45
+ - Are backup restore and failover tested?
46
+
47
+ ### Security
48
+ - Are authentication, authorization, tenant isolation and secrets handled correctly?
49
+ - Are external inputs, webhooks, files and URLs validated?
50
+ - Are logs and analytics free from unnecessary sensitive data?
51
+
52
+ ### Operations
53
+ - Are metrics, logs, traces, alerts and runbooks present?
54
+ - Are rollout, rollback, ownership, cost, and decommissioning understood?
55
+
56
+ ## 3. Finding severity
57
+
58
+ - **Critical**: likely data breach, data corruption/loss, unsafe financial/state transition, or broad outage with immediate exposure.
59
+ - **High**: likely major degradation/outage or important invariant violation under plausible conditions.
60
+ - **Medium**: localized reliability, performance, security-hardening, or maintainability issue with a workaround.
61
+ - **Low**: improvement opportunity or low-impact maintainability concern.
62
+
63
+ Severity must be evidence-based. Include likelihood and impact separately when useful. Do not inflate severity for style preferences.
64
+
65
+ ## 4. Decision matrix
66
+
67
+ Use the structured architecture decision process in `13-architecture-intelligence.md`. Separate hard constraints from weighted preferences; compare the simplest viable baseline with credible alternatives; assess performance, consistency, reliability, security, TCO, operational complexity, team capability, migration/exit cost, and ecosystem risk. Attach evidence and confidence to each assessment, test sensitivity to uncertain assumptions, and record revisit triggers in an ADR. Never let an aggregate score compensate for a violated hard constraint.
68
+
69
+ ## 5. Architecture debt and evolution triggers
70
+
71
+ Write down observable triggers for a future change, e.g.:
72
+
73
+ - DB CPU or I/O above target for sustained periods despite query/index tuning;
74
+ - replica lag violates read freshness SLO;
75
+ - queue oldest-message age exceeds business deadline;
76
+ - p99 latency budget consumed by a single dependency;
77
+ - a module requires independent deploy/scale/security ownership;
78
+ - a tenant causes resource starvation;
79
+ - recovery time exceeds RTO.
80
+
81
+ Avoid arbitrary user counts as the sole migration trigger. Workload shape and bottleneck evidence matter more.
82
+
83
+ ## 6. Implementation quality gate
84
+
85
+ Before declaring architecture work done:
86
+
87
+ - tests cover business invariants, concurrency, idempotency, retries and failure paths;
88
+ - migrations are tested and reversible or have an explicit forward-recovery plan;
89
+ - load tests use representative data and realistic concurrency;
90
+ - telemetry can confirm success and detect regressions;
91
+ - threat model and authorization checks are reviewed;
92
+ - deployment and rollback are documented;
93
+ - operational owner and runbook exist.
94
+
95
+ ## 7. System design interview structure
96
+
97
+ Use: clarify requirements → state assumptions → estimate scale → API/data model → high-level diagram → deep dive into bottlenecks → consistency/reliability/security → alternatives/trade-offs. Spend time according to the interviewer and problem scope; do not attempt to name every technology.
@@ -0,0 +1,74 @@
1
+ # Case studies: apply patterns, do not memorize diagrams
2
+
3
+ These are starting points for reasoning, not universal reference architectures.
4
+
5
+ ## 1. URL shortener
6
+
7
+ Core paths: create mapping, redirect lookup, optional analytics, expiration/abuse control.
8
+
9
+ - Data: unique short code → canonical URL + metadata. Enforce uniqueness in the database, not only in application code.
10
+ - Reads often dominate writes; a cache may help after measuring hot-key distribution.
11
+ - Choose redirect status based on permanence, analytics, and client caching semantics.
12
+ - Generate codes with collision handling and rate/abuse protection. Base62 is an encoding alphabet, not a collision-proof generator.
13
+ - Analytics should not add unbounded latency to redirects; consider asynchronous events with dedupe.
14
+ - Consider malicious URL reporting, phishing, private links, expiry, and tenant ownership.
15
+
16
+ ## 2. Social feed / timeline
17
+
18
+ Core paths: publish post, follow/unfollow, read feed, rank/filter items.
19
+
20
+ - Fan-out-on-write lowers read work but makes celebrity/high-fanout writes expensive.
21
+ - Fan-out-on-read reduces write amplification but increases read-time work and dependency fan-out.
22
+ - A hybrid approach may treat high-follower authors differently.
23
+ - Define ordering, pagination cursor, deletion propagation, privacy checks, and feed freshness.
24
+ - Materialized feeds are derived data; define rebuild/reconciliation strategy.
25
+
26
+ ## 3. Chat / messaging
27
+
28
+ Core paths: send, persist, deliver online, queue offline, acknowledge, sync devices.
29
+
30
+ - WebSocket or another persistent transport can support server push, but connection lifecycle and routing need ownership.
31
+ - Persist before acknowledging durable acceptance if offline delivery matters.
32
+ - Define per-conversation ordering, message IDs, dedupe, delivery/read receipts, retention, attachments, and end-to-end encryption requirements.
33
+ - Online presence is approximate and must tolerate disconnect races.
34
+ - Avoid claiming “delivered” when only accepted by an intermediary.
35
+
36
+ ## 4. Image/video upload
37
+
38
+ Typical flow: client obtains scoped upload authorization → uploads directly to object storage → metadata/status persisted → async validation/transcoding/thumbnails → CDN delivery.
39
+
40
+ - Separate blob storage from relational metadata.
41
+ - Use short-lived scoped upload credentials, content/size limits, checksum, malware scanning when appropriate, and safe content serving.
42
+ - Make processing jobs idempotent and observable.
43
+ - Define private/public access, lifecycle/retention, deletion, and CDN invalidation.
44
+
45
+ ## 5. Web crawler
46
+
47
+ - Frontier queue schedules URLs with per-host politeness and rate limits.
48
+ - Normalize URLs and deduplicate with canonicalization plus durable seen state.
49
+ - Fetch workers need deadlines, robots policy, SSRF protections, retry ceilings, and content size limits.
50
+ - Parse/index pipeline should be asynchronous and replayable.
51
+ - Track frontier age, fetch success, duplicate rate, per-host throughput, and storage growth.
52
+
53
+ ## 6. Rate limiter
54
+
55
+ - Token bucket supports bursts with average rate control; leaky bucket smooths output; fixed/sliding windows have different boundary behavior.
56
+ - Decide enforcement point (edge, gateway, service, shared store) and key dimension (user, tenant, API key, IP, route).
57
+ - Distributed counters add consistency and network costs; local limits can be approximate but protect each instance.
58
+ - Define fail-open/fail-closed behavior and return retry guidance.
59
+
60
+ ## 7. Search / autocomplete
61
+
62
+ - Separate source-of-truth writes from derived search index updates.
63
+ - Use async indexing with freshness SLO, replay/rebuild, delete propagation, and backfill.
64
+ - Apply tenant and authorization filters before returning results; indexes must not leak private records.
65
+ - Optimize prefix/query workload and ranking based on real usage; cache only where key cardinality and freshness allow.
66
+
67
+ ## 8. E-commerce checkout / payment
68
+
69
+ - Define order state machine and inventory reservation semantics.
70
+ - Make checkout/payment requests idempotent; persist idempotency key and result.
71
+ - Treat provider timeouts as ambiguous; query/reconcile provider state before issuing a new charge.
72
+ - Use outbox for durable event publication and reconciliation jobs for stuck states.
73
+ - Distinguish authorization, capture, refund, cancellation, and settlement.
74
+ - Do not use eventual consistency where it violates money/inventory invariants.
@@ -0,0 +1,139 @@
1
+ # AI system architecture: LLMs, RAG, agents, and inference platforms
2
+
3
+ This reference applies system-design methods to AI-backed products. Treat model behavior as probabilistic, model outputs as untrusted input, and vendor-specific limits as values to verify against current official documentation.
4
+
5
+ ## 1. Workload and service objectives
6
+
7
+ Model the complete user journey rather than estimating only model calls.
8
+
9
+ | Dimension | Questions |
10
+ |---|---|
11
+ | Demand | requests/sec, concurrent sessions, burst shape, batch vs interactive |
12
+ | Context | input tokens, retrieved tokens, system prompt size, history growth |
13
+ | Output | output token distribution, max generation length, streaming behavior |
14
+ | Latency | time to first token, inter-token latency, total completion latency |
15
+ | Quality | task success, groundedness, refusal quality, tool success, human override |
16
+ | Cost | input/output tokens, embeddings, reranking, storage, GPU time, tool calls |
17
+ | Limits | per-model quota, tenant budget, context window, concurrency and regional capacity |
18
+
19
+ Separate time to first token (TTFT), time per output token, and end-to-end completion time. Streaming improves perceived responsiveness but does not reduce total generation work.
20
+
21
+ Estimate token consumption from measured distributions, not character-count rules of thumb. Model long-context tails and retry amplification. Apply tenant and user budgets to both synchronous calls and background workflows.
22
+
23
+ ## 2. Model gateway and routing
24
+
25
+ A model gateway can centralize provider credentials, policy, quotas, telemetry, retries, and routing. Keep routing policy explicit and testable.
26
+
27
+ - Route by task capability, quality target, context size, latency, region, and cost ceiling.
28
+ - Use fallbacks only when the alternate model preserves required behavior and data-handling constraints.
29
+ - Treat provider timeouts as ambiguous: a request may have been processed and billed even when the response was lost.
30
+ - Use bounded retries, request IDs, cancellation, and idempotency for side-effecting tool workflows.
31
+ - Preserve model, prompt, routing-policy, and tool versions in traces for reproducibility.
32
+ - Do not silently send regulated or tenant-restricted data to a different provider or region.
33
+
34
+ A fallback is not automatically equivalent. Define quality degradation, structured-output compatibility, refusal behavior, and when to fail closed.
35
+
36
+ ## 3. Inference capacity and GPU planning
37
+
38
+ For self-hosted inference, distinguish model weights, KV cache, runtime workspace, activations, batching buffers, and fragmentation.
39
+
40
+ A first-order weight-memory estimate is parameter count multiplied by bytes per parameter. This excludes runtime memory and is not a capacity guarantee. Quantization changes weight size and may change quality or kernel efficiency. KV cache depends on layers, attention heads/KV heads, head dimension, token count, batch concurrency, and cache precision; use the selected runtime's documented memory model.
41
+
42
+ Capacity planning must consider:
43
+ - model size, precision/quantization, tensor/pipeline parallelism;
44
+ - prompt and output length distributions;
45
+ - concurrency and continuous/dynamic batching;
46
+ - prefill vs decode throughput;
47
+ - GPU memory headroom and fragmentation;
48
+ - queue wait, admission control, cancellation, and autoscaling delay;
49
+ - cold start, model loading, health checks, and failover capacity.
50
+
51
+ Benchmark with representative prompts and concurrency. Record tokens/sec, TTFT, inter-token latency, GPU utilization, memory, queue age, error rate, and quality. Never infer production capacity from a single throughput benchmark.
52
+
53
+ ## 4. Prompt, context, and caching
54
+
55
+ Treat prompts as versioned application contracts. Record system prompt, template, model, decoding settings, retrieval policy, and tool schema versions.
56
+
57
+ - Bound conversation history; summarize or retrieve older context with provenance.
58
+ - Reserve context budget for system instructions, tool schemas, retrieved evidence, and output.
59
+ - Cache only when the full cache key includes model, prompt version, parameters, tenant/auth scope, and normalized inputs.
60
+ - Understand provider prompt-cache behavior, eligibility, retention, and pricing before relying on it.
61
+ - Do not cache sensitive cross-tenant output under shared keys.
62
+
63
+ A cache hit is not proof of correctness. Re-check authorization and data freshness at the appropriate boundary.
64
+
65
+ ## 5. Retrieval-augmented generation
66
+
67
+ Design ingestion and query paths separately.
68
+
69
+ Ingestion path:
70
+ 1. collect authorized source content and metadata;
71
+ 2. normalize, classify, and remove or quarantine unsafe content;
72
+ 3. chunk with source and access-control metadata;
73
+ 4. generate embeddings with a versioned model;
74
+ 5. write vectors and metadata with a durable source-of-truth reference;
75
+ 6. monitor indexing lag, failures, duplicates, and deletion propagation.
76
+
77
+ Query path:
78
+ 1. authenticate and establish tenant/resource scope;
79
+ 2. rewrite or expand the query only within policy;
80
+ 3. retrieve candidates using vector, lexical, or hybrid search;
81
+ 4. apply authorization filters before evidence reaches the model;
82
+ 5. rerank and trim to a measured context budget;
83
+ 6. generate with citations/provenance where required;
84
+ 7. validate output and measure groundedness or task success.
85
+
86
+ Define freshness SLO, chunk IDs, embedding version, metadata schema, deduplication, backfill, replay, deletion, and full-index rebuild. An index is derived data; it must be possible to reconstruct it from authorized source records.
87
+
88
+ ## 6. Agent and tool architecture
89
+
90
+ Treat an agent as a workflow engine with probabilistic planning, not as an authorization boundary.
91
+
92
+ - Give each tool a narrow schema, explicit permission, timeout, and resource budget.
93
+ - Authorize every tool call in application code against the current user, tenant, and object.
94
+ - Validate tool arguments and outputs; do not trust model-generated paths, URLs, SQL, shell commands, or identity claims.
95
+ - Make side effects idempotent and auditable; use approval gates for irreversible, financial, privacy-sensitive, or externally visible actions.
96
+ - Bound steps, tokens, wall-clock time, concurrency, recursion, and tool retries.
97
+ - Persist workflow state and checkpoint long-running jobs; support cancellation, resume, and human handoff.
98
+ - Treat retrieved documents, web pages, files, and tool results as untrusted data that may contain prompt injection.
99
+
100
+ Do not place unrestricted credentials in model context. Use brokered tools and least-privilege service identities.
101
+
102
+ ## 7. Security, privacy, and governance
103
+
104
+ Threat-model prompt injection, data exfiltration, cross-tenant retrieval, indirect instruction attacks, unsafe tool invocation, model/provider retention, and sensitive output leakage.
105
+
106
+ Enforce authorization before retrieval and again before side effects. Apply data minimization, regional controls, retention/deletion, encryption, redaction, audit trails, and provider policy review. A model refusal is not a substitute for application authorization or safety validation.
107
+
108
+ ## 8. Evaluation and observability
109
+
110
+ Build a versioned evaluation set from representative and adversarial cases. Include correctness, groundedness, citation accuracy, tool selection, argument validity, policy compliance, latency, cost, and robustness. Use offline regression tests plus staged online evaluation; do not rely on subjective spot checks alone.
111
+
112
+ Trace the complete chain:
113
+ - request and tenant-safe correlation identifiers;
114
+ - model/provider, model version, prompt/template version;
115
+ - token counts, latency phases, retries, and estimated cost;
116
+ - retrieval queries, document IDs, filter outcomes, and index version;
117
+ - tool names, validated arguments, authorization decisions, and outcomes;
118
+ - quality signals, user feedback, escalation, and human override.
119
+
120
+ Avoid logging raw prompts, secrets, or sensitive retrieved content by default. Define redaction and retention before enabling traces.
121
+
122
+ ## 9. Failure modes and degradation
123
+
124
+ Plan for provider outage, quota exhaustion, model retirement, slow generation, malformed structured output, retrieval lag, vector-store outage, tool timeout, partial side effects, and evaluation regression.
125
+
126
+ Define per-task fallback, queueing, cancellation, graceful degradation, circuit breaking, dead-letter and replay behavior. For consequential workflows, expose explicit pending/unknown states and reconcile ambiguous provider or tool outcomes rather than repeating a side effect blindly.
127
+
128
+ ## 10. Delivery checklist
129
+
130
+ - [ ] Workload, token distributions, latency phases, quality and cost objectives are measurable.
131
+ - [ ] Model/provider routing and fallback policy are explicit.
132
+ - [ ] Capacity accounts for context, concurrency, KV cache, batching and headroom.
133
+ - [ ] Prompts, model choices, tools, retrieval and schemas are versioned.
134
+ - [ ] RAG authorization is applied before evidence enters model context.
135
+ - [ ] Agent tools are narrow, authenticated, bounded, idempotent and auditable.
136
+ - [ ] Prompt injection and untrusted retrieval content are tested.
137
+ - [ ] Evaluation datasets and regression thresholds exist.
138
+ - [ ] Telemetry is tenant-safe and sensitive data is redacted.
139
+ - [ ] Provider, quota, retrieval, and tool failures have safe degraded modes.
@@ -0,0 +1,91 @@
1
+ # Architecture intelligence: decisions and evidence-based reviews
2
+
3
+ Use this reference when selecting an architecture, comparing technology options, or conducting a structured architecture review. The goal is a defensible decision under stated constraints—not a universal ranking of patterns or vendors.
4
+
5
+ ## 1. Decision process
6
+
7
+ 1. Frame the decision and the business outcome it enables.
8
+ 2. Record hard constraints separately from preferences: compliance, data residency, latency ceiling, budget ceiling, team skills, existing contracts, and delivery deadline.
9
+ 3. Define workload shape and critical paths using measured or explicitly assumed inputs.
10
+ 4. Identify viable options, including the simplest baseline and the option to defer the decision.
11
+ 5. Eliminate options that violate hard constraints.
12
+ 6. Compare remaining options against criteria that matter to this workload.
13
+ 7. Run sensitivity analysis: identify which assumptions could reverse the ranking.
14
+ 8. Record the decision, rejected alternatives, risks, validation plan, and revisit triggers in an ADR.
15
+
16
+ Do not start with a preferred technology and retrofit requirements to justify it.
17
+
18
+ ## 2. Architecture decision matrix
19
+
20
+ Use a qualitative rating (Strong / Adequate / Weak / Unknown) or a numeric scale only when the team defines what each score means. Never imply false precision. Suggested dimensions:
21
+
22
+ | Dimension | Questions and evidence |
23
+ |---|---|
24
+ | Performance and scalability | Does it meet measured peak throughput, tail latency, concurrency, and growth needs? What is the bottleneck and scaling unit? |
25
+ | Consistency and data integrity | Can it enforce business invariants, transaction boundaries, ordering, idempotency, and recovery semantics? |
26
+ | Reliability and availability | What are failure domains, dependency assumptions, degraded modes, RPO/RTO, and operational SLOs? |
27
+ | Security and compliance | Does it support required identity, authorization, isolation, encryption, audit, residency, and retention? |
28
+ | Total cost of ownership | Include infrastructure, licenses, egress, observability, support, on-call, migration, and engineering time. |
29
+ | Operational complexity | What must be deployed, upgraded, monitored, backed up, restored, and debugged? Who owns it? |
30
+ | Team capability | Does the team have the skills and capacity to operate it safely? What training or hiring is required? |
31
+ | Migration and exit cost | How difficult is adoption, data movement, rollback, portability, and eventual replacement? |
32
+ | Ecosystem and delivery risk | Are integrations, support lifecycle, maturity, and vendor dependencies acceptable? |
33
+
34
+ ### Weighting rules
35
+
36
+ - Use weights only when stakeholders provide priorities or the architect clearly labels them as provisional assumptions.
37
+ - Record the reason for each weight and test whether small changes alter the preferred option.
38
+ - A hard constraint is a gate, not a weighted preference. A compliance failure cannot be compensated for by a high performance score.
39
+ - Unknown is not zero and must not silently become a favorable score. Identify the experiment or evidence needed.
40
+ - Include confidence and evidence references beside each rating.
41
+ - Prefer a short written rationale over an aggregate score when options are close or evidence is weak.
42
+
43
+ ### Decision output
44
+
45
+ State: recommended option; viable alternatives; constraints satisfied; material trade-offs; assumptions; evidence gaps; implementation and migration plan; rollback; and measurable revisit triggers.
46
+
47
+ ## 3. Architecture review scorecard
48
+
49
+ Assess each dimension as **Pass**, **Needs work**, **Fail**, or **Not assessed**. Attach evidence and business impact. A score without evidence is not a finding.
50
+
51
+ | Dimension | Review questions |
52
+ |---|---|
53
+ | Architecture and boundaries | Are responsibilities, dependencies, ownership, and sync/async boundaries explicit? Is complexity justified? |
54
+ | Data integrity | Are source of truth, invariants, transactions, concurrency, idempotency, and schema evolution correct? |
55
+ | Scalability and performance | Are workload assumptions credible? Are hotspots, tail latency, capacity limits, and resource bounds measured? |
56
+ | Reliability and recovery | Are failure domains, timeouts, retries, overload, degradation, backups, restore, RPO, and RTO addressed? |
57
+ | Security and privacy | Are identity, authorization, tenant isolation, secrets, data minimization, and threat boundaries enforced? |
58
+ | Observability and operations | Can operators detect, diagnose, mitigate, and learn from failures? Are ownership and runbooks clear? |
59
+ | Maintainability and delivery | Are contracts, tests, deployment, rollback, dependency lifecycle, and team ownership sustainable? |
60
+ | Cost efficiency | Are cost drivers visible and bounded? Are cost/performance trade-offs supported by workload evidence? |
61
+
62
+ ### Finding quality
63
+
64
+ Every material finding includes:
65
+ - Status: Confirmed, Likely, or Unknown.
66
+ - Evidence: file path and symbol/configuration, runtime metric, test, trace, or reproducible observation.
67
+ - Impact: user/business consequence and affected boundary.
68
+ - Trigger or conditions: when the issue manifests.
69
+ - Severity: Critical, High, Medium, or Low, based on impact and likelihood.
70
+ - Recommendation: a proportionate fix and alternatives.
71
+ - Validation: test, measurement, or operational signal that proves the fix.
72
+
73
+ Do not average scorecard dimensions into a single health percentage. A critical security or data-integrity failure must remain visible and cannot be diluted by strong scores elsewhere.
74
+
75
+ ## 4. Trade-off and sensitivity analysis
76
+
77
+ For each leading option, ask:
78
+ - Which workload assumption has the greatest effect on the decision?
79
+ - What happens at 2x or 10x traffic, during a burst, or after one failure domain is lost?
80
+ - What happens if the team cannot hire or train the required operator?
81
+ - What are the recurring and one-time costs, including migration and exit?
82
+ - Which requirement could make the option unacceptable?
83
+ - What is the cheapest experiment that would reduce uncertainty?
84
+
85
+ Use ranges for uncertain costs and capacity. Do not invent vendor benchmarks. Compare alternatives under the same workload and operational assumptions.
86
+
87
+ ## 5. Evolution triggers
88
+
89
+ Define observable thresholds rather than arbitrary user-count milestones. Examples include sustained p99 budget exhaustion, database saturation after query tuning, replica lag violating freshness objectives, queue age exceeding business deadlines, recovery tests missing RTO, or a stable team/domain boundary requiring independent deployment.
90
+
91
+ Revisit the decision when a trigger is met, not merely because a newer technology exists.
@@ -0,0 +1,80 @@
1
+ # API and data contract design
2
+
3
+ Treat public APIs, internal service APIs, events, database schemas, and cached representations as contracts with consumers and compatibility windows. Design them around business capabilities and data ownership rather than exposing persistence internals.
4
+
5
+ ## 1. Contract definition
6
+
7
+ For each API operation or event, document:
8
+ - Purpose, owner, consumers, and trust boundary.
9
+ - Method/topic, resource or command semantics, and stable identifiers.
10
+ - Request/response or event schema, required/optional fields, validation, and size limits.
11
+ - Authentication, authorization, tenant scope, and data classification.
12
+ - Error taxonomy, retryability, timeout/deadline, and rate limits.
13
+ - Idempotency, ordering, concurrency, and duplicate handling.
14
+ - Pagination/filtering, consistency and freshness expectations.
15
+ - Versioning, deprecation policy, compatibility window, and telemetry.
16
+
17
+ Prefer explicit business commands for state transitions. Do not expose internal database table layouts as the external API contract.
18
+
19
+ ## 2. HTTP and RPC semantics
20
+
21
+ Use HTTP method semantics consistently, but do not confuse protocol semantics with business idempotency. A payment-creation POST, for example, needs a persisted idempotency key and operation result. Reusing a key with a different normalized payload should be rejected or follow a documented conflict rule.
22
+
23
+ Use stable error codes and structured error bodies. Distinguish validation failures, authorization failures, conflicts, rate limits, transient dependency failures, and unknown outcomes. Avoid leaking stack traces or sensitive internals.
24
+
25
+ For pagination, prefer opaque cursor pagination for large, changing datasets where offset drift is problematic. Bind cursors to the tenant, authorization scope, filter, sort order, and snapshot semantics; integrity-protect them. Set maximum page sizes.
26
+
27
+ ## 3. Versioning and compatibility
28
+
29
+ Version only when needed by a clear compatibility policy. Common approaches include URI, header, or media-type versioning; choose one deliberately and document how long versions coexist.
30
+
31
+ Prefer additive, backward-compatible changes:
32
+ - Add optional fields with safe defaults.
33
+ - Do not change field meaning, units, nullability, ordering, or enum behavior silently.
34
+ - Consumers should tolerate unknown fields where the format permits.
35
+ - Producers should not emit new enum values until consumers can safely handle them.
36
+ - Removing or changing a field requires usage evidence, a deprecation window, migration guidance, and a removal date.
37
+ - Breaking changes require a new contract version or an explicitly coordinated migration.
38
+
39
+ Compatibility must be tested across producer and consumer versions, not only against the latest schema.
40
+
41
+ ## 4. Data ownership and source of truth
42
+
43
+ Assign one authoritative owner for each business entity or invariant. Other services should access owned data through contracts, events, or explicitly governed read models—not by writing directly into another service's private tables.
44
+
45
+ For every entity, define:
46
+ - Stable identity and uniqueness constraints.
47
+ - Authoritative writer and permitted mutation paths.
48
+ - Transaction boundary and invariants.
49
+ - Lifecycle, retention, deletion, and audit requirements.
50
+ - Read models, replication lag, and acceptable staleness.
51
+ - Event publication and reconciliation strategy.
52
+
53
+ A shared database can be appropriate for a modular monolith, but table ownership and module boundaries should still be explicit.
54
+
55
+ ## 5. Schema and event evolution
56
+
57
+ Use expand-migrate-contract for online schema changes:
58
+ 1. Expand: add compatible schema and code paths.
59
+ 2. Migrate: backfill or dual-read with progress and reconciliation checks.
60
+ 3. Switch: move traffic or writers after compatibility is proven.
61
+ 4. Contract: remove obsolete fields/paths only after all consumers have migrated.
62
+
63
+ Avoid uncoordinated dual writes. Prefer transactional outbox/inbox or CDC where appropriate; define deduplication, ordering, replay, and poison-message behavior. Event schemas need stable event IDs, event type/version, occurred-at time, aggregate identity, and trace/correlation context where useful.
64
+
65
+ ## 6. Contract validation
66
+
67
+ Use schema validation, consumer-driven contract tests, compatibility checks in CI, migration tests against representative data, and end-to-end tests for critical business invariants. Include negative cases: malformed payloads, unknown fields/enum values, duplicate requests, replay, stale versions, authorization mismatch, and partial rollout.
68
+
69
+ ## 7. Review checklist
70
+
71
+ - [ ] Every contract has an owner and identified consumers.
72
+ - [ ] Authentication, authorization, and tenant scope are enforced server-side.
73
+ - [ ] Idempotency and concurrency semantics are explicit for writes.
74
+ - [ ] Errors distinguish retryable, terminal, conflict, and ambiguous outcomes.
75
+ - [ ] Pagination, filtering, payload size, and rate limits are bounded.
76
+ - [ ] Versioning and deprecation windows are documented.
77
+ - [ ] Data ownership and source of truth are unambiguous.
78
+ - [ ] Schema evolution supports old and new versions during rollout.
79
+ - [ ] Contract, migration, and compatibility tests run in CI.
80
+ - [ ] Sensitive fields are minimized and redacted from telemetry.
@@ -0,0 +1,84 @@
1
+ # Failure analysis and resilience design
2
+
3
+ Analyze failures across the full request, event, and operational lifecycle. A component diagram is not a failure model. Identify how faults propagate, what users observe, how the system degrades, and how it recovers.
4
+
5
+ ## 1. Failure-analysis sequence
6
+
7
+ 1. Identify critical business journeys, assets, and invariants.
8
+ 2. Enumerate dependencies and failure modes: application, database, network, identity, queue, cache, provider, operator, region, and deployment.
9
+ 3. Record trigger, detection signal, affected scope, and blast radius.
10
+ 4. Trace propagation: retries, fan-out, shared pools, queues, locks, and common-mode dependencies.
11
+ 5. Define prevention, containment, graceful degradation, and recovery.
12
+ 6. Specify reconciliation for ambiguous or partial outcomes.
13
+ 7. Define validation through fault injection, game days, restore drills, and incident exercises.
14
+
15
+ ## 2. Failure mode register
16
+
17
+ For each failure mode capture:
18
+
19
+ | Field | Description |
20
+ |---|---|
21
+ | Component / dependency | The affected system and owner |
22
+ | Failure mode | Timeout, crash, stale data, overload, corruption, credential expiry, bad deploy, etc. |
23
+ | Trigger / preconditions | Conditions under which it occurs |
24
+ | Detection | Metric, log, trace, alert, synthetic check, or user report |
25
+ | Blast radius | Users, tenants, regions, data, and downstream systems affected |
26
+ | User/business impact | Failed, delayed, duplicated, stale, or incorrect operation |
27
+ | Containment | Bulkhead, rate limit, circuit breaker, isolation, load shedding, feature disablement |
28
+ | Degraded behavior | Safe fallback, read-only mode, queued work, cached response, or explicit unavailability |
29
+ | Recovery | Restart, failover, replay, restore, rollback, reconciliation |
30
+ | Validation | Test or exercise and expected result |
31
+
32
+ ## 3. Blast-radius analysis
33
+
34
+ Check:
35
+ - Shared database, cache, queue, credentials, network, DNS, identity, and control-plane dependencies.
36
+ - Retry storms and fan-out amplification.
37
+ - Connection/thread/worker pool exhaustion.
38
+ - Noisy tenants and hot partitions.
39
+ - Region/AZ failure and common-mode dependencies.
40
+ - Deployment and schema compatibility across mixed versions.
41
+ - Correlated failures from shared providers or software supply chain.
42
+
43
+ Bound queues, concurrency, payload sizes, retries, cache memory, and per-tenant resource use. A retry is additional load; use deadlines, exponential backoff with jitter, retry budgets, and idempotency. Never retry an ambiguous side effect blindly.
44
+
45
+ ## 4. Degradation and recovery
46
+
47
+ For each critical journey, define:
48
+ - What can continue safely and what must fail closed?
49
+ - Which data may be stale, for how long, and how is freshness communicated?
50
+ - Can writes be queued safely? What is the durability and expiry guarantee?
51
+ - How are partial/unknown outcomes reconciled?
52
+ - What is the failover authority and how is split brain prevented?
53
+ - What is the rollback or forward-fix path?
54
+ - Who declares recovery and verifies backlog drain and data correctness?
55
+
56
+ Do not equate process restart with recovery. Recovery includes restoring business invariants, reconciling side effects, draining queues safely, and validating user-visible outcomes.
57
+
58
+ ## 5. RTO and RPO
59
+
60
+ - **RTO (Recovery Time Objective):** target maximum time to restore an acceptable service level after disruption.
61
+ - **RPO (Recovery Point Objective):** target maximum tolerable data loss measured in time.
62
+ - Define the service level that counts as recovered; partial read-only operation may differ from full write capability.
63
+ - Derive backup frequency, replication, restore procedures, and staffing from business RTO/RPO.
64
+ - Test restore and failover. A successful backup job does not prove recoverability.
65
+ - Include dependencies, credentials, encryption keys, DNS, and application compatibility in recovery exercises.
66
+
67
+ ## 6. Failure testing
68
+
69
+ Test realistic combinations: dependency timeout plus retry, zone loss under peak load, stale replica reads, queue backlog and poison messages, expired credentials, partial deployment, schema skew, cache outage, restore from backup, and ambiguous provider responses.
70
+
71
+ Set expected outcomes and stop conditions before fault injection. Protect production and obtain explicit authorization for disruptive exercises.
72
+
73
+ ## 7. Operational review checklist
74
+
75
+ - [ ] Critical journeys and invariants are identified.
76
+ - [ ] Failure modes and owners are recorded.
77
+ - [ ] Detection signals and alert thresholds map to user impact.
78
+ - [ ] Blast radius and shared dependencies are understood.
79
+ - [ ] Retries, deadlines, queues, and concurrency are bounded.
80
+ - [ ] Degraded modes preserve correctness and security.
81
+ - [ ] Ambiguous writes and partial side effects are reconciled.
82
+ - [ ] RTO/RPO are business-defined and restore-tested.
83
+ - [ ] Runbooks, escalation, and recovery ownership are clear.
84
+ - [ ] Failure tests cover the highest-risk scenarios.