system-design-primer-skill 1.2.0 → 2.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (128) hide show
  1. package/README.md +114 -60
  2. package/SKILL.md +437 -93
  3. package/VERSION +1 -1
  4. package/evals/README.md +1022 -0
  5. package/evals/system-design-cases.json +414 -0
  6. package/package.json +46 -4
  7. package/references/03-data-and-storage.md +8 -2
  8. package/references/04-caching.md +3 -3
  9. package/references/05-async-and-messaging.md +4 -0
  10. package/references/06-distributed-systems.md +2 -2
  11. package/references/07-reliability-and-resilience.md +7 -3
  12. package/references/08-api-security-observability.md +4 -0
  13. package/references/10-workflows-and-checklists.md +1 -1
  14. package/references/13-architecture-intelligence.md +91 -0
  15. package/references/14-api-and-data-contracts.md +80 -0
  16. package/references/15-failure-analysis.md +88 -0
  17. package/references/16-multi-agent-architecture.md +76 -0
  18. package/references/17-agent-harness-and-runtime.md +87 -0
  19. package/references/18-mcp-architecture.md +67 -0
  20. package/references/19-agent-security-and-permissions.md +90 -0
  21. package/references/20-agent-state-and-memory.md +76 -0
  22. package/references/21-agent-evaluation-and-observability.md +90 -0
  23. package/references/22-kubernetes-and-container-platforms.md +29 -0
  24. package/references/23-service-mesh-and-traffic-management.md +21 -0
  25. package/references/24-multi-region-and-disaster-recovery.md +28 -0
  26. package/references/25-platform-engineering-and-developer-experience.md +26 -0
  27. package/references/26-finops-and-cloud-cost.md +29 -0
  28. package/references/27-production-observability-and-slo.md +33 -0
  29. package/references/28-event-streaming-and-kafka-pulsar.md +33 -0
  30. package/references/29-event-sourcing-and-cqrs.md +23 -0
  31. package/references/30-change-data-capture-and-debezium.md +27 -0
  32. package/references/31-data-lakehouse-and-stream-batch.md +24 -0
  33. package/references/32-workflow-orchestration-and-temporal.md +28 -0
  34. package/references/33-schema-registry-and-event-governance.md +22 -0
  35. package/references/34-postgresql-internals-and-performance.md +45 -0
  36. package/references/35-mysql-innodb-and-performance.md +29 -0
  37. package/references/36-sharding-and-distributed-sql.md +22 -0
  38. package/references/37-consensus-and-distributed-transactions.md +29 -0
  39. package/references/38-search-architecture-elasticsearch-opensearch.md +24 -0
  40. package/references/39-vector-databases-and-hybrid-retrieval.md +31 -0
  41. package/references/40-zero-downtime-database-migrations.md +27 -0
  42. package/references/41-api-gateway-and-edge-architecture.md +23 -0
  43. package/references/42-rest-api-design-and-lifecycle.md +23 -0
  44. package/references/43-graphql-architecture-and-governance.md +16 -0
  45. package/references/44-grpc-and-service-contracts.md +16 -0
  46. package/references/45-websocket-sse-and-realtime-systems.md +16 -0
  47. package/references/46-realtime-collaboration-ot-crdt.md +16 -0
  48. package/references/47-api-security-rate-limiting-and-abuse.md +16 -0
  49. package/references/48-oauth-oidc-and-passkeys.md +20 -0
  50. package/references/49-iam-rbac-abac-rebac.md +20 -0
  51. package/references/50-zero-trust-and-workload-identity.md +16 -0
  52. package/references/51-secrets-kms-pki-and-key-management.md +20 -0
  53. package/references/52-supply-chain-security-sbom-slsa.md +78 -0
  54. package/references/53-threat-modeling-stride-and-attack-trees.md +16 -0
  55. package/references/54-privacy-engineering-and-data-governance.md +20 -0
  56. package/references/55-opentelemetry-and-instrumentation.md +54 -0
  57. package/references/56-distributed-tracing-and-context-propagation.md +47 -0
  58. package/references/57-continuous-profiling-and-performance-diagnostics.md +45 -0
  59. package/references/58-load-stress-soak-testing-and-workload-modeling.md +44 -0
  60. package/references/59-capacity-planning-and-performance-modeling.md +38 -0
  61. package/references/60-chaos-engineering-and-resilience-experiments.md +45 -0
  62. package/references/61-incident-management-and-postmortems.md +38 -0
  63. package/references/62-dns-and-service-discovery.md +35 -0
  64. package/references/63-load-balancing-and-global-traffic-management.md +31 -0
  65. package/references/64-cdn-and-edge-caching.md +34 -0
  66. package/references/65-cloud-networking-and-connectivity.md +34 -0
  67. package/references/66-edge-computing-architecture.md +33 -0
  68. package/references/67-storage-infrastructure-and-storage-systems.md +34 -0
  69. package/references/68-tcp-quic-and-network-performance.md +27 -0
  70. package/references/69-task-routing-and-evidence.md +115 -0
  71. package/scripts/audit_chain.py +36 -0
  72. package/scripts/compare_reports.py +43 -0
  73. package/scripts/evaluate.py +475 -0
  74. package/scripts/evaluation_catalog.py +129 -0
  75. package/scripts/evaluation_fingerprints.py +20 -0
  76. package/scripts/evidence_archive.py +79 -0
  77. package/scripts/evidence_signing.py +98 -0
  78. package/scripts/evidence_timestamp.py +441 -0
  79. package/scripts/install.ps1 +12 -3
  80. package/scripts/install.sh +15 -8
  81. package/scripts/report_integrity.py +184 -0
  82. package/scripts/review_consensus.py +77 -0
  83. package/scripts/review_evidence.py +105 -0
  84. package/scripts/review_provenance.py +44 -0
  85. package/scripts/reviewer_calibration.py +215 -0
  86. package/scripts/rubric_compatibility.py +134 -0
  87. package/scripts/run_evals.py +130 -0
  88. package/scripts/run_manifest.py +179 -0
  89. package/scripts/statistical_analysis.py +308 -0
  90. package/scripts/tsa_attestation.py +709 -0
  91. package/scripts/tsa_validation_report.py +627 -0
  92. package/scripts/validate.py +356 -11
  93. package/scripts/verify_build_provenance.py +625 -0
  94. package/scripts/verify_tsa_ocsp.py +962 -0
  95. package/scripts/verify_tsa_revocation.py +782 -0
  96. package/scripts/windows_broker_protocol.py +274 -0
  97. package/scripts/windows_broker_transport.py +141 -0
  98. package/scripts/windows_broker_win32.py +713 -0
  99. package/templates/agent-evaluation-plan-template.md +57 -0
  100. package/templates/agent-security-review-template.md +46 -0
  101. package/templates/agent-system-design-template.md +74 -0
  102. package/templates/api-architecture-design-template.md +33 -0
  103. package/templates/api-security-and-rate-limit-review-template.md +29 -0
  104. package/templates/architecture-decision-matrix-template.md +48 -0
  105. package/templates/architecture-review-scorecard-template.md +36 -0
  106. package/templates/cdc-data-pipeline-template.md +41 -0
  107. package/templates/cdn-and-edge-strategy-template.md +36 -0
  108. package/templates/chaos-experiment-template.md +39 -0
  109. package/templates/cloud-cost-model-template.md +37 -0
  110. package/templates/database-architecture-review-template.md +33 -0
  111. package/templates/edge-computing-readiness-template.md +36 -0
  112. package/templates/event-sourcing-cqrs-template.md +41 -0
  113. package/templates/event-streaming-design-template.md +47 -0
  114. package/templates/failure-analysis-template.md +36 -0
  115. package/templates/identity-and-access-architecture-template.md +29 -0
  116. package/templates/incident-postmortem-template.md +39 -0
  117. package/templates/multi-region-disaster-recovery-template.md +43 -0
  118. package/templates/network-architecture-design-template.md +38 -0
  119. package/templates/observability-architecture-template.md +52 -0
  120. package/templates/performance-test-plan-template.md +52 -0
  121. package/templates/privacy-data-governance-template.md +24 -0
  122. package/templates/production-readiness-checklist-template.md +52 -0
  123. package/templates/realtime-system-design-template.md +28 -0
  124. package/templates/security-threat-model-template.md +26 -0
  125. package/templates/sharding-distributed-sql-design-template.md +34 -0
  126. package/templates/storage-architecture-design-template.md +35 -0
  127. package/templates/supply-chain-security-review-template.md +20 -0
  128. package/templates/zero-downtime-database-migration-template.md +32 -0
@@ -0,0 +1,76 @@
1
+ # Agent state, memory, and durable workflow design
2
+
3
+ Do not conflate model context, conversation history, long-term memory, and durable workflow state. Each has different ownership, consistency, retention, privacy, and recovery requirements.
4
+
5
+ ## Scope and boundaries
6
+
7
+ This document owns agent-specific state taxonomy, durable run/checkpoint design, conversation history, and memory lifecycle. Use [privacy engineering](54-privacy-engineering-and-data-governance.md) for retention, deletion, legal holds, and downstream processor obligations; [workflow orchestration](32-workflow-orchestration-and-temporal.md) for durable timers, replay, and orchestration semantics; and [distributed transactions](37-consensus-and-distributed-transactions.md) for atomic coordination with external side effects. Deleting a conversation record does not prove deletion of derived memory, embeddings, caches, replicas, or backups.
8
+
9
+ ## 1. State taxonomy
10
+
11
+ | State type | Purpose | Typical lifecycle and controls |
12
+ |---|---|---|
13
+ | Ephemeral context | Current model invocation and bounded working context | Short-lived; rebuildable; never sole source of truth |
14
+ | Conversation history | User-visible interaction record | User/tenant scoped; retention and deletion policy |
15
+ | Summaries / memory | Derived facts or preferences used across sessions | Provenance, confidence, consent, correction, expiry, deletion |
16
+ | Workflow state | Durable task status, inputs, checkpoints, and outputs | Transactional ownership, versioning, idempotency, recovery |
17
+ | External business data | Authoritative customer/order/payment/project records | Owned by domain service; access via authorized APIs |
18
+
19
+ Define which store is authoritative for each state item. Model-generated summaries are derived data and may be incomplete or wrong; preserve source references and allow correction.
20
+
21
+ ## 2. Memory lifecycle and governance
22
+
23
+ For every memory class define:
24
+ - Purpose and allowed content.
25
+ - Creation authority and provenance.
26
+ - User visibility, consent, correction, and deletion semantics.
27
+ - Tenant isolation, encryption, retention, expiry, and backup deletion behavior.
28
+ - Access policy and audit.
29
+ - Confidence, freshness, and conflict resolution.
30
+
31
+ Do not silently promote a model inference into durable user memory. Do not store secrets or unnecessary sensitive data. Apply deletion to derived indexes, caches, summaries, and replicas according to a documented lifecycle.
32
+
33
+ ## 3. Workflow persistence
34
+
35
+ Represent a durable run with stable run ID, tenant/principal, workflow version, state version, status, deadline, budget, checkpoint pointer, and correlation metadata. Persist state transitions atomically with an event or outbox record when external work must be scheduled.
36
+
37
+ Use optimistic concurrency or explicit locking to prevent concurrent workers from overwriting newer state. Define allowed transitions and reject stale writes. Make task creation and message handling idempotent.
38
+
39
+ ## 4. Checkpoint and resume
40
+
41
+ A checkpoint should capture the minimum information needed to resume safely:
42
+ - Workflow state and version.
43
+ - Completed steps and validated outputs with provenance.
44
+ - Pending actions and their idempotency/operation IDs.
45
+ - Remaining budget/deadline and cancellation state.
46
+ - References to external data, not copies of secrets.
47
+
48
+ Checkpoint before and after meaningful side effects. On resume, revalidate identity, permissions, resource versions, and policy. Do not replay a side effect merely because its completion record is missing; query the downstream operation or reconcile using its idempotency key.
49
+
50
+ ## 5. Event history and replay
51
+
52
+ Use append-only transition/event history when auditability and recovery justify it. Distinguish event sourcing from ordinary logs: an event log used to rebuild authoritative state requires strict schema evolution, ordering, retention, and replay tests.
53
+
54
+ Define duplicate, late, out-of-order, and poison events. Replay must be deterministic enough to reproduce state transitions and must not re-trigger external side effects. Store schema/workflow versions and provide upcasters or migration procedures.
55
+
56
+ ## 6. Consistency and retention
57
+
58
+ Choose consistency based on business invariants. Conversation summaries can often be eventually consistent; payment or authorization workflow state generally cannot. Define transactional boundaries across state, queue publication, and side effects using outbox/inbox, idempotency, and reconciliation.
59
+
60
+ Set retention and deletion policies for conversation history, memory, workflow state, audit, and derived embeddings independently. Ensure deletion and tenant offboarding propagate to indexes, caches, backups, and downstream processors according to policy.
61
+
62
+ ## 7. Recovery and operations
63
+
64
+ Plan for worker crash, lease expiry, duplicate delivery, partial checkpoint, schema upgrade, provider timeout, tool timeout, cancellation, and tenant deletion during a run. Define stale-run detection, lease renewal, retry budget, dead-letter handling, operator repair, and recovery verification.
65
+
66
+ ## 8. Review checklist
67
+
68
+ - [ ] Ephemeral context, history, memory, workflow state, and business data are separated.
69
+ - [ ] Every state class has an owner, purpose, access policy, and retention lifecycle.
70
+ - [ ] Memory provenance, consent/correction, confidence, and deletion are defined.
71
+ - [ ] Workflow transitions are versioned and concurrency-safe.
72
+ - [ ] Checkpoints capture pending side effects and idempotency references.
73
+ - [ ] Resume revalidates identity, permission, and resource state.
74
+ - [ ] Replay cannot duplicate external effects.
75
+ - [ ] Deletion propagates to derived stores and indexes.
76
+ - [ ] Recovery procedures are tested, not merely documented.
@@ -0,0 +1,90 @@
1
+ # Agent evaluation, observability, and operational quality
2
+
3
+ Evaluate agents as complete systems that interact with users, tools, data, and external services. A plausible answer or successful model call is not proof that the task was completed correctly or safely.
4
+
5
+ ## 1. Define task-level outcomes
6
+
7
+ For each task class define:
8
+ - Preconditions and authorized scope.
9
+ - Expected outcome and business invariants.
10
+ - Acceptable alternatives and partial completion.
11
+ - Prohibited actions and safety constraints.
12
+ - Required evidence/provenance.
13
+ - Latency, token, cost, and human-intervention budgets.
14
+
15
+ Separate task success from stylistic preference. Define deterministic assertions wherever possible.
16
+
17
+ ## 2. Evaluation dataset
18
+
19
+ Maintain a versioned dataset containing representative, edge, adversarial, and regression cases. Record source, expected behavior, risk class, tenant/data assumptions, and evaluation version. Protect sensitive examples and ensure test data is authorized and sanitized.
20
+
21
+ Cover:
22
+ - Normal requests and ambiguous instructions.
23
+ - Tool selection and argument construction.
24
+ - Invalid, unavailable, or malicious tool results.
25
+ - Prompt injection in user and retrieved content.
26
+ - Authorization denial and cross-tenant attempts.
27
+ - Duplicate requests, timeout, partial side effect, and resume.
28
+ - Long-context, token-budget, cancellation, and concurrency behavior.
29
+ - Multi-agent delegation, disagreement, and aggregation.
30
+
31
+ ## 3. Metrics and scorecards
32
+
33
+ Measure multiple dimensions independently:
34
+ - Task completion and correctness.
35
+ - Groundedness, citation/provenance accuracy.
36
+ - Tool selection, argument validity, and tool outcome.
37
+ - Policy compliance, unauthorized action rate, and data leakage.
38
+ - Robustness to adversarial and malformed inputs.
39
+ - Recovery, cancellation, and duplicate-side-effect rate.
40
+ - End-to-end latency, TTFT where relevant, token use, and cost.
41
+ - Human approval, override, escalation, and abandonment rates.
42
+
43
+ Do not collapse safety-critical dimensions into a single weighted quality score. Report sample size, confidence interval where appropriate, dataset version, and known limitations. Avoid optimizing to a leaked or static benchmark.
44
+
45
+ ## 4. Evaluation methods
46
+
47
+ Combine:
48
+ - Unit tests for policy, schemas, state transitions, and deterministic logic.
49
+ - Contract tests for model adapters and tools.
50
+ - Scenario tests for complete user journeys.
51
+ - Trajectory review for intermediate reasoning/actions without treating hidden chain-of-thought as a required artifact.
52
+ - Adversarial/red-team tests for injection and authorization bypass.
53
+ - Fault injection for provider/tool outages, timeout, queue backlog, and worker crash.
54
+ - Human review for high-impact or ambiguous cases.
55
+ - Staged online experiments with rollback thresholds.
56
+
57
+ Use model-based graders cautiously: validate them against human labels, measure disagreement, and never let an LLM judge be the sole gate for critical security or financial invariants.
58
+
59
+ ## 5. Regression and release gates
60
+
61
+ Version model/provider, prompt, tool schema, retrieval index, policy, workflow, and dataset. Run the same critical regression suite when any of these changes. Define release thresholds per risk class, minimum sample size, and rollback criteria. Block release on critical policy violations even if average task quality improves.
62
+
63
+ Separate offline evaluation from online monitoring. Canary new versions on bounded traffic and compare task outcomes, safety, latency, cost, and escalation.
64
+
65
+ ## 6. Tracing and observability
66
+
67
+ Trace a run using correlation IDs across orchestration, model calls, tool calls, queues, and child agents. Capture:
68
+ - Run/task/parent-child IDs and workflow version.
69
+ - Model/provider and prompt/config version.
70
+ - Tool name/version, schema validation, authorization decision, and outcome.
71
+ - State transitions, checkpoint/resume, retries, cancellation, and escalation.
72
+ - Latency phases, token counts, budget consumption, and cost estimate.
73
+ - Evaluation signals, human intervention, and terminal status.
74
+
75
+ Redact prompts, secrets, PII, and sensitive tool payloads by default. Define access control and retention for traces and audit separately.
76
+
77
+ ## 7. Operations and incident response
78
+
79
+ Alert on user-impacting SLOs: task failure, policy violation, stuck runs, queue age, provider errors, tool timeouts, budget overruns, and recovery failures. Define runbooks for pausing workflows, revoking tool credentials, disabling a tool/server, rolling back prompt/model/policy versions, reconciling side effects, and replaying safe tasks.
80
+
81
+ ## 8. Review checklist
82
+
83
+ - [ ] Task-level success and prohibited behavior are explicit.
84
+ - [ ] Evaluation data is versioned, representative, authorized, and protected.
85
+ - [ ] Tool, policy, injection, recovery, and multi-agent cases are covered.
86
+ - [ ] Safety-critical metrics remain independent release gates.
87
+ - [ ] Graders are calibrated and human-validated.
88
+ - [ ] Model/prompt/tool/policy/dataset versions are traceable.
89
+ - [ ] Telemetry is end-to-end, redacted, access-controlled, and retained appropriately.
90
+ - [ ] Canary, rollback, and incident runbooks are defined.
@@ -0,0 +1,29 @@
1
+ # Kubernetes and Container Platform Architecture
2
+
3
+ Use Kubernetes when workload density, deployment automation, scheduling, service discovery, self-healing, policy enforcement, or platform standardization justify operating a cluster. It is not a default requirement for high traffic. Compare managed container platforms, serverless containers, and simpler VM or PaaS deployment first.
4
+
5
+ ## Workload and cluster model
6
+ - Classify stateless services, stateful services, batch jobs, scheduled jobs, operators, and platform components. State which workloads truly need orchestration.
7
+ - Define cluster boundaries by environment, region, trust domain, regulatory boundary, and blast-radius target. Avoid one global cluster as a universal control plane.
8
+ - Treat namespaces as organizational and policy boundaries, not hard security isolation. Use separate clusters or stronger runtime isolation for mutually untrusted tenants.
9
+ - Model requests and limits, QoS classes, pod disruption budgets, topology spread, anti-affinity, priority classes, and autoscaling together. CPU limits can cause throttling; memory limits can cause OOM kills. Validate under representative load.
10
+ - Keep readiness, liveness, and startup probes distinct. Liveness must not restart a process merely because a downstream dependency is temporarily unavailable. Readiness should remove an instance that cannot safely serve.
11
+ - Design graceful termination: stop new work, drain connections, finish or checkpoint bounded in-flight work, and exit before the termination grace period. Align application shutdown, load-balancer deregistration, and rollout settings.
12
+ - Treat persistent volumes, CSI drivers, storage classes, snapshots, and zone failure behavior as explicit data architecture. Replicated Pods do not replicate a database.
13
+
14
+ ## Scheduling, scaling, and delivery
15
+ Set resource requests from measured steady-state and burst behavior. Use HPA only with reliable signals; use custom or external metrics only when their pipeline is dependable. Cluster autoscaling has provisioning delay, quota limits, and bin-packing constraints. Model the complete scale-up path and retain headroom for node failure and rolling updates.
16
+
17
+ Avoid feedback loops where HPA, KEDA, cluster autoscaler, and application concurrency controls amplify one another. Bound queue consumers and scale on backlog age or processing demand when appropriate, not CPU alone.
18
+
19
+ Use immutable image digests, signed artifacts where available, SBOMs, vulnerability scanning, admission policy, and provenance verification. Pin and review Helm charts, operators, CRDs, and controller versions. Separate build permissions from deploy permissions. Prefer declarative delivery with drift detection and auditable promotion. Kubernetes component version-skew limits and supported release branches change over time; check the current [Kubernetes version-skew policy](https://kubernetes.io/releases/version-skew-policy/) and the compatibility matrix for each add-on before planning control-plane or node upgrades. Do not encode a remembered skew allowance as a timeless rule.
20
+
21
+ Rollouts must define max unavailable/surge, readiness gates, canary or progressive delivery, automated abort criteria, and rollback compatibility. Database and event schema changes must remain compatible with old and new application versions during rollout.
22
+
23
+ ## Security and operations
24
+ Apply least-privilege RBAC, workload identity, short-lived credentials, network policies, restricted Pod Security settings, admission controls, secret encryption, and node hardening. Do not mount the container runtime socket or privileged host paths into ordinary workloads. Restrict egress and metadata-service access. Namespace boundaries alone do not prevent noisy-neighbor or control-plane risks.
25
+
26
+ Define API-server and etcd availability, node capacity, DNS, CNI, CSI, ingress, certificate, and control-plane upgrade dependencies. Test node drains, zone loss, cluster upgrades, quota exhaustion, image-pull failure, DNS degradation, and storage attachment failure. Keep cluster recovery procedures and application-data backups separate.
27
+
28
+ ## Decision and exit criteria
29
+ Compare total cost: control plane, worker nodes, idle headroom, networking, observability, security tooling, on-call, upgrades, and specialist staffing. Define when to consolidate, split, or leave Kubernetes. Success means reduced developer lead time and operational risk, not maximum feature usage.
@@ -0,0 +1,21 @@
1
+ # Service Mesh and Traffic Management
2
+
3
+ A service mesh can centralize service identity, encryption, traffic policy, retries, telemetry, and progressive delivery. Adopt one only when these capabilities solve measured cross-cutting problems that application libraries, gateways, or the platform cannot solve more simply.
4
+
5
+ ## Traffic architecture
6
+ Separate north-south traffic from east-west traffic. Identify ingress, API gateway, load balancers, DNS, service discovery, mesh data plane, and egress gateways. Assign each layer a clear responsibility; overlapping retries, timeouts, routing, and rate limits create unpredictable behavior.
7
+
8
+ Define routing by service identity and version, not unstable Pod IPs. For canaries, specify traffic split, eligible requests, session affinity, metric windows, minimum sample size, abort thresholds, and rollback action. Percentage splits may not create representative cohorts when traffic is skewed by tenant, region, or request type.
9
+
10
+ ## Identity and security
11
+ Use workload identity and mutual TLS where justified. Define certificate issuance, rotation, trust roots, revocation behavior, identity mapping, and failure policy. mTLS authenticates a workload identity and encrypts a channel; it does not authorize every business operation. Enforce application-level authorization and tenant checks independently.
12
+
13
+ Model ingress and egress explicitly. Restrict outbound destinations, DNS resolution, metadata endpoints, and internal control planes. Validate bypass paths, host networking, UDP, init containers, and node-level traffic rather than assuming every flow is captured by a sidecar.
14
+
15
+ ## Resilience semantics
16
+ Set one end-to-end deadline and allocate budgets across hops. Retries must be bounded, jittered, and limited to safe or idempotent operations. Avoid retries at multiple layers; they multiply load during incidents. Define circuit-breaker thresholds, outlier detection, connection pools, concurrency limits, and load shedding.
17
+
18
+ A mesh control-plane outage may leave existing data-plane configuration active while preventing updates; verify behavior for the chosen implementation and version. Decide whether policy failures are fail-open or fail-closed per traffic class. Test certificate expiry, control-plane loss, stale configuration, DNS failure, and partial proxy rollout.
19
+
20
+ ## Cost and adoption gate
21
+ Measure proxy CPU/memory, connection overhead, tail latency, telemetry volume, configuration complexity, and incident debugging time. Assign ownership for upgrades, policy review, certificate lifecycle, and version skew. Compare application libraries, API gateways, ingress controllers, cloud-native service networking, and a mesh. Start with one bounded service group and define success metrics and an exit path. Verify feature support against current official versioned documentation.
@@ -0,0 +1,28 @@
1
+ # Multi-Region Architecture and Disaster Recovery
2
+
3
+ Multi-region architecture is a business continuity and latency decision, not a checkbox for availability. Begin with business impact analysis, failure scenarios, and tested recovery objectives.
4
+
5
+ ## Scope and boundaries
6
+
7
+ This document owns regional topology, data ownership across regions, replication/failover/failback, and business recovery objectives. Use [service reliability](07-reliability-and-resilience.md) for service-level timeout, retry, degradation, and availability design; [storage and database references](03-data-and-storage.md) plus the engine-specific guides for local transaction semantics; and [incident management](61-incident-management-and-postmortems.md) for live incident command. DNS or traffic steering changes do not fence a former writer, and a regional recovery plan must prove data ownership and write safety independently of routing.
8
+
9
+ ## Objectives and failure domains
10
+ For each critical journey define RTO, RPO, availability target, error-budget policy, residency constraints, and dependencies that must recover together. Distinguish zone, region, provider, control-plane, account, and global dependency failures. A second region is not independent if both share identity, DNS, deployment, encryption-key, or quota dependencies.
11
+
12
+ ## Topology choices
13
+ Compare backup and restore, pilot light, warm standby, active-passive, and active-active. Choose per workload and data domain. Stateless services may be active-active while transactional data remains single-writer. Do not claim active-active availability without proving write semantics and dependency independence.
14
+
15
+ ## Data replication and consistency
16
+ For every dataset define source of truth, replication mode, lag objective, ordering, conflict policy, deletion propagation, encryption, and restore procedure. Synchronous replication can increase write latency and couple regional availability; asynchronous replication permits nonzero RPO and stale reads. Specify whether consistency is per-key, per-entity, per-session, or global, and what a read in each region can observe after a write. Multi-primary writes require conflict resolution that preserves business invariants, not merely last-write-wins timestamps. For invariants that cannot be merged safely (for example, spending a shared balance or allocating a unique scarce item), use a single-writer/ownership boundary or coordination rather than assuming merge or timestamp ordering can preserve correctness.
17
+
18
+ Test replication lag, network partition, duplicate delivery, out-of-order updates, tombstones, schema changes, and recovery from corrupted or logically deleted data. Backups must be isolated from production credentials and deletion paths. Replication is not a backup.
19
+
20
+ ## Failover and failback
21
+ Specify detection signals, decision authority, fencing of the old primary, traffic steering, data promotion, write enablement, queue ownership, cache invalidation, and customer communication. Prevent split brain with leases, quorum, or an operator-controlled fencing mechanism appropriate to the data system. DNS TTL alone is not a failover strategy.
22
+
23
+ Failback is a separate migration: reconcile writes made in the recovery region, validate consistency, re-establish replication, and switch traffic only after safety gates pass. Define the cutover epoch/ownership change and fence the former writer before enabling writes in the restored region; DNS convergence or a routing change is not a write-fencing mechanism. Document manual and automated paths, including when automation must stop for human approval.
24
+
25
+ ## Validation and economics
26
+ Run game days for region isolation, degraded inter-region links, unavailable KMS, exhausted quotas, stale DNS, failed deployment, and data corruption. Measure actual RTO/RPO from detection through verified business recovery. Perform restore tests, not just backup-job checks. Record evidence, gaps, owners, and remediation dates.
27
+
28
+ Estimate duplicate compute, replicated storage, inter-region transfer, global load balancing, support tiers, idle capacity, and operational staffing. Compare expected loss reduction with recurring and migration costs.
@@ -0,0 +1,26 @@
1
+ # Platform Engineering and Developer Experience
2
+
3
+ A platform is an internal product that provides paved paths for software delivery. Its purpose is to reduce cognitive load, improve consistency, and make secure and reliable practices easier—not to centralize every engineering decision.
4
+
5
+ ## Product model
6
+ Identify developer personas, lifecycle journeys, friction points, and measurable outcomes. Treat application teams as customers. Publish a service catalog, ownership metadata, supported golden paths, lifecycle policies, and a support model. Define platform product ownership and a feedback loop.
7
+
8
+ Measure lead time to first production deployment, deployment frequency, change failure rate, recovery time, toil, onboarding time, platform adoption, and developer satisfaction. Avoid vanity metrics such as number of templates or features shipped.
9
+
10
+ ## Platform capabilities
11
+ Typical capabilities include secure service scaffolding; CI/CD building blocks, artifact provenance, and progressive delivery; infrastructure provisioning through reviewed modules; identity, secrets, certificates, network, and policy integrations; observability defaults and SLO templates; service catalog and dependency mapping; and ephemeral preview environments.
12
+
13
+ Build a capability only when there is repeated demand and a clear support owner. Prefer composable APIs and documented interfaces over a monolithic portal that becomes a mandatory bottleneck.
14
+
15
+ ## Golden paths and guardrails
16
+ A golden path is an opinionated, supported route—not the only possible route. Make the default secure, observable, cost-aware, and operable. Provide escape hatches with explicit risk ownership and review. Enforce non-negotiable controls through policy-as-code and runtime boundaries rather than documentation alone.
17
+
18
+ Version templates and modules, define compatibility windows, publish deprecation notices, and provide migration tooling. Avoid breaking all teams through uncoordinated platform upgrades.
19
+
20
+ ## Team topology and reliability
21
+ Clarify platform team responsibilities versus product team responsibilities. Product teams retain accountability for service behavior, SLOs, data, and on-call unless a documented operating model says otherwise. Define escalation, incident handoff, platform SLOs, and shared responsibility for dependencies.
22
+
23
+ Treat the platform as production software: availability objectives, capacity, backup, disaster recovery, security review, release management, support, and incident response. Roll out incrementally, observe adoption and friction, and retire duplicated tooling only after migration.
24
+
25
+ ## Economics
26
+ Calculate total cost including engineering time, support, compute, licensing, maintenance, and duplicated or unused capabilities. Compare build-versus-buy and centralization benefits against queueing delays and reduced team autonomy.
@@ -0,0 +1,29 @@
1
+ # FinOps and Cloud Cost Architecture
2
+
3
+ FinOps is a continuous operating practice connecting engineering, finance, and business value. Cost optimization must preserve reliability, security, performance, and delivery outcomes.
4
+
5
+ ## Cost visibility and allocation
6
+ Create a common allocation model using accounts/projects, environments, services, teams, products, tenants, and cost centers. Require ownership metadata and consistent tags/labels. Separate direct, shared, unallocated, and one-time costs. Define allocation rules for shared clusters, network egress, observability, support plans, and platform services.
7
+
8
+ Track cost by workload and business unit. Useful measures include cost per active user, transaction, request, GB processed, inference, or unit of business value. Normalize for demand and product mix; raw monthly spend alone cannot distinguish growth from inefficiency.
9
+
10
+ ## Cost model
11
+ Include compute requests and idle capacity, autoscaling headroom, storage tiers and IOPS, database licenses, replicas, backups, snapshots, network egress, inter-region transfer, load balancers, observability ingestion and retention, support, marketplace software, and engineering operations. Model burst, peak, growth, retention, and disaster-recovery capacity.
12
+
13
+ Distinguish committed-use discounts and reserved capacity from flexible on-demand demand. Evaluate break-even utilization, commitment term, cancellation risk, portability, and forecast confidence. Spot/preemptible capacity suits only interruption-tolerant workloads with checkpointing and retry-safe execution.
14
+
15
+ ## Optimization workflow
16
+ 1. Establish a baseline and unit economics.
17
+ 2. Identify the largest cost drivers and waste, not merely the easiest line items.
18
+ 3. Form a hypothesis with an expected saving range and performance/reliability risk.
19
+ 4. Test one change under representative load.
20
+ 5. Compare cost and service-level outcomes before and after.
21
+ 6. Roll out gradually, monitor regressions, and document ownership.
22
+ 7. Revisit continuously as workload and pricing change.
23
+
24
+ Common levers include measured rightsizing, autoscaling, scheduling non-production environments, storage lifecycle policies, query and data-transfer reduction, cache efficiency, batching, retention controls, and commitment optimization. Do not lower replica counts, backups, redundancy, or telemetry blindly to hit a budget.
25
+
26
+ ## Budgets and architecture decisions
27
+ Use budgets and anomaly detection as alerts, not the only control. Define approval for commitments, quota increases, expensive resources, and exceptions. Apply policy-as-code for resource classes, regions, labels, and retention where appropriate.
28
+
29
+ Compare lifecycle TCO rather than unit price alone. Include migration, lock-in, support, incident risk, idle capacity, staff expertise, and exit cost. Model expected, peak, and degraded modes. Kubernetes, service mesh, and multi-region designs may increase resilience while materially increasing fixed cost; quantify both benefits and operational burden.
@@ -0,0 +1,33 @@
1
+ # Production Observability and SLO Engineering
2
+
3
+ Observability should answer whether users can complete critical journeys, why they cannot, and what action is safe. Instrumentation volume is not a substitute for useful operational signals.
4
+
5
+ ## Scope and boundaries
6
+
7
+ This document owns production telemetry architecture, SLIs/SLOs, error budgets, instrumentation quality, and observability-backend operations. Use [distributed tracing](56-distributed-tracing-and-context-propagation.md) for trace-context propagation details; [OpenTelemetry instrumentation](55-opentelemetry-and-instrumentation.md) for SDK/collector conventions; [service reliability](07-reliability-and-resilience.md) for timeout, retry, failover, and recovery controls; and [incident management](61-incident-management-and-postmortems.md) for command roles, communications, and postmortems. Observability signals support diagnosis; they do not establish causality or prove a recovery action is safe.
8
+
9
+ ## Service-level objectives
10
+ Define SLIs from user-visible behavior: availability, latency, correctness, freshness, durability, or successful completion. Specify event population, good-event definition, measurement window, exclusions, and aggregation. Set SLO targets based on user and business needs, not arbitrary industry defaults.
11
+
12
+ Use error budgets to govern release risk. Define policy for budget consumption, freeze thresholds, exception approval, and recovery. Pair service SLOs with dependency and platform indicators while preserving ownership. Do not treat every infrastructure metric as an SLI.
13
+
14
+ ## Metrics, logs, traces, and profiles
15
+ - Metrics: bounded-cardinality time series for rates, errors, duration, saturation, queue depth, and resource use.
16
+ - Logs: structured events with correlation identifiers, severity, and controlled fields. Redact secrets and personal data; define retention and access.
17
+ - Distributed traces: propagate trace context across asynchronous boundaries and record meaningful spans, status, and dependency calls. Sampling must preserve error and tail-latency evidence.
18
+ - Profiles: use continuous or on-demand profiling to identify CPU, allocation, lock, and runtime hotspots when metrics and traces are insufficient.
19
+
20
+ Use consistent service, environment, region, version, and ownership attributes. Avoid unbounded labels such as user IDs, request IDs, raw URLs, or arbitrary exception strings. Keep high-cardinality identifiers in logs/traces where access and cost controls are appropriate.
21
+
22
+ ## Alerting and incident response
23
+ Alert on user impact, SLO burn rate, actionable saturation, and failure conditions with a clear owner and runbook. Use multi-window, multi-burn-rate alerts where appropriate. Avoid paging on every transient symptom or on metrics without an action. Distinguish page, ticket, dashboard, and automated remediation signals.
24
+
25
+ Define severity, incident commander, communication channels, escalation, evidence preservation, rollback authority, and post-incident review. Runbooks should state symptoms, safe checks, mitigation, rollback, escalation, and verification. Automated remediation must be bounded, idempotent, observable, and reversible.
26
+
27
+ ## Telemetry reliability and cost
28
+ Model SDKs, agents, collectors, gateways, queues, storage, query engines, and alert delivery as production dependencies. Bound buffers, retries, cardinality, sampling, and backpressure. Define behavior when telemetry backends are unavailable; application availability must not depend on synchronous log or trace delivery.
29
+
30
+ Set retention by operational and regulatory purpose. Monitor dropped telemetry, ingestion lag, query latency, alert delivery, and collector resource use. Estimate ingestion and retention cost before enabling high-volume instrumentation.
31
+
32
+ ## Release and evaluation
33
+ Correlate deployments and configuration changes with SLOs and incidents. Validate dashboards and alerts as code. Use synthetic checks for critical user journeys and real-user monitoring where useful. Test alert routing and on-call schedules. Verify telemetry supports diagnosis without exposing sensitive data.
@@ -0,0 +1,33 @@
1
+ # Event Streaming Architecture: Kafka and Pulsar
2
+
3
+ Choose an event-streaming platform based on durability, ordering, replay, throughput, multi-tenancy, operations, and ecosystem requirements—not brand popularity. Compare Kafka, Pulsar, managed cloud streams, and simpler queues.
4
+
5
+ ## Scope and boundaries
6
+
7
+ This document owns Kafka/Pulsar and event-streaming-specific design: partitions, broker replication, producer/consumer behavior, offsets, retention, rebalances, and platform operations. Use [asynchronous processing](05-async-and-messaging.md) for broker-agnostic delivery and queue fundamentals; [CDC](30-change-data-capture-and-debezium.md) for database-log capture; [event sourcing and CQRS](29-event-sourcing-and-cqrs.md) for domain event history and projections; and [schema governance](33-schema-registry-and-event-governance.md) for compatibility policy. Broker transactions do not by themselves make external side effects atomic.
8
+
9
+ ## Domain and topic design
10
+ Define event ownership, topic purpose, key, partitioning strategy, retention, compaction, schema, compatibility, and access policy. Use stable business keys when per-entity ordering matters. Partition count determines parallelism and affects ordering, storage, and reassignment cost. Hot keys can bottleneck a partition even when cluster capacity is available.
11
+
12
+ Treat events as immutable facts about something that happened. Separate event time, ingestion time, and processing time. Include event ID, type, schema version, aggregate/entity key, producer identity, correlation/causation IDs, timestamp semantics, and trace context. Do not put secrets or unnecessary personal data in event payloads.
13
+
14
+ ## Delivery and ordering semantics
15
+ Document producer acknowledgements, replication factor, minimum in-sync replicas, leader election behavior, idempotent producer settings, transaction scope, consumer offset commits, and failure behavior. Understand the limits of ordering: ordering is generally scoped to a partition or key, not a whole topic or multiple topics.
16
+
17
+ Consumers should be idempotent and tolerate duplicates, retries, rebalances, and replay. Commit offsets only after the corresponding side effect is durable, or use an atomic pattern that binds offset progress to output effects. A broker transaction does not automatically make an external database or API side effect exactly once.
18
+
19
+ For Apache Kafka, distinguish producer idempotence from transactions. In Kafka 4.0, producer idempotence is enabled by default only when no conflicting configuration disables it; it suppresses duplicates caused by producer retries within a producer session, not arbitrary application-level resends or effects in another system. A transactional producer requires a stable, appropriately unique `transactional.id`; consumers must use `read_committed` to hide aborted transactional records. End-to-end exactly-once processing across Kafka and an external sink still requires a sink/offset integration or an application-level idempotency and reconciliation design. Re-check defaults and constraints against the deployed client/broker version; see the [Kafka producer API](https://kafka.apache.org/40/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html) and [producer configuration](https://kafka.apache.org/40/generated/producer_config.html).
20
+
21
+ ## Consumer operations
22
+ Bound poll batches, processing concurrency, in-flight records, and retry duration. Preserve per-key ordering when parallelizing. Distinguish retryable transient errors from poison events; use bounded retries, delayed retry topics or queues, and a dead-letter workflow with alerting, ownership, replay controls, and audit. Avoid hot retry loops.
23
+
24
+ Monitor consumer lag and lag age, throughput, rebalance frequency, fetch latency, partition skew, under-replicated partitions, ISR shrink, disk capacity, request errors, produce latency, and broker/controller health. Lag alone is insufficient: measure the age of the oldest unprocessed business event.
25
+
26
+ ## Platform and topology
27
+ Design broker placement across zones, replication, rack awareness, storage, network throughput, quotas, TLS, authentication, ACLs, tenant isolation, schema registry, and control-plane availability. For Kafka, account for partition leadership, controller quorum, log retention, compaction, and replication. For Pulsar, account for brokers, BookKeeper storage/quorums, metadata store, namespace policies, and tiered storage. Verify implementation-specific semantics against the selected release's official docs.
28
+
29
+ ## Capacity and recovery
30
+ Estimate bytes/sec, peak multiplier, retention bytes, replication factor, compaction overhead, consumer fan-out, network egress, and recovery/replay throughput. Plan disk and network headroom for broker failure, partition reassignment, rolling upgrades, and consumer catch-up. Test broker loss, zone loss, disk pressure, network partition, credential expiry, schema incompatibility, and poison messages. Back up configuration and critical metadata; retained logs are not a substitute for application-level backup.
31
+
32
+ ## Decision gate
33
+ Use streaming when multiple consumers, durable replay, high sustained throughput, or event-time processing justify it. Use a task queue for work dispatch and a database outbox for reliable publication where appropriate. Document operational ownership, SLOs, upgrade policy, cost, and exit path.
@@ -0,0 +1,23 @@
1
+ # Event Sourcing and CQRS
2
+
3
+ Event Sourcing stores state changes as an append-only event history; CQRS separates command-side writes from query-side read models. They are independent patterns and should be adopted only when auditability, temporal reconstruction, multiple read projections, or domain complexity justify their costs.
4
+
5
+ ## Event-sourced aggregate design
6
+ Define aggregate boundaries around invariants and transactional consistency. Commands express intent; the aggregate validates business rules and emits domain events. Events describe facts in past tense and should be immutable. Persist the event append and expected stream version atomically using optimistic concurrency or an equivalent concurrency mechanism.
7
+
8
+ Specify event identity, aggregate ID, sequence/version, event type, schema version, occurred-at semantics, actor, correlation/causation, and payload. Avoid storing mutable derived state as if it were a fact. Model correction as compensating or superseding events rather than rewriting history, subject to legal deletion requirements.
9
+
10
+ ## Concurrency and idempotency
11
+ Use expected-version checks to prevent lost updates and conflicting commands. Define command idempotency keys and deduplicate retries. Handle duplicate delivery, concurrent append, and command timeout where the caller cannot know whether a write committed. Make command outcomes queryable by idempotency key when necessary.
12
+
13
+ ## CQRS projections
14
+ Build read models from the event log or change stream. Define projection ownership, ordering, checkpoint, rebuild, replay, lag, and versioning. Reads may be eventually consistent; state acceptable staleness and provide read-your-writes options when required. Rebuild into a new projection generation, validate counts/invariants, then switch traffic atomically. Do not let a projection become an accidental source of truth.
15
+
16
+ ## Schema evolution and event upcasting
17
+ Events live longer than application deployments. Use additive evolution where possible, explicit versioning, tolerant readers, upcasters, and compatibility tests against historical samples. Avoid changing the meaning of an existing field or event type. Keep a documented policy for retiring event versions and replaying old data.
18
+
19
+ ## Snapshots and retention
20
+ Snapshots accelerate aggregate rehydration but are disposable derived state; prove that replay from the event stream remains possible. Define snapshot version, consistency point, validation, and rebuild path. Event retention, archival, legal hold, deletion, and privacy requirements can conflict with immutability. Define a lawful data model before adopting event sourcing, including PII minimization, tokenization, encryption-key strategies, and deletion or redaction semantics.
21
+
22
+ ## Operational and organizational costs
23
+ Account for event-store availability, write contention, projection lag, replay storms, storage growth, schema governance, observability, debugging, and developer learning curve. Event sourcing is not a replacement for backups, and CQRS does not require microservices or separate databases. Prefer conventional transactional storage plus an outbox when the event history itself is not a product requirement.
@@ -0,0 +1,27 @@
1
+ # Change Data Capture and Debezium Architecture
2
+
3
+ Change Data Capture (CDC) streams committed database changes to downstream systems. It is useful for search indexes, analytics, cache invalidation, data integration, and migration, but it introduces log-retention, schema, ordering, and operational dependencies.
4
+
5
+ ## Source and connector model
6
+ Identify the source database, transaction log mechanism, connector, offset/checkpoint store, schema history, broker, and downstream consumers. For Debezium, verify connector-specific support and configuration for the exact database/version. Understand whether capture uses WAL, binlog, redo log, or another engine-specific mechanism.
7
+
8
+ CDC is not a substitute for domain events. Database change records expose persistence representation and may omit business intent. Use domain events for business facts and CDC for reliable propagation of committed storage changes where that coupling is acceptable.
9
+
10
+ ## Snapshot and streaming lifecycle
11
+ Define initial snapshot mode, snapshot consistency, table selection, incremental snapshot behavior, log position handoff, and recovery from connector restart. Ensure log retention is long enough to cover the maximum expected outage and catch-up time. If the source log is purged before the connector resumes, a new snapshot or controlled recovery may be required.
12
+
13
+ Model initial snapshot load separately from steady-state change throughput. Avoid overwhelming downstream systems with snapshot bursts. Throttle, stage, and monitor snapshot progress.
14
+
15
+ ## Ordering, transactions, and duplicates
16
+ Capture transaction metadata where available. Define ordering guarantees by source partition, table, key, and transaction; do not assume a global order across all tables or connectors. Consumers must handle duplicate delivery, restarts, and replay idempotently. Preserve delete/tombstone semantics and distinguish a deleted row from a null field.
17
+
18
+ For multi-table invariants, understand transaction boundaries and whether the connector emits transaction metadata. Downstream consumers should not expose partially applied business transactions as if they were atomic.
19
+
20
+ ## Schema and data governance
21
+ Plan schema evolution, type mapping, decimal/timestamp precision, primary-key changes, column filtering, masking, and compatibility. Prevent sensitive columns from entering topics by default. Define retention, access control, encryption, residency, lineage, and deletion propagation. CDC topics can leak PII and internal database structure.
22
+
23
+ ## Operations and failure modes
24
+ Monitor connector task state, source log lag, snapshot progress, queue/broker lag, offset persistence, schema history, heartbeat, error rate, and source database impact. Bound connector retries and handle poison records explicitly. Test source failover, log truncation, connector rebalance, schema incompatibility, network isolation, and destination outage.
25
+
26
+ ## Migration and dual-running
27
+ For database migration, define snapshot boundary, CDC catch-up, consistency validation, write ownership, cutover, rollback, and old-source retirement. Avoid uncoordinated dual writes. Use checksums, row counts, invariant queries, and sampled business-level comparisons. CDC replication alone does not prove semantic equivalence.
@@ -0,0 +1,24 @@
1
+ # Data Lakehouse and Stream/Batch Data Architecture
2
+
3
+ A lakehouse combines low-cost object storage with table formats, metadata/catalog services, and analytical query engines. Choose the architecture based on latency, concurrency, governance, workload isolation, and operational maturity—not the label.
4
+
5
+ ## Data layers and contracts
6
+ Define source systems, ingestion, raw/bronze, cleaned/silver, curated/gold, serving, and semantic layers. Assign ownership, schema, quality expectations, freshness SLO, lineage, classification, retention, and access controls to each dataset. Distinguish raw immutable capture from curated business meaning.
7
+
8
+ Use open table formats where interoperability, schema evolution, partition evolution, snapshots, and transactional writes matter. Compare the capabilities and ecosystem of Apache Iceberg, Delta Lake, and Apache Hudi for the target engine and deployment. Validate catalog, locking, concurrency, compaction, and metadata behavior for chosen versions.
9
+
10
+ ## Batch and streaming
11
+ Use batch for bounded periodic processing and streaming for low-latency continuous updates. A unified API does not erase differences in event-time semantics, late arrivals, replay, state, watermarking, exactly-once scope, and cost. Define freshness requirements before choosing the processing mode.
12
+
13
+ For stream processing, model event time vs processing time, watermarks, late data, state backend, checkpoint interval, savepoints, backpressure, and replay. For batch, define partition sizing, incremental processing, idempotent writes, retries, and backfill strategy. Prevent concurrent writers from corrupting table metadata or producing small-file storms.
14
+
15
+ ## Data quality and governance
16
+ Define schema contracts, nullability, uniqueness, referential checks, freshness, volume, distribution, and business-rule tests. Quarantine bad records with reason, owner, and replay path. Maintain lineage from source to report/model. Apply column- and row-level authorization, masking, encryption, audit, and purpose-based access. Catalog metadata is not itself an access-control boundary.
17
+
18
+ ## Performance and cost
19
+ Optimize partitioning for query predicates and data distribution; avoid over-partitioning and high-cardinality directory layouts. Plan file sizes, compaction, clustering, statistics, metadata cleanup, cache, and query concurrency. Monitor scan bytes, small files, shuffle, spill, skew, queue wait, freshness, and compute utilization.
20
+
21
+ Include object storage requests, data transfer, catalog, compute, orchestration, observability, retention, compaction, and egress in TCO. Separate interactive BI, ETL, ad hoc exploration, and ML workloads where noisy-neighbor risks justify workload isolation.
22
+
23
+ ## Recovery and portability
24
+ Define reproducible pipeline code, versioned datasets, checkpoints, table snapshots, catalog recovery, object versioning, and restore testing. Do not assume object storage durability alone recovers catalog metadata or application semantics. Test region failure, accidental overwrite, schema breakage, late data, and replay.
@@ -0,0 +1,28 @@
1
+ # Workflow Orchestration and Durable Execution
2
+
3
+ Workflow orchestration coordinates long-running business processes across services, human tasks, timers, and retries. Choose between a task queue, scheduler, state machine, and durable workflow engine based on process duration, recovery, visibility, and compensation needs.
4
+
5
+ ## Scope and boundaries
6
+
7
+ This document owns durable business-process orchestration: workflow state, deterministic replay, activity execution, timers, signals, cancellation, versioning, and workflow-engine operations. Use [asynchronous processing](05-async-and-messaging.md) for queue and broker delivery semantics, [consensus and distributed transactions](37-consensus-and-distributed-transactions.md) for cross-resource atomicity and saga guarantees, and [incident management](61-incident-management-and-postmortems.md) for operational response. A workflow engine's durable history does not make an external activity exactly once.
8
+
9
+ ## Workflow model
10
+ Represent the process as explicit states and transitions. Define start conditions, inputs, outputs, deadlines, cancellation, pause/resume, human approval, terminal states, and invariants. Separate workflow definition from activity implementation. Give each workflow a stable ID and each activity a deterministic or unique execution identity.
11
+
12
+ Use a durable engine such as Temporal when workflows must survive process restarts, wait for external events or long timers, coordinate retries, and expose execution history. Compare with cloud-native state machines, queue-based workers, and application-managed orchestration. Verify SDK, server, and deployment-specific semantics for the selected version.
13
+
14
+ ## Durable execution and determinism
15
+ Workflow code may be replayed to reconstruct state. Keep nondeterministic I/O, wall-clock access, random values, and external side effects inside activities or supported workflow APIs. Version workflow definitions safely; old executions may continue while new code is deployed. Define worker compatibility and retirement policy.
16
+
17
+ Activities should be idempotent or protected by idempotency keys because timeouts can leave completion uncertain. Use heartbeats for long-running work, bounded retries with backoff, deadlines, cancellation propagation, and compensation where business semantics require it. A timeout does not prove that an external side effect did not happen.
18
+
19
+ ## State, signals, and versioning
20
+ Define workflow state retention, payload limits, search attributes, signal/update semantics, event history growth, continue-as-new, and archival. Keep large business data in durable domain storage and reference it from workflow state. Protect history and payloads as sensitive data. Define encryption, access, retention, and deletion behavior.
21
+
22
+ ## Scaling and operations
23
+ Scale workers by task queue backlog and activity capacity, not only CPU. Bound concurrency per dependency and tenant. Separate task queues by workload or trust boundary where useful. Monitor schedule-to-start, execution latency, task failures, retries, timeouts, stuck workflows, poller capacity, history size, and dependency saturation.
24
+
25
+ Design engine cluster availability, persistence, namespace/tenant isolation, visibility store, backups, upgrades, and disaster recovery. Test worker loss, engine failover, dependency outage, poison activities, duplicate signals, cancellation races, and version skew.
26
+
27
+ ## Decision gate
28
+ Do not use a durable workflow engine for a simple cron task or short request-response flow. Account for operational ownership, learning curve, visibility benefits, replay constraints, and migration/exit cost. Prefer explicit state machines over hidden orchestration spread across callbacks and queues.
@@ -0,0 +1,22 @@
1
+ # Schema Registry and Event Governance
2
+
3
+ Event governance makes independently deployed producers and consumers interoperable over time. A schema registry helps manage schema identity and compatibility; it does not replace ownership, data contracts, authorization, or consumer testing.
4
+
5
+ ## Contract design
6
+ Define event name, semantic meaning, owner, key, payload schema, required/optional fields, units, timestamp semantics, PII classification, retention, and consumer expectations. Use stable names and explicit schema versions. Distinguish additive fields from semantic changes. Do not repurpose a field or change units without a migration plan.
7
+
8
+ ## Compatibility policy
9
+ Choose backward, forward, or full compatibility based on producer/consumer deployment order and replay requirements. Test compatibility against all supported historical versions and representative consumers. Compatibility checks are necessary but insufficient: a schema can remain structurally compatible while business meaning changes.
10
+
11
+ Plan expand-migrate-contract: publish additive schema, update consumers to tolerate both forms, migrate producers, measure adoption, then retire old fields only after the compatibility window and consumer inventory are satisfied. Version event semantics deliberately rather than blindly incrementing every payload.
12
+
13
+ ## Registry architecture and operations
14
+ Define registry availability, replication, authentication, authorization, tenancy, subject naming, schema ID behavior, cache behavior, soft/hard deletion, and backup/restore. Producers and consumers need a documented response to registry unavailability; cached schema IDs may help but cannot be assumed indefinitely.
15
+
16
+ Use least-privilege subject permissions and protect registry APIs. Monitor compatibility failures, unknown schema IDs, serialization errors, consumer parse failures, and orphaned schemas. Define ownership and lifecycle for unused subjects and old versions.
17
+
18
+ ## Data governance and evolution
19
+ Document data classification, PII minimization, encryption, retention, deletion propagation, lineage, and regulatory obligations. Avoid secrets and unneeded personal data in event payloads. Schema metadata should link to source owner, producer, consumers, and business glossary. Use contract tests and sampled production payload validation to catch semantic drift.
20
+
21
+ ## Rollout checklist
22
+ Inventory producers and consumers; define compatibility policy; register and review schema; deploy tolerant consumers; roll producers gradually; monitor failures and lag; verify replay from retained data; deprecate old fields with notice; remove only after all consumer and archival constraints are satisfied.
@@ -0,0 +1,45 @@
1
+ # PostgreSQL Internals and Performance
2
+
3
+ ## Purpose and decision context
4
+ Use this guide when designing or diagnosing PostgreSQL-backed systems. Confirm the deployed major version, extensions, hosting model, workload, and operational constraints before applying version-specific advice.
5
+
6
+ ## Storage and execution model
7
+ - Explain heap pages, tuples, TOAST, visibility metadata, WAL, checkpoints, background writer, and vacuum as a connected lifecycle.
8
+ - Distinguish heap from index storage. Indexes accelerate selected access paths but increase write amplification, storage, cache pressure, and maintenance.
9
+ - Use EXPLAIN (ANALYZE, BUFFERS) on representative queries. Inspect estimated versus actual rows, loops, join strategy, sort/hash spills, buffer hits/reads, and planning/execution time.
10
+ - Treat ANALYZE statistics as workload-sensitive estimates. Check stale statistics, skew, correlation, and extended statistics before forcing planner settings.
11
+ - Evaluate B-tree, GIN, GiST, BRIN, partial, and expression indexes against predicates, ordering, selectivity, write rate, and maintenance cost.
12
+
13
+ ## MVCC, transactions, and vacuum
14
+ PostgreSQL MVCC retains row versions so readers and writers can often proceed concurrently. Updates and deletes create dead tuples; vacuum makes space reusable and maintains visibility information, but does not normally shrink relation files for the operating system.
15
+ - Explain READ COMMITTED, REPEATABLE READ, and SERIALIZABLE semantics, including serialization failures and retry requirements.
16
+ - PostgreSQL's REPEATABLE READ is snapshot isolation: it prevents the standard phantom-read phenomenon but can still permit serialization anomalies. SERIALIZABLE adds Serializable Snapshot Isolation checks and may abort transactions with SQLSTATE `40001`; retry the entire transaction, not only the failed statement. Treat this as versioned PostgreSQL behavior and verify the deployed major release in the [official transaction isolation documentation](https://www.postgresql.org/docs/current/transaction-iso.html).
17
+ - Sequence changes (including `nextval`) are not rolled back with the surrounding transaction; do not use gap-free sequence values as a transactional business invariant.
18
+ - Monitor transaction age, long-running snapshots, idle-in-transaction sessions, dead tuples, autovacuum progress, bloat indicators, and replication slots retaining WAL.
19
+ - Set autovacuum thresholds and scale factors based on table size and churn; do not disable autovacuum globally.
20
+ - Diagnose lock waits and deadlocks with lock graphs and transaction traces. Keep transactions short and acquire locks consistently.
21
+ - Use row locks or advisory locks only when lock scope and contention are explicit.
22
+
23
+ ## Connection and workload management
24
+ - Size pools from database capacity, query concurrency, CPU, memory, and latency—not application replica count alone.
25
+ - Consider PgBouncer transaction pooling only after checking session state, prepared statements, temporary tables, advisory locks, and other session-affine behavior.
26
+ - Separate OLTP, analytical, maintenance, and migration workloads where contention warrants it.
27
+ - Use statement and lock timeouts, bounded queues, cancellation, and workload priorities.
28
+
29
+ ## Replication and operations
30
+ Distinguish physical streaming replication from logical replication. Document synchronous commit policy, replica lag, promotion/fencing, slot lifecycle, failover authority, and read-after-write behavior. A replica is not a backup.
31
+ Define tested backup/PITR procedures, WAL retention, restore time, major-version upgrade path, extension compatibility, and rollback constraints.
32
+
33
+ ## Diagnostic checklist
34
+ 1. Capture query plans with realistic parameters and data volume.
35
+ 2. Check CPU, memory, storage latency/IOPS, cache behavior, WAL rate, checkpoints, and connection saturation.
36
+ 3. Inspect lock waits, deadlocks, long transactions, vacuum lag, and replication lag.
37
+ 4. Change one variable at a time; compare p95/p99 latency and write amplification.
38
+ 5. Record version, schema, indexes, workload, and rollback plan.
39
+
40
+ ## Anti-patterns
41
+ - Adding indexes without measuring write and storage costs.
42
+ - Treating VACUUM FULL as routine maintenance; it requires stronger locking and extra space.
43
+ - Increasing max_connections without resource and pool analysis.
44
+ - Using planner settings to mask incorrect statistics or query shape.
45
+ - Assuming replica reads are current or failover is lossless without a documented durability policy.