system-design-primer-skill 0.0.0-stage → 2.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (112) hide show
  1. package/ATTRIBUTION.md +10 -0
  2. package/LICENSE +7 -0
  3. package/README.md +180 -2
  4. package/SKILL.md +447 -0
  5. package/VERSION +1 -0
  6. package/package.json +78 -4
  7. package/references/00-primer-map-and-attribution.md +29 -0
  8. package/references/01-requirements-and-capacity.md +107 -0
  9. package/references/02-architecture-and-networking.md +94 -0
  10. package/references/03-data-and-storage.md +88 -0
  11. package/references/04-caching.md +78 -0
  12. package/references/05-async-and-messaging.md +77 -0
  13. package/references/06-distributed-systems.md +59 -0
  14. package/references/07-reliability-and-resilience.md +71 -0
  15. package/references/08-api-security-observability.md +62 -0
  16. package/references/09-modern-engineering.md +58 -0
  17. package/references/10-workflows-and-checklists.md +97 -0
  18. package/references/11-case-studies.md +74 -0
  19. package/references/12-ai-systems-architecture.md +139 -0
  20. package/references/13-architecture-intelligence.md +91 -0
  21. package/references/14-api-and-data-contracts.md +80 -0
  22. package/references/15-failure-analysis.md +84 -0
  23. package/references/16-multi-agent-architecture.md +76 -0
  24. package/references/17-agent-harness-and-runtime.md +87 -0
  25. package/references/18-mcp-architecture.md +67 -0
  26. package/references/19-agent-security-and-permissions.md +74 -0
  27. package/references/20-agent-state-and-memory.md +72 -0
  28. package/references/21-agent-evaluation-and-observability.md +90 -0
  29. package/references/22-kubernetes-and-container-platforms.md +29 -0
  30. package/references/23-service-mesh-and-traffic-management.md +21 -0
  31. package/references/24-multi-region-and-disaster-recovery.md +24 -0
  32. package/references/25-platform-engineering-and-developer-experience.md +26 -0
  33. package/references/26-finops-and-cloud-cost.md +29 -0
  34. package/references/27-production-observability-and-slo.md +29 -0
  35. package/references/28-event-streaming-and-kafka-pulsar.md +27 -0
  36. package/references/29-event-sourcing-and-cqrs.md +23 -0
  37. package/references/30-change-data-capture-and-debezium.md +27 -0
  38. package/references/31-data-lakehouse-and-stream-batch.md +24 -0
  39. package/references/32-workflow-orchestration-and-temporal.md +24 -0
  40. package/references/33-schema-registry-and-event-governance.md +22 -0
  41. package/references/34-postgresql-internals-and-performance.md +43 -0
  42. package/references/35-mysql-innodb-and-performance.md +27 -0
  43. package/references/36-sharding-and-distributed-sql.md +22 -0
  44. package/references/37-consensus-and-distributed-transactions.md +25 -0
  45. package/references/38-search-architecture-elasticsearch-opensearch.md +24 -0
  46. package/references/39-vector-databases-and-hybrid-retrieval.md +31 -0
  47. package/references/40-zero-downtime-database-migrations.md +27 -0
  48. package/references/41-api-gateway-and-edge-architecture.md +23 -0
  49. package/references/42-rest-api-design-and-lifecycle.md +23 -0
  50. package/references/43-graphql-architecture-and-governance.md +16 -0
  51. package/references/44-grpc-and-service-contracts.md +16 -0
  52. package/references/45-websocket-sse-and-realtime-systems.md +16 -0
  53. package/references/46-realtime-collaboration-ot-crdt.md +16 -0
  54. package/references/47-api-security-rate-limiting-and-abuse.md +16 -0
  55. package/references/48-oauth-oidc-and-passkeys.md +20 -0
  56. package/references/49-iam-rbac-abac-rebac.md +16 -0
  57. package/references/50-zero-trust-and-workload-identity.md +16 -0
  58. package/references/51-secrets-kms-pki-and-key-management.md +16 -0
  59. package/references/52-supply-chain-security-sbom-slsa.md +16 -0
  60. package/references/53-threat-modeling-stride-and-attack-trees.md +16 -0
  61. package/references/54-privacy-engineering-and-data-governance.md +16 -0
  62. package/references/55-opentelemetry-and-instrumentation.md +50 -0
  63. package/references/56-distributed-tracing-and-context-propagation.md +43 -0
  64. package/references/57-continuous-profiling-and-performance-diagnostics.md +45 -0
  65. package/references/58-load-stress-soak-testing-and-workload-modeling.md +44 -0
  66. package/references/59-capacity-planning-and-performance-modeling.md +34 -0
  67. package/references/60-chaos-engineering-and-resilience-experiments.md +41 -0
  68. package/references/61-incident-management-and-postmortems.md +34 -0
  69. package/references/62-dns-and-service-discovery.md +35 -0
  70. package/references/63-load-balancing-and-global-traffic-management.md +31 -0
  71. package/references/64-cdn-and-edge-caching.md +34 -0
  72. package/references/65-cloud-networking-and-connectivity.md +34 -0
  73. package/references/66-edge-computing-architecture.md +33 -0
  74. package/references/67-storage-infrastructure-and-storage-systems.md +30 -0
  75. package/references/68-tcp-quic-and-network-performance.md +27 -0
  76. package/references/69-task-routing-and-evidence.md +58 -0
  77. package/scripts/install.ps1 +124 -0
  78. package/scripts/install.sh +105 -0
  79. package/scripts/validate.py +376 -0
  80. package/templates/adr-template.md +32 -0
  81. package/templates/agent-evaluation-plan-template.md +57 -0
  82. package/templates/agent-security-review-template.md +46 -0
  83. package/templates/agent-system-design-template.md +74 -0
  84. package/templates/api-architecture-design-template.md +33 -0
  85. package/templates/api-security-and-rate-limit-review-template.md +29 -0
  86. package/templates/architecture-decision-matrix-template.md +48 -0
  87. package/templates/architecture-design-template.md +95 -0
  88. package/templates/architecture-review-scorecard-template.md +36 -0
  89. package/templates/architecture-review-template.md +52 -0
  90. package/templates/cdc-data-pipeline-template.md +41 -0
  91. package/templates/cdn-and-edge-strategy-template.md +36 -0
  92. package/templates/chaos-experiment-template.md +39 -0
  93. package/templates/cloud-cost-model-template.md +37 -0
  94. package/templates/database-architecture-review-template.md +33 -0
  95. package/templates/edge-computing-readiness-template.md +36 -0
  96. package/templates/event-sourcing-cqrs-template.md +41 -0
  97. package/templates/event-streaming-design-template.md +47 -0
  98. package/templates/failure-analysis-template.md +36 -0
  99. package/templates/identity-and-access-architecture-template.md +29 -0
  100. package/templates/incident-postmortem-template.md +39 -0
  101. package/templates/multi-region-disaster-recovery-template.md +43 -0
  102. package/templates/network-architecture-design-template.md +38 -0
  103. package/templates/observability-architecture-template.md +52 -0
  104. package/templates/performance-test-plan-template.md +52 -0
  105. package/templates/privacy-data-governance-template.md +24 -0
  106. package/templates/production-readiness-checklist-template.md +52 -0
  107. package/templates/realtime-system-design-template.md +28 -0
  108. package/templates/security-threat-model-template.md +26 -0
  109. package/templates/sharding-distributed-sql-design-template.md +34 -0
  110. package/templates/storage-architecture-design-template.md +35 -0
  111. package/templates/supply-chain-security-review-template.md +20 -0
  112. package/templates/zero-downtime-database-migration-template.md +32 -0
@@ -0,0 +1,76 @@
1
+ # Multi-agent architecture and orchestration
2
+
3
+ Use multiple agents only when the workload benefits from bounded specialization, independent context, parallel execution, or distinct trust boundaries. A multi-agent design is not inherently more capable or reliable than a single agent or deterministic workflow.
4
+
5
+ ## 1. Start with the baseline
6
+
7
+ Compare:
8
+ - Deterministic workflow or state machine.
9
+ - Single agent with a bounded toolset.
10
+ - Single agent with specialist tools or structured subroutines.
11
+ - Multi-agent orchestration.
12
+
13
+ Estimate coordination cost: additional model calls, context duplication, message volume, retries, synchronization, evaluation complexity, and failure surface. Prefer the least complex design that meets quality, latency, cost, and safety objectives.
14
+
15
+ ## 2. Common topologies
16
+
17
+ | Topology | Suitable when | Main risks / controls |
18
+ |---|---|---|
19
+ | Sequential pipeline | Work has clear ordered stages and stable handoffs | Error propagation; validate each stage output |
20
+ | Router / specialist | Requests fall into separable domains | Misrouting; confidence threshold and safe fallback |
21
+ | Supervisor / worker | A coordinator can decompose tasks and integrate bounded results | Supervisor bottleneck, runaway delegation; cap depth/fan-out and validate outputs |
22
+ | Parallel fan-out / aggregation | Independent subtasks can run concurrently | Cost amplification, inconsistent evidence; bound concurrency and define merge rules |
23
+ | Peer collaboration | Agents need iterative critique or negotiation | Non-termination and consensus theater; cap rounds and require evidence |
24
+ | Hierarchical delegation | Large tasks have stable subdomains and ownership | Deep call trees and opaque accountability; enforce depth and trace parent-child runs |
25
+
26
+ Choose topology based on dependency structure and task semantics, not fashion. A graph of agents must have explicit termination and cycle controls.
27
+
28
+ ## 3. Agent contracts
29
+
30
+ For every agent define:
31
+ - Stable role and responsibility; explicit non-goals.
32
+ - Input schema, output schema, and validation rules.
33
+ - Allowed tools, data scopes, and permissions.
34
+ - Context sources and trust classification.
35
+ - Token, time, step, concurrency, and cost budgets.
36
+ - Success criteria, stop conditions, and escalation behavior.
37
+ - Owner, version, and compatibility policy.
38
+
39
+ Treat peer outputs as untrusted proposals. Validate facts, schemas, citations, and permissions at the receiving boundary. Do not let one agent grant another agent authority by assertion.
40
+
41
+ ## 4. Orchestration and state
42
+
43
+ Represent long-running work as an explicit workflow with run ID, parent/child task IDs, state transitions, deadlines, cancellation, and durable checkpoints. Separate orchestration state from model conversation context. Persist task ownership and result provenance.
44
+
45
+ Define:
46
+ - Dispatch, queueing, backpressure, concurrency and tenant quotas.
47
+ - Correlation IDs and causality across parent/child traces.
48
+ - Duplicate delivery and idempotent task creation.
49
+ - Partial success, retries, timeout, cancellation, and compensation.
50
+ - Result aggregation, conflict resolution, evidence precedence, and unresolved disagreement.
51
+ - Human approval and handoff points.
52
+
53
+ Use deterministic code for critical transitions, budget enforcement, authorization, and side effects. Model-generated plans may propose transitions but must not bypass the workflow state machine.
54
+
55
+ ## 5. Consistency and coordination
56
+
57
+ Avoid shared mutable agent memory as an implicit coordination mechanism. Prefer explicit messages, versioned artifacts, or a workflow-owned state store. Define the source of truth and concurrency strategy for shared artifacts. Use optimistic concurrency/version checks when multiple workers can update the same task. Record provenance and distinguish observed facts, inferences, and hypotheses.
58
+
59
+ For parallel work, define merge semantics before fan-out. Examples include set union with provenance, deterministic conflict resolution, human review, or rejecting incompatible outputs. Do not silently select the most confident-sounding response.
60
+
61
+ ## 6. Reliability and cost controls
62
+
63
+ Bound fan-out, nesting depth, rounds, tokens, wall-clock duration, queue age, and total spend. Apply admission control, deadlines, cancellation propagation, retry budgets, and per-tenant quotas. A failed child task should have a defined retry, fallback, partial-result, or escalation path. Prevent retries from creating duplicate external effects.
64
+
65
+ Measure end-to-end task success and total orchestration cost—not only each agent's local completion rate.
66
+
67
+ ## 7. Review checklist
68
+
69
+ - [ ] A single-agent/deterministic baseline was considered.
70
+ - [ ] Each agent has a bounded role, contract, owner, and permission set.
71
+ - [ ] Topology and dependency graph have explicit termination rules.
72
+ - [ ] Fan-out, depth, concurrency, time, tokens, and cost are bounded.
73
+ - [ ] State ownership, message correlation, and conflict resolution are explicit.
74
+ - [ ] Partial success, cancellation, retry, and human escalation are designed.
75
+ - [ ] Child outputs are validated and retain provenance.
76
+ - [ ] Evaluation covers end-to-end tasks and coordination failures.
@@ -0,0 +1,87 @@
1
+ # Agent harness and runtime architecture
2
+
3
+ An agent harness is the deterministic application/runtime layer that hosts a model-driven loop, exposes capabilities, enforces policy, manages state, and controls execution. The model is a probabilistic component inside this system—not the runtime, policy engine, or source of truth.
4
+
5
+ ## 1. Reference execution loop
6
+
7
+ A robust loop typically performs:
8
+
9
+ 1. Receive a request and authenticate the principal.
10
+ 2. Load authorized context and durable workflow state.
11
+ 3. Construct bounded model context from versioned instructions, relevant data, and tool schemas.
12
+ 4. Request a model response under explicit token, latency, and cost limits.
13
+ 5. Parse and validate the response against an expected schema.
14
+ 6. If a tool action is proposed, resolve the tool and authorize the specific action.
15
+ 7. Execute within a bounded runtime and validate the result.
16
+ 8. Persist state/checkpoint and append an auditable event.
17
+ 9. Continue, return a result, request approval, or stop according to deterministic policy.
18
+
19
+ The model may suggest an action; only the application decides whether it is permitted and whether the workflow can advance.
20
+
21
+ ## 2. Runtime components
22
+
23
+ | Component | Responsibility |
24
+ |---|---|
25
+ | Request/session gateway | Authentication, tenant scope, request limits, cancellation |
26
+ | Orchestrator / state machine | Run lifecycle, transitions, deadlines, retries, human handoff |
27
+ | Model adapter | Provider abstraction, structured output, timeout, quota and routing |
28
+ | Context builder | Authorized retrieval, token budgeting, provenance and redaction |
29
+ | Tool registry | Typed schemas, versions, ownership, permission metadata |
30
+ | Policy decision point | Principal/resource/action authorization and approvals |
31
+ | Tool executor | Sandboxing, network/filesystem constraints, secrets broker |
32
+ | State store | Durable run state, checkpoints, idempotency and event history |
33
+ | Telemetry pipeline | Traces, metrics, audit events, redaction and retention |
34
+
35
+ Keep policy evaluation and execution separate. A tool registry entry is not itself permission to call the tool.
36
+
37
+ ## 3. Control plane and execution plane
38
+
39
+ The control plane owns identity, configuration, tool registration, policy, budgets, scheduling, and audit. The execution plane runs model calls and tools in isolated workers. Separate them when the threat model, scale, or operational requirements justify it.
40
+
41
+ Use workload identity and short-lived scoped credentials. Do not pass broad platform credentials into model context or untrusted tool processes. Apply network egress allowlists, filesystem boundaries, CPU/memory/process limits, and execution timeouts appropriate to the tool.
42
+
43
+ ## 4. Context assembly
44
+
45
+ Treat context as a derived, bounded view—not durable storage. Define precedence and provenance for system policy, developer configuration, user request, retrieved evidence, conversation summary, and tool results. Keep untrusted content clearly delimited and never elevate retrieved text into trusted instructions.
46
+
47
+ Budget context explicitly:
48
+ - Fixed instruction and tool-schema overhead.
49
+ - Current user request and relevant conversation.
50
+ - Authorized retrieved evidence and state summary.
51
+ - Reserved output and tool-call capacity.
52
+ - Safety margin for tokenizer/provider variation.
53
+
54
+ Summarize with provenance and retain links to source records. Re-check authorization when data is retrieved and before side effects.
55
+
56
+ ## 5. Tool execution contract
57
+
58
+ Every tool needs a stable ID/version, owner, typed input/output schema, authentication model, permission scope, timeout, resource budget, side-effect classification, idempotency strategy, and audit policy.
59
+
60
+ Validate arguments server-side. Reject unknown or excessive fields, unsafe paths/URLs, oversized payloads, invalid identifiers, and unbounded queries. Validate outputs before returning them to the model. Use structured errors that distinguish validation, authorization, rate limit, timeout, transient failure, and ambiguous outcome.
61
+
62
+ ## 6. Stop conditions and budgets
63
+
64
+ Enforce deterministic maximums for:
65
+ - Model turns and tool calls per run.
66
+ - Wall-clock deadline and per-call timeout.
67
+ - Input/output tokens and total cost.
68
+ - Tool concurrency, recursion/delegation depth, and queue wait.
69
+ - Retry count and retry budget.
70
+ - Output size and retained state.
71
+
72
+ Stop on completion, denial, budget exhaustion, cancellation, deadline, repeated invalid output, or policy-defined escalation. Do not allow the model to override these controls through a tool argument or prompt.
73
+
74
+ ## 7. Runtime resilience
75
+
76
+ Use bounded queues, backpressure, worker isolation, circuit breakers for failing providers, cancellation propagation, and explicit unknown/pending states for ambiguous external effects. Persist checkpoints before and after meaningful side effects. Use idempotency keys and reconciliation rather than blindly repeating uncertain actions.
77
+
78
+ ## 8. Review checklist
79
+
80
+ - [ ] Application code owns the execution loop and state transitions.
81
+ - [ ] Authentication and per-action authorization precede tool execution.
82
+ - [ ] Context is bounded, authorized, versioned, and provenance-aware.
83
+ - [ ] Tool schemas and outputs are validated.
84
+ - [ ] Execution is isolated and resource/network limits are enforced.
85
+ - [ ] Budgets, deadlines, cancellation, and stop conditions are deterministic.
86
+ - [ ] Checkpoints, idempotency, and ambiguous outcomes are handled.
87
+ - [ ] Telemetry and audit are redacted and access-controlled.
@@ -0,0 +1,67 @@
1
+ # Model Context Protocol (MCP) architecture
2
+
3
+ Use MCP as a protocol boundary for connecting AI applications to external capabilities and context. Treat protocol support, server implementation, and authorization policy as separate concerns. MCP does not automatically make a tool safe or authorize a caller.
4
+
5
+ ## 1. Roles and boundaries
6
+
7
+ Model the main roles explicitly:
8
+ - **Host application:** owns the user experience, policy, model orchestration, and trust decisions.
9
+ - **MCP client:** manages a protocol connection to a server and negotiates supported capabilities.
10
+ - **MCP server:** exposes declared capabilities and mediates access to underlying resources or services.
11
+
12
+ A host may manage multiple client/server connections. Each connection is a separate trust and failure boundary. Do not assume that a server can authenticate the end user merely because a host connected to it.
13
+
14
+ ## 2. Capability categories
15
+
16
+ | Capability | Architectural responsibility | Security considerations |
17
+ |---|---|---|
18
+ | Tools | Expose callable operations | Treat as potential side effects; authorize each invocation and validate arguments |
19
+ | Resources | Expose addressable context/data | Enforce access control, tenant scope, freshness, and data minimization |
20
+ | Prompts | Provide reusable prompt templates | Treat content as untrusted configuration unless reviewed and versioned |
21
+ | Sampling / elicitation (where supported) | Request model interaction or user input through the host | Keep user consent, policy, and host authority explicit |
22
+
23
+ Negotiate and inspect capabilities rather than assuming every server implements every feature. Verify the exact protocol revision and supported capability behavior against current official MCP documentation.
24
+
25
+ ## 3. Transport and session architecture
26
+
27
+ Choose transport based on deployment and trust boundaries. Local process transports introduce executable, filesystem, environment, and process-lifecycle risks. Network transports introduce endpoint authentication, TLS, network egress, session lifecycle, and proxy risks.
28
+
29
+ Define connection establishment, initialization, capability negotiation, request correlation, timeouts, cancellation, reconnection, and shutdown. Treat session identifiers as protocol state, not as proof of user identity. Avoid sharing a privileged server connection across tenants unless isolation and authorization are explicitly enforced.
30
+
31
+ ## 4. Authorization model
32
+
33
+ For each operation evaluate:
34
+ - Authenticated principal and tenant.
35
+ - Requested tool/resource and specific action.
36
+ - Object-level authorization and data classification.
37
+ - Scope, rate limit, budget, and approval requirement.
38
+ - Credential audience, expiry, and revocation.
39
+
40
+ Use delegated or workload identity with narrow scopes. Avoid ambient credentials, static broad tokens, and passing secrets through prompts or tool arguments. The host should mediate user consent where required; the server must still enforce its own access policy.
41
+
42
+ ## 5. Reliability and operations
43
+
44
+ Set per-server concurrency, request size, timeouts, retry budgets, and circuit breakers. Distinguish protocol errors from business errors and ambiguous side-effect outcomes. Make mutating operations idempotent where possible and provide operation IDs/status lookup for long-running work.
45
+
46
+ Observe server identity/version, negotiated capabilities, transport health, request latency, error classes, authorization decisions, and tool outcomes. Redact sensitive arguments and resource content. Define server upgrade, compatibility, rollback, and decommissioning procedures.
47
+
48
+ ## 6. Threat model
49
+
50
+ Consider malicious or compromised servers, tool-description prompt injection, confused-deputy attacks, SSRF, local process compromise, path traversal, credential theft, cross-tenant resource exposure, oversized responses, and denial of service. Treat tool descriptions and returned content as untrusted data. Keep protocol discovery separate from permission grants.
51
+
52
+ ## 7. Implementation guidance
53
+
54
+ Pin and review server packages and launch configuration. Restrict local executable paths and environment variables. For remote servers, enforce endpoint allowlists, TLS, authentication, DNS/IP validation where relevant, and outbound network policy. Validate every argument and result at the host/server boundary. Do not assume a protocol handshake is a security review.
55
+
56
+ Protocol details evolve. Before implementing a client or server, consult the official MCP specification and SDK documentation for the exact protocol version, transport, authorization, and capability semantics in use.
57
+
58
+ ## 8. Review checklist
59
+
60
+ - [ ] Host, client, and server responsibilities are explicit.
61
+ - [ ] Capabilities are negotiated and reviewed rather than blindly trusted.
62
+ - [ ] Every tool call and resource read is authorized in the correct tenant/object scope.
63
+ - [ ] Transport, process, endpoint, and credential risks are controlled.
64
+ - [ ] Tool schemas, payload sizes, timeouts, concurrency, and retries are bounded.
65
+ - [ ] Side effects are idempotent or have reconciliation/status mechanisms.
66
+ - [ ] Tool descriptions and results are treated as untrusted content.
67
+ - [ ] Version compatibility, audit, telemetry, and decommissioning are planned.
@@ -0,0 +1,74 @@
1
+ # Agent security, permissions, and isolation
2
+
3
+ An agent is not a security principal with inherent authority. The application must bind each action to an authenticated principal, tenant, resource, and policy decision. Prompt instructions and model refusals are not authorization controls.
4
+
5
+ ## 1. Threat model
6
+
7
+ Include:
8
+ - Direct and indirect prompt injection in user input, files, retrieved documents, web pages, and tool results.
9
+ - Confused-deputy behavior where an agent uses its host's authority for an untrusted requester.
10
+ - Excessive tool permissions, privilege escalation, and cross-tenant access.
11
+ - SSRF, DNS rebinding, path traversal, unsafe deserialization, command/code execution, and malicious file content.
12
+ - Secret leakage through prompts, traces, errors, generated artifacts, or tool output.
13
+ - Data exfiltration, unsafe outbound communication, and covert channels.
14
+ - Compromised MCP servers, dependencies, plugins, model providers, and tool supply chain.
15
+ - Denial of service through recursion, fan-out, large context, expensive calls, or unbounded tool loops.
16
+
17
+ ## 2. Principal and authorization model
18
+
19
+ Represent the effective authority for an operation as the intersection of:
20
+ - Authenticated end-user or initiating service identity.
21
+ - Tenant and organization scope.
22
+ - Agent/workflow identity and delegated scopes.
23
+ - Tool/service identity and resource permissions.
24
+ - Current policy, consent, approval, and environmental constraints.
25
+
26
+ Authorize each action at execution time, not only when the session begins. Enforce object-level checks at the service that owns the resource. Re-check authorization after long waits or privilege changes. Never trust model-supplied user IDs, tenant IDs, role claims, file paths, or URLs.
27
+
28
+ ## 3. Least privilege and capability design
29
+
30
+ Prefer narrow tools that perform one business operation over generic shell, SQL, browser, or filesystem access. Use explicit allowlists for operations, resources, domains, and paths. Scope credentials by audience, tenant, action, and lifetime. Use short-lived tokens and a secrets broker; do not expose raw credentials to model context.
31
+
32
+ Separate read, draft, and commit capabilities. A workflow that can inspect a payment should not automatically be able to issue a refund. Keep high-impact actions behind independent policy checks and approval gates.
33
+
34
+ ## 4. Human approval
35
+
36
+ Require explicit approval for irreversible, financial, legal, privacy-sensitive, externally visible, or privilege-changing actions according to risk. Bind approval to a canonical action summary, target resource, parameters, and expiry. Reject stale or modified approvals. Record approver identity, decision, timestamp, and the exact action authorized.
37
+
38
+ Human approval does not replace server-side validation or transaction controls.
39
+
40
+ ## 5. Isolation and execution safety
41
+
42
+ Run untrusted tools and generated code in isolated workers or sandboxes. Apply CPU, memory, process, filesystem, network egress, and wall-clock limits. Use read-only mounts and ephemeral workspaces where possible. Separate tenant workloads and prevent cross-run state leakage. Restrict environment variables and inherited file descriptors. Do not mount host credentials or container sockets into untrusted execution.
43
+
44
+ ## 6. Data protection and audit
45
+
46
+ Classify context and outputs. Minimize data before model/provider transmission; apply tenant filters before retrieval results enter context. Define retention, deletion, residency, encryption, and provider policy. Redact secrets and sensitive content from logs by default.
47
+
48
+ Audit principal, tenant, run ID, tool/version, action, target, authorization decision, approval, result status, and correlation IDs. Protect audit integrity and restrict who can inspect it. Avoid storing full sensitive payloads when metadata and hashes suffice.
49
+
50
+ ## 7. Defense in depth
51
+
52
+ Use layered controls:
53
+ 1. Input validation and content classification.
54
+ 2. Context isolation and untrusted-content delimiting.
55
+ 3. Model/tool selection policy.
56
+ 4. Deterministic authorization and approval.
57
+ 5. Sandboxed execution and egress restrictions.
58
+ 6. Output validation and data-loss prevention.
59
+ 7. Audit, alerting, and incident response.
60
+
61
+ No single prompt filter or model classifier is a sufficient defense.
62
+
63
+ ## 8. Security review checklist
64
+
65
+ - [ ] Threat model includes direct/indirect prompt injection and confused deputy.
66
+ - [ ] Principal, tenant, agent, and tool identities are distinct.
67
+ - [ ] Every action has server-side object-level authorization.
68
+ - [ ] Tool scopes and credentials follow least privilege and short lifetimes.
69
+ - [ ] High-impact actions use parameter-bound approvals.
70
+ - [ ] Untrusted execution is isolated with resource and egress limits.
71
+ - [ ] Retrieved data is tenant-filtered before model context.
72
+ - [ ] Secrets and sensitive content are redacted from telemetry.
73
+ - [ ] Audit records capture decisions and outcomes without leaking payloads.
74
+ - [ ] Revocation, incident response, and permission review are operationalized.
@@ -0,0 +1,72 @@
1
+ # Agent state, memory, and durable workflow design
2
+
3
+ Do not conflate model context, conversation history, long-term memory, and durable workflow state. Each has different ownership, consistency, retention, privacy, and recovery requirements.
4
+
5
+ ## 1. State taxonomy
6
+
7
+ | State type | Purpose | Typical lifecycle and controls |
8
+ |---|---|---|
9
+ | Ephemeral context | Current model invocation and bounded working context | Short-lived; rebuildable; never sole source of truth |
10
+ | Conversation history | User-visible interaction record | User/tenant scoped; retention and deletion policy |
11
+ | Summaries / memory | Derived facts or preferences used across sessions | Provenance, confidence, consent, correction, expiry, deletion |
12
+ | Workflow state | Durable task status, inputs, checkpoints, and outputs | Transactional ownership, versioning, idempotency, recovery |
13
+ | External business data | Authoritative customer/order/payment/project records | Owned by domain service; access via authorized APIs |
14
+
15
+ Define which store is authoritative for each state item. Model-generated summaries are derived data and may be incomplete or wrong; preserve source references and allow correction.
16
+
17
+ ## 2. Memory lifecycle and governance
18
+
19
+ For every memory class define:
20
+ - Purpose and allowed content.
21
+ - Creation authority and provenance.
22
+ - User visibility, consent, correction, and deletion semantics.
23
+ - Tenant isolation, encryption, retention, expiry, and backup deletion behavior.
24
+ - Access policy and audit.
25
+ - Confidence, freshness, and conflict resolution.
26
+
27
+ Do not silently promote a model inference into durable user memory. Do not store secrets or unnecessary sensitive data. Apply deletion to derived indexes, caches, summaries, and replicas according to a documented lifecycle.
28
+
29
+ ## 3. Workflow persistence
30
+
31
+ Represent a durable run with stable run ID, tenant/principal, workflow version, state version, status, deadline, budget, checkpoint pointer, and correlation metadata. Persist state transitions atomically with an event or outbox record when external work must be scheduled.
32
+
33
+ Use optimistic concurrency or explicit locking to prevent concurrent workers from overwriting newer state. Define allowed transitions and reject stale writes. Make task creation and message handling idempotent.
34
+
35
+ ## 4. Checkpoint and resume
36
+
37
+ A checkpoint should capture the minimum information needed to resume safely:
38
+ - Workflow state and version.
39
+ - Completed steps and validated outputs with provenance.
40
+ - Pending actions and their idempotency/operation IDs.
41
+ - Remaining budget/deadline and cancellation state.
42
+ - References to external data, not copies of secrets.
43
+
44
+ Checkpoint before and after meaningful side effects. On resume, revalidate identity, permissions, resource versions, and policy. Do not replay a side effect merely because its completion record is missing; query the downstream operation or reconcile using its idempotency key.
45
+
46
+ ## 5. Event history and replay
47
+
48
+ Use append-only transition/event history when auditability and recovery justify it. Distinguish event sourcing from ordinary logs: an event log used to rebuild authoritative state requires strict schema evolution, ordering, retention, and replay tests.
49
+
50
+ Define duplicate, late, out-of-order, and poison events. Replay must be deterministic enough to reproduce state transitions and must not re-trigger external side effects. Store schema/workflow versions and provide upcasters or migration procedures.
51
+
52
+ ## 6. Consistency and retention
53
+
54
+ Choose consistency based on business invariants. Conversation summaries can often be eventually consistent; payment or authorization workflow state generally cannot. Define transactional boundaries across state, queue publication, and side effects using outbox/inbox, idempotency, and reconciliation.
55
+
56
+ Set retention and deletion policies for conversation history, memory, workflow state, audit, and derived embeddings independently. Ensure deletion and tenant offboarding propagate to indexes, caches, backups, and downstream processors according to policy.
57
+
58
+ ## 7. Recovery and operations
59
+
60
+ Plan for worker crash, lease expiry, duplicate delivery, partial checkpoint, schema upgrade, provider timeout, tool timeout, cancellation, and tenant deletion during a run. Define stale-run detection, lease renewal, retry budget, dead-letter handling, operator repair, and recovery verification.
61
+
62
+ ## 8. Review checklist
63
+
64
+ - [ ] Ephemeral context, history, memory, workflow state, and business data are separated.
65
+ - [ ] Every state class has an owner, purpose, access policy, and retention lifecycle.
66
+ - [ ] Memory provenance, consent/correction, confidence, and deletion are defined.
67
+ - [ ] Workflow transitions are versioned and concurrency-safe.
68
+ - [ ] Checkpoints capture pending side effects and idempotency references.
69
+ - [ ] Resume revalidates identity, permission, and resource state.
70
+ - [ ] Replay cannot duplicate external effects.
71
+ - [ ] Deletion propagates to derived stores and indexes.
72
+ - [ ] Recovery procedures are tested, not merely documented.
@@ -0,0 +1,90 @@
1
+ # Agent evaluation, observability, and operational quality
2
+
3
+ Evaluate agents as complete systems that interact with users, tools, data, and external services. A plausible answer or successful model call is not proof that the task was completed correctly or safely.
4
+
5
+ ## 1. Define task-level outcomes
6
+
7
+ For each task class define:
8
+ - Preconditions and authorized scope.
9
+ - Expected outcome and business invariants.
10
+ - Acceptable alternatives and partial completion.
11
+ - Prohibited actions and safety constraints.
12
+ - Required evidence/provenance.
13
+ - Latency, token, cost, and human-intervention budgets.
14
+
15
+ Separate task success from stylistic preference. Define deterministic assertions wherever possible.
16
+
17
+ ## 2. Evaluation dataset
18
+
19
+ Maintain a versioned dataset containing representative, edge, adversarial, and regression cases. Record source, expected behavior, risk class, tenant/data assumptions, and evaluation version. Protect sensitive examples and ensure test data is authorized and sanitized.
20
+
21
+ Cover:
22
+ - Normal requests and ambiguous instructions.
23
+ - Tool selection and argument construction.
24
+ - Invalid, unavailable, or malicious tool results.
25
+ - Prompt injection in user and retrieved content.
26
+ - Authorization denial and cross-tenant attempts.
27
+ - Duplicate requests, timeout, partial side effect, and resume.
28
+ - Long-context, token-budget, cancellation, and concurrency behavior.
29
+ - Multi-agent delegation, disagreement, and aggregation.
30
+
31
+ ## 3. Metrics and scorecards
32
+
33
+ Measure multiple dimensions independently:
34
+ - Task completion and correctness.
35
+ - Groundedness, citation/provenance accuracy.
36
+ - Tool selection, argument validity, and tool outcome.
37
+ - Policy compliance, unauthorized action rate, and data leakage.
38
+ - Robustness to adversarial and malformed inputs.
39
+ - Recovery, cancellation, and duplicate-side-effect rate.
40
+ - End-to-end latency, TTFT where relevant, token use, and cost.
41
+ - Human approval, override, escalation, and abandonment rates.
42
+
43
+ Do not collapse safety-critical dimensions into a single weighted quality score. Report sample size, confidence interval where appropriate, dataset version, and known limitations. Avoid optimizing to a leaked or static benchmark.
44
+
45
+ ## 4. Evaluation methods
46
+
47
+ Combine:
48
+ - Unit tests for policy, schemas, state transitions, and deterministic logic.
49
+ - Contract tests for model adapters and tools.
50
+ - Scenario tests for complete user journeys.
51
+ - Trajectory review for intermediate reasoning/actions without treating hidden chain-of-thought as a required artifact.
52
+ - Adversarial/red-team tests for injection and authorization bypass.
53
+ - Fault injection for provider/tool outages, timeout, queue backlog, and worker crash.
54
+ - Human review for high-impact or ambiguous cases.
55
+ - Staged online experiments with rollback thresholds.
56
+
57
+ Use model-based graders cautiously: validate them against human labels, measure disagreement, and never let an LLM judge be the sole gate for critical security or financial invariants.
58
+
59
+ ## 5. Regression and release gates
60
+
61
+ Version model/provider, prompt, tool schema, retrieval index, policy, workflow, and dataset. Run the same critical regression suite when any of these changes. Define release thresholds per risk class, minimum sample size, and rollback criteria. Block release on critical policy violations even if average task quality improves.
62
+
63
+ Separate offline evaluation from online monitoring. Canary new versions on bounded traffic and compare task outcomes, safety, latency, cost, and escalation.
64
+
65
+ ## 6. Tracing and observability
66
+
67
+ Trace a run using correlation IDs across orchestration, model calls, tool calls, queues, and child agents. Capture:
68
+ - Run/task/parent-child IDs and workflow version.
69
+ - Model/provider and prompt/config version.
70
+ - Tool name/version, schema validation, authorization decision, and outcome.
71
+ - State transitions, checkpoint/resume, retries, cancellation, and escalation.
72
+ - Latency phases, token counts, budget consumption, and cost estimate.
73
+ - Evaluation signals, human intervention, and terminal status.
74
+
75
+ Redact prompts, secrets, PII, and sensitive tool payloads by default. Define access control and retention for traces and audit separately.
76
+
77
+ ## 7. Operations and incident response
78
+
79
+ Alert on user-impacting SLOs: task failure, policy violation, stuck runs, queue age, provider errors, tool timeouts, budget overruns, and recovery failures. Define runbooks for pausing workflows, revoking tool credentials, disabling a tool/server, rolling back prompt/model/policy versions, reconciling side effects, and replaying safe tasks.
80
+
81
+ ## 8. Review checklist
82
+
83
+ - [ ] Task-level success and prohibited behavior are explicit.
84
+ - [ ] Evaluation data is versioned, representative, authorized, and protected.
85
+ - [ ] Tool, policy, injection, recovery, and multi-agent cases are covered.
86
+ - [ ] Safety-critical metrics remain independent release gates.
87
+ - [ ] Graders are calibrated and human-validated.
88
+ - [ ] Model/prompt/tool/policy/dataset versions are traceable.
89
+ - [ ] Telemetry is end-to-end, redacted, access-controlled, and retained appropriately.
90
+ - [ ] Canary, rollback, and incident runbooks are defined.
@@ -0,0 +1,29 @@
1
+ # Kubernetes and Container Platform Architecture
2
+
3
+ Use Kubernetes when workload density, deployment automation, scheduling, service discovery, self-healing, policy enforcement, or platform standardization justify operating a cluster. It is not a default requirement for high traffic. Compare managed container platforms, serverless containers, and simpler VM or PaaS deployment first.
4
+
5
+ ## Workload and cluster model
6
+ - Classify stateless services, stateful services, batch jobs, scheduled jobs, operators, and platform components. State which workloads truly need orchestration.
7
+ - Define cluster boundaries by environment, region, trust domain, regulatory boundary, and blast-radius target. Avoid one global cluster as a universal control plane.
8
+ - Treat namespaces as organizational and policy boundaries, not hard security isolation. Use separate clusters or stronger runtime isolation for mutually untrusted tenants.
9
+ - Model requests and limits, QoS classes, pod disruption budgets, topology spread, anti-affinity, priority classes, and autoscaling together. CPU limits can cause throttling; memory limits can cause OOM kills. Validate under representative load.
10
+ - Keep readiness, liveness, and startup probes distinct. Liveness must not restart a process merely because a downstream dependency is temporarily unavailable. Readiness should remove an instance that cannot safely serve.
11
+ - Design graceful termination: stop new work, drain connections, finish or checkpoint bounded in-flight work, and exit before the termination grace period. Align application shutdown, load-balancer deregistration, and rollout settings.
12
+ - Treat persistent volumes, CSI drivers, storage classes, snapshots, and zone failure behavior as explicit data architecture. Replicated Pods do not replicate a database.
13
+
14
+ ## Scheduling, scaling, and delivery
15
+ Set resource requests from measured steady-state and burst behavior. Use HPA only with reliable signals; use custom or external metrics only when their pipeline is dependable. Cluster autoscaling has provisioning delay, quota limits, and bin-packing constraints. Model the complete scale-up path and retain headroom for node failure and rolling updates.
16
+
17
+ Avoid feedback loops where HPA, KEDA, cluster autoscaler, and application concurrency controls amplify one another. Bound queue consumers and scale on backlog age or processing demand when appropriate, not CPU alone.
18
+
19
+ Use immutable image digests, signed artifacts where available, SBOMs, vulnerability scanning, admission policy, and provenance verification. Pin and review Helm charts, operators, CRDs, and controller versions. Separate build permissions from deploy permissions. Prefer declarative delivery with drift detection and auditable promotion.
20
+
21
+ Rollouts must define max unavailable/surge, readiness gates, canary or progressive delivery, automated abort criteria, and rollback compatibility. Database and event schema changes must remain compatible with old and new application versions during rollout.
22
+
23
+ ## Security and operations
24
+ Apply least-privilege RBAC, workload identity, short-lived credentials, network policies, restricted Pod Security settings, admission controls, secret encryption, and node hardening. Do not mount the container runtime socket or privileged host paths into ordinary workloads. Restrict egress and metadata-service access. Namespace boundaries alone do not prevent noisy-neighbor or control-plane risks.
25
+
26
+ Define API-server and etcd availability, node capacity, DNS, CNI, CSI, ingress, certificate, and control-plane upgrade dependencies. Test node drains, zone loss, cluster upgrades, quota exhaustion, image-pull failure, DNS degradation, and storage attachment failure. Keep cluster recovery procedures and application-data backups separate.
27
+
28
+ ## Decision and exit criteria
29
+ Compare total cost: control plane, worker nodes, idle headroom, networking, observability, security tooling, on-call, upgrades, and specialist staffing. Define when to consolidate, split, or leave Kubernetes. Success means reduced developer lead time and operational risk, not maximum feature usage.
@@ -0,0 +1,21 @@
1
+ # Service Mesh and Traffic Management
2
+
3
+ A service mesh can centralize service identity, encryption, traffic policy, retries, telemetry, and progressive delivery. Adopt one only when these capabilities solve measured cross-cutting problems that application libraries, gateways, or the platform cannot solve more simply.
4
+
5
+ ## Traffic architecture
6
+ Separate north-south traffic from east-west traffic. Identify ingress, API gateway, load balancers, DNS, service discovery, mesh data plane, and egress gateways. Assign each layer a clear responsibility; overlapping retries, timeouts, routing, and rate limits create unpredictable behavior.
7
+
8
+ Define routing by service identity and version, not unstable Pod IPs. For canaries, specify traffic split, eligible requests, session affinity, metric windows, minimum sample size, abort thresholds, and rollback action. Percentage splits may not create representative cohorts when traffic is skewed by tenant, region, or request type.
9
+
10
+ ## Identity and security
11
+ Use workload identity and mutual TLS where justified. Define certificate issuance, rotation, trust roots, revocation behavior, identity mapping, and failure policy. mTLS authenticates a workload identity and encrypts a channel; it does not authorize every business operation. Enforce application-level authorization and tenant checks independently.
12
+
13
+ Model ingress and egress explicitly. Restrict outbound destinations, DNS resolution, metadata endpoints, and internal control planes. Validate bypass paths, host networking, UDP, init containers, and node-level traffic rather than assuming every flow is captured by a sidecar.
14
+
15
+ ## Resilience semantics
16
+ Set one end-to-end deadline and allocate budgets across hops. Retries must be bounded, jittered, and limited to safe or idempotent operations. Avoid retries at multiple layers; they multiply load during incidents. Define circuit-breaker thresholds, outlier detection, connection pools, concurrency limits, and load shedding.
17
+
18
+ A mesh control-plane outage may leave existing data-plane configuration active while preventing updates; verify behavior for the chosen implementation and version. Decide whether policy failures are fail-open or fail-closed per traffic class. Test certificate expiry, control-plane loss, stale configuration, DNS failure, and partial proxy rollout.
19
+
20
+ ## Cost and adoption gate
21
+ Measure proxy CPU/memory, connection overhead, tail latency, telemetry volume, configuration complexity, and incident debugging time. Assign ownership for upgrades, policy review, certificate lifecycle, and version skew. Compare application libraries, API gateways, ingress controllers, cloud-native service networking, and a mesh. Start with one bounded service group and define success metrics and an exit path. Verify feature support against current official versioned documentation.
@@ -0,0 +1,24 @@
1
+ # Multi-Region Architecture and Disaster Recovery
2
+
3
+ Multi-region architecture is a business continuity and latency decision, not a checkbox for availability. Begin with business impact analysis, failure scenarios, and tested recovery objectives.
4
+
5
+ ## Objectives and failure domains
6
+ For each critical journey define RTO, RPO, availability target, error-budget policy, residency constraints, and dependencies that must recover together. Distinguish zone, region, provider, control-plane, account, and global dependency failures. A second region is not independent if both share identity, DNS, deployment, encryption-key, or quota dependencies.
7
+
8
+ ## Topology choices
9
+ Compare backup and restore, pilot light, warm standby, active-passive, and active-active. Choose per workload and data domain. Stateless services may be active-active while transactional data remains single-writer. Do not claim active-active availability without proving write semantics and dependency independence.
10
+
11
+ ## Data replication and consistency
12
+ For every dataset define source of truth, replication mode, lag objective, ordering, conflict policy, deletion propagation, encryption, and restore procedure. Synchronous replication can increase write latency and couple regional availability; asynchronous replication permits nonzero RPO and stale reads. Multi-primary writes require conflict resolution that preserves business invariants, not merely last-write-wins timestamps.
13
+
14
+ Test replication lag, network partition, duplicate delivery, out-of-order updates, tombstones, schema changes, and recovery from corrupted or logically deleted data. Backups must be isolated from production credentials and deletion paths. Replication is not a backup.
15
+
16
+ ## Failover and failback
17
+ Specify detection signals, decision authority, fencing of the old primary, traffic steering, data promotion, write enablement, queue ownership, cache invalidation, and customer communication. Prevent split brain with leases, quorum, or an operator-controlled fencing mechanism appropriate to the data system. DNS TTL alone is not a failover strategy.
18
+
19
+ Failback is a separate migration: reconcile writes made in the recovery region, validate consistency, re-establish replication, and switch traffic only after safety gates pass. Document manual and automated paths, including when automation must stop for human approval.
20
+
21
+ ## Validation and economics
22
+ Run game days for region isolation, degraded inter-region links, unavailable KMS, exhausted quotas, stale DNS, failed deployment, and data corruption. Measure actual RTO/RPO from detection through verified business recovery. Perform restore tests, not just backup-job checks. Record evidence, gaps, owners, and remediation dates.
23
+
24
+ Estimate duplicate compute, replicated storage, inter-region transfer, global load balancing, support tiers, idle capacity, and operational staffing. Compare expected loss reduction with recurring and migration costs.
@@ -0,0 +1,26 @@
1
+ # Platform Engineering and Developer Experience
2
+
3
+ A platform is an internal product that provides paved paths for software delivery. Its purpose is to reduce cognitive load, improve consistency, and make secure and reliable practices easier—not to centralize every engineering decision.
4
+
5
+ ## Product model
6
+ Identify developer personas, lifecycle journeys, friction points, and measurable outcomes. Treat application teams as customers. Publish a service catalog, ownership metadata, supported golden paths, lifecycle policies, and a support model. Define platform product ownership and a feedback loop.
7
+
8
+ Measure lead time to first production deployment, deployment frequency, change failure rate, recovery time, toil, onboarding time, platform adoption, and developer satisfaction. Avoid vanity metrics such as number of templates or features shipped.
9
+
10
+ ## Platform capabilities
11
+ Typical capabilities include secure service scaffolding; CI/CD building blocks, artifact provenance, and progressive delivery; infrastructure provisioning through reviewed modules; identity, secrets, certificates, network, and policy integrations; observability defaults and SLO templates; service catalog and dependency mapping; and ephemeral preview environments.
12
+
13
+ Build a capability only when there is repeated demand and a clear support owner. Prefer composable APIs and documented interfaces over a monolithic portal that becomes a mandatory bottleneck.
14
+
15
+ ## Golden paths and guardrails
16
+ A golden path is an opinionated, supported route—not the only possible route. Make the default secure, observable, cost-aware, and operable. Provide escape hatches with explicit risk ownership and review. Enforce non-negotiable controls through policy-as-code and runtime boundaries rather than documentation alone.
17
+
18
+ Version templates and modules, define compatibility windows, publish deprecation notices, and provide migration tooling. Avoid breaking all teams through uncoordinated platform upgrades.
19
+
20
+ ## Team topology and reliability
21
+ Clarify platform team responsibilities versus product team responsibilities. Product teams retain accountability for service behavior, SLOs, data, and on-call unless a documented operating model says otherwise. Define escalation, incident handoff, platform SLOs, and shared responsibility for dependencies.
22
+
23
+ Treat the platform as production software: availability objectives, capacity, backup, disaster recovery, security review, release management, support, and incident response. Roll out incrementally, observe adoption and friction, and retire duplicated tooling only after migration.
24
+
25
+ ## Economics
26
+ Calculate total cost including engineering time, support, compute, licensing, maintenance, and duplicated or unused capabilities. Compare build-versus-buy and centralization benefits against queueing delays and reduced team autonomy.
@@ -0,0 +1,29 @@
1
+ # FinOps and Cloud Cost Architecture
2
+
3
+ FinOps is a continuous operating practice connecting engineering, finance, and business value. Cost optimization must preserve reliability, security, performance, and delivery outcomes.
4
+
5
+ ## Cost visibility and allocation
6
+ Create a common allocation model using accounts/projects, environments, services, teams, products, tenants, and cost centers. Require ownership metadata and consistent tags/labels. Separate direct, shared, unallocated, and one-time costs. Define allocation rules for shared clusters, network egress, observability, support plans, and platform services.
7
+
8
+ Track cost by workload and business unit. Useful measures include cost per active user, transaction, request, GB processed, inference, or unit of business value. Normalize for demand and product mix; raw monthly spend alone cannot distinguish growth from inefficiency.
9
+
10
+ ## Cost model
11
+ Include compute requests and idle capacity, autoscaling headroom, storage tiers and IOPS, database licenses, replicas, backups, snapshots, network egress, inter-region transfer, load balancers, observability ingestion and retention, support, marketplace software, and engineering operations. Model burst, peak, growth, retention, and disaster-recovery capacity.
12
+
13
+ Distinguish committed-use discounts and reserved capacity from flexible on-demand demand. Evaluate break-even utilization, commitment term, cancellation risk, portability, and forecast confidence. Spot/preemptible capacity suits only interruption-tolerant workloads with checkpointing and retry-safe execution.
14
+
15
+ ## Optimization workflow
16
+ 1. Establish a baseline and unit economics.
17
+ 2. Identify the largest cost drivers and waste, not merely the easiest line items.
18
+ 3. Form a hypothesis with an expected saving range and performance/reliability risk.
19
+ 4. Test one change under representative load.
20
+ 5. Compare cost and service-level outcomes before and after.
21
+ 6. Roll out gradually, monitor regressions, and document ownership.
22
+ 7. Revisit continuously as workload and pricing change.
23
+
24
+ Common levers include measured rightsizing, autoscaling, scheduling non-production environments, storage lifecycle policies, query and data-transfer reduction, cache efficiency, batching, retention controls, and commitment optimization. Do not lower replica counts, backups, redundancy, or telemetry blindly to hit a budget.
25
+
26
+ ## Budgets and architecture decisions
27
+ Use budgets and anomaly detection as alerts, not the only control. Define approval for commitments, quota increases, expensive resources, and exceptions. Apply policy-as-code for resource classes, regions, labels, and retention where appropriate.
28
+
29
+ Compare lifecycle TCO rather than unit price alone. Include migration, lock-in, support, incident risk, idle capacity, staff expertise, and exit cost. Model expected, peak, and degraded modes. Kubernetes, service mesh, and multi-region designs may increase resilience while materially increasing fixed cost; quantify both benefits and operational burden.