system-design-primer-skill 1.2.0 → 2.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +114 -60
- package/SKILL.md +437 -93
- package/VERSION +1 -1
- package/evals/README.md +1022 -0
- package/evals/system-design-cases.json +414 -0
- package/package.json +46 -4
- package/references/03-data-and-storage.md +8 -2
- package/references/04-caching.md +3 -3
- package/references/05-async-and-messaging.md +4 -0
- package/references/06-distributed-systems.md +2 -2
- package/references/07-reliability-and-resilience.md +7 -3
- package/references/08-api-security-observability.md +4 -0
- package/references/10-workflows-and-checklists.md +1 -1
- package/references/13-architecture-intelligence.md +91 -0
- package/references/14-api-and-data-contracts.md +80 -0
- package/references/15-failure-analysis.md +88 -0
- package/references/16-multi-agent-architecture.md +76 -0
- package/references/17-agent-harness-and-runtime.md +87 -0
- package/references/18-mcp-architecture.md +67 -0
- package/references/19-agent-security-and-permissions.md +90 -0
- package/references/20-agent-state-and-memory.md +76 -0
- package/references/21-agent-evaluation-and-observability.md +90 -0
- package/references/22-kubernetes-and-container-platforms.md +29 -0
- package/references/23-service-mesh-and-traffic-management.md +21 -0
- package/references/24-multi-region-and-disaster-recovery.md +28 -0
- package/references/25-platform-engineering-and-developer-experience.md +26 -0
- package/references/26-finops-and-cloud-cost.md +29 -0
- package/references/27-production-observability-and-slo.md +33 -0
- package/references/28-event-streaming-and-kafka-pulsar.md +33 -0
- package/references/29-event-sourcing-and-cqrs.md +23 -0
- package/references/30-change-data-capture-and-debezium.md +27 -0
- package/references/31-data-lakehouse-and-stream-batch.md +24 -0
- package/references/32-workflow-orchestration-and-temporal.md +28 -0
- package/references/33-schema-registry-and-event-governance.md +22 -0
- package/references/34-postgresql-internals-and-performance.md +45 -0
- package/references/35-mysql-innodb-and-performance.md +29 -0
- package/references/36-sharding-and-distributed-sql.md +22 -0
- package/references/37-consensus-and-distributed-transactions.md +29 -0
- package/references/38-search-architecture-elasticsearch-opensearch.md +24 -0
- package/references/39-vector-databases-and-hybrid-retrieval.md +31 -0
- package/references/40-zero-downtime-database-migrations.md +27 -0
- package/references/41-api-gateway-and-edge-architecture.md +23 -0
- package/references/42-rest-api-design-and-lifecycle.md +23 -0
- package/references/43-graphql-architecture-and-governance.md +16 -0
- package/references/44-grpc-and-service-contracts.md +16 -0
- package/references/45-websocket-sse-and-realtime-systems.md +16 -0
- package/references/46-realtime-collaboration-ot-crdt.md +16 -0
- package/references/47-api-security-rate-limiting-and-abuse.md +16 -0
- package/references/48-oauth-oidc-and-passkeys.md +20 -0
- package/references/49-iam-rbac-abac-rebac.md +20 -0
- package/references/50-zero-trust-and-workload-identity.md +16 -0
- package/references/51-secrets-kms-pki-and-key-management.md +20 -0
- package/references/52-supply-chain-security-sbom-slsa.md +78 -0
- package/references/53-threat-modeling-stride-and-attack-trees.md +16 -0
- package/references/54-privacy-engineering-and-data-governance.md +20 -0
- package/references/55-opentelemetry-and-instrumentation.md +54 -0
- package/references/56-distributed-tracing-and-context-propagation.md +47 -0
- package/references/57-continuous-profiling-and-performance-diagnostics.md +45 -0
- package/references/58-load-stress-soak-testing-and-workload-modeling.md +44 -0
- package/references/59-capacity-planning-and-performance-modeling.md +38 -0
- package/references/60-chaos-engineering-and-resilience-experiments.md +45 -0
- package/references/61-incident-management-and-postmortems.md +38 -0
- package/references/62-dns-and-service-discovery.md +35 -0
- package/references/63-load-balancing-and-global-traffic-management.md +31 -0
- package/references/64-cdn-and-edge-caching.md +34 -0
- package/references/65-cloud-networking-and-connectivity.md +34 -0
- package/references/66-edge-computing-architecture.md +33 -0
- package/references/67-storage-infrastructure-and-storage-systems.md +34 -0
- package/references/68-tcp-quic-and-network-performance.md +27 -0
- package/references/69-task-routing-and-evidence.md +115 -0
- package/scripts/audit_chain.py +36 -0
- package/scripts/compare_reports.py +43 -0
- package/scripts/evaluate.py +475 -0
- package/scripts/evaluation_catalog.py +129 -0
- package/scripts/evaluation_fingerprints.py +20 -0
- package/scripts/evidence_archive.py +79 -0
- package/scripts/evidence_signing.py +98 -0
- package/scripts/evidence_timestamp.py +441 -0
- package/scripts/install.ps1 +12 -3
- package/scripts/install.sh +15 -8
- package/scripts/report_integrity.py +184 -0
- package/scripts/review_consensus.py +77 -0
- package/scripts/review_evidence.py +105 -0
- package/scripts/review_provenance.py +44 -0
- package/scripts/reviewer_calibration.py +215 -0
- package/scripts/rubric_compatibility.py +134 -0
- package/scripts/run_evals.py +130 -0
- package/scripts/run_manifest.py +179 -0
- package/scripts/statistical_analysis.py +308 -0
- package/scripts/tsa_attestation.py +709 -0
- package/scripts/tsa_validation_report.py +627 -0
- package/scripts/validate.py +356 -11
- package/scripts/verify_build_provenance.py +625 -0
- package/scripts/verify_tsa_ocsp.py +962 -0
- package/scripts/verify_tsa_revocation.py +782 -0
- package/scripts/windows_broker_protocol.py +274 -0
- package/scripts/windows_broker_transport.py +141 -0
- package/scripts/windows_broker_win32.py +713 -0
- package/templates/agent-evaluation-plan-template.md +57 -0
- package/templates/agent-security-review-template.md +46 -0
- package/templates/agent-system-design-template.md +74 -0
- package/templates/api-architecture-design-template.md +33 -0
- package/templates/api-security-and-rate-limit-review-template.md +29 -0
- package/templates/architecture-decision-matrix-template.md +48 -0
- package/templates/architecture-review-scorecard-template.md +36 -0
- package/templates/cdc-data-pipeline-template.md +41 -0
- package/templates/cdn-and-edge-strategy-template.md +36 -0
- package/templates/chaos-experiment-template.md +39 -0
- package/templates/cloud-cost-model-template.md +37 -0
- package/templates/database-architecture-review-template.md +33 -0
- package/templates/edge-computing-readiness-template.md +36 -0
- package/templates/event-sourcing-cqrs-template.md +41 -0
- package/templates/event-streaming-design-template.md +47 -0
- package/templates/failure-analysis-template.md +36 -0
- package/templates/identity-and-access-architecture-template.md +29 -0
- package/templates/incident-postmortem-template.md +39 -0
- package/templates/multi-region-disaster-recovery-template.md +43 -0
- package/templates/network-architecture-design-template.md +38 -0
- package/templates/observability-architecture-template.md +52 -0
- package/templates/performance-test-plan-template.md +52 -0
- package/templates/privacy-data-governance-template.md +24 -0
- package/templates/production-readiness-checklist-template.md +52 -0
- package/templates/realtime-system-design-template.md +28 -0
- package/templates/security-threat-model-template.md +26 -0
- package/templates/sharding-distributed-sql-design-template.md +34 -0
- package/templates/storage-architecture-design-template.md +35 -0
- package/templates/supply-chain-security-review-template.md +20 -0
- package/templates/zero-downtime-database-migration-template.md +32 -0
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# Distributed Tracing and Context Propagation
|
|
2
|
+
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns distributed trace structure, context propagation, baggage, sampling behavior, and trace-based latency interpretation. Use [OpenTelemetry instrumentation](55-opentelemetry-and-instrumentation.md) for APIs/SDKs, semantic conventions, collectors, and exporters; [production observability](27-production-observability-and-slo.md) for SLIs/SLOs, alerting, and telemetry-backend operations; and [privacy engineering](54-privacy-engineering-and-data-governance.md) for authoritative data classification, retention, and deletion policy. A trace is diagnostic evidence, not proof of causality, authorization, or business completion; missing spans do not prove that work did not occur.
|
|
6
|
+
|
|
7
|
+
## Trace model
|
|
8
|
+
A trace is a causal graph of spans, not necessarily a single linear tree. Define trace boundaries, parent-child relationships, span links, baggage policy, and the behavior of asynchronous work before interpreting latency or causality.
|
|
9
|
+
|
|
10
|
+
## Propagation
|
|
11
|
+
- Standardize W3C Trace Context (traceparent and tracestate) where interoperable; explicitly document legacy formats and translation boundaries.
|
|
12
|
+
- Propagate context through HTTP, RPC, message headers, job metadata, and workflow state. Verify framework and broker instrumentation behavior for deployed versions.
|
|
13
|
+
- For fan-out/fan-in, preserve causal relationships. Use span links when work is initiated by multiple messages or has multiple causal parents.
|
|
14
|
+
- Baggage is propagated metadata, not a trusted identity or authorization claim. Allowlist keys, constrain size, and exclude secrets and PII.
|
|
15
|
+
- At trust boundaries, validate and optionally restart trace context to prevent untrusted callers from forging internal trace relationships or injecting excessive baggage.
|
|
16
|
+
|
|
17
|
+
## Latency interpretation
|
|
18
|
+
- Distinguish wall-clock end-to-end latency from summed span durations. Concurrent spans overlap; summing them overstates request duration.
|
|
19
|
+
- Identify queue wait, network time, server processing, lock waits, retries, and downstream dependency time separately where instrumentation permits.
|
|
20
|
+
- Check clock skew before interpreting cross-host timestamps. Prefer duration measurements from a single process clock for local spans.
|
|
21
|
+
- A missing span can mean instrumentation gaps, sampling, propagation loss, or exporter/backend loss; do not infer that the operation did not happen.
|
|
22
|
+
|
|
23
|
+
## Sampling
|
|
24
|
+
- Define head/tail sampling by service criticality, error status, latency, tenant, and cost.
|
|
25
|
+
- Tail sampling needs trace affinity, bounded decision windows, memory limits, and explicit behavior when late spans arrive.
|
|
26
|
+
- Ensure sampling decisions are consistent across services or understand why partial traces are expected.
|
|
27
|
+
- Preserve representative baseline traffic; error-only retention cannot measure normal latency distributions.
|
|
28
|
+
|
|
29
|
+
## Security and privacy
|
|
30
|
+
- Do not expose credentials, request bodies, raw query strings, or sensitive identifiers in span attributes.
|
|
31
|
+
- Sanitize untrusted header values and cap attribute/event sizes.
|
|
32
|
+
- Restrict trace search and export by tenant and role. Consider cross-tenant leakage through shared trace IDs and query interfaces.
|
|
33
|
+
|
|
34
|
+
## Troubleshooting workflow
|
|
35
|
+
1. Start from a user-visible symptom and identify the affected journey, time window, and cohort.
|
|
36
|
+
2. Compare golden signals and SLO burn against baseline.
|
|
37
|
+
3. Inspect representative traces, including successful and failed requests.
|
|
38
|
+
4. Separate application time, queue wait, retries, and dependency time.
|
|
39
|
+
5. Correlate with deploys, configuration changes, saturation, and dependency incidents.
|
|
40
|
+
6. Validate the hypothesis with logs, metrics, profiles, or a controlled experiment.
|
|
41
|
+
|
|
42
|
+
## Checklist
|
|
43
|
+
- [ ] Propagation covers sync and async boundaries.
|
|
44
|
+
- [ ] Baggage is bounded, allowlisted, and non-sensitive.
|
|
45
|
+
- [ ] Sampling preserves useful normal and exceptional traffic.
|
|
46
|
+
- [ ] Trace gaps and partial traces are understood.
|
|
47
|
+
- [ ] Latency interpretation accounts for concurrency and clock behavior.
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# Continuous Profiling and Performance Diagnostics
|
|
2
|
+
|
|
3
|
+
## Purpose
|
|
4
|
+
Profiling explains where CPU, memory, allocation, lock, and execution time are spent. Use it to test a specific performance hypothesis; do not treat a flame graph as a root cause without workload and runtime context.
|
|
5
|
+
|
|
6
|
+
## Profile types
|
|
7
|
+
- CPU profiles: sampled on-CPU execution; distinguish hot code from time blocked on I/O.
|
|
8
|
+
- Heap/allocation profiles: retained objects, allocation rate, and allocation hot spots.
|
|
9
|
+
- Wall-clock profiles: on-CPU plus off-CPU waiting, subject to profiler/runtime support.
|
|
10
|
+
- Lock and contention profiles: mutex waits, blocked goroutines/threads, and synchronization bottlenecks.
|
|
11
|
+
- Runtime profiles: GC pauses, scheduling, thread pools, JIT behavior, and event-loop stalls.
|
|
12
|
+
|
|
13
|
+
## Method
|
|
14
|
+
1. Define the affected endpoint/job, service version, environment, workload, and target metric.
|
|
15
|
+
2. Capture representative profiles under controlled and production-safe conditions.
|
|
16
|
+
3. Compare baseline and candidate profiles using equivalent traffic, data shape, warm-up, and runtime settings.
|
|
17
|
+
4. Attribute costs to code paths; correlate with traces, request mix, allocation rates, GC, and saturation.
|
|
18
|
+
5. Change one material factor at a time and repeat the measurement.
|
|
19
|
+
6. Validate user-visible latency/throughput and resource cost after the change.
|
|
20
|
+
|
|
21
|
+
## Continuous profiling architecture
|
|
22
|
+
- Select agent, in-process, or eBPF-based collection according to language/runtime support, privilege boundaries, overhead, and operational model.
|
|
23
|
+
- Define sampling frequency, profile duration, upload cadence, symbolization, retention, and tenant isolation.
|
|
24
|
+
- Monitor profiler overhead, missing samples, upload failures, and storage cost.
|
|
25
|
+
- Restrict access to profiles: symbols, stack traces, and labels can reveal proprietary code paths or sensitive metadata.
|
|
26
|
+
|
|
27
|
+
## Performance diagnosis
|
|
28
|
+
- CPU saturation: inspect hot functions, serialization, compression, cryptography, regex, and inefficient algorithms.
|
|
29
|
+
- Memory pressure: inspect allocation churn, retained references, caches, fragmentation, and GC behavior.
|
|
30
|
+
- Lock contention: identify shared mutable state, coarse locks, hot keys, and queueing around critical sections.
|
|
31
|
+
- I/O waits: correlate profiles with traces, storage latency, connection pools, and network retransmissions.
|
|
32
|
+
- Tail latency: inspect rare paths, GC pauses, lock convoys, retries, cold caches, noisy neighbors, and dependency variance.
|
|
33
|
+
|
|
34
|
+
## Pitfalls
|
|
35
|
+
- Comparing profiles from different workloads or runtime configurations.
|
|
36
|
+
- Assuming sampled CPU time explains blocked time.
|
|
37
|
+
- Optimizing a hot function that contributes little to end-to-end latency.
|
|
38
|
+
- Ignoring profiler overhead, symbolization gaps, and production data governance.
|
|
39
|
+
|
|
40
|
+
## Checklist
|
|
41
|
+
- [ ] Hypothesis, workload, baseline, and target are recorded.
|
|
42
|
+
- [ ] Profile type matches the suspected bottleneck.
|
|
43
|
+
- [ ] Samples are representative and overhead is measured.
|
|
44
|
+
- [ ] Findings are correlated with traces and service metrics.
|
|
45
|
+
- [ ] Before/after results include user impact and resource cost.
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
# Load, Stress, Soak Testing, and Workload Modeling
|
|
2
|
+
|
|
3
|
+
## Workload model
|
|
4
|
+
Model production traffic rather than a single peak RPS number. Include arrival patterns, concurrency, request mix, payload sizes, user think time, read/write ratio, tenant skew, cache state, data volume, geographic distribution, and background jobs.
|
|
5
|
+
|
|
6
|
+
Distinguish:
|
|
7
|
+
- Open workload: arrivals continue independently of response time; useful for detecting queue growth and overload.
|
|
8
|
+
- Closed workload: a fixed population waits for responses before issuing more work; useful for concurrency-limited clients but can hide overload through coordinated omission.
|
|
9
|
+
- Burst workload: short high-rate arrivals and recovery periods.
|
|
10
|
+
- Stateful workload: realistic sessions, hot keys, cache behavior, and data growth.
|
|
11
|
+
|
|
12
|
+
## Test types
|
|
13
|
+
- Baseline/load: expected steady-state and forecast traffic.
|
|
14
|
+
- Stress: increase load to discover saturation, queue growth, and failure boundaries.
|
|
15
|
+
- Spike: abrupt arrival-rate changes and autoscaling response.
|
|
16
|
+
- Soak: sustained load to expose leaks, resource drift, compaction, GC, and backlog accumulation.
|
|
17
|
+
- Capacity breakpoint: find the highest load that still meets latency/error/SLO constraints with headroom.
|
|
18
|
+
- Resilience: combine dependency degradation and partial failures with representative load.
|
|
19
|
+
|
|
20
|
+
## Test validity
|
|
21
|
+
- Define arrival-rate model and concurrency separately.
|
|
22
|
+
- Prevent coordinated omission; use tools and metrics that account for delayed request scheduling.
|
|
23
|
+
- Validate generator capacity, network path, TLS, DNS, and client-side limits.
|
|
24
|
+
- Use production-like data distributions and data sizes; avoid testing only warm caches or tiny datasets.
|
|
25
|
+
- Include retries and background work, or explicitly isolate them.
|
|
26
|
+
- Warm up JITs, caches, and connection pools consistently; report warm-up and steady-state windows separately.
|
|
27
|
+
|
|
28
|
+
## Safety and controls
|
|
29
|
+
- Obtain explicit authorization for production tests. Prefer isolated environments and synthetic or anonymized data.
|
|
30
|
+
- Set hard limits for RPS, concurrency, duration, spend, and generated data.
|
|
31
|
+
- Define abort thresholds for error rate, latency, saturation, queue depth, and downstream impact.
|
|
32
|
+
- Coordinate with on-call owners and external providers; use allowlisted test identities and clear traffic tagging.
|
|
33
|
+
- Ensure cleanup and data reset are safe and repeatable.
|
|
34
|
+
|
|
35
|
+
## Reporting
|
|
36
|
+
Record build/configuration, infrastructure shape, dataset, workload profile, generator placement, test duration, warm-up, run IDs, and environmental differences. Report throughput, P50/P95/P99, errors/timeouts, saturation, queue depth/age, resource use, and SLO attainment. Include confidence, variance, and limitations.
|
|
37
|
+
|
|
38
|
+
## Checklist
|
|
39
|
+
- [ ] Workload represents realistic request mix and arrival behavior.
|
|
40
|
+
- [ ] Open/closed model and coordinated omission treatment are explicit.
|
|
41
|
+
- [ ] Data size, skew, cache state, and background work are representative.
|
|
42
|
+
- [ ] Generator and network are not the bottleneck.
|
|
43
|
+
- [ ] Abort controls and production authorization are defined.
|
|
44
|
+
- [ ] Results include tail latency, errors, saturation, and queue behavior.
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
# Capacity Planning and Performance Modeling
|
|
2
|
+
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns workload modeling, demand/capacity estimates, queue stability, headroom, failure capacity, and scaling thresholds. Use [load and stress testing](58-load-stress-soak-testing-and-workload-modeling.md) for experimental workload design and measured test results; [production observability](27-production-observability-and-slo.md) for live SLI/SLO signals; and [FinOps](26-finops-and-cloud-cost.md) for cost allocation and economic trade-offs. A capacity estimate is conditional on workload mix, service-time distribution, topology, quotas, and failure assumptions; benchmark throughput or replica count alone does not establish production headroom.
|
|
6
|
+
|
|
7
|
+
## Planning inputs
|
|
8
|
+
Use measured traffic and growth where available. Record average/peak arrival rates, concurrency, request mix, payload sizes, CPU time/request, memory/request, database queries/request, connection occupancy, queue arrival/service rates, storage growth, replication, retention, and failure scenarios. Mark forecasts and assumptions explicitly.
|
|
9
|
+
|
|
10
|
+
## Capacity model
|
|
11
|
+
- Little's Law: L = λW, where L is average in-system work, λ is throughput, and W is average time in the system. Apply to a stable system and a consistent boundary using long-run averages; it does not by itself predict tail latency or size a transient burst. State whether the boundary includes queued, executing, or downstream work.
|
|
12
|
+
- CPU demand: approximate fully utilized core-equivalents as throughput × CPU-seconds/request for the measured workload. Convert that demand to provisioned cores using a target utilization/headroom policy, and account for CPU throttling, uneven request cost, burst behavior, and the CPU spent on retries and background work.
|
|
13
|
+
- Connection demand: estimate concurrency from arrival rate × connection occupancy time; validate pool limits and downstream capacity.
|
|
14
|
+
- Queue stability: for a simple work-conserving queue, long-run arrival work must remain below sustainable service capacity for backlog to be stable; equality leaves no recovery headroom, and bursty arrivals or variable service times can produce large waits well before the average rates meet. Model utilization, service-time variability, burst size, queue limits, deadline expiry, retry amplification, and backlog drain time. Estimate drain time from net spare capacity (service rate minus ongoing arrival rate), not total service rate alone.
|
|
15
|
+
- Storage: model logical growth, indexes, replicas, WAL/binlogs, backups, retention, compaction, and migration headroom separately.
|
|
16
|
+
- Network: include payload, protocol/TLS overhead, retries, replication, telemetry, and cross-zone/region traffic.
|
|
17
|
+
|
|
18
|
+
## Bottleneck and headroom
|
|
19
|
+
- Find the first constrained resource across the full dependency chain; scaling stateless application replicas may simply move the bottleneck to the database or a shared service.
|
|
20
|
+
- Define safe operating envelopes using SLOs, not theoretical maximum throughput.
|
|
21
|
+
- Include N+1 failure capacity, rolling deployments, zone loss, and maintenance events where required. Define the failure scenario explicitly (for example, loss of the largest zone or one node) and verify that surviving dependencies, connection pools, quotas, and network paths can carry the shifted load; replica count alone is not proof of N+1 capacity.
|
|
22
|
+
- Model autoscaling detection and provisioning delay; reactive scaling cannot prevent every burst.
|
|
23
|
+
- Evaluate vertical, horizontal, queue-based, and workload-shaping options before introducing partitioning or sharding.
|
|
24
|
+
|
|
25
|
+
## Sensitivity and uncertainty
|
|
26
|
+
Build low/base/high scenarios for growth, peak factors, request mix, retry amplification, payload size, and dependency latency.
|
|
27
|
+
Show which assumptions dominate the result and which measurements would reduce uncertainty.
|
|
28
|
+
Do not convert benchmark throughput into production capacity without accounting for workload differences, failure headroom, and SLO limits.
|
|
29
|
+
|
|
30
|
+
## Operational planning
|
|
31
|
+
Define thresholds and owners for scaling, load shedding, admission control, queue limits, and capacity procurement. Review forecasts after major launches, architectural changes, and observed growth deviations.
|
|
32
|
+
|
|
33
|
+
## Checklist
|
|
34
|
+
- [ ] Inputs are measured or explicitly labeled assumptions.
|
|
35
|
+
- [ ] End-to-end dependency constraints and failure headroom are modeled.
|
|
36
|
+
- [ ] Queue stability and burst recovery are included.
|
|
37
|
+
- [ ] Low/base/high sensitivity analysis is available.
|
|
38
|
+
- [ ] Scaling and load-shedding actions have owners and triggers.
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# Chaos Engineering and Resilience Experiments
|
|
2
|
+
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns the methodology and safety controls for planned, hypothesis-driven resilience experiments. It does not define live incident command or replace incident response; use [incident management](61-incident-management-and-postmortems.md) for active incidents and postmortems, [failure analysis](15-failure-analysis.md) to structure failure modes, and [reliability design](07-reliability-and-resilience.md) for the underlying resilience controls. Experiment results validate only the tested hypothesis, workload, topology, and time window.
|
|
6
|
+
|
|
7
|
+
## Principles
|
|
8
|
+
Chaos engineering is a controlled method for testing a falsifiable resilience hypothesis. It is not random disruption in production. Begin with steady-state signals that represent user-visible behavior, then introduce the smallest fault that can test the hypothesis.
|
|
9
|
+
|
|
10
|
+
## Experiment design
|
|
11
|
+
1. State the hypothesis and expected steady state.
|
|
12
|
+
2. Identify target service, dependency, cohort, environment, and blast-radius boundary.
|
|
13
|
+
3. Enumerate hazards, prerequisites, and possible data-integrity consequences.
|
|
14
|
+
4. Define measurable abort conditions, owner, observer, and communication channel.
|
|
15
|
+
5. Start in a safe environment with a small cohort; increase scope only after evidence supports it.
|
|
16
|
+
6. Inject one fault class at a time and observe detection, containment, degradation, recovery, and reconciliation.
|
|
17
|
+
7. Remove the fault, verify recovery, and document findings and follow-up actions.
|
|
18
|
+
|
|
19
|
+
## Fault classes
|
|
20
|
+
- Network: latency, packet loss, partitions, DNS failures, connection resets.
|
|
21
|
+
- Dependency: slow/unavailable database, cache, queue, identity provider, or third-party API.
|
|
22
|
+
- Compute: process termination, CPU pressure, memory pressure, node/zone loss.
|
|
23
|
+
- Control plane: configuration errors, certificate expiry, deployment failure, autoscaler or scheduler disruption.
|
|
24
|
+
- Data path: delayed/out-of-order/duplicate messages, partial writes, replay, and stale replicas. Use non-destructive synthetic data and strict invariants.
|
|
25
|
+
|
|
26
|
+
## Safety controls
|
|
27
|
+
- Explicit authorization and change window for production experiments.
|
|
28
|
+
- Hard scope allowlists, tenant/test cohort boundaries, concurrency and duration caps.
|
|
29
|
+
- Automated kill switch and independent abort monitoring.
|
|
30
|
+
- Exclude critical systems and irreversible operations unless separately approved and safely simulated.
|
|
31
|
+
- Verify backups, restore paths, and data reconciliation before experiments that can affect persistence.
|
|
32
|
+
- Coordinate on-call, security, platform, and vendor stakeholders.
|
|
33
|
+
|
|
34
|
+
## What to measure
|
|
35
|
+
User journey success, SLO burn, latency percentiles, error rate, queue age/depth, saturation, failover time, data correctness, alert detection time, mitigation time, and recovery time. A system that remains available but corrupts state has not passed.
|
|
36
|
+
|
|
37
|
+
## Learning loop
|
|
38
|
+
Convert findings into engineering work: missing timeout, unbounded retry, weak isolation, inadequate capacity, poor alerting, unsafe failover, or incomplete runbooks. Assign owners and deadlines; repeat the experiment after remediation. Track experiment coverage and validated hypotheses, not a vanity count of injected faults.
|
|
39
|
+
|
|
40
|
+
## Checklist
|
|
41
|
+
- [ ] Hypothesis and steady-state metrics are defined.
|
|
42
|
+
- [ ] Scope, authorization, blast radius, and abort criteria are explicit.
|
|
43
|
+
- [ ] Data integrity and recovery are included.
|
|
44
|
+
- [ ] On-call owners can observe and stop the experiment.
|
|
45
|
+
- [ ] Findings become tracked remediation and retested controls.
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
# Incident Management, On-call, and Postmortems
|
|
2
|
+
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns operational incident response: severity, command roles, communications, triage, recovery verification, on-call practice, and postmortem follow-through. Use [chaos engineering](60-chaos-engineering-and-resilience-experiments.md) for planned fault injection, [failure analysis](15-failure-analysis.md) for design-time fault registers, and [reliability architecture](07-reliability-and-resilience.md) for service-level resilience patterns. Technical replay, failover, and reconciliation procedures must link to the relevant queue, data, and regional-architecture references rather than being inferred from incident process alone.
|
|
6
|
+
|
|
7
|
+
## Operational model
|
|
8
|
+
Define severity levels by user/business impact, not by component count or internal effort. Establish incident command, technical lead, communications lead, and scribe roles appropriate to incident scale. Publish escalation paths, service ownership, dependency contacts, and decision authority.
|
|
9
|
+
|
|
10
|
+
## Detection and response
|
|
11
|
+
- Alert on user-visible symptoms and SLO error-budget burn; use metrics, logs, traces, and synthetic checks as complementary evidence.
|
|
12
|
+
- Make alerts actionable: condition, impact, owner, runbook, and escalation. Avoid paging on every transient threshold breach.
|
|
13
|
+
- Triage scope, affected cohorts, start time, recent changes, dependencies, and data-integrity risk.
|
|
14
|
+
- Stabilize first: rollback, traffic shift, disable a feature, shed load, isolate a dependency, or invoke a tested degraded mode.
|
|
15
|
+
- Preserve evidence and timeline. Communicate known facts, uncertainty, impact, and next update time.
|
|
16
|
+
- Verify recovery using user-facing signals and reconciliation, not merely healthy processes.
|
|
17
|
+
|
|
18
|
+
## On-call sustainability
|
|
19
|
+
- Define primary/secondary coverage, escalation timeout, handoff, compensation/rest, and backup ownership.
|
|
20
|
+
- Track page volume, actionable-page ratio, toil, after-hours load, and recurring incidents.
|
|
21
|
+
- Runbooks should include symptoms, diagnostic queries, safe mitigations, rollback, verification, and escalation. Keep commands environment-specific and reviewed.
|
|
22
|
+
- Automate repetitive safe actions with guardrails and audit; do not automate irreversible actions without explicit controls.
|
|
23
|
+
|
|
24
|
+
## Postmortems
|
|
25
|
+
Use blameless, evidence-based analysis. Capture impact, timeline, detection, contributing conditions, response effectiveness, what went well, where response struggled, and prioritized corrective actions with owners and due dates. Distinguish trigger from systemic contributors; avoid assigning root cause to the last person or deploy.
|
|
26
|
+
- Corrective actions should address prevention, detection, containment, and recovery.
|
|
27
|
+
- Rank actions by risk reduction and recurrence likelihood; verify completion and effectiveness.
|
|
28
|
+
- Share learnings across teams without exposing secrets or sensitive customer data.
|
|
29
|
+
|
|
30
|
+
## Operational metrics
|
|
31
|
+
Track MTTD, time to acknowledge, time to mitigate, time to recover, recurrence, SLO impact, customer impact, and action completion. These metrics diagnose process friction; do not use them as simplistic individual performance scores.
|
|
32
|
+
|
|
33
|
+
## Checklist
|
|
34
|
+
- [ ] Severity, roles, escalation, and communications are defined.
|
|
35
|
+
- [ ] Alerts are actionable and tied to user impact.
|
|
36
|
+
- [ ] Runbooks contain safe, tested mitigation and verification steps.
|
|
37
|
+
- [ ] Recovery includes data correctness and backlog reconciliation.
|
|
38
|
+
- [ ] Postmortem actions have owners, due dates, and effectiveness checks.
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
# DNS and Service Discovery Architecture
|
|
2
|
+
|
|
3
|
+
## Responsibilities and boundaries
|
|
4
|
+
DNS resolves names to records; service discovery identifies healthy, eligible service instances and their metadata. Decide whether discovery is client-side, server-side, DNS-based, registry-backed, or provided by a platform control plane. Do not assume DNS alone provides immediate health-aware routing.
|
|
5
|
+
|
|
6
|
+
## Naming and record design
|
|
7
|
+
- Define stable service names independently of ephemeral instance addresses.
|
|
8
|
+
- Choose A/AAAA, CNAME, SRV, TXT, and HTTPS/SVCB records according to client and protocol support.
|
|
9
|
+
- Document authoritative zones, delegation, split-horizon behavior, private/public namespaces, and ownership.
|
|
10
|
+
- Set TTLs based on change frequency, resolver behavior, failover objectives, and cache layers. TTL is not a hard upper bound on propagation because clients and resolvers may cache differently.
|
|
11
|
+
- Plan negative caching, NXDOMAIN behavior, stale answers, and resolver failure modes.
|
|
12
|
+
|
|
13
|
+
## Service discovery
|
|
14
|
+
- Model registration, health state, metadata, endpoint eligibility, deregistration, and stale-instance cleanup.
|
|
15
|
+
- Prefer platform-native discovery when it meets requirements; add a registry only for a demonstrated cross-platform or control-plane need.
|
|
16
|
+
- Define client-side versus server-side load balancing, endpoint refresh cadence, connection reuse, and behavior when no healthy endpoint exists.
|
|
17
|
+
- Avoid synchronized refresh storms; use jitter, bounded caching, and backoff.
|
|
18
|
+
- Treat discovery data as routing input, not authorization. Enforce identity and policy at the connection and request layers.
|
|
19
|
+
|
|
20
|
+
## Failure and security
|
|
21
|
+
- Consider authoritative DNS outage, resolver saturation, stale cache, split-brain registration, control-plane partition, and partial network reachability.
|
|
22
|
+
- Use redundant resolvers and authoritative services; test recursive resolver dependencies and local caching behavior.
|
|
23
|
+
- Protect zone administration with least privilege, MFA, audit, DNSSEC where appropriate, and controlled change processes.
|
|
24
|
+
- Defend against cache poisoning, dangling records/subdomain takeover, unauthorized zone transfer, and SSRF through attacker-controlled DNS answers. Revalidate resolved destinations where threat models require it.
|
|
25
|
+
- Do not use DNS as a substitute for service identity, mTLS, or application authorization.
|
|
26
|
+
|
|
27
|
+
## Operations
|
|
28
|
+
Track query volume, latency, SERVFAIL/NXDOMAIN, cache hit ratio, registration churn, stale endpoint rate, and resolution failures by resolver and zone. Test failover and record changes from real client resolver paths.
|
|
29
|
+
|
|
30
|
+
## Checklist
|
|
31
|
+
- [ ] Namespaces, zones, ownership, and record lifecycle are documented.
|
|
32
|
+
- [ ] TTL and negative-cache behavior match failover objectives.
|
|
33
|
+
- [ ] Discovery freshness, health, and empty-endpoint behavior are defined.
|
|
34
|
+
- [ ] Resolver and control-plane failure modes are tested.
|
|
35
|
+
- [ ] DNS administration and destination validation are secured.
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# Load Balancing and Global Traffic Management
|
|
2
|
+
|
|
3
|
+
## Traffic layers
|
|
4
|
+
Separate global traffic steering (DNS, anycast, or global load balancer), regional ingress, application load balancing, and service-to-service balancing. Document which layer owns health decisions, retries, TLS termination, affinity, and routing policy.
|
|
5
|
+
|
|
6
|
+
## Algorithms and selection
|
|
7
|
+
- Compare round robin, weighted round robin, least connections, least response time, consistent hashing, and locality-aware routing against workload characteristics.
|
|
8
|
+
- Account for long-lived connections, uneven request cost, HTTP/2 and HTTP/3 multiplexing, connection pools, and slow-start behavior.
|
|
9
|
+
- Use weights for controlled rollout and capacity-aware distribution; avoid assuming equal request counts imply equal resource cost.
|
|
10
|
+
- Define session affinity only when required; explain failover and rebalancing behavior when affinity breaks.
|
|
11
|
+
- Prevent retry amplification by assigning retry ownership, deadlines, retry budgets, and idempotency requirements.
|
|
12
|
+
|
|
13
|
+
## Health and failover
|
|
14
|
+
- Distinguish liveness, readiness, synthetic user-journey checks, and dependency health. A process responding to a health endpoint may still be unable to serve traffic correctly.
|
|
15
|
+
- Use passive health signals with active probes carefully; define thresholds, hysteresis, and recovery slow start to avoid flapping.
|
|
16
|
+
- Specify zone and region failure policy, drain behavior, connection termination, and failback criteria.
|
|
17
|
+
- Avoid removing every instance simultaneously due to a shared dependency failure; use dependency-aware and failure-domain-aware policies.
|
|
18
|
+
- Define behavior when health state is unavailable or inconsistent.
|
|
19
|
+
|
|
20
|
+
## Global traffic management
|
|
21
|
+
Compare DNS-based steering, anycast, and proxy-based global load balancing. Consider resolver caching, client location accuracy, BGP convergence, TLS, data residency, active-active data consistency, and regional control-plane dependencies. Route users to a region that can actually serve their data and authorization context.
|
|
22
|
+
|
|
23
|
+
## Security and operations
|
|
24
|
+
Define trusted proxy boundaries, forwarded-header normalization, source IP preservation, DDoS controls, TLS policy, certificate lifecycle, and administrative access. Monitor distribution skew, queueing, connection saturation, health transitions, failover time, rejected traffic, and per-region SLOs.
|
|
25
|
+
|
|
26
|
+
## Checklist
|
|
27
|
+
- [ ] Each routing layer has explicit ownership and policy.
|
|
28
|
+
- [ ] Algorithms match connection and request-cost patterns.
|
|
29
|
+
- [ ] Health checks reflect serving ability without causing correlated ejection.
|
|
30
|
+
- [ ] Retry, drain, failover, and failback behavior are bounded and tested.
|
|
31
|
+
- [ ] Global routing accounts for data locality, consistency, and resolver behavior.
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# CDN and Edge Caching Architecture
|
|
2
|
+
|
|
3
|
+
## Cache placement
|
|
4
|
+
Choose browser, application, reverse-proxy, CDN point-of-presence, and regional cache layers based on content ownership, latency objectives, freshness, privacy, and invalidation needs. A cache is a derived copy, not automatically a source of truth.
|
|
5
|
+
|
|
6
|
+
## Cache key and policy
|
|
7
|
+
- Define cache keys from normalized scheme, host, path, query parameters, selected headers, content encoding, and tenant context.
|
|
8
|
+
- Avoid accidental variation explosion and cache poisoning caused by inconsistent normalization.
|
|
9
|
+
- Set Cache-Control, s-maxage, stale-while-revalidate, stale-if-error, and Vary policies deliberately; verify CDN-specific semantics.
|
|
10
|
+
- Separate public immutable assets from personalized, authenticated, and sensitive responses. Never share-cache user-specific content without a proven safe key and authorization model.
|
|
11
|
+
- Use versioned asset URLs for immutable content and explicit TTL/invalidation for mutable content.
|
|
12
|
+
|
|
13
|
+
## Invalidation and consistency
|
|
14
|
+
- Prefer versioned keys or bounded TTLs where possible; use purge APIs for urgent invalidation with documented propagation guarantees.
|
|
15
|
+
- Define invalidation fan-out, retry, ordering, and observability. Purge success at the control plane does not prove every edge has evicted an object.
|
|
16
|
+
- Use surrogate keys/tags where supported to invalidate related content groups.
|
|
17
|
+
- Decide whether stale content is acceptable during origin failure; encode this per content class.
|
|
18
|
+
- Prevent stampedes with request coalescing, stale-while-revalidate, jittered refresh, and origin shielding.
|
|
19
|
+
|
|
20
|
+
## Dynamic and API traffic
|
|
21
|
+
Evaluate edge compute for authentication prechecks, redirects, routing, lightweight personalization, and request normalization only when data locality, runtime limits, and debugging support are acceptable. Keep authorization and sensitive business invariants at trusted services unless the edge has equivalent policy and identity guarantees.
|
|
22
|
+
|
|
23
|
+
## Security
|
|
24
|
+
Protect origins from direct access and bypass; authenticate origin pulls; validate host and forwarding headers; defend against cache deception, poisoning, request smuggling, and cross-tenant cache leakage. Define WAF, bot management, rate limits, TLS, and DDoS responsibilities across provider and origin.
|
|
25
|
+
|
|
26
|
+
## Economics and observability
|
|
27
|
+
Model cache hit ratio by bytes and requests, origin offload, egress, purge volume, request fees, shield traffic, and regional distribution. Monitor hit/miss, stale responses, origin latency, revalidation, purge lag, cache-key cardinality, and origin errors.
|
|
28
|
+
|
|
29
|
+
## Checklist
|
|
30
|
+
- [ ] Cacheability and privacy classes are explicit.
|
|
31
|
+
- [ ] Cache keys include every representation/tenant dimension and exclude unsafe noise.
|
|
32
|
+
- [ ] Freshness, invalidation, and stale-serving policies are documented.
|
|
33
|
+
- [ ] Origin protection and cache-bypass paths are secured.
|
|
34
|
+
- [ ] Hit ratio, origin offload, purge lag, and cost are measured.
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Cloud Networking and Connectivity Architecture
|
|
2
|
+
|
|
3
|
+
## Network model
|
|
4
|
+
Document accounts/projects, regions, availability zones, VPC/VNet boundaries, subnets, route tables, gateways, firewalls, DNS, private endpoints, and shared network services. Distinguish control-plane configuration from packet data paths.
|
|
5
|
+
|
|
6
|
+
## Addressing and segmentation
|
|
7
|
+
- Establish non-overlapping CIDR plans across environments, regions, acquisitions, and partner networks.
|
|
8
|
+
- Segment by trust, workload, environment, and operational ownership; avoid relying on subnet boundaries as the only authorization control.
|
|
9
|
+
- Define ingress, egress, east-west, and management planes with explicit default-deny policies where practical.
|
|
10
|
+
- Plan IPv4 exhaustion, IPv6 dual-stack support, NAT port capacity, ephemeral ports, and overlapping-address translation.
|
|
11
|
+
- Document route propagation, longest-prefix matching, asymmetric paths, and blackhole behavior.
|
|
12
|
+
|
|
13
|
+
## Connectivity patterns
|
|
14
|
+
Compare public endpoints, NAT, VPN, dedicated private connectivity, peering, transit hubs, private service endpoints, and service-mesh connectivity. Evaluate bandwidth, latency, resiliency, routing limits, encryption, DNS integration, and cost. Peering is not transitive unless the platform explicitly provides a transit design.
|
|
15
|
+
|
|
16
|
+
## Availability and scale
|
|
17
|
+
- Design redundant gateways and paths across failure domains; define failover convergence and route ownership.
|
|
18
|
+
- Model throughput, packets per second, connection tracking, NAT port exhaustion, MTU, fragmentation, and cross-zone/region egress.
|
|
19
|
+
- Avoid central inspection bottlenecks and single-region transit dependencies.
|
|
20
|
+
- Test asymmetric routing, failover, DNS resolution, and effective policy from workload subnets.
|
|
21
|
+
- Treat network policy and firewall changes as versioned infrastructure with review, staged rollout, and rollback.
|
|
22
|
+
|
|
23
|
+
## Security and governance
|
|
24
|
+
Apply least privilege to network administration, log flow decisions, restrict management access, and use workload identity and encryption in addition to network location. Review broad CIDRs, unrestricted egress, exposed control planes, stale rules, and shadow connectivity. Keep segmentation aligned with data classification and threat models.
|
|
25
|
+
|
|
26
|
+
## Operations
|
|
27
|
+
Monitor flow logs, route changes, drops, retransmits, latency, connection tracking, NAT utilization, gateway saturation, and cross-zone/region transfer. Maintain diagrams generated from deployed state and reconcile drift.
|
|
28
|
+
|
|
29
|
+
## Checklist
|
|
30
|
+
- [ ] Address plan avoids overlap and includes IPv6/NAT capacity.
|
|
31
|
+
- [ ] Trust zones and ingress/egress policies are explicit.
|
|
32
|
+
- [ ] Routing, DNS, MTU, and asymmetric paths are validated.
|
|
33
|
+
- [ ] Redundancy and failover are tested across independent failure domains.
|
|
34
|
+
- [ ] Flow visibility, change control, and cost allocation are available.
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Edge Computing Architecture
|
|
2
|
+
|
|
3
|
+
## When edge is justified
|
|
4
|
+
Use edge computing when latency, intermittent connectivity, data locality, bandwidth reduction, physical proximity, or local autonomy materially changes the solution. Compare with regional cloud deployment and client-side execution before adding distributed edge operations.
|
|
5
|
+
|
|
6
|
+
## Placement and topology
|
|
7
|
+
Define device, site, metro, regional, and cloud tiers. Assign workloads to tiers based on latency budgets, data gravity, compute limits, connectivity, and failure domains. Document which tier is authoritative for identity, configuration, state, and policy.
|
|
8
|
+
|
|
9
|
+
## Workload lifecycle
|
|
10
|
+
- Package immutable, signed workloads with reproducible builds and versioned configuration.
|
|
11
|
+
- Define enrollment, attestation, provisioning, staged rollout, health, rollback, decommissioning, and secure recovery.
|
|
12
|
+
- Support offline operation with bounded local queues, durable state, clock uncertainty, and explicit conflict/reconciliation rules.
|
|
13
|
+
- Design for intermittent links, bandwidth caps, delayed updates, duplicate delivery, and long periods without control-plane contact.
|
|
14
|
+
- Keep secrets short-lived where possible; support rotation and revocation during disconnection with a documented risk window.
|
|
15
|
+
|
|
16
|
+
## Data and consistency
|
|
17
|
+
Classify data as local ephemeral, locally durable, replicated, or cloud-authoritative. Define write ownership, synchronization direction, conflict resolution, retention, deletion propagation, and behavior after device replacement. Do not assume eventual connectivity guarantees automatic convergence.
|
|
18
|
+
|
|
19
|
+
## Security and fleet management
|
|
20
|
+
Use hardware-backed identity/secure boot where available, signed artifacts, least-privilege workload identities, encrypted storage, tamper-aware logs, secure update channels, and remote revocation. Treat physical access and device compromise as realistic threats. Bound local privileges and lateral movement.
|
|
21
|
+
|
|
22
|
+
## Reliability and observability
|
|
23
|
+
Define local degraded modes and safe shutdown behavior. Buffer telemetry during disconnection with bounded storage and priority; avoid unbounded replay storms after reconnect. Monitor fleet version skew, last-seen time, update success, resource pressure, synchronization lag, and policy compliance.
|
|
24
|
+
|
|
25
|
+
## Economics
|
|
26
|
+
Include edge hardware, replacement, connectivity, field support, remote hands, energy, local storage, fleet management, and lifecycle costs—not only cloud compute savings.
|
|
27
|
+
|
|
28
|
+
## Checklist
|
|
29
|
+
- [ ] Edge placement solves a measured latency, locality, or connectivity constraint.
|
|
30
|
+
- [ ] Offline behavior, local authority, and reconciliation are explicit.
|
|
31
|
+
- [ ] Artifact signing, device identity, updates, and revocation are designed.
|
|
32
|
+
- [ ] Fleet observability and bounded telemetry replay are available.
|
|
33
|
+
- [ ] Full lifecycle and field-support costs are modeled.
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Storage Infrastructure and Storage Systems
|
|
2
|
+
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns infrastructure-level block, file, and object storage behavior: performance, durability, replication, lifecycle tiers, backup integration, and storage cost. Use [data and storage](03-data-and-storage.md) for application data models and transaction semantics; [privacy engineering](54-privacy-engineering-and-data-governance.md) for retention and deletion obligations; and [secrets/KMS/PKI](51-secrets-kms-pki-and-key-management.md) for cryptographic key lifecycle. Storage durability, replication, snapshots, and retention locks do not by themselves prove application consistency, recoverability, or compliance with a deletion request.
|
|
6
|
+
|
|
7
|
+
## Workload characterization
|
|
8
|
+
Classify data by access pattern, durability, latency, throughput, IOPS, block size, concurrency, growth, retention, sharing, locality, and recovery objectives. Separate application database semantics from underlying block, file, and object storage guarantees.
|
|
9
|
+
|
|
10
|
+
## Storage types
|
|
11
|
+
- Block storage: low-level volumes for databases and filesystems; evaluate IOPS, throughput, latency, queue depth, burst credits, multi-attach semantics, snapshots, and resize behavior.
|
|
12
|
+
- File storage: shared hierarchical namespace; evaluate metadata operations, locking, consistency, throughput, client caching, and concurrent writers.
|
|
13
|
+
- Object storage: API-addressed objects; evaluate durability, consistency model, multipart uploads, lifecycle tiers, versioning, retention lock, event notifications, and request cost.
|
|
14
|
+
- Ephemeral/local NVMe: high performance with host or instance lifecycle coupling; define replication and rebuild behavior.
|
|
15
|
+
- Distributed storage: understand replication/erasure coding, quorum, failure domains, repair bandwidth, rebalancing, and correlated failure risks.
|
|
16
|
+
|
|
17
|
+
## Durability and recovery
|
|
18
|
+
Distinguish durability from availability and backup. Define replication scope, versioning, immutable backups, retention, encryption keys, restore testing, point-in-time recovery, and deletion protection. Validate recovery time and data integrity, not only snapshot creation. Understand consistency and atomicity boundaries across objects, files, and volumes.
|
|
19
|
+
|
|
20
|
+
## Performance and capacity
|
|
21
|
+
Model IOPS, throughput, queue depth, request size, latency percentiles, metadata hot spots, cache hit ratio, compaction, garbage collection, and rebuild overhead. Include burst limits, throttling, noisy neighbors, and performance degradation during repair or snapshots. Benchmark with production-like data and concurrency.
|
|
22
|
+
|
|
23
|
+
## Cost and lifecycle
|
|
24
|
+
Include provisioned capacity, provisioned performance, operations/requests, snapshots, replication, retrieval, egress, minimum retention, and early deletion fees. Define tiering, lifecycle transitions, orphan detection, capacity alerts, and ownership.
|
|
25
|
+
|
|
26
|
+
## Security
|
|
27
|
+
Use encryption at rest and in transit, scoped identities, private endpoints, least-privilege access, access logging, key lifecycle, tenant isolation, and secure deletion. For object storage, explicitly review public access, bucket policies, presigned URL expiry, and cross-account access.
|
|
28
|
+
|
|
29
|
+
## Checklist
|
|
30
|
+
- [ ] Storage class matches access pattern and latency/throughput needs.
|
|
31
|
+
- [ ] Durability, availability, replication, backup, and restore are distinguished.
|
|
32
|
+
- [ ] IOPS, throughput, queue depth, burst, and rebuild behavior are modeled.
|
|
33
|
+
- [ ] Lifecycle, retention, encryption, and tenant access policies are explicit.
|
|
34
|
+
- [ ] Restore and failure-domain recovery are tested.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# TCP, QUIC, and Network Performance Engineering
|
|
2
|
+
|
|
3
|
+
## Transport selection
|
|
4
|
+
Understand TCP connection setup, congestion control, retransmission, flow control, head-of-line blocking, keepalive, and connection reuse. Evaluate QUIC/HTTP/3 for connection migration, stream multiplexing, handshake behavior, and deployment compatibility; do not assume it improves every workload. QUIC loss recovery retransmits information in newly formed packets/frames rather than retransmitting an entire lost packet; do not describe it as TCP-style packet retransmission. Connection migration is subject to connection-ID and path-validation rules and can be blocked by deployment policy or intermediaries. Verify transport claims against [RFC 9000](https://www.rfc-editor.org/rfc/rfc9000.html) and the relevant recovery specification.
|
|
5
|
+
|
|
6
|
+
## Latency budget
|
|
7
|
+
Decompose DNS lookup, connection establishment, TLS handshake, network RTT, congestion/window growth, server queueing, application processing, and response transfer. Separate cold and warm connections, client and server time, and network from application delay.
|
|
8
|
+
|
|
9
|
+
## Throughput and congestion
|
|
10
|
+
- Estimate bandwidth-delay product (BDP = bandwidth × RTT) to understand in-flight data requirements.
|
|
11
|
+
- Investigate packet loss, retransmissions, congestion window, receive window, pacing, and path MTU.
|
|
12
|
+
- Avoid increasing socket buffers or concurrency without evidence; buffers can increase queueing and memory pressure.
|
|
13
|
+
- Account for NAT, firewalls, proxies, load balancers, and middleboxes that alter idle timeout, connection reuse, or protocol support.
|
|
14
|
+
- Test realistic RTT, loss, jitter, bandwidth constraints, and asymmetric paths.
|
|
15
|
+
|
|
16
|
+
## Connection management
|
|
17
|
+
Define pool sizing, reuse, idle timeout, keepalive, maximum lifetime, DNS refresh, connection draining, and retry behavior. Avoid synchronized reconnect storms after deploys or network restoration. For streaming and long-lived connections, define heartbeat, backpressure, reconnect jitter, and resume semantics.
|
|
18
|
+
|
|
19
|
+
## Diagnostics
|
|
20
|
+
Use packet captures, flow logs, socket metrics, retransmission counters, DNS/TLS timing, and distributed traces together. Packet captures can contain sensitive payloads and identifiers; restrict capture scope, access, and retention. eBPF tools can improve visibility but require kernel compatibility, privilege controls, and overhead evaluation.
|
|
21
|
+
|
|
22
|
+
## Checklist
|
|
23
|
+
- [ ] Transport and protocol behavior are verified for clients and intermediaries.
|
|
24
|
+
- [ ] Cold/warm connection and end-to-end latency are measured separately.
|
|
25
|
+
- [ ] RTT, loss, MTU, retransmissions, and BDP inform tuning.
|
|
26
|
+
- [ ] Pools, keepalive, timeouts, draining, and reconnect jitter are bounded.
|
|
27
|
+
- [ ] Packet and kernel-level diagnostics follow privacy and privilege controls.
|