system-design-primer-skill 0.0.0-stage → 2.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ATTRIBUTION.md +10 -0
- package/LICENSE +7 -0
- package/README.md +180 -2
- package/SKILL.md +447 -0
- package/VERSION +1 -0
- package/package.json +78 -4
- package/references/00-primer-map-and-attribution.md +29 -0
- package/references/01-requirements-and-capacity.md +107 -0
- package/references/02-architecture-and-networking.md +94 -0
- package/references/03-data-and-storage.md +88 -0
- package/references/04-caching.md +78 -0
- package/references/05-async-and-messaging.md +77 -0
- package/references/06-distributed-systems.md +59 -0
- package/references/07-reliability-and-resilience.md +71 -0
- package/references/08-api-security-observability.md +62 -0
- package/references/09-modern-engineering.md +58 -0
- package/references/10-workflows-and-checklists.md +97 -0
- package/references/11-case-studies.md +74 -0
- package/references/12-ai-systems-architecture.md +139 -0
- package/references/13-architecture-intelligence.md +91 -0
- package/references/14-api-and-data-contracts.md +80 -0
- package/references/15-failure-analysis.md +84 -0
- package/references/16-multi-agent-architecture.md +76 -0
- package/references/17-agent-harness-and-runtime.md +87 -0
- package/references/18-mcp-architecture.md +67 -0
- package/references/19-agent-security-and-permissions.md +74 -0
- package/references/20-agent-state-and-memory.md +72 -0
- package/references/21-agent-evaluation-and-observability.md +90 -0
- package/references/22-kubernetes-and-container-platforms.md +29 -0
- package/references/23-service-mesh-and-traffic-management.md +21 -0
- package/references/24-multi-region-and-disaster-recovery.md +24 -0
- package/references/25-platform-engineering-and-developer-experience.md +26 -0
- package/references/26-finops-and-cloud-cost.md +29 -0
- package/references/27-production-observability-and-slo.md +29 -0
- package/references/28-event-streaming-and-kafka-pulsar.md +27 -0
- package/references/29-event-sourcing-and-cqrs.md +23 -0
- package/references/30-change-data-capture-and-debezium.md +27 -0
- package/references/31-data-lakehouse-and-stream-batch.md +24 -0
- package/references/32-workflow-orchestration-and-temporal.md +24 -0
- package/references/33-schema-registry-and-event-governance.md +22 -0
- package/references/34-postgresql-internals-and-performance.md +43 -0
- package/references/35-mysql-innodb-and-performance.md +27 -0
- package/references/36-sharding-and-distributed-sql.md +22 -0
- package/references/37-consensus-and-distributed-transactions.md +25 -0
- package/references/38-search-architecture-elasticsearch-opensearch.md +24 -0
- package/references/39-vector-databases-and-hybrid-retrieval.md +31 -0
- package/references/40-zero-downtime-database-migrations.md +27 -0
- package/references/41-api-gateway-and-edge-architecture.md +23 -0
- package/references/42-rest-api-design-and-lifecycle.md +23 -0
- package/references/43-graphql-architecture-and-governance.md +16 -0
- package/references/44-grpc-and-service-contracts.md +16 -0
- package/references/45-websocket-sse-and-realtime-systems.md +16 -0
- package/references/46-realtime-collaboration-ot-crdt.md +16 -0
- package/references/47-api-security-rate-limiting-and-abuse.md +16 -0
- package/references/48-oauth-oidc-and-passkeys.md +20 -0
- package/references/49-iam-rbac-abac-rebac.md +16 -0
- package/references/50-zero-trust-and-workload-identity.md +16 -0
- package/references/51-secrets-kms-pki-and-key-management.md +16 -0
- package/references/52-supply-chain-security-sbom-slsa.md +16 -0
- package/references/53-threat-modeling-stride-and-attack-trees.md +16 -0
- package/references/54-privacy-engineering-and-data-governance.md +16 -0
- package/references/55-opentelemetry-and-instrumentation.md +50 -0
- package/references/56-distributed-tracing-and-context-propagation.md +43 -0
- package/references/57-continuous-profiling-and-performance-diagnostics.md +45 -0
- package/references/58-load-stress-soak-testing-and-workload-modeling.md +44 -0
- package/references/59-capacity-planning-and-performance-modeling.md +34 -0
- package/references/60-chaos-engineering-and-resilience-experiments.md +41 -0
- package/references/61-incident-management-and-postmortems.md +34 -0
- package/references/62-dns-and-service-discovery.md +35 -0
- package/references/63-load-balancing-and-global-traffic-management.md +31 -0
- package/references/64-cdn-and-edge-caching.md +34 -0
- package/references/65-cloud-networking-and-connectivity.md +34 -0
- package/references/66-edge-computing-architecture.md +33 -0
- package/references/67-storage-infrastructure-and-storage-systems.md +30 -0
- package/references/68-tcp-quic-and-network-performance.md +27 -0
- package/references/69-task-routing-and-evidence.md +58 -0
- package/scripts/install.ps1 +124 -0
- package/scripts/install.sh +105 -0
- package/scripts/validate.py +376 -0
- package/templates/adr-template.md +32 -0
- package/templates/agent-evaluation-plan-template.md +57 -0
- package/templates/agent-security-review-template.md +46 -0
- package/templates/agent-system-design-template.md +74 -0
- package/templates/api-architecture-design-template.md +33 -0
- package/templates/api-security-and-rate-limit-review-template.md +29 -0
- package/templates/architecture-decision-matrix-template.md +48 -0
- package/templates/architecture-design-template.md +95 -0
- package/templates/architecture-review-scorecard-template.md +36 -0
- package/templates/architecture-review-template.md +52 -0
- package/templates/cdc-data-pipeline-template.md +41 -0
- package/templates/cdn-and-edge-strategy-template.md +36 -0
- package/templates/chaos-experiment-template.md +39 -0
- package/templates/cloud-cost-model-template.md +37 -0
- package/templates/database-architecture-review-template.md +33 -0
- package/templates/edge-computing-readiness-template.md +36 -0
- package/templates/event-sourcing-cqrs-template.md +41 -0
- package/templates/event-streaming-design-template.md +47 -0
- package/templates/failure-analysis-template.md +36 -0
- package/templates/identity-and-access-architecture-template.md +29 -0
- package/templates/incident-postmortem-template.md +39 -0
- package/templates/multi-region-disaster-recovery-template.md +43 -0
- package/templates/network-architecture-design-template.md +38 -0
- package/templates/observability-architecture-template.md +52 -0
- package/templates/performance-test-plan-template.md +52 -0
- package/templates/privacy-data-governance-template.md +24 -0
- package/templates/production-readiness-checklist-template.md +52 -0
- package/templates/realtime-system-design-template.md +28 -0
- package/templates/security-threat-model-template.md +26 -0
- package/templates/sharding-distributed-sql-design-template.md +34 -0
- package/templates/storage-architecture-design-template.md +35 -0
- package/templates/supply-chain-security-review-template.md +20 -0
- package/templates/zero-downtime-database-migration-template.md +32 -0
|
@@ -0,0 +1,107 @@
|
|
|
1
|
+
# Requirements, non-functional goals, and capacity estimation
|
|
2
|
+
|
|
3
|
+
## 1. Frame the problem
|
|
4
|
+
|
|
5
|
+
Separate:
|
|
6
|
+
|
|
7
|
+
- **Functional requirements**: user-visible capabilities and business operations.
|
|
8
|
+
- **Non-functional requirements**: latency, throughput, availability, durability, consistency, security, privacy, cost, operability, and maintainability.
|
|
9
|
+
- **Constraints**: current stack, team size, budget, cloud/vendor, compliance, deadlines, data residency, integrations, and legacy compatibility.
|
|
10
|
+
- **Out of scope**: features not required in this phase.
|
|
11
|
+
|
|
12
|
+
Write requirements as observable outcomes. Prefer “95% of interactive reads finish within 300 ms at 2,000 RPS” over “the system must be fast.” Treat numbers as user-supplied goals, measured facts, or explicit assumptions; label which.
|
|
13
|
+
|
|
14
|
+
## 2. Workload model
|
|
15
|
+
|
|
16
|
+
Capture at minimum:
|
|
17
|
+
|
|
18
|
+
| Dimension | Questions |
|
|
19
|
+
|---|---|
|
|
20
|
+
| Users | DAU/MAU, tenants, geographic distribution, concurrency |
|
|
21
|
+
| Traffic | average/peak RPS, burst duration, read/write ratio, seasonality |
|
|
22
|
+
| Payload | request/response size, media size, batch size |
|
|
23
|
+
| Data | new rows/day, size/row, retention, growth, hot/cold ratio |
|
|
24
|
+
| Latency | P50/P95/P99 by critical operation |
|
|
25
|
+
| Availability | SLO, maintenance tolerance, dependency SLOs |
|
|
26
|
+
| Recovery | RPO, RTO, backup frequency, restore test cadence |
|
|
27
|
+
|
|
28
|
+
Do not infer RPS directly from DAU without modeling sessions, actions per session, and request fan-out.
|
|
29
|
+
|
|
30
|
+
## 3. Estimation formulas
|
|
31
|
+
|
|
32
|
+
Use consistent units and show substitutions.
|
|
33
|
+
|
|
34
|
+
```text
|
|
35
|
+
average requests/sec = requests/day / 86,400
|
|
36
|
+
peak requests/sec = average requests/sec × peak-to-average factor
|
|
37
|
+
writes/sec = total requests/sec × write fraction
|
|
38
|
+
raw bytes = record count × average serialized bytes/record
|
|
39
|
+
retained bytes = raw bytes × retention periods
|
|
40
|
+
primary dataset bytes ≈ logical bytes × (1 + index overhead ratio + engine overhead ratio)
|
|
41
|
+
total database copies ≈ primary dataset bytes × total copy count
|
|
42
|
+
backup / snapshot / WAL / temporary migration space = estimate separately
|
|
43
|
+
bandwidth bytes/sec ≈ RPS × mean payload bytes
|
|
44
|
+
queue backlog growth/sec = arrival rate - successful processing rate
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
For storage, distinguish logical data from indexes, replicas, snapshots, backups, WAL/binlogs, compaction headroom, and temporary migration copies. For network estimates, distinguish bytes from bits and include protocol overhead and peak concurrency.
|
|
48
|
+
|
|
49
|
+
### Storage model and caveats
|
|
50
|
+
|
|
51
|
+
Use separate layers rather than adding unlike factors together:
|
|
52
|
+
|
|
53
|
+
1. Logical retained data = daily logical ingest × retention days, adjusted for growth, updates, compression and deletion behavior.
|
|
54
|
+
2. Primary dataset storage ≈ logical retained data × (1 + index overhead ratio + engine overhead ratio). Define the ratios from measurements or clearly labeled assumptions.
|
|
55
|
+
3. Total database storage ≈ primary dataset storage × total copy count. The copy count includes the primary and every full replica; do not count a replica factor twice.
|
|
56
|
+
4. Backups, snapshots, point-in-time recovery logs, object-store exports, temporary migration copies and free-space/headroom are separate line items. Their retention and deduplication behavior differ from live database storage.
|
|
57
|
+
|
|
58
|
+
This remains a first-order model: indexes may differ between replicas, compression may vary, and storage engines can have compaction or vacuum headroom requirements. Validate against production-like data and the selected database's metrics.
|
|
59
|
+
|
|
60
|
+
## 4. Example estimate
|
|
61
|
+
|
|
62
|
+
Assumptions: 5 million daily requests; average request/response payload is 2 KiB; peak-to-average factor is 8.
|
|
63
|
+
|
|
64
|
+
```text
|
|
65
|
+
average RPS = 5,000,000 / 86,400 ≈ 58
|
|
66
|
+
peak RPS ≈ 58 × 8 = 464
|
|
67
|
+
mean egress ≈ 58 × 2 KiB ≈ 116 KiB/s
|
|
68
|
+
peak egress ≈ 464 × 2 KiB ≈ 928 KiB/s
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
This is only a first-order estimate. It does not account for image/video payloads, retries, multiple downstream calls per request, TLS, cache hits, compression, or response fan-out. State those omissions.
|
|
72
|
+
|
|
73
|
+
## 5. Latency and throughput
|
|
74
|
+
|
|
75
|
+
- Latency is elapsed time for an operation; throughput is completed work per unit time.
|
|
76
|
+
- Measure end-to-end and per dependency. A fast service can still have poor end-to-end latency if it performs serial network calls.
|
|
77
|
+
- Report percentiles, especially P95/P99. Averages hide long-tail behavior.
|
|
78
|
+
- Little's Law: `L = λW` (average in-flight work = arrival rate × average time in system), when the system is stable and the terms use consistent units. It helps estimate concurrency, not capacity by itself.
|
|
79
|
+
- Distinguish queue wait time from execution time.
|
|
80
|
+
|
|
81
|
+
## 6. Capacity safety margin
|
|
82
|
+
|
|
83
|
+
Never size to exactly the average. Include:
|
|
84
|
+
|
|
85
|
+
- peak and burst traffic;
|
|
86
|
+
- failure of one instance or availability zone;
|
|
87
|
+
- rolling deploy capacity;
|
|
88
|
+
- background jobs and replays;
|
|
89
|
+
- growth horizon and data retention;
|
|
90
|
+
- noisy neighbors and resource headroom.
|
|
91
|
+
|
|
92
|
+
Avoid universal “always use 2x” rules. Choose margins based on measured variance, failure policy, cost, and business risk.
|
|
93
|
+
|
|
94
|
+
## 7. Cost model
|
|
95
|
+
|
|
96
|
+
Estimate recurring costs as a workload model: compute-hours, memory, database IOPS/storage, network egress, managed service request units, cache memory, queue throughput/storage, observability ingestion, backups, and cross-region replication. Explain which component dominates and what assumption changes the result.
|
|
97
|
+
|
|
98
|
+
## 8. Output checklist
|
|
99
|
+
|
|
100
|
+
- [ ] User journeys and critical paths identified.
|
|
101
|
+
- [ ] FR/NFR and out-of-scope items documented.
|
|
102
|
+
- [ ] Inputs classified as measured, user-provided, or assumed.
|
|
103
|
+
- [ ] Average and peak workload estimated.
|
|
104
|
+
- [ ] Storage, network, and queue growth estimated.
|
|
105
|
+
- [ ] Latency percentiles and SLO are explicit.
|
|
106
|
+
- [ ] Failure capacity and growth margin considered.
|
|
107
|
+
- [ ] Estimates include caveats and validation plan.
|
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
# Architecture, request paths, and networking
|
|
2
|
+
|
|
3
|
+
## 1. Start with a request path
|
|
4
|
+
|
|
5
|
+
Map the user request end-to-end:
|
|
6
|
+
|
|
7
|
+
```text
|
|
8
|
+
Client → DNS → Edge/CDN/WAF → Load Balancer/Ingress → API/Application
|
|
9
|
+
→ Domain logic → Database/Cache/Queue/External dependency
|
|
10
|
+
→ Response or asynchronous event
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
The exact path depends on deployment. Do not draw components that the repository or chosen platform does not actually use.
|
|
14
|
+
|
|
15
|
+
## 2. Scaling choices
|
|
16
|
+
|
|
17
|
+
### Vertical scaling
|
|
18
|
+
|
|
19
|
+
Increase CPU, RAM, IOPS, or network capacity of one node. It is often simpler and can be the right first step, but has hardware/instance ceilings and may not remove a single-node failure domain.
|
|
20
|
+
|
|
21
|
+
### Horizontal scaling
|
|
22
|
+
|
|
23
|
+
Add instances and distribute work. It requires attention to statelessness, shared state, session handling, connection limits, distributed coordination, and uneven load.
|
|
24
|
+
|
|
25
|
+
Keep application instances stateless where practical. Put durable/shared state in a database, cache, object store, or dedicated session service. If local state is required, document affinity and failover limitations.
|
|
26
|
+
|
|
27
|
+
## 3. DNS and CDN
|
|
28
|
+
|
|
29
|
+
- DNS maps names to records and can steer traffic, but resolver caching and TTL mean changes are not instant everywhere.
|
|
30
|
+
- A CDN caches eligible content at edge locations. Define cache keys, TTL, purge/invalidation, authorization, and behavior for personalized content.
|
|
31
|
+
- Separate immutable versioned assets from mutable/private content.
|
|
32
|
+
- Do not cache personalized or sensitive responses publicly unless authorization and cache-key boundaries are proven correct.
|
|
33
|
+
- CDN does not automatically make dynamic APIs fast; origin latency and cacheability still matter.
|
|
34
|
+
|
|
35
|
+
## 4. Load balancing and proxies
|
|
36
|
+
|
|
37
|
+
- L4 routing uses transport-level information such as IP/port; L7 routing can consider HTTP host/path/headers and application semantics.
|
|
38
|
+
- A reverse proxy is an intermediary for upstream services; a load balancer distributes traffic. One product may perform both roles.
|
|
39
|
+
- Health checks must distinguish liveness (process should restart) from readiness (instance can receive traffic). Poor health checks can cause cascading removal of healthy nodes.
|
|
40
|
+
- Consider connection draining, keep-alive, TLS termination, request body size, header limits, timeouts, and proxy buffer limits.
|
|
41
|
+
- Do not trust client-supplied forwarding headers unless the proxy chain is controlled and configured.
|
|
42
|
+
|
|
43
|
+
## 5. Application architecture
|
|
44
|
+
|
|
45
|
+
### Modular monolith
|
|
46
|
+
|
|
47
|
+
A single deployable system with explicit internal modules and boundaries. Often a good default: simpler transactions, deployment, debugging, and local development. Enforce boundaries in code, not only by folder names.
|
|
48
|
+
|
|
49
|
+
### Service-oriented / microservices
|
|
50
|
+
|
|
51
|
+
Separate deployable services can improve independent scaling, ownership, release cadence, and fault isolation. They also add network failure, distributed tracing, versioning, deployment, data ownership, and consistency costs. Split along stable business capabilities and team ownership, not technical layers alone.
|
|
52
|
+
|
|
53
|
+
### API gateway and BFF
|
|
54
|
+
|
|
55
|
+
An API gateway commonly provides routing, authentication integration, quotas, and cross-cutting policies. A Backend-for-Frontend tailors APIs for a client experience. Avoid embedding core business logic into a gateway.
|
|
56
|
+
|
|
57
|
+
### Service discovery
|
|
58
|
+
|
|
59
|
+
Services need a reliable way to locate healthy instances (DNS, platform discovery, registry, or service mesh). Define registration, health, TTL, failover, and stale endpoint behavior.
|
|
60
|
+
|
|
61
|
+
## 6. Communication choices
|
|
62
|
+
|
|
63
|
+
| Pattern | Good fit | Costs / cautions |
|
|
64
|
+
|---|---|---|
|
|
65
|
+
| HTTP/REST | public APIs, resource-oriented interfaces | payload/versioning design, chatty interactions |
|
|
66
|
+
| gRPC/RPC | typed internal service calls, streaming use cases | generated contracts, compatibility and debugging |
|
|
67
|
+
| WebSocket | bidirectional real-time client sessions | connection lifecycle, fan-out, backpressure |
|
|
68
|
+
| Async events | decoupled workflows and buffering | eventual consistency, retries, duplicate delivery |
|
|
69
|
+
| Batch jobs | bulk and periodic work | freshness delay, checkpointing and restartability |
|
|
70
|
+
|
|
71
|
+
TCP is a reliable ordered byte stream; UDP is a datagram transport without TCP's built-in delivery/order guarantees. HTTP/3 uses QUIC over UDP. Do not equate “UDP” with “unreliable application” or assume every protocol has the same semantics.
|
|
72
|
+
|
|
73
|
+
## 7. Design a reliable request boundary
|
|
74
|
+
|
|
75
|
+
For each inbound API define:
|
|
76
|
+
|
|
77
|
+
- authentication and authorization;
|
|
78
|
+
- request size and schema limits;
|
|
79
|
+
- rate/concurrency limits;
|
|
80
|
+
- deadline and timeout budget;
|
|
81
|
+
- idempotency semantics for writes;
|
|
82
|
+
- validation and error contract;
|
|
83
|
+
- trace/correlation identifiers;
|
|
84
|
+
- response caching policy;
|
|
85
|
+
- compatibility and deprecation policy.
|
|
86
|
+
|
|
87
|
+
## 8. Review questions
|
|
88
|
+
|
|
89
|
+
- Is the critical path synchronous for a reason?
|
|
90
|
+
- Are there serial network hops that could be removed or parallelized safely?
|
|
91
|
+
- Are service-to-service calls bounded by deadlines?
|
|
92
|
+
- Can one tenant or hot key monopolize a shared resource?
|
|
93
|
+
- What happens if DNS, CDN, ingress, gateway, or a downstream service is degraded?
|
|
94
|
+
- Does horizontal scaling actually scale the database and all shared dependencies?
|
|
@@ -0,0 +1,88 @@
|
|
|
1
|
+
# Data modeling, databases, and storage scaling
|
|
2
|
+
|
|
3
|
+
## 1. Choose by access patterns and invariants
|
|
4
|
+
|
|
5
|
+
Select storage after defining entities, queries, update patterns, consistency, transaction boundaries, retention, and operational skills. “SQL vs NoSQL” is not a speed contest.
|
|
6
|
+
|
|
7
|
+
| Data model | Typical fit | Important trade-off |
|
|
8
|
+
|---|---|---|
|
|
9
|
+
| Relational | transactions, joins, constraints, structured entities | schema design, write contention, scale topology |
|
|
10
|
+
| Key-value | direct lookup by key, session/cache state | limited ad-hoc querying |
|
|
11
|
+
| Document | aggregate-shaped, evolving records | embedding/reference trade-offs, index discipline |
|
|
12
|
+
| Wide-column | high write volume and known key-oriented queries | query-first data modeling, partition design |
|
|
13
|
+
| Graph | relationship traversal and graph algorithms | traversal cost and graph operations |
|
|
14
|
+
| Object storage | blobs, media, exports, backups | not a transactional row store; metadata/index elsewhere |
|
|
15
|
+
|
|
16
|
+
## 2. Relational data design
|
|
17
|
+
|
|
18
|
+
- Model business invariants in constraints where feasible: primary key, unique key, foreign key, check constraints.
|
|
19
|
+
- Normalize to avoid update anomalies; denormalize deliberately for a measured read path, with an explicit refresh/consistency strategy.
|
|
20
|
+
- Design indexes for actual predicates, joins, sort order, and selectivity. Composite index column order matters.
|
|
21
|
+
- Avoid `SELECT *` in high-volume paths when it loads unused columns.
|
|
22
|
+
- Use `EXPLAIN` / query plans and production-like data distributions.
|
|
23
|
+
- Watch N+1 queries, unbounded result sets, offset pagination at high offsets, lock waits, connection pool saturation, and long transactions.
|
|
24
|
+
|
|
25
|
+
## 3. ACID and isolation
|
|
26
|
+
|
|
27
|
+
- Atomicity: a transaction commits all intended changes or none.
|
|
28
|
+
- Consistency: constraints and application invariants remain true.
|
|
29
|
+
- Isolation: concurrent transactions obey a defined isolation behavior.
|
|
30
|
+
- Durability: committed data survives the failures covered by the database's guarantees.
|
|
31
|
+
|
|
32
|
+
Isolation levels differ by database and implementation. Consider dirty reads, non-repeatable reads, phantom reads, write skew, lost updates, and serialization failures. Do not assume a level's name fully describes exact behavior across products.
|
|
33
|
+
|
|
34
|
+
For inventory-like counters, a conditional update such as `UPDATE inventory SET available = available - 1 WHERE sku = ? AND available > 0` can be safer than read-then-write, provided affected-row checks and transaction boundaries are correct.
|
|
35
|
+
|
|
36
|
+
## 4. Replication and read/write split
|
|
37
|
+
|
|
38
|
+
- Primary/replica separates write authority from read capacity or availability copies.
|
|
39
|
+
- Replication can be synchronous or asynchronous; asynchronous replication introduces lag and stale reads.
|
|
40
|
+
- Read-after-write requirements may need primary reads, session consistency, replica waiting, or sticky routing. Define the guarantee explicitly.
|
|
41
|
+
- Failover needs leader election/coordination, fencing against an old primary, routing update, and data-loss policy. “Two DBs” alone does not provide safe HA.
|
|
42
|
+
- Multi-primary writes require conflict resolution, ownership partitioning, or coordination. Last-write-wins can silently discard meaningful business updates.
|
|
43
|
+
|
|
44
|
+
## 5. Partitioning and sharding
|
|
45
|
+
|
|
46
|
+
Partitioning divides a logical dataset into manageable subsets. Sharding distributes subsets across independent nodes.
|
|
47
|
+
|
|
48
|
+
Common shard keys: tenant ID, user ID, geography, time bucket, or a compound key. Evaluate:
|
|
49
|
+
|
|
50
|
+
- cardinality and distribution;
|
|
51
|
+
- hot tenants/keys and write skew;
|
|
52
|
+
- ability to target a query to one shard;
|
|
53
|
+
- cross-shard joins, transactions, and aggregations;
|
|
54
|
+
- resharding and data movement;
|
|
55
|
+
- ID generation and uniqueness;
|
|
56
|
+
- backup, restore, and rebalancing.
|
|
57
|
+
|
|
58
|
+
Hash sharding spreads keys but can make range queries harder. Range sharding supports range locality but can create hot ranges. Consistent hashing can reduce movement in some topologies but does not remove hot-key or rebalancing design.
|
|
59
|
+
|
|
60
|
+
Do not shard before observing that simpler tuning, replicas, partitioning, archiving, or workload separation is insufficient.
|
|
61
|
+
|
|
62
|
+
## 6. Schema evolution and migrations
|
|
63
|
+
|
|
64
|
+
Prefer expand-migrate-contract:
|
|
65
|
+
|
|
66
|
+
1. Add backward-compatible schema/field/column.
|
|
67
|
+
2. Deploy code that can read old and new formats.
|
|
68
|
+
3. Backfill in bounded, resumable batches.
|
|
69
|
+
4. Validate counts, checksums, and business invariants.
|
|
70
|
+
5. Switch reads/writes gradually.
|
|
71
|
+
6. Stop old writes and observe.
|
|
72
|
+
7. Remove obsolete schema only after rollback window closes.
|
|
73
|
+
|
|
74
|
+
Make migrations idempotent or safely restartable. Avoid long table locks and unbounded backfills. Test against production-like data volume. Keep rollback realistic: a destructive schema change may not be reversible without a restore or forward-fix.
|
|
75
|
+
|
|
76
|
+
## 7. Data ownership across services
|
|
77
|
+
|
|
78
|
+
A service should own its authoritative write model. Other services consume APIs/events or materialized read models instead of directly updating its tables. Cross-service workflows may need Saga/orchestration, outbox, compensation, and reconciliation rather than a distributed ACID transaction.
|
|
79
|
+
|
|
80
|
+
## 8. Checklist
|
|
81
|
+
|
|
82
|
+
- [ ] Data model follows actual queries and business invariants.
|
|
83
|
+
- [ ] Transactions and isolation anomalies are understood.
|
|
84
|
+
- [ ] Indexes have evidence and write-cost review.
|
|
85
|
+
- [ ] Read/write replica lag is addressed where relevant.
|
|
86
|
+
- [ ] Shard key avoids skew and supports common queries.
|
|
87
|
+
- [ ] Schema migration is online, observable, and recoverable.
|
|
88
|
+
- [ ] Backups are restored in a tested exercise.
|
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
# Caching patterns and cache correctness
|
|
2
|
+
|
|
3
|
+
## 1. Cache only with a reason
|
|
4
|
+
|
|
5
|
+
A cache is justified when repeated reads, expensive computation, latency goals, or load reduction outweigh staleness, memory, invalidation, and operational costs. Measure hit rate, miss penalty, hot-key concentration, eviction, memory usage, and stale-read impact.
|
|
6
|
+
|
|
7
|
+
Potential layers:
|
|
8
|
+
|
|
9
|
+
- browser/client cache;
|
|
10
|
+
- CDN/edge cache;
|
|
11
|
+
- reverse-proxy cache;
|
|
12
|
+
- application-local cache;
|
|
13
|
+
- distributed cache (e.g. Redis);
|
|
14
|
+
- database buffer/query cache.
|
|
15
|
+
|
|
16
|
+
Each layer has a different consistency and invalidation boundary.
|
|
17
|
+
|
|
18
|
+
## 2. Common patterns
|
|
19
|
+
|
|
20
|
+
### Cache-aside
|
|
21
|
+
|
|
22
|
+
Read: cache lookup → on miss read source of truth → populate cache → return. Write: update source of truth → invalidate cache (or use a carefully designed refresh). Simple and common, but concurrent misses can stampede and stale refill races need handling.
|
|
23
|
+
|
|
24
|
+
### Read-through
|
|
25
|
+
|
|
26
|
+
Application asks the cache; cache abstraction loads from backing store on miss. Centralizes loading but couples cache to data access behavior.
|
|
27
|
+
|
|
28
|
+
### Write-through
|
|
29
|
+
|
|
30
|
+
Write path updates cache and backing store synchronously through a coordinated abstraction. Better immediate read freshness in some designs; write latency and failure semantics must be explicit.
|
|
31
|
+
|
|
32
|
+
### Write-behind / write-back
|
|
33
|
+
|
|
34
|
+
Accept write in cache and persist asynchronously. Fast but increases data-loss and recovery risk. Use only with durable buffering, explicit loss tolerance, replay, and monitoring.
|
|
35
|
+
|
|
36
|
+
### Refresh-ahead
|
|
37
|
+
|
|
38
|
+
Refresh hot entries before expiry. Useful for predictable hot data, but can waste work and requires jitter and failure handling.
|
|
39
|
+
|
|
40
|
+
## 3. Invalidation and race conditions
|
|
41
|
+
|
|
42
|
+
Cache invalidation is a correctness problem. Consider this race:
|
|
43
|
+
|
|
44
|
+
1. Request A reads old database value.
|
|
45
|
+
2. Request B updates DB and deletes cache.
|
|
46
|
+
3. Request A writes its old value into cache after B's deletion.
|
|
47
|
+
|
|
48
|
+
Possible mitigations include versioned values, write ordering, short TTL as a safety net, compare-and-set, single-flight loading, or designing cache as a derived/replaceable view. Choose based on business tolerance.
|
|
49
|
+
|
|
50
|
+
For multi-instance caches, ensure the invalidation event reaches all relevant cache layers. Define behavior on invalidation failure.
|
|
51
|
+
|
|
52
|
+
## 4. Stampedes and hot keys
|
|
53
|
+
|
|
54
|
+
- **Penetration**: repeated misses for keys that do not exist. Validate inputs, short negative-cache selected misses, or use a Bloom filter where appropriate.
|
|
55
|
+
- **Breakdown / stampede**: a popular key expires and many requests hit the source simultaneously. Use request coalescing/single-flight, refresh-ahead, stale-while-revalidate, jittered TTL, or bounded locks.
|
|
56
|
+
- **Avalanche**: many keys expire together or the cache service fails. Use TTL jitter, warm-up, fallback policy, circuit breaking, and source protection.
|
|
57
|
+
- **Hot key**: a small number of keys dominate load. Consider local near-cache, key replication, request coalescing, precomputation, or partitioning; each has consistency costs.
|
|
58
|
+
|
|
59
|
+
## 5. TTL and eviction
|
|
60
|
+
|
|
61
|
+
TTL is not a substitute for a correctness strategy. Use TTL jitter to desynchronize expiry. LRU/LFU/random/volatile policies have different behavior; choose based on the workload and cache product. Reserve memory headroom and monitor eviction, fragmentation, and replication overhead.
|
|
62
|
+
|
|
63
|
+
## 6. Cache key design
|
|
64
|
+
|
|
65
|
+
Keys must include every input that changes the result: tenant, user/authorization scope, locale, feature flags, schema version, query filters, and pagination where appropriate. Avoid caching sensitive cross-user data under shared keys. Normalize key serialization and define invalidation ownership.
|
|
66
|
+
|
|
67
|
+
## 7. Cache failure policy
|
|
68
|
+
|
|
69
|
+
Decide whether cache failure is fail-open (fall back to DB), fail-closed (reject), or selective degradation. Protect the database with concurrency limits, rate limiting, request shedding, and load tests. A cache outage must not accidentally create unbounded database traffic.
|
|
70
|
+
|
|
71
|
+
## 8. Checklist
|
|
72
|
+
|
|
73
|
+
- [ ] There is a measured reason to cache.
|
|
74
|
+
- [ ] Source of truth is clear.
|
|
75
|
+
- [ ] Cache key includes all relevant authorization and query dimensions.
|
|
76
|
+
- [ ] TTL, invalidation, and stale-read contract are defined.
|
|
77
|
+
- [ ] Stampede, hot-key, and cache-outage behavior are bounded.
|
|
78
|
+
- [ ] Memory, hit rate, evictions, errors, and latency are monitored.
|
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
# Asynchronous processing, queues, and event-driven design
|
|
2
|
+
|
|
3
|
+
## 1. Why async
|
|
4
|
+
|
|
5
|
+
Use asynchronous work to decouple latency, smooth bursts, isolate failures, or process work that does not need to finish before the user receives a response. Do not move a step async if the user requires a synchronous guarantee and there is no durable acknowledgement/compensation design.
|
|
6
|
+
|
|
7
|
+
Typical flow:
|
|
8
|
+
|
|
9
|
+
```text
|
|
10
|
+
API → validate + persist accepted intent → durable event/outbox
|
|
11
|
+
→ broker/queue → consumer group/workers → side effect
|
|
12
|
+
→ status/event/reconciliation
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
A `202 Accepted` response should mean the system has durably accepted the job according to a documented contract, not merely placed it in volatile process memory.
|
|
16
|
+
|
|
17
|
+
## 2. Queue vs event log vs task queue
|
|
18
|
+
|
|
19
|
+
- Work queue: consumers compete to perform a task; acknowledgement and redelivery matter.
|
|
20
|
+
- Event log/stream: durable ordered records per partition, replay and consumer offsets matter.
|
|
21
|
+
- Pub/sub: fan-out to independent subscribers; retention and delivery model vary by broker.
|
|
22
|
+
- Task scheduler: determines when work becomes eligible; worker queue executes it.
|
|
23
|
+
|
|
24
|
+
Do not treat Kafka, RabbitMQ, cloud queues, and Redis lists/streams as interchangeable. Compare ordering scope, retention, replay, delivery guarantee, routing, throughput, operations, and ecosystem.
|
|
25
|
+
|
|
26
|
+
## 3. Delivery semantics
|
|
27
|
+
|
|
28
|
+
- At-most-once: message may be lost but is not redelivered after acknowledgement.
|
|
29
|
+
- At-least-once: message can be redelivered; consumer must be idempotent.
|
|
30
|
+
- Exactly-once claims are scoped to a specific broker transaction or processing boundary. End-to-end external side effects still need idempotency or deduplication.
|
|
31
|
+
|
|
32
|
+
Design for duplicates unless the complete end-to-end guarantee is explicitly proven.
|
|
33
|
+
|
|
34
|
+
## 4. Idempotent consumer
|
|
35
|
+
|
|
36
|
+
Use a stable message/event ID and a durable deduplication record or business unique constraint. Atomically couple “event processed” state with the business mutation where possible. Define dedupe retention long enough for replay and delayed redelivery.
|
|
37
|
+
|
|
38
|
+
Do not dedupe solely by payload hash if two legitimate events can have identical payloads.
|
|
39
|
+
|
|
40
|
+
## 5. Transactional outbox and inbox
|
|
41
|
+
|
|
42
|
+
**Outbox** solves the DB-write + publish dual-write gap: write business state and an outbox record in one local DB transaction; a relay publishes outbox records; consumers dedupe by event ID. Monitor relay lag and poison records.
|
|
43
|
+
|
|
44
|
+
**Inbox** records incoming message IDs in the consumer's local transaction alongside its business mutation to prevent duplicate effects.
|
|
45
|
+
|
|
46
|
+
These patterns provide reliable eventual publication/processing, not a global distributed transaction. Handle ordering, retries, retention, and replay explicitly.
|
|
47
|
+
|
|
48
|
+
## 6. Ordering and partitioning
|
|
49
|
+
|
|
50
|
+
Ordering is often guaranteed only within a queue, session, or partition. Partition by the entity whose events must be ordered (e.g. account ID or order ID), but beware hot partitions. Distinguish event time from processing time and define how late/out-of-order events are handled.
|
|
51
|
+
|
|
52
|
+
## 7. Retries and dead letters
|
|
53
|
+
|
|
54
|
+
- Retry transient failures only; do not retry permanent validation failures.
|
|
55
|
+
- Use exponential backoff with jitter, maximum attempts/age, and a retry budget.
|
|
56
|
+
- Make retry handlers idempotent.
|
|
57
|
+
- Route poison messages to a DLQ with alerting, inspection, and a safe redrive procedure.
|
|
58
|
+
- Redrive should preserve original identity and be auditable; avoid retry storms.
|
|
59
|
+
|
|
60
|
+
## 8. Backpressure and queue stability
|
|
61
|
+
|
|
62
|
+
If arrival rate λ exceeds successful processing rate μ, backlog grows approximately at λ - μ messages per second. Monitor oldest-message age and queue depth, not depth alone. Bound worker concurrency, prefetch, payload size, retry volume, and queue retention. Apply admission control or load shedding upstream when the system cannot catch up.
|
|
63
|
+
|
|
64
|
+
## 9. Event schema evolution
|
|
65
|
+
|
|
66
|
+
Use versioned, compatibility-aware event contracts. Prefer additive optional fields; define defaults and unknown-field behavior. Consumers should tolerate fields they do not understand where the encoding allows it. Test old/new producer-consumer combinations. Avoid publishing internal DB row shapes as long-lived public events.
|
|
67
|
+
|
|
68
|
+
## 10. Checklist
|
|
69
|
+
|
|
70
|
+
- [ ] Queue chosen for needed semantics, not popularity.
|
|
71
|
+
- [ ] Durable acceptance boundary documented.
|
|
72
|
+
- [ ] Duplicate delivery and idempotency implemented.
|
|
73
|
+
- [ ] Ordering scope and partition key chosen.
|
|
74
|
+
- [ ] Retries bounded with jitter and DLQ.
|
|
75
|
+
- [ ] Backlog age, throughput, errors, and lag monitored.
|
|
76
|
+
- [ ] Outbox/inbox or other dual-write solution evaluated.
|
|
77
|
+
- [ ] Schema compatibility and replay procedure documented.
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# Distributed systems, consistency, and coordination
|
|
2
|
+
|
|
3
|
+
## 1. Failure is normal
|
|
4
|
+
|
|
5
|
+
Distributed components communicate over networks that can delay, drop, duplicate, reorder, or partition messages. A timeout does not prove the remote operation failed; it may have committed and only the response was lost. Design every remote call with bounded deadlines and an ambiguity strategy.
|
|
6
|
+
|
|
7
|
+
## 2. CAP theorem in context
|
|
8
|
+
|
|
9
|
+
CAP describes the trade-off during a network partition between consistency and availability for a distributed data system. Partition tolerance is not an optional feature in a networked system; during a partition, an operation may be rejected/blocked to preserve a stronger consistency guarantee, or accepted with potentially stale/conflicting state to preserve availability.
|
|
10
|
+
|
|
11
|
+
Avoid the misleading shorthand “a database is CP/AP.” Behavior varies by operation, topology, consistency mode, quorum, and failure condition. State the actual read/write guarantee and failure scenario.
|
|
12
|
+
|
|
13
|
+
## 3. Consistency vocabulary
|
|
14
|
+
|
|
15
|
+
- **Strong / linearizable**: operations appear to take effect atomically in a single real-time-respecting order.
|
|
16
|
+
- **Sequential consistency**: operations appear in one order consistent with each process's program order, but not necessarily real-time order.
|
|
17
|
+
- **Causal consistency**: causally related operations are observed in causal order.
|
|
18
|
+
- **Eventual consistency**: absent new writes and with successful propagation, replicas converge; it does not specify a fixed convergence deadline by itself.
|
|
19
|
+
- **Read-your-writes / monotonic reads**: useful session guarantees that may be weaker than global linearizability.
|
|
20
|
+
|
|
21
|
+
Match the required guarantee to business invariants. A follower count can tolerate eventual convergence; a payment/ledger invariant usually requires transactional or carefully coordinated semantics.
|
|
22
|
+
|
|
23
|
+
## 4. Replication and quorum
|
|
24
|
+
|
|
25
|
+
Replication improves availability and can increase read capacity, but adds lag, conflict, failover, and stale-read concerns. In quorum systems, read/write quorum sizes and replica count influence overlap and guarantees, but exact behavior depends on the database's consistency and conflict model. Never infer linearizability from `R + W > N` without checking the implementation assumptions (concurrent writes, sloppy quorums, hinted handoff, clock behavior, repair, and topology).
|
|
26
|
+
|
|
27
|
+
## 5. Leader election and split brain
|
|
28
|
+
|
|
29
|
+
A lease or election alone may not prevent a paused old leader from resuming and writing. Use fencing tokens/epochs or storage-enforced generation checks so an obsolete leader cannot mutate authoritative state. Define behavior when the coordinator is unavailable or clocks drift.
|
|
30
|
+
|
|
31
|
+
## 6. Time and ordering
|
|
32
|
+
|
|
33
|
+
Wall clocks on different machines are not a safe universal ordering mechanism. Use monotonic clocks for elapsed timeouts; use database sequence/transaction order, logical clocks, epochs, or per-entity versions when ordering matters. Timestamps are metadata, not proof of causality.
|
|
34
|
+
|
|
35
|
+
## 7. Distributed transactions and Sagas
|
|
36
|
+
|
|
37
|
+
Prefer a local transaction within one service boundary. For cross-service workflows consider:
|
|
38
|
+
|
|
39
|
+
- Saga orchestration or choreography;
|
|
40
|
+
- compensating actions that are business-valid (not always a literal rollback);
|
|
41
|
+
- outbox/inbox for reliable event transfer;
|
|
42
|
+
- explicit pending/confirmed/failed states;
|
|
43
|
+
- reconciliation for lost or ambiguous outcomes.
|
|
44
|
+
|
|
45
|
+
A compensation can itself fail and require retries or manual intervention. Document intermediate states and customer-visible behavior.
|
|
46
|
+
|
|
47
|
+
## 8. Concurrency controls
|
|
48
|
+
|
|
49
|
+
Choose among unique constraints, compare-and-swap/version columns, optimistic locking, pessimistic locking, leases, partitioned ownership, or serialized queues. State the invariant and show how the chosen mechanism enforces it under concurrent requests.
|
|
50
|
+
|
|
51
|
+
## 9. Checklist
|
|
52
|
+
|
|
53
|
+
- [ ] Remote timeout is treated as an ambiguous outcome where appropriate.
|
|
54
|
+
- [ ] Read/write consistency requirements are per use case.
|
|
55
|
+
- [ ] Replica lag and stale-read behavior are explicit.
|
|
56
|
+
- [ ] Leader fencing/split-brain protection is addressed.
|
|
57
|
+
- [ ] Clocks are not used as an unquestioned global order.
|
|
58
|
+
- [ ] Cross-service workflows have compensations and reconciliation.
|
|
59
|
+
- [ ] Concurrency invariants are enforced at the authoritative boundary.
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# Reliability, availability, recovery, and resilience
|
|
2
|
+
|
|
3
|
+
## 1. Availability math
|
|
4
|
+
|
|
5
|
+
For serial dependencies that must all work, approximate availability is the product of their availabilities only when the independence assumptions are reasonable. For independent redundant paths where either can serve, approximate availability is `1 - product(1 - A_i)`. Real systems have common-mode failures (shared region, network, identity provider, deployment, configuration, operator error), so independence is often overstated.
|
|
6
|
+
|
|
7
|
+
Do not promise “five nines” from node counts alone. Define the measurement window, eligible requests, exclusions, and dependency boundary.
|
|
8
|
+
|
|
9
|
+
## 2. SLI, SLO, SLA, error budget
|
|
10
|
+
|
|
11
|
+
- **SLI**: measured indicator (availability, latency, freshness, correctness).
|
|
12
|
+
- **SLO**: internal target over a time window.
|
|
13
|
+
- **SLA**: external contractual commitment with defined remedies/exclusions.
|
|
14
|
+
- **Error budget**: allowed unreliability derived from the SLO; use it to balance release velocity and reliability work.
|
|
15
|
+
|
|
16
|
+
Choose user-centered SLIs. A process-health check is not equivalent to successful completion of a user transaction.
|
|
17
|
+
|
|
18
|
+
## 3. Timeout and retry policy
|
|
19
|
+
|
|
20
|
+
- Set explicit connect, request, and total deadlines.
|
|
21
|
+
- Propagate deadline budgets across downstream calls; do not let each hop independently wait the full client timeout.
|
|
22
|
+
- Retry only transient conditions and only if the operation is idempotent or protected by an idempotency key.
|
|
23
|
+
- Use exponential backoff plus jitter and cap attempts/elapsed time.
|
|
24
|
+
- Use retry budgets to prevent overload amplification.
|
|
25
|
+
- Avoid retrying at every layer; one well-defined layer should own policy.
|
|
26
|
+
|
|
27
|
+
## 4. Circuit breaker and bulkhead
|
|
28
|
+
|
|
29
|
+
Circuit breakers temporarily stop calls to a persistently failing dependency and later allow controlled probes. They do not replace timeouts or fix the dependency. Bulkheads isolate resource pools, concurrency, queues, or tenants so one failure does not consume all capacity.
|
|
30
|
+
|
|
31
|
+
## 5. Rate limiting and load shedding
|
|
32
|
+
|
|
33
|
+
Rate limiting controls demand; load shedding rejects lower-priority work under overload. Algorithms include token bucket, leaky bucket, fixed/sliding window. Define identity dimension (user, tenant, API key, IP), burst allowance, fairness, distributed coordination, and what happens when the limiter itself fails.
|
|
34
|
+
|
|
35
|
+
Protect critical workloads from optional analytics, expensive searches, exports, and background work. Return clear retry guidance when safe.
|
|
36
|
+
|
|
37
|
+
## 6. Graceful degradation
|
|
38
|
+
|
|
39
|
+
Define reduced modes before the outage: stale-but-safe reads, disable recommendations, postpone analytics, serve cached public content, or reject non-critical operations. Never silently degrade security checks, payment correctness, or authorization.
|
|
40
|
+
|
|
41
|
+
## 7. Failover and redundancy
|
|
42
|
+
|
|
43
|
+
Active-passive and active-active topologies have different cost and conflict behavior. Failover needs detection, routing, state continuity, fencing, and a tested recovery plan. Replicas are not backups: replication can copy accidental deletion or corruption.
|
|
44
|
+
|
|
45
|
+
## 8. Backup and disaster recovery
|
|
46
|
+
|
|
47
|
+
- Define RPO (maximum acceptable data loss) and RTO (maximum acceptable recovery time).
|
|
48
|
+
- Use backups/snapshots plus log-based recovery when needed.
|
|
49
|
+
- Protect backups with separate credentials, encryption, immutability or isolation, and retention policy.
|
|
50
|
+
- Test restores and failover regularly; a successful backup job does not prove recoverability.
|
|
51
|
+
- Document regional dependencies, DNS/client caches, secrets, identity, and external provider recovery.
|
|
52
|
+
|
|
53
|
+
## 9. Failure-mode analysis
|
|
54
|
+
|
|
55
|
+
For each critical dependency ask:
|
|
56
|
+
|
|
57
|
+
- What fails if it is unavailable, slow, inconsistent, or returns malformed data?
|
|
58
|
+
- Does failure create queue growth, retries, thread exhaustion, connection exhaustion, or cascading load?
|
|
59
|
+
- What is the detection signal and alert threshold?
|
|
60
|
+
- What can be degraded safely?
|
|
61
|
+
- How is recovery verified and backlog drained without a second outage?
|
|
62
|
+
|
|
63
|
+
## 10. Checklist
|
|
64
|
+
|
|
65
|
+
- [ ] User-facing SLI/SLO defined.
|
|
66
|
+
- [ ] Deadlines and retry budgets are end-to-end.
|
|
67
|
+
- [ ] Circuit breakers/bulkheads used only where useful.
|
|
68
|
+
- [ ] Rate limits and overload behavior are specified.
|
|
69
|
+
- [ ] Critical and optional paths are isolated.
|
|
70
|
+
- [ ] Failover, backup, restore, RPO and RTO tested.
|
|
71
|
+
- [ ] Common-mode failures considered.
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
# API design, security, and observability
|
|
2
|
+
|
|
3
|
+
## 1. API design
|
|
4
|
+
|
|
5
|
+
Define resources/commands, request/response schema, error model, pagination, filtering, sorting, idempotency, versioning, and compatibility. Use stable identifiers and explicit validation. Avoid leaking database schemas into public contracts.
|
|
6
|
+
|
|
7
|
+
HTTP method semantics are conventions, not a substitute for business idempotency. A POST that creates a payment should accept an idempotency key and persist its result/operation status. A repeated key with a different payload should be rejected or handled by a documented rule.
|
|
8
|
+
|
|
9
|
+
For pagination, cursor pagination is often more stable for changing large datasets than large offset pagination. Cursor contents should be opaque or integrity-protected and bound to the query/tenant context.
|
|
10
|
+
|
|
11
|
+
## 2. Security threat boundaries
|
|
12
|
+
|
|
13
|
+
Perform lightweight threat modeling for sensitive paths:
|
|
14
|
+
|
|
15
|
+
- assets: user data, payment state, credentials, uploaded files, tenant data;
|
|
16
|
+
- trust boundaries: browser, API, internal service, queue, cloud control plane, vendor webhook;
|
|
17
|
+
- threat actors: unauthenticated user, malicious tenant, compromised dependency, insider, bot;
|
|
18
|
+
- abuse cases: privilege escalation, enumeration, injection, replay, SSRF, exfiltration, denial of service.
|
|
19
|
+
|
|
20
|
+
## 3. Authentication and authorization
|
|
21
|
+
|
|
22
|
+
- Authentication establishes identity; authorization decides permitted actions on a resource.
|
|
23
|
+
- Enforce authorization server-side on every operation, including object-level and tenant-level checks.
|
|
24
|
+
- Use least privilege for users, services, database roles, CI, and cloud identities.
|
|
25
|
+
- Store secrets in a secret manager or secure environment injection; rotate and audit access.
|
|
26
|
+
- Treat JWT claims and client-provided tenant IDs as untrusted until validated.
|
|
27
|
+
|
|
28
|
+
## 4. Common application risks
|
|
29
|
+
|
|
30
|
+
- SQL/NoSQL injection: parameterized queries and safe query builders.
|
|
31
|
+
- XSS: context-aware output encoding and safe templating; sanitize rich text when needed.
|
|
32
|
+
- CSRF: appropriate SameSite cookies, anti-CSRF token and origin checks for cookie-authenticated flows.
|
|
33
|
+
- SSRF: restrict outbound destinations, resolve/validate IPs, block metadata/internal ranges, and handle redirects/DNS rebinding.
|
|
34
|
+
- File upload: content/type/size limits, malware scanning where appropriate, safe filenames, isolated storage, and non-executable serving.
|
|
35
|
+
- Webhooks: signature verification, timestamp/replay control, idempotency, and secret rotation.
|
|
36
|
+
- Secrets/logging: redact tokens, passwords, session IDs, and sensitive PII; avoid production dumps in prompts or test fixtures.
|
|
37
|
+
- Supply chain: pin and review dependencies, scan vulnerabilities, protect build provenance and CI secrets.
|
|
38
|
+
|
|
39
|
+
## 5. Encryption and data governance
|
|
40
|
+
|
|
41
|
+
Use TLS for data in transit and appropriate encryption at rest. Protect encryption keys separately from data. Classify data, minimize collection, define retention/deletion, audit access, and understand backups, replicas, analytics pipelines, and logs as data copies.
|
|
42
|
+
|
|
43
|
+
## 6. Observability
|
|
44
|
+
|
|
45
|
+
Use three complementary signals:
|
|
46
|
+
|
|
47
|
+
- **Metrics**: request rate, error rate, duration, saturation, queue lag, cache hit, DB pool wait, job age.
|
|
48
|
+
- **Logs**: structured events with severity, trace/request IDs, tenant-safe context, and redaction.
|
|
49
|
+
- **Traces**: spans across service and dependency boundaries to expose serial latency and fan-out.
|
|
50
|
+
|
|
51
|
+
Instrument business outcomes too: checkout success, payment settlement lag, failed notifications, inventory reconciliation drift. Avoid unbounded-cardinality labels such as raw user IDs or URLs.
|
|
52
|
+
|
|
53
|
+
## 7. Operational telemetry checklist
|
|
54
|
+
|
|
55
|
+
- [ ] RED/USE metrics for critical services (Rate, Errors, Duration; Utilization, Saturation, Errors).
|
|
56
|
+
- [ ] P95/P99 by route and dependency.
|
|
57
|
+
- [ ] Trace propagation across async boundaries.
|
|
58
|
+
- [ ] Queue age/depth, retries, DLQ, and consumer lag.
|
|
59
|
+
- [ ] DB slow queries, lock waits, pool saturation, replication lag.
|
|
60
|
+
- [ ] Cache hit/miss, eviction, memory and timeout rates.
|
|
61
|
+
- [ ] Alerts tied to user impact and actionable runbooks.
|
|
62
|
+
- [ ] Sensitive fields are redacted and access is restricted.
|