system-design-primer-skill 0.0.0-stage → 1.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,88 @@
1
+ # Data modeling, databases, and storage scaling
2
+
3
+ ## 1. Choose by access patterns and invariants
4
+
5
+ Select storage after defining entities, queries, update patterns, consistency, transaction boundaries, retention, and operational skills. “SQL vs NoSQL” is not a speed contest.
6
+
7
+ | Data model | Typical fit | Important trade-off |
8
+ |---|---|---|
9
+ | Relational | transactions, joins, constraints, structured entities | schema design, write contention, scale topology |
10
+ | Key-value | direct lookup by key, session/cache state | limited ad-hoc querying |
11
+ | Document | aggregate-shaped, evolving records | embedding/reference trade-offs, index discipline |
12
+ | Wide-column | high write volume and known key-oriented queries | query-first data modeling, partition design |
13
+ | Graph | relationship traversal and graph algorithms | traversal cost and graph operations |
14
+ | Object storage | blobs, media, exports, backups | not a transactional row store; metadata/index elsewhere |
15
+
16
+ ## 2. Relational data design
17
+
18
+ - Model business invariants in constraints where feasible: primary key, unique key, foreign key, check constraints.
19
+ - Normalize to avoid update anomalies; denormalize deliberately for a measured read path, with an explicit refresh/consistency strategy.
20
+ - Design indexes for actual predicates, joins, sort order, and selectivity. Composite index column order matters.
21
+ - Avoid `SELECT *` in high-volume paths when it loads unused columns.
22
+ - Use `EXPLAIN` / query plans and production-like data distributions.
23
+ - Watch N+1 queries, unbounded result sets, offset pagination at high offsets, lock waits, connection pool saturation, and long transactions.
24
+
25
+ ## 3. ACID and isolation
26
+
27
+ - Atomicity: a transaction commits all intended changes or none.
28
+ - Consistency: constraints and application invariants remain true.
29
+ - Isolation: concurrent transactions obey a defined isolation behavior.
30
+ - Durability: committed data survives the failures covered by the database's guarantees.
31
+
32
+ Isolation levels differ by database and implementation. Consider dirty reads, non-repeatable reads, phantom reads, write skew, lost updates, and serialization failures. Do not assume a level's name fully describes exact behavior across products.
33
+
34
+ For inventory-like counters, a conditional update such as `UPDATE inventory SET available = available - 1 WHERE sku = ? AND available > 0` can be safer than read-then-write, provided affected-row checks and transaction boundaries are correct.
35
+
36
+ ## 4. Replication and read/write split
37
+
38
+ - Primary/replica separates write authority from read capacity or availability copies.
39
+ - Replication can be synchronous or asynchronous; asynchronous replication introduces lag and stale reads.
40
+ - Read-after-write requirements may need primary reads, session consistency, replica waiting, or sticky routing. Define the guarantee explicitly.
41
+ - Failover needs leader election/coordination, fencing against an old primary, routing update, and data-loss policy. “Two DBs” alone does not provide safe HA.
42
+ - Multi-primary writes require conflict resolution, ownership partitioning, or coordination. Last-write-wins can silently discard meaningful business updates.
43
+
44
+ ## 5. Partitioning and sharding
45
+
46
+ Partitioning divides a logical dataset into manageable subsets. Sharding distributes subsets across independent nodes.
47
+
48
+ Common shard keys: tenant ID, user ID, geography, time bucket, or a compound key. Evaluate:
49
+
50
+ - cardinality and distribution;
51
+ - hot tenants/keys and write skew;
52
+ - ability to target a query to one shard;
53
+ - cross-shard joins, transactions, and aggregations;
54
+ - resharding and data movement;
55
+ - ID generation and uniqueness;
56
+ - backup, restore, and rebalancing.
57
+
58
+ Hash sharding spreads keys but can make range queries harder. Range sharding supports range locality but can create hot ranges. Consistent hashing can reduce movement in some topologies but does not remove hot-key or rebalancing design.
59
+
60
+ Do not shard before observing that simpler tuning, replicas, partitioning, archiving, or workload separation is insufficient.
61
+
62
+ ## 6. Schema evolution and migrations
63
+
64
+ Prefer expand-migrate-contract:
65
+
66
+ 1. Add backward-compatible schema/field/column.
67
+ 2. Deploy code that can read old and new formats.
68
+ 3. Backfill in bounded, resumable batches.
69
+ 4. Validate counts, checksums, and business invariants.
70
+ 5. Switch reads/writes gradually.
71
+ 6. Stop old writes and observe.
72
+ 7. Remove obsolete schema only after rollback window closes.
73
+
74
+ Make migrations idempotent or safely restartable. Avoid long table locks and unbounded backfills. Test against production-like data volume. Keep rollback realistic: a destructive schema change may not be reversible without a restore or forward-fix.
75
+
76
+ ## 7. Data ownership across services
77
+
78
+ A service should own its authoritative write model. Other services consume APIs/events or materialized read models instead of directly updating its tables. Cross-service workflows may need Saga/orchestration, outbox, compensation, and reconciliation rather than a distributed ACID transaction.
79
+
80
+ ## 8. Checklist
81
+
82
+ - [ ] Data model follows actual queries and business invariants.
83
+ - [ ] Transactions and isolation anomalies are understood.
84
+ - [ ] Indexes have evidence and write-cost review.
85
+ - [ ] Read/write replica lag is addressed where relevant.
86
+ - [ ] Shard key avoids skew and supports common queries.
87
+ - [ ] Schema migration is online, observable, and recoverable.
88
+ - [ ] Backups are restored in a tested exercise.
@@ -0,0 +1,78 @@
1
+ # Caching patterns and cache correctness
2
+
3
+ ## 1. Cache only with a reason
4
+
5
+ A cache is justified when repeated reads, expensive computation, latency goals, or load reduction outweigh staleness, memory, invalidation, and operational costs. Measure hit rate, miss penalty, hot-key concentration, eviction, memory usage, and stale-read impact.
6
+
7
+ Potential layers:
8
+
9
+ - browser/client cache;
10
+ - CDN/edge cache;
11
+ - reverse-proxy cache;
12
+ - application-local cache;
13
+ - distributed cache (e.g. Redis);
14
+ - database buffer/query cache.
15
+
16
+ Each layer has a different consistency and invalidation boundary.
17
+
18
+ ## 2. Common patterns
19
+
20
+ ### Cache-aside
21
+
22
+ Read: cache lookup → on miss read source of truth → populate cache → return. Write: update source of truth → invalidate cache (or use a carefully designed refresh). Simple and common, but concurrent misses can stampede and stale refill races need handling.
23
+
24
+ ### Read-through
25
+
26
+ Application asks the cache; cache abstraction loads from backing store on miss. Centralizes loading but couples cache to data access behavior.
27
+
28
+ ### Write-through
29
+
30
+ Write path updates cache and backing store synchronously through a coordinated abstraction. Better immediate read freshness in some designs; write latency and failure semantics must be explicit.
31
+
32
+ ### Write-behind / write-back
33
+
34
+ Accept write in cache and persist asynchronously. Fast but increases data-loss and recovery risk. Use only with durable buffering, explicit loss tolerance, replay, and monitoring.
35
+
36
+ ### Refresh-ahead
37
+
38
+ Refresh hot entries before expiry. Useful for predictable hot data, but can waste work and requires jitter and failure handling.
39
+
40
+ ## 3. Invalidation and race conditions
41
+
42
+ Cache invalidation is a correctness problem. Consider this race:
43
+
44
+ 1. Request A reads old database value.
45
+ 2. Request B updates DB and deletes cache.
46
+ 3. Request A writes its old value into cache after B's deletion.
47
+
48
+ Possible mitigations include versioned values, write ordering, short TTL as a safety net, compare-and-set, single-flight loading, or designing cache as a derived/replaceable view. Choose based on business tolerance.
49
+
50
+ For multi-instance caches, ensure the invalidation event reaches all relevant cache layers. Define behavior on invalidation failure.
51
+
52
+ ## 4. Stampedes and hot keys
53
+
54
+ - **Penetration**: repeated misses for keys that do not exist. Validate inputs, short negative-cache selected misses, or use a Bloom filter where appropriate.
55
+ - **Breakdown / stampede**: a popular key expires and many requests hit the source simultaneously. Use request coalescing/single-flight, refresh-ahead, stale-while-revalidate, jittered TTL, or bounded locks.
56
+ - **Avalanche**: many keys expire together or the cache service fails. Use TTL jitter, warm-up, fallback policy, circuit breaking, and source protection.
57
+ - **Hot key**: a small number of keys dominate load. Consider local near-cache, key replication, request coalescing, precomputation, or partitioning; each has consistency costs.
58
+
59
+ ## 5. TTL and eviction
60
+
61
+ TTL is not a substitute for a correctness strategy. Use TTL jitter to desynchronize expiry. LRU/LFU/random/volatile policies have different behavior; choose based on the workload and cache product. Reserve memory headroom and monitor eviction, fragmentation, and replication overhead.
62
+
63
+ ## 6. Cache key design
64
+
65
+ Keys must include every input that changes the result: tenant, user/authorization scope, locale, feature flags, schema version, query filters, and pagination where appropriate. Avoid caching sensitive cross-user data under shared keys. Normalize key serialization and define invalidation ownership.
66
+
67
+ ## 7. Cache failure policy
68
+
69
+ Decide whether cache failure is fail-open (fall back to DB), fail-closed (reject), or selective degradation. Protect the database with concurrency limits, rate limiting, request shedding, and load tests. A cache outage must not accidentally create unbounded database traffic.
70
+
71
+ ## 8. Checklist
72
+
73
+ - [ ] There is a measured reason to cache.
74
+ - [ ] Source of truth is clear.
75
+ - [ ] Cache key includes all relevant authorization and query dimensions.
76
+ - [ ] TTL, invalidation, and stale-read contract are defined.
77
+ - [ ] Stampede, hot-key, and cache-outage behavior are bounded.
78
+ - [ ] Memory, hit rate, evictions, errors, and latency are monitored.
@@ -0,0 +1,77 @@
1
+ # Asynchronous processing, queues, and event-driven design
2
+
3
+ ## 1. Why async
4
+
5
+ Use asynchronous work to decouple latency, smooth bursts, isolate failures, or process work that does not need to finish before the user receives a response. Do not move a step async if the user requires a synchronous guarantee and there is no durable acknowledgement/compensation design.
6
+
7
+ Typical flow:
8
+
9
+ ```text
10
+ API → validate + persist accepted intent → durable event/outbox
11
+ → broker/queue → consumer group/workers → side effect
12
+ → status/event/reconciliation
13
+ ```
14
+
15
+ A `202 Accepted` response should mean the system has durably accepted the job according to a documented contract, not merely placed it in volatile process memory.
16
+
17
+ ## 2. Queue vs event log vs task queue
18
+
19
+ - Work queue: consumers compete to perform a task; acknowledgement and redelivery matter.
20
+ - Event log/stream: durable ordered records per partition, replay and consumer offsets matter.
21
+ - Pub/sub: fan-out to independent subscribers; retention and delivery model vary by broker.
22
+ - Task scheduler: determines when work becomes eligible; worker queue executes it.
23
+
24
+ Do not treat Kafka, RabbitMQ, cloud queues, and Redis lists/streams as interchangeable. Compare ordering scope, retention, replay, delivery guarantee, routing, throughput, operations, and ecosystem.
25
+
26
+ ## 3. Delivery semantics
27
+
28
+ - At-most-once: message may be lost but is not redelivered after acknowledgement.
29
+ - At-least-once: message can be redelivered; consumer must be idempotent.
30
+ - Exactly-once claims are scoped to a specific broker transaction or processing boundary. End-to-end external side effects still need idempotency or deduplication.
31
+
32
+ Design for duplicates unless the complete end-to-end guarantee is explicitly proven.
33
+
34
+ ## 4. Idempotent consumer
35
+
36
+ Use a stable message/event ID and a durable deduplication record or business unique constraint. Atomically couple “event processed” state with the business mutation where possible. Define dedupe retention long enough for replay and delayed redelivery.
37
+
38
+ Do not dedupe solely by payload hash if two legitimate events can have identical payloads.
39
+
40
+ ## 5. Transactional outbox and inbox
41
+
42
+ **Outbox** solves the DB-write + publish dual-write gap: write business state and an outbox record in one local DB transaction; a relay publishes outbox records; consumers dedupe by event ID. Monitor relay lag and poison records.
43
+
44
+ **Inbox** records incoming message IDs in the consumer's local transaction alongside its business mutation to prevent duplicate effects.
45
+
46
+ These patterns provide reliable eventual publication/processing, not a global distributed transaction. Handle ordering, retries, retention, and replay explicitly.
47
+
48
+ ## 6. Ordering and partitioning
49
+
50
+ Ordering is often guaranteed only within a queue, session, or partition. Partition by the entity whose events must be ordered (e.g. account ID or order ID), but beware hot partitions. Distinguish event time from processing time and define how late/out-of-order events are handled.
51
+
52
+ ## 7. Retries and dead letters
53
+
54
+ - Retry transient failures only; do not retry permanent validation failures.
55
+ - Use exponential backoff with jitter, maximum attempts/age, and a retry budget.
56
+ - Make retry handlers idempotent.
57
+ - Route poison messages to a DLQ with alerting, inspection, and a safe redrive procedure.
58
+ - Redrive should preserve original identity and be auditable; avoid retry storms.
59
+
60
+ ## 8. Backpressure and queue stability
61
+
62
+ If arrival rate λ exceeds successful processing rate μ, backlog grows approximately at λ - μ messages per second. Monitor oldest-message age and queue depth, not depth alone. Bound worker concurrency, prefetch, payload size, retry volume, and queue retention. Apply admission control or load shedding upstream when the system cannot catch up.
63
+
64
+ ## 9. Event schema evolution
65
+
66
+ Use versioned, compatibility-aware event contracts. Prefer additive optional fields; define defaults and unknown-field behavior. Consumers should tolerate fields they do not understand where the encoding allows it. Test old/new producer-consumer combinations. Avoid publishing internal DB row shapes as long-lived public events.
67
+
68
+ ## 10. Checklist
69
+
70
+ - [ ] Queue chosen for needed semantics, not popularity.
71
+ - [ ] Durable acceptance boundary documented.
72
+ - [ ] Duplicate delivery and idempotency implemented.
73
+ - [ ] Ordering scope and partition key chosen.
74
+ - [ ] Retries bounded with jitter and DLQ.
75
+ - [ ] Backlog age, throughput, errors, and lag monitored.
76
+ - [ ] Outbox/inbox or other dual-write solution evaluated.
77
+ - [ ] Schema compatibility and replay procedure documented.
@@ -0,0 +1,59 @@
1
+ # Distributed systems, consistency, and coordination
2
+
3
+ ## 1. Failure is normal
4
+
5
+ Distributed components communicate over networks that can delay, drop, duplicate, reorder, or partition messages. A timeout does not prove the remote operation failed; it may have committed and only the response was lost. Design every remote call with bounded deadlines and an ambiguity strategy.
6
+
7
+ ## 2. CAP theorem in context
8
+
9
+ CAP describes the trade-off during a network partition between consistency and availability for a distributed data system. Partition tolerance is not an optional feature in a networked system; during a partition, an operation may be rejected/blocked to preserve a stronger consistency guarantee, or accepted with potentially stale/conflicting state to preserve availability.
10
+
11
+ Avoid the misleading shorthand “a database is CP/AP.” Behavior varies by operation, topology, consistency mode, quorum, and failure condition. State the actual read/write guarantee and failure scenario.
12
+
13
+ ## 3. Consistency vocabulary
14
+
15
+ - **Strong / linearizable**: operations appear to take effect atomically in a single real-time-respecting order.
16
+ - **Sequential consistency**: operations appear in one order consistent with each process's program order, but not necessarily real-time order.
17
+ - **Causal consistency**: causally related operations are observed in causal order.
18
+ - **Eventual consistency**: absent new writes and with successful propagation, replicas converge; it does not specify a fixed convergence deadline by itself.
19
+ - **Read-your-writes / monotonic reads**: useful session guarantees that may be weaker than global linearizability.
20
+
21
+ Match the required guarantee to business invariants. A follower count can tolerate eventual convergence; a payment/ledger invariant usually requires transactional or carefully coordinated semantics.
22
+
23
+ ## 4. Replication and quorum
24
+
25
+ Replication improves availability and can increase read capacity, but adds lag, conflict, failover, and stale-read concerns. In quorum systems, read/write quorum sizes and replica count influence overlap and guarantees, but exact behavior depends on the database's consistency and conflict model. Never infer linearizability from `R + W > N` without checking the implementation assumptions (concurrent writes, sloppy quorums, hinted handoff, clock behavior, repair, and topology).
26
+
27
+ ## 5. Leader election and split brain
28
+
29
+ A lease or election alone may not prevent a paused old leader from resuming and writing. Use fencing tokens/epochs or storage-enforced generation checks so an obsolete leader cannot mutate authoritative state. Define behavior when the coordinator is unavailable or clocks drift.
30
+
31
+ ## 6. Time and ordering
32
+
33
+ Wall clocks on different machines are not a safe universal ordering mechanism. Use monotonic clocks for elapsed timeouts; use database sequence/transaction order, logical clocks, epochs, or per-entity versions when ordering matters. Timestamps are metadata, not proof of causality.
34
+
35
+ ## 7. Distributed transactions and Sagas
36
+
37
+ Prefer a local transaction within one service boundary. For cross-service workflows consider:
38
+
39
+ - Saga orchestration or choreography;
40
+ - compensating actions that are business-valid (not always a literal rollback);
41
+ - outbox/inbox for reliable event transfer;
42
+ - explicit pending/confirmed/failed states;
43
+ - reconciliation for lost or ambiguous outcomes.
44
+
45
+ A compensation can itself fail and require retries or manual intervention. Document intermediate states and customer-visible behavior.
46
+
47
+ ## 8. Concurrency controls
48
+
49
+ Choose among unique constraints, compare-and-swap/version columns, optimistic locking, pessimistic locking, leases, partitioned ownership, or serialized queues. State the invariant and show how the chosen mechanism enforces it under concurrent requests.
50
+
51
+ ## 9. Checklist
52
+
53
+ - [ ] Remote timeout is treated as an ambiguous outcome where appropriate.
54
+ - [ ] Read/write consistency requirements are per use case.
55
+ - [ ] Replica lag and stale-read behavior are explicit.
56
+ - [ ] Leader fencing/split-brain protection is addressed.
57
+ - [ ] Clocks are not used as an unquestioned global order.
58
+ - [ ] Cross-service workflows have compensations and reconciliation.
59
+ - [ ] Concurrency invariants are enforced at the authoritative boundary.
@@ -0,0 +1,71 @@
1
+ # Reliability, availability, recovery, and resilience
2
+
3
+ ## 1. Availability math
4
+
5
+ For serial dependencies that must all work, approximate availability is the product of their availabilities only when the independence assumptions are reasonable. For independent redundant paths where either can serve, approximate availability is `1 - product(1 - A_i)`. Real systems have common-mode failures (shared region, network, identity provider, deployment, configuration, operator error), so independence is often overstated.
6
+
7
+ Do not promise “five nines” from node counts alone. Define the measurement window, eligible requests, exclusions, and dependency boundary.
8
+
9
+ ## 2. SLI, SLO, SLA, error budget
10
+
11
+ - **SLI**: measured indicator (availability, latency, freshness, correctness).
12
+ - **SLO**: internal target over a time window.
13
+ - **SLA**: external contractual commitment with defined remedies/exclusions.
14
+ - **Error budget**: allowed unreliability derived from the SLO; use it to balance release velocity and reliability work.
15
+
16
+ Choose user-centered SLIs. A process-health check is not equivalent to successful completion of a user transaction.
17
+
18
+ ## 3. Timeout and retry policy
19
+
20
+ - Set explicit connect, request, and total deadlines.
21
+ - Propagate deadline budgets across downstream calls; do not let each hop independently wait the full client timeout.
22
+ - Retry only transient conditions and only if the operation is idempotent or protected by an idempotency key.
23
+ - Use exponential backoff plus jitter and cap attempts/elapsed time.
24
+ - Use retry budgets to prevent overload amplification.
25
+ - Avoid retrying at every layer; one well-defined layer should own policy.
26
+
27
+ ## 4. Circuit breaker and bulkhead
28
+
29
+ Circuit breakers temporarily stop calls to a persistently failing dependency and later allow controlled probes. They do not replace timeouts or fix the dependency. Bulkheads isolate resource pools, concurrency, queues, or tenants so one failure does not consume all capacity.
30
+
31
+ ## 5. Rate limiting and load shedding
32
+
33
+ Rate limiting controls demand; load shedding rejects lower-priority work under overload. Algorithms include token bucket, leaky bucket, fixed/sliding window. Define identity dimension (user, tenant, API key, IP), burst allowance, fairness, distributed coordination, and what happens when the limiter itself fails.
34
+
35
+ Protect critical workloads from optional analytics, expensive searches, exports, and background work. Return clear retry guidance when safe.
36
+
37
+ ## 6. Graceful degradation
38
+
39
+ Define reduced modes before the outage: stale-but-safe reads, disable recommendations, postpone analytics, serve cached public content, or reject non-critical operations. Never silently degrade security checks, payment correctness, or authorization.
40
+
41
+ ## 7. Failover and redundancy
42
+
43
+ Active-passive and active-active topologies have different cost and conflict behavior. Failover needs detection, routing, state continuity, fencing, and a tested recovery plan. Replicas are not backups: replication can copy accidental deletion or corruption.
44
+
45
+ ## 8. Backup and disaster recovery
46
+
47
+ - Define RPO (maximum acceptable data loss) and RTO (maximum acceptable recovery time).
48
+ - Use backups/snapshots plus log-based recovery when needed.
49
+ - Protect backups with separate credentials, encryption, immutability or isolation, and retention policy.
50
+ - Test restores and failover regularly; a successful backup job does not prove recoverability.
51
+ - Document regional dependencies, DNS/client caches, secrets, identity, and external provider recovery.
52
+
53
+ ## 9. Failure-mode analysis
54
+
55
+ For each critical dependency ask:
56
+
57
+ - What fails if it is unavailable, slow, inconsistent, or returns malformed data?
58
+ - Does failure create queue growth, retries, thread exhaustion, connection exhaustion, or cascading load?
59
+ - What is the detection signal and alert threshold?
60
+ - What can be degraded safely?
61
+ - How is recovery verified and backlog drained without a second outage?
62
+
63
+ ## 10. Checklist
64
+
65
+ - [ ] User-facing SLI/SLO defined.
66
+ - [ ] Deadlines and retry budgets are end-to-end.
67
+ - [ ] Circuit breakers/bulkheads used only where useful.
68
+ - [ ] Rate limits and overload behavior are specified.
69
+ - [ ] Critical and optional paths are isolated.
70
+ - [ ] Failover, backup, restore, RPO and RTO tested.
71
+ - [ ] Common-mode failures considered.
@@ -0,0 +1,62 @@
1
+ # API design, security, and observability
2
+
3
+ ## 1. API design
4
+
5
+ Define resources/commands, request/response schema, error model, pagination, filtering, sorting, idempotency, versioning, and compatibility. Use stable identifiers and explicit validation. Avoid leaking database schemas into public contracts.
6
+
7
+ HTTP method semantics are conventions, not a substitute for business idempotency. A POST that creates a payment should accept an idempotency key and persist its result/operation status. A repeated key with a different payload should be rejected or handled by a documented rule.
8
+
9
+ For pagination, cursor pagination is often more stable for changing large datasets than large offset pagination. Cursor contents should be opaque or integrity-protected and bound to the query/tenant context.
10
+
11
+ ## 2. Security threat boundaries
12
+
13
+ Perform lightweight threat modeling for sensitive paths:
14
+
15
+ - assets: user data, payment state, credentials, uploaded files, tenant data;
16
+ - trust boundaries: browser, API, internal service, queue, cloud control plane, vendor webhook;
17
+ - threat actors: unauthenticated user, malicious tenant, compromised dependency, insider, bot;
18
+ - abuse cases: privilege escalation, enumeration, injection, replay, SSRF, exfiltration, denial of service.
19
+
20
+ ## 3. Authentication and authorization
21
+
22
+ - Authentication establishes identity; authorization decides permitted actions on a resource.
23
+ - Enforce authorization server-side on every operation, including object-level and tenant-level checks.
24
+ - Use least privilege for users, services, database roles, CI, and cloud identities.
25
+ - Store secrets in a secret manager or secure environment injection; rotate and audit access.
26
+ - Treat JWT claims and client-provided tenant IDs as untrusted until validated.
27
+
28
+ ## 4. Common application risks
29
+
30
+ - SQL/NoSQL injection: parameterized queries and safe query builders.
31
+ - XSS: context-aware output encoding and safe templating; sanitize rich text when needed.
32
+ - CSRF: appropriate SameSite cookies, anti-CSRF token and origin checks for cookie-authenticated flows.
33
+ - SSRF: restrict outbound destinations, resolve/validate IPs, block metadata/internal ranges, and handle redirects/DNS rebinding.
34
+ - File upload: content/type/size limits, malware scanning where appropriate, safe filenames, isolated storage, and non-executable serving.
35
+ - Webhooks: signature verification, timestamp/replay control, idempotency, and secret rotation.
36
+ - Secrets/logging: redact tokens, passwords, session IDs, and sensitive PII; avoid production dumps in prompts or test fixtures.
37
+ - Supply chain: pin and review dependencies, scan vulnerabilities, protect build provenance and CI secrets.
38
+
39
+ ## 5. Encryption and data governance
40
+
41
+ Use TLS for data in transit and appropriate encryption at rest. Protect encryption keys separately from data. Classify data, minimize collection, define retention/deletion, audit access, and understand backups, replicas, analytics pipelines, and logs as data copies.
42
+
43
+ ## 6. Observability
44
+
45
+ Use three complementary signals:
46
+
47
+ - **Metrics**: request rate, error rate, duration, saturation, queue lag, cache hit, DB pool wait, job age.
48
+ - **Logs**: structured events with severity, trace/request IDs, tenant-safe context, and redaction.
49
+ - **Traces**: spans across service and dependency boundaries to expose serial latency and fan-out.
50
+
51
+ Instrument business outcomes too: checkout success, payment settlement lag, failed notifications, inventory reconciliation drift. Avoid unbounded-cardinality labels such as raw user IDs or URLs.
52
+
53
+ ## 7. Operational telemetry checklist
54
+
55
+ - [ ] RED/USE metrics for critical services (Rate, Errors, Duration; Utilization, Saturation, Errors).
56
+ - [ ] P95/P99 by route and dependency.
57
+ - [ ] Trace propagation across async boundaries.
58
+ - [ ] Queue age/depth, retries, DLQ, and consumer lag.
59
+ - [ ] DB slow queries, lock waits, pool saturation, replication lag.
60
+ - [ ] Cache hit/miss, eviction, memory and timeout rates.
61
+ - [ ] Alerts tied to user impact and actionable runbooks.
62
+ - [ ] Sensitive fields are redacted and access is restricted.
@@ -0,0 +1,58 @@
1
+ # Modern engineering practices beyond the Primer
2
+
3
+ This reference extends the Primer's foundational patterns into day-to-day production engineering. Verify vendor-specific and version-specific details against official documentation.
4
+
5
+ ## 1. Architecture boundaries
6
+
7
+ - Prefer a modular monolith until independent deployment, ownership, scaling, security, or fault-isolation needs justify distributed services.
8
+ - Define module APIs and data ownership; avoid cross-module writes into private tables.
9
+ - Use dependency inversion at meaningful boundaries, not abstractions around every function.
10
+ - Keep domain logic separate from transport, persistence, and framework adapters where it improves testability.
11
+
12
+ ## 2. Microservice extraction criteria
13
+
14
+ A service extraction is justified when one or more are demonstrably important:
15
+
16
+ - independent scaling profile;
17
+ - different reliability/security boundary;
18
+ - different deployment cadence or team ownership;
19
+ - domain boundary is stable and data ownership can be isolated;
20
+ - blast-radius reduction is worth operational overhead.
21
+
22
+ Before extraction, define API contracts, data migration, event ownership, consistency, observability, deployment, on-call ownership, and rollback. Avoid distributed monoliths with synchronous call chains and shared database writes.
23
+
24
+ ## 3. Cloud and container deployment
25
+
26
+ Containers package process/runtime dependencies; they do not automatically provide resilience. Orchestration adds scheduling and rollout capabilities but also control-plane and configuration complexity. For each service define:
27
+
28
+ - CPU/memory requests and limits or equivalent resource reservations;
29
+ - readiness/liveness/startup checks;
30
+ - graceful shutdown and connection draining;
31
+ - replica/zone placement and disruption budget where relevant;
32
+ - autoscaling signals (often queue lag or concurrency in addition to CPU);
33
+ - configuration and secret separation;
34
+ - rolling/blue-green/canary strategy and rollback criteria.
35
+
36
+ Serverless can reduce operational work for bursty/event-driven workloads, but review cold starts, execution limits, concurrency quotas, vendor coupling, local testing, and cost under sustained load.
37
+
38
+ ## 4. Progressive delivery
39
+
40
+ Use feature flags, canary, blue/green, or phased tenant rollout when risk warrants. Define success and abort thresholds before rollout. Feature flags need owners, expiry/review dates, and safe defaults; stale flags become complexity.
41
+
42
+ ## 5. Database and event contract evolution
43
+
44
+ Treat API schemas, event schemas, database schema, and cached representations as versioned contracts. Plan forward/backward compatibility and the window in which old and new versions coexist. Use expand-migrate-contract for schema evolution and compatibility tests in CI.
45
+
46
+ ## 6. AI application architecture
47
+
48
+ For LLM/agent features, system design also includes model/provider routing, token and GPU capacity, prompt/tool versioning, prompt-injection boundaries, least-privilege tool authorization, output validation, retrieval freshness and tenant isolation, evaluation, and safe cancellation/recovery. Treat model output as untrusted data. A prompt or generated plan must not bypass application authorization, validation, or transaction controls.
49
+
50
+ Use `references/12-ai-systems-architecture.md` for detailed workload modeling, inference/GPU planning, model gateways, RAG pipelines, agent tool boundaries, evaluation, cost and failure handling.
51
+
52
+ ## 7. Operational ownership
53
+
54
+ Every production component needs an owner, runbook, alert routing, capacity plan, backup/recovery plan if stateful, patch/upgrade policy, and decommission path. A diagram without ownership and operations is incomplete architecture.
55
+
56
+ ## 8. Architecture decision records (ADR)
57
+
58
+ Record decisions that are expensive to reverse. An ADR should contain context, decision, alternatives, consequences, risks, and status. Do not write an ADR for every small implementation detail.
@@ -0,0 +1,97 @@
1
+ # Workflows, design review checklists, and decision process
2
+
3
+ ## 1. Design review sequence
4
+
5
+ 1. Problem statement and users.
6
+ 2. Functional requirements and critical journeys.
7
+ 3. NFRs with measurable targets.
8
+ 4. Workload/capacity model.
9
+ 5. Data model and ownership.
10
+ 6. High-level architecture and trust boundaries.
11
+ 7. Critical request and event flows.
12
+ 8. Consistency/concurrency invariants.
13
+ 9. Failure modes and recovery.
14
+ 10. Security/privacy/compliance.
15
+ 11. Observability and operations.
16
+ 12. Cost, alternatives, migration and rollback.
17
+
18
+ ## 2. Architecture review checklist
19
+
20
+ ### Product and requirements
21
+ - Is the problem scoped and tied to measurable outcomes?
22
+ - Are assumptions, unknowns, and exclusions explicit?
23
+ - Is the critical path identified?
24
+
25
+ ### Architecture
26
+ - Are module/service responsibilities and ownership clear?
27
+ - Are synchronous and asynchronous boundaries intentional?
28
+ - Is the design simpler than necessary, or unnecessarily complex?
29
+ - Are single points of failure and common-mode failures understood?
30
+
31
+ ### Data
32
+ - Is source of truth explicit?
33
+ - Are transaction boundaries and invariants enforced?
34
+ - Are indexes and queries aligned to access patterns?
35
+ - Are replication lag, duplicate events, and migration paths addressed?
36
+
37
+ ### Performance and capacity
38
+ - Are peak and burst loads modeled?
39
+ - Are tail latencies and downstream fan-out measured?
40
+ - Are connection pools, queues, caches and hot partitions bounded?
41
+
42
+ ### Reliability
43
+ - Are timeouts/deadlines, retries, idempotency and overload behavior specified?
44
+ - Can the system degrade safely?
45
+ - Are backup restore and failover tested?
46
+
47
+ ### Security
48
+ - Are authentication, authorization, tenant isolation and secrets handled correctly?
49
+ - Are external inputs, webhooks, files and URLs validated?
50
+ - Are logs and analytics free from unnecessary sensitive data?
51
+
52
+ ### Operations
53
+ - Are metrics, logs, traces, alerts and runbooks present?
54
+ - Are rollout, rollback, ownership, cost, and decommissioning understood?
55
+
56
+ ## 3. Finding severity
57
+
58
+ - **Critical**: likely data breach, data corruption/loss, unsafe financial/state transition, or broad outage with immediate exposure.
59
+ - **High**: likely major degradation/outage or important invariant violation under plausible conditions.
60
+ - **Medium**: localized reliability, performance, security-hardening, or maintainability issue with a workaround.
61
+ - **Low**: improvement opportunity or low-impact maintainability concern.
62
+
63
+ Severity must be evidence-based. Include likelihood and impact separately when useful. Do not inflate severity for style preferences.
64
+
65
+ ## 4. Decision matrix
66
+
67
+ Compare candidate options using criteria relevant to the workload: correctness, latency, throughput, availability, consistency, complexity, operational burden, cost, migration risk, ecosystem, team familiarity, and vendor lock-in. Weight criteria only when the user supplies priorities or explain the weights as assumptions.
68
+
69
+ ## 5. Architecture debt and evolution triggers
70
+
71
+ Write down observable triggers for a future change, e.g.:
72
+
73
+ - DB CPU or I/O above target for sustained periods despite query/index tuning;
74
+ - replica lag violates read freshness SLO;
75
+ - queue oldest-message age exceeds business deadline;
76
+ - p99 latency budget consumed by a single dependency;
77
+ - a module requires independent deploy/scale/security ownership;
78
+ - a tenant causes resource starvation;
79
+ - recovery time exceeds RTO.
80
+
81
+ Avoid arbitrary user counts as the sole migration trigger. Workload shape and bottleneck evidence matter more.
82
+
83
+ ## 6. Implementation quality gate
84
+
85
+ Before declaring architecture work done:
86
+
87
+ - tests cover business invariants, concurrency, idempotency, retries and failure paths;
88
+ - migrations are tested and reversible or have an explicit forward-recovery plan;
89
+ - load tests use representative data and realistic concurrency;
90
+ - telemetry can confirm success and detect regressions;
91
+ - threat model and authorization checks are reviewed;
92
+ - deployment and rollback are documented;
93
+ - operational owner and runbook exist.
94
+
95
+ ## 7. System design interview structure
96
+
97
+ Use: clarify requirements → state assumptions → estimate scale → API/data model → high-level diagram → deep dive into bottlenecks → consistency/reliability/security → alternatives/trade-offs. Spend time according to the interviewer and problem scope; do not attempt to name every technology.