system-design-primer-skill 0.0.0-stage → 2.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ATTRIBUTION.md +10 -0
- package/LICENSE +7 -0
- package/README.md +180 -2
- package/SKILL.md +447 -0
- package/VERSION +1 -0
- package/package.json +78 -4
- package/references/00-primer-map-and-attribution.md +29 -0
- package/references/01-requirements-and-capacity.md +107 -0
- package/references/02-architecture-and-networking.md +94 -0
- package/references/03-data-and-storage.md +88 -0
- package/references/04-caching.md +78 -0
- package/references/05-async-and-messaging.md +77 -0
- package/references/06-distributed-systems.md +59 -0
- package/references/07-reliability-and-resilience.md +71 -0
- package/references/08-api-security-observability.md +62 -0
- package/references/09-modern-engineering.md +58 -0
- package/references/10-workflows-and-checklists.md +97 -0
- package/references/11-case-studies.md +74 -0
- package/references/12-ai-systems-architecture.md +139 -0
- package/references/13-architecture-intelligence.md +91 -0
- package/references/14-api-and-data-contracts.md +80 -0
- package/references/15-failure-analysis.md +84 -0
- package/references/16-multi-agent-architecture.md +76 -0
- package/references/17-agent-harness-and-runtime.md +87 -0
- package/references/18-mcp-architecture.md +67 -0
- package/references/19-agent-security-and-permissions.md +74 -0
- package/references/20-agent-state-and-memory.md +72 -0
- package/references/21-agent-evaluation-and-observability.md +90 -0
- package/references/22-kubernetes-and-container-platforms.md +29 -0
- package/references/23-service-mesh-and-traffic-management.md +21 -0
- package/references/24-multi-region-and-disaster-recovery.md +24 -0
- package/references/25-platform-engineering-and-developer-experience.md +26 -0
- package/references/26-finops-and-cloud-cost.md +29 -0
- package/references/27-production-observability-and-slo.md +29 -0
- package/references/28-event-streaming-and-kafka-pulsar.md +27 -0
- package/references/29-event-sourcing-and-cqrs.md +23 -0
- package/references/30-change-data-capture-and-debezium.md +27 -0
- package/references/31-data-lakehouse-and-stream-batch.md +24 -0
- package/references/32-workflow-orchestration-and-temporal.md +24 -0
- package/references/33-schema-registry-and-event-governance.md +22 -0
- package/references/34-postgresql-internals-and-performance.md +43 -0
- package/references/35-mysql-innodb-and-performance.md +27 -0
- package/references/36-sharding-and-distributed-sql.md +22 -0
- package/references/37-consensus-and-distributed-transactions.md +25 -0
- package/references/38-search-architecture-elasticsearch-opensearch.md +24 -0
- package/references/39-vector-databases-and-hybrid-retrieval.md +31 -0
- package/references/40-zero-downtime-database-migrations.md +27 -0
- package/references/41-api-gateway-and-edge-architecture.md +23 -0
- package/references/42-rest-api-design-and-lifecycle.md +23 -0
- package/references/43-graphql-architecture-and-governance.md +16 -0
- package/references/44-grpc-and-service-contracts.md +16 -0
- package/references/45-websocket-sse-and-realtime-systems.md +16 -0
- package/references/46-realtime-collaboration-ot-crdt.md +16 -0
- package/references/47-api-security-rate-limiting-and-abuse.md +16 -0
- package/references/48-oauth-oidc-and-passkeys.md +20 -0
- package/references/49-iam-rbac-abac-rebac.md +16 -0
- package/references/50-zero-trust-and-workload-identity.md +16 -0
- package/references/51-secrets-kms-pki-and-key-management.md +16 -0
- package/references/52-supply-chain-security-sbom-slsa.md +16 -0
- package/references/53-threat-modeling-stride-and-attack-trees.md +16 -0
- package/references/54-privacy-engineering-and-data-governance.md +16 -0
- package/references/55-opentelemetry-and-instrumentation.md +50 -0
- package/references/56-distributed-tracing-and-context-propagation.md +43 -0
- package/references/57-continuous-profiling-and-performance-diagnostics.md +45 -0
- package/references/58-load-stress-soak-testing-and-workload-modeling.md +44 -0
- package/references/59-capacity-planning-and-performance-modeling.md +34 -0
- package/references/60-chaos-engineering-and-resilience-experiments.md +41 -0
- package/references/61-incident-management-and-postmortems.md +34 -0
- package/references/62-dns-and-service-discovery.md +35 -0
- package/references/63-load-balancing-and-global-traffic-management.md +31 -0
- package/references/64-cdn-and-edge-caching.md +34 -0
- package/references/65-cloud-networking-and-connectivity.md +34 -0
- package/references/66-edge-computing-architecture.md +33 -0
- package/references/67-storage-infrastructure-and-storage-systems.md +30 -0
- package/references/68-tcp-quic-and-network-performance.md +27 -0
- package/references/69-task-routing-and-evidence.md +58 -0
- package/scripts/install.ps1 +124 -0
- package/scripts/install.sh +105 -0
- package/scripts/validate.py +376 -0
- package/templates/adr-template.md +32 -0
- package/templates/agent-evaluation-plan-template.md +57 -0
- package/templates/agent-security-review-template.md +46 -0
- package/templates/agent-system-design-template.md +74 -0
- package/templates/api-architecture-design-template.md +33 -0
- package/templates/api-security-and-rate-limit-review-template.md +29 -0
- package/templates/architecture-decision-matrix-template.md +48 -0
- package/templates/architecture-design-template.md +95 -0
- package/templates/architecture-review-scorecard-template.md +36 -0
- package/templates/architecture-review-template.md +52 -0
- package/templates/cdc-data-pipeline-template.md +41 -0
- package/templates/cdn-and-edge-strategy-template.md +36 -0
- package/templates/chaos-experiment-template.md +39 -0
- package/templates/cloud-cost-model-template.md +37 -0
- package/templates/database-architecture-review-template.md +33 -0
- package/templates/edge-computing-readiness-template.md +36 -0
- package/templates/event-sourcing-cqrs-template.md +41 -0
- package/templates/event-streaming-design-template.md +47 -0
- package/templates/failure-analysis-template.md +36 -0
- package/templates/identity-and-access-architecture-template.md +29 -0
- package/templates/incident-postmortem-template.md +39 -0
- package/templates/multi-region-disaster-recovery-template.md +43 -0
- package/templates/network-architecture-design-template.md +38 -0
- package/templates/observability-architecture-template.md +52 -0
- package/templates/performance-test-plan-template.md +52 -0
- package/templates/privacy-data-governance-template.md +24 -0
- package/templates/production-readiness-checklist-template.md +52 -0
- package/templates/realtime-system-design-template.md +28 -0
- package/templates/security-threat-model-template.md +26 -0
- package/templates/sharding-distributed-sql-design-template.md +34 -0
- package/templates/storage-architecture-design-template.md +35 -0
- package/templates/supply-chain-security-review-template.md +20 -0
- package/templates/zero-downtime-database-migration-template.md +32 -0
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# Production Observability and SLO Engineering
|
|
2
|
+
|
|
3
|
+
Observability should answer whether users can complete critical journeys, why they cannot, and what action is safe. Instrumentation volume is not a substitute for useful operational signals.
|
|
4
|
+
|
|
5
|
+
## Service-level objectives
|
|
6
|
+
Define SLIs from user-visible behavior: availability, latency, correctness, freshness, durability, or successful completion. Specify event population, good-event definition, measurement window, exclusions, and aggregation. Set SLO targets based on user and business needs, not arbitrary industry defaults.
|
|
7
|
+
|
|
8
|
+
Use error budgets to govern release risk. Define policy for budget consumption, freeze thresholds, exception approval, and recovery. Pair service SLOs with dependency and platform indicators while preserving ownership. Do not treat every infrastructure metric as an SLI.
|
|
9
|
+
|
|
10
|
+
## Metrics, logs, traces, and profiles
|
|
11
|
+
- Metrics: bounded-cardinality time series for rates, errors, duration, saturation, queue depth, and resource use.
|
|
12
|
+
- Logs: structured events with correlation identifiers, severity, and controlled fields. Redact secrets and personal data; define retention and access.
|
|
13
|
+
- Distributed traces: propagate trace context across asynchronous boundaries and record meaningful spans, status, and dependency calls. Sampling must preserve error and tail-latency evidence.
|
|
14
|
+
- Profiles: use continuous or on-demand profiling to identify CPU, allocation, lock, and runtime hotspots when metrics and traces are insufficient.
|
|
15
|
+
|
|
16
|
+
Use consistent service, environment, region, version, and ownership attributes. Avoid unbounded labels such as user IDs, request IDs, raw URLs, or arbitrary exception strings. Keep high-cardinality identifiers in logs/traces where access and cost controls are appropriate.
|
|
17
|
+
|
|
18
|
+
## Alerting and incident response
|
|
19
|
+
Alert on user impact, SLO burn rate, actionable saturation, and failure conditions with a clear owner and runbook. Use multi-window, multi-burn-rate alerts where appropriate. Avoid paging on every transient symptom or on metrics without an action. Distinguish page, ticket, dashboard, and automated remediation signals.
|
|
20
|
+
|
|
21
|
+
Define severity, incident commander, communication channels, escalation, evidence preservation, rollback authority, and post-incident review. Runbooks should state symptoms, safe checks, mitigation, rollback, escalation, and verification. Automated remediation must be bounded, idempotent, observable, and reversible.
|
|
22
|
+
|
|
23
|
+
## Telemetry reliability and cost
|
|
24
|
+
Model SDKs, agents, collectors, gateways, queues, storage, query engines, and alert delivery as production dependencies. Bound buffers, retries, cardinality, sampling, and backpressure. Define behavior when telemetry backends are unavailable; application availability must not depend on synchronous log or trace delivery.
|
|
25
|
+
|
|
26
|
+
Set retention by operational and regulatory purpose. Monitor dropped telemetry, ingestion lag, query latency, alert delivery, and collector resource use. Estimate ingestion and retention cost before enabling high-volume instrumentation.
|
|
27
|
+
|
|
28
|
+
## Release and evaluation
|
|
29
|
+
Correlate deployments and configuration changes with SLOs and incidents. Validate dashboards and alerts as code. Use synthetic checks for critical user journeys and real-user monitoring where useful. Test alert routing and on-call schedules. Verify telemetry supports diagnosis without exposing sensitive data.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# Event Streaming Architecture: Kafka and Pulsar
|
|
2
|
+
|
|
3
|
+
Choose an event-streaming platform based on durability, ordering, replay, throughput, multi-tenancy, operations, and ecosystem requirements—not brand popularity. Compare Kafka, Pulsar, managed cloud streams, and simpler queues.
|
|
4
|
+
|
|
5
|
+
## Domain and topic design
|
|
6
|
+
Define event ownership, topic purpose, key, partitioning strategy, retention, compaction, schema, compatibility, and access policy. Use stable business keys when per-entity ordering matters. Partition count determines parallelism and affects ordering, storage, and reassignment cost. Hot keys can bottleneck a partition even when cluster capacity is available.
|
|
7
|
+
|
|
8
|
+
Treat events as immutable facts about something that happened. Separate event time, ingestion time, and processing time. Include event ID, type, schema version, aggregate/entity key, producer identity, correlation/causation IDs, timestamp semantics, and trace context. Do not put secrets or unnecessary personal data in event payloads.
|
|
9
|
+
|
|
10
|
+
## Delivery and ordering semantics
|
|
11
|
+
Document producer acknowledgements, replication factor, minimum in-sync replicas, leader election behavior, idempotent producer settings, transaction scope, consumer offset commits, and failure behavior. Understand the limits of ordering: ordering is generally scoped to a partition or key, not a whole topic or multiple topics.
|
|
12
|
+
|
|
13
|
+
Consumers should be idempotent and tolerate duplicates, retries, rebalances, and replay. Commit offsets only after the corresponding side effect is durable, or use an atomic pattern that binds offset progress to output effects. A broker transaction does not automatically make an external database or API side effect exactly once.
|
|
14
|
+
|
|
15
|
+
## Consumer operations
|
|
16
|
+
Bound poll batches, processing concurrency, in-flight records, and retry duration. Preserve per-key ordering when parallelizing. Distinguish retryable transient errors from poison events; use bounded retries, delayed retry topics or queues, and a dead-letter workflow with alerting, ownership, replay controls, and audit. Avoid hot retry loops.
|
|
17
|
+
|
|
18
|
+
Monitor consumer lag and lag age, throughput, rebalance frequency, fetch latency, partition skew, under-replicated partitions, ISR shrink, disk capacity, request errors, produce latency, and broker/controller health. Lag alone is insufficient: measure the age of the oldest unprocessed business event.
|
|
19
|
+
|
|
20
|
+
## Platform and topology
|
|
21
|
+
Design broker placement across zones, replication, rack awareness, storage, network throughput, quotas, TLS, authentication, ACLs, tenant isolation, schema registry, and control-plane availability. For Kafka, account for partition leadership, controller quorum, log retention, compaction, and replication. For Pulsar, account for brokers, BookKeeper storage/quorums, metadata store, namespace policies, and tiered storage. Verify implementation-specific semantics against the selected release's official docs.
|
|
22
|
+
|
|
23
|
+
## Capacity and recovery
|
|
24
|
+
Estimate bytes/sec, peak multiplier, retention bytes, replication factor, compaction overhead, consumer fan-out, network egress, and recovery/replay throughput. Plan disk and network headroom for broker failure, partition reassignment, rolling upgrades, and consumer catch-up. Test broker loss, zone loss, disk pressure, network partition, credential expiry, schema incompatibility, and poison messages. Back up configuration and critical metadata; retained logs are not a substitute for application-level backup.
|
|
25
|
+
|
|
26
|
+
## Decision gate
|
|
27
|
+
Use streaming when multiple consumers, durable replay, high sustained throughput, or event-time processing justify it. Use a task queue for work dispatch and a database outbox for reliable publication where appropriate. Document operational ownership, SLOs, upgrade policy, cost, and exit path.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# Event Sourcing and CQRS
|
|
2
|
+
|
|
3
|
+
Event Sourcing stores state changes as an append-only event history; CQRS separates command-side writes from query-side read models. They are independent patterns and should be adopted only when auditability, temporal reconstruction, multiple read projections, or domain complexity justify their costs.
|
|
4
|
+
|
|
5
|
+
## Event-sourced aggregate design
|
|
6
|
+
Define aggregate boundaries around invariants and transactional consistency. Commands express intent; the aggregate validates business rules and emits domain events. Events describe facts in past tense and should be immutable. Persist the event append and expected stream version atomically using optimistic concurrency or an equivalent concurrency mechanism.
|
|
7
|
+
|
|
8
|
+
Specify event identity, aggregate ID, sequence/version, event type, schema version, occurred-at semantics, actor, correlation/causation, and payload. Avoid storing mutable derived state as if it were a fact. Model correction as compensating or superseding events rather than rewriting history, subject to legal deletion requirements.
|
|
9
|
+
|
|
10
|
+
## Concurrency and idempotency
|
|
11
|
+
Use expected-version checks to prevent lost updates and conflicting commands. Define command idempotency keys and deduplicate retries. Handle duplicate delivery, concurrent append, and command timeout where the caller cannot know whether a write committed. Make command outcomes queryable by idempotency key when necessary.
|
|
12
|
+
|
|
13
|
+
## CQRS projections
|
|
14
|
+
Build read models from the event log or change stream. Define projection ownership, ordering, checkpoint, rebuild, replay, lag, and versioning. Reads may be eventually consistent; state acceptable staleness and provide read-your-writes options when required. Rebuild into a new projection generation, validate counts/invariants, then switch traffic atomically. Do not let a projection become an accidental source of truth.
|
|
15
|
+
|
|
16
|
+
## Schema evolution and event upcasting
|
|
17
|
+
Events live longer than application deployments. Use additive evolution where possible, explicit versioning, tolerant readers, upcasters, and compatibility tests against historical samples. Avoid changing the meaning of an existing field or event type. Keep a documented policy for retiring event versions and replaying old data.
|
|
18
|
+
|
|
19
|
+
## Snapshots and retention
|
|
20
|
+
Snapshots accelerate aggregate rehydration but are disposable derived state; prove that replay from the event stream remains possible. Define snapshot version, consistency point, validation, and rebuild path. Event retention, archival, legal hold, deletion, and privacy requirements can conflict with immutability. Define a lawful data model before adopting event sourcing, including PII minimization, tokenization, encryption-key strategies, and deletion or redaction semantics.
|
|
21
|
+
|
|
22
|
+
## Operational and organizational costs
|
|
23
|
+
Account for event-store availability, write contention, projection lag, replay storms, storage growth, schema governance, observability, debugging, and developer learning curve. Event sourcing is not a replacement for backups, and CQRS does not require microservices or separate databases. Prefer conventional transactional storage plus an outbox when the event history itself is not a product requirement.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# Change Data Capture and Debezium Architecture
|
|
2
|
+
|
|
3
|
+
Change Data Capture (CDC) streams committed database changes to downstream systems. It is useful for search indexes, analytics, cache invalidation, data integration, and migration, but it introduces log-retention, schema, ordering, and operational dependencies.
|
|
4
|
+
|
|
5
|
+
## Source and connector model
|
|
6
|
+
Identify the source database, transaction log mechanism, connector, offset/checkpoint store, schema history, broker, and downstream consumers. For Debezium, verify connector-specific support and configuration for the exact database/version. Understand whether capture uses WAL, binlog, redo log, or another engine-specific mechanism.
|
|
7
|
+
|
|
8
|
+
CDC is not a substitute for domain events. Database change records expose persistence representation and may omit business intent. Use domain events for business facts and CDC for reliable propagation of committed storage changes where that coupling is acceptable.
|
|
9
|
+
|
|
10
|
+
## Snapshot and streaming lifecycle
|
|
11
|
+
Define initial snapshot mode, snapshot consistency, table selection, incremental snapshot behavior, log position handoff, and recovery from connector restart. Ensure log retention is long enough to cover the maximum expected outage and catch-up time. If the source log is purged before the connector resumes, a new snapshot or controlled recovery may be required.
|
|
12
|
+
|
|
13
|
+
Model initial snapshot load separately from steady-state change throughput. Avoid overwhelming downstream systems with snapshot bursts. Throttle, stage, and monitor snapshot progress.
|
|
14
|
+
|
|
15
|
+
## Ordering, transactions, and duplicates
|
|
16
|
+
Capture transaction metadata where available. Define ordering guarantees by source partition, table, key, and transaction; do not assume a global order across all tables or connectors. Consumers must handle duplicate delivery, restarts, and replay idempotently. Preserve delete/tombstone semantics and distinguish a deleted row from a null field.
|
|
17
|
+
|
|
18
|
+
For multi-table invariants, understand transaction boundaries and whether the connector emits transaction metadata. Downstream consumers should not expose partially applied business transactions as if they were atomic.
|
|
19
|
+
|
|
20
|
+
## Schema and data governance
|
|
21
|
+
Plan schema evolution, type mapping, decimal/timestamp precision, primary-key changes, column filtering, masking, and compatibility. Prevent sensitive columns from entering topics by default. Define retention, access control, encryption, residency, lineage, and deletion propagation. CDC topics can leak PII and internal database structure.
|
|
22
|
+
|
|
23
|
+
## Operations and failure modes
|
|
24
|
+
Monitor connector task state, source log lag, snapshot progress, queue/broker lag, offset persistence, schema history, heartbeat, error rate, and source database impact. Bound connector retries and handle poison records explicitly. Test source failover, log truncation, connector rebalance, schema incompatibility, network isolation, and destination outage.
|
|
25
|
+
|
|
26
|
+
## Migration and dual-running
|
|
27
|
+
For database migration, define snapshot boundary, CDC catch-up, consistency validation, write ownership, cutover, rollback, and old-source retirement. Avoid uncoordinated dual writes. Use checksums, row counts, invariant queries, and sampled business-level comparisons. CDC replication alone does not prove semantic equivalence.
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# Data Lakehouse and Stream/Batch Data Architecture
|
|
2
|
+
|
|
3
|
+
A lakehouse combines low-cost object storage with table formats, metadata/catalog services, and analytical query engines. Choose the architecture based on latency, concurrency, governance, workload isolation, and operational maturity—not the label.
|
|
4
|
+
|
|
5
|
+
## Data layers and contracts
|
|
6
|
+
Define source systems, ingestion, raw/bronze, cleaned/silver, curated/gold, serving, and semantic layers. Assign ownership, schema, quality expectations, freshness SLO, lineage, classification, retention, and access controls to each dataset. Distinguish raw immutable capture from curated business meaning.
|
|
7
|
+
|
|
8
|
+
Use open table formats where interoperability, schema evolution, partition evolution, snapshots, and transactional writes matter. Compare the capabilities and ecosystem of Apache Iceberg, Delta Lake, and Apache Hudi for the target engine and deployment. Validate catalog, locking, concurrency, compaction, and metadata behavior for chosen versions.
|
|
9
|
+
|
|
10
|
+
## Batch and streaming
|
|
11
|
+
Use batch for bounded periodic processing and streaming for low-latency continuous updates. A unified API does not erase differences in event-time semantics, late arrivals, replay, state, watermarking, exactly-once scope, and cost. Define freshness requirements before choosing the processing mode.
|
|
12
|
+
|
|
13
|
+
For stream processing, model event time vs processing time, watermarks, late data, state backend, checkpoint interval, savepoints, backpressure, and replay. For batch, define partition sizing, incremental processing, idempotent writes, retries, and backfill strategy. Prevent concurrent writers from corrupting table metadata or producing small-file storms.
|
|
14
|
+
|
|
15
|
+
## Data quality and governance
|
|
16
|
+
Define schema contracts, nullability, uniqueness, referential checks, freshness, volume, distribution, and business-rule tests. Quarantine bad records with reason, owner, and replay path. Maintain lineage from source to report/model. Apply column- and row-level authorization, masking, encryption, audit, and purpose-based access. Catalog metadata is not itself an access-control boundary.
|
|
17
|
+
|
|
18
|
+
## Performance and cost
|
|
19
|
+
Optimize partitioning for query predicates and data distribution; avoid over-partitioning and high-cardinality directory layouts. Plan file sizes, compaction, clustering, statistics, metadata cleanup, cache, and query concurrency. Monitor scan bytes, small files, shuffle, spill, skew, queue wait, freshness, and compute utilization.
|
|
20
|
+
|
|
21
|
+
Include object storage requests, data transfer, catalog, compute, orchestration, observability, retention, compaction, and egress in TCO. Separate interactive BI, ETL, ad hoc exploration, and ML workloads where noisy-neighbor risks justify workload isolation.
|
|
22
|
+
|
|
23
|
+
## Recovery and portability
|
|
24
|
+
Define reproducible pipeline code, versioned datasets, checkpoints, table snapshots, catalog recovery, object versioning, and restore testing. Do not assume object storage durability alone recovers catalog metadata or application semantics. Test region failure, accidental overwrite, schema breakage, late data, and replay.
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# Workflow Orchestration and Durable Execution
|
|
2
|
+
|
|
3
|
+
Workflow orchestration coordinates long-running business processes across services, human tasks, timers, and retries. Choose between a task queue, scheduler, state machine, and durable workflow engine based on process duration, recovery, visibility, and compensation needs.
|
|
4
|
+
|
|
5
|
+
## Workflow model
|
|
6
|
+
Represent the process as explicit states and transitions. Define start conditions, inputs, outputs, deadlines, cancellation, pause/resume, human approval, terminal states, and invariants. Separate workflow definition from activity implementation. Give each workflow a stable ID and each activity a deterministic or unique execution identity.
|
|
7
|
+
|
|
8
|
+
Use a durable engine such as Temporal when workflows must survive process restarts, wait for external events or long timers, coordinate retries, and expose execution history. Compare with cloud-native state machines, queue-based workers, and application-managed orchestration. Verify SDK, server, and deployment-specific semantics for the selected version.
|
|
9
|
+
|
|
10
|
+
## Durable execution and determinism
|
|
11
|
+
Workflow code may be replayed to reconstruct state. Keep nondeterministic I/O, wall-clock access, random values, and external side effects inside activities or supported workflow APIs. Version workflow definitions safely; old executions may continue while new code is deployed. Define worker compatibility and retirement policy.
|
|
12
|
+
|
|
13
|
+
Activities should be idempotent or protected by idempotency keys because timeouts can leave completion uncertain. Use heartbeats for long-running work, bounded retries with backoff, deadlines, cancellation propagation, and compensation where business semantics require it. A timeout does not prove that an external side effect did not happen.
|
|
14
|
+
|
|
15
|
+
## State, signals, and versioning
|
|
16
|
+
Define workflow state retention, payload limits, search attributes, signal/update semantics, event history growth, continue-as-new, and archival. Keep large business data in durable domain storage and reference it from workflow state. Protect history and payloads as sensitive data. Define encryption, access, retention, and deletion behavior.
|
|
17
|
+
|
|
18
|
+
## Scaling and operations
|
|
19
|
+
Scale workers by task queue backlog and activity capacity, not only CPU. Bound concurrency per dependency and tenant. Separate task queues by workload or trust boundary where useful. Monitor schedule-to-start, execution latency, task failures, retries, timeouts, stuck workflows, poller capacity, history size, and dependency saturation.
|
|
20
|
+
|
|
21
|
+
Design engine cluster availability, persistence, namespace/tenant isolation, visibility store, backups, upgrades, and disaster recovery. Test worker loss, engine failover, dependency outage, poison activities, duplicate signals, cancellation races, and version skew.
|
|
22
|
+
|
|
23
|
+
## Decision gate
|
|
24
|
+
Do not use a durable workflow engine for a simple cron task or short request-response flow. Account for operational ownership, learning curve, visibility benefits, replay constraints, and migration/exit cost. Prefer explicit state machines over hidden orchestration spread across callbacks and queues.
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
# Schema Registry and Event Governance
|
|
2
|
+
|
|
3
|
+
Event governance makes independently deployed producers and consumers interoperable over time. A schema registry helps manage schema identity and compatibility; it does not replace ownership, data contracts, authorization, or consumer testing.
|
|
4
|
+
|
|
5
|
+
## Contract design
|
|
6
|
+
Define event name, semantic meaning, owner, key, payload schema, required/optional fields, units, timestamp semantics, PII classification, retention, and consumer expectations. Use stable names and explicit schema versions. Distinguish additive fields from semantic changes. Do not repurpose a field or change units without a migration plan.
|
|
7
|
+
|
|
8
|
+
## Compatibility policy
|
|
9
|
+
Choose backward, forward, or full compatibility based on producer/consumer deployment order and replay requirements. Test compatibility against all supported historical versions and representative consumers. Compatibility checks are necessary but insufficient: a schema can remain structurally compatible while business meaning changes.
|
|
10
|
+
|
|
11
|
+
Plan expand-migrate-contract: publish additive schema, update consumers to tolerate both forms, migrate producers, measure adoption, then retire old fields only after the compatibility window and consumer inventory are satisfied. Version event semantics deliberately rather than blindly incrementing every payload.
|
|
12
|
+
|
|
13
|
+
## Registry architecture and operations
|
|
14
|
+
Define registry availability, replication, authentication, authorization, tenancy, subject naming, schema ID behavior, cache behavior, soft/hard deletion, and backup/restore. Producers and consumers need a documented response to registry unavailability; cached schema IDs may help but cannot be assumed indefinitely.
|
|
15
|
+
|
|
16
|
+
Use least-privilege subject permissions and protect registry APIs. Monitor compatibility failures, unknown schema IDs, serialization errors, consumer parse failures, and orphaned schemas. Define ownership and lifecycle for unused subjects and old versions.
|
|
17
|
+
|
|
18
|
+
## Data governance and evolution
|
|
19
|
+
Document data classification, PII minimization, encryption, retention, deletion propagation, lineage, and regulatory obligations. Avoid secrets and unneeded personal data in event payloads. Schema metadata should link to source owner, producer, consumers, and business glossary. Use contract tests and sampled production payload validation to catch semantic drift.
|
|
20
|
+
|
|
21
|
+
## Rollout checklist
|
|
22
|
+
Inventory producers and consumers; define compatibility policy; register and review schema; deploy tolerant consumers; roll producers gradually; monitor failures and lag; verify replay from retained data; deprecate old fields with notice; remove only after all consumer and archival constraints are satisfied.
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# PostgreSQL Internals and Performance
|
|
2
|
+
|
|
3
|
+
## Purpose and decision context
|
|
4
|
+
Use this guide when designing or diagnosing PostgreSQL-backed systems. Confirm the deployed major version, extensions, hosting model, workload, and operational constraints before applying version-specific advice.
|
|
5
|
+
|
|
6
|
+
## Storage and execution model
|
|
7
|
+
- Explain heap pages, tuples, TOAST, visibility metadata, WAL, checkpoints, background writer, and vacuum as a connected lifecycle.
|
|
8
|
+
- Distinguish heap from index storage. Indexes accelerate selected access paths but increase write amplification, storage, cache pressure, and maintenance.
|
|
9
|
+
- Use EXPLAIN (ANALYZE, BUFFERS) on representative queries. Inspect estimated versus actual rows, loops, join strategy, sort/hash spills, buffer hits/reads, and planning/execution time.
|
|
10
|
+
- Treat ANALYZE statistics as workload-sensitive estimates. Check stale statistics, skew, correlation, and extended statistics before forcing planner settings.
|
|
11
|
+
- Evaluate B-tree, GIN, GiST, BRIN, partial, and expression indexes against predicates, ordering, selectivity, write rate, and maintenance cost.
|
|
12
|
+
|
|
13
|
+
## MVCC, transactions, and vacuum
|
|
14
|
+
PostgreSQL MVCC retains row versions so readers and writers can often proceed concurrently. Updates and deletes create dead tuples; vacuum makes space reusable and maintains visibility information, but does not normally shrink relation files for the operating system.
|
|
15
|
+
- Explain READ COMMITTED, REPEATABLE READ, and SERIALIZABLE semantics, including serialization failures and retry requirements.
|
|
16
|
+
- Monitor transaction age, long-running snapshots, idle-in-transaction sessions, dead tuples, autovacuum progress, bloat indicators, and replication slots retaining WAL.
|
|
17
|
+
- Set autovacuum thresholds and scale factors based on table size and churn; do not disable autovacuum globally.
|
|
18
|
+
- Diagnose lock waits and deadlocks with lock graphs and transaction traces. Keep transactions short and acquire locks consistently.
|
|
19
|
+
- Use row locks or advisory locks only when lock scope and contention are explicit.
|
|
20
|
+
|
|
21
|
+
## Connection and workload management
|
|
22
|
+
- Size pools from database capacity, query concurrency, CPU, memory, and latency—not application replica count alone.
|
|
23
|
+
- Consider PgBouncer transaction pooling only after checking session state, prepared statements, temporary tables, advisory locks, and other session-affine behavior.
|
|
24
|
+
- Separate OLTP, analytical, maintenance, and migration workloads where contention warrants it.
|
|
25
|
+
- Use statement and lock timeouts, bounded queues, cancellation, and workload priorities.
|
|
26
|
+
|
|
27
|
+
## Replication and operations
|
|
28
|
+
Distinguish physical streaming replication from logical replication. Document synchronous commit policy, replica lag, promotion/fencing, slot lifecycle, failover authority, and read-after-write behavior. A replica is not a backup.
|
|
29
|
+
Define tested backup/PITR procedures, WAL retention, restore time, major-version upgrade path, extension compatibility, and rollback constraints.
|
|
30
|
+
|
|
31
|
+
## Diagnostic checklist
|
|
32
|
+
1. Capture query plans with realistic parameters and data volume.
|
|
33
|
+
2. Check CPU, memory, storage latency/IOPS, cache behavior, WAL rate, checkpoints, and connection saturation.
|
|
34
|
+
3. Inspect lock waits, deadlocks, long transactions, vacuum lag, and replication lag.
|
|
35
|
+
4. Change one variable at a time; compare p95/p99 latency and write amplification.
|
|
36
|
+
5. Record version, schema, indexes, workload, and rollback plan.
|
|
37
|
+
|
|
38
|
+
## Anti-patterns
|
|
39
|
+
- Adding indexes without measuring write and storage costs.
|
|
40
|
+
- Treating VACUUM FULL as routine maintenance; it requires stronger locking and extra space.
|
|
41
|
+
- Increasing max_connections without resource and pool analysis.
|
|
42
|
+
- Using planner settings to mask incorrect statistics or query shape.
|
|
43
|
+
- Assuming replica reads are current or failover is lossless without a documented durability policy.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# MySQL and InnoDB Architecture and Performance
|
|
2
|
+
|
|
3
|
+
## Storage and indexing
|
|
4
|
+
InnoDB stores row data in the clustered primary-key B-tree. Secondary-index records also carry the primary-key value, so a wide primary key increases secondary-index footprint.
|
|
5
|
+
- Choose stable, compact primary keys and verify insertion locality and hotspot behavior.
|
|
6
|
+
- Design composite indexes around leftmost-prefix, filtering, and ordering requirements; measure covering-index benefits against write amplification.
|
|
7
|
+
- Use EXPLAIN and, where supported, EXPLAIN ANALYZE. Compare estimated/actual rows, access paths, join order, temporary tables, and filesort.
|
|
8
|
+
- Evaluate partitioning only when pruning, lifecycle management, or operational needs justify its constraints; it is not a substitute for indexing or sharding.
|
|
9
|
+
|
|
10
|
+
## Transactions, locks, and MVCC
|
|
11
|
+
InnoDB uses undo records for consistent reads and rollback, and redo logging for crash recovery. Under REPEATABLE READ, consistent reads and locking reads differ; next-key and gap locks can prevent phantom inserts but increase contention.
|
|
12
|
+
- Define isolation and read/write semantics for each critical transaction.
|
|
13
|
+
- Diagnose deadlocks from engine reports; treat them as expected concurrency outcomes and retry bounded, idempotent transactions.
|
|
14
|
+
- Keep transactions short, access rows in consistent order, and index predicates used by locking statements.
|
|
15
|
+
- Monitor history-list length, purge lag, undo growth, lock waits, and long-running transactions.
|
|
16
|
+
- Do not assume a plain SELECT protects a business invariant; use unique constraints, conditional writes, or explicit locking.
|
|
17
|
+
|
|
18
|
+
## Redo, binlog, and replication
|
|
19
|
+
Document redo flush policy, binary-log durability, GTID usage, replication topology, semi-synchronous behavior, failover fencing, and replica consistency. Binlog retention must account for lagging replicas and CDC consumers. Replication does not replace backups.
|
|
20
|
+
|
|
21
|
+
## Query and resource performance
|
|
22
|
+
Measure buffer-pool hit behavior, dirty-page pressure, redo throughput, fsync latency, temporary-table spills, connection usage, and storage IOPS. Tune memory and redo settings from measured workload and headroom, not copied defaults.
|
|
23
|
+
Use slow-query evidence, Performance Schema, sys schema, and query plans. Check N+1 access, unbounded scans, filesort, temporary tables, skew, and parameter-sensitive plans.
|
|
24
|
+
|
|
25
|
+
## Operational readiness
|
|
26
|
+
Plan online schema changes with explicit algorithm/lock behavior and version-specific verification. Large ALTER operations may rebuild tables, consume disk, increase replication lag, or block writes. Test on production-like data and retain rollback capacity.
|
|
27
|
+
Define backup/PITR restore drills, upgrade compatibility, plugin/storage-engine dependencies, replication recovery, and capacity headroom.
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
# Sharding and Distributed SQL
|
|
2
|
+
|
|
3
|
+
## When to partition data
|
|
4
|
+
Start with a well-operated single database. Consider sharding only after measured limits in write throughput, storage, working-set size, or fault isolation cannot be addressed economically through query design, indexing, vertical scaling, read replicas, partitioning, or workload isolation.
|
|
5
|
+
- Partitioning: logical subdivision managed within a database system.
|
|
6
|
+
- Sharding: data ownership distributed across independent database nodes.
|
|
7
|
+
- Distributed SQL: a database system coordinating SQL transactions across nodes, often using consensus and distributed transaction protocols.
|
|
8
|
+
|
|
9
|
+
## Shard-key design
|
|
10
|
+
Choose a key with high cardinality, even distribution, query alignment, and stable ownership. Model hot tenants, celebrity keys, monotonically increasing keys, and skew. Hashing distributes load but weakens range locality; range sharding improves range scans but risks hotspots.
|
|
11
|
+
Define routing metadata and shard-map versioning, tenant affinity/isolation, rebalancing and split/merge, unique constraints and ID generation, and cross-shard joins/transactions/pagination/aggregation semantics.
|
|
12
|
+
|
|
13
|
+
## Correctness and operations
|
|
14
|
+
Cross-shard transactions may require two-phase commit or database-specific distributed protocols. Explain coordinator failure, prepared transaction recovery, lock duration, and availability implications. Prefer sagas only when business invariants tolerate intermediate states and compensations.
|
|
15
|
+
Plan online resharding as a state machine: establish target ownership, copy a consistent snapshot, stream changes, verify counts/checksums/invariants, fence old writers, switch routing, monitor, and retain a bounded rollback path. Prevent dual-writer ambiguity.
|
|
16
|
+
Track per-shard QPS, p95/p99 latency, CPU, storage, hot-key distribution, replication lag, queue depth, rebalance progress, and cross-shard traffic.
|
|
17
|
+
|
|
18
|
+
## Distributed SQL evaluation
|
|
19
|
+
Evaluate transaction isolation, serializability claims, consensus replication, geographic placement, follower reads, transaction retries, schema changes, online rebalancing, and failure behavior under partitions. Test actual consistency semantics and workload patterns.
|
|
20
|
+
|
|
21
|
+
## Exit criteria
|
|
22
|
+
Define thresholds for adding shards and consolidating or migrating away. Include staffing, upgrades, backup/restore, observability, and vendor/format exit costs in TCO.
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# Consensus and Distributed Transactions
|
|
2
|
+
|
|
3
|
+
## Consensus fundamentals
|
|
4
|
+
Consensus lets distributed replicas agree on an ordered sequence or value despite specified failures. Raft and Paxos-family protocols rely on quorum intersection and explicit safety/liveness assumptions; they do not make arbitrary networks synchronous.
|
|
5
|
+
For Raft-style systems, understand leader election, terms, log replication, commit index, membership changes, quorum availability, and snapshot/log compaction. A majority quorum is necessary for progress in the common crash-fault model, but latency and availability depend on placement and network conditions.
|
|
6
|
+
|
|
7
|
+
## Design questions
|
|
8
|
+
- What failure model is assumed: crash-stop, crash-recovery, Byzantine, or correlated infrastructure failure?
|
|
9
|
+
- What is the quorum and which regions/zones participate?
|
|
10
|
+
- Which writes are acknowledged: leader append, quorum commit, or durable flush?
|
|
11
|
+
- How are membership changes, snapshots, log compaction, and recovery handled?
|
|
12
|
+
- Can stale reads occur, and what read-index/lease/follower-read guarantees apply?
|
|
13
|
+
- How are fencing tokens used to prevent a former leader or stale owner from acting?
|
|
14
|
+
Do not equate leader election with application-level failover or data correctness.
|
|
15
|
+
|
|
16
|
+
## Distributed transactions
|
|
17
|
+
Two-phase commit coordinates participants through prepare and commit/abort. It can block while a coordinator decision is unavailable and hold locks/resources across phases. Explain durable coordinator logs, participant recovery, timeout semantics, and operational procedures.
|
|
18
|
+
Compare local ACID plus transactional outbox, saga orchestration/choreography with explicit compensations, two-phase commit where atomic cross-resource commit is required and supported, and idempotent workflows with reconciliation where eventual convergence is acceptable.
|
|
19
|
+
A saga is not an atomic transaction. Compensation is a new business action and may not reverse external side effects.
|
|
20
|
+
|
|
21
|
+
## Failure analysis
|
|
22
|
+
Model network partitions, lost acknowledgements, duplicate messages, coordinator crashes before/after prepare, participant restart, delayed retries, clock skew, and stale leaders. Define fencing, idempotency, durable decision records, retry budgets, and reconciliation.
|
|
23
|
+
|
|
24
|
+
## Validation
|
|
25
|
+
Use fault injection and deterministic integration tests to verify safety properties separately from liveness under stated assumptions. Document assumptions and limits.
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# Search Architecture: Elasticsearch and OpenSearch
|
|
2
|
+
|
|
3
|
+
## Search as a derived read model
|
|
4
|
+
Treat search indexes as derived data, not the authoritative source of truth unless the domain explicitly accepts that ownership model. Define ingestion from database/CDC/events, transformation, indexing, refresh visibility, deletion propagation, and reconciliation.
|
|
5
|
+
|
|
6
|
+
## Index and query design
|
|
7
|
+
- Map fields deliberately: analyzed text, keyword/exact values, numeric/date, geo, nested, and vector fields have different costs and semantics.
|
|
8
|
+
- Design analyzers, tokenization, synonyms, language handling, stemming, and normalization around user intent and relevance evaluation.
|
|
9
|
+
- Use filters for structured constraints and scoring for relevance. Avoid uncontrolled wildcard/regex queries and unbounded result windows.
|
|
10
|
+
- Use search_after with a stable sort for deep pagination; bound PIT/search-context lifecycle and resource costs.
|
|
11
|
+
- Evaluate shard count, shard size, replicas, routing, segment merges, refresh interval, and rollover policy using actual workload.
|
|
12
|
+
|
|
13
|
+
## Reliability and freshness
|
|
14
|
+
Define indexing lag and user-visible freshness SLOs. Make ingestion idempotent, handle out-of-order updates with versioning or source sequence, and propagate deletes/tombstones. Plan reindexing with aliases or versioned indexes, backfill throttling, dual-read comparison, atomic alias switch, and rollback.
|
|
15
|
+
Monitor rejected requests, thread pools, heap/GC, disk watermarks, segment count, merge pressure, shard skew, query latency, indexing latency, and recovery time.
|
|
16
|
+
|
|
17
|
+
## Relevance and quality
|
|
18
|
+
Build representative query sets and graded relevance judgments. Measure NDCG, MRR, Recall@K, precision, zero-result rate, and query reformulation. Segment by intent, language, tenant, and query class.
|
|
19
|
+
|
|
20
|
+
## Security and tenancy
|
|
21
|
+
Enforce tenant filters server-side and test bypass attempts. Protect document-level authorization, field-level restrictions, PII deletion, audit, and snapshot access. Do not rely on a client-supplied tenant filter as an authorization boundary.
|
|
22
|
+
|
|
23
|
+
## Alternatives
|
|
24
|
+
Compare database full-text search, managed search services, Elasticsearch/OpenSearch, and specialized retrieval systems by freshness, relevance control, operational burden, cost, and query complexity.
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# Vector Databases and Hybrid Retrieval
|
|
2
|
+
|
|
3
|
+
## Retrieval requirements
|
|
4
|
+
Define corpus size, embedding dimensions/model, update rate, metadata filters, tenant boundaries, query QPS, latency SLO, recall target, freshness, deletion requirements, and cost. Distinguish semantic retrieval from lexical search and exact database lookup.
|
|
5
|
+
|
|
6
|
+
## Vector index choices
|
|
7
|
+
Compare exact nearest-neighbor search with approximate methods such as HNSW, IVF, and product quantization. Evaluate recall/latency/memory trade-offs, build time, incremental updates, deletes, filtering behavior, and compaction. Measure with representative queries and ground-truth neighbors.
|
|
8
|
+
Distance functions (cosine, dot product, Euclidean) and vector normalization must match the embedding model and index configuration. Do not compare scores across incompatible models or metrics.
|
|
9
|
+
|
|
10
|
+
## Hybrid retrieval pipeline
|
|
11
|
+
1. Normalize and classify the query.
|
|
12
|
+
2. Apply authorization and tenant constraints before retrieval.
|
|
13
|
+
3. Retrieve lexical candidates (BM25) and vector candidates independently.
|
|
14
|
+
4. Fuse rankings (for example, reciprocal rank fusion) with calibrated or learned alternatives.
|
|
15
|
+
5. Rerank a bounded candidate set with a cross-encoder or domain model.
|
|
16
|
+
6. Apply diversity, freshness, and business constraints.
|
|
17
|
+
7. Return citations/provenance and enforce post-retrieval authorization.
|
|
18
|
+
Treat reranking as a latency/cost budget. Cache only with tenant- and permission-aware keys.
|
|
19
|
+
|
|
20
|
+
## RAG correctness and safety
|
|
21
|
+
- Evaluate retrieval separately from generation. Measure Recall@K, MRR/NDCG, context precision/recall, grounded answer rate, citation correctness, and abstention quality.
|
|
22
|
+
- Preserve chunk-to-source lineage, version, ACL, timestamp, and deletion tombstones.
|
|
23
|
+
- Re-check authorization at retrieval and before returning content; revocation and deletion must propagate to vector and lexical indexes.
|
|
24
|
+
- Treat retrieved text as untrusted data. Prompt injection inside documents must not grant tools, permissions, or policy changes.
|
|
25
|
+
- Version embedding models and indexes. Re-embed through a shadow index with measured quality comparison and controlled cutover.
|
|
26
|
+
|
|
27
|
+
## Storage choices
|
|
28
|
+
Compare pgvector, dedicated vector databases, search engines with vector support, and managed retrieval services. Evaluate filtering, consistency, durability, backup/restore, multi-tenancy, observability, ecosystem, and operational skills—not only benchmark throughput.
|
|
29
|
+
|
|
30
|
+
## Operational metrics
|
|
31
|
+
Track ingestion lag, embedding failures, index build/rebuild time, recall under filters, p95/p99 retrieval latency, memory per vector, cache hit rate, delete propagation, and per-tenant cost.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# Database Migrations and Zero-Downtime Schema Evolution
|
|
2
|
+
|
|
3
|
+
## Compatibility-first change
|
|
4
|
+
Use expand-migrate-contract for changes spanning application versions or large datasets. Keep old and new application versions compatible during rolling deployment.
|
|
5
|
+
1. **Expand:** add nullable columns, new tables/indexes, or compatible fields without breaking old readers/writers.
|
|
6
|
+
2. **Deploy compatibility:** code can read old/new representations and write safely under a single ownership strategy.
|
|
7
|
+
3. **Backfill:** process bounded batches with checkpoints, rate limits, idempotency, and observable progress.
|
|
8
|
+
4. **Validate:** compare counts, checksums, nullability, uniqueness, and domain invariants; sample and reconcile differences.
|
|
9
|
+
5. **Switch:** gradually move reads/writes, monitor errors and latency, and preserve a tested rollback route.
|
|
10
|
+
6. **Contract:** remove obsolete fields/indexes only after consumers and rollback windows have expired.
|
|
11
|
+
|
|
12
|
+
## Online DDL risk
|
|
13
|
+
"Online" does not mean lock-free or risk-free. Verify engine/version-specific algorithm and lock behavior, metadata locks, table rebuilds, disk headroom, replication lag, and impact on backups/CDC. Test against production-like data and concurrent traffic.
|
|
14
|
+
|
|
15
|
+
## Backfill engineering
|
|
16
|
+
Use keyset/range pagination rather than large OFFSET scans. Bound batch size, concurrency, transaction duration, and replication impact. Persist checkpoints and make each batch restartable. Avoid uncoordinated dual writes; define an authoritative path and reconciliation.
|
|
17
|
+
|
|
18
|
+
## Cutover and rollback
|
|
19
|
+
Use feature flags, shadow reads, comparison metrics, canaries, and progressive traffic shifts. Define rollback triggers and whether rollback remains possible after writes begin in the new schema. Defer irreversible contract steps until confidence and recovery evidence meet explicit gates.
|
|
20
|
+
|
|
21
|
+
## Validation checklist
|
|
22
|
+
- Schema compatibility across deployed application versions.
|
|
23
|
+
- Query plans and index behavior at expected cardinality.
|
|
24
|
+
- Backfill throughput, lag, retries, and idempotency.
|
|
25
|
+
- Data invariants, counts, checksums, and reconciliation.
|
|
26
|
+
- Lock duration, replication/CDC lag, disk, WAL/binlog, and backup capacity.
|
|
27
|
+
- Cutover, rollback, restore, and post-migration cleanup.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# API Gateway and Edge Architecture
|
|
2
|
+
|
|
3
|
+
## Responsibilities and boundaries
|
|
4
|
+
An API gateway is an edge policy and routing layer, not a replacement for domain services. Define which concerns belong at the edge (TLS termination, routing, authentication token validation, coarse rate limits, request size limits, protocol adaptation, and telemetry) and which remain in services (object-level authorization, business quotas, domain validation, transaction semantics, and authoritative decisions).
|
|
5
|
+
|
|
6
|
+
## Topology choices
|
|
7
|
+
Compare a managed gateway, ingress controller, service-mesh ingress, reverse proxy, and application BFF. Evaluate latency overhead, availability dependencies, policy expressiveness, multi-region behavior, deployment ownership, observability, and cost. Avoid stacking multiple gateways that each retry, buffer, or transform requests without a clear contract.
|
|
8
|
+
|
|
9
|
+
## Routing and resilience
|
|
10
|
+
Specify host/path/header routing, versioned routes, canary weights, health-based endpoint selection, connection draining, deadlines, retry budgets, circuit breakers, and load shedding. Retry only safe or explicitly idempotent operations; enforce one end-to-end deadline rather than multiplying per-hop timeouts. Bound request/response buffering and concurrent connections.
|
|
11
|
+
|
|
12
|
+
## Security
|
|
13
|
+
Terminate TLS with a documented trust model; validate issuer, audience, signature, expiry, and key rotation for tokens. Strip untrusted forwarding headers and reconstruct client identity only from trusted proxies. Apply WAF rules as defense in depth, not as authorization. Prevent request smuggling by consistent HTTP parsing and hop-by-hop header handling. Keep secrets out of logs.
|
|
14
|
+
|
|
15
|
+
## Operations and validation
|
|
16
|
+
Track route-level request rate, status, latency percentiles, upstream saturation, retries, rejected traffic, connection errors, and policy changes. Test gateway failure, upstream brownouts, slow clients, oversized payloads, certificate rotation, and configuration rollback. Gateway availability is part of the critical path; define its SLO and recovery plan.
|
|
17
|
+
|
|
18
|
+
## Decision checklist
|
|
19
|
+
- [ ] Edge responsibilities and service responsibilities are explicit.
|
|
20
|
+
- [ ] Trust proxy chain and client-IP derivation are documented.
|
|
21
|
+
- [ ] Deadlines, retries, payload limits, and connection limits are bounded.
|
|
22
|
+
- [ ] Route changes support staged rollout and fast rollback.
|
|
23
|
+
- [ ] Gateway failure and regional failover have been exercised.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# REST API Design and Lifecycle
|
|
2
|
+
|
|
3
|
+
## Resource and contract design
|
|
4
|
+
Model stable resources and relationships; use HTTP methods and status codes consistently. Define canonical identifiers, filtering, sorting, pagination, partial updates, field masks, and error envelopes. Prefer cursor pagination for mutable or large datasets; document cursor stability and expiry. Avoid leaking storage schemas as public contracts.
|
|
5
|
+
|
|
6
|
+
## Correctness
|
|
7
|
+
Use conditional requests (ETag/If-Match) or explicit versions to prevent lost updates. Define idempotency-key scope, retention, payload binding, concurrent duplicate behavior, and replay response semantics for retryable creates or payments. A client timeout leaves the outcome unknown: provide a way to query operation status or safely retry with the same key.
|
|
8
|
+
|
|
9
|
+
## Compatibility and evolution
|
|
10
|
+
Treat additive changes as potentially breaking when clients use strict decoders or exhaustive enums. Version only when compatibility policy requires it; define deprecation notice, telemetry, support window, sunset criteria, and owner. Maintain consumer contract tests and generated-client compatibility checks. Document stable versus experimental fields.
|
|
11
|
+
|
|
12
|
+
## Security and usability
|
|
13
|
+
Specify authentication, object- and function-level authorization, tenant scoping, input validation, mass-assignment defenses, rate limits, and data classification. Use consistent machine-readable errors with correlation IDs, but never expose stack traces or secrets. Define CORS deliberately; it is not an authorization control.
|
|
14
|
+
|
|
15
|
+
## Operations
|
|
16
|
+
Publish latency and availability objectives per critical operation. Set request/response limits, pagination caps, deadlines, and retry guidance. Instrument business outcomes as well as HTTP status. Test concurrent updates, duplicate requests, stale ETags, malformed inputs, and old-client compatibility.
|
|
17
|
+
|
|
18
|
+
## Review checklist
|
|
19
|
+
- [ ] OpenAPI or equivalent contract is version-controlled and validated.
|
|
20
|
+
- [ ] Idempotency and concurrency semantics are explicit.
|
|
21
|
+
- [ ] Pagination, errors, and filtering are bounded and deterministic.
|
|
22
|
+
- [ ] AuthZ is enforced at object and function level.
|
|
23
|
+
- [ ] Deprecation and compatibility are observable and owned.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# GraphQL Architecture and Governance
|
|
2
|
+
|
|
3
|
+
## Schema and ownership
|
|
4
|
+
Design the graph around domain capabilities and consumer needs. Define field ownership, resolver ownership, nullability semantics, deprecation policy, schema composition, and federation boundaries. Avoid a graph that becomes a shared database facade with unclear authorization and change ownership.
|
|
5
|
+
|
|
6
|
+
## Query execution controls
|
|
7
|
+
GraphQL moves query shape to clients; it does not remove backend cost. Apply depth, complexity, alias, batch, and response-size limits. Use persisted/allowlisted operations for high-risk or mobile clients where appropriate. Prevent N+1 access with request-scoped DataLoader-style batching while preserving tenant and authorization context. Bound resolver concurrency and downstream fan-out.
|
|
8
|
+
|
|
9
|
+
## Authorization and data protection
|
|
10
|
+
Enforce authorization in resolvers or domain services for every object and field; never rely solely on hiding schema fields. Prevent cross-tenant cache leakage by including principal/tenant and policy context in cache keys. Restrict introspection according to threat model, but do not treat disabling it as a substitute for access control. Redact sensitive fields from traces and error payloads.
|
|
11
|
+
|
|
12
|
+
## Federation and consistency
|
|
13
|
+
For federated graphs, define entity ownership, key stability, reference resolution, timeout budgets, partial failure semantics, and schema rollout ordering. Avoid cyclic dependencies and excessive cross-subgraph waterfalls. Treat composed schema checks as a release gate; test old and new subgraph combinations.
|
|
14
|
+
|
|
15
|
+
## Evaluation
|
|
16
|
+
Measure operation-level latency, resolver fan-out, cache hit ratio, downstream calls, error rate, and query-cost rejection. Test adversarial deep/aliased queries, partial data errors, null propagation, authorization bypass, and schema compatibility. Compare GraphQL with REST/BFF for actual client needs rather than assuming one is universally superior.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# gRPC and Service Contracts
|
|
2
|
+
|
|
3
|
+
## Protocol fit
|
|
4
|
+
Use gRPC when typed contracts, efficient binary transport, streaming, generated clients, or polyglot service communication justify the operational and browser-compatibility costs. Compare REST/JSON, gRPC, and asynchronous messaging by consumer environment, debugging, proxy support, latency, and evolution requirements.
|
|
5
|
+
|
|
6
|
+
## Protobuf evolution
|
|
7
|
+
Treat field numbers as permanent identifiers: never reuse deleted numbers or names; reserve them. Add fields compatibly, define defaults and presence semantics, and avoid changing field meaning or wire type. Enums need an unspecified zero value and tolerant handling of unknown values. Validate compatibility in CI across supported producer/consumer versions.
|
|
8
|
+
|
|
9
|
+
## RPC semantics
|
|
10
|
+
Define deadlines, cancellation propagation, status codes, retryable conditions, idempotency, metadata limits, message size limits, and streaming flow control. A transport retry can duplicate a side effect; retries must respect method semantics and an end-to-end budget. Distinguish application errors from transport failures.
|
|
11
|
+
|
|
12
|
+
## Security and operations
|
|
13
|
+
Use workload identity and mTLS where appropriate; authorize every method and resource. Protect reflection and health endpoints according to exposure. Instrument method-level latency, status, message sizes, active streams, cancellations, and retry attempts. Test rolling upgrades, proxy/load-balancer behavior, slow consumers, stream resets, and mixed-version compatibility.
|
|
14
|
+
|
|
15
|
+
## Streaming
|
|
16
|
+
For client/server or bidirectional streams, define message ordering, backpressure, heartbeats, idle timeout, reconnect/resume tokens, replay boundaries, and maximum stream duration. Do not assume a broken stream means the server did not process the last message.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# WebSocket, Server-Sent Events, and Real-Time Systems
|
|
2
|
+
|
|
3
|
+
## Transport selection
|
|
4
|
+
Choose request/response for ordinary interactions, Server-Sent Events (SSE) for one-way server-to-client updates, and WebSocket for bidirectional low-latency sessions. Consider WebRTC for peer media/data only when its NAT traversal and media semantics are needed. Evaluate browser/proxy compatibility, connection duration, fan-out, and operational complexity.
|
|
5
|
+
|
|
6
|
+
## Connection architecture
|
|
7
|
+
Model connection establishment, authentication, authorization, heartbeat, idle timeout, reconnect, session affinity, regional routing, and graceful drain. Keep connection state disposable where possible; store durable business state outside gateway memory. Use pub/sub or partitioned brokers to distribute events to connection-owning nodes, with explicit delivery and ordering guarantees.
|
|
8
|
+
|
|
9
|
+
## Backpressure and scale
|
|
10
|
+
Bound per-connection outbound queues, message size, fan-out, subscriptions, and total concurrent connections. Define slow-consumer policy (drop, coalesce, disconnect, or durable replay) by message class. Capacity-plan file descriptors, memory per connection, heartbeat traffic, TLS, broker throughput, and reconnect storms.
|
|
11
|
+
|
|
12
|
+
## Delivery semantics
|
|
13
|
+
Treat live transports as best-effort unless backed by durable offsets or application acknowledgements. Include event IDs, sequence numbers, resume cursors, deduplication, and snapshot-plus-delta recovery when missed updates matter. Define whether ordering is per entity, room, partition, or connection; avoid claiming global ordering without a mechanism.
|
|
14
|
+
|
|
15
|
+
## Security and reliability
|
|
16
|
+
Authorize subscriptions and every sensitive message; revalidate long-lived sessions and revoke access after permission changes. Validate Origin for browser WebSockets, protect against cross-site WebSocket hijacking, enforce quotas, and avoid logging payloads containing PII. Test node loss, broker partitions, reconnect storms, slow consumers, and regional failover.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Real-Time Collaboration: OT and CRDT
|
|
2
|
+
|
|
3
|
+
## Problem framing
|
|
4
|
+
Define collaboration semantics before choosing an algorithm: document model, concurrent edit types, offline duration, expected participants, latency, conflict tolerance, undo/redo, presence, and persistence. Presence and cursors are ephemeral; document operations and snapshots are durable domain state.
|
|
5
|
+
|
|
6
|
+
## Operational Transformation
|
|
7
|
+
OT transforms concurrent operations against one another and depends on a well-defined transformation function, ordering authority, and server/client protocol. Specify revision numbers, operation identity, acknowledgements, reconnect catch-up, and transformation correctness. Test convergence across permutations and stale clients.
|
|
8
|
+
|
|
9
|
+
## Conflict-free Replicated Data Types
|
|
10
|
+
CRDTs encode merge behavior so replicas converge under stated assumptions. Choose a data type whose semantics match the product; convergence does not guarantee intuitive user intent, bounded metadata, or low storage cost. Model tombstones/garbage collection, causal metadata, offline replicas, compaction, and schema evolution. State whether the design is state-based or operation-based and its delivery assumptions.
|
|
11
|
+
|
|
12
|
+
## Persistence and security
|
|
13
|
+
Persist operations or snapshots with atomic revision/checkpoint boundaries. Make operation application idempotent and validate authorization at the document/operation boundary. Enforce tenant isolation, document ACL changes, retention, export, and deletion across replicas and backups. Do not trust client-generated operations.
|
|
14
|
+
|
|
15
|
+
## Testing and trade-offs
|
|
16
|
+
Use property-based and randomized concurrency tests, network partitions, duplicate/reordered delivery, offline edits, and mixed-version clients. Measure convergence time, metadata growth, sync bandwidth, conflict UX, and recovery. Choose centralized sequencing when it is simpler and meets offline/latency requirements; CRDT is not a default requirement for every collaborative feature.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# API Security, Rate Limiting, and Abuse Prevention
|
|
2
|
+
|
|
3
|
+
## Threat model
|
|
4
|
+
Cover broken object/function authorization, credential stuffing, scraping, enumeration, replay, resource exhaustion, injection, SSRF, request smuggling, webhook spoofing, and tenant escape. Map controls to principals, endpoints, resources, and business impact. Authentication alone does not establish permission to access an object.
|
|
5
|
+
|
|
6
|
+
## Rate-limit dimensions
|
|
7
|
+
Apply limits at appropriate layers: global protection, per-IP or network, per-credential, per-user, per-tenant, per-route, and expensive-operation budgets. Combine request rate with concurrency, cost units, payload size, and burst allowance. Avoid IP-only controls for NAT/mobile users and avoid user-only controls that permit anonymous abuse.
|
|
8
|
+
|
|
9
|
+
## Algorithms and distributed enforcement
|
|
10
|
+
Compare token bucket, leaky bucket, fixed window, and sliding window. Document burst semantics, clock assumptions, key cardinality, storage consistency, and failure mode. Distributed counters may be approximate; state whether enforcement is best-effort or strict. Use local emergency limits when the shared limiter is unavailable, and avoid a limiter dependency that becomes a single point of failure.
|
|
11
|
+
|
|
12
|
+
## Fairness and degradation
|
|
13
|
+
Define tenant tiers, weighted fairness, priority classes, retry-after behavior, and quota reset semantics. Return actionable 429/503 responses without leaking internal policy. Apply load shedding and queue admission controls to protect critical traffic. Avoid retry storms by communicating backoff and enforcing client retry budgets.
|
|
14
|
+
|
|
15
|
+
## Abuse operations
|
|
16
|
+
Monitor rejected requests, credential anomalies, expensive query patterns, cost per tenant, and false positives. Build alerting, investigation, appeal/override, and emergency block workflows. Test bypass attempts, distributed bursts, hot keys, limiter outage, clock skew, and noisy-neighbor isolation. Rate limiting complements authorization, WAF, and business fraud controls; it does not replace them.
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
# OAuth 2.1, OpenID Connect, and Passkeys
|
|
2
|
+
|
|
3
|
+
## Protocol roles
|
|
4
|
+
OAuth is an authorization framework; OpenID Connect adds an identity layer. Distinguish the authorization server, resource server, client, subject, and relying party. Use Authorization Code with PKCE for interactive public clients; do not embed client secrets in mobile or browser applications. Avoid implicit and resource-owner-password flows. Verify current standards and provider-specific behavior before deployment.
|
|
5
|
+
|
|
6
|
+
## Token validation and lifecycle
|
|
7
|
+
Validate signature using trusted issuer metadata and rotating keys, issuer, audience, expiry, not-before, authorized party where applicable, and allowed algorithms. Access tokens should be audience-restricted and short-lived according to risk. Define refresh-token rotation, reuse detection, revocation, session termination, and key rollover. Do not treat an ID token as an API access token.
|
|
8
|
+
|
|
9
|
+
## Redirects and browser security
|
|
10
|
+
Use exact registered redirect URIs, state binding, nonce for OIDC authentication, PKCE, and strict callback handling. Protect cookies with Secure, HttpOnly, and appropriate SameSite attributes. Defend against open redirects, login CSRF, token leakage through URLs, and unsafe post-logout redirects.
|
|
11
|
+
|
|
12
|
+
## Passkeys and WebAuthn
|
|
13
|
+
Passkeys use public-key credentials scoped to a relying party. Verify challenge freshness and one-time use, origin, RP ID, user presence/verification policy, credential ownership, and signature counter behavior according to authenticator capabilities. Design recovery, device replacement, account linking, and phishing-resistant step-up flows; passkeys do not eliminate account-recovery risk.
|
|
14
|
+
|
|
15
|
+
## Review checklist
|
|
16
|
+
- [ ] OAuth and OIDC responsibilities are separated correctly.
|
|
17
|
+
- [ ] PKCE, state, nonce, exact redirect URI, issuer and audience checks are enforced.
|
|
18
|
+
- [ ] Refresh rotation, revocation, and key rotation are tested.
|
|
19
|
+
- [ ] Passkey registration, assertion, origin, challenge, and recovery flows are tested.
|
|
20
|
+
- [ ] Tokens and authentication codes never enter logs, analytics, or referrers.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Identity and Access Management: RBAC, ABAC, and ReBAC
|
|
2
|
+
|
|
3
|
+
## Authorization model selection
|
|
4
|
+
RBAC assigns permissions through roles and suits stable organizational responsibilities. ABAC evaluates subject, resource, action, and environment attributes. ReBAC evaluates relationships such as owner, member, parent, and delegated access. Real systems may combine models, but policy composition and precedence must be explicit.
|
|
5
|
+
|
|
6
|
+
## Policy design
|
|
7
|
+
Define principal types, resource hierarchy, actions, scopes, tenant boundaries, deny/allow precedence, default-deny behavior, and delegation. Separate authentication from authorization and coarse gateway checks from authoritative domain checks. Enforce object-level authorization on every read, write, export, and background operation.
|
|
8
|
+
|
|
9
|
+
## Policy evaluation
|
|
10
|
+
Centralized policy engines improve consistency but add latency and availability dependencies. Define policy distribution, cache TTL, invalidation, fail-open/fail-closed behavior, versioning, and explainability. Cache keys must include principal, tenant, resource, action, relevant attributes, and policy version. Avoid stale grants after role removal or ACL changes.
|
|
11
|
+
|
|
12
|
+
## Lifecycle and governance
|
|
13
|
+
Model joiner/mover/leaver workflows, periodic access reviews, privileged access, service accounts, break-glass access, separation of duties, and audit evidence. Prefer time-bound grants and just-in-time elevation. Reconcile identity-provider groups with application permissions; group membership alone is not a complete authorization decision.
|
|
14
|
+
|
|
15
|
+
## Testing
|
|
16
|
+
Test horizontal and vertical privilege escalation, cross-tenant access, inherited permissions, revoked access, policy conflicts, stale caches, and background jobs executing under the wrong identity. Use policy unit tests and representative end-to-end authorization matrices.
|