system-design-primer-skill 1.2.0 → 2.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +114 -60
- package/SKILL.md +437 -93
- package/VERSION +1 -1
- package/evals/README.md +1022 -0
- package/evals/system-design-cases.json +414 -0
- package/package.json +46 -4
- package/references/03-data-and-storage.md +8 -2
- package/references/04-caching.md +3 -3
- package/references/05-async-and-messaging.md +4 -0
- package/references/06-distributed-systems.md +2 -2
- package/references/07-reliability-and-resilience.md +7 -3
- package/references/08-api-security-observability.md +4 -0
- package/references/10-workflows-and-checklists.md +1 -1
- package/references/13-architecture-intelligence.md +91 -0
- package/references/14-api-and-data-contracts.md +80 -0
- package/references/15-failure-analysis.md +88 -0
- package/references/16-multi-agent-architecture.md +76 -0
- package/references/17-agent-harness-and-runtime.md +87 -0
- package/references/18-mcp-architecture.md +67 -0
- package/references/19-agent-security-and-permissions.md +90 -0
- package/references/20-agent-state-and-memory.md +76 -0
- package/references/21-agent-evaluation-and-observability.md +90 -0
- package/references/22-kubernetes-and-container-platforms.md +29 -0
- package/references/23-service-mesh-and-traffic-management.md +21 -0
- package/references/24-multi-region-and-disaster-recovery.md +28 -0
- package/references/25-platform-engineering-and-developer-experience.md +26 -0
- package/references/26-finops-and-cloud-cost.md +29 -0
- package/references/27-production-observability-and-slo.md +33 -0
- package/references/28-event-streaming-and-kafka-pulsar.md +33 -0
- package/references/29-event-sourcing-and-cqrs.md +23 -0
- package/references/30-change-data-capture-and-debezium.md +27 -0
- package/references/31-data-lakehouse-and-stream-batch.md +24 -0
- package/references/32-workflow-orchestration-and-temporal.md +28 -0
- package/references/33-schema-registry-and-event-governance.md +22 -0
- package/references/34-postgresql-internals-and-performance.md +45 -0
- package/references/35-mysql-innodb-and-performance.md +29 -0
- package/references/36-sharding-and-distributed-sql.md +22 -0
- package/references/37-consensus-and-distributed-transactions.md +29 -0
- package/references/38-search-architecture-elasticsearch-opensearch.md +24 -0
- package/references/39-vector-databases-and-hybrid-retrieval.md +31 -0
- package/references/40-zero-downtime-database-migrations.md +27 -0
- package/references/41-api-gateway-and-edge-architecture.md +23 -0
- package/references/42-rest-api-design-and-lifecycle.md +23 -0
- package/references/43-graphql-architecture-and-governance.md +16 -0
- package/references/44-grpc-and-service-contracts.md +16 -0
- package/references/45-websocket-sse-and-realtime-systems.md +16 -0
- package/references/46-realtime-collaboration-ot-crdt.md +16 -0
- package/references/47-api-security-rate-limiting-and-abuse.md +16 -0
- package/references/48-oauth-oidc-and-passkeys.md +20 -0
- package/references/49-iam-rbac-abac-rebac.md +20 -0
- package/references/50-zero-trust-and-workload-identity.md +16 -0
- package/references/51-secrets-kms-pki-and-key-management.md +20 -0
- package/references/52-supply-chain-security-sbom-slsa.md +78 -0
- package/references/53-threat-modeling-stride-and-attack-trees.md +16 -0
- package/references/54-privacy-engineering-and-data-governance.md +20 -0
- package/references/55-opentelemetry-and-instrumentation.md +54 -0
- package/references/56-distributed-tracing-and-context-propagation.md +47 -0
- package/references/57-continuous-profiling-and-performance-diagnostics.md +45 -0
- package/references/58-load-stress-soak-testing-and-workload-modeling.md +44 -0
- package/references/59-capacity-planning-and-performance-modeling.md +38 -0
- package/references/60-chaos-engineering-and-resilience-experiments.md +45 -0
- package/references/61-incident-management-and-postmortems.md +38 -0
- package/references/62-dns-and-service-discovery.md +35 -0
- package/references/63-load-balancing-and-global-traffic-management.md +31 -0
- package/references/64-cdn-and-edge-caching.md +34 -0
- package/references/65-cloud-networking-and-connectivity.md +34 -0
- package/references/66-edge-computing-architecture.md +33 -0
- package/references/67-storage-infrastructure-and-storage-systems.md +34 -0
- package/references/68-tcp-quic-and-network-performance.md +27 -0
- package/references/69-task-routing-and-evidence.md +115 -0
- package/scripts/audit_chain.py +36 -0
- package/scripts/compare_reports.py +43 -0
- package/scripts/evaluate.py +475 -0
- package/scripts/evaluation_catalog.py +129 -0
- package/scripts/evaluation_fingerprints.py +20 -0
- package/scripts/evidence_archive.py +79 -0
- package/scripts/evidence_signing.py +98 -0
- package/scripts/evidence_timestamp.py +441 -0
- package/scripts/install.ps1 +12 -3
- package/scripts/install.sh +15 -8
- package/scripts/report_integrity.py +184 -0
- package/scripts/review_consensus.py +77 -0
- package/scripts/review_evidence.py +105 -0
- package/scripts/review_provenance.py +44 -0
- package/scripts/reviewer_calibration.py +215 -0
- package/scripts/rubric_compatibility.py +134 -0
- package/scripts/run_evals.py +130 -0
- package/scripts/run_manifest.py +179 -0
- package/scripts/statistical_analysis.py +308 -0
- package/scripts/tsa_attestation.py +709 -0
- package/scripts/tsa_validation_report.py +627 -0
- package/scripts/validate.py +356 -11
- package/scripts/verify_build_provenance.py +625 -0
- package/scripts/verify_tsa_ocsp.py +962 -0
- package/scripts/verify_tsa_revocation.py +782 -0
- package/scripts/windows_broker_protocol.py +274 -0
- package/scripts/windows_broker_transport.py +141 -0
- package/scripts/windows_broker_win32.py +713 -0
- package/templates/agent-evaluation-plan-template.md +57 -0
- package/templates/agent-security-review-template.md +46 -0
- package/templates/agent-system-design-template.md +74 -0
- package/templates/api-architecture-design-template.md +33 -0
- package/templates/api-security-and-rate-limit-review-template.md +29 -0
- package/templates/architecture-decision-matrix-template.md +48 -0
- package/templates/architecture-review-scorecard-template.md +36 -0
- package/templates/cdc-data-pipeline-template.md +41 -0
- package/templates/cdn-and-edge-strategy-template.md +36 -0
- package/templates/chaos-experiment-template.md +39 -0
- package/templates/cloud-cost-model-template.md +37 -0
- package/templates/database-architecture-review-template.md +33 -0
- package/templates/edge-computing-readiness-template.md +36 -0
- package/templates/event-sourcing-cqrs-template.md +41 -0
- package/templates/event-streaming-design-template.md +47 -0
- package/templates/failure-analysis-template.md +36 -0
- package/templates/identity-and-access-architecture-template.md +29 -0
- package/templates/incident-postmortem-template.md +39 -0
- package/templates/multi-region-disaster-recovery-template.md +43 -0
- package/templates/network-architecture-design-template.md +38 -0
- package/templates/observability-architecture-template.md +52 -0
- package/templates/performance-test-plan-template.md +52 -0
- package/templates/privacy-data-governance-template.md +24 -0
- package/templates/production-readiness-checklist-template.md +52 -0
- package/templates/realtime-system-design-template.md +28 -0
- package/templates/security-threat-model-template.md +26 -0
- package/templates/sharding-distributed-sql-design-template.md +34 -0
- package/templates/storage-architecture-design-template.md +35 -0
- package/templates/supply-chain-security-review-template.md +20 -0
- package/templates/zero-downtime-database-migration-template.md +32 -0
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# MySQL and InnoDB Architecture and Performance
|
|
2
|
+
|
|
3
|
+
## Storage and indexing
|
|
4
|
+
InnoDB stores row data in the clustered primary-key B-tree. Secondary-index records also carry the primary-key value, so a wide primary key increases secondary-index footprint.
|
|
5
|
+
- Choose stable, compact primary keys and verify insertion locality and hotspot behavior.
|
|
6
|
+
- Design composite indexes around leftmost-prefix, filtering, and ordering requirements; measure covering-index benefits against write amplification.
|
|
7
|
+
- Use EXPLAIN and, where supported, EXPLAIN ANALYZE. Compare estimated/actual rows, access paths, join order, temporary tables, and filesort.
|
|
8
|
+
- Evaluate partitioning only when pruning, lifecycle management, or operational needs justify its constraints; it is not a substitute for indexing or sharding.
|
|
9
|
+
|
|
10
|
+
## Transactions, locks, and MVCC
|
|
11
|
+
InnoDB uses undo records for consistent reads and rollback, and redo logging for crash recovery. Under REPEATABLE READ, consistent reads and locking reads differ; next-key and gap locks can prevent phantom inserts but increase contention.
|
|
12
|
+
|
|
13
|
+
For MySQL 8.4, REPEATABLE READ is the InnoDB default: plain consistent reads use the snapshot established by the first consistent read, while locking reads and DML use current data and acquire locks according to the index and search predicate. A unique-index equality lookup can lock only the matching index record; range predicates can take next-key/gap locks. READ COMMITTED creates a fresh snapshot for each consistent read and disables most gap locking, with documented exceptions such as foreign-key and duplicate-key checks. Verify these details for the exact MySQL release and statement shape in the [official isolation-level reference](https://dev.mysql.com/doc/refman/8.4/en/innodb-transaction-isolation-levels.html); do not infer lock coverage from SQL text alone.
|
|
14
|
+
- Define isolation and read/write semantics for each critical transaction.
|
|
15
|
+
- Diagnose deadlocks from engine reports; treat them as expected concurrency outcomes and retry bounded, idempotent transactions.
|
|
16
|
+
- Keep transactions short, access rows in consistent order, and index predicates used by locking statements.
|
|
17
|
+
- Monitor history-list length, purge lag, undo growth, lock waits, and long-running transactions.
|
|
18
|
+
- Do not assume a plain SELECT protects a business invariant; use unique constraints, conditional writes, or explicit locking.
|
|
19
|
+
|
|
20
|
+
## Redo, binlog, and replication
|
|
21
|
+
Document redo flush policy, binary-log durability, GTID usage, replication topology, semi-synchronous behavior, failover fencing, and replica consistency. Binlog retention must account for lagging replicas and CDC consumers. Replication does not replace backups.
|
|
22
|
+
|
|
23
|
+
## Query and resource performance
|
|
24
|
+
Measure buffer-pool hit behavior, dirty-page pressure, redo throughput, fsync latency, temporary-table spills, connection usage, and storage IOPS. Tune memory and redo settings from measured workload and headroom, not copied defaults.
|
|
25
|
+
Use slow-query evidence, Performance Schema, sys schema, and query plans. Check N+1 access, unbounded scans, filesort, temporary tables, skew, and parameter-sensitive plans.
|
|
26
|
+
|
|
27
|
+
## Operational readiness
|
|
28
|
+
Plan online schema changes with explicit algorithm/lock behavior and version-specific verification. Large ALTER operations may rebuild tables, consume disk, increase replication lag, or block writes. Test on production-like data and retain rollback capacity.
|
|
29
|
+
Define backup/PITR restore drills, upgrade compatibility, plugin/storage-engine dependencies, replication recovery, and capacity headroom.
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
# Sharding and Distributed SQL
|
|
2
|
+
|
|
3
|
+
## When to partition data
|
|
4
|
+
Start with a well-operated single database. Consider sharding only after measured limits in write throughput, storage, working-set size, or fault isolation cannot be addressed economically through query design, indexing, vertical scaling, read replicas, partitioning, or workload isolation.
|
|
5
|
+
- Partitioning: logical subdivision managed within a database system.
|
|
6
|
+
- Sharding: data ownership distributed across independent database nodes.
|
|
7
|
+
- Distributed SQL: a database system coordinating SQL transactions across nodes, often using consensus and distributed transaction protocols.
|
|
8
|
+
|
|
9
|
+
## Shard-key design
|
|
10
|
+
Choose a key with high cardinality, even distribution, query alignment, and stable ownership. Model hot tenants, celebrity keys, monotonically increasing keys, and skew. Hashing distributes load but weakens range locality; range sharding improves range scans but risks hotspots.
|
|
11
|
+
Define routing metadata and shard-map versioning, tenant affinity/isolation, rebalancing and split/merge, unique constraints and ID generation, and cross-shard joins/transactions/pagination/aggregation semantics.
|
|
12
|
+
|
|
13
|
+
## Correctness and operations
|
|
14
|
+
Cross-shard transactions may require two-phase commit or database-specific distributed protocols. Explain coordinator failure, prepared transaction recovery, lock duration, and availability implications. Prefer sagas only when business invariants tolerate intermediate states and compensations.
|
|
15
|
+
Plan online resharding as a state machine: establish target ownership, copy a consistent snapshot, stream changes from a well-defined log position, verify counts/checksums/invariants, fence old writers, switch routing at a defined ownership epoch, monitor, and retain a bounded rollback path. Prevent dual-writer ambiguity. Define how requests already in flight at cutover are drained or rejected, how retries discover the new owner, and how stale shard-map caches are invalidated or safely rejected.
|
|
16
|
+
Track per-shard QPS, p95/p99 latency, CPU, storage, hot-key distribution, replication lag, queue depth, rebalance progress, and cross-shard traffic.
|
|
17
|
+
|
|
18
|
+
## Distributed SQL evaluation
|
|
19
|
+
Evaluate transaction isolation, serializability claims, consensus replication, geographic placement, follower reads, transaction retries, schema changes, online rebalancing, and failure behavior under partitions. Test actual consistency semantics and workload patterns.
|
|
20
|
+
|
|
21
|
+
## Exit criteria
|
|
22
|
+
Define thresholds for adding shards and consolidating or migrating away. Include staffing, upgrades, backup/restore, observability, and vendor/format exit costs in TCO.
|
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# Consensus and Distributed Transactions
|
|
2
|
+
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns consensus protocol safety/liveness assumptions and cross-resource transaction guarantees, including quorum decisions, two-phase commit, saga boundaries, and ambiguous outcomes. Use [distributed systems](06-distributed-systems.md) for the broader consistency and coordination vocabulary; [data and storage](03-data-and-storage.md) for database-local transaction and isolation semantics; [asynchronous processing](05-async-and-messaging.md) for broker-agnostic delivery and outbox/inbox patterns; [event streaming](28-event-streaming-and-kafka-pulsar.md) for broker-specific guarantees; and [durable workflows](32-workflow-orchestration-and-temporal.md) for long-running process execution. Consensus commit, message acknowledgement, workflow history, and business completion are distinct evidence boundaries.
|
|
6
|
+
|
|
7
|
+
## Consensus fundamentals
|
|
8
|
+
Consensus lets distributed replicas agree on an ordered sequence or value despite specified failures. Raft and Paxos-family protocols rely on quorum intersection and explicit safety/liveness assumptions; they do not make arbitrary networks synchronous.
|
|
9
|
+
For Raft-style systems, understand leader election, terms, log replication, commit index, membership changes, quorum availability, and snapshot/log compaction. A majority quorum is necessary for progress in the common crash-fault model, but latency and availability depend on placement and network conditions. A committed log entry is not automatically equivalent to an acknowledged application-level business operation: specify the storage durability contract, state-machine application, client response point, and behavior after an ambiguous timeout. Linearizable reads require an appropriate protocol (for example, a quorum-confirmed read or a valid lease with bounded-clock assumptions); a local read from a leader or follower is not inherently linearizable.
|
|
10
|
+
|
|
11
|
+
## Design questions
|
|
12
|
+
- What failure model is assumed: crash-stop, crash-recovery, Byzantine, or correlated infrastructure failure?
|
|
13
|
+
- What is the quorum and which regions/zones participate?
|
|
14
|
+
- Which writes are acknowledged: leader append, quorum commit, or durable flush?
|
|
15
|
+
- How are membership changes, snapshots, log compaction, and recovery handled?
|
|
16
|
+
- Can stale reads occur, and what read-index/lease/follower-read guarantees apply?
|
|
17
|
+
- How are fencing tokens used to prevent a former leader or stale owner from acting?
|
|
18
|
+
Do not equate leader election with application-level failover or data correctness.
|
|
19
|
+
|
|
20
|
+
## Distributed transactions
|
|
21
|
+
Two-phase commit coordinates participants through prepare and commit/abort. It can block while a coordinator decision is unavailable and hold locks/resources across phases. A participant that has voted prepared must not unilaterally guess commit or abort after losing the coordinator; it needs the durable global decision or a protocol-defined recovery path. Explain durable coordinator logs, participant recovery, timeout semantics, in-doubt transaction monitoring, lock/resource limits, and operational procedures. A timeout or lost commit acknowledgement creates an unknown outcome, not proof of rollback; clients need a transaction-status lookup or idempotent retry keyed to the same business operation.
|
|
22
|
+
Compare local ACID plus transactional outbox, saga orchestration/choreography with explicit compensations, two-phase commit where atomic cross-resource commit is required and supported, and idempotent workflows with reconciliation where eventual convergence is acceptable. The outbox makes the database state change and event intent atomic within one local transaction; a relay can publish more than once, so consumers still need deduplication/idempotency. A saga needs durable step state, bounded retries, timeouts, compensation policy, and a terminal/manual-reconciliation path. Choose based on the invariant and participant capabilities, not on a blanket preference for one pattern.
|
|
23
|
+
A saga is not an atomic transaction. Compensation is a new business action and may not reverse external side effects.
|
|
24
|
+
|
|
25
|
+
## Failure analysis
|
|
26
|
+
Model network partitions, lost acknowledgements, duplicate messages, coordinator crashes before/after prepare, participant restart, delayed retries, clock skew, and stale leaders. Define fencing, idempotency, durable decision records, retry budgets, and reconciliation.
|
|
27
|
+
|
|
28
|
+
## Validation
|
|
29
|
+
Use fault injection and deterministic integration tests to verify safety properties separately from liveness under stated assumptions. Document assumptions and limits.
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# Search Architecture: Elasticsearch and OpenSearch
|
|
2
|
+
|
|
3
|
+
## Search as a derived read model
|
|
4
|
+
Treat search indexes as derived data, not the authoritative source of truth unless the domain explicitly accepts that ownership model. Define ingestion from database/CDC/events, transformation, indexing, refresh visibility, deletion propagation, and reconciliation.
|
|
5
|
+
|
|
6
|
+
## Index and query design
|
|
7
|
+
- Map fields deliberately: analyzed text, keyword/exact values, numeric/date, geo, nested, and vector fields have different costs and semantics.
|
|
8
|
+
- Design analyzers, tokenization, synonyms, language handling, stemming, and normalization around user intent and relevance evaluation.
|
|
9
|
+
- Use filters for structured constraints and scoring for relevance. Avoid uncontrolled wildcard/regex queries and unbounded result windows.
|
|
10
|
+
- Use search_after with a stable sort for deep pagination; bound PIT/search-context lifecycle and resource costs.
|
|
11
|
+
- Evaluate shard count, shard size, replicas, routing, segment merges, refresh interval, and rollover policy using actual workload.
|
|
12
|
+
|
|
13
|
+
## Reliability and freshness
|
|
14
|
+
Define indexing lag and user-visible freshness SLOs. Make ingestion idempotent, handle out-of-order updates with versioning or source sequence, and propagate deletes/tombstones. Plan reindexing with aliases or versioned indexes, backfill throttling, dual-read comparison, atomic alias switch, and rollback.
|
|
15
|
+
Monitor rejected requests, thread pools, heap/GC, disk watermarks, segment count, merge pressure, shard skew, query latency, indexing latency, and recovery time.
|
|
16
|
+
|
|
17
|
+
## Relevance and quality
|
|
18
|
+
Build representative query sets and graded relevance judgments. Measure NDCG, MRR, Recall@K, precision, zero-result rate, and query reformulation. Segment by intent, language, tenant, and query class.
|
|
19
|
+
|
|
20
|
+
## Security and tenancy
|
|
21
|
+
Enforce tenant filters server-side and test bypass attempts. Protect document-level authorization, field-level restrictions, PII deletion, audit, and snapshot access. Do not rely on a client-supplied tenant filter as an authorization boundary.
|
|
22
|
+
|
|
23
|
+
## Alternatives
|
|
24
|
+
Compare database full-text search, managed search services, Elasticsearch/OpenSearch, and specialized retrieval systems by freshness, relevance control, operational burden, cost, and query complexity.
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# Vector Databases and Hybrid Retrieval
|
|
2
|
+
|
|
3
|
+
## Retrieval requirements
|
|
4
|
+
Define corpus size, embedding dimensions/model, update rate, metadata filters, tenant boundaries, query QPS, latency SLO, recall target, freshness, deletion requirements, and cost. Distinguish semantic retrieval from lexical search and exact database lookup.
|
|
5
|
+
|
|
6
|
+
## Vector index choices
|
|
7
|
+
Compare exact nearest-neighbor search with approximate methods such as HNSW, IVF, and product quantization. Evaluate recall/latency/memory trade-offs, build time, incremental updates, deletes, filtering behavior, and compaction. Measure with representative queries and ground-truth neighbors.
|
|
8
|
+
Distance functions (cosine, dot product, Euclidean) and vector normalization must match the embedding model and index configuration. Do not compare scores across incompatible models or metrics.
|
|
9
|
+
|
|
10
|
+
## Hybrid retrieval pipeline
|
|
11
|
+
1. Normalize and classify the query.
|
|
12
|
+
2. Apply authorization and tenant constraints before retrieval.
|
|
13
|
+
3. Retrieve lexical candidates (BM25) and vector candidates independently.
|
|
14
|
+
4. Fuse rankings (for example, reciprocal rank fusion) with calibrated or learned alternatives.
|
|
15
|
+
5. Rerank a bounded candidate set with a cross-encoder or domain model.
|
|
16
|
+
6. Apply diversity, freshness, and business constraints.
|
|
17
|
+
7. Return citations/provenance and enforce post-retrieval authorization.
|
|
18
|
+
Treat reranking as a latency/cost budget. Cache only with tenant- and permission-aware keys.
|
|
19
|
+
|
|
20
|
+
## RAG correctness and safety
|
|
21
|
+
- Evaluate retrieval separately from generation. Measure Recall@K, MRR/NDCG, context precision/recall, grounded answer rate, citation correctness, and abstention quality.
|
|
22
|
+
- Preserve chunk-to-source lineage, version, ACL, timestamp, and deletion tombstones.
|
|
23
|
+
- Re-check authorization at retrieval and before returning content; revocation and deletion must propagate to vector and lexical indexes.
|
|
24
|
+
- Treat retrieved text as untrusted data. Prompt injection inside documents must not grant tools, permissions, or policy changes.
|
|
25
|
+
- Version embedding models and indexes. Re-embed through a shadow index with measured quality comparison and controlled cutover.
|
|
26
|
+
|
|
27
|
+
## Storage choices
|
|
28
|
+
Compare pgvector, dedicated vector databases, search engines with vector support, and managed retrieval services. Evaluate filtering, consistency, durability, backup/restore, multi-tenancy, observability, ecosystem, and operational skills—not only benchmark throughput.
|
|
29
|
+
|
|
30
|
+
## Operational metrics
|
|
31
|
+
Track ingestion lag, embedding failures, index build/rebuild time, recall under filters, p95/p99 retrieval latency, memory per vector, cache hit rate, delete propagation, and per-tenant cost.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# Database Migrations and Zero-Downtime Schema Evolution
|
|
2
|
+
|
|
3
|
+
## Compatibility-first change
|
|
4
|
+
Use expand-migrate-contract for changes spanning application versions or large datasets. Keep old and new application versions compatible during rolling deployment.
|
|
5
|
+
1. **Expand:** add nullable columns, new tables/indexes, or compatible fields without breaking old readers/writers. Check whether adding a constraint or index scans/rebuilds the table or takes metadata locks on the specific engine/version; use the engine-supported online/concurrent operation only after verifying its failure and retry behavior.
|
|
6
|
+
2. **Deploy compatibility:** code can read old/new representations and write safely under a single ownership strategy.
|
|
7
|
+
3. **Backfill:** process bounded batches with checkpoints, rate limits, idempotency, and observable progress. Make progress monotonic where possible, define how rows changed during the backfill are captured, and ensure a restart cannot overwrite a newer application write with stale data.
|
|
8
|
+
4. **Validate:** compare counts, checksums, nullability, uniqueness, and domain invariants; sample and reconcile differences.
|
|
9
|
+
5. **Switch:** gradually move reads/writes, monitor errors and latency, and preserve a tested rollback route.
|
|
10
|
+
6. **Contract:** remove obsolete fields/indexes only after consumers and rollback windows have expired.
|
|
11
|
+
|
|
12
|
+
## Online DDL risk
|
|
13
|
+
"Online" does not mean lock-free or risk-free. Verify engine/version-specific algorithm and lock behavior, metadata locks, table rebuilds, disk headroom, replication lag, and impact on backups/CDC. Test against production-like data and concurrent traffic.
|
|
14
|
+
|
|
15
|
+
## Backfill engineering
|
|
16
|
+
Use keyset/range pagination rather than large OFFSET scans. Bound batch size, concurrency, transaction duration, and replication impact. Persist checkpoints and make each batch restartable. Avoid uncoordinated dual writes; define an authoritative path and reconciliation.
|
|
17
|
+
|
|
18
|
+
## Cutover and rollback
|
|
19
|
+
Use feature flags, shadow reads, comparison metrics, canaries, and progressive traffic shifts. Define rollback triggers and whether rollback remains possible after writes begin in the new schema. If new code writes data that old code cannot understand, application rollback may require a forward-fix or a separately tested reverse transformation; switching a flag does not restore data compatibility. Defer irreversible contract steps until confidence and recovery evidence meet explicit gates. After contract, treat rollback as a restore/recovery operation unless backward compatibility was deliberately retained.
|
|
20
|
+
|
|
21
|
+
## Validation checklist
|
|
22
|
+
- Schema compatibility across deployed application versions.
|
|
23
|
+
- Query plans and index behavior at expected cardinality.
|
|
24
|
+
- Backfill throughput, lag, retries, and idempotency.
|
|
25
|
+
- Data invariants, counts, checksums, and reconciliation.
|
|
26
|
+
- Lock duration, replication/CDC lag, disk, WAL/binlog, and backup capacity.
|
|
27
|
+
- Cutover, rollback, restore, and post-migration cleanup.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# API Gateway and Edge Architecture
|
|
2
|
+
|
|
3
|
+
## Responsibilities and boundaries
|
|
4
|
+
An API gateway is an edge policy and routing layer, not a replacement for domain services. Define which concerns belong at the edge (TLS termination, routing, authentication token validation, coarse rate limits, request size limits, protocol adaptation, and telemetry) and which remain in services (object-level authorization, business quotas, domain validation, transaction semantics, and authoritative decisions).
|
|
5
|
+
|
|
6
|
+
## Topology choices
|
|
7
|
+
Compare a managed gateway, ingress controller, service-mesh ingress, reverse proxy, and application BFF. Evaluate latency overhead, availability dependencies, policy expressiveness, multi-region behavior, deployment ownership, observability, and cost. Avoid stacking multiple gateways that each retry, buffer, or transform requests without a clear contract.
|
|
8
|
+
|
|
9
|
+
## Routing and resilience
|
|
10
|
+
Specify host/path/header routing, versioned routes, canary weights, health-based endpoint selection, connection draining, deadlines, retry budgets, circuit breakers, and load shedding. Retry only safe or explicitly idempotent operations; enforce one end-to-end deadline rather than multiplying per-hop timeouts. Bound request/response buffering and concurrent connections.
|
|
11
|
+
|
|
12
|
+
## Security
|
|
13
|
+
Terminate TLS with a documented trust model; validate issuer, audience, signature, expiry, and key rotation for tokens. Strip untrusted forwarding headers and reconstruct client identity only from trusted proxies. Apply WAF rules as defense in depth, not as authorization. Prevent request smuggling by consistent HTTP parsing and hop-by-hop header handling. Keep secrets out of logs.
|
|
14
|
+
|
|
15
|
+
## Operations and validation
|
|
16
|
+
Track route-level request rate, status, latency percentiles, upstream saturation, retries, rejected traffic, connection errors, and policy changes. Test gateway failure, upstream brownouts, slow clients, oversized payloads, certificate rotation, and configuration rollback. Gateway availability is part of the critical path; define its SLO and recovery plan.
|
|
17
|
+
|
|
18
|
+
## Decision checklist
|
|
19
|
+
- [ ] Edge responsibilities and service responsibilities are explicit.
|
|
20
|
+
- [ ] Trust proxy chain and client-IP derivation are documented.
|
|
21
|
+
- [ ] Deadlines, retries, payload limits, and connection limits are bounded.
|
|
22
|
+
- [ ] Route changes support staged rollout and fast rollback.
|
|
23
|
+
- [ ] Gateway failure and regional failover have been exercised.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# REST API Design and Lifecycle
|
|
2
|
+
|
|
3
|
+
## Resource and contract design
|
|
4
|
+
Model stable resources and relationships; use HTTP methods and status codes consistently. Define canonical identifiers, filtering, sorting, pagination, partial updates, field masks, and error envelopes. Prefer cursor pagination for mutable or large datasets; document cursor stability and expiry. Avoid leaking storage schemas as public contracts.
|
|
5
|
+
|
|
6
|
+
## Correctness
|
|
7
|
+
Use conditional requests (ETag/If-Match) or explicit versions to prevent lost updates. Define idempotency-key scope, retention, payload binding, concurrent duplicate behavior, and replay response semantics for retryable creates or payments. A client timeout leaves the outcome unknown: provide a way to query operation status or safely retry with the same key.
|
|
8
|
+
|
|
9
|
+
## Compatibility and evolution
|
|
10
|
+
Treat additive changes as potentially breaking when clients use strict decoders or exhaustive enums. Version only when compatibility policy requires it; define deprecation notice, telemetry, support window, sunset criteria, and owner. Maintain consumer contract tests and generated-client compatibility checks. Document stable versus experimental fields.
|
|
11
|
+
|
|
12
|
+
## Security and usability
|
|
13
|
+
Specify authentication, object- and function-level authorization, tenant scoping, input validation, mass-assignment defenses, rate limits, and data classification. Use consistent machine-readable errors with correlation IDs, but never expose stack traces or secrets. Define CORS deliberately; it is not an authorization control.
|
|
14
|
+
|
|
15
|
+
## Operations
|
|
16
|
+
Publish latency and availability objectives per critical operation. Set request/response limits, pagination caps, deadlines, and retry guidance. Instrument business outcomes as well as HTTP status. Test concurrent updates, duplicate requests, stale ETags, malformed inputs, and old-client compatibility.
|
|
17
|
+
|
|
18
|
+
## Review checklist
|
|
19
|
+
- [ ] OpenAPI or equivalent contract is version-controlled and validated.
|
|
20
|
+
- [ ] Idempotency and concurrency semantics are explicit.
|
|
21
|
+
- [ ] Pagination, errors, and filtering are bounded and deterministic.
|
|
22
|
+
- [ ] AuthZ is enforced at object and function level.
|
|
23
|
+
- [ ] Deprecation and compatibility are observable and owned.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# GraphQL Architecture and Governance
|
|
2
|
+
|
|
3
|
+
## Schema and ownership
|
|
4
|
+
Design the graph around domain capabilities and consumer needs. Define field ownership, resolver ownership, nullability semantics, deprecation policy, schema composition, and federation boundaries. Avoid a graph that becomes a shared database facade with unclear authorization and change ownership.
|
|
5
|
+
|
|
6
|
+
## Query execution controls
|
|
7
|
+
GraphQL moves query shape to clients; it does not remove backend cost. Apply depth, complexity, alias, batch, and response-size limits. Use persisted/allowlisted operations for high-risk or mobile clients where appropriate. Prevent N+1 access with request-scoped DataLoader-style batching while preserving tenant and authorization context. Bound resolver concurrency and downstream fan-out.
|
|
8
|
+
|
|
9
|
+
## Authorization and data protection
|
|
10
|
+
Enforce authorization in resolvers or domain services for every object and field; never rely solely on hiding schema fields. Prevent cross-tenant cache leakage by including principal/tenant and policy context in cache keys. Restrict introspection according to threat model, but do not treat disabling it as a substitute for access control. Redact sensitive fields from traces and error payloads.
|
|
11
|
+
|
|
12
|
+
## Federation and consistency
|
|
13
|
+
For federated graphs, define entity ownership, key stability, reference resolution, timeout budgets, partial failure semantics, and schema rollout ordering. Avoid cyclic dependencies and excessive cross-subgraph waterfalls. Treat composed schema checks as a release gate; test old and new subgraph combinations.
|
|
14
|
+
|
|
15
|
+
## Evaluation
|
|
16
|
+
Measure operation-level latency, resolver fan-out, cache hit ratio, downstream calls, error rate, and query-cost rejection. Test adversarial deep/aliased queries, partial data errors, null propagation, authorization bypass, and schema compatibility. Compare GraphQL with REST/BFF for actual client needs rather than assuming one is universally superior.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# gRPC and Service Contracts
|
|
2
|
+
|
|
3
|
+
## Protocol fit
|
|
4
|
+
Use gRPC when typed contracts, efficient binary transport, streaming, generated clients, or polyglot service communication justify the operational and browser-compatibility costs. Compare REST/JSON, gRPC, and asynchronous messaging by consumer environment, debugging, proxy support, latency, and evolution requirements.
|
|
5
|
+
|
|
6
|
+
## Protobuf evolution
|
|
7
|
+
Treat field numbers as permanent identifiers: never reuse deleted numbers or names; reserve them. Add fields compatibly, define defaults and presence semantics, and avoid changing field meaning or wire type. Enums need an unspecified zero value and tolerant handling of unknown values. Validate compatibility in CI across supported producer/consumer versions.
|
|
8
|
+
|
|
9
|
+
## RPC semantics
|
|
10
|
+
Define deadlines, cancellation propagation, status codes, retryable conditions, idempotency, metadata limits, message size limits, and streaming flow control. A transport retry can duplicate a side effect; retries must respect method semantics and an end-to-end budget. Distinguish application errors from transport failures.
|
|
11
|
+
|
|
12
|
+
## Security and operations
|
|
13
|
+
Use workload identity and mTLS where appropriate; authorize every method and resource. Protect reflection and health endpoints according to exposure. Instrument method-level latency, status, message sizes, active streams, cancellations, and retry attempts. Test rolling upgrades, proxy/load-balancer behavior, slow consumers, stream resets, and mixed-version compatibility.
|
|
14
|
+
|
|
15
|
+
## Streaming
|
|
16
|
+
For client/server or bidirectional streams, define message ordering, backpressure, heartbeats, idle timeout, reconnect/resume tokens, replay boundaries, and maximum stream duration. Do not assume a broken stream means the server did not process the last message.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# WebSocket, Server-Sent Events, and Real-Time Systems
|
|
2
|
+
|
|
3
|
+
## Transport selection
|
|
4
|
+
Choose request/response for ordinary interactions, Server-Sent Events (SSE) for one-way server-to-client updates, and WebSocket for bidirectional low-latency sessions. Consider WebRTC for peer media/data only when its NAT traversal and media semantics are needed. Evaluate browser/proxy compatibility, connection duration, fan-out, and operational complexity.
|
|
5
|
+
|
|
6
|
+
## Connection architecture
|
|
7
|
+
Model connection establishment, authentication, authorization, heartbeat, idle timeout, reconnect, session affinity, regional routing, and graceful drain. Keep connection state disposable where possible; store durable business state outside gateway memory. Use pub/sub or partitioned brokers to distribute events to connection-owning nodes, with explicit delivery and ordering guarantees.
|
|
8
|
+
|
|
9
|
+
## Backpressure and scale
|
|
10
|
+
Bound per-connection outbound queues, message size, fan-out, subscriptions, and total concurrent connections. Define slow-consumer policy (drop, coalesce, disconnect, or durable replay) by message class. Capacity-plan file descriptors, memory per connection, heartbeat traffic, TLS, broker throughput, and reconnect storms.
|
|
11
|
+
|
|
12
|
+
## Delivery semantics
|
|
13
|
+
Treat live transports as best-effort unless backed by durable offsets or application acknowledgements. Include event IDs, sequence numbers, resume cursors, deduplication, and snapshot-plus-delta recovery when missed updates matter. Define whether ordering is per entity, room, partition, or connection; avoid claiming global ordering without a mechanism.
|
|
14
|
+
|
|
15
|
+
## Security and reliability
|
|
16
|
+
Authorize subscriptions and every sensitive message; revalidate long-lived sessions and revoke access after permission changes. Validate Origin for browser WebSockets, protect against cross-site WebSocket hijacking, enforce quotas, and avoid logging payloads containing PII. Test node loss, broker partitions, reconnect storms, slow consumers, and regional failover.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Real-Time Collaboration: OT and CRDT
|
|
2
|
+
|
|
3
|
+
## Problem framing
|
|
4
|
+
Define collaboration semantics before choosing an algorithm: document model, concurrent edit types, offline duration, expected participants, latency, conflict tolerance, undo/redo, presence, and persistence. Presence and cursors are ephemeral; document operations and snapshots are durable domain state.
|
|
5
|
+
|
|
6
|
+
## Operational Transformation
|
|
7
|
+
OT transforms concurrent operations against one another and depends on a well-defined transformation function, ordering authority, and server/client protocol. Specify revision numbers, operation identity, acknowledgements, reconnect catch-up, and transformation correctness. Test convergence across permutations and stale clients.
|
|
8
|
+
|
|
9
|
+
## Conflict-free Replicated Data Types
|
|
10
|
+
CRDTs encode merge behavior so replicas converge under stated assumptions. Choose a data type whose semantics match the product; convergence does not guarantee intuitive user intent, bounded metadata, or low storage cost. Model tombstones/garbage collection, causal metadata, offline replicas, compaction, and schema evolution. State whether the design is state-based or operation-based and its delivery assumptions.
|
|
11
|
+
|
|
12
|
+
## Persistence and security
|
|
13
|
+
Persist operations or snapshots with atomic revision/checkpoint boundaries. Make operation application idempotent and validate authorization at the document/operation boundary. Enforce tenant isolation, document ACL changes, retention, export, and deletion across replicas and backups. Do not trust client-generated operations.
|
|
14
|
+
|
|
15
|
+
## Testing and trade-offs
|
|
16
|
+
Use property-based and randomized concurrency tests, network partitions, duplicate/reordered delivery, offline edits, and mixed-version clients. Measure convergence time, metadata growth, sync bandwidth, conflict UX, and recovery. Choose centralized sequencing when it is simpler and meets offline/latency requirements; CRDT is not a default requirement for every collaborative feature.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# API Security, Rate Limiting, and Abuse Prevention
|
|
2
|
+
|
|
3
|
+
## Threat model
|
|
4
|
+
Cover broken object/function authorization, credential stuffing, scraping, enumeration, replay, resource exhaustion, injection, SSRF, request smuggling, webhook spoofing, and tenant escape. Map controls to principals, endpoints, resources, and business impact. Authentication alone does not establish permission to access an object.
|
|
5
|
+
|
|
6
|
+
## Rate-limit dimensions
|
|
7
|
+
Apply limits at appropriate layers: global protection, per-IP or network, per-credential, per-user, per-tenant, per-route, and expensive-operation budgets. Combine request rate with concurrency, cost units, payload size, and burst allowance. Avoid IP-only controls for NAT/mobile users and avoid user-only controls that permit anonymous abuse.
|
|
8
|
+
|
|
9
|
+
## Algorithms and distributed enforcement
|
|
10
|
+
Compare token bucket, leaky bucket, fixed window, and sliding window. Document burst semantics, clock assumptions, key cardinality, storage consistency, and failure mode. Distributed counters may be approximate; state whether enforcement is best-effort or strict. Use local emergency limits when the shared limiter is unavailable, and avoid a limiter dependency that becomes a single point of failure.
|
|
11
|
+
|
|
12
|
+
## Fairness and degradation
|
|
13
|
+
Define tenant tiers, weighted fairness, priority classes, retry-after behavior, and quota reset semantics. Return actionable 429/503 responses without leaking internal policy. Apply load shedding and queue admission controls to protect critical traffic. Avoid retry storms by communicating backoff and enforcing client retry budgets.
|
|
14
|
+
|
|
15
|
+
## Abuse operations
|
|
16
|
+
Monitor rejected requests, credential anomalies, expensive query patterns, cost per tenant, and false positives. Build alerting, investigation, appeal/override, and emergency block workflows. Test bypass attempts, distributed bursts, hot keys, limiter outage, clock skew, and noisy-neighbor isolation. Rate limiting complements authorization, WAF, and business fraud controls; it does not replace them.
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
# OAuth 2.1, OpenID Connect, and Passkeys
|
|
2
|
+
|
|
3
|
+
## Protocol roles
|
|
4
|
+
OAuth is an authorization framework; OpenID Connect adds an identity layer. Distinguish the authorization server, resource server, client, subject, and relying party. Use Authorization Code with PKCE for interactive public clients; do not embed client secrets in mobile or browser applications. Avoid implicit and resource-owner-password flows. Verify current standards and provider-specific behavior before deployment.
|
|
5
|
+
|
|
6
|
+
## Token validation and lifecycle
|
|
7
|
+
First determine whether the access token is a JWT or opaque token; validate JWTs locally only under the issuer contract, and use the provider-supported introspection path for opaque tokens. For JWTs, pin an allowed algorithm set from trusted configuration (never accept an algorithm selected solely by the token), resolve keys only from trusted issuer metadata/JWKS, handle key rotation and unknown `kid` safely, and validate signature, exact issuer, intended audience, expiry, not-before, and `azp` when required by the OIDC profile. For ID tokens, also validate the authentication request binding (`nonce` when sent) and the applicable audience/authorized-party rules. Under OpenID Connect Core, `aud` must include this relying party's `client_id`; when `aud` contains multiple values, the client SHOULD require `azp` to be present; whenever `azp` is present, it MUST identify this client. Do not make `azp` a universal mandatory claim for every single-audience ID token unless the applicable profile or provider contract requires it. Apply the exact rules from the [OpenID Connect Core ID Token specification](https://openid.net/specs/openid-connect-core-1_0.html). Access tokens should be audience-restricted and short-lived according to risk. Define refresh-token rotation, reuse detection, revocation, session termination, and key rollover. Do not treat an ID token as an API access token or assume every access token is a JWT.
|
|
8
|
+
|
|
9
|
+
## Redirects and browser security
|
|
10
|
+
Use exact registered redirect URIs, state binding, nonce for OIDC authentication, PKCE with `S256` for public clients, and strict callback handling. Bind the authorization response to the initiating browser session; validate `state` even when PKCE is used, and validate `nonce` when an OIDC authentication request included one. Protect cookies with Secure, HttpOnly, and appropriate SameSite attributes. Defend against open redirects, login CSRF, token leakage through URLs, and unsafe post-logout redirects.
|
|
11
|
+
|
|
12
|
+
## Passkeys and WebAuthn
|
|
13
|
+
Passkeys use public-key credentials scoped to a relying party. Verify challenge freshness and one-time use, exact expected origin, RP ID hash, user presence/verification policy, credential ownership, and signature counter behavior according to authenticator capabilities. Do not treat a non-increasing signature counter as a universal clone detector: synced passkeys and authenticators differ, so apply the WebAuthn guidance for the authenticator model and use risk signals rather than rejecting every zero or unchanged counter. Design recovery, device replacement, account linking, and phishing-resistant step-up flows; passkeys do not eliminate account-recovery risk.
|
|
14
|
+
|
|
15
|
+
## Review checklist
|
|
16
|
+
- [ ] OAuth and OIDC responsibilities are separated correctly.
|
|
17
|
+
- [ ] PKCE, state, nonce, exact redirect URI, issuer and audience checks are enforced.
|
|
18
|
+
- [ ] Refresh rotation, revocation, and key rotation are tested.
|
|
19
|
+
- [ ] Passkey registration, assertion, origin, challenge, and recovery flows are tested.
|
|
20
|
+
- [ ] Tokens and authentication codes never enter logs, analytics, or referrers.
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
# Identity and Access Management: RBAC, ABAC, and ReBAC
|
|
2
|
+
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns authorization policy semantics: principal/resource/action evaluation, RBAC/ABAC/ReBAC, tenant and object checks, deny precedence, delegation, and revocation freshness. Use [agent security](19-agent-security-and-permissions.md) for model/tool-specific threat controls; [workload identity](50-zero-trust-and-workload-identity.md) for machine identity and mutual authentication; and [API security](47-api-security-rate-limiting-and-abuse.md) for request abuse controls. Authentication, network location, gateway admission, and cached policy results are inputs or enforcement layers—not substitutes for authoritative resource authorization.
|
|
6
|
+
|
|
7
|
+
## Authorization model selection
|
|
8
|
+
RBAC assigns permissions through roles and suits stable organizational responsibilities. ABAC evaluates subject, resource, action, and environment attributes. ReBAC evaluates relationships such as owner, member, parent, and delegated access. Real systems may combine models, but policy composition and precedence must be explicit.
|
|
9
|
+
|
|
10
|
+
## Policy design
|
|
11
|
+
Define principal types, resource hierarchy, actions, scopes, tenant boundaries, deny/allow precedence, default-deny behavior, and delegation. Specify whether a deny overrides an allow, how inherited grants interact with explicit denies, and which service is authoritative. Separate authentication from authorization and coarse gateway checks from authoritative domain checks. Enforce object-level authorization on every read, write, export, bulk operation, and background job. Derive tenant and principal context from authenticated server-side identity, not a caller-controlled tenant header or object identifier.
|
|
12
|
+
|
|
13
|
+
## Policy evaluation
|
|
14
|
+
Centralized policy engines improve consistency but add latency and availability dependencies. Define policy distribution, cache TTL, invalidation, fail-open/fail-closed behavior, versioning, and explainability. Cache keys must include principal, tenant, resource, action, relevant attributes, and policy version. Define a revocation freshness objective and an invalidation strategy; a short TTL alone does not guarantee immediate revocation. For high-impact actions, consider authoritative re-checks or deny-list/version checks. Fail closed for sensitive authorization decisions unless a documented, bounded emergency policy says otherwise.
|
|
15
|
+
|
|
16
|
+
## Lifecycle and governance
|
|
17
|
+
Model joiner/mover/leaver workflows, periodic access reviews, privileged access, service accounts, break-glass access, separation of duties, and audit evidence. Prefer time-bound grants and just-in-time elevation. Reconcile identity-provider groups with application permissions; group membership alone is not a complete authorization decision.
|
|
18
|
+
|
|
19
|
+
## Testing
|
|
20
|
+
Test horizontal and vertical privilege escalation, cross-tenant access, caller-supplied tenant spoofing, inherited permissions, explicit-deny precedence, revoked access during cache TTL, policy-engine outage, stale caches, bulk/export endpoints, and background jobs executing under the wrong identity. Use policy unit tests and representative end-to-end authorization matrices.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Zero Trust and Workload Identity
|
|
2
|
+
|
|
3
|
+
## Trust model
|
|
4
|
+
Zero Trust removes implicit trust based on network location. Authenticate and authorize each workload-to-workload interaction using explicit identity, resource, action, and context. Network segmentation remains useful defense in depth, not proof of identity.
|
|
5
|
+
|
|
6
|
+
## Workload identity
|
|
7
|
+
Prefer short-lived, automatically rotated identities bound to workload properties and environment. Use SPIFFE/SPIRE or cloud-native workload identity where appropriate; compare federation, operational maturity, trust-domain boundaries, and portability. Avoid long-lived static service credentials and shared identities that erase attribution.
|
|
8
|
+
|
|
9
|
+
## Service-to-service controls
|
|
10
|
+
Use mTLS for peer authentication and transport protection where suitable, but still authorize methods and resources. Define certificate issuance, trust bundles, rotation overlap, revocation, identity mapping, and failure behavior. Prevent confused-deputy behavior by constraining audience and delegated authority.
|
|
11
|
+
|
|
12
|
+
## Policy and segmentation
|
|
13
|
+
Apply least privilege at service, method, data, and administrative boundaries. Combine identity-aware policy with network policy, egress restrictions, and workload isolation. Define policy propagation and emergency revocation latency.
|
|
14
|
+
|
|
15
|
+
## Operations
|
|
16
|
+
Measure authentication failures, certificate expiry, identity issuance latency, policy denials, and trust-bundle drift. Test clock skew, CA outage, certificate rotation, compromised workload containment, failover, and recovery. Document bootstrap trust and break-glass procedures.
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
# Secrets Management, KMS, PKI, and Key Management
|
|
2
|
+
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns secret, cryptographic key, certificate authority, KMS/HSM, rotation, revocation, and key-recovery lifecycle. Use [workload identity](50-zero-trust-and-workload-identity.md) for workload identity issuance and federation; [supply-chain security](52-supply-chain-security-sbom-slsa.md) for build provenance and artifact signing; and [storage infrastructure](67-storage-infrastructure-and-storage-systems.md) for storage encryption and backup/restore integration. Rotating a key does not prove historical ciphertext was re-encrypted, old credentials were revoked, or encrypted backups remain recoverable.
|
|
6
|
+
|
|
7
|
+
## Secret lifecycle
|
|
8
|
+
Inventory secrets, owners, consumers, environments, purpose, classification, and expiry. Prefer workload identity and dynamic credentials over static secrets. Centralize issuance, access policy, rotation, revocation, and audit; avoid distributing a universal root secret to every service.
|
|
9
|
+
|
|
10
|
+
## Cryptographic key hierarchy
|
|
11
|
+
Separate data-encryption keys from key-encryption keys. Use envelope encryption and managed KMS/HSM capabilities where appropriate. Define key purpose, algorithm, allowed operations, region, rotation, backup, recovery, deletion, and separation of duties. Key rotation does not automatically re-encrypt all historical data.
|
|
12
|
+
|
|
13
|
+
## PKI operations
|
|
14
|
+
Define root and intermediate CA custody, issuance policy, certificate identity, validity, renewal, revocation, trust distribution, and emergency rollover. Protect CA signing keys with stronger controls than ordinary application secrets. Plan overlapping trust during rotations and verify client behavior under expired or unknown certificates.
|
|
15
|
+
|
|
16
|
+
## Application integration
|
|
17
|
+
Avoid secrets in source, images, build logs, crash dumps, environment diagnostics, and telemetry. Use narrowly scoped runtime retrieval with bounded caching and fail-safe behavior. Do not log secret values when reporting access failures.
|
|
18
|
+
|
|
19
|
+
## Resilience and audit
|
|
20
|
+
KMS and secret stores are critical dependencies: model quotas, regional availability, latency, caching, outage behavior, and recovery. Test restore and key-loss scenarios before relying on encrypted backups. Audit key use, policy changes, export, deletion, and break-glass operations.
|
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
# Software Supply Chain Security: SBOM, Provenance, and SLSA
|
|
2
|
+
|
|
3
|
+
## Threat model
|
|
4
|
+
Cover dependency confusion, typosquatting, compromised maintainers, malicious build steps, poisoned artifacts, leaked signing keys, untrusted registries, and release pipeline compromise. Map trust boundaries from source and dependencies through build, signing, registry, deployment, and runtime.
|
|
5
|
+
|
|
6
|
+
## Dependency governance
|
|
7
|
+
Pin and review dependencies using lockfiles, trusted registries, provenance where available, vulnerability and license scanning, and controlled update automation. Pinning alone does not establish trust. Define exceptions, remediation SLAs, ownership, and reachability-based prioritization.
|
|
8
|
+
|
|
9
|
+
## SBOM and provenance
|
|
10
|
+
Generate machine-readable SBOMs for shipped artifacts and preserve component identity and version. Attest build provenance with authenticated builder identity, source revision, build parameters, and artifact digest. Verify attestations at promotion/deployment; do not trust unsigned metadata supplied beside an artifact.
|
|
11
|
+
|
|
12
|
+
## Build integrity
|
|
13
|
+
Use isolated ephemeral builders, least-privilege tokens, protected branches, reviewed workflows, reproducible or independently verifiable builds where feasible, and separation between build and release authority. Protect signing keys and enforce artifact immutability.
|
|
14
|
+
|
|
15
|
+
## Release and response
|
|
16
|
+
Define artifact signing, verification, promotion, rollback, and incident procedures for compromised dependencies or builders. Test provenance verification and quarantine paths. Treat SLSA levels as a framework for incremental assurance, not a blanket security certification.
|
|
17
|
+
|
|
18
|
+
## Repository build provenance
|
|
19
|
+
|
|
20
|
+
On pushes to `main`, after the Linux validation job builds and verifies the deterministic ZIP, a separate least-privilege job generates a GitHub SLSA provenance attestation for the ZIP. Pull-request runs do not receive attestation-writing permissions at job execution time and do not publish provenance. The attestation job uses GitHub OIDC and the GitHub attestation service; it does not use a long-lived signing key.
|
|
21
|
+
|
|
22
|
+
Verify a downloaded ZIP against the exact main-branch commit that produced it:
|
|
23
|
+
|
|
24
|
+
```sh
|
|
25
|
+
python scripts/verify_build_provenance.py dist/system-design-primer-v2.3.0.zip \
|
|
26
|
+
--repo jack-001-a/system-design-primer-skill \
|
|
27
|
+
--signer-workflow jack-001-a/system-design-primer-skill/.github/workflows/ci.yml \
|
|
28
|
+
--source-ref refs/heads/main \
|
|
29
|
+
--source-digest <full-40-character-source-commit-sha>
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
The verifier delegates cryptographic signature, certificate, transparency-log, and trusted-root validation to the installed GitHub CLI's `gh attestation verify` command. It additionally requires the repository, signer workflow, source ref, source commit, GitHub Actions OIDC issuer, SLSA provenance v1 predicate, and GitHub-hosted runner policy. Install and authenticate a current GitHub CLI before running it. A successful result proves the artifact is covered by a GitHub-verified attestation matching that policy; it does not prove the build was bug-free, reproducible, or free from compromise inside the authorized workflow.
|
|
33
|
+
|
|
34
|
+
|
|
35
|
+
### Verifier version and subject policy
|
|
36
|
+
|
|
37
|
+
The repository verifier requires GitHub CLI **2.102.0 or newer**. Versions through 2.101.0 had security advisories affecting case-sensitive source-ref matching and exact signer-workflow identity matching; do not use those versions for provenance authorization. The wrapper also checks the verified in-toto statement's predicate type and SHA-256 subject against a private snapshot of the exact artifact bytes passed to the CLI. This supplements, but does not replace, the CLI's cryptographic verification.
|
|
38
|
+
|
|
39
|
+
|
|
40
|
+
## Reproducible and offline verification
|
|
41
|
+
|
|
42
|
+
Download the artifact's attestation bundle while online:
|
|
43
|
+
|
|
44
|
+
```sh
|
|
45
|
+
gh attestation download dist/system-design-primer-v2.3.0.zip \
|
|
46
|
+
--repo jack-001-a/system-design-primer-skill \
|
|
47
|
+
--predicate-type https://slsa.dev/provenance/v1
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
The CLI writes a digest-named JSONL bundle. Preserve it alongside the artifact. The verifier's offline mode requires all three of `--bundle`, `--trusted-root`, and `--repository-id`; it fails closed if any is missing. Obtain a Sigstore trusted-root JSONL through `gh attestation trusted-root` and authenticate it through an independent trusted channel:
|
|
51
|
+
|
|
52
|
+
```sh
|
|
53
|
+
python scripts/verify_build_provenance.py dist/system-design-primer-v2.3.0.zip \
|
|
54
|
+
--repo jack-001-a/system-design-primer-skill \
|
|
55
|
+
--signer-workflow jack-001-a/system-design-primer-skill/.github/workflows/ci.yml \
|
|
56
|
+
--source-ref refs/heads/main \
|
|
57
|
+
--source-digest <full-40-character-source-commit-sha> \
|
|
58
|
+
--bundle sha256:<artifact-digest>.jsonl \
|
|
59
|
+
--trusted-root trusted_root.jsonl \
|
|
60
|
+
--repository-id 1462786926
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Obtain the expected numeric repository ID online from the canonical repository using `gh api repos/jack-001-a/system-design-primer-skill --jq .id`, and pass it with `--repository-id`. The verifier compares it with `verificationResult.signature.certificate.sourceRepositoryIdentifier` for every verified statement. Never trust a bundle or trusted-root file merely because it is adjacent to the artifact. A bundle is evidence to verify, not a trust anchor; the trusted root must be authenticated independently. Offline verification cannot learn about signing-key or certificate revocations published after the trusted-root snapshot was obtained. Refresh and independently authenticate the trusted root whenever importing new signed material, and understand that a stale root may continue to accept signatures made with subsequently revoked key material. GitHub CLI's name check alone cannot detect subsequent repository renames or name reuse; the additional repository-ID check above protects this offline path. If the repository has been renamed, pass the owner/repository name recorded in the signing certificate to `--repo`, while keeping the stable numeric ID from the canonical repository. Use online API-backed verification when current repository-name ownership matters. The bundle download command is currently public preview.
|
|
64
|
+
|
|
65
|
+
### Attestation trust boundary and stronger isolation
|
|
66
|
+
|
|
67
|
+
The current `.github/workflows/ci.yml` design provides **job-level privilege separation**, not an independent trusted-builder boundary:
|
|
68
|
+
|
|
69
|
+
- The attestation job runs only for a push to `refs/heads/main`; pull-request runs do not receive attestation-writing permissions.
|
|
70
|
+
- The job runs on a fresh runner, downloads the Linux validation job's ZIP artifact, and invokes GitHub's attestation action. It does not check out the source tree or execute project build scripts.
|
|
71
|
+
- Its token permissions are scoped to that job: `contents: read`, `id-token: write`, `attestations: write`, and `artifact-metadata: write`. The workflow default remains `contents: read`.
|
|
72
|
+
- The downloaded artifact is still produced by code and workflow definitions from the same repository revision. A compromised or malicious build on the trusted main branch can therefore produce a malicious ZIP that the attestation job will faithfully attest. Provenance authenticates the builder and its declared inputs; it is not an independent audit of the artifact's behavior.
|
|
73
|
+
|
|
74
|
+
This model depends on protecting `main` and reviewing changes to `.github/workflows/**`, build scripts, packaging logic, and dependency inputs. Configure branch protection or rulesets outside the workflow to require review and required checks; consider a CODEOWNERS rule for workflow and release files, and ensure the rule is actually enforced by repository settings. Do not treat a CODEOWNERS file alone as enforcement.
|
|
75
|
+
|
|
76
|
+
For a stronger trust boundary, move the privileged build/sign operation into a separately administered trusted-builder repository. Pin calls to that reusable workflow by a full commit SHA, restrict which caller repositories and source refs it accepts, and grant only the permissions required by the signing job. Keep caller-controlled inputs narrow and data-only (for example, an artifact identifier, expected digest, source repository ID, and source commit); validate each against the run context before signing. Never accept caller-supplied shell commands, scripts, arbitrary workflow paths, or arbitrary artifact URLs, and do not use `secrets: inherit`. Transfer artifacts through a mechanism whose run and repository identity can be verified, then independently verify the exact artifact digest before attestation. Protect the builder repository and its release process at least as strongly as the artifacts it signs.
|
|
77
|
+
|
|
78
|
+
A reusable workflow in the same repository can reduce duplication and centralize policy, but it does not by itself create an independent administrative trust boundary. SHA pinning protects the called workflow from mutable refs; it does not protect against an authorized change to that pinned workflow's source repository or against unsafe caller-controlled inputs. See GitHub's guidance on [reusable workflows](https://docs.github.com/en/actions/how-tos/reuse-automations/reuse-workflows), [secure use](https://docs.github.com/en/actions/reference/security/secure-use), and [workflow dependency lockfiles](https://docs.github.com/en/actions/concepts/security/workflow-dependency-lockfiles).
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Threat Modeling: STRIDE, Attack Trees, and Risk Analysis
|
|
2
|
+
|
|
3
|
+
## Scope and assets
|
|
4
|
+
Record system boundaries, data flows, stores, actors, trust boundaries, entry points, sensitive assets, and security objectives. Include control planes, CI/CD, support tooling, third parties, and operational access—not only runtime APIs.
|
|
5
|
+
|
|
6
|
+
## Analysis methods
|
|
7
|
+
Use STRIDE to systematically identify spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege. Use attack trees to decompose attacker goals into feasible paths. Supplement with abuse cases, misuse cases, and privacy harms. No single taxonomy replaces domain expertise.
|
|
8
|
+
|
|
9
|
+
## Risk prioritization
|
|
10
|
+
Describe each scenario as actor, preconditions, attack path, affected asset, impact, likelihood rationale, existing controls, and residual risk. Use a consistent scoring model and distinguish technical severity from business risk. Avoid false precision in numeric scores.
|
|
11
|
+
|
|
12
|
+
## Mitigation and verification
|
|
13
|
+
Prefer eliminating or reducing attack surface, then enforce preventive controls, detection, and recovery. Assign each mitigation an owner, due date, and verification test. Track accepted risks with an approver and expiry/review date.
|
|
14
|
+
|
|
15
|
+
## Continuous lifecycle
|
|
16
|
+
Revisit threat models when architecture, trust boundaries, data classes, dependencies, deployment, or attacker capabilities change. Validate assumptions with security tests, tabletop exercises, and incident learnings.
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
# Privacy Engineering and Data Governance
|
|
2
|
+
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns data-purpose governance, classification, minimization, retention, subject rights, deletion propagation, residency, and privacy risk. Use [agent state and memory](20-agent-state-and-memory.md) for agent-specific state categories and checkpoints; [OpenTelemetry](55-opentelemetry-and-instrumentation.md) and [distributed tracing](56-distributed-tracing-and-context-propagation.md) for telemetry-specific controls; and [storage infrastructure](67-storage-infrastructure-and-storage-systems.md) for physical durability, backup, and storage deletion mechanisms. A deletion workflow is complete only against a documented inventory of primary, derived, cached, replicated, exported, and backup copies, with exceptions and expiry behavior stated.
|
|
6
|
+
|
|
7
|
+
## Data inventory and purpose
|
|
8
|
+
Map personal and sensitive data across collection, inference, processing, sharing, storage, logs, analytics, backups, and deletion. Document purpose, lawful basis where applicable, data owner, recipients, retention, residency, and user expectations. Minimize collection and avoid retaining data merely because storage is cheap.
|
|
9
|
+
|
|
10
|
+
## Privacy by design
|
|
11
|
+
Use purpose limitation, minimization, transparency, access controls, pseudonymization, encryption, and privacy-preserving defaults. Assess re-identification risk in supposedly de-identified datasets and telemetry. Treat derived attributes and embeddings as potentially sensitive when they can reveal or link information.
|
|
12
|
+
|
|
13
|
+
## Lifecycle controls
|
|
14
|
+
Define retention schedules, legal holds, correction, export, deletion, backup expiry, replica propagation, and downstream processor obligations. A deletion request is incomplete if data remains searchable in caches, indexes, analytics, or derived stores without a documented policy.
|
|
15
|
+
|
|
16
|
+
## Governance and accountability
|
|
17
|
+
Assign data stewards, classification, quality rules, lineage, access approval, sharing agreements, and incident ownership. Apply residency and cross-border transfer controls based on applicable law and contractual commitments; obtain qualified legal review rather than inferring legal compliance from technical controls.
|
|
18
|
+
|
|
19
|
+
## Verification
|
|
20
|
+
Test data discovery, access logs, retention enforcement, deletion propagation, backup expiration, subject-access workflows, and least-privilege analytics. Measure policy exceptions and stale data inventories.
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# OpenTelemetry Architecture and Instrumentation
|
|
2
|
+
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns OpenTelemetry APIs/SDKs, semantic conventions, collectors, processors, exporters, and telemetry pipeline operations. Use [distributed tracing](56-distributed-tracing-and-context-propagation.md) for trace context, baggage, propagation, and sampling semantics; [production observability](27-production-observability-and-slo.md) for SLIs/SLOs and service-level interpretation; and [privacy engineering](54-privacy-engineering-and-data-governance.md) for data classification and retention policy. Instrumentation can be dropped, sampled, delayed, or misconfigured; absence of telemetry is not evidence that an operation did not occur.
|
|
6
|
+
|
|
7
|
+
## Purpose and scope
|
|
8
|
+
OpenTelemetry (OTel) provides vendor-neutral APIs, SDKs, semantic conventions, and telemetry pipelines for traces, metrics, and logs. Treat it as an instrumentation and data-delivery architecture, not as an observability product or a guarantee that telemetry is correct.
|
|
9
|
+
|
|
10
|
+
## Architecture decisions
|
|
11
|
+
- Define signal ownership and collection paths: application SDK/auto-instrumentation → OTLP → Collector gateway/agent → processors/exporters → backend.
|
|
12
|
+
- Choose agent, sidecar, gateway, or direct export based on deployment model, network boundaries, buffering needs, tenancy, and operational ownership.
|
|
13
|
+
- Prefer OTLP and stable semantic conventions where supported. Pin SDK, Collector, and instrumentation versions; verify stability status before adopting conventions.
|
|
14
|
+
- Separate application instrumentation from vendor-specific exporters to preserve portability.
|
|
15
|
+
|
|
16
|
+
## Instrumentation design
|
|
17
|
+
- Instrument user journeys and service boundaries; add spans around meaningful operations, not every function.
|
|
18
|
+
- Record HTTP/RPC/database/messaging operations with consistent service.name, service.version, deployment.environment, and resource attributes.
|
|
19
|
+
- Propagate W3C Trace Context across synchronous and asynchronous boundaries. Define behavior for queues, batch jobs, scheduled work, and context loss.
|
|
20
|
+
- Use span events and links for meaningful milestones and causal relationships; do not model every event as a child span.
|
|
21
|
+
- Use histograms for latency distributions; counters for monotonic event totals; gauges/observable instruments for sampled state. Avoid high-cardinality labels.
|
|
22
|
+
- Correlate logs with trace_id and span_id where available. Keep logs useful when no trace exists.
|
|
23
|
+
|
|
24
|
+
## Collector pipeline
|
|
25
|
+
- Define receiver protocols, authentication, TLS/mTLS, tenant routing, memory limits, batching, queueing, retry, and exporter timeouts.
|
|
26
|
+
- Apply filtering, attribute normalization, redaction, sampling, and routing deliberately. Processors must not silently remove fields needed for incident response.
|
|
27
|
+
- Size queues and memory limiters for bursts; monitor dropped data, export failures, queue occupancy, and retry exhaustion.
|
|
28
|
+
- Protect the Collector as a production service: capacity, HA, upgrades, configuration rollout, and failure isolation matter.
|
|
29
|
+
|
|
30
|
+
## Sampling and data governance
|
|
31
|
+
- Use head sampling for inexpensive early decisions; use tail sampling when outcomes or errors determine retention. Tail sampling requires trace affinity and bounded state.
|
|
32
|
+
- Define separate policies for errors, slow requests, critical business journeys, and routine traffic. Validate that parent/child spans remain coherent.
|
|
33
|
+
- Treat telemetry as potentially sensitive data. Minimize attributes, redact secrets and PII, restrict access, define retention, and document residency.
|
|
34
|
+
- Never put user IDs, request IDs, raw URLs, SQL literals, or arbitrary error text in metric labels.
|
|
35
|
+
|
|
36
|
+
## Validation and operations
|
|
37
|
+
- Test instrumentation with representative traffic and verify propagation across services and queues.
|
|
38
|
+
- Monitor telemetry pipeline health independently of the application: ingest rate, export success, dropped spans/metrics/logs, queue age, memory, and backend rejection.
|
|
39
|
+
- Use synthetic spans and known test requests to detect broken propagation and missing instrumentation.
|
|
40
|
+
- Establish telemetry cost budgets and retention tiers; telemetry volume is a workload and a cost center.
|
|
41
|
+
|
|
42
|
+
## Anti-patterns
|
|
43
|
+
- Assuming auto-instrumentation captures business semantics or all async context.
|
|
44
|
+
- Exporting directly from every workload without a failure and backpressure strategy.
|
|
45
|
+
- Treating trace sampling as statistically neutral for every question.
|
|
46
|
+
- Using unbounded attributes or cardinality as a substitute for useful dimensions.
|
|
47
|
+
|
|
48
|
+
## Design checklist
|
|
49
|
+
- [ ] Signal coverage and ownership are defined for critical journeys.
|
|
50
|
+
- [ ] Resource attributes and semantic conventions are consistent.
|
|
51
|
+
- [ ] Context propagation is tested across process and messaging boundaries.
|
|
52
|
+
- [ ] Collector topology, security, buffering, and failure behavior are explicit.
|
|
53
|
+
- [ ] Sampling, redaction, retention, access, and cost policies are documented.
|
|
54
|
+
- [ ] Pipeline loss and instrumentation regressions are observable.
|