system-design-primer-skill 2.3.0 → 2.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +15 -7
- package/SKILL.md +22 -2
- package/VERSION +1 -1
- package/evals/README.md +1022 -0
- package/evals/system-design-cases.json +414 -0
- package/package.json +3 -2
- package/references/03-data-and-storage.md +8 -2
- package/references/04-caching.md +3 -3
- package/references/05-async-and-messaging.md +4 -0
- package/references/06-distributed-systems.md +2 -2
- package/references/07-reliability-and-resilience.md +7 -3
- package/references/08-api-security-observability.md +4 -0
- package/references/15-failure-analysis.md +4 -0
- package/references/19-agent-security-and-permissions.md +22 -6
- package/references/20-agent-state-and-memory.md +4 -0
- package/references/22-kubernetes-and-container-platforms.md +1 -1
- package/references/24-multi-region-and-disaster-recovery.md +6 -2
- package/references/27-production-observability-and-slo.md +4 -0
- package/references/28-event-streaming-and-kafka-pulsar.md +6 -0
- package/references/32-workflow-orchestration-and-temporal.md +4 -0
- package/references/34-postgresql-internals-and-performance.md +2 -0
- package/references/35-mysql-innodb-and-performance.md +2 -0
- package/references/36-sharding-and-distributed-sql.md +1 -1
- package/references/37-consensus-and-distributed-transactions.md +7 -3
- package/references/40-zero-downtime-database-migrations.md +3 -3
- package/references/48-oauth-oidc-and-passkeys.md +3 -3
- package/references/49-iam-rbac-abac-rebac.md +7 -3
- package/references/51-secrets-kms-pki-and-key-management.md +4 -0
- package/references/52-supply-chain-security-sbom-slsa.md +62 -0
- package/references/54-privacy-engineering-and-data-governance.md +4 -0
- package/references/55-opentelemetry-and-instrumentation.md +4 -0
- package/references/56-distributed-tracing-and-context-propagation.md +4 -0
- package/references/59-capacity-planning-and-performance-modeling.md +8 -4
- package/references/60-chaos-engineering-and-resilience-experiments.md +4 -0
- package/references/61-incident-management-and-postmortems.md +4 -0
- package/references/67-storage-infrastructure-and-storage-systems.md +4 -0
- package/references/68-tcp-quic-and-network-performance.md +1 -1
- package/references/69-task-routing-and-evidence.md +75 -18
- package/scripts/audit_chain.py +36 -0
- package/scripts/compare_reports.py +43 -0
- package/scripts/evaluate.py +475 -0
- package/scripts/evaluation_catalog.py +129 -0
- package/scripts/evaluation_fingerprints.py +20 -0
- package/scripts/evidence_archive.py +79 -0
- package/scripts/evidence_signing.py +98 -0
- package/scripts/evidence_timestamp.py +441 -0
- package/scripts/install.ps1 +2 -2
- package/scripts/install.sh +2 -2
- package/scripts/report_integrity.py +184 -0
- package/scripts/review_consensus.py +77 -0
- package/scripts/review_evidence.py +105 -0
- package/scripts/review_provenance.py +44 -0
- package/scripts/reviewer_calibration.py +215 -0
- package/scripts/rubric_compatibility.py +134 -0
- package/scripts/run_evals.py +130 -0
- package/scripts/run_manifest.py +179 -0
- package/scripts/statistical_analysis.py +308 -0
- package/scripts/tsa_attestation.py +709 -0
- package/scripts/tsa_validation_report.py +627 -0
- package/scripts/validate.py +233 -21
- package/scripts/verify_build_provenance.py +625 -0
- package/scripts/verify_tsa_ocsp.py +962 -0
- package/scripts/verify_tsa_revocation.py +782 -0
- package/scripts/windows_broker_protocol.py +274 -0
- package/scripts/windows_broker_transport.py +141 -0
- package/scripts/windows_broker_win32.py +713 -0
|
@@ -2,6 +2,10 @@
|
|
|
2
2
|
|
|
3
3
|
An agent is not a security principal with inherent authority. The application must bind each action to an authenticated principal, tenant, resource, and policy decision. Prompt instructions and model refusals are not authorization controls.
|
|
4
4
|
|
|
5
|
+
## Scope and boundaries
|
|
6
|
+
|
|
7
|
+
This document owns the security model for model-driven agents: principal binding, tool permissions, prompt-injection containment, execution isolation, and approval controls. Use [IAM](49-iam-rbac-abac-rebac.md) for reusable authorization policy semantics; [workload identity](50-zero-trust-and-workload-identity.md) for service-to-service identity and trust; [secrets and key management](51-secrets-kms-pki-and-key-management.md) for credential lifecycle; and [privacy engineering](54-privacy-engineering-and-data-governance.md) for data-purpose and retention obligations. Tool allowlists and model refusals do not replace resource-owner authorization at execution time.
|
|
8
|
+
|
|
5
9
|
## 1. Threat model
|
|
6
10
|
|
|
7
11
|
Include:
|
|
@@ -31,23 +35,35 @@ Prefer narrow tools that perform one business operation over generic shell, SQL,
|
|
|
31
35
|
|
|
32
36
|
Separate read, draft, and commit capabilities. A workflow that can inspect a payment should not automatically be able to issue a refund. Keep high-impact actions behind independent policy checks and approval gates.
|
|
33
37
|
|
|
34
|
-
## 4.
|
|
38
|
+
## 4. Prompt injection and instruction provenance
|
|
39
|
+
|
|
40
|
+
Treat all retrieved or tool-supplied content as tainted data, including text that imitates system/developer messages, policy documents, tool descriptions, code comments, or approval notices. Indirect prompt injection is a data-integrity threat; delimit and label untrusted content, preserve source provenance, and never promote embedded instructions into user intent or policy.
|
|
41
|
+
|
|
42
|
+
Apply these invariants:
|
|
43
|
+
- The user's authenticated request defines the task scope. Retrieved content may be evidence for the task but cannot authorize a new action, tenant, recipient, destination, or privilege.
|
|
44
|
+
- A tool call proposed after reading untrusted content must independently pass deterministic policy checks for principal, tenant, resource, operation, and parameters at execution time.
|
|
45
|
+
- Treat tool output as untrusted too. Do not chain a second tool call merely because the first tool's output requests it; validate the result and re-evaluate the original user intent.
|
|
46
|
+
- Validate structured arguments and generated output against schemas, allowlists, destination controls, and resource limits before passing them to tools or downstream interpreters.
|
|
47
|
+
- Before sending data outside its trust boundary, enforce purpose limitation, recipient authorization, tenant isolation, and field-level minimization. Deny by default when destination or authorization is ambiguous.
|
|
48
|
+
- Test direct and indirect injection in retrieved documents, tool metadata/results, and multi-step workflows. Include attempts to override policy, exfiltrate another tenant's data, change recipients, and trigger unauthorized writes.
|
|
49
|
+
|
|
50
|
+
## 5. Human approval
|
|
35
51
|
|
|
36
|
-
Require explicit approval for irreversible, financial, legal, privacy-sensitive, externally visible, or privilege-changing actions according to risk. Bind approval to a canonical action summary, target resource, parameters, and expiry. Reject stale or modified approvals. Record approver identity, decision, timestamp, and the exact action authorized.
|
|
52
|
+
Require explicit approval for irreversible, financial, legal, privacy-sensitive, externally visible, or privilege-changing actions according to risk. Bind approval to a canonical action summary, tenant, target resource, exact parameters (including amount, recipient, and destination where applicable), and expiry. Reject stale, replayed, or modified approvals; revalidate the same parameters and current authorization immediately before execution. Record approver identity, decision, timestamp, and the exact action authorized.
|
|
37
53
|
|
|
38
54
|
Human approval does not replace server-side validation or transaction controls.
|
|
39
55
|
|
|
40
|
-
##
|
|
56
|
+
## 6. Isolation and execution safety
|
|
41
57
|
|
|
42
58
|
Run untrusted tools and generated code in isolated workers or sandboxes. Apply CPU, memory, process, filesystem, network egress, and wall-clock limits. Use read-only mounts and ephemeral workspaces where possible. Separate tenant workloads and prevent cross-run state leakage. Restrict environment variables and inherited file descriptors. Do not mount host credentials or container sockets into untrusted execution.
|
|
43
59
|
|
|
44
|
-
##
|
|
60
|
+
## 7. Data protection and audit
|
|
45
61
|
|
|
46
62
|
Classify context and outputs. Minimize data before model/provider transmission; apply tenant filters before retrieval results enter context. Define retention, deletion, residency, encryption, and provider policy. Redact secrets and sensitive content from logs by default.
|
|
47
63
|
|
|
48
64
|
Audit principal, tenant, run ID, tool/version, action, target, authorization decision, approval, result status, and correlation IDs. Protect audit integrity and restrict who can inspect it. Avoid storing full sensitive payloads when metadata and hashes suffice.
|
|
49
65
|
|
|
50
|
-
##
|
|
66
|
+
## 8. Defense in depth
|
|
51
67
|
|
|
52
68
|
Use layered controls:
|
|
53
69
|
1. Input validation and content classification.
|
|
@@ -60,7 +76,7 @@ Use layered controls:
|
|
|
60
76
|
|
|
61
77
|
No single prompt filter or model classifier is a sufficient defense.
|
|
62
78
|
|
|
63
|
-
##
|
|
79
|
+
## 9. Security review checklist
|
|
64
80
|
|
|
65
81
|
- [ ] Threat model includes direct/indirect prompt injection and confused deputy.
|
|
66
82
|
- [ ] Principal, tenant, agent, and tool identities are distinct.
|
|
@@ -2,6 +2,10 @@
|
|
|
2
2
|
|
|
3
3
|
Do not conflate model context, conversation history, long-term memory, and durable workflow state. Each has different ownership, consistency, retention, privacy, and recovery requirements.
|
|
4
4
|
|
|
5
|
+
## Scope and boundaries
|
|
6
|
+
|
|
7
|
+
This document owns agent-specific state taxonomy, durable run/checkpoint design, conversation history, and memory lifecycle. Use [privacy engineering](54-privacy-engineering-and-data-governance.md) for retention, deletion, legal holds, and downstream processor obligations; [workflow orchestration](32-workflow-orchestration-and-temporal.md) for durable timers, replay, and orchestration semantics; and [distributed transactions](37-consensus-and-distributed-transactions.md) for atomic coordination with external side effects. Deleting a conversation record does not prove deletion of derived memory, embeddings, caches, replicas, or backups.
|
|
8
|
+
|
|
5
9
|
## 1. State taxonomy
|
|
6
10
|
|
|
7
11
|
| State type | Purpose | Typical lifecycle and controls |
|
|
@@ -16,7 +16,7 @@ Set resource requests from measured steady-state and burst behavior. Use HPA onl
|
|
|
16
16
|
|
|
17
17
|
Avoid feedback loops where HPA, KEDA, cluster autoscaler, and application concurrency controls amplify one another. Bound queue consumers and scale on backlog age or processing demand when appropriate, not CPU alone.
|
|
18
18
|
|
|
19
|
-
Use immutable image digests, signed artifacts where available, SBOMs, vulnerability scanning, admission policy, and provenance verification. Pin and review Helm charts, operators, CRDs, and controller versions. Separate build permissions from deploy permissions. Prefer declarative delivery with drift detection and auditable promotion.
|
|
19
|
+
Use immutable image digests, signed artifacts where available, SBOMs, vulnerability scanning, admission policy, and provenance verification. Pin and review Helm charts, operators, CRDs, and controller versions. Separate build permissions from deploy permissions. Prefer declarative delivery with drift detection and auditable promotion. Kubernetes component version-skew limits and supported release branches change over time; check the current [Kubernetes version-skew policy](https://kubernetes.io/releases/version-skew-policy/) and the compatibility matrix for each add-on before planning control-plane or node upgrades. Do not encode a remembered skew allowance as a timeless rule.
|
|
20
20
|
|
|
21
21
|
Rollouts must define max unavailable/surge, readiness gates, canary or progressive delivery, automated abort criteria, and rollback compatibility. Database and event schema changes must remain compatible with old and new application versions during rollout.
|
|
22
22
|
|
|
@@ -2,6 +2,10 @@
|
|
|
2
2
|
|
|
3
3
|
Multi-region architecture is a business continuity and latency decision, not a checkbox for availability. Begin with business impact analysis, failure scenarios, and tested recovery objectives.
|
|
4
4
|
|
|
5
|
+
## Scope and boundaries
|
|
6
|
+
|
|
7
|
+
This document owns regional topology, data ownership across regions, replication/failover/failback, and business recovery objectives. Use [service reliability](07-reliability-and-resilience.md) for service-level timeout, retry, degradation, and availability design; [storage and database references](03-data-and-storage.md) plus the engine-specific guides for local transaction semantics; and [incident management](61-incident-management-and-postmortems.md) for live incident command. DNS or traffic steering changes do not fence a former writer, and a regional recovery plan must prove data ownership and write safety independently of routing.
|
|
8
|
+
|
|
5
9
|
## Objectives and failure domains
|
|
6
10
|
For each critical journey define RTO, RPO, availability target, error-budget policy, residency constraints, and dependencies that must recover together. Distinguish zone, region, provider, control-plane, account, and global dependency failures. A second region is not independent if both share identity, DNS, deployment, encryption-key, or quota dependencies.
|
|
7
11
|
|
|
@@ -9,14 +13,14 @@ For each critical journey define RTO, RPO, availability target, error-budget pol
|
|
|
9
13
|
Compare backup and restore, pilot light, warm standby, active-passive, and active-active. Choose per workload and data domain. Stateless services may be active-active while transactional data remains single-writer. Do not claim active-active availability without proving write semantics and dependency independence.
|
|
10
14
|
|
|
11
15
|
## Data replication and consistency
|
|
12
|
-
For every dataset define source of truth, replication mode, lag objective, ordering, conflict policy, deletion propagation, encryption, and restore procedure. Synchronous replication can increase write latency and couple regional availability; asynchronous replication permits nonzero RPO and stale reads. Multi-primary writes require conflict resolution that preserves business invariants, not merely last-write-wins timestamps.
|
|
16
|
+
For every dataset define source of truth, replication mode, lag objective, ordering, conflict policy, deletion propagation, encryption, and restore procedure. Synchronous replication can increase write latency and couple regional availability; asynchronous replication permits nonzero RPO and stale reads. Specify whether consistency is per-key, per-entity, per-session, or global, and what a read in each region can observe after a write. Multi-primary writes require conflict resolution that preserves business invariants, not merely last-write-wins timestamps. For invariants that cannot be merged safely (for example, spending a shared balance or allocating a unique scarce item), use a single-writer/ownership boundary or coordination rather than assuming merge or timestamp ordering can preserve correctness.
|
|
13
17
|
|
|
14
18
|
Test replication lag, network partition, duplicate delivery, out-of-order updates, tombstones, schema changes, and recovery from corrupted or logically deleted data. Backups must be isolated from production credentials and deletion paths. Replication is not a backup.
|
|
15
19
|
|
|
16
20
|
## Failover and failback
|
|
17
21
|
Specify detection signals, decision authority, fencing of the old primary, traffic steering, data promotion, write enablement, queue ownership, cache invalidation, and customer communication. Prevent split brain with leases, quorum, or an operator-controlled fencing mechanism appropriate to the data system. DNS TTL alone is not a failover strategy.
|
|
18
22
|
|
|
19
|
-
Failback is a separate migration: reconcile writes made in the recovery region, validate consistency, re-establish replication, and switch traffic only after safety gates pass. Document manual and automated paths, including when automation must stop for human approval.
|
|
23
|
+
Failback is a separate migration: reconcile writes made in the recovery region, validate consistency, re-establish replication, and switch traffic only after safety gates pass. Define the cutover epoch/ownership change and fence the former writer before enabling writes in the restored region; DNS convergence or a routing change is not a write-fencing mechanism. Document manual and automated paths, including when automation must stop for human approval.
|
|
20
24
|
|
|
21
25
|
## Validation and economics
|
|
22
26
|
Run game days for region isolation, degraded inter-region links, unavailable KMS, exhausted quotas, stale DNS, failed deployment, and data corruption. Measure actual RTO/RPO from detection through verified business recovery. Perform restore tests, not just backup-job checks. Record evidence, gaps, owners, and remediation dates.
|
|
@@ -2,6 +2,10 @@
|
|
|
2
2
|
|
|
3
3
|
Observability should answer whether users can complete critical journeys, why they cannot, and what action is safe. Instrumentation volume is not a substitute for useful operational signals.
|
|
4
4
|
|
|
5
|
+
## Scope and boundaries
|
|
6
|
+
|
|
7
|
+
This document owns production telemetry architecture, SLIs/SLOs, error budgets, instrumentation quality, and observability-backend operations. Use [distributed tracing](56-distributed-tracing-and-context-propagation.md) for trace-context propagation details; [OpenTelemetry instrumentation](55-opentelemetry-and-instrumentation.md) for SDK/collector conventions; [service reliability](07-reliability-and-resilience.md) for timeout, retry, failover, and recovery controls; and [incident management](61-incident-management-and-postmortems.md) for command roles, communications, and postmortems. Observability signals support diagnosis; they do not establish causality or prove a recovery action is safe.
|
|
8
|
+
|
|
5
9
|
## Service-level objectives
|
|
6
10
|
Define SLIs from user-visible behavior: availability, latency, correctness, freshness, durability, or successful completion. Specify event population, good-event definition, measurement window, exclusions, and aggregation. Set SLO targets based on user and business needs, not arbitrary industry defaults.
|
|
7
11
|
|
|
@@ -2,6 +2,10 @@
|
|
|
2
2
|
|
|
3
3
|
Choose an event-streaming platform based on durability, ordering, replay, throughput, multi-tenancy, operations, and ecosystem requirements—not brand popularity. Compare Kafka, Pulsar, managed cloud streams, and simpler queues.
|
|
4
4
|
|
|
5
|
+
## Scope and boundaries
|
|
6
|
+
|
|
7
|
+
This document owns Kafka/Pulsar and event-streaming-specific design: partitions, broker replication, producer/consumer behavior, offsets, retention, rebalances, and platform operations. Use [asynchronous processing](05-async-and-messaging.md) for broker-agnostic delivery and queue fundamentals; [CDC](30-change-data-capture-and-debezium.md) for database-log capture; [event sourcing and CQRS](29-event-sourcing-and-cqrs.md) for domain event history and projections; and [schema governance](33-schema-registry-and-event-governance.md) for compatibility policy. Broker transactions do not by themselves make external side effects atomic.
|
|
8
|
+
|
|
5
9
|
## Domain and topic design
|
|
6
10
|
Define event ownership, topic purpose, key, partitioning strategy, retention, compaction, schema, compatibility, and access policy. Use stable business keys when per-entity ordering matters. Partition count determines parallelism and affects ordering, storage, and reassignment cost. Hot keys can bottleneck a partition even when cluster capacity is available.
|
|
7
11
|
|
|
@@ -12,6 +16,8 @@ Document producer acknowledgements, replication factor, minimum in-sync replicas
|
|
|
12
16
|
|
|
13
17
|
Consumers should be idempotent and tolerate duplicates, retries, rebalances, and replay. Commit offsets only after the corresponding side effect is durable, or use an atomic pattern that binds offset progress to output effects. A broker transaction does not automatically make an external database or API side effect exactly once.
|
|
14
18
|
|
|
19
|
+
For Apache Kafka, distinguish producer idempotence from transactions. In Kafka 4.0, producer idempotence is enabled by default only when no conflicting configuration disables it; it suppresses duplicates caused by producer retries within a producer session, not arbitrary application-level resends or effects in another system. A transactional producer requires a stable, appropriately unique `transactional.id`; consumers must use `read_committed` to hide aborted transactional records. End-to-end exactly-once processing across Kafka and an external sink still requires a sink/offset integration or an application-level idempotency and reconciliation design. Re-check defaults and constraints against the deployed client/broker version; see the [Kafka producer API](https://kafka.apache.org/40/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html) and [producer configuration](https://kafka.apache.org/40/generated/producer_config.html).
|
|
20
|
+
|
|
15
21
|
## Consumer operations
|
|
16
22
|
Bound poll batches, processing concurrency, in-flight records, and retry duration. Preserve per-key ordering when parallelizing. Distinguish retryable transient errors from poison events; use bounded retries, delayed retry topics or queues, and a dead-letter workflow with alerting, ownership, replay controls, and audit. Avoid hot retry loops.
|
|
17
23
|
|
|
@@ -2,6 +2,10 @@
|
|
|
2
2
|
|
|
3
3
|
Workflow orchestration coordinates long-running business processes across services, human tasks, timers, and retries. Choose between a task queue, scheduler, state machine, and durable workflow engine based on process duration, recovery, visibility, and compensation needs.
|
|
4
4
|
|
|
5
|
+
## Scope and boundaries
|
|
6
|
+
|
|
7
|
+
This document owns durable business-process orchestration: workflow state, deterministic replay, activity execution, timers, signals, cancellation, versioning, and workflow-engine operations. Use [asynchronous processing](05-async-and-messaging.md) for queue and broker delivery semantics, [consensus and distributed transactions](37-consensus-and-distributed-transactions.md) for cross-resource atomicity and saga guarantees, and [incident management](61-incident-management-and-postmortems.md) for operational response. A workflow engine's durable history does not make an external activity exactly once.
|
|
8
|
+
|
|
5
9
|
## Workflow model
|
|
6
10
|
Represent the process as explicit states and transitions. Define start conditions, inputs, outputs, deadlines, cancellation, pause/resume, human approval, terminal states, and invariants. Separate workflow definition from activity implementation. Give each workflow a stable ID and each activity a deterministic or unique execution identity.
|
|
7
11
|
|
|
@@ -13,6 +13,8 @@ Use this guide when designing or diagnosing PostgreSQL-backed systems. Confirm t
|
|
|
13
13
|
## MVCC, transactions, and vacuum
|
|
14
14
|
PostgreSQL MVCC retains row versions so readers and writers can often proceed concurrently. Updates and deletes create dead tuples; vacuum makes space reusable and maintains visibility information, but does not normally shrink relation files for the operating system.
|
|
15
15
|
- Explain READ COMMITTED, REPEATABLE READ, and SERIALIZABLE semantics, including serialization failures and retry requirements.
|
|
16
|
+
- PostgreSQL's REPEATABLE READ is snapshot isolation: it prevents the standard phantom-read phenomenon but can still permit serialization anomalies. SERIALIZABLE adds Serializable Snapshot Isolation checks and may abort transactions with SQLSTATE `40001`; retry the entire transaction, not only the failed statement. Treat this as versioned PostgreSQL behavior and verify the deployed major release in the [official transaction isolation documentation](https://www.postgresql.org/docs/current/transaction-iso.html).
|
|
17
|
+
- Sequence changes (including `nextval`) are not rolled back with the surrounding transaction; do not use gap-free sequence values as a transactional business invariant.
|
|
16
18
|
- Monitor transaction age, long-running snapshots, idle-in-transaction sessions, dead tuples, autovacuum progress, bloat indicators, and replication slots retaining WAL.
|
|
17
19
|
- Set autovacuum thresholds and scale factors based on table size and churn; do not disable autovacuum globally.
|
|
18
20
|
- Diagnose lock waits and deadlocks with lock graphs and transaction traces. Keep transactions short and acquire locks consistently.
|
|
@@ -9,6 +9,8 @@ InnoDB stores row data in the clustered primary-key B-tree. Secondary-index reco
|
|
|
9
9
|
|
|
10
10
|
## Transactions, locks, and MVCC
|
|
11
11
|
InnoDB uses undo records for consistent reads and rollback, and redo logging for crash recovery. Under REPEATABLE READ, consistent reads and locking reads differ; next-key and gap locks can prevent phantom inserts but increase contention.
|
|
12
|
+
|
|
13
|
+
For MySQL 8.4, REPEATABLE READ is the InnoDB default: plain consistent reads use the snapshot established by the first consistent read, while locking reads and DML use current data and acquire locks according to the index and search predicate. A unique-index equality lookup can lock only the matching index record; range predicates can take next-key/gap locks. READ COMMITTED creates a fresh snapshot for each consistent read and disables most gap locking, with documented exceptions such as foreign-key and duplicate-key checks. Verify these details for the exact MySQL release and statement shape in the [official isolation-level reference](https://dev.mysql.com/doc/refman/8.4/en/innodb-transaction-isolation-levels.html); do not infer lock coverage from SQL text alone.
|
|
12
14
|
- Define isolation and read/write semantics for each critical transaction.
|
|
13
15
|
- Diagnose deadlocks from engine reports; treat them as expected concurrency outcomes and retry bounded, idempotent transactions.
|
|
14
16
|
- Keep transactions short, access rows in consistent order, and index predicates used by locking statements.
|
|
@@ -12,7 +12,7 @@ Define routing metadata and shard-map versioning, tenant affinity/isolation, reb
|
|
|
12
12
|
|
|
13
13
|
## Correctness and operations
|
|
14
14
|
Cross-shard transactions may require two-phase commit or database-specific distributed protocols. Explain coordinator failure, prepared transaction recovery, lock duration, and availability implications. Prefer sagas only when business invariants tolerate intermediate states and compensations.
|
|
15
|
-
Plan online resharding as a state machine: establish target ownership, copy a consistent snapshot, stream changes, verify counts/checksums/invariants, fence old writers, switch routing, monitor, and retain a bounded rollback path. Prevent dual-writer ambiguity.
|
|
15
|
+
Plan online resharding as a state machine: establish target ownership, copy a consistent snapshot, stream changes from a well-defined log position, verify counts/checksums/invariants, fence old writers, switch routing at a defined ownership epoch, monitor, and retain a bounded rollback path. Prevent dual-writer ambiguity. Define how requests already in flight at cutover are drained or rejected, how retries discover the new owner, and how stale shard-map caches are invalidated or safely rejected.
|
|
16
16
|
Track per-shard QPS, p95/p99 latency, CPU, storage, hot-key distribution, replication lag, queue depth, rebalance progress, and cross-shard traffic.
|
|
17
17
|
|
|
18
18
|
## Distributed SQL evaluation
|
|
@@ -1,8 +1,12 @@
|
|
|
1
1
|
# Consensus and Distributed Transactions
|
|
2
2
|
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns consensus protocol safety/liveness assumptions and cross-resource transaction guarantees, including quorum decisions, two-phase commit, saga boundaries, and ambiguous outcomes. Use [distributed systems](06-distributed-systems.md) for the broader consistency and coordination vocabulary; [data and storage](03-data-and-storage.md) for database-local transaction and isolation semantics; [asynchronous processing](05-async-and-messaging.md) for broker-agnostic delivery and outbox/inbox patterns; [event streaming](28-event-streaming-and-kafka-pulsar.md) for broker-specific guarantees; and [durable workflows](32-workflow-orchestration-and-temporal.md) for long-running process execution. Consensus commit, message acknowledgement, workflow history, and business completion are distinct evidence boundaries.
|
|
6
|
+
|
|
3
7
|
## Consensus fundamentals
|
|
4
8
|
Consensus lets distributed replicas agree on an ordered sequence or value despite specified failures. Raft and Paxos-family protocols rely on quorum intersection and explicit safety/liveness assumptions; they do not make arbitrary networks synchronous.
|
|
5
|
-
For Raft-style systems, understand leader election, terms, log replication, commit index, membership changes, quorum availability, and snapshot/log compaction. A majority quorum is necessary for progress in the common crash-fault model, but latency and availability depend on placement and network conditions.
|
|
9
|
+
For Raft-style systems, understand leader election, terms, log replication, commit index, membership changes, quorum availability, and snapshot/log compaction. A majority quorum is necessary for progress in the common crash-fault model, but latency and availability depend on placement and network conditions. A committed log entry is not automatically equivalent to an acknowledged application-level business operation: specify the storage durability contract, state-machine application, client response point, and behavior after an ambiguous timeout. Linearizable reads require an appropriate protocol (for example, a quorum-confirmed read or a valid lease with bounded-clock assumptions); a local read from a leader or follower is not inherently linearizable.
|
|
6
10
|
|
|
7
11
|
## Design questions
|
|
8
12
|
- What failure model is assumed: crash-stop, crash-recovery, Byzantine, or correlated infrastructure failure?
|
|
@@ -14,8 +18,8 @@ For Raft-style systems, understand leader election, terms, log replication, comm
|
|
|
14
18
|
Do not equate leader election with application-level failover or data correctness.
|
|
15
19
|
|
|
16
20
|
## Distributed transactions
|
|
17
|
-
Two-phase commit coordinates participants through prepare and commit/abort. It can block while a coordinator decision is unavailable and hold locks/resources across phases. Explain durable coordinator logs, participant recovery, timeout semantics, and operational procedures.
|
|
18
|
-
Compare local ACID plus transactional outbox, saga orchestration/choreography with explicit compensations, two-phase commit where atomic cross-resource commit is required and supported, and idempotent workflows with reconciliation where eventual convergence is acceptable.
|
|
21
|
+
Two-phase commit coordinates participants through prepare and commit/abort. It can block while a coordinator decision is unavailable and hold locks/resources across phases. A participant that has voted prepared must not unilaterally guess commit or abort after losing the coordinator; it needs the durable global decision or a protocol-defined recovery path. Explain durable coordinator logs, participant recovery, timeout semantics, in-doubt transaction monitoring, lock/resource limits, and operational procedures. A timeout or lost commit acknowledgement creates an unknown outcome, not proof of rollback; clients need a transaction-status lookup or idempotent retry keyed to the same business operation.
|
|
22
|
+
Compare local ACID plus transactional outbox, saga orchestration/choreography with explicit compensations, two-phase commit where atomic cross-resource commit is required and supported, and idempotent workflows with reconciliation where eventual convergence is acceptable. The outbox makes the database state change and event intent atomic within one local transaction; a relay can publish more than once, so consumers still need deduplication/idempotency. A saga needs durable step state, bounded retries, timeouts, compensation policy, and a terminal/manual-reconciliation path. Choose based on the invariant and participant capabilities, not on a blanket preference for one pattern.
|
|
19
23
|
A saga is not an atomic transaction. Compensation is a new business action and may not reverse external side effects.
|
|
20
24
|
|
|
21
25
|
## Failure analysis
|
|
@@ -2,9 +2,9 @@
|
|
|
2
2
|
|
|
3
3
|
## Compatibility-first change
|
|
4
4
|
Use expand-migrate-contract for changes spanning application versions or large datasets. Keep old and new application versions compatible during rolling deployment.
|
|
5
|
-
1. **Expand:** add nullable columns, new tables/indexes, or compatible fields without breaking old readers/writers.
|
|
5
|
+
1. **Expand:** add nullable columns, new tables/indexes, or compatible fields without breaking old readers/writers. Check whether adding a constraint or index scans/rebuilds the table or takes metadata locks on the specific engine/version; use the engine-supported online/concurrent operation only after verifying its failure and retry behavior.
|
|
6
6
|
2. **Deploy compatibility:** code can read old/new representations and write safely under a single ownership strategy.
|
|
7
|
-
3. **Backfill:** process bounded batches with checkpoints, rate limits, idempotency, and observable progress.
|
|
7
|
+
3. **Backfill:** process bounded batches with checkpoints, rate limits, idempotency, and observable progress. Make progress monotonic where possible, define how rows changed during the backfill are captured, and ensure a restart cannot overwrite a newer application write with stale data.
|
|
8
8
|
4. **Validate:** compare counts, checksums, nullability, uniqueness, and domain invariants; sample and reconcile differences.
|
|
9
9
|
5. **Switch:** gradually move reads/writes, monitor errors and latency, and preserve a tested rollback route.
|
|
10
10
|
6. **Contract:** remove obsolete fields/indexes only after consumers and rollback windows have expired.
|
|
@@ -16,7 +16,7 @@ Use expand-migrate-contract for changes spanning application versions or large d
|
|
|
16
16
|
Use keyset/range pagination rather than large OFFSET scans. Bound batch size, concurrency, transaction duration, and replication impact. Persist checkpoints and make each batch restartable. Avoid uncoordinated dual writes; define an authoritative path and reconciliation.
|
|
17
17
|
|
|
18
18
|
## Cutover and rollback
|
|
19
|
-
Use feature flags, shadow reads, comparison metrics, canaries, and progressive traffic shifts. Define rollback triggers and whether rollback remains possible after writes begin in the new schema. Defer irreversible contract steps until confidence and recovery evidence meet explicit gates.
|
|
19
|
+
Use feature flags, shadow reads, comparison metrics, canaries, and progressive traffic shifts. Define rollback triggers and whether rollback remains possible after writes begin in the new schema. If new code writes data that old code cannot understand, application rollback may require a forward-fix or a separately tested reverse transformation; switching a flag does not restore data compatibility. Defer irreversible contract steps until confidence and recovery evidence meet explicit gates. After contract, treat rollback as a restore/recovery operation unless backward compatibility was deliberately retained.
|
|
20
20
|
|
|
21
21
|
## Validation checklist
|
|
22
22
|
- Schema compatibility across deployed application versions.
|
|
@@ -4,13 +4,13 @@
|
|
|
4
4
|
OAuth is an authorization framework; OpenID Connect adds an identity layer. Distinguish the authorization server, resource server, client, subject, and relying party. Use Authorization Code with PKCE for interactive public clients; do not embed client secrets in mobile or browser applications. Avoid implicit and resource-owner-password flows. Verify current standards and provider-specific behavior before deployment.
|
|
5
5
|
|
|
6
6
|
## Token validation and lifecycle
|
|
7
|
-
|
|
7
|
+
First determine whether the access token is a JWT or opaque token; validate JWTs locally only under the issuer contract, and use the provider-supported introspection path for opaque tokens. For JWTs, pin an allowed algorithm set from trusted configuration (never accept an algorithm selected solely by the token), resolve keys only from trusted issuer metadata/JWKS, handle key rotation and unknown `kid` safely, and validate signature, exact issuer, intended audience, expiry, not-before, and `azp` when required by the OIDC profile. For ID tokens, also validate the authentication request binding (`nonce` when sent) and the applicable audience/authorized-party rules. Under OpenID Connect Core, `aud` must include this relying party's `client_id`; when `aud` contains multiple values, the client SHOULD require `azp` to be present; whenever `azp` is present, it MUST identify this client. Do not make `azp` a universal mandatory claim for every single-audience ID token unless the applicable profile or provider contract requires it. Apply the exact rules from the [OpenID Connect Core ID Token specification](https://openid.net/specs/openid-connect-core-1_0.html). Access tokens should be audience-restricted and short-lived according to risk. Define refresh-token rotation, reuse detection, revocation, session termination, and key rollover. Do not treat an ID token as an API access token or assume every access token is a JWT.
|
|
8
8
|
|
|
9
9
|
## Redirects and browser security
|
|
10
|
-
Use exact registered redirect URIs, state binding, nonce for OIDC authentication, PKCE, and strict callback handling. Protect cookies with Secure, HttpOnly, and appropriate SameSite attributes. Defend against open redirects, login CSRF, token leakage through URLs, and unsafe post-logout redirects.
|
|
10
|
+
Use exact registered redirect URIs, state binding, nonce for OIDC authentication, PKCE with `S256` for public clients, and strict callback handling. Bind the authorization response to the initiating browser session; validate `state` even when PKCE is used, and validate `nonce` when an OIDC authentication request included one. Protect cookies with Secure, HttpOnly, and appropriate SameSite attributes. Defend against open redirects, login CSRF, token leakage through URLs, and unsafe post-logout redirects.
|
|
11
11
|
|
|
12
12
|
## Passkeys and WebAuthn
|
|
13
|
-
Passkeys use public-key credentials scoped to a relying party. Verify challenge freshness and one-time use, origin, RP ID, user presence/verification policy, credential ownership, and signature counter behavior according to authenticator capabilities. Design recovery, device replacement, account linking, and phishing-resistant step-up flows; passkeys do not eliminate account-recovery risk.
|
|
13
|
+
Passkeys use public-key credentials scoped to a relying party. Verify challenge freshness and one-time use, exact expected origin, RP ID hash, user presence/verification policy, credential ownership, and signature counter behavior according to authenticator capabilities. Do not treat a non-increasing signature counter as a universal clone detector: synced passkeys and authenticators differ, so apply the WebAuthn guidance for the authenticator model and use risk signals rather than rejecting every zero or unchanged counter. Design recovery, device replacement, account linking, and phishing-resistant step-up flows; passkeys do not eliminate account-recovery risk.
|
|
14
14
|
|
|
15
15
|
## Review checklist
|
|
16
16
|
- [ ] OAuth and OIDC responsibilities are separated correctly.
|
|
@@ -1,16 +1,20 @@
|
|
|
1
1
|
# Identity and Access Management: RBAC, ABAC, and ReBAC
|
|
2
2
|
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns authorization policy semantics: principal/resource/action evaluation, RBAC/ABAC/ReBAC, tenant and object checks, deny precedence, delegation, and revocation freshness. Use [agent security](19-agent-security-and-permissions.md) for model/tool-specific threat controls; [workload identity](50-zero-trust-and-workload-identity.md) for machine identity and mutual authentication; and [API security](47-api-security-rate-limiting-and-abuse.md) for request abuse controls. Authentication, network location, gateway admission, and cached policy results are inputs or enforcement layers—not substitutes for authoritative resource authorization.
|
|
6
|
+
|
|
3
7
|
## Authorization model selection
|
|
4
8
|
RBAC assigns permissions through roles and suits stable organizational responsibilities. ABAC evaluates subject, resource, action, and environment attributes. ReBAC evaluates relationships such as owner, member, parent, and delegated access. Real systems may combine models, but policy composition and precedence must be explicit.
|
|
5
9
|
|
|
6
10
|
## Policy design
|
|
7
|
-
Define principal types, resource hierarchy, actions, scopes, tenant boundaries, deny/allow precedence, default-deny behavior, and delegation. Separate authentication from authorization and coarse gateway checks from authoritative domain checks. Enforce object-level authorization on every read, write, export, and background
|
|
11
|
+
Define principal types, resource hierarchy, actions, scopes, tenant boundaries, deny/allow precedence, default-deny behavior, and delegation. Specify whether a deny overrides an allow, how inherited grants interact with explicit denies, and which service is authoritative. Separate authentication from authorization and coarse gateway checks from authoritative domain checks. Enforce object-level authorization on every read, write, export, bulk operation, and background job. Derive tenant and principal context from authenticated server-side identity, not a caller-controlled tenant header or object identifier.
|
|
8
12
|
|
|
9
13
|
## Policy evaluation
|
|
10
|
-
Centralized policy engines improve consistency but add latency and availability dependencies. Define policy distribution, cache TTL, invalidation, fail-open/fail-closed behavior, versioning, and explainability. Cache keys must include principal, tenant, resource, action, relevant attributes, and policy version.
|
|
14
|
+
Centralized policy engines improve consistency but add latency and availability dependencies. Define policy distribution, cache TTL, invalidation, fail-open/fail-closed behavior, versioning, and explainability. Cache keys must include principal, tenant, resource, action, relevant attributes, and policy version. Define a revocation freshness objective and an invalidation strategy; a short TTL alone does not guarantee immediate revocation. For high-impact actions, consider authoritative re-checks or deny-list/version checks. Fail closed for sensitive authorization decisions unless a documented, bounded emergency policy says otherwise.
|
|
11
15
|
|
|
12
16
|
## Lifecycle and governance
|
|
13
17
|
Model joiner/mover/leaver workflows, periodic access reviews, privileged access, service accounts, break-glass access, separation of duties, and audit evidence. Prefer time-bound grants and just-in-time elevation. Reconcile identity-provider groups with application permissions; group membership alone is not a complete authorization decision.
|
|
14
18
|
|
|
15
19
|
## Testing
|
|
16
|
-
Test horizontal and vertical privilege escalation, cross-tenant access, inherited permissions, revoked access, policy
|
|
20
|
+
Test horizontal and vertical privilege escalation, cross-tenant access, caller-supplied tenant spoofing, inherited permissions, explicit-deny precedence, revoked access during cache TTL, policy-engine outage, stale caches, bulk/export endpoints, and background jobs executing under the wrong identity. Use policy unit tests and representative end-to-end authorization matrices.
|
|
@@ -1,5 +1,9 @@
|
|
|
1
1
|
# Secrets Management, KMS, PKI, and Key Management
|
|
2
2
|
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns secret, cryptographic key, certificate authority, KMS/HSM, rotation, revocation, and key-recovery lifecycle. Use [workload identity](50-zero-trust-and-workload-identity.md) for workload identity issuance and federation; [supply-chain security](52-supply-chain-security-sbom-slsa.md) for build provenance and artifact signing; and [storage infrastructure](67-storage-infrastructure-and-storage-systems.md) for storage encryption and backup/restore integration. Rotating a key does not prove historical ciphertext was re-encrypted, old credentials were revoked, or encrypted backups remain recoverable.
|
|
6
|
+
|
|
3
7
|
## Secret lifecycle
|
|
4
8
|
Inventory secrets, owners, consumers, environments, purpose, classification, and expiry. Prefer workload identity and dynamic credentials over static secrets. Centralize issuance, access policy, rotation, revocation, and audit; avoid distributing a universal root secret to every service.
|
|
5
9
|
|
|
@@ -14,3 +14,65 @@ Use isolated ephemeral builders, least-privilege tokens, protected branches, rev
|
|
|
14
14
|
|
|
15
15
|
## Release and response
|
|
16
16
|
Define artifact signing, verification, promotion, rollback, and incident procedures for compromised dependencies or builders. Test provenance verification and quarantine paths. Treat SLSA levels as a framework for incremental assurance, not a blanket security certification.
|
|
17
|
+
|
|
18
|
+
## Repository build provenance
|
|
19
|
+
|
|
20
|
+
On pushes to `main`, after the Linux validation job builds and verifies the deterministic ZIP, a separate least-privilege job generates a GitHub SLSA provenance attestation for the ZIP. Pull-request runs do not receive attestation-writing permissions at job execution time and do not publish provenance. The attestation job uses GitHub OIDC and the GitHub attestation service; it does not use a long-lived signing key.
|
|
21
|
+
|
|
22
|
+
Verify a downloaded ZIP against the exact main-branch commit that produced it:
|
|
23
|
+
|
|
24
|
+
```sh
|
|
25
|
+
python scripts/verify_build_provenance.py dist/system-design-primer-v2.3.0.zip \
|
|
26
|
+
--repo jack-001-a/system-design-primer-skill \
|
|
27
|
+
--signer-workflow jack-001-a/system-design-primer-skill/.github/workflows/ci.yml \
|
|
28
|
+
--source-ref refs/heads/main \
|
|
29
|
+
--source-digest <full-40-character-source-commit-sha>
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
The verifier delegates cryptographic signature, certificate, transparency-log, and trusted-root validation to the installed GitHub CLI's `gh attestation verify` command. It additionally requires the repository, signer workflow, source ref, source commit, GitHub Actions OIDC issuer, SLSA provenance v1 predicate, and GitHub-hosted runner policy. Install and authenticate a current GitHub CLI before running it. A successful result proves the artifact is covered by a GitHub-verified attestation matching that policy; it does not prove the build was bug-free, reproducible, or free from compromise inside the authorized workflow.
|
|
33
|
+
|
|
34
|
+
|
|
35
|
+
### Verifier version and subject policy
|
|
36
|
+
|
|
37
|
+
The repository verifier requires GitHub CLI **2.102.0 or newer**. Versions through 2.101.0 had security advisories affecting case-sensitive source-ref matching and exact signer-workflow identity matching; do not use those versions for provenance authorization. The wrapper also checks the verified in-toto statement's predicate type and SHA-256 subject against a private snapshot of the exact artifact bytes passed to the CLI. This supplements, but does not replace, the CLI's cryptographic verification.
|
|
38
|
+
|
|
39
|
+
|
|
40
|
+
## Reproducible and offline verification
|
|
41
|
+
|
|
42
|
+
Download the artifact's attestation bundle while online:
|
|
43
|
+
|
|
44
|
+
```sh
|
|
45
|
+
gh attestation download dist/system-design-primer-v2.3.0.zip \
|
|
46
|
+
--repo jack-001-a/system-design-primer-skill \
|
|
47
|
+
--predicate-type https://slsa.dev/provenance/v1
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
The CLI writes a digest-named JSONL bundle. Preserve it alongside the artifact. The verifier's offline mode requires all three of `--bundle`, `--trusted-root`, and `--repository-id`; it fails closed if any is missing. Obtain a Sigstore trusted-root JSONL through `gh attestation trusted-root` and authenticate it through an independent trusted channel:
|
|
51
|
+
|
|
52
|
+
```sh
|
|
53
|
+
python scripts/verify_build_provenance.py dist/system-design-primer-v2.3.0.zip \
|
|
54
|
+
--repo jack-001-a/system-design-primer-skill \
|
|
55
|
+
--signer-workflow jack-001-a/system-design-primer-skill/.github/workflows/ci.yml \
|
|
56
|
+
--source-ref refs/heads/main \
|
|
57
|
+
--source-digest <full-40-character-source-commit-sha> \
|
|
58
|
+
--bundle sha256:<artifact-digest>.jsonl \
|
|
59
|
+
--trusted-root trusted_root.jsonl \
|
|
60
|
+
--repository-id 1462786926
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Obtain the expected numeric repository ID online from the canonical repository using `gh api repos/jack-001-a/system-design-primer-skill --jq .id`, and pass it with `--repository-id`. The verifier compares it with `verificationResult.signature.certificate.sourceRepositoryIdentifier` for every verified statement. Never trust a bundle or trusted-root file merely because it is adjacent to the artifact. A bundle is evidence to verify, not a trust anchor; the trusted root must be authenticated independently. Offline verification cannot learn about signing-key or certificate revocations published after the trusted-root snapshot was obtained. Refresh and independently authenticate the trusted root whenever importing new signed material, and understand that a stale root may continue to accept signatures made with subsequently revoked key material. GitHub CLI's name check alone cannot detect subsequent repository renames or name reuse; the additional repository-ID check above protects this offline path. If the repository has been renamed, pass the owner/repository name recorded in the signing certificate to `--repo`, while keeping the stable numeric ID from the canonical repository. Use online API-backed verification when current repository-name ownership matters. The bundle download command is currently public preview.
|
|
64
|
+
|
|
65
|
+
### Attestation trust boundary and stronger isolation
|
|
66
|
+
|
|
67
|
+
The current `.github/workflows/ci.yml` design provides **job-level privilege separation**, not an independent trusted-builder boundary:
|
|
68
|
+
|
|
69
|
+
- The attestation job runs only for a push to `refs/heads/main`; pull-request runs do not receive attestation-writing permissions.
|
|
70
|
+
- The job runs on a fresh runner, downloads the Linux validation job's ZIP artifact, and invokes GitHub's attestation action. It does not check out the source tree or execute project build scripts.
|
|
71
|
+
- Its token permissions are scoped to that job: `contents: read`, `id-token: write`, `attestations: write`, and `artifact-metadata: write`. The workflow default remains `contents: read`.
|
|
72
|
+
- The downloaded artifact is still produced by code and workflow definitions from the same repository revision. A compromised or malicious build on the trusted main branch can therefore produce a malicious ZIP that the attestation job will faithfully attest. Provenance authenticates the builder and its declared inputs; it is not an independent audit of the artifact's behavior.
|
|
73
|
+
|
|
74
|
+
This model depends on protecting `main` and reviewing changes to `.github/workflows/**`, build scripts, packaging logic, and dependency inputs. Configure branch protection or rulesets outside the workflow to require review and required checks; consider a CODEOWNERS rule for workflow and release files, and ensure the rule is actually enforced by repository settings. Do not treat a CODEOWNERS file alone as enforcement.
|
|
75
|
+
|
|
76
|
+
For a stronger trust boundary, move the privileged build/sign operation into a separately administered trusted-builder repository. Pin calls to that reusable workflow by a full commit SHA, restrict which caller repositories and source refs it accepts, and grant only the permissions required by the signing job. Keep caller-controlled inputs narrow and data-only (for example, an artifact identifier, expected digest, source repository ID, and source commit); validate each against the run context before signing. Never accept caller-supplied shell commands, scripts, arbitrary workflow paths, or arbitrary artifact URLs, and do not use `secrets: inherit`. Transfer artifacts through a mechanism whose run and repository identity can be verified, then independently verify the exact artifact digest before attestation. Protect the builder repository and its release process at least as strongly as the artifacts it signs.
|
|
77
|
+
|
|
78
|
+
A reusable workflow in the same repository can reduce duplication and centralize policy, but it does not by itself create an independent administrative trust boundary. SHA pinning protects the called workflow from mutable refs; it does not protect against an authorized change to that pinned workflow's source repository or against unsafe caller-controlled inputs. See GitHub's guidance on [reusable workflows](https://docs.github.com/en/actions/how-tos/reuse-automations/reuse-workflows), [secure use](https://docs.github.com/en/actions/reference/security/secure-use), and [workflow dependency lockfiles](https://docs.github.com/en/actions/concepts/security/workflow-dependency-lockfiles).
|
|
@@ -1,5 +1,9 @@
|
|
|
1
1
|
# Privacy Engineering and Data Governance
|
|
2
2
|
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns data-purpose governance, classification, minimization, retention, subject rights, deletion propagation, residency, and privacy risk. Use [agent state and memory](20-agent-state-and-memory.md) for agent-specific state categories and checkpoints; [OpenTelemetry](55-opentelemetry-and-instrumentation.md) and [distributed tracing](56-distributed-tracing-and-context-propagation.md) for telemetry-specific controls; and [storage infrastructure](67-storage-infrastructure-and-storage-systems.md) for physical durability, backup, and storage deletion mechanisms. A deletion workflow is complete only against a documented inventory of primary, derived, cached, replicated, exported, and backup copies, with exceptions and expiry behavior stated.
|
|
6
|
+
|
|
3
7
|
## Data inventory and purpose
|
|
4
8
|
Map personal and sensitive data across collection, inference, processing, sharing, storage, logs, analytics, backups, and deletion. Document purpose, lawful basis where applicable, data owner, recipients, retention, residency, and user expectations. Minimize collection and avoid retaining data merely because storage is cheap.
|
|
5
9
|
|
|
@@ -1,5 +1,9 @@
|
|
|
1
1
|
# OpenTelemetry Architecture and Instrumentation
|
|
2
2
|
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns OpenTelemetry APIs/SDKs, semantic conventions, collectors, processors, exporters, and telemetry pipeline operations. Use [distributed tracing](56-distributed-tracing-and-context-propagation.md) for trace context, baggage, propagation, and sampling semantics; [production observability](27-production-observability-and-slo.md) for SLIs/SLOs and service-level interpretation; and [privacy engineering](54-privacy-engineering-and-data-governance.md) for data classification and retention policy. Instrumentation can be dropped, sampled, delayed, or misconfigured; absence of telemetry is not evidence that an operation did not occur.
|
|
6
|
+
|
|
3
7
|
## Purpose and scope
|
|
4
8
|
OpenTelemetry (OTel) provides vendor-neutral APIs, SDKs, semantic conventions, and telemetry pipelines for traces, metrics, and logs. Treat it as an instrumentation and data-delivery architecture, not as an observability product or a guarantee that telemetry is correct.
|
|
5
9
|
|
|
@@ -1,5 +1,9 @@
|
|
|
1
1
|
# Distributed Tracing and Context Propagation
|
|
2
2
|
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns distributed trace structure, context propagation, baggage, sampling behavior, and trace-based latency interpretation. Use [OpenTelemetry instrumentation](55-opentelemetry-and-instrumentation.md) for APIs/SDKs, semantic conventions, collectors, and exporters; [production observability](27-production-observability-and-slo.md) for SLIs/SLOs, alerting, and telemetry-backend operations; and [privacy engineering](54-privacy-engineering-and-data-governance.md) for authoritative data classification, retention, and deletion policy. A trace is diagnostic evidence, not proof of causality, authorization, or business completion; missing spans do not prove that work did not occur.
|
|
6
|
+
|
|
3
7
|
## Trace model
|
|
4
8
|
A trace is a causal graph of spans, not necessarily a single linear tree. Define trace boundaries, parent-child relationships, span links, baggage policy, and the behavior of asynchronous work before interpreting latency or causality.
|
|
5
9
|
|
|
@@ -1,20 +1,24 @@
|
|
|
1
1
|
# Capacity Planning and Performance Modeling
|
|
2
2
|
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns workload modeling, demand/capacity estimates, queue stability, headroom, failure capacity, and scaling thresholds. Use [load and stress testing](58-load-stress-soak-testing-and-workload-modeling.md) for experimental workload design and measured test results; [production observability](27-production-observability-and-slo.md) for live SLI/SLO signals; and [FinOps](26-finops-and-cloud-cost.md) for cost allocation and economic trade-offs. A capacity estimate is conditional on workload mix, service-time distribution, topology, quotas, and failure assumptions; benchmark throughput or replica count alone does not establish production headroom.
|
|
6
|
+
|
|
3
7
|
## Planning inputs
|
|
4
8
|
Use measured traffic and growth where available. Record average/peak arrival rates, concurrency, request mix, payload sizes, CPU time/request, memory/request, database queries/request, connection occupancy, queue arrival/service rates, storage growth, replication, retention, and failure scenarios. Mark forecasts and assumptions explicitly.
|
|
5
9
|
|
|
6
10
|
## Capacity model
|
|
7
|
-
- Little's Law: L = λW, where L is average in-system work, λ is throughput, and W is average time in the system. Apply
|
|
8
|
-
- CPU demand: approximate
|
|
11
|
+
- Little's Law: L = λW, where L is average in-system work, λ is throughput, and W is average time in the system. Apply to a stable system and a consistent boundary using long-run averages; it does not by itself predict tail latency or size a transient burst. State whether the boundary includes queued, executing, or downstream work.
|
|
12
|
+
- CPU demand: approximate fully utilized core-equivalents as throughput × CPU-seconds/request for the measured workload. Convert that demand to provisioned cores using a target utilization/headroom policy, and account for CPU throttling, uneven request cost, burst behavior, and the CPU spent on retries and background work.
|
|
9
13
|
- Connection demand: estimate concurrency from arrival rate × connection occupancy time; validate pool limits and downstream capacity.
|
|
10
|
-
- Queue stability:
|
|
14
|
+
- Queue stability: for a simple work-conserving queue, long-run arrival work must remain below sustainable service capacity for backlog to be stable; equality leaves no recovery headroom, and bursty arrivals or variable service times can produce large waits well before the average rates meet. Model utilization, service-time variability, burst size, queue limits, deadline expiry, retry amplification, and backlog drain time. Estimate drain time from net spare capacity (service rate minus ongoing arrival rate), not total service rate alone.
|
|
11
15
|
- Storage: model logical growth, indexes, replicas, WAL/binlogs, backups, retention, compaction, and migration headroom separately.
|
|
12
16
|
- Network: include payload, protocol/TLS overhead, retries, replication, telemetry, and cross-zone/region traffic.
|
|
13
17
|
|
|
14
18
|
## Bottleneck and headroom
|
|
15
19
|
- Find the first constrained resource across the full dependency chain; scaling stateless application replicas may simply move the bottleneck to the database or a shared service.
|
|
16
20
|
- Define safe operating envelopes using SLOs, not theoretical maximum throughput.
|
|
17
|
-
- Include N+1 failure capacity, rolling deployments, zone loss, and maintenance events where required.
|
|
21
|
+
- Include N+1 failure capacity, rolling deployments, zone loss, and maintenance events where required. Define the failure scenario explicitly (for example, loss of the largest zone or one node) and verify that surviving dependencies, connection pools, quotas, and network paths can carry the shifted load; replica count alone is not proof of N+1 capacity.
|
|
18
22
|
- Model autoscaling detection and provisioning delay; reactive scaling cannot prevent every burst.
|
|
19
23
|
- Evaluate vertical, horizontal, queue-based, and workload-shaping options before introducing partitioning or sharding.
|
|
20
24
|
|
|
@@ -1,5 +1,9 @@
|
|
|
1
1
|
# Chaos Engineering and Resilience Experiments
|
|
2
2
|
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns the methodology and safety controls for planned, hypothesis-driven resilience experiments. It does not define live incident command or replace incident response; use [incident management](61-incident-management-and-postmortems.md) for active incidents and postmortems, [failure analysis](15-failure-analysis.md) to structure failure modes, and [reliability design](07-reliability-and-resilience.md) for the underlying resilience controls. Experiment results validate only the tested hypothesis, workload, topology, and time window.
|
|
6
|
+
|
|
3
7
|
## Principles
|
|
4
8
|
Chaos engineering is a controlled method for testing a falsifiable resilience hypothesis. It is not random disruption in production. Begin with steady-state signals that represent user-visible behavior, then introduce the smallest fault that can test the hypothesis.
|
|
5
9
|
|
|
@@ -1,5 +1,9 @@
|
|
|
1
1
|
# Incident Management, On-call, and Postmortems
|
|
2
2
|
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns operational incident response: severity, command roles, communications, triage, recovery verification, on-call practice, and postmortem follow-through. Use [chaos engineering](60-chaos-engineering-and-resilience-experiments.md) for planned fault injection, [failure analysis](15-failure-analysis.md) for design-time fault registers, and [reliability architecture](07-reliability-and-resilience.md) for service-level resilience patterns. Technical replay, failover, and reconciliation procedures must link to the relevant queue, data, and regional-architecture references rather than being inferred from incident process alone.
|
|
6
|
+
|
|
3
7
|
## Operational model
|
|
4
8
|
Define severity levels by user/business impact, not by component count or internal effort. Establish incident command, technical lead, communications lead, and scribe roles appropriate to incident scale. Publish escalation paths, service ownership, dependency contacts, and decision authority.
|
|
5
9
|
|
|
@@ -1,5 +1,9 @@
|
|
|
1
1
|
# Storage Infrastructure and Storage Systems
|
|
2
2
|
|
|
3
|
+
## Scope and boundaries
|
|
4
|
+
|
|
5
|
+
This document owns infrastructure-level block, file, and object storage behavior: performance, durability, replication, lifecycle tiers, backup integration, and storage cost. Use [data and storage](03-data-and-storage.md) for application data models and transaction semantics; [privacy engineering](54-privacy-engineering-and-data-governance.md) for retention and deletion obligations; and [secrets/KMS/PKI](51-secrets-kms-pki-and-key-management.md) for cryptographic key lifecycle. Storage durability, replication, snapshots, and retention locks do not by themselves prove application consistency, recoverability, or compliance with a deletion request.
|
|
6
|
+
|
|
3
7
|
## Workload characterization
|
|
4
8
|
Classify data by access pattern, durability, latency, throughput, IOPS, block size, concurrency, growth, retention, sharing, locality, and recovery objectives. Separate application database semantics from underlying block, file, and object storage guarantees.
|
|
5
9
|
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# TCP, QUIC, and Network Performance Engineering
|
|
2
2
|
|
|
3
3
|
## Transport selection
|
|
4
|
-
Understand TCP connection setup, congestion control, retransmission, flow control, head-of-line blocking, keepalive, and connection reuse. Evaluate QUIC/HTTP/3 for connection migration, stream multiplexing, handshake behavior, and deployment compatibility; do not assume it improves every workload.
|
|
4
|
+
Understand TCP connection setup, congestion control, retransmission, flow control, head-of-line blocking, keepalive, and connection reuse. Evaluate QUIC/HTTP/3 for connection migration, stream multiplexing, handshake behavior, and deployment compatibility; do not assume it improves every workload. QUIC loss recovery retransmits information in newly formed packets/frames rather than retransmitting an entire lost packet; do not describe it as TCP-style packet retransmission. Connection migration is subject to connection-ID and path-validation rules and can be blocked by deployment policy or intermediaries. Verify transport claims against [RFC 9000](https://www.rfc-editor.org/rfc/rfc9000.html) and the relevant recovery specification.
|
|
5
5
|
|
|
6
6
|
## Latency budget
|
|
7
7
|
Decompose DNS lookup, connection establishment, TLS handshake, network RTT, congestion/window growth, server queueing, application processing, and response transfer. Separate cold and warm connections, client and server time, and network from application delay.
|