system-design-primer-skill 1.2.0 → 2.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +106 -60
- package/SKILL.md +417 -93
- package/VERSION +1 -1
- package/package.json +44 -3
- package/references/10-workflows-and-checklists.md +1 -1
- package/references/13-architecture-intelligence.md +91 -0
- package/references/14-api-and-data-contracts.md +80 -0
- package/references/15-failure-analysis.md +84 -0
- package/references/16-multi-agent-architecture.md +76 -0
- package/references/17-agent-harness-and-runtime.md +87 -0
- package/references/18-mcp-architecture.md +67 -0
- package/references/19-agent-security-and-permissions.md +74 -0
- package/references/20-agent-state-and-memory.md +72 -0
- package/references/21-agent-evaluation-and-observability.md +90 -0
- package/references/22-kubernetes-and-container-platforms.md +29 -0
- package/references/23-service-mesh-and-traffic-management.md +21 -0
- package/references/24-multi-region-and-disaster-recovery.md +24 -0
- package/references/25-platform-engineering-and-developer-experience.md +26 -0
- package/references/26-finops-and-cloud-cost.md +29 -0
- package/references/27-production-observability-and-slo.md +29 -0
- package/references/28-event-streaming-and-kafka-pulsar.md +27 -0
- package/references/29-event-sourcing-and-cqrs.md +23 -0
- package/references/30-change-data-capture-and-debezium.md +27 -0
- package/references/31-data-lakehouse-and-stream-batch.md +24 -0
- package/references/32-workflow-orchestration-and-temporal.md +24 -0
- package/references/33-schema-registry-and-event-governance.md +22 -0
- package/references/34-postgresql-internals-and-performance.md +43 -0
- package/references/35-mysql-innodb-and-performance.md +27 -0
- package/references/36-sharding-and-distributed-sql.md +22 -0
- package/references/37-consensus-and-distributed-transactions.md +25 -0
- package/references/38-search-architecture-elasticsearch-opensearch.md +24 -0
- package/references/39-vector-databases-and-hybrid-retrieval.md +31 -0
- package/references/40-zero-downtime-database-migrations.md +27 -0
- package/references/41-api-gateway-and-edge-architecture.md +23 -0
- package/references/42-rest-api-design-and-lifecycle.md +23 -0
- package/references/43-graphql-architecture-and-governance.md +16 -0
- package/references/44-grpc-and-service-contracts.md +16 -0
- package/references/45-websocket-sse-and-realtime-systems.md +16 -0
- package/references/46-realtime-collaboration-ot-crdt.md +16 -0
- package/references/47-api-security-rate-limiting-and-abuse.md +16 -0
- package/references/48-oauth-oidc-and-passkeys.md +20 -0
- package/references/49-iam-rbac-abac-rebac.md +16 -0
- package/references/50-zero-trust-and-workload-identity.md +16 -0
- package/references/51-secrets-kms-pki-and-key-management.md +16 -0
- package/references/52-supply-chain-security-sbom-slsa.md +16 -0
- package/references/53-threat-modeling-stride-and-attack-trees.md +16 -0
- package/references/54-privacy-engineering-and-data-governance.md +16 -0
- package/references/55-opentelemetry-and-instrumentation.md +50 -0
- package/references/56-distributed-tracing-and-context-propagation.md +43 -0
- package/references/57-continuous-profiling-and-performance-diagnostics.md +45 -0
- package/references/58-load-stress-soak-testing-and-workload-modeling.md +44 -0
- package/references/59-capacity-planning-and-performance-modeling.md +34 -0
- package/references/60-chaos-engineering-and-resilience-experiments.md +41 -0
- package/references/61-incident-management-and-postmortems.md +34 -0
- package/references/62-dns-and-service-discovery.md +35 -0
- package/references/63-load-balancing-and-global-traffic-management.md +31 -0
- package/references/64-cdn-and-edge-caching.md +34 -0
- package/references/65-cloud-networking-and-connectivity.md +34 -0
- package/references/66-edge-computing-architecture.md +33 -0
- package/references/67-storage-infrastructure-and-storage-systems.md +30 -0
- package/references/68-tcp-quic-and-network-performance.md +27 -0
- package/references/69-task-routing-and-evidence.md +58 -0
- package/scripts/install.ps1 +10 -1
- package/scripts/install.sh +13 -6
- package/scripts/validate.py +133 -0
- package/templates/agent-evaluation-plan-template.md +57 -0
- package/templates/agent-security-review-template.md +46 -0
- package/templates/agent-system-design-template.md +74 -0
- package/templates/api-architecture-design-template.md +33 -0
- package/templates/api-security-and-rate-limit-review-template.md +29 -0
- package/templates/architecture-decision-matrix-template.md +48 -0
- package/templates/architecture-review-scorecard-template.md +36 -0
- package/templates/cdc-data-pipeline-template.md +41 -0
- package/templates/cdn-and-edge-strategy-template.md +36 -0
- package/templates/chaos-experiment-template.md +39 -0
- package/templates/cloud-cost-model-template.md +37 -0
- package/templates/database-architecture-review-template.md +33 -0
- package/templates/edge-computing-readiness-template.md +36 -0
- package/templates/event-sourcing-cqrs-template.md +41 -0
- package/templates/event-streaming-design-template.md +47 -0
- package/templates/failure-analysis-template.md +36 -0
- package/templates/identity-and-access-architecture-template.md +29 -0
- package/templates/incident-postmortem-template.md +39 -0
- package/templates/multi-region-disaster-recovery-template.md +43 -0
- package/templates/network-architecture-design-template.md +38 -0
- package/templates/observability-architecture-template.md +52 -0
- package/templates/performance-test-plan-template.md +52 -0
- package/templates/privacy-data-governance-template.md +24 -0
- package/templates/production-readiness-checklist-template.md +52 -0
- package/templates/realtime-system-design-template.md +28 -0
- package/templates/security-threat-model-template.md +26 -0
- package/templates/sharding-distributed-sql-design-template.md +34 -0
- package/templates/storage-architecture-design-template.md +35 -0
- package/templates/supply-chain-security-review-template.md +20 -0
- package/templates/zero-downtime-database-migration-template.md +32 -0
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# Consensus and Distributed Transactions
|
|
2
|
+
|
|
3
|
+
## Consensus fundamentals
|
|
4
|
+
Consensus lets distributed replicas agree on an ordered sequence or value despite specified failures. Raft and Paxos-family protocols rely on quorum intersection and explicit safety/liveness assumptions; they do not make arbitrary networks synchronous.
|
|
5
|
+
For Raft-style systems, understand leader election, terms, log replication, commit index, membership changes, quorum availability, and snapshot/log compaction. A majority quorum is necessary for progress in the common crash-fault model, but latency and availability depend on placement and network conditions.
|
|
6
|
+
|
|
7
|
+
## Design questions
|
|
8
|
+
- What failure model is assumed: crash-stop, crash-recovery, Byzantine, or correlated infrastructure failure?
|
|
9
|
+
- What is the quorum and which regions/zones participate?
|
|
10
|
+
- Which writes are acknowledged: leader append, quorum commit, or durable flush?
|
|
11
|
+
- How are membership changes, snapshots, log compaction, and recovery handled?
|
|
12
|
+
- Can stale reads occur, and what read-index/lease/follower-read guarantees apply?
|
|
13
|
+
- How are fencing tokens used to prevent a former leader or stale owner from acting?
|
|
14
|
+
Do not equate leader election with application-level failover or data correctness.
|
|
15
|
+
|
|
16
|
+
## Distributed transactions
|
|
17
|
+
Two-phase commit coordinates participants through prepare and commit/abort. It can block while a coordinator decision is unavailable and hold locks/resources across phases. Explain durable coordinator logs, participant recovery, timeout semantics, and operational procedures.
|
|
18
|
+
Compare local ACID plus transactional outbox, saga orchestration/choreography with explicit compensations, two-phase commit where atomic cross-resource commit is required and supported, and idempotent workflows with reconciliation where eventual convergence is acceptable.
|
|
19
|
+
A saga is not an atomic transaction. Compensation is a new business action and may not reverse external side effects.
|
|
20
|
+
|
|
21
|
+
## Failure analysis
|
|
22
|
+
Model network partitions, lost acknowledgements, duplicate messages, coordinator crashes before/after prepare, participant restart, delayed retries, clock skew, and stale leaders. Define fencing, idempotency, durable decision records, retry budgets, and reconciliation.
|
|
23
|
+
|
|
24
|
+
## Validation
|
|
25
|
+
Use fault injection and deterministic integration tests to verify safety properties separately from liveness under stated assumptions. Document assumptions and limits.
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# Search Architecture: Elasticsearch and OpenSearch
|
|
2
|
+
|
|
3
|
+
## Search as a derived read model
|
|
4
|
+
Treat search indexes as derived data, not the authoritative source of truth unless the domain explicitly accepts that ownership model. Define ingestion from database/CDC/events, transformation, indexing, refresh visibility, deletion propagation, and reconciliation.
|
|
5
|
+
|
|
6
|
+
## Index and query design
|
|
7
|
+
- Map fields deliberately: analyzed text, keyword/exact values, numeric/date, geo, nested, and vector fields have different costs and semantics.
|
|
8
|
+
- Design analyzers, tokenization, synonyms, language handling, stemming, and normalization around user intent and relevance evaluation.
|
|
9
|
+
- Use filters for structured constraints and scoring for relevance. Avoid uncontrolled wildcard/regex queries and unbounded result windows.
|
|
10
|
+
- Use search_after with a stable sort for deep pagination; bound PIT/search-context lifecycle and resource costs.
|
|
11
|
+
- Evaluate shard count, shard size, replicas, routing, segment merges, refresh interval, and rollover policy using actual workload.
|
|
12
|
+
|
|
13
|
+
## Reliability and freshness
|
|
14
|
+
Define indexing lag and user-visible freshness SLOs. Make ingestion idempotent, handle out-of-order updates with versioning or source sequence, and propagate deletes/tombstones. Plan reindexing with aliases or versioned indexes, backfill throttling, dual-read comparison, atomic alias switch, and rollback.
|
|
15
|
+
Monitor rejected requests, thread pools, heap/GC, disk watermarks, segment count, merge pressure, shard skew, query latency, indexing latency, and recovery time.
|
|
16
|
+
|
|
17
|
+
## Relevance and quality
|
|
18
|
+
Build representative query sets and graded relevance judgments. Measure NDCG, MRR, Recall@K, precision, zero-result rate, and query reformulation. Segment by intent, language, tenant, and query class.
|
|
19
|
+
|
|
20
|
+
## Security and tenancy
|
|
21
|
+
Enforce tenant filters server-side and test bypass attempts. Protect document-level authorization, field-level restrictions, PII deletion, audit, and snapshot access. Do not rely on a client-supplied tenant filter as an authorization boundary.
|
|
22
|
+
|
|
23
|
+
## Alternatives
|
|
24
|
+
Compare database full-text search, managed search services, Elasticsearch/OpenSearch, and specialized retrieval systems by freshness, relevance control, operational burden, cost, and query complexity.
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# Vector Databases and Hybrid Retrieval
|
|
2
|
+
|
|
3
|
+
## Retrieval requirements
|
|
4
|
+
Define corpus size, embedding dimensions/model, update rate, metadata filters, tenant boundaries, query QPS, latency SLO, recall target, freshness, deletion requirements, and cost. Distinguish semantic retrieval from lexical search and exact database lookup.
|
|
5
|
+
|
|
6
|
+
## Vector index choices
|
|
7
|
+
Compare exact nearest-neighbor search with approximate methods such as HNSW, IVF, and product quantization. Evaluate recall/latency/memory trade-offs, build time, incremental updates, deletes, filtering behavior, and compaction. Measure with representative queries and ground-truth neighbors.
|
|
8
|
+
Distance functions (cosine, dot product, Euclidean) and vector normalization must match the embedding model and index configuration. Do not compare scores across incompatible models or metrics.
|
|
9
|
+
|
|
10
|
+
## Hybrid retrieval pipeline
|
|
11
|
+
1. Normalize and classify the query.
|
|
12
|
+
2. Apply authorization and tenant constraints before retrieval.
|
|
13
|
+
3. Retrieve lexical candidates (BM25) and vector candidates independently.
|
|
14
|
+
4. Fuse rankings (for example, reciprocal rank fusion) with calibrated or learned alternatives.
|
|
15
|
+
5. Rerank a bounded candidate set with a cross-encoder or domain model.
|
|
16
|
+
6. Apply diversity, freshness, and business constraints.
|
|
17
|
+
7. Return citations/provenance and enforce post-retrieval authorization.
|
|
18
|
+
Treat reranking as a latency/cost budget. Cache only with tenant- and permission-aware keys.
|
|
19
|
+
|
|
20
|
+
## RAG correctness and safety
|
|
21
|
+
- Evaluate retrieval separately from generation. Measure Recall@K, MRR/NDCG, context precision/recall, grounded answer rate, citation correctness, and abstention quality.
|
|
22
|
+
- Preserve chunk-to-source lineage, version, ACL, timestamp, and deletion tombstones.
|
|
23
|
+
- Re-check authorization at retrieval and before returning content; revocation and deletion must propagate to vector and lexical indexes.
|
|
24
|
+
- Treat retrieved text as untrusted data. Prompt injection inside documents must not grant tools, permissions, or policy changes.
|
|
25
|
+
- Version embedding models and indexes. Re-embed through a shadow index with measured quality comparison and controlled cutover.
|
|
26
|
+
|
|
27
|
+
## Storage choices
|
|
28
|
+
Compare pgvector, dedicated vector databases, search engines with vector support, and managed retrieval services. Evaluate filtering, consistency, durability, backup/restore, multi-tenancy, observability, ecosystem, and operational skills—not only benchmark throughput.
|
|
29
|
+
|
|
30
|
+
## Operational metrics
|
|
31
|
+
Track ingestion lag, embedding failures, index build/rebuild time, recall under filters, p95/p99 retrieval latency, memory per vector, cache hit rate, delete propagation, and per-tenant cost.
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# Database Migrations and Zero-Downtime Schema Evolution
|
|
2
|
+
|
|
3
|
+
## Compatibility-first change
|
|
4
|
+
Use expand-migrate-contract for changes spanning application versions or large datasets. Keep old and new application versions compatible during rolling deployment.
|
|
5
|
+
1. **Expand:** add nullable columns, new tables/indexes, or compatible fields without breaking old readers/writers.
|
|
6
|
+
2. **Deploy compatibility:** code can read old/new representations and write safely under a single ownership strategy.
|
|
7
|
+
3. **Backfill:** process bounded batches with checkpoints, rate limits, idempotency, and observable progress.
|
|
8
|
+
4. **Validate:** compare counts, checksums, nullability, uniqueness, and domain invariants; sample and reconcile differences.
|
|
9
|
+
5. **Switch:** gradually move reads/writes, monitor errors and latency, and preserve a tested rollback route.
|
|
10
|
+
6. **Contract:** remove obsolete fields/indexes only after consumers and rollback windows have expired.
|
|
11
|
+
|
|
12
|
+
## Online DDL risk
|
|
13
|
+
"Online" does not mean lock-free or risk-free. Verify engine/version-specific algorithm and lock behavior, metadata locks, table rebuilds, disk headroom, replication lag, and impact on backups/CDC. Test against production-like data and concurrent traffic.
|
|
14
|
+
|
|
15
|
+
## Backfill engineering
|
|
16
|
+
Use keyset/range pagination rather than large OFFSET scans. Bound batch size, concurrency, transaction duration, and replication impact. Persist checkpoints and make each batch restartable. Avoid uncoordinated dual writes; define an authoritative path and reconciliation.
|
|
17
|
+
|
|
18
|
+
## Cutover and rollback
|
|
19
|
+
Use feature flags, shadow reads, comparison metrics, canaries, and progressive traffic shifts. Define rollback triggers and whether rollback remains possible after writes begin in the new schema. Defer irreversible contract steps until confidence and recovery evidence meet explicit gates.
|
|
20
|
+
|
|
21
|
+
## Validation checklist
|
|
22
|
+
- Schema compatibility across deployed application versions.
|
|
23
|
+
- Query plans and index behavior at expected cardinality.
|
|
24
|
+
- Backfill throughput, lag, retries, and idempotency.
|
|
25
|
+
- Data invariants, counts, checksums, and reconciliation.
|
|
26
|
+
- Lock duration, replication/CDC lag, disk, WAL/binlog, and backup capacity.
|
|
27
|
+
- Cutover, rollback, restore, and post-migration cleanup.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# API Gateway and Edge Architecture
|
|
2
|
+
|
|
3
|
+
## Responsibilities and boundaries
|
|
4
|
+
An API gateway is an edge policy and routing layer, not a replacement for domain services. Define which concerns belong at the edge (TLS termination, routing, authentication token validation, coarse rate limits, request size limits, protocol adaptation, and telemetry) and which remain in services (object-level authorization, business quotas, domain validation, transaction semantics, and authoritative decisions).
|
|
5
|
+
|
|
6
|
+
## Topology choices
|
|
7
|
+
Compare a managed gateway, ingress controller, service-mesh ingress, reverse proxy, and application BFF. Evaluate latency overhead, availability dependencies, policy expressiveness, multi-region behavior, deployment ownership, observability, and cost. Avoid stacking multiple gateways that each retry, buffer, or transform requests without a clear contract.
|
|
8
|
+
|
|
9
|
+
## Routing and resilience
|
|
10
|
+
Specify host/path/header routing, versioned routes, canary weights, health-based endpoint selection, connection draining, deadlines, retry budgets, circuit breakers, and load shedding. Retry only safe or explicitly idempotent operations; enforce one end-to-end deadline rather than multiplying per-hop timeouts. Bound request/response buffering and concurrent connections.
|
|
11
|
+
|
|
12
|
+
## Security
|
|
13
|
+
Terminate TLS with a documented trust model; validate issuer, audience, signature, expiry, and key rotation for tokens. Strip untrusted forwarding headers and reconstruct client identity only from trusted proxies. Apply WAF rules as defense in depth, not as authorization. Prevent request smuggling by consistent HTTP parsing and hop-by-hop header handling. Keep secrets out of logs.
|
|
14
|
+
|
|
15
|
+
## Operations and validation
|
|
16
|
+
Track route-level request rate, status, latency percentiles, upstream saturation, retries, rejected traffic, connection errors, and policy changes. Test gateway failure, upstream brownouts, slow clients, oversized payloads, certificate rotation, and configuration rollback. Gateway availability is part of the critical path; define its SLO and recovery plan.
|
|
17
|
+
|
|
18
|
+
## Decision checklist
|
|
19
|
+
- [ ] Edge responsibilities and service responsibilities are explicit.
|
|
20
|
+
- [ ] Trust proxy chain and client-IP derivation are documented.
|
|
21
|
+
- [ ] Deadlines, retries, payload limits, and connection limits are bounded.
|
|
22
|
+
- [ ] Route changes support staged rollout and fast rollback.
|
|
23
|
+
- [ ] Gateway failure and regional failover have been exercised.
|
|
@@ -0,0 +1,23 @@
|
|
|
1
|
+
# REST API Design and Lifecycle
|
|
2
|
+
|
|
3
|
+
## Resource and contract design
|
|
4
|
+
Model stable resources and relationships; use HTTP methods and status codes consistently. Define canonical identifiers, filtering, sorting, pagination, partial updates, field masks, and error envelopes. Prefer cursor pagination for mutable or large datasets; document cursor stability and expiry. Avoid leaking storage schemas as public contracts.
|
|
5
|
+
|
|
6
|
+
## Correctness
|
|
7
|
+
Use conditional requests (ETag/If-Match) or explicit versions to prevent lost updates. Define idempotency-key scope, retention, payload binding, concurrent duplicate behavior, and replay response semantics for retryable creates or payments. A client timeout leaves the outcome unknown: provide a way to query operation status or safely retry with the same key.
|
|
8
|
+
|
|
9
|
+
## Compatibility and evolution
|
|
10
|
+
Treat additive changes as potentially breaking when clients use strict decoders or exhaustive enums. Version only when compatibility policy requires it; define deprecation notice, telemetry, support window, sunset criteria, and owner. Maintain consumer contract tests and generated-client compatibility checks. Document stable versus experimental fields.
|
|
11
|
+
|
|
12
|
+
## Security and usability
|
|
13
|
+
Specify authentication, object- and function-level authorization, tenant scoping, input validation, mass-assignment defenses, rate limits, and data classification. Use consistent machine-readable errors with correlation IDs, but never expose stack traces or secrets. Define CORS deliberately; it is not an authorization control.
|
|
14
|
+
|
|
15
|
+
## Operations
|
|
16
|
+
Publish latency and availability objectives per critical operation. Set request/response limits, pagination caps, deadlines, and retry guidance. Instrument business outcomes as well as HTTP status. Test concurrent updates, duplicate requests, stale ETags, malformed inputs, and old-client compatibility.
|
|
17
|
+
|
|
18
|
+
## Review checklist
|
|
19
|
+
- [ ] OpenAPI or equivalent contract is version-controlled and validated.
|
|
20
|
+
- [ ] Idempotency and concurrency semantics are explicit.
|
|
21
|
+
- [ ] Pagination, errors, and filtering are bounded and deterministic.
|
|
22
|
+
- [ ] AuthZ is enforced at object and function level.
|
|
23
|
+
- [ ] Deprecation and compatibility are observable and owned.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# GraphQL Architecture and Governance
|
|
2
|
+
|
|
3
|
+
## Schema and ownership
|
|
4
|
+
Design the graph around domain capabilities and consumer needs. Define field ownership, resolver ownership, nullability semantics, deprecation policy, schema composition, and federation boundaries. Avoid a graph that becomes a shared database facade with unclear authorization and change ownership.
|
|
5
|
+
|
|
6
|
+
## Query execution controls
|
|
7
|
+
GraphQL moves query shape to clients; it does not remove backend cost. Apply depth, complexity, alias, batch, and response-size limits. Use persisted/allowlisted operations for high-risk or mobile clients where appropriate. Prevent N+1 access with request-scoped DataLoader-style batching while preserving tenant and authorization context. Bound resolver concurrency and downstream fan-out.
|
|
8
|
+
|
|
9
|
+
## Authorization and data protection
|
|
10
|
+
Enforce authorization in resolvers or domain services for every object and field; never rely solely on hiding schema fields. Prevent cross-tenant cache leakage by including principal/tenant and policy context in cache keys. Restrict introspection according to threat model, but do not treat disabling it as a substitute for access control. Redact sensitive fields from traces and error payloads.
|
|
11
|
+
|
|
12
|
+
## Federation and consistency
|
|
13
|
+
For federated graphs, define entity ownership, key stability, reference resolution, timeout budgets, partial failure semantics, and schema rollout ordering. Avoid cyclic dependencies and excessive cross-subgraph waterfalls. Treat composed schema checks as a release gate; test old and new subgraph combinations.
|
|
14
|
+
|
|
15
|
+
## Evaluation
|
|
16
|
+
Measure operation-level latency, resolver fan-out, cache hit ratio, downstream calls, error rate, and query-cost rejection. Test adversarial deep/aliased queries, partial data errors, null propagation, authorization bypass, and schema compatibility. Compare GraphQL with REST/BFF for actual client needs rather than assuming one is universally superior.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# gRPC and Service Contracts
|
|
2
|
+
|
|
3
|
+
## Protocol fit
|
|
4
|
+
Use gRPC when typed contracts, efficient binary transport, streaming, generated clients, or polyglot service communication justify the operational and browser-compatibility costs. Compare REST/JSON, gRPC, and asynchronous messaging by consumer environment, debugging, proxy support, latency, and evolution requirements.
|
|
5
|
+
|
|
6
|
+
## Protobuf evolution
|
|
7
|
+
Treat field numbers as permanent identifiers: never reuse deleted numbers or names; reserve them. Add fields compatibly, define defaults and presence semantics, and avoid changing field meaning or wire type. Enums need an unspecified zero value and tolerant handling of unknown values. Validate compatibility in CI across supported producer/consumer versions.
|
|
8
|
+
|
|
9
|
+
## RPC semantics
|
|
10
|
+
Define deadlines, cancellation propagation, status codes, retryable conditions, idempotency, metadata limits, message size limits, and streaming flow control. A transport retry can duplicate a side effect; retries must respect method semantics and an end-to-end budget. Distinguish application errors from transport failures.
|
|
11
|
+
|
|
12
|
+
## Security and operations
|
|
13
|
+
Use workload identity and mTLS where appropriate; authorize every method and resource. Protect reflection and health endpoints according to exposure. Instrument method-level latency, status, message sizes, active streams, cancellations, and retry attempts. Test rolling upgrades, proxy/load-balancer behavior, slow consumers, stream resets, and mixed-version compatibility.
|
|
14
|
+
|
|
15
|
+
## Streaming
|
|
16
|
+
For client/server or bidirectional streams, define message ordering, backpressure, heartbeats, idle timeout, reconnect/resume tokens, replay boundaries, and maximum stream duration. Do not assume a broken stream means the server did not process the last message.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# WebSocket, Server-Sent Events, and Real-Time Systems
|
|
2
|
+
|
|
3
|
+
## Transport selection
|
|
4
|
+
Choose request/response for ordinary interactions, Server-Sent Events (SSE) for one-way server-to-client updates, and WebSocket for bidirectional low-latency sessions. Consider WebRTC for peer media/data only when its NAT traversal and media semantics are needed. Evaluate browser/proxy compatibility, connection duration, fan-out, and operational complexity.
|
|
5
|
+
|
|
6
|
+
## Connection architecture
|
|
7
|
+
Model connection establishment, authentication, authorization, heartbeat, idle timeout, reconnect, session affinity, regional routing, and graceful drain. Keep connection state disposable where possible; store durable business state outside gateway memory. Use pub/sub or partitioned brokers to distribute events to connection-owning nodes, with explicit delivery and ordering guarantees.
|
|
8
|
+
|
|
9
|
+
## Backpressure and scale
|
|
10
|
+
Bound per-connection outbound queues, message size, fan-out, subscriptions, and total concurrent connections. Define slow-consumer policy (drop, coalesce, disconnect, or durable replay) by message class. Capacity-plan file descriptors, memory per connection, heartbeat traffic, TLS, broker throughput, and reconnect storms.
|
|
11
|
+
|
|
12
|
+
## Delivery semantics
|
|
13
|
+
Treat live transports as best-effort unless backed by durable offsets or application acknowledgements. Include event IDs, sequence numbers, resume cursors, deduplication, and snapshot-plus-delta recovery when missed updates matter. Define whether ordering is per entity, room, partition, or connection; avoid claiming global ordering without a mechanism.
|
|
14
|
+
|
|
15
|
+
## Security and reliability
|
|
16
|
+
Authorize subscriptions and every sensitive message; revalidate long-lived sessions and revoke access after permission changes. Validate Origin for browser WebSockets, protect against cross-site WebSocket hijacking, enforce quotas, and avoid logging payloads containing PII. Test node loss, broker partitions, reconnect storms, slow consumers, and regional failover.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Real-Time Collaboration: OT and CRDT
|
|
2
|
+
|
|
3
|
+
## Problem framing
|
|
4
|
+
Define collaboration semantics before choosing an algorithm: document model, concurrent edit types, offline duration, expected participants, latency, conflict tolerance, undo/redo, presence, and persistence. Presence and cursors are ephemeral; document operations and snapshots are durable domain state.
|
|
5
|
+
|
|
6
|
+
## Operational Transformation
|
|
7
|
+
OT transforms concurrent operations against one another and depends on a well-defined transformation function, ordering authority, and server/client protocol. Specify revision numbers, operation identity, acknowledgements, reconnect catch-up, and transformation correctness. Test convergence across permutations and stale clients.
|
|
8
|
+
|
|
9
|
+
## Conflict-free Replicated Data Types
|
|
10
|
+
CRDTs encode merge behavior so replicas converge under stated assumptions. Choose a data type whose semantics match the product; convergence does not guarantee intuitive user intent, bounded metadata, or low storage cost. Model tombstones/garbage collection, causal metadata, offline replicas, compaction, and schema evolution. State whether the design is state-based or operation-based and its delivery assumptions.
|
|
11
|
+
|
|
12
|
+
## Persistence and security
|
|
13
|
+
Persist operations or snapshots with atomic revision/checkpoint boundaries. Make operation application idempotent and validate authorization at the document/operation boundary. Enforce tenant isolation, document ACL changes, retention, export, and deletion across replicas and backups. Do not trust client-generated operations.
|
|
14
|
+
|
|
15
|
+
## Testing and trade-offs
|
|
16
|
+
Use property-based and randomized concurrency tests, network partitions, duplicate/reordered delivery, offline edits, and mixed-version clients. Measure convergence time, metadata growth, sync bandwidth, conflict UX, and recovery. Choose centralized sequencing when it is simpler and meets offline/latency requirements; CRDT is not a default requirement for every collaborative feature.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# API Security, Rate Limiting, and Abuse Prevention
|
|
2
|
+
|
|
3
|
+
## Threat model
|
|
4
|
+
Cover broken object/function authorization, credential stuffing, scraping, enumeration, replay, resource exhaustion, injection, SSRF, request smuggling, webhook spoofing, and tenant escape. Map controls to principals, endpoints, resources, and business impact. Authentication alone does not establish permission to access an object.
|
|
5
|
+
|
|
6
|
+
## Rate-limit dimensions
|
|
7
|
+
Apply limits at appropriate layers: global protection, per-IP or network, per-credential, per-user, per-tenant, per-route, and expensive-operation budgets. Combine request rate with concurrency, cost units, payload size, and burst allowance. Avoid IP-only controls for NAT/mobile users and avoid user-only controls that permit anonymous abuse.
|
|
8
|
+
|
|
9
|
+
## Algorithms and distributed enforcement
|
|
10
|
+
Compare token bucket, leaky bucket, fixed window, and sliding window. Document burst semantics, clock assumptions, key cardinality, storage consistency, and failure mode. Distributed counters may be approximate; state whether enforcement is best-effort or strict. Use local emergency limits when the shared limiter is unavailable, and avoid a limiter dependency that becomes a single point of failure.
|
|
11
|
+
|
|
12
|
+
## Fairness and degradation
|
|
13
|
+
Define tenant tiers, weighted fairness, priority classes, retry-after behavior, and quota reset semantics. Return actionable 429/503 responses without leaking internal policy. Apply load shedding and queue admission controls to protect critical traffic. Avoid retry storms by communicating backoff and enforcing client retry budgets.
|
|
14
|
+
|
|
15
|
+
## Abuse operations
|
|
16
|
+
Monitor rejected requests, credential anomalies, expensive query patterns, cost per tenant, and false positives. Build alerting, investigation, appeal/override, and emergency block workflows. Test bypass attempts, distributed bursts, hot keys, limiter outage, clock skew, and noisy-neighbor isolation. Rate limiting complements authorization, WAF, and business fraud controls; it does not replace them.
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
# OAuth 2.1, OpenID Connect, and Passkeys
|
|
2
|
+
|
|
3
|
+
## Protocol roles
|
|
4
|
+
OAuth is an authorization framework; OpenID Connect adds an identity layer. Distinguish the authorization server, resource server, client, subject, and relying party. Use Authorization Code with PKCE for interactive public clients; do not embed client secrets in mobile or browser applications. Avoid implicit and resource-owner-password flows. Verify current standards and provider-specific behavior before deployment.
|
|
5
|
+
|
|
6
|
+
## Token validation and lifecycle
|
|
7
|
+
Validate signature using trusted issuer metadata and rotating keys, issuer, audience, expiry, not-before, authorized party where applicable, and allowed algorithms. Access tokens should be audience-restricted and short-lived according to risk. Define refresh-token rotation, reuse detection, revocation, session termination, and key rollover. Do not treat an ID token as an API access token.
|
|
8
|
+
|
|
9
|
+
## Redirects and browser security
|
|
10
|
+
Use exact registered redirect URIs, state binding, nonce for OIDC authentication, PKCE, and strict callback handling. Protect cookies with Secure, HttpOnly, and appropriate SameSite attributes. Defend against open redirects, login CSRF, token leakage through URLs, and unsafe post-logout redirects.
|
|
11
|
+
|
|
12
|
+
## Passkeys and WebAuthn
|
|
13
|
+
Passkeys use public-key credentials scoped to a relying party. Verify challenge freshness and one-time use, origin, RP ID, user presence/verification policy, credential ownership, and signature counter behavior according to authenticator capabilities. Design recovery, device replacement, account linking, and phishing-resistant step-up flows; passkeys do not eliminate account-recovery risk.
|
|
14
|
+
|
|
15
|
+
## Review checklist
|
|
16
|
+
- [ ] OAuth and OIDC responsibilities are separated correctly.
|
|
17
|
+
- [ ] PKCE, state, nonce, exact redirect URI, issuer and audience checks are enforced.
|
|
18
|
+
- [ ] Refresh rotation, revocation, and key rotation are tested.
|
|
19
|
+
- [ ] Passkey registration, assertion, origin, challenge, and recovery flows are tested.
|
|
20
|
+
- [ ] Tokens and authentication codes never enter logs, analytics, or referrers.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Identity and Access Management: RBAC, ABAC, and ReBAC
|
|
2
|
+
|
|
3
|
+
## Authorization model selection
|
|
4
|
+
RBAC assigns permissions through roles and suits stable organizational responsibilities. ABAC evaluates subject, resource, action, and environment attributes. ReBAC evaluates relationships such as owner, member, parent, and delegated access. Real systems may combine models, but policy composition and precedence must be explicit.
|
|
5
|
+
|
|
6
|
+
## Policy design
|
|
7
|
+
Define principal types, resource hierarchy, actions, scopes, tenant boundaries, deny/allow precedence, default-deny behavior, and delegation. Separate authentication from authorization and coarse gateway checks from authoritative domain checks. Enforce object-level authorization on every read, write, export, and background operation.
|
|
8
|
+
|
|
9
|
+
## Policy evaluation
|
|
10
|
+
Centralized policy engines improve consistency but add latency and availability dependencies. Define policy distribution, cache TTL, invalidation, fail-open/fail-closed behavior, versioning, and explainability. Cache keys must include principal, tenant, resource, action, relevant attributes, and policy version. Avoid stale grants after role removal or ACL changes.
|
|
11
|
+
|
|
12
|
+
## Lifecycle and governance
|
|
13
|
+
Model joiner/mover/leaver workflows, periodic access reviews, privileged access, service accounts, break-glass access, separation of duties, and audit evidence. Prefer time-bound grants and just-in-time elevation. Reconcile identity-provider groups with application permissions; group membership alone is not a complete authorization decision.
|
|
14
|
+
|
|
15
|
+
## Testing
|
|
16
|
+
Test horizontal and vertical privilege escalation, cross-tenant access, inherited permissions, revoked access, policy conflicts, stale caches, and background jobs executing under the wrong identity. Use policy unit tests and representative end-to-end authorization matrices.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Zero Trust and Workload Identity
|
|
2
|
+
|
|
3
|
+
## Trust model
|
|
4
|
+
Zero Trust removes implicit trust based on network location. Authenticate and authorize each workload-to-workload interaction using explicit identity, resource, action, and context. Network segmentation remains useful defense in depth, not proof of identity.
|
|
5
|
+
|
|
6
|
+
## Workload identity
|
|
7
|
+
Prefer short-lived, automatically rotated identities bound to workload properties and environment. Use SPIFFE/SPIRE or cloud-native workload identity where appropriate; compare federation, operational maturity, trust-domain boundaries, and portability. Avoid long-lived static service credentials and shared identities that erase attribution.
|
|
8
|
+
|
|
9
|
+
## Service-to-service controls
|
|
10
|
+
Use mTLS for peer authentication and transport protection where suitable, but still authorize methods and resources. Define certificate issuance, trust bundles, rotation overlap, revocation, identity mapping, and failure behavior. Prevent confused-deputy behavior by constraining audience and delegated authority.
|
|
11
|
+
|
|
12
|
+
## Policy and segmentation
|
|
13
|
+
Apply least privilege at service, method, data, and administrative boundaries. Combine identity-aware policy with network policy, egress restrictions, and workload isolation. Define policy propagation and emergency revocation latency.
|
|
14
|
+
|
|
15
|
+
## Operations
|
|
16
|
+
Measure authentication failures, certificate expiry, identity issuance latency, policy denials, and trust-bundle drift. Test clock skew, CA outage, certificate rotation, compromised workload containment, failover, and recovery. Document bootstrap trust and break-glass procedures.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Secrets Management, KMS, PKI, and Key Management
|
|
2
|
+
|
|
3
|
+
## Secret lifecycle
|
|
4
|
+
Inventory secrets, owners, consumers, environments, purpose, classification, and expiry. Prefer workload identity and dynamic credentials over static secrets. Centralize issuance, access policy, rotation, revocation, and audit; avoid distributing a universal root secret to every service.
|
|
5
|
+
|
|
6
|
+
## Cryptographic key hierarchy
|
|
7
|
+
Separate data-encryption keys from key-encryption keys. Use envelope encryption and managed KMS/HSM capabilities where appropriate. Define key purpose, algorithm, allowed operations, region, rotation, backup, recovery, deletion, and separation of duties. Key rotation does not automatically re-encrypt all historical data.
|
|
8
|
+
|
|
9
|
+
## PKI operations
|
|
10
|
+
Define root and intermediate CA custody, issuance policy, certificate identity, validity, renewal, revocation, trust distribution, and emergency rollover. Protect CA signing keys with stronger controls than ordinary application secrets. Plan overlapping trust during rotations and verify client behavior under expired or unknown certificates.
|
|
11
|
+
|
|
12
|
+
## Application integration
|
|
13
|
+
Avoid secrets in source, images, build logs, crash dumps, environment diagnostics, and telemetry. Use narrowly scoped runtime retrieval with bounded caching and fail-safe behavior. Do not log secret values when reporting access failures.
|
|
14
|
+
|
|
15
|
+
## Resilience and audit
|
|
16
|
+
KMS and secret stores are critical dependencies: model quotas, regional availability, latency, caching, outage behavior, and recovery. Test restore and key-loss scenarios before relying on encrypted backups. Audit key use, policy changes, export, deletion, and break-glass operations.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Software Supply Chain Security: SBOM, Provenance, and SLSA
|
|
2
|
+
|
|
3
|
+
## Threat model
|
|
4
|
+
Cover dependency confusion, typosquatting, compromised maintainers, malicious build steps, poisoned artifacts, leaked signing keys, untrusted registries, and release pipeline compromise. Map trust boundaries from source and dependencies through build, signing, registry, deployment, and runtime.
|
|
5
|
+
|
|
6
|
+
## Dependency governance
|
|
7
|
+
Pin and review dependencies using lockfiles, trusted registries, provenance where available, vulnerability and license scanning, and controlled update automation. Pinning alone does not establish trust. Define exceptions, remediation SLAs, ownership, and reachability-based prioritization.
|
|
8
|
+
|
|
9
|
+
## SBOM and provenance
|
|
10
|
+
Generate machine-readable SBOMs for shipped artifacts and preserve component identity and version. Attest build provenance with authenticated builder identity, source revision, build parameters, and artifact digest. Verify attestations at promotion/deployment; do not trust unsigned metadata supplied beside an artifact.
|
|
11
|
+
|
|
12
|
+
## Build integrity
|
|
13
|
+
Use isolated ephemeral builders, least-privilege tokens, protected branches, reviewed workflows, reproducible or independently verifiable builds where feasible, and separation between build and release authority. Protect signing keys and enforce artifact immutability.
|
|
14
|
+
|
|
15
|
+
## Release and response
|
|
16
|
+
Define artifact signing, verification, promotion, rollback, and incident procedures for compromised dependencies or builders. Test provenance verification and quarantine paths. Treat SLSA levels as a framework for incremental assurance, not a blanket security certification.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Threat Modeling: STRIDE, Attack Trees, and Risk Analysis
|
|
2
|
+
|
|
3
|
+
## Scope and assets
|
|
4
|
+
Record system boundaries, data flows, stores, actors, trust boundaries, entry points, sensitive assets, and security objectives. Include control planes, CI/CD, support tooling, third parties, and operational access—not only runtime APIs.
|
|
5
|
+
|
|
6
|
+
## Analysis methods
|
|
7
|
+
Use STRIDE to systematically identify spoofing, tampering, repudiation, information disclosure, denial of service, and elevation of privilege. Use attack trees to decompose attacker goals into feasible paths. Supplement with abuse cases, misuse cases, and privacy harms. No single taxonomy replaces domain expertise.
|
|
8
|
+
|
|
9
|
+
## Risk prioritization
|
|
10
|
+
Describe each scenario as actor, preconditions, attack path, affected asset, impact, likelihood rationale, existing controls, and residual risk. Use a consistent scoring model and distinguish technical severity from business risk. Avoid false precision in numeric scores.
|
|
11
|
+
|
|
12
|
+
## Mitigation and verification
|
|
13
|
+
Prefer eliminating or reducing attack surface, then enforce preventive controls, detection, and recovery. Assign each mitigation an owner, due date, and verification test. Track accepted risks with an approver and expiry/review date.
|
|
14
|
+
|
|
15
|
+
## Continuous lifecycle
|
|
16
|
+
Revisit threat models when architecture, trust boundaries, data classes, dependencies, deployment, or attacker capabilities change. Validate assumptions with security tests, tabletop exercises, and incident learnings.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Privacy Engineering and Data Governance
|
|
2
|
+
|
|
3
|
+
## Data inventory and purpose
|
|
4
|
+
Map personal and sensitive data across collection, inference, processing, sharing, storage, logs, analytics, backups, and deletion. Document purpose, lawful basis where applicable, data owner, recipients, retention, residency, and user expectations. Minimize collection and avoid retaining data merely because storage is cheap.
|
|
5
|
+
|
|
6
|
+
## Privacy by design
|
|
7
|
+
Use purpose limitation, minimization, transparency, access controls, pseudonymization, encryption, and privacy-preserving defaults. Assess re-identification risk in supposedly de-identified datasets and telemetry. Treat derived attributes and embeddings as potentially sensitive when they can reveal or link information.
|
|
8
|
+
|
|
9
|
+
## Lifecycle controls
|
|
10
|
+
Define retention schedules, legal holds, correction, export, deletion, backup expiry, replica propagation, and downstream processor obligations. A deletion request is incomplete if data remains searchable in caches, indexes, analytics, or derived stores without a documented policy.
|
|
11
|
+
|
|
12
|
+
## Governance and accountability
|
|
13
|
+
Assign data stewards, classification, quality rules, lineage, access approval, sharing agreements, and incident ownership. Apply residency and cross-border transfer controls based on applicable law and contractual commitments; obtain qualified legal review rather than inferring legal compliance from technical controls.
|
|
14
|
+
|
|
15
|
+
## Verification
|
|
16
|
+
Test data discovery, access logs, retention enforcement, deletion propagation, backup expiration, subject-access workflows, and least-privilege analytics. Measure policy exceptions and stale data inventories.
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
# OpenTelemetry Architecture and Instrumentation
|
|
2
|
+
|
|
3
|
+
## Purpose and scope
|
|
4
|
+
OpenTelemetry (OTel) provides vendor-neutral APIs, SDKs, semantic conventions, and telemetry pipelines for traces, metrics, and logs. Treat it as an instrumentation and data-delivery architecture, not as an observability product or a guarantee that telemetry is correct.
|
|
5
|
+
|
|
6
|
+
## Architecture decisions
|
|
7
|
+
- Define signal ownership and collection paths: application SDK/auto-instrumentation → OTLP → Collector gateway/agent → processors/exporters → backend.
|
|
8
|
+
- Choose agent, sidecar, gateway, or direct export based on deployment model, network boundaries, buffering needs, tenancy, and operational ownership.
|
|
9
|
+
- Prefer OTLP and stable semantic conventions where supported. Pin SDK, Collector, and instrumentation versions; verify stability status before adopting conventions.
|
|
10
|
+
- Separate application instrumentation from vendor-specific exporters to preserve portability.
|
|
11
|
+
|
|
12
|
+
## Instrumentation design
|
|
13
|
+
- Instrument user journeys and service boundaries; add spans around meaningful operations, not every function.
|
|
14
|
+
- Record HTTP/RPC/database/messaging operations with consistent service.name, service.version, deployment.environment, and resource attributes.
|
|
15
|
+
- Propagate W3C Trace Context across synchronous and asynchronous boundaries. Define behavior for queues, batch jobs, scheduled work, and context loss.
|
|
16
|
+
- Use span events and links for meaningful milestones and causal relationships; do not model every event as a child span.
|
|
17
|
+
- Use histograms for latency distributions; counters for monotonic event totals; gauges/observable instruments for sampled state. Avoid high-cardinality labels.
|
|
18
|
+
- Correlate logs with trace_id and span_id where available. Keep logs useful when no trace exists.
|
|
19
|
+
|
|
20
|
+
## Collector pipeline
|
|
21
|
+
- Define receiver protocols, authentication, TLS/mTLS, tenant routing, memory limits, batching, queueing, retry, and exporter timeouts.
|
|
22
|
+
- Apply filtering, attribute normalization, redaction, sampling, and routing deliberately. Processors must not silently remove fields needed for incident response.
|
|
23
|
+
- Size queues and memory limiters for bursts; monitor dropped data, export failures, queue occupancy, and retry exhaustion.
|
|
24
|
+
- Protect the Collector as a production service: capacity, HA, upgrades, configuration rollout, and failure isolation matter.
|
|
25
|
+
|
|
26
|
+
## Sampling and data governance
|
|
27
|
+
- Use head sampling for inexpensive early decisions; use tail sampling when outcomes or errors determine retention. Tail sampling requires trace affinity and bounded state.
|
|
28
|
+
- Define separate policies for errors, slow requests, critical business journeys, and routine traffic. Validate that parent/child spans remain coherent.
|
|
29
|
+
- Treat telemetry as potentially sensitive data. Minimize attributes, redact secrets and PII, restrict access, define retention, and document residency.
|
|
30
|
+
- Never put user IDs, request IDs, raw URLs, SQL literals, or arbitrary error text in metric labels.
|
|
31
|
+
|
|
32
|
+
## Validation and operations
|
|
33
|
+
- Test instrumentation with representative traffic and verify propagation across services and queues.
|
|
34
|
+
- Monitor telemetry pipeline health independently of the application: ingest rate, export success, dropped spans/metrics/logs, queue age, memory, and backend rejection.
|
|
35
|
+
- Use synthetic spans and known test requests to detect broken propagation and missing instrumentation.
|
|
36
|
+
- Establish telemetry cost budgets and retention tiers; telemetry volume is a workload and a cost center.
|
|
37
|
+
|
|
38
|
+
## Anti-patterns
|
|
39
|
+
- Assuming auto-instrumentation captures business semantics or all async context.
|
|
40
|
+
- Exporting directly from every workload without a failure and backpressure strategy.
|
|
41
|
+
- Treating trace sampling as statistically neutral for every question.
|
|
42
|
+
- Using unbounded attributes or cardinality as a substitute for useful dimensions.
|
|
43
|
+
|
|
44
|
+
## Design checklist
|
|
45
|
+
- [ ] Signal coverage and ownership are defined for critical journeys.
|
|
46
|
+
- [ ] Resource attributes and semantic conventions are consistent.
|
|
47
|
+
- [ ] Context propagation is tested across process and messaging boundaries.
|
|
48
|
+
- [ ] Collector topology, security, buffering, and failure behavior are explicit.
|
|
49
|
+
- [ ] Sampling, redaction, retention, access, and cost policies are documented.
|
|
50
|
+
- [ ] Pipeline loss and instrumentation regressions are observable.
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# Distributed Tracing and Context Propagation
|
|
2
|
+
|
|
3
|
+
## Trace model
|
|
4
|
+
A trace is a causal graph of spans, not necessarily a single linear tree. Define trace boundaries, parent-child relationships, span links, baggage policy, and the behavior of asynchronous work before interpreting latency or causality.
|
|
5
|
+
|
|
6
|
+
## Propagation
|
|
7
|
+
- Standardize W3C Trace Context (traceparent and tracestate) where interoperable; explicitly document legacy formats and translation boundaries.
|
|
8
|
+
- Propagate context through HTTP, RPC, message headers, job metadata, and workflow state. Verify framework and broker instrumentation behavior for deployed versions.
|
|
9
|
+
- For fan-out/fan-in, preserve causal relationships. Use span links when work is initiated by multiple messages or has multiple causal parents.
|
|
10
|
+
- Baggage is propagated metadata, not a trusted identity or authorization claim. Allowlist keys, constrain size, and exclude secrets and PII.
|
|
11
|
+
- At trust boundaries, validate and optionally restart trace context to prevent untrusted callers from forging internal trace relationships or injecting excessive baggage.
|
|
12
|
+
|
|
13
|
+
## Latency interpretation
|
|
14
|
+
- Distinguish wall-clock end-to-end latency from summed span durations. Concurrent spans overlap; summing them overstates request duration.
|
|
15
|
+
- Identify queue wait, network time, server processing, lock waits, retries, and downstream dependency time separately where instrumentation permits.
|
|
16
|
+
- Check clock skew before interpreting cross-host timestamps. Prefer duration measurements from a single process clock for local spans.
|
|
17
|
+
- A missing span can mean instrumentation gaps, sampling, propagation loss, or exporter/backend loss; do not infer that the operation did not happen.
|
|
18
|
+
|
|
19
|
+
## Sampling
|
|
20
|
+
- Define head/tail sampling by service criticality, error status, latency, tenant, and cost.
|
|
21
|
+
- Tail sampling needs trace affinity, bounded decision windows, memory limits, and explicit behavior when late spans arrive.
|
|
22
|
+
- Ensure sampling decisions are consistent across services or understand why partial traces are expected.
|
|
23
|
+
- Preserve representative baseline traffic; error-only retention cannot measure normal latency distributions.
|
|
24
|
+
|
|
25
|
+
## Security and privacy
|
|
26
|
+
- Do not expose credentials, request bodies, raw query strings, or sensitive identifiers in span attributes.
|
|
27
|
+
- Sanitize untrusted header values and cap attribute/event sizes.
|
|
28
|
+
- Restrict trace search and export by tenant and role. Consider cross-tenant leakage through shared trace IDs and query interfaces.
|
|
29
|
+
|
|
30
|
+
## Troubleshooting workflow
|
|
31
|
+
1. Start from a user-visible symptom and identify the affected journey, time window, and cohort.
|
|
32
|
+
2. Compare golden signals and SLO burn against baseline.
|
|
33
|
+
3. Inspect representative traces, including successful and failed requests.
|
|
34
|
+
4. Separate application time, queue wait, retries, and dependency time.
|
|
35
|
+
5. Correlate with deploys, configuration changes, saturation, and dependency incidents.
|
|
36
|
+
6. Validate the hypothesis with logs, metrics, profiles, or a controlled experiment.
|
|
37
|
+
|
|
38
|
+
## Checklist
|
|
39
|
+
- [ ] Propagation covers sync and async boundaries.
|
|
40
|
+
- [ ] Baggage is bounded, allowlisted, and non-sensitive.
|
|
41
|
+
- [ ] Sampling preserves useful normal and exceptional traffic.
|
|
42
|
+
- [ ] Trace gaps and partial traces are understood.
|
|
43
|
+
- [ ] Latency interpretation accounts for concurrency and clock behavior.
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# Continuous Profiling and Performance Diagnostics
|
|
2
|
+
|
|
3
|
+
## Purpose
|
|
4
|
+
Profiling explains where CPU, memory, allocation, lock, and execution time are spent. Use it to test a specific performance hypothesis; do not treat a flame graph as a root cause without workload and runtime context.
|
|
5
|
+
|
|
6
|
+
## Profile types
|
|
7
|
+
- CPU profiles: sampled on-CPU execution; distinguish hot code from time blocked on I/O.
|
|
8
|
+
- Heap/allocation profiles: retained objects, allocation rate, and allocation hot spots.
|
|
9
|
+
- Wall-clock profiles: on-CPU plus off-CPU waiting, subject to profiler/runtime support.
|
|
10
|
+
- Lock and contention profiles: mutex waits, blocked goroutines/threads, and synchronization bottlenecks.
|
|
11
|
+
- Runtime profiles: GC pauses, scheduling, thread pools, JIT behavior, and event-loop stalls.
|
|
12
|
+
|
|
13
|
+
## Method
|
|
14
|
+
1. Define the affected endpoint/job, service version, environment, workload, and target metric.
|
|
15
|
+
2. Capture representative profiles under controlled and production-safe conditions.
|
|
16
|
+
3. Compare baseline and candidate profiles using equivalent traffic, data shape, warm-up, and runtime settings.
|
|
17
|
+
4. Attribute costs to code paths; correlate with traces, request mix, allocation rates, GC, and saturation.
|
|
18
|
+
5. Change one material factor at a time and repeat the measurement.
|
|
19
|
+
6. Validate user-visible latency/throughput and resource cost after the change.
|
|
20
|
+
|
|
21
|
+
## Continuous profiling architecture
|
|
22
|
+
- Select agent, in-process, or eBPF-based collection according to language/runtime support, privilege boundaries, overhead, and operational model.
|
|
23
|
+
- Define sampling frequency, profile duration, upload cadence, symbolization, retention, and tenant isolation.
|
|
24
|
+
- Monitor profiler overhead, missing samples, upload failures, and storage cost.
|
|
25
|
+
- Restrict access to profiles: symbols, stack traces, and labels can reveal proprietary code paths or sensitive metadata.
|
|
26
|
+
|
|
27
|
+
## Performance diagnosis
|
|
28
|
+
- CPU saturation: inspect hot functions, serialization, compression, cryptography, regex, and inefficient algorithms.
|
|
29
|
+
- Memory pressure: inspect allocation churn, retained references, caches, fragmentation, and GC behavior.
|
|
30
|
+
- Lock contention: identify shared mutable state, coarse locks, hot keys, and queueing around critical sections.
|
|
31
|
+
- I/O waits: correlate profiles with traces, storage latency, connection pools, and network retransmissions.
|
|
32
|
+
- Tail latency: inspect rare paths, GC pauses, lock convoys, retries, cold caches, noisy neighbors, and dependency variance.
|
|
33
|
+
|
|
34
|
+
## Pitfalls
|
|
35
|
+
- Comparing profiles from different workloads or runtime configurations.
|
|
36
|
+
- Assuming sampled CPU time explains blocked time.
|
|
37
|
+
- Optimizing a hot function that contributes little to end-to-end latency.
|
|
38
|
+
- Ignoring profiler overhead, symbolization gaps, and production data governance.
|
|
39
|
+
|
|
40
|
+
## Checklist
|
|
41
|
+
- [ ] Hypothesis, workload, baseline, and target are recorded.
|
|
42
|
+
- [ ] Profile type matches the suspected bottleneck.
|
|
43
|
+
- [ ] Samples are representative and overhead is measured.
|
|
44
|
+
- [ ] Findings are correlated with traces and service metrics.
|
|
45
|
+
- [ ] Before/after results include user impact and resource cost.
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
# Load, Stress, Soak Testing, and Workload Modeling
|
|
2
|
+
|
|
3
|
+
## Workload model
|
|
4
|
+
Model production traffic rather than a single peak RPS number. Include arrival patterns, concurrency, request mix, payload sizes, user think time, read/write ratio, tenant skew, cache state, data volume, geographic distribution, and background jobs.
|
|
5
|
+
|
|
6
|
+
Distinguish:
|
|
7
|
+
- Open workload: arrivals continue independently of response time; useful for detecting queue growth and overload.
|
|
8
|
+
- Closed workload: a fixed population waits for responses before issuing more work; useful for concurrency-limited clients but can hide overload through coordinated omission.
|
|
9
|
+
- Burst workload: short high-rate arrivals and recovery periods.
|
|
10
|
+
- Stateful workload: realistic sessions, hot keys, cache behavior, and data growth.
|
|
11
|
+
|
|
12
|
+
## Test types
|
|
13
|
+
- Baseline/load: expected steady-state and forecast traffic.
|
|
14
|
+
- Stress: increase load to discover saturation, queue growth, and failure boundaries.
|
|
15
|
+
- Spike: abrupt arrival-rate changes and autoscaling response.
|
|
16
|
+
- Soak: sustained load to expose leaks, resource drift, compaction, GC, and backlog accumulation.
|
|
17
|
+
- Capacity breakpoint: find the highest load that still meets latency/error/SLO constraints with headroom.
|
|
18
|
+
- Resilience: combine dependency degradation and partial failures with representative load.
|
|
19
|
+
|
|
20
|
+
## Test validity
|
|
21
|
+
- Define arrival-rate model and concurrency separately.
|
|
22
|
+
- Prevent coordinated omission; use tools and metrics that account for delayed request scheduling.
|
|
23
|
+
- Validate generator capacity, network path, TLS, DNS, and client-side limits.
|
|
24
|
+
- Use production-like data distributions and data sizes; avoid testing only warm caches or tiny datasets.
|
|
25
|
+
- Include retries and background work, or explicitly isolate them.
|
|
26
|
+
- Warm up JITs, caches, and connection pools consistently; report warm-up and steady-state windows separately.
|
|
27
|
+
|
|
28
|
+
## Safety and controls
|
|
29
|
+
- Obtain explicit authorization for production tests. Prefer isolated environments and synthetic or anonymized data.
|
|
30
|
+
- Set hard limits for RPS, concurrency, duration, spend, and generated data.
|
|
31
|
+
- Define abort thresholds for error rate, latency, saturation, queue depth, and downstream impact.
|
|
32
|
+
- Coordinate with on-call owners and external providers; use allowlisted test identities and clear traffic tagging.
|
|
33
|
+
- Ensure cleanup and data reset are safe and repeatable.
|
|
34
|
+
|
|
35
|
+
## Reporting
|
|
36
|
+
Record build/configuration, infrastructure shape, dataset, workload profile, generator placement, test duration, warm-up, run IDs, and environmental differences. Report throughput, P50/P95/P99, errors/timeouts, saturation, queue depth/age, resource use, and SLO attainment. Include confidence, variance, and limitations.
|
|
37
|
+
|
|
38
|
+
## Checklist
|
|
39
|
+
- [ ] Workload represents realistic request mix and arrival behavior.
|
|
40
|
+
- [ ] Open/closed model and coordinated omission treatment are explicit.
|
|
41
|
+
- [ ] Data size, skew, cache state, and background work are representative.
|
|
42
|
+
- [ ] Generator and network are not the bottleneck.
|
|
43
|
+
- [ ] Abort controls and production authorization are defined.
|
|
44
|
+
- [ ] Results include tail latency, errors, saturation, and queue behavior.
|