perf-skills 2.0.0 → 4.0.0

package/skills/perf/references/tools/neoload.md

@@ -57,5 +57,59 @@ curl -X POST "https://neoload-api.tricentis.com/v3/tests/{testId}/start" \
 - Integrate with **Dynatrace/AppDynamics** via built-in APM connectors.
 - Use **as-code YAML definitions** (NeoLoad as Code) for version-controlled test configs.
 
+---
+
+## NeoLoad as Code (YAML)
+
+Version-control your test definitions instead of relying on the GUI `.nlp` file:
+
+```yaml
+# neoload-project.yml
+name: Checkout-Load-Test
+ populations:
+  - name: Checkout_Users
+    user_paths:
+      - name: Checkout_Flow
+        distribution: 70%
+      - name: Browse_Flow
+        distribution: 30%
+
+scenarios:
+  - name: Regression_Load
+    populations:
+      - name: Checkout_Users
+        load_policy:
+          steps:
+            - type: ramp
+              from: 0
+              to: 200
+              duration: 120s
+            - type: constant
+              value: 200
+              duration: 600s
+            - type: ramp
+              from: 200
+              to: 0
+              duration: 60s
+
+sla_profiles:
+  - name: Checkout_SLA
+    thresholds:
+      - per_transaction: Checkout
+        avg_response_time:
+          warn: 500ms
+          fail: 1000ms
+        error_rate:
+          warn: 1%
+          fail: 5%
+```
+
+```bash
+# Validate and run from CLI
+neoload-cli validate neoload-project.yml
+neoload-cli run --scenario Regression_Load
+```
+
 > For anti-patterns, assertions, think time, and parameterization principles, see **Key Principles** in `SKILL.md`.
 > For CI/CD integration details, see `../topics/test-execution.md`.
+> For SLO threshold design, see `../topics/slo-capacity.md`.

package/skills/perf/references/topics/correlation.md

@@ -404,3 +404,7 @@ If you need help configuring correlation, copy and paste the following prompt te
 3. Write the exact extraction syntax for the target tool.
 4. Show the exact injection syntax (including variable syntax, e.g., `${varName}`, `#{varName}`, `{varName}`).
 5. Explain how to debug and verify that the correlation succeeded.
+
+---
+
+> **See also:** Tool-specific extractor syntax in `../tools/jmeter.md`, `../tools/k6.md`, `../tools/gatling.md`, `../tools/locust.md`, `../tools/loadrunner.md`, `../tools/artillery.md`. Script-level correlation patterns in `script-generation.md`.

package/skills/perf/references/topics/database-testing.md

@@ -212,3 +212,5 @@ WHERE NOT bl.granted;
 - [ ] Replication lag monitoring active (if using read replicas)
 - [ ] Connection pool metrics exposed and dashboarded
 - [ ] Cleanup script ready for test data removal post-test
+
+> **See also:** JDBC sampler setup in `../tools/jmeter.md`. Monitoring and dashboards in `observability.md`. Interpreting saturation and bottleneck signals in `results-analysis.md`.

package/skills/perf/references/topics/llm-inference.md

@@ -0,0 +1,179 @@
+# LLM Inference Performance
+
+> Scope: load and capacity testing of LLM inference servers (vLLM, TRT-LLM, SGLang, OpenAI-compatible endpoints, Ray Serve LLM, KServe). Covers metric definitions, workload design, tools, and SLO methodology for generative AI.
+
+LLM inference is **not conventional HTTP load testing**. Responses stream token-by-token over an open connection, output length is unbounded, and the dominant cost is GPU compute, not network. A generic `k6`/`JMeter` run that only measures request latency will badly misreport performance - it ignores the streaming shape entirely. This topic defines the metrics that actually matter, how to design realistic LLM workloads, and which tools to use.
+
+> For general workload theory (Little's Law, open vs closed models), see `workload-design.md`. For analyzing the percentiles and SLO failures this produces, see `results-analysis.md`. For running these tests in CI/CD, see `test-execution.md`.
+
+---
+
+## Why LLM load testing is different
+
+| Aspect | Traditional API | LLM inference |
+|---|---|---|
+| Response | Single body, fixed size | Streamed tokens, variable length |
+| Latency signal | TTFB + total | TTFT + inter-token gaps + E2E |
+| Cost driver | CPU/network | GPU memory (KV cache), compute |
+| Throughput unit | Requests/sec | Tokens/sec (not requests/sec) |
+| Concurrency limit | Threads/sockets | KV cache capacity (`max_num_seqs`) |
+| Key risk | Timeouts | Queueing delay, GPU saturation, OOM |
+
+Two phases dominate every request:
+- **Prefill** - the model processes the full prompt to build the KV cache. Compute-heavy, determines **TTFT**. Scales with prompt length.
+- **Decode** - tokens generated one at a time using the KV cache. Memory-bandwidth-bound, determines **ITL/TPOT**. Scales with output length.
+
+---
+
+## Core Metrics
+
+### TTFT - Time to First Token
+Time from request send to first streamed token. Driven by prompt length, queue/self.time, and prefill speed. The primary *perceived responsiveness* metric for chat/coding assistants. High TTFT under load usually means the scheduler is queuing requests (KV cache exhausted) rather than a slow model.
+
+### TPOT / ITL - Generation smoothness
+- **TPOT (Time Per Output Token)** - average gap between tokens for a single request: `(E2E - TTFT) / (output_tokens - 1)`. Report the mean of per-request TPOTs.
+- **ITL (Inter-Token Latency)** - same gaps, but token-weighted across all requests (mean of every gap). This is the system's steady streaming speed.
+- **Which to use**: TPOT compares per-request behavior (each request equal); ITL estimates system-wide streaming feel across mixed traffic. A bursty ITL (high p99 vs mean) makes output appear in clumps - the classic "GPU contention" signature.
+
+### E2E Latency
+Total time from prompt to final token. `E2E = TTFT + (output_tokens - 1) * TPOT`. Matters for batch/codegen/summarization where the full response is needed before the next step.
+
+### TPS - Token Throughput
+- **System TPS** - total output tokens/sec across all requests. Raw capacity; rises with load until GPU-saturated. `TPS = output_tokens / (T_last - T_first)`.
+- **User TPS** - tokens/sec a single user experiences; `≈ 1 / ITL` at long outputs. Drops as concurrency rises because the engine shares the GPU.
+
+### RPS - Request Throughput
+Completed requests/sec. Dominant metric for high-volume short-prompt traffic (chatbots, search, API gateways). Shorter prompts + higher `max_num_seqs` raise RPS.
+
+### Goodput (the metric that matters most)
+`Goodput = (requests meeting ALL SLOs) / total_requests * 100%`. Unlike TPS/RPS, goodput tells you what fraction of users got an *acceptable* experience. A system can show high TPS while most requests violate latency SLOs. Always define goodput with explicit thresholds, e.g. TTFT < 500ms, TPOT < 15ms, E2E < 2s. As load climbs, goodput falls even as raw throughput keeps rising - that crossover is your real capacity limit.
+
+### Percentiles
+Report **p50/p95/p99** for TTFT, ITL, and E2E - never just averages. Averages hide the unlucky 5% whose tokens arrive in bursts. p99 is the near-worst-case; if p99 meets SLO, the system is consistent.
+
+---
+
+## Workload Dimensions
+
+Controlling these is what separates a useful LLM test from a misleading one:
+
+| Dimension | Effect | How to set |
+|---|---|---|
+| **Prompt length (ISL)** | Longer → higher TTFT, more KV cache | Use real traffic (ShareGPT) or synthetic ranges |
+| **Output length (OSL)** | Longer → higher E2E, more decode cost | Real distributions; `--random-output-len` for synthetic |
+| **Concurrency** | Bounded by KV cache, not sockets | Set `--max-concurrency` to simulate gateway limits |
+| **Request rate** | Open-model arrival (Poisson) | `--request-rate`; `inf` for max throughput |
+| **Burstiness** | Gamma-distributed arrivals | `1.0` realistic, `0.1-0.5` stress, `2-5` uniform |
+| **Batching (`max_num_seqs`)** | Higher → more RPS, worse per-user latency | Tune on the server, not the client |
+
+**KV cache math**: `max_concurrency ≈ KV_cache_tokens / max_model_len`. vLLM prints this at startup. Set test concurrency to 80-90% of it for capacity planning; use the full value as the SLA limit.
+
+---
+
+## Tools
+
+Purpose-built LLM benchmarkers surface token metrics natively. Prefer them over generic HTTP load testers.
+
+| Tool | Best for | Notes |
+|---|---|---|
+| **vLLM `bench serve`** | Serving benchmarks against an OpenAI-compatible endpoint | Native TTFT/TPOT/ITL/TPS, ShareGPT + synthetic datasets, ramp-up, goodput via percentiles |
+| **GuideLLM** | Production vLLM SLA/capacity studies | Auto reports, live progress, profile-based; recommended by vLLM for production |
+| **NVIDIA GenAI-Perf** | Multi-backend (TRT-LLM, Triton, vLLM), concurrency/rate modes | Emits TTFT, ITL, output TPS, goodput; pairs with Perf Analyzer |
+| **llmperf / LLM-Perf** | Quick pointwise latency/throughput checks | Lightweight, great for smoke and regression |
+| **LLM Locust** | Distributed load on Locust, GenAI metrics | Use when you already run Locust fleets |
+| **k6 + custom metrics** | Unified CI with existing k6 stacks | Must instrument streaming manually: capture TTFT from first SSE chunk, ITL between chunks, count tokens. See `../tools/k6.md` for metric primitives |
+
+> **Anti-pattern (per current research):** do not misuse model-server micro-benchmarkers (vLLM `bench`, SGLang bench) as production-level evaluators. They optimize for regression/feature testing, not realistic arrival patterns. Use them for component baselines; use GuideLLM/GenAI-Perf/production-style load for capacity and SLO validation.
+
+---
+
+## Quick Start Commands
+
+```bash
+# vLLM bench serve - max throughput probe with ShareGPT data
+python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-3-8B &
+python -m vllm bench serve \
+  --model meta-llama/Llama-3-8B \
+  --endpoint http://localhost:8000/v1/completions \
+  --dataset-name sharegpt \
+  --request-rate inf \
+  --max-concurrency 256 \
+  --num-prompts 500
+
+# vLLM bench serve - realistic arrival pattern
+python -m vllm bench serve \
+  --model meta-llama/Llama-3-8B \
+  --endpoint http://localhost:8000/v1/completions \
+  --dataset-name sharegpt \
+  --request-rate 10 \
+  --burstiness 1.0 \
+  --num-prompts 200
+
+# GuideLLM - production SLA study
+pip install guidedllm
+guidedllm evaluate \
+  --model meta-llama/Llama-3-8B \
+  --base-url http://localhost:8000/v1 \
+  --data emulated --rate 10 --duration 120
+
+# NVIDIA GenAI-Perf - multi-backend benchmark
+genai-perf profile \
+  --model meta-llama/Llama-3-8B \
+  --endpoint-type chat \
+  --url localhost:8000 \
+  --concurrency 64 \
+  --request-count 200
+
+# llmperf - quick latency/throughput check
+python token_benchmark_ray.py \
+  --model meta-llama/Llama-3-8B \
+  --mean-input-tokens 512 \
+  --mean-output-tokens 128 \
+  --num-concurrent-requests 32 \
+  --results-dir ./results
+```
+
+---
+
+## Test Design & Methodology
+
+1. **Start from real traffic shapes.** Use ShareGPT or captured production traces for prompt/output length distributions. Synthetic `random` datasets are fine for stress but unrealistic for sizing.
+2. **Define SLOs as goodput thresholds first.** e.g. "TTFT p95 < 500ms, ITL p95 < 20ms, E2E p95 < 3s" → goodput target 99%.
+3. **Run a max-throughput probe** (`--request-rate inf --max-concurrency <limit>`) to find the concurrency ceiling and baseline TPS.
+4. **Sweep concurrency / request rate** to find the goodput cliff - the load where SLO compliance drops. That is capacity.
+5. **Test ramp-up and spikes** (linear/exponential ramp, bursty arrival) to validate autoscaling and queue behavior.
+6. **Always assert on token metrics, not just HTTP 200.** A 200 with a 10s TTFT is a failed request.
+
+### Workload pattern recipes (vLLM bench semantics)
+| Goal | `--request-rate` | `--burstiness` | `--max-concurrency` |
+|---|---|---|---|
+| Max throughput | `inf` | n/a | limited |
+| Realistic baseline | 5-20 | 1.0 | inf |
+| Stress / resilience | 20-100 | 0.1-0.5 | inf |
+| Latency profiling | 1-10 | 2-5 | inf |
+| Capacity / SLA | target rate | 1.0 | SLA limit |
+
+---
+
+## Common Mistakes
+
+- **Measuring only HTTP latency** - TTFT and streaming gaps are invisible to `http_req_duration`. You'll report a green test while users stare at a blank cursor.
+- **Ignoring output length variance** - fixed `max_tokens` hides the real E2E spread; use realistic OSL distributions.
+- **Unbounded concurrency** - without `--max-concurrency`, the client fires until the GPU chokes; you measure collapse, not capacity. The KV cache, not sockets, is the real cap.
+- **Confusing TPS with user experience** - high system TPS can coexist with terrible per-user ITL. Track both; report goodput.
+- **Averaging instead of percentiles** - a 200ms mean TTFT with 8s p99 is a broken chat UX.
+- **Using a micro-benchmarker for production SLOs** - `vllm bench`/SGLang bench validate the engine, not your serving capacity under real arrival patterns.
+- **No streaming instrumentation in k6** - a naive `http.get` to a streaming endpoint counts only the final byte; you must parse SSE chunks to get TTFT/ITL.
+- **Forgetting prompt caching** - prefix caching dramatically cuts TTFT for repeated prefixes; test with and without it (e.g. RAG, system prompts) to size the win.
+
+---
+
+## Observability
+
+Track these server-side alongside the client metrics above:
+- **KV cache utilization** - the true saturation signal; when it pins at 100%, TTFT inflates via queueing.
+- **Batch size / `num_running` vs `num_waiting`** - waiting > 0 means you're over concurrency.
+- **Prefill vs decode time split** - isolates whether slowness is prompt-side or generation-side.
+- **GPU util + memory** - confirms you're compute-bound, not starved.
+
+> For dashboards/APM integration patterns, see `observability.md`. For interpreting the goodput cliff and saturation, see `results-analysis.md`.

package/skills/perf/references/topics/modern-architectures.md

@@ -312,3 +312,5 @@ Define performance budgets in CI to prevent regressions:
 - [ ] Concurrency limits and provisioned concurrency validated
 - [ ] Frontend Core Web Vitals measured under backend load
 - [ ] Performance budgets defined and enforced in CI
+
+> **See also:** CI/CD pipeline patterns in `test-execution.md`. Kubernetes and mesh monitoring in `observability.md`. SLO headroom for autoscaling decisions in `slo-capacity.md`.

package/skills/perf/references/topics/observability.md

@@ -199,3 +199,5 @@ Synthetic monitors catch regressions that slip through staging without requiring
 - [ ] Log aggregation active and searchable
 - [ ] Baseline metrics screenshot taken before test starts (for comparison)
 - [ ] Metrics retention configured to keep results for trend analysis
+
+> **See also:** k6 metric primitives in `../tools/k6.md`. JMeter Backend Listener in `../tools/jmeter.md`. Interpreting the metrics in `results-analysis.md`. SLO thresholds in `slo-capacity.md`.

package/skills/perf/references/topics/production-testing.md

@@ -212,3 +212,5 @@ If something breaks during a test:
 4. **Roll back if needed** - restore previous version, restart services.
 5. **Preserve evidence** - don't restart services without capturing logs and heap dumps.
 6. **Post-mortem** - document what broke, at what load, what the root cause was.
+
+> **See also:** CI/CD execution patterns in `test-execution.md`. Monitoring during production tests in `observability.md`. SLO and error budget framing in `slo-capacity.md`.

package/skills/perf/references/topics/protocol-testing.md

@@ -58,6 +58,36 @@ export default function () {
 - **Connection reuse**: Keep connections open across iterations; don't connect/close per request.
 - **Streaming throughput**: For server-streaming RPCs, measure messages-per-second, not just request latency.
 - **Deadline propagation**: Set gRPC deadlines in tests to match production timeouts.
+
+### Gatling gRPC Example
+
+```scala
+// Requires: gatling-grpc plugin (io.gatling:gatling-grpc)
+import io.gatling.core.Predef._
+import io.gatling.grpc.Predef._
+import io.grpc.ManagedChannelBuilder
+import scala.concurrent.duration._
+
+class GrpcSimulation extends Simulation {
+  val channel = ManagedChannelBuilder
+    .forAddress("grpc-server", 50051)
+    .usePlaintext()
+    .build()
+
+  val scn = scenario("gRPC Unary")
+    .exec(
+      grpc("SayHello")
+        .rpc(HelloServiceGrpc.METHOD_SAY_HELLO)
+        .payload(HelloRequest(greeting = "perf-test"))
+        .check(statusCode.is(StatusCode.OK))
+    )
+    .pause(1)
+
+  setUp(
+    scn.injectOpen(rampUsers(100).during(30.seconds))
+  ).protocols(grpc(channel))
+}
+```
 - **Load balancer awareness**: gRPC over HTTP/2 with persistent connections can cause uneven load across backends - test with client-side load balancing or L7 proxy.
 - **Protobuf payload size**: Binary encoding is smaller than JSON - adjust throughput expectations accordingly.
 
@@ -199,6 +229,39 @@ export default function () {
 3. **Measure server-side**: File descriptor count, memory per connection, event loop lag.
 4. **Test reconnection storms**: Kill server, observe client reconnection behavior and server recovery.
 
+### Gatling WebSocket Example
+
+```scala
+import io.gatling.core.Predef._
+import io.gatling.http.Predef._
+import scala.concurrent.duration._
+
+class WebSocketSimulation extends Simulation {
+  val httpProtocol = http.baseUrl("https://app.example.com")
+
+  val scn = scenario("WebSocket Chat")
+    .exec(
+      ws("Connect WS")
+        .connect("/ws/chat")
+        .header("Authorization", "Bearer #{token}")
+    )
+    .exec(
+      ws("Subscribe")
+        .sendText("""{"type":"subscribe","channel":"updates"}""")
+        .await(30.seconds)(
+          ws.checkTextMessage("subscribed")
+            .check(bodyJsonPath("$.type").is("subscribed"))
+        )
+    )
+    .pause(30)  // hold connection open
+    .exec(ws("Close").close)
+
+  setUp(
+    scn.injectOpen(rampUsers(500).during(60.seconds))
+  ).protocols(httpProtocol)
+}
+```
+
 ---
 
 ## Message Queue / Event Streaming Testing
@@ -272,3 +335,5 @@ export function teardown() {
 - [ ] Backpressure / flow control behavior validated
 - [ ] Error codes and retry behavior tested (gRPC status codes, AMQP nacks, etc.)
 - [ ] End-to-end latency measured (not just request latency)
+
+> **See also:** k6 scripting (all examples above) in `../tools/k6.md`. Gatling gRPC/WebSocket in `../tools/gatling.md`. JMeter JMS/plugins in `../tools/jmeter.md`. Workload design for streaming protocols in `workload-design.md`.

package/skills/perf/references/topics/results-analysis.md

@@ -242,3 +242,5 @@ Structure your report:
    - Error log samples
    - Environment configuration
 ```
+
+> **See also:** Metric collection and dashboards in `observability.md`. SLO interpretation and error budgets in `slo-capacity.md`. LLM-specific metrics (TTFT, TPOT, goodput) in `llm-inference.md`.

package/skills/perf/references/topics/script-generation.md

@@ -39,26 +39,17 @@ Record a HAR file from the browser → import into your tool → clean up and pa
 
 ---
 
-## Correlation Deep Dive
+## Correlation
 
 Correlation is extracting a dynamic value from a response and using it in a subsequent request. Missing correlation is the #1 cause of performance script failure.
 
-### Values that always need correlation
-- **Session tokens** (JSESSIONID, ASP.NET_SessionId)
-- **Auth tokens** (JWT, OAuth access_token, refresh_token)
-- **CSRF tokens** (`_token`, `authenticity_token`, `__RequestVerificationToken`)
-- **View state** (ASP.NET `__VIEWSTATE`, `__EVENTVALIDATION`)
-- **Resource IDs** (created order ID, cart ID, uploaded file ID)
-- **One-time codes** (OTP, nonce, challenge)
-- **Timestamps/signatures** used in request signing
-
-### Correlation debugging approach
-1. Run the script with 1 VU.
-2. Look for HTTP 4xx errors (especially 403 Forbidden, 422 Unprocessable Entity).
-3. Check the request body/headers - is a token missing or stale?
-4. Use the tool's debug output or proxy (Fiddler, Charles) to inspect live traffic.
-5. Identify where the value appears in a *previous* response.
-6. Add an extractor at that response, reference the variable in the failing request.
+> **Full reference:** see `correlation.md` for the complete correlation workflow, tool-specific extractor syntax, tech-stack rules (ASP.NET, Java, SAP, OAuth), brittle-vs-robust regex guidance, and the diagnostic playbook.
+
+### Quick checklist
+- Session tokens, CSRF tokens, OAuth codes, resource IDs, and one-time nonces must always be correlated.
+- Run with 1 VU first; 403/422 errors usually mean a missing or stale token.
+- Use a proxy (Fiddler, Charles, mitmproxy) to find where the value appears in a previous response.
+- Add the extractor at that response, reference the variable in the failing request.
 
 ---
 
@@ -219,3 +210,5 @@ Treat performance scripts as production code:
 - [ ] Single VU smoke test passes cleanly
 - [ ] Script loads test data from external source (CSV, API)
 - [ ] Environment-specific values in config/env vars
+
+> **See also:** Full correlation reference in `correlation.md`. Parameterization and data feeds in `test-data.md`. Tool-specific syntax in `../tools/k6.md`, `../tools/jmeter.md`, `../tools/gatling.md`, `../tools/locust.md`.

package/skills/perf/references/topics/slo-capacity.md

@@ -0,0 +1,136 @@
+# SLO & Capacity Planning
+
+> Scope: turning load-test results into service level objectives, error budgets, and capacity decisions. Complements `workload-design.md` (designing the load) and `results-analysis.md` (reading the numbers). Read those first if you have not designed or run the test yet.
+
+This topic covers the *decision* layer of performance engineering: how to define SLIs/SLOs that mean something, how to convert a load-test curve into a capacity number and a headroom plan, and how to enforce all of it as an automated CI gate.
+
+> For LLM-specific SLOs (TTFT, goodput), see `llm-inference.md`. For the tool syntax that enforces SLOs (`ensure`, `thresholds`), see the relevant tool file.
+
+---
+
+## SLI / SLO / Error Budget
+
+### Definitions
+- **SLI (Service Level Indicator)** - the actual measured signal, e.g. request p95 latency, error rate, availability.
+- **SLO (Service Level Objective)** - the target you commit to for an SLI over a window, e.g. "p95 < 500ms for 99% of requests over 28 days."
+- **Error Budget** - `1 - SLO`. A 99% SLO means a 1% error budget: you may violate the objective 1% of the time before it's a breach. Budgets make trade-offs explicit (ship features vs. burn reliability).
+
+### Choosing SLIs
+Pick SLIs users actually feel, not ones that are easy to measure:
+
+| User-facing concern | SLI | Typical SLO |
+|---|---|---|
+| Responsiveness | Request p95 latency | < 500ms |
+| Worst-case tail | Request p99 latency | < 1,500ms |
+| Reliability | Error rate (5xx + timeouts) | < 1% (99% success) |
+| Availability | Successfully served requests / total | 99.9% |
+| Throughput | Sustained RPS at SLO latency | ≥ peak + headroom |
+| Freshness (data) | Staleness of served data | < 60s |
+
+For streaming/LLM, use token-aware SLIs (TTFT, ITL) and **goodput** (fraction of requests meeting all thresholds) instead of raw latency - see `llm-inference.md`.
+
+### Windowing
+- Use a **rolling window** (e.g. 28 days) so a single bad day does not trigger an alert; it consumes budget gradually.
+- Report SLO attainment as `good_events / total_events` over the window. If attainment < SLO, the budget is exhausted.
+- Set **alerts on burn rate**, not on instantaneous SLO pass/fail. Fast burn (e.g. 14× rate) means a major incident; slow burn means a trend to watch.
+
+### Multi-window burn-rate alerting (pattern)
+Define fast and slow burn signals so you catch both outages and slow erosion:
+```
+Fast burn:  burn rate ≥ 14  over 1h   → page
+Slow burn:  burn rate ≥ 2   over 6h   → ticket
+```
+A burn rate of `X` means the budget is being consumed `X` times faster than the 28-day baseline allows.
+
+---
+
+## From Load Test to Capacity
+
+A load test produces a curve of latency/error vs load. Capacity is the point on that curve where SLOs still hold, **plus** the headroom you keep for safety.
+
+### The saturation curve
+```
+latency
+  │              ╱‾‾‾‾‾‾‾   ← cliff: SLO breach
+  │ ─────────────
+  └──────────────────────── load (RPS/VUs)
+   [safe zone]   [capacity point]
+```
+- **Capacity point** = highest sustained load at which all SLOs (p95, error rate, goodput) still pass.
+- **Saturation point** = where throughput flatlines and latency explodes (the "cliff" in `results-analysis.md`).
+- **Never plan to capacity.** Plan to capacity minus headroom.
+
+### Headroom rule
+- Target **peak production load at ~60-70% of measured capacity** under normal operation.
+- Keep **~30-40% headroom** to absorb traffic spikes, node failures, and deployment blips without breaching SLO.
+- For autoscaling systems, headroom is what lets new replicas come up before the SLO burns.
+
+### Inverting Little's Law to size capacity
+Given a target arrival rate and response time, concurrency is `N = λ × W` (see `workload-design.md`). Use it both ways:
+- **To size load**: known λ and W → required VUs (already in workload-design).
+- **To size infrastructure**: if the test shows a single replica sustains `C` RPS at SLO, replicas needed = `target_RPS / (C × utilization_target)`. With a 70% utilization target and 1,000 target RPS at 200 RPS/replica → `1000 / (200 × 0.7)` ≈ **8 replicas** (round up, add 1 for failure tolerance).
+
+### Capacity from the goodput cliff
+For LLM and streaming services, capacity = load at the goodput cliff (see `llm-inference.md`). Report "sustains 200 concurrent users at 99% goodput; goodput drops to 80% at 350" - that 200 is your planning number, 350 is your hard ceiling.
+
+---
+
+## Worked Example
+
+**Inputs:** Peak prod = 800 RPS. Test shows p95 < 500ms and error < 1% hold up to 1,200 RPS on the current 6-replica cluster; saturation at ~1,500 RPS.
+
+| Decision | Calculation | Result |
+|---|---|---|
+| Capacity point | measured SLO-holding load | 1,200 RPS |
+| Planning target | 60-70% of capacity | 720-840 RPS |
+| Headroom | 1,200 - 800 peak | 33% (acceptable) |
+| Replicas at growth | if peak grows to 1,500 RPS: `1500 / (200 × 0.7)` | 11 replicas (was 6) |
+| Alert threshold | 80% of capacity as early warning | alert at ~960 RPS |
+
+---
+
+## CI Regression Gating
+
+SLOs are useless if they are only in a report. Enforce them automatically:
+
+1. **Gate on thresholds, not on "it ran."** Every tool has an SLO mechanism - k6 `thresholds`, Artillery `ensure`, Gatling Enterprise assertions, JMeter exit codes. A run with no gate always passes CI (see each tool file's Common Mistakes).
+2. **Compare against baseline, not just absolute.** A p95 of 480ms passes a 500ms SLO but is a +60% regression vs last week's 300ms. Track percentile deltas build-over-build.
+3. **Set a regression threshold.** e.g. "fail the build if p95 rises > 10% vs baseline, or error rate > 0.5%." Avoid zero-tolerance (noise); avoid loose (misses real regressions).
+4. **Require statistical significance for soak/large runs.** A 1-run blip should not fail CI; use multiple iterations or a confidence band.
+5. **Publish the SLO report in the pipeline.** Surface p50/p95/p99, throughput, error rate, and budget consumption on every run so regressions are visible, not buried.
+
+> For the per-tool gate syntax, see `../tools/k6.md` (`thresholds`), `../tools/artillery.md` (`ensure`), `../tools/gatling.md`, `../tools/jmeter.md`. For running these in pipelines, see `test-execution.md`.
+
+---
+
+## Common Mistakes
+
+- **Averaging for SLOs** - an average of 200ms can hide a p99 of 5s. SLOs must be percentile-based.
+- **SLO with no error budget** - a target with no burn policy becomes a meaningless number; you can never "spend" reliability intentionally.
+- **Planning to the saturation point** - sizing for max measured throughput leaves zero margin; the first spike breaches SLO.
+- **Confusing capacity with peak** - if peak ≈ capacity, you have no headroom. Target 60-70%.
+- **One-shot capacity number** - capacity drifts as code, data, and dependencies change. Re-baseline on major releases.
+- **Gating CI on run success only** - "test passed" ≠ "SLO met." Without a threshold/ensure gate, regressions ship green.
+- **Ignoring tail latency in capacity** - a system can hold p95 at capacity while p99 is 10× worse; gate on the tail users actually feel.
+- **No burn-rate alerting** - alerting on instantaneous SLO state misses slow erosion until the budget is already gone.
+
+---
+
+## Reporting SLO & Capacity
+
+Structure the decision section for stakeholders:
+
+```
+1. SLO SUMMARY
+   - p95 < 500ms: 99.2% attainment (budget: 1% used 0.8%)
+   - Error rate < 1%: 99.97% attainment
+2. CAPACITY
+   - Measured capacity: 1,200 RPS @ SLO
+   - Current peak: 800 RPS (33% headroom)
+   - Saturation: ~1,500 RPS
+3. RECOMMENDATION
+   - Scale to 8 replicas before peak season (growth to 1,500 RPS)
+   - Alert at 960 RPS (80% of capacity)
+4. REGRESSION GATE
+   - p95 > +10% vs baseline fails CI
+```

package/skills/perf/references/topics/test-data.md

@@ -194,3 +194,5 @@ const user = users[__VU - 1];  // VU 1 gets row 0, VU 2 gets row 1, etc.
 - [ ] CSV files available on all injector nodes (distributed tests)
 - [ ] Sensitive data masked/anonymized (not using real PII in test data)
 - [ ] Date/time sensitive records accounted for (e.g., expiry dates set far in future)
+
+> **See also:** Tool-specific parameterization syntax in `../tools/k6.md` (SharedArray), `../tools/jmeter.md` (CSV Data Set Config), `../tools/gatling.md` (feeders), `../tools/locust.md`. Distributing data to injectors in `test-execution.md`.

package/skills/perf/references/topics/test-execution.md

@@ -114,12 +114,37 @@ bzt tests/load-test.jmx \
 ```
 
 ### AWS (DIY Cloud Execution)
-```bash
-# Spin up EC2 injectors via Terraform, then run JMeter/k6
-# Use Auto Scaling Groups for burst capacity
-# Use S3 to store results, CloudWatch for metrics
 
-# Cost optimization: use Spot instances for injectors (price vs reliability tradeoff)
+```bash
+# 1. Launch injectors with AWS CLI (or Terraform)
+aws ec2 run-instances \
+  --image-id ami-0abcdef1234567890 \
+  --instance-type c5.2xlarge \
+  --count 4 \
+  --spot-price "0.15" \
+  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=load-injector}]'
+
+# 2. Install k6 on each injector (via user-data or SSM)
+aws ssm send-command \
+  --instance-ids i-0abc i-0def i-0ghi i-0jkl \
+  --document-name "AWS-RunShellScript" \
+  --parameters 'commands=["sudo apt-get install -y k6"]'
+
+# 3. Distribute the test script and run in parallel
+for ip in 10.0.1.{10..13}; do
+  scp script.js ec2-user@$ip:~/ && \
+  ssh ec2-user@$ip "k6 run --vus 250 --duration 10m script.js --out json=/tmp/results.json" &
+done
+wait
+
+# 4. Collect results to S3
+for ip in 10.0.1.{10..13}; do
+  scp ec2-user@$ip:/tmp/results.json ./results/$(echo $ip | tr '.' '-')-results.json
+done
+aws s3 cp ./results/ s3://my-load-test-bucket/runs/$(date +%Y%m%d)/ --recursive
+
+# Cost optimization: use Spot instances for injectors (60-90% cheaper).
+# Handle interruptions with a 2-min warning hook that flushes partial results.
 ```
 
 ---
@@ -279,3 +304,5 @@ Always monitor the injector resources alongside the SUT:
 - [ ] Smoke test (1 VU) passed before full run
 - [ ] Team notified of test window
 - [ ] Rollback plan in place (especially for production testing)
+
+> **See also:** Tool-specific CLI and distributed setup in `../tools/jmeter.md`, `../tools/k6.md`, `../tools/locust.md`, `../tools/gatling.md`, `../tools/artillery.md`. Monitoring during execution in `observability.md`. Production safety controls in `production-testing.md`.