perf-skills 2.0.0 → 3.0.0

package/README.md

@@ -1,165 +1,182 @@
-# perf-skills
-
-> *"The more you sweat in training, the less you bleed in battle."* - Richard Marcinko
->
-> The more relevant skills your AI loads during development, the fewer fires you fight in production.
-
-## What Is This?
-
-`perf-skills` is a structured knowledge base that gives AI coding assistants deep, opinionated expertise in performance testing. It works with any AI tool that supports custom instructions, rules, or skill files - helping you plan, script, execute, and analyze load tests using any major tool.
-
-### Supported Tools
-
-| Open Source | Commercial |
-|---|---|
-| Apache JMeter | NeoLoad (Tricentis) |
-| k6 (Grafana Labs) | LoadRunner (OpenText) |
-| Gatling | OctoPerf (SaaS) |
-| Locust | |
-
-### Topics Covered
-
-- **Workload design** - concurrency models, Little's Law, transaction mix, SLA targets
-- **Test data** - parameterization, CSV feeds, synthetic data, data isolation
-- **Script generation** - correlation, assertions, error handling, session management
-- **Test execution** - local, distributed, CI/CD pipelines, cloud execution
-- **Results analysis** - percentiles, bottleneck identification, trend comparison, reporting
-- **Observability** - APM integration, Prometheus/Grafana, distributed tracing, log correlation
-- **Production testing** - canary, shadow traffic, chaos engineering, safety controls
-- **Protocol testing** - gRPC, GraphQL, WebSocket/SSE, Kafka/message queues
-- **Database testing** - JDBC load testing, connection pools, query concurrency, replication lag
-- **Modern architectures** - microservices, Kubernetes (HPA, service mesh), serverless (cold starts), frontend (Core Web Vitals)
-
-## Compatible AI Coding Assistants
-
-| Tool | Integration Method | Setup |
-|---|---|---|
-| **Windsurf (Cascade)** | Skills | Copy to skills directory - auto-triggers on perf questions |
-| **Cursor** | Rules / Docs | Add as project rules or index via `@Docs` |
-| **Claude Code** | CLAUDE.md / Custom instructions | Reference files in `CLAUDE.md` or feed as context |
-| **Cline** | Custom instructions / `.clinerules` | Add to `.clinerules` or workspace instructions |
-| **Roo Code** | Custom instructions / Rules | Add as workspace rules or custom instructions |
-| **Aider** | Conventions / Chat context | Add to `.aider.conf.yml` conventions or `/read` files |
-| **OpenCode** | Custom instructions | Add to project-level instructions |
-| **Antigravity** | Context files | Add as context / knowledge files |
-| **Pochi** | Custom instructions | Reference files in project instructions |
-| **GitHub Copilot** | Custom instructions / `.github/copilot-instructions.md` | Reference in repo-level instructions |
-
-## Installation
+# ⚡ Performance Testing Skills (`perf-skills`)
 
-### Claude Code Plugin (Recommended)
+> **Turn your AI assistant into a senior performance engineer.** An opinionated, token-optimized knowledge base for planning, scripting, executing, and analyzing load tests across any major tool.
 
-**Add the marketplace and install:**
-```bash
-/plugin marketplace add QAInsights/perf-skills
-/plugin install perf@qainsights
-/reload-plugins
-```
+> *"The more you sweat in training, the less you bleed in battle."* - Richard Marcinko  
+> *"The more relevant skills your AI loads during development, the fewer fires you fight in production."*
 
-After install, the `/perf` skill is available and auto-activates on performance testing questions.
+---
 
-**Update to latest version:**
-```bash
-/plugin marketplace update
-/plugin uninstall perf
-/plugin install perf@qainsights
-/reload-plugins
-```
+## 📖 Table of Contents
 
-### Install as Skills (npx)
+- [🔍 What is perf-skills?](#-what-is-perf-skills)
+- [📦 Supported Tools & Protocols](#-supported-tools--protocols)
+- [⛓️ Dynamic Value Correlation](#️-dynamic-value-correlation)
+- [🛠️ Compatible AI Assistants](#️-compatible-ai-assistants)
+- [🚀 Installation & Setup](#-installation--setup)
+- [📂 File Structure & Routing](#-file-structure--routing)
+- [💡 How the Skill Works](#-how-the-skill-works)
+- [🤝 Contributing & License](#-contributing--license)
 
-```bash
-npx skills add QAInsights/perf-skills
-```
+---
 
-### Install as Skills (Manual)
+## 🔍 What is perf-skills?
 
-```bash
-# Clone and copy to Claude skills directory
-git clone https://github.com/QAInsights/perf-skills.git
-cp -r perf-skills/skills/perf ~/.claude/skills/
-```
+`perf-skills` is a highly structured knowledge base designed specifically for AI coding assistants (like Claude, Cursor, Windsurf, etc.). It injects deep, expert-level performance testing context on-demand, allowing your AI to generate correct scripts, follow best practices, and debug bottlenecks without bloating its context window.
 
-### Windsurf (Skills)
+---
 
-1. Clone the repository: `git clone https://github.com/QAInsights/perf-skills.git`
-2. Copy the skill directory: `cp -r perf-skills/skills/perf ~/.windsurf/skills/`
-3. The skill auto-triggers when you ask about performance testing, load testing, or any supported tool.
+## 📦 Supported Tools & Protocols
 
-### Cursor (Rules / Docs)
+| Category | Supported Technologies |
+| :--- | :--- |
+| **Open Source** | Apache JMeter, k6 (Grafana Labs), Gatling, Locust |
+| **Commercial/SaaS** | NeoLoad (Tricentis), LoadRunner (OpenText), OctoPerf |
+| **Protocols** | HTTP/REST, gRPC, GraphQL, WebSocket, SSE, JDBC, SOAP, Kafka/MQ, Citrix, SAP |
 
-**Option A - Project Rules:**
-1. Clone the repository: `git clone https://github.com/QAInsights/perf-skills.git`
-2. Create `.cursor/rules/perf.mdc` in your project root.
-3. Copy content from `perf-skills/skills/perf/SKILL.md` into the rule file.
+---
 
-**Option B - @Docs indexing:**
-1. Clone the repository into your project or a known location.
-2. Open Cursor Settings → Features → Docs.
-3. Add the `perf-skills/skills/perf/` directory as a doc source.
-4. Reference with `@Docs perf` in chat.
+## ⛓️ Dynamic Value Correlation
 
-### Claude Code (CLAUDE.md)
+A major highlight of this library is the **Dynamic Value Correlation** engine ([correlation.md](skills/perf/references/topics/correlation.md)). Correlation is often the most fragile part of performance scripting; `perf-skills` equips your AI with robust strategies to handle it flawlessly:
 
-1. Clone the repository: `git clone https://github.com/QAInsights/perf-skills.git`
-2. Copy the skill directory into your project: `cp -r perf-skills/skills/perf ./perf`
-3. In your `CLAUDE.md`, add:
-```markdown
-For performance testing questions, read `perf/SKILL.md` for routing,
-then load the relevant reference file(s) from `perf/references/`.
-```
+- **The Correlation Workflow**: Standardizes identifying, extracting, injecting, and verifying dynamic tokens.
+- **Advanced Scanning**: Strategies for Next.js/React SSR hydration payloads (`__NEXT_DATA__`), SPA client-side token retrieval, runtime hooking, and Swagger contract mapping.
+- **Robust vs. Brittle Regex**: Guides your AI to write expressions that don't break on minor HTML structure modifications.
+- **Tech Stack Rules**: Pre-defined extraction rules for enterprise environments:
+  - **ASP.NET**: ViewState, RequestVerificationToken, EventValidation.
+  - **Java**: JSF ViewState, Spring Security `_csrf`, JSESSIONID.
+  - **SAP & Oracle**: NetWeaver `sap-contextid`, ADF `_afrLoop`.
+  - **OAuth 2.0 / SAML**: SAMLResponse, code, state tokens.
+  - **Modern SSR/SPA**: Hydration states, Laravel `_token`, Django `csrfmiddlewaretoken`, Rails `authenticity_token`.
+- **Diagnostic Playbook**: Step-by-step troubleshooting for URL-encoding, base64 url-safe transforms, JSON-escaping, and thread safety.
 
-### Cline / Roo Code
+---
 
-1. Clone the repository: `git clone https://github.com/QAInsights/perf-skills.git`
-2. Copy the skill directory into your project: `cp -r perf-skills/skills/perf ./perf`
-3. Add to your custom instructions or `.clinerules`:
-```
-For performance testing guidance, consult the perf knowledge base.
-Start with perf/SKILL.md for routing to the correct reference file.
-```
+## 🛠️ Compatible AI Assistants
 
-### Aider
+`perf-skills` works seamlessly with any modern AI development tool:
 
-1. Clone the repository: `git clone https://github.com/QAInsights/perf-skills.git`
-2. Copy the skill directory into your project: `cp -r perf-skills/skills/perf ./perf`
-3. Use `/read perf/SKILL.md` to load the routing file.
-4. Then `/read` the specific reference file(s) relevant to your question.
+| AI Assistant | Integration Method |
+| :--- | :--- |
+| **Claude Code** | Native Plugin or `CLAUDE.md` context referencing |
+| **Windsurf (Cascade)** | Local Skill configuration |
+| **Cursor** | Custom rules `.cursor/rules/` or index via `@Docs` |
+| **Cline / Roo Code** | Workspace instructions / custom rules |
+| **Aider** | Conventions file or `/read` context loading |
+| **GitHub Copilot** | Repository-level custom instructions |
+| **Antigravity / Pochi** | Context files / instruction references |
 
-### GitHub Copilot
+---
 
-1. Clone the repository: `git clone https://github.com/QAInsights/perf-skills.git`
-2. Copy the skill directory into your project: `cp -r perf-skills/skills/perf ./perf`
-3. In `.github/copilot-instructions.md`, add:
-```markdown
-For performance testing questions, reference the perf knowledge base.
-Start with perf/SKILL.md, then load relevant files from perf/references/.
-```
+## 🚀 Installation & Setup
 
-### OpenCode / Antigravity / Pochi
+### Claude Code Plugin (Recommended)
 
-1. Clone the repository: `git clone https://github.com/QAInsights/perf-skills.git`
-2. Copy the skill directory into your project: `cp -r perf-skills/skills/perf ./perf`
-3. Add to your project-level custom instructions or context files:
-```
-For performance testing guidance, consult the perf knowledge base.
-Start with perf/SKILL.md for routing to the correct reference file.
+To install as a Claude Code plugin, run:
+```bash
+/plugin marketplace add QAInsights/perf-skills
+/plugin install perf@qainsights
+/reload-plugins
 ```
 
-### Any Other AI Tool
-
-The skill is plain markdown files. Any AI tool that can read files or accept custom instructions can use it:
-1. Point the tool to `SKILL.md` as the entry point.
-2. Let the Reference Map in `SKILL.md` guide which file(s) to load.
+Once installed, the `/perf` skill will be available and automatically load when you ask performance-related questions.
 
-### As a Standalone Knowledge Base
+To update to the latest version:
+```bash
+/plugin marketplace update
+/plugin uninstall perf
+/plugin install perf@qainsights
+/reload-plugins
+```
 
-Browse the markdown files directly - they're self-contained references useful even without an AI assistant.
+---
 
-## File Structure
+### Install as local Skills (npx)
 
+If you are using a tool that supports `npx skills`, add it directly:
+```bash
+npx skills add QAInsights/perf-skills
 ```
-perf-skills/                              # Repository root
+
+---
+
+### Setup for Other Editors
+
+<details>
+<summary><b>🌊 Windsurf (Cascade)</b></summary>
+<br>
+
+1. Clone this repository:
+   ```bash
+   git clone https://github.com/QAInsights/perf-skills.git
+   ```
+2. Copy the skill directory:
+   ```bash
+   cp -r perf-skills/skills/perf ~/.windsurf/skills/
+   ```
+The skill will auto-trigger when you ask about performance testing, load testing, or any supported tool.
+</details>
+
+<details>
+<summary><b>🎯 Cursor</b></summary>
+<br>
+
+**Option A (Project Rules):**
+1. Copy [SKILL.md](skills/perf/SKILL.md) content into a rule file: `.cursor/rules/perf.mdc`.
+2. Configure rules to trigger on file paths matching performance scripts or performance-related prompts.
+
+**Option B (@Docs indexing):**
+1. Open Cursor Settings → Features → Docs.
+2. Add the path to `skills/perf/` directory as a doc source.
+3. Reference with `@Docs perf` in your prompts.
+</details>
+
+<details>
+<summary><b>🤖 Claude Code (CLAUDE.md)</b></summary>
+<br>
+
+1. Copy the `skills/perf` directory into your project root:
+   ```bash
+   cp -r perf-skills/skills/perf ./perf
+   ```
+2. In your `CLAUDE.md`, add:
+   ```markdown
+   For performance testing questions, read `perf/SKILL.md` for routing, then load the relevant reference files from `perf/references/`.
+   ```
+</details>
+
+<details>
+<summary><b> Cline / Roo Code</b></summary>
+<br>
+
+1. Copy the `skills/perf` directory into your project root.
+2. In your `.clinerules` or custom instructions, append:
+   ```text
+   For performance testing guidance, consult the perf knowledge base starting at perf/SKILL.md.
+   ```
+</details>
+
+<details>
+<summary><b>⚓ Aider</b></summary>
+<br>
+
+1. Copy the `skills/perf` directory into your project root.
+2. Tell Aider to read the entrypoint:
+   ```bash
+   /read perf/SKILL.md
+   ```
+3. Ask your question. Aider will load the corresponding tool/topic files from `perf/references/`.
+</details>
+
+---
+
+## 📂 File Structure & Routing
+
+Here is how the knowledge base is organized. The entry point [SKILL.md](skills/perf/SKILL.md) routes queries to specific reference guides to keep prompt sizes minimal.
+
+```text
+perf-skills/
 ├── .claude-plugin/
 │   ├── marketplace.json                  # Claude Code marketplace catalog
 │   └── plugin.json                       # Claude Code plugin manifest
@@ -167,73 +184,65 @@ perf-skills/                              # Repository root
 ├── LICENSE.md
 └── skills/
     └── perf/                             # Skill module
-        ├── SKILL.md                      # Entry point - tool selection, lifecycle, key principles
+        ├── SKILL.md                      # [Entry Point] Routing, Key Principles
         └── references/
-            ├── tools/                    # Tool-specific syntax and configuration
-            │   ├── jmeter.md             # JMeter 5.6+ - samplers, extractors, plugins, Groovy
-            │   ├── k6.md                 # k6 v0.50+ - executors, checks, thresholds, modules
-            │   ├── gatling.md            # Gatling 3.10+ - Scala/Java DSL, feeders, injection
-            │   ├── locust.md             # Locust 2.20+ - Python scripts, events, FastHttpUser
-            │   ├── neoload.md            # NeoLoad - GUI workflow, CLI, API execution
-            │   ├── loadrunner.md         # LoadRunner - VuGen, protocols, Controller scenarios
-            │   └── octoperf.md           # OctoPerf - JMeter-based SaaS, HAR import, cloud
-            └── topics/                   # Cross-cutting concepts (tool-agnostic)
-                ├── workload-design.md        # Concurrency models, load profiles, Little's Law
-                ├── test-data.md              # CSV, DB seeding, Faker, data isolation patterns
-                ├── script-generation.md      # Assertions, error handling, naming, structure
-                ├── correlation.md            # Dynamic values, extractors, regex, tool mappings
-                ├── test-execution.md         # Distributed, CI/CD (GitHub Actions, GitLab, Jenkins)
-                ├── results-analysis.md       # Percentiles, bottleneck framework, reporting
-                ├── observability.md          # APM, Prometheus, Grafana, tracing, JVM metrics
-                ├── production-testing.md     # Canary, shadow traffic, chaos, safety controls
-                ├── protocol-testing.md       # gRPC, GraphQL, WebSocket, Kafka/message queues
-                ├── database-testing.md       # JDBC, connection pools, slow queries, deadlocks
-                └── modern-architectures.md   # Microservices, K8s, serverless, browser/Web Vitals
+            ├── tools/                    # Tool-specific syntax & configs
+            │   ├── jmeter.md             # Apache JMeter (Groovy, Samplers, Extractors)
+            │   ├── k6.md                 # k6 JS/TS scripting, thresholds, VU options
+            │   ├── gatling.md            # Gatling Scala/Java DSL & feeders
+            │   ├── locust.md             # Locust Python scripting & FastHttpUser
+│   ├── artillery.md          # Artillery YAML/JS/TS scripting, phases, ensure SLOs, cloud
+            │   ├── neoload.md            # NeoLoad GUI configs & API execution
+            │   ├── loadrunner.md         # LoadRunner VuGen C scripting
+            │   └── octoperf.md           # OctoPerf JMeter-based cloud execution
+            └── topics/                   # Cross-cutting topics (Tool-agnostic)
+                ├── workload-design.md    # Little's Law, concurrency, pacing
+                ├── test-data.md          # Parameterization, CSV, data isolation
+                ├── script-generation.md  # Naming conventions, error handling
+                ├── correlation.md        # Dynamic values, Regex, JSONPath (Highlighted!)
+                ├── test-execution.md     # Distributed load, CI/CD pipelines
+                ├── results-analysis.md   # Latency percentiles, bottleneck triage
+                ├── observability.md      # Grafana, APM integration, VM profiling
+                ├── production-testing.md # Chaos engineering, canary safety controls
+                ├── protocol-testing.md   # gRPC, GraphQL, WebSockets, Kafka
+                ├── database-testing.md   # JDBC, pools, deadlocks, slow queries
+                ├── modern-architectures.md # K8s scaling, HPA, Frontend CWV
+                ├── llm-inference.md     # LLM serving: TTFT, TPOT/ITL, TPS, goodput, tools
+                └── slo-capacity.md      # SLOs, error budgets, headroom, CI gating
 ```
 
-## How the Skill Works
-
-### Routing Logic
-
-`SKILL.md` acts as the entry point and router. It contains:
+---
 
-1. **Loading Priority Rules** - tells the AI which file(s) to load based on the user's question (never all at once).
-2. **Reference Map** - maps user intent to the right file.
-3. **Protocol Routing Table** - maps protocols (gRPC, GraphQL, etc.) to recommended tools and reference files.
-4. **Tool Selection Matrix** - helps recommend a tool when the user hasn't chosen one.
-5. **Key Principles** - the single source of truth for cross-cutting best practices (assertions, think time, parameterization, correlation).
+## 💡 How the Skill Works
 
-### Design Principles
+To avoid hitting context window limits, the skill operates on a **selective loading** design:
 
-- **Token-efficient**: Tool files contain only tool-specific syntax. Cross-cutting concepts live in topic files. No duplication.
-- **Selective loading**: The AI loads 1-2 files per question, not the entire knowledge base.
-- **Single source of truth**: Each concept is defined in exactly one place. Tool files cross-reference topic files for shared concepts.
-- **Opinionated**: The skill prescribes best practices, not just documentation. It tells you what to do, not just what's possible.
+1. **Routing**: [SKILL.md](skills/perf/SKILL.md) is always read first.
+2. **Selective Context**: Based on the query, only the exact tool file (e.g., `k6.md`) and the relevant topic file (e.g., `correlation.md`) are loaded.
+3. **Consistency**: Global rules (like realistic think times and assertions) are enforced uniformly across all tool files by anchoring them in [SKILL.md](skills/perf/SKILL.md).
 
-## Example Questions This Skill Handles
+### ❓ Example Queries and Loaded Files
 
-| Question | Files Loaded |
-|---|---|
-| "Help me write a k6 load test for our REST API" | `k6.md` |
-| "How should I design the workload for our e-commerce app?" | `workload-design.md` |
-| "Set up JMeter in our GitHub Actions pipeline" | `test-execution.md` + `jmeter.md` |
-| "Our p95 latency is spiking at 500 VUs - how do I debug?" | `results-analysis.md` + `observability.md` |
-| "How do I load test a gRPC service?" | `protocol-testing.md` + `k6.md` |
-| "What tool should I use? We're a Python team." | `SKILL.md` (Tool Selection Matrix) |
-| "Test our Kafka consumer throughput" | `protocol-testing.md` |
-| "Validate our K8s HPA scales correctly under load" | `modern-architectures.md` |
-| "Load test our PostgreSQL connection pool" | `database-testing.md` + `jmeter.md` |
+| Query | Files Loaded |
+| :--- | :--- |
+| *"Help me write a k6 load test for our REST API"* | `k6.md` |
+| *"How should I correlate a dynamic JSESSIONID in JMeter?"* | `jmeter.md` + `correlation.md` |
+| *"Set up a distributed Locust execution in GitLab CI"* | `locust.md` + `test-execution.md` |
+| *"Our p95 latency spikes during database writes, how do we fix?"* | `database-testing.md` + `results-analysis.md` |
+| *"What TTFT/throughput can our vLLM endpoint sustain at 200 concurrent users?"* | `llm-inference.md` |
+| *"How many replicas do we need, and what SLO headroom should we keep?"* | `slo-capacity.md` |
 
-## Contributing
+---
 
-To add or update content:
+## 🤝 Contributing
 
-1. **Tool-specific content** goes in `references/tools/<tool>.md` - syntax, config, tool-unique tips only.
-2. **Cross-cutting concepts** go in `references/topics/<topic>.md` - patterns that apply across tools.
-3. **Never duplicate** - if a concept exists in a topic file, tool files should cross-reference it, not restate it.
-4. **Update SKILL.md** if you add a new file - add it to the Reference Map and Protocol Routing Table if applicable.
-5. **Add a version indicator** (`> Targets: ...`) to new tool files.
+We welcome contributions to expand the knowledge base:
+1. **Tool-specific syntax** belongs in `references/tools/`.
+2. **Generic concepts** belong in `references/topics/`.
+3. **Keep it DRY**: Reference topic files from tool files instead of duplicating.
+4. **Update the Index**: Remember to update the Reference Map in [SKILL.md](skills/perf/SKILL.md) when adding files.
 
-## License
+## 📄 License
 
-See repository license.
+This project is licensed under the MIT License. See [LICENSE.md](LICENSE.md) for details.
+tails.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "perf-skills",
-  "version": "2.0.0",
+  "version": "3.0.0",
   "description": "Expert AI skill for performance testing - covers JMeter, k6, Gatling, Locust, NeoLoad, LoadRunner, OctoPerf across the full lifecycle",
   "main": "skills/perf/SKILL.md",
   "files": [

package/skills/perf/SKILL.md

@@ -1,14 +1,17 @@
 ---
 name: perf
 description: Performance testing expert covering the full lifecycle for
-  JMeter, k6, Gatling, Locust, NeoLoad, and LoadRunner. Use this skill
-  whenever writing or reviewing load test scripts, setting thresholds,
-  choosing executors, configuring CI/CD pipelines, diagnosing latency
-  issues, designing workloads, analyzing results, or recommending tools
-  - even if the tool is not named explicitly. Always consult before
-  suggesting thresholds, executor types, or output configuration.
-  Prefer this skill over general knowledge for any performance testing
-  decision, debugging session, or tool comparison.
+  JMeter, k6, Gatling, Locust, NeoLoad, LoadRunner, and Artillery, plus
+  LLM inference (vLLM, TRT-LLM, SGLang, OpenAI-compatible endpoints).
+  Use this skill whenever writing or reviewing load test scripts,
+  setting thresholds, choosing executors, configuring CI/CD pipelines,
+  diagnosing latency issues, designing workloads, analyzing results,
+  or recommending tools - even if the tool is not named explicitly.
+  Always consult before suggesting thresholds, executor types, or
+  output configuration. Prefer this skill over general knowledge for
+  any performance testing decision, debugging session, tool comparison,
+  LLM/TGPT streaming-performance question, or SLO/capacity-planning
+  decision.
 ---
 
 # Performance Testing Skill
@@ -46,6 +49,7 @@ Multiple files may apply.
 | k6 scripting, extensions, cloud                  | `references/tools/k6.md`                          |
 | Gatling simulations, Scala/Java DSL              | `references/tools/gatling.md`                     |
 | Locust Python tests, distributed                 | `references/tools/locust.md`                      |
+| Artillery YAML/JS/TS scripts, cloud              | `references/tools/artillery.md`                   |
 | NeoLoad projects, GUI, APIs                      | `references/tools/neoload.md`                     |
 | LoadRunner scripts, protocols, VuGen             | `references/tools/loadrunner.md`                  |
 | OctoPerf cloud test management                   | `references/tools/octoperf.md`                    |
@@ -60,6 +64,8 @@ Multiple files may apply.
 | gRPC, GraphQL, WebSocket, messaging protocols    | `references/topics/protocol-testing.md`           |
 | Database load testing (JDBC, connection pools)   | `references/topics/database-testing.md`           |
 | Microservices, K8s, serverless performance       | `references/topics/modern-architectures.md`       |
+| LLM inference: TTFT, TPOT/ITL, TPS, goodput      | `references/topics/llm-inference.md`             |
+| SLOs, error budgets, capacity & headroom         | `references/topics/slo-capacity.md`             |
 
 ### Protocol Routing Table
 
@@ -76,6 +82,8 @@ right tool and reference:
 | Kafka / Message Queues | k6 (xk6-kafka), JMeter       | `references/topics/protocol-testing.md`                      |
 | SOAP / WSDL            | LoadRunner, JMeter           | Tool file                                                     |
 | SAP / Citrix           | LoadRunner, NeoLoad          | Tool file                                                     |
+| LLM inference (streaming) | vLLM bench, GenAI-Perf, GuideLLM, llmperf | `references/topics/llm-inference.md`             |
+| SLO/capacity (error budgets, headroom) | any | `references/topics/slo-capacity.md` |
 
 ---
 
@@ -83,16 +91,16 @@ right tool and reference:
 
 Use this to recommend the right tool when the user hasn't decided yet.
 
-| Criteria             | JMeter              | k6                    | Gatling              | Locust         | NeoLoad          | LoadRunner              | OctoPerf              |
+| Criteria             | JMeter              | k6                    | Gatling              | Locust         | Artillery        | NeoLoad          | LoadRunner              | OctoPerf              |
 |----------------------|---------------------|-----------------------|----------------------|----------------|------------------|-------------------------|-----------------------|
 | **Language**         | GUI/XML + Groovy    | JavaScript/TypeScript | Scala/Java           | Python         | GUI + NeoLoad DSL| VuGen C-like            | Web UI (JMeter-based) |
 | **Open source**      | ✅                  | ✅                    | ✅                   | ✅             | ❌               | ❌                      | ❌ (SaaS)             |
 | **Protocol support** | HTTP, JDBC, JMS, MQTT, FTP, gRPC | HTTP, gRPC, WS | HTTP, JMS, gRPC | HTTP, gRPC | HTTP, gRPC, WS, SAP | HTTP, Citrix, SAP, Flex | HTTP (JMeter-backed) |
 | **Developer-friendly** | Medium            | High                  | High                 | High           | Low              | Low                     | Medium                |
-| **Enterprise support** | Community + BlazeMeter | Grafana Cloud    | Gatling Enterprise   | Limited        | ✅               | ✅                      | ✅                    |
+| **Enterprise support** | Community + BlazeMeter | Grafana Cloud    | Gatling Enterprise   | Limited        | Artillery Cloud | ✅               | ✅                      | ✅                    |
 | **CI/CD integration** | Good (Maven/Gradle) | Excellent            | Excellent            | Good           | Good             | Moderate                | Good                  |
-| **Cloud execution**  | BlazeMeter, OctoPerf | Grafana Cloud        | Gatling Enterprise   | Self-managed   | NeoLoad Cloud    | AWS/on-prem             | OctoPerf Cloud        |
-| **Best for**         | Legacy systems, JDBC, protocols | Modern APIs, TypeScript devs | High-throughput HTTP | Python teams, flexible | SAP/Citrix enterprise | Mainframe, legacy enterprise | JMeter teams needing cloud UI |
+| **Cloud execution**  | BlazeMeter, OctoPerf | Grafana Cloud        | Gatling Enterprise   | Self-managed   | Artillery Cloud (Lambda/Fargate) | NeoLoad Cloud    | AWS/on-prem             | OctoPerf Cloud        |
+| **Best for**         | Legacy systems, JDBC, protocols | Modern APIs, TypeScript devs | High-throughput HTTP | Python teams, flexible | Node teams, YAML tests, cloud scale | SAP/Citrix enterprise | Mainframe, legacy enterprise | JMeter teams needing cloud UI |
 
 ### Quick decision rules
 
@@ -107,6 +115,7 @@ Use this to recommend the right tool when the user hasn't decided yet.
   Correlation Recorder) or LoadRunner
 - **gRPC or GraphQL APIs** → k6 or Gatling
 - **Message queues (Kafka, RabbitMQ)** → k6 (xk6-kafka) or JMeter
+- **Node shop / prefer YAML over code** → Artillery (YAML, JS, TS; easy Lambda/Fargate cloud scale)
 
 ---
 
@@ -181,26 +190,44 @@ ask about them.
   that skews both throughput and latency measurements. Always run
   workers on separate machines or containers for distributed tests.
 
----
+### Artillery
+
+- **No `ensure` block** - without `ensure`, Artillery reports metrics
+  but always exits 0, so CI never fails on latency or error spikes.
+  Always add `ensure.thresholds` (or `conditions`) for SLA gates.
+- **`arrivalRate` mistaken for concurrency** - it is new users per
+  second (open model). On a slow backend, pending VUs pile up unbounded.
+  Set `maxVusers` to cap real concurrency, or use `arrivalCount`.
+- **Strict captures aborting VUs** - captures are strict by default;
+  a missed extractor stops the whole VU. Only set `strict: false`
+  when a downstream 404 is acceptable.
+- **`payload.order: sequence` in distributed runs** - sequential CSV
+  consumption breaks under Lambda/Fargate workers (each has its own
+  copy). Use the default `random` ordering for distributed tests.
+- **Zero `think` time** - 1 VU/sec with no think time fires the
+  maximum RPS for the journey; add `think` to model real pacing.
+- **Forgetting `http.response_time` is TTFB** - latency metrics are
+  time-to-first-byte by default. Enable `config.http.extendedMetrics`
+  for full download timing (`http.total.*`) when that matters.
 
-## Cross-Tool Concept Mapping
+---
 
 Use this when users are migrating between tools or asking how a
 concept from one tool maps to another. Claude should always provide
 the specific mapping rather than a generic explanation.
 
-| Concept          | JMeter                  | k6                      | Gatling                  | Locust                   | LoadRunner          |
-|------------------|-------------------------|-------------------------|--------------------------|--------------------------|---------------------|
-| Virtual user     | Thread                  | VU                      | User                     | User                     | Vuser               |
-| Test plan        | .jmx file               | .js / .ts script        | Simulation class         | .py file                 | VuGen script (.usr) |
-| User entrypoint  | Thread Group            | `default()` function    | `scenario()`             | task methods             | `Action()`          |
-| Concurrency ctrl | Thread Group settings   | executor                | `inject()`               | `spawn_rate`             | Vuser Group         |
-| Think time       | Constant/Uniform Timer  | `sleep()`               | `pause()`                | `time.sleep()`           | `lr_think_time()`   |
+| Concept          | JMeter                  | k6                      | Gatling                  | Locust                   | Artillery                | LoadRunner          |
+|------------------|-------------------------|-------------------------|--------------------------|--------------------------|--------------------------|---------------------|
+| Virtual user     | Thread                  | VU                      | User                     | User                     | VU (arrival per sec)     | Vuser               |
+| Test plan        | .jmx file               | .js / .ts script        | Simulation class         | .py file                 | .yml / .js / .ts script  | VuGen script (.usr) |
+| User entrypoint  | Thread Group            | `default()` function    | `scenario()`             | task methods             | `flow` in scenario       | `Action()`          |
+| Concurrency ctrl | Thread Group settings   | executor                | `inject()`               | `spawn_rate`             | `maxVusers` / `arrivalRate` | Vuser Group         |
+| Think time       | Constant/Uniform Timer  | `sleep()`               | `pause()`                | `time.sleep()`           | `think`                  | `lr_think_time()`   |
 | Inline assertion | Response Assertion      | `check()`               | `.check()`               | `catch_response`         | `lr_eval_string()`  |
-| SLA enforcement  | Duration Assertion      | `thresholds`            | Assertions (Enterprise)  | custom + exit code       | SLA definition      |
-| Correlation      | Regex / CSS Extractor   | `res.json()` / regex    | `.check()` + `saveAs()`  | `response.text` + regex  | `web_reg_save_param`|
-| Data feed        | CSV Data Set Config     | `SharedArray`           | `feeder`                 | CSV reader               | `lr_paramarr()`     |
-| Grouping         | Transaction Controller  | `group()`               | `group()`                | task sets                | Transaction         |
+| SLA enforcement  | Duration Assertion      | `thresholds`            | Assertions (Enterprise)  | custom + exit code       | `ensure` plugin          | SLA definition      |
+| Correlation      | Regex / CSS Extractor   | `res.json()` / regex    | `.check()` + `saveAs()`  | `response.text` + regex  | `capture` (json/xpath/regexp/header) | `web_reg_save_param`|
+| Data feed        | CSV Data Set Config     | `SharedArray`           | `feeder`                 | CSV reader               | `payload` / `variables`  | `lr_paramarr()`     |
+| Grouping         | Transaction Controller  | `group()`               | `group()`                | task sets                | `name` on scenario       | Transaction         |
 | Distributed      | Controller + Agents     | k6 cloud / k6 operator  | Gatling Enterprise       | master + workers         | Load Generator      |
 | Results output   | .jtl (CSV/XML)          | JSON / InfluxDB / cloud | simulation.log           | CSV / Locust web UI      | .lrr file           |
 
@@ -317,4 +344,4 @@ prescribing a solution:
 - Where will tests **run from** (local, CI, cloud)?
 - What **environment** is being tested (dev, staging, prod)?
 - Is there an **APM tool** in place (Datadog, Dynatrace, Grafana,
-  New Relic)?
+  New Relic)??)??

package/skills/perf/references/tools/artillery.md

@@ -0,0 +1,331 @@
+# Artillery Reference
+
+> Targets: Artillery v2.x (YAML, JS, and TS test definitions; `http`, `playwright`, `socketio`, `ws` engines)
+
+Artillery is a developer-centric, open-source load testing tool that runs on Node.js. Tests can be written in YAML, JavaScript, or TypeScript, and it scales from a single laptop to distributed AWS Lambda / Fargate runs (Artillery Cloud). It is the closest JS/TS-native alternative to k6 and is a strong fit for teams already in the Node ecosystem.
+
+---
+
+## Core Concepts
+
+| Concept | Description |
+|---|---|
+| **Virtual User (VU)** | A single simulated user executing a `flow` |
+| **Arrival (open model)** | New VUs generated per second - the default load model |
+| **Phase** | A timed load segment (`arrivalRate`, `rampTo`, `arrivalCount`, `pause`) |
+| **Scenario** | Named user journey; a `flow` of requests/actions |
+| **Flow** | Ordered list of actions (request, `think`, `capture`, `loop`, `function`) |
+| **Capture** | Extract a dynamic value from a response for later reuse (correlation) |
+| **Processor** | Custom JS/TS module supplying hooks and metric logic |
+| **`ensure`** | SLO/assertion plugin - FAILS the run (non-zero exit) if breached |
+| **Environment** | Named config profile switched with `-e` |
+
+> Artillery uses an **open (arrival-rate) load model by default**. `arrivalRate` is *new users per second*, NOT concurrent users. Use `maxVusers` to cap real concurrency. For closed/concurrency modeling, see `../topics/workload-design.md`.
+
+---
+
+## Script Structure (YAML)
+
+```yaml
+config:
+  target: 'https://staging.example.com'
+  phases:
+    - duration: '2m'
+      arrivalRate: 10
+      rampTo: 50
+      name: ramp-up
+    - duration: '5m'
+      arrivalRate: 50
+      maxVusers: 200
+      name: sustain
+  ensure:
+    thresholds:
+      - 'http.response_time.p95': 500
+      - 'http.response_time.p99': 1000
+    conditions:
+      - expression: 'http.codes.5xx < http.codes.2xx * 0.01'  # <1% 5xx
+  processor: './helpers.js'
+
+scenarios:
+  - name: 'Browse + Checkout'
+    weight: 1
+    flow:
+      - post:
+          url: '/auth'
+          json:
+            username: '{{ username }}'
+            password: '{{ password }}'
+          capture:
+            - json: '$.id_token'
+              as: token
+      - think: 2
+      - get:
+          url: '/products'
+          headers:
+            authorization: 'Bearer {{ token }}'
+      - post:
+          url: '/checkout'
+          json:
+            itemId: '{{ $uuid }}'
+          capture:
+            - json: '$.orderId'
+              as: orderId
+      - get:
+          url: '/orders/{{ orderId }}'
+          headers:
+            authorization: 'Bearer {{ token }}'
+```
+
+---
+
+## Script Structure (JS / TS)
+
+```javascript
+export const config = {
+  target: 'https://staging.example.com',
+  phases: [
+    { duration: '2m', arrivalRate: 10, rampTo: 50, name: 'ramp-up' },
+    { duration: '5m', arrivalRate: 50, maxVusers: 200, name: 'sustain' },
+  ],
+  ensure: {
+    thresholds: [
+      'http.response_time.p95: 500',
+      'http.response_time.p99: 1000',
+    ],
+  },
+  processor: './helpers.js',
+};
+
+export const scenarios = [
+  {
+    name: 'Browse + Checkout',
+    flow: [
+      {
+        post: {
+          url: '/auth',
+          json: { username: '{{ username }}', password: '{{ password }}' },
+          capture: [{ json: '$.id_token', as: 'token' }],
+        },
+      },
+      { think: 2 },
+      {
+        get: {
+          url: '/products',
+          headers: { authorization: 'Bearer {{ token }}' },
+        },
+      },
+    ],
+  },
+];
+```
+
+---
+
+## Load Phases
+
+`config.phases` is an array executed sequentially. Four phase kinds:
+
+| Phase kind | Key options | Use case |
+|---|---|---|
+| **Constant arrival** | `arrivalRate` | Steady RPS-style load (open model) |
+| **Ramp** | `arrivalRate` + `rampTo` (both over `duration`) | Warm-up / ramp-up |
+| **Fixed count** | `arrivalCount` | Exact N total users spread over `duration` |
+| **Pause** | `pause` | Idle gap (soak cool-down, between spikes) |
+
+```yaml
+phases:
+  - duration: '30m'
+    arrivalRate: 1
+    rampTo: 100
+    name: ramp-up
+  - duration: '3h'
+    arrivalRate: 100
+    name: sustain          # soak/endurance
+  - duration: '1m'
+    arrivalRate: 500
+    name: spike            # spike test
+  - pause: 60
+```
+
+- `duration` / `pause` accept human-readable units (`'5m'`, `'3h'`) as well as seconds.
+- `maxVusers` caps in-flight VUs for any phase - essential to bound concurrency on slow servers (open-model load otherwise queues unbounded pending VUs).
+- `name` makes phases identifiable in CLI output and Artillery Cloud.
+
+---
+
+## Correlation (Capture / Dynamic Values)
+
+Use `capture` on a request to extract a value for later steps. Requires `as` and one extractor:
+
+| Extractor | Syntax | Example |
+|---|---|---|
+| JSONPath | `json: '$.path'` | `json: '$.id_token'` |
+| XPath | `xpath: '//node/text()'` | SOAP / XML bodies |
+| Regex | `regexp: 'pattern'`, optional `group`, `flags` | `regexp: 'sid=([^&]+)'` |
+| Header | `header: 'X-Custom'` | `header: 'Set-Cookie'` |
+| Selector | `selector: 'a.product'`, `attr`, `index` | HTML scraping |
+
+```yaml
+- get:
+    url: '/login'
+    capture:
+      - json: '$.csrf'
+        as: csrf
+      - header: 'set-cookie'
+        as: cookie
+- post:
+    url: '/submit'
+    headers:
+      x-csrf-token: '{{ csrf }}'
+    cookie:
+      session: '{{ cookie }}'
+```
+
+- **Captures are strict by default**: a failed capture stops that VU. Set `strict: false` only when a later request can safely 404.
+- For multi-step journeys, capture once near the top and reuse via `{{ var }}` in every later request.
+- Capture multiple values from one response with an array of capture specs.
+
+> For framework-specific extraction rules (ASP.NET ViewState, JSF, JWT, SAP `sap-contextid`), see `../topics/correlation.md`.
+
+---
+
+## Parameterization
+
+### CSV payload (`config.payload`)
+```yaml
+config:
+  payload:
+    path: 'users.csv'
+    fields: ['username', 'password']
+    skipHeader: true
+    order: sequence        # 'random' (default) | 'sequence'
+scenarios:
+  - flow:
+      - post:
+          url: '/auth'
+          json:
+            username: '{{ username }}'
+            password: '{{ password }}'
+```
+- `order: sequence` is deterministic but **breaks under distributed runs** (each worker has its own copy). Use `random` (default) for distributed tests.
+- `loadAll: true` + `name` exposes the whole dataset to each VU for `loop`.
+- `cast: false` keeps values as strings; `delimiter` overrides the comma.
+
+### Inline variables (`config.variables`)
+```yaml
+config:
+  variables:
+    postcode: ['SE1', 'EC1', 'E8']
+    id: ['8731', '9965', '2806']
+```
+One value is picked at random per VU. Cannot template `config` values.
+
+### Environment variables (`$env`)
+```yaml
+headers:
+  x-api-key: '{{ $env.API_KEY }}'
+```
+Run with `API_KEY=xxx artillery run script.yml` or `--env-file .env`. Keeps secrets out of source.
+
+### Environments (`-e`)
+Reuse one script across dev/staging/prod by defining `config.environments` with per-env `target` and `phases`:
+```bash
+artillery run -e production script.yml
+```
+Access the active name via `{{ $environment }}` (e.g. to pick a CSV: `path: '{{ $environment }}-logins.csv'`).
+
+---
+
+## SLO Checks with `ensure` (Assertions)
+
+`ensure` is Artillery's SLA gate - **without it, the run reports metrics but always exits 0**, so CI never fails on latency. Always add it for CI.
+
+```yaml
+config:
+  plugins:
+    ensure:
+      thresholds:                 # value must be LESS than this
+        - 'http.response_time.p95': 500
+        - 'http.response_time.p99': 1000
+      conditions:                 # advanced boolean/numeric expressions
+        - expression: 'http.response_time.p95 < 500 and http.request_rate > 1000'
+        - expression: 'http.codes.5xx <= http.codes.2xx * 0.01'
+          strict: false           # optional check; failure won't fail the run
+```
+
+- `thresholds` check a metric's aggregate is **below** the integer.
+- `conditions` combine metrics with `+ - * / % ^`, comparisons, `and`/`or`/`not`, and `ceil/floor/round`.
+- `strict: true` (default) fails the run on breach; `strict: false` reports only.
+- Using a non-existent metric name makes that check fail.
+
+### Key metrics for `ensure`
+| Metric | Meaning |
+|---|---|
+| `http.response_time.p95` / `.p99` | Latency percentile (ms) |
+| `http.request_rate` | Requests/sec |
+| `http.codes.2xx` / `.4xx` / `.5xx` | Status-code counters |
+| `http.downloaded_bytes` | Total payload bytes |
+| `vusers.completed` / `vusers.failed` | VU outcomes |
+
+> `http.response_time.*` is **TTFB** by default. Enable `config.http.extendedMetrics: true` for full `http.total.*` (download-complete) timing. For SLA baseline tables, see SKILL.md "Threshold Starting Points" and `../topics/results-analysis.md`.
+
+---
+
+## Custom Logic (Processor Hooks)
+
+Load JS/TS via `config.processor`:
+
+```javascript
+// helpers.js
+module.exports = {
+  setApiKey(context, events, done) {
+    context.vars.apiKey = process.env.API_KEY;
+    return done();
+  },
+  assertOrder(context, events, done) {
+    const status = context.vars.orderStatus;
+    if (status !== 'confirmed') {
+      events.emit('counter', 'order_failures', 1);
+    }
+    return done();
+  },
+};
+```
+
+Hook points:
+- **`beforeRequest` / `afterResponse`** - set on a request; customize/inspect URL, headers, body.
+- **`beforeScenario` / `afterScenario`** - set on a scenario.
+- **`function`** step - run arbitrary code mid-flow.
+
+Async hooks are supported (v2.0.7+). Use `events.emit('counter'|'histogram'|'rate', name, value)` for custom metrics.
+
+---
+
+## Output and Observability
+
+| Output | How |
+|---|---|
+| Terminal summary | Default (`artillery run script.yml`) |
+| JSON report | `artillery run -o json=report.json script.yml` |
+| Artillery Cloud | `artillery run --record script.yml` (dashboards, historical trends) |
+| CSV | `artillery run -o csv=results.csv script.yml` |
+| Distributed | AWS Lambda / Fargate workers via `artillery run-fargate` |
+
+Enable `config.http.distributedTracing: true` to attach a W3C `traceparent` header to every request - correlates load with backend spans in your APM.
+
+---
+
+## Artillery-Specific Tips
+
+- **`arrivalRate` is new-users-per-second, not concurrency.** A slow backend makes pending VUs pile up. Always set `maxVusers` to bound real concurrency, or switch to `arrivalCount`/closed-model thinking via `workload-design.md`.
+- **Always add `ensure` for CI.** A test with no `ensure` exits 0 regardless of latency or error spikes - it generates traffic but enforces nothing.
+- **Captures are strict by default.** A missed extractor aborts the whole VU. Only set `strict: false` when a downstream 404 is acceptable.
+- **Add `think` between steps.** Zero think time maximizes RPS unrealistically; use `think` (seconds or `ms` units) to model real pacing.
+- **Default `payload.order` is `random`** - deterministic `sequence` ordering does not work correctly in distributed runs.
+- **`http.response_time` is TTFB.** Turn on `extendedMetrics` if you need full download time (`http.total.*`).
+- **Secrets via `$env` / `--env-file`**, never inline. Use `config.environments` + `-e` to promote the same script dev → staging → prod.
+- **Avoid heavy `log` actions under load** - they add overhead; prefer `ensure`/custom counters for visibility.
+- **Browser load?** Use the `playwright` engine (`engines: { playwright: {} }`) to drive real pages; note it is far heavier per VU than HTTP.
+
+> For CI/CD integration (GitHub Actions, GitLab CI, distributed execution), see `../topics/test-execution.md`.
+> For anti-patterns, assertions, think time, and parameterization principles, see **Key Principles** in `SKILL.md`.

package/skills/perf/references/topics/llm-inference.md

@@ -0,0 +1,131 @@
+# LLM Inference Performance
+
+> Scope: load and capacity testing of LLM inference servers (vLLM, TRT-LLM, SGLang, OpenAI-compatible endpoints, Ray Serve LLM, KServe). Covers metric definitions, workload design, tools, and SLO methodology for generative AI.
+
+LLM inference is **not conventional HTTP load testing**. Responses stream token-by-token over an open connection, output length is unbounded, and the dominant cost is GPU compute, not network. A generic `k6`/`JMeter` run that only measures request latency will badly misreport performance - it ignores the streaming shape entirely. This topic defines the metrics that actually matter, how to design realistic LLM workloads, and which tools to use.
+
+> For general workload theory (Little's Law, open vs closed models), see `workload-design.md`. For analyzing the percentiles and SLO failures this produces, see `results-analysis.md`. For running these tests in CI/CD, see `test-execution.md`.
+
+---
+
+## Why LLM load testing is different
+
+| Aspect | Traditional API | LLM inference |
+|---|---|---|
+| Response | Single body, fixed size | Streamed tokens, variable length |
+| Latency signal | TTFB + total | TTFT + inter-token gaps + E2E |
+| Cost driver | CPU/network | GPU memory (KV cache), compute |
+| Throughput unit | Requests/sec | Tokens/sec (not requests/sec) |
+| Concurrency limit | Threads/sockets | KV cache capacity (`max_num_seqs`) |
+| Key risk | Timeouts | Queueing delay, GPU saturation, OOM |
+
+Two phases dominate every request:
+- **Prefill** - the model processes the full prompt to build the KV cache. Compute-heavy, determines **TTFT**. Scales with prompt length.
+- **Decode** - tokens generated one at a time using the KV cache. Memory-bandwidth-bound, determines **ITL/TPOT**. Scales with output length.
+
+---
+
+## Core Metrics
+
+### TTFT - Time to First Token
+Time from request send to first streamed token. Driven by prompt length, queue/self.time, and prefill speed. The primary *perceived responsiveness* metric for chat/coding assistants. High TTFT under load usually means the scheduler is queuing requests (KV cache exhausted) rather than a slow model.
+
+### TPOT / ITL - Generation smoothness
+- **TPOT (Time Per Output Token)** - average gap between tokens for a single request: `(E2E - TTFT) / (output_tokens - 1)`. Report the mean of per-request TPOTs.
+- **ITL (Inter-Token Latency)** - same gaps, but token-weighted across all requests (mean of every gap). This is the system's steady streaming speed.
+- **Which to use**: TPOT compares per-request behavior (each request equal); ITL estimates system-wide streaming feel across mixed traffic. A bursty ITL (high p99 vs mean) makes output appear in clumps - the classic "GPU contention" signature.
+
+### E2E Latency
+Total time from prompt to final token. `E2E = TTFT + (output_tokens - 1) * TPOT`. Matters for batch/codegen/summarization where the full response is needed before the next step.
+
+### TPS - Token Throughput
+- **System TPS** - total output tokens/sec across all requests. Raw capacity; rises with load until GPU-saturated. `TPS = output_tokens / (T_last - T_first)`.
+- **User TPS** - tokens/sec a single user experiences; `≈ 1 / ITL` at long outputs. Drops as concurrency rises because the engine shares the GPU.
+
+### RPS - Request Throughput
+Completed requests/sec. Dominant metric for high-volume short-prompt traffic (chatbots, search, API gateways). Shorter prompts + higher `max_num_seqs` raise RPS.
+
+### Goodput (the metric that matters most)
+`Goodput = (requests meeting ALL SLOs) / total_requests * 100%`. Unlike TPS/RPS, goodput tells you what fraction of users got an *acceptable* experience. A system can show high TPS while most requests violate latency SLOs. Always define goodput with explicit thresholds, e.g. TTFT < 500ms, TPOT < 15ms, E2E < 2s. As load climbs, goodput falls even as raw throughput keeps rising - that crossover is your real capacity limit.
+
+### Percentiles
+Report **p50/p95/p99** for TTFT, ITL, and E2E - never just averages. Averages hide the unlucky 5% whose tokens arrive in bursts. p99 is the near-worst-case; if p99 meets SLO, the system is consistent.
+
+---
+
+## Workload Dimensions
+
+Controlling these is what separates a useful LLM test from a misleading one:
+
+| Dimension | Effect | How to set |
+|---|---|---|
+| **Prompt length (ISL)** | Longer → higher TTFT, more KV cache | Use real traffic (ShareGPT) or synthetic ranges |
+| **Output length (OSL)** | Longer → higher E2E, more decode cost | Real distributions; `--random-output-len` for synthetic |
+| **Concurrency** | Bounded by KV cache, not sockets | Set `--max-concurrency` to simulate gateway limits |
+| **Request rate** | Open-model arrival (Poisson) | `--request-rate`; `inf` for max throughput |
+| **Burstiness** | Gamma-distributed arrivals | `1.0` realistic, `0.1-0.5` stress, `2-5` uniform |
+| **Batching (`max_num_seqs`)** | Higher → more RPS, worse per-user latency | Tune on the server, not the client |
+
+**KV cache math**: `max_concurrency ≈ KV_cache_tokens / max_model_len`. vLLM prints this at startup. Set test concurrency to 80-90% of it for capacity planning; use the full value as the SLA limit.
+
+---
+
+## Tools
+
+Purpose-built LLM benchmarkers surface token metrics natively. Prefer them over generic HTTP load testers.
+
+| Tool | Best for | Notes |
+|---|---|---|
+| **vLLM `bench serve`** | Serving benchmarks against an OpenAI-compatible endpoint | Native TTFT/TPOT/ITL/TPS, ShareGPT + synthetic datasets, ramp-up, goodput via percentiles |
+| **GuideLLM** | Production vLLM SLA/capacity studies | Auto reports, live progress, profile-based; recommended by vLLM for production |
+| **NVIDIA GenAI-Perf** | Multi-backend (TRT-LLM, Triton, vLLM), concurrency/rate modes | Emits TTFT, ITL, output TPS, goodput; pairs with Perf Analyzer |
+| **llmperf / LLM-Perf** | Quick pointwise latency/throughput checks | Lightweight, great for smoke and regression |
+| **LLM Locust** | Distributed load on Locust, GenAI metrics | Use when you already run Locust fleets |
+| **k6 + custom metrics** | Unified CI with existing k6 stacks | Must instrument streaming manually: capture TTFT from first SSE chunk, ITL between chunks, count tokens. See `../tools/k6.md` for metric primitives |
+
+> **Anti-pattern (per current research):** do not misuse model-server micro-benchmarkers (vLLM `bench`, SGLang bench) as production-level evaluators. They optimize for regression/feature testing, not realistic arrival patterns. Use them for component baselines; use GuideLLM/GenAI-Perf/production-style load for capacity and SLO validation.
+
+---
+
+## Test Design & Methodology
+
+1. **Start from real traffic shapes.** Use ShareGPT or captured production traces for prompt/output length distributions. Synthetic `random` datasets are fine for stress but unrealistic for sizing.
+2. **Define SLOs as goodput thresholds first.** e.g. "TTFT p95 < 500ms, ITL p95 < 20ms, E2E p95 < 3s" → goodput target 99%.
+3. **Run a max-throughput probe** (`--request-rate inf --max-concurrency <limit>`) to find the concurrency ceiling and baseline TPS.
+4. **Sweep concurrency / request rate** to find the goodput cliff - the load where SLO compliance drops. That is capacity.
+5. **Test ramp-up and spikes** (linear/exponential ramp, bursty arrival) to validate autoscaling and queue behavior.
+6. **Always assert on token metrics, not just HTTP 200.** A 200 with a 10s TTFT is a failed request.
+
+### Workload pattern recipes (vLLM bench semantics)
+| Goal | `--request-rate` | `--burstiness` | `--max-concurrency` |
+|---|---|---|---|
+| Max throughput | `inf` | n/a | limited |
+| Realistic baseline | 5-20 | 1.0 | inf |
+| Stress / resilience | 20-100 | 0.1-0.5 | inf |
+| Latency profiling | 1-10 | 2-5 | inf |
+| Capacity / SLA | target rate | 1.0 | SLA limit |
+
+---
+
+## Common Mistakes
+
+- **Measuring only HTTP latency** - TTFT and streaming gaps are invisible to `http_req_duration`. You'll report a green test while users stare at a blank cursor.
+- **Ignoring output length variance** - fixed `max_tokens` hides the real E2E spread; use realistic OSL distributions.
+- **Unbounded concurrency** - without `--max-concurrency`, the client fires until the GPU chokes; you measure collapse, not capacity. The KV cache, not sockets, is the real cap.
+- **Confusing TPS with user experience** - high system TPS can coexist with terrible per-user ITL. Track both; report goodput.
+- **Averaging instead of percentiles** - a 200ms mean TTFT with 8s p99 is a broken chat UX.
+- **Using a micro-benchmarker for production SLOs** - `vllm bench`/SGLang bench validate the engine, not your serving capacity under real arrival patterns.
+- **No streaming instrumentation in k6** - a naive `http.get` to a streaming endpoint counts only the final byte; you must parse SSE chunks to get TTFT/ITL.
+- **Forgetting prompt caching** - prefix caching dramatically cuts TTFT for repeated prefixes; test with and without it (e.g. RAG, system prompts) to size the win.
+
+---
+
+## Observability
+
+Track these server-side alongside the client metrics above:
+- **KV cache utilization** - the true saturation signal; when it pins at 100%, TTFT inflates via queueing.
+- **Batch size / `num_running` vs `num_waiting`** - waiting > 0 means you're over concurrency.
+- **Prefill vs decode time split** - isolates whether slowness is prompt-side or generation-side.
+- **GPU util + memory** - confirms you're compute-bound, not starved.
+
+> For dashboards/APM integration patterns, see `observability.md`. For interpreting the goodput cliff and saturation, see `results-analysis.md`.

package/skills/perf/references/topics/slo-capacity.md

@@ -0,0 +1,136 @@
+# SLO & Capacity Planning
+
+> Scope: turning load-test results into service level objectives, error budgets, and capacity decisions. Complements `workload-design.md` (designing the load) and `results-analysis.md` (reading the numbers). Read those first if you have not designed or run the test yet.
+
+This topic covers the *decision* layer of performance engineering: how to define SLIs/SLOs that mean something, how to convert a load-test curve into a capacity number and a headroom plan, and how to enforce all of it as an automated CI gate.
+
+> For LLM-specific SLOs (TTFT, goodput), see `llm-inference.md`. For the tool syntax that enforces SLOs (`ensure`, `thresholds`), see the relevant tool file.
+
+---
+
+## SLI / SLO / Error Budget
+
+### Definitions
+- **SLI (Service Level Indicator)** - the actual measured signal, e.g. request p95 latency, error rate, availability.
+- **SLO (Service Level Objective)** - the target you commit to for an SLI over a window, e.g. "p95 < 500ms for 99% of requests over 28 days."
+- **Error Budget** - `1 - SLO`. A 99% SLO means a 1% error budget: you may violate the objective 1% of the time before it's a breach. Budgets make trade-offs explicit (ship features vs. burn reliability).
+
+### Choosing SLIs
+Pick SLIs users actually feel, not ones that are easy to measure:
+
+| User-facing concern | SLI | Typical SLO |
+|---|---|---|
+| Responsiveness | Request p95 latency | < 500ms |
+| Worst-case tail | Request p99 latency | < 1,500ms |
+| Reliability | Error rate (5xx + timeouts) | < 1% (99% success) |
+| Availability | Successfully served requests / total | 99.9% |
+| Throughput | Sustained RPS at SLO latency | ≥ peak + headroom |
+| Freshness (data) | Staleness of served data | < 60s |
+
+For streaming/LLM, use token-aware SLIs (TTFT, ITL) and **goodput** (fraction of requests meeting all thresholds) instead of raw latency - see `llm-inference.md`.
+
+### Windowing
+- Use a **rolling window** (e.g. 28 days) so a single bad day does not trigger an alert; it consumes budget gradually.
+- Report SLO attainment as `good_events / total_events` over the window. If attainment < SLO, the budget is exhausted.
+- Set **alerts on burn rate**, not on instantaneous SLO pass/fail. Fast burn (e.g. 14× rate) means a major incident; slow burn means a trend to watch.
+
+### Multi-window burn-rate alerting (pattern)
+Define fast and slow burn signals so you catch both outages and slow erosion:
+```
+Fast burn:  burn rate ≥ 14  over 1h   → page
+Slow burn:  burn rate ≥ 2   over 6h   → ticket
+```
+A burn rate of `X` means the budget is being consumed `X` times faster than the 28-day baseline allows.
+
+---
+
+## From Load Test to Capacity
+
+A load test produces a curve of latency/error vs load. Capacity is the point on that curve where SLOs still hold, **plus** the headroom you keep for safety.
+
+### The saturation curve
+```
+latency
+  │              ╱‾‾‾‾‾‾‾   ← cliff: SLO breach
+  │ ─────────────
+  └──────────────────────── load (RPS/VUs)
+   [safe zone]   [capacity point]
+```
+- **Capacity point** = highest sustained load at which all SLOs (p95, error rate, goodput) still pass.
+- **Saturation point** = where throughput flatlines and latency explodes (the "cliff" in `results-analysis.md`).
+- **Never plan to capacity.** Plan to capacity minus headroom.
+
+### Headroom rule
+- Target **peak production load at ~60-70% of measured capacity** under normal operation.
+- Keep **~30-40% headroom** to absorb traffic spikes, node failures, and deployment blips without breaching SLO.
+- For autoscaling systems, headroom is what lets new replicas come up before the SLO burns.
+
+### Inverting Little's Law to size capacity
+Given a target arrival rate and response time, concurrency is `N = λ × W` (see `workload-design.md`). Use it both ways:
+- **To size load**: known λ and W → required VUs (already in workload-design).
+- **To size infrastructure**: if the test shows a single replica sustains `C` RPS at SLO, replicas needed = `target_RPS / (C × utilization_target)`. With a 70% utilization target and 1,000 target RPS at 200 RPS/replica → `1000 / (200 × 0.7)` ≈ **8 replicas** (round up, add 1 for failure tolerance).
+
+### Capacity from the goodput cliff
+For LLM and streaming services, capacity = load at the goodput cliff (see `llm-inference.md`). Report "sustains 200 concurrent users at 99% goodput; goodput drops to 80% at 350" - that 200 is your planning number, 350 is your hard ceiling.
+
+---
+
+## Worked Example
+
+**Inputs:** Peak prod = 800 RPS. Test shows p95 < 500ms and error < 1% hold up to 1,200 RPS on the current 6-replica cluster; saturation at ~1,500 RPS.
+
+| Decision | Calculation | Result |
+|---|---|---|
+| Capacity point | measured SLO-holding load | 1,200 RPS |
+| Planning target | 60-70% of capacity | 720-840 RPS |
+| Headroom | 1,200 - 800 peak | 33% (acceptable) |
+| Replicas at growth | if peak grows to 1,500 RPS: `1500 / (200 × 0.7)` | 11 replicas (was 6) |
+| Alert threshold | 80% of capacity as early warning | alert at ~960 RPS |
+
+---
+
+## CI Regression Gating
+
+SLOs are useless if they are only in a report. Enforce them automatically:
+
+1. **Gate on thresholds, not on "it ran."** Every tool has an SLO mechanism - k6 `thresholds`, Artillery `ensure`, Gatling Enterprise assertions, JMeter exit codes. A run with no gate always passes CI (see each tool file's Common Mistakes).
+2. **Compare against baseline, not just absolute.** A p95 of 480ms passes a 500ms SLO but is a +60% regression vs last week's 300ms. Track percentile deltas build-over-build.
+3. **Set a regression threshold.** e.g. "fail the build if p95 rises > 10% vs baseline, or error rate > 0.5%." Avoid zero-tolerance (noise); avoid loose (misses real regressions).
+4. **Require statistical significance for soak/large runs.** A 1-run blip should not fail CI; use multiple iterations or a confidence band.
+5. **Publish the SLO report in the pipeline.** Surface p50/p95/p99, throughput, error rate, and budget consumption on every run so regressions are visible, not buried.
+
+> For the per-tool gate syntax, see `../tools/k6.md` (`thresholds`), `../tools/artillery.md` (`ensure`), `../tools/gatling.md`, `../tools/jmeter.md`. For running these in pipelines, see `test-execution.md`.
+
+---
+
+## Common Mistakes
+
+- **Averaging for SLOs** - an average of 200ms can hide a p99 of 5s. SLOs must be percentile-based.
+- **SLO with no error budget** - a target with no burn policy becomes a meaningless number; you can never "spend" reliability intentionally.
+- **Planning to the saturation point** - sizing for max measured throughput leaves zero margin; the first spike breaches SLO.
+- **Confusing capacity with peak** - if peak ≈ capacity, you have no headroom. Target 60-70%.
+- **One-shot capacity number** - capacity drifts as code, data, and dependencies change. Re-baseline on major releases.
+- **Gating CI on run success only** - "test passed" ≠ "SLO met." Without a threshold/ensure gate, regressions ship green.
+- **Ignoring tail latency in capacity** - a system can hold p95 at capacity while p99 is 10× worse; gate on the tail users actually feel.
+- **No burn-rate alerting** - alerting on instantaneous SLO state misses slow erosion until the budget is already gone.
+
+---
+
+## Reporting SLO & Capacity
+
+Structure the decision section for stakeholders:
+
+```
+1. SLO SUMMARY
+   - p95 < 500ms: 99.2% attainment (budget: 1% used 0.8%)
+   - Error rate < 1%: 99.97% attainment
+2. CAPACITY
+   - Measured capacity: 1,200 RPS @ SLO
+   - Current peak: 800 RPS (33% headroom)
+   - Saturation: ~1,500 RPS
+3. RECOMMENDATION
+   - Scale to 8 replicas before peak season (growth to 1,500 RPS)
+   - Alert at 960 RPS (80% of capacity)
+4. REGRESSION GATE
+   - p95 > +10% vs baseline fails CI
+```