perf-skills 1.0.0

package/perf-skills/references/topics/results-analysis.md

@@ -0,0 +1,244 @@
+# Results Analysis
+
+Interpreting performance test results is a skill in itself. Raw numbers are meaningless without context — this reference covers how to read, analyze, and act on performance data.
+
+---
+
+## Core Metrics
+
+| Metric | Description | Target |
+|---|---|---|
+| **Throughput** | Requests per second (RPS / TPS) | ≥ defined baseline |
+| **Response Time (p50)** | Median — half of requests faster than this | Per SLA |
+| **Response Time (p95)** | 95% of requests faster than this — main SLA metric | Per SLA |
+| **Response Time (p99)** | Tail latency — reveals worst-case behavior | Per SLA |
+| **Error Rate** | % of failed requests | < 1% (or as defined) |
+| **Active VUs / Concurrency** | Number of simulated users at any point | Matches load profile |
+| **Network I/O** | Bytes/sec in and out — detects bandwidth bottlenecks | Headroom vs NIC capacity |
+
+---
+
+## Response Time Percentiles
+
+Never rely on averages alone. They mask the tail:
+
+```
+Example: 100 requests
+  98 requests at 100ms  ← average pulled down
+   2 requests at 5000ms ← 2% of users wait 5 seconds
+
+  Average: 198ms   ← looks fine
+  p95:     100ms   ← looks fine
+  p99:     5000ms  ← alerts you to the problem
+  Max:     5100ms  ← confirms the problem
+```
+
+**Which percentile to use for SLAs:**
+- p50 = typical user experience
+- p95 = most users' worst case (standard SLA metric)
+- p99 = tail latency (critical for high-volume services)
+- p99.9 = extreme tail (needed for financial/medical systems)
+
+---
+
+## Reading JMeter Reports
+
+### Summary Report columns
+| Column | Meaning |
+|---|---|
+| # Samples | Total requests sent |
+| Average | Mean response time (ms) — use sparingly |
+| Min / Max | Absolute floor and ceiling |
+| Std. Dev. | Variance — high deviation = inconsistent behavior |
+| Error % | Percentage of failed requests |
+| Throughput | Requests/second |
+| KB/sec | Network throughput |
+| p90 / p95 / p99 | Percentile response times |
+
+### HTML Dashboard
+Generated with `-e -o results/dashboard` flag. Key graphs:
+- **Response Time Over Time** — spot degradation trends
+- **Transactions Per Second** — confirm throughput matches expectation
+- **Response Time Percentiles** — see distribution
+- **Active Threads Over Time** — correlate with response time
+- **Errors Over Time** — when and what errors spike
+
+### JTL Analysis (programmatic)
+```python
+import pandas as pd
+
+df = pd.read_csv('results.jtl')
+df['elapsed'] = df['elapsed'].astype(int)
+df['success'] = df['success'].astype(bool)
+
+# Filter to successful requests only for latency analysis
+success = df[df['success'] == True]
+
+print(f"p50:  {success['elapsed'].quantile(0.50):.0f}ms")
+print(f"p95:  {success['elapsed'].quantile(0.95):.0f}ms")
+print(f"p99:  {success['elapsed'].quantile(0.99):.0f}ms")
+print(f"Max:  {success['elapsed'].max():.0f}ms")
+print(f"RPS:  {len(df) / (df['timeStamp'].max() - df['timeStamp'].min()) * 1000:.1f}")
+print(f"Error rate: {(~df['success']).mean()*100:.2f}%")
+```
+
+---
+
+## Reading k6 Reports
+
+### Terminal summary output
+```
+✓ http_req_duration............: avg=182ms min=45ms med=156ms max=2.1s p(90)=312ms p(95)=489ms
+✓ http_req_failed..............: 0.52% ✓ 52 ✗ 9948
+✓ login_errors.................: count=2
+  http_reqs.....................: 10000  83.33/s
+  iterations....................: 2000   16.67/s
+  vus...........................: 100
+  vus_max.......................: 100
+```
+
+**Thresholds status:** `✓` = passed, `✗` = failed (non-zero exit code).
+
+### k6 JSON output analysis
+```python
+import json
+
+with open('results.json') as f:
+    for line in f:
+        point = json.loads(line)
+        if point['type'] == 'Point' and point['metric'] == 'http_req_duration':
+            # Process individual data points
+            pass
+```
+
+---
+
+## Bottleneck Identification Framework
+
+When you see high response times or errors, use this framework:
+
+```
+High Response Time / Errors
+        │
+        ├─ Check: Is the error rate high?
+        │    ├─ YES → What status codes? (503 = server overloaded, 504 = timeout, 429 = rate limit)
+        │    └─ NO → Latency issue, not availability
+        │
+        ├─ Check: Is throughput (RPS) flat-lining below target?
+        │    ├─ YES → Server saturated; find the saturated resource
+        │    └─ NO → Throughput ok; issue is latency distribution
+        │
+        ├─ Check: Server CPU > 80%?
+        │    ├─ YES → CPU bottleneck → profile code, optimize algorithms, scale horizontally
+        │    └─ NO → CPU is not the limit
+        │
+        ├─ Check: Database slow query log showing queries > 100ms?
+        │    ├─ YES → DB bottleneck → add indexes, optimize queries, connection pool tuning
+        │    └─ NO → Look elsewhere
+        │
+        ├─ Check: JVM GC pauses (Java services)?
+        │    ├─ YES → Memory tuning, GC algorithm selection, heap sizing
+        │    └─ NO → Not GC
+        │
+        ├─ Check: Connection pool exhausted? (connection refused, pool timeout errors)
+        │    ├─ YES → Tune pool size, connection timeout, check for connection leaks
+        │    └─ NO → Not pool
+        │
+        └─ Check: Network I/O saturated?
+             ├─ YES → Response payload too large? Bandwidth ceiling? CDN needed?
+             └─ NO → Dig deeper with APM traces
+```
+
+---
+
+## Analyzing Latency Patterns
+
+### Response Time Over Time Shapes
+
+**Stable (good):**
+```
+ms │ ~~~~~~~~~~~~~~~~~~~~
+   └──────────────────────── Time
+```
+Consistent response time = healthy, stable system.
+
+**Degrading (memory leak / connection exhaustion):**
+```
+ms │                    ╱‾‾‾
+   │          ╱‾‾‾‾‾‾‾‾
+   │ ╱‾‾‾‾‾‾‾
+   └──────────────────────── Time
+```
+Gradual increase = something is accumulating over time.
+
+**Sawtooth (GC or periodic batch job):**
+```
+ms │ ▁▂▃▄▅▆▇█▁▂▃▄▅▆▇█▁▂▃▄▅
+   └──────────────────────── Time
+```
+Periodic spikes = GC pauses, scheduled jobs, connection pool refresh.
+
+**Cliff (saturation point):**
+```
+ms │              ╱‾‾‾‾‾‾‾
+   │ ─────────────
+   └──────────────────────── Time
+   [normal load]  [saturation]
+```
+Sudden jump at a load threshold = system has hit a constraint (thread limit, DB connection pool, queue depth).
+
+---
+
+## Comparing Runs (Trend Analysis)
+
+Always compare the current run against a baseline — never evaluate a run in isolation.
+
+| Metric | Run 1 (Baseline) | Run 2 | Delta | Status |
+|---|---|---|---|---|
+| p95 response time | 320ms | 380ms | +18.7% | ⚠️ Regression |
+| Error rate | 0.1% | 0.08% | -20% | ✅ Improved |
+| Throughput | 450 RPS | 480 RPS | +6.7% | ✅ Improved |
+| p99 | 1200ms | 2100ms | +75% | ❌ Major regression |
+
+Tools for trend analysis:
+- **Grafana + InfluxDB**: Overlay runs as time-series overlays.
+- **Gatling Enterprise**: Built-in run comparison.
+- **JMeter + Jenkins Performance Plugin**: Trend charts per build.
+- **Custom scripting**: Python/Pandas on JTL files.
+
+---
+
+## Reporting to Stakeholders
+
+Structure your report:
+
+```
+1. EXECUTIVE SUMMARY
+   - Test objective and load profile (1 paragraph)
+   - Overall result: PASS / FAIL against SLA
+   - Top finding (e.g., "p95 exceeded SLA by 40% above 500 VUs")
+
+2. TEST PARAMETERS
+   - Tool, VU count, duration, environment, data set
+
+3. KEY METRICS TABLE
+   - Throughput, p50/p95/p99, error rate vs target
+
+4. GRAPHS
+   - Response time over time
+   - Throughput over time
+   - Error rate over time
+   - Server metrics (CPU, memory) if available
+
+5. BOTTLENECK ANALYSIS
+   - What degraded, at what load level, probable cause
+
+6. RECOMMENDATIONS
+   - Specific, actionable items (e.g., "Add index on orders.customer_id")
+   - Priority and estimated impact
+
+7. APPENDIX
+   - Full metrics breakdown per endpoint
+   - Error log samples
+   - Environment configuration
+```

package/perf-skills/references/topics/script-generation.md

@@ -0,0 +1,221 @@
+# Script Generation & Best Practices
+
+This reference covers how to write robust, realistic, and maintainable performance test scripts — regardless of tool. Apply these principles to JMeter, k6, Gatling, Locust, or any other tool.
+
+---
+
+## Scripting Workflow
+
+```
+1. Understand the flow       ← Get sequence from BA, Swagger, or HAR
+2. Record or scaffold        ← Record via proxy or write from API docs
+3. Smoke test (1 VU)         ← Validate correctness before load
+4. Correlate dynamic values  ← Session tokens, CSRF, IDs, ViewState
+5. Parameterize data         ← Replace hardcoded values with variables
+6. Add assertions             ← Validate response correctness
+7. Add think time            ← Make VU behavior realistic
+8. Add error handling        ← Handle expected and unexpected failures
+9. Smoke test again          ← Confirm script still works after changes
+10. Ramp up gradually        ← 5 → 10 → 25 → 50 → 100 VUs
+```
+
+---
+
+## Recording vs. Coding
+
+### Recording (proxy-based)
+- **Pros:** Fast, captures real browser behavior, catches all requests (including hidden calls).
+- **Cons:** Captures noise (analytics, CDN, fonts), needs cleanup and correlation.
+- **Best for:** Complex web apps, SPAs, unfamiliar APIs.
+- **Tools:** JMeter HTTP(S) Test Script Recorder, BlazeMeter Chrome Extension, Gatling Recorder, HAR import (OctoPerf, k6).
+
+### Coding from Scratch
+- **Pros:** Clean, no recording noise, easier to version control.
+- **Cons:** Slower for complex flows.
+- **Best for:** REST/GraphQL APIs with Swagger/OpenAPI docs.
+
+### Hybrid (Recommended)
+Record a HAR file from the browser → import into your tool → clean up and parameterize → enhance with dynamic logic.
+
+---
+
+## Correlation Deep Dive
+
+Correlation is extracting a dynamic value from a response and using it in a subsequent request. Missing correlation is the #1 cause of performance script failure.
+
+### Values that always need correlation
+- **Session tokens** (JSESSIONID, ASP.NET_SessionId)
+- **Auth tokens** (JWT, OAuth access_token, refresh_token)
+- **CSRF tokens** (`_token`, `authenticity_token`, `__RequestVerificationToken`)
+- **View state** (ASP.NET `__VIEWSTATE`, `__EVENTVALIDATION`)
+- **Resource IDs** (created order ID, cart ID, uploaded file ID)
+- **One-time codes** (OTP, nonce, challenge)
+- **Timestamps/signatures** used in request signing
+
+### Correlation debugging approach
+1. Run the script with 1 VU.
+2. Look for HTTP 4xx errors (especially 403 Forbidden, 422 Unprocessable Entity).
+3. Check the request body/headers — is a token missing or stale?
+4. Use the tool's debug output or proxy (Fiddler, Charles) to inspect live traffic.
+5. Identify where the value appears in a *previous* response.
+6. Add an extractor at that response, reference the variable in the failing request.
+
+---
+
+## Assertions (Why They're Non-Negotiable)
+
+Without assertions, your test might generate 1000 RPS of 404 responses or empty bodies — and your metrics will look "fine."
+
+### What to assert
+- **Status code** — the obvious one; but also check for 200s that contain error bodies.
+- **Response body** — key field exists and has expected value.
+- **Response time** — flag individual responses over SLA as failures.
+- **Response size** — detect truncated or empty bodies.
+- **Content-Type header** — ensure you got JSON, not an HTML error page.
+
+### Assertion layering
+
+```
+Level 1: HTTP status (global, every request)
+Level 2: Body content check (per transaction type)
+Level 3: Business logic check (e.g., balance is non-negative)
+Level 4: Duration assertion (flag outliers per request)
+```
+
+### Handling false failures
+- Exclude known retryable errors (rate limits, 503 during scale events) from SLA breach calculation.
+- Add error handling blocks that retry on specific status codes.
+- Log all failures with request/response detail for post-test debugging.
+
+---
+
+## Error Handling
+
+### JMeter
+Use **If Controller** + **Retry** or set "Continue" behavior in Thread Group on sampler error.
+
+### k6
+```javascript
+import { check } from 'k6';
+import { sleep } from 'k6';
+
+const res = http.post(url, body, params);
+if (res.status === 429) {
+    sleep(5); // Back off on rate limit
+    return;
+}
+check(res, { 'status 200': (r) => r.status === 200 });
+```
+
+### Gatling
+```scala
+.exec(http("Submit Order")
+  .post("/orders")
+  .check(status.in(200, 201, 202))  // Accept multiple valid codes
+  .checkIf((response, session) => response.status != 200) {
+    jsonPath("$.error").saveAs("errorMessage")
+  }
+)
+```
+
+### Locust
+```python
+with self.client.post("/orders", json=payload, catch_response=True) as res:
+    if res.status_code == 200:
+        res.success()
+    elif res.status_code == 429:
+        res.success()  # Don't count rate limits as failures
+        time.sleep(5)
+    else:
+        res.failure(f"Unexpected {res.status_code}: {res.text[:200]}")
+```
+
+---
+
+## Session Management Patterns
+
+### Cookie-based sessions
+Most tools handle cookies automatically. Verify the Cookie Manager is enabled.
+
+### Token-based (JWT/OAuth)
+1. Authenticate in `setup()` or `on_start()`.
+2. Store the token in session variables.
+3. Add as Authorization header to subsequent requests.
+4. Handle token expiry — check for 401 responses and re-authenticate.
+
+```javascript
+// k6: handle token refresh
+function getToken() {
+    const res = http.post('/auth/token', credentials);
+    if (res.status !== 200) throw new Error('Auth failed');
+    return res.json('access_token');
+}
+
+let token = getToken();
+
+export default function () {
+    let res = http.get('/api/profile', { headers: { Authorization: `Bearer ${token}` } });
+    if (res.status === 401) {
+        token = getToken(); // Refresh token
+        res = http.get('/api/profile', { headers: { Authorization: `Bearer ${token}` } });
+    }
+    check(res, { 'profile 200': (r) => r.status === 200 });
+}
+```
+
+---
+
+## Naming Conventions
+
+Consistent request naming makes results readable:
+
+```
+BAD:  "Request 1", "HTTP Request", "sampler_3"
+GOOD: "POST /api/orders", "GET /products/{id}", "Login - Authenticate"
+```
+
+Use naming patterns:
+- REST APIs: `{METHOD} {path}` e.g., `POST /api/v2/orders`
+- Business transactions: `{UserAction} - {Step}` e.g., `Checkout - Submit Payment`
+- Background calls: `BG - {description}` e.g., `BG - Poll Status`
+
+For parameterized URLs, use a fixed name:
+```javascript
+// k6
+http.get(`/products/${productId}`, { tags: { name: 'GET /products/{id}' } });
+```
+
+---
+
+## Script Structure Best Practices
+
+### Modularize reusable flows
+Extract common flows (login, logout, navigation) into separate functions/files. Import them in scenarios.
+
+### Use config files for environment portability
+```javascript
+// k6 config
+const config = {
+    baseUrl: __ENV.BASE_URL || 'https://staging.example.com',
+    timeout: '30s',
+    headers: { 'X-Test-Run': __ENV.BUILD_ID || 'manual' },
+};
+```
+
+### Version control your scripts
+Treat performance scripts as production code:
+- Git repository alongside application code
+- PR reviews for script changes
+- Tag script versions to match application releases
+- Store results per tag for trend comparison
+
+### Script review checklist
+- [ ] No hardcoded URLs, credentials, or IDs
+- [ ] All dynamic values correlated and parameterized
+- [ ] Assertions on every request (minimum: status code)
+- [ ] Think time applied between transactions
+- [ ] Error handling for known failure cases (401, 429, 503)
+- [ ] Request naming follows convention
+- [ ] Single VU smoke test passes cleanly
+- [ ] Script loads test data from external source (CSV, API)
+- [ ] Environment-specific values in config/env vars

package/perf-skills/references/topics/test-data.md

@@ -0,0 +1,196 @@
+# Test Data Strategy
+
+Test data is a first-class concern in performance testing. Bad data causes false failures, data collisions, unrealistic server behavior, and post-test cleanup nightmares.
+
+---
+
+## Why Test Data Matters
+
+- **Authentication tokens** expire — hardcoded tokens fail within minutes.
+- **Unique constraints** (email, username, order number) cause failures at concurrency > 1.
+- **State-dependent flows** (checkout, approval) require data in the right state before the test.
+- **Cache effects** — if 1000 VUs all hit the same product ID, cache hit rates are unrealistically high; spread data across IDs.
+
+---
+
+## Data Strategy by Test Type
+
+| Test Type | Data Strategy |
+|---|---|
+| Smoke (1 VU) | Minimal — single known-good user/record |
+| Load (realistic VUs) | Pool of pre-created users/records matching production volume |
+| Stress (pushing limits) | Large pool; no uniqueness conflicts |
+| Soak (hours) | Rotating/recyclable data; handle state cleanup |
+| Spike (burst) | Same as load; ensure pool size exceeds spike VU count |
+
+---
+
+## Data Sources
+
+### Strategy 1: Pre-Generated CSV Files
+
+Best for: stable, static data (user credentials, product IDs, account numbers).
+
+```
+data/
+├── users.csv          (username, password, account_id)
+├── products.csv       (product_id, sku, price)
+├── accounts.csv       (account_number, routing_number, balance)
+└── orders.csv         (order_id, status, customer_id)
+```
+
+**Sizing:** CSV row count should be ≥ peak VU count to prevent multiple VUs sharing the same row and causing collisions.
+
+**JMeter:** `CSV Data Set Config` element  
+**k6:** `SharedArray` with `open()` or `papaparse`  
+**Gatling:** `csv("users.csv").circular`  
+**Locust:** Read CSV in `on_start()` or `__init__.py`
+
+---
+
+### Strategy 2: Database-Seeded Data
+
+Best for: complex relational state (orders in specific statuses, accounts with balances, workflows pending approval).
+
+**Approach:**
+1. Write a seeding script (Python, SQL, or test framework) to create N records in the right state.
+2. Export IDs to CSV for the load tool to consume.
+3. After the test, run a cleanup script.
+
+```sql
+-- Seed 5000 users for load test
+INSERT INTO test_users (username, password_hash, tenant_id)
+SELECT 
+    'loadtest_user_' || generate_series(1, 5000),
+    '$2b$12$fixedhash',
+    'perf-test-tenant'
+;
+
+-- Export to CSV
+COPY (SELECT username, 'testpass123' AS password FROM test_users WHERE username LIKE 'loadtest_%')
+TO '/tmp/users.csv' CSV HEADER;
+```
+
+---
+
+### Strategy 3: API-Based Data Setup (Setup Hooks)
+
+Best for: data that must be in a specific state per VU (unique cart, session, transaction).
+
+```javascript
+// k6 setup() — run before VUs start
+export function setup() {
+  const orders = [];
+  for (let i = 0; i < 200; i++) {
+    const res = http.post('https://api.example.com/admin/orders', JSON.stringify({
+      status: 'pending',
+      amount: Math.random() * 1000
+    }), { headers: adminHeaders });
+    orders.push(res.json('orderId'));
+  }
+  return { orders };  // Passed to default() as data argument
+}
+```
+
+```python
+# Locust on_start()
+def on_start(self):
+    res = self.client.post("/api/checkout/initiate", json={"cartId": new_uuid()})
+    self.checkout_id = res.json()["checkoutId"]
+```
+
+---
+
+### Strategy 4: Faker / Synthetic Data Generation
+
+Best for: registration flows, profile creation, form submission — where each VU needs unique PII-like data.
+
+```python
+# Python (Locust or data generation script)
+from faker import Faker
+fake = Faker()
+
+user = {
+    "name": fake.name(),
+    "email": fake.unique.email(),
+    "address": fake.address(),
+    "phone": fake.phone_number()
+}
+```
+
+```javascript
+// k6 with randomString
+import { randomString, randomIntBetween } from 'https://jslib.k6.io/k6-utils/1.4.0/index.js';
+
+const email = `user_${randomString(8)}@loadtest.example.com`;
+```
+
+```groovy
+// JMeter JSR223
+import java.util.UUID
+vars.put("email", "user_${UUID.randomUUID()}@loadtest.example.com")
+```
+
+---
+
+## Data Isolation Patterns
+
+### Tenant Isolation
+Create a dedicated `perf-test` tenant in multi-tenant systems. This prevents test data from polluting other tenants and enables easy cleanup.
+
+### Test Data Tagging
+Tag all test records (e.g., `source: "perf-test"`, `env: "load-test"`) so they can be identified and deleted post-test.
+
+### Ephemeral Environments
+In cloud-native architectures, spin up a fresh environment for each test run (Infrastructure as Code), then destroy it. Eliminates test data pollution entirely.
+
+### Data Reset Hooks
+Register teardown hooks to delete or reset test data after the run:
+
+```python
+# Locust teardown
+@events.test_stop.add_listener
+def cleanup(environment, **kwargs):
+    requests.delete(
+        f"{HOST}/admin/test-data",
+        headers=admin_headers,
+        json={"tag": "perf-test"}
+    )
+```
+
+---
+
+## Parameterization Patterns
+
+### Round-Robin (default for most tools)
+Each VU/iteration picks the next row. Ensures even distribution.
+
+### Random
+Better for cache-busting tests where you want realistic cache miss rates.
+
+### Unique-Per-VU
+Each VU gets a dedicated row — critical for state-dependent flows (user owns specific order). Use VU index (`__VU` in k6, `${__threadNum}` in JMeter) to deterministically select a row.
+
+```javascript
+// k6: VU-indexed data
+const user = users[__VU - 1];  // VU 1 gets row 0, VU 2 gets row 1, etc.
+```
+
+```xml
+<!-- JMeter: use __threadNum to assign data -->
+<!-- In CSV Data Set Config, Sharing Mode = Current Thread Group -->
+<!-- Or use: ${__groovy(vars.get("__threadNum").toInteger() - 1)} -->
+```
+
+---
+
+## Test Data Checklist
+
+- [ ] Sufficient data volume (≥ peak VU count rows for unique-per-VU flows)
+- [ ] Data is in the correct initial state for each scenario
+- [ ] No hardcoded credentials or tokens
+- [ ] Data isolated from production (separate tenant, tagging, or ephemeral env)
+- [ ] Cleanup mechanism in place post-test
+- [ ] CSV files available on all injector nodes (distributed tests)
+- [ ] Sensitive data masked/anonymized (not using real PII in test data)
+- [ ] Date/time sensitive records accounted for (e.g., expiry dates set far in future)