perf-skills 1.0.0

package/perf-skills/references/topics/test-execution.md

@@ -0,0 +1,281 @@
+# Test Execution
+
+Covers how to run performance tests reliably — locally, in CI/CD pipelines, distributed across injectors, and in the cloud.
+
+---
+
+## Execution Modes
+
+| Mode | When to Use | Tooling |
+|---|---|---|
+| Local (single process) | Development, debugging, smoke tests | All tools |
+| Distributed (multiple agents) | > 500–1000 VUs, high RPS | JMeter, Locust, k6 |
+| Cloud (managed) | Production-scale, geographic distribution | Grafana Cloud, BlazeMeter, OctoPerf, Gatling Enterprise, NeoLoad Cloud |
+| CI/CD (automated) | Regression gate, scheduled runs | All tools via CLI |
+
+---
+
+## Local Execution Best Practices
+
+- Always use **non-GUI / headless mode** for actual load runs (GUI mode adds overhead).
+- Run scripts from the same network segment as the system under test when possible.
+- Monitor injector machine resources during the test — if CPU/memory of the injector saturates, results are invalid.
+- Set JVM heap appropriately for JMeter: `JVM_ARGS="-Xms2g -Xmx4g" jmeter -n -t test.jmx`
+
+---
+
+## Distributed Execution
+
+### JMeter Distributed
+
+**Architecture:**
+```
+Controller (orchestrate)  →  Injector 1 (1/N VUs)
+                          →  Injector 2 (1/N VUs)
+                          →  Injector 3 (1/N VUs)
+```
+
+**Setup:**
+```bash
+# On each injector
+./bin/jmeter-server -Djava.rmi.server.hostname=<injector-ip>
+
+# On controller (bin/jmeter.properties)
+remote_hosts=injector-1:1099,injector-2:1099,injector-3:1099
+
+# Run (controller distributes test plan and aggregates results)
+jmeter -n -t test.jmx -r -l results.jtl -Jthreads=300 -Jduration=600
+```
+
+**Ports to open:** 1099 (RMI control), 50000+ (dynamic data ports) — firewall rules critical.
+
+**Scaling:** Each injector can typically handle 300–500 VUs for HTTP at moderate response times. For 10k VUs, plan 20–30 injectors.
+
+### k6 Distributed
+
+```bash
+# k6 Operator on Kubernetes (recommended for large-scale k6)
+# 1. Install k6 Operator
+kubectl apply -f https://github.com/grafana/k6-operator/releases/latest/download/bundle.yaml
+
+# 2. Create TestRun resource
+apiVersion: k6.io/v1alpha1
+kind: TestRun
+metadata:
+  name: k6-load-test
+spec:
+  parallelism: 5         # 5 pods, each running part of the test
+  script:
+    configMap:
+      name: k6-test-script
+      file: test.js
+  arguments: --out influxdb=http://influx:8086/k6
+```
+
+### Locust Distributed
+
+```bash
+# Master (web UI on port 8089)
+locust -f locustfile.py --master --expect-workers=4
+
+# Workers (connect to master)
+locust -f locustfile.py --worker --master-host=<master-ip>
+
+# Headless distributed
+locust -f locustfile.py --master --headless \
+  --users 2000 --spawn-rate 50 --run-time 10m \
+  --expect-workers=4 --html=report.html
+```
+
+---
+
+## Cloud Execution
+
+### k6 → Grafana Cloud
+```bash
+# Authenticate
+k6 login cloud --token $K6_CLOUD_TOKEN
+
+# Run in cloud
+k6 cloud --vus 1000 --duration 10m script.js
+
+# Or tag the project
+k6 cloud --project-id $PROJECT_ID script.js
+```
+
+### JMeter → BlazeMeter / OctoPerf
+```bash
+# BlazeMeter CLI
+bzt tests/load-test.jmx \
+  -o modules.blazemeter.token=$BZ_TOKEN \
+  -o modules.blazemeter.project="MyApp Load Test" \
+  -o modules.blazemeter.concurrency=1000 \
+  -o modules.blazemeter.duration=600
+```
+
+### AWS (DIY Cloud Execution)
+```bash
+# Spin up EC2 injectors via Terraform, then run JMeter/k6
+# Use Auto Scaling Groups for burst capacity
+# Use S3 to store results, CloudWatch for metrics
+
+# Cost optimization: use Spot instances for injectors (price vs reliability tradeoff)
+```
+
+---
+
+## CI/CD Integration Patterns
+
+### Performance Gate Pattern
+```
+Code PR  →  Build  →  Unit/Integration Tests  →  [Performance Gate]  →  Deploy
+                                                         │
+                                                         ├─ Run smoke (1 VU)
+                                                         ├─ Run load test (target VUs)
+                                                         ├─ Check SLA thresholds
+                                                         └─ PASS → Deploy / FAIL → Block
+```
+
+### GitHub Actions — k6
+
+```yaml
+name: Performance Gate
+
+on:
+  pull_request:
+    branches: [main]
+  schedule:
+    - cron: '0 2 * * *'  # Nightly soak test
+
+jobs:
+  load-test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Start Application (staging)
+        run: docker-compose up -d
+        
+      - name: Wait for readiness
+        run: |
+          timeout 60 sh -c 'until curl -sf http://localhost:8080/health; do sleep 2; done'
+
+      - name: Run k6 Load Test
+        uses: grafana/k6-action@v0.3.1
+        with:
+          filename: tests/load/main.js
+          flags: --out json=results/results.json
+        env:
+          BASE_URL: http://localhost:8080
+          TARGET_VUS: 50
+          DURATION: 5m
+
+      - name: Upload Results
+        uses: actions/upload-artifact@v4
+        if: always()
+        with:
+          name: k6-results-${{ github.run_id }}
+          path: results/
+
+      - name: Check SLA Thresholds
+        run: |
+          # k6 already exits non-zero if thresholds fail
+          # For additional reporting, parse results.json here
+```
+
+### GitHub Actions — JMeter
+
+```yaml
+      - name: Run JMeter Tests
+        run: |
+          jmeter -n \
+            -t tests/load.jmx \
+            -l results/results.jtl \
+            -e -o results/dashboard \
+            -Jbase_url=${{ vars.STAGING_URL }} \
+            -Jthreads=100 \
+            -Jduration=300
+          
+      - name: Check Error Rate
+        run: |
+          python3 scripts/check_jtl.py results/results.jtl \
+            --max-error-rate 1.0 \
+            --max-p95 500
+```
+
+### GitLab CI
+
+```yaml
+performance-test:
+  stage: performance
+  image: grafana/k6:latest
+  script:
+    - k6 run --vus $VUS --duration $DURATION tests/load.js
+  variables:
+    VUS: "100"
+    DURATION: "5m"
+    BASE_URL: $STAGING_URL
+  artifacts:
+    reports:
+      performance: results.json
+    paths:
+      - results.json
+  only:
+    - main
+    - /^release\/.*$/
+```
+
+### Jenkins Pipeline
+
+```groovy
+stage('Performance Test') {
+    steps {
+        sh '''
+            k6 run \
+              --out json=results.json \
+              -e BASE_URL=${STAGING_URL} \
+              tests/load.js
+        '''
+    }
+    post {
+        always {
+            archiveArtifacts artifacts: 'results.json'
+            perfReport 'results.json'  // Jenkins Performance Plugin
+        }
+    }
+}
+```
+
+---
+
+## Environment Isolation for Tests
+
+### Dedicated Staging Environment
+- Mirror production architecture (same instance types, same DB configuration).
+- No shared services with other teams during test window.
+- Disable rate limiting or coordinate with the team running the test.
+
+### Test Window Management
+- Book a test window and notify all stakeholders.
+- Disable non-essential background jobs that would skew results.
+- Pre-warm caches if testing steady-state behavior (not cold-start).
+
+### Infrastructure Monitoring During Test
+Always monitor the injector resources alongside the SUT:
+- CPU, memory, network I/O on each injector.
+- If injector CPU hits 80%+, the injector is the bottleneck, not the SUT.
+
+---
+
+## Test Execution Checklist
+
+- [ ] Non-GUI / headless mode configured
+- [ ] Injector capacity verified (CPU, network bandwidth, thread limit)
+- [ ] Data files deployed to all injector nodes
+- [ ] SUT environment isolated and confirmed healthy
+- [ ] APM/observability hooks active
+- [ ] SLA thresholds configured for CI pass/fail gate
+- [ ] Results output path configured (JTL, JSON, InfluxDB)
+- [ ] Smoke test (1 VU) passed before full run
+- [ ] Team notified of test window
+- [ ] Rollback plan in place (especially for production testing)

package/perf-skills/references/topics/workload-design.md

@@ -0,0 +1,207 @@
+# Workload Design
+
+Workload design is the most impactful phase of performance testing. A poorly designed workload produces results that are irrelevant to production behavior — no matter how well-scripted the test is.
+
+---
+
+## Step 1: Understand Production Traffic
+
+Before writing a single line of script, gather real data:
+
+| Data Source | What to Extract |
+|---|---|
+| Web server access logs | Request URLs, methods, frequency, user session patterns |
+| APM (Datadog, Dynatrace) | Transaction mix, throughput by endpoint, peak hours |
+| CDN logs | Real geographic distribution and peak RPS |
+| Analytics (GA, Mixpanel) | User journeys, funnel drop-offs, session duration |
+| Database slow query logs | Expensive queries to include in test scenarios |
+
+**Key questions to answer:**
+- What is the **peak concurrent user count** (not sessions per day)?
+- What is the **throughput target** (RPS or TPS) at peak?
+- What is the **transaction mix** (% login, % browse, % checkout)?
+- What is the **session length** and **average think time**?
+- Are there **batch or background jobs** running during peak?
+
+---
+
+## Step 2: Define the Workload Model
+
+### Concurrency Model Selection
+
+**Closed Model (VU-based)**
+- VU count is fixed; next iteration starts only when previous completes.
+- Think time is part of the model.
+- Best for: web apps where users "hold" sessions.
+- Tools: JMeter (Thread Groups), Gatling (`rampUsers`), Locust.
+
+**Open Model (Arrival-rate)**
+- New requests arrive at a fixed rate regardless of outstanding requests.
+- More realistic for APIs, microservices, and public-facing systems.
+- Best for: REST APIs, event-driven systems, queueing scenarios.
+- Tools: k6 (`constant-arrival-rate`), Gatling (`constantUsersPerSec`), JMeter (Throughput Shaping Timer).
+
+### When to Use Each
+
+| Scenario | Model |
+|---|---|
+| E-commerce web app with sessions | Closed |
+| REST API serving mobile clients | Open |
+| Microservice with queue consumer | Open |
+| Banking portal with session timeouts | Closed |
+| Public API with rate limiting | Open |
+
+---
+
+## Step 3: Define the Load Profile
+
+### Common Load Profiles
+
+**Ramp-Up + Steady State + Ramp-Down**
+```
+VUs  │    ▁▂▃▄▅▆▇█████████▇▆▅▄▃▂▁
+     │
+     └────────────────────────────── Time
+     [Ramp 5m][  Hold 10m  ][Down 2m]
+```
+
+**Step Load (Staircase)**
+```
+VUs  │         ████
+     │    ████ ████ ████
+     │ ████ ████ ████ ████
+     └──────────────────────── Time
+```
+Use for: finding the load level at which behavior degrades.
+
+**Spike Test**
+```
+VUs  │              ▐█▌
+     │              ▐█▌
+     │    ██████████▐█▌█████████
+     └──────────────────────────── Time
+```
+Use for: testing autoscaling response, queue behavior under burst.
+
+**Soak/Endurance**
+```
+VUs  │    ████████████████████████
+     └──────────────────────────── Time (2–8 hours)
+```
+Use for: detecting memory leaks, connection pool exhaustion, log rotation issues.
+
+---
+
+## Step 4: Calculate Concurrency
+
+### Little's Law (the fundamental formula)
+
+```
+N = λ × W
+
+N = average concurrency (VUs)
+λ = arrival rate (requests/sec = throughput)
+W = average response time + think time (seconds)
+```
+
+**Example:**
+- Target: 1,000 RPS
+- Average response time: 200ms = 0.2s
+- Average think time: 3s
+- W = 0.2 + 3 = 3.2s
+- N = 1,000 × 3.2 = **3,200 VUs needed**
+
+This is why VU count alone is meaningless without knowing think time and response time targets.
+
+---
+
+## Step 5: Define Transaction Mix
+
+Break down the workload into realistic business transactions:
+
+```
+Example: E-commerce application
+
+Transaction          | % of Traffic | Think Time
+---------------------|--------------|------------
+Homepage browse      |     40%      |   5–15s
+Product search       |     25%      |   3–8s
+Product detail view  |     20%      |   5–10s
+Add to cart          |     10%      |   2–5s
+Checkout + payment   |      5%      |   30–60s
+```
+
+Implement in scripts:
+- **JMeter**: Use Throughput Controller (% mode) or separate Thread Groups with ratios.
+- **k6**: Use scenarios with `weight` or multiple VU groups with proportional counts.
+- **Gatling**: Use `Population` with `userWeight`.
+- **Locust**: Use `@task(weight)` decorator.
+
+---
+
+## Step 6: Set SLA Targets
+
+Every test must have explicit, measurable SLA targets before execution. Negotiate these with stakeholders:
+
+| Metric | Threshold Example |
+|---|---|
+| p50 (median) response time | < 200ms |
+| p95 response time | < 500ms |
+| p99 response time | < 1,500ms |
+| Error rate | < 1% |
+| Throughput floor | ≥ 500 RPS |
+| Max response time | < 5,000ms |
+
+**Why percentiles matter more than averages:**
+Average response time hides outliers. p95 tells you what 95% of users experience — averages can look fine while 10% of users timeout.
+
+---
+
+## Think Time Design
+
+Think time simulates the time a real user spends reading a page, filling a form, or deciding what to do next.
+
+| Distribution | When to use |
+|---|---|
+| Constant (e.g., 3s) | Simple tests, known fixed pacing |
+| Uniform random (1–5s) | Basic variability, most common |
+| Gaussian (mean=3s, std=1s) | Most realistic for web users |
+| Negative exponential | Rare; models Poisson arrival patterns |
+
+**Rule of thumb:** Never set think time to 0 in closed-model tests unless explicitly modeling a batch job or API benchmark with no user interaction.
+
+---
+
+## Pacing vs Think Time
+
+- **Think time**: Time a VU sleeps *within* a transaction (between pages).
+- **Pacing**: Time between the *start* of each iteration (controls throughput more directly).
+
+Pacing is used when you want `N` iterations per hour per VU regardless of response time. Example:
+- Target: 360 transactions/hour per VU = 1 transaction every 10 seconds
+- Pacing = 10s (start next iteration 10s after previous started)
+
+---
+
+## Ramp-Up Strategy
+
+Bad ramp-up causes a "thundering herd" — thousands of VUs hitting the server simultaneously, warming up connection pools, JVM, and caches all at once. This is unrealistic and produces misleading results.
+
+**Good ramp-up:**
+- Rule of thumb: Ramp to 100% over at least 5–10 minutes for large tests.
+- Allow cache warm-up (exclude ramp period from SLA analysis).
+- For step tests, hold each step for at least 2–3× the transaction response time.
+
+---
+
+## Checklist Before Writing Scripts
+
+- [ ] Production traffic baselines collected (RPS, response time, error rate)
+- [ ] Concurrency calculated (Little's Law applied)
+- [ ] Transaction mix defined (% per transaction type)
+- [ ] Think time distribution specified
+- [ ] Load profile drawn (ramp, hold, spike, soak)
+- [ ] SLA thresholds defined and signed off by stakeholders
+- [ ] Environment capacity verified (enough headroom to push load)
+- [ ] Test data strategy defined (see test-data.md)
+- [ ] Monitoring and APM hooks confirmed (see observability.md)