@harness-engineering/cli 1.14.0 → 1.16.0

package/dist/agents/skills/codex/harness-load-testing/SKILL.md

@@ -0,0 +1,274 @@
+# Harness Load Testing
+
+> Stress testing, capacity planning, and performance benchmarking with k6, Artillery, and Gatling. Detects existing load test infrastructure, designs test scenarios for critical paths, executes tests, and analyzes results against defined thresholds.
+
+## When to Use
+
+- Before major releases or migrations to validate capacity and identify breaking points
+- At milestone boundaries to establish performance baselines for critical endpoints
+- When scaling infrastructure and needing to verify new capacity meets demand projections
+- NOT for unit-level microbenchmarks (use harness-perf for function-level performance)
+- NOT for real-time production monitoring (use observability tooling like Datadog or Grafana)
+- NOT for frontend performance audits (use Lighthouse or harness-perf for client-side metrics)
+
+## Process
+
+### Phase 1: DETECT -- Inventory Existing Tests and Infrastructure
+
+1. **Discover load testing tooling.** Scan the project for load test infrastructure:
+   - k6: `*.k6.js`, `k6/` directory, `import { check } from 'k6'` patterns
+   - Artillery: `artillery.yml`, `artillery/` directory, `config.target` in YAML files
+   - Gatling: `gatling/`, `*Simulation.scala`, `gatling.conf`
+   - JMeter: `*.jmx` files, `jmeter/` directory
+   - Custom: `load-tests/`, `perf/`, `benchmark/` directories
+
+2. **Inventory existing test scenarios.** For each discovered test file:
+   - Extract target endpoints and their HTTP methods
+   - Identify virtual user counts, ramp-up profiles, and duration
+   - Note defined thresholds (p95, p99, error rate, throughput)
+   - Check for test data generation or seed data requirements
+
+3. **Map critical endpoints.** Identify endpoints that should be load tested:
+   - High-traffic endpoints from route definitions and API documentation
+   - Endpoints involved in revenue-critical flows (checkout, payment, subscription)
+   - Endpoints with known performance sensitivity (search, aggregation, file upload)
+   - Recently changed endpoints from `git log --oneline --since="30 days ago"`
+
+4. **Identify coverage gaps.** Compare critical endpoints against existing test scenarios:
+   - Endpoints with no load test coverage
+   - Tests with outdated thresholds or missing scenarios (spike, soak)
+   - Tests that do not match current API contracts (changed parameters, new endpoints)
+
+5. **Check test infrastructure.** Verify the testing environment:
+   - Target environment configuration (staging URL, test database, mock services)
+   - CI/CD integration (is load testing part of the pipeline?)
+   - Test data availability (seed scripts, fixtures, data generators)
+
+---
+
+### Phase 2: DESIGN -- Define Scenarios and Thresholds
+
+1. **Select test profiles.** Design scenarios for each critical endpoint based on the testing goal:
+   - **Smoke test:** 1-5 VUs for 1 minute. Validates the test script works correctly.
+   - **Load test:** Expected production traffic for 5-15 minutes. Validates normal operation.
+   - **Stress test:** 2-3x expected traffic with ramp-up. Finds the breaking point.
+   - **Spike test:** Sudden burst from baseline to 10x traffic. Tests auto-scaling and recovery.
+   - **Soak test:** Expected traffic for 1-4 hours. Detects memory leaks and connection pool exhaustion.
+
+2. **Define virtual user profiles.** Model realistic user behavior:
+   - Think time between requests (sleep 1-3 seconds between page interactions)
+   - Request sequences that mirror real user journeys (browse -> search -> add to cart -> checkout)
+   - Authentication flows (login, token refresh, session maintenance)
+   - Data variation (different product IDs, user accounts, search queries)
+
+3. **Set performance thresholds.** Define pass/fail criteria per endpoint:
+   - **Latency:** p95 < 500ms, p99 < 1000ms (adjust per endpoint SLO)
+   - **Error rate:** < 1% for load tests, < 5% for stress tests
+   - **Throughput:** Minimum requests per second for the target VU count
+   - Thresholds should be derived from SLOs if they exist, or from baseline measurements
+
+4. **Generate test scripts.** Produce test files in the detected tool format:
+   - k6: JavaScript test with `stages`, `thresholds`, and `checks`
+   - Artillery: YAML config with `phases`, `ensure`, and scenario `flow`
+   - Include parameterized data, proper assertions, and tagged metrics
+
+5. **Design ramp-up stages.** For each profile, define the VU ramp schedule:
+   - Gradual ramp-up to avoid thundering herd at test start
+   - Plateau at target load for measurement stability
+   - Optional step-up stages for stress tests to find the exact breaking point
+   - Cool-down period to observe recovery behavior
+
+---
+
+### Phase 3: EXECUTE -- Run Tests and Collect Metrics
+
+1. **Validate test environment.** Before executing:
+   - Confirm the target URL is a staging/test environment (never run load tests against production without explicit approval)
+   - Verify test database is seeded and isolated
+   - Check that external dependencies are stubbed or rate-limited appropriately
+   - Confirm monitoring is active on the target environment to correlate metrics
+
+2. **Run smoke test first.** Execute each test script with minimal load:
+   - 1-2 VUs for 30 seconds
+   - Verify all requests return expected status codes
+   - Confirm test data and authentication work correctly
+   - If smoke test fails, fix the script before proceeding to load tests
+
+3. **Execute the selected test profile.** Run the load test and capture:
+   - Request latency distribution (p50, p90, p95, p99, max)
+   - Throughput (requests per second over time)
+   - Error rate and error type distribution (4xx vs 5xx)
+   - Virtual user count over time
+   - If using k6: `k6 run --out json=results.json test.k6.js`
+   - If using Artillery: `artillery run --output report.json artillery.yml`
+
+4. **Monitor system resources during execution.** If accessible:
+   - CPU and memory utilization on target servers
+   - Database connection pool usage and query times
+   - Queue depths and consumer lag
+   - Network I/O and connection counts
+
+5. **Capture baseline or compare to previous run.** Store results for trend analysis:
+   - Save raw results to `load-tests/results/YYYY-MM-DD-<profile>.json`
+   - If a previous baseline exists, prepare a comparison dataset
+   - Tag results with git commit SHA for traceability
+
+---
+
+### Phase 4: ANALYZE -- Interpret Results and Report
+
+1. **Evaluate against thresholds.** For each endpoint tested:
+   - Did p95 latency meet the threshold? If not, by how much?
+   - Did error rate stay within acceptable bounds?
+   - Did throughput meet the minimum target?
+   - At what VU count did performance degrade (stress test breaking point)?
+
+2. **Identify bottlenecks.** Correlate performance data with system metrics:
+   - High latency + high CPU = compute-bound (optimize algorithm or scale horizontally)
+   - High latency + normal CPU = I/O-bound (database queries, external API calls, disk)
+   - Increasing errors at specific VU count = resource exhaustion (connection pools, file descriptors, memory)
+   - Latency spike at ramp-up = cold start issue (JIT compilation, cache warming, connection establishment)
+
+3. **Calculate capacity projections.** Based on stress test results:
+   - Maximum sustainable throughput before p95 exceeds threshold
+   - Current headroom as percentage of maximum capacity
+   - Projected capacity needed for growth targets (e.g., 2x traffic in 6 months)
+   - Scaling recommendation: vertical (bigger instances) vs horizontal (more instances)
+
+4. **Compare to baseline.** If previous results exist:
+   - Latency delta per endpoint (regression or improvement)
+   - Throughput delta (higher or lower capacity)
+   - Error rate delta
+   - Flag any regressions greater than 10% as requiring investigation
+
+5. **Produce the load test report.** Output a structured summary:
+
+   ```
+   Load Test Report: <profile> — <date>
+   Target: <URL> | Tool: <k6/Artillery/Gatling> | Duration: <time>
+   Commit: <SHA>
+
+   Results:
+     Endpoint          | p50    | p95    | p99    | Errors | RPS   | Status
+     GET /api/products | 45ms   | 120ms  | 340ms  | 0.1%   | 850   | PASS
+     POST /api/orders  | 180ms  | 520ms  | 1100ms | 0.8%   | 120   | WARN (p99 > 1000ms)
+     GET /api/search   | 95ms   | 680ms  | 2100ms | 2.3%   | 340   | FAIL (p99 > 1000ms, errors > 1%)
+
+   Capacity: Max sustainable 1200 RPS at p95 < 500ms (current production: ~400 RPS, 3x headroom)
+   Bottleneck: /api/search becomes I/O-bound at 500 RPS — database full-text search query
+   Recommendation: Add search index or migrate to Elasticsearch for /api/search
+   ```
+
+6. **Archive results.** Save the full report and raw data for historical comparison.
+
+---
+
+## Harness Integration
+
+- **`harness skill run harness-load-testing`** -- Primary CLI entry point. Runs all four phases.
+- **`harness validate`** -- Run after generating test scripts to verify project structure.
+- **`harness check-deps`** -- Verify load testing tool dependencies are declared (k6, artillery npm package).
+- **`emit_interaction`** -- Used before execution (checkpoint:human-verify) to confirm target environment and test profile.
+- **`Glob`** -- Discover existing load test files, result archives, and configuration.
+- **`Grep`** -- Search for endpoint definitions, route handlers, and threshold configurations.
+- **`Write`** -- Generate load test scripts and result reports.
+- **`Edit`** -- Update existing test scripts with new scenarios or adjusted thresholds.
+
+## Success Criteria
+
+- All critical endpoints are identified and have corresponding load test scenarios
+- Test scripts are syntactically valid and pass a smoke test before full execution
+- Thresholds are derived from SLOs or documented baseline measurements, not arbitrary values
+- Results include latency percentiles (p50, p95, p99), error rates, and throughput
+- Bottlenecks are identified with specific causes (CPU, I/O, connection pool, memory)
+- Capacity projections include headroom percentage and scaling recommendations
+
+## Examples
+
+### Example: k6 Load Test for Express.js REST API
+
+```
+Phase 1: DETECT
+  Tool: k6 (found k6/ directory with 2 existing scripts)
+  Existing tests:
+    - k6/smoke-api.k6.js: GET /api/health (1 VU, 10s)
+    - k6/load-products.k6.js: GET /api/products (50 VUs, 5m, p95 < 300ms)
+  Coverage gaps:
+    - POST /api/orders — revenue-critical, no load test
+    - GET /api/search — high-traffic, no load test
+    - No stress or soak test profiles exist
+
+Phase 2: DESIGN
+  New scenarios:
+    - k6/load-orders.k6.js: POST /api/orders
+      Stages: ramp to 100 VUs over 2m, hold 5m, ramp down 1m
+      Thresholds: p95 < 800ms, errors < 1%, RPS > 80
+    - k6/stress-api.k6.js: All endpoints
+      Stages: step-up from 50 to 500 VUs in 50-VU increments, 2m per step
+      Thresholds: find breaking point, record p95 at each step
+    - k6/soak-api.k6.js: Critical endpoints at expected load
+      Duration: 2 hours at 200 VUs
+      Thresholds: p95 < 500ms, memory growth < 50MB/hour
+
+Phase 3: EXECUTE
+  Environment: https://staging.example.com (verified non-production)
+  Smoke: All scripts pass with 1 VU
+  Load test results captured to load-tests/results/2026-03-27-load.json
+
+Phase 4: ANALYZE
+  Results: POST /api/orders p95=620ms (PASS), GET /api/search p99=2100ms (FAIL)
+  Bottleneck: Full-text search on PostgreSQL LIKE query at 300+ RPS
+  Capacity: 800 RPS sustainable, current production 250 RPS (3.2x headroom)
+  Recommendation: Add pg_trgm index or migrate search to Elasticsearch
+```
+
+### Example: Artillery Test for NestJS GraphQL API
+
+```
+Phase 1: DETECT
+  Tool: Artillery (found artillery.yml with 1 scenario)
+  Existing: query { products } at 20 RPS for 60s
+  Gaps: mutations not tested, no spike profile, thresholds not defined
+
+Phase 2: DESIGN
+  New config: artillery/graphql-load.yml
+    Phases:
+      - warm-up: 5 RPS for 30s
+      - load: 50 RPS for 5m
+      - spike: jump to 200 RPS for 30s, back to 50 RPS
+    Scenarios:
+      - query { products(limit: 20) } — 60% weight
+      - mutation { createOrder(input: $input) } — 25% weight
+      - query { user(id: $id) { orders } } — 15% weight
+    Ensure:
+      - p99 < 1500ms
+      - maxErrorRate < 2
+
+Phase 3: EXECUTE
+  Target: http://staging.internal:3000/graphql
+  Smoke: PASS (all queries resolve, auth tokens valid)
+  Full run: artillery run --output report.json artillery/graphql-load.yml
+
+Phase 4: ANALYZE
+  Results:
+    query products: p95=89ms, p99=210ms — PASS
+    mutation createOrder: p95=340ms, p99=890ms — PASS
+    query user.orders: p95=520ms, p99=2400ms — FAIL
+  Bottleneck: N+1 query in user.orders resolver (no DataLoader)
+  Spike recovery: System recovered to baseline within 15s after spike — PASS
+  Recommendation: Add DataLoader for orders resolver, re-test after fix
+```
+
+## Gates
+
+- **No load tests against production without explicit human approval.** Load tests can cause real outages. The target environment must be verified as non-production before execution. If production testing is required, a `[checkpoint:human-verify]` must be passed with documented approval.
+- **No test execution with failing smoke tests.** If the smoke test fails, the test script is broken. Fix the script before running at load. Running broken scripts at scale wastes time and produces meaningless results.
+- **No capacity claims without stress test data.** Capacity projections require finding the actual breaking point. A load test that passes at the expected level does not establish maximum capacity. Stress test data is required for credible projections.
+- **No threshold changes without documented justification.** If thresholds are relaxed from a previous baseline, the reason must be documented (e.g., "p95 threshold increased from 300ms to 500ms due to added encryption overhead per SEC-2026-04 requirement").
+
+## Escalation
+
+- **When the staging environment does not match production configuration:** Load test results against a differently-sized environment are misleading. Report: "Staging has [N] instances with [M] CPU/RAM. Production has [X] instances with [Y] CPU/RAM. Results should be scaled by a factor of [ratio], but this is approximate. Recommend matching staging to production configuration for accurate capacity planning."
+- **When external dependencies cannot be stubbed for load testing:** Real external APIs may rate-limit or block load test traffic. Report: "The [dependency] cannot be stubbed and will rate-limit at [N] RPS. Options: (1) use a mock service like WireMock, (2) test with a dedicated sandbox API key, (3) test only the internal service layer with the external call short-circuited."
+- **When load test results are inconsistent between runs:** Noisy results indicate environmental interference. Report: "Variance between runs exceeds 15% for [endpoint]. Possible causes: shared staging environment, garbage collection pauses, network congestion. Recommend running 3 iterations and using the median, or running on a dedicated isolated environment."
+- **When the breaking point is below expected production traffic:** This is a capacity emergency. Report: "The system breaks at [N] RPS but production receives [M] RPS. Immediate scaling action required. Short-term: increase instance count by [factor]. Long-term: address the identified bottleneck in [component]."

package/dist/agents/skills/codex/harness-load-testing/skill.yaml

@@ -0,0 +1,79 @@
+name: harness-load-testing
+version: "1.0.0"
+description: Stress testing, capacity planning, and performance benchmarking with k6/Artillery/Gatling
+cognitive_mode: meticulous-verifier
+triggers:
+  - manual
+  - on_milestone
+platforms:
+  - claude-code
+  - gemini-cli
+tools:
+  - Bash
+  - Read
+  - Write
+  - Edit
+  - Glob
+  - Grep
+  - emit_interaction
+cli:
+  command: harness skill run harness-load-testing
+  args:
+    - name: path
+      description: Project root path
+      required: false
+    - name: tool
+      description: "Load testing tool: k6, artillery, or gatling. Auto-detected when omitted."
+      required: false
+    - name: profile
+      description: "Test profile: smoke, load, stress, spike, or soak. Defaults to load."
+      required: false
+mcp:
+  tool: run_skill
+  input:
+    skill: harness-load-testing
+    path: string
+type: rigid
+tier: 3
+internal: false
+keywords:
+  - load test
+  - stress test
+  - performance test
+  - k6
+  - Artillery
+  - Gatling
+  - JMeter
+  - throughput
+  - latency
+  - p99
+  - capacity planning
+  - benchmark
+  - soak test
+  - spike test
+stack_signals:
+  - "load-tests/"
+  - "k6/"
+  - "artillery/"
+  - "*.k6.js"
+  - "artillery.yml"
+  - "gatling/"
+  - "src/**/load-test*"
+  - "perf/"
+phases:
+  - name: detect
+    description: Identify existing load tests, tooling, endpoints, and performance baselines
+    required: true
+  - name: design
+    description: Define test scenarios, virtual user profiles, thresholds, and ramp-up stages
+    required: true
+  - name: execute
+    description: Run load tests and collect metrics for throughput, latency, and error rates
+    required: true
+  - name: analyze
+    description: Compare results against thresholds, identify bottlenecks, and produce capacity report
+    required: true
+state:
+  persistent: false
+  files: []
+depends_on: []