@assertrx/dino 1.0.10 → 1.0.12

package/skills/designing-test-scenarios/reference/executors.md

@@ -0,0 +1,215 @@
+# k6 Executors Reference
+
+## Overview
+
+k6 provides 6 executor types organized into two models:
+
+**Closed Model** (VU-based): Iteration rate depends on system response time
+- `shared-iterations`, `per-vu-iterations`, `constant-vus`, `ramping-vus`
+
+**Open Model** (arrival-rate): Iteration rate is fixed regardless of response time
+- `constant-arrival-rate`, `ramping-arrival-rate`
+
+---
+
+## shared-iterations
+
+A fixed total number of iterations shared between VUs. Test ends when all iterations complete.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `vus` | integer | 1 | Number of VUs to run concurrently |
+| `iterations` | integer | 1 | Total iterations across all VUs |
+| `maxDuration` | string | `"10m"` | Max scenario duration |
+
+**Distribution is NOT guaranteed even.** Faster VUs execute more iterations.
+
+```javascript
+export const options = {
+  scenarios: {
+    quick_test: {
+      executor: 'shared-iterations',
+      vus: 10,
+      iterations: 200,
+      maxDuration: '30s',
+    },
+  },
+};
+```
+
+**Use when:** You need a fixed total iteration count and don't care about even distribution.
+
+---
+
+## per-vu-iterations
+
+Each VU executes an exact number of iterations. Total = VUs x iterations.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `vus` | integer | 1 | Number of VUs to run concurrently |
+| `iterations` | integer | 1 | Iterations per VU |
+| `maxDuration` | string | `"10m"` | Max scenario duration |
+
+```javascript
+export const options = {
+  scenarios: {
+    even_distribution: {
+      executor: 'per-vu-iterations',
+      vus: 10,
+      iterations: 20,  // Each VU does exactly 20 iterations (200 total)
+      maxDuration: '1m',
+    },
+  },
+};
+```
+
+**Use when:** Each VU needs a guaranteed number of iterations (e.g., partitioned test data).
+
+---
+
+## constant-vus
+
+Fixed VU count runs for a set duration. VUs execute as many iterations as possible.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `duration` | string | required | Scenario duration |
+| `vus` | integer | 1 | Number of concurrent VUs |
+
+```javascript
+export const options = {
+  scenarios: {
+    constant_load: {
+      executor: 'constant-vus',
+      vus: 50,
+      duration: '5m',
+    },
+  },
+};
+```
+
+**Use when:** Simple constant load for a fixed duration.
+
+---
+
+## ramping-vus
+
+VU count changes over time via stages. VUs execute as many iterations as possible.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `stages` | array | required | Array of `{ duration, target }` objects |
+| `startVUs` | integer | 1 | Initial VU count |
+| `gracefulRampDown` | string | `"30s"` | Wait time during ramp-down |
+
+```javascript
+export const options = {
+  scenarios: {
+    ramping_load: {
+      executor: 'ramping-vus',
+      startVUs: 0,
+      stages: [
+        { duration: '5m', target: 100 },   // Ramp up to 100
+        { duration: '30m', target: 100 },  // Stay at 100
+        { duration: '5m', target: 0 },     // Ramp down to 0
+      ],
+      gracefulRampDown: '30s',
+    },
+  },
+};
+```
+
+**Stage behavior:** VU count transitions linearly from previous target to current target over the stage duration.
+
+**Use when:** Load tests with ramp-up/ramp-down phases (load, stress, soak, spike tests).
+
+---
+
+## constant-arrival-rate
+
+Fixed iteration rate independent of system response time (open model).
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `duration` | string | required | Scenario duration |
+| `rate` | integer | required | Iterations per `timeUnit` |
+| `timeUnit` | string | `"1s"` | Time period for rate |
+| `preAllocatedVUs` | integer | required | VUs pre-allocated before start |
+| `maxVUs` | integer | preAllocatedVUs | Maximum VU count |
+
+**Important:** Iterations start at evenly spaced intervals. At `rate: 10, timeUnit: '1s'`, one iteration starts every 100ms.
+
+**Do NOT use `sleep()` at end of iteration** — pacing is handled by rate/timeUnit.
+
+```javascript
+export const options = {
+  scenarios: {
+    constant_rps: {
+      executor: 'constant-arrival-rate',
+      rate: 50,                // 50 iterations per second
+      timeUnit: '1s',
+      duration: '5m',
+      preAllocatedVUs: 20,     // Start with 20 VUs
+      maxVUs: 100,             // Scale up to 100 if needed
+    },
+  },
+};
+```
+
+**Use when:** You need a guaranteed request rate (RPS), regardless of system performance. Preferred for realistic load testing.
+
+**Tuning preAllocatedVUs:** If too low, k6 spends time allocating VUs and may not hit target rate initially. Set to expected concurrent VU count based on `rate × average_iteration_duration`.
+
+---
+
+## ramping-arrival-rate
+
+Variable iteration rate over time via stages (open model).
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `stages` | array | required | Array of `{ duration, target }` objects |
+| `startRate` | integer | 0 | Initial iteration rate per `timeUnit` |
+| `timeUnit` | string | `"1s"` | Time period for rate (constant for entire scenario) |
+| `preAllocatedVUs` | integer | required | VUs pre-allocated before start |
+| `maxVUs` | integer | preAllocatedVUs | Maximum VU count |
+
+**Do NOT use `sleep()` at end of iteration.**
+
+```javascript
+export const options = {
+  scenarios: {
+    ramping_rps: {
+      executor: 'ramping-arrival-rate',
+      startRate: 10,
+      timeUnit: '1s',
+      preAllocatedVUs: 50,
+      maxVUs: 200,
+      stages: [
+        { duration: '2m', target: 10 },   // Stay at 10 RPS
+        { duration: '5m', target: 100 },  // Ramp to 100 RPS
+        { duration: '5m', target: 100 },  // Stay at 100 RPS
+        { duration: '2m', target: 0 },    // Ramp down to 0
+      ],
+    },
+  },
+};
+```
+
+**Use when:** Ramp request rate up/down over time. Ideal for breakpoint tests (finding system limits) and realistic traffic simulation.
+
+---
+
+## Executor Selection Quick Reference
+
+| Scenario | Recommended Executor |
+|----------|---------------------|
+| Quick smoke test | `shared-iterations` (low vus, few iterations) |
+| Each VU tests different data | `per-vu-iterations` |
+| Simple constant load | `constant-vus` |
+| Load/stress/soak with ramp | `ramping-vus` |
+| Maintain exact RPS | `constant-arrival-rate` |
+| Ramp RPS up/down | `ramping-arrival-rate` |
+| Breakpoint (find limits) | `ramping-arrival-rate` |
+| Spike test | `ramping-vus` (fast ramp) or `ramping-arrival-rate` |

package/skills/designing-test-scenarios/reference/test-types.md

@@ -0,0 +1,215 @@
+# k6 Test Types Reference
+
+## Summary
+
+| Test Type | VUs | Duration | Purpose |
+|-----------|-----|----------|---------|
+| Smoke | 2-5 | 30s-3m | Verify script works, baseline metrics |
+| Load | Average production | 5-60min | Typical load performance |
+| Stress | 50-100%+ above avg | 10-60min | Heavy load behavior |
+| Soak | Average production | 3-72 hours | Long-running reliability |
+| Spike | Very high, sudden | 2-5min | Sudden traffic burst |
+| Breakpoint | Incremental | Until failure | Find system limits |
+
+---
+
+## Smoke Test
+
+Minimal load to verify the script works and establish baseline metrics.
+
+```javascript
+export const options = {
+  vus: 3,
+  duration: '1m',
+  thresholds: {
+    http_req_failed: ['rate<0.01'],
+    http_req_duration: ['p(95)<500'],
+  },
+};
+```
+
+**When to run:** After every script change. Before any other test type.
+
+---
+
+## Load Test (Average Load)
+
+Assess performance under typical production load.
+
+**Stage pattern:** Ramp-up (5-15% of total) → Plateau (5x ramp-up) → Ramp-down
+
+```javascript
+export const options = {
+  stages: [
+    { duration: '5m', target: 100 },   // Ramp up
+    { duration: '30m', target: 100 },  // Plateau
+    { duration: '5m', target: 0 },     // Ramp down
+  ],
+  thresholds: {
+    http_req_duration: ['p(95)<500'],
+    http_req_failed: ['rate<0.01'],
+  },
+};
+```
+
+**When to run:** Regularly after code changes. Foundation for all other test types.
+
+**What to look for:**
+- Compare response times during ramp-up vs full load
+- Response time stability during plateau
+- Error rate trends
+
+---
+
+## Stress Test
+
+Assess performance under heavier-than-normal load (50-100%+ above average).
+
+```javascript
+export const options = {
+  stages: [
+    { duration: '10m', target: 200 },  // Ramp up to 2x normal
+    { duration: '30m', target: 200 },  // Sustained heavy load
+    { duration: '5m', target: 0 },     // Ramp down
+  ],
+  thresholds: {
+    http_req_duration: ['p(95)<1000'],
+    http_req_failed: ['rate<0.05'],
+  },
+};
+```
+
+**When to run:** After successful load tests. Before anticipated high-traffic events.
+
+**What to look for:**
+- Performance degradation rate
+- Error rate increase patterns
+- Recovery behavior during ramp-down
+
+---
+
+## Soak Test
+
+Test reliability and performance over extended periods. Same load as load test but much longer duration.
+
+```javascript
+export const options = {
+  stages: [
+    { duration: '5m', target: 100 },   // Ramp up
+    { duration: '8h', target: 100 },   // Extended plateau (hours!)
+    { duration: '5m', target: 0 },     // Ramp down
+  ],
+  thresholds: {
+    http_req_duration: ['p(95)<500'],
+    http_req_failed: ['rate<0.01'],
+  },
+};
+```
+
+**When to run:** After successful load and stress tests. Before production deployments.
+
+**What to look for:**
+- Memory leaks (gradual increase in response times)
+- Database connection pool exhaustion
+- Log/storage growth issues
+- Gradual metric degradation over time
+
+---
+
+## Spike Test
+
+Verify system survives sudden, massive traffic bursts.
+
+```javascript
+export const options = {
+  stages: [
+    { duration: '2m', target: 2000 },  // Fast ramp to extreme load
+    { duration: '1m', target: 0 },     // Quick ramp down
+  ],
+};
+```
+
+**Alternative — multiple spikes:**
+
+```javascript
+export const options = {
+  stages: [
+    { duration: '1m', target: 100 },    // Normal load
+    { duration: '30s', target: 1500 },  // Spike 1
+    { duration: '30s', target: 100 },   // Recovery
+    { duration: '30s', target: 1500 },  // Spike 2
+    { duration: '1m', target: 0 },      // Ramp down
+  ],
+};
+```
+
+**When to run:** For systems expecting sudden traffic (sales events, broadcasts, deadlines).
+
+**What to look for:**
+- System survival (does it crash?)
+- Auto-scaling response time
+- Recovery time after spike
+- Error patterns during peak
+
+---
+
+## Breakpoint Test
+
+Find system capacity limits by continuously increasing load until failure.
+
+**Recommended executor:** `ramping-arrival-rate` (maintains throughput despite degradation)
+
+```javascript
+export const options = {
+  scenarios: {
+    breakpoint: {
+      executor: 'ramping-arrival-rate',
+      preAllocatedVUs: 100,
+      maxVUs: 2000,
+      stages: [
+        { duration: '2h', target: 20000 },  // Slow continuous ramp
+      ],
+    },
+  },
+  thresholds: {
+    http_req_failed: [{
+      threshold: 'rate>0.1',
+      abortOnFail: true,
+      delayAbortEval: '1m',
+    }],
+  },
+};
+```
+
+**When to run:** To discover system capacity limits. After tuning system to verify new limits.
+
+**What to look for (progressive failure stages):**
+1. Degraded performance → response times increase
+2. Troublesome → severe user experience degradation
+3. Timeouts → requests start timing out
+4. Errors → HTTP error codes appear
+5. System failure → complete collapse
+
+**Warning:** Avoid in elastic cloud environments without disabling auto-scaling, or you may only discover your cloud bill limit.
+
+---
+
+## Test Type Decision Guide
+
+```
+What do you want to learn?
+│
+├─ "Does my script work?" → Smoke Test
+│
+├─ "How does it perform normally?" → Load Test
+│
+├─ "Can it handle peak traffic?" → Stress Test
+│
+├─ "Is it reliable over time?" → Soak Test
+│
+├─ "Can it survive sudden bursts?" → Spike Test
+│
+└─ "Where does it break?" → Breakpoint Test
+```
+
+**Recommended progression:** Smoke → Load → Stress → Soak → Spike → Breakpoint

package/skills/designing-test-scenarios/reference/thresholds.md

@@ -0,0 +1,208 @@
+# k6 Thresholds Reference
+
+## Defining Thresholds
+
+### Short Format (Array of Strings)
+
+```javascript
+export const options = {
+  thresholds: {
+    http_req_duration: ['p(95)<500', 'p(99)<1000'],
+    http_req_failed: ['rate<0.01'],
+  },
+};
+```
+
+### Long Format (With Abort Control)
+
+```javascript
+export const options = {
+  thresholds: {
+    http_req_duration: [{
+      threshold: 'p(99)<1000',
+      abortOnFail: true,
+      delayAbortEval: '30s',
+    }],
+  },
+};
+```
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `threshold` | string | Threshold expression |
+| `abortOnFail` | boolean | Stop test if threshold fails |
+| `delayAbortEval` | string | Wait before evaluating abort (e.g., `"10s"`) |
+
+## Threshold Expression Syntax
+
+```
+<aggregation_method> <operator> <value>
+```
+
+**Operators:** `<`, `<=`, `>`, `>=`, `==`, `!=`
+
+## Aggregation Methods by Metric Type
+
+| Metric Type | Methods | Example Metrics |
+|-------------|---------|-----------------|
+| **Counter** | `count`, `rate` | `http_reqs`, `data_received` |
+| **Gauge** | `value` | `vus`, `vus_max` |
+| **Rate** | `rate` | `http_req_failed`, `checks` |
+| **Trend** | `avg`, `min`, `max`, `med`, `p(N)` | `http_req_duration`, `iteration_duration` |
+
+**Percentile syntax:** `p(N)` where N is 0.0 to 100 (e.g., `p(90)`, `p(95)`, `p(99)`, `p(99.9)`)
+
+## Common Threshold Patterns
+
+### Response Time
+
+```javascript
+thresholds: {
+  http_req_duration: [
+    'p(50)<200',     // Median under 200ms
+    'p(90)<400',     // 90th percentile under 400ms
+    'p(95)<500',     // 95th percentile under 500ms
+    'p(99)<1000',    // 99th percentile under 1s
+    'avg<300',       // Average under 300ms
+    'max<3000',      // Max under 3s
+  ],
+}
+```
+
+### Error Rate
+
+```javascript
+thresholds: {
+  http_req_failed: ['rate<0.01'],   // Less than 1% errors
+  checks: ['rate>0.99'],            // 99% of checks pass
+}
+```
+
+### Throughput
+
+```javascript
+thresholds: {
+  http_reqs: ['rate>100'],          // At least 100 RPS
+  http_reqs: ['count>10000'],       // At least 10000 total requests
+}
+```
+
+### Custom Metrics
+
+```javascript
+import { Trend, Rate, Counter, Gauge } from 'k6/metrics';
+
+const apiLatency = new Trend('api_latency');
+const successRate = new Rate('success_rate');
+const errorCount = new Counter('error_count');
+const queueSize = new Gauge('queue_size');
+
+export const options = {
+  thresholds: {
+    api_latency: ['p(95)<500', 'avg<200'],   // Trend
+    success_rate: ['rate>0.95'],              // Rate
+    error_count: ['count<100'],              // Counter
+    queue_size: ['value<1000'],              // Gauge
+  },
+};
+```
+
+## Tag-Based Thresholds
+
+Apply different thresholds to different request groups:
+
+```javascript
+export const options = {
+  thresholds: {
+    // Global threshold
+    http_req_duration: ['p(95)<500'],
+    // Per-endpoint thresholds
+    'http_req_duration{name:GetUsers}': ['p(95)<300'],
+    'http_req_duration{name:CreateOrder}': ['p(95)<800'],
+    'http_req_duration{name:UploadFile}': ['p(95)<2000'],
+    // Per-type thresholds
+    'http_req_duration{type:api}': ['p(95)<500'],
+    'http_req_duration{type:static}': ['p(95)<100'],
+  },
+};
+
+export default function () {
+  http.get('https://api.example.com/users', {
+    tags: { name: 'GetUsers', type: 'api' },
+  });
+  http.post('https://api.example.com/orders', payload, {
+    tags: { name: 'CreateOrder', type: 'api' },
+  });
+}
+```
+
+### Group Duration Thresholds
+
+```javascript
+export const options = {
+  thresholds: {
+    'group_duration{group:::login_flow}': ['avg<3000'],
+    'group_duration{group:::checkout}': ['avg<5000'],
+  },
+};
+```
+
+Note: Group names in threshold tags use `:::` as separator.
+
+## Combining Checks with Thresholds
+
+Checks alone do NOT affect test exit status. Use thresholds on check rate:
+
+```javascript
+export const options = {
+  thresholds: {
+    checks: ['rate>0.99'],                          // All checks combined
+    'checks{myTag:critical}': ['rate>0.999'],       // Tagged checks
+  },
+};
+
+export default function () {
+  const res = http.get(url);
+
+  // Regular check
+  check(res, { 'status 200': (r) => r.status === 200 });
+
+  // Tagged check (higher threshold)
+  check(res,
+    { 'body valid': (r) => r.body.includes('expected') },
+    { myTag: 'critical' }
+  );
+}
+```
+
+## Typical SLA Thresholds
+
+### Web Application
+
+```javascript
+thresholds: {
+  http_req_duration: ['p(95)<500', 'p(99)<1500'],
+  http_req_failed: ['rate<0.01'],
+  checks: ['rate>0.99'],
+}
+```
+
+### API Service
+
+```javascript
+thresholds: {
+  http_req_duration: ['p(95)<200', 'p(99)<500'],
+  http_req_failed: ['rate<0.001'],
+  http_reqs: ['rate>500'],
+}
+```
+
+### Real-Time Application
+
+```javascript
+thresholds: {
+  http_req_duration: ['p(99)<100'],
+  http_req_failed: ['rate<0.0001'],
+  ws_session_duration: ['p(95)<30000'],
+}
+```