qa-skills 3.0.0

package/skills/qa-jmeter-writer/references/best-practices.md

@@ -0,0 +1,157 @@
+# JMeter Best Practices
+
+## Non-GUI Mode
+
+| Do | Don't |
+|----|-------|
+| Always use `-n` for load tests | Run load tests in GUI mode |
+| Disable View Results Tree in load plans | Keep View Results Tree enabled for 1000+ threads |
+| Use Aggregate Report, Summary Report | Use listeners that store full response data |
+| Set `jmeter.save.saveservice.response_data=false` | Save response data in JTL for load runs |
+
+**Why:** GUI mode consumes significant memory and is not designed for high load. View Results Tree stores every response and causes OOM.
+
+---
+
+## Parameterization
+
+### CSV Data Set Config
+
+| Setting | Recommendation |
+|--------|----------------|
+| **Sharing mode** | `All threads` for shared data; `Current thread` for unique per user |
+| **Delimiter** | `,` (comma) or `\t` (tab) |
+| **Recycle on EOF** | `true` for repeated use; `false` to stop when exhausted |
+| **Stop thread on EOF** | `true` when recycle=false and you want graceful stop |
+| **File path** | Use `${__P(csvPath, default.csv)}` for CI flexibility |
+
+### Unique Data
+
+- Use enough rows for `threads × loops` when recycle=false
+- For user simulation: unique emails, usernames per row
+- Consider UUID or timestamp in JSR223 for dynamic uniqueness
+
+### Avoid
+
+- Hardcoded URLs, credentials, IDs
+- Same data for all users when testing concurrency (e.g., same login)
+
+---
+
+## Correlation
+
+### Order of Execution
+
+1. **Sampler** (e.g., Login) → **PostProcessor** (extract token) → **Next Sampler** (use `${token}`)
+2. Extractors must be children of the sampler whose response they parse
+3. Use **JSON Extractor** or **Regular Expression Extractor** as PostProcessor
+
+### Variable Scope
+
+- `vars` (JMeterVariables): per-thread, cleared each iteration
+- `props` (JMeterProperties): global, shared across threads
+- Use `vars.put("name", value)` in JSR223; `${name}` in samplers
+
+### Regex Tips
+
+- Prefer non-greedy: `"token":"([^"]+)"` over `"token":"(.+)"`
+- Use `$1$` in Template for first capture group
+- Test regex in View Results Tree (debug only) or regex101.com
+
+### JSON Extraction
+
+- Use **JSON Extractor** or **JSR223 PostProcessor** with JsonSlurper
+- JSONPath: `$.data.token`, `$.items[0].id`
+
+---
+
+## Result Analysis
+
+### JTL Columns
+
+| Column | Purpose |
+|--------|---------|
+| `timeStamp` | Request time |
+| `elapsed` | Total time (ms) |
+| `label` | Sampler name |
+| `responseCode` | HTTP status |
+| `success` | true/false |
+| `bytes` | Response size |
+| `latency` | Time to first byte |
+| `connect` | Connection time |
+
+### HTML Report
+
+- **Dashboard**: Overview, APDEX, response times, throughput
+- **APDEX**: Satisfied (fast), Tolerated (ok), Frustrated (slow)
+- **Throughput**: Requests per second
+- **Response Times**: Percentiles (90, 95, 99)
+
+### Interpreting Results
+
+| Metric | Good | Investigate |
+|--------|------|-------------|
+| Error rate | < 1% | > 1% |
+| p95 latency | Within SLA | Exceeds SLA |
+| Throughput | Meets target | Below target |
+| Connect time | Low, stable | High or spiking |
+
+---
+
+## Test Design
+
+### Ramp-Up
+
+- **Rule of thumb**: Ramp-up = 10–20% of total test duration
+- Too short: artificial spike, connection storms
+- Too long: wastes time before steady state
+
+### Think Time
+
+- Add **Constant Timer** or **Uniform Random Timer** between requests
+- Typical: 1–3 seconds for web; 0–500 ms for API
+- Omit for throughput-only tests
+
+### Assertions
+
+- Use **Response Assertion** for status codes (200, 201)
+- Use **Duration Assertion** for SLA (e.g., 2000 ms)
+- Use **JSON Assertion** for API contract validation
+- Avoid overly strict assertions that fail on minor variations
+
+### Naming
+
+- Use descriptive **labels** for samplers (e.g., `GET /users`, `POST /login`)
+- Labels appear in reports and JTL; avoid generic names like "HTTP Request"
+
+---
+
+## Performance of JMeter Itself
+
+| Tweak | Purpose |
+|-------|---------|
+| Increase JVM heap | `-Xmx4g` for 5k+ threads |
+| Disable unnecessary listeners | Reduce memory and I/O |
+| Use `jmeter.save.saveservice.*` to minimize JTL | Smaller files, faster writes |
+| Run workers on separate machines | Distribute load, avoid single-node bottleneck |
+| Use latest JMeter | Bug fixes, performance improvements |
+
+---
+
+## Security
+
+- Never commit credentials; use `${__P()}` or environment variables
+- Use `user.properties` (gitignored) for local overrides
+- In CI: use secrets for API keys, tokens
+- Test against staging/QA, not production, unless explicitly approved
+
+---
+
+## Integration with QA Skills
+
+| Skill | Use Case |
+|-------|----------|
+| **qa-plan-creator** | Performance plan → thread groups, duration, load profile |
+| **qa-nfr-analyst** | NFR specs → SLA assertions, duration thresholds |
+| **qa-api-contract-curator** | OpenAPI → HTTP samplers, JSON assertions |
+| **qa-requirements-generator** | Requirements → test scenarios, endpoints |

package/skills/qa-jmeter-writer/references/config.md

@@ -0,0 +1,204 @@
+# JMeter Configuration
+
+## CLI Mode (Non-GUI)
+
+### Basic Run
+
+```bash
+jmeter -n -t test.jmx -l results.jtl
+```
+
+| Option | Purpose |
+|--------|---------|
+| `-n` | Non-GUI mode (required for CI/headless) |
+| `-t test.jmx` | Test plan file |
+| `-l results.jtl` | Results file (JTL/CSV) |
+| `-j jmeter.log` | Log file |
+| `-q user.properties` | User properties file |
+
+### HTML Report Generation
+
+```bash
+jmeter -n -t test.jmx -l results.jtl -e -o report/
+```
+
+| Option | Purpose |
+|--------|---------|
+| `-e` | Generate HTML report after run |
+| `-o report/` | Output directory for HTML report |
+
+Report includes: dashboard, APDEX, response times, throughput, errors.
+
+### Property Overrides
+
+```bash
+jmeter -n -t test.jmx -l results.jtl -Jhost=api.staging.example.com -Jusers=50
+```
+
+| Option | Purpose |
+|--------|---------|
+| `-Jprop=value` | Set JMeter property (single) |
+| `-Gprop=value` | Global property (distributed mode) |
+| `-q user.properties` | Load from file |
+
+Use in test plan: `${__P(host, default)}`, `${__P(users, 10)}`.
+
+---
+
+## Key Properties
+
+### JTL Output Format
+
+```properties
+# user.properties or -J
+jmeter.save.saveservice.output_format=csv
+jmeter.save.saveservice.response_data=false
+jmeter.save.saveservice.samplerData=false
+jmeter.save.saveservice.requestHeaders=false
+jmeter.save.saveservice.responseHeaders=false
+jmeter.save.saveservice.url=false
+jmeter.save.saveservice.filename=false
+jmeter.save.saveservice.hostname=false
+jmeter.save.saveservice.thread_counts=false
+jmeter.save.saveservice.latency=true
+jmeter.save.saveservice.timestamp_format=yyyy/MM/dd HH:mm:ss
+jmeter.save.saveservice.print_field_names=true
+```
+
+Minimal JTL for performance: `timeStamp`, `elapsed`, `label`, `responseCode`, `success`, `bytes`, `latency`.
+
+### HTML Report
+
+```properties
+jmeter.reportgenerator.overall_granularity=60000
+jmeter.reportgenerator.apdex_satisfied_threshold=500
+jmeter.reportgenerator.apdex_tolerated_threshold=1500
+```
+
+### JVM Heap
+
+```bash
+# Set in jmeter or jmeter.bat
+HEAP="-Xms1g -Xmx4g -XX:MaxMetaspaceSize=256m"
+```
+
+Increase for high thread counts (e.g., 10k+ threads).
+
+---
+
+## Plugins
+
+### JMeter Plugins Manager
+
+Install via Plugins Manager (Options → Plugins Manager):
+
+| Plugin | Purpose |
+|--------|---------|
+| **Custom Thread Groups** | Ultimate Thread Group, Stepping Thread Group |
+| **3 Basic Graphs** | Response Times, Throughput, Active Threads |
+| **PerfMon** | Server metrics (CPU, memory) |
+| **Dummy Sampler** | Simulate responses for debugging |
+| **JSON Path Assertion** | JSONPath assertions (built-in 5.2+) |
+
+### Maven/Gradle
+
+```xml
+<!-- For programmatic JMeter execution -->
+<dependency>
+  <groupId>org.apache.jmeter</groupId>
+  <artifactId>ApacheJMeter_core</artifactId>
+  <version>5.6.3</version>
+</dependency>
+```
+
+---
+
+## Distributed Testing
+
+### Architecture
+
+- **Controller**: Runs GUI/non-GUI, orchestrates workers
+- **Workers**: Run actual load, send results to controller
+
+### Setup
+
+1. **Same JMeter version** on all nodes
+2. **Same plugins** on all nodes
+3. **RMI ports**: 1099 (default), 4000 (server)
+4. **Firewall**: Allow controller ↔ workers
+
+### Start Workers
+
+```bash
+# On each worker machine
+jmeter-server
+# or
+jmeter -s -Jserver.rmi.localport=50000
+```
+
+### Run from Controller
+
+```bash
+jmeter -n -t test.jmx -l results.jtl -R worker1,worker2,worker3
+# or
+jmeter -n -t test.jmx -l results.jtl -Gremote_hosts=worker1,worker2
+```
+
+| Option | Purpose |
+|--------|---------|
+| `-R host1,host2` | Remote hosts (workers) |
+| `-Gprop=value` | Global property for all workers |
+| `-Jserver.rmi.ssl.disable=true` | Disable RMI SSL if needed |
+
+### Remote Hosts Property
+
+```properties
+# jmeter.properties
+remote_hosts=127.0.0.1:1099,worker2:1099
+```
+
+---
+
+## Directory Layout
+
+```
+project/
+  jmeter/
+    plans/
+      api-load.jmx
+      api-stress.jmx
+    data/
+      users.csv
+    results/          # JTL output
+    report/           # HTML report
+  user.properties     # Overrides
+  jmeter.log
+```
+
+---
+
+## CI Integration (GitHub Actions)
+
+```yaml
+- name: Run JMeter
+  run: |
+    jmeter -n -t jmeter/plans/api-load.jmx \
+      -l jmeter/results/results.jtl \
+      -e -o jmeter/report \
+      -Jhost=${{ env.API_HOST }} \
+      -Jusers=${{ env.LOAD_USERS }}
+```
+
+---
+
+## User Defined Variables
+
+Define at Test Plan or Thread Group level:
+
+| Variable | Value | Usage |
+|----------|-------|-------|
+| `host` | api.example.com | `${host}` |
+| `basePath` | /v1 | `${basePath}` |
+| `protocol` | https | `${protocol}` |
+
+Prefer `${__P(var, default)}` for CI overrides.

package/skills/qa-jmeter-writer/references/patterns.md

@@ -0,0 +1,242 @@
+# JMeter Test Plan Patterns
+
+## Thread Groups
+
+### Basic Thread Group
+
+| Property | Purpose | Example |
+|----------|---------|---------|
+| `num_threads` | Number of virtual users | 10, 100, 1000 |
+| `ramp_time` | Ramp-up period (seconds) | 60, 300 |
+| `loops` | Iterations per user (-1 = forever) | 1, 10, -1 |
+| `duration` | Test duration (seconds) | Use with loops=-1 |
+| `scheduler` | Enable scheduler | true/false |
+| `delay` | Startup delay (seconds) | 0 |
+| `on_sample_error` | Action on failure | continue, stopthread, stoppage, stopnow |
+
+### Load Profile Mapping
+
+| Test Type | Threads | Ramp-up | Loops/Duration |
+|-----------|---------|---------|---------------|
+| Smoke | 1–5 | 10s | 1–5 |
+| Load | Target VUs | 10–20% of duration | Duration-based |
+| Stress | Ramp to breaking point | Gradual | Until failure |
+| Soak | Steady VUs | Ramp-up | Hours |
+| Spike | Sudden surge | Very short | Short burst |
+
+### Ultimate Thread Group (Plugin)
+
+For complex load profiles (stages), use Ultimate Thread Group or Stepping Thread Group plugins when available.
+
+---
+
+## Samplers
+
+### HTTP Request Sampler
+
+```xml
+<HTTPSamplerProxy>
+  <stringProp name="HTTPSampler.domain">${__P(host,api.example.com)}</stringProp>
+  <stringProp name="HTTPSampler.path">/users</stringProp>
+  <stringProp name="HTTPSampler.method">GET</stringProp>
+  <boolProp name="HTTPSampler.follow_redirects">true</boolProp>
+  <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
+  <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
+  <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
+  <elementProp name="HTTPsampler.Arguments">
+    <collectionProp name="Arguments.arguments">
+      <elementProp name="" elementType="HTTPArgument">
+        <boolProp name="HTTPArgument.always_encode">false</boolProp>
+        <stringProp name="Argument.value">{"email":"${email}"}</stringProp>
+        <stringProp name="Argument.metadata">=</stringProp>
+      </elementProp>
+    </collectionProp>
+  </elementProp>
+</HTTPSamplerProxy>
+```
+
+| Property | Purpose |
+|----------|---------|
+| `HTTPSampler.domain` | Host (or use full URL in path) |
+| `HTTPSampler.path` | Path + query |
+| `HTTPSampler.method` | GET, POST, PUT, DELETE, PATCH |
+| `HTTPSampler.protocol` | http or https |
+| `HTTPSampler.port` | Port (empty = default) |
+| `HTTPsampler.Arguments` | Request body (POST/PUT) |
+| `HTTPSampler.contentEncoding` | UTF-8 |
+
+### HTTP Header Manager
+
+Add as child of sampler or test plan:
+
+```xml
+<HeaderManager>
+  <collectionProp name="HeaderManager.headers">
+    <elementProp name="Content-Type" elementType="Header">
+      <stringProp name="Header.name">Content-Type</stringProp>
+      <stringProp name="Header.value">application/json</stringProp>
+    </elementProp>
+    <elementProp name="Authorization" elementType="Header">
+      <stringProp name="Header.name">Authorization</stringProp>
+      <stringProp name="Header.value">Bearer ${token}</stringProp>
+    </elementProp>
+  </collectionProp>
+</HeaderManager>
+```
+
+### JDBC Request Sampler
+
+For database load testing:
+
+| Property | Purpose |
+|----------|---------|
+| `dataSource` | JDBC connection pool variable |
+| `queryType` | Select, Update, Callable, Prepared |
+| `query` | SQL statement |
+
+### JMS Sampler
+
+For messaging (JMS Point-to-Point, Pub/Sub) when testing message brokers.
+
+---
+
+## Assertions
+
+### Response Assertion
+
+| Field | Purpose |
+|-------|---------|
+| `Assertion.test_field` | Response Code, Response Message, Response Headers, Response Body |
+| `Assertion.pattern` | String or regex to match |
+| `Assertion.test_type` | Contains, Matches, Equals, Substring |
+| `Assertion.assume_success` | Treat as success if not applied |
+
+Common patterns:
+- Response Code: `200`, `201`, `2\d{2}`
+- Response Body: `"success":\s*true`, `"id":\s*\d+`
+
+### Duration Assertion
+
+| Property | Purpose |
+|----------|---------|
+| `DurationAssertion.duration` | Max allowed response time (ms) |
+
+Use for SLA validation (e.g., 2000 ms for p95).
+
+### Size Assertion
+
+| Property | Purpose |
+|----------|---------|
+| `SizeAssertion.size` | Expected size in bytes |
+| `SizeAssertion.assume_success` | Treat as success if not applied |
+
+### JSON Assertion (JMeter 5.2+)
+
+| Property | Purpose |
+|----------|---------|
+| `JSON_PATH` | JSONPath expression (e.g., `$.data.id`) |
+| `EXPECTED_VALUE` | Expected value or leave empty for existence |
+| `ASSERT_JSON_PATH_EXISTS` | true/false |
+
+---
+
+## Extractors
+
+### Regular Expression Extractor
+
+| Property | Purpose |
+|----------|---------|
+| `Refname` | Variable name (e.g., `token`) |
+| `Regex` | Regex with capture group, e.g., `"token":"([^"]+)"` |
+| `Template` | `$1$` for first group |
+| `Match Nr` | 1 = first, -1 = random, 0 = all |
+| `Default` | Default value if no match |
+
+### JSON Extractor
+
+| Property | Purpose |
+|----------|---------|
+| `JSON_PATH` | JSONPath, e.g., `$.data.token` |
+| `VAR` | Variable name |
+| `DEFAULT` | Default if not found |
+
+### Boundary Extractor
+
+For simple left/right boundary extraction when regex is overkill.
+
+---
+
+## Timers
+
+### Constant Timer
+
+| Property | Purpose |
+|----------|---------|
+| `ConstantTimer.delay` | Delay in ms (e.g., 1000 for 1s think time) |
+
+### Uniform Random Timer
+
+| Property | Purpose |
+|----------|---------|
+| `ConstantTimer.delay` | Minimum delay (ms) |
+| `RandomTimer.range` | Additional random range (ms) |
+
+Total delay = delay + random(0, range).
+
+### Gaussian Random Timer
+
+Similar to Uniform but with Gaussian distribution for more realistic think time.
+
+---
+
+## Controllers
+
+### Loop Controller
+
+| Property | Purpose |
+|----------|---------|
+| `LoopController.loops` | Number of iterations |
+
+### If Controller
+
+| Property | Purpose |
+|----------|---------|
+| `IfController.condition` | JavaScript or `${__jexl3(...)}` expression |
+
+Example: `${__jexl3(${status} == 200)}`
+
+### Once Only Controller
+
+Execute child elements only on first iteration (e.g., login once).
+
+### Transaction Controller
+
+Group samplers into a transaction for aggregate timing.
+
+---
+
+## Pre/Post Processors
+
+### JSR223 PreProcessor (Groovy)
+
+```groovy
+// Set variable from previous response
+vars.put("timestamp", String.valueOf(System.currentTimeMillis()));
+
+// Correlation: extract from prev sampler (use PostProcessor on prior sampler instead)
+// vars.put("token", prev.getResponseDataAsString());
+```
+
+Prefer Groovy over BeanShell (faster, maintained).
+
+### JSR223 PostProcessor
+
+```groovy
+import groovy.json.JsonSlurper
+def json = new JsonSlurper().parseText(prev.getResponseDataAsString())
+vars.put("userId", json.data.id.toString())
+```
+
+### User Parameters
+
+For per-iteration parameter overrides without CSV.