@assertrx/dino 1.0.10 → 1.0.11
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package.json +1 -1
- package/prompts/generate-k6-test-from-endpoint.md +68 -0
- package/prompts/generate-k6-test-from-openapi.md +85 -0
- package/prompts/setup-k6.md +130 -0
- package/scripts/api-server.mjs +23 -0
- package/skills/analyzing-test-results/SKILL.md +150 -0
- package/skills/analyzing-test-results/reference/bottleneck-patterns.md +130 -0
- package/skills/analyzing-test-results/reference/metrics.md +134 -0
- package/skills/analyzing-test-results/reference/visualization.md +152 -0
- package/skills/designing-test-scenarios/SKILL.md +247 -0
- package/skills/designing-test-scenarios/reference/executors.md +215 -0
- package/skills/designing-test-scenarios/reference/test-types.md +215 -0
- package/skills/designing-test-scenarios/reference/thresholds.md +208 -0
- package/skills/generating-api-load-tests/SKILL.md +199 -0
- package/skills/generating-api-load-tests/reference/data-parameterization.md +343 -0
- package/skills/generating-api-load-tests/reference/grpc.md +315 -0
- package/skills/generating-api-load-tests/reference/http.md +380 -0
- package/skills/generating-api-load-tests/reference/websocket.md +369 -0
- package/skills/generating-browser-tests/SKILL.md +269 -0
- package/skills/generating-browser-tests/reference/browser-api.md +251 -0
- package/skills/generating-browser-tests/reference/web-vitals.md +148 -0
- package/skills/generating-tests-from-code/SKILL.md +142 -0
- package/skills/generating-tests-from-code/reference/auth-analysis.md +220 -0
- package/skills/generating-tests-from-code/reference/framework-patterns.md +242 -0
- package/skills/generating-tests-from-openapi/SKILL.md +234 -0
- package/skills/generating-tests-from-openapi/reference/openapi-auth-mapping.md +247 -0
- package/skills/generating-tests-from-openapi/reference/openapi-parsing.md +194 -0
- package/skills/operating-k6-in-ci-cd/SKILL.md +98 -0
- package/skills/operating-k6-in-ci-cd/reference/ci-cd-pipelines.md +265 -0
- package/skills/operating-k6-in-ci-cd/reference/extensions.md +144 -0
- package/skills/operating-k6-in-ci-cd/reference/k6-operator.md +214 -0
- package/skills/testing-resilience/SKILL.md +230 -0
- package/skills/testing-resilience/reference/fault-injection.md +153 -0
- package/skills/testing-resilience/reference/resilience-patterns.md +500 -0
package/package.json
CHANGED
|
@@ -0,0 +1,68 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: Generate a k6 load test for a single provided endpoint using all the k6 skills
|
|
3
|
+
argument-hint: "<endpoint-url> [output-file] [test-type]"
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Generate a complete, runnable k6 load test for the endpoint at `$1`, guided by the k6 skills in this package. The default output file is `${2:-tests/k6/endpoint-test.js}` and the default test type is `${3:-load}`.
|
|
7
|
+
|
|
8
|
+
## Step 1 — Load the k6 skills
|
|
9
|
+
|
|
10
|
+
Before writing any code, use `read` to load the SKILL.md files for these skills from the package `skills/` directory (follow the reference files they point to when relevant):
|
|
11
|
+
|
|
12
|
+
- **designing-test-scenarios** — always; needed for the load profile and thresholds.
|
|
13
|
+
- **generating-api-load-tests** — always; needed for the script structure and protocol patterns.
|
|
14
|
+
- **analyzing-test-results** — always; needed to verify and interpret the validation run.
|
|
15
|
+
|
|
16
|
+
Load these additional skills only when the corresponding condition applies:
|
|
17
|
+
|
|
18
|
+
- **generating-tests-from-openapi** — the user provides an OpenAPI spec for the endpoint.
|
|
19
|
+
- **generating-tests-from-code** — the endpoint's source code exists in the current workspace.
|
|
20
|
+
- **generating-browser-tests** — the endpoint serves HTML and the user wants browser-level (Web Vitals) testing instead of protocol-level.
|
|
21
|
+
- **testing-resilience** — the user asks for fault injection or chaos testing against the endpoint.
|
|
22
|
+
- **operating-k6-in-ci-cd** — the user asks for a CI pipeline snippet along with the test.
|
|
23
|
+
|
|
24
|
+
## Step 2 — Probe the endpoint
|
|
25
|
+
|
|
26
|
+
- Use `bash` with `curl -s -o /dev/null -w "%{http_code} %{time_total}"` (or similar) to verify the endpoint is reachable and note the status, latency, content type, and auth requirements.
|
|
27
|
+
- If the endpoint requires authentication, determine the scheme (Bearer token, basic, cookie, API key) and ask the user to supply credentials via an environment variable or env file. **Never hardcode credentials in the generated script** — read them from `__ENV`.
|
|
28
|
+
- If the endpoint is unreachable, still generate the script against the given URL and note that verification was skipped.
|
|
29
|
+
|
|
30
|
+
## Step 3 — Confirm the test design
|
|
31
|
+
|
|
32
|
+
Ask the user (or infer sensibly and state the assumptions) for:
|
|
33
|
+
|
|
34
|
+
- **Test type** (`$3`, default `load`): map it to an executor and load profile using the *designing-test-scenarios* skill — smoke, load, stress, spike, soak, or breakpoint.
|
|
35
|
+
- **Performance targets**: p95/p99 latency and error-rate thresholds; use the skill's threshold guidance (e.g. `http_req_failed: ['rate<0.01']`, `http_req_duration: ['p(95)<500']`) if none are given.
|
|
36
|
+
- **Auth and test data**: environment variables needed, any request body for POST/PUT endpoints, and whether test data should be parameterized per the *generating-api-load-tests* skill.
|
|
37
|
+
|
|
38
|
+
## Step 4 — Generate the script
|
|
39
|
+
|
|
40
|
+
- Write the test to `${2:-tests/k6/endpoint-test.js}` using kebab-case file names, following the *generating-api-load-tests* skill patterns:
|
|
41
|
+
- All config via `__ENV` with sensible defaults (`BASE_URL`, `HEADERS`, `STAGES_*`).
|
|
42
|
+
- `export const options` with the chosen executor, stages, thresholds, and tags (e.g. `{ name: '$1' }`).
|
|
43
|
+
- A `default` function that hits the endpoint, applies `check`s for status and content, and `sleep`s with realistic think time.
|
|
44
|
+
- `setup()`/`teardown()` only if auth tokens or shared data require them.
|
|
45
|
+
- If an OpenAPI spec was provided, follow *generating-tests-from-openapi* for request construction and auth mapping; if the API source is in the workspace, follow *generating-tests-from-code* to mirror real routes and payloads.
|
|
46
|
+
- If browser testing applies, generate the browser variant per the *generating-browser-tests* skill (import from `k6/browser`, `export async function default(page)`, Web Vitals measurements).
|
|
47
|
+
- If CI is requested, add a pipeline snippet per *operating-k6-in-ci-cd* (GitHub Actions unless specified otherwise).
|
|
48
|
+
|
|
49
|
+
## Step 5 — Validate and analyze
|
|
50
|
+
|
|
51
|
+
- Run `k6 inspect <output-file>` to confirm the script parses and options are exported.
|
|
52
|
+
- With the user's permission (and only for endpoints you are authorized to test), run a short smoke validation, e.g. `k6 run --iterations 2 <output-file>`.
|
|
53
|
+
- If a run was executed, apply the *analyzing-test-results* skill to read the summary: threshold pass/fail, percentiles, and any bottleneck signals.
|
|
54
|
+
- Report the results and suggested next steps (e.g. tuning VUs, adding scenarios, running a soak).
|
|
55
|
+
|
|
56
|
+
## Rules
|
|
57
|
+
|
|
58
|
+
- Only target endpoints the user owns or has explicit authorization to load test.
|
|
59
|
+
- Never fire load at production endpoints without the user's explicit confirmation; default to a 2-iteration smoke validation only.
|
|
60
|
+
- Never echo, log, or persist credentials; use `__ENV` exclusively.
|
|
61
|
+
- Do not overwrite existing test files; pick a new file name if the target exists.
|
|
62
|
+
- Keep the script idiomatic k6 (init context, options export, checks, tags); avoid dead code and unused imports.
|
|
63
|
+
|
|
64
|
+
After generating, summarize:
|
|
65
|
+
- The endpoint, chosen test type, executor, and thresholds (and why).
|
|
66
|
+
- The output file path(s) and any env variables required to run it.
|
|
67
|
+
- The probe/validation results and analysis from any executed run.
|
|
68
|
+
- Which k6 skills were loaded and how each shaped the result.
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: Generate a k6 load test suite from a provided OpenAPI or Swagger specification
|
|
3
|
+
argument-hint: "<spec-path-or-url> [output-dir] [scope]"
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Generate a complete, runnable k6 load test suite from the OpenAPI/Swagger specification at `$1`, guided by the k6 skills in this package. Default output directory is `${2:-tests/k6/}` and default scope is `${3:-all}` (all suitable operations, or a tag/path filter like `Users` or `/users`).
|
|
7
|
+
|
|
8
|
+
## Step 1 — Load the k6 skills
|
|
9
|
+
|
|
10
|
+
Before writing any code, use `read` to load the SKILL.md files (follow their reference files when relevant):
|
|
11
|
+
|
|
12
|
+
- **generating-tests-from-openapi** — primary; drives the whole workflow (spec parsing, auth mapping, request body generation).
|
|
13
|
+
- **designing-test-scenarios** — always; for executors, load profiles, and thresholds per endpoint type.
|
|
14
|
+
- **generating-api-load-tests** — always; for idiomatic HTTP script patterns (headers, params, checks, data handling).
|
|
15
|
+
- **analyzing-test-results** — always; to interpret the validation run.
|
|
16
|
+
|
|
17
|
+
Load these additional skills only when the condition applies:
|
|
18
|
+
|
|
19
|
+
- **operating-k6-in-ci-cd** — the user asks for a CI pipeline snippet alongside the tests.
|
|
20
|
+
- **testing-resilience** — the user asks for fault injection during the load tests.
|
|
21
|
+
|
|
22
|
+
## Step 2 — Load and validate the spec
|
|
23
|
+
|
|
24
|
+
- If `$1` is a URL, download it with `curl -sSL`; if it is a local file, read it directly.
|
|
25
|
+
- Support both **OpenAPI 3.x** (`openapi:` key, `servers`, `components.securitySchemes`) and **Swagger 2.0** (`swagger: '2.0'`, `basePath`/`host`, `securityDefinitions`) — normalize Swagger 2.0 structures to their OpenAPI 3 equivalents while parsing.
|
|
26
|
+
- Support both YAML and JSON. If the file fails to parse, stop and report the exact error instead of guessing.
|
|
27
|
+
- Note the API title, version, base URL (`servers[0].url` or `host + basePath`), and whether a base URL is missing (then require it via `__ENV.BASE_URL` and say so).
|
|
28
|
+
|
|
29
|
+
## Step 3 — Parse the spec per the skill
|
|
30
|
+
|
|
31
|
+
Follow the *generating-tests-from-openapi* skill workflow and extract:
|
|
32
|
+
|
|
33
|
+
- All paths and operations (method, `operationId`, `tags`, parameters, request bodies, response codes).
|
|
34
|
+
- **Security analysis (prerequisite)**: global `security`, per-operation overrides, and public endpoints (`security: []`); map schemes per the skill's auth-mapping reference (OAuth2 flows, API key, Bearer/Basic, OpenID Connect).
|
|
35
|
+
- Request body schemas, resolving `$ref`s to `components`/`definitions`, and building example values from the skill's schema→example table (prefer `example`/`default` values from the spec when present).
|
|
36
|
+
|
|
37
|
+
## Step 4 — Confirm the scope
|
|
38
|
+
|
|
39
|
+
- Apply `$3` (default `all`): filter to the requested tag or path prefix if given.
|
|
40
|
+
- By default **exclude**: deprecated operations, destructive operations (`DELETE`, and state-destroying `PUT`/`POST` like reset endpoints), and health/doc endpoints unless trivial — but list everything excluded and let the user re-include any.
|
|
41
|
+
- If the API has many tags, propose one script per tag under the output directory; otherwise a single script organized with `group()`s by tag (per the skill's Step 5 pattern).
|
|
42
|
+
|
|
43
|
+
## Step 5 — Design the load profile
|
|
44
|
+
|
|
45
|
+
Use the *designing-test-scenarios* skill:
|
|
46
|
+
|
|
47
|
+
- Pick executors and load levels per endpoint characteristic (per the OpenAPI skill's Step 6): read-heavy GETs get higher rates, writes get lower rates, admin endpoints are excluded or isolated.
|
|
48
|
+
- Set sensible default thresholds (e.g. `http_req_failed: ['rate<0.01']`, `http_req_duration: ['p(95)<500']`) and let the user override with their SLAs.
|
|
49
|
+
- Confirm the target environment: default `BASE_URL` from the spec's `servers`, but **warn and require explicit confirmation before running against any production URL**.
|
|
50
|
+
|
|
51
|
+
## Step 6 — Generate the test suite
|
|
52
|
+
|
|
53
|
+
Following the *generating-tests-from-openapi* script structure (and *generating-api-load-tests* patterns):
|
|
54
|
+
|
|
55
|
+
- All config via `__ENV` with defaults (`BASE_URL`, auth credentials, scenario tuning); never hardcode secrets.
|
|
56
|
+
- `setup()` implementing the required auth flow from the security analysis (token fetch, API key injection) — pass tokens to the default function via the setup data argument.
|
|
57
|
+
- `export const options` with the designed scenarios and thresholds.
|
|
58
|
+
- Default function with one `group()` per API tag; each operation tagged `{ name: '<operationId>' }` and wrapped in `check`s for the documented success status code (200/201/etc. from the spec).
|
|
59
|
+
- **Chain dependent operations** where the spec implies it (e.g. POST `/users` → use the created `id` in GET `/users/{id}` → DELETE it in cleanup), so writes are self-cleaning and don't pollute the target.
|
|
60
|
+
- Request bodies built from the schema example values; override placeholders like `test@example.com` are fine for load tests.
|
|
61
|
+
- Realistic `sleep(1)` think time between operations.
|
|
62
|
+
- If CI was requested, add the pipeline snippet per *operating-k6-in-ci-cd*; if resilience was requested, follow *testing-resilience* for the fault-injection wrapper.
|
|
63
|
+
|
|
64
|
+
## Step 7 — Validate and analyze
|
|
65
|
+
|
|
66
|
+
- Run `k6 inspect` on each generated script to confirm they parse and options export correctly.
|
|
67
|
+
- With the user's explicit permission (non-production target preferred), run a short smoke validation, e.g. `k6 run --iterations 2 <script>` per script or on one representative script.
|
|
68
|
+
- If a run was executed, apply the *analyzing-test-results* skill to interpret threshold pass/fail, percentiles, and error rates.
|
|
69
|
+
- Report results and suggested next steps (tuning VUs, per-endpoint scenarios, soak runs).
|
|
70
|
+
|
|
71
|
+
## Rules
|
|
72
|
+
|
|
73
|
+
- Only target APIs the user owns or has explicit authorization to load test; never run against production without explicit confirmation.
|
|
74
|
+
- Never echo, log, or persist credentials; read auth secrets exclusively from `__ENV`.
|
|
75
|
+
- Do not overwrite existing test files; pick new file names if targets exist.
|
|
76
|
+
- Do not invent endpoints or fields that are not in the spec; if something is ambiguous (missing base URL, circular `$ref`, unresolved schema), ask or state the assumption explicitly.
|
|
77
|
+
- Keep generated scripts idiomatic k6 (init context, options export, groups, checks, tags); no dead code.
|
|
78
|
+
|
|
79
|
+
After generating, summarize:
|
|
80
|
+
- The spec (title, version, OpenAPI/Swagger format), base URL, and number of operations covered vs. excluded.
|
|
81
|
+
- The auth scheme detected and how it was implemented in `setup()`.
|
|
82
|
+
- The scenarios/thresholds designed per endpoint type and why.
|
|
83
|
+
- Output file path(s), required env variables, and how to run them.
|
|
84
|
+
- Validation/analysis results from any executed run.
|
|
85
|
+
- Which k6 skills were loaded and how each shaped the result.
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: Set up k6 and a skeleton load testing project in the current workspace
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
Set up k6 and a runnable skeleton load-testing project in the current workspace root. The goal is to end up with the k6 CLI available and a minimal `tests/` folder with working scripts covering a smoke test and a load test.
|
|
6
|
+
|
|
7
|
+
## Steps
|
|
8
|
+
|
|
9
|
+
1. **Inspect the workspace**.
|
|
10
|
+
- Use `bash` to list the current directory and check for `package.json`.
|
|
11
|
+
- Do not touch existing k6 scripts, options files, or CI pipelines that reference k6.
|
|
12
|
+
|
|
13
|
+
2. **Check k6 installation**.
|
|
14
|
+
- Run `k6 version`.
|
|
15
|
+
- If it prints a version (v0.5x or newer), skip installation.
|
|
16
|
+
|
|
17
|
+
3. **Install k6 if needed**. Pick the first method that works on the current OS:
|
|
18
|
+
- Windows: `winget install grafana.k6` (or `choco install k6`, or `scoop install k6`).
|
|
19
|
+
- macOS: `brew install k6`.
|
|
20
|
+
- Linux: follow the official apt/dnf instructions from https://grafana.com/docs/k6/latest/set-up/install/.
|
|
21
|
+
- Fallback for any OS: download a binary from the grafana/k6 GitHub releases and put it on `PATH`, or use Docker: `docker run --rm -i grafana/k6 run - < tests/smoke.js`.
|
|
22
|
+
- Re-run `k6 version` to confirm the install. If no method works, create the project files anyway and document the install commands in a `README.md`.
|
|
23
|
+
|
|
24
|
+
4. **Create the project structure**.
|
|
25
|
+
- Keep it minimal and idiomatic:
|
|
26
|
+
```
|
|
27
|
+
tests/
|
|
28
|
+
smoke.js
|
|
29
|
+
load.js
|
|
30
|
+
```
|
|
31
|
+
- Use plain JavaScript; k6 runs scripts natively with no build step.
|
|
32
|
+
|
|
33
|
+
5. **Create a smoke test**.
|
|
34
|
+
- Write `tests/smoke.js` with a minimal working script:
|
|
35
|
+
```js
|
|
36
|
+
import http from 'k6/http';
|
|
37
|
+
import { check, sleep } from 'k6';
|
|
38
|
+
|
|
39
|
+
const BASE_URL = __ENV.BASE_URL || 'https://quickpizza.grafana.com';
|
|
40
|
+
|
|
41
|
+
export const options = {
|
|
42
|
+
vus: 1,
|
|
43
|
+
iterations: 5,
|
|
44
|
+
thresholds: {
|
|
45
|
+
http_req_failed: ['rate<0.01'], // http errors should be less than 1%
|
|
46
|
+
http_req_duration: ['p(95)<500'], // 95% of requests under 500ms
|
|
47
|
+
},
|
|
48
|
+
};
|
|
49
|
+
|
|
50
|
+
export default function () {
|
|
51
|
+
const res = http.get(BASE_URL);
|
|
52
|
+
check(res, {
|
|
53
|
+
'status is 200': (r) => r.status === 200,
|
|
54
|
+
});
|
|
55
|
+
sleep(1);
|
|
56
|
+
}
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
6. **Create a load test**.
|
|
60
|
+
- Write `tests/load.js` demonstrating executor stages and thresholds:
|
|
61
|
+
```js
|
|
62
|
+
import http from 'k6/http';
|
|
63
|
+
import { check, sleep } from 'k6';
|
|
64
|
+
|
|
65
|
+
const BASE_URL = __ENV.BASE_URL || 'https://quickpizza.grafana.com';
|
|
66
|
+
|
|
67
|
+
export const options = {
|
|
68
|
+
scenarios: {
|
|
69
|
+
ramping_vus: {
|
|
70
|
+
executor: 'ramping-vus',
|
|
71
|
+
startVUs: 1,
|
|
72
|
+
stages: [
|
|
73
|
+
{ duration: '30s', target: 10 }, // ramp up
|
|
74
|
+
{ duration: '1m', target: 10 }, // stay at load
|
|
75
|
+
{ duration: '30s', target: 0 }, // ramp down
|
|
76
|
+
],
|
|
77
|
+
gracefulRampDown: '15s',
|
|
78
|
+
},
|
|
79
|
+
},
|
|
80
|
+
thresholds: {
|
|
81
|
+
http_req_failed: ['rate<0.01'],
|
|
82
|
+
http_req_duration: ['p(95)<500'],
|
|
83
|
+
},
|
|
84
|
+
};
|
|
85
|
+
|
|
86
|
+
export default function () {
|
|
87
|
+
const res = http.get(BASE_URL);
|
|
88
|
+
check(res, {
|
|
89
|
+
'status is 200': (r) => r.status === 200,
|
|
90
|
+
});
|
|
91
|
+
sleep(1);
|
|
92
|
+
}
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
7. **Add convenience scripts**.
|
|
96
|
+
- If `package.json` is missing, run `npm init -y` first.
|
|
97
|
+
- Use `edit` to add scripts to `package.json`:
|
|
98
|
+
```json
|
|
99
|
+
{
|
|
100
|
+
"k6:smoke": "k6 run tests/smoke.js",
|
|
101
|
+
"k6:load": "k6 run tests/load.js"
|
|
102
|
+
}
|
|
103
|
+
```
|
|
104
|
+
- Do not duplicate scripts that already exist.
|
|
105
|
+
- If the workspace has no Node.js at all, write a short `README.md` documenting the raw `k6 run tests/smoke.js` / `k6 run tests/load.js` commands instead.
|
|
106
|
+
|
|
107
|
+
8. **Verify the setup**.
|
|
108
|
+
- Run `k6 version` and report the version.
|
|
109
|
+
- Run `k6 inspect tests/smoke.js` to confirm the script parses and the options export is picked up.
|
|
110
|
+
- Optionally run `k6 run --iterations 2 tests/smoke.js` for a quick live check (it hits `quickpizza.grafana.com`, so skip if offline). Confirm the threshold summary passes.
|
|
111
|
+
- Report the final k6 version and the list of created files.
|
|
112
|
+
|
|
113
|
+
## Optional: TypeScript
|
|
114
|
+
|
|
115
|
+
Only if the user explicitly asks for TypeScript, add esbuild bundling (`k6` has no native TS runtime):
|
|
116
|
+
- `npm install -D esbuild typescript` and add a `build` script that bundles `src/**/*.ts` into `tests/`.
|
|
117
|
+
- Otherwise skip this step entirely; plain JavaScript is the default.
|
|
118
|
+
|
|
119
|
+
## Rules
|
|
120
|
+
|
|
121
|
+
- Do not overwrite existing user files unless they are missing or clearly incomplete.
|
|
122
|
+
- Keep the setup minimal and idiomatic; do not add CI pipelines, dashboards, or cloud integrations unless asked.
|
|
123
|
+
- Never hardcode secrets or tokens in the scripts; use `__ENV` and environment variables.
|
|
124
|
+
- Use `edit` for precise changes to `package.json`.
|
|
125
|
+
- Use `write` only for new files or complete rewrites of files you created.
|
|
126
|
+
|
|
127
|
+
After completing the setup, summarize:
|
|
128
|
+
- Which files were created or modified.
|
|
129
|
+
- The k6 version installed.
|
|
130
|
+
- Any verification results or errors encountered.
|
package/scripts/api-server.mjs
CHANGED
|
@@ -88,6 +88,29 @@ const PROMPT_REGISTRY = {
|
|
|
88
88
|
],
|
|
89
89
|
description: "Detect, diagnose, and auto-heal failed Playwright tests",
|
|
90
90
|
},
|
|
91
|
+
"setup-k6": {
|
|
92
|
+
file: "setup-k6.md",
|
|
93
|
+
params: [],
|
|
94
|
+
description: "Install k6 and create a runnable skeleton load-testing project in the workspace",
|
|
95
|
+
},
|
|
96
|
+
"generate-k6-test-from-endpoint": {
|
|
97
|
+
file: "generate-k6-test-from-endpoint.md",
|
|
98
|
+
params: [
|
|
99
|
+
{ name: "url", required: true, description: "Target endpoint URL to load test" },
|
|
100
|
+
{ name: "outputFile", required: false, description: "Output test file (default: tests/k6/endpoint-test.js)" },
|
|
101
|
+
{ name: "testType", required: false, description: "smoke, load, stress, spike, soak or breakpoint (default: load)" },
|
|
102
|
+
],
|
|
103
|
+
description: "Generate a k6 load test for a single endpoint using the k6 skills",
|
|
104
|
+
},
|
|
105
|
+
"generate-k6-test-from-openapi": {
|
|
106
|
+
file: "generate-k6-test-from-openapi.md",
|
|
107
|
+
params: [
|
|
108
|
+
{ name: "specUrl", required: true, description: "Path or URL of the OpenAPI/Swagger spec (.yaml/.yml/.json)" },
|
|
109
|
+
{ name: "outputDir", required: false, description: "Output directory (default: tests/k6/)" },
|
|
110
|
+
{ name: "scope", required: false, description: "Tag or path filter, e.g. Users or /users (default: all operations)" },
|
|
111
|
+
],
|
|
112
|
+
description: "Generate a k6 load test suite from an OpenAPI/Swagger specification",
|
|
113
|
+
},
|
|
91
114
|
};
|
|
92
115
|
|
|
93
116
|
// ── Server-level agent defaults ────────────────────────────────────────
|
|
@@ -0,0 +1,150 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: analyzing-test-results
|
|
3
|
+
description: Use when interpreting k6 test output, analyzing performance metrics, identifying bottlenecks, tuning thresholds, or optimizing k6 test scripts. Use when the user has k6 test results and needs to understand response times, error rates, percentiles (p95, p99), or performance degradation patterns.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Analyzing k6 Test Results
|
|
7
|
+
|
|
8
|
+
Interpret k6 test output, identify performance bottlenecks, and provide optimization recommendations.
|
|
9
|
+
|
|
10
|
+
## Analysis Workflow
|
|
11
|
+
|
|
12
|
+
1. **Review summary output** — Check overall pass/fail, iteration count, VU count
|
|
13
|
+
2. **Check thresholds** — Which passed/failed? By how much?
|
|
14
|
+
3. **Analyze response times** — Look at p50, p90, p95, p99 distribution
|
|
15
|
+
4. **Break down request duration** — Examine blocked, connecting, TLS, waiting, receiving
|
|
16
|
+
5. **Identify bottlenecks** — Map timing components to root causes
|
|
17
|
+
6. **Suggest optimizations** — Script improvements, threshold tuning, infrastructure changes
|
|
18
|
+
|
|
19
|
+
## Understanding k6 Output
|
|
20
|
+
|
|
21
|
+
### End-of-Test Summary
|
|
22
|
+
|
|
23
|
+
```
|
|
24
|
+
data_received..................: 148 kB 4.9 kB/s
|
|
25
|
+
data_sent......................: 17 kB 564 B/s
|
|
26
|
+
http_req_blocked...............: avg=23.38ms min=1µs med=5µs max=235.44ms p(90)=10µs p(95)=210.47ms
|
|
27
|
+
http_req_connecting............: avg=11.12ms min=0s med=0s max=117.4ms p(90)=0s p(95)=99.38ms
|
|
28
|
+
✓ http_req_duration..............: avg=131.01ms min=116.4ms med=127.6ms max=196.38ms p(90)=146.21ms p(95)=176.91ms
|
|
29
|
+
http_req_failed................: 0.00% ✓ 0 ✗ 200
|
|
30
|
+
http_req_receiving.............: avg=5.11ms min=26µs med=4.88ms max=14.82ms p(90)=9.31ms p(95)=10.82ms
|
|
31
|
+
http_req_sending...............: avg=27µs min=4µs med=23µs max=240µs p(90)=42µs p(95)=66µs
|
|
32
|
+
http_req_tls_handshaking.......: avg=11.7ms min=0s med=0s max=116.76ms p(90)=0s p(95)=105.06ms
|
|
33
|
+
http_req_waiting...............: avg=125.86ms min=116.27ms med=122.6ms max=185.5ms p(90)=137.21ms p(95)=170.53ms
|
|
34
|
+
http_reqs......................: 200 6.63/s
|
|
35
|
+
iteration_duration.............: avg=1.16s min=1.12s med=1.13s max=1.43s p(90)=1.15s p(95)=1.41s
|
|
36
|
+
iterations.....................: 200 6.63/s
|
|
37
|
+
vus............................: 10 min=10 max=10
|
|
38
|
+
vus_max........................: 10 min=10 max=10
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
### Reading the Output
|
|
42
|
+
|
|
43
|
+
- **✓** = threshold passed, **✗** = threshold failed
|
|
44
|
+
- Each metric shows: `avg`, `min`, `med` (median), `max`, `p(90)`, `p(95)`
|
|
45
|
+
- `http_reqs` rate = requests per second (RPS)
|
|
46
|
+
|
|
47
|
+
## HTTP Request Duration Breakdown
|
|
48
|
+
|
|
49
|
+
The total `http_req_duration` is composed of:
|
|
50
|
+
|
|
51
|
+
```
|
|
52
|
+
blocked → connecting → tls_handshaking → sending → waiting → receiving
|
|
53
|
+
↑
|
|
54
|
+
(TTFB / server time)
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
| Component | What It Measures | Typical Cause of High Values |
|
|
58
|
+
|-----------|-----------------|------------------------------|
|
|
59
|
+
| `http_req_blocked` | Time in queue before request starts | Connection pool exhaustion, DNS lookup |
|
|
60
|
+
| `http_req_connecting` | TCP connection time | Network latency, server far away |
|
|
61
|
+
| `http_req_tls_handshaking` | TLS/SSL handshake | Missing keep-alive, new connections per request |
|
|
62
|
+
| `http_req_sending` | Time to send request body | Large request body, slow upload |
|
|
63
|
+
| `http_req_waiting` | Server processing time (TTFB) | Slow backend, DB queries, server overload |
|
|
64
|
+
| `http_req_receiving` | Time to download response | Large response body, slow network |
|
|
65
|
+
| `http_req_duration` | sending + waiting + receiving | Overall request time |
|
|
66
|
+
|
|
67
|
+
See [reference/bottleneck-patterns.md](reference/bottleneck-patterns.md) for detailed bottleneck analysis patterns.
|
|
68
|
+
|
|
69
|
+
## Key Metrics Quick Reference
|
|
70
|
+
|
|
71
|
+
See [reference/metrics.md](reference/metrics.md) for the complete k6 metrics reference.
|
|
72
|
+
|
|
73
|
+
### Most Important Metrics to Monitor
|
|
74
|
+
|
|
75
|
+
| Metric | Type | What to Watch |
|
|
76
|
+
|--------|------|---------------|
|
|
77
|
+
| `http_req_duration` | Trend | p95 and p99 — are they within SLA? |
|
|
78
|
+
| `http_req_failed` | Rate | Error rate — should be < 1% for normal load |
|
|
79
|
+
| `http_reqs` | Counter | Request rate (RPS) — is target throughput achieved? |
|
|
80
|
+
| `iteration_duration` | Trend | Full iteration time including sleeps |
|
|
81
|
+
| `checks` | Rate | Assertion pass rate — should be > 99% |
|
|
82
|
+
| `vus` | Gauge | Active VU count — did it reach target? |
|
|
83
|
+
|
|
84
|
+
## Percentile Interpretation
|
|
85
|
+
|
|
86
|
+
| Percentile | Meaning |
|
|
87
|
+
|------------|---------|
|
|
88
|
+
| `p(50)` / `med` | Half of requests are faster. Typical user experience. |
|
|
89
|
+
| `p(90)` | 90% of requests are faster. Most users' experience. |
|
|
90
|
+
| `p(95)` | 95% of requests are faster. Common SLA target. |
|
|
91
|
+
| `p(99)` | 99% of requests are faster. Worst-case for most users. |
|
|
92
|
+
| `p(99.9)` | 1 in 1000 requests slower. Tail latency. |
|
|
93
|
+
|
|
94
|
+
**Rule of thumb:** If p95 is 2x+ the median, there's likely a bottleneck affecting some requests.
|
|
95
|
+
|
|
96
|
+
## handleSummary() Customization
|
|
97
|
+
|
|
98
|
+
Override the default summary output:
|
|
99
|
+
|
|
100
|
+
```javascript
|
|
101
|
+
import { textSummary } from 'https://jslib.k6.io/k6-summary/0.1.0/index.js';
|
|
102
|
+
|
|
103
|
+
export function handleSummary(data) {
|
|
104
|
+
return {
|
|
105
|
+
'summary.json': JSON.stringify(data, null, 2),
|
|
106
|
+
stdout: textSummary(data, { indent: ' ', enableColors: true }),
|
|
107
|
+
};
|
|
108
|
+
}
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
Available output targets:
|
|
112
|
+
- `stdout` — Console output
|
|
113
|
+
- `stderr` — Error output
|
|
114
|
+
- `'filename.json'` — Write to file
|
|
115
|
+
- `'filename.html'` — Write HTML report
|
|
116
|
+
|
|
117
|
+
## Results Output Options
|
|
118
|
+
|
|
119
|
+
### JSON Lines (Real-Time)
|
|
120
|
+
|
|
121
|
+
```bash
|
|
122
|
+
k6 run --out json=results.json script.js
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
### CSV Output
|
|
126
|
+
|
|
127
|
+
```bash
|
|
128
|
+
k6 run --out csv=results.csv script.js
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
### Web Dashboard (Built-in)
|
|
132
|
+
|
|
133
|
+
```bash
|
|
134
|
+
K6_WEB_DASHBOARD=true k6 run script.js
|
|
135
|
+
# Dashboard at http://localhost:5665
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
### Grafana Cloud k6
|
|
139
|
+
|
|
140
|
+
```bash
|
|
141
|
+
k6 cloud run --local-execution script.js
|
|
142
|
+
# Or set K6_CLOUD_TOKEN environment variable
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
See [reference/visualization.md](reference/visualization.md) for detailed visualization options.
|
|
146
|
+
|
|
147
|
+
## Related Skills
|
|
148
|
+
|
|
149
|
+
- For designing test scenarios and thresholds: `/k6:designing-test-scenarios`
|
|
150
|
+
- For generating test scripts: `/k6:generating-api-load-tests`
|
|
@@ -0,0 +1,130 @@
|
|
|
1
|
+
# k6 Bottleneck Patterns Reference
|
|
2
|
+
|
|
3
|
+
## Diagnostic Matrix
|
|
4
|
+
|
|
5
|
+
| Symptom | Primary Metric | Root Cause | Action |
|
|
6
|
+
|---------|---------------|------------|--------|
|
|
7
|
+
| High blocked time | `http_req_blocked` > 100ms | Connection pool exhaustion, DNS issues | Increase connection pool, use HTTP keep-alive |
|
|
8
|
+
| High connecting time | `http_req_connecting` > 50ms | Server distance, TCP overhead | Use closer servers, CDN, connection reuse |
|
|
9
|
+
| High TLS time | `http_req_tls_handshaking` > 100ms | New TLS connection per request | Enable keep-alive, TLS session resumption |
|
|
10
|
+
| High waiting time | `http_req_waiting` > 500ms | Slow server processing | Optimize backend, DB queries, caching |
|
|
11
|
+
| High receiving time | `http_req_receiving` > 100ms | Large response body | Compress responses, paginate, reduce payload |
|
|
12
|
+
| High error rate | `http_req_failed` > 1% | Server overload, bugs | Check error codes, server logs, capacity |
|
|
13
|
+
| Low throughput | `http_reqs` rate declining | Resource saturation | Scale infrastructure, optimize code |
|
|
14
|
+
| Dropped iterations | `dropped_iterations` > 0 | Not enough VUs for target rate | Increase `preAllocatedVUs` or `maxVUs` |
|
|
15
|
+
|
|
16
|
+
## Pattern 1: Connection Pool Exhaustion
|
|
17
|
+
|
|
18
|
+
**Symptoms:**
|
|
19
|
+
- `http_req_blocked` p95 > 100ms (often > 1s)
|
|
20
|
+
- `http_req_blocked` increases over time
|
|
21
|
+
- `http_req_connecting` stays low
|
|
22
|
+
|
|
23
|
+
**Diagnosis:** k6 cannot open new connections fast enough. Requests queue waiting for available connections.
|
|
24
|
+
|
|
25
|
+
**Solutions:**
|
|
26
|
+
- Check target server connection limits
|
|
27
|
+
- Reduce concurrent VU count
|
|
28
|
+
- Add think time (`sleep()`) between requests
|
|
29
|
+
- Check if server supports HTTP/2 (multiplexing)
|
|
30
|
+
|
|
31
|
+
## Pattern 2: TLS Overhead
|
|
32
|
+
|
|
33
|
+
**Symptoms:**
|
|
34
|
+
- `http_req_tls_handshaking` consistently > 50ms
|
|
35
|
+
- First requests have much higher duration than subsequent ones
|
|
36
|
+
- `http_req_blocked` correlates with TLS timing
|
|
37
|
+
|
|
38
|
+
**Diagnosis:** New TLS connections being established for each request instead of reusing.
|
|
39
|
+
|
|
40
|
+
**Solutions:**
|
|
41
|
+
- k6 reuses connections by default within an iteration
|
|
42
|
+
- Ensure server supports TLS session resumption
|
|
43
|
+
- Check if `noConnectionReuse` option is accidentally set
|
|
44
|
+
- Consider HTTP/2 for multiplexing over single connection
|
|
45
|
+
|
|
46
|
+
## Pattern 3: Server-Side Bottleneck
|
|
47
|
+
|
|
48
|
+
**Symptoms:**
|
|
49
|
+
- `http_req_waiting` (TTFB) increases as VUs increase
|
|
50
|
+
- `http_req_waiting` is the dominant component of duration
|
|
51
|
+
- `http_req_sending` and `http_req_receiving` remain low
|
|
52
|
+
|
|
53
|
+
**Diagnosis:** Server is struggling under load. Could be CPU, memory, database, or application bottleneck.
|
|
54
|
+
|
|
55
|
+
**Solutions:**
|
|
56
|
+
- Profile server-side code
|
|
57
|
+
- Optimize database queries (add indexes, caching)
|
|
58
|
+
- Scale horizontally (add servers)
|
|
59
|
+
- Implement caching (Redis, CDN)
|
|
60
|
+
- Check for thread/connection pool limits
|
|
61
|
+
|
|
62
|
+
## Pattern 4: Response Size Bottleneck
|
|
63
|
+
|
|
64
|
+
**Symptoms:**
|
|
65
|
+
- `http_req_receiving` is high relative to total duration
|
|
66
|
+
- `data_received` is very high
|
|
67
|
+
- Throughput drops as response sizes increase
|
|
68
|
+
|
|
69
|
+
**Diagnosis:** Large response payloads consuming bandwidth and processing time.
|
|
70
|
+
|
|
71
|
+
**Solutions:**
|
|
72
|
+
- Enable response compression (gzip, brotli)
|
|
73
|
+
- Implement pagination for list endpoints
|
|
74
|
+
- Return only necessary fields (field selection)
|
|
75
|
+
- Use `discardResponseBodies: true` in k6 options if body isn't needed
|
|
76
|
+
|
|
77
|
+
## Pattern 5: Gradual Degradation (Soak Test)
|
|
78
|
+
|
|
79
|
+
**Symptoms:**
|
|
80
|
+
- Metrics gradually worsen over time
|
|
81
|
+
- `http_req_duration` slowly increases during soak test
|
|
82
|
+
- Error rate slowly creeps up
|
|
83
|
+
|
|
84
|
+
**Diagnosis:** Memory leak, connection leak, or resource exhaustion in server.
|
|
85
|
+
|
|
86
|
+
**Solutions:**
|
|
87
|
+
- Monitor server-side memory usage
|
|
88
|
+
- Check for connection/handle leaks
|
|
89
|
+
- Verify database connection pool sizing
|
|
90
|
+
- Check log file growth
|
|
91
|
+
- Monitor disk space
|
|
92
|
+
|
|
93
|
+
## Pattern 6: Sudden Failure (Breakpoint)
|
|
94
|
+
|
|
95
|
+
**Symptoms:**
|
|
96
|
+
- Metrics suddenly spike at specific VU count or RPS
|
|
97
|
+
- Error rate jumps from ~0% to high percentage
|
|
98
|
+
- `http_req_duration` increases dramatically
|
|
99
|
+
|
|
100
|
+
**Diagnosis:** System has hit a hard limit (connection limit, thread pool, memory).
|
|
101
|
+
|
|
102
|
+
**Solutions:**
|
|
103
|
+
- Identify the exact threshold where failure begins
|
|
104
|
+
- Check server error logs at that timestamp
|
|
105
|
+
- Common limits: max connections, max threads, memory, file descriptors
|
|
106
|
+
- Tune the specific limit or scale infrastructure
|
|
107
|
+
|
|
108
|
+
## Correlation Between Metrics
|
|
109
|
+
|
|
110
|
+
**Healthy test:**
|
|
111
|
+
- `http_req_duration` stable across all percentiles
|
|
112
|
+
- `http_req_failed` near 0%
|
|
113
|
+
- `http_reqs` rate matches expected throughput
|
|
114
|
+
- `dropped_iterations` = 0
|
|
115
|
+
|
|
116
|
+
**Signs of trouble:**
|
|
117
|
+
- p99 >> p50 (high variance = inconsistent performance)
|
|
118
|
+
- `http_req_failed` increasing over time
|
|
119
|
+
- `http_reqs` rate declining while VUs are constant
|
|
120
|
+
- `http_req_blocked` growing (connection issues)
|
|
121
|
+
- `dropped_iterations` increasing (can't keep up with arrival rate)
|
|
122
|
+
|
|
123
|
+
## Performance Ratio Guidelines
|
|
124
|
+
|
|
125
|
+
| Ratio | Healthy | Warning | Critical |
|
|
126
|
+
|-------|---------|---------|----------|
|
|
127
|
+
| p95 / p50 | < 2x | 2-5x | > 5x |
|
|
128
|
+
| p99 / p50 | < 3x | 3-10x | > 10x |
|
|
129
|
+
| Error rate | < 0.1% | 0.1-1% | > 1% |
|
|
130
|
+
| TTFB / Duration | < 80% | 80-95% | > 95% |
|