mmt-testlight 0.4.3 → 0.4.5

package/dist/guides/generate-suite.md

@@ -0,0 +1,167 @@
+This file tells the AI how to generate **`type: suite`** `.mmt` files.
+
+Always follow these rules:
+- Output must be valid YAML.
+- The first non-comment line must be `type: suite`.
+- Suites should primarily **list other mmt files** (tests, APIs, or other suites) to be executed.
+
+---
+
+## Schema (mental model for the AI)
+
+These fields match `SuiteData` in `core/src/SuiteData.ts` and the public docs in `docs/files/suite/index.md`.
+
+Top-level keys and types:
+
+```yaml
+type: suite                      # REQUIRED, must be exactly "suite"
+
+title: string                    # optional, human-readable suite name
+tags:                            # optional, for grouping
+  - string
+
+description: string              # optional, short explanation
+
+servers:                         # optional, root-only: mock servers started before items
+  - path/to/server.mmt           # type: server files; kept running for the entire suite
+
+items:                           # REQUIRED, array of files to run
+  - path/to/file1.mmt
+  - path/to/file2.mmt
+  - then
+  - path/to/file3.mmt
+
+environment:                     # optional, root-only: configure env for suite runs
+  preset?: string                # preset name from multimeter.mmt (or from file)
+  file?: string                  # path to an env file to load
+  variables?:                    # inline key-value environment variables
+    <name>: <value>
+
+export:                          # optional, root-only: generate reports after completion
+  - path/to/report.xml           # supported: .xml (JUnit), .html, .md, .mmt
+```
+
+> **Root-only fields**: `servers`, `environment`, and `export` only take effect when the suite is run directly. When imported by another suite, these fields are ignored.
+
+> **Legacy alias**: `tests` is still accepted as an alias for `items`, but new suites should use `items`.
+
+---
+
+## Execution Flow (`items` array)
+
+The `items` array defines the execution flow.
+- All files listed between `then` separators (or before the first one) are run in parallel.
+- The groups of files separated by `then` are run sequentially.
+- **Server files** (`type: server`) can be included — they start before items in the same stage and stop when the suite completes.
+
+Example: `[a, b, then, c]` will run `a` and `b` in parallel, and once both are finished, it will run `c`.
+
+---
+
+## Common patterns the AI should generate
+
+### 1. Simple suite running all items in parallel
+
+User asks: "Create a suite to run smoke tests `test1.mmt` and `test2.mmt`."
+
+```yaml
+type: suite
+title: Smoke Tests
+tags: [smoke]
+items:
+  - ./tests/test1.mmt
+  - ./tests/test2.mmt
+```
+
+### 2. Sequential execution of items
+
+User asks: "Create a suite that first runs `login.mmt`, and then `get_user.mmt`."
+
+```yaml
+type: suite
+title: Login and Get User
+tags: [smoke, auth]
+items:
+  - ./tests/login.mmt
+  - then
+  - ./tests/get_user.mmt
+```
+
+### 3. Mixed parallel and sequential execution
+
+User asks: "Create a suite that runs `test1.mmt` and `test2.mmt` in parallel, and after they are done, runs `test3.mmt`."
+
+```yaml
+type: suite
+title: Mixed Execution Suite
+tags: [regression]
+items:
+  - ./tests/test1.mmt
+  - ./tests/test2.mmt
+  - then
+  - ./tests/test3.mmt
+```
+
+### 4. Suite with mock servers
+
+User asks: "Create a suite that starts a mock server before running integration tests."
+
+```yaml
+type: suite
+title: Integration with Mock Server
+tags: [integration]
+items:
+  - ./mocks/user-service.mmt    # type: server — starts first
+  - then
+  - ./tests/user_crud.mmt
+  - ./tests/user_auth.mmt
+```
+
+Servers start before items in the same stage and stop when the suite finishes.
+
+### 5. Suite with top-level servers and environment
+
+User asks: "Create an integration suite with a mock server, using the staging preset."
+
+```yaml
+type: suite
+title: Integration Suite
+tags: [integration]
+servers:
+  - ./mocks/user-service.mmt
+  - ./mocks/auth-service.mmt
+environment:
+  preset: staging
+items:
+  - ./tests/login.mmt
+  - ./tests/profile.mmt
+```
+
+Using the `servers` field is recommended over placing servers in `items` — it keeps them running for the entire suite and avoids ordering issues.
+
+### 6. Suite with report exports
+
+User asks: "Create a CI suite that exports JUnit XML and HTML reports."
+
+```yaml
+type: suite
+title: CI Suite
+export:
+  - ./reports/results.xml
+  - ./reports/results.html
+items:
+  - ./tests/login.mmt
+  - ./tests/profile.mmt
+```
+
+---
+
+## Style rules for the AI
+
+- Use 2-space indentation.
+- Prefer clear and descriptive titles.
+- Always include a `title`.
+- Use `tags` to categorize suites where appropriate.
+- Ensure file paths in the `items` array are plausible.
+
+When unsure, generate a **minimal valid suite** that includes the requested files.

package/dist/guides/generate-test-skill.md

@@ -0,0 +1,60 @@
+# Auto Test Generation Skill
+
+Use this skill when the user wants a Multimeter test generated from an API file, an OpenAPI/Postman spec, or a natural-language scenario.
+
+## Goal
+
+Produce a valid Multimeter `type: test` YAML file that is deterministic, minimal, and easy to validate — with **low token cost**.
+
+## Required workflow (API → test)
+
+1. Optional few-shot: `list_examples` → mirror **`goldenSmoke`** (`examples/ai/golden_smoke/`).
+2. Identify the target API `.mmt` path (`discover_api` / `api_card` only if needed).
+3. Call MCP **`scaffold_test({ workspaceRoot, apiPath })`** — **required**. Do not invent a blank test.
+   - Offline: `testlight scaffold test --from <api.mmt>`
+4. Write the returned YAML to `suggestedPath` (or the user path).
+5. Apply **only minimal** edits (title, expects, inputs). Prefer smoke unless asked for more.
+6. **`validate` until valid, then `format`** — both required before finishing.
+7. Stop — no polish rewrite.
+8. If the user wants stronger asserts after a run: **`suggest_assertions` → patch → validate → format**.
+
+## Modify discipline
+
+- **Patch only.** Never replace the whole file unless the user explicitly says rewrite/regenerate.
+- Prefer surgical edits (one step, one expect, one input).
+
+## Rules
+
+- Output only valid YAML for the generated test.
+- Start the file with `type: test`.
+- Never add YAML comments (`#`).
+- Never web-search Multimeter syntax.
+- Use tokens such as `e:`, `i:`, `r:`, and `c:` when appropriate.
+- If the source is ambiguous, ask a short clarifying question before generating.
+
+## Golden smoke shape (mirror this)
+
+See `docs/AI/golden-smoke.md` / `list_examples.goldenSmoke`. Scaffold already produces:
+
+```yaml
+type: test
+title: <API title> smoke test
+tags: [smoke, ...]
+import:
+  <alias>: <relative-api-path>
+inputs:
+  <name>: i:<name>
+steps:
+  - call: <alias>
+    id: <step_id>
+    inputs:
+      <name>: i:<name>
+    expect:
+      status: 200
+```
+
+## When to use this skill
+
+- “Generate a test for this API.”
+- “Create a smoke test from this OpenAPI spec.”
+- “Turn this endpoint into a Multimeter test.”

package/dist/guides/generate-test.md

@@ -0,0 +1,335 @@
+This file tells the AI how to generate **`type: test`** `.mmt` files.
+
+**Prefer MCP `scaffold_test` (or `testlight scaffold test --from`) for new tests from an API.** Use this guide for fields and step types beyond the scaffold baseline — not as an excuse to invent a full file from scratch.
+
+Always follow these rules:
+- Output must be valid YAML.
+- The first non-comment line must be `type: test`.
+- Tests should primarily **call APIs or other tests**, assert results, and optionally loop or branch.
+
+---
+
+## Schema (mental model for the AI)
+
+These fields match `TestData` in `core/src/TestData.ts` and the public docs in `docs/files/test/index.md`.
+
+Top-level keys and types:
+
+```yaml
+type: test                      # REQUIRED, must be exactly "test"
+
+title: string                   # REQUIRED, human-readable test name
+tags:                           # REQUIRED (can be empty array) for grouping
+  - string
+
+description: string             # optional, short explanation
+
+import:                         # alias -> file path (apis, tests, CSV)
+  <alias>: path/to/file.mmt
+
+inputs:                         # input variables with default values
+  <name>: string                # or number/boolean/null (JSON-compatible)
+
+outputs:                        # top-level outputs for this test
+  <name>: string                # expression or literal
+
+cache: 5m                       # optional; when this test is imported+called again
+                                # in the same root run with same title+inputs,
+                                # reuse outputs (skip callee body). Duration like
+                                # repeat/delay, epoch number, or datetime with ":"
+
+steps:                          # linear flow of steps (no stages)
+  - <step>
+
+stages:                         # parallel/grouped flows
+  - id: string
+    title?: string
+    condition?: string          # optional condition string
+    after?: string | string[]
+    steps:                      # steps inside this stage
+      - <step>
+```
+
+`steps` and `stages` are mutually exclusive in typical usage — use **one** or the other unless you have a very deliberate reason.
+
+---
+
+## Step types
+
+All steps are objects with a `type` (implicit from the key). They map to `TestFlowStep` in `TestData.ts`.
+
+Supported step forms:
+
+```yaml
+# 1) call: invoke another test or an API (imported by alias)
+- call: <alias>
+  id: <stepId>                  # REQUIRED, used to reference outputs later
+  title?: string                # optional human-readable label
+  inputs?:                      # passed as JSON object
+    <name>: <value>
+  expect?:                      # inline assertions (shorthand for assert)
+    <field>: <value>
+  debug?:                       # inline debug inspections (same syntax as expect, never fails)
+    <field>: <value>
+  report?: all | fails | none   # controls reporting level
+
+# 2) run: start an imported mock server (type: server file)
+- run: <alias>                  # alias of an imported server file
+
+# 3) check: soft assertion (logs failure, continues)
+- check: <comparison>
+
+# 4) assert: hard assertion (stops on failure)
+- assert: <comparison>
+
+# 5) if: conditional block (optional else)
+- if: <comparison>
+  steps:
+    - <step>
+  else:
+    - <step>
+
+# 6) repeat: repeat a block N times or by string
+- repeat: <number | string>
+  steps:
+    - <step>
+
+# 7) for: loop header (JS-style or shorthand)
+- for: <header or expression>
+  steps:
+    - <step>
+
+# 8) js: inline JavaScript
+- js: |
+    // javascript code
+
+# 9) print: log a message
+- print: "text"
+
+# 10) delay: sleep
+- delay: <number | string>      # e.g. 200, "2s", "1m"
+
+# 11) set: assign/update variables
+- set:
+    <name>: <value>
+
+# 12) var / const / let: JS-style declarations
+- var:
+    <name>: <value>
+- const:
+    <name>: <value>
+- let:
+    <name>: <value>
+
+# 13) data: bind CSV alias (from import)
+- data: <alias>
+
+# 14) setenv: promote values into environment variables
+- setenv:
+    <env_name>: <value>       # e.g. token: ${loginStep.token}
+```
+
+### Additional `call` fields
+
+Beyond the basic `call`, `id`, and `inputs`, a call step also supports:
+
+```yaml
+- call: <alias>
+  id: <stepId>
+  title?: string              # optional human-readable label for logs/reports
+  inputs?: { ... }
+  expect?:                    # inline assertions on the call result
+    <field>: <value>          # e.g. status: 200, body.name: "John"
+  debug?:                     # inline debug inspections (same syntax as expect, never fails)
+    <field>: <value>          # e.g. status: 200, body: ${someVar}
+  report?: all | fails | none # controls pass/fail reporting level
+```
+
+`expect` is a shorthand for common assertions — each key is a dotted path into the response, and the value is compared with `==` by default. You can prefix with an operator (e.g. `>= 1`, `!= null`).
+
+`debug` uses the same syntax as `expect` but never fails the test — it always logs results with a debug icon for troubleshooting. Debug entries are excluded from exported reports.
+
+---
+
+`<comparison>` is a string expression using operators from `opsList` in `TestData.ts`:
+
+- `<`, `>`, `<=`, `>=`, `==`, `!=`
+- `=@` (is in: left is contained in right), `!@` (is not in)
+- `=C` (contains: left contains right), `!C` (does not contain)
+- `=^` (starts with), `!^` (not starts with)
+- `=$` (ends with), `!$` (not ends with)
+- `=*` (regex match), `!*` (regex not match)
+- `=~` (equal as string / type-unsafe), `!~` (not equal as string) — for XML/text string outputs vs bare YAML booleans/numbers
+- `=#` (string/number character length equals), `!#` (not equal)
+- `>N%`(fuzzy match at least N% similar), `<N%` (fuzzy match less than N%). Any whole percent from 0 to 100 can be used, for example `>80%`. In the visual UI these appear as `>%` and `<%` with a separate percentage selector.
+
+Example comparisons:
+
+```yaml
+- assert: ${login.status} == 200
+- check: ${profile.email} =* /@example.com$/
+- assert: ${response.body.total} >= 1
+```
+
+---
+
+## Naming and references
+
+- Every `call` step **must** have a unique `id` within the test.
+- Use that `id` to reference its outputs in later steps (the exact JS shape comes from the imported file, but following patterns from the docs is enough):
+
+```yaml
+- call: login
+  id: loginStep
+  inputs:
+    username: i:username
+    password: i:password
+
+- assert: ${loginStep.status} == 200
+```
+
+- Use `set` to place values into `outputs` if you want the test itself to expose results:
+
+```yaml
+- set:
+    outputs.token: ${loginStep.token}
+```
+
+---
+
+## Tokens the AI can use in tests
+
+You can use the same token syntaxes as for APIs:
+
+- Environment `variables:` `e:api_url`, `e:auth_token`, etc., or `<<e:api_url>>`
+- Test `inputs:` `i:user_id` or `<<i:user_id>>`
+- Random values: `r:name` or `<<r:name>>`
+- Current/time values: `c:name` or `<<c:name>>`
+
+Guidelines:
+- Use bare tokens when they are the **entire value**.
+- Use `<< >>` when embedding in text.
+
+Example:
+
+```yaml
+inputs:
+  username: user@example.com
+  password: string
+
+steps:
+  - call: login
+    id: loginStep
+    inputs:
+      username: i:username
+      password: i:password
+  - assert: ${loginStep.status} == 200
+  - set:
+      outputs.token: ${loginStep.token}
+```
+
+---
+
+## Common patterns the AI should generate
+
+### 1. Simple smoke test for a single API
+
+User asks: "Create a test that logs in and checks status 200. The API alias is `login`."
+
+```yaml
+type: test
+title: Login smoke test
+tags: [smoke, auth]
+inputs:
+  username: user@example.com
+  password: string
+steps:
+  - call: login
+    id: loginStep
+    inputs:
+      username: i:username
+      password: i:password
+  - assert: ${loginStep.status} == 200
+```
+
+### 2. Chained calls: login then get profile
+
+```yaml
+type: test
+title: Login and get profile
+tags: [smoke, auth, profile]
+import:
+  login: ./login.mmt
+  get_profile: ./get_profile.mmt
+inputs:
+  username: user@example.com
+  password: string
+steps:
+  - call: login
+    id: loginStep
+    inputs:
+      username: i:username
+      password: i:password
+  - assert: ${loginStep.status} == 200
+
+  - call: get_profile
+    id: profileStep
+    inputs:
+      token: ${loginStep.token}
+  - assert: ${profileStep.status} == 200
+```
+
+### 3. Data-driven loop over CSV
+
+```yaml
+type: test
+title: Login for multiple users
+tags: [load, auth]
+import:
+  users: ./users.csv
+  login: ./login.mmt
+
+steps:
+  - data: users
+  - for: const user of users
+    steps:
+      - call: login
+        id: loginStep
+        inputs:
+          username: ${user.username}
+          password: ${user.password}
+      - check: ${loginStep.status} == 200
+```
+
+### 4. Conditional flows
+
+```yaml
+type: test
+title: Conditional retry on failure
+tags: [resilience]
+import:
+  login: ./login.mmt
+
+steps:
+  - call: login
+    id: login1
+  - if: ${login1.status} != 200
+    steps:
+      - print: "Retrying login"
+      - call: login
+        id: login2
+      - assert: ${login2.status} == 200
+    else:
+      - print: "Login succeeded on first attempt"
+```
+
+---
+
+## Style rules for the AI
+
+- Use 2-space indentation.
+- Prefer small, readable tests focused on a single behavior.
+- Prefer `steps` for simple linear flows; only use `stages` when explicit parallelism is needed.
+- Always include `title` and `tags` (even if tags is a small list like `[smoke]`).
+- Avoid adding `js` steps unless the user needs custom logic that can’t be expressed with other constructs. YAML comments (`#`) are fine — Format Document preserves them. Prefer `title` / `description` for structured documentation.
+When unsure, generate a **minimal valid test** that clearly calls the described APIs and asserts the most important property (typically HTTP status or a key field in the response).

package/dist/guides/generate.md

@@ -0,0 +1,60 @@
+This file is the **single entry point** when the AI needs to **generate or modify** any `.mmt` file.
+
+**Copilot / MCP:** Before editing YAML, call `read_documentation` for the file type, then `validate` after every edit. See `AI/agent-workflow.md`.
+
+The AI must:
+- Always output valid **YAML**.
+- Always put `type` as the **first key** in the file.
+- Choose the correct `type` based on the user request (rules below).
+- Use the exact structures outlined and never deviate them.
+- Use the fence language identifier yaml (i.e. ```yaml).
+- Do not use stray backticks inside YAML.
+- YAML comments (`#`) are preserved by Format Document. Prefer `description` / `title` for structured docs that also survive UI edits.
+
+The only file types are:
+- `type: api` – describe a single HTTP/WebSocket endpoint.
+- `type: test` – describe a test flow that calls APIs/tests.
+- `type: env` – define environment variables and presets.
+- `type: doc` – describe API documentation over a set of `.mmt` files.
+- `type: suite` – group and run multiple tests, APIs, or other suites.
+- `type: loadtest` – run one `type: test` file with load configuration.
+- `type: server` – define a mock server with endpoints, matching, and responses.
+- `type: report` – structured test results generated by runs (viewable in the editor; usually not hand-authored).
+
+When the user asks you to **create or change** a `.mmt` file, follow this order:
+
+1. **Decide the type**
+	- If the user talks about an endpoint, URL, method, headers, body, or examples → use `type: api`.
+	- If the user talks about scenarios, flows, assertions, retries, loops, or "tests" → use `type: test`.
+	- If the user talks about base URLs, credentials, modes, feature flags, presets → use `type: env`.
+	- If the user talks about documentation, catalogs, grouping APIs → use `type: doc`.
+	- If the user talks about running multiple tests, orchestration, collections of tests → use `type: suite`.
+	- If the user talks about concurrency, ramp-up, repeated execution, or load against one test flow → use `type: loadtest`.
+	- If the user talks about mocking, fake servers, stubbing responses, or "mock server" → use `type: server`.
+	- If the user talks about test reports, CI results, or exporting run results → `type: report` files are generated automatically. Explain the `--report` flag (see `docs/files/report/index.md`) rather than generating a report file.
+
+2. **Jump to the detailed generator**
+	- For `type: api`, use **`AI/generate-api.md`** to shape fields and examples.
+	- For `type: test`, use **`AI/generate-test.md`**.
+	- For `type: env`, use **`AI/generate-env.md`**.
+	- For `type: doc`, use **`AI/generate-doc.md`**.
+	- For `type: suite`, use **`AI/generate-suite.md`**.
+	- For `type: loadtest`, use **`AI/generate-loadtest.md`**.
+
+3. **Token syntax (always snake_case)**
+	- Environment `variables:` `e:api_url`, `e:auth_token`, etc.
+	- Embedded in strings/URLs: `<<e:api_url>>/users`, `Bearer <<e:auth_token>>`.
+	- Test/API `inputs:` `i:user_id`, `<<i:user_id>>`.
+	- Random: `r:uuid`, `r:first_name`, etc.
+	- Current/time: `c:date`, `c:epoch_ms`, etc.
+
+	Rules:
+	- Prefer **snake_case** for names inside tokens: `e:api_url`, `i:user_id`, not `e:api_url`.
+	- Use bare tokens as the **entire YAML value** when possible.
+	- Use `<< >>` only when embedding tokens inside a larger string.
+
+4. **Editing vs. creating**
+	- When creating a new `file:` start from the minimal valid skeleton in the corresponding `generate-*.md`.
+	- When editing: keep the existing structure, only add/change what the user asked, and do not remove unrelated fields.
+
+If you only need to **reason** about a user file (and not generate new YAML), prefer reading **only the relevant `AI/generate-*.md`** file to save tokens.

package/dist/guides/golden-smoke.md

@@ -0,0 +1,52 @@
+# Golden smoke pair (AI)
+
+Canonical low-token example. Mirror this shape; prefer `scaffold_test` over inventing YAML.
+
+## API — `examples/ai/golden_smoke/apis/echo.mmt`
+
+```yaml
+type: api
+title: Echo message
+description: Golden smoke API for AI agents — minimal POST with outputs
+method: post
+url: https://test.mmt.dev/echo
+format: json
+inputs:
+  message: hello
+outputs:
+  echoed: body.message
+body:
+  message: i:message
+```
+
+## Test — `examples/ai/golden_smoke/tests/echo-smoke.mmt`
+
+Produced by `scaffold_test` / `testlight scaffold test --from` (do not hand-author from scratch):
+
+```yaml
+type: test
+title: Echo message smoke test
+description: Smoke test for API "Echo message".
+tags:
+  - smoke
+  - post
+import:
+  echo: ../apis/echo.mmt
+inputs:
+  message: i:message
+steps:
+  - call: echo
+    id: iEcho
+    inputs:
+      message: i:message
+    expect:
+      status: 200
+      echoed: != null
+```
+
+## Agent rules
+
+1. New API test → `scaffold_test` first.
+2. Then `validate` → `format`.
+3. Modify → patch only (never full-file rewrite unless the user asks).
+4. After a run, to tighten asserts → `suggest_assertions` → patch → `validate` → `format`.

package/dist/guides/min/api.md

@@ -0,0 +1,29 @@
+# Generate `type: api` (min)
+
+```yaml
+type: api
+title: Login
+method: post
+url: <<e:api_url>>/login
+format: json
+inputs:
+  username: i:username
+  password: i:password
+body:
+  username: i:username
+  password: i:password
+outputs:
+  token: body.token
+```
+
+## Essentials
+
+- First line: `type: api`
+- `url` required; `method` required for HTTP
+- `protocol`: `http` or `ws` (often inferred)
+- `format`: `json` | `text` | `xml` | `urlencoded` | …
+- Optional: `headers`, `query`, `cookies`, `examples`, `setenv`
+- Tokens: `e:`, `i:`, `r:`, `c:`
+
+For tests against this API use `scaffold_test` / `api_card`, not a full OpenAPI dump.
+Request `pack: full` for WS, GraphQL, gRPC, and rare fields.