mmt-testlight 0.4.4 → 0.4.5

package/dist/guides/agent-workflow.md

@@ -0,0 +1,75 @@
+# Multimeter agent workflow (MCP-first)
+
+When the user asks to **create, modify, validate, format, or run** any `.mmt` file, use the **Multimeter MCP server tools first**. Do not explore npm packages, CLIs, or shell workarounds before calling MCP. Do not web-search Multimeter syntax.
+
+## Golden rule
+
+**If the task involves a `.mmt` file, start with Multimeter MCP tools in the first tool-call batch.**
+
+Do **not** first try:
+- `testlight`, `npx testlight`, or npm install/run (unless MCP is unavailable — then use `testlight scaffold` / docs)
+- `node dist/mcp/server.js` or the `mmt-mcp` binary
+- Reading Multimeter source, tarball contents, or package.json to find a runner
+- Guessing YAML syntax from memory or random repo files
+- Inventing a new test from blank without `scaffold_test`
+- Rewriting an entire `.mmt` file when the user asked to change one part
+
+## Decision tree
+
+| User intent | First MCP tool(s) | Then |
+|-------------|-------------------|------|
+| Generate test from API | **`scaffold_test`** | Write yaml → minimal edits → **`validate` → `format`** |
+| Need few-shot shape | **`list_examples`** (see `goldenSmoke`) | Mirror pair; still call `scaffold_test` |
+| Inspect one API | **`api_card`** | Prefer over full file / OpenAPI dump |
+| Tighten asserts (after run or from outputs) | **`suggest_assertions`** | Patch only → **`validate` → `format`** |
+| Create or change other types | `read_documentation(topic)` (pack **min**) | Patch → **`validate` → `format`** |
+| Change existing `.mmt` file | (docs only if needed) | **Patch only** → **`validate` → `format`** |
+| List APIs | `discover_api` | Then `api_card` / `scaffold_test` |
+| Run or execute `.mmt` | `run({ file, workspaceRoot })` | Report tool JSON; if user wants stronger asserts → `suggest_assertions` |
+| Unsure of syntax | `read_documentation` (min; `pack: full` if needed) | Local docs only — no web |
+| Offline / no MCP | see `offline-agent.md` | `testlight docs` → `scaffold` → `validate` |
+
+## Generate test from API (required)
+
+1. Optional: `list_examples` and mirror **`goldenSmoke`** (do not invent a different structure).
+2. `scaffold_test({ workspaceRoot, apiPath, strategy?: "smoke"|"example" })`
+3. Write `yaml` to `suggestedPath` (or user path).
+4. Apply **only minimal** edits (asserts, inputs, title).
+5. **`validate({ file, workspaceRoot })` until `valid: true`.**
+6. **`format({ file, workspaceRoot })` after validate passes** (required on generate).
+7. `run` only when the user asks to execute.
+8. If the user then wants stronger asserts from a response: `suggest_assertions` → patch → validate → format.
+
+## Modify workflow (required)
+
+When the user asks to **modify**, **update**, **fix**, or **add steps** to a `.mmt` `file:`
+
+1. `read_documentation(topic: "<type>")` only when syntax is unclear (pack **min**).
+2. If helpful, `api_card` / `discover_api`.
+3. **Patch only** — change the few lines needed. **Do not rewrite the whole file** unless the user explicitly asks for a rewrite/regenerate.
+4. **`validate` immediately after every edit** until `valid: true`.
+5. **`format` after validate passes.**
+6. Only call `run` when the user asks to execute.
+
+## Run → tighten asserts (optional)
+
+When the user wants better assertions after a successful (or inspected) run:
+
+1. `suggest_assertions` with `apiPath` and/or response `body` / `bodyFile`
+2. Patch `expect` / `assert` lines only
+3. `validate` → `format`
+
+## Validation + format are mandatory
+
+After **every** generate or modify on `.mmt` files:
+
+1. `validate` until valid  
+2. `format`  
+before telling the user the task is done.
+
+## YAML rules (always)
+
+- Output valid Multimeter YAML only.
+- First non-comment line must be `type: <api|test|env|...>`.
+- Never add YAML comments (`#`).
+- Use snake_case tokens: `e:api_url`, `i:user_id`, `r:uuid`, `c:epoch_ms`.

package/dist/guides/general.md

@@ -0,0 +1,64 @@
+This folder teaches an AI how to **answer questions** about Multimeter and decide **when to generate or edit** `.mmt` files.
+
+**When using VS Code Copilot with the Multimeter extension:** call Multimeter **MCP tools first** (`read_documentation`, `discover_api`, `validate`, `format`, `run`). See `AI/agent-workflow.md`.
+
+For actual YAML generation or modification, always go through `AI/generate.md`.
+
+The AI should:
+- Understand what each `type` (`api`, `test`, `env`, `doc`, `suite`) is for.
+- Decide whether the user needs **an explanation** or **new/edited YAML**.
+- Minimize tokens by reading only the specific generator file it needs.
+
+Short mental model of file types:
+- `type: api` → describe one HTTP/WebSocket endpoint.
+- `type: test` → describe flows that call APIs/tests and assert behavior.
+- `type: env` → define environment variables and presets.
+- `type: doc` → describe documentation over existing `.mmt` files.
+- `type: suite` → group and run multiple tests, APIs, or other suites.
+- `type: server` → define a mock server with endpoints, matching, and responses.
+- `type: report` → structured test results (generated by runs, viewable in the editor).
+
+When you are the AI, follow these rules:
+
+1. **Decide intent first**
+	- If the user asks "how" or "why" (no explicit request to create/edit a file):
+	  - Explain concepts in plain language.
+	  - Use small inline examples.
+	  - Do **not** open generator files unless you need exact shapes.
+	- If the user explicitly asks to create/edit a `.mmt` `file:`
+	  - Open **`AI/generate.md`**.
+	  - Follow its instructions to choose the type and generator.
+
+2. **Token conventions (snake_case)**
+	- Always show environment tokens in **snake_case**, for example:
+	  - `e:api_url`, `e:auth_token`, `e:user_id`.
+	  - Embedded: `<<e:api_url>>/users`, `Bearer <<e:auth_token>>`.
+	- Inputs: `i:user_id`, `<<i:user_id>>`.
+	- Random values: `r:uuid`, `r:first_name`, `r:last_name`, `r:epoch_ms`.
+	- Current/time values: `c:date`, `c:epoch`, `c:epoch_ms`.
+	- Use bare tokens as standalone YAML values; use `<< >>` only when embedding inside larger strings.
+
+3. **Answering common question types**
+	- "How do I call this endpoint?"
+	  - Explain briefly.
+	  - If the user wants a file, go to `generate.md` and build a `type: api` example.
+	- "How do I write a test that does X then Y?"
+	  - Explain tests and flows.
+	  - If a file is requested, go to `generate.md` and build a `type: test` example.
+	- "How do I manage/dev/prod environments?"
+	  - Explain env files and presets.
+	  - If a file is requested, go to `generate.md` and build a `type: env` example.
+	- "How do I generate API docs?"
+	  - Explain doc files.
+	  - If a file is requested, go to `generate.md` and build a `type: doc` example.
+	- "How do I run multiple tests together?"
+	  - Explain suite files.
+	  - If a file is requested, go to `generate.md` and build a `type: suite` example.
+
+4. **Safety and style**
+	- Use 2‑space indentation in all YAML examples.
+	- Prefer small, focused examples.
+	- Do not invent new fields or token formats.
+	- When unsure, propose the **minimal valid change** and explain it.
+
+Your goal here is to **decide what to do** and provide good explanations; when it is time to actually write YAML, defer to `AI/generate.md` and the more detailed `AI/generate-*.md` files.

package/dist/guides/generate-api.md

@@ -0,0 +1,221 @@
+This file tells the AI how to generate **`type: api`** `.mmt` files.
+
+Always follow these rules:
+- Output must be valid YAML.
+- The **first non-comment line** must be `type: api`.
+- Prefer explicit, descriptive names and short, readable examples.
+- Only include fields that make sense for the user’s request.
+
+---
+
+## Schema (mental model for the AI)
+
+These fields mirror the internal `api` data model.
+
+Top‑level keys and types (with descriptions the AI should remember):
+
+```yaml
+type: api                      # REQUIRED. Declares this file as an API description.
+
+# Documentation / metadata (optional but recommended)
+title: string                  # Short human title, e.g. "Login API" or "List users".
+description: string            # 1–3 sentence explanation of what this endpoint does.
+tags:                          # Optional tags for search/filters (e.g. [auth, smoke]).
+	- string
+
+# Reuse / composition
+import:                        # Optional. Map of aliases to other .mmt files this API depends on.
+	<alias>: path/to/file.mmt
+inputs:                        # Optional. Input parameters for this API, with default values.
+	<name>: JSON value           # string | number | boolean | null | object.
+outputs:                       # Optional. Named values extracted from the response (body/headers/cookies).
+	<name>: string               # Extractor expression, e.g. body[token] or body[/regex_pattern/]
+setenv:                        # Optional. Promote response values into env vars after a run (same extraction DSL as outputs).
+	<env_name>: <expression>     # e.g. body[token] or body.access_token (legacy: outputs key name still works).
+
+# Request description (REQUIRED)
+url: string                    # REQUIRED. Full or relative URL; may include env/input tokens like <<e:api_url>>.
+query:                         # Optional. Query parameters appended to url; merged with any inline query string.
+	<name>: string               # Value is always a string expression (can contain tokens).
+protocol: http | ws            # Optional. "http" for HTTP(S); "ws" for WebSocket.
+                               # Inferred from URL if omitted: ws:// or wss:// → ws, otherwise http.
+format: json | text | xml | xmle | urlencoded | { request, response }
+method:                        # HTTP method (REQUIRED when protocol is http).
+	get | post | put | delete | patch | head | options | trace
+headers:                       # Optional. HTTP/WS headers to send with the request.
+	<Header-Name>: string        # Values can include tokens; header names are case-insensitive.
+cookies:                       # Optional. HTTP cookies to send.
+	<cookie_name>: string
+body: object | string | null   # Optional. Request body (HTTP) or initial WS message; type depends on format.
+
+# Examples (optional)
+examples:                      # Optional. Example invocations for smoke tests / documentation.
+	- name: string               # REQUIRED per example. Identifier used in UI/CLI.
+		description?: string       # Optional explanation for this example.
+		inputs?: object            # JSON record overriding top-level inputs for this example.
+		outputs?: object           # JSON record of expected outputs for this example.
+```
+
+Notes for the AI:
+- `inputs`, `setenv`, and `examples` are optional but powerful when the user wants reusable APIs or smoke‑testable examples.
+- `import` is rarely needed when the user only describes a single endpoint.
+
+---
+
+## When to use which fields
+
+- Use **`title`** and **`description`** whenever the user describes intent or business meaning (`"login endpoint"`, `"create user"`, etc.).
+- Use **`tags`** when the user mentions categories like "smoke", "load", "auth", "payments".
+- Use **`inputs`** when the user expects to change parameters between runs (username, pagination, filters, ids, etc.).
+- Use **`outputs`** + **`setenv`** when the user wants to **capture response data** for later steps (e.g. tokens, IDs).
+- Use **`examples`** when the user asks for **examples, smoke tests, or sample calls**.
+- Use **`protocol: ws`** when the user asks for **websocket api**. For **REST or SOAP api**, protocol can be omitted (defaults to http based on URL).
+
+
+If the user gives:
+- **Only URL + method** → make a minimal `file:` `type`, `title`, `format`, `method`, `url` (protocol is inferred).
+- **Request + response contract** → also add `inputs`, `outputs`, maybe `setenv`.
+- **Multiple usage scenarios** → use `examples` to capture them.
+
+---
+
+## Tokens the AI can use
+
+The API body, headers, query, cookies, and url can contain tokens:
+
+- Environment `variables:` `e:api_url`, `e:auth_token`, etc., or `<<e:api_url>>`
+- Test/API `inputs:` `i:var` or `<<i:var>>`
+- Random values: `r:name` or `<<r:name>>`
+- Current/time values: `c:name` or `<<c:name>>`
+
+Guidelines:
+- If a value **is exactly** a single token (e.g. `username: i:user` or `id: r:int`), do **not** quote it unless the surrounding examples already use quotes.
+- If a token is used **inside text**, wrap it as `<<token>>`, for example:
+	- `url: <<e:api_url>>/users/<<i:user_id>>`
+	- `x-request-id: req-<<r:uuid>>`
+
+Common random tokens: `r:uuid`, `r:int`, `r:bool`, `r:email`, `r:phone`, `r:first_name`, `r:last_name`, `r:full_name`, `r:epoch`, `r:epoch_ms`.
+Common current tokens: `c:date`, `c:time`, `c:epoch`, `c:epoch_ms`.
+
+Do **not** invent new token syntaxes.
+
+---
+
+## Authoring patterns for common requests
+
+### 1. Simple GET with query and headers
+
+Use when the user asks for something like “GET `/users` with pagination and auth header”.
+
+```yaml
+type: api
+title: List users
+description: Returns a paginated list of users.
+tags: [users, list]
+inputs:
+	limit: 20
+	page: 1
+url: <<e:api_url>>/users
+protocol: http
+format: json
+method: get
+headers:
+	authorization: Bearer <<e:auth_token>>
+query:
+	limit: "<<i:limit>>"
+	page: "<<i:page>>"
+```
+
+### 2. Login POST with JSON body and extracted token
+
+Use when the user describes a login/auth endpoint that returns a token.
+
+```yaml
+type: api
+title: Login
+description: Authenticate user and return a session token.
+tags: [auth, login]
+inputs:
+	username: user@example.com
+	password: string
+outputs:
+	token: body[token]
+	status: body[status]
+setenv:
+	auth_token: body[token]
+url: <<e:api_url>>/login
+protocol: http
+method: post
+format: json
+headers:
+	content-type: application/json
+body:
+	username: i:username
+	password: i:password
+examples:
+	- name: valid-user
+		description: Login with a valid account.
+		inputs:
+			username: alice@example.com
+			password: secret
+	- name: invalid-password
+		description: Login with wrong password.
+		inputs:
+			username: alice@example.com
+			password: wrong
+```
+
+### 3. Raw text or XML body
+
+Use `format: text`, `format: xml`, `format: xmle`, or `format: urlencoded` when the user explicitly mentions plain text, HTML, XML, or form URL-encoded bodies. Use `xmle` when expanded empty tags are required. Use `urlencoded` for `application/x-www-form-urlencoded` key/value bodies.
+
+```yaml
+type: api
+title: Echo text
+description: Send and echo back a plain text payload.
+protocol: http
+format: text
+method: post
+url: <<e:api_url>>/echo
+body: |
+	hello world
+```
+
+```yaml
+type: api
+title: Submit XML payload
+description: Send an XML document as the request body.
+protocol: http
+format: xml
+method: post
+url: <<e:api_url>>/xml
+body: |
+	<root>
+		<value>42</value>
+	</root>
+```
+
+### 4. WebSocket connection description
+
+Use when the user describes WS endpoints (chat, notifications, streams). Actual send/receive flows are usually defined in `type: test` files calling this API.
+
+```yaml
+type: api
+title: Notifications stream
+description: WebSocket stream of user notifications.
+protocol: ws
+format: json
+url: wss://example.com/notifications
+```
+
+---
+
+## Style rules for the AI
+
+- Prefer **2 spaces** for indentation.
+- Avoid trailing spaces.
+- Prefer lower‑case, hyphen‑separated tags (e.g. `smoke`, `auth`, `user-profile`).
+- Keep titles short (3–6 words) and descriptions 1–3 sentences.
+- Do **not** rely on YAML comments (`#`) for important docs — prefer `description`. Format Document preserves comments, but UI form sync may still rebuild YAML without them.
+
+When unsure, **favor simpler files** with fewer fields rather than guessing complex outputs or headers.

package/dist/guides/generate-doc.md

@@ -0,0 +1,103 @@
+This file tells the AI how to generate **`type: doc`** `.mmt` files.
+
+Doc files describe how to build HTML/Markdown documentation from existing `.mmt` API files.
+
+Always follow these rules:
+- Output must be valid YAML.
+- The first non-comment line must be `type: doc`.
+- Do **not** redefine APIs here; instead, point to files/folders containing `type: api` definitions.
+
+---
+
+## Schema (mental model for the AI)
+
+This matches `DocData` in `core/src/DocData.ts` and `docs/files/doc/index.md`.
+
+Top-level keys and types:
+
+```yaml
+type: doc                       # REQUIRED, must be exactly "doc"
+
+title?: string                  # title of the documentation page
+description?: string            # optional description or intro text
+logo?: string                   # optional path/URL to a logo image
+sources?:                       # list of folders/files to scan for APIs
+  - string
+services?:                      # optional logical groupings
+  - name?: string
+    description?: string
+    sources?:
+      - string
+```
+
+Notes:
+- `sources` and each service’s `sources` can include:
+  - Direct paths to `.mmt` files.
+  - Folders containing `.mmt` files (recursively scanned).
+- Only `type: api` entries discovered in those paths are rendered as API boxes.
+
+---
+
+## Common patterns the AI should generate
+
+### 1. Single-folder API catalog
+
+User asks: "Generate docs for all APIs in `./apis`."
+
+```yaml
+type: doc
+title: API Catalog
+sources:
+  - ./apis
+```
+
+### 2. Multi-service catalog with groupings
+
+User asks: "We have user and order services under different folders; build docs for both."
+
+```yaml
+type: doc
+title: Platform APIs
+description: Documentation for core platform services.
+sources:
+  - ./shared
+services:
+  - name: Users
+    description: User registration, login, and profile APIs.
+    sources:
+      - ./services/users
+  - name: Orders
+    description: Order creation, tracking, and status APIs.
+    sources:
+      - ./services/orders
+```
+
+### 3. Mixed single files and folders
+
+```yaml
+type: doc
+title: Authentication APIs
+sources:
+  - ./auth/login.mmt
+  - ./auth/refresh.mmt
+  - ./auth
+```
+
+---
+
+## How the AI should answer doc-related questions
+
+- If the user mentions **"generate documentation"**, **"API catalog"**, or similar, prefer `type: doc`.
+- If the user provides a list of API files or folders, put them in `sources` and optionally group them under `services` if they describe separate domains.
+- If the user wants a **logo** or branding, set `logo` to a relative path or URL (do not invent images; reuse what they mention).
+
+---
+
+## Style rules for the AI
+
+- Use 2-space indentation.
+- Keep titles short and descriptive.
+- Keep service names in Title Case (e.g. `Users`, `Billing`).
+- Prefer concise descriptions (1–2 sentences per service).
+
+When unsure, generate a **small doc file** that points to the most obvious folder(s) the user mentioned (such as `./apis` or `./examples`).

package/dist/guides/generate-env.md

@@ -0,0 +1,147 @@
+This file tells the AI how to generate **`type: env`** `.mmt` files.
+
+Environment files define variables and presets used by APIs and tests.
+
+Always follow these rules:
+- Output must be valid YAML.
+- The first non-comment line must be `type: env`.
+- Favor clear, descriptive variable names (snake_case or SCREAMING_SNAKE_CASE), for example:
+  - `api_url`, `auth_token`, `mode`, or
+  - `api_url`, `auth_token`, `mode`.
+
+---
+
+## Schema (mental model for the AI)
+
+This matches `EnvironmentData` in `mmtview/src/environment/EnvironmentData.tsx` and `docs/files/env/index.md`.
+
+Top-level keys and types:
+
+```yaml
+type: env                       # REQUIRED, must be exactly "env"
+
+variables:                      # REQUIRED
+  <name>:
+    # One of:
+    # 1) mapping of labels -> string values (key-value choices)
+    # 2) array of strings (allowed values)
+
+presets?:                       # OPTIONAL
+  <presetGroup>:
+    <presetName>:
+      <env_name>: string        # value *must* match one of the variable options
+```
+
+Interpretation:
+- `variables[NAME]` can be:
+  - `{ dev: "http://localhost:8080", prod: "https://test.mmt.dev" }` → named key-value choices.
+  - `["debug", "info", "warn"]` → a list of allowed values.
+- `presets[group][name]` tells the UI what to pick for each variable when that preset is active.
+
+---
+
+## Tokens in other files
+
+Env variables are **read in APIs/tests**, not stored as tokens here. In `type: api` or `type: test` files, you read them with:
+
+- `e:api_url`        # standalone value
+- `<<e:api_url>>`    # inside strings/URLs
+
+The AI should not put `e:` tokens into env files; here we only define raw values or options.
+
+---
+
+## Common patterns the AI should generate
+
+### 1. Simple local + prod configuration
+
+```yaml
+type: env
+variables:
+  api_url:
+    local: "http://localhost:8080"
+    prod: "https://test.mmt.dev"
+  default_user:
+    - "alice@example.com"
+  default_password:
+    - "secret"
+  log_level:
+    - debug
+    - info
+    - warn
+    - error
+presets:
+  runner:
+    local:
+      api_url: local
+      log_level: debug
+    prod:
+      api_url: prod
+      log_level: info
+```
+
+### 2. Testing modes and feature flags
+
+```yaml
+type: env
+variables:
+  mode:
+    dev: dev
+    staging: staging
+    prod: prod
+  feature_x_enabled:
+    - "true"
+    - "false"
+  timeout_ms:
+    - "1000"
+    - "2000"
+    - "5000"
+presets:
+  runner:
+    dev:
+      mode: dev
+      feature_x_enabled: "true"
+      timeout_ms: "1000"
+    prod:
+      mode: prod
+      feature_x_enabled: "false"
+      timeout_ms: "2000"
+```
+
+---
+
+## How the AI should answer env-related questions
+
+- If the user wants to **switch between environments** (local/staging/prod), generate:
+  - Variables with **key-value choice mappings**, e.g. `api_url: { local: ..., prod: ... }`.
+  - A `runner` preset group with `local`, `staging`, `prod` entries.
+- If the user wants **valid options** for a flag or numeric setting, use **arrays** of strings.
+- If a variable has only a single known value, wrap it in a single-element array (e.g. `token: ["your-token"]`).
+- Always keep values as strings in env files; typing happens when they are read and interpreted.
+
+---
+
+## Style rules for the AI
+
+- Use 2-space indentation.
+- Use ALL_CAPS for variable names by convention (not enforced but clearer).
+- Keep presets small and focused; do not create many unused presets.
+- Prefer explicit URLs and modes instead of opaque values.
+
+When unsure, generate a **small env file** with the most clearly requested variables and a single preset group named `runner` with 1–2 named presets.
+
+---
+
+## Certificates (optional)
+
+Env files can also contain a `certificates` section for SSL/TLS settings. The AI generally does not need to generate this unless the user specifically asks about TLS, mTLS, or client certificates. See `docs/features/certificates/index.md` for full details.
+
+```yaml
+certificates:
+  server_ca: ./certs/ca.pem
+  clients:
+    - name: my-client
+      host: api.example.com
+      cert: ./certs/client.pem
+      key: ./certs/client-key.pem
+```

package/dist/guides/generate-loadtest.md

@@ -0,0 +1,55 @@
+This file tells the AI how to generate `type:` loadtest .mmt files.
+
+Always follow these rules:
+- Output must be valid YAML.
+- The first non-comment line must be `type:` loadtest.
+- Load tests should reference exactly one existing `type:` test file via the top-level test field.
+
+---
+
+## Schema (mental model for the AI)
+
+Top-level keys and types:
+
+```yaml
+type: loadtest
+
+title: string
+
+tags:
+  - string
+
+description: string
+
+environment:
+  preset?: string
+  file?: string
+  variables?:
+    <name>: <value>
+
+threads: number
+repeat: string | number
+rampup: string
+
+export:
+  - path/to/report.mmt
+
+test: path/to/test.mmt
+```
+
+Guidelines:
+- Use test, not tests.
+- Do not add servers; load tests do not support suite-level servers.
+- Keep the referenced file a `type:` test document.
+
+Example:
+
+```yaml
+type: loadtest
+title: Login load test
+tags: [load, auth]
+threads: 100
+repeat: 1m
+rampup: 10s
+test: ./tests/login.mmt
+```