@postman/postman-plugin 0.3.0 → 0.4.0

package/skills/api-simulation/reference/sim_yaml_schema.md

@@ -1,112 +1,50 @@
-# `.sim.yaml` schema
+# `.sim.yaml` reference
 
-The file `postman simulation run <filepath>` reads. It lives in
-`postman/simulations/NAME.sim.yaml`, beside the `postman/mocks/` folders it
-composes. `postman simulation create` writes one; the Postman app writes the
-same shape with `id:` fields added.
+Read this when writing or debugging a `.sim.yaml` by hand.
+`postman simulation create` writes the same shape; the Postman app adds
+`id:` fields, which a local run ignores.
 
 ```yaml
-id: 01a110fc-e3f5-77eb-af59-0a711a509960        # written by the Postman app
-simulation: checkout-dev
-routing: path
-mocks:
-  - id: 01a110fc-e344-72ca-9d0c-1bb83b99ec2a    # written by the Postman app
-    routeKey: products
-    path: ../mocks/products/config.yaml
-    scenarios:
-      - path: ../mocks/products/default.js
-  - routeKey: shopping-cart
-    path: ../mocks/shopping-cart/config.yaml
+simulation: checkout-dev               # required: the name
+routing: path                          # path (default) or header
+mocks:                                 # required: one entry per member
+  - routeKey: catalog                  # lowercase letters, digits, - and _
+    path: ../mocks/catalog/config.yaml # a mock's config.yaml, or a .js handler
+    scenarios:                         # optional: conditions, below
+      - overrides:
+          conditions:
+            latency: { delay_ms: 250 }
 ```
 
-## Root
+Run the `.sim.yaml` rather than a simulation id: a run by id drops every
+member's conditions (CLI 1.70.0–1.73.0).
 
-| Key | Required | Notes |
-| --- | --- | --- |
-| `simulation` | Yes | The simulation's name. |
-| `mocks` | Yes | Non-empty list of members. |
-| `routing` | No | `path` (default) or `header`. Any other value means `path`. |
-| `id` | No | The simulation's id in its workspace. A local run ignores it; leave it as written. |
-
-## Member
-
-| Key | Required | Notes |
-| --- | --- | --- |
-| `path` | Yes | The mock's `config.yaml`, or a `.js` handler. Relative paths resolve from the `.sim.yaml`'s folder, then from the working directory. `file` is a legacy alias. |
-| `routeKey` | No | The member's address. Lowercase letters, digits, `-` and `_`, at most 64 characters, unique per simulation. Defaults to the kebab-cased name. |
-| `name` | No | Display name. Defaults to the mock's folder name for a `config.yaml`, else the file name. Must be unique. |
-| `scenarios` | No | Fault conditions, below. |
-| `id` | No | The mock's id in its workspace. A local run ignores it. |
-| `port` | No | Ignored by `simulation run`, which gives each member an internal port. |
-
-## Routing
-
-- **`path`**: a member is at `http://localhost:<port>/<routeKey>`. The first
-  path segment selects the member and is stripped, so `/payments/cart` reaches
-  the `payments` member as `/cart`. Point the service's base URL for that
-  dependency at `http://localhost:<port>/<routeKey>`.
-- **`header`**: every member shares `http://localhost:<port>`, and the caller
-  picks one with an `x-mock-slug: <routeKey>` header. Use it only when the
-  caller can send that header.
-
-The router answers on its own, without reaching a member:
-
-| Response | Cause |
-| --- | --- |
-| `400 {"error":"Prefix the request path with the member routing token, e.g. /<routeKey>/path."}` | No routeKey under `path` routing: the path was `/`. Under `header` routing the error asks for the `x-mock-slug` header instead. |
-| `404 {"error":"Unknown simulation member '<x>'.","available":[...]}` | `<x>` isn't a routeKey. Under `path` routing this is often a client that dropped the base URL's path, so `<x>` is the endpoint's own first segment. |
-
-## `scenarios`
-
-Each entry is a selection, and only its `overrides.conditions` inject faults:
-
-```yaml
-scenarios:
-  - overrides:
-      conditions:
-        latency:
-          delay_ms: 250              # hold each request 250 ms, then go on
-        error:
-          status_code: 503           # every request fails with 503
-        rate_limit:
-          requests_per_minute: 30    # the 31st request in a minute gets 429
-        chaos:
-          failure_rate: 10           # 10% of requests fail with 500
-```
-
-This example sets all four to show their fields. They run in a fixed order,
-`latency`, `error`, `rate_limit`, `chaos`, whatever order you write them in,
-and the first to fail a request ends it: with `error` set, `rate_limit` and
-`chaos` never fire. Set one failure at a time, optionally with `latency`.
-
-The mock's default handler always serves, and conditions run in front of it.
-An entry with only `path` or `id`, as the Postman app writes, injects nothing.
-Unknown condition names are ignored, so a misspelled one fails silently.
-`overrides.bypass` prints a warning and is ignored.
-
-The legacy form `- type: error` with `config: { status_code: 503 }` is still
-accepted and validated strictly: an unknown `type` is an error.
+## Conditions
 
 | Condition | Field | Effect |
 | --- | --- | --- |
-| `latency` | `delay_ms` (positive number) | Holds each request that long, then serves it normally. |
-| `error` | `status_code` (integer 400–599) | Every request returns that status with `{"error":{...},"scenario":"error"}`. |
-| `rate_limit` | `requests_per_minute` (positive integer) | Requests over the limit return `429` with a `Retry-After` header and `"scenario":"rate_limit"`. |
-| `chaos` | `failure_rate` (0–100) | That percentage of requests, chosen at random, return `500` with `"scenario":"chaos"`. |
-
-A condition applies to every route on its member. To fail one endpoint while
-its siblings succeed, split that endpoint into its own mock.
-
-## `simulation run`
-
-| Flag | Notes |
-| --- | --- |
-| `-p, --port <n\|auto>` | Port to serve on. Defaults to 3000, and silently falls back to a free port if 3000 is busy. An explicit port fails instead. |
-| `--output ndjson` | One JSON event per line on stdout, instead of the default readable log; messages go to stderr: `listening` (port, url, each member's `routeKey` and `address`); `request`, one per request a member served (`routeKey`, `method`, `path`, `statusCode`, `duration` in ms, headers and bodies); and `summary` on shutdown (`requestsServed`, `requestsFailed`). The router's own `400` and `404`, and injected faults, aren't logged. |
-| `-e`, `-g`, `--dataset` | Environment, globals and datasets for every member, as for `mock run`. |
-| `--no-history` | Record nothing. A `.sim.yaml` linked to a workspace otherwise records each start there; an unlinked one records only with `-w` and `--simulation`. |
-
-The argument can also be a simulation id from a workspace: the members are
-fetched and served locally. All members start before the port opens, so one
-`listening` event means the whole simulation is up. A member that fails to
-start stops the run with `Error: ...` and exit code 1. Ctrl+C stops everything.
+| `latency` | `delay_ms` | Holds each request that long, then serves it. |
+| `error` | `status_code` (400–599) | Every request gets that status as JSON, with `"scenario":"error"`. |
+| `rate_limit` | `requests_per_minute` | Requests over the limit get `429` and `Retry-After`. |
+| `chaos` | `failure_rate` (0–100), optional `status_code` | That share of requests, at random, get `500` or that status. |
+
+- A condition covers every route on its member. To fail one endpoint while
+  its siblings succeed, give that endpoint a mock of its own.
+- They run in a fixed order (`latency`, `error`, `rate_limit`, `chaos`), and
+  the first to fail a request ends it, so set one failure at a time,
+  optionally with `latency`.
+- An unknown condition name is ignored silently, so a misspelling injects
+  nothing. `overrides.bypass` prints a warning and is ignored.
+- A `latency` request is served and logged, with a `duration` that leaves
+  out the delay.
+
+## The log (`--output ndjson`)
+
+One JSON event per line on stdout; notices go to stderr, so keep them apart.
+
+- `listening`: every member has started; a member that fails to start
+  stops the run with `Error: …` and exit code 1.
+- `request`: one per request a member served.
+- `summary`, on stop: `requestsServed` (under 400), `requestsFailed` (400
+  and up), `uptimeMs` and `signal`. Injected failures and the simulation's
+  own `400`, `404` and `502` aren't counted.

package/skills/api-testing/SKILL.md

@@ -1,104 +1,57 @@
 ---
 name: api-testing
-description: Runs tests against an API from the command line — a single ad-hoc request, a full collection of pm.test assertions, or matching real captured app traffic against a collection contract. Use when the user asks to "test this endpoint," "run this collection," "check the API still works," or "verify my app's requests match the contract." Covers `postman request`, `postman collection run`, and `postman application test`. Depends on bootstrap when the target is a cloud collection or workspace-bound environment; a bare URL or local collection needs nothing from bootstrap.
+description: Use when testing an endpoint, running collection assertions, or checking real application traffic against an API contract. Reuses saved requests and reports what live, mocked, or captured-traffic results actually establish. Use for requests such as "test this endpoint".
 ---
 
 # API Testing
 
-## Overview
-
-Three tools, matched to what already exists:
-
-| Have | Use |
-| --- | --- |
-| Just a URL to check, with no saved request | `postman request` |
-| A request already saved in a collection | `postman collection run <collection-path> -i <request-id-name-or-path>` |
-| A collection with `pm.test` assertions saved in it | `postman collection run` |
-| A real app (browser flow, CLI, service) whose traffic should match a collection's contract | `postman application test` |
-
-Don't reach for the heavier tool when the lighter one already answers the
-question — a one-off endpoint check doesn't need a collection, and a
-collection run doesn't need Playwright.
-
-## `postman request` — over curl, not instead of testing
-
-A single request with Postman's resolution built in: `-e` resolves
-`{{variables}}` from an environment file the same way a collection run
-would, `--auth-*` flags cover basic/bearer/digest/oauth/aws/etc. without
-hand-building headers, and `--retry`/`--timeout` handle flaky endpoints.
-`--script-post-request` can run `pm.test(...)` assertions inline — the exit
-code counts *failed assertions*, not just HTTP status, so a 200 with a
-failing test still exits nonzero. Useful for a quick check or a CI health
-check; not the place to accumulate assertions that should outlive one
-command — those belong saved in a collection.
-
-Before constructing a URL, headers, auth, and body on the command line, look
-for a matching `*.request.yaml` under `postman/collections/`. If it exists,
-execute the saved request through its collection:
-
-```bash
-postman collection run "postman/collections/Orders API" -i "create order"
-```
-
-Do not copy the saved YAML fields into `postman request`; that bypasses the
-collection's inherited variables, auth, scripts, and maintained payload. The
-`postman request` positional target is a URL, not a `.request.yaml` path. When
-there is no saved request but its body already lives in a separate file, keep
-the file as the source of truth with `--body @path/to/payload.json` instead of
-inlining its contents.
-
-## `collection run` — the assertion suite
-
-Debugging a request's behavior during a run (why a `pm.test` failed, what a
-request actually sends) means reading the request's own YAML — see the
-`collection-schema-v3` skill for that file format before assuming a field's
-shape.
-
-Runs every request in a collection (or a subset via `-i`, repeatable),
-executing whatever `pm.test` scripts are already saved in it.
-`-d`/`--iteration-data` (or the beta `--iteration-data-dataset` +
-`--iteration-data-view` pair) drives data-driven runs across a CSV/JSON
-file or a Postman Dataset. `-r junit,html` for CI-consumable reports.
-`--use-mock`/`--mock` redirects the run at a mock instead of a real backend
-(see the `api-mocking` skill) — reach for this to test request/assertion
-logic without depending on a live service.
-
-## `application test` — contract-matching real traffic
-
-This doesn't send its own requests. It runs your existing test command
-(`--command "npx playwright test"`, or config-driven via
-`postman.config.cjs` targets), captures the network traffic that command
-generates, and matches/asserts it against your Postman collections —
-answering "did my app's actual calls conform to the contract," not "does
-this endpoint respond correctly." `--capture-only` skips the matching step
-entirely and just exports what was captured as a new v3 collection,
-organized by host — a way to bootstrap a collection from real traffic
-rather than authoring one from scratch. Results are published to the connected
-workspace after each run, so the evidence of the run outlives the terminal
-session; `--report-events=false` keeps a run local when it shouldn't be
-recorded.
-
-## Critical Rules
-
-1. **Don't reach for `application test` for something a plain
-   `collection run` covers.** It exists specifically for matching *captured*
-   app traffic (via Playwright or similar), not for driving requests itself.
-2. **`postman request`'s exit code reflects failed `pm.test` assertions, not
-   HTTP status alone.** A nonzero exit on a 200 response usually means a
-   post-request script assertion failed, not a network problem.
-3. **Assertions meant to be reused belong in the collection, not on the
-   command line.** A `--script-post-request` test on `collection run` runs
-   once and leaves nothing for the next person; save it as a `pm.test` in
-   the request instead.
-4. **`--use-mock` is the way to test without a live backend** — prefer it
-   over standing up ad hoc fakes or skipping tests that need a dependency.
-5. **Reuse a saved request instead of reconstructing it.** If a matching v3
-   request exists, use `collection run <collection-path> -i <request>`; reserve
-   `postman request` for genuinely ad-hoc requests.
+Saved requests preserve the auth, payloads, variables, and assertions that make
+an API check reproducible. The useful result is evidence about an intended
+guarantee; a successful HTTP exchange alone does not establish correctness.
+
+1. Choose the smallest execution mode that answers the question:
+
+   | Evidence needed | Use |
+   | --- | --- |
+   | One endpoint with no saved request | `postman request <url>` |
+   | Behavior of an existing saved request | `postman collection run <collection-path> -i <request-id-name-or-path>` |
+   | Reusable regression checks | `postman collection run <path-or-id>` |
+   | Whether a real app's calls match the contract | `postman application test` |
+
+2. Reuse a matching request under `postman/collections/` before reconstructing
+   its HTTP fields. Running it through the collection retains inherited auth,
+   variables, and scripts. `postman request` takes a URL, not a request YAML
+   path; for an unsaved request with an existing payload file, `--body @path`
+   preserves that file as the source. See **collection-schema-v3** when
+   editing the saved request.
+3. Derive assertions from the contract and the behavior at risk. A test that
+   merely repeats the current implementation can bless its bug. Preserve
+   existing guarantees and add checks for the changed behavior; put reusable
+   `pm.test` assertions in the collection so later runs and teammates inherit
+   them. Inline `--script-post-request` assertions suit a one-off check.
+4. Choose a real or mocked target according to the claim. `--use-mock` lets a
+   collection run exercise request and assertion logic without a live
+   backend; `--mock` alone only starts one. `--use-mock` replaces the
+   variable's whole value, base path included (see **api-mocking**). Passing
+   against that mock does not verify the service the mock stands in for.
+5. Reuse environment resolution with `-e` and **variables**. Use a CSV/JSON
+   file through `-d` for simple iteration; use **datasets** when the inputs
+   need queries, joins, or reusable views. Export reports such as
+   `-r junit,html` when **ci-integration** needs evidence beyond terminal output.
+6. Use `application test` when the application's actual traffic is the subject.
+   It runs an existing command, such as `--command "npx playwright test"`,
+   captures its traffic, and matches it against collections. `--capture-only`
+   instead exports a v3 collection organized by host; that captures behavior
+   without validating it. Results are recorded in the connected workspace by
+   default; `--report-events=false` keeps the run local when required.
+
+Use command-specific `-h` for supported options and **bootstrap** for unresolved
+CLI setup or authentication when using workspace resources. A bare URL or
+local collection does not require a workspace binding.
 
 ## Verification
 
-State the actual result (pass/fail counts, exit code), not just that the
-command ran. For `application test`, state whether it ran in match mode or
-`--capture-only` — they answer different questions and shouldn't be
-reported the same way.
+Report the target, tested behavior, assertion counts, and exit code. The exit
+code reflects failed assertions, not HTTP status alone: a 200 can fail a test.
+For application testing, distinguish matching from capture-only mode, and for
+mocked runs state which real behavior remains unverified.