@postman/postman-plugin 0.3.0 → 0.4.1-rc.0

package/skills/api-simulation/SKILL.md

@@ -1,186 +1,138 @@
 ---
 name: api-simulation
-description: The local development loop for a service that calls other APIs — find its dependencies in the code and the Context Graph, give each one a mock (the owning team's, or one generated from its contract), serve them together from one `.sim.yaml` on one port, run the real service against it, read the simulation's log to see what each dependency served, and ship it with the simulation in the PR. Can also inject latency, error, rate-limit or chaos per dependency. Use when the user is building or changing an endpoint that calls other services, or asks to "simulate my dependencies," "mock everything this service calls," or "run this end to end locally." Covers `postman simulation`. Builds on api-discovery for finding dependencies and their owners' mocks, and on api-mocking for the rest.
+description: Runs your code for real against stand-ins for the HTTP services it calls, served together as one environment on one local port or one deployed URL, whether that code is a service you're changing, a frontend, a test suite or a demo. Use when you add or change code that calls other services and need to run it, test it, prove a fix or reproduce an upstream failure (timeouts, 5xx, 409, 429) end to end, locally or in CI, without the real services because they're unreachable, need credentials, cost money, have side effects or don't exist yet. Also answers what happens to a request, or to the data it writes, when a service it calls goes down, is slow or returns errors ("what happens to a booking if the email service is down?"), by running each failure instead of reading the code. Applies even when nobody says mock or simulation. To give a frontend or another team a stand-in for an API, use api-mocking. Covers `postman simulation`.
 ---
 
 # API Simulation
 
-## Overview
-
-A simulation serves several mocks on one local port. Each member has a
-**routeKey**, its address: with `routing: path` (the default) a dependency
-lives at `http://localhost:<port>/<routeKey>`, and the routeKey is stripped
-before the mock sees the request. The service under test runs for real and is
-never a member.
-
-```yaml
-# postman/simulations/feeds-dev.sim.yaml
-simulation: feeds-dev
-routing: path
-mocks:
-  - routeKey: acs
-    path: ../mocks/acs/config.yaml
-  - routeKey: workspaces
-    path: ../.dependencies/mocks/workspace-service/config.yaml
+A simulation serves several mocks as one environment, each reached by its
+routeKey: as a path prefix by default, or as an `x-mock-slug` header when the
+caller's paths can't change.
+
+Run your code against it rather than stubbing its HTTP client in tests or
+calling the real services. A stub skips the client, config, timeouts and
+error handling that will run in production; the real services need
+credentials and leave side effects. A simulation runs that code over real
+HTTP, the same way in CI, and its log shows what each dependency was asked.
+
+Do only what the task needs. If a `.sim.yaml` in `postman/simulations/`
+already covers it, run that one. One request through the changed path proves
+a change; build failure variants only when asked how the code fails or when
+it must survive a failure.
+
+`postman simulation` needs CLI 1.70.0 or later (`postman --version`; an older
+CLI answers `postman simulation -h` with its general help).
+
+## Building it
+
+- **Members are the code's HTTP dependencies.** Databases, queues and caches
+  aren't; run them locally. **api-discovery** can find calls made through
+  shared code; ask about the repo by its `git remote` name, because a close
+  name matches another repo.
+- **Each member needs a mock: see `api-mocking`,** which looks for the owning
+  team's published mock before building one. For a question, one built from
+  the contract will do; one that invents responses can send the caller down a
+  path the real service never would. In a mock you build, answer `501` for calls this path
+  doesn't make: a `404` reads to the caller like a real "not found".
+- **Prefix each mock's `pm.state` keys with its name.** As of CLI 1.73.0,
+  members of a local run read each other's keys, so two mocks storing
+  `orders` overwrite each other.
+
+## Running it
+
+The run is a server that never exits, so a foreground command never returns.
+Start it in the background with an explicit port, its log in a file:
+
+```bash
+postman simulation run postman/simulations/checkout-dev.sim.yaml --port 4900 --output ndjson > sim.ndjson 2> sim.err &
 ```
 
-Running a local `.sim.yaml` works signed out. Running one by id, the Context
-Graph, `search`, `dependency` and anything in a workspace need `postman login`.
-`postman simulation` needs Postman CLI 1.70.0 or later (`postman update`);
-earlier releases have only the older `postman simulate run`.
-
-## Commands
-
-Most commands work either on a local `.sim.yaml` or on a simulation in a
-workspace:
-
-| Command | With a `.sim.yaml` path | With a simulation id |
-| --- | --- | --- |
-| `create -n <name> --mock <ref>=<routeKey>` | Writes `postman/simulations/<name>.sim.yaml` from mock paths | Creates one in the workspace from mock ids (`-w`). Don't mix the two |
-| `list` | Lists `postman/simulations/` | `-w <workspaceId>` lists the workspace's |
-| `get <pathOrId>` | Reads the file | Fetches it |
-| `run <pathOrId>` | Serves the members on one port | Fetches the members, then serves them locally |
-| `push <path>` | Uploads each member mock, then the simulation, to the linked workspace. A member pulled with `dependency add` goes up as a copy | — |
-| `deploy <id>` | — | Serves it at a URL, **public by default** (rule 7). `--private` requires an API key |
-| `delete <pathOrId>` | Deletes the file, not its mocks | Deletes it and takes its URL down |
-
-## Process
-
-1. **Find what the change calls.** If a `.sim.yaml` in `postman/simulations/`
-   already covers the change, skip to step 4. Otherwise read the code path
-   you're changing: its HTTP clients, their base-URL settings, and any call
-   the change adds. Only HTTP dependencies become members; queues, databases
-   and caches run as real local instances. When signed in, check that list
-   against the Context Graph, which also sees calls made through shared code
-   and other repositories:
-   `postman context-graph ask "What services and APIs does <repo-name> call?" --wait`.
-   Use the repository's name from `git remote get-url origin`, not the
-   folder's: a wrong name answers "no dependencies", not an error. A call the
-   change adds isn't in the graph yet.
-
-2. **Get each dependency's mock from its owner first.** This needs
-   `postman login`; signed out, ask the user to sign in rather than writing
-   mocks (rule 5). For every dependency, in this order:
-   1. Find the owner's workspace:
-      `postman context-graph ask "Which Postman workspace holds the collections and mocks for <service>?" --wait`.
-      If the workspace it names has neither, find the owner's with
-      `postman search workspaces "<service>"`.
-   2. List that workspace's mocks:
-      `postman search mocks --filter "workspaceId=<id>"`. A name match in
-      another workspace is not ownership.
-   3. If the owner has one, pull it with `postman dependency add mock <mockId>`
-      and use the copy as it is, even if the owner also deployed it.
-   4. Only if the owner has none, make one: pull their collection with
-      `postman dependency add collection <id>` and run `postman mock generate`
-      on it (see `api-mocking`). Keep the endpoints this path calls and answer
-      `501` for the rest, so a missed call fails loudly. Make it stateful when
-      the change writes something and reads it back.
-
-3. **Compose the simulation.**
-
-   ```bash
-   postman simulation create -n feeds-dev \
-     --mock ./postman/mocks/acs=acs \
-     --mock ./postman/.dependencies/mocks/workspace-service=workspaces
-   ```
-
-   Each `=<routeKey>` names a member: lowercase letters, digits, `-` and `_`.
-   Schema: [reference/sim_yaml_schema.md](reference/sim_yaml_schema.md).
-
-4. **Run it and point the service at it.**
-
-   ```bash
-   postman simulation run postman/simulations/feeds-dev.sim.yaml --port 4900 --output ndjson > sim.ndjson 2> sim.err &
-   ```
-
-   Keep stderr out of `sim.ndjson` so every line parses; start-up errors land
-   in `sim.err`. It's up when the `listening` event lists each member's
-   address. Set each upstream base URL to `http://localhost:4900/<routeKey>`,
-   never to a mock server's URL.
-
-   Stop it with
-   `pkill -INT -f "simulation run postman/simulations/feeds-dev.sim.yaml"`:
-   every member stops, and the closing `summary` event counts the requests
-   served and failed. `kill $!` reaches only the `postman` launcher, which
-   ignores it, so the simulation keeps serving. In a terminal of its own,
-   Ctrl+C works too.
-
-5. **Develop and test, reading the simulation log after each call.** Call
-   the changed route on the real service with `postman request` or its
-   collection (see `api-testing`). Every request a member served is a
-   `request` event in `sim.ndjson`, with its `routeKey`, `method`, `path`,
-   `statusCode`, `duration` in milliseconds, and both bodies. Check that each
-   upstream call you expect is there, on the right member, with the status you
-   expect. Restart the service after a code change, and the simulation after a
-   mock edit. A response from the simulation itself means wiring, not code:
-   - `404 Unknown simulation member`: the path has no valid routeKey. The body
-     lists the valid ones. It isn't logged.
-   - `501`: an upstream call you haven't modelled yet. Add it to your own
-     mock. For a pulled one, fork it into `postman/mocks/` first.
-   - `502 Proxy error`: the member's handler crashed. See `sim.ndjson`.
-
-6. **Optionally, fail a dependency.** Worth doing when the change has to
-   survive an upstream that errors or is slow, but not required. Copy the
-   baseline `.sim.yaml`, give that member a condition, run the service
-   against it, and report what it did.
-
-   ```yaml
-     - routeKey: workspaces
-       path: ../.dependencies/mocks/workspace-service/config.yaml
-       scenarios:
-         - overrides:
-             conditions:
-               error: { status_code: 503 }
-   ```
-
-   The conditions are `latency`, `error`, `rate_limit` and `chaos`; the schema
-   has all four. Each applies to every route on that member, and the first to
-   fail a request ends it, so fail a dependency one way at a time. An injected
-   error, `429` or chaos failure never reaches the mock and isn't logged, so
-   judge it by what the service returned.
-
-7. **Ship.** Commit your mocks, the `.sim.yaml` files and
-   `.postman/resources.yaml`, but not `sim.ndjson`: it records request
-   headers and bodies, tokens included. After a clone,
-   `postman dependency install` restores the pulled mocks. Running the
-   simulation in CI is optional: start it as in step 4 before the test step,
-   and stop it after.
-
-## Critical Rules
-
-1. **The service under test is never a member.** If nothing real runs, use
-   `api-mocking`.
-2. **Unique routeKeys and an explicit `--port`.** A member's own `port:` is
-   ignored. Without `--port`, a busy 3000 silently moves to a random port.
-3. **Members share one `pm.state` store.** Two stateful mocks that use the
-   same key read and overwrite each other's records, so give each its own keys.
-4. **Only `overrides.conditions` inject faults.** A bare
-   `scenarios: - path: …/default.js` entry, as the Postman app writes, adds
-   none. `overrides.bypass` is ignored.
-5. **Never write or generate a mock for a dependency before searching its
-   owner's workspace.** A mock you write encodes your assumptions; the
-   owner's encodes their contract and their state.
-6. **Pull an owner's mock even when it's deployed.** Only members take faults
-   and log requests, and a private mock server needs an `x-api-key` your
-   service won't send. Never edit `postman/.dependencies/`:
-   `dependency update` overwrites it. To change a pulled mock, copy it into
-   `postman/mocks/` and call it a fork.
-7. **Confirm before a public deploy.** `simulation deploy` without
-   `--private`, or with `-y`, serves every member's mock to anyone with the
-   URL. Ask the user first; a local `run` needs no deploy at all.
-
-## Verification
-
-Starting is not the same as being used. Before you report, read the whole
-log: every member on the path should have `request` events with the statuses
-you expect. A member with none isn't wired or isn't on this path.
-
-Report:
-
-- **How the simulation performed:** for each member, the requests it served
-  and their statuses, and anything unexpected, such as a `501`, a `502` or a
-  member that was never called.
-- **Where each mock came from:** for each dependency, the owner workspace you
-  searched, what you found there, and whether its mock was pulled, forked or
-  generated here.
-- **What ran for real.**
+Without `--port`, a busy 3000 silently becomes a random port and every base
+URL points at nothing. It's up when `sim.ndjson` has a `listening` event; a
+start-up error is in `sim.err`. It listens on 127.0.0.1 only, so a caller in
+a container or VM can't reach it. Stop it with
+`pkill -INT -f "simulation run <file>"`; `kill $!` leaves it running,
+because the `postman` launcher ignores the signal. Nothing reloads: restart
+it after a mock edit.
+
+Point the caller at it without changing what the code does once deployed.
+Under path routing, swap only the host and keep the path:
+`https://catalog.internal.example.com/v2` becomes
+`http://127.0.0.1:4900/catalog/v2`; a dropped `/v2` is the usual cause of a
+`404`. A hard-coded base URL should read an environment variable with
+today's value as the default. If the code mustn't change at all,
+`simulation run --proxy` reaches the members through `HTTPS_PROXY` instead;
+it applies no conditions, and passes any other host through to the real
+service.
+
+## Reading the log
+
+A passing test command doesn't show the dependency path ran: read
+`sim.ndjson`. A member with no `request` events isn't wired or isn't on this
+path. Some answers come from the simulation itself and mean wiring, not code:
+
+- `404 Unknown simulation member`: a base URL lost its routeKey, or a client
+  sent the wrong `x-mock-slug`.
+- `501`, or a generated mock's `404 Endpoint not defined`: a call the mock
+  doesn't model yet.
+- `502 Proxy error`: the member's process died, usually an async throw in a
+  mock without `interceptRequests: true`. It stays down and logs nothing
+  until you restart; run that mock alone with `postman mock run` to see why.
+
+## When a dependency goes down
+
+Run it rather than reason from the code: each way of going down takes a
+different path through the caller, and a stub skips the client code that
+picks the path. Fail the one dependency in each shape that matters, a run
+each, with the others answering normally:
+
+- **Unreachable:** point its base URL at a port nothing listens on.
+- **Hanging:** give its member a `latency` condition longer than the
+  client's timeout. If the client sets none, that is the finding: Node's
+  `fetch` waits about five minutes, so say so rather than wait it out.
+- **An error with a JSON body:** the `error` condition.
+- **An error with an HTML body,** as proxies send: a copy of its mock whose
+  default handler answers `500` with HTML. No condition can do this.
+
+Only `overrides.conditions` on a member change it. Members always serve
+their default handler and ignore `x-mock-scenario`, and a `scenarios` entry
+with just a `path`, as the Postman app writes, does nothing. Fields and the
+other conditions are in [reference/sim_yaml_schema.md](reference/sim_yaml_schema.md). An injected
+failure never reaches the mock and isn't logged, so judge it by what the
+caller did, and check the other members' logs for side effects, such as
+stock reserved and never released.
+
+## Keeping it
+
+Commit your own mocks, the `.sim.yaml` and what `postman dependency add`
+changed, so the next session or CI reruns it in one command. Don't commit
+`postman/.dependencies/`: a copy goes stale when its owner updates theirs,
+and `postman dependency install` restores it. Don't commit `sim.ndjson`: the
+CLI redacts common auth headers and fields, but other headers and bodies are
+logged as sent. In CI, `postman dependency install` needs `POSTMAN_API_KEY`,
+and `postman collection run <collection> --simulation <file> --port <n>`
+starts and stops the simulation around a collection run.
+
+`postman simulation push` uploads every member to the workspace, so ask
+first. It fails on a pulled member, which keeps its owner's `id:`; copy it to
+`postman/mocks/` and delete that line.
+
+## Rules
+
+1. **What you're testing is never a member.** It must run for real, or the
+   test only proves the mock.
+2. **Pull the owner's mock even if it's deployed.** Only members take
+   conditions and appear in the log, and a private mock server wants a key
+   your code won't send.
+3. **`simulation deploy` is public by default,** unlike `mock deploy`. Ask
+   before deploying without `--private`; a local run needs no deploy.
+
+## Before you say it works
+
+Claim nothing until what you're testing actually ran: a service printed
+that it's listening and answered your request, or the frontend or test
+suite finished, and its calls are in the log. If its start command exited
+or hung, the run proved nothing; say so and why. Say what a pass proves,
+your code against these stand-ins rather than the real services, what still
+needs the real service before release, and where each mock came from,
+including when you skipped the owner's.

package/skills/api-simulation/reference/sim_yaml_schema.md

@@ -1,112 +1,50 @@
-# `.sim.yaml` schema
+# `.sim.yaml` reference
 
-The file `postman simulation run <filepath>` reads. It lives in
-`postman/simulations/NAME.sim.yaml`, beside the `postman/mocks/` folders it
-composes. `postman simulation create` writes one; the Postman app writes the
-same shape with `id:` fields added.
+Read this when writing or debugging a `.sim.yaml` by hand.
+`postman simulation create` writes the same shape; the Postman app adds
+`id:` fields, which a local run ignores.
 
 ```yaml
-id: 01a110fc-e3f5-77eb-af59-0a711a509960        # written by the Postman app
-simulation: checkout-dev
-routing: path
-mocks:
-  - id: 01a110fc-e344-72ca-9d0c-1bb83b99ec2a    # written by the Postman app
-    routeKey: products
-    path: ../mocks/products/config.yaml
-    scenarios:
-      - path: ../mocks/products/default.js
-  - routeKey: shopping-cart
-    path: ../mocks/shopping-cart/config.yaml
+simulation: checkout-dev               # required: the name
+routing: path                          # path (default) or header
+mocks:                                 # required: one entry per member
+  - routeKey: catalog                  # lowercase letters, digits, - and _
+    path: ../mocks/catalog/config.yaml # a mock's config.yaml, or a .js handler
+    scenarios:                         # optional: conditions, below
+      - overrides:
+          conditions:
+            latency: { delay_ms: 250 }
 ```
 
-## Root
+Run the `.sim.yaml` rather than a simulation id: a run by id drops every
+member's conditions (CLI 1.70.0–1.73.0).
 
-| Key | Required | Notes |
-| --- | --- | --- |
-| `simulation` | Yes | The simulation's name. |
-| `mocks` | Yes | Non-empty list of members. |
-| `routing` | No | `path` (default) or `header`. Any other value means `path`. |
-| `id` | No | The simulation's id in its workspace. A local run ignores it; leave it as written. |
-
-## Member
-
-| Key | Required | Notes |
-| --- | --- | --- |
-| `path` | Yes | The mock's `config.yaml`, or a `.js` handler. Relative paths resolve from the `.sim.yaml`'s folder, then from the working directory. `file` is a legacy alias. |
-| `routeKey` | No | The member's address. Lowercase letters, digits, `-` and `_`, at most 64 characters, unique per simulation. Defaults to the kebab-cased name. |
-| `name` | No | Display name. Defaults to the mock's folder name for a `config.yaml`, else the file name. Must be unique. |
-| `scenarios` | No | Fault conditions, below. |
-| `id` | No | The mock's id in its workspace. A local run ignores it. |
-| `port` | No | Ignored by `simulation run`, which gives each member an internal port. |
-
-## Routing
-
-- **`path`**: a member is at `http://localhost:<port>/<routeKey>`. The first
-  path segment selects the member and is stripped, so `/payments/cart` reaches
-  the `payments` member as `/cart`. Point the service's base URL for that
-  dependency at `http://localhost:<port>/<routeKey>`.
-- **`header`**: every member shares `http://localhost:<port>`, and the caller
-  picks one with an `x-mock-slug: <routeKey>` header. Use it only when the
-  caller can send that header.
-
-The router answers on its own, without reaching a member:
-
-| Response | Cause |
-| --- | --- |
-| `400 {"error":"Prefix the request path with the member routing token, e.g. /<routeKey>/path."}` | No routeKey under `path` routing: the path was `/`. Under `header` routing the error asks for the `x-mock-slug` header instead. |
-| `404 {"error":"Unknown simulation member '<x>'.","available":[...]}` | `<x>` isn't a routeKey. Under `path` routing this is often a client that dropped the base URL's path, so `<x>` is the endpoint's own first segment. |
-
-## `scenarios`
-
-Each entry is a selection, and only its `overrides.conditions` inject faults:
-
-```yaml
-scenarios:
-  - overrides:
-      conditions:
-        latency:
-          delay_ms: 250              # hold each request 250 ms, then go on
-        error:
-          status_code: 503           # every request fails with 503
-        rate_limit:
-          requests_per_minute: 30    # the 31st request in a minute gets 429
-        chaos:
-          failure_rate: 10           # 10% of requests fail with 500
-```
-
-This example sets all four to show their fields. They run in a fixed order,
-`latency`, `error`, `rate_limit`, `chaos`, whatever order you write them in,
-and the first to fail a request ends it: with `error` set, `rate_limit` and
-`chaos` never fire. Set one failure at a time, optionally with `latency`.
-
-The mock's default handler always serves, and conditions run in front of it.
-An entry with only `path` or `id`, as the Postman app writes, injects nothing.
-Unknown condition names are ignored, so a misspelled one fails silently.
-`overrides.bypass` prints a warning and is ignored.
-
-The legacy form `- type: error` with `config: { status_code: 503 }` is still
-accepted and validated strictly: an unknown `type` is an error.
+## Conditions
 
 | Condition | Field | Effect |
 | --- | --- | --- |
-| `latency` | `delay_ms` (positive number) | Holds each request that long, then serves it normally. |
-| `error` | `status_code` (integer 400–599) | Every request returns that status with `{"error":{...},"scenario":"error"}`. |
-| `rate_limit` | `requests_per_minute` (positive integer) | Requests over the limit return `429` with a `Retry-After` header and `"scenario":"rate_limit"`. |
-| `chaos` | `failure_rate` (0–100) | That percentage of requests, chosen at random, return `500` with `"scenario":"chaos"`. |
-
-A condition applies to every route on its member. To fail one endpoint while
-its siblings succeed, split that endpoint into its own mock.
-
-## `simulation run`
-
-| Flag | Notes |
-| --- | --- |
-| `-p, --port <n\|auto>` | Port to serve on. Defaults to 3000, and silently falls back to a free port if 3000 is busy. An explicit port fails instead. |
-| `--output ndjson` | One JSON event per line on stdout, instead of the default readable log; messages go to stderr: `listening` (port, url, each member's `routeKey` and `address`); `request`, one per request a member served (`routeKey`, `method`, `path`, `statusCode`, `duration` in ms, headers and bodies); and `summary` on shutdown (`requestsServed`, `requestsFailed`). The router's own `400` and `404`, and injected faults, aren't logged. |
-| `-e`, `-g`, `--dataset` | Environment, globals and datasets for every member, as for `mock run`. |
-| `--no-history` | Record nothing. A `.sim.yaml` linked to a workspace otherwise records each start there; an unlinked one records only with `-w` and `--simulation`. |
-
-The argument can also be a simulation id from a workspace: the members are
-fetched and served locally. All members start before the port opens, so one
-`listening` event means the whole simulation is up. A member that fails to
-start stops the run with `Error: ...` and exit code 1. Ctrl+C stops everything.
+| `latency` | `delay_ms` | Holds each request that long, then serves it. |
+| `error` | `status_code` (400–599) | Every request gets that status as JSON, with `"scenario":"error"`. |
+| `rate_limit` | `requests_per_minute` | Requests over the limit get `429` and `Retry-After`. |
+| `chaos` | `failure_rate` (0–100), optional `status_code` | That share of requests, at random, get `500` or that status. |
+
+- A condition covers every route on its member. To fail one endpoint while
+  its siblings succeed, give that endpoint a mock of its own.
+- They run in a fixed order (`latency`, `error`, `rate_limit`, `chaos`), and
+  the first to fail a request ends it, so set one failure at a time,
+  optionally with `latency`.
+- An unknown condition name is ignored silently, so a misspelling injects
+  nothing. `overrides.bypass` prints a warning and is ignored.
+- A `latency` request is served and logged, with a `duration` that leaves
+  out the delay.
+
+## The log (`--output ndjson`)
+
+One JSON event per line on stdout; notices go to stderr, so keep them apart.
+
+- `listening`: every member has started; a member that fails to start
+  stops the run with `Error: …` and exit code 1.
+- `request`: one per request a member served.
+- `summary`, on stop: `requestsServed` (under 400), `requestsFailed` (400
+  and up), `uptimeMs` and `signal`. Injected failures and the simulation's
+  own `400`, `404` and `502` aren't counted.