@postman/postman-plugin 0.2.2 → 0.4.0

package/skills/api-simulation/SKILL.md

@@ -0,0 +1,138 @@
+---
+name: api-simulation
+description: Runs your code for real against stand-ins for the HTTP services it calls, served together as one environment on one local port or one deployed URL, whether that code is a service you're changing, a frontend, a test suite or a demo. Use when you add or change code that calls other services and need to run it, test it, prove a fix or reproduce an upstream failure (timeouts, 5xx, 409, 429) end to end, locally or in CI, without the real services because they're unreachable, need credentials, cost money, have side effects or don't exist yet. Also answers what happens to a request, or to the data it writes, when a service it calls goes down, is slow or returns errors ("what happens to a booking if the email service is down?"), by running each failure instead of reading the code. Applies even when nobody says mock or simulation. To give a frontend or another team a stand-in for an API, use api-mocking. Covers `postman simulation`.
+---
+
+# API Simulation
+
+A simulation serves several mocks as one environment, each reached by its
+routeKey: as a path prefix by default, or as an `x-mock-slug` header when the
+caller's paths can't change.
+
+Run your code against it rather than stubbing its HTTP client in tests or
+calling the real services. A stub skips the client, config, timeouts and
+error handling that will run in production; the real services need
+credentials and leave side effects. A simulation runs that code over real
+HTTP, the same way in CI, and its log shows what each dependency was asked.
+
+Do only what the task needs. If a `.sim.yaml` in `postman/simulations/`
+already covers it, run that one. One request through the changed path proves
+a change; build failure variants only when asked how the code fails or when
+it must survive a failure.
+
+`postman simulation` needs CLI 1.70.0 or later (`postman --version`; an older
+CLI answers `postman simulation -h` with its general help).
+
+## Building it
+
+- **Members are the code's HTTP dependencies.** Databases, queues and caches
+  aren't; run them locally. **api-discovery** can find calls made through
+  shared code; ask about the repo by its `git remote` name, because a close
+  name matches another repo.
+- **Each member needs a mock: see `api-mocking`,** which looks for the owning
+  team's published mock before building one. For a question, one built from
+  the contract will do; one that invents responses can send the caller down a
+  path the real service never would. In a mock you build, answer `501` for calls this path
+  doesn't make: a `404` reads to the caller like a real "not found".
+- **Prefix each mock's `pm.state` keys with its name.** As of CLI 1.73.0,
+  members of a local run read each other's keys, so two mocks storing
+  `orders` overwrite each other.
+
+## Running it
+
+The run is a server that never exits, so a foreground command never returns.
+Start it in the background with an explicit port, its log in a file:
+
+```bash
+postman simulation run postman/simulations/checkout-dev.sim.yaml --port 4900 --output ndjson > sim.ndjson 2> sim.err &
+```
+
+Without `--port`, a busy 3000 silently becomes a random port and every base
+URL points at nothing. It's up when `sim.ndjson` has a `listening` event; a
+start-up error is in `sim.err`. It listens on 127.0.0.1 only, so a caller in
+a container or VM can't reach it. Stop it with
+`pkill -INT -f "simulation run <file>"`; `kill $!` leaves it running,
+because the `postman` launcher ignores the signal. Nothing reloads: restart
+it after a mock edit.
+
+Point the caller at it without changing what the code does once deployed.
+Under path routing, swap only the host and keep the path:
+`https://catalog.internal.example.com/v2` becomes
+`http://127.0.0.1:4900/catalog/v2`; a dropped `/v2` is the usual cause of a
+`404`. A hard-coded base URL should read an environment variable with
+today's value as the default. If the code mustn't change at all,
+`simulation run --proxy` reaches the members through `HTTPS_PROXY` instead;
+it applies no conditions, and passes any other host through to the real
+service.
+
+## Reading the log
+
+A passing test command doesn't show the dependency path ran: read
+`sim.ndjson`. A member with no `request` events isn't wired or isn't on this
+path. Some answers come from the simulation itself and mean wiring, not code:
+
+- `404 Unknown simulation member`: a base URL lost its routeKey, or a client
+  sent the wrong `x-mock-slug`.
+- `501`, or a generated mock's `404 Endpoint not defined`: a call the mock
+  doesn't model yet.
+- `502 Proxy error`: the member's process died, usually an async throw in a
+  mock without `interceptRequests: true`. It stays down and logs nothing
+  until you restart; run that mock alone with `postman mock run` to see why.
+
+## When a dependency goes down
+
+Run it rather than reason from the code: each way of going down takes a
+different path through the caller, and a stub skips the client code that
+picks the path. Fail the one dependency in each shape that matters, a run
+each, with the others answering normally:
+
+- **Unreachable:** point its base URL at a port nothing listens on.
+- **Hanging:** give its member a `latency` condition longer than the
+  client's timeout. If the client sets none, that is the finding: Node's
+  `fetch` waits about five minutes, so say so rather than wait it out.
+- **An error with a JSON body:** the `error` condition.
+- **An error with an HTML body,** as proxies send: a copy of its mock whose
+  default handler answers `500` with HTML. No condition can do this.
+
+Only `overrides.conditions` on a member change it. Members always serve
+their default handler and ignore `x-mock-scenario`, and a `scenarios` entry
+with just a `path`, as the Postman app writes, does nothing. Fields and the
+other conditions are in [reference/sim_yaml_schema.md](reference/sim_yaml_schema.md). An injected
+failure never reaches the mock and isn't logged, so judge it by what the
+caller did, and check the other members' logs for side effects, such as
+stock reserved and never released.
+
+## Keeping it
+
+Commit your own mocks, the `.sim.yaml` and what `postman dependency add`
+changed, so the next session or CI reruns it in one command. Don't commit
+`postman/.dependencies/`: a copy goes stale when its owner updates theirs,
+and `postman dependency install` restores it. Don't commit `sim.ndjson`: the
+CLI redacts common auth headers and fields, but other headers and bodies are
+logged as sent. In CI, `postman dependency install` needs `POSTMAN_API_KEY`,
+and `postman collection run <collection> --simulation <file> --port <n>`
+starts and stops the simulation around a collection run.
+
+`postman simulation push` uploads every member to the workspace, so ask
+first. It fails on a pulled member, which keeps its owner's `id:`; copy it to
+`postman/mocks/` and delete that line.
+
+## Rules
+
+1. **What you're testing is never a member.** It must run for real, or the
+   test only proves the mock.
+2. **Pull the owner's mock even if it's deployed.** Only members take
+   conditions and appear in the log, and a private mock server wants a key
+   your code won't send.
+3. **`simulation deploy` is public by default,** unlike `mock deploy`. Ask
+   before deploying without `--private`; a local run needs no deploy.
+
+## Before you say it works
+
+Claim nothing until what you're testing actually ran: a service printed
+that it's listening and answered your request, or the frontend or test
+suite finished, and its calls are in the log. If its start command exited
+or hung, the run proved nothing; say so and why. Say what a pass proves,
+your code against these stand-ins rather than the real services, what still
+needs the real service before release, and where each mock came from,
+including when you skipped the owner's.

package/skills/api-simulation/reference/sim_yaml_schema.md

@@ -0,0 +1,50 @@
+# `.sim.yaml` reference
+
+Read this when writing or debugging a `.sim.yaml` by hand.
+`postman simulation create` writes the same shape; the Postman app adds
+`id:` fields, which a local run ignores.
+
+```yaml
+simulation: checkout-dev               # required: the name
+routing: path                          # path (default) or header
+mocks:                                 # required: one entry per member
+  - routeKey: catalog                  # lowercase letters, digits, - and _
+    path: ../mocks/catalog/config.yaml # a mock's config.yaml, or a .js handler
+    scenarios:                         # optional: conditions, below
+      - overrides:
+          conditions:
+            latency: { delay_ms: 250 }
+```
+
+Run the `.sim.yaml` rather than a simulation id: a run by id drops every
+member's conditions (CLI 1.70.0–1.73.0).
+
+## Conditions
+
+| Condition | Field | Effect |
+| --- | --- | --- |
+| `latency` | `delay_ms` | Holds each request that long, then serves it. |
+| `error` | `status_code` (400–599) | Every request gets that status as JSON, with `"scenario":"error"`. |
+| `rate_limit` | `requests_per_minute` | Requests over the limit get `429` and `Retry-After`. |
+| `chaos` | `failure_rate` (0–100), optional `status_code` | That share of requests, at random, get `500` or that status. |
+
+- A condition covers every route on its member. To fail one endpoint while
+  its siblings succeed, give that endpoint a mock of its own.
+- They run in a fixed order (`latency`, `error`, `rate_limit`, `chaos`), and
+  the first to fail a request ends it, so set one failure at a time,
+  optionally with `latency`.
+- An unknown condition name is ignored silently, so a misspelling injects
+  nothing. `overrides.bypass` prints a warning and is ignored.
+- A `latency` request is served and logged, with a `duration` that leaves
+  out the delay.
+
+## The log (`--output ndjson`)
+
+One JSON event per line on stdout; notices go to stderr, so keep them apart.
+
+- `listening`: every member has started; a member that fails to start
+  stops the run with `Error: …` and exit code 1.
+- `request`: one per request a member served.
+- `summary`, on stop: `requestsServed` (under 400), `requestsFailed` (400
+  and up), `uptimeMs` and `signal`. Injected failures and the simulation's
+  own `400`, `404` and `502` aren't counted.

package/skills/api-testing/SKILL.md

@@ -1,104 +1,57 @@
 ---
 name: api-testing
-description: Runs tests against an API from the command line — a single ad-hoc request, a full collection of pm.test assertions, or matching real captured app traffic against a collection contract. Use when the user asks to "test this endpoint," "run this collection," "check the API still works," or "verify my app's requests match the contract." Covers `postman request`, `postman collection run`, and `postman application test`. Depends on bootstrap when the target is a cloud collection or workspace-bound environment; a bare URL or local collection needs nothing from bootstrap.
+description: Use when testing an endpoint, running collection assertions, or checking real application traffic against an API contract. Reuses saved requests and reports what live, mocked, or captured-traffic results actually establish. Use for requests such as "test this endpoint".
 ---
 
 # API Testing
 
-## Overview
-
-Three tools, matched to what already exists:
-
-| Have | Use |
-| --- | --- |
-| Just a URL to check, with no saved request | `postman request` |
-| A request already saved in a collection | `postman collection run <collection-path> -i <request-id-name-or-path>` |
-| A collection with `pm.test` assertions saved in it | `postman collection run` |
-| A real app (browser flow, CLI, service) whose traffic should match a collection's contract | `postman application test` |
-
-Don't reach for the heavier tool when the lighter one already answers the
-question — a one-off endpoint check doesn't need a collection, and a
-collection run doesn't need Playwright.
-
-## `postman request` — over curl, not instead of testing
-
-A single request with Postman's resolution built in: `-e` resolves
-`{{variables}}` from an environment file the same way a collection run
-would, `--auth-*` flags cover basic/bearer/digest/oauth/aws/etc. without
-hand-building headers, and `--retry`/`--timeout` handle flaky endpoints.
-`--script-post-request` can run `pm.test(...)` assertions inline — the exit
-code counts *failed assertions*, not just HTTP status, so a 200 with a
-failing test still exits nonzero. Useful for a quick check or a CI health
-check; not the place to accumulate assertions that should outlive one
-command — those belong saved in a collection.
-
-Before constructing a URL, headers, auth, and body on the command line, look
-for a matching `*.request.yaml` under `postman/collections/`. If it exists,
-execute the saved request through its collection:
-
-```bash
-postman collection run "postman/collections/Orders API" -i "create order"
-```
-
-Do not copy the saved YAML fields into `postman request`; that bypasses the
-collection's inherited variables, auth, scripts, and maintained payload. The
-`postman request` positional target is a URL, not a `.request.yaml` path. When
-there is no saved request but its body already lives in a separate file, keep
-the file as the source of truth with `--body @path/to/payload.json` instead of
-inlining its contents.
-
-## `collection run` — the assertion suite
-
-Debugging a request's behavior during a run (why a `pm.test` failed, what a
-request actually sends) means reading the request's own YAML — see the
-`collection-schema-v3` skill for that file format before assuming a field's
-shape.
-
-Runs every request in a collection (or a subset via `-i`, repeatable),
-executing whatever `pm.test` scripts are already saved in it.
-`-d`/`--iteration-data` (or the beta `--iteration-data-dataset` +
-`--iteration-data-view` pair) drives data-driven runs across a CSV/JSON
-file or a Postman Dataset. `-r junit,html` for CI-consumable reports.
-`--use-mock`/`--mock` redirects the run at a mock instead of a real backend
-(see the `api-mocking` skill) — reach for this to test request/assertion
-logic without depending on a live service.
-
-## `application test` — contract-matching real traffic
-
-This doesn't send its own requests. It runs your existing test command
-(`--command "npx playwright test"`, or config-driven via
-`postman.config.cjs` targets), captures the network traffic that command
-generates, and matches/asserts it against your Postman collections —
-answering "did my app's actual calls conform to the contract," not "does
-this endpoint respond correctly." `--capture-only` skips the matching step
-entirely and just exports what was captured as a new v3 collection,
-organized by host — a way to bootstrap a collection from real traffic
-rather than authoring one from scratch. Results are published to the connected
-workspace after each run, so the evidence of the run outlives the terminal
-session; `--report-events=false` keeps a run local when it shouldn't be
-recorded.
-
-## Critical Rules
-
-1. **Don't reach for `application test` for something a plain
-   `collection run` covers.** It exists specifically for matching *captured*
-   app traffic (via Playwright or similar), not for driving requests itself.
-2. **`postman request`'s exit code reflects failed `pm.test` assertions, not
-   HTTP status alone.** A nonzero exit on a 200 response usually means a
-   post-request script assertion failed, not a network problem.
-3. **Assertions meant to be reused belong in the collection, not on the
-   command line.** A `--script-post-request` test on `collection run` runs
-   once and leaves nothing for the next person; save it as a `pm.test` in
-   the request instead.
-4. **`--use-mock` is the way to test without a live backend** — prefer it
-   over standing up ad hoc fakes or skipping tests that need a dependency.
-5. **Reuse a saved request instead of reconstructing it.** If a matching v3
-   request exists, use `collection run <collection-path> -i <request>`; reserve
-   `postman request` for genuinely ad-hoc requests.
+Saved requests preserve the auth, payloads, variables, and assertions that make
+an API check reproducible. The useful result is evidence about an intended
+guarantee; a successful HTTP exchange alone does not establish correctness.
+
+1. Choose the smallest execution mode that answers the question:
+
+   | Evidence needed | Use |
+   | --- | --- |
+   | One endpoint with no saved request | `postman request <url>` |
+   | Behavior of an existing saved request | `postman collection run <collection-path> -i <request-id-name-or-path>` |
+   | Reusable regression checks | `postman collection run <path-or-id>` |
+   | Whether a real app's calls match the contract | `postman application test` |
+
+2. Reuse a matching request under `postman/collections/` before reconstructing
+   its HTTP fields. Running it through the collection retains inherited auth,
+   variables, and scripts. `postman request` takes a URL, not a request YAML
+   path; for an unsaved request with an existing payload file, `--body @path`
+   preserves that file as the source. See **collection-schema-v3** when
+   editing the saved request.
+3. Derive assertions from the contract and the behavior at risk. A test that
+   merely repeats the current implementation can bless its bug. Preserve
+   existing guarantees and add checks for the changed behavior; put reusable
+   `pm.test` assertions in the collection so later runs and teammates inherit
+   them. Inline `--script-post-request` assertions suit a one-off check.
+4. Choose a real or mocked target according to the claim. `--use-mock` lets a
+   collection run exercise request and assertion logic without a live
+   backend; `--mock` alone only starts one. `--use-mock` replaces the
+   variable's whole value, base path included (see **api-mocking**). Passing
+   against that mock does not verify the service the mock stands in for.
+5. Reuse environment resolution with `-e` and **variables**. Use a CSV/JSON
+   file through `-d` for simple iteration; use **datasets** when the inputs
+   need queries, joins, or reusable views. Export reports such as
+   `-r junit,html` when **ci-integration** needs evidence beyond terminal output.
+6. Use `application test` when the application's actual traffic is the subject.
+   It runs an existing command, such as `--command "npx playwright test"`,
+   captures its traffic, and matches it against collections. `--capture-only`
+   instead exports a v3 collection organized by host; that captures behavior
+   without validating it. Results are recorded in the connected workspace by
+   default; `--report-events=false` keeps the run local when required.
+
+Use command-specific `-h` for supported options and **bootstrap** for unresolved
+CLI setup or authentication when using workspace resources. A bare URL or
+local collection does not require a workspace binding.
 
 ## Verification
 
-State the actual result (pass/fail counts, exit code), not just that the
-command ran. For `application test`, state whether it ran in match mode or
-`--capture-only` — they answer different questions and shouldn't be
-reported the same way.
+Report the target, tested behavior, assertion counts, and exit code. The exit
+code reflects failed assertions, not HTTP status alone: a 200 can fail a test.
+For application testing, distinguish matching from capture-only mode, and for
+mocked runs state which real behavior remains unverified.