@postman/postman-plugin 0.4.1-rc.0 → 0.5.0-rc.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +7 -1
- package/mcp.pi.json +2 -2
- package/package.json +1 -1
- package/skills/api-engineer/SKILL.md +16 -12
- package/skills/api-simulation/SKILL.md +66 -16
- package/skills/api-simulation/reference/context_file.md +159 -0
- package/skills/api-simulation/reference/sim_yaml_schema.md +33 -3
- package/skills/application-testing/SKILL.md +144 -0
- package/skills/bootstrap/SKILL.md +11 -0
package/README.md
CHANGED
|
@@ -73,7 +73,7 @@ codex plugin add postman@postman
|
|
|
73
73
|
|
|
74
74
|
These commands track `main`; review the session-start hook in Codex after a
|
|
75
75
|
manual install. The npx installer also approves the hook after each install or
|
|
76
|
-
update
|
|
76
|
+
update.
|
|
77
77
|
|
|
78
78
|
### Cursor
|
|
79
79
|
|
|
@@ -232,6 +232,12 @@ exercise failure paths.
|
|
|
232
232
|
|
|
233
233
|
## How Postman builds context
|
|
234
234
|
|
|
235
|
+
`postman init` is idempotent and safe to run in any project. Run it before API
|
|
236
|
+
engineering work to create the relevant local directories and make Postman
|
|
237
|
+
skills available to guide the task, even if the project is already initialized.
|
|
238
|
+
See [bootstrap](https://github.com/postmanlabs/postman-plugin/blob/main/skills/bootstrap/SKILL.md#start-with-postman-init) for setup
|
|
239
|
+
options and workspace binding.
|
|
240
|
+
|
|
235
241
|
Work done in a Postman folder produces artifacts: collection runs, lint
|
|
236
242
|
results, request history, mock deployments. When that folder is connected to a
|
|
237
243
|
Postman workspace after `postman init`, those artifacts are published to it as
|
package/mcp.pi.json
CHANGED
|
@@ -6,8 +6,8 @@
|
|
|
6
6
|
"description": "Postman workspaces, collections, specs, environments, mocks and monitors",
|
|
7
7
|
"headers": {
|
|
8
8
|
"X-Source": "postman-pi-plugin",
|
|
9
|
-
"X-Plugin-Version": "0.
|
|
10
|
-
"User-Agent": "postman-pi-plugin/0.
|
|
9
|
+
"X-Plugin-Version": "0.5.0-rc.0",
|
|
10
|
+
"User-Agent": "postman-pi-plugin/0.5.0-rc.0"
|
|
11
11
|
}
|
|
12
12
|
}
|
|
13
13
|
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@postman/postman-plugin",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.5.0-rc.0",
|
|
4
4
|
"description": "Postman's API engineering skills for coding agents: a Pi package, and an npx installer that sets up Claude Code, Codex, Cursor, Kimi Code, Factory Droid, OpenCode and Pi.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "Apache-2.0",
|
|
@@ -30,8 +30,7 @@ guesswork each time work resumes.
|
|
|
30
30
|
even when the provider is already chosen: those details help you assess fit and
|
|
31
31
|
implement reliably without relying on remembered API behavior or loading a
|
|
32
32
|
full vendor specification.
|
|
33
|
-
|
|
34
|
-
Context Graph can identify consumers and dependencies outside the checkout,
|
|
33
|
+
Context Graph can identify consumers and dependencies outside the current repo,
|
|
35
34
|
helping you focus investigation before traversing more repositories.
|
|
36
35
|
5. When you run or test code that calls other services, or are asked how it
|
|
37
36
|
behaves when one of them fails, use **api-simulation**, which runs that code
|
|
@@ -44,20 +43,25 @@ guesswork each time work resumes.
|
|
|
44
43
|
be diffed, reviewed, and reused in Git and CI. Use **bootstrap** when setup or
|
|
45
44
|
workspace bindings are unresolved: reuse an existing local layout, pull the
|
|
46
45
|
current service's existing workspace, or initialize a new project when neither
|
|
47
|
-
exists.
|
|
48
|
-
|
|
49
|
-
|
|
46
|
+
exists.
|
|
47
|
+
7. `postman init` gives API engineering work a ready state. It scaffolds local structure: `postman/`
|
|
48
|
+
directories for API artifacts. It installs skills in `postman/skills` folder with guidance on designing, implementing, and validating APIs.
|
|
49
|
+
The command is idempotent and safe to rerun in an initialized project.
|
|
50
|
+
It creates an agent-scoped session that lets you record work without signing
|
|
51
|
+
in. You can claim that work later if you create an account.
|
|
52
|
+
8. Another service's collection, environment, or mock can be added as a dependency
|
|
50
53
|
without replacing this repository's owning workspace binding. Use
|
|
51
54
|
**api-discovery** to bring in the relevant resource and preserve that separation.
|
|
52
|
-
|
|
53
|
-
for
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
55
|
+
9. Publishing local changes to the bound workspace makes changes available
|
|
56
|
+
for other agent sessions and teammates. Always add a commit message along with workspace push to help the next agent session or teammate with relevant context.
|
|
57
|
+
10. A Postman workspace is a place to capture context. Use this information as prior
|
|
58
|
+
knowledge of work done before. For example,
|
|
59
|
+
`postman workspace context --type workspacePush --limit 5 --brief` lists the
|
|
60
|
+
last five workspace push messages and log.
|
|
61
|
+
11. When requests need credentials, Postman secret variables let them resolve the
|
|
58
62
|
credentials without exposing their values to the agent or committed files.
|
|
59
63
|
See **variables**.
|
|
60
|
-
|
|
64
|
+
12. When implementing or debugging webhooks, use **webhook-testing** to
|
|
61
65
|
capture what the sender actually delivers.
|
|
62
66
|
Can also be used when the need is to forward incoming events
|
|
63
67
|
to a local handler and replay captured requests after fixes;
|
|
@@ -16,9 +16,12 @@ credentials and leave side effects. A simulation runs that code over real
|
|
|
16
16
|
HTTP, the same way in CI, and its log shows what each dependency was asked.
|
|
17
17
|
|
|
18
18
|
Do only what the task needs. If a `.sim.yaml` in `postman/simulations/`
|
|
19
|
-
already covers it, run that one
|
|
20
|
-
|
|
21
|
-
it
|
|
19
|
+
already covers it, run that one, and read the `<name>.context.md` beside it
|
|
20
|
+
first: it holds the commands, the preconditions and the failures earlier runs
|
|
21
|
+
hit, so reading it costs less than rebuilding any of that from the `.sim.yaml`
|
|
22
|
+
and the handlers. One request through the changed path proves a change; build
|
|
23
|
+
failure variants only when asked how the code fails or when it must survive a
|
|
24
|
+
failure.
|
|
22
25
|
|
|
23
26
|
`postman simulation` needs CLI 1.70.0 or later (`postman --version`; an older
|
|
24
27
|
CLI answers `postman simulation -h` with its general help).
|
|
@@ -34,6 +37,11 @@ CLI answers `postman simulation -h` with its general help).
|
|
|
34
37
|
the contract will do; one that invents responses can send the caller down a
|
|
35
38
|
path the real service never would. In a mock you build, answer `501` for calls this path
|
|
36
39
|
doesn't make: a `404` reads to the caller like a real "not found".
|
|
40
|
+
- **Bring in each mock you don't own as a dependency first, then build the
|
|
41
|
+
simulation over them.** `postman dependency add mock <name-or-id>` declares
|
|
42
|
+
it in `.postman/resources.yaml` and materialises it under
|
|
43
|
+
`postman/.dependencies/`, and the member's `path:` points there; `-w
|
|
44
|
+
<workspace>` resolves a name from the workspace that owns it.
|
|
37
45
|
- **Prefix each mock's `pm.state` keys with its name.** As of CLI 1.73.0,
|
|
38
46
|
members of a local run read each other's keys, so two mocks storing
|
|
39
47
|
`orders` overwrite each other.
|
|
@@ -41,18 +49,17 @@ CLI answers `postman simulation -h` with its general help).
|
|
|
41
49
|
## Running it
|
|
42
50
|
|
|
43
51
|
The run is a server that never exits, so a foreground command never returns.
|
|
44
|
-
Start it in the background
|
|
52
|
+
Start it in the background, its log in a file:
|
|
45
53
|
|
|
46
54
|
```bash
|
|
47
55
|
postman simulation run postman/simulations/checkout-dev.sim.yaml --port 4900 --output ndjson > sim.ndjson 2> sim.err &
|
|
48
56
|
```
|
|
49
57
|
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
`pkill -INT -f "simulation run
|
|
55
|
-
because the `postman` launcher ignores the signal. Nothing reloads: restart
|
|
58
|
+
`--port` is optional. It's up when `sim.ndjson` has a `listening` event; a
|
|
59
|
+
start-up error is in `sim.err`. That event carries the bound port and each
|
|
60
|
+
member's address, so take the base URLs from it rather than from the port you
|
|
61
|
+
asked for. Stop it with
|
|
62
|
+
`pkill -INT -f "simulation run .*<file>.sim.yaml"`. Nothing reloads: restart
|
|
56
63
|
it after a mock edit.
|
|
57
64
|
|
|
58
65
|
Point the caller at it without changing what the code does once deployed.
|
|
@@ -60,10 +67,30 @@ Under path routing, swap only the host and keep the path:
|
|
|
60
67
|
`https://catalog.internal.example.com/v2` becomes
|
|
61
68
|
`http://127.0.0.1:4900/catalog/v2`; a dropped `/v2` is the usual cause of a
|
|
62
69
|
`404`. A hard-coded base URL should read an environment variable with
|
|
63
|
-
today's value as the default. If the code mustn't change at all,
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
70
|
+
today's value as the default. If the code mustn't change at all, run it
|
|
71
|
+
behind the proxy instead, below.
|
|
72
|
+
|
|
73
|
+
## Through a proxy, when no base URL can change
|
|
74
|
+
|
|
75
|
+
`simulation run <file.sim.yaml> --proxy` puts the members behind a local
|
|
76
|
+
forward proxy that answers for the dependencies' own hostnames, so the code
|
|
77
|
+
keeps the base URLs it ships with. Any other host passes through to the real
|
|
78
|
+
service. Reach for it when hostnames are hard-coded, built from config the
|
|
79
|
+
change shouldn't touch, or owned by an SDK — and because nothing it answers
|
|
80
|
+
for has to resolve, an internal hostname with no DNS record works.
|
|
81
|
+
|
|
82
|
+
Read `postman simulation run --help` before driving it. The flags, what the
|
|
83
|
+
caller's own process needs in order to reach the proxy, and what to do about a
|
|
84
|
+
client that ignores it are all there and change with the CLI, so take them
|
|
85
|
+
from the command rather than from here.
|
|
86
|
+
|
|
87
|
+
Two things decide whether a proxied run proves anything. A member with no
|
|
88
|
+
`origin:` in the `.sim.yaml` is never matched, so its traffic reaches the real
|
|
89
|
+
service and a run can come up having intercepted nothing — `simulation create`
|
|
90
|
+
copies that field from the mock's manifest, so editing the manifest alone
|
|
91
|
+
won't fix it ([reference/sim_yaml_schema.md](reference/sim_yaml_schema.md)).
|
|
92
|
+
And conditions don't apply here: member `scenarios` are ignored, so failures
|
|
93
|
+
need the one-port run.
|
|
67
94
|
|
|
68
95
|
## Reading the log
|
|
69
96
|
|
|
@@ -102,10 +129,27 @@ failure never reaches the mock and isn't logged, so judge it by what the
|
|
|
102
129
|
caller did, and check the other members' logs for side effects, such as
|
|
103
130
|
stock reserved and never released.
|
|
104
131
|
|
|
132
|
+
## Its context document
|
|
133
|
+
|
|
134
|
+
Every simulation is created with a `<name>.context.md` beside it, holding what
|
|
135
|
+
running this one has taught: the commands that start it and prove it's up, the
|
|
136
|
+
preconditions and what breaks when one fails, what it takes for granted about
|
|
137
|
+
the dependencies themselves, the data seeded to test with, the failures earlier
|
|
138
|
+
runs already hit, and what a passing run still doesn't prove. The `.sim.yaml` says what the simulation is, not how it behaves, so
|
|
139
|
+
this is the file that saves a session the search. A new one is mostly
|
|
140
|
+
`@todo`s, which are the questions a first real run answers.
|
|
141
|
+
|
|
142
|
+
Every line passes one test — the repo doesn't already state it — so it stays
|
|
143
|
+
about a screenful, and a claim carries the date and the run that established
|
|
144
|
+
it, because the log it came from isn't committed. Its sections and conventions
|
|
145
|
+
are in [reference/context_file.md](reference/context_file.md); read that
|
|
146
|
+
before writing it back.
|
|
147
|
+
|
|
105
148
|
## Keeping it
|
|
106
149
|
|
|
107
|
-
Commit your own mocks, the `.sim.yaml
|
|
108
|
-
changed, so the next session or CI reruns it in one
|
|
150
|
+
Commit your own mocks, the `.sim.yaml`, its context document and what
|
|
151
|
+
`postman dependency add` changed, so the next session or CI reruns it in one
|
|
152
|
+
command. Don't commit
|
|
109
153
|
`postman/.dependencies/`: a copy goes stale when its owner updates theirs,
|
|
110
154
|
and `postman dependency install` restores it. Don't commit `sim.ndjson`: the
|
|
111
155
|
CLI redacts common auth headers and fields, but other headers and bodies are
|
|
@@ -136,3 +180,9 @@ or hung, the run proved nothing; say so and why. Say what a pass proves,
|
|
|
136
180
|
your code against these stand-ins rather than the real services, what still
|
|
137
181
|
needs the real service before release, and where each mock came from,
|
|
138
182
|
including when you skipped the owner's.
|
|
183
|
+
|
|
184
|
+
Then write the run back into `<name>.context.md`, so the next session starts
|
|
185
|
+
where this one ended: the `@todo`s it answered, whatever it found wrong, a
|
|
186
|
+
precondition you had to discover, what broke and why, and which members served
|
|
187
|
+
or stayed silent — the last is a line every run adds, so a member silent across
|
|
188
|
+
several runs reads as a pattern rather than a one-off.
|
|
@@ -0,0 +1,159 @@
|
|
|
1
|
+
# `<name>.context.md`
|
|
2
|
+
|
|
3
|
+
The handoff document beside a simulation: `postman/simulations/checkout.sim.yaml`
|
|
4
|
+
gets `checkout.context.md`, and a foldered simulation keeps both in its folder.
|
|
5
|
+
`simulation create` and the Postman app write it from what they already know,
|
|
6
|
+
`simulation update` renames it with the simulation, and `simulation delete`
|
|
7
|
+
removes it. An existing one is never overwritten, because it holds what earlier
|
|
8
|
+
sessions learned.
|
|
9
|
+
|
|
10
|
+
It is prose for whoever works here next, human or agent. Nothing parses it, so
|
|
11
|
+
there is no schema to satisfy — but the generated headings and the
|
|
12
|
+
`<!-- begin:observations -->` markers are what a reader and the next write
|
|
13
|
+
expect to find, so keep them.
|
|
14
|
+
|
|
15
|
+
Keep it to the generated sections plus `## What didn't work`, in Markdown,
|
|
16
|
+
under about a screenful, so the line that matters is the one a reader lands on
|
|
17
|
+
rather than one they have to search for.
|
|
18
|
+
|
|
19
|
+
## What the generated file contains
|
|
20
|
+
|
|
21
|
+
A fresh file states the simulation's name, the date, its members and the hosts
|
|
22
|
+
they stand in for, and the run command for this simulation's own path.
|
|
23
|
+
Everything needing someone who has actually run it is left as an `@todo` HTML
|
|
24
|
+
comment, so a half-filled file reads as half-filled instead of as fact.
|
|
25
|
+
|
|
26
|
+
| Section | Holds |
|
|
27
|
+
| --- | --- |
|
|
28
|
+
| Header | Generation date and what the mocks came from. |
|
|
29
|
+
| `## Purpose` | What the simulation stands in for and what work it unblocks. Generated as the member list and a pointer to the handlers; needs hindsight, so it is the last thing to rewrite. |
|
|
30
|
+
| `## Execution` | The commands, in order: start the simulation, start the service pointed at it, prove it is up, count what got called, stop. The generated block leaves the service's own command, the proof request, the port and the log path as `@todo`. |
|
|
31
|
+
| `## Assumptions` | Every precondition someone had to discover, what the simulation takes for granted about the dependencies' own behaviour, the seeded data worth reusing, and what the simulation cannot reproduce. Each is a dated claim, the consequence of it not holding, and how it was established. |
|
|
32
|
+
| `## Observations` | One dated line per run, newest first, between the `begin`/`end` markers: which members took traffic and which took none. A summary, not a request log, and a failure goes in `## What didn't work` instead. |
|
|
33
|
+
| `## What didn't work` | Failures this simulation has already hit, a dated line each: what broke, the signature you'd recognize it by, where it surfaced, and the cause. Not generated — add the heading at the end, after `## Observations`, the first time a run gives you one. |
|
|
34
|
+
|
|
35
|
+
`## Observations` and `## What didn't work` both accrete, so a run is visible
|
|
36
|
+
in the file after the next one; `## Purpose`, `## Execution` and
|
|
37
|
+
`## Assumptions` are rewritten in place, since they describe the simulation as
|
|
38
|
+
it stands rather than a run. Each accreting section is bounded: `## What didn't
|
|
39
|
+
work` keeps the ten most recent, a line staying until a run shows it fixed, and
|
|
40
|
+
`## Observations` the five most recent runs. Past that the oldest go.
|
|
41
|
+
|
|
42
|
+
A run that keeps its line is how a pattern becomes visible — a member silent
|
|
43
|
+
for three runs running is worth more than the same fact seen once.
|
|
44
|
+
|
|
45
|
+
Don't restate what the generated file already says: its `## Assumptions` ships
|
|
46
|
+
with the ones that hold for every simulation.
|
|
47
|
+
|
|
48
|
+
## What earns a line
|
|
49
|
+
|
|
50
|
+
Write what running it taught you, and only that. Each line passes one test:
|
|
51
|
+
the repo doesn't already state it. That leaves five kinds of thing, and they
|
|
52
|
+
are the file's whole value:
|
|
53
|
+
|
|
54
|
+
- **A precondition you had to discover** — the setting, the service, the
|
|
55
|
+
cleanup a run needs before it works at all. In `## Assumptions`.
|
|
56
|
+
- **A failure and its cause**, with where it surfaced, so the next session
|
|
57
|
+
recognizes it instead of debugging it again. In `## What didn't work`.
|
|
58
|
+
- **What the run takes for granted about a dependency itself** — that an
|
|
59
|
+
operation is idempotent, that a timeout means nothing committed, what order
|
|
60
|
+
events arrive in. A mock answers these whether or not anyone checked, so its
|
|
61
|
+
answer is a belief your code now relies on. In `## Assumptions`.
|
|
62
|
+
- **What the simulation can't reproduce, and what nobody has settled**, so a
|
|
63
|
+
passing run is read for what it actually proves. In `## Assumptions`.
|
|
64
|
+
- **What the last run served**: which members took traffic, which stayed
|
|
65
|
+
silent, and why. In `## Observations`.
|
|
66
|
+
|
|
67
|
+
For anything the repo does state — routes and request shapes, valid actions,
|
|
68
|
+
auth mechanics, each mock's hosts, config flags — name the file that holds it,
|
|
69
|
+
the way the generated `## Purpose` points at the handlers, and let the reader
|
|
70
|
+
go there. One source stays true; a copy here drifts, and this file gets
|
|
71
|
+
committed, so a named file is also how a credential stays out of it.
|
|
72
|
+
|
|
73
|
+
Write statements that stay true between runs. Counts, timings and coverage
|
|
74
|
+
belong to the run that produced them, so leave them in the log and keep the
|
|
75
|
+
pattern here: "every member served except `workspace`" rather than a tally.
|
|
76
|
+
|
|
77
|
+
## How a line carries its evidence
|
|
78
|
+
|
|
79
|
+
The log a claim came from isn't committed, so the line is all the next session
|
|
80
|
+
gets. Date every claim in `## Assumptions` and `## What didn't work`, and say
|
|
81
|
+
where it came from when a run didn't show it: `(observed 2026-10-22)` for what
|
|
82
|
+
a run established, `(reasoned 2026-10-22, src/payments/client.ts)` for what was
|
|
83
|
+
read rather than run.
|
|
84
|
+
|
|
85
|
+
A failure also carries the signature the next session will recognize it by —
|
|
86
|
+
the error text, the status, the member that sent it — since that is what
|
|
87
|
+
someone matches a new failure against. Quote the error itself, never a
|
|
88
|
+
response body. For the rest, name the committed file that holds it: the
|
|
89
|
+
handler that threw, the mock example captured from the real service, the test
|
|
90
|
+
that reproduces it. Not the run's log, which is wherever that run redirected
|
|
91
|
+
output and which the next run overwrites.
|
|
92
|
+
|
|
93
|
+
A good `## Assumptions` is a claim, its consequence and its date; a good
|
|
94
|
+
`## Observations` is a few lines; a good `## What didn't work` is one line per
|
|
95
|
+
failure:
|
|
96
|
+
|
|
97
|
+
```markdown
|
|
98
|
+
## Assumptions
|
|
99
|
+
|
|
100
|
+
- **`payments` treats a retried charge as idempotent on `Idempotency-Key`.**
|
|
101
|
+
If it doesn't, the retry path charges twice. Nobody has settled it and the
|
|
102
|
+
mock answers as though it does. (reasoned 2026-10-22, the mock's handler)
|
|
103
|
+
- **Redis holds no `EVENT_BATCH` key when a run starts.** A leftover one
|
|
104
|
+
crashes the aggregation flush before any request is sent. (observed
|
|
105
|
+
2026-10-22)
|
|
106
|
+
|
|
107
|
+
<!-- begin:observations -->
|
|
108
|
+
2026-10-22: every member served except `workspace`, which the service never
|
|
109
|
+
calls in development, and `god`, which models no endpoints.
|
|
110
|
+
2026-10-18: same, and `payments` served only `POST /charges`.
|
|
111
|
+
<!-- end:observations -->
|
|
112
|
+
|
|
113
|
+
## What didn't work
|
|
114
|
+
|
|
115
|
+
- 2026-10-22: the aggregation flush crashed on start —
|
|
116
|
+
`TypeError: batch.map is not a function` in `src/events/flush.ts`, from a
|
|
117
|
+
stale Redis `EVENT_BATCH` key an earlier run left behind. Not a simulation
|
|
118
|
+
error, so the log shows nothing.
|
|
119
|
+
- 2026-09-30: `payments` answered `501` for `POST /refunds` — the mock models
|
|
120
|
+
no refund path, so a run that reaches it proves nothing about refunds.
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
## Working with it
|
|
124
|
+
|
|
125
|
+
Read it before starting. An observed claim holds until your run contradicts
|
|
126
|
+
it; a reasoned one is a hypothesis, so check it, cheaply, when your run depends
|
|
127
|
+
on it. Check the assumptions and anything recorded about what a run doesn't
|
|
128
|
+
prove before concluding that something works. A claim this run contradicts is
|
|
129
|
+
a finding to report, not to quietly route around.
|
|
130
|
+
|
|
131
|
+
What it already answers, it answers: `## Execution` holds commands that ran,
|
|
132
|
+
and `## Assumptions` holds preconditions someone already established, so both
|
|
133
|
+
are usable as they stand rather than as a starting point to rebuild from. The
|
|
134
|
+
code you are about to change still needs reading; the setup around it is what
|
|
135
|
+
this file is for. A session that reads the document and then repeats the
|
|
136
|
+
search it was meant to replace has paid for it twice.
|
|
137
|
+
|
|
138
|
+
Then leave it truer than you found it:
|
|
139
|
+
|
|
140
|
+
- Replace the `@todo`s your run can answer, and delete each one you replace.
|
|
141
|
+
- Correct what you found wrong. Say what it is now, not what it used to be.
|
|
142
|
+
- Add the precondition you had to discover, or the thing the simulation turned
|
|
143
|
+
out not to reproduce, as another dated claim-and-consequence pair.
|
|
144
|
+
- Add what broke and why to `## What didn't work`, and delete the lines there
|
|
145
|
+
your run showed fixed.
|
|
146
|
+
- Add this run's summary to the top of the `## Observations` block, read from
|
|
147
|
+
the simulation's log, and drop the oldest line past five.
|
|
148
|
+
|
|
149
|
+
A run that only confirms the file still adds its `## Observations` line.
|
|
150
|
+
|
|
151
|
+
## Rules
|
|
152
|
+
|
|
153
|
+
- Leave another session's assumptions and limits standing unless your run
|
|
154
|
+
contradicted one: not exercised is not the same as not true.
|
|
155
|
+
- Keep this file for how the simulation behaves, and let the `.sim.yaml`
|
|
156
|
+
speak for what it is.
|
|
157
|
+
- Rewrite `## Purpose` when the simulation's scope genuinely changed.
|
|
158
|
+
- Prefer deleting a line to adding one. Everything here is something the next
|
|
159
|
+
session must either trust or re-check.
|
|
@@ -8,8 +8,11 @@ Read this when writing or debugging a `.sim.yaml` by hand.
|
|
|
8
8
|
simulation: checkout-dev # required: the name
|
|
9
9
|
routing: path # path (default) or header
|
|
10
10
|
mocks: # required: one entry per member
|
|
11
|
-
-
|
|
12
|
-
|
|
11
|
+
- name: catalog # optional display name
|
|
12
|
+
routeKey: catalog # lowercase letters, digits, - and _
|
|
13
|
+
origin: # optional: hostnames, for --proxy
|
|
14
|
+
- catalog.internal.example.com
|
|
15
|
+
path: postman/mocks/catalog # a mock's folder, config.yaml or handler
|
|
13
16
|
scenarios: # optional: conditions, below
|
|
14
17
|
- overrides:
|
|
15
18
|
conditions:
|
|
@@ -17,7 +20,17 @@ mocks: # required: one entry per member
|
|
|
17
20
|
```
|
|
18
21
|
|
|
19
22
|
Run the `.sim.yaml` rather than a simulation id: a run by id drops every
|
|
20
|
-
member's conditions (CLI 1.70.0–1.73.0)
|
|
23
|
+
member's conditions (CLI 1.70.0–1.73.0), and `--proxy` takes no id at all,
|
|
24
|
+
since the file is what declares the hostnames.
|
|
25
|
+
|
|
26
|
+
`simulation create` writes each `path:` relative to the repository root, the
|
|
27
|
+
directory holding `postman/`, and puts the simulation in a folder of its own
|
|
28
|
+
(`postman/simulations/checkout-dev/checkout-dev.sim.yaml`) so its
|
|
29
|
+
`<name>.context.md` sits beside it; the app records that document in a
|
|
30
|
+
`context:` ref. Reading is broader than writing — a path relative to
|
|
31
|
+
`postman/simulations/`, to the file, absolute or `~/`, naming the folder or
|
|
32
|
+
the manifest inside it, all resolve — and an unresolvable one names every
|
|
33
|
+
directory it looked in.
|
|
21
34
|
|
|
22
35
|
## Conditions
|
|
23
36
|
|
|
@@ -38,6 +51,23 @@ member's conditions (CLI 1.70.0–1.73.0).
|
|
|
38
51
|
- A `latency` request is served and logged, with a `duration` that leaves
|
|
39
52
|
out the delay.
|
|
40
53
|
|
|
54
|
+
## `origin`
|
|
55
|
+
|
|
56
|
+
The hostnames a member answers for under `--proxy`, and the only thing that
|
|
57
|
+
decides which member a proxied request reaches.
|
|
58
|
+
|
|
59
|
+
`simulation create` copies `origin` from the mock's manifest
|
|
60
|
+
(`origin: api.stripe.com`, a string or a list).
|
|
61
|
+
|
|
62
|
+
A scheme and port are stripped (`https://API.Stripe.com:443` →
|
|
63
|
+
`api.stripe.com`). An unmatchable value — a path left on a copied URL, a
|
|
64
|
+
wildcard — is warned about and skipped, leaving the member working; two
|
|
65
|
+
members claiming one hostname fails the run. A member whose mock declares
|
|
66
|
+
none has no `origin` key, since `origin: []` would reach the actual service instead of the mock.
|
|
67
|
+
|
|
68
|
+
Under `--proxy` the members run on throwaway ports, so a manifest's `port` is
|
|
69
|
+
ignored.
|
|
70
|
+
|
|
41
71
|
## The log (`--output ndjson`)
|
|
42
72
|
|
|
43
73
|
One JSON event per line on stdout; notices go to stderr, so keep them apart.
|
|
@@ -0,0 +1,144 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: application-testing
|
|
3
|
+
description: Sets up, runs, debugs, and interprets results from `postman application test` — the Postman layer that runs your Playwright tests, captures their network traffic, matches it against a collection contract, runs pm.test assertions, and publishes results to Application Inventory. Covers the Postman side of the integration only — not Playwright test authoring or Playwright failures unrelated to capture. Use when the user asks to "set up application testing," "test my application with Postman," "connect my Playwright tests to Postman," "capture network traffic from Playwright," "Playwright tests are running but nothing shows up in Postman," "why isn't application testing capturing anything," "exclude certain requests from capture," "generate a collection from my app's traffic," "set up application testing in CI," "what API coverage do my e2e tests have," "which APIs are my Playwright tests not hitting."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Application Test
|
|
7
|
+
|
|
8
|
+
## Overview
|
|
9
|
+
|
|
10
|
+
**Requires a Playwright project in the repository.** Without Playwright e2e tests there is nothing to capture — Playwright is the only supported runner.
|
|
11
|
+
|
|
12
|
+
`postman application test` does not send its own requests. It:
|
|
13
|
+
1. Spawns your Playwright test command (e.g. `npx playwright test`)
|
|
14
|
+
2. The capture plugin intercepts network traffic and writes it as artifacts
|
|
15
|
+
3. The CLI reads those artifacts, matches each request against your collections, and runs any `pm.test` assertions saved in the matched items
|
|
16
|
+
4. Publishes results to Application Inventory in your Postman workspace
|
|
17
|
+
|
|
18
|
+
The question it answers: **did my app's actual HTTP calls conform to the collection contract?** Not "does this endpoint respond correctly" — for that, use `postman collection run`.
|
|
19
|
+
|
|
20
|
+
**Application Inventory** is where results are published — a Postman dashboard that records each run's Playwright test results alongside captured/matched/assertion counts, giving you a combined view of test health and API contract conformance across runs.
|
|
21
|
+
|
|
22
|
+
---
|
|
23
|
+
|
|
24
|
+
## Prerequisites
|
|
25
|
+
|
|
26
|
+
**All `postman app` commands must be run from the repository root** — that is where `postman.config.cjs`, `.postman/resources.yaml`, and `postman/collections/` are expected to live.
|
|
27
|
+
|
|
28
|
+
Assess each prerequisite against the current state of the repository before taking any action. Most will already be satisfied in an established project; only act on what is actually missing.
|
|
29
|
+
|
|
30
|
+
### Playwright project
|
|
31
|
+
|
|
32
|
+
This skill applies only when the repository has Playwright e2e tests. Verify that a `playwright.config.*` file exists and `@playwright/test` is listed as a project dependency. If neither is present, stop — there is nothing to capture and this skill does not cover Playwright setup.
|
|
33
|
+
|
|
34
|
+
### Authentication
|
|
35
|
+
|
|
36
|
+
Publishing results to Postman requires credentials. Authenticate only when workspace publishing is needed:
|
|
37
|
+
|
|
38
|
+
- If `POSTMAN_API_KEY` is in the environment, run `postman login --with-api-key "$POSTMAN_API_KEY"`. This is the expected path in CI and agentic contexts.
|
|
39
|
+
- If the key is not available, surface the gap to the user — do not initiate `postman login` in an agentic context, as it requires a human to complete a browser flow.
|
|
40
|
+
|
|
41
|
+
If this run does not need to publish results (local debugging, dry run), authentication can be skipped by passing `--report-events false`.
|
|
42
|
+
|
|
43
|
+
### Workspace link
|
|
44
|
+
|
|
45
|
+
Publishing to Application Inventory requires a workspace binding. Check `.postman/resources.yaml` for a `workspace.id`. Reason about whether publishing is actually needed for this task before treating a missing workspace as a blocker:
|
|
46
|
+
|
|
47
|
+
- If publishing is needed and the workspace is missing, load the `bootstrap` skill to set it up. In a non-interactive context this must be done locally first, then `.postman/resources.yaml` committed.
|
|
48
|
+
- If publishing is not needed, pass `--report-events false` and proceed.
|
|
49
|
+
|
|
50
|
+
### Capture plugin
|
|
51
|
+
|
|
52
|
+
Network traffic capture requires the `postman-playwright` package to be installed and the Playwright config wrapped with `withPostman()`. Before attempting any setup, check whether it is already configured — `postman app setup-capture --dry-run` shows the current state without touching files.
|
|
53
|
+
|
|
54
|
+
Decide what action (if any) is needed based on what the dry run reports:
|
|
55
|
+
|
|
56
|
+
- Already configured → no action needed.
|
|
57
|
+
- Not configured → run `postman app setup-capture --yes`. If automatic patching fails (non-standard config export patterns), fall back to manual installation: install `postman-playwright` as a dev dependency and wrap the config export with `withPostman()`.
|
|
58
|
+
|
|
59
|
+
**Legacy pattern:** If the project uses `attachNetworkCapture()` in spec files instead of `withPostman()`, that is still supported — do not re-run setup. The CLI detects it automatically. Only migrate to `withPostman()` if the user asks to upgrade.
|
|
60
|
+
|
|
61
|
+
### Server reachability
|
|
62
|
+
|
|
63
|
+
`postman app test` spawns Playwright, which drives the live application — servers must be running before the command starts.
|
|
64
|
+
|
|
65
|
+
Check `playwright.config.*` for a `webServer` entry. If present unconditionally, Playwright manages server startup automatically. If absent or conditional (e.g. gated on an env var), verify the servers are reachable before running. If they are not, check `webServer.command`, `package.json` scripts, and any README for the startup command, and surface it to the user.
|
|
66
|
+
|
|
67
|
+
---
|
|
68
|
+
|
|
69
|
+
## Running
|
|
70
|
+
|
|
71
|
+
Before constructing any command, run `postman app test --help` to confirm the available flags for the installed CLI version. Do not guess flag names.
|
|
72
|
+
|
|
73
|
+
**No `postman.config.cjs` yet** — the first run generates and saves `postman.config.cjs`, writing these choices as the defaults for all future runs. Confirm both with the user before running:
|
|
74
|
+
|
|
75
|
+
- *Test command*: check `package.json` scripts for a project-specific test command (e.g. `npm run test:e2e`). Present what you find and ask the user to confirm — the CLI defaults to `npx playwright test` when a Playwright config is found, but do not assume that is what they want.
|
|
76
|
+
- *Collection*: check `postman/collections/` and list what is available. Ask the user which collection to use. If none exist, ask whether to generate one from captured traffic.
|
|
77
|
+
|
|
78
|
+
Once confirmed, run `postman app test` with the appropriate flags (e.g. `--target-collection <path>` or `--generate-collection`) — the CLI generates and saves `postman.config.cjs` automatically. Do not write the config file manually.
|
|
79
|
+
|
|
80
|
+
Running `postman app test` without a collection in a non-interactive context exits with code 2 unless a collection source is provided explicitly.
|
|
81
|
+
|
|
82
|
+
**`postman.config.cjs` exists** — the config is the source of truth. Run `postman app test` without extra flags. Individual config values can be overridden per-run without editing the file; consult `postman app test --help` for the override flags.
|
|
83
|
+
|
|
84
|
+
**Capture-only mode** — when the goal is to stage and review traffic before committing a collection manually, there is a mode that skips collection matching and writes captured traffic as a collection to disk. Use this only as a review step, not the normal run path.
|
|
85
|
+
|
|
86
|
+
---
|
|
87
|
+
|
|
88
|
+
## Config File (`postman.config.cjs`)
|
|
89
|
+
|
|
90
|
+
Lives at the **repository root**. The generated file includes comments explaining every field — read it for the full reference. It stays current with the CLI version installed in the repo.
|
|
91
|
+
|
|
92
|
+
To edit config for common needs:
|
|
93
|
+
- **Change the test command** — update `command`
|
|
94
|
+
- **Add or change a collection** — update `targets.default.collections`
|
|
95
|
+
- **Exclude noisy requests** — add `filters` (urlPatterns, methods, headers)
|
|
96
|
+
- **Rewrite URLs before matching** — add `transformers` (config entries that normalize staging/proxy prefixes before the CLI matches captured URLs against the collection)
|
|
97
|
+
- **Multiple named environments** — add entries under `targets`
|
|
98
|
+
|
|
99
|
+
A **target** is a named set of collections + optional environment. The `default` target is used when no `--target` flag is passed. Individual values can be overridden per-run without editing config — consult `postman app test --help` for the override flags.
|
|
100
|
+
|
|
101
|
+
---
|
|
102
|
+
|
|
103
|
+
## Debugging
|
|
104
|
+
|
|
105
|
+
**If a flag in this skill is unrecognized** (e.g. `--yes`, `--generate-collection`, `--dry-run`) or the behaviour described here doesn't match what you see, check the CLI version first:
|
|
106
|
+
```bash
|
|
107
|
+
postman --version
|
|
108
|
+
```
|
|
109
|
+
This skill targets the latest CLI release. If the version is outdated, load the `bootstrap` skill for installation and update steps, then re-run. The `--help` output for each command shows the flags actually available in the installed version.
|
|
110
|
+
|
|
111
|
+
Read the CLI error output first — it names the specific failure and what to fix.
|
|
112
|
+
|
|
113
|
+
### Requests show "not matched" in the summary table
|
|
114
|
+
|
|
115
|
+
Run with `--verbose --no-report-events` to see the unmatched URLs without publishing a diagnostic run to Application Inventory. Then:
|
|
116
|
+
|
|
117
|
+
- Wrong collection selected — check `targets.default.collections` in config or override with the target-collection flag.
|
|
118
|
+
- URLs have a staging prefix or proxy path — add a `transformers` entry in `postman.config.cjs` to rewrite them before matching.
|
|
119
|
+
|
|
120
|
+
---
|
|
121
|
+
|
|
122
|
+
> **Note — Exit code:** The CLI exits with the **test runner's exit code** (Playwright's), not the Postman assertion pass/fail count. A run where Postman assertions fail but all Playwright `expect()` calls pass will exit 0. If you need CI to fail on Postman assertion failures, check the summary table output or the published results.
|
|
123
|
+
|
|
124
|
+
---
|
|
125
|
+
|
|
126
|
+
## Critical Rules
|
|
127
|
+
|
|
128
|
+
1. **`postman application test` captures traffic; it does not drive requests.** The test runner (Playwright) drives the app; this command observes and asserts. If no test runner is running, nothing is captured.
|
|
129
|
+
2. **`not matched` is not a test failure — it is a coverage gap.** Requests the CLI cannot find in a collection run no assertions and do not affect the exit code. A high not-matched count means the collection is incomplete for the traffic the app generates.
|
|
130
|
+
3. **Transformers run before matching, not before capture.** The raw URL is stored; the transformer rewrites it for the matching step only. Use this to normalise staging/prod URL differences without losing the original for debugging.
|
|
131
|
+
4. **Results are published per run.** Each invocation creates a new entry in Application Inventory. Pass `--report-events false` for dry runs or local debugging you do not want recorded.
|
|
132
|
+
|
|
133
|
+
## Anti-patterns
|
|
134
|
+
|
|
135
|
+
- **Capture-only mode is for reviewing traffic before committing to a collection** — use it when you want to inspect what the app generates before deciding on a collection structure. It is not a substitute for `--generate-collection`, which generates, tests, and saves to config in one step when no collection is configured.
|
|
136
|
+
- **Do not run `postman app init`** — it is deprecated. `postman app test` runs the same setup on first use.
|
|
137
|
+
- **Do not manually write `postman.config.cjs`** — the CLI generates it automatically on first run when you pass `--target-collection` or `--generate-collection`. Writing it by hand bypasses the CLI's generation logic.
|
|
138
|
+
- **Do not combine capture-only mode with a target collection** — capture-only skips collection matching entirely; passing a collection alongside it has no effect and signals a misunderstanding of the mode.
|
|
139
|
+
|
|
140
|
+
## Verification
|
|
141
|
+
|
|
142
|
+
State captured / matched / not-matched / assertion counts from the summary table. Note whether results were published to Postman (`View results:` URL present) or kept local. If debugging a "not captured" issue, confirm which capture format was detected (Playwright JSON report or legacy NDJSON) and which directory the CLI scanned.
|
|
143
|
+
|
|
144
|
+
Postman docs: https://learning.postman.com/docs/postman-cli/postman-cli-application/#postman-app-test
|
|
@@ -11,6 +11,17 @@ in a PR. A workspace binding connects that work to maintained shared context.
|
|
|
11
11
|
Setup is useful when it resolves a missing capability or binding; a local API
|
|
12
12
|
check does not need a new workspace merely to proceed.
|
|
13
13
|
|
|
14
|
+
## Start with postman init
|
|
15
|
+
|
|
16
|
+
`postman init` is idempotent and safe to run in any project, including one
|
|
17
|
+
already initialized. You can run it before API engineering work to scaffold the
|
|
18
|
+
relevant `postman/` directories and make Postman skills available in
|
|
19
|
+
`postman/skills` to guide the task. It establishes an agent-scoped session for
|
|
20
|
+
local work without signing in; accessing existing workspace resources still
|
|
21
|
+
requires authentication.
|
|
22
|
+
See "Interpret initialization without losing partial success" below for flags
|
|
23
|
+
and outcomes; creating or connecting a workspace depends on the task.
|
|
24
|
+
|
|
14
25
|
## Resolve only the missing setup
|
|
15
26
|
|
|
16
27
|
Inspect `postman --version`, `.postman/resources.yaml`, and the relevant
|