@sitar_fiercer4c/skills 0.1.0

package/skills/record-api-traffic/SKILL.md

@@ -0,0 +1,434 @@
+---
+name: record-api-traffic
+description: Record the HTTP traffic of a Node/Next app on four legs — browser-to-frontend, browser-to-backend, and each server's own outbound calls (the frontend's server-side fetches, the backend's to Microsoft Graph and Datalab) — by running this skill's own recorder against a repo path, after a preflight that finds every blocker first. Everything lives in the skill; the repo is only read. Use this skill whenever the user wants to capture, record, trace, log or inspect those calls — "record the API traffic", "record the frontend traffic", "start the recorder", "capture what the frontend sends", "what pages did the browser load", "what does the backend call out to", "I want a log of the API calls" — and also whenever they hit trouble around it: "port 7100 is in use", "the recording is empty", "the backend won't start", "the dev server won't move off 4100", "mitmproxy won't bind", "what's holding 4100". Also use it when they ask what a finished recording contains or how to read the .jsonl and .log output, or when they want the pipeline's real input and output — the source Excel file or the stored quotation document — dumped for a given quotationId.
+---
+
+# Recording API traffic
+
+## What this is
+
+Everything this skill needs lives in the skill. `scripts/record-api-traffic.py` starts six
+processes — the two servers, each moved one port up, with a mitmproxy in front of it and a
+second mitmproxy recording its own outbound calls — and prints a link. The user opens it, uses
+the app, presses Ctrl-C, and is left with a single folder for that run holding four pairs of
+files: for each leg, an aligned table of every call and a JSONL file with full request and
+response bodies.
+
+```
+browser ──→ :4100 recorder ──→ :4101 frontend ──→ :4102 recorder ──→ the backend, external
+        └─→ :7100 recorder ──→ :7101 backend  ──→ :7102 recorder ──→ Graph, Datalab
+
+  :4100  frontend-inbound.*   what arrives at the frontend  (browser)
+  :4102  frontend-outbound.*  what the frontend calls out to  (backend, external)
+  :7100  backend-inbound.*    what arrives at the backend  (browser, frontend)
+  :7102  backend-outbound.*   what the backend calls out to  (Graph, Datalab)
+```
+
+Both servers are treated identically: the recorder takes the port that is already forwarded to
+the laptop, the server moves one up behind it, and a forward proxy catches what that server's
+own Node process calls out to. Nothing on the VS Code side changes and no config file in the
+repo is edited.
+
+**Ask for one thing: the repo path.** Every script here takes it as its single input, and once
+you have it you reuse it for all of them. Nothing else is asked of the user.
+
+**The repo is read, never written.** The backend's working directory is a temp dir under `/tmp`,
+recordings and dumps land in `skills/record-api-traffic-workspace/recordings/`, and no config
+file is edited.
+The one exception is honest and unavoidable: `npm run dev` writes `Frontend/.next/`, exactly as
+it would if the user started the dev server themselves. It is gitignored. If you ever find
+yourself about to create, edit or delete anything else under the repo, stop — that is a bug in
+what you are doing, not a step.
+
+The scripts work. The hard part is everything around them: ports are frequently occupied, the
+frontend can be silently pointed away from the recorder, and stopping a stray process is a
+decision that needs the user's judgment rather than yours. That is what this skill is for.
+
+Run the preflight, resolve what it finds *with* the user, launch, then prove the recording is
+actually capturing before calling it done.
+
+## Why the recorders sit on 7100 and 4100
+
+Worth understanding, because it explains several checks. The browser runs on the user's laptop
+and reaches the VM through VS Code port forwarding. `:4100` and `:7100` are already forwarded.
+So each recorder takes over the port the browser already calls and the server moves one up
+behind it. Nothing changes on the laptop, no new port needs forwarding, and no config file in
+the repo is edited. `BACKEND_PUBLIC_URL` and `AZURE_REDIRECT_URI` already point at `:7100`, so
+the Outlook OAuth callback keeps working and gets recorded too.
+
+The two servers resist being moved in different ways, and each needs its own trick.
+
+**The backend** ignores shell env vars: `app.js` calls `dotenv.config({ override: true })`, so
+`.env` wins. The recorder instead runs `node src/server.js` from a temp directory holding a copy
+of `.env` with `PORT=7101`, since dotenv resolves its file from the current working directory.
+That copy is deleted on teardown, and it is why nothing has to be written inside the repo to
+move the port.
+
+**The frontend** ignores `PORT` for a different reason: the dev script pins the port on the
+command line (`next dev -H 0.0.0.0 -p 4100`), and commander only consults `.env("PORT")` when
+the flag is absent. So the recorder appends a second one — `npm run dev -- -p 4101` — which both
+argument parsers Next has shipped resolve to the last occurrence. `PORT` is exported alongside
+for a dev script that leaves the port unpinned. Preflight reports which of the two applies, and
+warns when the dev script is not a Next one at all.
+
+## Why there are two forward proxies
+
+A reverse proxy only sees what the browser sends. Plenty of what these apps do never crosses one:
+the Excel attachment the backend pulls from Microsoft Graph server-side, the Datalab extraction
+calls, and anything the frontend fetches from a server component or route handler rather than
+from the browser. Those are outbound HTTPS from a Node process, so each server gets a
+`mitmdump --mode regular` of its own and is pointed at it with `HTTP_PROXY`/`HTTPS_PROXY` plus
+`NODE_USE_ENV_PROXY=1` — Node's global `fetch` ignores proxy env vars without that flag.
+`NODE_EXTRA_CA_CERTS` makes it trust mitmproxy's CA.
+
+The two differ in one respect, and it is deliberate. The backend excludes all of loopback from
+its proxy (`NO_PROXY=127.0.0.1,localhost`), because it has no loopback calls worth recording.
+The frontend excludes only its **own** two ports, so a server component calling the backend on
+`:7100` still goes through `:4102` — that call is the main thing the frontend outbound leg
+exists to catch. It is then recorded twice, once leaving the frontend and once arriving at the
+backend, which is the point: you can see both ends of the same call.
+
+Not covered on either: the AWS SDK (its own HTTP handler ignores the env) and MongoDB (a binary
+protocol, not HTTP). `--no-outbound` skips both forward proxies.
+
+## Step 1 — Preflight
+
+Always run this before launching. It is read-only: it never kills a process, writes a file, or
+edits config.
+
+```bash
+python3 skills/record-api-traffic/scripts/preflight.py <repo>
+```
+
+`--json` for machine-readable output. Exit code 0 means clear, 1 means blocked, 2 means the path
+is not an app repo.
+
+It reports every check at once rather than dying on the first failure, which matters
+because blockers cluster — an abandoned session typically leaves a proxy on 7100 *and* a dev
+server on 4100, and discovering those one launch-attempt at a time is slow and irritating.
+
+## Step 2 — Clear the blockers
+
+Present what preflight found in plain terms, then work through it. Two rules shape everything
+here.
+
+**Never stop a process you have not identified.** Preflight classifies each port holder as
+`[ours]` (a leftover from this recorder), `[app]` (the project's own dev server), or
+`[foreign]` (something else entirely), and prints the working directory and command line for
+foreign ones. That distinction is the whole point. A `mitmdump` on 7100 could be this tool's
+own orphan, or it could be a different project's recording session that someone is actively
+using — killing it destroys their work with no way to recover it.
+
+**Killing is the user's call, not yours.** For `[ours]` and `[app]` you can offer directly and
+act once they agree. For `[foreign]`, show them the command line and cwd and ask what it is
+before proposing anything. If they are unsure, leave it alone and say so — an unstarted
+recording costs a minute, a destroyed session costs an afternoon.
+
+### The blockers you will actually see
+
+**Port held by a foreign process.** The common real case: a `mitmdump` from another repo's
+recording setup, sometimes half-dead (proxy still listening, its backend long gone). Show the
+cwd and argv, ask, and only then offer `kill <pid>`. Prefer plain `kill` over `kill -9` so the
+process can shut down cleanly and flush whatever it was writing.
+
+**Port held by the app's own dev server.** Someone ran `npm run dev` by hand. Safe to offer to
+stop — but say what it is, because they may have work mid-flight in that browser tab.
+
+Six ports are checked now, not four: `4100`/`4101`/`4102` and `7100`/`7101`/`7102`. Only the
+two `x100` ports are ever forwarded or called from outside, so a holder on one of the other four
+is almost always debris. Preflight now also reads each holder's environment for the marker the
+recorder exports into every child, which is what lets it call a leftover dev server on `:4101`
+`[ours]` rather than `[app]` — the frontend has to run with the repo as its working directory,
+so cwd alone cannot tell the two apart.
+
+**The dev script cannot be moved off 4100.** Preflight reads `Frontend/package.json` and says
+how the port will be moved. A `next` script is fine. Anything else falls back to `PORT`, and if
+that script pins its own port instead, it will fight the recorder for `:4100` and one of them
+loses. Read what preflight prints rather than assuming; `--no-frontend` is the way out if the
+script cannot be moved.
+
+**Frontend env points somewhere else.** This is the one to care about most. If
+`NEXT_PUBLIC_API_URL` does not contain `:7100`, the browser talks straight to the backend and
+the API recorder sees nothing. Everything appears to work — processes start, the link opens, the
+app functions — and `backend-inbound.log` comes out empty. A crash is honest; this failure mode
+lies. Fix `Frontend/.env.local` before launching. That file is the user's to edit, not yours.
+
+Note what this does *not* affect: `frontend-inbound.log` fills up regardless, because the
+browser reaches the frontend through `:4100` either way. A run with pages in one file and
+nothing in the other is this bug, not a quiet recorder.
+
+**Dump scripts out of sync with the app.** See the next section — the one blocker that is new
+and easy to under-react to.
+
+**MongoDB unreachable.** The backend never passes its health check and the script gives up after
+90 seconds with no obvious cause. Preflight resolves this ahead of time. Note that Atlas
+hostnames are SRV-only, so a plain A/AAAA lookup fails even when the cluster is perfectly
+healthy — check `_mongodb._tcp.<host>` instead. Preflight already does this correctly; don't
+"fix" it with `ping` or `getent hosts` and conclude the network is down.
+
+**Missing mitmproxy CA.** `~/.mitmproxy/mitmproxy-ca-cert.pem` is written on mitmproxy's first
+ever run. Without it the backend cannot trust the outbound proxy and every outbound call fails.
+Generate it by starting `mitmdump` once, or launch with `--no-outbound`.
+
+**Orphans or a leftover scratch directory.** Debris from a previous run that crashed hard. The
+orphaned processes are this tool's own leftovers, so they are safe to offer to stop. A stale
+`/tmp/record-api-traffic-*` is only a warning. If preflight mentions `Backend/.mitm-run`, that is
+from an older version of this skill that wrote inside the repo; report it and let the user delete
+it — do not delete it yourself.
+
+One caveat, now that the frontend runs under the recorder: starting a dev server spawns
+short-lived helpers that inherit the recorder's marker, and one can outlive teardown by a while
+without holding any port. So an orphan report whose ports are all free, shortly after a session
+ended, is worth re-running before acting on — it usually clears on its own. A report that
+persists, or that comes with a busy port, is real.
+
+## Step 3 — Check the dump scripts still fit the app
+
+`scripts/dump-quotation.mjs` and `scripts/dump-source-excel.mjs` read the app's Mongoose model
+directly. They used to live in the app repo, where a schema change and the code reading it moved
+in the same commit. They no longer do, and nothing keeps them honest but this check:
+
+```bash
+node skills/record-api-traffic/scripts/check-schema.mjs --repo <repo>
+```
+
+Preflight runs it for you and folds the result into its report; run it directly when you want the
+per-field detail. It loads the real model and asserts every field the dumps touch — that
+`sourceWorkbookBuffer` exists **and is still a Buffer**, that `db.js` still default-exports
+`connectDB`, that `email.service.getAttachmentBuffer` still takes three arguments.
+
+Take a `FAIL` here seriously, because drift is silent. A renamed field does not crash: the
+document still loads, `quotation.sourceWorkbookBuffer` is merely `undefined`, and
+`dump-source-excel` falls through to the Graph refetch or writes a zero-byte file. You get an
+artifact that looks like an artifact and is worthless. Fix the script in the skill before
+recording, rather than after discovering the dump is empty.
+
+`warn` on a display-only field (`status`, `quotationItems`) means the summary line degrades but
+the dumped bytes are still right — worth mentioning, not worth blocking on. Exit code 2 means
+the check could not run at all, almost always because `Backend/node_modules` is missing; that is
+the dependency blocker, not a schema problem.
+
+## Step 4 — Launch
+
+The recorder is a long-lived foreground process that the user stops with Ctrl-C. Hand the
+command over rather than running it yourself in the background, where they cannot reach it and
+its output is buried:
+
+```
+python3 skills/record-api-traffic/scripts/record-api-traffic.py <repo>
+```
+
+Suggest they type `! python3 skills/record-api-traffic/scripts/record-api-traffic.py <repo>` in
+the prompt so the output lands in the conversation.
+
+Startup takes up to a couple of minutes — the backend waits on Mongo, and Next.js compiles the
+first page on demand. It prints progress for each stage, then a banner with the link
+(`http://localhost:4100`), the run directory and one path per active leg.
+
+If it dies during startup it prints the failing child's last 40 lines. Read those rather than
+guessing; the cause is nearly always in them.
+
+Three flags, none of them the normal case:
+
+- `--no-frontend` leaves the dev server on `:4100` unrecorded, exactly as it ran before the
+  frontend legs existed. Reach for it when the dev script cannot be moved off `:4100`, or when
+  fronting Next with a proxy is itself the suspect.
+- `--no-outbound` drops **both** forward proxies, not just the backend's. It is the answer when
+  `~/.mitmproxy/mitmproxy-ca-cert.pem` is missing or Node is older than 24, since either
+  condition breaks both equally.
+- `--frontend-all` keeps the static-asset traffic the frontend leg drops by default. Worth it
+  when the question is about asset loading or chunk sizes, and a bad idea otherwise — see the
+  quirk on noise below.
+
+## Step 5 — Prove it is recording
+
+Do not report success because processes started. Confirm a call was actually captured, on both
+browser-facing legs — they fail independently, so one row proves nothing about the other:
+
+```bash
+run=$(ls -d skills/record-api-traffic-workspace/recordings/*/run-* | tail -1)
+curl -s -o /dev/null http://127.0.0.1:7100/health
+curl -s -o /dev/null http://127.0.0.1:4100/
+tail -3 "$run/backend-inbound.log"
+tail -3 "$run/frontend-inbound.log"
+```
+
+A row should appear for `/health` in the first and for `/` in the second. If a file has only its
+header after the user has clicked around the app:
+
+- **`backend-inbound.log` empty, `frontend-inbound.log` filling.** The browser is reaching the
+  frontend but the frontend is not routing API calls through `:7100`. Re-check
+  `NEXT_PUBLIC_API_URL`.
+- **Both empty.** The browser is not reaching `:4100` at all — a stale forwarded port, or a tab
+  still pointed at an old URL.
+- **`frontend-outbound.log` empty.** Usually correct, not broken. It only fills when
+  the frontend fetches from Node — a server component, a route handler, or the dev server's own
+  telemetry. An app whose data fetching all happens in the browser leaves this leg nearly empty
+  by design.
+
+This step exists because the expensive failure is a session that felt fine and produced nothing.
+Thirty seconds here saves re-running the whole thing.
+
+## Step 6 — Read the output
+
+Recordings live in the skill's workspace, not the repo, under a folder named for the repo they
+came from, so two clones of the same project never mix. The workspace sits beside the skill
+rather than inside it, because `skills/*-workspace/` is already gitignored — which is what keeps
+the tokens in a recording out of this repo's history:
+
+```
+skills/record-api-traffic-workspace/
+  iteration-N/               eval and benchmark output, not yours to touch
+  recordings/
+    <last-two-path-segments-of-the-repo>/
+      run-20260907-015953/
+        run.json                 repo, time, and which legs were on
+        backend-inbound.log      into the backend, scannable table
+        backend-inbound.jsonl    the same calls with full bodies
+        backend-outbound.log     out of the backend  (Graph, Datalab)
+        backend-outbound.jsonl
+        frontend-inbound.log     into the frontend  (browser)
+        frontend-inbound.jsonl
+        frontend-outbound.log    out of the frontend  (backend, external)
+        frontend-outbound.jsonl
+        artifacts/               written by the dump scripts in Step 7
+```
+
+One run, one directory; nothing to correlate by timestamp across a flat listing.
+
+The `.log` is one row per call, time first. The `.jsonl` holds the full bodies, one JSON object
+per line, where line number equals the row's ID. The two forward-proxy legs put `host:port` in
+the PATH column, since they span several hosts — and, on the frontend's, several loopback ports
+that would otherwise all read as `127.0.0.1`.
+
+`run.json` records which legs were active. A missing file means that leg was switched off, not
+that it captured nothing; check there before concluding a recorder failed.
+
+The banner prints `Run dir` at startup. When working with "the recording" later, resolve the
+newest run for that repo rather than assuming, and check `run.json` if you are unsure which repo
+a folder belongs to.
+
+For formats, field meanings and jq recipes, read `references/reading-recordings.md`.
+
+## Step 7 — Stop, verify, dump
+
+When the user signals they are finished with the app — "done", "stop recording", "that's it" —
+that single signal means all three of the following. Carry them out in order without waiting to
+be asked for each one. A run that stops without its dumps is an incomplete recording, and the
+user will not know to ask for them.
+
+### 7a — Stop
+
+Ctrl-C stops all six processes, removes the temp scratch directory, and prints a count of calls
+recorded and failed for each leg.
+
+If you launched it in the background rather than handing it over, Ctrl-C cannot reach it. Send
+`kill -INT` to the `python3` process itself — not `kill -9`, which skips teardown and loses both
+the summary and the scratch cleanup:
+
+```bash
+pgrep -af '[r]ecord-api-traffic.py' | grep -v 'zsh -c'
+kill -INT <pid of the "python3 .../record-api-traffic.py" process>
+```
+
+Read that list and pick the right line rather than piping to `head -1`. It also contains the
+launching shell (`/bin/zsh -c ... eval 'python3 ...'`) and the mitmdump children — eight lines of
+them, since mitmproxy forks and each of the four recorders shows up twice. All match the pattern.
+Signalling the shell does nothing — the recorder keeps running and the ports stay bound, which
+looks exactly like a teardown that silently failed. Confirm with the port check in 7b rather than
+assuming the signal landed.
+
+### 7b — Verify the teardown
+
+The repo should be exactly as it was, apart from the dev server's gitignored `.next/`:
+
+```bash
+git -C <repo> status --short          # nothing from the recording
+ls -d /tmp/record-api-traffic-* 2>/dev/null   # gone
+```
+
+Then confirm the ports are free. Do not go looking for processes by name:
+
+```bash
+for p in 4100 4101 4102 7100 7101 7102; do
+  ss -ltnp | grep -q ":$p " && echo "$p BUSY" || echo "$p free"
+done
+```
+
+A `pgrep -af '[m]itmdump|[s]erver\.js|[n]ext dev'` sweep is deliberately broad and will catch
+unrelated things — VS Code's `tsserver.js` matches `server.js`, and other repos run their own
+mitmdump. The port check answers the actual question. If you do run pgrep, use bracketed patterns
+like `[m]itmdump`; without them the pattern matches the shell running it, which reports a phantom
+process and, with `pkill`, kills the shell itself. Report anything foreign rather than killing it.
+
+### 7c — Dump the input and the output
+
+The recording is HTTP only, so the pipeline's real input and output are not in it. Two scripts
+fill that gap, both taking the repo and a quotationId. Get the id from the recording rather than
+asking:
+
+```bash
+grep -oE '/api/quotations/[a-f0-9]{24}' <run-dir>/backend-inbound.log \
+  | grep -oE '[a-f0-9]{24}' | sort | uniq -c | sort -rn
+
+node skills/record-api-traffic/scripts/dump-source-excel.mjs --repo <repo> <quotationId>
+node skills/record-api-traffic/scripts/dump-quotation.mjs   --repo <repo> <quotationId>
+```
+
+Both default to `artifacts/` inside the newest run folder *for that repo*, so run them straight
+after a session and they land beside the traffic they belong to. Pass a run dir as an optional
+extra argument to target an older session. If several quotationIds appear in the log, dump each.
+
+The Excel comes from `sourceWorkbookBuffer` on the document — the raw xlsx bytes are kept there
+as the export template — and falls back to refetching from Graph when that field is null, which
+is the case for CSV attachments and oversized documents. That fallback is rarely exercised;
+treat a failure in it as untested code rather than a broken session.
+
+Say plainly that these files contain live JWTs, the login password and Outlook tokens, recorded
+raw by deliberate choice. The `skills/*-workspace/` gitignore rule keeps them out of this repo's
+history, and nothing is written into the recorded app's repo at all. They
+should not be pasted into issues or shared.
+
+## Known quirks
+
+**No record is written when the backend is unreachable.** mitmproxy aborts at the connection
+stage before a flow exists, so its error hook never fires. If the backend dies mid-session,
+expect silence in the log and a loud message in the terminal — the launcher notices within a
+second and shuts everything down. Absence of rows during an outage is expected, not a bug.
+
+**No hot reload on the backend.** It runs under plain `node`, not nodemon, because nodemon
+watches the working directory and that is now a temp directory. Backend edits need a restart of
+the recorder. The frontend is unaffected: `next dev` still runs normally, and its HMR websocket
+upgrades cleanly through the reverse proxy, so editing a component reloads the browser as usual.
+
+**The frontend leg drops static assets by default.** One page load in `next dev` pulls dozens of
+`/_next/static/` chunks; recorded in full they would bury the handful of rows that carry meaning
+and add tens of megabytes of JavaScript to the JSONL. So `/_next/static/`, `/_next/image`,
+`/_next/webpack-hmr`, `/__nextjs*` and `/favicon.ico` are skipped before the row counter moves,
+which keeps row IDs equal to JSONL line numbers. Navigations, RSC payloads, server actions and
+any route handler the frontend serves itself are all kept. `--frontend-all` records everything.
+
+**A frontend server-side call to the backend appears twice.** Once on
+`frontend-outbound` as it leaves Node, once on `backend-inbound` as it arrives. Same
+call, both ends. Match them on path and timestamp; the outbound row is the earlier of the two.
+
+**The frontend outbound leg picks up npm and Next's own chatter.** `next dev` checks the npm
+registry for a newer version at startup, and that request is a real outbound call from the
+frontend's Node process, so it is recorded like any other. A row for `registry.npmjs.org` at the
+top of the file is expected, not a leak.
+
+**Bodies are capped at 1 MB** and binary responses (images, PDFs, spreadsheets) are recorded as
+a size summary rather than base64. The quotation PUT path can carry 16 MB of inline images, so
+without this the file would be unusable. This applies to every leg: the multipart upload to
+Datalab shows as a size, not the file. Use `dump-source-excel.mjs` for the bytes.
+
+Note what the cap does *not* cover on the frontend leg. JavaScript and HTML are text, so a page
+or a chunk under 1 MB is stored in full — which is exactly why the noise filter above exists,
+and why `--frontend-all` should be a considered choice rather than a default.
+
+**The dump scripts resolve everything from `--repo` by hand.** They import the app's modules by
+absolute path and its `mongoose` through `createRequire` rooted at `Backend/`, because a bare
+`import mongoose` inside the skill would look in the skill's own non-existent `node_modules`.
+They also parse `Backend/.env` themselves rather than relying on `dotenv` finding it relative to
+the working directory. If you add a script here that touches the app, use `scripts/lib/repo.mjs`
+rather than reinventing that.

package/skills/record-api-traffic/references/reading-recordings.md

@@ -0,0 +1,224 @@
+# Reading a recording
+
+Every session writes its own folder in the skill's workspace,
+`skills/record-api-traffic-workspace/recordings/<repo-slug>/run-<YYYYMMDD-HHMMSS>/`, containing:
+
+- `run.json` — which repo this came from, when, and which legs were recorded
+- `backend-inbound.log` / `.jsonl` — everything arriving at the backend. Mostly the browser,
+  but the frontend's own server-side calls land here too
+- `backend-outbound.log` / `.jsonl` — what the backend calls out to: Microsoft Graph, Datalab
+- `frontend-inbound.log` / `.jsonl` — everything arriving at the frontend: page navigations,
+  RSC payloads, server actions, and any route handler the frontend serves itself
+- `frontend-outbound.log` / `.jsonl` — what the frontend calls out to, i.e. what its Node
+  process fetches rather than what the browser does: the backend, and anything external
+- `artifacts/` — the source spreadsheet and stored quotation, written by the dump scripts
+
+The names say which server is being watched and which way the traffic goes, so each server's
+pair sits together in a listing. Note that "inbound" is deliberate rather than "from the
+browser": `backend-inbound` catches whatever reaches `:7100`, whoever sent it.
+
+Each pair is an aligned table for scanning plus one JSON object per line with the full bodies.
+All four share a format, so every recipe below works on any of them — swap the filename.
+
+The recipes below assume you are inside a run folder. To work on the most recent session:
+
+```bash
+cd "$(ls -d skills/record-api-traffic-workspace/recordings/*/run-* | tail -1)"
+```
+
+The slug is the last two segments of the repo path, so two clones of the same project keep
+separate folders. When several repos have been recorded, pick the right slug rather than the
+newest run overall, and check `run.json` if you are unsure.
+
+The only formatting difference between legs: the two forward-proxy files
+(`backend-outbound`, `frontend-outbound`) put `host:port` in the `PATH` column,
+because they span several hosts — and on the frontend's, several loopback ports that would
+otherwise all read as `127.0.0.1`.
+
+All are flushed after every call, so `tail -f` on a `.log` works while a session is live.
+
+## Which file answers which question
+
+| Question | File |
+|---|---|
+| What did the user's browser actually load? | `frontend-inbound` |
+| Which API calls did the page make? | `backend-inbound` |
+| Did a server component or route handler fetch this, or the browser? | `frontend-outbound` — if it is there, it came from Node |
+| What did the backend call out to? | `backend-outbound` |
+
+A server-side call from the frontend to the backend lands in **two** files: once leaving Node,
+once arriving at the backend. That is not double-counting — it is both ends of one call, and
+comparing them is how you see what the proxy chain did to it in between.
+
+The frontend leg drops `/_next/static/`, `/_next/image`, `/_next/webpack-hmr`, `/__nextjs*` and
+`/favicon.ico` unless the run used `--frontend-all`; `run.json` records which. The rows are
+dropped before the counter moves, so IDs still equal line numbers.
+
+## The table
+
+```
+TIME          ID   MS     METHOD STATUS PATH                          REQ      RESP
+14:22:03.184  0001  1243  POST   200    /api/auth/login               86 B     412 B
+14:22:06.771  0003  4021  GET    200    /api/email/messages?top=25    -        184 KB
+14:22:31.004  0004 12908  POST   502    /api/quotation/extract        2.4 MB   311 B
+```
+
+`ID` is the line number of the matching record in the `.jsonl`, so row `0003` is line 3. To
+read a row's full bodies, jump straight to that line:
+
+```bash
+sed -n '3p' backend-inbound.jsonl | jq .
+```
+
+Two things to know about the columns. `TIME` is when the response *completed*, not when the
+request started — rows are appended on completion, which is what keeps ID equal to line number
+even when requests overlap. The start time is in the JSONL as `started_at`. And paths longer
+than 44 characters are ellipsized with `…` to hold the table's alignment; the full path and
+parsed query string are always intact in the JSONL.
+
+`STATUS` shows `ERR` for a flow that failed without a response.
+
+## The JSONL record
+
+```json
+{
+  "id": "0002",
+  "ts": "2026-09-03T18:45:27.323364+00:00",
+  "started_at": "2026-09-03T18:45:27.321536+00:00",
+  "duration_ms": 1,
+  "method": "POST",
+  "path": "/api/auth/login",
+  "host": "127.0.0.1",
+  "url": "http://127.0.0.1:7100/api/auth/login",
+  "query": {},
+  "client_ip": "127.0.0.1",
+  "request": {
+    "headers": { "Authorization": "Bearer ...", "Content-Type": "application/json" },
+    "body": { "email": "ops@example.com", "password": "hunter2" },
+    "body_encoding": "json",
+    "body_size": 48,
+    "truncated": false
+  },
+  "response": {
+    "status": 200,
+    "headers": { "Content-Type": "application/json" },
+    "body": { "success": true, "token": "eyJ..." },
+    "body_encoding": "json",
+    "body_size": 42,
+    "truncated": false
+  }
+}
+```
+
+`path` excludes the query string; `query` holds it parsed. `host` is the request's target host,
+which is the only way to tell Graph from Datalab in an outbound recording. `response` is absent
+entirely on a failed flow, and an `error` field appears instead.
+
+Which external services the backend called, and how often:
+
+```bash
+jq -r '.host' backend-outbound.jsonl | sort | uniq -c | sort -rn
+```
+
+On the frontend outbound leg `host` is nearly always `127.0.0.1`, so key off the origin instead —
+the port is what separates the backend from everything else:
+
+```bash
+jq -r '.url | split("/")[0:3] | join("/")' frontend-outbound.jsonl \
+  | sort | uniq -c | sort -rn
+```
+
+```
+      1 https://registry.npmjs.org
+      1 http://127.0.0.1:7100
+```
+
+Which calls on a page came from the server rather than the browser — the paths in this list are
+the ones you will also find in `backend-inbound.jsonl`, recorded a few milliseconds later at the
+other end:
+
+```bash
+jq -r 'select(.url | test(":7100")) | "\(.method) \(.path)"' \
+   frontend-outbound.jsonl | sort -u
+```
+
+### body_encoding
+
+Tells you how to interpret `body`, which matters because its type varies:
+
+- `json` — parsed, so jq can reach inside it directly
+- `text` — a string; either genuinely non-JSON, or JSON that was truncated before parsing
+- `binary` — a summary object `{"binary": true, "content_type": "image/png", "size": 5004}`.
+  Images, PDFs and spreadsheets are never base64'd into the file; the S3 image proxy alone
+  would otherwise add megabytes per thumbnail.
+- `null` — no body at all (`body_size` 0)
+
+`body_size` is always the true size on the wire. When `truncated` is `true` the body was larger
+than the 1 MB cap and `body` holds the first 1 MB as a string — so `body_size` and the actual
+length of `body` disagree, by design.
+
+## jq recipes
+
+Failures, with the reason:
+
+```bash
+jq -c 'select(.response.status >= 400 or has("error"))
+       | {id, method, path, status: .response.status, msg: .response.body.message}' \
+   backend-inbound.jsonl
+```
+
+Slowest calls first. Put `duration_ms` first and let `sort -rn` key off it — piping whole JSON
+objects into `sort` does not work, because the field separators inside the object throw the
+column numbering off and you get something that looks sorted but is not:
+
+```bash
+jq -r '[.duration_ms, .id, .method, .path] | @tsv' backend-inbound.jsonl \
+  | sort -rn | head -20
+```
+
+Which endpoints were exercised, and how often:
+
+```bash
+jq -r '.method + " " + .path' backend-inbound.jsonl | sort | uniq -c | sort -rn
+```
+
+Everything one endpoint did:
+
+```bash
+jq -c 'select(.path | test("/api/quotation")) | {id, method, status: .response.status}' \
+   backend-inbound.jsonl
+```
+
+Request and response bodies for one call:
+
+```bash
+jq 'select(.id == "0004") | {request: .request.body, response: .response.body}' \
+   backend-inbound.jsonl
+```
+
+Payloads big enough to have been truncated:
+
+```bash
+jq -c 'select(.request.truncated or .response.truncated) | {id, path, req: .request.body_size, resp: .response.body_size}' \
+   backend-inbound.jsonl
+```
+
+An API inventory to compare against documentation — unique endpoints with the status codes each
+returned:
+
+```bash
+jq -r '"\(.method) \(.path) -> \(.response.status)"' backend-inbound.jsonl \
+  | sort -u
+```
+
+If `jq` is unavailable, the same work is straightforward in a short Python loop over
+`json.loads` per line.
+
+## Secrets
+
+Recordings are raw by deliberate choice — nothing is masked. `request.headers.Authorization`
+carries live bearer tokens, `/api/auth/login` request bodies carry the real password, and the
+Outlook flow carries Microsoft access tokens. The `skills/*-workspace/` gitignore rule keeps the
+whole workspace out of this repo's history, and nothing is written into the recorded app's repo
+at all. Treat a
+recording like a credential: don't paste one into an issue, a PR, or a chat with anyone.