auto-model-router 0.2.22 → 0.2.23

package/.claude/skills/agentdox/SKILL.md

@@ -43,7 +43,8 @@ Deterministic, so the same folder always resolves to the same slug:
      stop and ask for a distinguishing slug.
    - **Near match** (`my-app` vs `my-app-v2`) — a stop sign: ask, rather than fork a second
      namespace for one project.
-4. `project_ensure {slug, name}` — `name` is the readable form of the folder.
+4. `project_ensure {slug}` — `name` is optional and only used if the project is new; it
+   defaults to the slug, so pass one only when the readable form differs.
 5. **Make `.env.agentdox` un-committable *before* writing it.** Run `git check-ignore -v
    .env.agentdox`. If it is not ignored, **add `.env.agentdox` to `.gitignore`** (create that
    file if there is none) — do not merely check and move on. Patterns like `.env` and
@@ -136,7 +137,7 @@ await fetch("http://localhost:3003/memory", { method: "POST", headers: H,
 
 | Step | MCP | REST |
 | --- | --- | --- |
-| Ensure the project | `project_ensure {slug, name}` | `POST /projects {slug,name}` (idempotent) |
+| Ensure the project | `project_ensure {slug}` | `POST /projects {slug}` (idempotent) |
 | Read the brief | `context_brief {scope}` | `GET /context/brief?scope=<scope>` |
 
 The brief is the cumulative on-ramp: overview, repo layout, code style, build/test,

package/README.md

@@ -32,6 +32,93 @@ This router exists for the things a prompt classifier structurally cannot do:
 | **Closed-loop trust** | Per-model escalation and error rates from *your* traffic demote cheap-but-flaky models automatically. |
 | **Explainability** | Every decision — candidates, rejections, forecasts, reasons — is persisted and replayable via `auto-model-router explain`. |
 
+## Measured against Claude Opus 5
+
+Five benchmark runs, 88 graded task runs, 2026-08-29. Each task is a real omp
+session working in a pristine git workspace from a written spec. Hidden tests are
+copied in only *after* the agent exits, so they cannot be read or edited by it;
+every task is verified to fail an untouched workspace and to pass a reference
+solution. Both arms are metered from omp's own event stream, run under an
+identical tool surface, and are checked per turn against their expected provider.
+The router arm routes freely — nothing pinned. The baseline is `claude-opus-5`
+on Anthropic first-party.
+
+### Core suite — 10 coding tasks × 3 trials
+
+| | auto-model-router | Claude Opus 5 |
+| --- | --- | --- |
+| Tasks solved | **30 / 30** | 30 / 30 |
+| Total cost | **$0.63** | $16.61 |
+| Cost per solved task | **$0.0209** | $0.5538 |
+| Turns to finish | **278** | 303 |
+| Tool calls | **265** | 337 |
+| Wall clock | **2 057 s** | 3 185 s |
+| Median time to first token | 5 776 ms | **1 490 ms** |
+
+**26.5× cheaper at identical correctness** — and in fewer turns, fewer tool
+calls, and 19 minutes less wall clock. The saving is not bought by grinding out
+extra turns. The one regression is time to first token: a routed turn pays for
+classification and dispatch before anything streams back.
+
+Per task the ratio ranges from 9× to 264×. The widest gaps are tasks where the
+single-model baseline entered long tool loops — `semver` and `queue-order` cost
+it $2.99 each across three trials against a $1.32 median, 36% of its entire bill.
+
+### Difficulty ladder — 7 rungs, run twice
+
+A second suite of deliberately escalating difficulty, ending in npm semver range
+semantics and a minimal diff with a specified tie-break.
+
+| | auto-model-router | Claude Opus 5 |
+| --- | --- | --- |
+| Run 1 | 5 / 7 · $0.30 | 5 / 7 · $6.25 |
+| Run 2 | 5 / 7 · $0.46 | **6 / 7** · $6.60 |
+
+At the top of the ladder the engines separate: they fail different rungs, and on
+the second run the single-model baseline finished one more. Both arms timed out
+on the semver rung at the 10-minute cap.
+
+### What it routed to
+
+Across 464 routed turns in all five runs:
+
+| Model | Turns | Input price | Role |
+| --- | --- | --- | --- |
+| `z-ai/glm-5.3-flash` | 389 (84%) | $0.07 / MTok | default |
+| `google/gemini-3.7-flash` | 56 (12%) | $0.75 / MTok | escalation target |
+| `x-ai/grok-4.6` | 18 (4%) | $2.00 / MTok | escalation target |
+
+**Tier escalation converts to a costlier model roughly one-for-one**: on the
+ladder, the count of turns classified `hard` matched the count served by
+something other than the default (6/6, 4/4, 3/3, 5/5, 7/7, 1/1 across rungs and
+runs). The escalation *target* is chosen live from trust and latency history, so
+it differs between runs on the same catalog — run 1 stepped up to
+`gemini-3.7-flash`, run 2 to `grok-4.6`.
+
+Escalation stays inside the cheaper half of the catalog. A model priced above a
+tier's `maxInputPerMtok` is excluded before ranking, and at `hard` the
+`(quality/100)^qualityExponent ÷ expected cost` score favours cheaper models that
+score nearly as well. If your workload needs a frontier model on hard turns,
+raise the tier price ceiling and `qualityExponent` — measured thresholds are in
+[`docs/routing-benchmark-findings.md`](docs/routing-benchmark-findings.md).
+
+### Scope
+
+These are small, self-contained tasks of one to three files, solved in under 25
+turns. On the core suite both engines solved everything, so it measures cost at
+equal correctness rather than capability; the ladder is where capability
+separates. The cost multiple varied between 14× and 32× across runs depending on
+which task the baseline stalled on — treat "well over an order of magnitude" as
+the claim, not a specific figure.
+
+For sustained work on a large codebase the economics differ: cost there is
+dominated by the conversation being resent each turn rather than by per-token
+price. Replaying a week of real omp traffic (6 918 billed turns, 410:1
+input-to-output) against a single-model baseline gives **≈15×**.
+
+Harness, tasks and raw per-turn data:
+[`docs/routing-benchmark-findings.md`](docs/routing-benchmark-findings.md).
+
 ## Architecture
 
 ```mermaid

package/docs/routing-benchmark-findings.md

@@ -13,15 +13,18 @@ of my own earlier conclusions were wrong and are corrected in place.
 
 1. **The router is 20–32× cheaper at equal solve rate.** On 10 easy tasks both
    arms solved 20/20; on a 7-rung difficulty ladder both solved 5/7.
-2. **It never escalates.** Across 86 routed turns on the ladder — including the
-   rungs it *failed* — there were zero escalation signals, zero retry
-   dispatches, and only two models ever served: `glm-5.3-flash` and
-   `gemini-3.7-flash`.
-3. **Root cause is two independent gates, and both must move.** The price
-   ceiling hard-excludes expensive models below `hard`; and the ranking
-   arithmetic makes quality unable to outweigh price at any sane exponent.
-4. **Escalation does work once both gates are opened** — verified end to end,
-   with `claude-opus-5` serving 3 turns of a task the cheap-only config failed.
+2. **Tier escalation works.** A turn classified `hard` reliably selects a
+   costlier model — the hard-turn count matches the non-flash turn count ~1:1
+   across two independent ladder runs (to `gemini-3.7-flash` in one, to
+   `grok-4.6` in the other). *An earlier version of this document claimed the
+   opposite; that was my error, corrected in §3.*
+3. **What never happens is reaching `claude-opus-5`.** Two gates: the price
+   ceiling excludes it below `hard`, and at `hard` it is outranked by cheaper
+   high-quality models. Whether that is a defect depends on whether
+   `grok-4.6`-class is "good enough" — this benchmark cannot say.
+4. **Mid-stream escalation signals never fired** — `escalation_signal` was empty
+   on every one of ~190 routed ladder turns. That is a different mechanism from
+   tier selection and may simply mean nothing malformed occurred.
 5. **The standalone `serve` process died 4 times inside one normally-completing
    run.** The crash is real and mid-run; the *trigger* is unidentified — my first
    explanation (client disconnect mid-stream) does not reproduce. Details in §7.
@@ -33,6 +36,11 @@ of my own earlier conclusions were wrong and are corrected in place.
 - **Harness**: omp in `-p --mode=json` print mode. Every turn's tokens,
   duration, TTFT and tool calls are read off omp's own event stream, so both
   arms are measured by the same instrument.
+- **Provider guard**: each arm declares its expected omp provider and every
+  completion turn is checked against it — `anthropic` + `claude-opus-5` for the
+  baseline (first-party, never OpenRouter), `benchrouter` for the router arm.
+  Zero violations across the re-runs. Added after an early pilot silently
+  resolved `--model auto` to OpenRouter's own `auto` meta-router.
 - **Router arm**: the standalone `auto-model-router serve` endpoint registered
   as a plain OpenAI-compatible provider — the documented non-omp path (README
   § "Standalone alternative"). Nothing pinned, nothing stubbed; the router
@@ -46,22 +54,29 @@ of my own earlier conclusions were wrong and are corrected in place.
   verified to fail an untouched workspace and to pass a reference solution
   before any run (`bench/validate-ladder.ts`).
 
-> **A note on print mode.** The embedded extension only registers the
-> `auto-model-router` provider inside an interactive session — `ctx.hasUI` is
-> false in `-p`, so it takes the subagent path and looks for an existing
-> `embed.port`. In print mode with no interactive session running, the provider
-> is absent from the model registry entirely and `--model auto-model-router/auto`
-> fails with "Model not found". Worth documenting, or worth having the subagent
-> path fall back to binding its own server.
+> **Print mode — resolved.** During the benchmark the embedded extension only
+> registered the provider inside a session with a UI, so `omp -p` could not see
+> `auto-model-router/auto` at all; that is why the harness drives the standalone
+> `serve` endpoint instead. Fixed upstream in **0.2.20** (`fix(embed): resolve the
+> router in print mode and subagents`) — verified: `omp -p --model
+> auto-model-router/auto` now resolves natively with no standalone server. The
+> harness keeps the `serve` path because it gives the benchmark its own isolated
+> ledger, not because the embedded path is broken.
 
 ---
 
 ## 2. Cost result
 
-| Suite | Router | Opus 5 | Ratio |
-|---|---|---|---|
-| 10 easy tasks × 2 trials | 20/20 · $0.3577 | 20/20 · $11.4040 | 31.9× |
-| 7-rung ladder × 1 trial | 5/7 · $0.3026 | 5/7 · $6.2506 | 20.7× |
+| Suite | Run | Router | Opus 5 | Ratio |
+|---|---|---|---|---|
+| 10 easy tasks | trials 1+2 | 20/20 · $0.3577 | 20/20 · $11.4040 | 31.9× |
+| 10 easy tasks | re-run | 10/10 · $0.2695 | 10/10 · $5.2092 | 19.3× |
+| 7-rung ladder | first | 5/7 · $0.3026 | 5/7 · $6.2506 | 20.7× |
+| 7-rung ladder | re-run | 5/7 · $0.4619 | **6/7** · $6.5967 | 14.3× |
+
+Across four independent runs the ratio lands between **14× and 32×**. On the
+ladder re-run Opus 5 edged the router on correctness for the first time (6/7 vs
+5/7), which is the outcome the ladder was built to be able to detect.
 
 Turn counts are comparable (184 vs 217 on the easy suite; 86 vs 91 on the
 ladder), so the saving is not bought with extra turns. Median TTFT is the
@@ -74,30 +89,56 @@ magnitude" is defensible; a precise figure is not.
 
 ---
 
-## 3. The router never escalates
-
-The ladder was built specifically to force an escalation decision: seven rungs
-ending in npm semver range semantics and a minimal-diff with a specified
-tie-break.
+## 3. Escalation — corrected
 
-| Rung | Task | Router | Opus 5 | Escalated? |
-|---|---|---|---|---|
-| 1 | in-range | PASS | PASS | no |
-| 2 | round-half-even | PASS | PASS | no |
-| 3 | csv-document | PASS | PASS | no |
-| 4 | sliding-limiter | PASS | **FAIL** | no |
-| 5 | savepoints | **FAIL** | PASS | **no** |
-| 6 | semver-ranges | timeout | timeout | no |
-| 7 | minimal-diff | PASS | PASS | no |
+**I got this wrong first time.** I reported that the router "never escalates",
+having looked at the served-model list, seen only `glm-5.3-flash` and
+`gemini-3.7-flash`, and mentally filed both as "flash models, therefore no
+escalation". I never checked whether the counts lined up with the tier. They do.
 
-Rung 6 timed out on both arms at the 10-minute cap; it is not evidence about
-capability either way.
+### Tier escalation works, and is close to deterministic
 
-**The router assigned its `hard` tier on six of the seven rungs**, so the
-classifier is not blind — the tier simply never converts into a more capable
-model. On rung 5 it failed while continuing to dispatch to `glm-5.3-flash`.
+Hard-tier turns versus turns served by something other than the cheap default:
 
----
+| Rung | ladder-1 hard / non-flash | v2-ladder hard / non-flash |
+|---|---|---|
+| L1 in-range | 0 / 0 | 0 / 0 |
+| L2 round-half-even | 6 / 6 | 0 / 0 |
+| L3 csv-document | 4 / 4 | 0 / 0 |
+| L4 sliding-limiter | 3 / 6 | 5 / 5 |
+| L5 savepoints | 3 / 3 | 7 / 7 |
+| L6 semver-ranges | 2 / 3 | 0 / 0 |
+| L7 minimal-diff | 3 / 4 | 1 / 1 |
+
+The escalation *target* differs by run, which is itself worth knowing:
+`ladder-1` stepped up to `gemini-3.7-flash` ($0.75/MTok, ~10× the default);
+`v2-ladder` stepped up to `grok-4.6` ($2.00/MTok, ~28×). Same config, same
+catalog — the choice moves with trust and latency history, which are live inputs
+to the ranking. Escalation behaviour is therefore **not reproducible run to run**,
+even though it is reliable *within* a run.
+
+### What genuinely never happens
+
+`anthropic/claude-opus-5` was never selected in any run. §4 explains why, and
+that analysis stands: the price ceiling excludes it below `hard`, and at `hard`
+it loses the quality-per-dollar ranking to cheaper models scoring nearly as well.
+
+Whether that is a *problem* is not something this benchmark answers. If
+`grok-4.6` is good enough for the hard turns, the router is behaving correctly
+and cheaply. The one weak signal available: rung 5 failed in `ladder-1` when it
+escalated only as far as `gemini-3.7-flash` (3 turns) and passed in `v2-ladder`
+when it escalated to `grok-4.6` (7 turns). Rung 4 pushes the other way — it
+passed in `ladder-1` without much escalation and failed in `v2-ladder` *with*
+`grok-4.6` on 5 turns. With n=1 per cell these cancel out; nothing is
+established.
+
+### Mid-stream escalation is a separate mechanism, and it never fired
+
+`escalation_signal` — the abort-and-redispatch-upward path for malformed tool
+calls, empty completions and repeated calls — was empty on every routed ladder
+turn, as was `attempt > 0`. That is consistent with "no turn ever came back
+malformed" rather than evidence of a defect; the ladder gave it nothing to react
+to.
 
 ## 4. Root cause: two gates, both hard
 

package/omp-extension/embed-logic.ts

@@ -80,17 +80,29 @@ export function deriveAgentdoxScope(cwd: string): string {
 }
 
 /**
- * Resolves the desired bind port: an explicit `AUTO_MODEL_ROUTER_PORT` when set and
- * valid, else `0` so Bun assigns a free ephemeral port (the "random port" that
- * lets multiple local omp sessions coexist without colliding). Returning 0
- * means "let the OS pick"; the caller must read the actual port back off the
- * started server.
+ * Resolves the port the embedded router should serve on, in precedence order:
+ * an explicit `AUTO_MODEL_ROUTER_PORT`, else the configured `server.port`, else
+ * 0 (let the OS pick a free one).
+ *
+ * A STABLE port is what keeps omp's model resolution honest. omp resolves
+ * `modelRoles.default` from `models.yml` during startup — BEFORE extensions
+ * load, so before this session can bind and rewrite that file. With an
+ * ephemeral port the block names the PREVIOUS session's port, which is dead
+ * once that session exits, and every main-agent turn fails with "Unable to
+ * connect" while utility calls (resolved later, from the live registration)
+ * still work. A deterministic port makes the pre-bind block correct by
+ * construction. Sessions sharing that port share one router, which is already
+ * how subagents behave.
+ *
+ * `0` is still honoured when asked for explicitly, and remains the fallback
+ * when the desired port is occupied by something that is not our router.
  */
-export function resolveEmbedPort(envPort: string | undefined): number {
+export function resolveEmbedPort(envPort: string | undefined, configuredPort = 0): number {
 	if (envPort !== undefined && envPort !== "") {
 		const port = Number.parseInt(envPort, 10);
 		if (Number.isInteger(port) && port >= 0 && port <= 65_535) return port;
 	}
+	if (Number.isInteger(configuredPort) && configuredPort > 0 && configuredPort <= 65_535) return configuredPort;
 	return 0;
 }
 

package/omp-extension/router-embed.ts

@@ -86,8 +86,6 @@ function registerRouterProvider(pi: ExtensionAPI, port: number, cfg: RouterConfi
 export default function (pi: ExtensionAPI): void {
 	pi.setLabel("auto-model-router embed");
 
-	const requestedPort = resolveEmbedPort(process.env.AUTO_MODEL_ROUTER_PORT);
-
 	// Shared port file, written only by the main session's router.
 	const homeRaw = process.env.AUTO_MODEL_ROUTER_HOME ?? join(homedir(), ".auto-model-router");
 	const home =
@@ -95,11 +93,15 @@ export default function (pi: ExtensionAPI): void {
 			? join(homedir(), homeRaw.slice(1))
 			: homeRaw;
 	const portFile = embedPortPath(home);
-	const cfg = loadConfig({ overrides: { server: { host: "127.0.0.1", port: requestedPort } } });
+	// Load first WITHOUT a port override so `server.port` from config.yml is
+	// visible, then let it (or the env var) decide the bind port.
+	const cfg = loadConfig({ overrides: { server: { host: "127.0.0.1" } } });
+	const requestedPort = resolveEmbedPort(process.env.AUTO_MODEL_ROUTER_PORT, cfg.server.port);
+	cfg.server.port = requestedPort;
 
 	let app: StartedServer | null = null;
 
-	pi.on("session_start", (_event, ctx) => {
+	pi.on("session_start", async (_event, ctx) => {
 		// The omp UI session id tags every request so the toast can scope its
 		// notifications to that exact session (see router-toast.ts).
 		const sessionId = ctx.sessionManager.getSessionId();
@@ -108,7 +110,7 @@ export default function (pi: ExtensionAPI): void {
 			// shared router: one process, one ledger, one place to inspect.
 			// The main writes the port file before spawning subagents.
 			const shared = readEmbedPort(portFile);
-			if (shared !== null && probeEmbed(shared)) {
+			if (shared !== null && (await probeEmbed(shared))) {
 				registerRouterProvider(pi, shared, cfg, sessionId);
 				return;
 			}
@@ -128,12 +130,35 @@ export default function (pi: ExtensionAPI): void {
 			return;
 		}
 
-		// Main interactive session: bind the router once, then register the
-		// provider against the exact bound port. Registration happens only
-		// here — never at factory load, where a stale shared port would be
-		// captured into omp's model registry and defeat the correct bound URL.
+		// Main interactive session. The port is deterministic (see
+		// resolveEmbedPort), which matters because omp resolves
+		// `modelRoles.default` from models.yml BEFORE this extension loads: the
+		// URL that block names must be one this session will actually serve.
 		if (app) return;
-		const started = startServer(cfg);
+
+		// Another live session already serving this port? Share it rather than
+		// fighting over the socket — subagents already share one router, and the
+		// ledger and DB are shared regardless.
+		if (requestedPort !== 0 && (await probeEmbed(requestedPort))) {
+			writeEmbedPort(portFile, requestedPort);
+			syncModelsYml(cfg, requestedPort);
+			registerRouterProvider(pi, requestedPort, cfg, sessionId);
+			pi.setLabel(`auto-model-router embed (shared :${requestedPort})`);
+			return;
+		}
+
+		// Bind the desired port; if something that is NOT our router holds it,
+		// fall back to an ephemeral port rather than leaving the session with no
+		// provider at all. models.yml is rewritten either way, so headless runs
+		// and subagents still resolve.
+		let started: StartedServer;
+		try {
+			started = startServer(cfg);
+		} catch (err) {
+			if (requestedPort === 0) throw err;
+			cfg.server.port = 0;
+			started = startServer(cfg);
+		}
 		const actualPort = started.server.port;
 		if (actualPort === undefined) return;
 		app = started;
@@ -145,8 +170,7 @@ export default function (pi: ExtensionAPI): void {
 		// subagent processes resolve models from models.yml in a FRESH registry
 		// — extension registration does not reach them — so without this they
 		// fail with "Model not found" when no interactive session is live
-		// (the print-mode gap the external benchmark hit). Registration still
-		// wins at runtime, so a live session always overrides a stale block.
+		// (the print-mode gap the external benchmark hit).
 		const syncAction = syncModelsYml(cfg, actualPort);
 		if (syncAction !== null) pi.setLabel(`auto-model-router embed (models.yml ${syncAction})`);
 		registerRouterProvider(pi, actualPort, cfg, sessionId);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "auto-model-router",
-  "version": "0.2.22",
+  "version": "0.2.23",
   "private": false,
   "description": "Local cost/complexity-aware model router for Oh My Pi, backed by OpenRouter",
   "type": "module",

package/test/embed-logic.test.ts

@@ -16,7 +16,7 @@ import {
 } from "../omp-extension/embed-logic.ts";
 
 describe("resolveEmbedPort", () => {
-	test("returns 0 (let the OS assign a free port) when AUTO_MODEL_ROUTER_PORT is absent", () => {
+	test("returns 0 (let the OS assign a free port) when nothing is configured", () => {
 		expect(resolveEmbedPort(undefined)).toBe(0);
 		expect(resolveEmbedPort("")).toBe(0);
 	});
@@ -31,6 +31,28 @@ describe("resolveEmbedPort", () => {
 		expect(resolveEmbedPort("-1")).toBe(0);
 		expect(resolveEmbedPort("70000")).toBe(0);
 	});
+
+	// A stable port is what keeps omp's PRE-extension model resolution correct:
+	// it reads models.yml before this extension can bind and rewrite it, so an
+	// ephemeral port leaves that block naming the previous session's dead port.
+	test("uses the configured server.port when no env override is set", () => {
+		expect(resolveEmbedPort(undefined, 8788)).toBe(8788);
+		expect(resolveEmbedPort("", 8788)).toBe(8788);
+	});
+
+	test("the env var wins over the configured port", () => {
+		expect(resolveEmbedPort("8812", 8788)).toBe(8812);
+	});
+
+	test("an explicit env 0 wins, so an ephemeral port stays requestable", () => {
+		expect(resolveEmbedPort("0", 8788)).toBe(0);
+	});
+
+	test("ignores a nonsense configured port rather than binding it", () => {
+		expect(resolveEmbedPort(undefined, 0)).toBe(0);
+		expect(resolveEmbedPort(undefined, -5)).toBe(0);
+		expect(resolveEmbedPort(undefined, 70_000)).toBe(0);
+	});
 });
 
 describe("embed port file", () => {

package/tools/verify-embed-port.ts

@@ -0,0 +1,91 @@
+/**
+ * Verifies the embed port contract that omp's model resolution depends on.
+ *
+ * omp resolves `modelRoles.default` from `models.yml` during startup — BEFORE
+ * extensions load, so before the router can bind and rewrite that file. With an
+ * ephemeral port the block therefore names the PREVIOUS session's port, which
+ * is dead once that session exits: every main-agent turn fails with "Unable to
+ * connect" while utility calls (resolved later, from the live registration)
+ * still work. This asserts the properties that make that impossible:
+ *
+ *  1. The bind port is deterministic across restarts (configured port wins).
+ *  2. A second session finds the first one healthy, so it can REUSE it.
+ *  3. A port held by something that is not our router is not mistaken for one,
+ *     and binding falls back instead of leaving the session with no provider.
+ *
+ * Run: bun tools/verify-embed-port.ts
+ */
+import { mkdtempSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { probeEmbed, resolveEmbedPort } from "../omp-extension/embed-logic.ts";
+import { loadConfig } from "../src/config/load.ts";
+import { startServer, type StartedServer } from "../src/server/http.ts";
+import type { RouterConfig } from "../src/config/types.ts";
+
+let failures = 0;
+const check = (label: string, ok: boolean, detail?: unknown): void => {
+	console.log(`  ${ok ? "PASS" : "FAIL"}  ${label}${ok || detail === undefined ? "" : `\n        ${JSON.stringify(detail)}`}`);
+	if (!ok) failures++;
+};
+
+const home = mkdtempSync(join(tmpdir(), "verify-embed-port-"));
+function baseCfg(): RouterConfig {
+	const cfg = loadConfig({ overrides: { server: { host: "127.0.0.1" } } });
+	cfg.ledger.path = join(home, "router.db");
+	cfg.logLevel = "error";
+	cfg.benchmarks.enabled = false;
+	cfg.context.enabled = false;
+	return cfg;
+}
+
+// 1. Determinism: two independent startups must choose the same port.
+const cfg1 = baseCfg();
+const configured = cfg1.server.port;
+const portA = resolveEmbedPort(undefined, configured);
+const portB = resolveEmbedPort(undefined, baseCfg().server.port);
+check("bind port is deterministic across restarts", portA === portB && portA !== 0, { portA, portB });
+check("deterministic port comes from config.yml server.port", portA === configured, { portA, configured });
+check("an explicit env port still wins", resolveEmbedPort("8812", configured) === 8812);
+check("an explicit env 0 still requests an ephemeral port", resolveEmbedPort("0", configured) === 0);
+
+// 2. First session binds it; a second must see it healthy (=> reuse, no bind war).
+cfg1.server.port = portA;
+let first: StartedServer | null = null;
+try {
+	first = startServer(cfg1);
+} catch (err) {
+	check("first session can bind the deterministic port", false, String(err));
+}
+
+if (first !== null) {
+	check("first session bound the deterministic port", first.server.port === portA, { bound: first.server.port, wanted: portA });
+	check("router answers /health there (so a peer session reuses it)", await probeEmbed(portA), { port: portA });
+
+	// 3. A non-router occupant must not be mistaken for our router.
+	const squatter = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("not a router", { status: 404 }) });
+	const squatted = squatter.port as number;
+	check("a non-router occupant fails the health probe", (await probeEmbed(squatted)) === false, { port: squatted });
+
+	// ...and binding over it must still leave the session with a usable router.
+	const cfgFallback = baseCfg();
+	cfgFallback.server.port = squatted;
+	let fell: StartedServer | null = null;
+	try {
+		fell = startServer(cfgFallback);
+		check("bind on an occupied port yields a different, usable port", fell.server.port !== squatted, { got: fell.server.port });
+	} catch {
+		cfgFallback.server.port = 0;
+		fell = startServer(cfgFallback);
+		check("falls back to an ephemeral port when the desired one is taken", fell.server.port !== undefined && fell.server.port !== squatted, {
+			got: fell.server.port,
+		});
+	}
+	if (fell !== null) await fell.stop();
+	squatter.stop(true);
+	await first.stop();
+}
+
+console.log(failures === 0 ? "\nAll checks passed." : `\n${failures} check(s) FAILED.`);
+process.exit(failures === 0 ? 0 : 1);