auto-model-router 0.2.22 → 0.2.24

package/.claude/skills/agentdox/SKILL.md

@@ -43,7 +43,8 @@ Deterministic, so the same folder always resolves to the same slug:
      stop and ask for a distinguishing slug.
    - **Near match** (`my-app` vs `my-app-v2`) — a stop sign: ask, rather than fork a second
      namespace for one project.
-4. `project_ensure {slug, name}` — `name` is the readable form of the folder.
+4. `project_ensure {slug}` — `name` is optional and only used if the project is new; it
+   defaults to the slug, so pass one only when the readable form differs.
 5. **Make `.env.agentdox` un-committable *before* writing it.** Run `git check-ignore -v
    .env.agentdox`. If it is not ignored, **add `.env.agentdox` to `.gitignore`** (create that
    file if there is none) — do not merely check and move on. Patterns like `.env` and
@@ -136,7 +137,7 @@ await fetch("http://localhost:3003/memory", { method: "POST", headers: H,
 
 | Step | MCP | REST |
 | --- | --- | --- |
-| Ensure the project | `project_ensure {slug, name}` | `POST /projects {slug,name}` (idempotent) |
+| Ensure the project | `project_ensure {slug}` | `POST /projects {slug}` (idempotent) |
 | Read the brief | `context_brief {scope}` | `GET /context/brief?scope=<scope>` |
 
 The brief is the cumulative on-ramp: overview, repo layout, code style, build/test,

package/README.md

@@ -32,6 +32,93 @@ This router exists for the things a prompt classifier structurally cannot do:
 | **Closed-loop trust** | Per-model escalation and error rates from *your* traffic demote cheap-but-flaky models automatically. |
 | **Explainability** | Every decision — candidates, rejections, forecasts, reasons — is persisted and replayable via `auto-model-router explain`. |
 
+## Measured against Claude Opus 5
+
+Five benchmark runs, 88 graded task runs, 2026-08-29. Each task is a real omp
+session working in a pristine git workspace from a written spec. Hidden tests are
+copied in only *after* the agent exits, so they cannot be read or edited by it;
+every task is verified to fail an untouched workspace and to pass a reference
+solution. Both arms are metered from omp's own event stream, run under an
+identical tool surface, and are checked per turn against their expected provider.
+The router arm routes freely — nothing pinned. The baseline is `claude-opus-5`
+on Anthropic first-party.
+
+### Core suite — 10 coding tasks × 3 trials
+
+| | auto-model-router | Claude Opus 5 |
+| --- | --- | --- |
+| Tasks solved | **30 / 30** | 30 / 30 |
+| Total cost | **$0.63** | $16.61 |
+| Cost per solved task | **$0.0209** | $0.5538 |
+| Turns to finish | **278** | 303 |
+| Tool calls | **265** | 337 |
+| Wall clock | **2 057 s** | 3 185 s |
+| Median time to first token | 5 776 ms | **1 490 ms** |
+
+**26.5× cheaper at identical correctness** — and in fewer turns, fewer tool
+calls, and 19 minutes less wall clock. The saving is not bought by grinding out
+extra turns. The one regression is time to first token: a routed turn pays for
+classification and dispatch before anything streams back.
+
+Per task the ratio ranges from 9× to 264×. The widest gaps are tasks where the
+single-model baseline entered long tool loops — `semver` and `queue-order` cost
+it $2.99 each across three trials against a $1.32 median, 36% of its entire bill.
+
+### Difficulty ladder — 7 rungs, run twice
+
+A second suite of deliberately escalating difficulty, ending in npm semver range
+semantics and a minimal diff with a specified tie-break.
+
+| | auto-model-router | Claude Opus 5 |
+| --- | --- | --- |
+| Run 1 | 5 / 7 · $0.30 | 5 / 7 · $6.25 |
+| Run 2 | 5 / 7 · $0.46 | **6 / 7** · $6.60 |
+
+At the top of the ladder the engines separate: they fail different rungs, and on
+the second run the single-model baseline finished one more. Both arms timed out
+on the semver rung at the 10-minute cap.
+
+### What it routed to
+
+Across 464 routed turns in all five runs:
+
+| Model | Turns | Input price | Role |
+| --- | --- | --- | --- |
+| `z-ai/glm-5.3-flash` | 389 (84%) | $0.07 / MTok | default |
+| `google/gemini-3.7-flash` | 56 (12%) | $0.75 / MTok | escalation target |
+| `x-ai/grok-4.6` | 18 (4%) | $2.00 / MTok | escalation target |
+
+**Tier escalation converts to a costlier model roughly one-for-one**: on the
+ladder, the count of turns classified `hard` matched the count served by
+something other than the default (6/6, 4/4, 3/3, 5/5, 7/7, 1/1 across rungs and
+runs). The escalation *target* is chosen live from trust and latency history, so
+it differs between runs on the same catalog — run 1 stepped up to
+`gemini-3.7-flash`, run 2 to `grok-4.6`.
+
+Escalation stays inside the cheaper half of the catalog. A model priced above a
+tier's `maxInputPerMtok` is excluded before ranking, and at `hard` the
+`(quality/100)^qualityExponent ÷ expected cost` score favours cheaper models that
+score nearly as well. If your workload needs a frontier model on hard turns,
+raise the tier price ceiling and `qualityExponent` — measured thresholds are in
+[`docs/routing-benchmark-findings.md`](docs/routing-benchmark-findings.md).
+
+### Scope
+
+These are small, self-contained tasks of one to three files, solved in under 25
+turns. On the core suite both engines solved everything, so it measures cost at
+equal correctness rather than capability; the ladder is where capability
+separates. The cost multiple varied between 14× and 32× across runs depending on
+which task the baseline stalled on — treat "well over an order of magnitude" as
+the claim, not a specific figure.
+
+For sustained work on a large codebase the economics differ: cost there is
+dominated by the conversation being resent each turn rather than by per-token
+price. Replaying a week of real omp traffic (6 918 billed turns, 410:1
+input-to-output) against a single-model baseline gives **≈15×**.
+
+Harness, tasks and raw per-turn data:
+[`docs/routing-benchmark-findings.md`](docs/routing-benchmark-findings.md).
+
 ## Architecture
 
 ```mermaid

package/docs/routing-benchmark-findings.md

@@ -13,15 +13,18 @@ of my own earlier conclusions were wrong and are corrected in place.
 
 1. **The router is 20–32× cheaper at equal solve rate.** On 10 easy tasks both
    arms solved 20/20; on a 7-rung difficulty ladder both solved 5/7.
-2. **It never escalates.** Across 86 routed turns on the ladder — including the
-   rungs it *failed* — there were zero escalation signals, zero retry
-   dispatches, and only two models ever served: `glm-5.3-flash` and
-   `gemini-3.7-flash`.
-3. **Root cause is two independent gates, and both must move.** The price
-   ceiling hard-excludes expensive models below `hard`; and the ranking
-   arithmetic makes quality unable to outweigh price at any sane exponent.
-4. **Escalation does work once both gates are opened** — verified end to end,
-   with `claude-opus-5` serving 3 turns of a task the cheap-only config failed.
+2. **Tier escalation works.** A turn classified `hard` reliably selects a
+   costlier model — the hard-turn count matches the non-flash turn count ~1:1
+   across two independent ladder runs (to `gemini-3.7-flash` in one, to
+   `grok-4.6` in the other). *An earlier version of this document claimed the
+   opposite; that was my error, corrected in §3.*
+3. **What never happens is reaching `claude-opus-5`.** Two gates: the price
+   ceiling excludes it below `hard`, and at `hard` it is outranked by cheaper
+   high-quality models. Whether that is a defect depends on whether
+   `grok-4.6`-class is "good enough" — this benchmark cannot say.
+4. **Mid-stream escalation signals never fired** — `escalation_signal` was empty
+   on every one of ~190 routed ladder turns. That is a different mechanism from
+   tier selection and may simply mean nothing malformed occurred.
 5. **The standalone `serve` process died 4 times inside one normally-completing
    run.** The crash is real and mid-run; the *trigger* is unidentified — my first
    explanation (client disconnect mid-stream) does not reproduce. Details in §7.
@@ -33,6 +36,11 @@ of my own earlier conclusions were wrong and are corrected in place.
 - **Harness**: omp in `-p --mode=json` print mode. Every turn's tokens,
   duration, TTFT and tool calls are read off omp's own event stream, so both
   arms are measured by the same instrument.
+- **Provider guard**: each arm declares its expected omp provider and every
+  completion turn is checked against it — `anthropic` + `claude-opus-5` for the
+  baseline (first-party, never OpenRouter), `benchrouter` for the router arm.
+  Zero violations across the re-runs. Added after an early pilot silently
+  resolved `--model auto` to OpenRouter's own `auto` meta-router.
 - **Router arm**: the standalone `auto-model-router serve` endpoint registered
   as a plain OpenAI-compatible provider — the documented non-omp path (README
   § "Standalone alternative"). Nothing pinned, nothing stubbed; the router
@@ -46,22 +54,29 @@ of my own earlier conclusions were wrong and are corrected in place.
   verified to fail an untouched workspace and to pass a reference solution
   before any run (`bench/validate-ladder.ts`).
 
-> **A note on print mode.** The embedded extension only registers the
-> `auto-model-router` provider inside an interactive session — `ctx.hasUI` is
-> false in `-p`, so it takes the subagent path and looks for an existing
-> `embed.port`. In print mode with no interactive session running, the provider
-> is absent from the model registry entirely and `--model auto-model-router/auto`
-> fails with "Model not found". Worth documenting, or worth having the subagent
-> path fall back to binding its own server.
+> **Print mode — resolved.** During the benchmark the embedded extension only
+> registered the provider inside a session with a UI, so `omp -p` could not see
+> `auto-model-router/auto` at all; that is why the harness drives the standalone
+> `serve` endpoint instead. Fixed upstream in **0.2.20** (`fix(embed): resolve the
+> router in print mode and subagents`) — verified: `omp -p --model
+> auto-model-router/auto` now resolves natively with no standalone server. The
+> harness keeps the `serve` path because it gives the benchmark its own isolated
+> ledger, not because the embedded path is broken.
 
 ---
 
 ## 2. Cost result
 
-| Suite | Router | Opus 5 | Ratio |
-|---|---|---|---|
-| 10 easy tasks × 2 trials | 20/20 · $0.3577 | 20/20 · $11.4040 | 31.9× |
-| 7-rung ladder × 1 trial | 5/7 · $0.3026 | 5/7 · $6.2506 | 20.7× |
+| Suite | Run | Router | Opus 5 | Ratio |
+|---|---|---|---|---|
+| 10 easy tasks | trials 1+2 | 20/20 · $0.3577 | 20/20 · $11.4040 | 31.9× |
+| 10 easy tasks | re-run | 10/10 · $0.2695 | 10/10 · $5.2092 | 19.3× |
+| 7-rung ladder | first | 5/7 · $0.3026 | 5/7 · $6.2506 | 20.7× |
+| 7-rung ladder | re-run | 5/7 · $0.4619 | **6/7** · $6.5967 | 14.3× |
+
+Across four independent runs the ratio lands between **14× and 32×**. On the
+ladder re-run Opus 5 edged the router on correctness for the first time (6/7 vs
+5/7), which is the outcome the ladder was built to be able to detect.
 
 Turn counts are comparable (184 vs 217 on the easy suite; 86 vs 91 on the
 ladder), so the saving is not bought with extra turns. Median TTFT is the
@@ -74,30 +89,56 @@ magnitude" is defensible; a precise figure is not.
 
 ---
 
-## 3. The router never escalates
-
-The ladder was built specifically to force an escalation decision: seven rungs
-ending in npm semver range semantics and a minimal-diff with a specified
-tie-break.
+## 3. Escalation — corrected
 
-| Rung | Task | Router | Opus 5 | Escalated? |
-|---|---|---|---|---|
-| 1 | in-range | PASS | PASS | no |
-| 2 | round-half-even | PASS | PASS | no |
-| 3 | csv-document | PASS | PASS | no |
-| 4 | sliding-limiter | PASS | **FAIL** | no |
-| 5 | savepoints | **FAIL** | PASS | **no** |
-| 6 | semver-ranges | timeout | timeout | no |
-| 7 | minimal-diff | PASS | PASS | no |
+**I got this wrong first time.** I reported that the router "never escalates",
+having looked at the served-model list, seen only `glm-5.3-flash` and
+`gemini-3.7-flash`, and mentally filed both as "flash models, therefore no
+escalation". I never checked whether the counts lined up with the tier. They do.
 
-Rung 6 timed out on both arms at the 10-minute cap; it is not evidence about
-capability either way.
+### Tier escalation works, and is close to deterministic
 
-**The router assigned its `hard` tier on six of the seven rungs**, so the
-classifier is not blind — the tier simply never converts into a more capable
-model. On rung 5 it failed while continuing to dispatch to `glm-5.3-flash`.
+Hard-tier turns versus turns served by something other than the cheap default:
 
----
+| Rung | ladder-1 hard / non-flash | v2-ladder hard / non-flash |
+|---|---|---|
+| L1 in-range | 0 / 0 | 0 / 0 |
+| L2 round-half-even | 6 / 6 | 0 / 0 |
+| L3 csv-document | 4 / 4 | 0 / 0 |
+| L4 sliding-limiter | 3 / 6 | 5 / 5 |
+| L5 savepoints | 3 / 3 | 7 / 7 |
+| L6 semver-ranges | 2 / 3 | 0 / 0 |
+| L7 minimal-diff | 3 / 4 | 1 / 1 |
+
+The escalation *target* differs by run, which is itself worth knowing:
+`ladder-1` stepped up to `gemini-3.7-flash` ($0.75/MTok, ~10× the default);
+`v2-ladder` stepped up to `grok-4.6` ($2.00/MTok, ~28×). Same config, same
+catalog — the choice moves with trust and latency history, which are live inputs
+to the ranking. Escalation behaviour is therefore **not reproducible run to run**,
+even though it is reliable *within* a run.
+
+### What genuinely never happens
+
+`anthropic/claude-opus-5` was never selected in any run. §4 explains why, and
+that analysis stands: the price ceiling excludes it below `hard`, and at `hard`
+it loses the quality-per-dollar ranking to cheaper models scoring nearly as well.
+
+Whether that is a *problem* is not something this benchmark answers. If
+`grok-4.6` is good enough for the hard turns, the router is behaving correctly
+and cheaply. The one weak signal available: rung 5 failed in `ladder-1` when it
+escalated only as far as `gemini-3.7-flash` (3 turns) and passed in `v2-ladder`
+when it escalated to `grok-4.6` (7 turns). Rung 4 pushes the other way — it
+passed in `ladder-1` without much escalation and failed in `v2-ladder` *with*
+`grok-4.6` on 5 turns. With n=1 per cell these cancel out; nothing is
+established.
+
+### Mid-stream escalation is a separate mechanism, and it never fired
+
+`escalation_signal` — the abort-and-redispatch-upward path for malformed tool
+calls, empty completions and repeated calls — was empty on every routed ladder
+turn, as was `attempt > 0`. That is consistent with "no turn ever came back
+malformed" rather than evidence of a defect; the ladder gave it nothing to react
+to.
 
 ## 4. Root cause: two gates, both hard
 

package/omp-extension/embed-logic.ts

@@ -80,20 +80,53 @@ export function deriveAgentdoxScope(cwd: string): string {
 }
 
 /**
- * Resolves the desired bind port: an explicit `AUTO_MODEL_ROUTER_PORT` when set and
- * valid, else `0` so Bun assigns a free ephemeral port (the "random port" that
- * lets multiple local omp sessions coexist without colliding). Returning 0
- * means "let the OS pick"; the caller must read the actual port back off the
- * started server.
+ * Resolves the port the embedded router should serve on, in precedence order:
+ * an explicit `AUTO_MODEL_ROUTER_PORT`, else the configured `server.port`, else
+ * 0 (let the OS pick a free one).
+ *
+ * A STABLE port is what keeps omp's model resolution honest. omp resolves
+ * `modelRoles.default` from `models.yml` during startup — BEFORE extensions
+ * load, so before this session can bind and rewrite that file. With an
+ * ephemeral port the block names the PREVIOUS session's port, which is dead
+ * once that session exits, and every main-agent turn fails with "Unable to
+ * connect" while utility calls (resolved later, from the live registration)
+ * still work. A deterministic port makes the pre-bind block correct by
+ * construction. Sessions sharing that port share one router, which is already
+ * how subagents behave.
+ *
+ * `0` is still honoured when asked for explicitly, and remains the fallback
+ * when the desired port is occupied by something that is not our router.
  */
-export function resolveEmbedPort(envPort: string | undefined): number {
+export function resolveEmbedPort(envPort: string | undefined, configuredPort = 0): number {
 	if (envPort !== undefined && envPort !== "") {
 		const port = Number.parseInt(envPort, 10);
 		if (Number.isInteger(port) && port >= 0 && port <= 65_535) return port;
 	}
+	if (Number.isInteger(configuredPort) && configuredPort > 0 && configuredPort <= 65_535) return configuredPort;
 	return 0;
 }
 
+/**
+ * The port omp's `models.yml` currently advertises for our provider, or null
+ * when the block is absent or unparseable.
+ *
+ * This is the port omp resolves `modelRoles.default` against DURING STARTUP,
+ * before this extension loads. If it disagrees with the port we end up serving,
+ * this session's main-model handle points somewhere we are not listening, and
+ * only a restart can rebuild it — there is no API to re-resolve an already
+ * built handle. Detecting the mismatch is what turns a baffling
+ * "Unable to connect" on every real turn into a message that names the cause.
+ */
+export function modelsYmlPort(text: string): number | null {
+	const block = /^\s*auto-model-router:\s*$/m.exec(text);
+	if (block === null) return null;
+	const rest = text.slice(block.index);
+	const url = /baseUrl:\s*http:\/\/[^\s:]+:(\d+)/.exec(rest);
+	if (url === null) return null;
+	const port = Number.parseInt(url[1] ?? "", 10);
+	return Number.isInteger(port) ? port : null;
+}
+
 /**
  * Absolute path of the shared embed port file under a router home directory.
  */

package/omp-extension/router-embed.ts

@@ -19,10 +19,11 @@
  *     - /path/to/auto-model-router/omp-extension/router-toast.ts
  */
 
+import { readFileSync } from "node:fs";
 import { homedir } from "node:os";
 import { join } from "node:path";
 
-import { syncModelsYml } from "../src/cli/config-cmd.ts";
+import { ompModelsPath, syncModelsYml } from "../src/cli/config-cmd.ts";
 import { loadConfig } from "../src/config/load.ts";
 import { startServer } from "../src/server/http.ts";
 import type { StartedServer } from "../src/server/http.ts";
@@ -36,11 +37,21 @@ import {
 	EMBED_PROVIDER_ID,
 	embedPortPath,
 	readEmbedPort,
+	modelsYmlPort,
 	probeEmbed,
 	resolveEmbedPort,
 	writeEmbedPort,
 } from "./embed-logic.ts";
 
+/** omp's models.yml as text, or "" when it does not exist / cannot be read. */
+function readModelsYml(): string {
+	try {
+		return readFileSync(ompModelsPath(), "utf8");
+	} catch {
+		return "";
+	}
+}
+
 /**
  * Registers the auto-model-router provider (and its virtual models) into omp's model
  * registry at a specific bound port.
@@ -86,8 +97,6 @@ function registerRouterProvider(pi: ExtensionAPI, port: number, cfg: RouterConfi
 export default function (pi: ExtensionAPI): void {
 	pi.setLabel("auto-model-router embed");
 
-	const requestedPort = resolveEmbedPort(process.env.AUTO_MODEL_ROUTER_PORT);
-
 	// Shared port file, written only by the main session's router.
 	const homeRaw = process.env.AUTO_MODEL_ROUTER_HOME ?? join(homedir(), ".auto-model-router");
 	const home =
@@ -95,11 +104,15 @@ export default function (pi: ExtensionAPI): void {
 			? join(homedir(), homeRaw.slice(1))
 			: homeRaw;
 	const portFile = embedPortPath(home);
-	const cfg = loadConfig({ overrides: { server: { host: "127.0.0.1", port: requestedPort } } });
+	// Load first WITHOUT a port override so `server.port` from config.yml is
+	// visible, then let it (or the env var) decide the bind port.
+	const cfg = loadConfig({ overrides: { server: { host: "127.0.0.1" } } });
+	const requestedPort = resolveEmbedPort(process.env.AUTO_MODEL_ROUTER_PORT, cfg.server.port);
+	cfg.server.port = requestedPort;
 
 	let app: StartedServer | null = null;
 
-	pi.on("session_start", (_event, ctx) => {
+	pi.on("session_start", async (_event, ctx) => {
 		// The omp UI session id tags every request so the toast can scope its
 		// notifications to that exact session (see router-toast.ts).
 		const sessionId = ctx.sessionManager.getSessionId();
@@ -108,7 +121,7 @@ export default function (pi: ExtensionAPI): void {
 			// shared router: one process, one ledger, one place to inspect.
 			// The main writes the port file before spawning subagents.
 			const shared = readEmbedPort(portFile);
-			if (shared !== null && probeEmbed(shared)) {
+			if (shared !== null && (await probeEmbed(shared))) {
 				registerRouterProvider(pi, shared, cfg, sessionId);
 				return;
 			}
@@ -128,12 +141,35 @@ export default function (pi: ExtensionAPI): void {
 			return;
 		}
 
-		// Main interactive session: bind the router once, then register the
-		// provider against the exact bound port. Registration happens only
-		// here — never at factory load, where a stale shared port would be
-		// captured into omp's model registry and defeat the correct bound URL.
+		// Main interactive session. The port is deterministic (see
+		// resolveEmbedPort), which matters because omp resolves
+		// `modelRoles.default` from models.yml BEFORE this extension loads: the
+		// URL that block names must be one this session will actually serve.
 		if (app) return;
-		const started = startServer(cfg);
+
+		// Another live session already serving this port? Share it rather than
+		// fighting over the socket — subagents already share one router, and the
+		// ledger and DB are shared regardless.
+		if (requestedPort !== 0 && (await probeEmbed(requestedPort))) {
+			writeEmbedPort(portFile, requestedPort);
+			syncModelsYml(cfg, requestedPort);
+			registerRouterProvider(pi, requestedPort, cfg, sessionId);
+			pi.setLabel(`auto-model-router embed (shared :${requestedPort})`);
+			return;
+		}
+
+		// Bind the desired port; if something that is NOT our router holds it,
+		// fall back to an ephemeral port rather than leaving the session with no
+		// provider at all. models.yml is rewritten either way, so headless runs
+		// and subagents still resolve.
+		let started: StartedServer;
+		try {
+			started = startServer(cfg);
+		} catch (err) {
+			if (requestedPort === 0) throw err;
+			cfg.server.port = 0;
+			started = startServer(cfg);
+		}
 		const actualPort = started.server.port;
 		if (actualPort === undefined) return;
 		app = started;
@@ -141,14 +177,30 @@ export default function (pi: ExtensionAPI): void {
 		// Publish the shared port; subagents and the toast read it from here.
 		writeEmbedPort(portFile, actualPort);
 
+		// What omp resolved `modelRoles.default` against at STARTUP, before this
+		// extension loaded. If it names a different port than we serve, this
+		// session's main-model handle points at a socket we are not listening on
+		// and every real turn fails with "Unable to connect" while utility calls
+		// (resolved later, from the registration below) still work. Nothing here
+		// can rebuild that handle, so say so plainly instead of leaving the user
+		// to diagnose it.
+		const advertised = modelsYmlPort(readModelsYml());
+		if (advertised !== null && advertised !== actualPort) {
+			pi.setLabel(`auto-model-router: RESTART NEEDED — omp resolved :${advertised}, router serves :${actualPort}`);
+			console.warn(
+				`[auto-model-router] models.yml advertised port ${advertised} at startup but this router serves ${actualPort}. ` +
+					`omp resolves the default model before extensions load, so this session's main model still points at ${advertised}. ` +
+					`models.yml has been corrected — restart omp once and it will be right.`,
+			);
+		}
+
 		// Keep models.yml pointing at this port. Headless runs (`-p`) and
 		// subagent processes resolve models from models.yml in a FRESH registry
 		// — extension registration does not reach them — so without this they
 		// fail with "Model not found" when no interactive session is live
-		// (the print-mode gap the external benchmark hit). Registration still
-		// wins at runtime, so a live session always overrides a stale block.
+		// (the print-mode gap the external benchmark hit).
 		const syncAction = syncModelsYml(cfg, actualPort);
-		if (syncAction !== null) pi.setLabel(`auto-model-router embed (models.yml ${syncAction})`);
+		if (syncAction !== null && advertised === actualPort) pi.setLabel(`auto-model-router embed (models.yml ${syncAction})`);
 		registerRouterProvider(pi, actualPort, cfg, sessionId);
 
 		pi.on("session_shutdown", () => {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "auto-model-router",
-  "version": "0.2.22",
+  "version": "0.2.24",
   "private": false,
   "description": "Local cost/complexity-aware model router for Oh My Pi, backed by OpenRouter",
   "type": "module",

package/test/embed-logic.test.ts

@@ -9,6 +9,7 @@ import {
 	EMBED_PORT_FILE,
 	EMBED_PROVIDER_ID,
 	embedPortPath,
+	modelsYmlPort,
 	readEmbedPort,
 	resolveEmbedPort,
 	writeEmbedPort,
@@ -16,7 +17,7 @@ import {
 } from "../omp-extension/embed-logic.ts";
 
 describe("resolveEmbedPort", () => {
-	test("returns 0 (let the OS assign a free port) when AUTO_MODEL_ROUTER_PORT is absent", () => {
+	test("returns 0 (let the OS assign a free port) when nothing is configured", () => {
 		expect(resolveEmbedPort(undefined)).toBe(0);
 		expect(resolveEmbedPort("")).toBe(0);
 	});
@@ -31,6 +32,68 @@ describe("resolveEmbedPort", () => {
 		expect(resolveEmbedPort("-1")).toBe(0);
 		expect(resolveEmbedPort("70000")).toBe(0);
 	});
+
+	// A stable port is what keeps omp's PRE-extension model resolution correct:
+	// it reads models.yml before this extension can bind and rewrite it, so an
+	// ephemeral port leaves that block naming the previous session's dead port.
+	test("uses the configured server.port when no env override is set", () => {
+		expect(resolveEmbedPort(undefined, 8788)).toBe(8788);
+		expect(resolveEmbedPort("", 8788)).toBe(8788);
+	});
+
+	test("the env var wins over the configured port", () => {
+		expect(resolveEmbedPort("8812", 8788)).toBe(8812);
+	});
+
+	test("an explicit env 0 wins, so an ephemeral port stays requestable", () => {
+		expect(resolveEmbedPort("0", 8788)).toBe(0);
+	});
+
+	test("ignores a nonsense configured port rather than binding it", () => {
+		expect(resolveEmbedPort(undefined, 0)).toBe(0);
+		expect(resolveEmbedPort(undefined, -5)).toBe(0);
+		expect(resolveEmbedPort(undefined, 70_000)).toBe(0);
+	});
+});
+
+describe("modelsYmlPort", () => {
+	// This is the port omp resolves modelRoles.default against at STARTUP,
+	// before the extension loads. A disagreement with the served port means
+	// every main-agent turn in that session fails with "Unable to connect"
+	// while utility calls still work, so the extension has to detect it.
+	const REAL = `providers:
+    # BEGIN auto-model-router
+    auto-model-router:
+      baseUrl: http://127.0.0.1:58724/v1
+      api: openai-completions
+      auth: none
+      models:
+        - id: auto
+          name: Auto (auto-model-router)
+`;
+
+	test("reads the advertised port out of a real block", () => {
+		expect(modelsYmlPort(REAL)).toBe(58724);
+	});
+
+	test("returns null when our provider block is absent", () => {
+		expect(modelsYmlPort("providers:\n    openrouter:\n      baseUrl: https://openrouter.ai/api/v1\n")).toBeNull();
+		expect(modelsYmlPort("")).toBeNull();
+	});
+
+	test("is not fooled by another provider's baseUrl appearing first", () => {
+		const mixed = `providers:
+    llama.cpp:
+      baseUrl: http://127.0.0.1:8080/v1
+    auto-model-router:
+      baseUrl: http://127.0.0.1:8788/v1
+`;
+		expect(modelsYmlPort(mixed)).toBe(8788);
+	});
+
+	test("returns null when the block carries no parseable url", () => {
+		expect(modelsYmlPort("providers:\n    auto-model-router:\n      api: openai-completions\n")).toBeNull();
+	});
 });
 
 describe("embed port file", () => {

package/tools/verify-embed-port.ts

@@ -0,0 +1,97 @@
+/**
+ * Verifies the embed port contract that omp's model resolution depends on.
+ *
+ * omp resolves `modelRoles.default` from `models.yml` during startup — BEFORE
+ * extensions load, so before the router can bind and rewrite that file. With an
+ * ephemeral port the block therefore names the PREVIOUS session's port, which
+ * is dead once that session exits: every main-agent turn fails with "Unable to
+ * connect" while utility calls (resolved later, from the live registration)
+ * still work. This asserts the properties that make that impossible:
+ *
+ *  1. The bind port is deterministic across restarts (configured port wins).
+ *  2. A second session finds the first one healthy, so it can REUSE it.
+ *  3. A port held by something that is not our router is not mistaken for one,
+ *     and binding falls back instead of leaving the session with no provider.
+ *
+ * Run: bun tools/verify-embed-port.ts
+ */
+import { mkdtempSync } from "node:fs";
+import { tmpdir } from "node:os";
+import { join } from "node:path";
+
+import { probeEmbed, resolveEmbedPort } from "../omp-extension/embed-logic.ts";
+import { loadConfig } from "../src/config/load.ts";
+import { startServer, type StartedServer } from "../src/server/http.ts";
+import type { RouterConfig } from "../src/config/types.ts";
+
+let failures = 0;
+const check = (label: string, ok: boolean, detail?: unknown): void => {
+	console.log(`  ${ok ? "PASS" : "FAIL"}  ${label}${ok || detail === undefined ? "" : `\n        ${JSON.stringify(detail)}`}`);
+	if (!ok) failures++;
+};
+
+const home = mkdtempSync(join(tmpdir(), "verify-embed-port-"));
+function baseCfg(): RouterConfig {
+	const cfg = loadConfig({ overrides: { server: { host: "127.0.0.1" } } });
+	cfg.ledger.path = join(home, "router.db");
+	cfg.logLevel = "error";
+	cfg.benchmarks.enabled = false;
+	cfg.context.enabled = false;
+	return cfg;
+}
+
+// 1. Determinism: two independent startups must choose the same port.
+const cfg1 = baseCfg();
+const configured = cfg1.server.port;
+const portA = resolveEmbedPort(undefined, configured);
+const portB = resolveEmbedPort(undefined, baseCfg().server.port);
+check("bind port is deterministic across restarts", portA === portB && portA !== 0, { portA, portB });
+check("deterministic port comes from config.yml server.port", portA === configured, { portA, configured });
+check("an explicit env port still wins", resolveEmbedPort("8812", configured) === 8812);
+check("an explicit env 0 still requests an ephemeral port", resolveEmbedPort("0", configured) === 0);
+
+// 2. First session binds it; a second must see it healthy (=> reuse, no bind war).
+//    Uses a DISCOVERED free port, not the machine's configured one: a real
+//    router may already be serving 8788 here, and this check must be hermetic.
+const scout = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("scout") });
+const freePort = scout.port as number;
+scout.stop(true);
+cfg1.server.port = freePort;
+
+let first: StartedServer | null = null;
+try {
+	first = startServer(cfg1);
+} catch (err) {
+	check("first session can bind its chosen port", false, String(err));
+}
+
+if (first !== null) {
+	check("first session bound the port it asked for", first.server.port === freePort, { bound: first.server.port, wanted: freePort });
+	check("router answers /health there (so a peer session reuses it)", await probeEmbed(freePort), { port: freePort });
+
+	// 3. A non-router occupant must not be mistaken for our router.
+	const squatter = Bun.serve({ port: 0, hostname: "127.0.0.1", fetch: () => new Response("not a router", { status: 404 }) });
+	const squatted = squatter.port as number;
+	check("a non-router occupant fails the health probe", (await probeEmbed(squatted)) === false, { port: squatted });
+
+	// ...and binding over it must still leave the session with a usable router.
+	const cfgFallback = baseCfg();
+	cfgFallback.server.port = squatted;
+	let fell: StartedServer | null = null;
+	try {
+		fell = startServer(cfgFallback);
+		check("bind on an occupied port yields a different, usable port", fell.server.port !== squatted, { got: fell.server.port });
+	} catch {
+		cfgFallback.server.port = 0;
+		fell = startServer(cfgFallback);
+		check("falls back to an ephemeral port when the desired one is taken", fell.server.port !== undefined && fell.server.port !== squatted, {
+			got: fell.server.port,
+		});
+	}
+	if (fell !== null) await fell.stop();
+	squatter.stop(true);
+	await first.stop();
+}
+
+console.log(failures === 0 ? "\nAll checks passed." : `\n${failures} check(s) FAILED.`);
+process.exit(failures === 0 ? 0 : 1);