llm-relay 0.0.1

package/README.md

@@ -0,0 +1,236 @@
+# llm-relay
+
+A standalone, **loopback** Anthropic-Messages-API reverse proxy. It forwards `/v1/messages` to any backend model and **validates tool-call responses** against the request's `tools[].input_schema`, so the Claude Code harness (or any `ANTHROPIC_BASE_URL` client) can run on non-Anthropic models without pre-filtering them by tool competence.
+
+**The one boundary:** it fixes/flags *protocol form* (malformed tool calls), never *judgment* (bad reasoning).
+
+## What it does
+
+- **Transparent passthrough** — forwards streaming and non-streaming `/v1/messages` byte-for-byte.
+- **`detect` mode** — deterministic tool_use validation (Ajv2020) with metadata-only logging of pass/fail/uncheckable. Behavior is unchanged; it only observes.
+- **`repair` mode** — on a validation failure, a cheap reshaper model corrects the call, the result is **re-validated**, and the corrected response is re-emitted (JSON or freshly-serialized SSE). Destructive-tool calls are **refused, never fabricated**; unrepairable calls **fail-clean** (502). Valid calls pass through untouched.
+- **OpenAI-compatible backends** (`backend.kind:"openai"`) — front NIM / vLLM / OpenRouter / LM Studio. Requests are translated Anthropic→OpenAI and responses back (streaming SSE + non-streaming) via [`llm-bridge`](https://github.com/supermemoryai/llm-bridge) (zero-dep). The validate/repair layer always sees Anthropic Messages, regardless of backend. Verified live end-to-end.
+- **Streaming repair** — text-block SSE frames stream to the client **as they arrive**; the proxy only withholds from the first `tool_use` block. A pure-text response is byte-for-byte passthrough with zero added latency; a valid tool call flushes the withheld frames verbatim; an invalid one is repaired with only the corrected trailing blocks re-emitted (`message_start` + leading text already delivered). A mid-stream repair failure surfaces as an SSE `error` event, never a fabricated call. Handles LF and CRLF frame delimiters and multibyte UTF-8 across chunk boundaries.
+
+### Live demo (no external creds)
+
+```bash
+npm run build && node scripts/live-demo.mjs
+```
+Runs the compiled CLI as a real process against a local flaky-model backend + stub reshaper, showing detect (logs the failure) then repair (delivers the fixed call).
+
+## Install & run
+
+```bash
+npm install
+npm run build
+cp config.example.json config.json   # edit backend + auth
+NVIDIA_API_KEY=nvapi-... node dist/cli.js --config config.json
+# or, no build step:
+NVIDIA_API_KEY=nvapi-... npm run dev -- --config config.json
+```
+
+## Use it from your projects
+
+Point the `claude` CLI at the running proxy. **The one thing that matters:** give claude an **isolated `CLAUDE_CONFIG_DIR`**. Without it, an active claude.ai subscription session conflicts with the proxy token and claude fails client-side with `Invalid API key` / `401 Invalid bearer token` before any request is even sent. With it, the proxy's provider token is the sole credential — and your subscription is never in the path (the safe direction).
+
+Wrappers do this for you (they also set the thinking/beta/attribution flags the harness needs against a non-Anthropic model):
+
+```powershell
+# PowerShell (from any project directory)
+C:\Code\repair-proxy\scripts\claude-proxied.ps1 -p "list the files here"
+```
+```bash
+# bash
+/c/Code/repair-proxy/scripts/claude-proxied.sh -p "list the files here"
+```
+
+Or inline, if you'd rather not use the wrapper:
+
+```bash
+env -u CLAUDECODE -u ANTHROPIC_API_KEY \
+  CLAUDE_CONFIG_DIR="$HOME/.repair-proxy-claude" \
+  ANTHROPIC_BASE_URL=http://127.0.0.1:8791 \
+  ANTHROPIC_AUTH_TOKEN=dummy \
+  CLAUDE_CODE_DISABLE_ADAPTIVE_THINKING=1 CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1 CLAUDE_CODE_ATTRIBUTION_HEADER=0 \
+  claude -p "list the files here"
+```
+
+`ANTHROPIC_AUTH_TOKEN` can be `dummy` — the proxy strips inbound auth and injects the real backend key itself (from `authEnv`). Override the wrapper defaults with `RP_PROXY_URL`, `RP_AUTH`, `RP_CONFIG_DIR`. Verified live end-to-end: a real `claude` agentic session (tool_use → tool_result → answer) completes through the proxy against NIM.
+
+> Backend note: weak models still fail *reasoning* (they may loop or skip a tool) — repair fixes malformed tool-call *form*, not judgment. Pick a strong tool-caller as the backend model. NIM also rate-limits (HTTP 429) under load; claude's own retry/backoff absorbs it.
+
+## Config — multi-provider registry
+
+A `providers{}` registry (any number of OpenAI-compatible or Anthropic backends) plus
+a `routing` block that maps each request's `model` to one provider + backend model:
+
+```jsonc
+{
+  "listen": "127.0.0.1:8791",              // loopback ONLY — startup refuses non-loopback
+  "providers": {
+    "nim":        { "base": "https://integrate.api.nvidia.com/v1", "kind": "openai", "authEnv": "NVIDIA_API_KEY" },
+    "openrouter": { "base": "https://openrouter.ai/api/v1",        "kind": "openai", "authEnv": "OPENROUTER_API_KEY" },
+    "gemini":     { "base": "https://generativelanguage.googleapis.com/v1beta/openai", "kind": "openai", "authEnv": "GEMINI_API_KEY" }
+  },
+  "routing": {
+    "default": "nim/z-ai/glm-5.2",           // fallback when nothing else matches
+    "tiers": {                                // Claude tier (substring match) → provider/model
+      "opus":   "nim/nvidia/nemotron-3-super-120b-a12b",
+      "sonnet": "nim/z-ai/glm-5.2",
+      "haiku":  "nim/openai/gpt-oss-20b",     // cheap/fast — also catches Claude's haiku side-calls
+      "fable":  "nim/openai/gpt-oss-20b"
+    }
+  },
+  "mode": "repair",                          // detect | repair (strict accepted, aliases detect)
+  "repair": { "maxAttempts": 2, "destructiveTools": ["rm","delete","push","force","overwrite","drop","reset"] },
+  "log": { "level": "metadata", "file": null }  // metadata-only; NEVER logs headers/bodies
+}
+```
+
+**Routing (lifted from free-claude-code's proven scheme — split on the first `/` only):**
+1. **Namespaced** — a request `model` of `provider/rest` where `provider` is a configured
+   provider routes there directly; the entire tail (nested slashes, `:free` suffixes) is the
+   backend model, verbatim. E.g. `nim/openai/gpt-oss-120b`, `openrouter/openai/gpt-5.2-codex`.
+2. **Tier** — otherwise the Claude model id is substring-matched against `routing.tiers`
+   (`opus`/`sonnet`/`haiku`/`fable`). This also fixes Claude's haiku-class side-calls, which
+   would otherwise blindly hit one model and 404.
+3. **Default** — anything unrecognized falls to `routing.default`.
+
+Each provider is `kind:"openai"` (translated Anthropic↔OpenAI via llm-bridge) or
+`kind:"anthropic"` (forwarded as-is). In `repair` mode an openai target reshapes on itself;
+an anthropic provider needs an explicit top-level `reshaper` block.
+
+### Repointing without editing the file
+
+Config strings may reference env vars as `${NAME}` (unset → loud startup error). Or override
+routing from the CLI (wins over the file):
+
+```bash
+node dist/cli.js --config config.json --default openrouter/openai/gpt-5.2-codex --mode repair
+```
+
+`repair-proxy --help` lists every override.
+
+### Model discovery (dynamic + cached)
+
+Model ids are **discovered live** from each provider's `/models` endpoint — never
+hand-maintained. The catalog is cached in `~/.repair-proxy/models-cache.json`
+(10-min TTL, fail-open: a fetch failure serves the last-known list).
+
+```bash
+repair-proxy models                      # list live models for every provider
+repair-proxy models --provider nim       # one provider
+repair-proxy models --provider nim --refresh   # force a re-fetch
+```
+
+On startup the proxy warms the cache and **warns about any routing target its
+provider doesn't serve** — so a stale/typo'd tier model is caught at boot, not
+silently at request time.
+
+> Provider notes: **Groq** returns `403 "check your network settings"` from some
+> IPs/regions (a network-side block, not a key issue) — it works once your network
+> allows it. **Mistral** needs `MISTRAL_API_KEY` set in your environment.
+
+### Discovery endpoint (`GET /registry`) — for a dispatcher
+
+For a caller that does its own selection (e.g. audit-tools dispatch, which weighs
+quota / rate limits / token budget), `GET http://127.0.0.1:8791/registry` returns one
+coherent JSON view:
+
+- **providers** — each with `base`, `kind`, `has_key` (auth env set?), `reachable`
+  (did the live `/models` catalog return anything?), and `models[]` where every model
+  carries a best-effort `capability` (raw BFCL + Arena scores, **never collapsed** to
+  tiers — `null` when no confident leaderboard match).
+- **routing** — the current default + tier map.
+- **capability_source** — the full raw leaderboard dataset, so a consumer can run a
+  finer id→score join than the built-in best-effort one.
+
+The consumer then dispatches by pointing its OpenAI-compatible pool at :8791 and
+setting each packet's model to a **namespaced** `provider/model` (it picked the exact
+backend). repair-proxy exposes an **OpenAI-compatible front** for exactly this —
+`POST /v1/chat/completions` (and `/chat/completions`): the request's `model` is routed
+by namespace/tier, rewritten to the backend id, and the upstream OpenAI response is
+returned verbatim (OpenAI in, OpenAI out — the Anthropic `/v1/messages` front with
+tool-call repair stays available in parallel for a Claude-harness client). Meanwhile a plain `claude` client that sends `claude-sonnet-…` still gets the
+**dumb tier/default routing** — both coexist, no mode switch. So the tier map stays the
+default, and dispatcher-style usage is just "send namespaced ids + read `/registry`".
+
+### Model tiers from leaderboards (never a hand-maintained table)
+
+`npm run sync:tiers` snapshots capability rankings from **BFCL** (Berkeley Function-Calling
+Leaderboard — tool-use accuracy, the primary signal for a tool-call proxy, incl. its
+Irrelevance-Detection metric = the malformed-call proxy) and **LMArena** (general capability)
+into `docs/tier-data.json`, and prints the top tool-callers so you can pick tier targets from
+real data. Both sources are synced-not-forked; a leaderboard schema change fails the sync loudly.
+
+The reshaper also takes `"kind": "openai"` — so `repair` mode can run entirely on an OpenAI-compatible provider (e.g. NIM) with no Anthropic key. The reshaper is asked only for the **corrected arguments per tool-call id** (not the full message envelope), which is far more reliable on weaker models; the proxy reconstructs the message and re-validates it.
+
+### Live run
+
+```bash
+node scripts/nim-front.mjs   # runs the compiled proxy fronting live NIM end-to-end (uses NVIDIA_API_KEY)
+```
+Then point a `claude` CLI at it (see "Install & run" above) and inspect the log to see which calls trip the validator on your traffic.
+
+## What it logs (per request, metadata only)
+
+`{ ts, path, backendModel, hadTools, streamed, backendStatus, validated: pass|fail|uncheckable|skipped, toolUseCount, uncheckableCount, errorKinds[], latencyMs }`
+
+`uncheckable` = a declared tool with no `input_schema` (built-in `bash`/`text_editor`/…) or a schema that wouldn't compile — surfaced distinctly so an unvalidatable call is never miscounted as a clean pass.
+
+This is the dataset for deciding which backend models are *format-broken* (reshapeable later) vs pass cleanly. Run in `detect` first, measure, then decide on repair.
+
+### Trip-rate dataset
+
+`node scripts/nim-trip-rate.mjs` probes a list of backend models across difficulty-graded tool schemas (× N trials), runs each call through the real validator, and repairs the failures — producing a per-model **trip rate** (share of tool calls that fail schema validation) and **repair-fix rate**. Latest live NIM run: [`docs/nim-trip-rate.md`](docs/nim-trip-rate.md) (raw records in `docs/nim-trip-rate.jsonl`). The sharp result: even strong Llama-3.1 models emit `days:"5"` (string) against an `integer` schema on every trial — and the proxy repairs it every time; the flat/enum/nested schemas pass clean.
+
+## Composing with headroom (optional)
+
+[headroom](../headroom) is a separate loopback proxy that **optimizes/compresses**
+context on the way to the model. Both it and repair-proxy are transparent
+Anthropic-Messages proxies, so they chain — but only in one order, because
+repair-proxy's backend speaks OpenAI/NIM while headroom only forwards Anthropic:
+
+```
+claude → headroom (:8787, context optimization, OUTER) → repair-proxy (:8791, validate/repair + translate, INNER) → NIM/…
+```
+
+repair-proxy must be **innermost**. To chain them, point headroom's upstream at
+repair-proxy — headroom exposes this as a launch flag, so its own code is untouched:
+
+```bash
+ANTHROPIC_TARGET_API_URL=http://127.0.0.1:8791   # headroom → repair-proxy
+```
+
+**Caveat:** that env var repoints *all* of headroom's Anthropic traffic — including
+your real (paid) Claude sessions — at repair-proxy. So run a **second, scoped
+headroom instance** for the multiplexed lane and leave your main one pointed at
+Anthropic:
+
+```bash
+HEADROOM_PORT=8788 ANTHROPIC_TARGET_API_URL=http://127.0.0.1:8791 headroom proxy
+# then point the claude client at :8788 (the wrapper's isolated CLAUDE_CONFIG_DIR keeps
+# your subscription out of the path); :8787 stays your normal Anthropic route.
+```
+
+Note the `claude-proxied` wrappers set `ANTHROPIC_BASE_URL` straight to :8791 and use
+an isolated `CLAUDE_CONFIG_DIR`, so **by default they bypass headroom entirely** — you
+only get the chain if you deliberately point the client at a headroom instance whose
+upstream is repair-proxy.
+
+**Is it worth it?** headroom's headline win is $/token savings vs *paid* Anthropic —
+**moot on the free NIM pool**. What still pays off through the chain: context
+**compression to fit a smaller backend context window** + lower latency, plus
+headroom's backend-agnostic memory/learn layer. So stack it for context-fit, not cost.
+
+## Design
+
+Consumers (audit-tools dispatch, plain `claude` CLI) point `ANTHROPIC_BASE_URL` at this proxy; it validates one backend per request. Target *selection* / token-prediction is a separate concern (the router/auditor), deliberately not here. For architecture, invariants, and the script inventory, see [CLAUDE.md](CLAUDE.md).
+
+## Dev
+
+```bash
+npm run typecheck   # tsc --noEmit
+npm test            # vitest (validator, SSE reconstruction, e2e transparency+detection)
+```

package/config.example.json

@@ -0,0 +1,45 @@
+{
+  "listen": "127.0.0.1:8791",
+  "providers": {
+    "nim": {
+      "base": "https://integrate.api.nvidia.com/v1",
+      "kind": "openai",
+      "authEnv": "NVIDIA_API_KEY"
+    },
+    "openrouter": {
+      "base": "https://openrouter.ai/api/v1",
+      "kind": "openai",
+      "authEnv": "OPENROUTER_API_KEY"
+    },
+    "gemini": {
+      "base": "https://generativelanguage.googleapis.com/v1beta/openai",
+      "kind": "openai",
+      "authEnv": "GEMINI_API_KEY"
+    },
+    "groq": {
+      "base": "https://api.groq.com/openai/v1",
+      "kind": "openai",
+      "authEnv": "GROQ_API_KEY"
+    },
+    "mistral": {
+      "base": "https://api.mistral.ai/v1",
+      "kind": "openai",
+      "authEnv": "MISTRAL_API_KEY"
+    }
+  },
+  "routing": {
+    "default": "nim/z-ai/glm-5.2",
+    "tiers": {
+      "opus": "nim/nvidia/nemotron-3-super-120b-a12b",
+      "sonnet": "nim/z-ai/glm-5.2",
+      "haiku": "nim/openai/gpt-oss-20b",
+      "fable": "nim/openai/gpt-oss-20b"
+    }
+  },
+  "mode": "repair",
+  "repair": {
+    "maxAttempts": 2,
+    "destructiveTools": ["rm", "delete", "push", "force", "overwrite", "drop", "reset"]
+  },
+  "log": { "level": "metadata", "file": null }
+}

package/dist/anthropic.d.ts

@@ -0,0 +1,49 @@
+/**
+ * Minimal Anthropic Messages API shapes — only the fields the proxy inspects.
+ * The proxy is byte-transparent for everything else; these types exist to
+ * validate tool_use blocks and reconstruct streamed responses, not to model the
+ * whole API.
+ */
+/** A JSON Schema object as it appears in a tool's `input_schema`. */
+export type JsonSchema = Record<string, unknown>;
+export interface Tool {
+    name: string;
+    description?: string;
+    input_schema: JsonSchema;
+}
+export interface TextBlock {
+    type: "text";
+    text: string;
+}
+export interface ToolUseBlock {
+    type: "tool_use";
+    id: string;
+    name: string;
+    input: unknown;
+}
+/** Anything we don't specifically model (thinking, redacted_thinking, …). */
+export interface OpaqueBlock {
+    type: string;
+    [k: string]: unknown;
+}
+export type ContentBlock = TextBlock | ToolUseBlock | OpaqueBlock;
+export type StopReason = "end_turn" | "max_tokens" | "stop_sequence" | "tool_use" | string | null;
+/** The assistant message the proxy validates (from JSON body or reconstructed from SSE). */
+export interface AssistantMessage {
+    content: ContentBlock[];
+    stop_reason: StopReason;
+    usage?: {
+        input_tokens?: number;
+        output_tokens?: number;
+    } | undefined;
+}
+export declare function isToolUseBlock(b: ContentBlock): b is ToolUseBlock;
+/**
+ * Extract the tools[] map (name → input_schema) from a parsed request body.
+ * Value is `null` for a tool that is DECLARED but has no JSON `input_schema` —
+ * notably Anthropic's built-in/typed tools (`bash`, `text_editor`, `computer`,
+ * `web_search`), which carry a `type` but no schema. Such tools are "known but
+ * unvalidatable": a tool_use naming them must NOT be flagged unknown_tool, but
+ * also cannot be schema-checked.
+ */
+export declare function toolSchemaMap(requestBody: unknown): Map<string, JsonSchema | null>;

package/dist/anthropic.js

@@ -0,0 +1,35 @@
+/**
+ * Minimal Anthropic Messages API shapes — only the fields the proxy inspects.
+ * The proxy is byte-transparent for everything else; these types exist to
+ * validate tool_use blocks and reconstruct streamed responses, not to model the
+ * whole API.
+ */
+export function isToolUseBlock(b) {
+    return b.type === "tool_use";
+}
+/**
+ * Extract the tools[] map (name → input_schema) from a parsed request body.
+ * Value is `null` for a tool that is DECLARED but has no JSON `input_schema` —
+ * notably Anthropic's built-in/typed tools (`bash`, `text_editor`, `computer`,
+ * `web_search`), which carry a `type` but no schema. Such tools are "known but
+ * unvalidatable": a tool_use naming them must NOT be flagged unknown_tool, but
+ * also cannot be schema-checked.
+ */
+export function toolSchemaMap(requestBody) {
+    const map = new Map();
+    if (typeof requestBody === "object" &&
+        requestBody !== null &&
+        Array.isArray(requestBody.tools)) {
+        for (const t of requestBody.tools) {
+            if (typeof t === "object" && t !== null && typeof t.name === "string") {
+                const rawSchema = t.input_schema;
+                const schema = typeof rawSchema === "object" && rawSchema !== null
+                    ? rawSchema
+                    : null;
+                map.set(t.name, schema);
+            }
+        }
+    }
+    return map;
+}
+//# sourceMappingURL=anthropic.js.map

package/dist/anthropic.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"anthropic.js","sourceRoot":"","sources":["../src/anthropic.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AA8CH,MAAM,UAAU,cAAc,CAAC,CAAe;IAC5C,OAAO,CAAC,CAAC,IAAI,KAAK,UAAU,CAAC;AAC/B,CAAC;AAED;;;;;;;GAOG;AACH,MAAM,UAAU,aAAa,CAAC,WAAoB;IAChD,MAAM,GAAG,GAAG,IAAI,GAAG,EAA6B,CAAC;IACjD,IACE,OAAO,WAAW,KAAK,QAAQ;QAC/B,WAAW,KAAK,IAAI;QACpB,KAAK,CAAC,OAAO,CAAE,WAAmC,CAAC,KAAK,CAAC,EACzD,CAAC;QACD,KAAK,MAAM,CAAC,IAAK,WAAoC,CAAC,KAAK,EAAE,CAAC;YAC5D,IAAI,OAAO,CAAC,KAAK,QAAQ,IAAI,CAAC,KAAK,IAAI,IAAI,OAAQ,CAAU,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;gBAChF,MAAM,SAAS,GAAI,CAAgC,CAAC,YAAY,CAAC;gBACjE,MAAM,MAAM,GACV,OAAO,SAAS,KAAK,QAAQ,IAAI,SAAS,KAAK,IAAI;oBACjD,CAAC,CAAE,SAAwB;oBAC3B,CAAC,CAAC,IAAI,CAAC;gBACX,GAAG,CAAC,GAAG,CAAE,CAAU,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;YACpC,CAAC;QACH,CAAC;IACH,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC"}

package/dist/backend.d.ts

@@ -0,0 +1,36 @@
+import { type ResolvedTarget } from "./config.js";
+/**
+ * Fetch the resolved provider target and return an ANTHROPIC-shaped `Response`,
+ * regardless of the backend's native wire format. For kind="anthropic" this is a
+ * passthrough. For kind="openai" (NIM/vLLM/OpenRouter/Gemini) the request is
+ * translated Anthropic→OpenAI and the response translated back (streaming via
+ * llm-bridge's SSE re-encoder, non-streaming via a direct mapper) — so the rest
+ * of the proxy (validate/repair) always sees Anthropic Messages.
+ */
+export declare function fetchBackend(target: ResolvedTarget, args: {
+    path: string;
+    method: string;
+    reqBuf: Buffer;
+    reqJson: unknown;
+    anthropicHeaders: Record<string, string>;
+    wantsStream: boolean;
+    signal: AbortSignal;
+}, fetchFn?: typeof fetch): Promise<Response>;
+/** Map a non-streaming OpenAI chat completion into an Anthropic message. */
+export declare function openAiResponseToAnthropic(j: Record<string, unknown>, model: string): object;
+/**
+ * OpenAI-compatible FRONT: an OpenAI `/chat/completions` request comes in, its `model`
+ * has already been resolved to a provider target by namespace/tier routing. For an
+ * openai-kind target this is a routing reverse-proxy — rewrite `model` to the backend
+ * id, inject the backend key, and stream the upstream OpenAI response straight back
+ * (OpenAI in, OpenAI out — no translation). This is the transport a dispatcher (e.g.
+ * audit-tools) consumes to reach many backends behind one endpoint.
+ *
+ * anthropic-kind targets are not served on the OpenAI front (they need OpenAI↔Anthropic
+ * translation and are not the dispatcher use case) — a clean 400, never a mistranslation.
+ */
+export declare function fetchOpenAiFront(target: ResolvedTarget, args: {
+    reqJson: unknown;
+    wantsStream: boolean;
+    signal: AbortSignal;
+}, fetchFn?: typeof fetch): Promise<Response>;

package/dist/backend.js

@@ -0,0 +1,135 @@
+import { translateBetweenProviders, handleUniversalStreamRequest } from "llm-bridge";
+/**
+ * Fetch the resolved provider target and return an ANTHROPIC-shaped `Response`,
+ * regardless of the backend's native wire format. For kind="anthropic" this is a
+ * passthrough. For kind="openai" (NIM/vLLM/OpenRouter/Gemini) the request is
+ * translated Anthropic→OpenAI and the response translated back (streaming via
+ * llm-bridge's SSE re-encoder, non-streaming via a direct mapper) — so the rest
+ * of the proxy (validate/repair) always sees Anthropic Messages.
+ */
+export async function fetchBackend(target, args, fetchFn = fetch) {
+    if (target.kind === "anthropic") {
+        const init = { method: args.method, headers: args.anthropicHeaders, signal: args.signal };
+        if (args.reqBuf.length)
+            init.body = args.reqBuf;
+        return fetchFn(target.base + args.path, init);
+    }
+    // kind === "openai"
+    let openaiBody;
+    try {
+        openaiBody = translateBetweenProviders("anthropic", "openai", (args.reqJson ?? {}));
+    }
+    catch (e) {
+        return anthropicError(502, `request translation failed: ${e.message}`);
+    }
+    openaiBody.model = target.model;
+    openaiBody.stream = args.wantsStream;
+    const headers = { "content-type": "application/json" };
+    const key = target.authEnv ? process.env[target.authEnv]?.trim() : undefined;
+    if (key) {
+        if (target.authHeader === "authorization")
+            headers["authorization"] = `Bearer ${key}`;
+        else
+            headers["x-api-key"] = key;
+    }
+    const res = await fetchFn(target.base + "/chat/completions", {
+        method: "POST",
+        headers,
+        body: JSON.stringify(openaiBody),
+        signal: args.signal,
+    });
+    if (!res.ok) {
+        const body = await res.text();
+        return anthropicError(res.status, `openai backend HTTP ${res.status}: ${body.slice(0, 300)}`);
+    }
+    if (args.wantsStream && res.body) {
+        const anthStream = handleUniversalStreamRequest(res.body, "openai", "anthropic");
+        return new Response(anthStream, { status: res.status, headers: { "content-type": "text/event-stream" } });
+    }
+    let anthropicJson;
+    try {
+        anthropicJson = openAiResponseToAnthropic((await res.json()), target.model ?? "");
+    }
+    catch (e) {
+        return anthropicError(502, `response translation failed: ${e.message}`);
+    }
+    return new Response(JSON.stringify(anthropicJson), { status: 200, headers: { "content-type": "application/json" } });
+}
+/** Map a non-streaming OpenAI chat completion into an Anthropic message. */
+export function openAiResponseToAnthropic(j, model) {
+    const choice = j.choices?.[0] ?? {};
+    const msg = choice.message ?? {};
+    const content = [];
+    if (typeof msg.content === "string" && msg.content.length > 0)
+        content.push({ type: "text", text: msg.content });
+    const toolCalls = msg.tool_calls ?? [];
+    for (const tc of toolCalls) {
+        const fn = tc.function ?? {};
+        let input;
+        try {
+            input = JSON.parse(fn.arguments ?? "{}");
+        }
+        catch {
+            input = fn.arguments ?? {};
+        }
+        content.push({ type: "tool_use", id: tc.id ?? "tu", name: fn.name ?? "", input });
+    }
+    const finish = choice.finish_reason;
+    const stopReason = toolCalls.length > 0 ? "tool_use" : finish === "length" ? "max_tokens" : finish === "stop" ? "end_turn" : finish ?? "end_turn";
+    const usage = j.usage ?? {};
+    return {
+        id: j.id ?? "msg_translated",
+        type: "message",
+        role: "assistant",
+        model: model || (j.model ?? ""),
+        content,
+        stop_reason: stopReason,
+        stop_sequence: null,
+        usage: { input_tokens: usage.prompt_tokens ?? 0, output_tokens: usage.completion_tokens ?? 0 },
+    };
+}
+function anthropicError(status, message) {
+    return new Response(JSON.stringify({ type: "error", error: { type: "api_error", message } }), {
+        status,
+        headers: { "content-type": "application/json" },
+    });
+}
+function openaiError(status, message) {
+    return new Response(JSON.stringify({ error: { message, type: "invalid_request_error" } }), {
+        status,
+        headers: { "content-type": "application/json" },
+    });
+}
+/**
+ * OpenAI-compatible FRONT: an OpenAI `/chat/completions` request comes in, its `model`
+ * has already been resolved to a provider target by namespace/tier routing. For an
+ * openai-kind target this is a routing reverse-proxy — rewrite `model` to the backend
+ * id, inject the backend key, and stream the upstream OpenAI response straight back
+ * (OpenAI in, OpenAI out — no translation). This is the transport a dispatcher (e.g.
+ * audit-tools) consumes to reach many backends behind one endpoint.
+ *
+ * anthropic-kind targets are not served on the OpenAI front (they need OpenAI↔Anthropic
+ * translation and are not the dispatcher use case) — a clean 400, never a mistranslation.
+ */
+export async function fetchOpenAiFront(target, args, fetchFn = fetch) {
+    if (target.kind !== "openai") {
+        return openaiError(400, `llm-relay: OpenAI front requires an openai-kind provider; "${target.provider}" is ${target.kind}`);
+    }
+    const base = (args.reqJson ?? {});
+    const body = { ...base, model: target.model, stream: args.wantsStream };
+    const headers = { "content-type": "application/json" };
+    const key = target.authEnv ? process.env[target.authEnv]?.trim() : undefined;
+    if (key) {
+        if (target.authHeader === "authorization")
+            headers["authorization"] = `Bearer ${key}`;
+        else
+            headers["x-api-key"] = key;
+    }
+    return fetchFn(target.base + "/chat/completions", {
+        method: "POST",
+        headers,
+        body: JSON.stringify(body),
+        signal: args.signal,
+    });
+}
+//# sourceMappingURL=backend.js.map

package/dist/backend.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"backend.js","sourceRoot":"","sources":["../src/backend.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,yBAAyB,EAAE,4BAA4B,EAAE,MAAM,YAAY,CAAC;AAGrF;;;;;;;GAOG;AACH,MAAM,CAAC,KAAK,UAAU,YAAY,CAChC,MAAsB,EACtB,IAQC,EACD,UAAwB,KAAK;IAE7B,IAAI,MAAM,CAAC,IAAI,KAAK,WAAW,EAAE,CAAC;QAChC,MAAM,IAAI,GAAgB,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,EAAE,OAAO,EAAE,IAAI,CAAC,gBAAgB,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,EAAE,CAAC;QACvG,IAAI,IAAI,CAAC,MAAM,CAAC,MAAM;YAAE,IAAI,CAAC,IAAI,GAAG,IAAI,CAAC,MAAM,CAAC;QAChD,OAAO,OAAO,CAAC,MAAM,CAAC,IAAI,GAAG,IAAI,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;IAChD,CAAC;IAED,oBAAoB;IACpB,IAAI,UAAmC,CAAC;IACxC,IAAI,CAAC;QACH,UAAU,GAAG,yBAAyB,CAAC,WAAW,EAAE,QAAQ,EAAE,CAAC,IAAI,CAAC,OAAO,IAAI,EAAE,CAAU,CAA4B,CAAC;IAC1H,CAAC;IAAC,OAAO,CAAC,EAAE,CAAC;QACX,OAAO,cAAc,CAAC,GAAG,EAAE,+BAAgC,CAAW,CAAC,OAAO,EAAE,CAAC,CAAC;IACpF,CAAC;IACD,UAAU,CAAC,KAAK,GAAG,MAAM,CAAC,KAAK,CAAC;IAChC,UAAU,CAAC,MAAM,GAAG,IAAI,CAAC,WAAW,CAAC;IAErC,MAAM,OAAO,GAA2B,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC;IAC/E,MAAM,GAAG,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC;IAC7E,IAAI,GAAG,EAAE,CAAC;QACR,IAAI,MAAM,CAAC,UAAU,KAAK,eAAe;YAAE,OAAO,CAAC,eAAe,CAAC,GAAG,UAAU,GAAG,EAAE,CAAC;;YACjF,OAAO,CAAC,WAAW,CAAC,GAAG,GAAG,CAAC;IAClC,CAAC;IAED,MAAM,GAAG,GAAG,MAAM,OAAO,CAAC,MAAM,CAAC,IAAI,GAAG,mBAAmB,EAAE;QAC3D,MAAM,EAAE,MAAM;QACd,OAAO;QACP,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,UAAU,CAAC;QAChC,MAAM,EAAE,IAAI,CAAC,MAAM;KACpB,CAAC,CAAC;IAEH,IAAI,CAAC,GAAG,CAAC,EAAE,EAAE,CAAC;QACZ,MAAM,IAAI,GAAG,MAAM,GAAG,CAAC,IAAI,EAAE,CAAC;QAC9B,OAAO,cAAc,CAAC,GAAG,CAAC,MAAM,EAAE,uBAAuB,GAAG,CAAC,MAAM,KAAK,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,CAAC,CAAC;IAChG,CAAC;IAED,IAAI,IAAI,CAAC,WAAW,IAAI,GAAG,CAAC,IAAI,EAAE,CAAC;QACjC,MAAM,UAAU,GAAG,4BAA4B,CAAC,GAAG,CAAC,IAAI,EAAE,QAAQ,EAAE,WAAW,CAAC,CAAC;QACjF,OAAO,IAAI,QAAQ,CAAC,UAAU,EAAE,EAAE,MAAM,EAAE,GAAG,CAAC,MAAM,EAAE,OAAO,EAAE,EAAE,cAAc,EAAE,mBAAmB,EAAE,EAAE,CAAC,CAAC;IAC5G,CAAC;IAED,IAAI,aAAqB,CAAC;IAC1B,IAAI,CAAC;QACH,aAAa,GAAG,yBAAyB,CAAC,CAAC,MAAM,GAAG,CAAC,IAAI,EAAE,CAA4B,EAAE,MAAM,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC;IAC/G,CAAC;IAAC,OAAO,CAAC,EAAE,CAAC;QACX,OAAO,cAAc,CAAC,GAAG,EAAE,gCAAiC,CAAW,CAAC,OAAO,EAAE,CAAC,CAAC;IACrF,CAAC;IACD,OAAO,IAAI,QAAQ,CAAC,IAAI,CAAC,SAAS,CAAC,aAAa,CAAC,EAAE,EAAE,MAAM,EAAE,GAAG,EAAE,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE,EAAE,CAAC,CAAC;AACvH,CAAC;AAED,4EAA4E;AAC5E,MAAM,UAAU,yBAAyB,CAAC,CAA0B,EAAE,KAAa;IACjF,MAAM,MAAM,GAAI,CAAC,CAAC,OAAsD,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;IACpF,MAAM,GAAG,GAAI,MAAM,CAAC,OAA+C,IAAI,EAAE,CAAC;IAC1E,MAAM,OAAO,GAAa,EAAE,CAAC;IAC7B,IAAI,OAAO,GAAG,CAAC,OAAO,KAAK,QAAQ,IAAI,GAAG,CAAC,OAAO,CAAC,MAAM,GAAG,CAAC;QAAE,OAAO,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,IAAI,EAAE,GAAG,CAAC,OAAO,EAAE,CAAC,CAAC;IACjH,MAAM,SAAS,GAAI,GAAG,CAAC,UAAyD,IAAI,EAAE,CAAC;IACvF,KAAK,MAAM,EAAE,IAAI,SAAS,EAAE,CAAC;QAC3B,MAAM,EAAE,GAAI,EAAE,CAAC,QAAgD,IAAI,EAAE,CAAC;QACtE,IAAI,KAAc,CAAC;QACnB,IAAI,CAAC;YAAC,KAAK,GAAG,IAAI,CAAC,KAAK,CAAE,EAAE,CAAC,SAAoB,IAAI,IAAI,CAAC,CAAC;QAAC,CAAC;QAAC,MAAM,CAAC;YAAC,KAAK,GAAG,EAAE,CAAC,SAAS,IAAI,EAAE,CAAC;QAAC,CAAC;QACnG,OAAO,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,UAAU,EAAE,EAAE,EAAG,EAAE,CAAC,EAAa,IAAI,IAAI,EAAE,IAAI,EAAG,EAAE,CAAC,IAAe,IAAI,EAAE,EAAE,KAAK,EAAE,CAAC,CAAC;IAC5G,CAAC;IACD,MAAM,MAAM,GAAG,MAAM,CAAC,aAAmC,CAAC;IAC1D,MAAM,UAAU,GACd,SAAS,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,UAAU,CAAC,CAAC,CAAC,MAAM,KAAK,QAAQ,CAAC,CAAC,CAAC,YAAY,CAAC,CAAC,CAAC,MAAM,KAAK,MAAM,CAAC,CAAC,CAAC,UAAU,CAAC,CAAC,CAAC,MAAM,IAAI,UAAU,CAAC;IACjI,MAAM,KAAK,GAAI,CAAC,CAAC,KAA4C,IAAI,EAAE,CAAC;IACpE,OAAO;QACL,EAAE,EAAG,CAAC,CAAC,EAAa,IAAI,gBAAgB;QACxC,IAAI,EAAE,SAAS;QACf,IAAI,EAAE,WAAW;QACjB,KAAK,EAAE,KAAK,IAAI,CAAE,CAAC,CAAC,KAAgB,IAAI,EAAE,CAAC;QAC3C,OAAO;QACP,WAAW,EAAE,UAAU;QACvB,aAAa,EAAE,IAAI;QACnB,KAAK,EAAE,EAAE,YAAY,EAAE,KAAK,CAAC,aAAa,IAAI,CAAC,EAAE,aAAa,EAAE,KAAK,CAAC,iBAAiB,IAAI,CAAC,EAAE;KAC/F,CAAC;AACJ,CAAC;AAED,SAAS,cAAc,CAAC,MAAc,EAAE,OAAe;IACrD,OAAO,IAAI,QAAQ,CAAC,IAAI,CAAC,SAAS,CAAC,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,EAAE,IAAI,EAAE,WAAW,EAAE,OAAO,EAAE,EAAE,CAAC,EAAE;QAC5F,MAAM;QACN,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE;KAChD,CAAC,CAAC;AACL,CAAC;AAED,SAAS,WAAW,CAAC,MAAc,EAAE,OAAe;IAClD,OAAO,IAAI,QAAQ,CAAC,IAAI,CAAC,SAAS,CAAC,EAAE,KAAK,EAAE,EAAE,OAAO,EAAE,IAAI,EAAE,uBAAuB,EAAE,EAAE,CAAC,EAAE;QACzF,MAAM;QACN,OAAO,EAAE,EAAE,cAAc,EAAE,kBAAkB,EAAE;KAChD,CAAC,CAAC;AACL,CAAC;AAED;;;;;;;;;;GAUG;AACH,MAAM,CAAC,KAAK,UAAU,gBAAgB,CACpC,MAAsB,EACtB,IAAqE,EACrE,UAAwB,KAAK;IAE7B,IAAI,MAAM,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;QAC7B,OAAO,WAAW,CAAC,GAAG,EAAE,8DAA8D,MAAM,CAAC,QAAQ,QAAQ,MAAM,CAAC,IAAI,EAAE,CAAC,CAAC;IAC9H,CAAC;IACD,MAAM,IAAI,GAAG,CAAC,IAAI,CAAC,OAAO,IAAI,EAAE,CAA4B,CAAC;IAC7D,MAAM,IAAI,GAAG,EAAE,GAAG,IAAI,EAAE,KAAK,EAAE,MAAM,CAAC,KAAK,EAAE,MAAM,EAAE,IAAI,CAAC,WAAW,EAAE,CAAC;IACxE,MAAM,OAAO,GAA2B,EAAE,cAAc,EAAE,kBAAkB,EAAE,CAAC;IAC/E,MAAM,GAAG,GAAG,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,SAAS,CAAC;IAC7E,IAAI,GAAG,EAAE,CAAC;QACR,IAAI,MAAM,CAAC,UAAU,KAAK,eAAe;YAAE,OAAO,CAAC,eAAe,CAAC,GAAG,UAAU,GAAG,EAAE,CAAC;;YACjF,OAAO,CAAC,WAAW,CAAC,GAAG,GAAG,CAAC;IAClC,CAAC;IACD,OAAO,OAAO,CAAC,MAAM,CAAC,IAAI,GAAG,mBAAmB,EAAE;QAChD,MAAM,EAAE,MAAM;QACd,OAAO;QACP,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC;QAC1B,MAAM,EAAE,IAAI,CAAC,MAAM;KACpB,CAAC,CAAC;AACL,CAAC"}

package/dist/catalog.d.ts

@@ -0,0 +1,55 @@
+import type { ProviderConfig } from "./config.js";
+/**
+ * Live per-provider model catalog — model ids are DISCOVERED from each provider's
+ * OpenAI-compatible `/models` endpoint, never hand-maintained. In-memory TTL cache
+ * backed by a small on-disk cache so restarts start warm. Fail-open everywhere: a
+ * fetch failure serves the last-known list (or an empty one), never blocks routing.
+ */
+export declare class ModelCatalog {
+    private mem;
+    private readonly ttlMs;
+    private readonly cachePath;
+    private loaded;
+    /** Providers with a background refresh in flight — dedups stampeding probes. */
+    private refreshing;
+    constructor(opts?: {
+        ttlMs?: number;
+        cachePath?: string | null;
+    });
+    private loadDisk;
+    private saveDisk;
+    /** Cached model ids for a provider; empty array if fetch fails and no prior cache. */
+    private cached;
+    /**
+     * Live model ids for a provider, cached with TTL. `force` bypasses the TTL.
+     *
+     * STALE-WHILE-REVALIDATE: a fresh cache (within TTL) is served directly; a STALE
+     * cache is ALSO served immediately while a background refresh updates it — so a
+     * discovery/liveness probe (`GET /registry`) NEVER blocks on an upstream refetch.
+     * Blocking `await` happens only on a genuine cold start (no prior at all, in memory
+     * or on disk). Serves a stale cache if a blocking refresh fails; returns [] only
+     * when there is no cache AND the fetch fails. `force` still awaits (explicit refresh).
+     */
+    list(name: string, cfg: ProviderConfig, opts?: {
+        force?: boolean;
+        now?: number;
+        fetchFn?: typeof fetch;
+    }): Promise<string[]>;
+    /**
+     * Fire-and-forget catalog refresh for a stale provider, deduped so repeated probes
+     * during one refresh window spawn at most one upstream fetch. A failed refresh
+     * leaves the stale entry in place (fail-open); it is retried on the next `list`.
+     */
+    private refreshInBackground;
+    /**
+     * Whether a provider serves a model. Returns null when the catalog is
+     * unavailable (no cache and fetch failed) — callers treat null as "unknown,
+     * proceed" so routing never hard-fails on a catalog miss.
+     */
+    has(name: string, cfg: ProviderConfig, model: string, opts?: {
+        force?: boolean;
+        now?: number;
+        fetchFn?: typeof fetch;
+    }): Promise<boolean | null>;
+    private fetch;
+}