@askalf/dario 6.1.0 → 6.2.0

package/README.md

@@ -289,7 +289,7 @@ That is a **chain**, read left to right; each provider takes the first entry it
 
 A single-entry chain is one-way and means what it always meant, so an existing config is unaffected. Failover is opt-in: without `--pool-fallback`, a drained pool still returns its honest 429/503. Only a **429 or 5xx** fails over; a 400 surfaces, because a bad request that fails over just reproduces itself on the other provider and buries the real cause. A 429 also cools that provider for a bounded interval, its `retry-after` if it sent one and 60 s otherwise, never longer than 15 min, and an entry that already declined is not asked again within the same request. When every entry is cooling, the request ends on one honest `429` with a `retry-after` instead of a retry storm. The Claude entry has to be a model the pool can actually serve, checked positively against the live catalog, so a typo can't trade a recoverable 429 for an unrecoverable 404.
 
-The chain also covers a stream that dies **mid-answer**. Until 6.1 that was the one failure nothing could catch: bytes were on the wire, so the socket reset, the in-band `overloaded_error`, the codex `response.failed` all ended the stream where they happened, with no `message_stop`, and the client threw away every word it already had. Now dario finishes the same stream from the other subscription — the resume picks up inside the still-open content block, a comment marks the seam (`: dario continuation gpt-5.6-sol (codex live) after 1240 chars`), and the client sees one message. The model is asked to repeat the last few words verbatim and dario trims the repeat, so the join is rendered by a model and cut by a parser, never guessed. Text only — a cut inside a tool call ends as it always did. On by default, `--no-midstream-continue` turns it off, and without a chain entry for the other provider it is inert. [How it works](docs/midstream-continuation.md).
+The chain also covers a stream that dies **mid-answer**. Until 6.1 that was the one failure nothing could catch: bytes were on the wire, so the socket reset, the in-band `overloaded_error`, the codex `response.failed` all ended the stream where they happened, with no `message_stop`, and the client threw away every word it already had. Now dario finishes the same stream — on the same model first, a fresh request through its own front door; on the other subscription when that delivers nothing or dies too. The resume picks up inside the still-open content block, a comment marks each seam (`: dario continuation claude-opus-5 (same model) after 1240 chars`), and the client sees one message. The model is asked to repeat the last few words verbatim and dario trims the repeat, so the join is rendered by a model and cut by a parser, never guessed. Text only — a cut inside a tool call ends as it always did. On by default, `--no-midstream-continue` turns it off; without a chain the second hop is simply not there. [How it works](docs/midstream-continuation.md).
 
 `dario doctor` tells you which of these you are actually in:
 

package/dist/midstream.d.ts

@@ -49,8 +49,22 @@
  */
 import type { ServerResponse } from 'node:http';
 export type WireShape = 'anthropic' | 'openai';
-/** Client-visible marker that a loopback request is a continuation, so the handler never nests one. */
+/**
+ * Marks a loopback request as a continuation and carries its DEPTH: `1` for
+ * the resume of a client request, `2` for the resume of that resume. A
+ * request at `MAX_CONTINUATION_DEPTH` is never continued itself.
+ */
 export declare const CONTINUATION_HEADER = "x-dario-continuation";
+/**
+ * Two hops. The first resume asks for the SAME model again through the front
+ * door — the pool picks a seat and the existing pre-byte failover applies —
+ * so a transient reset finishes on the model the client asked for. If that
+ * resume dies too, the second hop goes to the OTHER provider's chain entry.
+ * A third hop would be a third attempt at whatever is failing.
+ */
+export declare const MAX_CONTINUATION_DEPTH = 2;
+/** The depth a request carries, 0 for an ordinary client request. */
+export declare function continuationDepth(headerValue: string | string[] | undefined): number;
 /** Characters of the partial the model is asked to repeat verbatim (the seam anchor). */
 export declare const ANCHOR_CHARS = 40;
 export interface SseFrame {
@@ -219,6 +233,8 @@ export declare class Splicer {
         dropped: number;
         emitted: number;
     };
+    /** `message_start.model` of the resume — who actually served it. */
+    resumeModel: string | null;
     constructor(shape: WireShape, state: ClientStreamState, partial: string);
     /** Frames to write to the client for one resume frame. */
     feed(f: SseFrame): string[];
@@ -261,8 +277,13 @@ export interface ResumeOptions {
     loopbackBase: string;
     /** Auth + attribution headers for the loopback request. */
     loopbackHeaders: Record<string, string>;
-    /** Decides where to resume, once, at failure time. Null = nowhere; the stream ends as before. */
-    resolveTarget: () => Promise<ContinuationTarget | null>;
+    /**
+     * Where to resume, decided at failure time. `choice` is 1 for the same
+     * model again and 2 for the other provider's chain entry; the guard asks
+     * in that order and moves on when a choice is null or its loopback delivers
+     * nothing. Null for every choice = the stream ends as before.
+     */
+    resolveTarget: (choice: number) => Promise<ContinuationTarget | null>;
     /** Called right before the loopback request is made — the site releases its own queue slot here. */
     onBeforeResume?: () => void;
     timeoutMs: number;
@@ -277,6 +298,8 @@ export interface MidstreamGuardOptions {
     isClientGone: () => boolean;
     resume: ResumeOptions | null;
     requestNo: number;
+    /** Depth of the request THIS guard protects (0 = a client request). Its resume is `depth + 1`. */
+    depth?: number;
     verbose: boolean;
     log?: (line: string) => void;
 }

package/dist/midstream.js

@@ -47,8 +47,28 @@
  * the existing pre-byte failover covers them), and the api-key OpenAI backend.
  * A stream that cannot be continued ends exactly as it did before this module.
  */
-/** Client-visible marker that a loopback request is a continuation, so the handler never nests one. */
+/**
+ * Marks a loopback request as a continuation and carries its DEPTH: `1` for
+ * the resume of a client request, `2` for the resume of that resume. A
+ * request at `MAX_CONTINUATION_DEPTH` is never continued itself.
+ */
 export const CONTINUATION_HEADER = 'x-dario-continuation';
+/**
+ * Two hops. The first resume asks for the SAME model again through the front
+ * door — the pool picks a seat and the existing pre-byte failover applies —
+ * so a transient reset finishes on the model the client asked for. If that
+ * resume dies too, the second hop goes to the OTHER provider's chain entry.
+ * A third hop would be a third attempt at whatever is failing.
+ */
+export const MAX_CONTINUATION_DEPTH = 2;
+/** The depth a request carries, 0 for an ordinary client request. */
+export function continuationDepth(headerValue) {
+    const v = Array.isArray(headerValue) ? headerValue[0] : headerValue;
+    if (v === undefined)
+        return 0;
+    const n = Number.parseInt(v, 10);
+    return Number.isFinite(n) && n > 0 ? n : 1;
+}
 /** Characters of the partial the model is asked to repeat verbatim (the seam anchor). */
 export const ANCHOR_CHARS = 40;
 /** Upper bound on continuation text held back while looking for the anchor. */
@@ -457,6 +477,8 @@ export class Splicer {
     terminalSeen = false;
     /** Diagnostics for the log line. */
     stats = { anchor: 'n/a', dropped: 0, emitted: 0 };
+    /** `message_start.model` of the resume — who actually served it. */
+    resumeModel = null;
     constructor(shape, state, partial) {
         this.shape = shape;
         this.partial = partial;
@@ -467,7 +489,11 @@ export class Splicer {
     }
     /** Frames to write to the client for one resume frame. */
     feed(f) {
-        if (f.comment || f.dataText === null)
+        // A second hop's seam comment, written by the inner guard, rides through
+        // so a raw capture shows every takeover; every other comment is dropped.
+        if (f.comment)
+            return f.raw.startsWith(SEAM_COMMENT_PREFIX) ? [f.raw] : [];
+        if (f.dataText === null)
             return [];
         return this.shape === 'anthropic' ? this.feedAnthropic(f) : this.feedOpenAI(f);
     }
@@ -490,8 +516,12 @@ export class Splicer {
         switch (d.type) {
             case 'ping':
                 return [f.raw];
-            case 'message_start':
+            case 'message_start': {
+                const m = d.message;
+                if (typeof m?.model === 'string')
+                    this.resumeModel = m.model;
                 return []; // the client already has one
+            }
             case 'content_block_start': {
                 const idx = d.index;
                 const cb = d.content_block;
@@ -607,6 +637,8 @@ export class Splicer {
         const c = choices?.[0];
         if (!c)
             return [];
+        if (typeof d.model === 'string' && !this.resumeModel)
+            this.resumeModel = d.model;
         const out = [];
         if (typeof c.delta?.content === 'string' && c.delta.content.length > 0) {
             this.hold += c.delta.content;
@@ -661,6 +693,7 @@ export class Splicer {
         return [openaiChunk({ content: text }, null)];
     }
 }
+const SEAM_COMMENT_PREFIX = ': dario continuation';
 function openaiChunk(delta, finish) {
     return `data: ${JSON.stringify({ id: 'chatcmpl-dario', object: 'chat.completion.chunk', created: Math.floor(Date.now() / 1000), model: 'claude', choices: [{ index: 0, delta, finish_reason: finish }] })}\n\n`;
 }
@@ -722,27 +755,41 @@ export class MidstreamGuard {
             cleanEnd();
             return 'not-continuable';
         }
-        let target = null;
-        try {
-            target = await this.o.resume.resolveTarget();
-        }
-        catch {
-            target = null;
+        // The choices, in order: 1 = the same model again, 2 = the other
+        // provider's chain entry. A client request tries both; a request that IS
+        // a resume (depth 1) starts at 2 — its model is the one that just failed
+        // twice. A choice whose loopback delivers nothing (refused, unreachable,
+        // dead before its first byte) hands over to the next; the first one that
+        // puts content on the wire ends the search, finished or not.
+        const partial = s.textSoFar;
+        let tried = 0;
+        for (let choice = (this.o.depth ?? 0) + 1; choice <= MAX_CONTINUATION_DEPTH; choice++) {
+            let target = null;
+            try {
+                target = await this.o.resume.resolveTarget(choice);
+            }
+            catch {
+                target = null;
+            }
+            if (!target)
+                continue;
+            tried++;
+            this.log(`#${this.o.requestNo} stream died after ${partial.length} chars → continuing as ${target.label}`);
+            const outcome = await this.continueFrom(target, partial);
+            if (outcome === 'failed') {
+                this.log(`#${this.o.requestNo} continuation as ${target.label} delivered nothing${choice < MAX_CONTINUATION_DEPTH ? ' — trying the next choice' : ''}`);
+                continue;
+            }
+            this.o.end();
+            return outcome === 'finished' ? 'continued' : 'continued-unfinished';
         }
-        if (!target) {
-            this.log(`#${this.o.requestNo} stream died after ${s.textSoFar.length} chars — no continuation target (set --pool-fallback with an entry for the other provider)`);
+        if (tried === 0) {
+            this.log(`#${this.o.requestNo} stream died after ${partial.length} chars — no continuation target (set --pool-fallback with an entry for the other provider)`);
             cleanEnd();
             return 'no-target';
         }
-        const partial = s.textSoFar;
-        this.log(`#${this.o.requestNo} stream died after ${partial.length} chars → continuing as ${target.label}`);
-        const outcome = await this.continueFrom(target, partial);
-        if (outcome === 'failed') {
-            cleanEnd();
-            return 'resume-failed';
-        }
-        this.o.end();
-        return outcome === 'finished' ? 'continued' : 'continued-unfinished';
+        cleanEnd();
+        return 'resume-failed';
     }
     /**
      * 'failed': nothing of the resume reached the client — the site ends the
@@ -770,7 +817,7 @@ export class MidstreamGuard {
             r.onBeforeResume?.();
             const res = await fetchImpl(`${r.loopbackBase}${path}`, {
                 method: 'POST',
-                headers: { 'content-type': 'application/json', [CONTINUATION_HEADER]: String(this.o.requestNo), ...r.loopbackHeaders },
+                headers: { 'content-type': 'application/json', [CONTINUATION_HEADER]: String((this.o.depth ?? 0) + 1), ...r.loopbackHeaders },
                 body: JSON.stringify(body),
                 signal: abort.signal,
             });
@@ -781,7 +828,7 @@ export class MidstreamGuard {
             }
             // An SSE comment, ignored by every parser, so a raw capture shows where
             // the second provider took over.
-            this.o.write(`: dario continuation ${target.label} after ${partial.length} chars\n\n`);
+            this.o.write(`${SEAM_COMMENT_PREFIX} ${target.label} after ${partial.length} chars\n\n`);
             const reader = res.body.getReader();
             const split = new SseFrameSplitter();
             let sawContent = false;
@@ -826,7 +873,8 @@ export class MidstreamGuard {
                 return 'unfinished';
             }
             if (splicer.terminalSeen) {
-                this.log(`#${this.o.requestNo} continuation done: +${st.emitted} chars in ${Date.now() - startedAt}ms (anchor ${st.anchor}, trimmed ${st.dropped})`);
+                const by = splicer.resumeModel && splicer.resumeModel !== 'claude' ? ` by ${splicer.resumeModel}` : '';
+                this.log(`#${this.o.requestNo} continuation done${by}: +${st.emitted} chars in ${Date.now() - startedAt}ms (anchor ${st.anchor}, trimmed ${st.dropped})`);
                 return 'finished';
             }
             // The resume body ended without its terminal event — a reset on the

package/dist/proxy.js

@@ -26,7 +26,7 @@ import { createTokenBucket } from './rate-limit.js';
 import { getOpenAIBackend, isOpenAIModel, forwardToOpenAI } from './openai-backend.js';
 import { forwardToCodex, getCodexModelSlugs, peekCodexModelSlugs, isCodexModel, pickCodexFallback, pickClaudeTarget, CODEX_BACKEND_BASE_URL } from './codex-backend.js';
 import { effortForCodex } from './effort.js';
-import { MidstreamGuard, guardFor, loopbackBaseFor, CONTINUATION_HEADER } from './midstream.js';
+import { MidstreamGuard, guardFor, loopbackBaseFor, CONTINUATION_HEADER, MAX_CONTINUATION_DEPTH, continuationDepth } from './midstream.js';
 import { isClaudeServableModel } from './claude-model.js';
 import { MODEL_UNROUTABLE } from './upstream-rejection.js';
 import { readCompareTarget, teeResponse, runCompare, writeCompareRecord, COMPARE_RESULT_HEADER } from './compare.js';
@@ -2952,9 +2952,21 @@ export async function startProxy(opts = {}) {
             // re-issues the CLIENT's request, not the rewritten one, so dario's own
             // rules apply to the resume the same way they applied to the original.
             const clientBodyBytes = body;
-            // A loopback request made by a continuation. Never continued itself —
-            // one resume per client request, no nesting.
-            const isContinuation = req.headers[CONTINUATION_HEADER] !== undefined;
+            // How deep in a continuation chain this request sits: 0 for a client
+            // request, 1 for its resume, 2 for the resume of that resume — which is
+            // never continued itself (MAX_CONTINUATION_DEPTH).
+            const requestDepth = continuationDepth(req.headers[CONTINUATION_HEADER]);
+            const isContinuation = requestDepth >= MAX_CONTINUATION_DEPTH;
+            /**
+             * First hop: the SAME model again, through the front door. The pool
+             * picks a seat (sticky binding keeps the prompt cache warm), and if the
+             * provider cannot take it at all the existing pre-byte failover already
+             * hands it to the other one. Null when the client named no model.
+             */
+            const sameModelTarget = () => {
+                const m = parseClientBody()?.model;
+                return typeof m === 'string' && m.length > 0 ? { model: m, label: `${m} (same model)` } : null;
+            };
             const loopbackHeaders = () => {
                 const h = {};
                 if (apiKey)
@@ -3373,12 +3385,13 @@ export async function startProxy(opts = {}) {
                                     res.write(chunk); },
                                 isClientGone: () => res.destroyed || res.writableEnded,
                                 requestNo: codexReq,
+                                depth: requestDepth,
                                 verbose,
                                 resume: {
                                     clientBody: parseClientBody,
                                     loopbackBase,
                                     loopbackHeaders: loopbackHeaders(),
-                                    resolveTarget: async () => claudeContinuation,
+                                    resolveTarget: async (hop) => hop === 1 ? sameModelTarget() : claudeContinuation,
                                     onBeforeResume: releaseQueueSlot,
                                     timeoutMs: upstreamTimeoutMs,
                                 },
@@ -4704,12 +4717,13 @@ export async function startProxy(opts = {}) {
                             res.end(); },
                         isClientGone: () => clientDisconnected || res.destroyed || upstreamAbortReason === 'client_closed' || upstreamAbortReason === 'sse_overflow',
                         requestNo: requestCount,
+                        depth: requestDepth,
                         verbose,
                         resume: {
                             clientBody: parseClientBody,
                             loopbackBase,
                             loopbackHeaders: loopbackHeaders(),
-                            resolveTarget: codexContinuationTarget,
+                            resolveTarget: async (hop) => hop === 1 ? sameModelTarget() : codexContinuationTarget(),
                             onBeforeResume: releaseQueueSlot,
                             timeoutMs: upstreamTimeoutMs,
                         },

package/docs/midstream-continuation.md

@@ -46,23 +46,34 @@ Non-streaming requests are untouched; nothing was on the wire.
 
 Through dario's own front door. The resume is a loopback `POST` to the same
 proxy, so the pool, the codex translator, cch, the template, every rule that
-applied to the original request applies to the resume. The request carries
-`x-dario-continuation` so it is never itself continued — one resume per client
-request, no nesting.
-
-The target is the other provider's entry in `--pool-fallback`, exactly the entry
-a mid-flight 429 would use:
-
-- a Claude stream resumes on the codex half of the chain (`gpt-5.6-sol` in
-  `--pool-fallback=gpt-5.6-sol,claude:claude-sonnet-5`), resolved at failure
-  time against the account's live model list;
-- a codex stream resumes on the Claude half (`claude-sonnet-5` above), resolved
-  against the live catalog the way the chain already is.
-
-With no chain, or no entry for the other provider, the stream ends exactly as it
-did before 6.1, and the log says why once:
+applied to the original request applies to the resume.
+
+Two choices, in order:
+
+1. **The same model again.** A fresh request for the model the client asked
+   for. The pool picks a seat — the sticky binding keeps the prompt cache warm
+   — and if the provider cannot take the request at all, the existing pre-byte
+   failover already hands it to the other one. A transient reset therefore
+   finishes on the model the user chose, with no chain configured. This is
+   what most streams that die get, and most users have one plan.
+2. **The other provider's entry in `--pool-fallback`**, exactly the entry a
+   mid-flight 429 would use: a Claude stream goes to the codex half of the
+   chain (`gpt-5.6-sol` in `--pool-fallback=gpt-5.6-sol,claude:claude-sonnet-5`),
+   resolved at failure time against the account's live model list; a codex
+   stream goes to the Claude half, resolved against the live catalog.
+
+Choice 2 is taken when choice 1 delivers nothing — refused, unreachable, dead
+before its first byte — or when the resume itself dies mid-way. In the second
+case the resume's own guard makes the hop, so the client stream carries two
+seams: `(same model)` then `(codex live)`. The loopback carries
+`x-dario-continuation: <depth>`; a request at depth 2 is never continued. Two
+hops, never three: a third would be a third attempt at whatever is failing.
+
+With only one plan and no chain, a stream whose same-model resume also fails
+ends where the resume stopped, and the log says why once:
 
 ```
+[dario] #42 continuation as claude-opus-5 (same model) delivered nothing — trying the next choice
 [dario] #42 stream died after 1240 chars — no continuation target (set --pool-fallback with an entry for the other provider)
 ```
 
@@ -120,7 +131,7 @@ is never closed with a synthetic `end_turn`; only the resume's own
 |---|---|
 | `--no-midstream-continue` | off for this proxy |
 | `DARIO_MIDSTREAM_CONTINUE=0` | same, for the container |
-| `--pool-fallback=…` | where a stream resumes; no entry for the other provider means no resume |
+| `--pool-fallback=…` | where the second hop goes; without an entry for the other provider a stream gets the same-model resume only |
 
 On by default: it only ever acts where the alternative is a broken stream.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@askalf/dario",
-  "version": "6.1.0",
+  "version": "6.2.0",
   "description": "Use your Claude and ChatGPT subscriptions in Cursor, Cline, Aider, Claude Code and the Agent SDK — at subscription pricing, not per-token API bills. One local Anthropic + OpenAI-compatible endpoint: either plan answers either wire shape, with automatic failover when one hits its limit.",
   "type": "module",
   "bin": {