@askalf/dario 6.6.7 → 6.7.1

package/README.md

@@ -337,6 +337,17 @@ curl localhost:3456/v1/messages \
 
 You get the Claude answer, exactly as you would have. Beside it, dario runs the same prompt past `gpt-5.6-sol` and writes both to `~/.dario/compare/<timestamp>-<model>.json`, in your own wire shape, so you are comparing like with like. The comparison cannot degrade the request it observes: it only reads bytes already on their way out, your request is never held open for it, and a comparison that fails, times out or has nowhere to go is dropped with the record still written. Both sides are stored as raw payloads, because extracting text is where a bug would quietly make two answers look more alike than they are.
 
+Read them with **`dario compare`**: calls, success rate, median latency, how often each model's answer parsed as JSON, average length — and, above all, why any comparison was skipped. That last column is the point. The records had no reader until 6.7, and a week-long comparison on a box collected 919 of them without a single usable result, every one carrying its own reason inside the file. A log nobody can read is a log nobody reads.
+
+```
+  Records: 1,515  (2026-09-06 → 2026-09-13)
+  Compared: 1,515 with both sides
+
+  model          calls    200s    median   valid JSON   avg chars
+  gpt-5.6-luna    1515    100%    2736ms          98%          410
+  gpt-5.5         1515     99%    2729ms         100%          523
+```
+
 ## Many seats, one endpoint
 
 **Every dario is a pool.** A plain `dario login` is a pool of one; there is no separate mode to switch on. Hold more than one seat — a personal Max and a work Max, a couple of Pros, team seats — and the same `localhost:3456` routes every request to whichever seat has the most headroom, live, per request.
@@ -506,7 +517,7 @@ Longer version, with specifics: [#68](https://github.com/askalf/dario/discussion
 | `dario accounts list` / `add` / `remove` / `check <alias>` | Pool management; `check` sends one pinned request per model through the running proxy (admin API on) |
 | `dario backend list` / `add` / `remove` | OpenAI-compatible API-key backends |
 | `dario codex list` / `add` / `remove` | ChatGPT accounts (the long form of `dario add altman`) |
-| `dario usage` · `dario config` · `dario status` | Lifetime API-equivalent spend + burn rate for the last hour (`--card` writes the share card) · effective config, redacted · token health |
+| `dario usage` · `dario compare` · `dario config` · `dario status` | Lifetime API-equivalent spend + burn rate for the last hour (`--card` writes the share card) · read the shadow-compare log · effective config, redacted · token health |
 | `dario resume` · `dario refresh` · `dario logout` · `dario upgrade` | Clear an overage halt · force a token refresh · delete credentials · safe self-update |
 | `dario mcp` · `dario subagent install` / `remove` / `status` | Reach dario from inside any MCP client, or from inside a Claude Code session, read-only |
 

package/dist/cli.js

@@ -1594,6 +1594,13 @@ async function help() {
                              down). --card[=file.svg] writes a share
                              card of that number (default
                              dario-api-equivalent.svg). (v6.6)
+    dario compare            Read the shadow-compare log written by
+                             requests carrying \`x-dario-compare\`:
+                             per-model calls, success rate, median
+                             latency, how often the answer parsed as
+                             JSON, average length — plus why any
+                             comparison was skipped. --dir=PATH to read
+                             elsewhere, --json for the summary. (v6.7)
     dario upgrade            npm install -g @askalf/dario@latest with a
                              pre-flight current-vs-latest check.
 
@@ -2574,6 +2581,29 @@ async function add() {
     process.exit(1);
 }
 // Main
+/**
+ * `dario compare` — read the shadow-compare log.
+ *
+ * The records have existed since v6.0.0 with nothing to read them with, which
+ * is how a week-long bake-off on a production box collected 919 failed
+ * comparisons without anyone noticing (#1306): the reason was in the files,
+ * and nobody opens 919 files. This prints the answer instead.
+ *
+ * --dir=PATH   read somewhere other than ~/.dario/compare
+ * --json       the summary as JSON, for a dashboard or a diff over time
+ */
+async function compare() {
+    const { DEFAULT_COMPARE_DIR, readCompareDir, summarizeCompareRecords, formatCompareReport } = await import('./compare-report.js');
+    const dirArg = args.find(a => a.startsWith('--dir='));
+    const dir = dirArg ? dirArg.slice('--dir='.length) : DEFAULT_COMPARE_DIR;
+    const summary = summarizeCompareRecords(readCompareDir(dir));
+    if (args.includes('--json')) {
+        process.stdout.write(JSON.stringify({ dir, ...summary }, null, 2) + '\n');
+        return;
+    }
+    for (const line of formatCompareReport(summary, dir))
+        console.log(line);
+}
 const commands = {
     login,
     status,
@@ -2592,6 +2622,7 @@ const commands = {
     config,
     upgrade,
     usage,
+    compare,
     tui,
     help,
     version,

package/dist/compare-report.d.ts

@@ -0,0 +1,69 @@
+/**
+ * Reading the shadow-compare log.
+ *
+ * dario has written one JSON record per compared request since v6.0.0 and
+ * shipped nothing to read them with. That gap is not cosmetic: a bake-off on
+ * the box collected 919 records over a week, every one of them a failed
+ * comparison, with the reason sitting inside the files (#1306). Nobody opened
+ * them, because opening 919 JSON blobs is not a thing anyone does by hand.
+ *
+ * So the log gets a reader. `dario compare` answers the question the records
+ * were collected for — which model is faster, which returns what you asked
+ * for, how often either failed — and prints the skip reasons, so a comparison
+ * that is not running says so on the first line instead of in file 700.
+ *
+ * Pure except for the directory read: the summary is computed from records the
+ * caller supplies, so the shapes below are testable without a filesystem.
+ */
+export declare const DEFAULT_COMPARE_DIR: string;
+/** One side of a comparison, as written by src/compare.ts. */
+export interface CompareSideRecord {
+    status: number | null;
+    body: string;
+    ms: number;
+}
+export interface CompareRecordShape {
+    ts?: string;
+    primaryModel?: string;
+    comparedModel?: string;
+    primary?: CompareSideRecord | null;
+    compare?: CompareSideRecord | null;
+    skipped?: string;
+}
+export interface ModelStats {
+    model: string;
+    calls: number;
+    ok: number;
+    /** Median latency in ms, or null when nothing recorded one. */
+    medianMs: number | null;
+    /** Responses whose text parsed as JSON — the thing an extraction caller wants. */
+    json: number;
+    avgChars: number;
+}
+export interface CompareSummary {
+    records: number;
+    /** Records carrying BOTH sides — the only ones that compare anything. */
+    pairs: number;
+    first: string | null;
+    last: string | null;
+    perModel: ModelStats[];
+    /** Skip reason → count, most common first. */
+    skips: Array<{
+        reason: string;
+        count: number;
+    }>;
+}
+/**
+ * The assistant text inside a recorded body, whatever it is wrapped in: an
+ * Anthropic message, a legacy completion, or an SSE stream of text deltas.
+ * Returns '' when there is nothing quotable — a caller measuring "did this
+ * answer with JSON" wants the text, not the envelope.
+ */
+export declare function textOfBody(body: string | undefined | null): string;
+/** Whether text is JSON once a markdown fence is peeled off. */
+export declare function looksLikeJson(text: string): boolean;
+export declare function summarizeCompareRecords(records: readonly CompareRecordShape[]): CompareSummary;
+/** Read every record in a directory, skipping anything unparseable. */
+export declare function readCompareDir(dir: string): CompareRecordShape[];
+/** The report `dario compare` prints. Two-space indent, like the other commands. */
+export declare function formatCompareReport(s: CompareSummary, dir: string): string[];

package/dist/compare-report.js

@@ -0,0 +1,186 @@
+/**
+ * Reading the shadow-compare log.
+ *
+ * dario has written one JSON record per compared request since v6.0.0 and
+ * shipped nothing to read them with. That gap is not cosmetic: a bake-off on
+ * the box collected 919 records over a week, every one of them a failed
+ * comparison, with the reason sitting inside the files (#1306). Nobody opened
+ * them, because opening 919 JSON blobs is not a thing anyone does by hand.
+ *
+ * So the log gets a reader. `dario compare` answers the question the records
+ * were collected for — which model is faster, which returns what you asked
+ * for, how often either failed — and prints the skip reasons, so a comparison
+ * that is not running says so on the first line instead of in file 700.
+ *
+ * Pure except for the directory read: the summary is computed from records the
+ * caller supplies, so the shapes below are testable without a filesystem.
+ */
+import { readdirSync, readFileSync } from 'node:fs';
+import { join } from 'node:path';
+import { homedir } from 'node:os';
+export const DEFAULT_COMPARE_DIR = join(homedir(), '.dario', 'compare');
+/**
+ * The assistant text inside a recorded body, whatever it is wrapped in: an
+ * Anthropic message, a legacy completion, or an SSE stream of text deltas.
+ * Returns '' when there is nothing quotable — a caller measuring "did this
+ * answer with JSON" wants the text, not the envelope.
+ */
+export function textOfBody(body) {
+    if (!body)
+        return '';
+    try {
+        const j = JSON.parse(body);
+        if (Array.isArray(j.content))
+            return j.content.filter((b) => b?.type === 'text').map((b) => b.text ?? '').join('');
+        if (typeof j.completion === 'string')
+            return j.completion;
+    }
+    catch { /* not JSON — an SSE stream, most likely */ }
+    const deltas = [...String(body).matchAll(/"type"\s*:\s*"text_delta"\s*,\s*"text"\s*:\s*("(?:[^"\\]|\\.)*")/g)];
+    if (deltas.length > 0) {
+        return deltas.map((m) => { try {
+            return JSON.parse(m[1]);
+        }
+        catch {
+            return '';
+        } }).join('');
+    }
+    return '';
+}
+/** Whether text is JSON once a markdown fence is peeled off. */
+export function looksLikeJson(text) {
+    const s = String(text).trim().replace(/^```(?:json)?\s*/i, '').replace(/\s*```$/, '');
+    if (s.length === 0)
+        return false;
+    try {
+        JSON.parse(s);
+        return true;
+    }
+    catch {
+        return false;
+    }
+}
+function median(values) {
+    if (values.length === 0)
+        return null;
+    const sorted = [...values].sort((a, b) => a - b);
+    return sorted[Math.floor(sorted.length / 2)];
+}
+export function summarizeCompareRecords(records) {
+    const acc = new Map();
+    const skips = new Map();
+    let pairs = 0;
+    let first = null;
+    let last = null;
+    const add = (model, side) => {
+        if (!model || !side)
+            return;
+        let s = acc.get(model);
+        if (!s) {
+            s = { calls: 0, ok: 0, ms: [], json: 0, chars: 0 };
+            acc.set(model, s);
+        }
+        s.calls++;
+        if (side.status === 200)
+            s.ok++;
+        if (typeof side.ms === 'number' && Number.isFinite(side.ms))
+            s.ms.push(side.ms);
+        const text = textOfBody(side.body);
+        s.chars += text.length;
+        if (looksLikeJson(text))
+            s.json++;
+    };
+    for (const r of records) {
+        if (typeof r.ts === 'string') {
+            first ??= r.ts;
+            last = r.ts;
+        }
+        add(r.primaryModel, r.primary);
+        if (r.skipped) {
+            skips.set(r.skipped, (skips.get(r.skipped) ?? 0) + 1);
+            continue;
+        }
+        add(r.comparedModel, r.compare);
+        if (r.primary && r.compare)
+            pairs++;
+    }
+    const perModel = [...acc]
+        .map(([model, s]) => ({
+        model,
+        calls: s.calls,
+        ok: s.ok,
+        medianMs: median(s.ms),
+        json: s.json,
+        avgChars: Math.round(s.chars / Math.max(1, s.calls)),
+    }))
+        .sort((a, b) => b.calls - a.calls || a.model.localeCompare(b.model));
+    return {
+        records: records.length,
+        pairs,
+        first,
+        last,
+        perModel,
+        skips: [...skips].map(([reason, count]) => ({ reason, count })).sort((a, b) => b.count - a.count),
+    };
+}
+/** Read every record in a directory, skipping anything unparseable. */
+export function readCompareDir(dir) {
+    let names;
+    try {
+        names = readdirSync(dir).filter((f) => f.endsWith('.json')).sort();
+    }
+    catch {
+        return [];
+    }
+    const out = [];
+    for (const name of names) {
+        try {
+            out.push(JSON.parse(readFileSync(join(dir, name), 'utf-8')));
+        }
+        catch { /* a half-written or hand-edited file is not worth failing the report over */ }
+    }
+    return out;
+}
+const pct = (a, b) => (b === 0 ? '—' : `${Math.round((a / b) * 100)}%`);
+/** The report `dario compare` prints. Two-space indent, like the other commands. */
+export function formatCompareReport(s, dir) {
+    const out = [];
+    out.push('');
+    out.push('  dario — Shadow compare');
+    out.push('  ─────────────────────');
+    out.push('');
+    if (s.records === 0) {
+        out.push(`  No records in ${dir}.`);
+        out.push('');
+        out.push('  A comparison is requested per-request with the `x-dario-compare: <model>` header,');
+        out.push('  and the model has to be one your ChatGPT plan lists. Nothing is compared by default.');
+        out.push('');
+        return out;
+    }
+    const span = s.first && s.last && s.first.slice(0, 10) !== s.last.slice(0, 10)
+        ? `${s.first.slice(0, 10)} → ${s.last.slice(0, 10)}`
+        : (s.first ?? '').slice(0, 10);
+    out.push(`  Records: ${s.records.toLocaleString('en-US')}${span ? `  (${span})` : ''}`);
+    out.push(`  Compared: ${s.pairs.toLocaleString('en-US')} with both sides`);
+    out.push('');
+    const width = Math.max(8, ...s.perModel.map((m) => m.model.length));
+    out.push(`  ${'model'.padEnd(width)}   calls    200s    median   valid JSON   avg chars`);
+    for (const m of s.perModel) {
+        out.push(`  ${m.model.padEnd(width)} ${String(m.calls).padStart(7)} ${pct(m.ok, m.calls).padStart(7)} `
+            + `${(m.medianMs === null ? '—' : `${m.medianMs}ms`).padStart(9)} ${pct(m.json, m.calls).padStart(12)} `
+            + `${String(m.avgChars).padStart(11)}`);
+    }
+    if (s.skips.length > 0) {
+        out.push('');
+        out.push('  Skipped comparisons:');
+        for (const { reason, count } of s.skips) {
+            out.push(`    ${String(count).padStart(5)}  ${reason.length > 96 ? `${reason.slice(0, 93)}…` : reason}`);
+        }
+    }
+    if (s.pairs === 0) {
+        out.push('');
+        out.push('  Nothing has both sides yet, so there is nothing to compare — see the skip reasons above.');
+    }
+    out.push('');
+    return out;
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@askalf/dario",
-  "version": "6.6.7",
+  "version": "6.7.1",
   "description": "Use your Claude and ChatGPT subscriptions in Cursor, Cline, Aider, Claude Code and the Agent SDK — at subscription pricing, not per-token API bills. One local Anthropic + OpenAI-compatible endpoint: either plan answers either wire shape, with automatic failover when one hits its limit.",
   "type": "module",
   "bin": {