@askalf/dario 6.8.18 → 6.9.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +3 -3
- package/dist/analytics.d.ts +14 -0
- package/dist/analytics.js +3 -1
- package/dist/cli.js +6 -0
- package/dist/codex-backend.d.ts +16 -2
- package/dist/codex-backend.js +59 -11
- package/dist/metrics.js +26 -0
- package/dist/pacing.d.ts +29 -0
- package/dist/pacing.js +22 -0
- package/dist/proxy.d.ts +7 -0
- package/dist/proxy.js +66 -34
- package/dist/timing.d.ts +92 -0
- package/dist/timing.js +93 -0
- package/dist/tui/tabs/analytics.d.ts +9 -0
- package/dist/tui/tabs/analytics.js +7 -0
- package/docs/analytics.md +21 -1
- package/docs/wire-fidelity.md +1 -1
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -27,7 +27,7 @@
|
|
|
27
27
|
|
|
28
28
|
<p><strong>One local endpoint. Every AI tool you own. The subscriptions you already pay for.</strong></p>
|
|
29
29
|
|
|
30
|
-
<sub><code>npm i -g @askalf/dario</code> · <strong>0</strong> runtime deps · <a href="https://www.npmjs.com/package/@askalf/dario">SLSA-attested</a> every release · nothing phones home · ~
|
|
30
|
+
<sub><code>npm i -g @askalf/dario</code> · <strong>0</strong> runtime deps · <a href="https://www.npmjs.com/package/@askalf/dario">SLSA-attested</a> every release · nothing phones home · ~39k lines you can read in a weekend · independent, unofficial, third-party (<a href="DISCLAIMER.md">DISCLAIMER.md</a>)</sub>
|
|
31
31
|
|
|
32
32
|
<sub><a href="#start-in-60-seconds">Start</a> · <a href="#point-your-tools-at-it">Your tools</a> · <a href="#what-it-does-with-a-request">Routing</a> · <a href="#two-plans-one-endpoint">Two plans</a> · <a href="#many-seats-one-endpoint">Pool</a> · <a href="#it-tracks-a-moving-target">Drift</a> · <a href="#trust--transparency">Trust</a> · <a href="#will-my-account-get-suspended">Risk</a> · <a href="#commands">Commands</a> · <a href="#faq">FAQ</a> · <a href="docs/returning.md">Coming back after a while?</a></sub>
|
|
33
33
|
|
|
@@ -409,7 +409,7 @@ The rolling window forgets on every restart; the **ledger** does not. Since 6.6
|
|
|
409
409
|
|
|
410
410
|
Only served requests count. Traffic that was metered anyway — an API key upstream, or Anthropic's paid `extra_usage` overage — is kept in its own column and reported as spent, not saved. `dario usage --card` writes the headline as a 640×320 SVG you can drop in a README or a post, and `--donut` writes the same number as three rings — by model, by key, subscription vs metered; `--no-ledger` / `DARIO_LEDGER=0` turns the file off, `DARIO_LEDGER_PATH` moves it, and `GET /analytics/ledger` is the per-day table behind the number. Details: [api-equivalent-spend.md](./docs/api-equivalent-spend.md).
|
|
411
411
|
|
|
412
|
-
**Scrape it, or open it.** `GET /metrics` is the same state as Prometheus text exposition — window, seats, models, consumers, queue, latency quantiles, burn rates, ledger — so Grafana reads dario like anything else. `GET /analytics/ui` is a self-contained dashboard page with the headline, the rings and the tables, refreshing every minute. Both sit behind the same gate as `/analytics`; `--analytics-token` (env `DARIO_ANALYTICS_TOKEN`) adds a **read-only** credential accepted on those paths and nowhere else, so a scraper or a browser can hold the numbers without holding request rights. Families and the gate: [analytics.md](./docs/analytics.md).
|
|
412
|
+
**Scrape it, or open it.** `GET /metrics` is the same state as Prometheus text exposition — window, seats, models, consumers, queue, latency quantiles, burn rates, ledger — so Grafana reads dario like anything else. Since 6.9 every request also carries its **timing split**: queue wait, governor sleep, the provider's time to first byte, the provider's total, and what is left — dario's own overhead — as `x-dario-*-ms` response headers on the request itself, as `window.timing` on `/analytics`, and as five more `/metrics` families, so "was that Anthropic or the proxy?" is a `curl -i` away ([details](./docs/analytics.md#the-timing-split)). `GET /analytics/ui` is a self-contained dashboard page with the headline, the rings and the tables, refreshing every minute. Both sit behind the same gate as `/analytics`; `--analytics-token` (env `DARIO_ANALYTICS_TOKEN`) adds a **read-only** credential accepted on those paths and nowhere else, so a scraper or a browser can hold the numbers without holding request rights. Families and the gate: [analytics.md](./docs/analytics.md).
|
|
413
413
|
|
|
414
414
|
## It tracks a moving target
|
|
415
415
|
|
|
@@ -465,7 +465,7 @@ The split isn't live, but it was announced once on short notice and could return
|
|
|
465
465
|
|
|
466
466
|
| Signal | Status |
|
|
467
467
|
|---|---|
|
|
468
|
-
| Source | **~
|
|
468
|
+
| Source | **~39k** lines of TypeScript across **77** files, auditable in a weekend. One credential path since v5: the pool. |
|
|
469
469
|
| Dependencies | **0 runtime.** Verify: `npm ls --production` |
|
|
470
470
|
| Provenance | Every release [SLSA-attested](https://www.npmjs.com/package/@askalf/dario) via GitHub Actions + Sigstore, published with OIDC trusted publishing — no long-lived npm token exists to leak |
|
|
471
471
|
| Scanning | [CodeQL](https://github.com/askalf/dario/actions/workflows/codeql.yml) on every push and weekly · [ClusterFuzzLite](./.github/workflows/cflite.yml) fuzzes the SSE translator and rejection parsers weekly · [OpenSSF Scorecard](https://scorecard.dev/viewer/?uri=github.com/askalf/dario) and [Best Practices](https://www.bestpractices.dev/projects/13638) badges above are live |
|
package/dist/analytics.d.ts
CHANGED
|
@@ -17,6 +17,7 @@
|
|
|
17
17
|
* each subscriber listens for `'record'` and writes one SSE frame.
|
|
18
18
|
*/
|
|
19
19
|
import { EventEmitter } from 'node:events';
|
|
20
|
+
import { type RequestTiming, type TimingStats } from './timing.js';
|
|
20
21
|
export interface RequestRecord {
|
|
21
22
|
timestamp: number;
|
|
22
23
|
/**
|
|
@@ -47,6 +48,13 @@ export interface RequestRecord {
|
|
|
47
48
|
* delivered, which leg served the rest. Absent on every ordinary request.
|
|
48
49
|
*/
|
|
49
50
|
continuation?: RequestContinuation;
|
|
51
|
+
/**
|
|
52
|
+
* Where the wall-clock time went (src/timing.ts): queue wait, governor
|
|
53
|
+
* sleep, upstream time-to-first-byte, upstream total, end-to-end total.
|
|
54
|
+
* Absent on rows written before the split existed and on legs that never
|
|
55
|
+
* reached an upstream.
|
|
56
|
+
*/
|
|
57
|
+
timing?: RequestTiming;
|
|
50
58
|
}
|
|
51
59
|
export interface RequestContinuation {
|
|
52
60
|
/** See midstream.ts ContinuationOutcome. */
|
|
@@ -351,6 +359,12 @@ interface WindowStats {
|
|
|
351
359
|
cachedPromptPercent: number;
|
|
352
360
|
estimatedCost: number;
|
|
353
361
|
avgLatencyMs: number;
|
|
362
|
+
/**
|
|
363
|
+
* The latency split, averaged over the rows that carry one: queue wait,
|
|
364
|
+
* governor sleep, upstream TTFB, upstream total and dario's own overhead
|
|
365
|
+
* (src/timing.ts). `samples` says how many rows that was.
|
|
366
|
+
*/
|
|
367
|
+
timing: TimingStats;
|
|
354
368
|
errorRate: number;
|
|
355
369
|
/** Mid-stream continuations in the window and how they went (v6.1 guard, counted since v6.6.1). */
|
|
356
370
|
continuations: ContinuationStats;
|
package/dist/analytics.js
CHANGED
|
@@ -18,6 +18,7 @@
|
|
|
18
18
|
*/
|
|
19
19
|
import { EventEmitter } from 'node:events';
|
|
20
20
|
import { createHash } from 'node:crypto';
|
|
21
|
+
import { timingStats } from './timing.js';
|
|
21
22
|
export function continuationStats(records) {
|
|
22
23
|
const out = { attempted: 0, finished: 0, unfinished: 0, failed: 0, noTarget: 0 };
|
|
23
24
|
for (const r of records) {
|
|
@@ -397,7 +398,7 @@ export class Analytics extends EventEmitter {
|
|
|
397
398
|
return {
|
|
398
399
|
totalInputTokens: 0, totalOutputTokens: 0, totalThinkingTokens: 0,
|
|
399
400
|
totalCacheReadTokens: 0, totalCacheCreateTokens: 0, cachedPromptPercent: 0,
|
|
400
|
-
estimatedCost: 0, avgLatencyMs: 0, errorRate: 0,
|
|
401
|
+
estimatedCost: 0, avgLatencyMs: 0, timing: timingStats([]), errorRate: 0,
|
|
401
402
|
continuations: { attempted: 0, finished: 0, unfinished: 0, failed: 0, noTarget: 0 },
|
|
402
403
|
claimBreakdown: {},
|
|
403
404
|
billingBucketBreakdown: {
|
|
@@ -444,6 +445,7 @@ export class Analytics extends EventEmitter {
|
|
|
444
445
|
cachedPromptPercent: cachedPromptPercent(totalInput, totalCacheRead, totalCacheCreate),
|
|
445
446
|
estimatedCost: Math.round(cost * 10000) / 10000,
|
|
446
447
|
avgLatencyMs: Math.round(avgLatency),
|
|
448
|
+
timing: timingStats(records.map(r => r.timing)),
|
|
447
449
|
errorRate: Math.round((errors / records.length) * 10000) / 10000,
|
|
448
450
|
continuations: continuationStats(records),
|
|
449
451
|
claimBreakdown: claims,
|
package/dist/cli.js
CHANGED
|
@@ -2801,6 +2801,12 @@ async function usage() {
|
|
|
2801
2801
|
console.log(` Input tokens: ${(win.totalInputTokens ?? 0).toLocaleString()}`);
|
|
2802
2802
|
console.log(` Output tokens: ${(win.totalOutputTokens ?? 0).toLocaleString()}`);
|
|
2803
2803
|
console.log(` Avg latency: ${win.avgLatencyMs ?? 0} ms`);
|
|
2804
|
+
// Where that went (src/timing.ts): the provider, the waits, and dario itself.
|
|
2805
|
+
const t = win.timing;
|
|
2806
|
+
if (t && t.samples > 0) {
|
|
2807
|
+
console.log(` upstream: ${t.avgUpstreamMs} ms (TTFB ${t.avgUpstreamTtfbMs} ms)`);
|
|
2808
|
+
console.log(` dario: ${t.avgOverheadMs} ms overhead · ${t.avgQueueMs} ms queue · ${t.avgPacingMs} ms pacing`);
|
|
2809
|
+
}
|
|
2804
2810
|
if ((win.errorRate ?? 0) > 0) {
|
|
2805
2811
|
console.log(` Error rate: ${((win.errorRate ?? 0) * 100).toFixed(1)}%`);
|
|
2806
2812
|
}
|
package/dist/codex-backend.d.ts
CHANGED
|
@@ -34,6 +34,10 @@ export declare function fetchCodexModels(creds: CodexAccountCredentials, fetchIm
|
|
|
34
34
|
export interface CodexForwardOutcome {
|
|
35
35
|
status: number;
|
|
36
36
|
latencyMs: number;
|
|
37
|
+
/** First outbound fetch → upstream response headers, ms: the seat's time to first byte (the served attempt when a refresh forced a retry). 0 when nothing went out. */
|
|
38
|
+
upstreamTtfbMs: number;
|
|
39
|
+
/** First outbound fetch → the report, ms: the seat's time including its body. 0 when nothing went out. */
|
|
40
|
+
upstreamMs: number;
|
|
37
41
|
/** Net of the cached prefix (Anthropic convention; see splitResponsesUsage). */
|
|
38
42
|
inputTokens: number;
|
|
39
43
|
outputTokens: number;
|
|
@@ -311,7 +315,12 @@ onDecline?: (info: CodexDecline) => void,
|
|
|
311
315
|
/** When true a decline returns false WITHOUT writing, so the caller can
|
|
312
316
|
* retry the request on a healthy peer. False keeps the old behaviour: the
|
|
313
317
|
* upstream error is written through as the backend sent it. */
|
|
314
|
-
deferOnUnavailable?: boolean
|
|
318
|
+
deferOnUnavailable?: boolean,
|
|
319
|
+
/** Queue wait and arrival stamp from the proxy: the response carries the timing headers (src/timing.ts) like a Claude-path response does. */
|
|
320
|
+
timing?: {
|
|
321
|
+
queueMs: number;
|
|
322
|
+
arrivedAt: number;
|
|
323
|
+
}): Promise<boolean>;
|
|
315
324
|
/**
|
|
316
325
|
* Serve a request from a stored Codex account, in either client wire shape.
|
|
317
326
|
*
|
|
@@ -354,4 +363,9 @@ effort?: ResponsesReasoningConfig['effort'],
|
|
|
354
363
|
* stream that dies with content on the wire is finished from the Claude
|
|
355
364
|
* pool instead of ending truncated.
|
|
356
365
|
*/
|
|
357
|
-
midstream?: MidstreamGuard | null
|
|
366
|
+
midstream?: MidstreamGuard | null,
|
|
367
|
+
/** Queue wait and arrival stamp from the proxy: the response carries the timing headers (src/timing.ts) like a Claude-path response does. */
|
|
368
|
+
timing?: {
|
|
369
|
+
queueMs: number;
|
|
370
|
+
arrivedAt: number;
|
|
371
|
+
}): Promise<boolean>;
|
package/dist/codex-backend.js
CHANGED
|
@@ -22,6 +22,7 @@
|
|
|
22
22
|
* translation in both directions, including SSE.
|
|
23
23
|
*/
|
|
24
24
|
import { createHash } from 'node:crypto';
|
|
25
|
+
import { timingHeaders } from './timing.js';
|
|
25
26
|
import { forceRefreshCodexAccount } from './codex-accounts.js';
|
|
26
27
|
import { anthropicToResponsesRequest, anthropicUsageFromResponses, createResponsesSSEParser, formatResponsesAnthropicSSE, createAnthropicMessageAssembler, responsesStreamToAnthropicSSE, } from './anthropic-responses-translate.js';
|
|
27
28
|
import { resolveClaudeTarget } from './claude-model.js';
|
|
@@ -815,8 +816,30 @@ onDecline,
|
|
|
815
816
|
/** When true a decline returns false WITHOUT writing, so the caller can
|
|
816
817
|
* retry the request on a healthy peer. False keeps the old behaviour: the
|
|
817
818
|
* upstream error is written through as the backend sent it. */
|
|
818
|
-
deferOnUnavailable = false
|
|
819
|
+
deferOnUnavailable = false,
|
|
820
|
+
/** Queue wait and arrival stamp from the proxy: the response carries the timing headers (src/timing.ts) like a Claude-path response does. */
|
|
821
|
+
timing) {
|
|
819
822
|
const startedAt = Date.now();
|
|
823
|
+
// Timing split (src/timing.ts): the first outbound fetch and the headers of
|
|
824
|
+
// the attempt that was served. A refresh-and-retry keeps the first start —
|
|
825
|
+
// the rejected attempt was the seat's time too.
|
|
826
|
+
let fetchStartedAt = 0;
|
|
827
|
+
let upstreamHeadersAt = 0;
|
|
828
|
+
const timedFetch = async (input, init) => {
|
|
829
|
+
if (!fetchStartedAt)
|
|
830
|
+
fetchStartedAt = Date.now();
|
|
831
|
+
const r = await fetchImpl(input, init);
|
|
832
|
+
upstreamHeadersAt = Date.now();
|
|
833
|
+
return r;
|
|
834
|
+
};
|
|
835
|
+
const ttfbMs = () => (fetchStartedAt && upstreamHeadersAt ? Math.max(0, upstreamHeadersAt - fetchStartedAt) : 0);
|
|
836
|
+
const upstreamMsNow = () => (fetchStartedAt ? Math.max(0, Date.now() - fetchStartedAt) : 0);
|
|
837
|
+
// The four x-dario-*-ms headers, merged into every response this leg writes
|
|
838
|
+
// once the backend has answered. The governor never runs for codex: pacing 0.
|
|
839
|
+
const splitHeaders = () => timingHeaders({
|
|
840
|
+
queueMs: timing?.queueMs ?? 0, pacingMs: 0, arrivedAt: timing?.arrivedAt ?? startedAt,
|
|
841
|
+
fetchStartedAt: fetchStartedAt || Date.now(), upstreamTtfbMs: ttfbMs(),
|
|
842
|
+
});
|
|
820
843
|
const model = String(body.model ?? '');
|
|
821
844
|
let reported = false;
|
|
822
845
|
const report = (status, usage) => {
|
|
@@ -824,7 +847,7 @@ deferOnUnavailable = false) {
|
|
|
824
847
|
return;
|
|
825
848
|
reported = true;
|
|
826
849
|
try {
|
|
827
|
-
onDone({ status, latencyMs: Date.now() - startedAt, inputTokens: usage?.input ?? 0, outputTokens: usage?.output ?? 0,
|
|
850
|
+
onDone({ status, latencyMs: Date.now() - startedAt, upstreamTtfbMs: ttfbMs(), upstreamMs: upstreamMsNow(), inputTokens: usage?.input ?? 0, outputTokens: usage?.output ?? 0,
|
|
828
851
|
cacheReadTokens: usage?.cacheRead ?? 0, cacheCreateTokens: usage?.cacheCreate ?? 0, stream: true, model, alias: creds.alias });
|
|
829
852
|
}
|
|
830
853
|
catch { /* never break a served request */ }
|
|
@@ -856,13 +879,13 @@ deferOnUnavailable = false) {
|
|
|
856
879
|
if (verbose)
|
|
857
880
|
console.log(`[dario] → codex backend (responses passthrough): ${target} (model: ${model})`);
|
|
858
881
|
let activeCreds = creds;
|
|
859
|
-
let upstream = await
|
|
882
|
+
let upstream = await timedFetch(target, { method: 'POST', headers: buildCodexHeaders(activeCreds), body: JSON.stringify(upstreamBody), signal: abort.signal });
|
|
860
883
|
if (isCodexAuthFailure(upstream.status)) {
|
|
861
884
|
await upstream.text().catch(() => ''); // release the rejected response before retrying
|
|
862
885
|
const fresh = await refreshAfterCodexAuthFailure(activeCreds, verbose);
|
|
863
886
|
if (fresh) {
|
|
864
887
|
activeCreds = fresh;
|
|
865
|
-
upstream = await
|
|
888
|
+
upstream = await timedFetch(target, { method: 'POST', headers: buildCodexHeaders(activeCreds), body: JSON.stringify(upstreamBody), signal: abort.signal });
|
|
866
889
|
}
|
|
867
890
|
}
|
|
868
891
|
if (!upstream.ok || !upstream.body) {
|
|
@@ -890,14 +913,14 @@ deferOnUnavailable = false) {
|
|
|
890
913
|
return false;
|
|
891
914
|
}
|
|
892
915
|
if (!clientGone) {
|
|
893
|
-
res.writeHead(upstream.status, { 'Content-Type': 'application/json', ...securityHeaders });
|
|
916
|
+
res.writeHead(upstream.status, { 'Content-Type': 'application/json', ...securityHeaders, ...splitHeaders() });
|
|
894
917
|
// The backend's own error body, already in the client's shape.
|
|
895
918
|
res.end(detail || JSON.stringify({ error: { message: 'Upstream Codex backend error', type: 'server_error', code: null, param: null } }));
|
|
896
919
|
}
|
|
897
920
|
report(clientGone ? 499 : upstream.status, null);
|
|
898
921
|
return true;
|
|
899
922
|
}
|
|
900
|
-
res.writeHead(200, { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache', 'Connection': 'keep-alive', 'Access-Control-Allow-Origin': corsOrigin, ...securityHeaders });
|
|
923
|
+
res.writeHead(200, { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache', 'Connection': 'keep-alive', 'Access-Control-Allow-Origin': corsOrigin, ...securityHeaders, ...splitHeaders() });
|
|
901
924
|
const reader = upstream.body.getReader();
|
|
902
925
|
const decoder = new TextDecoder();
|
|
903
926
|
let tail = '';
|
|
@@ -1015,13 +1038,35 @@ effort,
|
|
|
1015
1038
|
* stream that dies with content on the wire is finished from the Claude
|
|
1016
1039
|
* pool instead of ending truncated.
|
|
1017
1040
|
*/
|
|
1018
|
-
midstream
|
|
1041
|
+
midstream,
|
|
1042
|
+
/** Queue wait and arrival stamp from the proxy: the response carries the timing headers (src/timing.ts) like a Claude-path response does. */
|
|
1043
|
+
timing) {
|
|
1019
1044
|
void req;
|
|
1020
1045
|
const isAnthropic = shape === 'anthropic';
|
|
1021
1046
|
// Reported exactly once, on every exit that answered the client. Without
|
|
1022
1047
|
// this the proxy had no idea a codex request happened: no analytics row, no
|
|
1023
1048
|
// log line, no per-account count.
|
|
1024
1049
|
const startedAt = Date.now();
|
|
1050
|
+
// Timing split (src/timing.ts): the first outbound fetch and the headers of
|
|
1051
|
+
// the attempt that was served. A refresh-and-retry keeps the first start —
|
|
1052
|
+
// the rejected attempt was the seat's time too.
|
|
1053
|
+
let fetchStartedAt = 0;
|
|
1054
|
+
let upstreamHeadersAt = 0;
|
|
1055
|
+
const timedFetch = async (input, init) => {
|
|
1056
|
+
if (!fetchStartedAt)
|
|
1057
|
+
fetchStartedAt = Date.now();
|
|
1058
|
+
const r = await fetchImpl(input, init);
|
|
1059
|
+
upstreamHeadersAt = Date.now();
|
|
1060
|
+
return r;
|
|
1061
|
+
};
|
|
1062
|
+
const ttfbMs = () => (fetchStartedAt && upstreamHeadersAt ? Math.max(0, upstreamHeadersAt - fetchStartedAt) : 0);
|
|
1063
|
+
const upstreamMsNow = () => (fetchStartedAt ? Math.max(0, Date.now() - fetchStartedAt) : 0);
|
|
1064
|
+
// The four x-dario-*-ms headers, merged into every response this leg writes
|
|
1065
|
+
// once the backend has answered. The governor never runs for codex: pacing 0.
|
|
1066
|
+
const splitHeaders = () => timingHeaders({
|
|
1067
|
+
queueMs: timing?.queueMs ?? 0, pacingMs: 0, arrivedAt: timing?.arrivedAt ?? startedAt,
|
|
1068
|
+
fetchStartedAt: fetchStartedAt || Date.now(), upstreamTtfbMs: ttfbMs(),
|
|
1069
|
+
});
|
|
1025
1070
|
let reported = false;
|
|
1026
1071
|
const report = (status, usage, stream, model) => {
|
|
1027
1072
|
if (reported || !onDone)
|
|
@@ -1029,7 +1074,7 @@ midstream) {
|
|
|
1029
1074
|
reported = true;
|
|
1030
1075
|
try {
|
|
1031
1076
|
onDone({
|
|
1032
|
-
status, latencyMs: Date.now() - startedAt,
|
|
1077
|
+
status, latencyMs: Date.now() - startedAt, upstreamTtfbMs: ttfbMs(), upstreamMs: upstreamMsNow(),
|
|
1033
1078
|
inputTokens: usage?.input ?? 0, outputTokens: usage?.output ?? 0,
|
|
1034
1079
|
cacheReadTokens: usage?.cacheRead ?? 0, cacheCreateTokens: usage?.cacheCreate ?? 0,
|
|
1035
1080
|
stream, model, alias: creds.alias,
|
|
@@ -1110,7 +1155,7 @@ midstream) {
|
|
|
1110
1155
|
if (verbose)
|
|
1111
1156
|
console.log(`[dario] → codex backend: ${target} (model: ${model})`);
|
|
1112
1157
|
let activeCreds = creds;
|
|
1113
|
-
let upstream = await
|
|
1158
|
+
let upstream = await timedFetch(target, {
|
|
1114
1159
|
method: 'POST',
|
|
1115
1160
|
headers: buildCodexHeaders(activeCreds),
|
|
1116
1161
|
body: JSON.stringify(scrubbed),
|
|
@@ -1123,7 +1168,7 @@ midstream) {
|
|
|
1123
1168
|
const fresh = await refreshAfterCodexAuthFailure(activeCreds, verbose);
|
|
1124
1169
|
if (fresh) {
|
|
1125
1170
|
activeCreds = fresh;
|
|
1126
|
-
upstream = await
|
|
1171
|
+
upstream = await timedFetch(target, {
|
|
1127
1172
|
method: 'POST',
|
|
1128
1173
|
headers: buildCodexHeaders(activeCreds),
|
|
1129
1174
|
body: JSON.stringify(scrubbed),
|
|
@@ -1175,7 +1220,7 @@ midstream) {
|
|
|
1175
1220
|
// (the decline was already recorded above, for both exits)
|
|
1176
1221
|
return false;
|
|
1177
1222
|
}
|
|
1178
|
-
res.writeHead(upstream.status, { 'Content-Type': 'application/json', ...securityHeaders });
|
|
1223
|
+
res.writeHead(upstream.status, { 'Content-Type': 'application/json', ...securityHeaders, ...splitHeaders() });
|
|
1179
1224
|
res.end(errBody('Upstream Codex backend error', { status: upstream.status, account: creds.alias }));
|
|
1180
1225
|
report(upstream.status, null, clientWantsStream, model);
|
|
1181
1226
|
return true;
|
|
@@ -1228,6 +1273,7 @@ midstream) {
|
|
|
1228
1273
|
'Connection': 'keep-alive',
|
|
1229
1274
|
'Access-Control-Allow-Origin': corsOrigin,
|
|
1230
1275
|
...securityHeaders,
|
|
1276
|
+
...splitHeaders(),
|
|
1231
1277
|
});
|
|
1232
1278
|
}
|
|
1233
1279
|
let buffered = '';
|
|
@@ -1311,6 +1357,7 @@ midstream) {
|
|
|
1311
1357
|
'Content-Type': 'application/json',
|
|
1312
1358
|
'Access-Control-Allow-Origin': corsOrigin,
|
|
1313
1359
|
...securityHeaders,
|
|
1360
|
+
...splitHeaders(),
|
|
1314
1361
|
});
|
|
1315
1362
|
antAssembler.push(antTranslator.end());
|
|
1316
1363
|
finished = true;
|
|
@@ -1326,6 +1373,7 @@ midstream) {
|
|
|
1326
1373
|
'Content-Type': 'application/json',
|
|
1327
1374
|
'Access-Control-Allow-Origin': corsOrigin,
|
|
1328
1375
|
...securityHeaders,
|
|
1376
|
+
...splitHeaders(),
|
|
1329
1377
|
});
|
|
1330
1378
|
finished = true;
|
|
1331
1379
|
res.end(JSON.stringify(translator.complete()));
|
package/dist/metrics.js
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { billingBucketFromClaim } from './analytics.js';
|
|
2
|
+
import { TIMING_METRIC_FAMILIES } from './timing.js';
|
|
2
3
|
const escapeLabel = (v) => v.replace(/\\/g, '\\\\').replace(/"/g, '\\"').replace(/\n/g, '\\n');
|
|
3
4
|
const labels = (kv) => {
|
|
4
5
|
const parts = Object.entries(kv).map(([k, v]) => `${k}="${escapeLabel(v)}"`);
|
|
@@ -47,6 +48,16 @@ export function renderPrometheus(input) {
|
|
|
47
48
|
const win = { window_minutes: String(w.minutes) };
|
|
48
49
|
metric('dario_window_requests', 'Requests in the rolling window.', [[win, w.requests]]);
|
|
49
50
|
metric('dario_window_avg_latency_ms', 'Mean request latency in the rolling window, ms.', [[win, w.avgLatencyMs]]);
|
|
51
|
+
// The split behind the mean (src/timing.ts). Absent when no row in the
|
|
52
|
+
// window carried one — an older proxy's rows, or nothing served yet.
|
|
53
|
+
const wt = w.timing;
|
|
54
|
+
if (wt && wt.samples > 0) {
|
|
55
|
+
metric('dario_window_avg_upstream_ttfb_ms', 'Mean upstream time to first byte in the rolling window, ms.', [[win, wt.avgUpstreamTtfbMs]]);
|
|
56
|
+
metric('dario_window_avg_upstream_latency_ms', 'Mean upstream time (first outbound byte to body consumed) in the rolling window, ms.', [[win, wt.avgUpstreamMs]]);
|
|
57
|
+
metric('dario_window_avg_overhead_ms', 'Mean time dario itself spent per request in the rolling window, ms.', [[win, wt.avgOverheadMs]]);
|
|
58
|
+
metric('dario_window_avg_queue_wait_ms', 'Mean wait for a concurrency slot in the rolling window, ms.', [[win, wt.avgQueueMs]]);
|
|
59
|
+
metric('dario_window_avg_pacing_wait_ms', 'Mean rate-governor sleep in the rolling window, ms.', [[win, wt.avgPacingMs]]);
|
|
60
|
+
}
|
|
50
61
|
metric('dario_window_error_rate', 'Share of requests that failed in the rolling window, 0..1.', [[win, w.errorRate]]);
|
|
51
62
|
metric('dario_window_cached_prompt_percent', 'Share of prompt tokens served from cache in the rolling window, 0..100.', [[win, w.cachedPromptPercent]]);
|
|
52
63
|
metric('dario_window_estimated_cost_usd', 'API-equivalent cost of the rolling window, USD.', [[win, w.estimatedCost]]);
|
|
@@ -91,6 +102,21 @@ export function renderPrometheus(input) {
|
|
|
91
102
|
out.push(`dario_request_latency_ms_sum ${num(lat.reduce((a, b) => a + b, 0))}`);
|
|
92
103
|
out.push(`dario_request_latency_ms_count ${lat.length}`);
|
|
93
104
|
}
|
|
105
|
+
// The same quantiles for each leg of the split, over the recent records
|
|
106
|
+
// that carry one. A row without timing (older proxy, pre-upstream reject)
|
|
107
|
+
// is left out rather than counted as zero.
|
|
108
|
+
const timed = recent.map(r => r.timing).filter((t) => t !== undefined);
|
|
109
|
+
if (timed.length > 0) {
|
|
110
|
+
for (const fam of TIMING_METRIC_FAMILIES) {
|
|
111
|
+
const vals = timed.map(fam.pick).filter(n => Number.isFinite(n)).sort((a, b) => a - b);
|
|
112
|
+
out.push(`# HELP ${fam.name} ${fam.help} Nearest-rank quantiles over the most recent records.`);
|
|
113
|
+
out.push(`# TYPE ${fam.name} summary`);
|
|
114
|
+
for (const q of [0.5, 0.9, 0.99])
|
|
115
|
+
out.push(`${fam.name}{quantile="${q}"} ${num(quantile(vals, q))}`);
|
|
116
|
+
out.push(`${fam.name}_sum ${num(vals.reduce((a, b) => a + b, 0))}`);
|
|
117
|
+
out.push(`${fam.name}_count ${vals.length}`);
|
|
118
|
+
}
|
|
119
|
+
}
|
|
94
120
|
// ---- predictions -------------------------------------------------------
|
|
95
121
|
const p = summary.predictions;
|
|
96
122
|
if (p.estimatedExhaustionMinutes !== null) {
|
package/dist/pacing.d.ts
CHANGED
|
@@ -141,3 +141,32 @@ export declare function resolveSessionStartConfig(explicit?: {
|
|
|
141
141
|
jitterMs?: number;
|
|
142
142
|
stealth?: boolean;
|
|
143
143
|
}, env?: NodeJS.ProcessEnv): SessionStartConfig;
|
|
144
|
+
/**
|
|
145
|
+
* The governor's clocks, one set per seat (v6.9.1).
|
|
146
|
+
*
|
|
147
|
+
* The floors above model ONE account's cadence — that is what the provider
|
|
148
|
+
* observes. Through v6.9.0 the proxy kept a single `lastRequestTime` for the
|
|
149
|
+
* whole process, so a pool paced every seat against every other: a three-seat
|
|
150
|
+
* pool could put at most one request per floor on the wire, and a request to
|
|
151
|
+
* an idle seat waited for a stranger's request on a busy one (dario#1244's
|
|
152
|
+
* family — the pool running behind one seat's limits). Keying the clocks by
|
|
153
|
+
* seat keeps each account's observed rhythm exactly as before while the pool
|
|
154
|
+
* as a whole moves at pool speed. API-key mode has one seat and one clock.
|
|
155
|
+
*/
|
|
156
|
+
export interface SeatPacingState {
|
|
157
|
+
/** When the seat's last request STARTED (feeds the inter-request floor). */
|
|
158
|
+
lastRequestTime: number;
|
|
159
|
+
/** When the seat's last 2xx response COMPLETED (feeds think-time). */
|
|
160
|
+
lastResponseTime: number;
|
|
161
|
+
/** Output tokens of that response (feeds think-time's per-token term). */
|
|
162
|
+
lastResponseTokens: number;
|
|
163
|
+
}
|
|
164
|
+
export declare class PacingRegistry {
|
|
165
|
+
private readonly seats;
|
|
166
|
+
/** The seat's clocks, created at zero on first sight (a fresh seat is never paced). */
|
|
167
|
+
seat(key: string): SeatPacingState;
|
|
168
|
+
/** Stamp a completed 2xx response on the seat, for the next request's think-time. */
|
|
169
|
+
noteResponse(key: string, at: number, outputTokens: number): void;
|
|
170
|
+
/** Seats the registry has seen, for tests and status. */
|
|
171
|
+
keys(): string[];
|
|
172
|
+
}
|
package/dist/pacing.js
CHANGED
|
@@ -140,3 +140,25 @@ export function resolveSessionStartConfig(explicit = {}, env = process.env) {
|
|
|
140
140
|
const jitter = pickNonNegativeInt(explicit.jitterMs, env.DARIO_SESSION_START_JITTER_MS) ?? (stealth ? 3000 : 0);
|
|
141
141
|
return { minMs: min, jitterMs: jitter };
|
|
142
142
|
}
|
|
143
|
+
export class PacingRegistry {
|
|
144
|
+
seats = new Map();
|
|
145
|
+
/** The seat's clocks, created at zero on first sight (a fresh seat is never paced). */
|
|
146
|
+
seat(key) {
|
|
147
|
+
let s = this.seats.get(key);
|
|
148
|
+
if (!s) {
|
|
149
|
+
s = { lastRequestTime: 0, lastResponseTime: 0, lastResponseTokens: 0 };
|
|
150
|
+
this.seats.set(key, s);
|
|
151
|
+
}
|
|
152
|
+
return s;
|
|
153
|
+
}
|
|
154
|
+
/** Stamp a completed 2xx response on the seat, for the next request's think-time. */
|
|
155
|
+
noteResponse(key, at, outputTokens) {
|
|
156
|
+
const s = this.seat(key);
|
|
157
|
+
s.lastResponseTime = at;
|
|
158
|
+
s.lastResponseTokens = outputTokens;
|
|
159
|
+
}
|
|
160
|
+
/** Seats the registry has seen, for tests and status. */
|
|
161
|
+
keys() {
|
|
162
|
+
return [...this.seats.keys()];
|
|
163
|
+
}
|
|
164
|
+
}
|
package/dist/proxy.d.ts
CHANGED
|
@@ -599,6 +599,13 @@ export interface ProxyLogEntry {
|
|
|
599
599
|
model?: string;
|
|
600
600
|
status?: number;
|
|
601
601
|
latency_ms?: number;
|
|
602
|
+
/** The latency split (src/timing.ts): where `latency_ms` and the time around it went. */
|
|
603
|
+
queue_ms?: number;
|
|
604
|
+
pacing_ms?: number;
|
|
605
|
+
upstream_ttfb_ms?: number;
|
|
606
|
+
upstream_ms?: number;
|
|
607
|
+
total_ms?: number;
|
|
608
|
+
overhead_ms?: number;
|
|
602
609
|
in_tokens?: number;
|
|
603
610
|
out_tokens?: number;
|
|
604
611
|
cache_read?: number;
|
package/dist/proxy.js
CHANGED
|
@@ -11,6 +11,7 @@ import { getServingProbe } from './serving-probe.js';
|
|
|
11
11
|
import { darioVersion } from './version.js';
|
|
12
12
|
import { buildCCRequest, applyCcPromptCaching, isGenuineCCClient, parseEffortSuffix, reverseMapResponse, createStreamingReverseMapper, orderHeadersForOutbound, overlayTemplateHeaderValues, forwardClientCCIdentityHeaders, isMcpToolName, CC_TEMPLATE, effectiveCacheControl, withForced1hBeta } from './cc-template.js';
|
|
13
13
|
import { stampCch, hasCchSeed } from './cch.js';
|
|
14
|
+
import { foldTiming, timingHeaders, timingLogFields } from './timing.js';
|
|
14
15
|
import { describeTemplate, detectDrift, checkCCCompat, probeInstalledCCVersion } from './live-fingerprint.js';
|
|
15
16
|
import { AccountPool, computeStickyKey, parseRateLimits, modelFamily, isInAuthCooldown, authCooldownMs, accountIneligibility, reportedAccountStatus, reconcilePoolAccounts, resolvePoolStrategy, resolvePoolHeadroomFloor, DEFAULT_POOL_HEADROOM_FLOOR, utilFreshness, rateLimitWindow, accountAction, accountPeers, distinctAccounts, describeRejection, maskEmail, isAccountEligible } from './pool.js';
|
|
16
17
|
import { backfillIdentity } from './accounts.js';
|
|
@@ -1895,15 +1896,13 @@ export async function startProxy(opts = {}) {
|
|
|
1895
1896
|
// 500ms floor keeps the default behavior identical to v3.23; `--pace-min`
|
|
1896
1897
|
// and `--pace-jitter` let callers tune the distribution. Pure calc lives
|
|
1897
1898
|
// in src/pacing.ts so the edge cases are unit-tested without timers.
|
|
1898
|
-
const { computePacingDelay, resolvePacingConfig, computeThinkTimeDelay, resolveThinkTimeConfig, computeSessionStartDelay, resolveSessionStartConfig, } = await import('./pacing.js');
|
|
1899
|
-
|
|
1900
|
-
//
|
|
1901
|
-
// how many output tokens
|
|
1902
|
-
//
|
|
1903
|
-
//
|
|
1904
|
-
|
|
1905
|
-
let lastResponseTime = 0;
|
|
1906
|
-
let lastResponseTokens = 0;
|
|
1899
|
+
const { computePacingDelay, resolvePacingConfig, computeThinkTimeDelay, resolveThinkTimeConfig, computeSessionStartDelay, resolveSessionStartConfig, PacingRegistry, } = await import('./pacing.js');
|
|
1900
|
+
// The governor's clocks, PER SEAT (v6.9.1, src/pacing.ts PacingRegistry):
|
|
1901
|
+
// when a seat's last request started (the inter-request floor) and when its
|
|
1902
|
+
// last 2xx response completed with how many output tokens (think-time).
|
|
1903
|
+
// One clock for the whole proxy used to pace every seat against every
|
|
1904
|
+
// other, so a pool moved at one seat's speed. API-key mode is one seat.
|
|
1905
|
+
const pacingClocks = new PacingRegistry();
|
|
1907
1906
|
// --stealth toggles the behavioral-stealth preset across all three
|
|
1908
1907
|
// pacing layers (pace, think-time, session-start). When on, each
|
|
1909
1908
|
// resolver's zero-default flips to its stealth preset; explicit flags
|
|
@@ -1931,7 +1930,7 @@ export async function startProxy(opts = {}) {
|
|
|
1931
1930
|
if (verbose) {
|
|
1932
1931
|
if (stealth)
|
|
1933
1932
|
console.log('[dario] stealth: behavioral-stealth preset active (pace+think+session-start defaults non-zero)');
|
|
1934
|
-
console.log(`[dario] pacing: min=${pacingCfg.minGapMs}ms jitter=${pacingCfg.jitterMs}ms`);
|
|
1933
|
+
console.log(`[dario] pacing: min=${pacingCfg.minGapMs}ms jitter=${pacingCfg.jitterMs}ms (per seat)`);
|
|
1935
1934
|
if (thinkTimeEnabled) {
|
|
1936
1935
|
console.log(`[dario] think-time: base=${thinkTimeCfg.baseMs}ms perToken=${thinkTimeCfg.perTokenMs}ms jitter=${thinkTimeCfg.jitterMs}ms max=${thinkTimeCfg.maxMs}ms`);
|
|
1937
1936
|
}
|
|
@@ -2188,7 +2187,9 @@ export async function startProxy(opts = {}) {
|
|
|
2188
2187
|
* a silently swapped model family is precisely the surprise this project
|
|
2189
2188
|
* exists to avoid.
|
|
2190
2189
|
*/
|
|
2191
|
-
const tryCodexPoolFallback = async (req, res, body, fallbackModels, shape, why, attempted
|
|
2190
|
+
const tryCodexPoolFallback = async (req, res, body, fallbackModels, shape, why, attempted,
|
|
2191
|
+
/** The request's queue wait and arrival, so the codex response carries the timing headers too. */
|
|
2192
|
+
timing) => {
|
|
2192
2193
|
if (fallbackModels.length === 0)
|
|
2193
2194
|
return false;
|
|
2194
2195
|
// Never a second codex attempt in the same request, and never one while it
|
|
@@ -2280,7 +2281,7 @@ export async function startProxy(opts = {}) {
|
|
|
2280
2281
|
// `--pool-fallback=gpt-5.6-terra:high` is choosing the effort the
|
|
2281
2282
|
// failover runs at, so the entry's own suffix reaches the request rather
|
|
2282
2283
|
// than the failover quietly running at the backend default.
|
|
2283
|
-
effortForCodex(fallbackPick.effort));
|
|
2284
|
+
effortForCodex(fallbackPick.effort), undefined, timing);
|
|
2284
2285
|
if (served || !peer)
|
|
2285
2286
|
break;
|
|
2286
2287
|
console.log(`[dario] codex seat ${seat.alias} declined — retrying this fallback on ${peer.alias}`);
|
|
@@ -2316,8 +2317,8 @@ export async function startProxy(opts = {}) {
|
|
|
2316
2317
|
* bug structurally impossible to reintroduce: a future third call site gets
|
|
2317
2318
|
* this by construction, not by remembering to copy six lines correctly.
|
|
2318
2319
|
*/
|
|
2319
|
-
const attemptPoolFallbackOn429 = async (req, res, body, isOpenAI, attempted) => {
|
|
2320
|
-
if (await tryCodexPoolFallback(req, res, body, selectPoolFallbackForBody(body), isOpenAI ? 'openai' : 'anthropic', 'pool exhausted mid-flight (429, no peer)', attempted)) {
|
|
2320
|
+
const attemptPoolFallbackOn429 = async (req, res, body, isOpenAI, attempted, timing) => {
|
|
2321
|
+
if (await tryCodexPoolFallback(req, res, body, selectPoolFallbackForBody(body), isOpenAI ? 'openai' : 'anthropic', 'pool exhausted mid-flight (429, no peer)', attempted, timing)) {
|
|
2321
2322
|
return true;
|
|
2322
2323
|
}
|
|
2323
2324
|
const fallbackModel = selectPoolFallbackForBody(body)[0] ?? null;
|
|
@@ -2333,6 +2334,9 @@ export async function startProxy(opts = {}) {
|
|
|
2333
2334
|
return false;
|
|
2334
2335
|
};
|
|
2335
2336
|
const server = createServer(async (req, res) => {
|
|
2337
|
+
// The request's first stamp (src/timing.ts): every split below is
|
|
2338
|
+
// measured from here, before any parsing, auth or queueing.
|
|
2339
|
+
const arrivedAt = Date.now();
|
|
2336
2340
|
if (req.method === 'OPTIONS') {
|
|
2337
2341
|
res.writeHead(204, CORS_HEADERS);
|
|
2338
2342
|
res.end();
|
|
@@ -2971,6 +2975,10 @@ export async function startProxy(opts = {}) {
|
|
|
2971
2975
|
// longer uses once its upstream is dead (a one-slot proxy would otherwise
|
|
2972
2976
|
// wait on itself until the queue timeout).
|
|
2973
2977
|
let queueSlotHeld = false;
|
|
2978
|
+
// The deliberate waits, reported on their own in the timing split so
|
|
2979
|
+
// "overhead" never has to be guessed at (src/timing.ts).
|
|
2980
|
+
let queueMs = 0;
|
|
2981
|
+
let pacingMs = 0;
|
|
2974
2982
|
const releaseQueueSlot = () => {
|
|
2975
2983
|
if (!queueSlotHeld)
|
|
2976
2984
|
return;
|
|
@@ -2978,7 +2986,9 @@ export async function startProxy(opts = {}) {
|
|
|
2978
2986
|
queue.release(consumerFromHeaders);
|
|
2979
2987
|
};
|
|
2980
2988
|
try {
|
|
2989
|
+
const queueEnteredAt = Date.now();
|
|
2981
2990
|
await queue.acquire(consumerFromHeaders);
|
|
2991
|
+
queueMs = Date.now() - queueEnteredAt;
|
|
2982
2992
|
queueSlotHeld = true;
|
|
2983
2993
|
}
|
|
2984
2994
|
catch (err) {
|
|
@@ -3845,6 +3855,7 @@ export async function startProxy(opts = {}) {
|
|
|
3845
3855
|
claim: CODEX_CLAIM, util5h: 0, util7d: 0, overageUtil: 0,
|
|
3846
3856
|
latencyMs: o.latencyMs, status: o.status, isStream: o.stream, isOpenAI,
|
|
3847
3857
|
continuation: continuationOf(codexGuard, requestDepth),
|
|
3858
|
+
timing: { queueMs, pacingMs: 0, upstreamTtfbMs: o.upstreamTtfbMs, upstreamMs: o.upstreamMs, totalMs: Math.max(0, Date.now() - arrivedAt) },
|
|
3848
3859
|
});
|
|
3849
3860
|
writeLogLine(logFileStream, {
|
|
3850
3861
|
ts: new Date().toISOString(), req: codexReq,
|
|
@@ -3919,14 +3930,14 @@ export async function startProxy(opts = {}) {
|
|
|
3919
3930
|
// outage this change exists to remove, surviving on the one shape Codex
|
|
3920
3931
|
// CLI actually speaks (caught in review of #1288).
|
|
3921
3932
|
if (isResponses && responsesBodyRaw) {
|
|
3922
|
-
served = await forwardResponsesToCodex(rawRes, { ...responsesBodyRaw, model: rawModel }, codexSeat, corsOrigin, SECURITY_HEADERS, upstreamTimeoutMs, verbose, codexFetch, codexOnDone, codexOnDecline, canDefer || codexPeer !== null);
|
|
3933
|
+
served = await forwardResponsesToCodex(rawRes, { ...responsesBodyRaw, model: rawModel }, codexSeat, corsOrigin, SECURITY_HEADERS, upstreamTimeoutMs, verbose, codexFetch, codexOnDone, codexOnDecline, canDefer || codexPeer !== null, { queueMs, arrivedAt });
|
|
3923
3934
|
}
|
|
3924
3935
|
else {
|
|
3925
3936
|
served = await forwardToCodex(req, res, body, codexSeat, corsOrigin, SECURITY_HEADERS, upstreamTimeoutMs, verbose, isOpenAI ? 'openai' : 'anthropic', codexFetch, canDefer || codexPeer !== null, codexOnDone, codexOnDecline,
|
|
3926
3937
|
// dario#1260 — the effort named by the model-name suffix stripped
|
|
3927
3938
|
// above. Undefined for every request that did not name one, which
|
|
3928
3939
|
// leaves the outbound body exactly as it was.
|
|
3929
|
-
effortForCodex(requestEffort), codexGuard);
|
|
3940
|
+
effortForCodex(requestEffort), codexGuard, { queueMs, arrivedAt });
|
|
3930
3941
|
}
|
|
3931
3942
|
if (served || !codexPeer)
|
|
3932
3943
|
break;
|
|
@@ -4056,7 +4067,7 @@ export async function startProxy(opts = {}) {
|
|
|
4056
4067
|
if (!upstreamApiKey && !poolAccount) {
|
|
4057
4068
|
attemptedProviders.add('claude');
|
|
4058
4069
|
}
|
|
4059
|
-
if (!upstreamApiKey && !poolAccount && !pinnedAccount && await tryCodexPoolFallback(req, res, body, selectPoolFallbackForBody(body), isOpenAI ? 'openai' : 'anthropic', 'pool exhausted', attemptedProviders)) {
|
|
4070
|
+
if (!upstreamApiKey && !poolAccount && !pinnedAccount && await tryCodexPoolFallback(req, res, body, selectPoolFallbackForBody(body), isOpenAI ? 'openai' : 'anthropic', 'pool exhausted', attemptedProviders, { queueMs, arrivedAt })) {
|
|
4060
4071
|
return;
|
|
4061
4072
|
}
|
|
4062
4073
|
// `isOpenAI` is REQUIRED here and was not, before v6.0.0 — the selector's
|
|
@@ -4493,19 +4504,25 @@ export async function startProxy(opts = {}) {
|
|
|
4493
4504
|
// Opt-in via --session-start-* flags.
|
|
4494
4505
|
// We take the max because each layer enforces an independent floor
|
|
4495
4506
|
// — waiting longer satisfies all of them, so we never need to sum.
|
|
4507
|
+
// The clocks are the selected seat's. A mid-flight failover to a peer
|
|
4508
|
+
// (below) does not re-pace on the peer: the retry is the rare path, and
|
|
4509
|
+
// holding it for a floor would stack a second wait on a request that
|
|
4510
|
+
// has already been refused once.
|
|
4511
|
+
const seatClock = pacingClocks.seat(poolAccount?.alias ?? ACCOUNT_KEY_APIKEY);
|
|
4496
4512
|
const nowForPacing = Date.now();
|
|
4497
|
-
const pacingDelay = computePacingDelay(nowForPacing, lastRequestTime, pacingCfg);
|
|
4513
|
+
const pacingDelay = computePacingDelay(nowForPacing, seatClock.lastRequestTime, pacingCfg);
|
|
4498
4514
|
const thinkDelay = thinkTimeEnabled
|
|
4499
|
-
? computeThinkTimeDelay(nowForPacing, lastResponseTime, lastResponseTokens, thinkTimeCfg)
|
|
4515
|
+
? computeThinkTimeDelay(nowForPacing, seatClock.lastResponseTime, seatClock.lastResponseTokens, thinkTimeCfg)
|
|
4500
4516
|
: 0;
|
|
4501
|
-
const sessionStartDelay = (sessionStartEnabled && lastResponseTime === 0 && lastRequestTime === 0)
|
|
4517
|
+
const sessionStartDelay = (sessionStartEnabled && seatClock.lastResponseTime === 0 && seatClock.lastRequestTime === 0)
|
|
4502
4518
|
? computeSessionStartDelay(sessionStartCfg)
|
|
4503
4519
|
: 0;
|
|
4504
4520
|
const totalDelay = Math.max(pacingDelay, thinkDelay, sessionStartDelay);
|
|
4505
4521
|
if (totalDelay > 0) {
|
|
4522
|
+
pacingMs += totalDelay;
|
|
4506
4523
|
await new Promise(r => setTimeout(r, totalDelay));
|
|
4507
4524
|
}
|
|
4508
|
-
lastRequestTime = Date.now();
|
|
4525
|
+
seatClock.lastRequestTime = Date.now();
|
|
4509
4526
|
// Session ID: resolved through the rotation registry keyed by the selected
|
|
4510
4527
|
// account (src/session-rotation.ts), applying the configured idle / jitter
|
|
4511
4528
|
// / max-age / per-client policy. The template-build path resolves it
|
|
@@ -4583,6 +4600,13 @@ export async function startProxy(opts = {}) {
|
|
|
4583
4600
|
};
|
|
4584
4601
|
req.on('close', onClientClose);
|
|
4585
4602
|
const startTime = Date.now();
|
|
4603
|
+
// Upstream stamps for the timing split (src/timing.ts). The fetch stamp is
|
|
4604
|
+
// set once, on the first attempt: a failover's earlier tries are provider
|
|
4605
|
+
// time too, and the headers stamp is the attempt that was served.
|
|
4606
|
+
let fetchStartedAt;
|
|
4607
|
+
let upstreamHeadersAt;
|
|
4608
|
+
let upstreamDoneAt;
|
|
4609
|
+
const timingNow = () => foldTiming({ arrivedAt, queueMs, pacingMs, fetchStartedAt, upstreamHeadersAt, upstreamDoneAt, endedAt: Date.now() });
|
|
4586
4610
|
// Tracks which accounts we've already tried this request — used by the
|
|
4587
4611
|
// inside-request 429 failover loop to avoid re-hitting exhausted accounts.
|
|
4588
4612
|
const triedAliases = new Set();
|
|
@@ -4611,12 +4635,14 @@ export async function startProxy(opts = {}) {
|
|
|
4611
4635
|
// Skipped in passthrough mode — passthrough means "don't shape the
|
|
4612
4636
|
// request to look like CC," and reordering is a form of shaping.
|
|
4613
4637
|
const outboundHeaders = passthrough ? headers : orderHeadersForOutbound(headers);
|
|
4638
|
+
fetchStartedAt ??= Date.now();
|
|
4614
4639
|
upstream = await upstreamFetch(targetBase, {
|
|
4615
4640
|
method: req.method ?? 'POST',
|
|
4616
4641
|
headers: outboundHeaders,
|
|
4617
4642
|
body: finalBody ? new Uint8Array(finalBody) : undefined,
|
|
4618
4643
|
signal: upstreamAbort.signal,
|
|
4619
4644
|
});
|
|
4645
|
+
upstreamHeadersAt = Date.now();
|
|
4620
4646
|
// Pool mode: capture rate-limit snapshot from the response. parseRateLimits
|
|
4621
4647
|
// returns status='rejected' on 429, which makes the next `select()` call
|
|
4622
4648
|
// route traffic away from this account until it resets.
|
|
@@ -4940,7 +4966,7 @@ export async function startProxy(opts = {}) {
|
|
|
4940
4966
|
// sent one, so the following request does not re-walk the chain.
|
|
4941
4967
|
attemptedProviders.add('claude');
|
|
4942
4968
|
providerCooldowns.note('claude', parseRetryAfterMs(upstream.headers.get('retry-after')));
|
|
4943
|
-
if (!pinnedAccount && await attemptPoolFallbackOn429(req, res, body, isOpenAI, attemptedProviders)) {
|
|
4969
|
+
if (!pinnedAccount && await attemptPoolFallbackOn429(req, res, body, isOpenAI, attemptedProviders, { queueMs, arrivedAt })) {
|
|
4944
4970
|
return;
|
|
4945
4971
|
}
|
|
4946
4972
|
if (allProvidersCooled(['codex', 'claude'], providerCooldowns)) {
|
|
@@ -4973,7 +4999,7 @@ export async function startProxy(opts = {}) {
|
|
|
4973
4999
|
model: requestModel,
|
|
4974
5000
|
inputTokens: 0, outputTokens: 0, cacheReadTokens: 0, cacheCreateTokens: 0, thinkingTokens: 0,
|
|
4975
5001
|
claim: rl.claim, util5h: rl.util5h, util7d: rl.util7d, overageUtil: rl.overageUtil,
|
|
4976
|
-
latencyMs: Date.now() - startTime, status: 429, isStream: false, isOpenAI,
|
|
5002
|
+
latencyMs: Date.now() - startTime, status: 429, isStream: false, isOpenAI, timing: timingNow(),
|
|
4977
5003
|
});
|
|
4978
5004
|
}
|
|
4979
5005
|
res.writeHead(429, responseHeaders);
|
|
@@ -5068,7 +5094,7 @@ export async function startProxy(opts = {}) {
|
|
|
5068
5094
|
// Same bookkeeping as the other mid-flight site — see there.
|
|
5069
5095
|
attemptedProviders.add('claude');
|
|
5070
5096
|
providerCooldowns.note('claude', parseRetryAfterMs(upstream.headers.get('retry-after')));
|
|
5071
|
-
if (!pinnedAccount && await attemptPoolFallbackOn429(req, res, body, isOpenAI, attemptedProviders)) {
|
|
5097
|
+
if (!pinnedAccount && await attemptPoolFallbackOn429(req, res, body, isOpenAI, attemptedProviders, { queueMs, arrivedAt })) {
|
|
5072
5098
|
return;
|
|
5073
5099
|
}
|
|
5074
5100
|
if (allProvidersCooled(['codex', 'claude'], providerCooldowns)) {
|
|
@@ -5097,7 +5123,7 @@ export async function startProxy(opts = {}) {
|
|
|
5097
5123
|
model: requestModel,
|
|
5098
5124
|
inputTokens: 0, outputTokens: 0, cacheReadTokens: 0, cacheCreateTokens: 0, thinkingTokens: 0,
|
|
5099
5125
|
claim: rl.claim, util5h: rl.util5h, util7d: rl.util7d, overageUtil: rl.overageUtil,
|
|
5100
|
-
latencyMs: Date.now() - startTime, status: 429, isStream: false, isOpenAI,
|
|
5126
|
+
latencyMs: Date.now() - startTime, status: 429, isStream: false, isOpenAI, timing: timingNow(),
|
|
5101
5127
|
});
|
|
5102
5128
|
}
|
|
5103
5129
|
res.writeHead(429, responseHeaders);
|
|
@@ -5172,6 +5198,11 @@ export async function startProxy(opts = {}) {
|
|
|
5172
5198
|
console.log(`[dario] #${requestCount} billing: headers absent (status=${upstream.status})`);
|
|
5173
5199
|
}
|
|
5174
5200
|
}
|
|
5201
|
+
Object.assign(responseHeaders, timingHeaders({
|
|
5202
|
+
queueMs, pacingMs, arrivedAt,
|
|
5203
|
+
fetchStartedAt: fetchStartedAt ?? Date.now(),
|
|
5204
|
+
upstreamTtfbMs: (upstreamHeadersAt ?? 0) - (fetchStartedAt ?? 0),
|
|
5205
|
+
}));
|
|
5175
5206
|
res.writeHead(upstream.status, responseHeaders);
|
|
5176
5207
|
if (isStream && upstream.body) {
|
|
5177
5208
|
// Analytics accumulators for streaming responses — filled by parsing
|
|
@@ -5252,8 +5283,10 @@ export async function startProxy(opts = {}) {
|
|
|
5252
5283
|
const MAX_LINE_LENGTH = 1_000_000; // 1MB max per SSE line
|
|
5253
5284
|
while (true) {
|
|
5254
5285
|
const { done, value } = await reader.read();
|
|
5255
|
-
if (done)
|
|
5286
|
+
if (done) {
|
|
5287
|
+
upstreamDoneAt = Date.now();
|
|
5256
5288
|
break;
|
|
5289
|
+
}
|
|
5257
5290
|
// Parse SSE events for analytics regardless of routing branch
|
|
5258
5291
|
if (analyticsDecoder && value) {
|
|
5259
5292
|
analyticsBuffer += analyticsDecoder.decode(value, { stream: true });
|
|
@@ -5371,8 +5404,7 @@ export async function startProxy(opts = {}) {
|
|
|
5371
5404
|
// would read, and using their (often zero) output_tokens would
|
|
5372
5405
|
// pin think time to baseMs+jitter on the next request needlessly.
|
|
5373
5406
|
if (upstream.status >= 200 && upstream.status < 300) {
|
|
5374
|
-
|
|
5375
|
-
lastResponseTokens = streamOutputTokens;
|
|
5407
|
+
pacingClocks.noteResponse(poolAccount?.alias ?? ACCOUNT_KEY_APIKEY, Date.now(), streamOutputTokens);
|
|
5376
5408
|
}
|
|
5377
5409
|
{
|
|
5378
5410
|
const rl = poolAccount?.rateLimit ?? parseRateLimits(upstream.headers);
|
|
@@ -5385,7 +5417,7 @@ export async function startProxy(opts = {}) {
|
|
|
5385
5417
|
cacheReadTokens: streamCacheReadTokens, cacheCreateTokens: streamCacheCreateTokens,
|
|
5386
5418
|
thinkingTokens: Math.round(streamThinkingChars / 4),
|
|
5387
5419
|
claim: rl.claim, util5h: rl.util5h, util7d: rl.util7d, overageUtil: rl.overageUtil,
|
|
5388
|
-
latencyMs: Date.now() - startTime, status: upstream.status, isStream: true, isOpenAI,
|
|
5420
|
+
latencyMs: Date.now() - startTime, status: upstream.status, isStream: true, isOpenAI, timing: timingNow(),
|
|
5389
5421
|
continuation: continuationOf(guard, requestDepth),
|
|
5390
5422
|
});
|
|
5391
5423
|
}
|
|
@@ -5393,7 +5425,7 @@ export async function startProxy(opts = {}) {
|
|
|
5393
5425
|
ts: new Date().toISOString(), req: requestCount,
|
|
5394
5426
|
method: req.method ?? '', path: urlPath,
|
|
5395
5427
|
model: requestModel || undefined,
|
|
5396
|
-
status: upstream.status, latency_ms: Date.now() - startTime,
|
|
5428
|
+
status: upstream.status, latency_ms: Date.now() - startTime, ...timingLogFields(timingNow()),
|
|
5397
5429
|
in_tokens: streamInputTokens, out_tokens: streamOutputTokens,
|
|
5398
5430
|
cache_read: streamCacheReadTokens, cache_create: streamCacheCreateTokens,
|
|
5399
5431
|
claim: poolAccount?.rateLimit.claim,
|
|
@@ -5415,6 +5447,7 @@ export async function startProxy(opts = {}) {
|
|
|
5415
5447
|
else {
|
|
5416
5448
|
// Buffer and forward
|
|
5417
5449
|
let responseBody = await upstream.text();
|
|
5450
|
+
upstreamDoneAt = Date.now();
|
|
5418
5451
|
// Reverse tool name mapping so client sees original names
|
|
5419
5452
|
if (ccToolMap)
|
|
5420
5453
|
responseBody = reverseMapResponse(responseBody, ccToolMap, reqCtx);
|
|
@@ -5441,8 +5474,7 @@ export async function startProxy(opts = {}) {
|
|
|
5441
5474
|
// tokens when the body wasn't JSON or had no usage block — base +
|
|
5442
5475
|
// jitter still apply but the per-token component is 0.
|
|
5443
5476
|
if (upstream.status >= 200 && upstream.status < 300) {
|
|
5444
|
-
|
|
5445
|
-
lastResponseTokens = bufferedUsage?.outputTokens ?? 0;
|
|
5477
|
+
pacingClocks.noteResponse(poolAccount?.alias ?? ACCOUNT_KEY_APIKEY, Date.now(), bufferedUsage?.outputTokens ?? 0);
|
|
5446
5478
|
}
|
|
5447
5479
|
if (bufferedUsage) {
|
|
5448
5480
|
try {
|
|
@@ -5456,7 +5488,7 @@ export async function startProxy(opts = {}) {
|
|
|
5456
5488
|
cacheReadTokens: bufferedUsage.cacheReadTokens, cacheCreateTokens: bufferedUsage.cacheCreateTokens,
|
|
5457
5489
|
thinkingTokens: bufferedUsage.thinkingTokens,
|
|
5458
5490
|
claim: rl.claim, util5h: rl.util5h, util7d: rl.util7d, overageUtil: rl.overageUtil,
|
|
5459
|
-
latencyMs: Date.now() - startTime, status: upstream.status, isStream: false, isOpenAI,
|
|
5491
|
+
latencyMs: Date.now() - startTime, status: upstream.status, isStream: false, isOpenAI, timing: timingNow(),
|
|
5460
5492
|
});
|
|
5461
5493
|
}
|
|
5462
5494
|
catch { /* don't let analytics errors break responses */ }
|
|
@@ -5465,7 +5497,7 @@ export async function startProxy(opts = {}) {
|
|
|
5465
5497
|
ts: new Date().toISOString(), req: requestCount,
|
|
5466
5498
|
method: req.method ?? '', path: urlPath,
|
|
5467
5499
|
model: bufferedUsage?.model || requestModel || undefined,
|
|
5468
|
-
status: upstream.status, latency_ms: Date.now() - startTime,
|
|
5500
|
+
status: upstream.status, latency_ms: Date.now() - startTime, ...timingLogFields(timingNow()),
|
|
5469
5501
|
in_tokens: bufferedUsage?.inputTokens, out_tokens: bufferedUsage?.outputTokens,
|
|
5470
5502
|
cache_read: bufferedUsage?.cacheReadTokens, cache_create: bufferedUsage?.cacheCreateTokens,
|
|
5471
5503
|
claim: poolAccount?.rateLimit.claim,
|
package/dist/timing.d.ts
ADDED
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Per-request timing split — where a request's wall-clock time went.
|
|
3
|
+
*
|
|
4
|
+
* Until now a request carried one number, `latencyMs`, measured from the
|
|
5
|
+
* moment dario was ready to dispatch to the moment the response ended. It
|
|
6
|
+
* left out the wait for a concurrency slot and the rate governor's sleep,
|
|
7
|
+
* and it folded the provider's time and dario's own work into one figure.
|
|
8
|
+
* A user seeing 6 s per request could not tell whether that was Anthropic,
|
|
9
|
+
* the queue, the 500 ms pacing floor, or the proxy itself (dario#1341 asked
|
|
10
|
+
* for exactly LiteLLM's split: total, provider, overhead, queue, TTFT).
|
|
11
|
+
*
|
|
12
|
+
* Five stamps, all milliseconds, all measured by dario on the same clock:
|
|
13
|
+
*
|
|
14
|
+
* queueMs waited for a `--max-concurrent` slot
|
|
15
|
+
* pacingMs slept in the rate governor (pacing / think-time /
|
|
16
|
+
* session-start floors)
|
|
17
|
+
* upstreamTtfbMs first outbound byte → upstream response headers
|
|
18
|
+
* (the provider's time-to-first-byte, failover attempts
|
|
19
|
+
* included — that is provider time, not dario's)
|
|
20
|
+
* upstreamMs first outbound byte → upstream body fully consumed
|
|
21
|
+
* totalMs request arrived at dario → response ended
|
|
22
|
+
*
|
|
23
|
+
* and one derived figure, `overheadMs = total − upstream − queue − pacing`:
|
|
24
|
+
* the time dario itself spent reading the body, building the template,
|
|
25
|
+
* translating shapes and relaying SSE. Deliberate waits (queue, pacing) are
|
|
26
|
+
* reported on their own so nobody has to guess whether "overhead" includes
|
|
27
|
+
* them; it does not.
|
|
28
|
+
*
|
|
29
|
+
* Pure over its inputs. The proxy stamps the clock; this module only folds.
|
|
30
|
+
*/
|
|
31
|
+
export interface RequestTiming {
|
|
32
|
+
queueMs: number;
|
|
33
|
+
pacingMs: number;
|
|
34
|
+
upstreamTtfbMs: number;
|
|
35
|
+
upstreamMs: number;
|
|
36
|
+
totalMs: number;
|
|
37
|
+
}
|
|
38
|
+
/** Averages over the records in a window that carry a timing split. */
|
|
39
|
+
export interface TimingStats {
|
|
40
|
+
/** Records the averages are over. 0 when no request carried timing (older rows, codex legs without a split). */
|
|
41
|
+
samples: number;
|
|
42
|
+
avgQueueMs: number;
|
|
43
|
+
avgPacingMs: number;
|
|
44
|
+
avgUpstreamTtfbMs: number;
|
|
45
|
+
avgUpstreamMs: number;
|
|
46
|
+
avgOverheadMs: number;
|
|
47
|
+
}
|
|
48
|
+
/** Response headers the client sees before the body, so a curl can read them without /analytics. */
|
|
49
|
+
export declare const TIMING_HEADERS: {
|
|
50
|
+
readonly queue: "x-dario-queue-ms";
|
|
51
|
+
readonly pacing: "x-dario-pacing-ms";
|
|
52
|
+
readonly ttfb: "x-dario-upstream-ttfb-ms";
|
|
53
|
+
/** Arrival → first outbound byte, minus queue and pacing: dario's own pre-upstream work. */
|
|
54
|
+
readonly prep: "x-dario-prep-ms";
|
|
55
|
+
};
|
|
56
|
+
/** dario's own processing time: everything that is neither the provider nor a deliberate wait. Never negative. */
|
|
57
|
+
export declare function overheadMs(t: RequestTiming): number;
|
|
58
|
+
/** Fold a set of stamps into a record-ready split; clocks that never ticked read 0, never NaN or negative. */
|
|
59
|
+
export declare function foldTiming(stamps: {
|
|
60
|
+
arrivedAt: number;
|
|
61
|
+
queueMs: number;
|
|
62
|
+
pacingMs: number;
|
|
63
|
+
/** Absent when nothing went upstream (rejected before dispatch). */
|
|
64
|
+
fetchStartedAt?: number;
|
|
65
|
+
upstreamHeadersAt?: number;
|
|
66
|
+
upstreamDoneAt?: number;
|
|
67
|
+
endedAt: number;
|
|
68
|
+
}): RequestTiming;
|
|
69
|
+
/** The headers known before the body starts. `prepMs` is what remains of arrival→fetch once the waits are taken out. */
|
|
70
|
+
export declare function timingHeaders(t: {
|
|
71
|
+
queueMs: number;
|
|
72
|
+
pacingMs: number;
|
|
73
|
+
upstreamTtfbMs: number;
|
|
74
|
+
arrivedAt: number;
|
|
75
|
+
fetchStartedAt: number;
|
|
76
|
+
}): Record<string, string>;
|
|
77
|
+
/** The split as the request log's snake_case columns. */
|
|
78
|
+
export declare function timingLogFields(t: RequestTiming): {
|
|
79
|
+
queue_ms: number;
|
|
80
|
+
pacing_ms: number;
|
|
81
|
+
upstream_ttfb_ms: number;
|
|
82
|
+
upstream_ms: number;
|
|
83
|
+
total_ms: number;
|
|
84
|
+
overhead_ms: number;
|
|
85
|
+
};
|
|
86
|
+
export declare function timingStats(timings: readonly (RequestTiming | undefined)[]): TimingStats;
|
|
87
|
+
/** The five families `/metrics` exports as summaries, each read off the recent records that carry a split. */
|
|
88
|
+
export declare const TIMING_METRIC_FAMILIES: ReadonlyArray<{
|
|
89
|
+
name: string;
|
|
90
|
+
help: string;
|
|
91
|
+
pick: (t: RequestTiming) => number;
|
|
92
|
+
}>;
|
package/dist/timing.js
ADDED
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Per-request timing split — where a request's wall-clock time went.
|
|
3
|
+
*
|
|
4
|
+
* Until now a request carried one number, `latencyMs`, measured from the
|
|
5
|
+
* moment dario was ready to dispatch to the moment the response ended. It
|
|
6
|
+
* left out the wait for a concurrency slot and the rate governor's sleep,
|
|
7
|
+
* and it folded the provider's time and dario's own work into one figure.
|
|
8
|
+
* A user seeing 6 s per request could not tell whether that was Anthropic,
|
|
9
|
+
* the queue, the 500 ms pacing floor, or the proxy itself (dario#1341 asked
|
|
10
|
+
* for exactly LiteLLM's split: total, provider, overhead, queue, TTFT).
|
|
11
|
+
*
|
|
12
|
+
* Five stamps, all milliseconds, all measured by dario on the same clock:
|
|
13
|
+
*
|
|
14
|
+
* queueMs waited for a `--max-concurrent` slot
|
|
15
|
+
* pacingMs slept in the rate governor (pacing / think-time /
|
|
16
|
+
* session-start floors)
|
|
17
|
+
* upstreamTtfbMs first outbound byte → upstream response headers
|
|
18
|
+
* (the provider's time-to-first-byte, failover attempts
|
|
19
|
+
* included — that is provider time, not dario's)
|
|
20
|
+
* upstreamMs first outbound byte → upstream body fully consumed
|
|
21
|
+
* totalMs request arrived at dario → response ended
|
|
22
|
+
*
|
|
23
|
+
* and one derived figure, `overheadMs = total − upstream − queue − pacing`:
|
|
24
|
+
* the time dario itself spent reading the body, building the template,
|
|
25
|
+
* translating shapes and relaying SSE. Deliberate waits (queue, pacing) are
|
|
26
|
+
* reported on their own so nobody has to guess whether "overhead" includes
|
|
27
|
+
* them; it does not.
|
|
28
|
+
*
|
|
29
|
+
* Pure over its inputs. The proxy stamps the clock; this module only folds.
|
|
30
|
+
*/
|
|
31
|
+
/** Response headers the client sees before the body, so a curl can read them without /analytics. */
|
|
32
|
+
export const TIMING_HEADERS = {
|
|
33
|
+
queue: 'x-dario-queue-ms',
|
|
34
|
+
pacing: 'x-dario-pacing-ms',
|
|
35
|
+
ttfb: 'x-dario-upstream-ttfb-ms',
|
|
36
|
+
/** Arrival → first outbound byte, minus queue and pacing: dario's own pre-upstream work. */
|
|
37
|
+
prep: 'x-dario-prep-ms',
|
|
38
|
+
};
|
|
39
|
+
const nonNeg = (n) => (Number.isFinite(n) && n > 0 ? n : 0);
|
|
40
|
+
/** dario's own processing time: everything that is neither the provider nor a deliberate wait. Never negative. */
|
|
41
|
+
export function overheadMs(t) {
|
|
42
|
+
return Math.max(0, Math.round(t.totalMs - t.upstreamMs - t.queueMs - t.pacingMs));
|
|
43
|
+
}
|
|
44
|
+
/** Fold a set of stamps into a record-ready split; clocks that never ticked read 0, never NaN or negative. */
|
|
45
|
+
export function foldTiming(stamps) {
|
|
46
|
+
const fetchAt = stamps.fetchStartedAt ?? stamps.endedAt;
|
|
47
|
+
const headersAt = stamps.upstreamHeadersAt ?? fetchAt;
|
|
48
|
+
const doneAt = stamps.upstreamDoneAt ?? headersAt;
|
|
49
|
+
return {
|
|
50
|
+
queueMs: Math.round(nonNeg(stamps.queueMs)),
|
|
51
|
+
pacingMs: Math.round(nonNeg(stamps.pacingMs)),
|
|
52
|
+
upstreamTtfbMs: Math.round(nonNeg(headersAt - fetchAt)),
|
|
53
|
+
upstreamMs: Math.round(nonNeg(doneAt - fetchAt)),
|
|
54
|
+
totalMs: Math.round(nonNeg(stamps.endedAt - stamps.arrivedAt)),
|
|
55
|
+
};
|
|
56
|
+
}
|
|
57
|
+
/** The headers known before the body starts. `prepMs` is what remains of arrival→fetch once the waits are taken out. */
|
|
58
|
+
export function timingHeaders(t) {
|
|
59
|
+
const prep = Math.max(0, Math.round(t.fetchStartedAt - t.arrivedAt - t.queueMs - t.pacingMs));
|
|
60
|
+
return {
|
|
61
|
+
[TIMING_HEADERS.queue]: String(Math.round(nonNeg(t.queueMs))),
|
|
62
|
+
[TIMING_HEADERS.pacing]: String(Math.round(nonNeg(t.pacingMs))),
|
|
63
|
+
[TIMING_HEADERS.ttfb]: String(Math.round(nonNeg(t.upstreamTtfbMs))),
|
|
64
|
+
[TIMING_HEADERS.prep]: String(prep),
|
|
65
|
+
};
|
|
66
|
+
}
|
|
67
|
+
/** The split as the request log's snake_case columns. */
|
|
68
|
+
export function timingLogFields(t) {
|
|
69
|
+
return { queue_ms: t.queueMs, pacing_ms: t.pacingMs, upstream_ttfb_ms: t.upstreamTtfbMs, upstream_ms: t.upstreamMs, total_ms: t.totalMs, overhead_ms: overheadMs(t) };
|
|
70
|
+
}
|
|
71
|
+
export function timingStats(timings) {
|
|
72
|
+
const rows = timings.filter((t) => t !== undefined);
|
|
73
|
+
if (rows.length === 0) {
|
|
74
|
+
return { samples: 0, avgQueueMs: 0, avgPacingMs: 0, avgUpstreamTtfbMs: 0, avgUpstreamMs: 0, avgOverheadMs: 0 };
|
|
75
|
+
}
|
|
76
|
+
const avg = (pick) => Math.round(rows.reduce((s, t) => s + pick(t), 0) / rows.length);
|
|
77
|
+
return {
|
|
78
|
+
samples: rows.length,
|
|
79
|
+
avgQueueMs: avg(t => t.queueMs),
|
|
80
|
+
avgPacingMs: avg(t => t.pacingMs),
|
|
81
|
+
avgUpstreamTtfbMs: avg(t => t.upstreamTtfbMs),
|
|
82
|
+
avgUpstreamMs: avg(t => t.upstreamMs),
|
|
83
|
+
avgOverheadMs: avg(overheadMs),
|
|
84
|
+
};
|
|
85
|
+
}
|
|
86
|
+
/** The five families `/metrics` exports as summaries, each read off the recent records that carry a split. */
|
|
87
|
+
export const TIMING_METRIC_FAMILIES = [
|
|
88
|
+
{ name: 'dario_queue_wait_ms', help: 'Time a request waited for a concurrency slot, ms.', pick: t => t.queueMs },
|
|
89
|
+
{ name: 'dario_pacing_wait_ms', help: 'Time a request slept in the rate governor (pacing / think-time / session-start floors), ms.', pick: t => t.pacingMs },
|
|
90
|
+
{ name: 'dario_upstream_ttfb_ms', help: 'First outbound byte to upstream response headers, ms (the provider\'s time to first byte).', pick: t => t.upstreamTtfbMs },
|
|
91
|
+
{ name: 'dario_upstream_latency_ms', help: 'First outbound byte to upstream body fully consumed, ms (the provider\'s time).', pick: t => t.upstreamMs },
|
|
92
|
+
{ name: 'dario_overhead_ms', help: 'Time dario itself spent on a request: total minus upstream, queue and pacing, ms.', pick: overheadMs },
|
|
93
|
+
];
|
|
@@ -22,6 +22,15 @@ interface SummaryShape {
|
|
|
22
22
|
totalThinkingTokens: number;
|
|
23
23
|
estimatedCost: number;
|
|
24
24
|
avgLatencyMs: number;
|
|
25
|
+
/** The latency split (v6.9, src/timing.ts); absent on older proxies. */
|
|
26
|
+
timing?: {
|
|
27
|
+
samples: number;
|
|
28
|
+
avgQueueMs: number;
|
|
29
|
+
avgPacingMs: number;
|
|
30
|
+
avgUpstreamTtfbMs: number;
|
|
31
|
+
avgUpstreamMs: number;
|
|
32
|
+
avgOverheadMs: number;
|
|
33
|
+
};
|
|
25
34
|
subscriptionPercent: number;
|
|
26
35
|
billingBucketBreakdown: Record<string, number>;
|
|
27
36
|
};
|
|
@@ -97,6 +97,13 @@ export const AnalyticsTab = {
|
|
|
97
97
|
counters.push(' ' + renderKvRow('Tokens out', formatNumber(s.window.totalOutputTokens), w - 4));
|
|
98
98
|
counters.push(' ' + renderKvRow('Thinking tokens', formatNumber(s.window.totalThinkingTokens), w - 4));
|
|
99
99
|
counters.push(' ' + renderKvRow('Avg latency', `${Math.round(s.window.avgLatencyMs)}ms`, w - 4));
|
|
100
|
+
// The split behind that number (src/timing.ts), when the proxy is new
|
|
101
|
+
// enough to send it and a request in the window carried it.
|
|
102
|
+
const t = s.window.timing;
|
|
103
|
+
if (t && t.samples > 0) {
|
|
104
|
+
counters.push(' ' + renderKvRow(' upstream TTFB', `${t.avgUpstreamTtfbMs}ms ${dim(`upstream ${t.avgUpstreamMs}ms`)}`, w - 4));
|
|
105
|
+
counters.push(' ' + renderKvRow(' dario overhead', `${t.avgOverheadMs}ms ${dim(`queue ${t.avgQueueMs}ms · pacing ${t.avgPacingMs}ms`)}`, w - 4));
|
|
106
|
+
}
|
|
100
107
|
counters.push(' ' + renderKvRow('Subscription %', `${s.window.subscriptionPercent.toFixed(0)}%`, w - 4));
|
|
101
108
|
// The ledger's number: what everything since the first request would
|
|
102
109
|
// have been billed on the metered API. Lifetime, not the window.
|
package/docs/analytics.md
CHANGED
|
@@ -41,10 +41,30 @@ Names ending in `_total` are counters; everything else is a gauge. Labels are es
|
|
|
41
41
|
| `dario_consumer_requests_total`, `dario_consumer_estimated_cost_usd` | `consumer` (named key or `x-dario-consumer`) | since start |
|
|
42
42
|
| `dario_queue_active`, `_queued`, `_max_concurrent`, `_max_queued`, `_stalled`, `_max_wait_ms`, `_consumers_active` | — | request queue |
|
|
43
43
|
| `dario_request_latency_ms{quantile}` + `_sum`, `_count` | `quantile` ∈ 0.5, 0.9, 0.99 | nearest-rank over the most recent 1,000 records |
|
|
44
|
+
| `dario_queue_wait_ms`, `dario_pacing_wait_ms`, `dario_upstream_ttfb_ms`, `dario_upstream_latency_ms`, `dario_overhead_ms` — each `{quantile}` + `_sum`, `_count` | `quantile` ∈ 0.5, 0.9, 0.99 | the timing split (below), nearest-rank over the recent records that carry one |
|
|
45
|
+
| `dario_window_avg_upstream_ttfb_ms`, `_avg_upstream_latency_ms`, `_avg_overhead_ms`, `_avg_queue_wait_ms`, `_avg_pacing_wait_ms` | `window_minutes` | the split averaged over the rolling window; omitted until a request has carried one |
|
|
44
46
|
| `dario_predicted_exhaustion_minutes` (omitted when unknown), `dario_burn_tokens_per_minute`, `dario_burn_cost_usd_per_minute` | — | window predictions |
|
|
45
47
|
| `dario_ledger_requests_total`, `_api_equivalent_usd`, `_metered_usd`, `_recent_api_equivalent_usd{window}`, `_model_api_equivalent_usd{model,provider}`, `_model_requests_total{model,provider}`, `_consumer_api_equivalent_usd{consumer}` | `window` ∈ today, 7d, 30d | ledger (absent when the ledger is off) |
|
|
46
48
|
|
|
47
|
-
|
|
49
|
+
`dario_request_latency_ms` is the number dario has always kept: dispatch to response end, as the client saw it. Since 6.9 every request also carries the split below, so the one figure can be read as its parts.
|
|
50
|
+
|
|
51
|
+
## The timing split
|
|
52
|
+
|
|
53
|
+
A request's wall-clock time through dario is five stamps (`src/timing.ts`), all in milliseconds, all on dario's clock:
|
|
54
|
+
|
|
55
|
+
| leg | measures | where it shows |
|
|
56
|
+
|---|---|---|
|
|
57
|
+
| `queueMs` | waited for a `--max-concurrent` slot | `x-dario-queue-ms`, `dario_queue_wait_ms`, log `queue_ms` |
|
|
58
|
+
| `pacingMs` | slept in the rate governor (`--pace-min`, think-time, session-start floors) | `x-dario-pacing-ms`, `dario_pacing_wait_ms`, log `pacing_ms` |
|
|
59
|
+
| `upstreamTtfbMs` | first outbound byte → upstream response headers; the provider's time to first byte, failover attempts included | `x-dario-upstream-ttfb-ms`, `dario_upstream_ttfb_ms`, log `upstream_ttfb_ms` |
|
|
60
|
+
| `upstreamMs` | first outbound byte → upstream body fully consumed | `dario_upstream_latency_ms`, log `upstream_ms` |
|
|
61
|
+
| `totalMs` | request arrived at dario → response ended | log `total_ms` |
|
|
62
|
+
|
|
63
|
+
and one derived figure, **overhead** = `total − upstream − queue − pacing`: the time dario itself spent reading the body, building the template, translating shapes and relaying SSE. The two deliberate waits are reported on their own so "overhead" never has to be guessed at; it does not include them. `x-dario-prep-ms` on the response is the pre-upstream part of that overhead (arrival → first outbound byte, minus the waits), the only part known before the body starts.
|
|
64
|
+
|
|
65
|
+
The four `x-dario-*-ms` response headers ride on every served `/v1/messages` and `/v1/chat/completions` response, streamed or not, so a `curl -i` answers "was that Anthropic or dario?" without opening `/analytics`. They are added to the response dario writes to the client and change nothing on the wire to the provider; `--passthrough` stays byte-identical upstream.
|
|
66
|
+
|
|
67
|
+
`GET /analytics` carries the split averaged over the window and since start as `window.timing` / `allTime.timing` (`samples` says how many rows had one), `dario status` prints it under **Avg latency**, and the TUI's Analytics tab shows it beneath the same row. A ChatGPT (codex) leg records its seat's TTFB and total the same way and its response carries the same four headers; the governor never runs for it, so its `pacingMs` is 0.
|
|
48
68
|
|
|
49
69
|
A minimal scrape config:
|
|
50
70
|
|
package/docs/wire-fidelity.md
CHANGED
|
@@ -6,7 +6,7 @@ Between v3.22 and v3.28, dario's Claude backend closed six axes along which a pr
|
|
|
6
6
|
|---|---|---|---|
|
|
7
7
|
| **Request body key order** | v3.22 | Top-level JSON key order of the outbound `/v1/messages` body is captured from CC's wire serialization and replayed byte-for-byte. Schema bumped v2 → v3; stale caches quarantined. | Automatic once a live capture exists. The baked fallback carries a v2.1.112 snapshot. |
|
|
8
8
|
| **Runtime / TLS ClientHello** | v3.23 | Classifies the runtime as `bun-match` / `bun-ja3-unverified` / `bun-bypassed` / `node-only` and surfaces the class + hint in `dario doctor`. Bun yields the BoringSSL ClientHello CC presents; Node yields OpenSSL's (distinct JA3). Being on Bun is necessary but not sufficient — only Bun ≥ v1.3.14 is measured to reproduce CC's JA3, so an older Bun is flagged `bun-ja3-unverified` rather than green (#813). | `--strict-tls` (or `DARIO_STRICT_TLS=1`) refuses to start proxy mode unless `bun-match`. `DARIO_QUIET_TLS=1` silences the startup banner in known-fine environments. |
|
|
9
|
-
| **Inter-request timing** | v3.24 | Replaces the hardcoded 500 ms floor with a configurable floor + uniform jitter. A fixed 500 ms minimum-inter-arrival is an observable edge at scale; jitter dissolves the edge. | `--pace-min=MS`, `--pace-jitter=MS`, or `DARIO_PACE_MIN_MS` / `DARIO_PACE_JITTER_MS`. Legacy `DARIO_MIN_INTERVAL_MS` still honored. |
|
|
9
|
+
| **Inter-request timing** | v3.24 | Replaces the hardcoded 500 ms floor with a configurable floor + uniform jitter. A fixed 500 ms minimum-inter-arrival is an observable edge at scale; jitter dissolves the edge. Since 6.9.1 the floor (and think-time / session-start) is kept **per seat**: the cadence the provider sees per account is unchanged, and a pool no longer paces its seats against each other. | `--pace-min=MS`, `--pace-jitter=MS`, or `DARIO_PACE_MIN_MS` / `DARIO_PACE_JITTER_MS`. Legacy `DARIO_MIN_INTERVAL_MS` still honored. |
|
|
10
10
|
| **Stream-consumption shape** | v3.25 | When a downstream client disconnects mid-stream, CC keeps reading SSE to EOF. Dario now offers the same: drain upstream to completion even when the consumer has left. Default off — don't silently burn tokens. | `--drain-on-close` / `DARIO_DRAIN_ON_CLOSE=1`. Bounded by the existing 5-minute upstream timeout. |
|
|
11
11
|
| **Session-ID lifecycle** | v3.28 | Generalizes the v3.19 hardcoded 15-minute idle rotation into a tunable `SessionRegistry` with jitter, max-age, and per-client bucketing. Fixes a v3.27 body/header rotation race as a side effect. | `--session-idle-rotate=MS` (default 900000), `--session-rotate-jitter=MS`, `--session-max-age=MS`, `--session-per-client`. Env mirrors `DARIO_SESSION_*`. Defaults are bit-identical to v3.27. |
|
|
12
12
|
| **MCP / sub-agent reach** | v3.26 + v3.27 | Not a wire axis — a *surface* axis. CC-aware tools can now address dario directly (sub-agent from inside CC, MCP server for any MCP client), so operators don't have to switch terminals to introspect the proxy. Read-only by design. | `dario subagent install` / `dario mcp`. See [`mcp-server.md`](./mcp-server.md) and [`sub-agent.md`](./sub-agent.md). |
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@askalf/dario",
|
|
3
|
-
"version": "6.
|
|
3
|
+
"version": "6.9.1",
|
|
4
4
|
"description": "Use your Claude and ChatGPT subscriptions in Cursor, Cline, Aider, Claude Code and the Agent SDK — at subscription pricing, not per-token API bills. One local Anthropic + OpenAI-compatible endpoint: either plan answers either wire shape, with automatic failover when one hits its limit.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|