oira666_pi-limits-wait 0.6.5 → 0.6.7

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -12,19 +12,21 @@ pi install npm:oira666_pi-limits-wait
12
12
 
13
13
  Restart Pi or run `/reload` after installing.
14
14
 
15
+ Tested against Pi 1.4.0.
16
+
15
17
  ## What it does
16
18
 
17
- - Targets Pi 0.80.10+ (developed and tested against Pi 0.83.0) and works with model providers that use `ModelRuntime.streamSimple`.
19
+ - Works with model providers that use `ModelRuntime.streamSimple`.
18
20
  - Intercepts the current runtime without registering synthetic providers; every retry and fallback re-enters Pi's normal provider/auth/base URL/configured-header composition, including OAuth refresh checks.
19
- - On Pi 0.86+, stops background cache warming via `cache_warming_decision` while retry interception is enabled, preventing background refreshes from triggering foreground retries or fallbacks. Normal request caching is unchanged. With `PI_LIMITS_WAIT_DISABLE_ALL_WAITING=true`, Pi's cache-warming policy is left untouched. As with any handler for this event, a later extension can override the decision.
21
+ - Stops background cache warming via `cache_warming_decision` while retry interception is enabled, preventing background refreshes from triggering foreground retries or fallbacks. Normal request caching is unchanged. With `PI_LIMITS_WAIT_DISABLE_ALL_WAITING=true`, Pi's cache-warming policy is left untouched. As with any handler for this event, a later extension can override the decision.
20
22
  - Reload-safe ownership ensures `/reload` transfers interception cleanly and aborts waits still owned by the previous extension instance. The interception has one process-global active extension owner, matching the interactive Pi CLI's single active session runtime lifecycle; SDK processes running multiple concurrent Pi sessions are not supported.
21
23
  - On rate-limit errors (`429`, `rate_limit`, `too many requests`, quota/reset messages), waits and retries in a loop.
22
- - Remembers provider retry timing exposed through response callbacks or provider errors (`retry-after`, `retry-after-ms`, `retry in ...`, reset messages). With fallback probing enabled, explicit Retry-After deadlines may intentionally be probed early; set the probe interval to `0` to honor those deadlines (unless a wait is manually skipped).
24
+ - Remembers provider retry timing exposed through response callbacks or provider errors (`retry-after`, `retry-after-ms`, `retry in ...`, reset messages). Explicit provider retry/reset deadlines are always honored: no early probes occur during that period (unless a wait is manually skipped).
23
25
  - If no retry timing is available for a rate limit, waits 30 minutes before retrying.
24
26
  - On `server_is_overloaded`, waits 5 minutes, then retries. If the provider is still overloaded after Pi's normal retries, it waits another 5 minutes and repeats.
25
27
  - On transient network/transport failures — including undici idle-timeout aborts (`UND_ERR_HEADERS_TIMEOUT` / `UND_ERR_BODY_TIMEOUT`), `fetch failed`, `terminated`, `ECONNRESET`, `ETIMEDOUT`, etc. — treats the error as retryable with a short backoff (default 15s, or the provider's `retry-after`) instead of giving up. This prevents a stalled streaming request from turning into a silent hang.
26
28
  - Leaves context-overflow recovery entirely to Pi. The extension uses Pi's own `isContextOverflow()` detector and immediately surfaces matching failures without waiting, retrying, or selecting another fallback. If the overflow came from a fallback already attempted for the request, the extension makes that attempted model current when Pi is still on the request's original model and accepts the switch; it never overrides a newer user model selection. Pi intentionally ignores overflow errors from non-current models. Successful over-window responses and zero-output length stops also pass through unchanged, allowing Pi to compact and perform its built-in recovery.
27
- - For any other (unclassified) provider error — including an API `Internal server error` and HTTP `400`/`403` failures — waits and retries the same model. There is deliberately no permanent-HTTP-status allowlist. This retry-all behavior is enabled by default and uses a configurable retry limit and interval; it does not alter model-limit wait intervals.
29
+ - For any other (unclassified) provider error — including an API `Internal server error` and HTTP `400`/`403` failures — waits and retries the same model. There is deliberately no permanent-HTTP-status allowlist. This retry-all behavior is enabled by default and uses a configurable retry limit and interval; it does not alter model-limit wait intervals. ChatGPT Business workspace-admin usage limits use this bounded policy, even when accompanied by HTTP `429` or `Retry-After`, rather than an endless timed-limit loop. Disable `PI_LIMITS_WAIT_DEFAULT_WAITING` to surface them immediately, matching Pi's non-retryable classification; the provider's admin guidance is preserved.
28
30
  - Reports retry/fallback waits in chat notifications without adding persistent TUI status lines.
29
31
  - In non-interactive hosts (`rpc` and `json` modes) the extension never reports a planned retry as an error. While it is waiting to retry, it publishes `info`-level liveliness notifications (plus a `setStatus` entry) describing what it is doing, how long it will wait, and why. An `error`-level notification is emitted only when the extension stops retrying and surfaces the terminal failure. See [Non-interactive (RPC) reporting](#non-interactive-rpc-reporting).
30
32
  - During a wait, pressing Enter in an empty editor retries immediately. Typed commands, autocomplete, and model-picker navigation still work normally. Choosing another model with `/model` immediately ends the pending wait and tries that exact model, even if it was previously marked unavailable.
@@ -34,9 +36,17 @@ Restart Pi or run `/reload` after installing.
34
36
 
35
37
  ### Unsupported interception limitation
36
38
 
37
- Pi 0.83 has no public stream middleware that can intercept a failed provider stream, wait, and re-enter `ModelRuntime` for another attempt. For that reason, limits-wait still installs a reload-safe trampoline on `ModelRuntime.prototype.streamSimple`. This is an unsupported internal interception point and could require changes when Pi changes `ModelRuntime`; it also assumes the interactive CLI's one active session runtime per process. SDK processes with multiple concurrent Pi sessions are not supported.
39
+ Pi 1.4.0 has no public stream middleware that can intercept a failed provider stream, wait, and re-enter `ModelRuntime` for another attempt. For that reason, limits-wait still installs a reload-safe trampoline on `ModelRuntime.prototype.streamSimple`. This is an unsupported internal interception point and could require changes when Pi changes `ModelRuntime`; it also assumes the interactive CLI's one active session runtime per process. SDK processes with multiple concurrent Pi sessions are not supported.
40
+
41
+ `after_provider_response` is preserved for observers, but cannot replace stream-error interception or own retry/fallback control flow. Pi 1.4.0 also does not expose the active header dispatcher to extensions. To preserve `before_provider_headers` on different-model fallbacks without replaying arbitrary source-bound `transformHeaders` closures, limits-wait intercepts exported `ExtensionRunner.prototype.createContext` to associate the active registry with its runner, then calls `emitBeforeProviderHeaders` on freshly resolved target headers. This is another unsupported integration point, isolated in `src/header-dispatch.ts`. Target auth, endpoint, configured headers and native client attribution remain runtime-owned; hook insertions and `null` deletions apply to the target, including attribution assembled after the hook. Client identity and provider-stream instrumentation are retained across fallback. Standalone SDK calls without an active extension runner still discard source transforms.
42
+
43
+ ## Compatibility tests
44
+
45
+ Run `npm test`, `npm run test:e2e`, and `npm run pack:check`. There is no separate build or lint task; `npm test` includes the strict TypeScript check.
46
+
47
+ `test:e2e` requires `PI_EVERGREEN_PI_BIN` pointing to the installed global Pi executable. It explicitly loads this checkout and never installs a CLI. Loopback HTTP tests use dummy OAuth-shaped credentials in an isolated agent directory and exercise retries, tools, prompt changes, manual/overflow compaction, abort, session replacement, and reload. They also verify real cross-provider fallback headers (target insertion/deletion, native attribution, source-secret exclusion and parsed-stream events) across multiple turns and lifecycle transitions, plus OpenAI Responses workspace limits with retry-all enabled/disabled and Pi auto-retry enabled. A negative control without the extension proves the transport assertions detect prompt leaks.
38
48
 
39
- `after_provider_response` is preserved for observers, but it is not an equivalent retry hook: it cannot replace stream-error interception, own retry/fallback control flow, or recreate a complete target-model request. In particular, Pi 0.83 still does not expose a public factory for rebuilding the SDK's target-model-aware `transformHeaders` callback. Reusing the source request's closure on a fallback would apply source-model attribution and could leak extension-added headers, so limits-wait intentionally omits it for every different-model fallback. The target `ModelRuntime` still resolves target auth, base URL, configured provider/model headers, environment, and provider implementation canonically, but `before_provider_headers` cannot be re-fired for that fallback through the current public API.
49
+ The required live two-turn smoke uses normal Pi authentication without reading credential files. Configure it with `PI_EVERGREEN_E2E_PROVIDER`, `PI_EVERGREEN_E2E_MODEL`, and `PI_EVERGREEN_E2E_REASONING` (defaults: `openai`, `gpt-6-luna`, `medium`). It captures request bodies, never auth headers, and removes its repository-local temporary artifacts afterward. Missing CLI or model access fails the suite rather than skipping coverage.
40
50
 
41
51
  ## Non-interactive (RPC) reporting
42
52
 
@@ -234,7 +244,7 @@ Each `fallback-models` entry supports:
234
244
 
235
245
  Top-level settings keys:
236
246
 
237
- - `probe-interval-seconds` — optional. How often a limited/frozen higher-priority model is re-probed. Default `60`, `0` disables probing. `PI_LIMITS_WAIT_PROBE_INTERVAL` overrides it. Both sources accept only safe non-negative integers (numeric strings are also accepted); values above 2147483 seconds are capped. Invalid file values use the default, and invalid environment values use the file value or default.
247
+ - `probe-interval-seconds` — optional. How often a higher-priority model with an estimated limit/freeze is silently re-probed. Explicit provider retry/reset deadlines are never probed early. Default `60`, `0` disables probing. `PI_LIMITS_WAIT_PROBE_INTERVAL` overrides it. Both sources accept only safe non-negative integers (numeric strings are also accepted); values above 2147483 seconds are capped. Invalid file values use the default, and invalid environment values use the file value or default.
238
248
 
239
249
  Fallback priority and manual selection:
240
250
 
@@ -245,10 +255,10 @@ Fallback priority and manual selection:
245
255
  - the remembered limit for that model expires;
246
256
  - the model answers a probe (see below);
247
257
  - successful output clears the limit/freeze known when its attempt began, but never a newer concurrent failure. Committed streams are never interrupted to reclaim a model.
248
- 5. A still-limited higher-priority model is eligible for a probe every **`probe-interval-seconds`** (default 60s), on the next LLM call or while waiting with all candidates blocked. This intentionally permits early probes even before an explicit provider `Retry-After` deadline; `0` disables early probes and honors remembered deadlines unless manually skipped. A probe is one normal request sent to that model instead of the current fallback:
258
+ 5. A higher-priority model with an estimated limit/freeze (no explicit provider timing) is eligible for a probe every **`probe-interval-seconds`** (default 60s), on the next LLM call or while waiting with all candidates blocked. Explicit provider `Retry-After` and reset deadlines always block early probes, regardless of this interval; `0` disables early probes altogether. Manual wait skipping can still retry immediately. Probes and their recoverable failures are silent: no probe notices or retry/freeze warnings are shown. A probe is one normal request sent to that model instead of the current fallback:
249
259
  - if it succeeds, the model is reclaimed immediately and the request continues on it;
250
260
  - if it fails before output is committed, it costs just one attempt—even for HTTP 403, network or unknown errors—then falls back down the order and postpones the next probe by another interval. Abort and context-overflow errors remain terminal rather than triggering fallback. With freezing disabled, unknown/HTTP 403 probe failures only postpone probes of the existing limit (without extending its deadline or creating a freeze); that candidate is exhausted for the current request. Exhausted and not-yet-due blocked candidates are skipped, and if no eligible candidate remains the unknown error is surfaced rather than looping or waiting on a freeze.
251
- 6. When *every* configured model is unavailable, the extension waits — but only until the earliest deadline **or the next due probe**, whichever comes first, so recovery is never delayed by a long remembered deadline.
261
+ 6. When *every* configured model is unavailable, the extension waits — but only until the earliest deadline **or the next eligible probe**, whichever comes first (never probing before an explicit provider deadline), so recovery is never delayed by a long remembered deadline.
252
262
  7. Rate-limit/freeze memory lives only in memory and is cleared when Pi restarts.
253
263
 
254
264
  The extension rereads these files before every LLM call, so changes made during a long-running task take effect on the next LLM call without restarting Pi or using `/reload`. Tool calls do not trigger a settings read. When the settings are loaded, the extension shows the full usable fallback model list. When models become rate-limited, it reports the wait in chat notifications.
@@ -263,10 +273,23 @@ The extension rereads these files before every LLM call, so changes made during
263
273
  | `PI_LIMITS_WAIT_LIVELINESS_INTERVAL` | `15` | Seconds between liveliness notifications published while waiting in non-interactive (`rpc`/`json`) modes. Must be a non-negative integer; values below 1 second are clamped to 1 second. Ignored in TUI mode. Also drives the structured status channel and is reported inside it as `livelinessIntervalMs`. |
264
274
  | `PI_LIMITS_WAIT_STATUS_JSON` | `true` | Whether the machine-readable status channel (`statusKey` `oira666.pi-limits-wait.json`) is published in non-interactive modes. Set to `false` (also accepts `0`, `no`, `off`) to restore exact 0.5.6 wire behavior. The prose channel is never affected. |
265
275
  | `PI_LIMITS_WAIT_CONTROL_FILE` | *(unset)* | Absolute path of the out-of-band control file polled once per second **while waiting**, used by the host to skip the current wait and retry immediately. Unset disables the feature and all filesystem access. See [Skipping a wait from the host](#skipping-a-wait-from-the-host). |
266
- | `PI_LIMITS_WAIT_PROBE_INTERVAL` | `60` | Seconds between re-probes of a higher-priority model that is still remembered as rate-limited or frozen. Lower values return to the preferred model sooner at the cost of more rejected probe requests; `0` disables probing, so a model is only reconsidered when its remembered deadline expires. Overrides the `probe-interval-seconds` settings key. Must be a safe non-negative integer; fractions, Infinity and unsafe values are rejected. Both file and environment values above 2147483 seconds are capped to Node's reliable timeout range. Invalid environment values use the file setting (or default). |
276
+ | `PI_LIMITS_WAIT_PROBE_INTERVAL` | `60` | Seconds between silent re-probes of a higher-priority model with an estimated rate limit or freeze. Explicit provider retry/reset deadlines always block early probes. Lower values return to the preferred model sooner when no provider timing is known, at the cost of more rejected probe requests; `0` disables probing, so a model is only reconsidered when its remembered deadline expires. Overrides the `probe-interval-seconds` settings key. Must be a safe non-negative integer; fractions, Infinity and unsafe values are rejected. Both file and environment values above 2147483 seconds are capped to Node's reliable timeout range. Invalid environment values use the file setting (or default). |
267
277
  | `PI_LIMITS_WAIT_RETRY_INTERVAL` | `5` | Seconds to wait between unclassified/unknown provider-error retries. Must be a non-negative integer (values above 2147483 are capped to Node's maximum reliable timer delay). This does not affect model-limit wait intervals. |
268
278
  | `PI_LIMITS_WAIT_FREEZING_ENABLED` | `true` | When a model keeps failing with a non-retryable error, it is normally "frozen" for 10 minutes and skipped in favour of other configured models. Set this to `false` (also accepts `0`, `no`, `off`) to disable freezing entirely: the extension will instead try each configured candidate once (after the bounded retries) and then surface the error, never blocking on a long "model-frozen" wait. Useful for non-interactive / subagent runs where no one can press Enter to skip. |
269
279
 
280
+ ## Development
281
+
282
+ ```sh
283
+ npm ci
284
+ mkdir -p tmp
285
+ TMPDIR="$PWD/tmp" npm test
286
+ npm run pack:check
287
+ ```
288
+
289
+ `npm test` installs the direct Pi development dependencies at `@latest` with install scripts disabled, then runs `npm run check` and the tests. Tests target only the latest npm Pi release. Before each release, update the tested version above and the Pi peer dependency baseline after all checks pass. No lint command is configured.
290
+
291
+ Tests cover retries against a local HTTP server and editor focus/key routing. They do not call live model providers or separately execute README snippets.
292
+
270
293
  ## License
271
294
 
272
295
  MIT
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "oira666_pi-limits-wait",
3
- "version": "0.6.5",
3
+ "version": "0.6.7",
4
4
  "description": "Extension for PI that allows to automatically wait for limits to reset and retry the request",
5
5
  "type": "module",
6
6
  "exports": {
@@ -29,7 +29,9 @@
29
29
  },
30
30
  "scripts": {
31
31
  "check": "tsc -p tsconfig.json --noEmit",
32
+ "pretest": "npm run check",
32
33
  "test": "tsx src/__tests__/test-extension.ts && tsx src/__tests__/test-wait-ui.ts",
34
+ "test:e2e": "node scripts/test-e2e.mjs",
33
35
  "pack:check": "npm pack --dry-run",
34
36
  "prepublishOnly": "npm run check && npm test && npm run pack:check"
35
37
  },
@@ -38,14 +40,13 @@
38
40
  "./src/index.ts"
39
41
  ]
40
42
  },
41
- "dependencies": {},
42
43
  "peerDependencies": {
43
- "@earendil-works/pi-ai": "*",
44
- "@earendil-works/pi-coding-agent": "*"
44
+ "@earendil-works/pi-ai": ">=1.4.0",
45
+ "@earendil-works/pi-coding-agent": ">=1.4.0"
45
46
  },
46
47
  "devDependencies": {
47
- "@earendil-works/pi-ai": "0.83.0",
48
- "@earendil-works/pi-coding-agent": "0.83.0",
48
+ "@earendil-works/pi-ai": "1.4.0",
49
+ "@earendil-works/pi-coding-agent": "1.4.0",
49
50
  "tsx": "^4.20.6",
50
51
  "typescript": "^5.9.3"
51
52
  }
package/src/constants.ts CHANGED
@@ -25,8 +25,8 @@ export const DEFAULT_OVERLOADED_WAIT_MS = 5 * 60 * 1_000; // 5 minutes
25
25
  export const DEFAULT_NETWORK_WAIT_MS = 15 * 1_000; // 15 seconds
26
26
  export const DEFAULT_NON_RETRYABLE_FREEZE_MS = 10 * 60 * 1_000; // 10 minutes
27
27
 
28
- // A model that is currently limited/frozen is re-probed this often, so a
29
- // higher-priority model can be reclaimed long before its remembered deadline.
28
+ // Estimated limits/freezes may be silently re-probed this often. Explicit
29
+ // provider retry/reset deadlines always prevent early probes.
30
30
  export const DEFAULT_PROBE_INTERVAL_MS = 60 * 1_000; // 1 minute
31
31
 
32
32
  // Unknown provider errors are retried separately from rate-limit, overload,
@@ -43,7 +43,7 @@ export const DEFAULT_LIVELINESS_INTERVAL_MS = 15 * 1_000; // 15 seconds
43
43
  export const LIVELINESS_JSON_STATUS_KEY = "oira666.pi-limits-wait.json";
44
44
  /** Payload schema version for LIVELINESS_JSON_STATUS_KEY. Bump only on a breaking change. */
45
45
  export const LIVELINESS_JSON_VERSION = 1;
46
- export const EXTENSION_VERSION = "0.6.5";
46
+ export const EXTENSION_VERSION = "0.6.6";
47
47
 
48
48
  export const SETTINGS_FILE_NAME = "limits-wait.json";
49
49
  export const FALLBACK_MODELS_KEY = "fallback-models";
@@ -0,0 +1,35 @@
1
+ import { ExtensionRunner, type ModelRegistry } from "@earendil-works/pi-coding-agent";
2
+ import { state } from "./state.js";
3
+ import type { ModelsSimpleStreamOptions } from "@earendil-works/pi-ai";
4
+
5
+ // Pi does not expose its header dispatcher to extensions. Recover the runner
6
+ // from the contexts it creates, rather than replaying a source-model closure
7
+ // (which may inject source credentials) against another provider's headers.
8
+ // This exported-class interception is unsupported; keep it isolated and cover
9
+ // actual CLI fallback, reload and session replacement at the transport boundary.
10
+ const SLOT = Symbol.for("oira666.pi-limits-wait.header-dispatch.v1");
11
+ interface DispatchSlot {
12
+ runners: WeakMap<ModelRegistry, ExtensionRunner>;
13
+ wrapper: ExtensionRunner["createContext"];
14
+ }
15
+ type RunnerPrototype = ExtensionRunner & { [SLOT]?: DispatchSlot };
16
+
17
+ export function installHeaderDispatch(): void {
18
+ const prototype = ExtensionRunner.prototype as RunnerPrototype;
19
+ if (prototype[SLOT]?.wrapper === prototype.createContext) return;
20
+ const previous = prototype.createContext;
21
+ const runners = new WeakMap<ModelRegistry, ExtensionRunner>();
22
+ const wrapper = function (this: ExtensionRunner) {
23
+ const context = previous.call(this);
24
+ runners.set(context.modelRegistry, this);
25
+ return context;
26
+ };
27
+ Object.defineProperty(prototype, SLOT, { configurable: true, value: { runners, wrapper } });
28
+ prototype.createContext = wrapper;
29
+ }
30
+
31
+ export function fallbackHeaderTransform(): ModelsSimpleStreamOptions["transformHeaders"] {
32
+ const registry = state.sharedCtx?.modelRegistry;
33
+ const runner = registry && (ExtensionRunner.prototype as RunnerPrototype)[SLOT]?.runners.get(registry);
34
+ return runner ? (headers) => runner.emitBeforeProviderHeaders(headers) : undefined;
35
+ }
package/src/index.ts CHANGED
@@ -117,13 +117,8 @@ export default function (pi: ExtensionAPI) {
117
117
  const retrySummaries = new RetrySummaryCoordinator(pi);
118
118
  const releaseInterception = installModelRuntimeInterception(() => retrySummaries.createStream());
119
119
 
120
- // Pi 0.86+ warms caches through the same runtime as foreground requests.
121
- // Stop background refreshes before they can enter our retry/fallback logic.
122
- // Keep the older Pi dependency range: its event registry accepts this hook
123
- // but never emits it. The narrow overload bridges the pre-0.86 typings.
124
- (pi as ExtensionAPI & {
125
- on(event: "cache_warming_decision", handler: () => { action: "stop" }): void;
126
- }).on("cache_warming_decision", () => ({ action: "stop" }));
120
+ // Background refreshes must not enter foreground retry/fallback logic.
121
+ pi.on("cache_warming_decision", () => ({ action: "stop" }));
127
122
 
128
123
  pi.on("session_start", (event, ctx) => {
129
124
  state.sharedCtx = ctx;
package/src/models.ts CHANGED
@@ -1,6 +1,7 @@
1
1
  import type { Api, Model } from "@earendil-works/pi-ai";
2
2
  import { DEFAULT_NON_RETRYABLE_FREEZE_MS } from "./constants.js";
3
3
  import { expectModelSelection, state } from "./state.js";
4
+ import { parseRetryDelayMs } from "./retry-errors.js";
4
5
  import type { FallbackModel, RetryableError } from "./types.js";
5
6
  import { formatErrorDetail, formatDuration, notifyLiveliness, reasonLabel } from "./ui.js";
6
7
 
@@ -82,19 +83,13 @@ export function notifyRetryingAfterError(model: Model<Api>, waitMs: number, erro
82
83
  notifyLiveliness(`${formatModel(model)} retrying after error in ${formatDuration(waitMs)}. Error: ${formatErrorDetail(errorMessage, Number.MAX_SAFE_INTEGER)}`);
83
84
  }
84
85
 
85
- function notifyProbe(model: Model<Api>, remainingMs: number): void {
86
- const ctx = state.sharedCtx;
87
- if (!ctx) return;
88
- const remaining = remainingMs > 0 ? ` (${formatDuration(remainingMs)} left on its remembered limit)` : "";
89
- try {
90
- ctx.ui.notify(`Probing higher-priority model ${formatModel(model)}${remaining}.`, "info");
91
- } catch { /* UI unavailable */ }
92
- }
93
-
94
- export function rememberRateLimit(model: Model<Api>, retryable: RetryableError, errorMessage?: string): void {
86
+ export function rememberRateLimit(model: Model<Api>, retryable: RetryableError, errorMessage?: string, silent = false): void {
95
87
  const deadline = Date.now() + retryable.waitMs;
96
- state.rateLimitMemory.set(modelKey(model), { reason: retryable.reason, limitedAt: Date.now(), deadline });
97
- notifyRetryableError(model, retryable, errorMessage);
88
+ state.rateLimitMemory.set(modelKey(model), {
89
+ reason: retryable.reason, limitedAt: Date.now(), deadline,
90
+ explicitDeadline: errorMessage !== undefined && parseRetryDelayMs(errorMessage) !== undefined,
91
+ });
92
+ if (!silent) notifyRetryableError(model, retryable, errorMessage);
98
93
  ensureRateLimitedModelsStatus();
99
94
  }
100
95
 
@@ -118,9 +113,8 @@ export function probeIntervalMs(): number {
118
113
  /**
119
114
  * Remembered unusability of a model, merged across the rate-limit and frozen
120
115
  * memories, plus the moment it becomes worth re-probing. Remembered deadlines
121
- * are estimates (`retry-after` is often pessimistic, or absent and defaulted),
122
- * so a blocked model is re-probed periodically instead of being trusted until
123
- * its deadline.
116
+ * without provider timing are estimates and may be re-probed periodically.
117
+ * Explicit provider retry/reset deadlines must expire before any early probe.
124
118
  */
125
119
  export function blockedState(model: Model<Api>): { deadline: number; probeAt: number } | undefined {
126
120
  const limit = activeLimit(model);
@@ -134,7 +128,8 @@ export function blockedState(model: Model<Api>): { deadline: number; probeAt: nu
134
128
  frozen ? frozen.lastProbeAt ?? frozen.failedAt : 0,
135
129
  );
136
130
  const interval = probeIntervalMs();
137
- const probeAt = interval > 0 ? Math.min(lastEvidence + interval, deadline) : deadline;
131
+ const estimatedProbeAt = interval > 0 ? Math.min(lastEvidence + interval, deadline) : deadline;
132
+ const probeAt = Math.max(limit?.explicitDeadline ? limit.deadline : 0, estimatedProbeAt);
138
133
  return { deadline, probeAt };
139
134
  }
140
135
 
@@ -158,9 +153,7 @@ export function clearUnavailability(model: Model<Api>, snapshot: ReturnType<type
158
153
 
159
154
  /** Record that a blocked model was just re-probed, so probes stay rate-limited. */
160
155
  export function markProbeAttempt(model: Model<Api>, now = Date.now()): void {
161
- const remainingMs = (blockedState(model)?.deadline ?? now) - now;
162
156
  postponeProbe(model, now);
163
- notifyProbe(model, remainingMs);
164
157
  }
165
158
 
166
159
  /** Postpone probing existing evidence without creating or extending a freeze. */
@@ -198,10 +191,10 @@ export function nextAvailableCandidate(current: Model<Api>, excluded?: ReadonlyS
198
191
  return nextAttemptCandidate(current, excluded)?.entry;
199
192
  }
200
193
 
201
- export function rememberNonRetryableFailure(model: Model<Api>, errorMessage: string): void {
194
+ export function rememberNonRetryableFailure(model: Model<Api>, errorMessage: string, silent = false): void {
202
195
  const deadline = Date.now() + DEFAULT_NON_RETRYABLE_FREEZE_MS;
203
196
  state.nonRetryableFailureMemory.set(modelKey(model), { failedAt: Date.now(), deadline, errorMessage });
204
- notifyLiveliness(`${formatModel(model)} failed (${formatErrorDetail(errorMessage)}); freezing it for ${formatDuration(DEFAULT_NON_RETRYABLE_FREEZE_MS)} and trying another configured model if available.`);
197
+ if (!silent) notifyLiveliness(`${formatModel(model)} failed (${formatErrorDetail(errorMessage)}); freezing it for ${formatDuration(DEFAULT_NON_RETRYABLE_FREEZE_MS)} and trying another configured model if available.`);
205
198
  ensureRateLimitedModelsStatus();
206
199
  }
207
200
 
@@ -215,6 +215,11 @@ export function retryAfterHeaderMs(headers: Headers): number | undefined {
215
215
  }
216
216
 
217
217
  export function getRetryableError(msg: string): RetryableError | undefined {
218
+ // Business workspace limits require an administrator, not a timed reset.
219
+ // Keep retry-all configurable, but use its bounded unknown-error budget even
220
+ // when transport metadata adds HTTP 429 / Retry-After to the provider text.
221
+ if (msg.includes("subscription_sharing_workplace_individual_usage_limit_exceeded")
222
+ || msg.includes("You've reached your ChatGPT workspace's usage limit.")) return undefined;
218
223
  if (isServerOverloadedError(msg)) {
219
224
  return { reason: "overloaded", waitMs: parseRetryDelayMs(msg) ?? DEFAULT_OVERLOADED_WAIT_MS };
220
225
  }
package/src/stream.ts CHANGED
@@ -34,6 +34,7 @@ import {
34
34
  switchPiModel,
35
35
  } from "./models.js";
36
36
  import { withAttemptResponseObserver } from "./response-observer.js";
37
+ import { fallbackHeaderTransform, installHeaderDispatch } from "./header-dispatch.js";
37
38
  import { errorMessageWithCauses, getRetryableError, isAbortMessage, parseRetryDelayMs, retryAfterHeaderMs } from "./retry-errors.js";
38
39
  import type { RetryPeriodTracker } from "./retry-summary.js";
39
40
  import { loadFallbackSettings } from "./settings.js";
@@ -166,6 +167,11 @@ function attemptOptions(
166
167
  signal,
167
168
  onPayload,
168
169
  onResponse,
170
+ // Dispatch the host's hooks on freshly resolved TARGET headers. Do not
171
+ // replay options.transformHeaders: arbitrary SDK callers may bind secrets.
172
+ transformHeaders: fallbackHeaderTransform(),
173
+ identity: options?.identity,
174
+ onProviderStreamEvent: options?.onProviderStreamEvent,
169
175
  cacheRetention: options?.cacheRetention,
170
176
  sessionId: options?.sessionId,
171
177
  timeoutMs: options?.timeoutMs,
@@ -611,8 +617,8 @@ export function streamWithLimitsRetry(
611
617
  // classification. Abort and context overflow have already terminated.
612
618
  const failure = retryableErrorMessage || nonRetryableError?.message || "Provider stream ended without a terminal event.";
613
619
  if (retryable || freezingEnabled()) {
614
- if (retryable) rememberRateLimit(attempt.model, retryable, failure);
615
- else rememberNonRetryableFailure(attempt.model, failure);
620
+ if (retryable) rememberRateLimit(attempt.model, retryable, failure, true);
621
+ else rememberNonRetryableFailure(attempt.model, failure, true);
616
622
  retryPeriod?.recordImmediateRetry(failure);
617
623
  attempt = nextAvailableCandidate(attempt.model) ?? attempt;
618
624
  continue;
@@ -815,6 +821,7 @@ export function disableModelRuntimeInterception(): void {
815
821
  export function installModelRuntimeInterception(
816
822
  createRetryPeriod?: () => RetryPeriodTracker,
817
823
  ): () => boolean {
824
+ installHeaderDispatch();
818
825
  const prototype = ModelRuntime.prototype as InterceptablePrototype;
819
826
  const owner = {};
820
827
  const controller = new AbortController();
package/src/types.ts CHANGED
@@ -30,6 +30,8 @@ export type RateLimitMemory = {
30
30
  reason: RetryReason;
31
31
  limitedAt: number;
32
32
  deadline: number;
33
+ /** Provider supplied retry/reset timing; never probe before this deadline. */
34
+ explicitDeadline?: boolean;
33
35
  /** Last time this model was re-probed while still remembered as limited. */
34
36
  lastProbeAt?: number;
35
37
  };