pi-durable-subagents 1.0.9 → 1.0.10

package/CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## 1.0.10
+
+- Provider failover for a used-up usage window. An error such as `503 No
+  available accounts`, "usage limit" or "quota exceeded" (after pi's own
+  retries) marks the provider used up instead of counting as a lost
+  execution: a call in a pool continues in the same session on the pool's next
+  model, and new calls skip the provider. After `k.probeMs` (15 minutes) the
+  next call that wants it is admitted to it alone; when it answers, new calls
+  and new generations go back to it. A call with a single model waits for the
+  provider instead of failing. `status` lists used-up providers with their
+  next try. Billing errors (402, insufficient balance) still fail at once.
+  A short request rate limit ("429 … resets in 1 second") is not a used-up
+  window and is retried as before. A follow-up naming a model runs on it even
+  where the pool would start over.
+- After a call moved to another model by a relaunch, the orchestrator now
+  reads the session's model as pi restores it (from the last answer), so a
+  later relaunch holds the slot of the provider it actually uses.
+
 ## 1.0.9
 
 - `status` shows the model a call actually uses: the model of its last

package/README.md

@@ -55,6 +55,7 @@ the npx cache, so `install-service` refuses to run from there.
 | You steer a subagent while it is asking you a question | Your message reaches it, in order. Nothing is rejected or lost. |
 | Two steers arrive out of order and the second replaces the first | Only the second one applies. |
 | A step is refused, or a dependency fails | The workflow stops that branch cleanly. Nothing is retried in vain. |
+| A provider's usage window runs out (`No available accounts`, usage limit, quota exceeded) | A call in a pool continues **in the same session** on the pool's next model; new calls skip that provider. After 15 minutes the next call that wants it tries it once; when it answers, new calls and new generations use it again. A call with a single model waits for it instead of failing. Billing errors (402, insufficient balance) still fail at once. |
 | A subagent waits for an answer for a long time | It releases its model slot and memory, then resumes exactly once when you answer. |
 
 ## Use it
@@ -272,7 +273,14 @@ State lives in `~/.pi/durable-subagents`; set `DSA_HOME` to move it.
 ```
 
 - **Pools:** a model can name a pool. The first candidate with a free slot is
-  used, and a candidate that keeps failing is skipped for 10 minutes.
+  used, and a candidate that keeps failing is skipped for 10 minutes. The
+  order is the preference: list the provider you want to use first.
+- **A used-up provider** is not sent new calls until its next try, 15 minutes
+  after it last refused (`"k": { "probeMs": 900000 }`). Then one call at a
+  time goes to it, so finding out costs no extra request. A call that moved to
+  another provider stays there for the rest of its generation (switching back
+  mid-task would lose the prompt cache); a follow-up starts on the first
+  candidate again. `status` lists each used-up provider with its next try.
 - **Provider slots:** never exceeded, including while a model switch is in
   progress.
 - **Memory:** new subagents wait while memory is short. Running ones are

package/dist/agent/main.js

@@ -326,7 +326,7 @@ export function registerMain(pi, ui) {
             "run (action optional for exactly one launch form): agent+task; tasks:[call specs] parallel; chain:[call specs] sequential ({previous}); workflow:'./script.js' or source (runs.run(key,spec), runs.all([...]), emit(value), args, runs.input(name)). Optional name, cwd, usageBudget, maxCalls, inputs. With tasks/chain, top-level model, timeoutMs, budget, isolation, context, tools, skills, once are defaults for every step (a step's own value wins); a workflow/source script sets them per runs.run call. timeoutMs is milliseconds of active time (a number); omit it unless a hard limit is needed. Explicit unknown agents are rejected BEFORE creation, with available names; unknown script agents fail only their call.",
             "agents: list names, descriptions, default models and source for this cwd; use these names for run.",
             "send to:'<wid>/<key>' (bare '<wid>' only for a single-call workflow): steer on a running call delivers at the next safe point (receipt in status/UI); a steer to a call waiting on its question interrupts the question and the subagent usually asks again — use answer to answer it; sealed → finished:<status> — use kind 'follow-up'. follow-up continues a sealed call as generation g+1 or queues after a running turn; follow-up model:'provider/id' runs that generation on it. answer: give the qid (or just the call, or nothing when one question is open); to and rev are filled in. A question that needs the user's decision goes to the user; if you answer one yourself, tell the user what you chose. model: a running call switches at its next provider request; an asking, hibernated or queued call launches on it when it runs again; the reply's model/effect (next-request|next-execution|next-generation) says which. status model = model actually used by the last request; switching = requested, not used yet; switchFailed = refused. A provider content refusal (ToS/usage policy) fails the call at once, not retried. Unknown targets list valid addresses. replaces:[rid] supersedes an earlier send.",
-            "stop target:<wid|<wid>/<key>> is terminal stopped (usage and partial edits kept); a sealed call → already-sealed:<status>, a finished workflow → terminal:<status>. drain holds existing workflows reversibly (new runs unaffected); resume [wid] releases held workflows. status: without wid, what runs, asks (with its answer address; hibernated:true holds no slot) or failed, finished workflows one line each, provider slots held/limit and the config in effect; wid: one workflow, outputs clipped; wid+key: one call's full result; full:true: everything. A run's rid from {submitted:{rid}} works wherever a wid is expected. revise wid + workflow/source/args starts a revision.",
+            "stop target:<wid|<wid>/<key>> is terminal stopped (usage and partial edits kept); a sealed call → already-sealed:<status>, a finished workflow → terminal:<status>. drain holds existing workflows reversibly (new runs unaffected); resume [wid] releases held workflows. status: without wid, what runs, asks (with its answer address; hibernated:true holds no slot) or failed, finished workflows one line each, provider slots held/limit, the config in effect and providers whose usage window is used up (avoided until a probe finds them answering again); wid: one workflow, outputs clipped; wid+key: one call's full result; full:true: everything. A run's rid from {submitted:{rid}} works wherever a wid is expected. revise wid + workflow/source/args starts a revision.",
             "Control replies are {applied:true,rid} or {applied:false,reason,rid} when decided; otherwise {submitted:{rid}} after 10s.",
             ...(agents ? [`Available agents: ${agents}.`] : []),
             "User sees a summary line above the editor; ↓ on an empty editor (or /subagents) opens the list, Enter watches live OR finished calls (finished transcripts remain on disk) and expands finished workflows. List keys: s steer (paste-capable input), x stop (confirm y), m model, a answer when asked, f follow-up on finished calls; action feedback appears in footer.",

package/dist/cli/main.js

@@ -71,7 +71,7 @@ export function renderView(view) {
         lines.unshift(`${view.paused} (pi-durable-subagents resume)`);
     if (view.olderFinished)
         lines.push(`(+${view.olderFinished} older finished workflows; status <wid> shows one in detail)`);
-    const footer = [view.slots?.length ? `slots: ${view.slots.join(", ")}` : "", view.config ? `config: ${view.config}` : "", view.configRejected ? `config.json rejected: ${view.configRejected}` : ""].filter(Boolean);
+    const footer = [view.slots?.length ? `slots: ${view.slots.join(", ")}` : "", view.config ? `config: ${view.config}` : "", view.configRejected ? `config.json rejected: ${view.configRejected}` : "", ...(view.exhausted ?? [])].filter(Boolean);
     if (!lines.length)
         lines.push("No workflows");
     return [...lines, ...footer].join("\n");

package/dist/orchestrator/config.js

@@ -9,7 +9,7 @@ import { join } from "node:path";
 import { contentHash } from "../kernel/ids.js";
 /** The keys the orchestrator reads; config.json also holds pi-side settings (ui, onQuit) that it ignores. */
 const KEYS = ["defaultModel", "pools", "providers", "memory", "k"];
-const K = ["lossBound", "checkpointMs", "stallMs", "progressMs", "switchTimeoutMs", "idleExitMs", "trackerMs", "hibernateMs", "spawnBudget"];
+const K = ["lossBound", "checkpointMs", "stallMs", "progressMs", "switchTimeoutMs", "idleExitMs", "trackerMs", "hibernateMs", "spawnBudget", "probeMs"];
 export const configPath = (home) => join(home, "config.json");
 /** The orchestrator's part of a parsed config.json. */
 export function orchestratorSettings(raw) {

package/dist/orchestrator/executor/index.js

@@ -22,7 +22,8 @@ import { buildCallResult } from "../../compat/result.js";
 import createEffects from "./effects/index.js";
 import { continueSession } from "./generation.js";
 import { hibernation, openQuestion } from "./hibernate.js";
-import { evidence, fatalProviderError, refusedByProvider, forgetSession, readSessionState, receiptId, sessionModel } from "./session.js";
+import { foldExhaustion } from "../providers.js";
+import { evidence, fatalProviderError, quotaExhausted, refusedByProvider, forgetSession, readSessionState, receiptId, sessionModel } from "./session.js";
 import { activeTotal } from "./time.js";
 import { observeExecution } from "./observe.js";
 import { availableMemory } from "./memory.js";
@@ -115,7 +116,7 @@ export default function createExecutor(ledgers, options = {}) {
     const wake = () => { for (const fn of waiters)
         fn(); waiters.clear(); };
     // F3: fold only orchestrator entries appended since the last fold (holdings, observed switches, K7 skips).
-    const ledger = { seen: 0, held: new Map(), observed: new Set(), skips: new Map() };
+    const ledger = { seen: 0, held: new Map(), observed: new Set(), skips: new Map(), exhausted: new Map() };
     const folded = () => {
         const entries = orch.entries();
         for (; ledger.seen < entries.length; ledger.seen++) {
@@ -128,11 +129,17 @@ export default function createExecutor(ledgers, options = {}) {
                 ledger.observed.add(`${e.exec}\n${e.rid}`);
             else if (e.type === "skip")
                 ledger.skips.set(`${e.pool}\n${e.model}`, Math.max(Number(e.until), ledger.skips.get(`${e.pool}\n${e.model}`) ?? 0));
+            foldExhaustion(ledger.exhausted, e);
         }
         return ledger;
     };
     const holdings = () => [...folded().held.values()];
     const skipped = (pool, model) => (folded().skips.get(`${pool}\n${model.provider}/${model.id}`) ?? 0) > Date.now();
+    /** A provider whose usage window is used up admits no call until its next try, and then one probe at a time. */
+    const unavailable = (provider) => {
+        const x = provider ? folded().exhausted.get(provider) : undefined;
+        return !!x && (Date.now() < x.nextTry || x.probe !== undefined);
+    };
     async function release(exec) {
         await serial(async () => { for (const h of holdings().filter(e => e.exec === exec))
             await orch.append("release", { pool: h.pool, slot: h.slot, exec }); });
@@ -443,6 +450,9 @@ export default function createExecutor(ledgers, options = {}) {
                     if (!continuation && pool && skipped(pool, model))
                         continue;
                     const provider = model.provider;
+                    if (unavailable(provider))
+                        continue;
+                    const probe = provider !== undefined && folded().exhausted.has(provider);
                     const holders = holdings().filter(e => e.pool === provider);
                     const limit = config.providers?.[provider ?? ""]?.slots ?? Infinity;
                     if (!capacity({ kind: "provider", holders: holders.length, capacity: limit }))
@@ -470,6 +480,10 @@ export default function createExecutor(ledgers, options = {}) {
                         while (holders.some(e => e.slot === slot))
                             slot++;
                         await orch.append("hold", { pool: provider, slot, exec });
+                        // After its next try, the first call admitted to a used-up provider is its probe: its first request
+                        // either goes through (the provider is available again) or is refused, which uses no quota.
+                        if (probe)
+                            await orch.append("provider-probe", { provider, exec });
                     }
                     await a.ticket.journal.append("selected", { exec, model, ...(pool ? { pool } : {}) });
                     return model;
@@ -498,11 +512,16 @@ export default function createExecutor(ledgers, options = {}) {
         const pool = raw && config.pools?.[raw] ? raw : undefined;
         const candidates = raw ? resolveModel(raw, config.pools) : [{ id: "" }];
         const candidate = recorded && candidates.some(m => m.provider === recorded.provider && m.id === recorded.id);
-        const skip = previous && ownSegment && pool && candidate && skipped(pool, recorded);
+        // Leave the session's model for the pool's others when its pool skips it after losses, or its provider's usage
+        // window is used up; and at a new generation, go back to the pool's order of preference.
+        const skip = pool && candidate && (previous && ownSegment && skipped(pool, recorded) || unavailable(recorded.provider) || !ownSegment && !!t.continueFrom);
         // A model the call was asked to use replaces the session's: launched with it, and holding its provider's slot.
         const wanted = requestedModel(t.journal, t.callId, t.model);
-        if (wanted && !(recorded && !freshFork && recorded.provider === wanted.provider && recorded.id === wanted.id))
-            return { candidates: [wanted], continuation: false, pool: undefined };
+        // It outranks the pool's order at a new generation too, also when it names the model the session already has.
+        if (wanted)
+            return recorded && !freshFork && recorded.provider === wanted.provider && recorded.id === wanted.id
+                ? { candidates: [recorded], continuation: true, pool: undefined }
+                : { candidates: [wanted], continuation: false, pool: undefined };
         if (recorded && !freshFork && !skip)
             return { candidates: [recorded], continuation: true, pool: candidate ? pool : undefined };
         return { candidates, continuation: false, pool };
@@ -560,6 +579,32 @@ export default function createExecutor(ledgers, options = {}) {
         });
         wake();
     }
+    /** The model an execution last answered with, or was launched on. */
+    function modelOf(journal, exec) {
+        return journal.entries().findLast(e => (e.type === "selected" || e.type === "model-used") && e.exec === exec)?.model;
+    }
+    /** Record a used-up provider once per window: again only when its probe (or any call after the next try) is refused. */
+    async function recordExhausted(provider, exec, error) {
+        const x = folded().exhausted.get(provider), now = Date.now();
+        // While a probe runs, its outcome alone decides: a late refusal of an execution admitted earlier changes nothing.
+        if (x && (x.probe ? x.probe !== exec : now < x.nextTry))
+            return;
+        if (orch.entries().some(e => e.type === "provider-exhausted" && e.exec === exec))
+            return;
+        await orch.append("provider-exhausted", { provider, exec, since: x?.since ?? now, nextTry: now + (config.k?.probeMs ?? 900_000), error: error.slice(0, 300) });
+    }
+    /** An answer from a used-up provider, requested after it was found used up: available again. */
+    async function answered(exec, event) {
+        const message = event.message, provider = message?.provider;
+        if (message?.role !== "assistant" || !provider || message.stopReason === "error")
+            return;
+        await serial(async () => {
+            const x = folded().exhausted.get(provider);
+            if (x && (x.probe === exec || Number(message.timestamp) > x.since))
+                await orch.append("provider-available", { provider, exec });
+        });
+        wake();
+    }
     function pendingSwitch(exec) {
         return holdings().find(h => h.exec === exec && h.reserved && !folded().observed.has(`${exec}\n${h.rid}`));
     }
@@ -639,11 +684,17 @@ export default function createExecutor(ledgers, options = {}) {
                     // A refusal of the content is deterministic: the same request is refused again, so it is reported, not retried.
                     if (has(journal, "settled", exec) && !ev.text && ev.error && refusedByProvider(ev.error))
                         return finish(journal, t.callId, exec, makeResult("failed", "", `Refused by the provider (not retried): ${ev.error.slice(0, 500)}`));
-                    await serial(async () => {
-                        if (!has(journal, "loss", exec))
-                            await journal.append("loss", { exec });
-                        await skipLostCandidate(journal, orch, exec);
-                    });
+                    // A used-up usage window is no loss: the provider is avoided until a probe finds it accepting requests again,
+                    // and the call goes on with the pool's next model, or waits for that provider.
+                    const exhausted = has(journal, "settled", exec) && !ev.text && ev.error && quotaExhausted(ev.error) ? modelOf(journal, exec)?.provider : undefined;
+                    if (exhausted)
+                        await serial(() => recordExhausted(exhausted, exec, ev.error));
+                    else
+                        await serial(async () => {
+                            if (!has(journal, "loss", exec))
+                                await journal.append("loss", { exec });
+                            await skipLostCandidate(journal, orch, exec);
+                        });
                     const losses = journal.entries().filter(e => e.type === "loss" && String(e.exec).startsWith(`${t.callId}#`)).length;
                     if (losses >= (config.k?.lossBound ?? 5))
                         return finish(journal, t.callId, exec, makeResult("failed", "", `lost ×${losses}${ev.error ? `; last error: ${ev.error.slice(0, 300)}` : ""}`));
@@ -739,7 +790,7 @@ export default function createExecutor(ledgers, options = {}) {
                             }
                         });
                     }, recordUsage: values => recordUsage(t, values),
-                    switched: event => switched(exec, journal, event), pendingSwitch: () => pendingSwitch(exec),
+                    switched: event => switched(exec, journal, event), answered: event => answered(exec, event), pendingSwitch: () => pendingSwitch(exec),
                 });
             }
             finally {

package/dist/orchestrator/executor/observe.js

@@ -110,8 +110,10 @@ export async function observeExecution(d) {
                 await serial(() => t.journal.append("observation", { exec, event: slim }));
                 if (event.type === "message_start")
                     await d.switched(event);
-                if (event.type === "message_end")
+                if (event.type === "message_end") {
                     await d.recordUsage([{ id: String(slim.id), usage: slim.usage }]);
+                    await d.answered?.(event);
+                }
             }
             await limits();
             await stall();

package/dist/orchestrator/executor/session.js

@@ -107,9 +107,24 @@ export function evidence(entries, exec) {
     const error = last?.stopReason === "error" ? last.errorMessage : undefined;
     return { report, budget, text, error, dangling: [...tools].map(([id, name]) => `${name} (${id})`), usage };
 }
-/** Only explicit quota/payment failures are terminal; rate limits, overload and transport errors still retry. */
+/** Only explicit payment failures are terminal; rate limits, overload and transport errors still retry, and a used-up
+ *  usage window (`quotaExhausted`) waits for the provider or moves to another one. */
 export function fatalProviderError(text) {
-    return /\b402\b|insufficient[_ ]?(quota|balance|funds)|quota (exceeded|exhausted)|billing|credit balance|额度|余额|usage limit/i.test(text);
+    return /\b402\b|insufficient[_ ]?(quota|balance|funds)|billing|credit balance|余额/i.test(text);
+}
+/** A provider's usage window is used up: its requests are refused (and not counted) until the window resets, hours
+ *  later. Seen as a gateway's `503 No available accounts` once pi's own retries are spent, or a usage-limit message.
+ *  The provider is then avoided until a probe finds it accepting requests again. */
+export function quotaExhausted(text) {
+    if (fatalProviderError(text))
+        return false;
+    if (/no available accounts?/i.test(text))
+        return true;
+    // A request rate limit clears in seconds ("rate limit exceeded; resets in 1 second", "quota exceeded for requests
+    // per minute"): pi's retries and the lost-execution path handle it; it must not take the provider out for minutes.
+    if (/rate.?limit|too many requests|request limit|per (second|minute)|\b[RT]PM\b|resets? in \d+ ?(ms|s|secs?|seconds?|minutes?)\b/i.test(text))
+        return false;
+    return /usage limit|quota (exceeded|exhausted)|exceeded your (current )?(usage|quota)|limit (reached|exceeded)[^.]*resets?\b|额度/i.test(text);
 }
 /** A refusal of the request's content (terms of service, usage or content policy): the same request is refused again,
  *  on this provider and usually on another, so it is reported at once instead of retried as a lost execution. */
@@ -119,8 +134,16 @@ export function refusedByProvider(text) {
         return false;
     return /terms of service|usage polic(y|ies)|acceptable use|content[_ ]?(policy|filter|management policy)|safety (system|filter)|flagged as (unsafe|harmful)/i.test(text);
 }
-/** P13, C8: Restore the effective provider from the native session's model changes. */
+/** P13, C8: Restore the effective provider as pi does: from the last model change or assistant message. */
 export function sessionModel(entries) {
-    const last = entries.findLast(e => e.type === "model_change" && e.provider && e.modelId);
-    return last ? { provider: last.provider, id: last.modelId } : undefined;
+    // pi restores the model of the last model change or assistant message: a relaunch with `--model` on an existing
+    // session records no model change, so only the answer tells which model the session went on with.
+    const last = entries.findLast(e => e.type === "model_change" && e.provider && e.modelId
+        || e.type === "message" && e.message?.role === "assistant" && !!e.message.provider && !!e.message.model);
+    if (!last)
+        return undefined;
+    if (last.type === "model_change")
+        return { provider: last.provider, id: last.modelId };
+    const m = last.message;
+    return { provider: m.provider, id: m.model };
 }

package/dist/orchestrator/providers.js

@@ -0,0 +1,21 @@
+/** Apply one orchestrator ledger entry to the map of used-up providers. */
+export function foldExhaustion(exhausted, e) {
+    const provider = String(e.provider ?? e.pool ?? "");
+    if (e.type === "provider-exhausted") {
+        // A refusal of the probe ends it; any other execution's (the executor records none while a probe runs) keeps it.
+        const probe = exhausted.get(provider)?.probe;
+        exhausted.set(provider, { since: Number(e.since), nextTry: Number(e.nextTry), error: String(e.error ?? ""), ...(probe && probe !== e.exec ? { probe } : {}) });
+    }
+    else if (e.type === "provider-available")
+        exhausted.delete(provider);
+    else if (e.type === "provider-probe") {
+        const x = exhausted.get(provider);
+        if (x)
+            x.probe = String(e.exec);
+    }
+    else if (e.type === "release") {
+        const x = exhausted.get(provider);
+        if (x && x.probe === e.exec)
+            delete x.probe;
+    }
+}

package/dist/orchestrator/snapshot.js

@@ -6,6 +6,7 @@ import { compileFanout } from "../compat/fanout.js";
 import { readJournalSnapshot } from "../kernel/journal.js";
 import { journalPath, orchLedger, pinnedDir, workflowDir } from "../paths.js";
 import { JT } from "../types.js";
+import { foldExhaustion } from "./providers.js";
 /** A workflow has live work: it runs, or follow-ups opened on it after it finished have not ended yet. */
 export function isLive(wf) {
     return wf.status === "running" || (wf.followUps ?? 0) > 0;
@@ -469,7 +470,7 @@ const latestCalls = (wf) => [...new Map(wf.calls.map(c => [c.key, c])).values()]
 /** Provider slots from the orchestrator ledger: holders per provider (hold/release{pool,slot,exec}) and the limits of
  *  the settings in effect (the latest config{hash,config}); config-rejected after it is reported too. */
 export function slotsView(home, now = Date.now()) {
-    const held = new Map();
+    const held = new Map(), used = new Map();
     let config, rejected;
     for (const e of readJournalSnapshot(orchLedger(home))) {
         if (e.type === "hold")
@@ -482,7 +483,10 @@ export function slotsView(home, now = Date.now()) {
         }
         else if (e.type === "config-rejected")
             rejected = e;
+        foldExhaustion(used, e);
     }
+    const exhausted = [...used].sort(([a], [b]) => a.localeCompare(b)).map(([p, x]) => `${p} exhausted since ${age(now - x.since)} ago (${clip(x.error, 80)}), ` +
+        (x.probe ? `probing with ${x.probe.split("#")[0]}` : x.nextTry > now ? `next try in ${age(x.nextTry - now)}` : "next call probes it"));
     const limits = (config?.config?.providers) ?? {};
     const holders = new Map();
     for (const e of held.values())
@@ -491,6 +495,7 @@ export function slotsView(home, now = Date.now()) {
     const names = [...new Set([...Object.keys(limits), ...holders.keys()])].sort();
     const slots = names.map(p => { const n = holders.get(p) ?? 0, limit = limits[p]?.slots; return typeof limit === "number" ? `${p} ${n}/${limit}` : `${p} ${n} (no limit)`; });
     return { ...(slots.length ? { slots } : {}), ...(config ? { config: `${String(config.hash)} since ${age(now - config.ts)} ago` } : {}),
+        ...(exhausted.length ? { exhausted } : {}),
         ...(rejected ? { configRejected: `${clip(String(rejected.error), 200)} (${age(now - rejected.ts)} ago); ${config ? String(config.hash) : "the start settings"} stay in effect` } : {}) };
 }
 /** Tool status without a wid: what runs, what waits for an answer and what failed, with finished workflows one line each.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-durable-subagents",
-  "version": "1.0.9",
+  "version": "1.0.10",
   "description": "Subagents for pi that never lose work and never do it twice. Crash-safe workflows, automatic recovery, and a live view just like the main agent.",
   "type": "module",
   "license": "MIT",