agentfences 0.7.0 → 0.9.0

package/README.md

@@ -30,7 +30,7 @@ const research = governed(async (question: string) => {
 
 Pass the model's response to `checkpoint()` and Fences works out what the call cost from its model and token usage, including cache reads and writes. It understands results from the OpenAI and Anthropic SDKs, the Vercel AI SDK (`generateText`), Gemini and LangChain.js, or any object with the same usage fields. Prices come from a built-in table of about 290 models (the same one as the Python SDK).
 
-- Unknown model: its tokens are counted and Fences warns once. Add prices in USD per 1M tokens: `init({ localOnly: true, prices: { "my-finetune": { input: 2, output: 6 } } })`.
+- Unknown model: its tokens are counted and Fences warns once. Add prices in USD per 1M tokens: `init({ localOnly: true, prices: { "my-finetune": { input: 2, output: 6 } } })`. Both prices are required; an invalid one throws at `init()`.
 - Extra costs, or no response at all: `checkpoint(response, { costDeltaUsd: 0.01 })` or `checkpoint(null, { costDeltaUsd: 0.02, tokensUsed: 450 })`.
 
 ## Limits
@@ -47,7 +47,7 @@ const webSearch = tool(async ({ query }: { query: string }) => search(query));
 await context({ user_id: "u_123", session_id: "s_9" }, () => research(question));  // tags every run inside
 ```
 
-- Tool and model calls are recorded as OpenTelemetry-shaped spans (`execute_tool webSearch`, `gen_ai.*` attributes), on `getActiveRun().events`.
+- Tool and model calls are recorded as OpenTelemetry-shaped spans (`execute_tool webSearch`, `gen_ai.*` attributes), on `getActiveRun().events`. A tool call carries its argument names (`query, limit`), not their values; `init({ toolArguments: "full" })` records the values as JSON (300 characters at most), `"none"` neither (from 0.9.0). A guarded tool's check and an approval request still send the values.
 - When a governed run throws, Fences records the error's type, message and stack with the run; the error still reaches your code.
 - `init({ redact: (event) => ... })` sees everything before it leaves the process and can change or drop it.
 
@@ -57,7 +57,7 @@ await context({ user_id: "u_123", session_id: "s_9" }, () => research(question))
 init({ apiKey: process.env.FENCES_API_KEY, endpoint: "https://your-fences-host", environment: "prod", release: "v1.4.2" });
 ```
 
-The server is authoritative for limits and sees spend across processes. If it can't be reached, limits are enforced locally; pass `failClosed: true` to stop instead. A rejected key throws `FencesAuthError`; a quarantined agent throws `AgentQuarantined` before any of its code runs. In serverless handlers, `await flush()` before returning so queued decisions and spans are sent.
+The server is authoritative for limits and sees spend across processes. If it can't be reached, limits are enforced locally; pass `failClosed: true` to stop instead. A rejected key throws `FencesAuthError` when the run starts, and a key revoked during a run ends it at its next checkpoint (`fences_unreachable`); a quarantined agent throws `AgentQuarantined` before any of its code runs. A cost that isn't a finite number of zero or more throws instead of being counted. In serverless handlers, `await flush()` before returning so queued decisions and spans are sent.
 
 ## Live control (from 0.4.0)
 
@@ -117,6 +117,23 @@ mcp.wrap(client, { guard: ["send_email"], costUsd: { refund: (args) => args.amou
 
 `guard: true` guards every tool, and `approve: [...]` names tools a person approves first. A result with `isError` is recorded as a failed call and still comes back to the agent.
 
+## Model calls (from 0.8.0)
+
+`instrument(client)` records every OpenAI and Anthropic call made inside a governed run (model, tokens, cost, latency, errors) and checks it against the budget **before** it's sent:
+
+```ts
+import OpenAI from "openai";
+import { instrument } from "agentfences";
+
+const openai = instrument(new OpenAI());  // one call covers every client of that SDK
+```
+
+- If a call's worst case could take the run past its budget, it rejects with `FencesStop` (`budget_exceeded`) and never reaches the provider. The worst case is the request's input, estimated high, plus its output cap: `max_tokens` (or `max_completion_tokens`, `max_output_tokens`) if set, otherwise the model's whole output limit, so **set `max_tokens` on calls in runs with small budgets**.
+- Each call's worst case is held while it's in flight, so parallel calls can't overspend together. A failed call is released; a finished one is replaced by its actual cost, and the next `checkpoint()` releases it. With `onBreach: "hold"`, a call that doesn't fit waits for someone to raise the budget.
+- Calls are recorded, not counted as steps: keep calling `checkpoint(response)`, which adds the cost and checks the other limits.
+- The SDK's own promise comes back, with `.withResponse()` and `.asResponse()`. After `.asResponse()` the call is recorded without usage, and its worst case stays held until the next checkpoint.
+- OpenAI: `chat.completions.create` and `responses.create`, including OpenAI-compatible clients. Anthropic: `messages.create`. Streams are recorded when they start; pass the final usage to `checkpoint()`.
+
 ## Not yet in the TypeScript SDK
 
-The Python SDK also records OpenAI/Anthropic calls automatically (`instrument`) and integrates with LangGraph and the OpenAI Agents SDK. Those are also how it checks a model call against the budget **before** it's sent, so the TypeScript SDK doesn't do that yet: a model call is checked afterwards, by `checkpoint()`, and a run can go over its budget by one call. Everything here follows the same server protocol ([PROTOCOL.md](../PROTOCOL.md)).
+The Python SDK also integrates with LangGraph and the OpenAI Agents SDK. Model calls made without `instrument(client)` are checked afterwards, by `checkpoint()`, so a run can go over its budget by one such call. Everything here follows the same server protocol ([PROTOCOL.md](../PROTOCOL.md)).

package/dist/approvals.d.ts

@@ -0,0 +1,24 @@
+export interface Approval {
+    granted: boolean;
+    /** Who answered, as the dashboard reports it. */
+    by?: string | null;
+    /** Their note, or why no one could answer. */
+    note?: string | null;
+    /** The amount approved, if one was asked for. */
+    amountUsd?: number | null;
+    approvalId?: string | null;
+}
+/**
+ * Ask a person, through the Fences dashboard, to approve something, and wait for the answer:
+ *
+ *     const approval = await requestApproval("Refund $240 to order 1182?", { amountUsd: 240 });
+ *     if (!approval.granted) return `I didn't issue the refund: ${approval.note}`;
+ *
+ * If an amount is approved, the run's budget goes up by that much. Returns denied right away in
+ * local mode, outside a run, or when the server can't take requests, and denied after timeoutS
+ * (default 600) with no answer: it never hangs and never throws.
+ */
+export declare function requestApproval(reason: string, options?: {
+    amountUsd?: number;
+    timeoutS?: number;
+}): Promise<Approval>;

package/dist/approvals.js

@@ -0,0 +1,57 @@
+/** Approval requests: ask a person, through the Fences dashboard, and wait for the answer. */
+import { randomUUID } from "node:crypto";
+import * as control from "./control.js";
+import { recordSpan } from "./core.js";
+import { getActiveRun } from "./run.js";
+/**
+ * Ask a person, through the Fences dashboard, to approve something, and wait for the answer:
+ *
+ *     const approval = await requestApproval("Refund $240 to order 1182?", { amountUsd: 240 });
+ *     if (!approval.granted) return `I didn't issue the refund: ${approval.note}`;
+ *
+ * If an amount is approved, the run's budget goes up by that much. Returns denied right away in
+ * local mode, outside a run, or when the server can't take requests, and denied after timeoutS
+ * (default 600) with no answer: it never hangs and never throws.
+ */
+export async function requestApproval(reason, options = {}) {
+    const run = getActiveRun(), client = control._client();
+    const timeoutS = options.timeoutS ?? 600, amountUsd = options.amountUsd ?? null;
+    if (!run)
+        return { granted: false, note: "requestApproval() was called outside a governed run" };
+    if (!client)
+        return { granted: false, note: "Local mode: there is no one to approve this request" };
+    if (!control.supported.approvals)
+        return { granted: false, note: "This Fences server doesn't support approvals" };
+    const approvalId = randomUUID(), started = performance.now();
+    // timeout_s lets the server expire the request when we stop waiting
+    const resp = await client.requestApproval(run.runId, { approval_id: approvalId, reason, amount_usd: amountUsd, timeout_s: timeoutS });
+    if (resp.unsupported) {
+        control.unsupported("approvals", "approval requests", "they are denied");
+        return { granted: false, note: "This Fences server doesn't support approvals" };
+    }
+    if (resp.network_error)
+        return { granted: false, note: "Couldn't reach Fences to ask for approval" };
+    if (resp.ok === false)
+        return { granted: false, note: `Fences didn't accept the approval request (HTTP ${resp.status})` }; // no one will be asked
+    run.control.approvals.set(approvalId, null);
+    run.control.awaiting += 1;
+    control.schedule(0); // poll fast while someone decides
+    const deadline = Date.now() + timeoutS * 1000;
+    while (run.control.approvals.get(approvalId) === null && Date.now() < deadline) {
+        await new Promise((ok) => setTimeout(ok, 100));
+    }
+    run.control.awaiting -= 1;
+    const c = run.control.approvals.get(approvalId); // the dashboard's approval command, or null: no answer in time
+    run.control.approvals.delete(approvalId);
+    const answer = c
+        ? { granted: !!c.granted, by: c.by ?? null, note: c.note ?? null, amountUsd: c.amount_usd ?? null, approvalId }
+        : { granted: false, note: "No answer before the timeout", approvalId };
+    if (answer.granted && answer.amountUsd && run.budgetUsd !== null)
+        run.budgetUsd += answer.amountUsd; // an unlimited run stays unlimited
+    recordSpan("fences.approval", performance.now() - started, {
+        "fences.approval.id": approvalId, "fences.approval.reason": reason, "fences.approval.amount_usd": amountUsd,
+        "fences.approval.granted": answer.granted, "fences.approval.by": answer.by, "fences.approval.note": answer.note,
+        "fences.approval.granted_usd": answer.amountUsd,
+    });
+    return answer;
+}

package/dist/client.d.ts

@@ -1,10 +1,28 @@
 export type Reply = Record<string, any>;
+export declare const FAILURES_TO_OPEN = 3;
+/**
+ * A circuit breaker: after FAILURES_TO_OPEN failed calls in a row (no connection, a timeout, a 5xx,
+ * or a 429 on anything but the heartbeat), skip calls for about 30 seconds, then let one through to test the server. While it's open
+ * a call fails at once instead of after the timeout: checkpoints are enforced locally and guarded
+ * tools are blocked, without adding seconds to every step.
+ */
+export declare class Breaker {
+    failures: number;
+    openUntil: number;
+    private probing;
+    allow(): boolean;
+    record(ok: boolean): void;
+}
 export declare class FencesClient {
     readonly endpoint: string;
     private readonly timeoutMs;
     private readonly headers;
+    readonly breaker: Breaker;
+    /** The server echoed a step_id: it counts a repeated step once, so retrying a checkpoint is safe. */
+    dedupesSteps: boolean;
     constructor(apiKey: string, endpoint: string, timeoutMs?: number);
     startRun(body: Record<string, unknown>): Promise<Reply>;
+    /** body.step_id, if set, lets the server count the step once however often it's sent. */
     checkpoint(runId: string, body: Record<string, unknown>): Promise<Reply>;
     logDecision(runId: string, body: Record<string, unknown>): Promise<Reply>;
     endRun(runId: string, body: Record<string, unknown>): Promise<Reply>;
@@ -15,8 +33,12 @@ export declare class FencesClient {
     guard(runId: string, body: Record<string, unknown>): Promise<Reply>;
     requestApproval(runId: string, body: Record<string, unknown>): Promise<Reply>;
     /**
-     * optional: an endpoint older servers may not have. A missing route returns
-     * { unsupported: true } and nothing throws, not even a rejected key.
+     * optional: an endpoint older servers may not have. A missing route returns { unsupported: true },
+     * and any other error { ok: false, status }.
+     * start: a rejected key throws FencesAuthError, before any agent code runs. Later in a run it
+     * returns { network_error, auth_rejected: true }: throwing there would replace the agent's own result or error.
+     * retry: a call that's safe to repeat goes once more after a connection error (not after a timeout,
+     * which would double the wait).
      */
-    post(path: string, payload: unknown, optional?: boolean): Promise<Reply>;
+    post(path: string, payload: unknown, optional?: boolean, start?: boolean, retry?: boolean): Promise<Reply>;
 }

package/dist/client.js

@@ -1,10 +1,55 @@
-/** HTTP calls to a Fences server, as specified in PROTOCOL.md. Uses the fetch built into Node 18+. */
+/**
+ * HTTP calls to a Fences server, as specified in PROTOCOL.md, with the fetch built into Node 18+.
+ * Nothing throws except a rejected key at run start.
+ */
 import { FencesAuthError } from "./errors.js";
 import { VERSION } from "./version.js";
+export const FAILURES_TO_OPEN = 3; // failed calls in a row before the breaker opens
+const OPEN_MS = 30_000; // how long calls are skipped once it's open (with jitter), before one goes through to test
+/**
+ * A circuit breaker: after FAILURES_TO_OPEN failed calls in a row (no connection, a timeout, a 5xx,
+ * or a 429 on anything but the heartbeat), skip calls for about 30 seconds, then let one through to test the server. While it's open
+ * a call fails at once instead of after the timeout: checkpoints are enforced locally and guarded
+ * tools are blocked, without adding seconds to every step.
+ */
+export class Breaker {
+    failures = 0;
+    openUntil = 0;
+    probing = false;
+    allow() {
+        if (this.failures < FAILURES_TO_OPEN)
+            return true;
+        if (this.probing || Date.now() < this.openUntil)
+            return false;
+        this.probing = true; // one test call at a time
+        return true;
+    }
+    record(ok) {
+        const wasOpen = this.failures >= FAILURES_TO_OPEN;
+        this.probing = false;
+        if (ok) {
+            this.failures = 0;
+            if (wasOpen)
+                console.warn("agentfences: Fences is reachable again");
+            return;
+        }
+        this.failures += 1;
+        if (this.failures < FAILURES_TO_OPEN)
+            return;
+        this.openUntil = Date.now() + OPEN_MS * (0.75 + Math.random() / 2); // jitter: SDKs cut off together don't all come back at once
+        if (!wasOpen) {
+            console.warn(`agentfences: Fences failed ${this.failures} calls in a row; skipping calls for about ${OPEN_MS / 1000}s ` +
+                "(limits enforced locally, guarded tools blocked)");
+        }
+    }
+}
 export class FencesClient {
     endpoint;
     timeoutMs;
     headers;
+    breaker = new Breaker();
+    /** The server echoed a step_id: it counts a repeated step once, so retrying a checkpoint is safe. */
+    dedupesSteps = false;
     constructor(apiKey, endpoint, timeoutMs = 3000) {
         this.endpoint = endpoint;
         this.timeoutMs = timeoutMs;
@@ -12,8 +57,14 @@ export class FencesClient {
         // The server uses the SDK version to warn about old SDKs and to skip commands they don't understand
         this.headers = { "Content-Type": "application/json", "X-API-Key": apiKey, "X-Fences-SDK": `js/${VERSION}` };
     }
-    startRun(body) { return this.post("/api/runs/start", body); }
-    checkpoint(runId, body) { return this.post(`/api/runs/${runId}/checkpoint`, body); }
+    startRun(body) { return this.post("/api/runs/start", body, false, true); }
+    /** body.step_id, if set, lets the server count the step once however often it's sent. */
+    async checkpoint(runId, body) {
+        const resp = await this.post(`/api/runs/${runId}/checkpoint`, body, false, false, !!body.step_id && this.dedupesSteps);
+        if (body.step_id && resp.step_id === body.step_id)
+            this.dedupesSteps = true;
+        return resp;
+    }
     logDecision(runId, body) { return this.post(`/api/runs/${runId}/decisions`, body); }
     endRun(runId, body) { return this.post(`/api/runs/${runId}/end`, body); }
     logSpans(runId, spans) { return this.post(`/api/runs/${runId}/spans`, { spans }, true); }
@@ -22,28 +73,54 @@ export class FencesClient {
         return this.post("/api/heartbeat", acks.length ? { runs, acks } : { runs }, true);
     }
     /** Check a guarded tool call before it runs ({ phase: "check" }), or settle it after ({ phase: "settle" }). */
-    guard(runId, body) { return this.post(`/api/runs/${runId}/guard`, body, true); }
+    guard(runId, body) { return this.post(`/api/runs/${runId}/guard`, body, true, false, true); } // guard_id: safe to repeat
     requestApproval(runId, body) { return this.post(`/api/runs/${runId}/approvals`, body, true); }
     /**
-     * optional: an endpoint older servers may not have. A missing route returns
-     * { unsupported: true } and nothing throws, not even a rejected key.
+     * optional: an endpoint older servers may not have. A missing route returns { unsupported: true },
+     * and any other error { ok: false, status }.
+     * start: a rejected key throws FencesAuthError, before any agent code runs. Later in a run it
+     * returns { network_error, auth_rejected: true }: throwing there would replace the agent's own result or error.
+     * retry: a call that's safe to repeat goes once more after a connection error (not after a timeout,
+     * which would double the wait).
      */
-    async post(path, payload, optional = false) {
+    async post(path, payload, optional = false, start = false, retry = false) {
+        if (!this.breaker.allow())
+            return { network_error: "Fences unreachable: calls paused after repeated failures" };
+        let json;
+        try {
+            json = JSON.stringify(payload);
+        }
+        catch (e) { // a payload JSON can't encode: not the server's fault
+            this.breaker.record(true);
+            return { network_error: String(e) };
+        }
+        const send = () => fetch(this.endpoint + path, {
+            method: "POST", headers: this.headers, body: json, signal: AbortSignal.timeout(this.timeoutMs),
+        });
         let resp;
         try {
-            resp = await fetch(this.endpoint + path, {
-                method: "POST", headers: this.headers, body: JSON.stringify(payload), signal: AbortSignal.timeout(this.timeoutMs),
-            });
+            try {
+                resp = await send();
+            }
+            catch (e) {
+                if (!retry || e?.name === "TimeoutError")
+                    throw e;
+                await new Promise((ok) => setTimeout(ok, 100 + Math.random() * 200)); // a blip: once more, after a moment
+                resp = await send();
+            }
         }
         catch (e) {
+            this.breaker.record(false);
             return { network_error: String(e) };
         }
+        // A 429 on the heartbeat is its own budget, not an outage: it mustn't cut off checkpoints
+        this.breaker.record(resp.status < 500 && (resp.status !== 429 || path === "/api/heartbeat"));
         const text = await resp.text().catch(() => "");
         let body = {}, isJson = false;
         try {
             const parsed = JSON.parse(text);
-            if (parsed && typeof parsed === "object")
-                [body, isJson] = [parsed, true];
+            if (parsed && typeof parsed === "object" && !Array.isArray(parsed))
+                [body, isJson] = [parsed, true]; // a list isn't a Fences reply
         }
         catch { /* not JSON: the status code still decides */ }
         const detail = typeof body.detail === "string" ? body.detail : "";
@@ -53,8 +130,11 @@ export class FencesClient {
             }
             return resp.ok ? body : { ok: false, status: resp.status };
         }
-        if (resp.status === 401 || resp.status === 403)
-            throw new FencesAuthError(`Fences API key rejected: ${text}`);
+        if (resp.status === 401 || resp.status === 403) {
+            if (start)
+                throw new FencesAuthError(`Fences API key rejected: ${detail || text.slice(0, 200)}`);
+            return { network_error: `API key rejected (HTTP ${resp.status})`, auth_rejected: true };
+        }
         if (resp.status === 423)
             return { ok: false, quarantined: true, detail }; // the agent is quarantined: the run must not start
         if (resp.status === 409)

package/dist/control.d.ts

@@ -1,15 +1,12 @@
+/**
+ * Live control from the Fences dashboard: a heartbeat that reports live runs and brings back
+ * commands (stop, pause, resume, quarantine, limits, approval answers) and acknowledges them,
+ * containment hooks, pausing at checkpoints, waiting at a limit, and cancelling a run stuck after
+ * a stop. Cloud mode only; a server without these endpoints gets one warning and runs carry
+ * on as before. Same behaviour as the Python SDK (PROTOCOL.md, "Live control").
+ */
 import type { FencesClient, Reply } from "./client.js";
-import type { Run } from "./core.js";
-export interface Approval {
-    granted: boolean;
-    /** Who answered, as the dashboard reports it. */
-    by?: string | null;
-    /** Their note, or why no one could answer. */
-    note?: string | null;
-    /** The amount approved, if one was asked for. */
-    amountUsd?: number | null;
-    approvalId?: string | null;
-}
+import type { Run } from "./run.js";
 /** What onStop and onQuarantine receive. */
 export interface ControlEvent {
     type: "stop" | "quarantine";
@@ -36,13 +33,15 @@ export interface Control {
     quarantined: boolean;
     /** While waiting at a limit: "breach" or "budget". */
     held: "breach" | "budget" | null;
-    /** approval id -> the answer, or null while waiting */
-    approvals: Map<string, Approval | null>;
+    /** approval id -> the dashboard's answer (an approval command), or null while waiting */
+    approvals: Map<string, Reply | null>;
     awaiting: number;
     /** command ids already acknowledged */
     acked: Set<string>;
 }
 export declare const newControl: () => Control;
+/** @internal */
+export declare const _client: () => FencesClient | null;
 export declare let pauseTimeoutMs: number;
 export declare const supported: {
     heartbeat: boolean;
@@ -60,6 +59,19 @@ export declare function configure(c: FencesClient | null, o: ControlOptions): vo
 export declare function register(run: Run): void;
 export declare function unregister(run: Run): void;
 export declare function unsupported(kind: keyof typeof supported, what: string, consequence: string): void;
+export declare const LIMITS: {
+    readonly budget_usd: "budgetUsd";
+    readonly max_iterations: "maxIterations";
+    readonly max_duration_ms: "maxDurationMs";
+    readonly max_tokens: "maxTokens";
+};
+export declare const TOTALS: {
+    readonly spent_usd: "costUsd";
+    readonly iterations: "iterations";
+    readonly tokens_used: "tokensUsed";
+};
+/** Set the run's fields from a server reply: numbers only, so a null or odd value can't break local enforcement. nullBudget: a null budget_usd means unlimited spend. */
+export declare function adopt(run: Run, reply: Reply, fields: Record<string, string>, nullBudget?: boolean): void;
 /** Apply the commands addressed to this run (from a heartbeat or a checkpoint reply), and acknowledge them. Each is safe to apply twice. */
 export declare function apply(run: Run, commands: Reply[] | undefined): void;
 /**
@@ -68,6 +80,8 @@ export declare function apply(run: Run, commands: Reply[] | undefined): void;
  * its containment, and says so.
  */
 export declare function drain(timeoutMs?: number): Promise<void>;
+/** @internal Send a heartbeat after delayMs (0: now), replacing the one scheduled. */
+export declare function schedule(delayMs: number): void;
 /** Breaches a person can lift by raising a limit, so onBreach "hold" waits at them. */
 export declare const HOLDABLE: Set<string>;
 export declare const BUDGET_BREACHES: Set<string>;
@@ -79,17 +93,3 @@ export declare function startHold(run: Run, kind: "breach" | "budget"): void;
 export declare function endHold(run: Run): "released" | "stopped" | "timeout";
 /** Hold the agent at its checkpoint while its run is paused, up to pauseTimeoutS. */
 export declare function waitIfPaused(run: Run): Promise<void>;
-/**
- * Ask a person, through the Fences dashboard, to approve something, and wait for the answer:
- *
- *     const approval = await requestApproval("Refund $240 to order 1182?", { amountUsd: 240 });
- *     if (!approval.granted) return `I didn't issue the refund: ${approval.note}`;
- *
- * If an amount is approved, the run's budget goes up by that much. Returns denied right away in
- * local mode, outside a run, or when the server can't take requests, and denied after timeoutS
- * (default 600) with no answer: it never hangs and never throws.
- */
-export declare function requestApproval(reason: string, options?: {
-    amountUsd?: number;
-    timeoutS?: number;
-}): Promise<Approval>;

package/dist/control.js

@@ -1,21 +1,16 @@
-/**
- * Live control from the Fences dashboard: a heartbeat that reports live runs and brings back
- * commands (stop, pause, resume, quarantine, limits, approval answers) and acknowledges them,
- * containment hooks, pausing at checkpoints, cancelling a run stuck after a stop, and approval
- * requests. Cloud mode only; a server without these endpoints gets one warning and runs carry
- * on as before. Same behaviour as the Python SDK (PROTOCOL.md, "Live control").
- */
-import { randomUUID } from "node:crypto";
 /** Heartbeat statuses whose acks go again with the next heartbeat: rate limited, or a server error. */
 const retryable = (status) => status === undefined || status === 429 || status >= 500;
 const FAST_MS = 2000; // heartbeat interval while a run is paused, waiting for an approval, or stopping
 const HOOK_TIMEOUT_MS = 30_000; // a hook still running after this is reported as timed out
+const MAX_BACKOFF_MS = 60_000; // longest wait between heartbeats after failures (jittered)
 export const newControl = () => ({
     stopped: false, stoppedAt: 0, stopSeen: false, cancelled: false, onCancel: null, paused: false, pauseExpired: false,
     quarantined: false, held: null, approvals: new Map(), awaiting: 0, acked: new Set(),
 });
 const live = new Map(); // run id -> runs in progress
 let client = null;
+/** @internal */
+export const _client = () => client;
 let intervalMs = 15_000;
 export let pauseTimeoutMs = 3_600_000;
 let stopGraceMs = 30_000;
@@ -23,6 +18,7 @@ const hooks = { stop: null, quarantine: null };
 export const supported = { heartbeat: true, approvals: true, guard: true };
 let timer = null;
 let beating = false;
+const backoff = { failed: 0, nextTry: 0 }; // heartbeats failed in a row, and when the next may go
 let acks = []; // acknowledgements waiting for the next heartbeat
 const hooked = new Set(); // quarantine ids whose hook has run in this process
 const runningHooks = new Set();
@@ -34,6 +30,7 @@ export function configure(c, o) {
     hooks.stop = o.onStop ?? null;
     hooks.quarantine = o.onQuarantine ?? null;
     supported.heartbeat = supported.approvals = supported.guard = true;
+    Object.assign(backoff, { failed: 0, nextTry: 0 });
     schedule(0); // a new client or interval takes effect now, not after the old wait
 }
 export function register(run) {
@@ -51,7 +48,17 @@ export function unsupported(kind, what, consequence) {
     console.warn(`agentfences: this Fences server doesn't support ${what} yet, so ${consequence}`);
 }
 // ── Commands ──────────────────────────────────────────────────────────────────
-const LIMITS = { budget_usd: "budgetUsd", max_iterations: "maxIterations", max_duration_ms: "maxDurationMs", max_tokens: "maxTokens" };
+export const LIMITS = { budget_usd: "budgetUsd", max_iterations: "maxIterations", max_duration_ms: "maxDurationMs", max_tokens: "maxTokens" };
+export const TOTALS = { spent_usd: "costUsd", iterations: "iterations", tokens_used: "tokensUsed" };
+/** Set the run's fields from a server reply: numbers only, so a null or odd value can't break local enforcement. nullBudget: a null budget_usd means unlimited spend. */
+export function adopt(run, reply, fields, nullBudget = false) {
+    for (const [wire, field] of Object.entries(fields)) {
+        const v = reply[wire];
+        if ((typeof v === "number" && !Number.isNaN(v)) || (nullBudget && wire === "budget_usd" && v === null)) {
+            run[field] = v; // eslint-disable-line @typescript-eslint/no-explicit-any
+        }
+    }
+}
 /** Apply the commands addressed to this run (from a heartbeat or a checkpoint reply), and acknowledge them. Each is safe to apply twice. */
 export function apply(run, commands) {
     // Limits first: a resume in the same reply may release a call held for budget, which re-checks against them at once
@@ -69,8 +76,10 @@ export function apply(run, commands) {
             ctl.paused = ctl.quarantined = false;
         }
         else if (c.type === "pause" || c.type === "quarantine") { // a quarantined agent's runs hold at their next checkpoint
-            ctl.paused = true;
-            schedule(0);
+            if (!ctl.paused) { // only a change wakes the heartbeat: a quarantine is re-sent on every reply while it lasts
+                ctl.paused = true;
+                schedule(0);
+            }
             if (c.type === "quarantine") {
                 ctl.quarantined = true; // guarded tools are blocked here too, even if the server can't be reached
                 hook("quarantine", run, c);
@@ -80,15 +89,11 @@ export function apply(run, commands) {
             ctl.paused = ctl.quarantined = false; // releasing a quarantine resumes its runs
         }
         else if (c.type === "limits") {
-            for (const [wire, field] of Object.entries(LIMITS)) {
-                if (c[wire] !== undefined && c[wire] !== null)
-                    run[field] = c[wire]; // eslint-disable-line @typescript-eslint/no-explicit-any
-            }
+            adopt(run, c, LIMITS);
         }
         else if (c.type === "approval") {
-            if (ctl.approvals.has(c.approval_id)) {
-                ctl.approvals.set(c.approval_id, { granted: !!c.granted, by: c.by ?? null, note: c.note ?? null, amountUsd: c.amount_usd ?? null });
-            }
+            if (ctl.approvals.has(c.approval_id))
+                ctl.approvals.set(c.approval_id, c); // approvals.ts turns it into the answer
         }
         else {
             continue; // unknown: ignored, and not acknowledged
@@ -174,7 +179,8 @@ if (typeof process?.on === "function") { // edge runtimes have no process events
 const unseenStop = (r) => r.control.stopped && !r.control.stopSeen && !r.control.cancelled && r.control.onCancel !== null;
 const busy = () => [...live.values()].some((r) => r.control.paused || r.control.awaiting > 0 || unseenStop(r));
 const nextDelay = () => (busy() ? Math.min(FAST_MS, intervalMs, stopGraceMs ?? FAST_MS) : intervalMs); // faster while busy, never slower
-function schedule(delayMs) {
+/** @internal Send a heartbeat after delayMs (0: now), replacing the one scheduled. */
+export function schedule(delayMs) {
     if (timer)
         clearTimeout(timer);
     timer = null;
@@ -183,6 +189,13 @@ function schedule(delayMs) {
     timer = setTimeout(beat, delayMs);
     timer.unref(); // the heartbeat never keeps the process alive on its own
 }
+/** After a failed heartbeat: wait longer before the next, doubling from the shortest interval in use, with jitter,
+ *  so SDKs coming back from the same outage don't all beat at once. */
+function backOff() {
+    backoff.failed += 1;
+    const wait = Math.min(Math.min(FAST_MS, intervalMs) * 2 ** (backoff.failed - 1), MAX_BACKOFF_MS);
+    backoff.nextTry = Date.now() + wait * (0.5 + Math.random() / 2);
+}
 /** Halt governed calls that haven't reached a checkpoint within stopGraceS of a stop. */
 function cancelOverdue() {
     if (stopGraceMs === null)
@@ -196,8 +209,8 @@ function cancelOverdue() {
 }
 async function beat() {
     timer = null;
-    cancelOverdue();
-    if (beating || !client || !(live.size || acks.length))
+    cancelOverdue(); // local, so it keeps its schedule while heartbeats back off
+    if (beating || !client || !(live.size || acks.length) || Date.now() < backoff.nextTry)
         return schedule(nextDelay());
     beating = true;
     const runs = [...live.values()], batch = acks;
@@ -207,20 +220,24 @@ async function beat() {
             tokens_used: r.tokensUsed, paused: r.control.paused, stopped: r.control.stopped,
             ...(r.control.held ? { held: r.control.held } : {}) })), batch);
         if (resp.network_error || (resp.ok === false && !resp.unsupported)) {
-            if (resp.network_error || retryable(resp.status))
+            if (resp.network_error || retryable(resp.status)) {
                 acks = [...batch, ...acks]; // at least once; any other 4xx won't get better
+                backOff();
+            }
         }
         else if (resp.unsupported) {
             unsupported("heartbeat", "the heartbeat", "a stop from the dashboard applies at the next checkpoint and pause isn't available");
             return;
         }
         else {
+            Object.assign(backoff, { failed: 0, nextTry: 0 });
             for (const run of runs)
                 apply(run, resp.commands?.filter((c) => c.run_id === run.runId));
         }
     }
     catch {
         acks = [...batch, ...acks]; // a failed heartbeat is retried next time
+        backOff();
     }
     finally {
         beating = false;
@@ -266,53 +283,3 @@ export async function waitIfPaused(run) {
         await new Promise((ok) => setTimeout(ok, 200));
     }
 }
-// ── Approvals ─────────────────────────────────────────────────────────────────
-/**
- * Ask a person, through the Fences dashboard, to approve something, and wait for the answer:
- *
- *     const approval = await requestApproval("Refund $240 to order 1182?", { amountUsd: 240 });
- *     if (!approval.granted) return `I didn't issue the refund: ${approval.note}`;
- *
- * If an amount is approved, the run's budget goes up by that much. Returns denied right away in
- * local mode, outside a run, or when the server can't take requests, and denied after timeoutS
- * (default 600) with no answer: it never hangs and never throws.
- */
-export async function requestApproval(reason, options = {}) {
-    const { getActiveRun, recordSpan } = await import("./core.js");
-    const run = getActiveRun();
-    const timeoutS = options.timeoutS ?? 600, amountUsd = options.amountUsd ?? null;
-    if (!run)
-        return { granted: false, note: "requestApproval() was called outside a governed run" };
-    if (!client)
-        return { granted: false, note: "Local mode: there is no one to approve this request" };
-    if (!supported.approvals)
-        return { granted: false, note: "This Fences server doesn't support approvals" };
-    const approvalId = randomUUID(), started = performance.now();
-    // timeout_s lets the server expire the request when we stop waiting
-    const resp = await client.requestApproval(run.runId, { approval_id: approvalId, reason, amount_usd: amountUsd, timeout_s: timeoutS });
-    if (resp.unsupported) {
-        unsupported("approvals", "approval requests", "they are denied");
-        return { granted: false, note: "This Fences server doesn't support approvals" };
-    }
-    if (resp.network_error)
-        return { granted: false, note: "Couldn't reach Fences to ask for approval" };
-    run.control.approvals.set(approvalId, null);
-    run.control.awaiting += 1;
-    schedule(0); // poll fast while someone decides
-    const deadline = Date.now() + timeoutS * 1000;
-    while (run.control.approvals.get(approvalId) === null && Date.now() < deadline) {
-        await new Promise((ok) => setTimeout(ok, 100));
-    }
-    run.control.awaiting -= 1;
-    const answer = run.control.approvals.get(approvalId) ?? { granted: false, note: "No answer before the timeout" };
-    answer.approvalId = approvalId;
-    run.control.approvals.delete(approvalId);
-    if (answer.granted && answer.amountUsd && run.budgetUsd !== null)
-        run.budgetUsd += answer.amountUsd; // an unlimited run stays unlimited
-    recordSpan("fences.approval", performance.now() - started, {
-        "fences.approval.id": approvalId, "fences.approval.reason": reason, "fences.approval.amount_usd": amountUsd,
-        "fences.approval.granted": answer.granted, "fences.approval.by": answer.by, "fences.approval.note": answer.note,
-        "fences.approval.granted_usd": answer.amountUsd,
-    });
-    return answer;
-}