pi-braid 0.43.60 → 0.44.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -200,6 +200,86 @@ tarball did not have. All three move here, with the fixes they already hold:
200
200
  instead of throwing out of a timer and ending the process.
201
201
  - `pi-braid-contrib` 0.1.5: the git tool, argv-only and read-only by default.
202
202
 
203
+ ## Core 0.44.4, Telegram 0.2.0, Contrib 0.3.0 — a message names the process that sent it (2026-11-15)
204
+
205
+ Two agents writing through one bot were indistinguishable in the owner's chat, and a reply reaches whichever
206
+ process holds the poller. Every message the Telegram package sends now opens with one line rendered from what
207
+ the sending process can observe, `[braid · worker · pi-braid · session 01a1da43 · pid 98675]`: the role
208
+ marker, the working directory's name, the short session id and the pid. It is attached in `planOutbound`, the
209
+ boundary every owner-facing message already crosses, so a new send path cannot forget it and no caller can
210
+ label a message as another process's. An unobservable field says so and a marker the build does not know is
211
+ reported raw rather than upgraded to a role the code does not have; a body split into parts carries the line
212
+ once, in the first part.
213
+
214
+ The same gate closed a hole it had opened for itself: `rich:true` or a bare `parse_mode` exempted any body
215
+ from the 280-character rule, so a wall of prose could ship past it by asking for the format instead of using
216
+ it. Past the cap a sender-facing body must now carry structure a reader can see, and the refusal names the
217
+ length, the rule and the remedy. Automatic sources keep their delivery, because a refused crafted summary
218
+ has no sender to rewrite it.
219
+
220
+ Three smaller changes landed in the same window. The tracked ack is keyed by chat and resolves only when the
221
+ last holder ends, so two seats cannot both decide the chat has none; the adherence meter folds crossings
222
+ into at most one digest per hour and states how many it suppressed. A forwarded message carries its origin
223
+ now, read into the four kinds with the fields each actually carries, and two clocks are kept apart:
224
+ `forwardedAt` is arrival here, `originDate` is publication there, and `timeSource` says which one a consumer
225
+ holds, so a hand-copied message cannot inherit a forwarded timestamp. The `t.me` link is rebuilt from ids
226
+ only, with the `/c/` form for a channel-range id, and no link at all rather than a guess. And
227
+ `scripts/check-telegram-senders.mjs` walks the repository, the skills tree and the orchestrator's scripts,
228
+ comparing every file that can reach the Bot API with an allowlist where each row states why the file exists
229
+ and which gate it passes; an unlisted sender exits non-zero and names the file.
230
+
231
+ The journal checker stopped calling correct records fabrications: a supersedes target in a rotated sibling
232
+ was DANGLING and an evidence path recorded in another repository's coordinates was ABSENT. A seq now
233
+ resolves across the live file and its rotated siblings through one path helper the writer and the checker
234
+ both import, and a reference whose first component is not this tree's is UNRESOLVED and informational,
235
+ naming the root tried and the raw string given. Every run ends with one line counting findings by class, so
236
+ a checker that got quieter instead of more honest shows up as a shifted count.
237
+
238
+ Telegram moves a minor because a delivered body gains a line the caller did not author and the gate's answer
239
+ to a rich body over the cap flips from delivery to refusal. Contrib moves a minor for the same shape: its
240
+ network monitor's probe URL can no longer name a Bot API call or carry a bot token, since a configurable
241
+ endpoint is the one way a file that posts nothing can post, and the refusal is reported through the settings
242
+ error channel. Core stays a patch: the checker fix and the census script are not things a caller calls, and
243
+ `braid.schema.json` is regenerated with only its `x-source` line references moved.
244
+
245
+ Core ships at 0.44.4, and the three numbers before it are why. The 0.44.1 tag's own workflow never got past
246
+ its gates: the census sweeps three roots, two of which live under the operator's home, and it read a root a
247
+ runner does not have as a row that stopped being true, so both home rows came back stale on every CI machine
248
+ while passing on the machine that wrote them. Presence is now decided once per root, a row under an absent
249
+ root is reported as not checked rather than clean or stale, and `test-telegram-senders-census.mjs` holds both
250
+ halves: an absent root leaves its rows alone, while a root that IS present and holds a listed file that
251
+ reaches nothing still fails. That fixture also moved the suite's own census, which `LOCAL_DEV.md` states as
252
+ a number, so 0.44.2 was recorded and superseded before it could be published. Then 0.44.3's run died on that
253
+ same fixture's last arm, which asserted the operator's home instead of what every machine shares: the arm
254
+ now states the property it was meant to hold, that the shipped tree passes wherever the roots are. The tags
255
+ for 0.44.1 and 0.44.3 stay where they are, each with its failed run beside it.
256
+
257
+ ## Core 0.44.0, Contrib 0.2.0, Providers 1.0.8 — the ruler gains eyes, and refusals learn to speak (2026-11-12)
258
+
259
+ `design_audit` could measure a rendered page and say nothing about whether it looked cheap. The same
260
+ adapter's full-page PNGs now go to the model the session resolves for vision, one call per width, in
261
+ `visual_judge` beside the audit. The result names the model that answered, its route, the tokens and the
262
+ elapsed time, because an opinion with no author is not a report; the default rubric asks for named
263
+ observations and forbids scores, since a judge that answers 7/10 has measured nothing.
264
+
265
+ The generic half stayed in `lib/ai`, where the retry ladder, the pre-send budget and the key-ring
266
+ feedback already live. `callLlm` gained an optional image part and `callVision` is the named entry that
267
+ refuses a text-only model or a malformed part before a request is built, so an image is priced at the
268
+ budget where the text estimator had counted it as free. `resolveVisionModel` and `searchVisionModel`
269
+ share one walk, so a refusal prints every rung it passed over and why it was skipped.
270
+
271
+ A test-harness defect fell out on the way, and it was the kind that hides: contrib's fixtures import core as
272
+ `pi-braid/lib`, which in a checkout resolves through `node_modules` to whichever pi-braid is installed
273
+ rather than the tree under test, so a core export added in the same tree was invisible and the failure read
274
+ as a broken import. The shared hook maps that specifier to this tree's build, and an arm compares the
275
+ build's mtime against the newest core source, because a hook that reads an artifact can otherwise report a
276
+ green suite about code the tree no longer contains. Touching a source without rebuilding reddens it;
277
+ removing the export from the artifact reddens it naming the export; reverting the image part fails one arm
278
+ of twelve.
279
+
280
+ Contrib moves a minor because a new tool is something a caller notices, core because `pi-braid/lib` gained
281
+ five names; providers carries the rotation census at 1.0.8 and is unchanged here.
282
+
203
283
  ## Core — a prefix is ranked, never qualifying on its own (2026-11-11)
204
284
 
205
285
  The same defect as the class noun, one leg over: `improve` starts `improvement`, so the token-prefix leg
package/build-info.json CHANGED
@@ -1,9 +1,9 @@
1
1
  {
2
2
  "name": "pi-braid",
3
- "version": "0.43.60",
4
- "sha": "3cb04fa6bd3910a6db7f89ed536b045237920875",
5
- "shaShort": "3cb04fa6",
3
+ "version": "0.44.4",
4
+ "sha": "867ae9adcf06547838bf83c7c8dcf43e290eda3f",
5
+ "shaShort": "867ae9ad",
6
6
  "branch": "detached",
7
7
  "dirty": false,
8
- "packedAt": "2026-11-12T13:57:51.351Z"
8
+ "packedAt": "2026-11-15T15:38:22.037Z"
9
9
  }
@@ -7,10 +7,13 @@
7
7
  * family `rule` but no whisper source; there is no durable per-rule
8
8
  * violation record, so this is the only honest count and it is
9
9
  * current-session only.
10
- * 2. On turn_end, aggregate the active branch and escalate a rule that
11
- * repeats past K to the owner over the existing TG signal seam
12
- * (`WATCHDOG_TG_SIGNAL_KEY`, the same route the watchdog page uses),
13
- * bounded by the milestone ladder and the per-session budget.
10
+ * 2. On turn_end, aggregate the active branch and report a rule that repeats
11
+ * past K to the owner over the existing TG signal seam
12
+ * (`WATCHDOG_TG_SIGNAL_KEY`, the same route the watchdog page uses), as at
13
+ * most ONE digest per hour that states how many crossings it folded. The
14
+ * milestone ladder and the per-session budget are unchanged: a crossing is
15
+ * still planned by `planEscalation` and its rung is still spent only when
16
+ * the digest was accepted.
14
17
  * 3. On session shutdown, append the owner report
15
18
  * (`~/.pi/agent/braid/adherence/report.jsonl`).
16
19
  *
@@ -19,19 +22,47 @@
19
22
  */
20
23
  import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent";
21
24
  import { type AdherenceWriteResult } from "../../../lib/enforcement/adherence-meter.js";
25
+ /** What a crossing may ask the owner channel for. */
26
+ export interface AdherenceOwnerSender {
27
+ (text: string, opts?: {
28
+ parseMode?: "HTML" | "Markdown";
29
+ source?: string;
30
+ /** Escalations folded into this message instead of sent one by one. */
31
+ suppressed?: number;
32
+ }): boolean | Promise<boolean>;
33
+ }
22
34
  /** Injectables for fixtures; production callers pass nothing. */
23
35
  export interface AdherenceWiringDeps {
24
36
  now?: () => number;
25
37
  /** Owner-channel factory; defaults to the published TG signal sender. */
26
- sender?: () => ((text: string) => boolean | Promise<boolean>) | null;
38
+ sender?: () => AdherenceOwnerSender | null;
39
+ /**
40
+ * True when THIS session is the one that reads the owner's chat. Defaults to
41
+ * the Telegram poller's live state (the seat that owns the poller is the seat
42
+ * whose escalations the owner must hear). A second seat shares the bot token
43
+ * but not the conversation, and its escalations never become owner messages;
44
+ * a spawned seat registers no Telegram surface at all, so it publishes no
45
+ * sender either.
46
+ */
47
+ ownerSession?: () => boolean;
48
+ /** How often the owner may receive a digest (default one hour). */
49
+ digestWindowMs?: number;
27
50
  reportDir?: string;
28
51
  k?: number;
29
52
  }
30
53
  /**
31
- * Aggregate the current branch and escalate every rule that crosses its next
32
- * milestone. Returns what fired. The rung is marked only after the owner
33
- * channel accepted the message, so a missing/failing channel retries on the
34
- * next turn instead of silently spending the rung.
54
+ * Aggregate the current branch and fold every crossing into the owner's
55
+ * hourly digest.
56
+ *
57
+ * What did NOT change: the ladder. `planEscalation` is asked exactly as before,
58
+ * and a rung is still spent only when the owner channel accepted the message
59
+ * that reported it, so the milestones, thresholds and detection are untouched.
60
+ *
61
+ * What changed: a crossing is buffered (`recordEscalation`) and delivered as
62
+ * ONE digest per window (`planDigest`), which states how many crossings it
63
+ * folded instead of sending each one as its own message. Two seats sharing a
64
+ * bot token no longer both write into the owner's chat: only the session that
65
+ * owns the inbound channel escalates.
35
66
  */
36
67
  export declare function checkAdherence(ctx: ExtensionContext, deps?: AdherenceWiringDeps): Promise<{
37
68
  fired: Array<{
@@ -7,10 +7,13 @@
7
7
  * family `rule` but no whisper source; there is no durable per-rule
8
8
  * violation record, so this is the only honest count and it is
9
9
  * current-session only.
10
- * 2. On turn_end, aggregate the active branch and escalate a rule that
11
- * repeats past K to the owner over the existing TG signal seam
12
- * (`WATCHDOG_TG_SIGNAL_KEY`, the same route the watchdog page uses),
13
- * bounded by the milestone ladder and the per-session budget.
10
+ * 2. On turn_end, aggregate the active branch and report a rule that repeats
11
+ * past K to the owner over the existing TG signal seam
12
+ * (`WATCHDOG_TG_SIGNAL_KEY`, the same route the watchdog page uses), as at
13
+ * most ONE digest per hour that states how many crossings it folded. The
14
+ * milestone ladder and the per-session budget are unchanged: a crossing is
15
+ * still planned by `planEscalation` and its rung is still spent only when
16
+ * the digest was accepted.
14
17
  * 3. On session shutdown, append the owner report
15
18
  * (`~/.pi/agent/braid/adherence/report.jsonl`).
16
19
  *
@@ -24,11 +27,21 @@ import { registerTurnEndHandler } from "../../../lib/events/turn-end-coordinator
24
27
  import { registerSessionStartHandler, SessionPhase, } from "../../../lib/state/session-start-queue.js";
25
28
  import { getExtensionState, getSessionIdSafe, safeSessionManagerTyped, } from "../../../lib/state/extension-state.js";
26
29
  import { getWatchdogTgSignalSender } from "../../../lib/state/global-state.js";
30
+ import { readTelegramPollerLiveness } from "../../../lib/state/telegram-poller-liveness.js";
27
31
  import { formatLocalTime } from "../../../lib/utils/local-time.js";
28
- import { activeBranchEntries, aggregateRuleAdherence, createEscalationState, markEscalated, planEscalation, readSessionEntriesFromFile, writeAdherenceReport, } from "../../../lib/enforcement/adherence-meter.js";
32
+ import { activeBranchEntries, adherenceReportPath, aggregateRuleAdherence, commitDigest, createEscalationState, planDigest, planEscalation, readSessionEntriesFromFile, recordEscalation, writeAdherenceReport, } from "../../../lib/enforcement/adherence-meter.js";
29
33
  const log = createLogger("adherence-meter");
30
34
  /** Extension-state bucket key for the per-session meter state. */
31
35
  const ADHERENCE_STATE_KEY = "adherence-meter";
36
+ /**
37
+ * Owner-facing session predicate. The liveness probe is core's own registry
38
+ * (`telegram-poller-liveness.ts`): "running" or "unknown" means no other seat
39
+ * is holding the owner's inbound channel, "stopped" means this process loaded
40
+ * Telegram and its poller is not the one reading the chat.
41
+ */
42
+ function isOwnerChannelSession() {
43
+ return readTelegramPollerLiveness() !== "stopped";
44
+ }
32
45
  function adherenceState(ctx) {
33
46
  return getExtensionState(ctx, ADHERENCE_STATE_KEY, createEscalationState);
34
47
  }
@@ -42,54 +55,121 @@ function branchEntries(ctx) {
42
55
  return [];
43
56
  }
44
57
  }
45
- /** Compact owner ping: headline rung, source + signal, counters, provenance. */
46
- function renderEscalation(rule, milestone, state, ctx, nowMs) {
47
- const signal = rule.signal.length > 140 ? `${rule.signal.slice(0, 137)}...` : rule.signal;
48
- const sessionId = getSessionIdSafe(ctx);
49
- return (`[adherence] rule repeating · rung ${milestone}\n` +
50
- `${rule.source}: "${signal}"\n` +
51
- `\n` +
52
- `deliveries ${rule.deliveries} · repeats ${rule.repeats} · suppressed ${rule.suppressions} · escalation ${state.total + 1}\n` +
53
- `session ${sessionId} · ${formatLocalTime(new Date(nowMs))}`);
58
+ /** Escape the three characters Telegram's HTML parser reserves. */
59
+ function escapeHtml(text) {
60
+ return text.replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;");
61
+ }
62
+ function shorten(text, max) {
63
+ const trimmed = text.trim();
64
+ return trimmed.length <= max ? trimmed : `${trimmed.slice(0, max - 3)}...`;
65
+ }
66
+ /** The rule's matched text (the key's `source::pattern` tail), shortened. */
67
+ function ruleLabel(entry) {
68
+ const tail = entry.ruleKey.includes("::")
69
+ ? entry.ruleKey.split("::").slice(1).join("::")
70
+ : entry.ruleKey;
71
+ return shorten(`${entry.source}: ${tail}`, 64);
54
72
  }
55
73
  /**
56
- * Aggregate the current branch and escalate every rule that crosses its next
57
- * milestone. Returns what fired. The rung is marked only after the owner
58
- * channel accepted the message, so a missing/failing channel retries on the
59
- * next turn instead of silently spending the rung.
74
+ * The hourly digest: a bold subject naming the highest rung it reports, then
75
+ * labelled lines. Short enough to survive as plain text (the owner's
76
+ * tweet-length rule) and valid under `parse_mode: HTML` (the Telegram outbound
77
+ * gate validates it before any send).
78
+ */
79
+ function renderDigest(digest, ctx, nowMs) {
80
+ // The subject is the highest rung in the digest; ties go to the latest.
81
+ const subject = digest.pending.reduce((best, entry) => entry.milestone >= best.milestone ? entry : best);
82
+ const lines = [
83
+ `<b>adherence · rung ${subject.milestone}</b>`,
84
+ `rule ${escapeHtml(ruleLabel(subject))}`,
85
+ ...(digest.pending.length > 1 ? [`rules ${digest.pending.length}`] : []),
86
+ `deliveries ${subject.deliveries}`,
87
+ `repeats ${subject.repeats}`,
88
+ `suppressed ${digest.suppressed}`,
89
+ `session ${escapeHtml(shorten(getSessionIdSafe(ctx) || "unknown", 32))}`,
90
+ `at ${formatLocalTime(new Date(nowMs))}`,
91
+ // The next step is a pointer, not advice: the full signal text and the
92
+ // per-rule counts live in the session report.
93
+ `where ${escapeHtml(shorten(adherenceReportPath(), 72))}`,
94
+ ];
95
+ return lines.join("\n");
96
+ }
97
+ /**
98
+ * Aggregate the current branch and fold every crossing into the owner's
99
+ * hourly digest.
100
+ *
101
+ * What did NOT change: the ladder. `planEscalation` is asked exactly as before,
102
+ * and a rung is still spent only when the owner channel accepted the message
103
+ * that reported it, so the milestones, thresholds and detection are untouched.
104
+ *
105
+ * What changed: a crossing is buffered (`recordEscalation`) and delivered as
106
+ * ONE digest per window (`planDigest`), which states how many crossings it
107
+ * folded instead of sending each one as its own message. Two seats sharing a
108
+ * bot token no longer both write into the owner's chat: only the session that
109
+ * owns the inbound channel escalates.
60
110
  */
61
111
  export async function checkAdherence(ctx, deps = {}) {
62
112
  const state = adherenceState(ctx);
63
113
  const entries = branchEntries(ctx);
64
- if (entries.length === 0)
65
- return { fired: [] };
66
- const rules = aggregateRuleAdherence(activeBranchEntries(entries));
114
+ const rules = entries.length > 0 ? aggregateRuleAdherence(activeBranchEntries(entries)) : [];
67
115
  const sender = (deps.sender ?? getWatchdogTgSignalSender)();
68
116
  const nowMs = (deps.now ?? Date.now)();
69
- const fired = [];
117
+ const sessionId = getSessionIdSafe(ctx);
70
118
  for (const rule of rules) {
71
119
  const plan = planEscalation(state, rule.rule, rule.repeats);
72
120
  if (!plan.fire)
73
121
  continue;
74
- if (!sender) {
75
- log.debug(`[adherence] ${rule.rule} crossed rung ${plan.milestone}; no owner channel published`);
76
- continue;
77
- }
78
- let ok = false;
79
- try {
80
- ok =
81
- (await Promise.resolve(sender(renderEscalation(rule, plan.milestone, state, ctx, nowMs)))) === true;
82
- }
83
- catch (e) {
84
- log.debug(`[adherence] escalation send threw for ${rule.rule}: ${e.message}`);
85
- }
86
- if (!ok) {
87
- log.debug(`[adherence] escalation not delivered for ${rule.rule}; rung stays unspent`);
88
- continue;
89
- }
90
- markEscalated(state, rule.rule);
91
- fired.push({ rule: rule.rule, milestone: plan.milestone });
122
+ recordEscalation(state, {
123
+ ruleKey: rule.rule,
124
+ rung: (state.rules[rule.rule]?.rung ?? 0) + 1,
125
+ milestone: plan.milestone,
126
+ source: rule.source,
127
+ deliveries: rule.deliveries,
128
+ repeats: rule.repeats,
129
+ sessionId,
130
+ at: nowMs,
131
+ });
132
+ }
133
+ const buffered = state.pending ?? [];
134
+ if (buffered.length === 0)
135
+ return { fired: [] };
136
+ if (!sender) {
137
+ log.debug(`[adherence] ${buffered.length} escalation(s) buffered; no owner channel published`);
138
+ return { fired: [] };
139
+ }
140
+ if (!(deps.ownerSession ?? isOwnerChannelSession)()) {
141
+ log.debug(`[adherence] ${buffered.length} escalation(s) buffered; this session does not own the owner channel`);
142
+ return { fired: [] };
143
+ }
144
+ const digest = planDigest(state, nowMs, {
145
+ ...(deps.digestWindowMs !== undefined ? { windowMs: deps.digestWindowMs } : {}),
146
+ });
147
+ if (!digest.send) {
148
+ log.debug(`[adherence] ${digest.pending.length} escalation(s) buffered; digest window closed (${digest.suppressed} suppressed so far)`);
149
+ return { fired: [] };
150
+ }
151
+ let ok = false;
152
+ try {
153
+ ok =
154
+ (await Promise.resolve(sender(renderDigest(digest, ctx, nowMs), {
155
+ parseMode: "HTML",
156
+ source: "adherence-digest",
157
+ suppressed: digest.suppressed,
158
+ }))) === true;
159
+ }
160
+ catch (e) {
161
+ log.debug(`[adherence] digest send threw: ${e.message}`);
162
+ }
163
+ if (!ok) {
164
+ log.debug("[adherence] digest not delivered; crossings stay buffered, rungs unspent");
165
+ return { fired: [] };
92
166
  }
167
+ const fired = digest.pending.map((entry) => ({
168
+ rule: entry.ruleKey,
169
+ milestone: entry.milestone,
170
+ }));
171
+ commitDigest(state, nowMs);
172
+ log.debug(`[adherence] digest delivered: ${fired.length} crossing(s), ${digest.suppressed} suppressed`);
93
173
  return { fired };
94
174
  }
95
175
  /**
@@ -28,6 +28,11 @@
28
28
  * callLlm(model, apiKey, systemPrompt, userPrompt, options)
29
29
  * Returns { text } on success or { error } on failure (discriminated union).
30
30
  * const text = result.text;
31
+ *
32
+ * callVision(model, apiKey, systemPrompt, instruction, images, options)
33
+ * The same union with image parts attached to the user message. A model
34
+ * that declares no image input is refused before anything is sent, so a
35
+ * text-only model can never quietly answer about an image it never saw.
31
36
  */
32
37
  import type { Api, AssistantMessage, Context, Model, ModelThinkingLevel, ModelsSimpleStreamOptions } from "@earendil-works/pi-ai";
33
38
  /** One provider-neutral completion call. */
@@ -48,6 +53,32 @@ export declare function __resetRateLimitCooldownForTest(): void;
48
53
  import { setLlmCallListeners, type LlmErrorListener, type LlmSuccessListener } from "../state/global-state.js";
49
54
  export { setLlmCallListeners };
50
55
  export type { LlmErrorListener, LlmSuccessListener };
56
+ /**
57
+ * One image attached to an auxiliary request: base64 bytes and the MIME type
58
+ * they carry. `data` is the payload itself, without a `data:` prefix.
59
+ */
60
+ export interface LlmImagePart {
61
+ /** Base64 payload. */
62
+ data: string;
63
+ /** Declared MIME type, e.g. "image/png". The transport passes it through. */
64
+ mimeType: string;
65
+ }
66
+ /**
67
+ * What an auxiliary call returns: the text (with the stop reason and usage the
68
+ * provider reported) or one line naming why it failed. A vision call returns
69
+ * the same shape, so a caller never branches on which kind of call it made.
70
+ */
71
+ export type LlmCallResult = {
72
+ text: string;
73
+ stopReason?: string;
74
+ usage?: {
75
+ input?: number;
76
+ output?: number;
77
+ totalTokens?: number;
78
+ };
79
+ } | {
80
+ error: string;
81
+ };
51
82
  export interface LlmCallOptions {
52
83
  /** Temperature for generation. Default: 0.1 */
53
84
  temperature?: number;
@@ -85,6 +116,16 @@ export interface LlmCallOptions {
85
116
  sdkRetries?: number;
86
117
  /** Optional AbortSignal for cancellation */
87
118
  signal?: AbortSignal;
119
+ /**
120
+ * Image parts attached to the user message. `callVision` is the entry point
121
+ * that sets this: a text call leaves it out and its user content stays a
122
+ * plain string, so the request shape of every existing caller is unchanged.
123
+ *
124
+ * The model's declared input is checked here as well as in `callVision`,
125
+ * because the transport would otherwise drop the image silently or return a
126
+ * provider error that reads like a transient failure.
127
+ */
128
+ images?: readonly LlmImagePart[];
88
129
  }
89
130
  /**
90
131
  * Make an LLM call with two-tier retry, backoff, and rate-limit circuit breaker.
@@ -98,17 +139,25 @@ export interface LlmCallOptions {
98
139
  * @param options - Optional configuration
99
140
  * @returns `{ text }` on success or `{ error }` on failure
100
141
  */
101
- export declare function callLlm(model: Model<Api>, apiKey: string | undefined, systemPrompt: string, userPrompt: string, options?: LlmCallOptions): Promise<{
102
- text: string;
103
- stopReason?: string;
104
- usage?: {
105
- input?: number;
106
- output?: number;
107
- totalTokens?: number;
108
- };
109
- } | {
110
- error: string;
111
- }>;
142
+ export declare function callLlm(model: Model<Api>, apiKey: string | undefined, systemPrompt: string, userPrompt: string, options?: LlmCallOptions): Promise<LlmCallResult>;
143
+ /**
144
+ * Make an auxiliary call that carries images: the same dispatch, retry ladder,
145
+ * circuit breaker, pre-send budget and key-ring feedback as `callLlm`, with the
146
+ * images attached to the user message.
147
+ *
148
+ * Why this is a wrapper and not a second dispatcher: every policy the text
149
+ * path carries (per-model concurrency, transient retry with the backoff
150
+ * ceiling, the 413 emergency offload, the listener and key-ring seams) applies
151
+ * to a vision call exactly as it applies to a text one. A copy would drift from
152
+ * that policy at the first fix made in one place, and a vision call that does
153
+ * not feed the key ring back is a route that keeps failing without rotating.
154
+ * The image-specific decisions - is this model image-capable, is this part a
155
+ * real image - are the only ones the wrapper owns.
156
+ *
157
+ * `instruction` is the user turn; `images` are attached after it, in order, so
158
+ * a caller can refer to "the first image" and mean what it sent.
159
+ */
160
+ export declare function callVision(model: Model<Api>, apiKey: string | undefined, systemPrompt: string, instruction: string, images: readonly LlmImagePart[], options?: Omit<LlmCallOptions, "images">): Promise<LlmCallResult>;
112
161
  /**
113
162
  * The one yes/no question core asks the providers seam before re-using a
114
163
  * (provider, model) pair for an auxiliary call: can this pair serve a request
@@ -28,6 +28,11 @@
28
28
  * callLlm(model, apiKey, systemPrompt, userPrompt, options)
29
29
  * Returns { text } on success or { error } on failure (discriminated union).
30
30
  * const text = result.text;
31
+ *
32
+ * callVision(model, apiKey, systemPrompt, instruction, images, options)
33
+ * The same union with image parts attached to the user message. A model
34
+ * that declares no image input is refused before anything is sent, so a
35
+ * text-only model can never quietly answer about an image it never saw.
31
36
  */
32
37
  // Degraded transport only: see completeAuxCall below. Compat resolves the
33
38
  // credential from the model and the apiKey argument and nothing else, which is
@@ -231,6 +236,13 @@ export async function callLlm(model, apiKey, systemPrompt, userPrompt, options =
231
236
  log.debug(`Skipping LLM call - rate-limit cooldown ${remaining}ms remaining`);
232
237
  return { error: "Rate-limit cooldown active" };
233
238
  }
239
+ // Images: a request the model cannot read is refused before it is built,
240
+ // not sent and reported as a provider error (or, worse, answered from the
241
+ // text alone with no complaint about the missing picture).
242
+ const images = options.images ?? [];
243
+ const imageRefusal = refuseImages(model, images);
244
+ if (imageRefusal !== null)
245
+ return { error: imageRefusal };
234
246
  const maxRetries = options.maxRetries ?? DEFAULT_MAX_RETRIES;
235
247
  const maxTokens = options.maxTokens ?? Math.min(Math.max(1000, Math.floor(userPrompt.length / 8)), 4000);
236
248
  // 10-minute default (was 30s). Aborts should be signal-driven (ESC,
@@ -254,7 +266,9 @@ export async function callLlm(model, apiKey, systemPrompt, userPrompt, options =
254
266
  }
255
267
  const context = {
256
268
  systemPrompt: systemSafe.text,
257
- messages: [{ role: "user", content: userSafe.text, timestamp: Date.now() }],
269
+ messages: [
270
+ { role: "user", content: userContent(userSafe.text, images), timestamp: Date.now() },
271
+ ],
258
272
  };
259
273
  // ── pre-send budget ──
260
274
  // A request over the window cannot succeed, and a provider 400 is a late,
@@ -265,7 +279,8 @@ export async function callLlm(model, apiKey, systemPrompt, userPrompt, options =
265
279
  // provider will refuse what the cap already refused.
266
280
  const resolvedWindow = resolveWindow(model.contextWindow);
267
281
  const budget = await guardAgainstDoomedSend({
268
- measure: () => calibrateEstimate(estimatePayloadTokens({ systemPrompt: context.systemPrompt, messages: context.messages })).tokens,
282
+ measure: () => calibrateEstimate(estimatePayloadTokens({ systemPrompt: context.systemPrompt, messages: context.messages })).tokens +
283
+ images.length * IMAGE_TOKEN_ALLOWANCE,
269
284
  outputTokens: maxTokens,
270
285
  contextWindow: resolvedWindow.contextWindow,
271
286
  // The only compaction available to an auxiliary call is the published
@@ -544,7 +559,94 @@ export async function callLlm(model, apiKey, systemPrompt, userPrompt, options =
544
559
  releaseModelSlot(model.id);
545
560
  }
546
561
  }
562
+ // ── Vision ────────────────────────────────────────────────────────────────────
563
+ /**
564
+ * Make an auxiliary call that carries images: the same dispatch, retry ladder,
565
+ * circuit breaker, pre-send budget and key-ring feedback as `callLlm`, with the
566
+ * images attached to the user message.
567
+ *
568
+ * Why this is a wrapper and not a second dispatcher: every policy the text
569
+ * path carries (per-model concurrency, transient retry with the backoff
570
+ * ceiling, the 413 emergency offload, the listener and key-ring seams) applies
571
+ * to a vision call exactly as it applies to a text one. A copy would drift from
572
+ * that policy at the first fix made in one place, and a vision call that does
573
+ * not feed the key ring back is a route that keeps failing without rotating.
574
+ * The image-specific decisions - is this model image-capable, is this part a
575
+ * real image - are the only ones the wrapper owns.
576
+ *
577
+ * `instruction` is the user turn; `images` are attached after it, in order, so
578
+ * a caller can refer to "the first image" and mean what it sent.
579
+ */
580
+ export async function callVision(model, apiKey, systemPrompt, instruction, images, options = {}) {
581
+ if (images.length === 0) {
582
+ return {
583
+ error: "Image input refused: a vision call needs at least one image part. " +
584
+ "Use callLlm for a text-only request.",
585
+ };
586
+ }
587
+ return callLlm(model, apiKey, systemPrompt, instruction, { ...options, images });
588
+ }
547
589
  // ── Helpers ────────────────────────────────────────────────────────────────────
590
+ /**
591
+ * Tokens the pre-send budget counts per attached image.
592
+ *
593
+ * The payload estimator reads text only: an image block contributes its
594
+ * `mimeType` and nothing of its `data` (see estimateMessageTokens in
595
+ * detection/context-usage.ts), so without this allowance an image-heavy
596
+ * request looks free and the guard waves through a payload the provider then
597
+ * rejects. An image is priced by its pixels, not its bytes, and a full-page
598
+ * capture is a few thousand tokens on every provider that accepts one: OpenAI
599
+ * at high detail tiles a 1440x3000 capture into 18 tiles (~3.2K tokens),
600
+ * Anthropic's pixel rule puts the same image near 1.6K after its own downscale.
601
+ * This constant is the conservative ceiling of that range, so the estimate errs
602
+ * toward refusing rather than toward a doomed send.
603
+ */
604
+ const IMAGE_TOKEN_ALLOWANCE = 4_000;
605
+ /**
606
+ * The one place that decides whether a request carrying images may be built.
607
+ * Null means yes. A refusal names the model, what it declares, and the offending
608
+ * part, because the caller's next move (pick another model, re-encode the
609
+ * image) depends on which of those failed.
610
+ */
611
+ function refuseImages(model, images) {
612
+ if (images.length === 0)
613
+ return null;
614
+ const declared = model.input ?? [];
615
+ if (!declared.includes("image")) {
616
+ return (`Image input refused: ${model.provider ?? "unknown"}/${model.id} declares input ` +
617
+ `[${declared.join(", ") || "text"}] and does not accept images. A vision call needs an ` +
618
+ `image-capable model; resolveVisionModel names one for the current session.`);
619
+ }
620
+ for (let index = 0; index < images.length; index++) {
621
+ const image = images[index];
622
+ const position = `Image part ${index + 1} of ${images.length}`;
623
+ if (typeof image.data !== "string" || image.data.length === 0) {
624
+ return `${position} carries no data. Expected base64 image bytes, not an empty string.`;
625
+ }
626
+ if (typeof image.mimeType !== "string" || !/^image\/[a-z0-9.+-]+$/i.test(image.mimeType)) {
627
+ return (`${position} declares no image MIME type (got ${JSON.stringify(image.mimeType)}). ` +
628
+ `Expected a type such as "image/png".`);
629
+ }
630
+ }
631
+ return null;
632
+ }
633
+ /**
634
+ * The user message content: a plain string for a text call, text-then-images for
635
+ * a vision call. The text block comes first so an instruction that says "the
636
+ * image below" reads in the order it was written.
637
+ */
638
+ function userContent(text, images) {
639
+ if (images.length === 0)
640
+ return text;
641
+ return [
642
+ { type: "text", text },
643
+ ...images.map((image) => ({
644
+ type: "image",
645
+ data: image.data,
646
+ mimeType: image.mimeType,
647
+ })),
648
+ ];
649
+ }
548
650
  /**
549
651
  * Run the published emergency offloader (context-manager's
550
652
  * context-emergency.ts) against the current session context. Lossy-safe:
@@ -27,7 +27,23 @@ export declare function normaliseRoleSetting(raw: unknown): string[];
27
27
  */
28
28
  export declare function resolveUtilityModels(ctx: ExtensionContext): Model<Api>[];
29
29
  /**
30
- * The model a vision call should use.
30
+ * What a vision-model resolution found and what it passed over.
31
+ *
32
+ * `resolveVisionModel` answers "which model"; a caller that reports to a human
33
+ * also has to answer "and then what should I change", which needs the rungs
34
+ * that produced nothing. Both come from one walk, so the model a report names
35
+ * and the trace beside it cannot disagree.
36
+ */
37
+ export interface VisionModelSearch {
38
+ /** The model a vision call should use, or undefined when the walk found none. */
39
+ model?: Model<Api>;
40
+ /** Which rung supplied `model`; null when none did. */
41
+ source: "current" | "utilityModel" | "available" | null;
42
+ /** One line per rung, in the order tried, saying what it found or why not. */
43
+ tried: string[];
44
+ }
45
+ /**
46
+ * The model a vision call should use, with the walk that produced it.
31
47
  *
32
48
  * Priority:
33
49
  * 1. The current model when it accepts images: the image stays in the call
@@ -38,6 +54,10 @@ export declare function resolveUtilityModels(ctx: ExtensionContext): Model<Api>[
38
54
  *
39
55
  * A text-only entry is never selected for an image call.
40
56
  */
57
+ export declare function searchVisionModel(ctx: ExtensionContext): VisionModelSearch;
58
+ /**
59
+ * The model a vision call should use: the `model` of the walk above.
60
+ */
41
61
  export declare function resolveVisionModel(ctx: ExtensionContext): Model<Api> | undefined;
42
62
  /**
43
63
  * Get the configured utility model for classification tasks.