pi-advisor-flow 0.3.6 → 0.4.0

package/CHANGELOG.md

@@ -4,6 +4,21 @@ All notable changes to this project are documented here.
 
 The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 0.4.0
+
+### Added
+
+- Exposed provider-reported Advisor and Scout token/cache/cost details per result, tracked cumulative direct Advisor usage in the session footer and optional summary, and attached normalized `ask_advisor` usage to Pi's built-in cost totals without double-counting manual consultations or automatic gates.
+
+### Removed
+
+- Removed the repository-only Benchmark suite and its benchmark-specific telemetry instrumentation.
+
+### Fixed
+
+- Kept Scout's serialized manifest limit separate from the Advisor conversation budget so required context does not fall back solely because of manifest metadata overhead.
+- Made unlimited Advisor call budgets live at every enforcement boundary, preventing stale finite limits from triggering false Herdr budget notifications after switching to unlimited.
+
 ## 0.3.6
 
 ### Fixed

package/README.md

@@ -21,6 +21,7 @@ The idea is simple: keep implementation on a fast model and borrow frontier reas
 - **Configurable review gates** before plans, after repeated failures, and before declaring completion.
 - **Automatic loop detection** for repeated tool calls, with explicit proceed, revise, or blocked decisions.
 - **Separate model and reasoning controls** for the Executor and Advisor.
+- **Advisor usage accounting** with per-response token/cost details and cumulative direct usage in the Pi footer and session summary.
 - **Privacy controls** for conversation history, repository context, explicit tracked/untracked file handoff, tool results, secret redaction, and outcome logging.
 - **Optional persistent activation, Simple mode, session summaries, and Herdr integration.**
 - **EXPERIMENTAL Advisor Scout** that uses the configured Executor model to curate conversation evidence before every Advisor call.
@@ -55,7 +56,7 @@ Unknown fields in `advisor.json` are preserved for forward compatibility and rep
 You can also enable the flow and select both models at once:
 
 ```text
-/advisor executor=anthropic/claude-sonnet-5 advisor=openai/gpt-5.6-sol
+/advisor executor=openai-codex/gpt-5.6-luna advisor=openai-codex/gpt-5.6-sol
 ```
 
 ## How it works
@@ -68,6 +69,8 @@ You can also enable the flow and select both models at once:
 
 A normal consultation never blocks execution. The optional automatic loop gate is different: it evaluates repeated tool calls and applies the configured failure policy when the Advisor says to revise, reports a block, is unavailable, or returns an invalid decision.
 
+Advisor responses show provider-reported input, output, cache, and cost details when available. Successful `ask_advisor` tool results also carry normalized usage into Pi's built-in `Tools/summaries` and `/cost` totals. Manual consultations and automatic gates remain in the separate session-local direct Advisor accounting because they are custom messages, so they are not double-counted in Pi's Executor totals. Missing or partial provider usage is shown as unavailable rather than fabricated as zero usage.
+
 Successful calls return an opaque `adviceId`. If global outcome logging is enabled, the Executor can call `record_advisor_outcome` once to record whether the advice was adopted and whether final validation passed.
 
 ### Experimental Advisor Scout
@@ -76,7 +79,7 @@ Experimental Advisor Scout is off by default. Enable `Experimental Advisor Scout
 
 Scout runs before `ask_advisor`, `/advisor-manual`, and automatic Advisor gates. It uses the configured Executor model and Executor reasoning effort in a separate model call. This adds cost and latency, but can reduce cost in the Advisor call. The compact result shows the model, selection counts, and elapsed time; `Ctrl+O` shows bounded selected labels and the synthesis.
 
-Scout receives a bounded manifest of conversation and tool-history groups after the normal tool disclosure, result-cap, and redaction policies are applied. The manifest and reconstructed Scout conversation share the Advisor's remaining context budget after repository context; a zero remaining budget produces no history groups. For a pending `ask_advisor` call, Scout receives only the allowlisted question and Git-context preference, never the draft or explicit attachment paths. Scout does not receive the deterministic Git context, draft, project preferences, or explicit tracked and untracked attachments. Those regions are appended later through their existing consent and cap rules.
+Scout receives a bounded manifest of conversation and tool-history groups after the normal tool disclosure, result-cap, and redaction policies are applied. The Scout manifest has its own fixed transport limit, while the reconstructed conversation remains bounded by the Advisor's remaining context budget after repository context; manifest metadata no longer consumes that Advisor conversation budget. A zero remaining budget produces no history groups. For a pending `ask_advisor` call, Scout receives only the allowlisted question and Git-context preference, never the draft or explicit attachment paths. Scout does not receive the deterministic Git context, draft, project preferences, or explicit tracked and untracked attachments. Those regions are appended later through their existing consent and cap rules.
 
 This experiment adapts the context-boundary idea from Zhang et al., ["FastContext: Training Efficient Repository Explorer for Coding Agents"](https://arxiv.org/html/2606.14066v1). It is not a reproduction of FastContext. pi-advisor Scout curates conversation history only.
 

package/extensions/index.ts

@@ -2,7 +2,6 @@ import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
 import { registerCommands } from "../src/commands.js";
 import { setHerdrBlockedEmitter } from "../src/herdr.js";
 import { AdvisorSessionState } from "../src/session-state.js";
-import { createBenchmarkTelemetry } from "../src/telemetry.js";
 import {
   consultAdvisor as consultAdvisorImplementation,
   parseAutomaticDecision as parseAutomaticDecisionImplementation,
@@ -37,20 +36,11 @@ export default function (pi: ExtensionAPI) {
   setHerdrBlockedEmitter((active, label) =>
     pi.events.emit("herdr:blocked", { active, label })
   );
-  const benchmarkTelemetry = createBenchmarkTelemetry(pi.events);
-  if (benchmarkTelemetry) {
-    // Diagnostics only: never rewrite provider payloads or affect production sessions.
-    pi.on("before_provider_request", (event) => {
-      benchmarkTelemetry.providerRequest(event.payload);
-    });
-  }
   registerAdvisorTool(pi, sessionState, {
     statusManager: scoutStatus,
-    telemetry: benchmarkTelemetry,
   });
   registerCommands(pi, {
     sessionState,
     statusManager: scoutStatus,
-    telemetry: benchmarkTelemetry,
   });
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pi-advisor-flow",
-  "version": "0.3.6",
+  "version": "0.4.0",
   "description": "Advanced Executor/Advisor flow for Pi, fully configurable and extendable.",
   "keywords": [
     "pi-package",
@@ -28,8 +28,6 @@
     "README.md"
   ],
   "scripts": {
-    "benchmark": "bun benchmarks/src/cli.ts",
-    "benchmark:report": "bun benchmarks/src/cli.ts report",
     "format": "bunx ultracite fix --linter-enabled=false",
     "lint": "bunx ultracite check",
     "lint:fix": "bunx ultracite fix",

package/src/commands.ts

@@ -6,12 +6,12 @@ import {
 import { Box, Markdown, Text } from "@earendil-works/pi-tui";
 import {
   advisorEffortRef,
-  advisorMaxCallsPerSessionRef,
   advisorRef,
   alwaysOnRef,
   contextMaxCharsRef,
   executorEffortRef,
   executorRef,
+  getAdvisorMaxCallsPerSession,
   getAdvisorSettings,
   isSimpleMode,
   loadConfig,
@@ -52,7 +52,6 @@ import {
 import { herdrAdvisorActivity, notifyHerdrAdvisorFailure } from "./herdr.js";
 import type { ScoutLifecycleEvent } from "./scout.js";
 import type { AdvisorSessionState } from "./session-state.js";
-import type { BenchmarkTelemetry } from "./telemetry.js";
 import {
   adviceForDisplay,
   appendScoutLifecycleEntry,
@@ -70,6 +69,11 @@ import {
   type ContextPreset,
   SearchableModelSelector,
 } from "./ui.js";
+import {
+  advisorUsageCost,
+  formatAdvisorUsage,
+  snapshotAdvisorUsage,
+} from "./usage.js";
 
 const DEFAULT_EFFORT_LEVEL = "Default (Model Default)";
 const SELECTED_PREFIX = "✓ ";
@@ -145,6 +149,7 @@ type ManualConsult = (
   thinkingText: string;
   draftBytes?: number;
   preferenceBytes?: number;
+  usage?: unknown;
 }>;
 type ThinkingLevel = Parameters<ExtensionAPI["setThinkingLevel"]>[0];
 
@@ -169,7 +174,6 @@ export const registerCommands = (
     consult?: ManualConsult;
     sessionState?: AdvisorSessionState;
     statusManager?: ScoutStatusManager;
-    telemetry?: BenchmarkTelemetry;
   } = {}
 ) => {
   const advisorSessionState =
@@ -190,10 +194,14 @@ export const registerCommands = (
         undefined,
         undefined,
         onScout,
-        undefined,
-        dependencies.telemetry
+        undefined
       ));
   const manualConsultations = new Map<AbortController, symbol>();
+  const updateAdvisorUsageStatus = (ctx: ExtensionContext) => {
+    if (ctx.hasUI) {
+      ctx.ui.setStatus("advisor-usage", advisorSessionState.usageStatus());
+    }
+  };
   const setManualStatus = (
     ctx: ExtensionContext,
     controller: AbortController,
@@ -257,21 +265,30 @@ export const registerCommands = (
         }
       }
     )
-      .then(({ markdown }) => {
+      .then(({ markdown, usage }) => {
         if (controller.signal.aborted) {
           return;
         }
         advisorSessionState.recordInvocation({
+          cost: advisorUsageCost(usage),
           executionEffect: "continued",
           kind: "markdown",
           model: advisorRef,
           trigger: "manual",
+          usage,
         });
+        updateAdvisorUsageStatus(ctx);
+        const normalizedUsage = snapshotAdvisorUsage(usage);
         pi.sendMessage(
           {
             content: `Manual Advisor consultation${question ? ` (${question})` : ""}:\n\n${markdown}`,
             customType: "advisor-manual-result",
-            details: { advisor: advisorRef, question, text: markdown },
+            details: {
+              advisor: advisorRef,
+              question,
+              text: markdown,
+              ...(normalizedUsage ? { usage: normalizedUsage } : {}),
+            },
             display: true,
           },
           {
@@ -294,6 +311,7 @@ export const registerCommands = (
           model: advisorRef,
           trigger: "manual",
         });
+        updateAdvisorUsageStatus(ctx);
         pi.sendMessage(
           {
             content: `Manual Advisor consultation failed: ${message}`,
@@ -424,7 +442,7 @@ export const registerCommands = (
     "advisor-manual-result",
     (message, { expanded }, theme) => {
       const details = message.details as
-        | { advisor?: string; text?: string }
+        | { advisor?: string; text?: string; usage?: unknown }
         | undefined;
       const box = new Box(1, 1, (text) => theme.bg("customMessageBg", text));
       const advice =
@@ -443,6 +461,10 @@ export const registerCommands = (
       if (details?.advisor) {
         box.addChild(new Text(theme.fg("dim", `  ${details.advisor}`), 0, 0));
       }
+      const usage = formatAdvisorUsage(details?.usage);
+      if (usage) {
+        box.addChild(new Text(theme.fg("dim", `  Usage: ${usage}`), 0, 0));
+      }
       box.addChild(
         new Markdown(
           adviceForDisplay(advice, expanded),
@@ -486,6 +508,7 @@ export const registerCommands = (
   pi.on("session_shutdown", (_event, ctx) => {
     if (ctx.hasUI) {
       ctx.ui.setStatus("advisor-manual", undefined);
+      ctx.ui.setStatus("advisor-usage", undefined);
     }
     for (const [controller, token] of manualConsultations) {
       controller.abort();
@@ -506,7 +529,7 @@ export const registerCommands = (
       if (
         !(
           isSimpleMode() ||
-          advisorSessionState.canConsult(advisorMaxCallsPerSessionRef)
+          advisorSessionState.canConsult(getAdvisorMaxCallsPerSession())
         )
       ) {
         const message = "Advisor call budget exhausted for this session.";

package/src/config.ts

@@ -13,8 +13,8 @@ import {
   isValidGitContextLevel,
 } from "./git.js";
 
-export const FALLBACK_EXECUTOR = "aikeys/claude-sonnet-5";
-export const FALLBACK_ADVISOR = "aikeys/claude-fable-5";
+export const FALLBACK_EXECUTOR = "openai-codex/gpt-5.6-luna";
+export const FALLBACK_ADVISOR = "openai-codex/gpt-5.6-sol";
 export const DEFAULT_CONTEXT_MAX_CHARS = 15_000;
 export const MAX_CONTEXT_MAX_CHARS = Number.MAX_SAFE_INTEGER;
 export const DEFAULT_ADVISOR_TOOL_RESULT_MAX_LINES = DEFAULT_MAX_LINES;
@@ -208,9 +208,13 @@ export const getAdvisorSettings = () => ({
   untrackedContent: advisorUntrackedContentRef,
 });
 
+/** Reads the live session budget at each enforcement boundary. */
+export const getAdvisorMaxCallsPerSession = () =>
+  getAdvisorSettings().maxCallsPerSession;
+
 export const splitRef = (ref: string): [string, string] => {
   const i = ref.indexOf("/");
-  return i === -1 ? ["aikeys", ref] : [ref.slice(0, i), ref.slice(i + 1)];
+  return i === -1 ? ["openai-codex", ref] : [ref.slice(0, i), ref.slice(i + 1)];
 };
 
 export const configPaths = (ctx: ExtensionContext) => [

package/src/scout-context.ts

@@ -169,9 +169,14 @@ const pendingInvocationDisclosure = (
 
 export interface BuildScoutManifestOptions {
   currentInvocationId?: string;
+  /** @deprecated Use maxManifestBytes for the Scout transport budget. */
   maxBytes?: number;
+  /** Maximum reconstructed Advisor conversation characters. */
+  maxConversationChars?: number;
   maxGroupBytes?: number;
   maxGroups?: number;
+  /** Maximum serialized group-manifest bytes sent to Scout. */
+  maxManifestBytes?: number;
   policies?: AdvisorToolPolicies;
   redact?: boolean;
   toolResultMaxBytes?: number;
@@ -191,9 +196,13 @@ export const buildScoutManifest = (
     options.toolResultMaxLines ?? advisorToolResultMaxLinesRef;
   const toolResultMaxBytes =
     options.toolResultMaxBytes ?? advisorToolResultMaxBytesRef;
-  const maxBytes = options.maxBytes ?? SCOUT_MANIFEST_MAX_BYTES;
-  const maxGroups = options.maxGroups ?? SCOUT_MANIFEST_MAX_GROUPS;
-  const maxGroupBytes = options.maxGroupBytes ?? SCOUT_GROUP_MAX_BYTES;
+  const {
+    maxBytes,
+    maxConversationChars,
+    maxGroupBytes = SCOUT_GROUP_MAX_BYTES,
+    maxGroups = SCOUT_MANIFEST_MAX_GROUPS,
+    maxManifestBytes = maxBytes ?? SCOUT_MANIFEST_MAX_BYTES,
+  } = options;
 
   let latestUserIndex = -1;
   const callOwners = new Map<string, { index: number; name: string }>();
@@ -423,7 +432,7 @@ export const buildScoutManifest = (
   const availableBytes =
     groups.reduce((sum, group) => sum + groupWireBytes(group), 0) +
     protocolOmittedBytes;
-  if (maxBytes <= 0) {
+  if (maxManifestBytes <= 0) {
     return {
       manifest: {
         availableBytes: 0,
@@ -439,22 +448,39 @@ export const buildScoutManifest = (
   if (
     required.some((group) => group.bytes > maxGroupBytes) ||
     required.length > maxGroups ||
-    required.reduce((sum, group) => sum + groupWireBytes(group), 0) > maxBytes
+    required.reduce((sum, group) => sum + groupWireBytes(group), 0) >
+      maxManifestBytes
   ) {
     return {
       message:
-        "Required Scout context exceeds a fixed manifest or per-group limit.",
+        "Required Scout context exceeds the Scout manifest transport limit.",
+      ok: false,
+      reason: "required-group-overflow",
+    };
+  }
+  const contentChars = (items: ScoutContextGroup[]) =>
+    items.reduce((sum, group) => sum + group.content.length, 0) +
+    Math.max(0, items.length - 1) * 2;
+  if (
+    maxConversationChars !== undefined &&
+    contentChars(required) > maxConversationChars
+  ) {
+    return {
+      message:
+        "Required Scout context exceeds the Advisor conversation budget.",
       ok: false,
       reason: "required-group-overflow",
     };
   }
-
   const selected = groups.filter(
     (group) => group.required || group.bytes <= maxGroupBytes
   );
   const fits = () =>
     selected.length <= maxGroups &&
-    selected.reduce((sum, group) => sum + groupWireBytes(group), 0) <= maxBytes;
+    selected.reduce((sum, group) => sum + groupWireBytes(group), 0) <=
+      maxManifestBytes &&
+    (maxConversationChars === undefined ||
+      contentChars(selected) <= maxConversationChars);
   while (!fits()) {
     const optionalIndex = selected.findIndex((group) => !group.required);
     if (optionalIndex < 0) {

package/src/scout.ts

@@ -13,7 +13,7 @@ import {
   SCOUT_SYNTHESIS_MAX_BYTES,
   type ScoutManifest,
 } from "./scout-context.js";
-import type { BenchmarkTelemetry } from "./telemetry.js";
+import { snapshotAdvisorUsage } from "./usage.js";
 
 export const SCOUT_TIMEOUT_MS = 30_000;
 
@@ -224,42 +224,12 @@ export const runAdvisorScout = async (
   parentSignal?: AbortSignal,
   onEvent?: (event: ScoutLifecycleEvent) => void,
   timeoutMs = SCOUT_TIMEOUT_MS,
-  dependencies: ScoutDependencies = defaultDependencies,
-  telemetry?: BenchmarkTelemetry
+  dependencies: ScoutDependencies = defaultDependencies
   // biome-ignore lint/complexity/noExcessiveCognitiveComplexity: cancellation, timeout, provider, and schema outcomes remain explicitly distinct.
 ): Promise<ScoutOutcome> => {
   const startedAt = Date.now();
   const publish = (event: ScoutLifecycleEvent) => {
     onEvent?.(event);
-    if (event.type === "chunk") {
-      return;
-    }
-    if (event.type === "call" || event.type === "cancelled") {
-      telemetry?.scout(event);
-    } else if (event.type === "success") {
-      telemetry?.scout({
-        availableCount: event.outcome.metrics.availableCount,
-        latencyMs: event.outcome.metrics.latencyMs,
-        model: event.outcome.model,
-        omittedBeforeScout: event.outcome.metrics.omittedBeforeScout,
-        selectedCount: event.outcome.metrics.selectedCount,
-        selectedLabels: event.outcome.selectedLabels,
-        synthesis: event.outcome.selection.synthesis,
-        type: "success",
-        usage: event.outcome.metrics.usage,
-      });
-    } else {
-      telemetry?.scout({
-        availableCount: event.outcome.metrics.availableCount,
-        fallback: `${event.outcome.category}: ${event.outcome.message}`,
-        latencyMs: event.outcome.metrics.latencyMs,
-        model: event.outcome.model,
-        omittedBeforeScout: event.outcome.metrics.omittedBeforeScout,
-        selectedCount: event.outcome.metrics.selectedCount,
-        type: "fallback",
-        usage: event.outcome.metrics.usage,
-      });
-    }
   };
   if (parentSignal?.aborted) {
     publish({ type: "cancelled" });
@@ -361,7 +331,10 @@ export const runAdvisorScout = async (
         ? ("invalid-selection" as const)
         : ("empty-response" as const),
       message,
-      metrics: { ...baseMetrics(manifest, startedAt), usage: streamed.usage },
+      metrics: {
+        ...baseMetrics(manifest, startedAt),
+        usage: snapshotAdvisorUsage(streamed.usage),
+      },
       model: executorRef,
       ok: false as const,
     };
@@ -383,7 +356,7 @@ export const runAdvisorScout = async (
           .filter((group) => group.required)
           .map((group) => group.id),
       ]).size,
-      usage: streamed.usage,
+      usage: snapshotAdvisorUsage(streamed.usage),
     },
     model: executorRef,
     ok: true as const,

package/src/session-state.ts

@@ -1,3 +1,11 @@
+import {
+  type AdvisorUsageTotals,
+  addAdvisorUsage,
+  emptyAdvisorUsageTotals,
+  formatAdvisorUsageStatus,
+  formatAdvisorUsageTotals,
+} from "./usage.js";
+
 export type GateDecision = "proceed" | "revise" | "blocked";
 export type ConsultationTrigger = "manual" | "executor-requested";
 export type GateTrigger =
@@ -120,6 +128,7 @@ export class AdvisorSessionState {
   #draftConsultations = 0;
   #outcomes = 0;
   #lastAdvice?: string;
+  #usage = emptyAdvisorUsageTotals();
 
   resetTask() {
     this.#previousSignature = undefined;
@@ -133,6 +142,7 @@ export class AdvisorSessionState {
     this.#draftConsultations = 0;
     this.#outcomes = 0;
     this.#lastAdvice = undefined;
+    this.#usage = emptyAdvisorUsageTotals();
   }
 
   clearBlocked() {
@@ -182,8 +192,19 @@ export class AdvisorSessionState {
     return this.#consumedCalls;
   }
 
+  /** Returns a copy of cumulative direct Advisor usage for this session. */
+  get usageTotals(): AdvisorUsageTotals {
+    return { ...this.#usage };
+  }
+
+  /** Returns the footer-ready direct Advisor usage status for this session. */
+  usageStatus() {
+    return formatAdvisorUsageStatus(this.#usage);
+  }
+
   recordInvocation(record: AdvisorInvocationRecord) {
     this.#invocations.push(record);
+    addAdvisorUsage(this.#usage, record.usage);
   }
   issueAdvice(
     id: string,
@@ -271,6 +292,7 @@ export class AdvisorSessionState {
       `Triggers: ${["manual", "executor-requested", "repeated-tool-call", "completion-review", "custom-rule"].filter((trigger) => countTrigger(trigger as AdvisorTrigger) > 0).join(", ") || "none"}`,
       `Models: ${models}`,
       `Budget: ${budget}`,
+      `Usage: ${formatAdvisorUsageTotals(this.#usage)}`,
       `Markdown advice: ${markdown.length} responses (${this.#draftConsultations} with drafts)`,
       `Outcome reports: ${this.#outcomes}`,
       `Gate decisions: ${decisions}`,