@bivelio/savings-layer 0.4.25 → 0.4.28

package/README.md

@@ -74,12 +74,14 @@ products charge a flat subscription, so there is no per‑token bill to cut. The
 story, in plain words:
 **[Where does the saving install?](https://savings.bivelio.com/en/how-it-works)**
 
-**Claude Code, Codex CLI, Goose or Qwen Code with an API key?** The package also
-ships a local measuring gateway: `bvsala claude` (or `bvsala codex`, `bvsala
-goose`, `bvsala qwen`) launches the terminal through it and your real spend shows
-up in your dashboard, token by token — nothing to configure, nothing persisted.
-For opencode, Crush, Droid, Cline or Kilo Code, `bvsala doctor` prints the exact
-config block for each one you have installed. Like the SDK, the gateway is
+**A coding agent with an API key?** The package also ships a local measuring
+gateway: `bvsala claude` (or `bvsala codex`, `goose`, `qwen`, `opencode`, `kilo`,
+`copilot` with your own Anthropic key, `aider`) launches the agent through it and your real
+spend shows up in your dashboard, token by token — nothing to configure, nothing
+persisted. On exit it tells you how many requests it metered and which ones it
+could not, and why. For Crush, Droid, Cline, Continue or Copilot in VS Code,
+`bvsala doctor` prints the exact config block. Gemini CLI and Cursor cannot be
+metered today, and the doctor says so. Like the SDK, the gateway is
 fail‑closed: it starts only with your service key (`BIVELIO_SERVICE_KEY`) and an
 active BiVelio license, which it collects with that key and verifies against the
 key embedded in the package. Once running it never interrupts a request in flight:
@@ -94,7 +96,8 @@ as for any other use.
 
 **The gateway forwards what your terminal sends.** With no lever armed, your
 provider receives the request body byte for byte and every header your client
-sends (see the limits): `cache_control` and its 1-hour TTL, tool search and
+sends except `x-bvsala-report`, which asks the gateway itself for a report (see
+the limits): `cache_control` and its 1-hour TTL, tool search and
 `tool_reference` blocks, the advisor, every `anthropic-beta` value. An armed lever
 changes only what it declares, and a request it cannot rewrite exactly travels
 untouched. A test suite checks this against a fake upstream on every change
@@ -140,6 +143,17 @@ is measurement, not savings: the saving levers of the SDK are TypeScript, and th
 gateway's own levers apply to Anthropic `/v1/messages` traffic with a PRO license.
 `bvsala doctor` prints the recipe for each SDK.
 
+Each request can also bring its report back to your code. Send the header
+`x-bvsala-report: 1` (in Python, `default_headers={"x-bvsala-report": "1"}` on
+the client) and the gateway answers with `x-bvsala-request-id`. On the same
+gateway, `GET /_bvsala/v1/reports/<that id>` returns what it recorded for your
+dashboard: `metered` with the `SavingsReport` (the provider's token counts, the
+cost at list price and `cost.basis`), or `not_metered` with the reason from the
+log line. `?response_id=<the provider's response id>` finds it too, and
+`wait_ms` waits for a request still in flight. The header never reaches your
+provider, and a request without it is not changed at all:
+[the details](./docs/terminales/README.md#what-the-gateway-guarantees-it-forwards-what-your-client-sends).
+
 ## Check it before you install
 
 What you can check before paying is what we have measured: the published benchmark,
@@ -149,6 +163,22 @@ and the commission calculator on the front page, which runs on the same price
 catalogue that bills you. The verifier that replays the optimizer on your own data
 lives in your dashboard once you sign in.
 
+## Set it up in one command
+
+```bash
+npx @bivelio/savings-layer@latest init
+```
+
+It reads your `package.json` (Next.js, Express, the Vercel AI SDK, Mastra,
+LangChain.js, the Anthropic SDK or the OpenAI SDK) and prints the three steps below,
+written for your stack — the install line for your package manager, which keys are
+missing, the exact code for your adapter — plus a free check: a question the layer
+answers on your machine, so the model is not called. It reads `package.json` files only
+(yours, and those in `node_modules` for the installed versions), never a `.env` file or a
+key's value; it uses no network of its own and writes nothing.
+`--write` creates the new files after asking you, and never overwrites one; `--json`
+prints the same plan for a script or a coding agent.
+
 ## Install in three steps
 
 **1 · The package**
@@ -226,6 +256,19 @@ exists, as for Anthropic models), and `"unmeasured"` when the provider returned
 usage. Only `measured` and `reference` figures reach your invoice; estimated ones
 never do.
 
+A repeat is valued by reference only when the call that ran reached the provider's
+own API (`https://api.openai.com`, `https://api.anthropic.com` or
+`https://generativelanguage.googleapis.com`), because the list price it is valued at
+is the invoice only there. When `OpenAICompatibleProvider` or `AnthropicProvider`
+points anywhere else — a proxy such as LiteLLM, a gateway, Azure, Bedrock, Vertex —
+or when the Anthropic SDK or LangChain adapter reads that its client does, that call
+leaves no reference: its repeats stay `estimated`, and the `reference` trace of the
+call that ran says `unofficial-origin`. When the layer cannot tell where the call
+went (a provider of your own, the AI SDK and Mastra adapters, a LangChain model that
+does not expose its base URL), the rule applies as it always has. The layer reads the
+configured base URL, not the wire: a custom `fetch` that sends the request elsewhere
+is not seen.
+
 **Streaming.** `generateStream()` takes the same request and hands you the answer as
 the provider writes it, then the same `SavingsReport`:
 
@@ -332,8 +375,10 @@ shared here. `emitReportPart: true` also streams each report as a
 
 ### LangChain.js
 
-`pnpm add @bivelio/savings-layer langchain @langchain/core` — an agent middleware for
-`createAgent` in `langchain` 1.x:
+`pnpm add @bivelio/savings-layer langchain @langchain/core @langchain/openai` — an agent
+middleware for `createAgent` in `langchain` 1.x. The model string (`"openai:…"`) is loaded
+through its integration package, so `@langchain/openai` (or `@langchain/anthropic`) has
+to be installed:
 
 ```ts
 import { createAgent } from "langchain";
@@ -402,6 +447,19 @@ single-shot requests only: requests that declare tools, carry tool calls or resu
 hold more than one user turn are left out unless you set
 `savings.semanticCache: "include-agentic"`.
 
+**Calibrated reuse (opt-in).** A fixed similarity floor means something different for
+every embedder and every workload, so you can make the cache earn its floor on your own
+traffic instead: `new InMemorySemanticCache({ embed, calibration: { maxErrorRate: 0.02 } })`.
+It then serves no reworded answer until your labels show, at 95 % confidence, that at
+most 2 % of the answers it would reuse from some score up are wrong, and it re-tests that
+floor as labels accumulate. Until then every candidate goes to your provider, exactly as
+it would without the cache, and the fresh answer is compared with the stored one by your
+`judge` (by default only identical answers count, which suits classification and
+structured output; pass your own for free text). Each calibrated reuse carries its bound
+and the labels behind it in `report.qualityLedger.cacheMatch.calibration`. Calibration
+costs reuses, never an extra call, and it does not change how anything is billed. If you
+already have labeled pairs, `calibrateSemanticFloor()` runs the same test offline.
+
 And a promise you can hold it to: **never worse than your JSON.** Payload encodings
 are chosen by counting real tokens with the destination model's own tokenizer, and
 BV‑SALA only moves away from plain JSON when the saving clearly clears the bar —
@@ -465,8 +523,8 @@ changes the answer, and what they are worth is reported next to the four ledgers
 never folded into `cost.*`, never part of any commission.
 
 **`savings.deferrable`** asks for the provider's Flex tier: OpenAI, and Gemini through
-its OpenAI‑compatible endpoint. Anthropic has no cheaper synchronous tier (its
-discount lives only in Message Batches, which the layer does not support yet).
+its OpenAI‑compatible endpoint. Anthropic has no cheaper synchronous tier: its
+discount lives only in Message Batches, which `layer.batches` (below) sends to.
 Bedrock and DeepSeek off‑peak pricing are not supported.
 
 - **Latency.** OpenAI publishes no latency commitment for Flex; Google publishes a
@@ -514,6 +572,195 @@ report.deferred?.tierSaving; // { amount, basis, standardCost, appliedCost, pric
 report.cacheWrites;          // { mode, breakpoints, avoidedWritePremium?, reason }
 ```
 
+**`layer.batches`** sends requests that can wait hours to the provider's own batch API,
+billed at its published batch rate card: Anthropic's Message Batches through
+`AnthropicProvider`, and OpenAI's Batch API through `OpenAICompatibleProvider` on
+`api.openai.com`. Gemini's batch mode is not supported yet.
+
+- **What runs.** Each request is translated by the same provider adapter as
+  `generate()` and sent as you wrote it: the layer's own per‑request savings (caching,
+  compression, model cascade, retrieval) do not run on batches. The layer splits the
+  job at the provider's limits (and, on OpenAI, one model per file).
+- **Latency.** The provider's: results arrive within its batch window, not on a
+  request/response path. `submit()` returns a plain‑JSON job you can store and read
+  from another process.
+- **What it is worth.** Each result's usage counter at the standard and at the batch
+  published rate cards, component by component, with the same `measured` /
+  `estimated` rule as above. The tier is read from the provider: Anthropic reports it
+  on every result; on OpenAI it is the batch the result came from. Nothing is
+  metered and the batch saving is never billed.
+
+```ts
+const tickets = [{ id: "t-1", text: "My invoice is wrong." }]; // your deferrable work
+
+const job = await layer.batches.submit(
+  tickets.map((t) => ({
+    customId: t.id,
+    request: { model: "openai/gpt-4o-mini", messages: [{ role: "user", content: t.text }] },
+  })),
+);
+// …later, even from another process (store `job` as JSON):
+const results = await layer.batches.results(job);
+results.ended;   // false while the provider is still working
+results.items;   // one per request: { customId, outcome, text, usage, tierSaving?, reason }
+results.summary; // { succeeded, errored, expired, credited, tierSaving, basis, pricesCheckedOn, reason, … }
+```
+
+</details>
+
+<details>
+<summary><b>Tool outputs, compacted where they are born (opt-in, off by default)</b></summary>
+
+In an agent loop your history is re-sent on every turn and your provider reads it
+from its prompt cache at a fraction of the input price. Rewriting anything it has
+already cached turns everything after it back into full-price cache writes, which is
+how "fewer tokens" ends up as a bigger bill. So this lever only ever changes a tool
+output on the turn it arrives, and re-sends exactly the same bytes afterwards.
+
+It removes terminal colour codes, superseded progress-bar states, whitespace between
+JSON tokens and runs of identical lines (folded into one line plus a count), and
+nothing else: no summaries, no dropped words. Each change either reverses exactly or
+leaves what a terminal would display. Outputs of calls that name a file are left
+verbatim, so an agent that edits by exact match still sees the file as it is.
+
+```ts
+const agentLayer = createSavingsLayer({
+  provider,
+  licenseKey: process.env.BIVELIO_LICENSE_KEY,
+  toolOutputCompaction: true, // or { exceptTools: ["read_logs"], minSavedChars: 512 }
+});
+// or per request: savings: { toolOutputCompaction: true }
+```
+
+- **Switch it on for whole conversations.** The decision depends only on each
+  output's own text, so every process and every turn produces the same bytes. A
+  conversation that started without it is rewritten once, on its first compacted
+  request.
+- **Counted, not billed.** What it removes shows in `report.tokens.compressedInput`
+  and in `report.traces`; it never enters `cost.*` or any commission. Your provider's
+  invoice shows the effect.
+- **Inside a framework,** call `compactToolOutput(text)` in your tool before you
+  return its result: your history then holds the compacted text itself.
+- **Measure it by conversation, not by request.** A verification copy re-sends one
+  request with the original history, which your provider has not cached, so on
+  turns that re-send compacted outputs the copy costs more than going without the
+  layer would have. Compare whole conversations with and without it.
+
+</details>
+
+<details>
+<summary><b>Prompt-cache breakpoints placed from your traffic (opt-in)</b></summary>
+
+By default the layer writes the provider's prompt cache at two fixed places: the end
+of the system header and the end of the history before your last user message, and
+stops marking a prefix whose measured hit rate does not pay the write premium.
+
+`cacheBreakpoints: "auto"` places them from what your traffic actually does. The layer
+compares each request's prefix with the ones it sent before, per model, tool set and
+system header, and learns how often each part is sent again and after how long. That
+costs nothing: it compares locally and marks nothing to find out. Once the evidence
+is conclusive:
+
+- **Tail.** When requests continue each other (agent tool loops, conversations), it
+  also marks the last block, so the next turn reads the whole prompt from cache.
+  Tool results arrive after your last user message, so the fixed placement never
+  caches them.
+- **History.** When the history before the question is not sent again (one-off
+  requests with their own document), it moves the breakpoint back to the system
+  header, so no write premium is paid on a prefix nobody reads.
+- **One-hour lifetime** (Anthropic). When turns come back after more than five
+  minutes but within the hour, and the size of the request makes the 2x write pay.
+- **Explicit mode** (GPT‑5.6 and later, on OpenAI's own API). When continuations are
+  rare, your question is not written to the cache. `savings.standalone` still wins
+  when you set it, and `report.cacheWrites` says when the layer chose it.
+
+Until then it behaves exactly like the default. Every decision and its reason is in
+`report.traces` (`prefix-cache-placement`), and the effect is read from the provider's
+own counters: if they do not confirm the cache reads the layer predicted, that
+traffic goes back to the default placement. The write premium it causes, one-hour
+writes at 2x included, is charged to `cost.actual`; the cache discount it earns is
+counted in ledger 4 like any provider cache read, never credited as the layer's
+saving and never part of any commission.
+
+```ts
+const placedLayer = createSavingsLayer({
+  provider, // Anthropic, or OpenAI GPT-5.6+ on OpenAI's own API
+  licenseKey: process.env.BIVELIO_LICENSE_KEY,
+  cacheBreakpoints: "auto",
+});
+```
+
+</details>
+
+<details>
+<summary><b>Serverless: keep what the layer learns between instances</b></summary>
+
+Calibrated reuse and `cacheBreakpoints: "auto"` learn from your traffic, and by default
+what they learn lives in the process. On serverless platforms (Vercel Functions, AWS
+Lambda) an instance may serve one request or live a few minutes, so neither would ever
+get past its starting point. Give each one somewhere to keep it:
+
+```ts
+import { InMemorySemanticCache } from "@bivelio/savings-layer";
+
+// Any key-value client your instances share, such as Redis.
+type SharedKeyValue = {
+  get(key: string): Promise<unknown>;
+  set(key: string, value: string, options?: { px: number }): Promise<unknown>;
+};
+
+async function serverlessLayer(kv: SharedKeyValue) {
+  const semanticCache = new InMemorySemanticCache({ calibration: { maxErrorRate: 0.02 } });
+  const saved = await kv.get("bvsala:semantic-calibration");
+  if (saved) semanticCache.importCalibration(saved); // the object or its JSON text
+
+  const layer = createSavingsLayer({
+    provider,
+    licenseKey: process.env.BIVELIO_LICENSE_KEY,
+    semanticCache,
+    cacheBreakpoints: "auto",
+    cacheBreakpointStore: {
+      get: (key) => kv.get(key),
+      set: (key, value, ttlMs) => kv.set(key, value, { px: ttlMs }),
+    },
+  });
+
+  // After handling requests with `layer`, save the calibration:
+  const save = async () => {
+    await semanticCache.settled();
+    await kv.set("bvsala:semantic-calibration", JSON.stringify(semanticCache.exportCalibration()));
+  };
+  return { layer, save };
+}
+```
+
+- **Breakpoint placement.** The store holds prompt-prefix fingerprints, timestamps and
+  counts, one key per traffic class, never prompt text. The layer reads it before
+  placing breakpoints and reads and rewrites it after the provider answers, so every
+  instance extends the same measurement. If the store fails, that request is placed as
+  a fresh instance would place it and nothing is written over what others learned.
+  Without a store nothing changes.
+- **Calibration.** The saved state is the whole experiment: both halves of the labels,
+  how many tests have already drawn on the confidence budget, and the floor in force.
+  Restoring it continues the same calibration, so the bound keeps holding across
+  restarts; it never starts the budget over. A state recorded under another embedder,
+  verifier, judge or configuration is refused with `CalibrationStateError`. If you pass
+  your own `embed`, `verify` or `judge`, name them (`embedderId`, `verifierId`,
+  `calibration.judgeId`) so a state can say what it was recorded with.
+- **One line of history.** The bound holds along one line: restore the latest state and
+  save from where it went. Instances that continue the same state at the same time
+  each spend its remaining confidence again, and the last one to save overwrites the
+  others' labels. For one bound across many concurrent instances, calibrate in a single
+  long-lived process, or offline with `calibrateSemanticFloor()`, and give every
+  instance the certified floor as `acceptThreshold`. The cached answers themselves
+  still live in each instance's memory.
+- **Why a predicted cache read did not happen** (Anthropic's own API). With
+  `cacheMissDiagnostics: true`, the layer asks Anthropic to compare each marked request
+  with the one whose cache entry it expected to read, and puts the provider's
+  `cache_miss_reason` in `report.traces` when that read does not arrive. Anthropic
+  keeps a short-lived fingerprint (hashes and token-count estimates) of each request
+  that opts in; check that against your data-retention terms first.
+
 </details>
 
 <details>
@@ -547,10 +794,11 @@ console.log(data.answer); // 8
 
 When you pass `tools` and the model asks to invoke one, the reply is not the final
 answer: `text` is empty by design and the request is in `toolCalls`. Execute it,
-append the result to `messages`, and call `generate()` again.
+append the assistant turn and one `role: "tool"` message per call to `messages`, and
+call `generate()` again.
 
 ```ts
-import type { ToolDescriptor } from "@bivelio/savings-layer";
+import type { ChatMessage, ToolDescriptor } from "@bivelio/savings-layer";
 
 const tools: ToolDescriptor[] = [
   {
@@ -567,26 +815,83 @@ const tools: ToolDescriptor[] = [
   },
 ];
 
-const res = await layer.generate({
-  model: "openai/gpt-4o-mini",
-  messages: [{ role: "user", content: "What's the weather in Madrid?" }],
-  tools,
-});
+// ← your tool
+const getWeather = async (city: string): Promise<string> => `${city}: 21 °C, clear`;
 
-if (res.toolCalls?.length) {
-  // res.finishReason === "tool_calls"
+const messages: ChatMessage[] = [{ role: "user", content: "What's the weather in Madrid?" }];
+let res = await layer.generate({ model: "openai/gpt-4o-mini", messages, tools });
+
+while (res.toolCalls?.length) {
+  // res.finishReason === "tool_calls". Send the calls back as they came, ids included.
+  messages.push({ role: "assistant", content: res.text, toolCalls: res.toolCalls });
   for (const call of res.toolCalls) {
     const args = JSON.parse(call.arguments) as { city: string };
-    console.log(call.name, args); // ← run your tool here, then append its result
+    const output = await getWeather(args.city);
+    messages.push({ role: "tool", toolCallId: call.id, name: call.name, content: output });
   }
-} else {
-  console.log(res.text);
+  res = await layer.generate({ model: "openai/gpt-4o-mini", messages, tools });
 }
+console.log(res.text);
 ```
 
+- **Ids.** Each result carries the `toolCallId` of the call it answers, and every call
+  of a turn gets its result before the next user or assistant message. On Anthropic
+  that is what lets the turn travel as `tool_use`/`tool_result` blocks (with the tool
+  still in that request's `tools`); otherwise it is sent as plain text, as before, and
+  the SDK warns once.
+- **Errors.** When a tool fails, put the error in `content` and set `isError: true`.
+  Anthropic receives it as `is_error`; providers without such a flag get the text.
+
 A tool‑call turn is never served from cache and never counted as an avoided call:
 an action that has not run yet cannot be a saving.
 
+**Large catalogues: let the provider search them (opt‑in).** With hundreds of tools
+(several MCP servers, say), `savings: { toolSearch: "auto" }` sends every tool deferred
+behind the provider's own tool search instead of letting the layer's selector keep a
+few. A catalogue that declares no `intents` or `risk` gives the selector nothing to go
+on, so it keeps the first ones and the model cannot call the rest; with the provider's
+search the model looks up what it needs, no tool you declared is out of reach, and the
+cached prompt prefix stays the same from one question to the next.
+
+- **Where.** The Anthropic adapter, on models in Anthropic's tool search compatibility
+  table. OpenAI documents its tool search for the Responses and Agents APIs, not for
+  Chat Completions, which is what this SDK's OpenAI adapter speaks, so OpenAI requests
+  keep the selector.
+- **When.** `"auto"` applies it when the adapter talks to Anthropic's own API, the
+  catalogue is over the provider's documented size, and deferring it pays for one
+  search on every request even with a warm prompt cache. `"always"` skips those checks
+  (for example behind a proxy that forwards the search blocks). Requests with an output
+  contract, or with the model cascade on, keep the selector.
+- **Across turns.** Append the assistant turn with its `toolCalls` (ids as returned)
+  before the tool results, as in any tool conversation. The Anthropic adapter then sends
+  the earlier search results back in that turn, so the model calls the tools it already
+  found without searching again. It keeps them in memory, per adapter instance and
+  bounded; a turn it no longer remembers (another process, say) goes out as before and
+  the model searches again.
+- **Whose saving.** The provider's. Your usage counter shows it; the layer credits
+  nothing for it and charges no commission on it.
+
+```ts
+import { AnthropicProvider } from "@bivelio/savings-layer";
+
+const claude = createSavingsLayer({
+  provider: new AnthropicProvider({ apiKey: process.env.ANTHROPIC_API_KEY }),
+  licenseKey: process.env.BIVELIO_LICENSE_KEY,
+});
+
+const mcpTools: ToolDescriptor[] = []; // every tool your MCP servers list
+
+const { report: searched } = await claude.generate({
+  model: "anthropic/claude-opus-5-5",
+  messages: [{ role: "user", content: "Which orders from service 17 are still open?" }],
+  tools: mcpTools,
+  savings: { toolSearch: { mode: "auto", alwaysLoaded: ["read_file"] } },
+});
+
+// Which path ran and why, and how many searches the provider ran.
+console.log(searched.traces.filter((t) => t.stage === "tool-search"));
+```
+
 </details>
 
 <details>