@bivelio/savings-layer 0.4.31 → 0.4.33

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE CHANGED
@@ -10,18 +10,23 @@ not install or use the Software.
10
10
 
11
11
  1. LICENSE GRANT. Subject to your continuous compliance with this Agreement and to
12
12
  an active BiVelio subscription (paid, or granted by BiVelio at no charge, for
13
- example through a promotional code or a pilot) with a valid license key
14
- ("License Key"), BiVelio grants you a limited, non-exclusive, non-transferable,
15
- non-sublicensable, revocable license to install and use the Software, as
16
- published, solely for your own internal business purposes, and only for the
17
- number of seats covered by your subscription. The Software is licensed, not sold.
13
+ example through a promotional code, a pilot, or the free Measure license) with a
14
+ valid license key ("License Key"), BiVelio grants you a limited, non-exclusive,
15
+ non-transferable, non-sublicensable, revocable license to install and use the
16
+ Software, as published, solely for your own internal business purposes, and only
17
+ for the number of seats covered by your subscription. The Software is licensed,
18
+ not sold. Under a Measure license the Software runs in shadow mode only: it
19
+ forwards your requests to your provider unchanged, applies no optimization, and
20
+ reports estimated savings that are never billed.
18
21
 
19
22
  2. LICENSE KEY; FAIL-CLOSED OPERATION. The Software requires a valid, active License
20
23
  Key and does not operate without one. Expiry, non-payment, revocation, suspension,
21
24
  or termination of your subscription immediately ends the license granted in
22
- Section 1, and the Software will cease to function. You will not circumvent,
23
- disable, remove, or tamper with the Software's licensing, metering, or revocation
24
- mechanisms, nor use the Software after your license has ended.
25
+ Section 1, and the Software will cease to function; when a paid subscription
26
+ ends, BiVelio may instead grant you a Measure license, under which the Software
27
+ continues in shadow mode only. You will not circumvent, disable, remove, or tamper
28
+ with the Software's licensing, metering, or revocation mechanisms, nor use the
29
+ Software after your license has ended.
25
30
 
26
31
  3. RESTRICTIONS. Except as expressly permitted above, you will not: (a) distribute,
27
32
  publish, resell, rent, lease, lend, offer as a hosted or managed service, or
@@ -74,8 +79,11 @@ not install or use the Software.
74
79
  (iv) cost figures computed by the Software from its price catalogue (or from
75
80
  a cost model you supply): baseline cost, actual cost, the net-saving
76
81
  ratio, the basis of the figures (measured, reference, estimated, shadow,
77
- or unmeasured), and the value, at the provider's cache-read price, of
78
- input tokens your provider reported as cleared from context.
82
+ or unmeasured), the value, at the provider's cache-read price, of
83
+ input tokens your provider reported as cleared from context, and the
84
+ discount your provider's own prompt cache gave the request and the
85
+ cache-write premium it billed, which are shown apart and never used to
86
+ calculate the fees in Section 6.
79
87
  Each batch of records also carries the Software's surface and version (for
80
88
  example, "gateway/0.4.23"). BiVelio's ingest rejects any record that carries a
81
89
  field not listed above. Separately, the Software fetches from BiVelio its current
@@ -102,8 +110,10 @@ not install or use the Software.
102
110
  6. FEES. Your subscription is subject to BiVelio's pricing as published at
103
111
  https://savings.bivelio.com/pricing (the "Published Pricing") for each billing
104
112
  period:
105
- (a) a subscription fee per seat, payable whether or not the Software produces any
106
- saving; and (b) a commission, at tiered marginal rates, charged only on measured
113
+ (a) a subscription fee, per seat or for a plan that includes a number of seats,
114
+ payable whether or not the Software produces any saving; and (b) only if your
115
+ subscription is on a plan that the Published Pricing lists as carrying a
116
+ commission, a commission, at tiered marginal rates, charged only on measured
107
117
  savings, meaning savings whose basis is labelled "measured" or "reference" in
108
118
  your usage records (Section 5(b)(iv)). A "measured" saving is established from
109
119
  your LLM provider's own usage counter for that request; a "reference" saving, for
package/README.md CHANGED
@@ -19,21 +19,19 @@
19
19
  Every request your app sends to an AI is paid for by the text going in and out.
20
20
  **Savings** sits on that path and lets through only what's needed — skipping calls
21
21
  that don't have to happen, and trimming what does — so you pay for less. It is not a
22
- promise you take on faith: **every saving we charge on is measured with your provider's
22
+ promise you take on faith: **every saving we report is measured with your provider's
23
23
  own usage counter, the one your bill is built from** (for a repeat served without
24
24
  calling the provider, the counter of the identical call that ran), and every saving —
25
25
  measured or estimated, and labelled as such — is handed back to you request by request.
26
26
 
27
- **A small per‑seat fee, and a commission only on savings measured against your
28
- provider's counter.** The seat fee is paid whether or not you save. Your
29
- first month carries no commission at all — it is the first calendar month (UTC) in
30
- which your organization reports usage, so starting late in a month makes it a short one —
31
- and at its close you see the statement of what the commission would have been. After
32
- that, the commission is charged on the **measured** saving only — never from estimates:
33
- a tiered rate from 10 % down to 4 % of what you saved, so **you keep
34
- at least 90 % of every euro saved** (before the invoice is rounded to the cent; the
35
- seat fee is separate). A month with no measured saving is zero commission. And your
36
- prompts and completions **never reach BiVelio**: what BiVelio receives is per‑request
27
+ **Measure for free, then pay per seat — with no commission on your savings.** The
28
+ free **Measure** license (no card) runs the layer in shadow mode on your own traffic and
29
+ shows what it would have saved, valued with your provider's counter. To turn the levers
30
+ on, **Pro** is **€15 / seat / month** or **€150 / seat / year**, and **Team** is **€129 / month
31
+ with 10 seats included** (€12 per extra seat; €1,290 / year, €120 per extra seat), EUR,
32
+ VAT not included — a fixed fee, paid whether or not you save, and no commission on what
33
+ you save. Enterprise (more seats, your own contract or invoicing terms) is by contract.
34
+ And your prompts and completions **never reach BiVelio**: what BiVelio receives is per‑request
37
35
  usage — token counts, cost, the model id and opaque ids — never your content
38
36
  ([LICENSE](./LICENSE) §5 lists every field; [savings.bivelio.com/privacy](https://savings.bivelio.com/privacy)
39
37
  explains each one and how long it is kept).
@@ -46,14 +44,16 @@ picks the cheapest execution that still meets your quality bar and hands you an
46
44
  auditable `SavingsReport` — the same numbers your dashboard reads. Traffic still
47
45
  goes straight from your app to your provider; the SDK is never a middleman server.
48
46
 
49
- > **Commercial software — © 2026 BiVelio Inc.** Free to install, **paid to run**:
50
- > BV‑SALA is fail‑closed and executes only with an active BiVelio license key.
47
+ > **Commercial software — © 2026 BiVelio Inc.** Free to install, **paid to save**:
48
+ > BV‑SALA is fail‑closed and executes only with an active BiVelio license key. The
49
+ > free **Measure** license (no card) runs it in shadow mode: nothing you send is
50
+ > changed, nothing is served from cache, and you see what it would have saved.
51
51
  > The license signature is verified **offline**; with `licenseServerUrl` set, the SDK
52
52
  > also calls `savings.bivelio.com` for the revocation list, license renewal and usage
53
53
  > metering. **Your prompts, completions and provider keys never reach BiVelio** — what
54
54
  > travels is per‑request usage: token counts, cost, the model id, a request id and
55
55
  > opaque seat and session ids (the exact list is in [LICENSE](./LICENSE) §5). Use is
56
- > governed by the LICENSE, which also sets out the seat fee and the commission, and the
56
+ > governed by the LICENSE, which also sets out the fees, and the
57
57
  > BiVelio Terms. Not a wrapper around any third‑party library.
58
58
 
59
59
  ## Where it installs
@@ -119,6 +119,12 @@ bvsala gateway --upstream openai --port 8403 # one gateway per upstream
119
119
  bvsala gateway # Anthropic, the default, on 8402
120
120
  ```
121
121
 
122
+ Flags and environment variables are in English (`--port`, `--project`, `--force`,
123
+ `BVSALA_GATEWAY_EFFORT`, `BVSALA_GATEWAY_SIBLING_MODEL`). The Spanish spellings from
124
+ earlier versions (`--puerto`, `--proyecto`, `--forzar`, `BVSALA_GATEWAY_ESFUERZO`,
125
+ `BVSALA_GATEWAY_MODELO_HERMANO`) keep working until 0.5 and print a deprecation
126
+ notice; `BVSALA_SILENCE_DEPRECATIONS=1` silences it. `bvsala --help` lists them all.
127
+
122
128
  ```python
123
129
  from openai import OpenAI
124
130
  client = OpenAI(base_url="http://127.0.0.1:8403/v1") # WITH /v1
@@ -159,8 +165,8 @@ provider, and a request without it is not changed at all:
159
165
  What you can check before paying is what we have measured: the published benchmark,
160
166
  trial by trial, with the raw CSV linked, at
161
167
  **[savings.bivelio.com/en/how-it-works](https://savings.bivelio.com/en/how-it-works)**,
162
- and the commission calculator on the front page, which runs on the same price
163
- catalogue that bills you. The verifier that replays the optimizer on your own data
168
+ and the free **Measure** license, which runs the layer in shadow mode on your own
169
+ traffic and shows what it would have saved, valued with your provider's counter. The verifier that replays the optimizer on your own data
164
170
  lives in your dashboard once you sign in.
165
171
 
166
172
  ## Set it up in one command
@@ -195,7 +201,8 @@ saving where the layer removed something is reported as `estimated`, and estimat
195
201
  savings are never billed — the SDK warns once when the tokenizer is missing.
196
202
 
197
203
  **2 · The license** — from your [dashboard](https://savings.bivelio.com/dashboard/install)
198
- (**€4 / seat / month** or **€39 / seat / year**, EUR)
204
+ (free **Measure** license, no card; **Pro** €15 / seat / month or €150 / seat / year;
205
+ **Team** €129 / month with 10 seats included — EUR)
199
206
 
200
207
  ```bash
201
208
  export BIVELIO_LICENSE_KEY=… # your license key
@@ -210,8 +217,10 @@ import { createSavingsLayer, OpenAICompatibleProvider } from "@bivelio/savings-l
210
217
 
211
218
  // Your provider. LLM traffic goes straight to it — the SDK never proxies it.
212
219
  // Any OpenAI-compatible endpoint works: OpenAI, Azure, vLLM, SGLang, a gateway.
220
+ // For Gemini, use the native `GoogleProvider` instead (see "Gemini: the native adapter").
213
221
  // Model ids are qualified (`openai/gpt-4o-mini`); the prefix is stripped on the
214
- // wire for OpenAI and Azure and kept for gateways — see `stripProviderPrefix`.
222
+ // wire for OpenAI, Azure and Google's Gemini API and kept for gateways — see
223
+ // `stripProviderPrefix`.
215
224
  const provider = new OpenAICompatibleProvider({
216
225
  baseUrl: "https://api.openai.com/v1",
217
226
  apiKey: process.env.OPENAI_API_KEY,
@@ -231,8 +240,8 @@ const { text, report } = await layer.generate({
231
240
 
232
241
  console.log(text); // "8" (computed locally — no LLM call)
233
242
  console.log(report.calls.avoidedLlm); // → 1: a whole call you never paid for
234
- console.log(report.cost.netSavings); // → 0..1 ratio of the baseline you did not pay (1 here)
235
- console.log(report.cost.baselineEstimate - report.cost.actual); // → the same saving in USD
243
+ console.log(report.cost.netSavingsRatio); // → 0..1 ratio of the baseline you did not pay (1 here)
244
+ console.log(report.cost.netSavingsUsd); // → the same saving in USD (baselineEstimate − actual)
236
245
  console.log(report.cost.basis); // → "estimated": nothing ran, so there is no provider counter
237
246
  ```
238
247
 
@@ -259,8 +268,8 @@ never do.
259
268
  A repeat is valued by reference only when the call that ran reached the provider's
260
269
  own API (`https://api.openai.com`, `https://api.anthropic.com` or
261
270
  `https://generativelanguage.googleapis.com`), because the list price it is valued at
262
- is the invoice only there. When `OpenAICompatibleProvider` or `AnthropicProvider`
263
- points anywhere else — a proxy such as LiteLLM, a gateway, Azure, Bedrock, Vertex —
271
+ is the invoice only there. When `OpenAICompatibleProvider`, `AnthropicProvider` or
272
+ `GoogleProvider` points anywhere else — a proxy such as LiteLLM, a gateway, Azure, Bedrock, Vertex —
264
273
  or when the Anthropic SDK or LangChain adapter reads that its client does, that call
265
274
  leaves no reference: its repeats stay `estimated`, and the `reference` trace of the
266
275
  call that ran says `unofficial-origin`. When the layer cannot tell where the call
@@ -276,7 +285,7 @@ tokens) with `stop_details.category` set to `cyber`, `general_harms` or `null`.
276
285
  documents these as
277
286
  [not billed](https://platform.claude.com/docs/en/build-with-claude/refusals-and-fallback#how-refusals-are-billed).
278
287
  Its tokens still appear in `report.tokens`. `cost.actual`, `cost.baselineEstimate` and
279
- `cost.netSavings` are all 0, so the request credits no saving: not measured, not
288
+ `cost.netSavingsRatio` are all 0, so the request credits no saving: not measured, not
280
289
  estimated and not by reference. A concurrent identical request served from it credits
281
290
  nothing either. The same rule applies in the gateway. Any other refusal is priced from
282
291
  its `usage` as usual. That covers `bio`, `frontier_llm` and `reasoning_extraction`, a
@@ -286,16 +295,41 @@ Most refusals carry no signal from the API: the model writes "I can't help with
286
295
  as ordinary text and ends with `end_turn` or `stop`. The layer also looks for these,
287
296
  with a conservative text heuristic. It checks only the opening of the answer, or the
288
297
  whole answer when it is short, and it needs a real refusal clause, such as "I'm sorry,
289
- but I can't help with that", "I cannot assist" or "As an AI, I cannot provide". A
290
- single word is not enough. It covers English, Spanish, Catalan, French, German,
291
- Italian and Portuguese. A match is measured and served as usual. It is not written to
292
- the exact or semantic cache, and it is not replayed by reference. A concurrent
293
- identical request served from it is `estimated`. The trace says why
294
- (`skipReason: "text_refusal"`). A refusal already stored by an earlier version is not
295
- served either. In the gateway, the coalesced twin of such a response is `estimated`
296
- (`primary-text-refusal`). A false positive costs one cache hit. A normal answer that
297
- starts with "Sorry for the delay, here is…", or has "I can't" later on, is cached as
298
- before.
298
+ but I can't help with that", "I cannot assist", "I can't determine…" or "As an AI, I
299
+ cannot provide". After an apology, any verb counts ("I'm sorry, but I can't
300
+ determine…"), and so does a stated reason with no "I can't" ("I'm sorry, but
301
+ accessing … is illegal", "… would be a violation of…"). When the answer opens with
302
+ the refusal itself ("I can't", "I won't", "I'm not going to", "I'm unable to", and
303
+ "No puedo", "Je ne peux pas", "Ich kann … nicht"…), any verb counts too ("I can't
304
+ argue that…"), except for a tested list of idioms that are not refusals ("I can't
305
+ wait", "I won't lie", "I can't find any issues", "I can't be sure, but…").
306
+ A single word is not enough.
307
+ An answer that opens with empathy and, in its first two sentences, sends the user to
308
+ a mental health professional, a crisis line, emergency services or someone they
309
+ trust, instead of answering, counts too (rule `redirection`). It covers English,
310
+ Spanish, Catalan, French, German, Italian and Portuguese. A match is measured and
311
+ served as usual. It is not written to the exact or semantic cache, and it is not
312
+ replayed by reference. A concurrent identical request served from it is `estimated`.
313
+ The trace says why (`skipReason: "text_refusal"`, with the `rule`). A refusal
314
+ already stored by an earlier version is not served either. In the gateway, the
315
+ coalesced twin of such a response is `estimated` (`primary-text-refusal`). A false
316
+ positive costs one cache hit. A normal answer that starts with "Sorry for the delay,
317
+ here is…", praises with "I can't recommend this enough", has "I can't" later on, or
318
+ offers empathy with practical help and no referral ("I'm sorry to hear about your
319
+ flight; here are three options…") is cached as before.
320
+
321
+ **Sensitive topics.** A request whose user turns touch self-harm or suicide, a mental
322
+ health crisis or a medical emergency ("How do I commit suicide?", "I'm having a panic
323
+ attack", "my friend is not breathing") is never read from or written to the exact or
324
+ semantic cache, even when the answer is useful, and leaves no reference record. It is
325
+ measured and served as usual; a concurrent identical request served from it is
326
+ `estimated`. The `policy` and `cache-write` traces say why
327
+ (`skipReason: "sensitive_topic"`, with the `category`). The check is lexical and
328
+ conservative: phrases, not single words ("kill a Python process", "Suicide Squad" and
329
+ "cut my hair" do not count). It covers the same twelve languages as the freshness
330
+ and personalization checks, with shorter lists outside English, Spanish, Catalan,
331
+ French, German, Italian and Portuguese. In the gateway, the coalesced twin of such a
332
+ request is `estimated` (`sensitive-topic`). A false positive costs one cache hit.
299
333
 
300
334
  **Streaming.** `generateStream()` takes the same request and hands you the answer as
301
335
  the provider writes it, then the same `SavingsReport`:
@@ -453,6 +487,45 @@ It runs once per HTTP attempt, inside the SDK's retries: an attempt the provider
453
487
  rejects reaches the SDK unchanged and is not recorded. Only message creation goes
454
488
  through the layer; `countTokens`, batches, models and files pass untouched.
455
489
 
490
+ The middleware forwards your request body unchanged, history included, so it never
491
+ edits a turn before a thinking block. It also does not remove parameters the model
492
+ rejects. On Claude Sonnet 5.5, Opus 5.5, Fable 5.1 and Mythos 5.1 a forced
493
+ `tool_choice` (`any` or `tool`) returns a 400. On those models, and on Opus 4.7/4.8,
494
+ Opus 5, Sonnet 5 and Fable 5, so does a non-default `temperature`, `top_p` or `top_k`.
495
+ The middleware warns once per model and cause (`silenceWarnings` turns it off) and
496
+ the API's 400 reaches you unchanged.
497
+
498
+ <details>
499
+ <summary><b>Claude Sonnet 5.5, Opus 5.5, Fable 5.1: what <code>AnthropicProvider</code> sends instead</b></summary>
500
+
501
+ These models reject some request parameters with a 400 ([forcing tool
502
+ use](https://platform.claude.com/docs/en/agents-and-tools/tool-use/define-tools#forcing-tool-use),
503
+ [Sonnet 5.5](https://platform.claude.com/docs/en/models/sonnet-5-5/overview)).
504
+ `AnthropicProvider` sends a request they accept instead of failing:
505
+
506
+ - **Output contract.** A forced `tool_choice` returns a 400 on these models, with or
507
+ without thinking. When your schema meets `strict` mode, the contract travels as
508
+ structured output (`output_config.format`). Without your tools, the provider still
509
+ enforces the contract. With your tools, they stay callable with `tool_choice: auto`
510
+ and the final text answer comes back in the contract's shape. When your schema does
511
+ not meet `strict`, the contract travels as a tool the model is not forced to call,
512
+ with one sentence in the system prompt that tells it to answer through that tool.
513
+ Models that support forced tool use (Opus 5, Sonnet 5 and the 4.x models) get the
514
+ same request as before.
515
+ - **`temperature`.** A non-default value is dropped, with one warning per model,
516
+ because it can only return a 400. The answer is then sampled, not deterministic.
517
+ `temperature: 1`, the API default, is sent as is.
518
+ - **A model the table does not list** (a new model or a gateway alias): when the API
519
+ rejects either parameter, the provider repeats the request once without it and
520
+ remembers the model for later requests. A 400 is not billed, so the retry is not
521
+ counted in `retries`.
522
+ - **Thinking blocks.** `ChatMessage` does not carry thinking blocks, so this adapter
523
+ never sends one. The preserved-thinking check, which returns a 400 when a turn
524
+ before a thinking block has been edited, has nothing to check on its requests, even
525
+ when the levers that rewrite history are on.
526
+
527
+ </details>
528
+
456
529
  ## What you get on every request
457
530
 
458
531
  A `SavingsReport` with the numbers behind every claim, kept in **four separate
@@ -465,6 +538,32 @@ ledgers** that are never summed into one flattering figure:
465
538
  | `compressed_tokens` | information kept, but encoded with fewer tokens |
466
539
  | `provider_cached_tokens` | tokens still in the prompt, billed/processed more cheaply |
467
540
 
541
+ **Your provider's own cache discount, shown apart and never billed.** With prompt
542
+ caching, most of what you save often comes from your provider's prefix cache
543
+ (Anthropic `cache_control`, OpenAI's automatic cache, Gemini's implicit cache), not
544
+ from the layer. `cost` prices cached tokens at the cached rate on both sides, so that
545
+ discount is never credited as the layer's saving. `report.nativeSavings` shows it on
546
+ its own, from your provider's counter at list price, so the report adds up to your
547
+ invoice:
548
+
549
+ ```ts
550
+ report.nativeSavings;
551
+ // {
552
+ // provider: "anthropic", mechanism: "explicit", layerMarked: true,
553
+ // cacheReadTokens: 40000, cacheWriteTokens: 3000, cacheWrite1hTokens: 0,
554
+ // inputPer1k: 0.003, cachedInputPer1k: 0.0003, writeMultiplier: 1.25, write1hMultiplier: 2,
555
+ // readDiscountUsd: 0.108, // reads × (input − cached-input rate)
556
+ // writePremiumUsd: 0.00225, // writes × input × (multiplier − 1)
557
+ // netUsd: 0.10575, // can be negative: a cold write pays before any read
558
+ // basis: "measured",
559
+ // }
560
+ ```
561
+
562
+ It is present only when a call ran and the provider reported cache reads or writes;
563
+ a cache hit or a coalesced request never carries it. It is not in `cost.netSavings`,
564
+ not in `cost.baselineEstimate`, and never in the commission base. Your dashboard shows
565
+ it as its own card, **Native savings unlocked**, next to the credited savings.
566
+
468
567
  **Quality, kept apart from money.** `report.qualityLedger` records how the answer you
469
568
  received earned its place: whether it came from your provider, a replay, or the answer
470
569
  to a reworded question (with its similarity score); whether it was checked against your
@@ -551,7 +650,7 @@ changes the answer, and what they are worth is reported next to the four ledgers
551
650
  never folded into `cost.*`, never part of any commission.
552
651
 
553
652
  **`savings.deferrable`** asks for the provider's Flex tier: OpenAI, and Gemini through
554
- its OpenAI‑compatible endpoint. Anthropic has no cheaper synchronous tier: its
653
+ `GoogleProvider` (`serviceTier: "flex"`) or its OpenAI‑compatible endpoint. Anthropic has no cheaper synchronous tier: its
555
654
  discount lives only in Message Batches, which `layer.batches` (below) sends to.
556
655
  Bedrock and DeepSeek off‑peak pricing are not supported.
557
656
 
@@ -681,7 +780,16 @@ const agentLayer = createSavingsLayer({
681
780
 
682
781
  By default the layer writes the provider's prompt cache at two fixed places: the end
683
782
  of the system header and the end of the history before your last user message, and
684
- stops marking a prefix whose measured hit rate does not pay the write premium.
783
+ stops marking a prefix whose measured hit rate does not pay the write premium. On
784
+ Anthropic, where a write costs 1.25x, the history and a tool catalogue sent without a
785
+ system header are marked only when the request itself shows they will be sent again:
786
+ a tool-loop step after your last user message, or a conversation already past its
787
+ second turn. The first follow-up of a conversation (`[question, answer, question]`)
788
+ marks only the system header, so two-turn chats pay no premium that nothing reads.
789
+ In an agent loop with no system header, the end of the tool list is marked once the
790
+ catalogue reaches the model's minimum cacheable size (512 tokens on Claude Sonnet 5.5
791
+ and Opus 5.5, 1,024 on Sonnet 5, 4,096 on Haiku 4.5). That discount is the provider's
792
+ cache on your bytes: it is never credited as the layer's saving.
685
793
 
686
794
  `cacheBreakpoints: "auto"` places them from what your traffic actually does. The layer
687
795
  compares each request's prefix with the ones it sent before, per model, tool set and
@@ -727,7 +835,7 @@ Calibrated reuse and `cacheBreakpoints: "auto"` learn from your traffic, and by
727
835
  what they learn lives in the process. On serverless platforms (Vercel Functions, AWS
728
836
  Lambda) an instance may serve one request or live a few minutes, so neither would ever
729
837
  get past its starting point. Give each one somewhere to keep it. Giving the layer a
730
- `semanticCache` only says where reworded questions are kept; `semanticCacheDefault: true`
838
+ `semanticCacheStore` only says where reworded questions are kept; `semanticCacheDefault: true`
731
839
  is what switches reuse on for every request (a request's own `savings.semanticCache`
732
840
  still wins), and without it, or `savings.semanticCache` on each request, the cache
733
841
  stays empty and learns nothing:
@@ -749,7 +857,7 @@ async function serverlessLayer(kv: SharedKeyValue) {
749
857
  const layer = createSavingsLayer({
750
858
  provider,
751
859
  licenseKey: process.env.BIVELIO_LICENSE_KEY,
752
- semanticCache,
860
+ semanticCacheStore: semanticCache,
753
861
  // Passing the cache does not switch it on; this asks for it on every request.
754
862
  semanticCacheDefault: true,
755
863
  cacheBreakpoints: "auto",
@@ -823,6 +931,60 @@ console.log(data.answer); // 8
823
931
 
824
932
  </details>
825
933
 
934
+ <details>
935
+ <summary><b>Gemini: the native adapter</b></summary>
936
+
937
+ For Gemini, use `GoogleProvider`: it speaks Gemini's own API (`generateContent` and
938
+ `streamGenerateContent`), where what the layer measures is documented. Keep
939
+ `OpenAICompatibleProvider` for every other provider.
940
+
941
+ ```ts
942
+ import { GoogleProvider } from "@bivelio/savings-layer";
943
+
944
+ const gemini = createSavingsLayer({
945
+ provider: new GoogleProvider({ apiKey: process.env.GEMINI_API_KEY, requireUsage: true }),
946
+ licenseKey: process.env.BIVELIO_LICENSE_KEY,
947
+ });
948
+
949
+ const { text: answer, report: geminiReport } = await gemini.generate({
950
+ model: "google/gemini-3.8-flash", // the `google/` prefix never travels
951
+ messages: [
952
+ { role: "system", content: "Answer in one sentence." },
953
+ { role: "user", content: "Why is the sky blue?" },
954
+ ],
955
+ maxOutputTokens: 400,
956
+ reasoningEffort: "low", // → thinkingLevel "LOW" (Gemini 3) or a thinking budget (2.5)
957
+ });
958
+ console.log(answer, geminiReport.cost.actual);
959
+ ```
960
+
961
+ What the native API gives the layer that the OpenAI-compatible endpoint hides:
962
+
963
+ - **Thinking is billed output.** Output is `candidatesTokenCount + thoughtsTokenCount`
964
+ (Google bills thinking as output) and the thinking is reported as reasoning. Thought
965
+ summaries (`thinkingConfig: { includeThoughts: true }` on the adapter) are never part
966
+ of `text`.
967
+ - **Refusals with their reason.** A blocked prompt (`promptFeedback.blockReason`) or a
968
+ filtered answer (`finishReason` `SAFETY`, `PROHIBITED_CONTENT`, `BLOCKLIST`, …) is
969
+ `finishReason: "content_filter"` with the native reason in `finishReasonDetail`:
970
+ measured from its usage, never cached, never credited.
971
+ - **Tools.** Calls come back with their `id` and `thoughtSignature`; send the assistant
972
+ turn back with its `toolCalls` as they came (as in the tool loop above) and the
973
+ signature travels back untouched. The results of one turn go back together.
974
+ - **Implicit caching.** Gemini 2.5 and later cache repeated prefixes on their own; the
975
+ cached part of the prompt (`cachedContentTokenCount`) is reported as provider-cached
976
+ tokens. Explicit context caching (`cachedContents`) is not used: it bills storage by
977
+ the hour, which a per-request report cannot price.
978
+ - **Output contract.** Your schema travels as written in `responseJsonSchema`. Google
979
+ does not document that every JSON Schema feature is enforced, so a repeat of a request
980
+ with a schema is not valued by reference.
981
+ - **Thinking level.** `reasoningEffort` follows Google's published mapping. A model that
982
+ rejects a level (`gemini-3.8-flash` answers 400 to `MINIMAL`) is asked again once at
983
+ `LOW`, and remembered; the rejected request was not billed. Thinking cannot be turned
984
+ off on Gemini 3 or 2.5 Pro, so `"none"` there is the lowest level, with a warning.
985
+
986
+ </details>
987
+
826
988
  <details>
827
989
  <summary><b>Output cap — with or without a contract</b></summary>
828
990
 
@@ -846,6 +1008,26 @@ request is repeated once with the other one (nothing was generated, so nothing i
846
1008
  billed twice) and the model is remembered. To choose the field yourself:
847
1009
  `new OpenAICompatibleProvider({ baseUrl, maxTokensParam: "max_completion_tokens" })`.
848
1010
 
1011
+ **OpenAI-compatible endpoints that are not OpenAI** (Gemini's
1012
+ `generativelanguage.googleapis.com/v1beta/openai`, LiteLLM, vLLM, Ollama, a router).
1013
+ `prompt_cache_key`, OpenAI's prefix-cache routing hint, goes only to OpenAI's own API:
1014
+ Gemini answers 400 to a field it does not know, and up to 0.4.31 every request with a
1015
+ cached prefix failed there. A proxy in front of OpenAI that forwards it can opt in with
1016
+ `sendPromptCacheKey: true`. Any optional field an endpoint still rejects by name
1017
+ (`prompt_cache_key`, `service_tier`, `seed`, `reasoning_effort`) is dropped, the request
1018
+ repeated once and the model remembered; `reasoning_effort` is never dropped on OpenAI's
1019
+ own API, where its 400 is the answer. Gemini leaves its thinking tokens out of
1020
+ `completion_tokens` and bills them as output, so the layer counts output as
1021
+ `total_tokens − prompt_tokens` whenever the total is larger (and reports the difference
1022
+ as reasoning); on OpenAI's API `completion_tokens` already include reasoning and nothing
1023
+ is added. Gemini stops a filtered answer with `finish_reason: "content_filter: OTHER"`
1024
+ (or another reason): it is `finishReason: "content_filter"` (the adapter keeps the
1025
+ reason, which reaches you as `result.finishReasonDetail` and is named in the report's
1026
+ cache trace; an Anthropic refusal brings its `stop_details.category` there too), and like every
1027
+ provider refusal it is measured, never
1028
+ cached and never credited. Google does not document that such refusals are free, so
1029
+ they are costed from their `usage` like any call.
1030
+
849
1031
  The cap is yours, not a saving: it is never credited as output avoided. An answer
850
1032
  it cut (`finishReason: "length"`) is not cached, and
851
1033
  `report.qualityLedger.truncation` says the limit was yours.
@@ -904,6 +1086,21 @@ console.log(res.text);
904
1086
  the SDK warns once.
905
1087
  - **Errors.** When a tool fails, put the error in `content` and set `isError: true`.
906
1088
  Anthropic receives it as `is_error`; providers without such a flag get the text.
1089
+ - **Gemini 3's thought signature.** With `GoogleProvider` it travels in each call's
1090
+ `thoughtSignature` and comes back in `call.extraContent` in the same shape as below,
1091
+ so a conversation can move between the two Gemini adapters. Through its
1092
+ OpenAI-compatible endpoint Gemini
1093
+ returns each call with `extra_content.google.thought_signature` and rejects the next
1094
+ turn with a 400 if the call comes back without it. It arrives in `call.extraContent`;
1095
+ pushing `res.toolCalls` back as they came (as above) sends it back untouched. It is
1096
+ opaque and not part of any cache key.
1097
+ - **gpt-5.6-sol, gpt-5.6-terra and gpt-5.6-luna with tools need `reasoningEffort: "none"`.**
1098
+ On Chat Completions OpenAI rejects function tools for these models unless
1099
+ `reasoning_effort` is `"none"` («Function tools with reasoning_effort are not supported for gpt-5.6-sol in
1100
+ /v1/chat/completions»). Declare it on the request:
1101
+ `layer.generate({ model: "openai/gpt-5.6-sol", messages, tools, reasoningEffort: "none" })`.
1102
+ The layer never picks an effort itself: without the field nothing is sent. It is part
1103
+ of the cache key, like `temperature` and `seed`.
907
1104
 
908
1105
  A tool‑call turn is never served from cache and never counted as an avoided call:
909
1106
  an action that has not run yet cannot be a saving. The answer that follows a tool loop
@@ -915,10 +1112,11 @@ difference.
915
1112
  **Which tools reach the model.** The layer never cuts a catalogue it cannot rank. If
916
1113
  your tools declare no `intents` and no `risk` (the usual shape of MCP and OpenAI tool
917
1114
  lists), or you pass `savings: { router: false }`, every tool goes to the provider
918
- exactly as you sent it, and the layer credits nothing for the tool block. Only when
919
- your tools declare `intents`/`risk` does the selector keep the relevant ones, up to
920
- `new ToolSelector({ maxTools })` (8 by default), and hold back write tools from a
921
- request that is not an action. Whatever it leaves out, the model cannot call in that
1115
+ exactly as you sent it, and the layer credits nothing for the tool block. When your
1116
+ tools declare `intents`/`risk`, the selector holds back write tools from a request
1117
+ that is not an action and, since 0.4.33, sends every other tool: it prunes by
1118
+ relevance only if you ask for it with `new ToolSelector({ maxTools })` (up to 0.4.32
1119
+ that was the default, with 8). Whatever it leaves out, the model cannot call in that
922
1120
  request: the report's `tool-selector` trace lists it, the tokens count as eliminated
923
1121
  input, and the SDK warns once per process (`silenceWarnings` turns the warning off).
924
1122
 
@@ -998,7 +1196,7 @@ console.log(tracedReport.cost.basis); // the span carries the same figures
998
1196
  - **Money is kept apart by basis.** The money saved is
999
1197
  `com.bivelio.cost.saved.measured`, `.reference` or `.estimated` — one key per span,
1000
1198
  the one of its basis — so a sum over one key never mixes measured and estimated
1001
- money; `unmeasured` spans carry counts and no money. `report.cost.netSavings` is a
1199
+ money; `unmeasured` spans carry counts and no money. `report.cost.netSavingsRatio` is a
1002
1200
  ratio and goes out as `com.bivelio.cost.net_savings_ratio`; the money is
1003
1201
  `baselineEstimate − actual`, and that is what `saved.*` holds.
1004
1202
  - **None of this talks to BiVelio.** Spans go to your exporter; files stay where you
@@ -1033,7 +1231,7 @@ console.log(tracedReport.cost.basis); // the span carries the same figures
1033
1231
  | `gen_ai.usage.input_tokens`, `.output_tokens`, `.cache_read.input_tokens` | your provider's counter; absent when it reported no usage |
1034
1232
  | `com.bivelio.cost.basis`, `.currency`, `.baseline`, `.actual` | where the money comes from, its currency (USD by default), and both sides |
1035
1233
  | `com.bivelio.cost.saved.measured` \| `.reference` \| `.estimated` | the money saved, in the key of its basis only |
1036
- | `com.bivelio.cost.net_savings_ratio` | `cost.netSavings`, a 0..1 ratio — not money |
1234
+ | `com.bivelio.cost.net_savings_ratio` | `cost.netSavingsRatio`, a 0..1 ratio — not money |
1037
1235
  | `com.bivelio.calls.*`, `com.bivelio.tokens.*` | the four ledgers: avoided calls, eliminated, compressed and provider‑cached tokens |
1038
1236
  | `com.bivelio.request_id`, `.model.id`, `.cache.result`, `.quality.status` | ids and outcomes |
1039
1237
 
@@ -1148,12 +1346,30 @@ BVSALA_SEAT_ID=worker-03 # stable per replica; [A-Za-z0-9._:/-], max 128 chars
1148
1346
 
1149
1347
  ## Licensing & privacy
1150
1348
 
1151
- BV‑SALA is commercial software: a per‑seat subscription (**€4 / seat / month** or
1152
- **€39 / seat / year**, EUR; VAT not included), paid whether or not you save, plus a
1153
- tiered commission on measured savings only, as described at the top of this page, in
1154
- [LICENSE](./LICENSE) §6 and in the Published Pricing it refers to,
1155
- **[savings.bivelio.com/pricing](https://savings.bivelio.com/pricing)**. No free tier and no trial: what you can check before
1156
- paying is the published measurement. The license is a signed token your SDK and the
1349
+ BV‑SALA is commercial software. The plans are **Measure** (free), **Pro** (per seat),
1350
+ **Team** (a number of seats included, plus each extra seat) and **Enterprise** (by
1351
+ contract); Pro and Team are paid whether or not you save and carry **no commission** on
1352
+ your savings. The prices are at the top of this page, in [LICENSE](./LICENSE) §6 and in
1353
+ the Published Pricing it refers to,
1354
+ **[savings.bivelio.com/pricing](https://savings.bivelio.com/pricing)**. No trial. There is one free license,
1355
+ **Measure** — no card, one organization per verified account, up to 5 seats and 20,000
1356
+ requests a day: the layer runs in **shadow mode**. Every request goes to your provider
1357
+ exactly as you built it, nothing is served from any cache, no lever is applied, and
1358
+ `report.measure.wouldHaveSaved` (with `basis: "shadow"`) says what the exact cache and
1359
+ in‑flight coalescing would have saved on that request, valued with your provider's own
1360
+ counter — a lower bound, never part of any commission. The levers need a paid plan: once
1361
+ you upgrade, a layer with `serviceAccountKey` and `licenseServerUrl` picks up the paid
1362
+ license by itself and turns them on without a restart.
1363
+
1364
+ The earlier plan — a lower per‑seat fee plus a tiered commission on measured savings
1365
+ only — is **closed to new customers**; a subscription already on it keeps it unchanged.
1366
+ On that plan your first month carries no commission (the first calendar month, UTC, in
1367
+ which your organization reports usage); after that the commission is charged on the
1368
+ measured saving only — never from estimates — at a tiered rate from 10 % down to 4 %, so
1369
+ you keep at least 90 % of every euro saved (before rounding to the cent; the seat fee is
1370
+ separate).
1371
+
1372
+ The license is a signed token your SDK and the
1157
1373
  gateway verify **offline**; the managed control plane and the revocation kill‑switch
1158
1374
  are served from BiVelio. Your LLM traffic goes **straight to your provider** —
1159
1375
  BiVelio receives per‑request usage (counts, cost, the model id and opaque ids), never