@bivelio/savings-layer 0.4.31 → 0.4.33
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +22 -12
- package/README.md +265 -49
- package/dist/adapters/anthropic-sdk.js +75 -68
- package/dist/gateway/bin.js +964 -805
- package/dist/index.d.ts +472 -24
- package/dist/index.js +1743 -1473
- package/package.json +7 -2
package/LICENSE
CHANGED
|
@@ -10,18 +10,23 @@ not install or use the Software.
|
|
|
10
10
|
|
|
11
11
|
1. LICENSE GRANT. Subject to your continuous compliance with this Agreement and to
|
|
12
12
|
an active BiVelio subscription (paid, or granted by BiVelio at no charge, for
|
|
13
|
-
example through a promotional code
|
|
14
|
-
("License Key"), BiVelio grants you a limited, non-exclusive,
|
|
15
|
-
non-sublicensable, revocable license to install and use the
|
|
16
|
-
published, solely for your own internal business purposes, and only
|
|
17
|
-
number of seats covered by your subscription. The Software is licensed,
|
|
13
|
+
example through a promotional code, a pilot, or the free Measure license) with a
|
|
14
|
+
valid license key ("License Key"), BiVelio grants you a limited, non-exclusive,
|
|
15
|
+
non-transferable, non-sublicensable, revocable license to install and use the
|
|
16
|
+
Software, as published, solely for your own internal business purposes, and only
|
|
17
|
+
for the number of seats covered by your subscription. The Software is licensed,
|
|
18
|
+
not sold. Under a Measure license the Software runs in shadow mode only: it
|
|
19
|
+
forwards your requests to your provider unchanged, applies no optimization, and
|
|
20
|
+
reports estimated savings that are never billed.
|
|
18
21
|
|
|
19
22
|
2. LICENSE KEY; FAIL-CLOSED OPERATION. The Software requires a valid, active License
|
|
20
23
|
Key and does not operate without one. Expiry, non-payment, revocation, suspension,
|
|
21
24
|
or termination of your subscription immediately ends the license granted in
|
|
22
|
-
Section 1, and the Software will cease to function
|
|
23
|
-
|
|
24
|
-
|
|
25
|
+
Section 1, and the Software will cease to function; when a paid subscription
|
|
26
|
+
ends, BiVelio may instead grant you a Measure license, under which the Software
|
|
27
|
+
continues in shadow mode only. You will not circumvent, disable, remove, or tamper
|
|
28
|
+
with the Software's licensing, metering, or revocation mechanisms, nor use the
|
|
29
|
+
Software after your license has ended.
|
|
25
30
|
|
|
26
31
|
3. RESTRICTIONS. Except as expressly permitted above, you will not: (a) distribute,
|
|
27
32
|
publish, resell, rent, lease, lend, offer as a hosted or managed service, or
|
|
@@ -74,8 +79,11 @@ not install or use the Software.
|
|
|
74
79
|
(iv) cost figures computed by the Software from its price catalogue (or from
|
|
75
80
|
a cost model you supply): baseline cost, actual cost, the net-saving
|
|
76
81
|
ratio, the basis of the figures (measured, reference, estimated, shadow,
|
|
77
|
-
or unmeasured),
|
|
78
|
-
input tokens your provider reported as cleared from context
|
|
82
|
+
or unmeasured), the value, at the provider's cache-read price, of
|
|
83
|
+
input tokens your provider reported as cleared from context, and the
|
|
84
|
+
discount your provider's own prompt cache gave the request and the
|
|
85
|
+
cache-write premium it billed, which are shown apart and never used to
|
|
86
|
+
calculate the fees in Section 6.
|
|
79
87
|
Each batch of records also carries the Software's surface and version (for
|
|
80
88
|
example, "gateway/0.4.23"). BiVelio's ingest rejects any record that carries a
|
|
81
89
|
field not listed above. Separately, the Software fetches from BiVelio its current
|
|
@@ -102,8 +110,10 @@ not install or use the Software.
|
|
|
102
110
|
6. FEES. Your subscription is subject to BiVelio's pricing as published at
|
|
103
111
|
https://savings.bivelio.com/pricing (the "Published Pricing") for each billing
|
|
104
112
|
period:
|
|
105
|
-
(a) a subscription fee per seat
|
|
106
|
-
|
|
113
|
+
(a) a subscription fee, per seat or for a plan that includes a number of seats,
|
|
114
|
+
payable whether or not the Software produces any saving; and (b) only if your
|
|
115
|
+
subscription is on a plan that the Published Pricing lists as carrying a
|
|
116
|
+
commission, a commission, at tiered marginal rates, charged only on measured
|
|
107
117
|
savings, meaning savings whose basis is labelled "measured" or "reference" in
|
|
108
118
|
your usage records (Section 5(b)(iv)). A "measured" saving is established from
|
|
109
119
|
your LLM provider's own usage counter for that request; a "reference" saving, for
|
package/README.md
CHANGED
|
@@ -19,21 +19,19 @@
|
|
|
19
19
|
Every request your app sends to an AI is paid for by the text going in and out.
|
|
20
20
|
**Savings** sits on that path and lets through only what's needed — skipping calls
|
|
21
21
|
that don't have to happen, and trimming what does — so you pay for less. It is not a
|
|
22
|
-
promise you take on faith: **every saving we
|
|
22
|
+
promise you take on faith: **every saving we report is measured with your provider's
|
|
23
23
|
own usage counter, the one your bill is built from** (for a repeat served without
|
|
24
24
|
calling the provider, the counter of the identical call that ran), and every saving —
|
|
25
25
|
measured or estimated, and labelled as such — is handed back to you request by request.
|
|
26
26
|
|
|
27
|
-
**
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
seat fee is separate). A month with no measured saving is zero commission. And your
|
|
36
|
-
prompts and completions **never reach BiVelio**: what BiVelio receives is per‑request
|
|
27
|
+
**Measure for free, then pay per seat — with no commission on your savings.** The
|
|
28
|
+
free **Measure** license (no card) runs the layer in shadow mode on your own traffic and
|
|
29
|
+
shows what it would have saved, valued with your provider's counter. To turn the levers
|
|
30
|
+
on, **Pro** is **€15 / seat / month** or **€150 / seat / year**, and **Team** is **€129 / month
|
|
31
|
+
with 10 seats included** (€12 per extra seat; €1,290 / year, €120 per extra seat), EUR,
|
|
32
|
+
VAT not included — a fixed fee, paid whether or not you save, and no commission on what
|
|
33
|
+
you save. Enterprise (more seats, your own contract or invoicing terms) is by contract.
|
|
34
|
+
And your prompts and completions **never reach BiVelio**: what BiVelio receives is per‑request
|
|
37
35
|
usage — token counts, cost, the model id and opaque ids — never your content
|
|
38
36
|
([LICENSE](./LICENSE) §5 lists every field; [savings.bivelio.com/privacy](https://savings.bivelio.com/privacy)
|
|
39
37
|
explains each one and how long it is kept).
|
|
@@ -46,14 +44,16 @@ picks the cheapest execution that still meets your quality bar and hands you an
|
|
|
46
44
|
auditable `SavingsReport` — the same numbers your dashboard reads. Traffic still
|
|
47
45
|
goes straight from your app to your provider; the SDK is never a middleman server.
|
|
48
46
|
|
|
49
|
-
> **Commercial software — © 2026 BiVelio Inc.** Free to install, **paid to
|
|
50
|
-
> BV‑SALA is fail‑closed and executes only with an active BiVelio license key.
|
|
47
|
+
> **Commercial software — © 2026 BiVelio Inc.** Free to install, **paid to save**:
|
|
48
|
+
> BV‑SALA is fail‑closed and executes only with an active BiVelio license key. The
|
|
49
|
+
> free **Measure** license (no card) runs it in shadow mode: nothing you send is
|
|
50
|
+
> changed, nothing is served from cache, and you see what it would have saved.
|
|
51
51
|
> The license signature is verified **offline**; with `licenseServerUrl` set, the SDK
|
|
52
52
|
> also calls `savings.bivelio.com` for the revocation list, license renewal and usage
|
|
53
53
|
> metering. **Your prompts, completions and provider keys never reach BiVelio** — what
|
|
54
54
|
> travels is per‑request usage: token counts, cost, the model id, a request id and
|
|
55
55
|
> opaque seat and session ids (the exact list is in [LICENSE](./LICENSE) §5). Use is
|
|
56
|
-
> governed by the LICENSE, which also sets out the
|
|
56
|
+
> governed by the LICENSE, which also sets out the fees, and the
|
|
57
57
|
> BiVelio Terms. Not a wrapper around any third‑party library.
|
|
58
58
|
|
|
59
59
|
## Where it installs
|
|
@@ -119,6 +119,12 @@ bvsala gateway --upstream openai --port 8403 # one gateway per upstream
|
|
|
119
119
|
bvsala gateway # Anthropic, the default, on 8402
|
|
120
120
|
```
|
|
121
121
|
|
|
122
|
+
Flags and environment variables are in English (`--port`, `--project`, `--force`,
|
|
123
|
+
`BVSALA_GATEWAY_EFFORT`, `BVSALA_GATEWAY_SIBLING_MODEL`). The Spanish spellings from
|
|
124
|
+
earlier versions (`--puerto`, `--proyecto`, `--forzar`, `BVSALA_GATEWAY_ESFUERZO`,
|
|
125
|
+
`BVSALA_GATEWAY_MODELO_HERMANO`) keep working until 0.5 and print a deprecation
|
|
126
|
+
notice; `BVSALA_SILENCE_DEPRECATIONS=1` silences it. `bvsala --help` lists them all.
|
|
127
|
+
|
|
122
128
|
```python
|
|
123
129
|
from openai import OpenAI
|
|
124
130
|
client = OpenAI(base_url="http://127.0.0.1:8403/v1") # WITH /v1
|
|
@@ -159,8 +165,8 @@ provider, and a request without it is not changed at all:
|
|
|
159
165
|
What you can check before paying is what we have measured: the published benchmark,
|
|
160
166
|
trial by trial, with the raw CSV linked, at
|
|
161
167
|
**[savings.bivelio.com/en/how-it-works](https://savings.bivelio.com/en/how-it-works)**,
|
|
162
|
-
and the
|
|
163
|
-
|
|
168
|
+
and the free **Measure** license, which runs the layer in shadow mode on your own
|
|
169
|
+
traffic and shows what it would have saved, valued with your provider's counter. The verifier that replays the optimizer on your own data
|
|
164
170
|
lives in your dashboard once you sign in.
|
|
165
171
|
|
|
166
172
|
## Set it up in one command
|
|
@@ -195,7 +201,8 @@ saving where the layer removed something is reported as `estimated`, and estimat
|
|
|
195
201
|
savings are never billed — the SDK warns once when the tokenizer is missing.
|
|
196
202
|
|
|
197
203
|
**2 · The license** — from your [dashboard](https://savings.bivelio.com/dashboard/install)
|
|
198
|
-
(
|
|
204
|
+
(free **Measure** license, no card; **Pro** €15 / seat / month or €150 / seat / year;
|
|
205
|
+
**Team** €129 / month with 10 seats included — EUR)
|
|
199
206
|
|
|
200
207
|
```bash
|
|
201
208
|
export BIVELIO_LICENSE_KEY=… # your license key
|
|
@@ -210,8 +217,10 @@ import { createSavingsLayer, OpenAICompatibleProvider } from "@bivelio/savings-l
|
|
|
210
217
|
|
|
211
218
|
// Your provider. LLM traffic goes straight to it — the SDK never proxies it.
|
|
212
219
|
// Any OpenAI-compatible endpoint works: OpenAI, Azure, vLLM, SGLang, a gateway.
|
|
220
|
+
// For Gemini, use the native `GoogleProvider` instead (see "Gemini: the native adapter").
|
|
213
221
|
// Model ids are qualified (`openai/gpt-4o-mini`); the prefix is stripped on the
|
|
214
|
-
// wire for OpenAI and
|
|
222
|
+
// wire for OpenAI, Azure and Google's Gemini API and kept for gateways — see
|
|
223
|
+
// `stripProviderPrefix`.
|
|
215
224
|
const provider = new OpenAICompatibleProvider({
|
|
216
225
|
baseUrl: "https://api.openai.com/v1",
|
|
217
226
|
apiKey: process.env.OPENAI_API_KEY,
|
|
@@ -231,8 +240,8 @@ const { text, report } = await layer.generate({
|
|
|
231
240
|
|
|
232
241
|
console.log(text); // "8" (computed locally — no LLM call)
|
|
233
242
|
console.log(report.calls.avoidedLlm); // → 1: a whole call you never paid for
|
|
234
|
-
console.log(report.cost.
|
|
235
|
-
console.log(report.cost.
|
|
243
|
+
console.log(report.cost.netSavingsRatio); // → 0..1 ratio of the baseline you did not pay (1 here)
|
|
244
|
+
console.log(report.cost.netSavingsUsd); // → the same saving in USD (baselineEstimate − actual)
|
|
236
245
|
console.log(report.cost.basis); // → "estimated": nothing ran, so there is no provider counter
|
|
237
246
|
```
|
|
238
247
|
|
|
@@ -259,8 +268,8 @@ never do.
|
|
|
259
268
|
A repeat is valued by reference only when the call that ran reached the provider's
|
|
260
269
|
own API (`https://api.openai.com`, `https://api.anthropic.com` or
|
|
261
270
|
`https://generativelanguage.googleapis.com`), because the list price it is valued at
|
|
262
|
-
is the invoice only there. When `OpenAICompatibleProvider
|
|
263
|
-
points anywhere else — a proxy such as LiteLLM, a gateway, Azure, Bedrock, Vertex —
|
|
271
|
+
is the invoice only there. When `OpenAICompatibleProvider`, `AnthropicProvider` or
|
|
272
|
+
`GoogleProvider` points anywhere else — a proxy such as LiteLLM, a gateway, Azure, Bedrock, Vertex —
|
|
264
273
|
or when the Anthropic SDK or LangChain adapter reads that its client does, that call
|
|
265
274
|
leaves no reference: its repeats stay `estimated`, and the `reference` trace of the
|
|
266
275
|
call that ran says `unofficial-origin`. When the layer cannot tell where the call
|
|
@@ -276,7 +285,7 @@ tokens) with `stop_details.category` set to `cyber`, `general_harms` or `null`.
|
|
|
276
285
|
documents these as
|
|
277
286
|
[not billed](https://platform.claude.com/docs/en/build-with-claude/refusals-and-fallback#how-refusals-are-billed).
|
|
278
287
|
Its tokens still appear in `report.tokens`. `cost.actual`, `cost.baselineEstimate` and
|
|
279
|
-
`cost.
|
|
288
|
+
`cost.netSavingsRatio` are all 0, so the request credits no saving: not measured, not
|
|
280
289
|
estimated and not by reference. A concurrent identical request served from it credits
|
|
281
290
|
nothing either. The same rule applies in the gateway. Any other refusal is priced from
|
|
282
291
|
its `usage` as usual. That covers `bio`, `frontier_llm` and `reasoning_extraction`, a
|
|
@@ -286,16 +295,41 @@ Most refusals carry no signal from the API: the model writes "I can't help with
|
|
|
286
295
|
as ordinary text and ends with `end_turn` or `stop`. The layer also looks for these,
|
|
287
296
|
with a conservative text heuristic. It checks only the opening of the answer, or the
|
|
288
297
|
whole answer when it is short, and it needs a real refusal clause, such as "I'm sorry,
|
|
289
|
-
but I can't help with that", "I cannot assist" or "As an AI, I
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
298
|
+
but I can't help with that", "I cannot assist", "I can't determine…" or "As an AI, I
|
|
299
|
+
cannot provide". After an apology, any verb counts ("I'm sorry, but I can't
|
|
300
|
+
determine…"), and so does a stated reason with no "I can't" ("I'm sorry, but
|
|
301
|
+
accessing … is illegal", "… would be a violation of…"). When the answer opens with
|
|
302
|
+
the refusal itself ("I can't", "I won't", "I'm not going to", "I'm unable to", and
|
|
303
|
+
"No puedo", "Je ne peux pas", "Ich kann … nicht"…), any verb counts too ("I can't
|
|
304
|
+
argue that…"), except for a tested list of idioms that are not refusals ("I can't
|
|
305
|
+
wait", "I won't lie", "I can't find any issues", "I can't be sure, but…").
|
|
306
|
+
A single word is not enough.
|
|
307
|
+
An answer that opens with empathy and, in its first two sentences, sends the user to
|
|
308
|
+
a mental health professional, a crisis line, emergency services or someone they
|
|
309
|
+
trust, instead of answering, counts too (rule `redirection`). It covers English,
|
|
310
|
+
Spanish, Catalan, French, German, Italian and Portuguese. A match is measured and
|
|
311
|
+
served as usual. It is not written to the exact or semantic cache, and it is not
|
|
312
|
+
replayed by reference. A concurrent identical request served from it is `estimated`.
|
|
313
|
+
The trace says why (`skipReason: "text_refusal"`, with the `rule`). A refusal
|
|
314
|
+
already stored by an earlier version is not served either. In the gateway, the
|
|
315
|
+
coalesced twin of such a response is `estimated` (`primary-text-refusal`). A false
|
|
316
|
+
positive costs one cache hit. A normal answer that starts with "Sorry for the delay,
|
|
317
|
+
here is…", praises with "I can't recommend this enough", has "I can't" later on, or
|
|
318
|
+
offers empathy with practical help and no referral ("I'm sorry to hear about your
|
|
319
|
+
flight; here are three options…") is cached as before.
|
|
320
|
+
|
|
321
|
+
**Sensitive topics.** A request whose user turns touch self-harm or suicide, a mental
|
|
322
|
+
health crisis or a medical emergency ("How do I commit suicide?", "I'm having a panic
|
|
323
|
+
attack", "my friend is not breathing") is never read from or written to the exact or
|
|
324
|
+
semantic cache, even when the answer is useful, and leaves no reference record. It is
|
|
325
|
+
measured and served as usual; a concurrent identical request served from it is
|
|
326
|
+
`estimated`. The `policy` and `cache-write` traces say why
|
|
327
|
+
(`skipReason: "sensitive_topic"`, with the `category`). The check is lexical and
|
|
328
|
+
conservative: phrases, not single words ("kill a Python process", "Suicide Squad" and
|
|
329
|
+
"cut my hair" do not count). It covers the same twelve languages as the freshness
|
|
330
|
+
and personalization checks, with shorter lists outside English, Spanish, Catalan,
|
|
331
|
+
French, German, Italian and Portuguese. In the gateway, the coalesced twin of such a
|
|
332
|
+
request is `estimated` (`sensitive-topic`). A false positive costs one cache hit.
|
|
299
333
|
|
|
300
334
|
**Streaming.** `generateStream()` takes the same request and hands you the answer as
|
|
301
335
|
the provider writes it, then the same `SavingsReport`:
|
|
@@ -453,6 +487,45 @@ It runs once per HTTP attempt, inside the SDK's retries: an attempt the provider
|
|
|
453
487
|
rejects reaches the SDK unchanged and is not recorded. Only message creation goes
|
|
454
488
|
through the layer; `countTokens`, batches, models and files pass untouched.
|
|
455
489
|
|
|
490
|
+
The middleware forwards your request body unchanged, history included, so it never
|
|
491
|
+
edits a turn before a thinking block. It also does not remove parameters the model
|
|
492
|
+
rejects. On Claude Sonnet 5.5, Opus 5.5, Fable 5.1 and Mythos 5.1 a forced
|
|
493
|
+
`tool_choice` (`any` or `tool`) returns a 400. On those models, and on Opus 4.7/4.8,
|
|
494
|
+
Opus 5, Sonnet 5 and Fable 5, so does a non-default `temperature`, `top_p` or `top_k`.
|
|
495
|
+
The middleware warns once per model and cause (`silenceWarnings` turns it off) and
|
|
496
|
+
the API's 400 reaches you unchanged.
|
|
497
|
+
|
|
498
|
+
<details>
|
|
499
|
+
<summary><b>Claude Sonnet 5.5, Opus 5.5, Fable 5.1: what <code>AnthropicProvider</code> sends instead</b></summary>
|
|
500
|
+
|
|
501
|
+
These models reject some request parameters with a 400 ([forcing tool
|
|
502
|
+
use](https://platform.claude.com/docs/en/agents-and-tools/tool-use/define-tools#forcing-tool-use),
|
|
503
|
+
[Sonnet 5.5](https://platform.claude.com/docs/en/models/sonnet-5-5/overview)).
|
|
504
|
+
`AnthropicProvider` sends a request they accept instead of failing:
|
|
505
|
+
|
|
506
|
+
- **Output contract.** A forced `tool_choice` returns a 400 on these models, with or
|
|
507
|
+
without thinking. When your schema meets `strict` mode, the contract travels as
|
|
508
|
+
structured output (`output_config.format`). Without your tools, the provider still
|
|
509
|
+
enforces the contract. With your tools, they stay callable with `tool_choice: auto`
|
|
510
|
+
and the final text answer comes back in the contract's shape. When your schema does
|
|
511
|
+
not meet `strict`, the contract travels as a tool the model is not forced to call,
|
|
512
|
+
with one sentence in the system prompt that tells it to answer through that tool.
|
|
513
|
+
Models that support forced tool use (Opus 5, Sonnet 5 and the 4.x models) get the
|
|
514
|
+
same request as before.
|
|
515
|
+
- **`temperature`.** A non-default value is dropped, with one warning per model,
|
|
516
|
+
because it can only return a 400. The answer is then sampled, not deterministic.
|
|
517
|
+
`temperature: 1`, the API default, is sent as is.
|
|
518
|
+
- **A model the table does not list** (a new model or a gateway alias): when the API
|
|
519
|
+
rejects either parameter, the provider repeats the request once without it and
|
|
520
|
+
remembers the model for later requests. A 400 is not billed, so the retry is not
|
|
521
|
+
counted in `retries`.
|
|
522
|
+
- **Thinking blocks.** `ChatMessage` does not carry thinking blocks, so this adapter
|
|
523
|
+
never sends one. The preserved-thinking check, which returns a 400 when a turn
|
|
524
|
+
before a thinking block has been edited, has nothing to check on its requests, even
|
|
525
|
+
when the levers that rewrite history are on.
|
|
526
|
+
|
|
527
|
+
</details>
|
|
528
|
+
|
|
456
529
|
## What you get on every request
|
|
457
530
|
|
|
458
531
|
A `SavingsReport` with the numbers behind every claim, kept in **four separate
|
|
@@ -465,6 +538,32 @@ ledgers** that are never summed into one flattering figure:
|
|
|
465
538
|
| `compressed_tokens` | information kept, but encoded with fewer tokens |
|
|
466
539
|
| `provider_cached_tokens` | tokens still in the prompt, billed/processed more cheaply |
|
|
467
540
|
|
|
541
|
+
**Your provider's own cache discount, shown apart and never billed.** With prompt
|
|
542
|
+
caching, most of what you save often comes from your provider's prefix cache
|
|
543
|
+
(Anthropic `cache_control`, OpenAI's automatic cache, Gemini's implicit cache), not
|
|
544
|
+
from the layer. `cost` prices cached tokens at the cached rate on both sides, so that
|
|
545
|
+
discount is never credited as the layer's saving. `report.nativeSavings` shows it on
|
|
546
|
+
its own, from your provider's counter at list price, so the report adds up to your
|
|
547
|
+
invoice:
|
|
548
|
+
|
|
549
|
+
```ts
|
|
550
|
+
report.nativeSavings;
|
|
551
|
+
// {
|
|
552
|
+
// provider: "anthropic", mechanism: "explicit", layerMarked: true,
|
|
553
|
+
// cacheReadTokens: 40000, cacheWriteTokens: 3000, cacheWrite1hTokens: 0,
|
|
554
|
+
// inputPer1k: 0.003, cachedInputPer1k: 0.0003, writeMultiplier: 1.25, write1hMultiplier: 2,
|
|
555
|
+
// readDiscountUsd: 0.108, // reads × (input − cached-input rate)
|
|
556
|
+
// writePremiumUsd: 0.00225, // writes × input × (multiplier − 1)
|
|
557
|
+
// netUsd: 0.10575, // can be negative: a cold write pays before any read
|
|
558
|
+
// basis: "measured",
|
|
559
|
+
// }
|
|
560
|
+
```
|
|
561
|
+
|
|
562
|
+
It is present only when a call ran and the provider reported cache reads or writes;
|
|
563
|
+
a cache hit or a coalesced request never carries it. It is not in `cost.netSavings`,
|
|
564
|
+
not in `cost.baselineEstimate`, and never in the commission base. Your dashboard shows
|
|
565
|
+
it as its own card, **Native savings unlocked**, next to the credited savings.
|
|
566
|
+
|
|
468
567
|
**Quality, kept apart from money.** `report.qualityLedger` records how the answer you
|
|
469
568
|
received earned its place: whether it came from your provider, a replay, or the answer
|
|
470
569
|
to a reworded question (with its similarity score); whether it was checked against your
|
|
@@ -551,7 +650,7 @@ changes the answer, and what they are worth is reported next to the four ledgers
|
|
|
551
650
|
never folded into `cost.*`, never part of any commission.
|
|
552
651
|
|
|
553
652
|
**`savings.deferrable`** asks for the provider's Flex tier: OpenAI, and Gemini through
|
|
554
|
-
its OpenAI‑compatible endpoint. Anthropic has no cheaper synchronous tier: its
|
|
653
|
+
`GoogleProvider` (`serviceTier: "flex"`) or its OpenAI‑compatible endpoint. Anthropic has no cheaper synchronous tier: its
|
|
555
654
|
discount lives only in Message Batches, which `layer.batches` (below) sends to.
|
|
556
655
|
Bedrock and DeepSeek off‑peak pricing are not supported.
|
|
557
656
|
|
|
@@ -681,7 +780,16 @@ const agentLayer = createSavingsLayer({
|
|
|
681
780
|
|
|
682
781
|
By default the layer writes the provider's prompt cache at two fixed places: the end
|
|
683
782
|
of the system header and the end of the history before your last user message, and
|
|
684
|
-
stops marking a prefix whose measured hit rate does not pay the write premium.
|
|
783
|
+
stops marking a prefix whose measured hit rate does not pay the write premium. On
|
|
784
|
+
Anthropic, where a write costs 1.25x, the history and a tool catalogue sent without a
|
|
785
|
+
system header are marked only when the request itself shows they will be sent again:
|
|
786
|
+
a tool-loop step after your last user message, or a conversation already past its
|
|
787
|
+
second turn. The first follow-up of a conversation (`[question, answer, question]`)
|
|
788
|
+
marks only the system header, so two-turn chats pay no premium that nothing reads.
|
|
789
|
+
In an agent loop with no system header, the end of the tool list is marked once the
|
|
790
|
+
catalogue reaches the model's minimum cacheable size (512 tokens on Claude Sonnet 5.5
|
|
791
|
+
and Opus 5.5, 1,024 on Sonnet 5, 4,096 on Haiku 4.5). That discount is the provider's
|
|
792
|
+
cache on your bytes: it is never credited as the layer's saving.
|
|
685
793
|
|
|
686
794
|
`cacheBreakpoints: "auto"` places them from what your traffic actually does. The layer
|
|
687
795
|
compares each request's prefix with the ones it sent before, per model, tool set and
|
|
@@ -727,7 +835,7 @@ Calibrated reuse and `cacheBreakpoints: "auto"` learn from your traffic, and by
|
|
|
727
835
|
what they learn lives in the process. On serverless platforms (Vercel Functions, AWS
|
|
728
836
|
Lambda) an instance may serve one request or live a few minutes, so neither would ever
|
|
729
837
|
get past its starting point. Give each one somewhere to keep it. Giving the layer a
|
|
730
|
-
`
|
|
838
|
+
`semanticCacheStore` only says where reworded questions are kept; `semanticCacheDefault: true`
|
|
731
839
|
is what switches reuse on for every request (a request's own `savings.semanticCache`
|
|
732
840
|
still wins), and without it, or `savings.semanticCache` on each request, the cache
|
|
733
841
|
stays empty and learns nothing:
|
|
@@ -749,7 +857,7 @@ async function serverlessLayer(kv: SharedKeyValue) {
|
|
|
749
857
|
const layer = createSavingsLayer({
|
|
750
858
|
provider,
|
|
751
859
|
licenseKey: process.env.BIVELIO_LICENSE_KEY,
|
|
752
|
-
semanticCache,
|
|
860
|
+
semanticCacheStore: semanticCache,
|
|
753
861
|
// Passing the cache does not switch it on; this asks for it on every request.
|
|
754
862
|
semanticCacheDefault: true,
|
|
755
863
|
cacheBreakpoints: "auto",
|
|
@@ -823,6 +931,60 @@ console.log(data.answer); // 8
|
|
|
823
931
|
|
|
824
932
|
</details>
|
|
825
933
|
|
|
934
|
+
<details>
|
|
935
|
+
<summary><b>Gemini: the native adapter</b></summary>
|
|
936
|
+
|
|
937
|
+
For Gemini, use `GoogleProvider`: it speaks Gemini's own API (`generateContent` and
|
|
938
|
+
`streamGenerateContent`), where what the layer measures is documented. Keep
|
|
939
|
+
`OpenAICompatibleProvider` for every other provider.
|
|
940
|
+
|
|
941
|
+
```ts
|
|
942
|
+
import { GoogleProvider } from "@bivelio/savings-layer";
|
|
943
|
+
|
|
944
|
+
const gemini = createSavingsLayer({
|
|
945
|
+
provider: new GoogleProvider({ apiKey: process.env.GEMINI_API_KEY, requireUsage: true }),
|
|
946
|
+
licenseKey: process.env.BIVELIO_LICENSE_KEY,
|
|
947
|
+
});
|
|
948
|
+
|
|
949
|
+
const { text: answer, report: geminiReport } = await gemini.generate({
|
|
950
|
+
model: "google/gemini-3.8-flash", // the `google/` prefix never travels
|
|
951
|
+
messages: [
|
|
952
|
+
{ role: "system", content: "Answer in one sentence." },
|
|
953
|
+
{ role: "user", content: "Why is the sky blue?" },
|
|
954
|
+
],
|
|
955
|
+
maxOutputTokens: 400,
|
|
956
|
+
reasoningEffort: "low", // → thinkingLevel "LOW" (Gemini 3) or a thinking budget (2.5)
|
|
957
|
+
});
|
|
958
|
+
console.log(answer, geminiReport.cost.actual);
|
|
959
|
+
```
|
|
960
|
+
|
|
961
|
+
What the native API gives the layer that the OpenAI-compatible endpoint hides:
|
|
962
|
+
|
|
963
|
+
- **Thinking is billed output.** Output is `candidatesTokenCount + thoughtsTokenCount`
|
|
964
|
+
(Google bills thinking as output) and the thinking is reported as reasoning. Thought
|
|
965
|
+
summaries (`thinkingConfig: { includeThoughts: true }` on the adapter) are never part
|
|
966
|
+
of `text`.
|
|
967
|
+
- **Refusals with their reason.** A blocked prompt (`promptFeedback.blockReason`) or a
|
|
968
|
+
filtered answer (`finishReason` `SAFETY`, `PROHIBITED_CONTENT`, `BLOCKLIST`, …) is
|
|
969
|
+
`finishReason: "content_filter"` with the native reason in `finishReasonDetail`:
|
|
970
|
+
measured from its usage, never cached, never credited.
|
|
971
|
+
- **Tools.** Calls come back with their `id` and `thoughtSignature`; send the assistant
|
|
972
|
+
turn back with its `toolCalls` as they came (as in the tool loop above) and the
|
|
973
|
+
signature travels back untouched. The results of one turn go back together.
|
|
974
|
+
- **Implicit caching.** Gemini 2.5 and later cache repeated prefixes on their own; the
|
|
975
|
+
cached part of the prompt (`cachedContentTokenCount`) is reported as provider-cached
|
|
976
|
+
tokens. Explicit context caching (`cachedContents`) is not used: it bills storage by
|
|
977
|
+
the hour, which a per-request report cannot price.
|
|
978
|
+
- **Output contract.** Your schema travels as written in `responseJsonSchema`. Google
|
|
979
|
+
does not document that every JSON Schema feature is enforced, so a repeat of a request
|
|
980
|
+
with a schema is not valued by reference.
|
|
981
|
+
- **Thinking level.** `reasoningEffort` follows Google's published mapping. A model that
|
|
982
|
+
rejects a level (`gemini-3.8-flash` answers 400 to `MINIMAL`) is asked again once at
|
|
983
|
+
`LOW`, and remembered; the rejected request was not billed. Thinking cannot be turned
|
|
984
|
+
off on Gemini 3 or 2.5 Pro, so `"none"` there is the lowest level, with a warning.
|
|
985
|
+
|
|
986
|
+
</details>
|
|
987
|
+
|
|
826
988
|
<details>
|
|
827
989
|
<summary><b>Output cap — with or without a contract</b></summary>
|
|
828
990
|
|
|
@@ -846,6 +1008,26 @@ request is repeated once with the other one (nothing was generated, so nothing i
|
|
|
846
1008
|
billed twice) and the model is remembered. To choose the field yourself:
|
|
847
1009
|
`new OpenAICompatibleProvider({ baseUrl, maxTokensParam: "max_completion_tokens" })`.
|
|
848
1010
|
|
|
1011
|
+
**OpenAI-compatible endpoints that are not OpenAI** (Gemini's
|
|
1012
|
+
`generativelanguage.googleapis.com/v1beta/openai`, LiteLLM, vLLM, Ollama, a router).
|
|
1013
|
+
`prompt_cache_key`, OpenAI's prefix-cache routing hint, goes only to OpenAI's own API:
|
|
1014
|
+
Gemini answers 400 to a field it does not know, and up to 0.4.31 every request with a
|
|
1015
|
+
cached prefix failed there. A proxy in front of OpenAI that forwards it can opt in with
|
|
1016
|
+
`sendPromptCacheKey: true`. Any optional field an endpoint still rejects by name
|
|
1017
|
+
(`prompt_cache_key`, `service_tier`, `seed`, `reasoning_effort`) is dropped, the request
|
|
1018
|
+
repeated once and the model remembered; `reasoning_effort` is never dropped on OpenAI's
|
|
1019
|
+
own API, where its 400 is the answer. Gemini leaves its thinking tokens out of
|
|
1020
|
+
`completion_tokens` and bills them as output, so the layer counts output as
|
|
1021
|
+
`total_tokens − prompt_tokens` whenever the total is larger (and reports the difference
|
|
1022
|
+
as reasoning); on OpenAI's API `completion_tokens` already include reasoning and nothing
|
|
1023
|
+
is added. Gemini stops a filtered answer with `finish_reason: "content_filter: OTHER"`
|
|
1024
|
+
(or another reason): it is `finishReason: "content_filter"` (the adapter keeps the
|
|
1025
|
+
reason, which reaches you as `result.finishReasonDetail` and is named in the report's
|
|
1026
|
+
cache trace; an Anthropic refusal brings its `stop_details.category` there too), and like every
|
|
1027
|
+
provider refusal it is measured, never
|
|
1028
|
+
cached and never credited. Google does not document that such refusals are free, so
|
|
1029
|
+
they are costed from their `usage` like any call.
|
|
1030
|
+
|
|
849
1031
|
The cap is yours, not a saving: it is never credited as output avoided. An answer
|
|
850
1032
|
it cut (`finishReason: "length"`) is not cached, and
|
|
851
1033
|
`report.qualityLedger.truncation` says the limit was yours.
|
|
@@ -904,6 +1086,21 @@ console.log(res.text);
|
|
|
904
1086
|
the SDK warns once.
|
|
905
1087
|
- **Errors.** When a tool fails, put the error in `content` and set `isError: true`.
|
|
906
1088
|
Anthropic receives it as `is_error`; providers without such a flag get the text.
|
|
1089
|
+
- **Gemini 3's thought signature.** With `GoogleProvider` it travels in each call's
|
|
1090
|
+
`thoughtSignature` and comes back in `call.extraContent` in the same shape as below,
|
|
1091
|
+
so a conversation can move between the two Gemini adapters. Through its
|
|
1092
|
+
OpenAI-compatible endpoint Gemini
|
|
1093
|
+
returns each call with `extra_content.google.thought_signature` and rejects the next
|
|
1094
|
+
turn with a 400 if the call comes back without it. It arrives in `call.extraContent`;
|
|
1095
|
+
pushing `res.toolCalls` back as they came (as above) sends it back untouched. It is
|
|
1096
|
+
opaque and not part of any cache key.
|
|
1097
|
+
- **gpt-5.6-sol, gpt-5.6-terra and gpt-5.6-luna with tools need `reasoningEffort: "none"`.**
|
|
1098
|
+
On Chat Completions OpenAI rejects function tools for these models unless
|
|
1099
|
+
`reasoning_effort` is `"none"` («Function tools with reasoning_effort are not supported for gpt-5.6-sol in
|
|
1100
|
+
/v1/chat/completions»). Declare it on the request:
|
|
1101
|
+
`layer.generate({ model: "openai/gpt-5.6-sol", messages, tools, reasoningEffort: "none" })`.
|
|
1102
|
+
The layer never picks an effort itself: without the field nothing is sent. It is part
|
|
1103
|
+
of the cache key, like `temperature` and `seed`.
|
|
907
1104
|
|
|
908
1105
|
A tool‑call turn is never served from cache and never counted as an avoided call:
|
|
909
1106
|
an action that has not run yet cannot be a saving. The answer that follows a tool loop
|
|
@@ -915,10 +1112,11 @@ difference.
|
|
|
915
1112
|
**Which tools reach the model.** The layer never cuts a catalogue it cannot rank. If
|
|
916
1113
|
your tools declare no `intents` and no `risk` (the usual shape of MCP and OpenAI tool
|
|
917
1114
|
lists), or you pass `savings: { router: false }`, every tool goes to the provider
|
|
918
|
-
exactly as you sent it, and the layer credits nothing for the tool block.
|
|
919
|
-
|
|
920
|
-
|
|
921
|
-
|
|
1115
|
+
exactly as you sent it, and the layer credits nothing for the tool block. When your
|
|
1116
|
+
tools declare `intents`/`risk`, the selector holds back write tools from a request
|
|
1117
|
+
that is not an action and, since 0.4.33, sends every other tool: it prunes by
|
|
1118
|
+
relevance only if you ask for it with `new ToolSelector({ maxTools })` (up to 0.4.32
|
|
1119
|
+
that was the default, with 8). Whatever it leaves out, the model cannot call in that
|
|
922
1120
|
request: the report's `tool-selector` trace lists it, the tokens count as eliminated
|
|
923
1121
|
input, and the SDK warns once per process (`silenceWarnings` turns the warning off).
|
|
924
1122
|
|
|
@@ -998,7 +1196,7 @@ console.log(tracedReport.cost.basis); // the span carries the same figures
|
|
|
998
1196
|
- **Money is kept apart by basis.** The money saved is
|
|
999
1197
|
`com.bivelio.cost.saved.measured`, `.reference` or `.estimated` — one key per span,
|
|
1000
1198
|
the one of its basis — so a sum over one key never mixes measured and estimated
|
|
1001
|
-
money; `unmeasured` spans carry counts and no money. `report.cost.
|
|
1199
|
+
money; `unmeasured` spans carry counts and no money. `report.cost.netSavingsRatio` is a
|
|
1002
1200
|
ratio and goes out as `com.bivelio.cost.net_savings_ratio`; the money is
|
|
1003
1201
|
`baselineEstimate − actual`, and that is what `saved.*` holds.
|
|
1004
1202
|
- **None of this talks to BiVelio.** Spans go to your exporter; files stay where you
|
|
@@ -1033,7 +1231,7 @@ console.log(tracedReport.cost.basis); // the span carries the same figures
|
|
|
1033
1231
|
| `gen_ai.usage.input_tokens`, `.output_tokens`, `.cache_read.input_tokens` | your provider's counter; absent when it reported no usage |
|
|
1034
1232
|
| `com.bivelio.cost.basis`, `.currency`, `.baseline`, `.actual` | where the money comes from, its currency (USD by default), and both sides |
|
|
1035
1233
|
| `com.bivelio.cost.saved.measured` \| `.reference` \| `.estimated` | the money saved, in the key of its basis only |
|
|
1036
|
-
| `com.bivelio.cost.net_savings_ratio` | `cost.
|
|
1234
|
+
| `com.bivelio.cost.net_savings_ratio` | `cost.netSavingsRatio`, a 0..1 ratio — not money |
|
|
1037
1235
|
| `com.bivelio.calls.*`, `com.bivelio.tokens.*` | the four ledgers: avoided calls, eliminated, compressed and provider‑cached tokens |
|
|
1038
1236
|
| `com.bivelio.request_id`, `.model.id`, `.cache.result`, `.quality.status` | ids and outcomes |
|
|
1039
1237
|
|
|
@@ -1148,12 +1346,30 @@ BVSALA_SEAT_ID=worker-03 # stable per replica; [A-Za-z0-9._:/-], max 128 chars
|
|
|
1148
1346
|
|
|
1149
1347
|
## Licensing & privacy
|
|
1150
1348
|
|
|
1151
|
-
BV‑SALA is commercial software
|
|
1152
|
-
|
|
1153
|
-
|
|
1154
|
-
[LICENSE](./LICENSE) §6 and in
|
|
1155
|
-
|
|
1156
|
-
|
|
1349
|
+
BV‑SALA is commercial software. The plans are **Measure** (free), **Pro** (per seat),
|
|
1350
|
+
**Team** (a number of seats included, plus each extra seat) and **Enterprise** (by
|
|
1351
|
+
contract); Pro and Team are paid whether or not you save and carry **no commission** on
|
|
1352
|
+
your savings. The prices are at the top of this page, in [LICENSE](./LICENSE) §6 and in
|
|
1353
|
+
the Published Pricing it refers to,
|
|
1354
|
+
**[savings.bivelio.com/pricing](https://savings.bivelio.com/pricing)**. No trial. There is one free license,
|
|
1355
|
+
**Measure** — no card, one organization per verified account, up to 5 seats and 20,000
|
|
1356
|
+
requests a day: the layer runs in **shadow mode**. Every request goes to your provider
|
|
1357
|
+
exactly as you built it, nothing is served from any cache, no lever is applied, and
|
|
1358
|
+
`report.measure.wouldHaveSaved` (with `basis: "shadow"`) says what the exact cache and
|
|
1359
|
+
in‑flight coalescing would have saved on that request, valued with your provider's own
|
|
1360
|
+
counter — a lower bound, never part of any commission. The levers need a paid plan: once
|
|
1361
|
+
you upgrade, a layer with `serviceAccountKey` and `licenseServerUrl` picks up the paid
|
|
1362
|
+
license by itself and turns them on without a restart.
|
|
1363
|
+
|
|
1364
|
+
The earlier plan — a lower per‑seat fee plus a tiered commission on measured savings
|
|
1365
|
+
only — is **closed to new customers**; a subscription already on it keeps it unchanged.
|
|
1366
|
+
On that plan your first month carries no commission (the first calendar month, UTC, in
|
|
1367
|
+
which your organization reports usage); after that the commission is charged on the
|
|
1368
|
+
measured saving only — never from estimates — at a tiered rate from 10 % down to 4 %, so
|
|
1369
|
+
you keep at least 90 % of every euro saved (before rounding to the cent; the seat fee is
|
|
1370
|
+
separate).
|
|
1371
|
+
|
|
1372
|
+
The license is a signed token your SDK and the
|
|
1157
1373
|
gateway verify **offline**; the managed control plane and the revocation kill‑switch
|
|
1158
1374
|
are served from BiVelio. Your LLM traffic goes **straight to your provider** —
|
|
1159
1375
|
BiVelio receives per‑request usage (counts, cost, the model id and opaque ids), never
|