@bivelio/savings-layer 0.4.24 → 0.4.25
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +105 -21
- package/README.md +391 -22
- package/dist/adapters/ai-sdk.d.ts +46 -0
- package/dist/adapters/ai-sdk.js +84 -0
- package/dist/adapters/anthropic-sdk.d.ts +35 -0
- package/dist/adapters/anthropic-sdk.js +99 -0
- package/dist/adapters/langchain.d.ts +29 -0
- package/dist/adapters/langchain.js +57 -0
- package/dist/adapters/mastra.d.ts +41 -0
- package/dist/adapters/mastra.js +74 -0
- package/dist/gateway/bin.js +662 -605
- package/dist/index.d.ts +2260 -1531
- package/dist/index.js +1066 -856
- package/docs/terminales/README.md +117 -9
- package/docs/terminales/claude-code.md +7 -2
- package/package.json +70 -4
package/LICENSE
CHANGED
|
@@ -9,11 +9,12 @@ copying, or using the Software, you agree to this Agreement. If you do not agree
|
|
|
9
9
|
not install or use the Software.
|
|
10
10
|
|
|
11
11
|
1. LICENSE GRANT. Subject to your continuous compliance with this Agreement and to
|
|
12
|
-
an active
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
12
|
+
an active BiVelio subscription (paid, or granted by BiVelio at no charge, for
|
|
13
|
+
example through a promotional code or a pilot) with a valid license key
|
|
14
|
+
("License Key"), BiVelio grants you a limited, non-exclusive, non-transferable,
|
|
15
|
+
non-sublicensable, revocable license to install and use the Software, as
|
|
16
|
+
published, solely for your own internal business purposes, and only for the
|
|
17
|
+
number of seats covered by your subscription. The Software is licensed, not sold.
|
|
17
18
|
|
|
18
19
|
2. LICENSE KEY; FAIL-CLOSED OPERATION. The Software requires a valid, active License
|
|
19
20
|
Key and does not operate without one. Expiry, non-payment, revocation, suspension,
|
|
@@ -25,13 +26,13 @@ not install or use the Software.
|
|
|
25
26
|
3. RESTRICTIONS. Except as expressly permitted above, you will not: (a) distribute,
|
|
26
27
|
publish, resell, rent, lease, lend, offer as a hosted or managed service, or
|
|
27
28
|
otherwise make the Software available to any third party; (b) sublicense, assign,
|
|
28
|
-
or transfer your rights; (c) use the Software for more seats than
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
notice; (e) reverse engineer, decompile, or disassemble the Software,
|
|
32
|
-
the limited extent applicable law expressly permits notwithstanding
|
|
33
|
-
restriction; or (f) use the Software to develop, train, or improve a product
|
|
34
|
-
service that competes with the Software.
|
|
29
|
+
or transfer your rights; (c) use the Software for more seats than your
|
|
30
|
+
subscription covers, or share a License Key across more machines or service
|
|
31
|
+
identities than it covers; (d) remove, alter, or obscure any proprietary or
|
|
32
|
+
copyright notice; (e) reverse engineer, decompile, or disassemble the Software,
|
|
33
|
+
except to the limited extent applicable law expressly permits notwithstanding
|
|
34
|
+
this restriction; or (f) use the Software to develop, train, or improve a product
|
|
35
|
+
or service that competes with the Software.
|
|
35
36
|
|
|
36
37
|
4. OWNERSHIP. The Software is the exclusive property of BiVelio and its licensors and
|
|
37
38
|
embodies valuable proprietary information and trade secrets. BiVelio retains all
|
|
@@ -39,27 +40,110 @@ not install or use the Software.
|
|
|
39
40
|
rights therein. No rights are granted to you except as expressly set out in this
|
|
40
41
|
Agreement, whether by implication, estoppel, or otherwise.
|
|
41
42
|
|
|
42
|
-
5. YOUR DATA; PRIVACY.
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
43
|
+
5. YOUR DATA; PRIVACY.
|
|
44
|
+
(a) What the Software never sends to BiVelio. The Software never transmits to
|
|
45
|
+
BiVelio the content of your requests or responses (including prompts, system
|
|
46
|
+
instructions, messages, tool definitions, tool arguments and results, retrieved
|
|
47
|
+
documents, and completions) or the credentials you use with your LLM provider
|
|
48
|
+
(API keys, OAuth tokens, or other authorization headers). That content and those
|
|
49
|
+
credentials travel only between your systems and the LLM provider you configure.
|
|
50
|
+
(b) What the Software sends to BiVelio. The SDK contacts BiVelio only when you
|
|
51
|
+
configure a license server or metering URL; the local gateway always does. With a
|
|
52
|
+
BiVelio service-account key, the Software sends BiVelio's control plane usage
|
|
53
|
+
records for the requests it meters, each limited to:
|
|
54
|
+
(i) identifiers: a request identifier generated on your side (at random, or by
|
|
55
|
+
your own generator); for a repeated request valued by reference, the
|
|
56
|
+
identifier and timestamp of the earlier request whose provider counter is
|
|
57
|
+
used; a seat identifier, which is either a value you set or an opaque value
|
|
58
|
+
computed on your machine as a keyed hash (HMAC-SHA-256 wherever the
|
|
59
|
+
runtime provides it), keyed with your service-account key, of the
|
|
60
|
+
platform, CPU architecture, runtime executable path, host name,
|
|
61
|
+
operating-system user name, and home directory (those underlying values
|
|
62
|
+
are never sent); for traffic through the local gateway,
|
|
63
|
+
the session and sub-agent identifiers that your coding agent attaches to
|
|
64
|
+
its own requests (for example, Claude Code's session id), only when they
|
|
65
|
+
are plain identifiers; an organization hint; and a flag marking CI or
|
|
66
|
+
ephemeral runners;
|
|
67
|
+
(ii) the identifier of the model that served the request (the model named in
|
|
68
|
+
the request or, where the Software sent the request to another model under
|
|
69
|
+
a setting you enabled, that model; for a self-hosted or Azure deployment,
|
|
70
|
+
the name you gave it) and the time of the request;
|
|
71
|
+
(iii) counts: LLM calls executed and avoided; input and output tokens before
|
|
72
|
+
and after the Software's optimizations, including tokens removed,
|
|
73
|
+
compressed, or read from the provider's cache; and the cache outcome; and
|
|
74
|
+
(iv) cost figures computed by the Software from its price catalogue (or from
|
|
75
|
+
a cost model you supply): baseline cost, actual cost, the net-saving
|
|
76
|
+
ratio, the basis of the figures (measured, reference, estimated, shadow,
|
|
77
|
+
or unmeasured), and the value, at the provider's cache-read price, of
|
|
78
|
+
input tokens your provider reported as cleared from context.
|
|
79
|
+
Each batch of records also carries the Software's surface and version (for
|
|
80
|
+
example, "gateway/0.4.23"). BiVelio's ingest rejects any record that carries a
|
|
81
|
+
field not listed above. Separately, the Software fetches from BiVelio its current
|
|
82
|
+
license token, the signed license revocation list and, for the gateway, your
|
|
83
|
+
organization's feature entitlements; these requests carry your service-account
|
|
84
|
+
key where authentication is required, and no request data. Like any HTTPS
|
|
85
|
+
request, all of the above also reveals ordinary network metadata, such as your
|
|
86
|
+
IP address. If you enable the SDK's optional verification, which is off by
|
|
87
|
+
default, the SDK also sends BiVelio, through a separate channel and for each
|
|
88
|
+
request it samples: a label marking the record as a verification sample; the
|
|
89
|
+
request identifier and time; the model requested and the model that served it;
|
|
90
|
+
the cache outcome; the category of the request and the outcome of the check;
|
|
91
|
+
the input, cached-input, and output token counts and the cost of the served
|
|
92
|
+
request and of the unoptimized copy the SDK sent to your LLM provider; how
|
|
93
|
+
that copy finished and whether it met the response contract; and the per-token
|
|
94
|
+
prices applied; together with daily counts of requests per category and a
|
|
95
|
+
random batch identifier. That channel likewise rejects any field not listed
|
|
96
|
+
here, and none of it is used to calculate the fees in Section 6.
|
|
97
|
+
(c) BiVelio uses this data to operate licensing and revocation, show your
|
|
98
|
+
dashboard, count billable seats, and calculate the fees in Section 6. Each
|
|
99
|
+
field, and how long BiVelio keeps it, is further described at
|
|
100
|
+
https://savings.bivelio.com/privacy.
|
|
46
101
|
|
|
47
|
-
6.
|
|
102
|
+
6. FEES. Your subscription is subject to BiVelio's pricing as published at
|
|
103
|
+
https://savings.bivelio.com/pricing (the "Published Pricing") for each billing
|
|
104
|
+
period:
|
|
105
|
+
(a) a subscription fee per seat, payable whether or not the Software produces any
|
|
106
|
+
saving; and (b) a commission, at tiered marginal rates, charged only on measured
|
|
107
|
+
savings, meaning savings whose basis is labelled "measured" or "reference" in
|
|
108
|
+
your usage records (Section 5(b)(iv)). A "measured" saving is established from
|
|
109
|
+
your LLM provider's own usage counter for that request; a "reference" saving, for
|
|
110
|
+
a repeated request the Software served without calling your provider, from your
|
|
111
|
+
provider's counter for the identical earlier request that did run. Both are
|
|
112
|
+
valued at the per-token prices the Software applied to that request: those of a
|
|
113
|
+
cost model you configure, where you configure one, and otherwise your provider's
|
|
114
|
+
list prices in the price catalogue included in the Software. Savings labelled
|
|
115
|
+
"estimated", "shadow", or "unmeasured" never carry commission. Traffic the
|
|
116
|
+
Software identifies as a flat-rate provider subscription (a Claude or ChatGPT
|
|
117
|
+
subscription used through the local gateway) is metered but credited no measured
|
|
118
|
+
saving, so it carries no commission. A period with no measured savings carries no
|
|
119
|
+
commission. The first calendar month (UTC) in which your organization reports any
|
|
120
|
+
usage is a calibration period, however little of that month remains, and carries
|
|
121
|
+
no commission. Nor is commission charged for a period that BiVelio closes while
|
|
122
|
+
your account is exempt from it. Your account is exempt if BiVelio grants it an
|
|
123
|
+
exemption and, until a payment is made on the account, if its subscription was
|
|
124
|
+
activated with a promotional code that waives the subscription fee in full for
|
|
125
|
+
its entire term. For a calibration or exempt period the statement still shows the
|
|
126
|
+
commission that would have applied. Each period's commission is
|
|
127
|
+
itemized on a statement in your dashboard, together with the exchange rate
|
|
128
|
+
applied. Rates, tier thresholds, and currency are those of the Published Pricing
|
|
129
|
+
and are not restated here.
|
|
130
|
+
|
|
131
|
+
7. TERM AND TERMINATION. This Agreement remains in effect while your subscription is
|
|
48
132
|
active. BiVelio may terminate this Agreement immediately if you breach it. Upon
|
|
49
133
|
expiry or termination you must cease all use of the Software and destroy your
|
|
50
|
-
copies. Sections 2 through
|
|
134
|
+
copies. Sections 2 through 10 survive termination.
|
|
51
135
|
|
|
52
|
-
|
|
136
|
+
8. DISCLAIMER OF WARRANTY. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY
|
|
53
137
|
KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
|
54
138
|
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NONINFRINGEMENT.
|
|
55
139
|
|
|
56
|
-
|
|
140
|
+
9. LIMITATION OF LIABILITY. TO THE MAXIMUM EXTENT PERMITTED BY LAW, IN NO EVENT SHALL
|
|
57
141
|
BIVELIO INC. BE LIABLE FOR ANY INDIRECT, INCIDENTAL, SPECIAL, CONSEQUENTIAL, OR
|
|
58
142
|
EXEMPLARY DAMAGES, OR FOR ANY LOSS OF PROFITS, REVENUE, OR DATA, ARISING FROM OR
|
|
59
143
|
RELATED TO THE SOFTWARE OR THIS AGREEMENT. BIVELIO'S TOTAL LIABILITY SHALL NOT
|
|
60
144
|
EXCEED THE FEES YOU PAID FOR THE SOFTWARE IN THE TWELVE MONTHS PRECEDING THE CLAIM.
|
|
61
145
|
|
|
62
|
-
|
|
146
|
+
10. GENERAL. This Agreement is governed by the laws of the State of Delaware, USA,
|
|
63
147
|
without regard to its conflict-of-laws rules. If any provision is held
|
|
64
148
|
unenforceable, the remaining provisions remain in effect. This Agreement, together
|
|
65
149
|
with the applicable BiVelio subscription terms, constitutes the entire agreement
|
package/README.md
CHANGED
|
@@ -19,17 +19,24 @@
|
|
|
19
19
|
Every request your app sends to an AI is paid for by the text going in and out.
|
|
20
20
|
**Savings** sits on that path and lets through only what's needed — skipping calls
|
|
21
21
|
that don't have to happen, and trimming what does — so you pay for less. It is not a
|
|
22
|
-
promise you take on faith: **every saving we charge on is measured with
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
22
|
+
promise you take on faith: **every saving we charge on is measured with your provider's
|
|
23
|
+
own usage counter, the one your bill is built from** (for a repeat served without
|
|
24
|
+
calling the provider, the counter of the identical call that ran), and every saving —
|
|
25
|
+
measured or estimated, and labelled as such — is handed back to you request by request.
|
|
26
|
+
|
|
27
|
+
**A small per‑seat fee, and a commission only on savings measured against your
|
|
28
|
+
provider's counter.** The seat fee is paid whether or not you save. Your
|
|
29
|
+
first month carries no commission at all — it is the first calendar month (UTC) in
|
|
30
|
+
which your organization reports usage, so starting late in a month makes it a short one —
|
|
31
|
+
and at its close you see the statement of what the commission would have been. After
|
|
32
|
+
that, the commission is charged on the **measured** saving only — never from estimates:
|
|
33
|
+
a tiered rate from 10 % down to 4 % of what you saved, so **you keep
|
|
34
|
+
at least 90 % of every euro saved** (before the invoice is rounded to the cent; the
|
|
35
|
+
seat fee is separate). A month with no measured saving is zero commission. And your
|
|
36
|
+
prompts and completions **never reach BiVelio**: what BiVelio receives is per‑request
|
|
37
|
+
usage — token counts, cost, the model id and opaque ids — never your content
|
|
38
|
+
([LICENSE](./LICENSE) §5 lists every field; [savings.bivelio.com/privacy](https://savings.bivelio.com/privacy)
|
|
39
|
+
explains each one and how long it is kept).
|
|
33
40
|
|
|
34
41
|
### For engineers, in one line
|
|
35
42
|
|
|
@@ -43,9 +50,11 @@ goes straight from your app to your provider; the SDK is never a middleman serve
|
|
|
43
50
|
> BV‑SALA is fail‑closed and executes only with an active BiVelio license key.
|
|
44
51
|
> The license signature is verified **offline**; with `licenseServerUrl` set, the SDK
|
|
45
52
|
> also calls `savings.bivelio.com` for the revocation list, license renewal and usage
|
|
46
|
-
> metering. **Your prompts never
|
|
47
|
-
> counts, cost, the model id
|
|
48
|
-
> and the
|
|
53
|
+
> metering. **Your prompts, completions and provider keys never reach BiVelio** — what
|
|
54
|
+
> travels is per‑request usage: token counts, cost, the model id, a request id and
|
|
55
|
+
> opaque seat and session ids (the exact list is in [LICENSE](./LICENSE) §5). Use is
|
|
56
|
+
> governed by the LICENSE, which also sets out the seat fee and the commission, and the
|
|
57
|
+
> BiVelio Terms. Not a wrapper around any third‑party library.
|
|
49
58
|
|
|
50
59
|
## Where it installs
|
|
51
60
|
|
|
@@ -83,6 +92,20 @@ subscription there is no per-token invoice, so that traffic is metered at
|
|
|
83
92
|
list-price equivalent and never carries commission; the per-seat license applies
|
|
84
93
|
as for any other use.
|
|
85
94
|
|
|
95
|
+
**The gateway forwards what your terminal sends.** With no lever armed, your
|
|
96
|
+
provider receives the request body byte for byte and every header your client
|
|
97
|
+
sends (see the limits): `cache_control` and its 1-hour TTL, tool search and
|
|
98
|
+
`tool_reference` blocks, the advisor, every `anthropic-beta` value. An armed lever
|
|
99
|
+
changes only what it declares, and a request it cannot rewrite exactly travels
|
|
100
|
+
untouched. A test suite checks this against a fake upstream on every change
|
|
101
|
+
(`test/gateway-no-empeora.test.ts`; the repository is private, so the linked
|
|
102
|
+
section lists exactly what it checks). The gateway also logs, once per session,
|
|
103
|
+
requests that come out worse than your client asked for: cache markers the
|
|
104
|
+
provider ignored, a 1-hour TTL written at 5 minutes upstream, and tool search off
|
|
105
|
+
(Claude Code turns it off by default behind any proxy, this one included;
|
|
106
|
+
`bvsala claude` turns it back on). What it checks, and its limits:
|
|
107
|
+
[docs/terminales/README.md](./docs/terminales/README.md#what-the-gateway-guarantees-it-forwards-what-your-client-sends).
|
|
108
|
+
|
|
86
109
|
### Python and any other language: the local gateway
|
|
87
110
|
|
|
88
111
|
The same gateway sits in front of any SDK that lets you set a base URL, whatever
|
|
@@ -228,6 +251,135 @@ still writing cancels its request, and nothing is cached or recorded; once its a
|
|
|
228
251
|
is complete, or when it arrives in one piece, nothing is cancelled.
|
|
229
252
|
`await stream.result` tells you which happened.
|
|
230
253
|
|
|
254
|
+
## Inside your framework
|
|
255
|
+
|
|
256
|
+
Already on an agent framework? The layer plugs in at that framework's own extension
|
|
257
|
+
point, from a subpath of this same package. The four adapters behave alike:
|
|
258
|
+
|
|
259
|
+
- **They answer without calling the model when they can** — a computed answer (the
|
|
260
|
+
arithmetic above), an exact repeat of a request that already ran, an identical
|
|
261
|
+
request already in flight — and otherwise let your framework make exactly the call
|
|
262
|
+
it was going to make. They never rewrite what your framework sends.
|
|
263
|
+
- **Tools are treated as having side effects** unless you list them in
|
|
264
|
+
`readOnlyTools`: a request that offers any other tool is never answered from a
|
|
265
|
+
cache, because the action has to run.
|
|
266
|
+
- **Fail‑closed, like the SDK**: without an active license the call fails with
|
|
267
|
+
`LicenseError` (a framework may wrap it, keeping its name and message) and the
|
|
268
|
+
model is not called. **Put the adapter last**, so nothing sits between it and the
|
|
269
|
+
model.
|
|
270
|
+
|
|
271
|
+
Every model call gets the same `SavingsReport` as with the SDK — its `basis` says
|
|
272
|
+
whether a figure comes from your provider's own counter — and `onReport` receives it
|
|
273
|
+
too.
|
|
274
|
+
|
|
275
|
+
### Vercel AI SDK
|
|
276
|
+
|
|
277
|
+
`pnpm add @bivelio/savings-layer ai @ai-sdk/openai` — a `LanguageModelMiddleware`
|
|
278
|
+
for `ai` 6 or 7:
|
|
279
|
+
|
|
280
|
+
```ts
|
|
281
|
+
import { generateText, wrapLanguageModel } from "ai";
|
|
282
|
+
import { openai } from "@ai-sdk/openai";
|
|
283
|
+
import { savingsMiddleware, getSavingsReport } from "@bivelio/savings-layer/ai-sdk";
|
|
284
|
+
|
|
285
|
+
const savings = savingsMiddleware({
|
|
286
|
+
licenseKey: process.env.BIVELIO_LICENSE_KEY,
|
|
287
|
+
licenseServerUrl: "https://savings.bivelio.com",
|
|
288
|
+
serviceAccountKey: process.env.BIVELIO_SERVICE_KEY,
|
|
289
|
+
readOnlyTools: ["get_weather"],
|
|
290
|
+
});
|
|
291
|
+
const model = wrapLanguageModel({ model: openai("gpt-4o-mini"), middleware: savings }); // last in the list
|
|
292
|
+
|
|
293
|
+
const result = await generateText({
|
|
294
|
+
model,
|
|
295
|
+
prompt: "What is 2 + 2 * 3?",
|
|
296
|
+
providerOptions: { bivelio: { context: { sessionId: "user-42" } } },
|
|
297
|
+
});
|
|
298
|
+
console.log(getSavingsReport(result)?.cost.basis); // "estimated": answered without calling the model
|
|
299
|
+
```
|
|
300
|
+
|
|
301
|
+
For `streamText`, read it with `getSavingsReport({ providerMetadata: await result.providerMetadata })`.
|
|
302
|
+
|
|
303
|
+
### Mastra
|
|
304
|
+
|
|
305
|
+
`pnpm add @bivelio/savings-layer @mastra/core` — an input processor for
|
|
306
|
+
`@mastra/core` 1.x:
|
|
307
|
+
|
|
308
|
+
```ts
|
|
309
|
+
import { Agent } from "@mastra/core/agent";
|
|
310
|
+
import { savingsProcessor } from "@bivelio/savings-layer/mastra";
|
|
311
|
+
|
|
312
|
+
const bivelio = savingsProcessor({
|
|
313
|
+
licenseKey: process.env.BIVELIO_LICENSE_KEY,
|
|
314
|
+
licenseServerUrl: "https://savings.bivelio.com",
|
|
315
|
+
serviceAccountKey: process.env.BIVELIO_SERVICE_KEY,
|
|
316
|
+
onReport: (r) => console.log(r.cost.basis),
|
|
317
|
+
});
|
|
318
|
+
const agent = new Agent({
|
|
319
|
+
id: "support",
|
|
320
|
+
name: "Support",
|
|
321
|
+
instructions: "Answer briefly.",
|
|
322
|
+
model: "openai/gpt-4o-mini",
|
|
323
|
+
inputProcessors: [bivelio], // last in the list
|
|
324
|
+
});
|
|
325
|
+
console.log((await agent.generate("What is 2 + 2 * 3?")).text); // "8"
|
|
326
|
+
```
|
|
327
|
+
|
|
328
|
+
It covers `generate()` and `stream()`, not `generateLegacy()`/`streamLegacy()` or a
|
|
329
|
+
durable agent that resumes in another process; identical requests in flight are not
|
|
330
|
+
shared here. `emitReportPart: true` also streams each report as a
|
|
331
|
+
`data-bivelio-savings-report` part.
|
|
332
|
+
|
|
333
|
+
### LangChain.js
|
|
334
|
+
|
|
335
|
+
`pnpm add @bivelio/savings-layer langchain @langchain/core` — an agent middleware for
|
|
336
|
+
`createAgent` in `langchain` 1.x:
|
|
337
|
+
|
|
338
|
+
```ts
|
|
339
|
+
import { createAgent } from "langchain";
|
|
340
|
+
import { savingsAgentMiddleware, getSavingsReportFromMessage } from "@bivelio/savings-layer/langchain";
|
|
341
|
+
|
|
342
|
+
const agentSavings = savingsAgentMiddleware({
|
|
343
|
+
licenseKey: process.env.BIVELIO_LICENSE_KEY,
|
|
344
|
+
licenseServerUrl: "https://savings.bivelio.com",
|
|
345
|
+
serviceAccountKey: process.env.BIVELIO_SERVICE_KEY,
|
|
346
|
+
});
|
|
347
|
+
const lcAgent = createAgent({ model: "openai:gpt-4o-mini", tools: [], middleware: [agentSavings] }); // last
|
|
348
|
+
const state = await lcAgent.invoke({ messages: [{ role: "user", content: "What is 2 + 2 * 3?" }] });
|
|
349
|
+
console.log(getSavingsReportFromMessage(state.messages.at(-1))?.cost.basis);
|
|
350
|
+
```
|
|
351
|
+
|
|
352
|
+
The model call is measured when it finishes; tokens you stream with
|
|
353
|
+
`streamMode: "messages"` keep coming from the model as usual. A chat model with its
|
|
354
|
+
own `cache` set is reported as `unmeasured`: its counter may be a cached copy.
|
|
355
|
+
|
|
356
|
+
### Anthropic TypeScript SDK
|
|
357
|
+
|
|
358
|
+
`pnpm add @bivelio/savings-layer @anthropic-ai/sdk` — a client middleware for
|
|
359
|
+
`@anthropic-ai/sdk` 0.103 or later:
|
|
360
|
+
|
|
361
|
+
```ts
|
|
362
|
+
import Anthropic from "@anthropic-ai/sdk";
|
|
363
|
+
import { anthropicSavings } from "@bivelio/savings-layer/anthropic-sdk";
|
|
364
|
+
|
|
365
|
+
const claudeSavings = anthropicSavings({
|
|
366
|
+
licenseKey: process.env.BIVELIO_LICENSE_KEY,
|
|
367
|
+
licenseServerUrl: "https://savings.bivelio.com",
|
|
368
|
+
serviceAccountKey: process.env.BIVELIO_SERVICE_KEY,
|
|
369
|
+
});
|
|
370
|
+
const client = new Anthropic({ middleware: [claudeSavings.middleware] }); // last in the list
|
|
371
|
+
const message = await client.messages.create({
|
|
372
|
+
model: "claude-opus-5",
|
|
373
|
+
max_tokens: 256,
|
|
374
|
+
messages: [{ role: "user", content: "What is 2 + 2 * 3?" }],
|
|
375
|
+
});
|
|
376
|
+
console.log(claudeSavings.reportFor(message)?.cost.basis);
|
|
377
|
+
```
|
|
378
|
+
|
|
379
|
+
It runs once per HTTP attempt, inside the SDK's retries: an attempt the provider
|
|
380
|
+
rejects reaches the SDK unchanged and is not recorded. Only message creation goes
|
|
381
|
+
through the layer; `countTokens`, batches, models and files pass untouched.
|
|
382
|
+
|
|
231
383
|
## What you get on every request
|
|
232
384
|
|
|
233
385
|
A `SavingsReport` with the numbers behind every claim, kept in **four separate
|
|
@@ -240,6 +392,16 @@ ledgers** that are never summed into one flattering figure:
|
|
|
240
392
|
| `compressed_tokens` | information kept, but encoded with fewer tokens |
|
|
241
393
|
| `provider_cached_tokens` | tokens still in the prompt, billed/processed more cheaply |
|
|
242
394
|
|
|
395
|
+
**Quality, kept apart from money.** `report.qualityLedger` records how the answer you
|
|
396
|
+
received earned its place: whether it came from your provider, a replay, or the answer
|
|
397
|
+
to a reworded question (with its similarity score); whether it was checked against your
|
|
398
|
+
output contract, repaired or cut off by the output limit; and what the model cascade
|
|
399
|
+
decided. It holds no money, is never added to the four ledgers, and never leaves your
|
|
400
|
+
process. Reusing answers to reworded questions is opt-in, and by default it covers
|
|
401
|
+
single-shot requests only: requests that declare tools, carry tool calls or results, or
|
|
402
|
+
hold more than one user turn are left out unless you set
|
|
403
|
+
`savings.semanticCache: "include-agentic"`.
|
|
404
|
+
|
|
243
405
|
And a promise you can hold it to: **never worse than your JSON.** Payload encodings
|
|
244
406
|
are chosen by counting real tokens with the destination model's own tokenizer, and
|
|
245
407
|
BV‑SALA only moves away from plain JSON when the saving clearly clears the bar —
|
|
@@ -295,6 +457,65 @@ and every pair as a CSV download.
|
|
|
295
457
|
|
|
296
458
|
</details>
|
|
297
459
|
|
|
460
|
+
<details>
|
|
461
|
+
<summary><b>Provider price levers (opt-in, off by default)</b></summary>
|
|
462
|
+
|
|
463
|
+
Two request flags ask your provider for a lower price on the same tokens. Neither
|
|
464
|
+
changes the answer, and what they are worth is reported next to the four ledgers:
|
|
465
|
+
never folded into `cost.*`, never part of any commission.
|
|
466
|
+
|
|
467
|
+
**`savings.deferrable`** asks for the provider's Flex tier: OpenAI, and Gemini through
|
|
468
|
+
its OpenAI‑compatible endpoint. Anthropic has no cheaper synchronous tier (its
|
|
469
|
+
discount lives only in Message Batches, which the layer does not support yet).
|
|
470
|
+
Bedrock and DeepSeek off‑peak pricing are not supported.
|
|
471
|
+
|
|
472
|
+
- **Latency.** OpenAI publishes no latency commitment for Flex; Google publishes a
|
|
473
|
+
target of minutes. Keep deferrable requests off user‑facing paths.
|
|
474
|
+
`report.deferred.latency` quotes what the provider documents.
|
|
475
|
+
- **Risk.** Without capacity the provider refuses the request (429, or 503 on Gemini)
|
|
476
|
+
instead of waiting; OpenAI documents that the refusal is not charged. With
|
|
477
|
+
`priceLevers: { onCapacityError: "standard" }` the layer repeats it once at the
|
|
478
|
+
standard tier and price, and says so in `report.deferred.capacityFallback`.
|
|
479
|
+
- **What it is worth.** Your provider's usage counter for that call, priced at the two
|
|
480
|
+
published rate cards (standard, and the tier that actually served it), component
|
|
481
|
+
by component, with the date the prices were checked. It is `measured` only when
|
|
482
|
+
the served tier was read back from the provider's own API, the prices are list
|
|
483
|
+
prices and the counter says how much of the input was read from (or written to)
|
|
484
|
+
the prompt cache; otherwise `estimated`, and `basisReason` says why. It never
|
|
485
|
+
exceeds the difference between the two published rate cards, even with your own
|
|
486
|
+
`costModels`. A tier the provider did not confirm is never credited.
|
|
487
|
+
|
|
488
|
+
**`savings.standalone`** (GPT‑5.6 and later, on OpenAI's own API) declares that nothing
|
|
489
|
+
after the stable prefix of this request will be sent again. The provider then writes
|
|
490
|
+
its prompt cache only at the end of that prefix, or nowhere, instead of at the end of
|
|
491
|
+
your message, which on these models costs more than plain input. The trade‑off:
|
|
492
|
+
repeats and continuations of this exact prompt lose the cache reads the provider
|
|
493
|
+
default would give them, so leave it off conversation turns you will continue. Its
|
|
494
|
+
figure, `report.cacheWrites.avoidedWritePremium`, is always `estimated`: the provider
|
|
495
|
+
default was not run.
|
|
496
|
+
|
|
497
|
+
```ts
|
|
498
|
+
const leveredLayer = createSavingsLayer({
|
|
499
|
+
provider,
|
|
500
|
+
licenseKey: process.env.BIVELIO_LICENSE_KEY,
|
|
501
|
+
priceLevers: { deferrable: true, onCapacityError: "standard" }, // layer-wide defaults
|
|
502
|
+
});
|
|
503
|
+
|
|
504
|
+
const { report } = await leveredLayer.generate({
|
|
505
|
+
model: "openai/gpt-5.6-luna",
|
|
506
|
+
messages: [
|
|
507
|
+
{ role: "system", content: "…your long, fixed instructions…" },
|
|
508
|
+
{ role: "user", content: "Classify ticket 42." },
|
|
509
|
+
],
|
|
510
|
+
savings: { standalone: true }, // a request's own value always wins
|
|
511
|
+
});
|
|
512
|
+
|
|
513
|
+
report.deferred?.tierSaving; // { amount, basis, standardCost, appliedCost, pricesCheckedOn, basisReason }
|
|
514
|
+
report.cacheWrites; // { mode, breakpoints, avoidedWritePremium?, reason }
|
|
515
|
+
```
|
|
516
|
+
|
|
517
|
+
</details>
|
|
518
|
+
|
|
298
519
|
<details>
|
|
299
520
|
<summary><b>Structured output — validated <code>data</code>, not just text</b></summary>
|
|
300
521
|
|
|
@@ -368,6 +589,144 @@ an action that has not run yet cannot be a saving.
|
|
|
368
589
|
|
|
369
590
|
</details>
|
|
370
591
|
|
|
592
|
+
<details>
|
|
593
|
+
<summary><b>Your ledgers in your tracing — OpenTelemetry, Langfuse, Datadog</b></summary>
|
|
594
|
+
|
|
595
|
+
Wrap the layer once and every request becomes a span on the OpenTelemetry pipeline
|
|
596
|
+
you already run (`@opentelemetry/api` is an optional peer; the SDK never imports it):
|
|
597
|
+
|
|
598
|
+
```ts
|
|
599
|
+
import { trace } from "@opentelemetry/api";
|
|
600
|
+
import { instrumentSavingsLayer } from "@bivelio/savings-layer";
|
|
601
|
+
|
|
602
|
+
// One span per request, on the OpenTelemetry pipeline you already run.
|
|
603
|
+
const traced = instrumentSavingsLayer(layer, { tracer: trace });
|
|
604
|
+
const { report: tracedReport } = await traced.generate({
|
|
605
|
+
model: "openai/gpt-4o-mini",
|
|
606
|
+
messages: [{ role: "user", content: "What is 2 + 2 * 3?" }],
|
|
607
|
+
});
|
|
608
|
+
console.log(tracedReport.cost.basis); // the span carries the same figures
|
|
609
|
+
```
|
|
610
|
+
|
|
611
|
+
- **Standard spans, plus your ledgers.** Spans follow the
|
|
612
|
+
[GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions/blob/v1.41.1/model/gen-ai/registry.yaml)
|
|
613
|
+
as last published (OpenTelemetry semantic conventions v1.41.1): `chat {model}`,
|
|
614
|
+
kind CLIENT, provider, model and usage from your provider's counter (0 on an avoided
|
|
615
|
+
call: no token was billed). On top, `com.bivelio.*` carries the four ledgers, the
|
|
616
|
+
`basis`, the cost and the ids; `com.bivelio.request_id` is the same id as your usage
|
|
617
|
+
record. Same results, same errors: `traced` hands back the very objects `layer` does.
|
|
618
|
+
- **Money is kept apart by basis.** The money saved is
|
|
619
|
+
`com.bivelio.cost.saved.measured`, `.reference` or `.estimated` — one key per span,
|
|
620
|
+
the one of its basis — so a sum over one key never mixes measured and estimated
|
|
621
|
+
money; `unmeasured` spans carry counts and no money. `report.cost.netSavings` is a
|
|
622
|
+
ratio and goes out as `com.bivelio.cost.net_savings_ratio`; the money is
|
|
623
|
+
`baselineEstimate − actual`, and that is what `saved.*` holds.
|
|
624
|
+
- **None of this talks to BiVelio.** Spans go to your exporter; files stay where you
|
|
625
|
+
write them. Your prompts, completions and provider keys never reach BiVelio, and no
|
|
626
|
+
span carries them either: ids, counts, money and enums only, and a failed request
|
|
627
|
+
records its `error.type`, never its message.
|
|
628
|
+
- **Langfuse.** Add the `LangfuseSpanProcessor` from `@langfuse/otel` to your tracer
|
|
629
|
+
provider. These spans carry `gen_ai.*`, so its default filter exports them
|
|
630
|
+
([Langfuse OpenTelemetry docs](https://langfuse.com/integrations/native/opentelemetry)).
|
|
631
|
+
`instrumentSavingsLayer(layer, { tracer: trace, langfuse: { metadata: true } })`
|
|
632
|
+
makes the figures filterable metadata, and `langfuse: { cost: true }` hands Langfuse
|
|
633
|
+
the report's own cost (in USD) instead of its estimate. Tracing with `@langfuse/tracing` by
|
|
634
|
+
hand? Pass `toLangfuseGeneration(report)` to `generation.update()`.
|
|
635
|
+
- **Datadog LLM Observability** reads these spans over OTLP
|
|
636
|
+
([Datadog OpenTelemetry docs](https://docs.datadoghq.com/llm_observability/instrumentation/otel_instrumentation/)):
|
|
637
|
+
|
|
638
|
+
```bash
|
|
639
|
+
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
|
|
640
|
+
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=<your Datadog site's OTLP traces endpoint>
|
|
641
|
+
OTEL_EXPORTER_OTLP_TRACES_HEADERS=dd-api-key=<your Datadog API key>,dd-otlp-source=llmobs
|
|
642
|
+
```
|
|
643
|
+
|
|
644
|
+
The `com.bivelio.*` attributes arrive as tags; with `datadog: { metadata: true }`
|
|
645
|
+
they also arrive as span metadata. Datadog drops spans that carry no `gen_ai.*`
|
|
646
|
+
attribute, so use `instrumentSavingsLayer` there rather than `recordSavingsOnSpan`
|
|
647
|
+
(which writes `com.bivelio.*` on a span you already have, and no `gen_ai.*` unless
|
|
648
|
+
you ask for it with `genAi: true`).
|
|
649
|
+
|
|
650
|
+
| Attribute | What it holds |
|
|
651
|
+
| --- | --- |
|
|
652
|
+
| `gen_ai.operation.name`, `gen_ai.provider.name`, `gen_ai.request.model`, `gen_ai.response.model` | `chat`, the provider and the model (without a known provider prefix) |
|
|
653
|
+
| `gen_ai.usage.input_tokens`, `.output_tokens`, `.cache_read.input_tokens` | your provider's counter; absent when it reported no usage |
|
|
654
|
+
| `com.bivelio.cost.basis`, `.currency`, `.baseline`, `.actual` | where the money comes from, its currency (USD by default), and both sides |
|
|
655
|
+
| `com.bivelio.cost.saved.measured` \| `.reference` \| `.estimated` | the money saved, in the key of its basis only |
|
|
656
|
+
| `com.bivelio.cost.net_savings_ratio` | `cost.netSavings`, a 0..1 ratio — not money |
|
|
657
|
+
| `com.bivelio.calls.*`, `com.bivelio.tokens.*` | the four ledgers: avoided calls, eliminated, compressed and provider‑cached tokens |
|
|
658
|
+
| `com.bivelio.request_id`, `.model.id`, `.cache.result`, `.quality.status` | ids and outcomes |
|
|
659
|
+
|
|
660
|
+
</details>
|
|
661
|
+
|
|
662
|
+
<details>
|
|
663
|
+
<summary><b>Your ledgers in your FinOps tool — FOCUS rows and a daily savings series</b></summary>
|
|
664
|
+
|
|
665
|
+
Keep each report with the time it came back, and export whenever you like:
|
|
666
|
+
|
|
667
|
+
```ts
|
|
668
|
+
import { writeFileSync } from "node:fs";
|
|
669
|
+
import { toFocusRows, toSavingsSeries, toCsv, type SavingsRecord } from "@bivelio/savings-layer";
|
|
670
|
+
|
|
671
|
+
const records: SavingsRecord[] = [];
|
|
672
|
+
const answer = await layer.generate({
|
|
673
|
+
model: "openai/gpt-4o-mini",
|
|
674
|
+
messages: [{ role: "user", content: "What is 2 + 2 * 3?" }],
|
|
675
|
+
});
|
|
676
|
+
records.push({ at: Date.now(), report: answer.report }); // keep reports as they come back
|
|
677
|
+
|
|
678
|
+
// FOCUS 1.4 rows, one per day, model and basis. Every cost column is 0 by design:
|
|
679
|
+
// your provider's bill already has the spend, so these rows add savings, never cost.
|
|
680
|
+
const rows = toFocusRows(records, { billingAccountId: "org-XXXXXXXX" }); // your provider billing account id
|
|
681
|
+
writeFileSync("bv-sala-focus.csv", toCsv(rows));
|
|
682
|
+
|
|
683
|
+
// A daily series for tools that take savings as a metric: one file per basis.
|
|
684
|
+
const measured = toSavingsSeries(records).filter((r) => r.measuredSavings !== null);
|
|
685
|
+
writeFileSync(
|
|
686
|
+
"vantage-measured-savings.csv",
|
|
687
|
+
toCsv(measured.map((r) => ({ date: r.date, amount: r.measuredSavings, model: r.model }))),
|
|
688
|
+
);
|
|
689
|
+
```
|
|
690
|
+
|
|
691
|
+
The rows follow [FOCUS 1.4](https://github.com/FinOps-Open-Cost-and-Usage-Spec/FOCUS_Spec/tree/v1.4),
|
|
692
|
+
the FinOps Open Cost and Usage Specification: `ChargeCategory` is `Adjustment`, the
|
|
693
|
+
service provider is the one in the model id (`OpenAI`, `Anthropic`…; pass
|
|
694
|
+
`serviceProviderName`, `invoiceIssuerName` or `hostProviderName` when a cloud or a
|
|
695
|
+
gateway bills you), and no SKU or unit price is invented. Nothing is sent anywhere:
|
|
696
|
+
`toFocusRows`, `toSavingsSeries` and `toCsv` are plain functions over the reports you
|
|
697
|
+
kept. Where each file goes, as each tool documents it in September 2026:
|
|
698
|
+
|
|
699
|
+
- **A FOCUS data lake or warehouse** reads every column, the `x_BiVelio*` ones included.
|
|
700
|
+
- **[Finout](https://docs.finout.io/get-started-with-finout/cost-and-usage-types/focus-in-finout)**
|
|
701
|
+
loads a FOCUS CSV as a custom cost center; you send the file to Finout support.
|
|
702
|
+
- **[Vantage](https://docs.vantage.sh/connecting_custom_providers)** Custom Providers
|
|
703
|
+
accept FOCUS 1.1–1.3 files but use only a few columns for cost reporting, and say
|
|
704
|
+
nothing about `x_` columns, so the savings would not show. Upload the series as a
|
|
705
|
+
[Business Metric](https://docs.vantage.sh/per_unit_costs) instead
|
|
706
|
+
(`date,amount,<labels>`), one file per basis, so a metric never mixes bases.
|
|
707
|
+
Vantage rejects two rows with the same date and labels: with `dimensions`, add
|
|
708
|
+
them as label columns too.
|
|
709
|
+
- **[CloudZero](https://docs.cloudzero.com/docs/anycost-common-bill-format-cbf)** AnyCost
|
|
710
|
+
takes its own Common Bill Format; the series can go in as a
|
|
711
|
+
[Telemetry Stream](https://docs.cloudzero.com/docs/telemetry-streams).
|
|
712
|
+
|
|
713
|
+
The custom columns, in the currency you pass (`currency`, USD by default):
|
|
714
|
+
|
|
715
|
+
| Column | What it holds |
|
|
716
|
+
| --- | --- |
|
|
717
|
+
| `x_BiVelioBasis`, `x_BiVelioModelId` | the basis of every figure in the row, and the model id |
|
|
718
|
+
| `x_BiVelioRequestCount`, `x_BiVelioProviderCallCount`, `x_BiVelioAvoidedCallCount` | requests, calls that reached the provider, calls avoided |
|
|
719
|
+
| `x_BiVelioBaselineCost`, `x_BiVelioActualCost` | both sides of the saving; empty when the basis is `unmeasured` |
|
|
720
|
+
| `x_BiVelioMeasuredSavings`, `x_BiVelioReferenceSavings`, `x_BiVelioEstimatedSavings` | `baseline − actual`, in the column of the row's basis only; negative when the layer cost you money |
|
|
721
|
+
| `x_BiVelioEliminatedInputFloorCost` | the separate floor lane for context your provider cleared |
|
|
722
|
+
| `x_BiVelio…InputTokens`, `x_BiVelio…OutputTokens` | the token ledgers: baseline, actual, eliminated, compressed, provider‑cached |
|
|
723
|
+
|
|
724
|
+
`Tags` carries `bivelio/basis` and `bivelio/model`, plus any tags you return from the
|
|
725
|
+
`tags` option. Sum `measured` and `reference` for what you can check against your
|
|
726
|
+
provider invoice; `estimated` has its own column on purpose.
|
|
727
|
+
|
|
728
|
+
</details>
|
|
729
|
+
|
|
371
730
|
<details>
|
|
372
731
|
<summary><b>Running a fleet? Seats and renewals</b></summary>
|
|
373
732
|
|
|
@@ -390,6 +749,10 @@ BVSALA_SEAT_ID=worker-03 # stable per replica; [A-Za-z0-9._:/-], max 128 chars
|
|
|
390
749
|
|
|
391
750
|
- **Node ≥ 20.** The package is **ESM‑only** (`"type": "module"`); from CommonJS,
|
|
392
751
|
`require()` works on Node ≥ 22.
|
|
752
|
+
- Framework adapters: `ai` 6 or 7, `@mastra/core` ≥ 1.68 < 2, `langchain` ≥ 1.1 < 2
|
|
753
|
+
with `@langchain/core` ≥ 1.1 < 2, and `@anthropic-ai/sdk` ≥ 0.103 < 1 — optional
|
|
754
|
+
peers, needed only for the adapter you import. `ai` 7 and `@mastra/core` declare
|
|
755
|
+
Node ≥ 22 in their own `engines`.
|
|
393
756
|
- Optional peer `gpt-tokenizer` for exact OpenAI token counts, loaded automatically
|
|
394
757
|
when installed — without it a built‑in estimator is used. It improves the accuracy
|
|
395
758
|
of estimated figures; what reaches your invoice is `measured` and `reference`, which
|
|
@@ -402,14 +765,20 @@ BVSALA_SEAT_ID=worker-03 # stable per replica; [A-Za-z0-9._:/-], max 128 chars
|
|
|
402
765
|
|
|
403
766
|
## Licensing & privacy
|
|
404
767
|
|
|
405
|
-
BV‑SALA is commercial
|
|
406
|
-
year**, EUR; VAT not included)
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
768
|
+
BV‑SALA is commercial software: a per‑seat subscription (**€4 / seat / month** or
|
|
769
|
+
**€39 / seat / year**, EUR; VAT not included), paid whether or not you save, plus a
|
|
770
|
+
tiered commission on measured savings only, as described at the top of this page, in
|
|
771
|
+
[LICENSE](./LICENSE) §6 and in the Published Pricing it refers to,
|
|
772
|
+
**[savings.bivelio.com/pricing](https://savings.bivelio.com/pricing)**. No free tier and no trial: what you can check before
|
|
773
|
+
paying is the published measurement. The license is a signed token your SDK and the
|
|
774
|
+
gateway verify **offline**; the managed control plane and the revocation kill‑switch
|
|
775
|
+
are served from BiVelio. Your LLM traffic goes **straight to your provider** —
|
|
776
|
+
BiVelio receives per‑request usage (counts, cost, the model id and opaque ids), never
|
|
777
|
+
prompts, completions or provider keys; [LICENSE](./LICENSE) §5 lists every field, and
|
|
778
|
+
**[savings.bivelio.com/privacy](https://savings.bivelio.com/privacy)** says how long
|
|
779
|
+
each one is kept.
|
|
780
|
+
|
|
781
|
+
Get a key at **[savings.bivelio.com](https://savings.bivelio.com)**.
|
|
413
782
|
Onboarding & general questions: **support@bivelio.com** · licensing & compliance:
|
|
414
783
|
**privacy@bivelio.com** · technical: **team@bivelio.com**.
|
|
415
784
|
|