@extrafemi-ai/lodestone 0.5.1 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +27 -0
- package/README.md +43 -14
- package/SECURITY.md +17 -0
- package/dist/catalog.js +11 -2
- package/dist/errors.d.ts +6 -0
- package/dist/errors.js +1 -2
- package/dist/explain.js +4 -0
- package/dist/health.js +19 -3
- package/dist/index.d.ts +10 -4
- package/dist/index.js +4 -1
- package/dist/limiter.d.ts +34 -0
- package/dist/limiter.js +35 -0
- package/dist/lodestone.d.ts +35 -12
- package/dist/lodestone.js +73 -16
- package/dist/providers/openaiCompatible.d.ts +2 -0
- package/dist/providers/openaiCompatible.js +7 -1
- package/dist/scorecard.d.ts +9 -3
- package/dist/scorecard.js +45 -2
- package/dist/spend.d.ts +33 -0
- package/dist/spend.js +72 -0
- package/dist/store.d.ts +2 -2
- package/dist/store.js +19 -3
- package/dist/stores/cloudflare.d.ts +10 -0
- package/dist/stores/cloudflare.js +11 -0
- package/dist/stores/deno.d.ts +14 -0
- package/dist/stores/deno.js +8 -0
- package/dist/stores/index.d.ts +9 -0
- package/dist/stores/index.js +5 -0
- package/dist/stores/netlify.d.ts +12 -0
- package/dist/stores/netlify.js +9 -0
- package/dist/stores/safeKey.d.ts +10 -0
- package/dist/stores/safeKey.js +20 -0
- package/dist/stores/upstash.d.ts +14 -0
- package/dist/stores/upstash.js +50 -0
- package/dist/types.d.ts +15 -3
- package/dist/util.d.ts +7 -0
- package/dist/util.js +17 -0
- package/docs/protecting-your-budget.md +48 -0
- package/docs/runtimes.md +19 -0
- package/docs/stores.md +45 -0
- package/examples/netlify-function.ts +10 -7
- package/package.json +5 -2
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,33 @@
|
|
|
2
2
|
|
|
3
3
|
Versions before 1.0 may change between minor releases; each change that could break a caller is marked **breaking**.
|
|
4
4
|
|
|
5
|
+
## 0.7.0
|
|
6
|
+
|
|
7
|
+
Security fixes, shared state that works on every platform, per-caller limits, and runtimes beyond Node. Found in a review of 0.6.0.
|
|
8
|
+
|
|
9
|
+
**Security**
|
|
10
|
+
- **Private calls are never cached.** Before, a call with `private: true` and a `cacheKey` saved its answer (made from a CV, say) in the shared store for a day. Now it is neither read from nor written to the cache. **Breaking** for anyone who relied on caching private calls.
|
|
11
|
+
- **A cached answer is tied to its input.** The cache stores a fingerprint of the system prompt, messages and tier with each answer; a `cacheKey` that is too coarse gives a miss, never another input's answer. Answers saved by 0.6.0 have no fingerprint and are treated as misses.
|
|
12
|
+
- **`private` now holds for fallback providers with no `trust` policy.** A fallback is used for a private call only if its own model list says the model is not trained on, or you vouch for it with the new `privacy: 'no-training'`. **Breaking** if you used `private: true` with an `anthropic()`, `google()` or other fallback that publishes nothing: add `privacy: 'no-training'` once you have checked that provider's terms.
|
|
13
|
+
- **The spend limit holds under a burst.** `createSpendLimit()` takes a hold for each paid attempt, settles it with the real cost, and gives it back when the attempt fails. `SpendControl` gains an optional `release()`, and `allow`/`record` receive an `id`.
|
|
14
|
+
- **Per-caller rate limits.** `createRateLimiter()` and `rateLimit` in the config; `caller` on a call; new failure kind `throttled` (with `detail.retryAfterMs` and an `explain()` sentence); `whenOver: 'free-only'` keeps serving an over-limit caller from free models. Only a hash of the caller is stored.
|
|
15
|
+
- **Anything read from a store is checked** (catalogue, circuit-breaker state, spend ledger, rate-limit counters) for shape and size.
|
|
16
|
+
- **`ProviderError.body` is no longer an enumerable property**, so logging the error does not print a provider's echo of the prompt.
|
|
17
|
+
- **`fetchScorecard`** now requires https (http only for localhost), reads at most `maxBytes` (1 MB default), and takes `allowedHosts`, checked again after a redirect. A scorecard with more than 20,000 entries is refused.
|
|
18
|
+
- **Supply chain.** GitHub Actions are pinned to commits, Dependabot and CodeQL are on, and the release job runs in a protected environment. The site sends `Strict-Transport-Security` with `includeSubDomains`, no longer allows inline style attributes in its Content-Security-Policy, and publishes `/.well-known/security.txt`.
|
|
19
|
+
|
|
20
|
+
**Reach**
|
|
21
|
+
- **Store adapters:** `netlifyBlobsStore`, `cloudflareKvStore`, `upstashStore` (which can count atomically) and `denoKvStore`, plus `safeKey`. `KeyValueStore.set` takes an optional `{ ttlMs }`; `AtomicStore` adds `incrBy`.
|
|
22
|
+
- **Runtimes.** CI now runs the same smoke checks on Node 20, 22 and 24, Cloudflare Workers (`workerd`, no Node compatibility flag), Deno and Bun. Nothing in the library needed to change to pass.
|
|
23
|
+
- **Python port** in `python/` (alpha; not on PyPI yet, install from the repository): failover, time budget, circuit breaker, spend holds, rate limits, private calls, `explain()`, standard library only.
|
|
24
|
+
- The site gains a docs section (stores, protecting your budget, runtimes, Python, security), `llms.txt`, and a sitemap that lists them; the live "Free model weather" section appears only once there are counts.
|
|
25
|
+
|
|
26
|
+
## 0.6.0
|
|
27
|
+
|
|
28
|
+
- **Schemas.** `json({ schema })` accepts anything with a `safeParse` method, such as a Zod schema, as well as a hand-written `validate` function. A model whose answer the schema rejects is skipped for the next, as before, and `data` is what the schema returns, so its defaults and transforms apply. Lodestone does not depend on any schema library; the `SafeParser` type is exported.
|
|
29
|
+
|
|
30
|
+
- **Groq's model list no longer includes non-chat models.** A live check on 28 Oct 2026 showed Groq lists speech (whisper), text-to-speech (orpheus) and safety-classifier (prompt-guard, safeguard) models beside its chat models, and `groq()` counted all 11 as chat. `normalizeGroq` leaves them out, so discovery cannot pick one. The same check showed `llama-3.3-70b-versatile` is no longer offered by Groq (a 404, reported as `withdrawn`).
|
|
31
|
+
|
|
5
32
|
## 0.5.1
|
|
6
33
|
|
|
7
34
|
No change to the library's behaviour.
|
package/README.md
CHANGED
|
@@ -38,6 +38,13 @@ try {
|
|
|
38
38
|
}
|
|
39
39
|
```
|
|
40
40
|
|
|
41
|
+
`validate` is a type guard you write. Or pass a schema, anything with `safeParse` (Zod, for example), and `data` is what it returns:
|
|
42
|
+
|
|
43
|
+
```ts
|
|
44
|
+
const Job = z.object({ title: z.string(), remote: z.boolean().default(false) });
|
|
45
|
+
const { data } = await ai.json({ task: 'extract-job', prompt, schema: Job }); // data: { title: string; remote: boolean }
|
|
46
|
+
```
|
|
47
|
+
|
|
41
48
|
`import { anthropic, google, openrouter, mock } from '@extrafemi-ai/lodestone'` for the provider adapters.
|
|
42
49
|
|
|
43
50
|
### Tiers
|
|
@@ -78,10 +85,10 @@ When a later phase exists, the free models get a short turn (9 s in all by defau
|
|
|
78
85
|
|
|
79
86
|
### Keeping the catalogue fresh
|
|
80
87
|
|
|
81
|
-
Give it somewhere to keep the catalogue (anything that can get and set JSON by key; the default is memory), and
|
|
88
|
+
Give it somewhere to keep the catalogue (anything that can get and set JSON by key; the default is memory). Adapters for Netlify Blobs, Cloudflare KV, Upstash (and Vercel KV) and Deno KV are included, see [docs/stores.md](docs/stores.md). Run `checkModels()` once a day:
|
|
82
89
|
|
|
83
90
|
```ts
|
|
84
|
-
const ai = createLodestone({ store:
|
|
91
|
+
const ai = createLodestone({ store: netlifyBlobsStore(getStore('lodestone')), /* ... */ });
|
|
85
92
|
|
|
86
93
|
// daily
|
|
87
94
|
const { ok, withdrawn, shortTiers } = await ai.checkModels();
|
|
@@ -108,17 +115,25 @@ A model whose last three attempts (within ten minutes) all failed is skipped, so
|
|
|
108
115
|
### Limiting what the platform spends
|
|
109
116
|
|
|
110
117
|
```ts
|
|
111
|
-
const
|
|
112
|
-
|
|
113
|
-
spend: {
|
|
114
|
-
allow: async ({ model, phase }) => (await spentToday()) < DAILY_LIMIT_USD,
|
|
115
|
-
record: async ({ costUsd }) => addToday(costUsd ?? 0),
|
|
116
|
-
},
|
|
117
|
-
});
|
|
118
|
+
const spend = createSpendLimit({ store, dailyUsd: 2, monthlyUsd: 30 });
|
|
119
|
+
const ai = createLodestone({ /* ... */, spend });
|
|
118
120
|
```
|
|
119
121
|
|
|
120
122
|
`allow` is asked before each attempt that spends the platform's money (the paid models and fallback providers on the platform's keys), and a `false` means that model is not used. `record` is told what each such call cost: the provider's own figure when it gives one (OpenRouter does), otherwise an estimate from the model's catalogue price. Free models and a caller's own key (their money, not yours) never go through either. When the limit stops every paid option the error is `kind: 'budget'`. A `record` that throws never fails the answer.
|
|
121
123
|
|
|
124
|
+
`createSpendLimit` takes a **hold** for each paid attempt as it is allowed and counts holds against the limit, so a burst of concurrent requests cannot all pass the check before any has been counted. The hold is swapped for the real cost on an answer and given back (`release`) when the attempt fails. You can write your own `{ allow, record, release? }`; each receives an `id` naming the attempt.
|
|
125
|
+
|
|
126
|
+
### One person emptying the budget (`rateLimit`)
|
|
127
|
+
|
|
128
|
+
A spend limit is one pot for everybody. Give each caller their own allowance, and name the caller on each call:
|
|
129
|
+
|
|
130
|
+
```ts
|
|
131
|
+
const ai = createLodestone({ /* ... */, rateLimit: { limiter: createRateLimiter({ store, max: 20, windowMs: 3_600_000 }), whenOver: 'free-only' } });
|
|
132
|
+
await ai.complete({ prompt, caller: userId ?? clientIp });
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
Over their allowance a caller gets `kind: 'throttled'` (`'reject'`, the default) or free models only (`'free-only'`). Only a hash of the caller is stored. See [docs/protecting-your-budget.md](docs/protecting-your-budget.md) for what is exact and what is approximate.
|
|
136
|
+
|
|
122
137
|
### Providers, and not depending on one
|
|
123
138
|
|
|
124
139
|
OpenRouter is the default, but one provider's outage should not stop your tool. Give Lodestone others as `fallbacks`, each with its own key. Any service that speaks OpenAI's chat API works through `openaiCompatible()`, and presets exist for the ones Lodestone has been checked against:
|
|
@@ -146,7 +161,17 @@ Model ids are each service's own. See [examples/multi-provider.ts](examples/mult
|
|
|
146
161
|
- **Outdated or shut down.** The daily `checkModels()` saves each provider's current catalogue. A model the provider no longer lists is dropped; one with an expiry date within a week (OpenRouter publishes it) is left out; one that is failing right now is skipped by the circuit breaker. Nobody edits a list.
|
|
147
162
|
- **New models.** `discover: { fill }` tops a short list up from the provider's own catalogue with models within a price limit ("free" is a limit of zero) from trusted labs. For OpenRouter's free models this happens by itself.
|
|
148
163
|
- **Not a longer wait.** Discovery never lengthens what a request walks. A background pool (up to 8 per tier) is ranked, the models that are down are filtered out, and a request tries at most three. When one fails, the next best takes its place for later requests, so adding models adds choice, not delay.
|
|
149
|
-
- **Safe by default.** A discovered model is used only if its publisher is on the trust list; `trust` also takes a block list, refuses models a provider says are trained on or that keep prompts too long, and `private: true` applies the same to one call. There is no vulnerability database for hosted models, so none is claimed: see [SECURITY.md](SECURITY.md) for exactly what is and is not checked.
|
|
164
|
+
- **Safe by default.** A discovered model is used only if its publisher is on the trust list; `trust` also takes a block list, refuses models a provider says are trained on or that keep prompts too long, and `private: true` applies the same to one call (see below). There is no vulnerability database for hosted models, so none is claimed: see [SECURITY.md](SECURITY.md) for exactly what is and is not checked.
|
|
165
|
+
|
|
166
|
+
### Private calls (`private: true`)
|
|
167
|
+
|
|
168
|
+
For a prompt that must not be kept or trained on (a CV, a medical note). What it does, exactly:
|
|
169
|
+
|
|
170
|
+
- OpenRouter is told to use only providers that do not keep or train on the prompt (`data_collection: deny`).
|
|
171
|
+
- A fallback provider is used only if its own model list says the model is not trained on (Requesty publishes this), **or** you vouched for it with `privacy: 'no-training'` on that fallback. A provider that publishes nothing and that you have not vouched for is skipped, even if you set no `trust` policy. Lodestone cannot see what a provider does with a prompt unless the provider says.
|
|
172
|
+
- The answer is never read from or written to the cache.
|
|
173
|
+
- Its failures are not counted against a model (few providers can meet the rule, and that says nothing about the model).
|
|
174
|
+
- An explicit `route` is your own choice of provider and is passed through as told.
|
|
150
175
|
|
|
151
176
|
### Cancelling
|
|
152
177
|
|
|
@@ -158,7 +183,11 @@ await ai.complete({ prompt, signal: controller.signal }); // throws a Lodeston
|
|
|
158
183
|
|
|
159
184
|
### Where it runs
|
|
160
185
|
|
|
161
|
-
Lodestone uses only
|
|
186
|
+
Lodestone uses only web-standard APIs and has no dependencies. CI runs the same smoke checks on Node 20, 22 and 24, on Cloudflare Workers (in `workerd`, without the Node compatibility flag), on Deno and on Bun. See [docs/runtimes.md](docs/runtimes.md) for what that does and does not show.
|
|
187
|
+
|
|
188
|
+
### Python
|
|
189
|
+
|
|
190
|
+
`python/` holds a Python port of the core: failover, the time budget, the circuit breaker, spend holds, per-caller rate limits, private calls and `explain()`, using only the standard library. It is alpha and does not yet read provider model lists or support images. See [python/README.md](python/README.md).
|
|
162
191
|
|
|
163
192
|
### Not yet
|
|
164
193
|
|
|
@@ -171,7 +200,7 @@ A scorecard (a small JSON file of measured results) can put the models that did
|
|
|
171
200
|
|
|
172
201
|
### Caching
|
|
173
202
|
|
|
174
|
-
Pass `cacheKey` for a pure, repeatable task (for example a hash of a job page) and the answer is saved in the store and reused for `cacheTtlMs` (a day by default). Failures are never cached
|
|
203
|
+
Pass `cacheKey` for a pure, repeatable task (for example a hash of a job page) and the answer is saved in the store and reused for `cacheTtlMs` (a day by default). The key must stand for the whole input; as a safeguard an answer is also tied to a fingerprint of the system prompt, messages and tier, so a key that is too coarse gives a miss, never another input's answer. Failures are never cached, and **a call marked `private` is never read from or written to the cache**.
|
|
175
204
|
|
|
176
205
|
### Observing it
|
|
177
206
|
|
|
@@ -179,7 +208,7 @@ Pass `cacheKey` for a pure, repeatable task (for example a hash of a job page) a
|
|
|
179
208
|
|
|
180
209
|
## Errors
|
|
181
210
|
|
|
182
|
-
`complete()` and `json()` throw a `LodestoneError` whose `kind` is one of `auth`, `rate_limit`, `slow`, `credit`, `withdrawn`, `unavailable`, `bad_output`, `not_configured` or `other`. `detail` carries the HTTP status and, for rate limits, which kind it was. `explain(err, { product, settingsHint })` turns any of them into a message for a person.
|
|
211
|
+
`complete()` and `json()` throw a `LodestoneError` whose `kind` is one of `auth`, `rate_limit`, `slow`, `credit`, `withdrawn`, `unavailable`, `bad_output`, `aborted`, `budget`, `throttled`, `not_configured` or `other`. `detail` carries the HTTP status and, for rate limits, which kind it was. `explain(err, { product, settingsHint })` turns any of them into a message for a person.
|
|
183
212
|
|
|
184
213
|
## Develop
|
|
185
214
|
|
|
@@ -189,7 +218,7 @@ npm test
|
|
|
189
218
|
npm run build
|
|
190
219
|
```
|
|
191
220
|
|
|
192
|
-
Releases: bump `version` in `package.json`, then push a tag (`git tag v0.1.0 && git push --tags`). The release workflow tests, builds and publishes to npm with provenance. It
|
|
221
|
+
`npm run smoke` and `npm run smoke:workers` run the cross-runtime checks. Releases: bump `version` in `package.json`, then push a tag (`git tag v0.1.0 && git push --tags`). The release workflow tests, builds and publishes to npm with provenance. It uses npm trusted publishing, so there is no token to keep: npm trusts this repository and `release.yml` directly (set up under the package's Settings on npmjs.com).
|
|
193
222
|
|
|
194
223
|
## Licence
|
|
195
224
|
|
package/SECURITY.md
CHANGED
|
@@ -13,6 +13,7 @@ within a few days. Please give a reasonable time to fix it before you publish de
|
|
|
13
13
|
- **Published with provenance.** Each release is built by a GitHub Actions workflow from a tagged commit and published to
|
|
14
14
|
npm with a signed attestation. `npm audit signatures` verifies that a package came from this repository.
|
|
15
15
|
- **Small.** About 1,600 lines of source, with tests beside it. Read it before you depend on it.
|
|
16
|
+
- **Pinned build.** Every third-party GitHub Action is pinned to a commit, not a tag; Dependabot proposes updates; CodeQL scans the TypeScript and Python on every push and weekly; the release job runs in a protected environment (`npm-publish`) that needs a human to approve before anything is published.
|
|
16
17
|
- **Your keys stay yours.** Keys are passed in by your code and sent only to the provider you name. They are never stored,
|
|
17
18
|
logged or sent anywhere else, and prompts and answers are never logged. `onAttempt` reports the provider, model, outcome
|
|
18
19
|
and timing only.
|
|
@@ -33,3 +34,19 @@ really be checked, and says where it cannot:
|
|
|
33
34
|
which also limits what a misbehaving model can hand to the rest of your code. Treat every model answer as untrusted input.
|
|
34
35
|
|
|
35
36
|
These are self-reported by providers and are only as good as what they report.
|
|
37
|
+
|
|
38
|
+
## What protects your users' data and your budget
|
|
39
|
+
|
|
40
|
+
- **A private call is never cached**, and goes only to providers that say, or that you vouch for, they do not keep or train on the prompt ([README](README.md#private-calls-private-true)).
|
|
41
|
+
- **A cached answer is tied to the input**, not only to the key you chose, so a `cacheKey` that is too coarse cannot hand one person's answer to another.
|
|
42
|
+
- **The spend limit holds under a burst** (`createSpendLimit` takes a hold per attempt), and **each caller has an allowance** (`rateLimit`), stored as a hash so an IP address is not kept. What is exact and what is approximate between processes is stated in [docs/protecting-your-budget.md](docs/protecting-your-budget.md).
|
|
43
|
+
- **Anything read back from a store is checked** for shape and size before it is used (catalogue, breaker state, spend ledger, rate-limit counters), so a store someone else can write to cannot hide every model or grow without limit.
|
|
44
|
+
- **Errors do not carry the provider's raw response into logs.** `ProviderError.body` is not an enumerable property, so `console.log(err)` and `JSON.stringify(err)` leave it out; read `err.body` deliberately.
|
|
45
|
+
- **A scorecard fetched from a URL** must be https, under 1 MB by default, and (if you pass `allowedHosts`) from a host you named, including after a redirect. A scorecard can only change the order models are tried in.
|
|
46
|
+
|
|
47
|
+
## Known limits
|
|
48
|
+
|
|
49
|
+
- Lodestone cannot tell a person from a script. `caller` is whatever you pass; an IP address is a weak identity.
|
|
50
|
+
- With a plain get/set store, counters are exact inside one process and approximate between processes. Use `upstashStore` where that matters.
|
|
51
|
+
- `private` cannot make a provider honour a promise it does not publish. It can only refuse providers that say nothing.
|
|
52
|
+
- Everything above is about what Lodestone does. The model's answer is untrusted input: treat it like user input.
|
package/dist/catalog.js
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import { vet } from './trust.js';
|
|
2
|
+
import { isRecord } from './util.js';
|
|
2
3
|
export const ROUTER = 'openrouter/free';
|
|
3
4
|
const MAX_ENTRIES = 3;
|
|
4
5
|
const isFree = (m) => m.id.endsWith(':free') || (Number(m.pricing?.prompt) === 0 && Number(m.pricing?.completion) === 0 && m.id !== ROUTER);
|
|
@@ -47,6 +48,14 @@ export function pickModels(tier, catalog, preferences, opts = {}) {
|
|
|
47
48
|
const KEY = 'lodestone:model-catalog';
|
|
48
49
|
const MEMORY_MS = 6 * 3_600_000;
|
|
49
50
|
const STALE_MS = 24 * 3_600_000;
|
|
51
|
+
const MAX_CATALOG = 5_000;
|
|
52
|
+
/** A catalogue read back from the store, checked: a list somebody else wrote is not trusted to be the right shape or size. Entries without a string id are dropped. */
|
|
53
|
+
function cleanSaved(raw) {
|
|
54
|
+
if (!isRecord(raw) || typeof raw.at !== 'string' || !Array.isArray(raw.models))
|
|
55
|
+
return null;
|
|
56
|
+
const models = raw.models.filter((m) => isRecord(m) && typeof m.id === 'string' && m.id.length > 0 && m.id.length <= 200).slice(0, MAX_CATALOG);
|
|
57
|
+
return models.length ? { at: raw.at, models } : null;
|
|
58
|
+
}
|
|
50
59
|
/** OpenRouter's catalogue, as a list of CatalogModel. */
|
|
51
60
|
export async function listOpenRouter(fetchImpl) {
|
|
52
61
|
const res = await fetchImpl('https://openrouter.ai/api/v1/models', { signal: AbortSignal.timeout(8_000) });
|
|
@@ -77,8 +86,8 @@ export function createCatalog(store, fetchImpl, now = Date.now, source = {}) {
|
|
|
77
86
|
if (memory && now() - memory.at < MEMORY_MS)
|
|
78
87
|
return memory.models;
|
|
79
88
|
try {
|
|
80
|
-
const saved = (await store.get(KEY_));
|
|
81
|
-
if (saved
|
|
89
|
+
const saved = cleanSaved(await store.get(KEY_));
|
|
90
|
+
if (saved) {
|
|
82
91
|
memory = { at: now(), models: saved.models };
|
|
83
92
|
if (now() - Date.parse(saved.at) > STALE_MS)
|
|
84
93
|
refresh().catch(() => { });
|
package/dist/errors.d.ts
CHANGED
|
@@ -2,6 +2,8 @@ import type { RateLimitReading } from './rateLimit.js';
|
|
|
2
2
|
/** A provider answered with an error status (or the request itself failed: status 599). */
|
|
3
3
|
export declare class ProviderError extends Error {
|
|
4
4
|
status: number;
|
|
5
|
+
/** The provider's raw response. Some providers repeat part of the prompt in an error, so this is not an own enumerable property:
|
|
6
|
+
* `console.log(err)` and JSON.stringify(err) leave it out. Read it deliberately, as `err.body`. */
|
|
5
7
|
body: string;
|
|
6
8
|
constructor(status: number, body: string);
|
|
7
9
|
}
|
|
@@ -26,6 +28,8 @@ export type FailureKind =
|
|
|
26
28
|
| 'aborted'
|
|
27
29
|
/** The platform's spend limit has been reached, so no paid model was used. */
|
|
28
30
|
| 'budget'
|
|
31
|
+
/** This caller has made too many calls in the time allowed (see `rateLimit`). `detail.retryAfterMs` says when to try again. */
|
|
32
|
+
| 'throttled'
|
|
29
33
|
/** No provider has a key. */
|
|
30
34
|
| 'not_configured' | 'other';
|
|
31
35
|
/** Thrown when no model could answer. `kind` says why, so the caller can decide what
|
|
@@ -38,6 +42,7 @@ export declare class LodestoneError extends Error {
|
|
|
38
42
|
ownKey?: boolean;
|
|
39
43
|
needsCredit?: boolean;
|
|
40
44
|
providerMessage?: string;
|
|
45
|
+
retryAfterMs?: number;
|
|
41
46
|
};
|
|
42
47
|
constructor(kind: FailureKind, detail?: {
|
|
43
48
|
status?: number;
|
|
@@ -45,5 +50,6 @@ export declare class LodestoneError extends Error {
|
|
|
45
50
|
ownKey?: boolean;
|
|
46
51
|
needsCredit?: boolean;
|
|
47
52
|
providerMessage?: string;
|
|
53
|
+
retryAfterMs?: number;
|
|
48
54
|
});
|
|
49
55
|
}
|
package/dist/errors.js
CHANGED
|
@@ -1,12 +1,11 @@
|
|
|
1
1
|
/** A provider answered with an error status (or the request itself failed: status 599). */
|
|
2
2
|
export class ProviderError extends Error {
|
|
3
3
|
status;
|
|
4
|
-
body;
|
|
5
4
|
constructor(status, body) {
|
|
6
5
|
super(`provider error ${status}`);
|
|
7
6
|
this.status = status;
|
|
8
|
-
this.body = body;
|
|
9
7
|
this.name = 'ProviderError';
|
|
8
|
+
Object.defineProperty(this, 'body', { value: body, enumerable: false, writable: true, configurable: true });
|
|
10
9
|
}
|
|
11
10
|
}
|
|
12
11
|
/** The human-readable message inside a provider's error body ({"error":{"message":"..."}} or {"message":"..."}), if it has one. */
|
package/dist/explain.js
CHANGED
|
@@ -46,6 +46,10 @@ export function explain(err, opts = {}) {
|
|
|
46
46
|
return 'Cancelled.';
|
|
47
47
|
case 'budget':
|
|
48
48
|
return `${product}'s AI budget for now is used up, and the free models are busy. Try again later${opts.settingsHint ? `, or add your own OpenRouter key ${opts.settingsHint.replace(/^in /, 'in ')} to keep going` : ''}.`;
|
|
49
|
+
case 'throttled': {
|
|
50
|
+
const mins = Math.max(1, Math.ceil((detail.retryAfterMs ?? 60_000) / 60_000));
|
|
51
|
+
return `You've made a lot of requests in a short time. Try again in ${mins === 1 ? 'a minute' : `about ${mins} minutes`}.`;
|
|
52
|
+
}
|
|
49
53
|
case 'not_configured':
|
|
50
54
|
return `${product}'s AI isn't set up yet.`;
|
|
51
55
|
default:
|
package/dist/health.js
CHANGED
|
@@ -1,6 +1,22 @@
|
|
|
1
|
+
import { isRecord } from './util.js';
|
|
1
2
|
const KEEP = 6;
|
|
2
3
|
const SHARE_MS = 15_000;
|
|
3
4
|
const id = (provider, model) => `${provider}\u0000${model}`;
|
|
5
|
+
const MAX_MODELS = 400;
|
|
6
|
+
/** What a store returned, checked: only well-formed samples for a bounded number of models are kept, so a store somebody else can write to cannot make the breaker hide every model or grow without limit. */
|
|
7
|
+
function clean(raw) {
|
|
8
|
+
const out = {};
|
|
9
|
+
if (!isRecord(raw))
|
|
10
|
+
return out;
|
|
11
|
+
for (const [k, list] of Object.entries(raw).slice(0, MAX_MODELS)) {
|
|
12
|
+
if (k.length > 300 || !Array.isArray(list))
|
|
13
|
+
continue;
|
|
14
|
+
const ok = list.filter((s) => Array.isArray(s) && s.length === 3 && Number.isFinite(s[0]) && (s[1] === 0 || s[1] === 1) && typeof s[2] === 'string' && s[2].length <= 16).slice(-KEEP);
|
|
15
|
+
if (ok.length)
|
|
16
|
+
out[k] = ok;
|
|
17
|
+
}
|
|
18
|
+
return out;
|
|
19
|
+
}
|
|
4
20
|
export function createHealth(opts = {}) {
|
|
5
21
|
const need = opts.failures ?? 3;
|
|
6
22
|
const windowMs = opts.windowMs ?? 10 * 60_000;
|
|
@@ -15,7 +31,7 @@ export function createHealth(opts = {}) {
|
|
|
15
31
|
return;
|
|
16
32
|
loadedAt = now();
|
|
17
33
|
try {
|
|
18
|
-
state = merge(state, (await opts.store.get(key))
|
|
34
|
+
state = merge(state, clean(await opts.store.get(key)));
|
|
19
35
|
}
|
|
20
36
|
catch { /* an unreadable store is the same as an empty one */ }
|
|
21
37
|
}
|
|
@@ -49,9 +65,9 @@ export function createHealth(opts = {}) {
|
|
|
49
65
|
if (!opts.store || (ok && !recovering))
|
|
50
66
|
return;
|
|
51
67
|
try {
|
|
52
|
-
const theirs = (await opts.store.get(key))
|
|
68
|
+
const theirs = clean(await opts.store.get(key));
|
|
53
69
|
state = merge(state, theirs);
|
|
54
|
-
await opts.store.set(key, state);
|
|
70
|
+
await opts.store.set(key, state, { ttlMs: windowMs * 3 });
|
|
55
71
|
loadedAt = now();
|
|
56
72
|
}
|
|
57
73
|
catch { /* not saved: the next instance just has to learn it again */ }
|
package/dist/index.d.ts
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
export { createLodestone } from './lodestone.js';
|
|
2
|
-
export type { Lodestone, LodestoneConfig, CompleteOptions, CompleteResult, Profile, FallbackConfig, AttemptEvent, RouteStep, SpendControl, CheckResult } from './lodestone.js';
|
|
2
|
+
export type { Lodestone, LodestoneConfig, CompleteOptions, CompleteResult, SafeParser, Profile, FallbackConfig, AttemptEvent, RouteStep, SpendControl, SpendInfo, CheckResult } from './lodestone.js';
|
|
3
3
|
export { LodestoneError, ProviderError, messageFrom } from './errors.js';
|
|
4
4
|
export type { FailureKind } from './errors.js';
|
|
5
5
|
export { explain } from './explain.js';
|
|
@@ -11,23 +11,29 @@ export type { CatalogModel } from './catalog.js';
|
|
|
11
11
|
export { DEFAULT_MODELS, DEFAULT_PAID_MODELS } from './models.js';
|
|
12
12
|
export { tryParseJson } from './json.js';
|
|
13
13
|
export { memoryStore } from './store.js';
|
|
14
|
+
export { createSpendLimit } from './spend.js';
|
|
15
|
+
export type { SpendLimitOptions, SpendReport } from './spend.js';
|
|
16
|
+
export { createRateLimiter } from './limiter.js';
|
|
17
|
+
export type { RateLimiter, RateLimiterOptions, RateLimitResult } from './limiter.js';
|
|
14
18
|
export { openrouter } from './providers/openrouter.js';
|
|
15
19
|
export { anthropic } from './providers/anthropic.js';
|
|
16
20
|
export { google } from './providers/google.js';
|
|
17
21
|
export { mock } from './providers/mock.js';
|
|
18
|
-
export type { Tier, Message, ContentPart, Provider, ProviderRequest, ProviderReply, KeyValueStore, Usage } from './types.js';
|
|
22
|
+
export type { Tier, Message, ContentPart, Provider, ProviderRequest, ProviderReply, KeyValueStore, AtomicStore, Usage } from './types.js';
|
|
19
23
|
export { SCORECARD_SCHEMA, ScorecardError, parseScorecard, fetchScorecard, rankModels, scoreFor, baseModelId } from './scorecard.js';
|
|
20
|
-
export type { Scorecard, ModelScore, QualityQuery, RankOptions } from './scorecard.js';
|
|
24
|
+
export type { Scorecard, ModelScore, QualityQuery, RankOptions, FetchScorecardOptions } from './scorecard.js';
|
|
21
25
|
export { createStats } from './stats.js';
|
|
22
26
|
export type { Stats, StatsOptions, StatsState } from './stats.js';
|
|
23
27
|
export { CANDIDATES_SCHEMA } from './candidates.js';
|
|
24
28
|
export type { Candidates, Candidate } from './candidates.js';
|
|
25
29
|
export { createHealth } from './health.js';
|
|
26
30
|
export type { Health, HealthOptions, ModelHealth } from './health.js';
|
|
27
|
-
export { openaiCompatible, requesty, llmapi, groq, cerebras, openai, mistral, together, ollama, normalizeRequesty, normalizeLlmapi } from './providers/openaiCompatible.js';
|
|
31
|
+
export { openaiCompatible, requesty, llmapi, groq, cerebras, openai, mistral, together, ollama, normalizeRequesty, normalizeLlmapi, normalizeGroq } from './providers/openaiCompatible.js';
|
|
28
32
|
export type { OpenAICompatibleOptions, PresetOptions } from './providers/openaiCompatible.js';
|
|
29
33
|
export { vet, publisherOf, DEFAULT_PUBLISHERS } from './trust.js';
|
|
30
34
|
export type { TrustPolicy, Verdict } from './trust.js';
|
|
31
35
|
export { resolveProviderModels } from './discover.js';
|
|
32
36
|
export type { FillOptions } from './discover.js';
|
|
33
37
|
export type { CatalogModel as ModelInfo } from './catalog.js';
|
|
38
|
+
export { netlifyBlobsStore, cloudflareKvStore, upstashStore, denoKvStore, safeKey } from './stores/index.js';
|
|
39
|
+
export type { NetlifyBlobsLike, KVNamespaceLike, UpstashOptions, DenoKvLike } from './stores/index.js';
|
package/dist/index.js
CHANGED
|
@@ -6,6 +6,8 @@ export { pickModels, eligible, ROUTER } from './catalog.js';
|
|
|
6
6
|
export { DEFAULT_MODELS, DEFAULT_PAID_MODELS } from './models.js';
|
|
7
7
|
export { tryParseJson } from './json.js';
|
|
8
8
|
export { memoryStore } from './store.js';
|
|
9
|
+
export { createSpendLimit } from './spend.js';
|
|
10
|
+
export { createRateLimiter } from './limiter.js';
|
|
9
11
|
export { openrouter } from './providers/openrouter.js';
|
|
10
12
|
export { anthropic } from './providers/anthropic.js';
|
|
11
13
|
export { google } from './providers/google.js';
|
|
@@ -14,6 +16,7 @@ export { SCORECARD_SCHEMA, ScorecardError, parseScorecard, fetchScorecard, rankM
|
|
|
14
16
|
export { createStats } from './stats.js';
|
|
15
17
|
export { CANDIDATES_SCHEMA } from './candidates.js';
|
|
16
18
|
export { createHealth } from './health.js';
|
|
17
|
-
export { openaiCompatible, requesty, llmapi, groq, cerebras, openai, mistral, together, ollama, normalizeRequesty, normalizeLlmapi } from './providers/openaiCompatible.js';
|
|
19
|
+
export { openaiCompatible, requesty, llmapi, groq, cerebras, openai, mistral, together, ollama, normalizeRequesty, normalizeLlmapi, normalizeGroq } from './providers/openaiCompatible.js';
|
|
18
20
|
export { vet, publisherOf, DEFAULT_PUBLISHERS } from './trust.js';
|
|
19
21
|
export { resolveProviderModels } from './discover.js';
|
|
22
|
+
export { netlifyBlobsStore, cloudflareKvStore, upstashStore, denoKvStore, safeKey } from './stores/index.js';
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
import type { AtomicStore, KeyValueStore } from './types.js';
|
|
2
|
+
/** Per-caller rate limiting. The spend limit is one pot for everybody, so one person calling a public tool in a
|
|
3
|
+
* loop can empty it for everyone else. This gives each caller (a user id, an IP address, an API key) their own
|
|
4
|
+
* allowance: `max` calls in every `windowMs`. Pass the caller on each call (`caller: userId`) and give the
|
|
5
|
+
* limiter to createLodestone as `rateLimit`.
|
|
6
|
+
*
|
|
7
|
+
* What is stored is a hash of the caller, never the caller: an IP address does not end up in your store.
|
|
8
|
+
* With a store that can count atomically (Upstash, the memory store) the count is exact across functions.
|
|
9
|
+
* With a plain get/set store it is exact inside one process and approximate between processes: two functions
|
|
10
|
+
* answering the same caller in the same moment can let a call or two through. */
|
|
11
|
+
export interface RateLimiterOptions {
|
|
12
|
+
store: KeyValueStore | AtomicStore;
|
|
13
|
+
/** Calls allowed per caller in each window. */
|
|
14
|
+
max: number;
|
|
15
|
+
/** Default one hour. */
|
|
16
|
+
windowMs?: number;
|
|
17
|
+
/** Prefix for the keys. Default lodestone:rl */
|
|
18
|
+
key?: string;
|
|
19
|
+
/** What to do when the store cannot be reached. Default 'allow': a broken store should not take the tool down. The spend limit is the backstop. */
|
|
20
|
+
onStoreError?: 'allow' | 'deny';
|
|
21
|
+
now?: () => number;
|
|
22
|
+
}
|
|
23
|
+
export interface RateLimitResult {
|
|
24
|
+
ok: boolean;
|
|
25
|
+
/** Calls left in this window, after this one. */
|
|
26
|
+
remaining: number;
|
|
27
|
+
/** When the window ends, if `ok` is false: how long this caller has to wait. */
|
|
28
|
+
retryAfterMs: number;
|
|
29
|
+
}
|
|
30
|
+
export interface RateLimiter {
|
|
31
|
+
/** Counts one call for this caller and says whether it is within the allowance. */
|
|
32
|
+
take(caller: string): Promise<RateLimitResult>;
|
|
33
|
+
}
|
|
34
|
+
export declare function createRateLimiter(o: RateLimiterOptions): RateLimiter;
|
package/dist/limiter.js
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
import { createMutex, isNonNegative, isRecord, sha256Hex } from './util.js';
|
|
2
|
+
const isAtomic = (s) => typeof s.incrBy === 'function';
|
|
3
|
+
export function createRateLimiter(o) {
|
|
4
|
+
const windowMs = o.windowMs ?? 3_600_000;
|
|
5
|
+
const now = o.now ?? Date.now;
|
|
6
|
+
const prefix = o.key ?? 'lodestone:rl';
|
|
7
|
+
const exclusive = createMutex();
|
|
8
|
+
const failure = o.onStoreError === 'deny' ? { ok: false, remaining: 0, retryAfterMs: 60_000 } : { ok: true, remaining: o.max, retryAfterMs: 0 };
|
|
9
|
+
return {
|
|
10
|
+
async take(caller) {
|
|
11
|
+
const who = await sha256Hex(caller);
|
|
12
|
+
const t = now();
|
|
13
|
+
const windowStart = Math.floor(t / windowMs) * windowMs;
|
|
14
|
+
const retryAfterMs = windowStart + windowMs - t;
|
|
15
|
+
try {
|
|
16
|
+
if (isAtomic(o.store)) {
|
|
17
|
+
const n = await o.store.incrBy(`${prefix}:${who}:${windowStart}`, 1, { ttlMs: windowMs * 2 });
|
|
18
|
+
return { ok: n <= o.max, remaining: Math.max(0, o.max - n), retryAfterMs };
|
|
19
|
+
}
|
|
20
|
+
return await exclusive(async () => {
|
|
21
|
+
const key = `${prefix}:${who}`;
|
|
22
|
+
const saved = await o.store.get(key);
|
|
23
|
+
const n = isRecord(saved) && saved.w === windowStart && isNonNegative(saved.n) ? saved.n : 0;
|
|
24
|
+
if (n >= o.max)
|
|
25
|
+
return { ok: false, remaining: 0, retryAfterMs };
|
|
26
|
+
await o.store.set(key, { w: windowStart, n: n + 1 }, { ttlMs: windowMs * 2 });
|
|
27
|
+
return { ok: true, remaining: o.max - n - 1, retryAfterMs };
|
|
28
|
+
});
|
|
29
|
+
}
|
|
30
|
+
catch {
|
|
31
|
+
return failure;
|
|
32
|
+
}
|
|
33
|
+
},
|
|
34
|
+
};
|
|
35
|
+
}
|
package/dist/lodestone.d.ts
CHANGED
|
@@ -5,6 +5,7 @@ import { type Candidates } from './candidates.js';
|
|
|
5
5
|
import type { Health } from './health.js';
|
|
6
6
|
import { type QualityQuery, type Scorecard } from './scorecard.js';
|
|
7
7
|
import type { Stats } from './stats.js';
|
|
8
|
+
import type { RateLimiter } from './limiter.js';
|
|
8
9
|
import type { KeyValueStore, Message, Provider, Tier, Usage } from './types.js';
|
|
9
10
|
export interface Profile {
|
|
10
11
|
tier: Tier;
|
|
@@ -27,6 +28,8 @@ export interface FallbackConfig {
|
|
|
27
28
|
};
|
|
28
29
|
/** Also use this provider when the caller supplied their own OpenRouter key. Default false: a fallback has the platform's key, and a caller's own key must never lead to the platform's money being spent. */
|
|
29
30
|
allowWithCallerKey?: boolean;
|
|
31
|
+
/** Your statement that this provider does not keep or train on API prompts (you have read its terms). A call marked `private` uses a fallback only if the provider's own model list says so for that model, or you say so here. Without either, `private` skips it: Lodestone cannot see what a provider does with a prompt unless the provider publishes it. */
|
|
32
|
+
privacy?: 'no-training';
|
|
30
33
|
}
|
|
31
34
|
export interface AttemptEvent {
|
|
32
35
|
provider: string;
|
|
@@ -39,21 +42,23 @@ export interface AttemptEvent {
|
|
|
39
42
|
}
|
|
40
43
|
/** Where the platform's money is counted. Only attempts that spend it go through: the paid models and the
|
|
41
44
|
* fallback providers on the platform's keys. A call on the caller's own key never does. */
|
|
45
|
+
export interface SpendInfo {
|
|
46
|
+
/** Names one allowed attempt, so allow(), record() and release() can be matched up. */
|
|
47
|
+
id: string;
|
|
48
|
+
provider: string;
|
|
49
|
+
model: string;
|
|
50
|
+
phase: 'paid' | 'fallback';
|
|
51
|
+
}
|
|
42
52
|
export interface SpendControl {
|
|
43
|
-
/** Asked before each such attempt. Return false once the limit is reached and that model is not used. */
|
|
44
|
-
allow(info:
|
|
45
|
-
provider: string;
|
|
46
|
-
model: string;
|
|
47
|
-
phase: 'paid' | 'fallback';
|
|
48
|
-
}): boolean | Promise<boolean>;
|
|
53
|
+
/** Asked before each such attempt. Return false once the limit is reached and that model is not used. Many calls can be in flight at once, so a limit that only compares a total with a number lets a burst through: take a hold under `info.id` here (createSpendLimit does). */
|
|
54
|
+
allow(info: SpendInfo): boolean | Promise<boolean>;
|
|
49
55
|
/** Told after each such attempt that answered, with what it cost: the provider's own figure when it gives one (OpenRouter does), otherwise an estimate from the model's catalogue price, otherwise nothing. */
|
|
50
|
-
record(info: {
|
|
51
|
-
provider: string;
|
|
52
|
-
model: string;
|
|
53
|
-
phase: 'paid' | 'fallback';
|
|
56
|
+
record(info: SpendInfo & {
|
|
54
57
|
usage?: Usage;
|
|
55
58
|
costUsd?: number;
|
|
56
59
|
}): void | Promise<void>;
|
|
60
|
+
/** Told when an attempt that allow() let through did not answer, so a hold taken in allow() can be given back. */
|
|
61
|
+
release?(info: SpendInfo): void | Promise<void>;
|
|
57
62
|
}
|
|
58
63
|
export interface LodestoneConfig {
|
|
59
64
|
openrouter?: {
|
|
@@ -85,6 +90,12 @@ export interface LodestoneConfig {
|
|
|
85
90
|
spend?: SpendControl;
|
|
86
91
|
/** Records every attempt so you can publish how each model does (stats.scorecard()). */
|
|
87
92
|
stats?: Stats;
|
|
93
|
+
/** Gives each caller their own allowance, so one person cannot spend everyone's budget. A call names its caller with `caller`; a call without one is not limited, and neither is a call on the caller's own key. See limiter.ts. */
|
|
94
|
+
rateLimit?: {
|
|
95
|
+
limiter: RateLimiter;
|
|
96
|
+
/** What happens to a caller over their allowance. 'reject' (default) fails the call with kind 'throttled'. 'free-only' still serves them, from free models only, so they never cost you anything. */
|
|
97
|
+
whenOver?: 'reject' | 'free-only';
|
|
98
|
+
};
|
|
88
99
|
/** Named tasks, so calls say what they are doing and not which model to use. */
|
|
89
100
|
profiles?: Record<string, Profile>;
|
|
90
101
|
/** Where the model catalogue and cached answers are kept. Default: memory. */
|
|
@@ -113,6 +124,15 @@ export interface RouteStep {
|
|
|
113
124
|
/** Longest this step may take. Default: most of what is left, keeping some for the steps after it. */
|
|
114
125
|
timeoutMs?: number;
|
|
115
126
|
}
|
|
127
|
+
/** What `json({ schema })` needs from a schema: Zod, Valibot (via its adapter) and ArkType-style libraries have it. */
|
|
128
|
+
export interface SafeParser<T> {
|
|
129
|
+
safeParse(value: unknown): {
|
|
130
|
+
success: true;
|
|
131
|
+
data: T;
|
|
132
|
+
} | {
|
|
133
|
+
success: false;
|
|
134
|
+
};
|
|
135
|
+
}
|
|
116
136
|
export interface CompleteOptions {
|
|
117
137
|
/** Skip tiers and phases: try exactly these steps, in order, until one answers. For callers that choose their own provider and model per request. Temperature is left to the provider unless set. */
|
|
118
138
|
route?: RouteStep[];
|
|
@@ -131,11 +151,13 @@ export interface CompleteOptions {
|
|
|
131
151
|
apiKey?: string;
|
|
132
152
|
/** An answer that fails this check counts as a failed attempt, and the next model is tried. */
|
|
133
153
|
accept?: (text: string) => boolean;
|
|
134
|
-
/** The prompt must not be kept or trained on (a CV, say). OpenRouter is told to use only providers that do not
|
|
154
|
+
/** The prompt must not be kept or trained on (a CV, say). OpenRouter is told to use only providers that do not. Another provider is used only if its own model list says the model is not trained on (Requesty does), or you vouched for it with `privacy: 'no-training'` on the fallback. Many free models cannot meet this, and the paid ones are used instead. A private call is never saved to the cache, whatever `cacheKey` says. A call with an explicit `route` is your own choice of provider: Lodestone passes the request to it as told. */
|
|
135
155
|
private?: boolean;
|
|
156
|
+
/** Who is making this call (a user id, an IP address), for `rateLimit`. Only a hash is stored. */
|
|
157
|
+
caller?: string;
|
|
136
158
|
/** Stops the call when the caller no longer wants the answer (the person left the page). */
|
|
137
159
|
signal?: AbortSignal;
|
|
138
|
-
/** Same key, same answer: the answer is saved and reused. Only for pure, repeatable tasks. */
|
|
160
|
+
/** Same key, same answer: the answer is saved and reused. Only for pure, repeatable tasks, and the key must stand for the whole input (use a hash of it). As a safeguard an answer is also tied to a fingerprint of the system prompt, messages and tier, so a key that is too coarse gives a miss, never another person's answer. Ignored when `private` is set. */
|
|
139
161
|
cacheKey?: string;
|
|
140
162
|
cacheTtlMs?: number;
|
|
141
163
|
}
|
|
@@ -167,6 +189,7 @@ export declare function createLodestone(config?: LodestoneConfig): {
|
|
|
167
189
|
complete: (opts: CompleteOptions) => Promise<CompleteResult>;
|
|
168
190
|
json: <T = unknown>(opts: CompleteOptions & {
|
|
169
191
|
validate?: (value: unknown) => value is T;
|
|
192
|
+
schema?: SafeParser<T>;
|
|
170
193
|
}) => Promise<CompleteResult & {
|
|
171
194
|
data: T;
|
|
172
195
|
}>;
|