@extrafemi-ai/lodestone 0.3.1 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md ADDED
@@ -0,0 +1,45 @@
1
+ # Changelog
2
+
3
+ Versions before 1.0 may change between minor releases; each change that could break a caller is marked **breaking**.
4
+
5
+ ## 0.5.0
6
+
7
+ Redundancy across providers, and model lists that keep themselves alive and safe.
8
+
9
+ - **Other providers.** `openaiCompatible()` and presets for [Requesty](https://www.requesty.ai) (`requesty()`, with an EU region), [LLM API](https://llmapi.ai) (`llmapi()`), Groq, Cerebras, OpenAI, Mistral, Together and Ollama. Each preset reads that service's model list, so a model it no longer offers is dropped. Use them as `fallbacks` so one provider's outage does not stop the tool.
10
+ - **Discovery that stays short.** A provider's list can be topped up from its own catalogue (`discover: { fill }`) with trusted models within a price limit, and never above `max`. For OpenRouter, a longer pool is kept and the live models are chosen from it, so a model that is down is replaced by the next best while a request still tries at most three.
11
+ - **Trust policy** (`trust`). Discovered models must come from a trusted lab (a default list, yours to change); a block list; refuse models a provider says are trained on, or that keep prompts too long (Requesty publishes both); drop models retiring within a week (OpenRouter publishes the date). **Breaking:** a free model from a lab that is not on the default list is no longer added by itself. Name it in your own list, or set `trust: { publishers: 'any' }`.
12
+ - **Private calls** (`private: true`). OpenRouter is told to use only providers that do not keep or train on the prompt; other models are used only if their provider's list says so.
13
+ - `checkModels()` now checks every provider that can list its models and reports models that are retiring (`expiring`).
14
+ - **Cancelling.** `signal` stops a call, and every model after it, when the caller no longer wants the answer.
15
+ - `peek(cacheKey)` reads a saved answer without calling a model.
16
+ - `free: true` on a fallback: a free tier or local server is not counted against, or stopped by, the spend limit.
17
+ - Fixed: an OpenRouter outage could use the whole time budget on free models before another provider got a turn.
18
+ - Fixed: a model's catalogue is fetched in the background the first time it is needed, not only by the daily check.
19
+
20
+ ## 0.4.0
21
+
22
+ - **Circuit breaker** (`createHealth`): a model whose last three attempts failed is skipped by every function until it recovers; one attempt is let through every two minutes to see.
23
+ - **Spend control** (`spend`): asked before the platform's money is used, told what each call cost (OpenRouter reports it; otherwise estimated from catalogue price). A caller's own key never goes through it.
24
+ - `openrouter.paidFirst`: try the paid models first on the platform's key and use the free ones as backup, for when free models are not worth waiting for.
25
+ - `openrouter({ reasoning })`: set how much each model thinks. Measured: turning it off for `claude-haiku-5.5` halved scoring time with the same answers; Gemini refuses to run with it off.
26
+ - New failure kind `budget`.
27
+
28
+ ## 0.3.1
29
+
30
+ - A call on the caller's own key no longer reaches the platform's fallback providers unless a fallback sets `allowWithCallerKey`.
31
+ - Attempts that fail with 401, 402 or 403 are not counted against a model in live stats.
32
+
33
+ ## 0.3.0
34
+
35
+ - **Scorecards**: a `model-scorecard/1` file of measured results (for example from [Naija Eval](https://naija-eval.extrafemi.com)) can put the best-measured model first, per task. `parseScorecard`, `fetchScorecard`, `rankModels`.
36
+ - `createStats()` publishes live success rate and speed in the same format. `ai.candidates()` lists the models in use for an evaluator. See `docs/scorecards.md`.
37
+
38
+ ## 0.2.0
39
+
40
+ - `route`: try exactly these providers, models and keys, in order, for callers that choose their own.
41
+ - Errors carry the provider's own message (`detail.providerMessage`). Temperature is sent only when set.
42
+
43
+ ## 0.1.0
44
+
45
+ - First release: `complete()` and `json()` with time-boxed attempts across OpenRouter's free models, a model list that replaces withdrawn models from the live catalogue, four kinds of rate limit told apart, a paid fallback, plain-language `explain()`, bring-your-own-key, and Anthropic, Google and mock adapters.
package/README.md CHANGED
@@ -87,6 +87,82 @@ const { ok, withdrawn, shortTiers } = await ai.checkModels();
87
87
 
88
88
  `withdrawn` lists preferred models that have gone. That is harmless, they are already replaced. `ok` is false only when the catalogue can't be read or a tier is left with fewer than two real models. Without a daily run, a request that finds the saved catalogue over a day old refreshes it in the background.
89
89
 
90
+ ### Free models that are not worth waiting for (`paidFirst`)
91
+
92
+ Free models can be down for hours (rate-limited, or holding a connection open for 45 seconds and never answering). On the platform's own key, `openrouter: { paidFirst: true }` tries the low-cost paid model first and the free ones only if it fails or the spend limit is reached, so a request is not slowed by free models that are not going to answer. A caller's own key is not affected: the paid models are their money, so their free models still come first. Reasoning is set per model with `openrouter({ reasoning: (model) => ... })`, because models differ: one thinks for a thousand tokens unless told not to (measured: turning it off halved scoring time, 10 s to 5 s, with the same answers), another refuses to run with reasoning off.
93
+
94
+ ### Not waiting on a model that is down (circuit breaker)
95
+
96
+ Free models fail in bursts: rate-limited, stuck, or answering nothing. Without memory, every request pays for the same dead models again. `createHealth()` remembers:
97
+
98
+ ```ts
99
+ import { createHealth } from '@extrafemi-ai/lodestone';
100
+ const health = createHealth({ store }); // the same store as the catalogue: every serverless function shares it
101
+ const ai = createLodestone({ /* ... */, health });
102
+ ```
103
+
104
+ A model whose last three attempts (within ten minutes) all failed is skipped, so the next request goes straight to one that works. Every two minutes one attempt is let through to see whether it has recovered, and one success closes it again. A timeout, a rate limit, a server error and an unusable answer count as failures; a rejected key or missing credit (401, 402, 403) say nothing about the model and never do. If every free model looks down and a paid one can follow, the free ones are skipped entirely; if nothing can follow, they are still tried, because trying is better than not answering. `health.snapshot()` lists what it has seen, for a health page.
105
+
106
+ ### Limiting what the platform spends
107
+
108
+ ```ts
109
+ const ai = createLodestone({
110
+ /* ... */
111
+ spend: {
112
+ allow: async ({ model, phase }) => (await spentToday()) < DAILY_LIMIT_USD,
113
+ record: async ({ costUsd }) => addToday(costUsd ?? 0),
114
+ },
115
+ });
116
+ ```
117
+
118
+ `allow` is asked before each attempt that spends the platform's money (the paid models and fallback providers on the platform's keys), and a `false` means that model is not used. `record` is told what each such call cost: the provider's own figure when it gives one (OpenRouter does), otherwise an estimate from the model's catalogue price. Free models and a caller's own key (their money, not yours) never go through either. When the limit stops every paid option the error is `kind: 'budget'`. A `record` that throws never fails the answer.
119
+
120
+ ### Providers, and not depending on one
121
+
122
+ OpenRouter is the default, but one provider's outage should not stop your tool. Give Lodestone others as `fallbacks`, each with its own key. Any service that speaks OpenAI's chat API works through `openaiCompatible()`, and presets exist for the ones Lodestone has been checked against:
123
+
124
+ | Preset | Service | Notes |
125
+ |---|---|---|
126
+ | `requesty()` | [Requesty](https://www.requesty.ai) | Gateway to 800+ models with failover. Publishes price, lab, whether a model is trained on and how long prompts are kept. `requesty({ region: 'eu' })` keeps prompts in the EU. No free models; a 5% markup. |
127
+ | `llmapi()` | [LLM API](https://llmapi.ai) | Gateway to 150+ chat models. Lists price; ignores its image, video and speech models. |
128
+ | `groq()`, `cerebras()` | Groq, Cerebras | Fast inference with free tiers. Use `free: true` so they are not counted against a spend limit. |
129
+ | `openai()`, `mistral()`, `together()` | the providers themselves | |
130
+ | `ollama()` | a local server | Free and private. |
131
+ | `anthropic()`, `google()` | direct | Not OpenAI-style, so they have their own adapters. |
132
+
133
+ ```ts
134
+ fallbacks: [
135
+ { provider: requesty(), apiKey: process.env.REQUESTY_API_KEY!, models: ['bedrock/claude-haiku-5-5'], discover: { fill: { maxPromptUsdPerMTok: 1, maxCompletionUsdPerMTok: 5 }, max: 3 } },
136
+ { provider: groq(), apiKey: process.env.GROQ_API_KEY!, models: ['llama-3.3-70b-versatile'], free: true },
137
+ ]
138
+ ```
139
+
140
+ Model ids are each service's own. See [examples/multi-provider.ts](examples/multi-provider.ts).
141
+
142
+ ### Model lists that keep themselves alive, and short
143
+
144
+ - **Outdated or shut down.** The daily `checkModels()` saves each provider's current catalogue. A model the provider no longer lists is dropped; one with an expiry date within a week (OpenRouter publishes it) is left out; one that is failing right now is skipped by the circuit breaker. Nobody edits a list.
145
+ - **New models.** `discover: { fill }` tops a short list up from the provider's own catalogue with models within a price limit ("free" is a limit of zero) from trusted labs. For OpenRouter's free models this happens by itself.
146
+ - **Not a longer wait.** Discovery never lengthens what a request walks. A background pool (up to 8 per tier) is ranked, the models that are down are filtered out, and a request tries at most three. When one fails, the next best takes its place for later requests, so adding models adds choice, not delay.
147
+ - **Safe by default.** A discovered model is used only if its publisher is on the trust list; `trust` also takes a block list, refuses models a provider says are trained on or that keep prompts too long, and `private: true` applies the same to one call. There is no vulnerability database for hosted models, so none is claimed: see [SECURITY.md](SECURITY.md) for exactly what is and is not checked.
148
+
149
+ ### Cancelling
150
+
151
+ ```ts
152
+ const controller = new AbortController();
153
+ req.on('close', () => controller.abort()); // the person left the page
154
+ await ai.complete({ prompt, signal: controller.signal }); // throws a LodestoneError of kind 'aborted'
155
+ ```
156
+
157
+ ### Where it runs
158
+
159
+ Lodestone uses only `fetch`, `AbortSignal` and `URL`, with no Node-specific APIs, so it should run on Cloudflare Workers, Deno and edge runtimes too. **It is tested on Node 20 and 22 only.** Reports from other runtimes are welcome.
160
+
161
+ ### Not yet
162
+
163
+ - **Streaming.** `complete()` and `json()` wait for the whole answer. Streaming needs a decision about a model that fails halfway; the likely rule is to fail over only before the first token.
164
+ - **Tool calling** (functions the model can call), and **embeddings or speech**. Lodestone is for one-shot text and JSON calls today.
165
+
90
166
  ### Letting measurements decide the order
91
167
 
92
168
  A scorecard (a small JSON file of measured results) can put the models that did best on your kind of task first, and Lodestone can publish its own live success rates and speeds in the same format. [Naija Eval](https://naija-eval.extrafemi.com) publishes translation and speech quality for Hausa, Igbo and Yoruba this way. It is opt-in, per task, and only changes the order models are tried in. See [docs/scorecards.md](docs/scorecards.md).
package/SECURITY.md ADDED
@@ -0,0 +1,35 @@
1
+ # Security
2
+
3
+ ## Reporting a problem
4
+
5
+ Please report a vulnerability privately, not in a public issue: use GitHub's **Report a vulnerability** button on the
6
+ Security tab of this repository. Say what you found, how to reproduce it, and which version. You will get a reply
7
+ within a few days. Please give a reasonable time to fix it before you publish details.
8
+
9
+ ## What this package is, for supply-chain purposes
10
+
11
+ - **No runtime dependencies.** Lodestone uses only `fetch`, `AbortSignal` and `URL`. Installing it brings in nothing else,
12
+ so there is nothing underneath it to be taken over.
13
+ - **Published with provenance.** Each release is built by a GitHub Actions workflow from a tagged commit and published to
14
+ npm with a signed attestation. `npm audit signatures` verifies that a package came from this repository.
15
+ - **Small.** About 1,600 lines of source, with tests beside it. Read it before you depend on it.
16
+ - **Your keys stay yours.** Keys are passed in by your code and sent only to the provider you name. They are never stored,
17
+ logged or sent anywhere else, and prompts and answers are never logged. `onAttempt` reports the provider, model, outcome
18
+ and timing only.
19
+
20
+ ## What Lodestone checks about models, and what it does not
21
+
22
+ There is no vulnerability database for hosted AI models, so Lodestone does not claim to scan for one. It checks what can
23
+ really be checked, and says where it cannot:
24
+
25
+ - **Who made it.** A model found by discovery is used only if its publisher is on a list of well-known labs (yours to
26
+ change). A model you name yourself is your decision.
27
+ - **What happens to the prompt.** Where a provider publishes it (Requesty does, for every model), a model that is trained on,
28
+ or that keeps prompts longer than you allow, is refused. For OpenRouter, a call marked `private` asks for providers that do
29
+ not keep or train on prompts. For other providers Lodestone cannot enforce this: it relies on the model's list entry, and
30
+ refuses a model whose list says nothing.
31
+ - **That it is not about to disappear.** A model with an expiry date within a week is left out.
32
+ - **That the answer is the shape you expect.** `json()` skips a model whose answer is not valid JSON or fails your check,
33
+ which also limits what a misbehaving model can hand to the rest of your code. Treat every model answer as untrusted input.
34
+
35
+ These are self-reported by providers and are only as good as what they report.
package/dist/catalog.d.ts CHANGED
@@ -1,3 +1,4 @@
1
+ import { type TrustPolicy } from './trust.js';
1
2
  import type { KeyValueStore, Tier } from './types.js';
2
3
  /** Works out which models to ask for from OpenRouter's own catalogue, so a model
3
4
  * that has been withdrawn is dropped and a current one takes its place without
@@ -14,17 +15,52 @@ export interface CatalogModel {
14
15
  prompt?: string | number;
15
16
  completion?: string | number;
16
17
  };
18
+ /** The lab that made it, when the provider says (Requesty's model_lab); otherwise the first part of the id. */
19
+ publisher?: string;
20
+ /** The day the provider stops offering it (OpenRouter publishes this for models being retired). */
21
+ expiration_date?: string;
22
+ /** What the provider says happens to prompts (Requesty publishes these for every model). */
23
+ data_used_for_training?: boolean;
24
+ data_retention_days?: number;
25
+ /** Zero price, as the provider says it. */
26
+ free?: boolean;
17
27
  }
18
28
  export declare const ROUTER = "openrouter/free";
19
- export declare function eligible(catalog: CatalogModel[], tier: Tier): CatalogModel[];
29
+ /** US dollars per million tokens, or undefined if the provider gives no price. */
30
+ export declare const promptPerMTok: (m: CatalogModel) => number | undefined;
31
+ export declare const completionPerMTok: (m: CatalogModel) => number | undefined;
32
+ /** Text models fit for the tier. Free ones, unless `price` allows more (then at most that, per million tokens). */
33
+ export declare function eligible(catalog: CatalogModel[], tier: Tier, price?: {
34
+ promptUsdPerMTok: number;
35
+ completionUsdPerMTok: number;
36
+ }): CatalogModel[];
37
+ /** How the candidates for a tier are ranked: bigger is stronger, lighter is faster, then longer context. */
38
+ export declare function rank(tier: Tier, a: CatalogModel, b: CatalogModel): number;
39
+ export interface PickOptions {
40
+ policy?: TrustPolicy;
41
+ /** Treat unknown privacy facts as a refusal even for a preferred model (a call marked private). */
42
+ strict?: boolean;
43
+ /** How many to return, the router included. Default 3. A larger number gives a pool to choose live models from. */
44
+ depth?: number;
45
+ now?: number;
46
+ }
20
47
  /** The ids to send, best first: preferred models that still exist, then the best
21
- * others by tier, then the router; at most three. With no catalogue at all, the
22
- * preferences are returned as they are. */
23
- export declare function pickModels(tier: Tier, catalog: CatalogModel[] | null, preferences: string[]): string[];
48
+ * others by tier, then the router; at most `depth` (three by default). With no catalogue at all, the
49
+ * preferences are returned as they are. A model the trust policy refuses, or that is about to be
50
+ * retired, is left out. */
51
+ export declare function pickModels(tier: Tier, catalog: CatalogModel[] | null, preferences: string[], opts?: PickOptions): string[];
24
52
  export interface Catalog {
25
53
  /** Fetches OpenRouter's catalogue and saves it. Returns null if it can't be read. */
26
54
  refresh(): Promise<CatalogModel[] | null>;
27
55
  /** The saved catalogue (six-hour copy in memory; refreshed in the background if over a day old). Never waits on the network unless nothing is saved. */
28
56
  load(): Promise<CatalogModel[] | null>;
29
57
  }
30
- export declare function createCatalog(store: KeyValueStore, fetchImpl: typeof fetch, now?: () => number): Catalog;
58
+ /** OpenRouter's catalogue, as a list of CatalogModel. */
59
+ export declare function listOpenRouter(fetchImpl: typeof fetch): Promise<CatalogModel[] | null>;
60
+ export interface CatalogSource {
61
+ /** Where it is saved. Default: the OpenRouter catalogue's own key. */
62
+ key?: string;
63
+ /** How to read the provider's list. Default: OpenRouter's. */
64
+ list?: () => Promise<CatalogModel[] | null>;
65
+ }
66
+ export declare function createCatalog(store: KeyValueStore, fetchImpl: typeof fetch, now?: () => number, source?: CatalogSource): Catalog;
package/dist/catalog.js CHANGED
@@ -1,6 +1,10 @@
1
+ import { vet } from './trust.js';
1
2
  export const ROUTER = 'openrouter/free';
2
3
  const MAX_ENTRIES = 3;
3
4
  const isFree = (m) => m.id.endsWith(':free') || (Number(m.pricing?.prompt) === 0 && Number(m.pricing?.completion) === 0 && m.id !== ROUTER);
5
+ /** US dollars per million tokens, or undefined if the provider gives no price. */
6
+ export const promptPerMTok = (m) => { const n = Number(m.pricing?.prompt); return m.pricing?.prompt !== undefined && m.pricing?.prompt !== '' && Number.isFinite(n) ? n * 1e6 : undefined; };
7
+ export const completionPerMTok = (m) => { const n = Number(m.pricing?.completion); return m.pricing?.completion !== undefined && m.pricing?.completion !== '' && Number.isFinite(n) ? n * 1e6 : undefined; };
4
8
  const textOut = (m) => (m.architecture?.output_modalities ?? ['text']).join() === 'text';
5
9
  const takesImages = (m) => (m.architecture?.input_modalities ?? []).includes('image');
6
10
  // Safety classifiers, audio and music models, code-only models and embedders are not for writing or reading text.
@@ -8,45 +12,64 @@ const EXCLUDE = /safety|guard|moderation|lyria|embed|rerank|tts|whisper|code|cod
8
12
  /** "550b" or "26b" in an id: bigger is the stronger pick. */
9
13
  const size = (id) => Number(id.match(/(\d{2,4})b\b/i)?.[1] ?? 0);
10
14
  const LIGHT = /lightning|flash|mini|small|nano|lite|fast/i;
11
- export function eligible(catalog, tier) {
12
- return catalog.filter((m) => isFree(m) && textOut(m) && (m.context_length ?? 0) >= 32_000 && !EXCLUDE.test(m.id) && (tier !== 'vision' || takesImages(m)));
15
+ /** Text models fit for the tier. Free ones, unless `price` allows more (then at most that, per million tokens). */
16
+ export function eligible(catalog, tier, price) {
17
+ return catalog.filter((m) => {
18
+ const affordable = price
19
+ ? (promptPerMTok(m) ?? Infinity) <= price.promptUsdPerMTok && (completionPerMTok(m) ?? Infinity) <= price.completionUsdPerMTok
20
+ : isFree(m);
21
+ return affordable && textOut(m) && (m.context_length ?? 0) >= 32_000 && !EXCLUDE.test(m.id) && (tier !== 'vision' || takesImages(m));
22
+ });
23
+ }
24
+ /** How the candidates for a tier are ranked: bigger is stronger, lighter is faster, then longer context. */
25
+ export function rank(tier, a, b) {
26
+ return (tier === 'strong' ? size(b.id) - size(a.id) : tier === 'fast' ? Number(LIGHT.test(b.id)) - Number(LIGHT.test(a.id)) || size(a.id) - size(b.id) : size(b.id) - size(a.id)) || (b.context_length ?? 0) - (a.context_length ?? 0);
13
27
  }
14
28
  /** The ids to send, best first: preferred models that still exist, then the best
15
- * others by tier, then the router; at most three. With no catalogue at all, the
16
- * preferences are returned as they are. */
17
- export function pickModels(tier, catalog, preferences) {
29
+ * others by tier, then the router; at most `depth` (three by default). With no catalogue at all, the
30
+ * preferences are returned as they are. A model the trust policy refuses, or that is about to be
31
+ * retired, is left out. */
32
+ export function pickModels(tier, catalog, preferences, opts = {}) {
33
+ const depth = opts.depth ?? MAX_ENTRIES;
18
34
  if (!catalog || catalog.length === 0)
19
- return preferences.slice(0, MAX_ENTRIES);
35
+ return preferences.slice(0, depth);
36
+ const now = opts.now ?? Date.now();
20
37
  const ok = new Map(eligible(catalog, tier).map((m) => [m.id, m]));
21
- const out = preferences.filter((id) => id !== ROUTER && ok.has(id));
22
- const rest = [...ok.values()]
23
- .filter((m) => !out.includes(m.id))
24
- .sort((a, b) => (tier === 'strong' ? size(b.id) - size(a.id) : tier === 'fast' ? Number(LIGHT.test(b.id)) - Number(LIGHT.test(a.id)) || size(a.id) - size(b.id) : size(b.id) - size(a.id)) || (b.context_length ?? 0) - (a.context_length ?? 0));
38
+ const out = preferences.filter((id) => id !== ROUTER && ok.has(id) && vet(ok.get(id), opts.policy, { discovered: false, strict: opts.strict, now }).ok);
39
+ const rest = [...ok.values()].filter((m) => !out.includes(m.id) && vet(m, opts.policy, { discovered: true, strict: opts.strict, now }).ok).sort((a, b) => rank(tier, a, b));
25
40
  for (const m of rest) {
26
- if (out.length >= MAX_ENTRIES - 1)
41
+ if (out.length >= depth - 1)
27
42
  break;
28
43
  out.push(m.id);
29
44
  }
30
- return [...out.slice(0, MAX_ENTRIES - 1), ROUTER];
45
+ return [...out.slice(0, depth - 1), ROUTER];
31
46
  }
32
47
  const KEY = 'lodestone:model-catalog';
33
48
  const MEMORY_MS = 6 * 3_600_000;
34
49
  const STALE_MS = 24 * 3_600_000;
35
- export function createCatalog(store, fetchImpl, now = Date.now) {
50
+ /** OpenRouter's catalogue, as a list of CatalogModel. */
51
+ export async function listOpenRouter(fetchImpl) {
52
+ const res = await fetchImpl('https://openrouter.ai/api/v1/models', { signal: AbortSignal.timeout(8_000) });
53
+ if (!res.ok)
54
+ return null;
55
+ const models = ((await res.json()).data ?? []).map((m) => ({
56
+ id: m.id,
57
+ context_length: m.context_length,
58
+ architecture: m.architecture && { input_modalities: m.architecture.input_modalities, output_modalities: m.architecture.output_modalities },
59
+ pricing: m.pricing && { prompt: m.pricing.prompt, completion: m.pricing.completion },
60
+ ...(m.expiration_date ? { expiration_date: m.expiration_date } : {}),
61
+ }));
62
+ return models.length ? models : null;
63
+ }
64
+ export function createCatalog(store, fetchImpl, now = Date.now, source = {}) {
36
65
  let memory = null;
66
+ let lastKick = -Infinity;
67
+ const KEY_ = source.key ?? KEY;
37
68
  const refresh = async () => {
38
- const res = await fetchImpl('https://openrouter.ai/api/v1/models', { signal: AbortSignal.timeout(8_000) });
39
- if (!res.ok)
69
+ const models = await (source.list ? source.list() : listOpenRouter(fetchImpl));
70
+ if (!models || models.length === 0)
40
71
  return null;
41
- const models = ((await res.json()).data ?? []).map((m) => ({
42
- id: m.id,
43
- context_length: m.context_length,
44
- architecture: m.architecture && { input_modalities: m.architecture.input_modalities, output_modalities: m.architecture.output_modalities },
45
- pricing: m.pricing && { prompt: m.pricing.prompt, completion: m.pricing.completion },
46
- }));
47
- if (models.length === 0)
48
- return null;
49
- await store.set(KEY, { at: new Date(now()).toISOString(), models }).catch(() => { });
72
+ await store.set(KEY_, { at: new Date(now()).toISOString(), models }).catch(() => { });
50
73
  memory = { at: now(), models };
51
74
  return models;
52
75
  };
@@ -54,13 +77,18 @@ export function createCatalog(store, fetchImpl, now = Date.now) {
54
77
  if (memory && now() - memory.at < MEMORY_MS)
55
78
  return memory.models;
56
79
  try {
57
- const saved = (await store.get(KEY));
80
+ const saved = (await store.get(KEY_));
58
81
  if (saved?.models?.length) {
59
82
  memory = { at: now(), models: saved.models };
60
83
  if (now() - Date.parse(saved.at) > STALE_MS)
61
84
  refresh().catch(() => { });
62
85
  return saved.models;
63
86
  }
87
+ // Nothing saved yet: use the preferences this time, and fetch it for the next request.
88
+ if (now() - lastKick > 5 * 60_000) {
89
+ lastKick = now();
90
+ refresh().catch(() => { });
91
+ }
64
92
  }
65
93
  catch { /* an unreadable store is the same as an empty one */ }
66
94
  return null;
@@ -0,0 +1,24 @@
1
+ import { type CatalogModel } from './catalog.js';
2
+ import { type TrustPolicy } from './trust.js';
3
+ import type { Tier } from './types.js';
4
+ /** Keeps a provider's list of models alive without anyone editing it: a model the provider no
5
+ * longer lists is dropped, one that is retiring soon or that the trust policy refuses is dropped,
6
+ * and if the list is then short, it is filled from the provider's own catalogue with models that are
7
+ * within a price limit and made by a trusted publisher. Filling only ever adds models after the
8
+ * app's own, and never above `max`, so the list a request walks stays short however many models
9
+ * a provider offers. */
10
+ export interface FillOptions {
11
+ /** The most a filled-in model may cost, in US dollars per million tokens. { prompt: 0, completion: 0 } is "free". */
12
+ maxPromptUsdPerMTok: number;
13
+ maxCompletionUsdPerMTok: number;
14
+ }
15
+ export interface ResolveOptions {
16
+ tier: Tier;
17
+ policy?: TrustPolicy;
18
+ strict?: boolean;
19
+ fill?: FillOptions;
20
+ /** The most models to return. Default 3. */
21
+ max?: number;
22
+ now: number;
23
+ }
24
+ export declare function resolveProviderModels(curated: string[], catalog: CatalogModel[] | null, o: ResolveOptions): string[];
@@ -0,0 +1,20 @@
1
+ import { eligible, rank } from './catalog.js';
2
+ import { vet } from './trust.js';
3
+ export function resolveProviderModels(curated, catalog, o) {
4
+ if (!catalog || catalog.length === 0)
5
+ return curated; // nothing known about what exists: use what the app named
6
+ const max = o.max ?? 3;
7
+ const byId = new Map(catalog.map((m) => [m.id, m]));
8
+ const kept = curated.filter((id) => { const m = byId.get(id); return m !== undefined && vet(m, o.policy, { discovered: false, strict: o.strict, now: o.now }).ok; });
9
+ if (!o.fill || kept.length >= max)
10
+ return kept;
11
+ const fill = eligible(catalog, o.tier, { promptUsdPerMTok: o.fill.maxPromptUsdPerMTok, completionUsdPerMTok: o.fill.maxCompletionUsdPerMTok })
12
+ .filter((m) => !kept.includes(m.id) && vet(m, o.policy, { discovered: true, strict: o.strict, now: o.now }).ok)
13
+ .sort((a, b) => rank(o.tier, a, b));
14
+ for (const m of fill) {
15
+ if (kept.length >= max)
16
+ break;
17
+ kept.push(m.id);
18
+ }
19
+ return kept;
20
+ }
package/dist/errors.d.ts CHANGED
@@ -22,6 +22,10 @@ export type FailureKind =
22
22
  | 'unavailable'
23
23
  /** A model answered but the answer was not usable (empty, not valid JSON, failed validation). */
24
24
  | 'bad_output'
25
+ /** The caller stopped the call (it passed a signal that was aborted). */
26
+ | 'aborted'
27
+ /** The platform's spend limit has been reached, so no paid model was used. */
28
+ | 'budget'
25
29
  /** No provider has a key. */
26
30
  | 'not_configured' | 'other';
27
31
  /** Thrown when no model could answer. `kind` says why, so the caller can decide what
package/dist/explain.js CHANGED
@@ -42,6 +42,10 @@ export function explain(err, opts = {}) {
42
42
  return "The AI gave back something that couldn't be read. Try again; it usually works on a second attempt.";
43
43
  case 'credit':
44
44
  return 'The AI account has no credit for the backup model. Add a little credit and try again.';
45
+ case 'aborted':
46
+ return 'Cancelled.';
47
+ case 'budget':
48
+ return `${product}'s AI budget for now is used up, and the free models are busy. Try again later${opts.settingsHint ? `, or add your own OpenRouter key ${opts.settingsHint.replace(/^in /, 'in ')} to keep going` : ''}.`;
45
49
  case 'not_configured':
46
50
  return `${product}'s AI isn't set up yet.`;
47
51
  default:
@@ -0,0 +1,39 @@
1
+ import type { KeyValueStore } from './types.js';
2
+ /** A circuit breaker: remembers which models have been failing, so a request does not spend its
3
+ * time on a model that is down. A model whose last few attempts all failed is skipped; every
4
+ * couple of minutes one attempt is let through to see whether it has recovered, and one success
5
+ * closes the breaker again. With a store, every instance (every serverless function) shares what
6
+ * it has learned, so one request's failure saves the next request's time.
7
+ *
8
+ * Counts a timeout, a rate limit, a server error and an unusable answer (empty, not JSON) as
9
+ * failures. A rejected key or missing credit (401, 402, 403) says nothing about the model and is
10
+ * never recorded. */
11
+ export interface HealthOptions {
12
+ /** Where to keep what it has learned, shared across instances. Default: memory only. */
13
+ store?: KeyValueStore;
14
+ key?: string;
15
+ /** A model is skipped when its last `failures` attempts all failed within `windowMs`. Default 3 and 10 minutes. */
16
+ failures?: number;
17
+ windowMs?: number;
18
+ /** While a model is skipped, one attempt is let through this often. Default 2 minutes. */
19
+ probeEveryMs?: number;
20
+ now?: () => number;
21
+ }
22
+ export interface ModelHealth {
23
+ provider: string;
24
+ model: string;
25
+ /** Skipped right now. */
26
+ open: boolean;
27
+ /** Failures among its last few attempts. */
28
+ recentFailures: number;
29
+ recentAttempts: number;
30
+ lastFailureAt: string | null;
31
+ }
32
+ export interface Health {
33
+ /** Should this model be skipped right now? */
34
+ shouldSkip(provider: string, model: string): Promise<boolean>;
35
+ record(provider: string, model: string, ok: boolean): Promise<void>;
36
+ /** Every model it has seen, for a health page. Models not seen in the window are left out. */
37
+ snapshot(): Promise<ModelHealth[]>;
38
+ }
39
+ export declare function createHealth(opts?: HealthOptions): Health;
package/dist/health.js ADDED
@@ -0,0 +1,73 @@
1
+ const KEEP = 6;
2
+ const SHARE_MS = 15_000;
3
+ const id = (provider, model) => `${provider}\u0000${model}`;
4
+ export function createHealth(opts = {}) {
5
+ const need = opts.failures ?? 3;
6
+ const windowMs = opts.windowMs ?? 10 * 60_000;
7
+ const probeMs = opts.probeEveryMs ?? 2 * 60_000;
8
+ const now = opts.now ?? Date.now;
9
+ const key = opts.key ?? 'lodestone:health';
10
+ let state = {};
11
+ let loadedAt = -Infinity;
12
+ /** Brings in what other instances have saved, at most every 15 seconds. */
13
+ async function refresh() {
14
+ if (!opts.store || now() - loadedAt < SHARE_MS)
15
+ return;
16
+ loadedAt = now();
17
+ try {
18
+ state = merge(state, (await opts.store.get(key)) ?? {});
19
+ }
20
+ catch { /* an unreadable store is the same as an empty one */ }
21
+ }
22
+ const merge = (a, b) => {
23
+ const out = {};
24
+ for (const k of new Set([...Object.keys(a), ...Object.keys(b)])) {
25
+ const seen = new Set();
26
+ out[k] = [...(a[k] ?? []), ...(b[k] ?? [])].filter(([at, , sid]) => { const u = `${at}:${sid}`; return seen.has(u) ? false : (seen.add(u), true); }).sort((x, y) => x[0] - y[0]).slice(-KEEP);
27
+ }
28
+ return out;
29
+ };
30
+ const recent = (k) => (state[k] ?? []).filter(([at]) => now() - at <= windowMs);
31
+ const isOpen = (r) => {
32
+ const last = r.slice(-need);
33
+ if (last.length < need || last.some(([, ok]) => ok === 1))
34
+ return false;
35
+ return now() - last[last.length - 1][0] < probeMs; // after a while, let one attempt through to see if it has recovered
36
+ };
37
+ return {
38
+ async shouldSkip(provider, model) {
39
+ await refresh();
40
+ return isOpen(recent(id(provider, model)));
41
+ },
42
+ async record(provider, model, ok) {
43
+ const k = id(provider, model);
44
+ const before = recent(k);
45
+ const recovering = before.length > 0 && before[before.length - 1][1] === 0; // this success follows a failure
46
+ state[k] = [...(state[k] ?? []), [now(), ok ? 1 : 0, Math.random().toString(36).slice(2, 8)]].slice(-KEEP);
47
+ // Only failures, and a success right after a failure, change what anyone else needs to know;
48
+ // a model that is simply working costs nothing extra.
49
+ if (!opts.store || (ok && !recovering))
50
+ return;
51
+ try {
52
+ const theirs = (await opts.store.get(key)) ?? {};
53
+ state = merge(state, theirs);
54
+ await opts.store.set(key, state);
55
+ loadedAt = now();
56
+ }
57
+ catch { /* not saved: the next instance just has to learn it again */ }
58
+ },
59
+ async snapshot() {
60
+ await refresh();
61
+ const out = [];
62
+ for (const k of Object.keys(state)) {
63
+ const r = recent(k);
64
+ if (!r.length)
65
+ continue;
66
+ const [provider, model] = k.split('\u0000');
67
+ const fails = r.filter(([, ok]) => ok === 0);
68
+ out.push({ provider, model, open: isOpen(r), recentFailures: fails.length, recentAttempts: r.length, lastFailureAt: fails.length ? new Date(fails[fails.length - 1][0]).toISOString() : null });
69
+ }
70
+ return out;
71
+ },
72
+ };
73
+ }
package/dist/index.d.ts CHANGED
@@ -1,5 +1,5 @@
1
1
  export { createLodestone } from './lodestone.js';
2
- export type { Lodestone, LodestoneConfig, CompleteOptions, CompleteResult, Profile, FallbackConfig, AttemptEvent, RouteStep } from './lodestone.js';
2
+ export type { Lodestone, LodestoneConfig, CompleteOptions, CompleteResult, Profile, FallbackConfig, AttemptEvent, RouteStep, SpendControl, CheckResult } from './lodestone.js';
3
3
  export { LodestoneError, ProviderError, messageFrom } from './errors.js';
4
4
  export type { FailureKind } from './errors.js';
5
5
  export { explain } from './explain.js';
@@ -22,3 +22,12 @@ export { createStats } from './stats.js';
22
22
  export type { Stats, StatsOptions, StatsState } from './stats.js';
23
23
  export { CANDIDATES_SCHEMA } from './candidates.js';
24
24
  export type { Candidates, Candidate } from './candidates.js';
25
+ export { createHealth } from './health.js';
26
+ export type { Health, HealthOptions, ModelHealth } from './health.js';
27
+ export { openaiCompatible, requesty, llmapi, groq, cerebras, openai, mistral, together, ollama, normalizeRequesty, normalizeLlmapi } from './providers/openaiCompatible.js';
28
+ export type { OpenAICompatibleOptions, PresetOptions } from './providers/openaiCompatible.js';
29
+ export { vet, publisherOf, DEFAULT_PUBLISHERS } from './trust.js';
30
+ export type { TrustPolicy, Verdict } from './trust.js';
31
+ export { resolveProviderModels } from './discover.js';
32
+ export type { FillOptions } from './discover.js';
33
+ export type { CatalogModel as ModelInfo } from './catalog.js';
package/dist/index.js CHANGED
@@ -13,3 +13,7 @@ export { mock } from './providers/mock.js';
13
13
  export { SCORECARD_SCHEMA, ScorecardError, parseScorecard, fetchScorecard, rankModels, scoreFor, baseModelId } from './scorecard.js';
14
14
  export { createStats } from './stats.js';
15
15
  export { CANDIDATES_SCHEMA } from './candidates.js';
16
+ export { createHealth } from './health.js';
17
+ export { openaiCompatible, requesty, llmapi, groq, cerebras, openai, mistral, together, ollama, normalizeRequesty, normalizeLlmapi } from './providers/openaiCompatible.js';
18
+ export { vet, publisherOf, DEFAULT_PUBLISHERS } from './trust.js';
19
+ export { resolveProviderModels } from './discover.js';