@extrafemi-ai/lodestone 0.7.0 → 0.8.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -2,6 +2,19 @@
2
2
 
3
3
  Versions before 1.0 may change between minor releases; each change that could break a caller is marked **breaking**.
4
4
 
5
+ ## 0.8.0
6
+
7
+ - **A spend share for each caller.** `createSpendLimit({ perCaller: { dailyUsd, monthlyUsd } })` gives each caller (`caller` on the call; only a hash is stored) their own share of the pool, so one person cannot use it up for everyone. A call over its share fails with `kind: 'budget'` and `detail.scope: 'caller'` (`'app'` when the whole pool ran out); `explain()` words each. `spendCap` on a call lets a person lower their own share, never raise it. `spend.callerReport(caller)` says what they have used. A custom `SpendControl` can implement `refusal(id)` to report the scope.
8
+ - **An optional cap on a person's own key, off by default.** `spendCap` on a call that carries the person's own `apiKey` counts their paid calls on that key in a ledger of their own (never the pool's) and stops at the number they chose (`scope: 'own-key'`). Without `spendCap` a caller's own key is still neither counted nor limited. A `spendCap` that cannot be kept (no `caller`, or a spend limit not made by `createSpendLimit`) throws a `TypeError` instead of being ignored. The provider's own limit on the key is the one that really holds; the docs say so. Python: `SpendLimit(per_caller=...)`, `spend_cap=` on a call, `caller_report()`.
9
+ - **Anthropic self-healing.** `anthropic()` can now list Anthropic's models (`listModels`), so a fallback on the direct API keeps its chain alive like OpenRouter's: a retired id, or one retiring within a week, is dropped. New opt-in `discover: { replaceRetired: true }` swaps a gone model for the newest active model of the same `line` (never another line, never deprecated, trust policy applied, image-capable for vision calls) and `checkModels()` reports every swap (`replaced`, `unreplaced`). `CatalogModel` gains `lifecycle`, `line` and `created_at`; `healProviderModels()` is exported. Python port: `Anthropic.list_models`, `Fallback(discover=Discover(replace_retired=True))`, `heal_provider_models`, and `Lodestone.check_providers()`. Tested on fake responses only: the real Models API has not been called.
10
+
11
+ ## 0.7.1
12
+
13
+ - **`omniakey()` preset** (TypeScript and Python) for [OmniaKey](https://omniakey.com), a prepaid gateway with an OpenAI-compatible text endpoint. Use it as a `fallbacks` entry. It is a preset only: a key was accepted and its model list read (98 entries, with image, video and music models mixed in and nothing to tell them apart, so name your models and do not use `discover` with it), but the first chat call was refused with `insufficient_user_quota`, so it has not been seen to answer, and the Providers page says so. Text and JSON calls only; OmniaKey's image, video and music models are outside what Lodestone does.
14
+
15
+ - **Fixed: the direct Anthropic adapter no longer sends a temperature.** Anthropic's current models reject one: Claude 4.7 and later answer 400 to a non-default value, and API version 2026-10-21 removes the parameter for every model. Lodestone's calls default to 0.2, so a direct call to a current Claude model would have failed. `anthropic()` now leaves it out and the model's own setting applies; `anthropic({ sendTemperature: true })` restores it for an older model. The Python port's `Anthropic(send_temperature=...)` works the same way. Calls through OpenRouter are unaffected. This was found by reading Anthropic's deprecations page, not from a failure: no live call to Anthropic has been made.
16
+ - README: the route example used `claude-sonnet-4-5`, which Anthropic has deprecated; it now uses `claude-sonnet-5-5`.
17
+
5
18
  ## 0.7.0
6
19
 
7
20
  Security fixes, shared state that works on every platform, per-caller limits, and runtimes beyond Node. Found in a review of 0.6.0.
package/README.md CHANGED
@@ -4,11 +4,11 @@
4
4
 
5
5
  Calls AI models so your tool keeps working when a model is withdrawn, slow or rate-limited.
6
6
 
7
- Free and cheap models come and go. A hard-coded model id breaks without warning, one slow model can use your whole time limit, and "429" can mean four different things that need four different messages. Lodestone handles those so each tool doesn't have to.
7
+ Free models come and go. A hard-coded model id breaks without warning, one slow model can use your whole time limit, and "429" can mean four different things that need four different messages. Lodestone handles those so each tool doesn't have to.
8
8
 
9
9
  - **Replaces withdrawn models by itself.** Your lists are preferences. They are checked against OpenRouter's live catalogue, and any model that has gone is swapped for a current one of the same kind.
10
10
  - **Keeps inside a time budget.** Each model gets a slice of it. A slow or failing model is skipped for the next, so a function never hits its gateway limit with an unreadable error.
11
- - **Falls back across providers.** OpenRouter's free models, then a low-cost paid model, then Anthropic or Google on their own keys.
11
+ - **Falls back across providers.** OpenRouter's free models, then a paid model, then Anthropic or Google on their own keys.
12
12
  - **Explains failures in plain words.** A daily limit, a per-minute limit, a busy upstream provider, a bad key and a slow model are different errors with different advice.
13
13
  - **Checks the answer.** `json()` skips a model whose answer isn't valid JSON, or doesn't pass your validator.
14
14
  - **Bring your own key.** A caller's key replaces the platform's and allows the paid fallback, without touching the platform's budget.
@@ -59,8 +59,8 @@ Tiers are for tools that don't care which model answers. A tool that picks its o
59
59
  await ai.complete({
60
60
  system, messages, maxTokens: 900,
61
61
  route: [
62
- { provider: 'anthropic', model: 'claude-sonnet-4-5', apiKey: customerKey },
63
- { provider: 'openrouter', model: 'anthropic/claude-sonnet-4-5', apiKey: backupKey },
62
+ { provider: 'anthropic', model: 'claude-sonnet-5-5', apiKey: customerKey },
63
+ { provider: 'openrouter', model: 'anthropic/claude-sonnet-5.5', apiKey: backupKey },
64
64
  ],
65
65
  });
66
66
  ```
@@ -78,7 +78,7 @@ Pass the customer's OpenRouter key as `apiKey` on a call. It replaces the platfo
78
78
  For each call, in order, until one answers:
79
79
 
80
80
  1. OpenRouter's free models for the tier (current ones, per the catalogue).
81
- 2. A low-cost paid OpenRouter model, only if the caller supplied their own key or you set `allowPaid`. With no credit OpenRouter refuses it and nothing is spent.
81
+ 2. A paid OpenRouter model, only if the caller supplied their own key or you set `allowPaid`. With no credit OpenRouter refuses it and nothing is spent.
82
82
  3. Each `fallbacks` provider, with its own key.
83
83
 
84
84
  When a later phase exists, the free models get a short turn (9 s in all by default), so there is always time left for it. The whole call stays inside `budgetMs` (24 s by default, under a 30 s gateway limit).
@@ -98,7 +98,7 @@ const { ok, withdrawn, shortTiers } = await ai.checkModels();
98
98
 
99
99
  ### Free models that are not worth waiting for (`paidFirst`)
100
100
 
101
- Free models can be down for hours (rate-limited, or holding a connection open for 45 seconds and never answering). On the platform's own key, `openrouter: { paidFirst: true }` tries the low-cost paid model first and the free ones only if it fails or the spend limit is reached, so a request is not slowed by free models that are not going to answer. A caller's own key is not affected: the paid models are their money, so their free models still come first. Reasoning is set per model with `openrouter({ reasoning: (model) => ... })`, because models differ: one thinks for a thousand tokens unless told not to (measured: turning it off halved scoring time, 10 s to 5 s, with the same answers), another refuses to run with reasoning off.
101
+ Free models can be down for hours (rate-limited, or holding a connection open for 45 seconds and never answering). On the platform's own key, `openrouter: { paidFirst: true }` tries the paid model first and the free ones only if it fails or the spend limit is reached, so a request is not slowed by free models that are not going to answer. A caller's own key is not affected: the paid models are their money, so their free models still come first. Reasoning is set per model with `openrouter({ reasoning: (model) => ... })`, because models differ: one thinks for a thousand tokens unless told not to (measured: turning it off halved scoring time, 10 s to 5 s, with the same answers), another refuses to run with reasoning off.
102
102
 
103
103
  ### Not waiting on a model that is down (circuit breaker)
104
104
 
@@ -123,6 +123,33 @@ const ai = createLodestone({ /* ... */, spend });
123
123
 
124
124
  `createSpendLimit` takes a **hold** for each paid attempt as it is allowed and counts holds against the limit, so a burst of concurrent requests cannot all pass the check before any has been counted. The hold is swapped for the real cost on an answer and given back (`release`) when the attempt fails. You can write your own `{ allow, record, release? }`; each receives an `id` naming the attempt.
125
125
 
126
+ #### A share for each person
127
+
128
+ ```ts
129
+ const spend = createSpendLimit({ store, dailyUsd: 20, perCaller: { dailyUsd: 0.10, monthlyUsd: 1 } });
130
+ await ai.complete({ prompt, caller: user.id });
131
+ ```
132
+
133
+ The limit above is one pool for everyone, so one heavy user can use it all up. `perCaller` gives each caller their own share of it, per day and per month. Name the caller on every call (`caller`, a user id or an IP address; only a hash is stored). A call without a caller is held to the pool only, so always pass one. Free models cost nothing and never use up a share.
134
+
135
+ When someone is over their share the call fails with `kind: 'budget'` and `detail.scope: 'caller'` (`'app'` when the whole pool ran out), and `explain()` words each case differently, so your app can say "you've used your share, add your own key to carry on". `spend.callerReport(caller)` returns what they have used and may use, for a "$0.04 of $0.10 today" display.
136
+
137
+ A person can lower their own share, never raise it: pass `spendCap: { dailyUsd: 0.05 }` on their calls (from a setting in your app), and the lower of that and `perCaller` applies.
138
+
139
+ #### A cap on a person's own key
140
+
141
+ When someone brings their own OpenRouter key, the spending is theirs, and Lodestone does not limit it. **That is the default: no limit, nothing counted.** If they want a safety net against a loop or a bug, let them set one, and pass it on their calls:
142
+
143
+ ```ts
144
+ await ai.complete({ prompt, caller: user.id, apiKey: theirKey, spendCap: { dailyUsd: 2 } });
145
+ ```
146
+
147
+ Their paid calls on that key are then counted in a ledger of their own, which never touches yours, and stop at the number they chose (`detail.scope: 'own-key'`). Your app stores the setting; Lodestone keeps the count.
148
+
149
+ **Set the limit on the key itself too.** An OpenRouter key has its own credit limit, and that is the one that really holds. Lodestone's count is best-effort: it only sees calls made through this app, and with a plain get/set store two functions answering the same person at the same moment can each take a hold the other did not see. Tell people to put a limit on the key at the provider, and treat Lodestone's cap as a second line.
150
+
151
+ `spendCap` needs a `caller` and a limit made by `createSpendLimit()`. Without them the call throws a `TypeError` instead of quietly ignoring a limit someone asked for.
152
+
126
153
  ### One person emptying the budget (`rateLimit`)
127
154
 
128
155
  A spend limit is one pot for everybody. Give each caller their own allowance, and name the caller on each call:
@@ -143,9 +170,10 @@ OpenRouter is the default, but one provider's outage should not stop your tool.
143
170
  | `requesty()` | [Requesty](https://www.requesty.ai) | Gateway to 800+ models with failover. Publishes price, lab, whether a model is trained on and how long prompts are kept. `requesty({ region: 'eu' })` keeps prompts in the EU. No free models; a 5% markup. |
144
171
  | `llmapi()` | [LLM API](https://llmapi.ai) | Gateway to 150+ chat models. Lists price; ignores its image, video and speech models. |
145
172
  | `groq()`, `cerebras()` | Groq, Cerebras | Fast inference with free tiers. Use `free: true` so they are not counted against a spend limit. |
173
+ | `omniakey()` | [OmniaKey](https://omniakey.com) | A prepaid gateway for text, image, video and music models, with an OpenAI-compatible text endpoint. Text and JSON calls only here. Its model list needs a key and mixes chat with image, video and music models, so name your models. Not yet called live: the account's first chat call was refused for insufficient quota. |
146
174
  | `openai()`, `mistral()`, `together()` | the providers themselves | |
147
175
  | `ollama()` | a local server | Free and private. |
148
- | `anthropic()`, `google()` | direct | Not OpenAI-style, so they have their own adapters. |
176
+ | `anthropic()`, `google()` | direct | Not OpenAI-style, so they have their own adapters. `anthropic()` leaves the temperature out, because Anthropic's current models reject one (Claude 4.7 and later answer 400 to a non-default value, and API version 2026-10-21 removes it); pass `anthropic({ sendTemperature: true })` for an older model that still accepts one. Model ids are retired on a schedule (`claude-sonnet-4-20250514` was retired on 15 June 2026). `anthropic()` reads Anthropic's own model list, so a retired id is dropped from the chain and one with a retirement date within a week is left out; see [Anthropic self-healing](#anthropic-self-healing). Keep a second model or provider in the chain as well, so a retired id costs one attempt, not the call. |
149
177
 
150
178
  ```ts
151
179
  fallbacks: [
@@ -160,6 +188,21 @@ Model ids are each service's own. See [examples/multi-provider.ts](examples/mult
160
188
 
161
189
  - **Outdated or shut down.** The daily `checkModels()` saves each provider's current catalogue. A model the provider no longer lists is dropped; one with an expiry date within a week (OpenRouter publishes it) is left out; one that is failing right now is skipped by the circuit breaker. Nobody edits a list.
162
190
  - **New models.** `discover: { fill }` tops a short list up from the provider's own catalogue with models within a price limit ("free" is a limit of zero) from trusted labs. For OpenRouter's free models this happens by itself.
191
+
192
+ #### Anthropic self-healing
193
+
194
+ ```ts
195
+ fallbacks: [{ provider: anthropic(), apiKey: process.env.ANTHROPIC_API_KEY!, models: ['claude-sonnet-4-20250514'], discover: { replaceRetired: true } }]
196
+ ```
197
+
198
+ Anthropic publishes where each model is in its life (`active`, `deprecated`, `retired`), when it retires, and which line it belongs to (`haiku`, `sonnet`, `opus`...). Lodestone reads that list the way it reads OpenRouter's:
199
+
200
+ - **By default** a retired id, or one that retires within a week, is dropped from the chain, so a call no longer wastes an attempt on a model that answers 404. Nothing replaces it.
201
+ - **With `discover: { replaceRetired: true }`** a named model that has gone is swapped for the newest *active* model of the **same line**, in the same place in the list. It never crosses lines (a Sonnet is never swapped for an Opus), never picks a deprecated model or one you already named, only picks a model your `trust` policy accepts, and for image calls only one that reads images. If the line has nothing current, or Anthropic no longer lists the id at all (so its line is unknown), nothing is swapped.
202
+ - **Never silent.** `checkModels()` returns, under `providers.anthropic`, `replaced: [{ from, to, reason }]` and `unreplaced: [{ id, reason, note }]`; every answer says which model produced it. Treat a `replaced` entry as a prompt to update your list.
203
+ - **Why it is opt-in.** A replacement can cost more, answer differently, or need other settings, and Anthropic's list has no prices, so the spend limit cannot estimate what a swapped-in model costs. Switch it on when staying up matters more than staying identical.
204
+
205
+ What has been checked: the logic is tested on fake responses, and the field names follow Anthropic's Models API reference. No call to the real endpoint has been made, and the documentation's own example is inconsistent about the auth header and API version (Lodestone sends `x-api-key` and `2023-06-01`, as for messages). If the list cannot be read, the models you named are used as they are.
163
206
  - **Not a longer wait.** Discovery never lengthens what a request walks. A background pool (up to 8 per tier) is ranked, the models that are down are filtered out, and a request tries at most three. When one fails, the next best takes its place for later requests, so adding models adds choice, not delay.
164
207
  - **Safe by default.** A discovered model is used only if its publisher is on the trust list; `trust` also takes a block list, refuses models a provider says are trained on or that keep prompts too long, and `private: true` applies the same to one call (see below). There is no vulnerability database for hosted models, so none is claimed: see [SECURITY.md](SECURITY.md) for exactly what is and is not checked.
165
208
 
package/dist/catalog.d.ts CHANGED
@@ -24,11 +24,18 @@ export interface CatalogModel {
24
24
  data_retention_days?: number;
25
25
  /** Zero price, as the provider says it. */
26
26
  free?: boolean;
27
+ /** Where the provider says the model is in its life: Anthropic publishes this. A retired model no longer answers. */
28
+ lifecycle?: 'active' | 'deprecated' | 'retired';
29
+ /** The family the provider puts the model in (Anthropic: haiku, sonnet, opus...). Models of one line can stand in for each other. */
30
+ line?: string;
31
+ /** When the model was released, as the provider says it. */
32
+ created_at?: string;
27
33
  }
28
34
  export declare const ROUTER = "openrouter/free";
29
35
  /** US dollars per million tokens, or undefined if the provider gives no price. */
30
36
  export declare const promptPerMTok: (m: CatalogModel) => number | undefined;
31
37
  export declare const completionPerMTok: (m: CatalogModel) => number | undefined;
38
+ export declare const takesImages: (m: CatalogModel) => boolean;
32
39
  /** Text models fit for the tier. Free ones, unless `price` allows more (then at most that, per million tokens). */
33
40
  export declare function eligible(catalog: CatalogModel[], tier: Tier, price?: {
34
41
  promptUsdPerMTok: number;
package/dist/catalog.js CHANGED
@@ -7,7 +7,7 @@ const isFree = (m) => m.id.endsWith(':free') || (Number(m.pricing?.prompt) === 0
7
7
  export const promptPerMTok = (m) => { const n = Number(m.pricing?.prompt); return m.pricing?.prompt !== undefined && m.pricing?.prompt !== '' && Number.isFinite(n) ? n * 1e6 : undefined; };
8
8
  export const completionPerMTok = (m) => { const n = Number(m.pricing?.completion); return m.pricing?.completion !== undefined && m.pricing?.completion !== '' && Number.isFinite(n) ? n * 1e6 : undefined; };
9
9
  const textOut = (m) => (m.architecture?.output_modalities ?? ['text']).join() === 'text';
10
- const takesImages = (m) => (m.architecture?.input_modalities ?? []).includes('image');
10
+ export const takesImages = (m) => (m.architecture?.input_modalities ?? []).includes('image');
11
11
  // Safety classifiers, audio and music models, code-only models and embedders are not for writing or reading text.
12
12
  const EXCLUDE = /safety|guard|moderation|lyria|embed|rerank|tts|whisper|code|coder|preview-audio|ocr/i;
13
13
  /** "550b" or "26b" in an id: bigger is the stronger pick. */
@@ -19,6 +19,28 @@ export interface ResolveOptions {
19
19
  fill?: FillOptions;
20
20
  /** The most models to return. Default 3. */
21
21
  max?: number;
22
+ /** When a model the app named has gone (retired, no longer listed, or about to be), put the provider's newest current model of the same line in its place. Off by default: a replacement can cost more, answer differently, or need other settings, so it is something you switch on, and it is always reported (see healProviderModels). Needs a provider that publishes a `line` for its models (Anthropic does). */
23
+ replaceRetired?: boolean;
22
24
  now: number;
23
25
  }
24
- export declare function resolveProviderModels(curated: string[], catalog: CatalogModel[] | null, o: ResolveOptions): string[];
26
+ export interface Healed {
27
+ models: string[];
28
+ /** A named model that was swapped for the newest current model of its line. */
29
+ replaced: {
30
+ from: string;
31
+ to: string;
32
+ reason: GoneReason;
33
+ }[];
34
+ /** A named model that has gone and could not be replaced, with why. */
35
+ unreplaced: {
36
+ id: string;
37
+ reason: GoneReason;
38
+ note?: string;
39
+ }[];
40
+ }
41
+ export type GoneReason = 'withdrawn' | 'retired' | 'expiring';
42
+ /** The list to use, and what was changed in it: a gone model is dropped, and with `replaceRetired` swapped
43
+ * for the newest model of the same line (never one the app already named, never a deprecated or retired
44
+ * one, and only one the trust policy accepts and that can read images when the tier needs it). */
45
+ export declare function healProviderModels(curated: string[], catalog: CatalogModel[] | null, o: ResolveOptions): Healed;
46
+ export declare const resolveProviderModels: (curated: string[], catalog: CatalogModel[] | null, o: ResolveOptions) => string[];
package/dist/discover.js CHANGED
@@ -1,20 +1,63 @@
1
- import { eligible, rank } from './catalog.js';
1
+ import { eligible, rank, takesImages } from './catalog.js';
2
2
  import { vet } from './trust.js';
3
- export function resolveProviderModels(curated, catalog, o) {
3
+ /** Why a named model cannot be used any more, or undefined if it still can. A model the trust policy refuses for another reason is dropped by the caller, not replaced. */
4
+ function gone(id, byId, o) {
5
+ const m = byId.get(id);
6
+ if (!m)
7
+ return 'withdrawn';
8
+ if (m.lifecycle === 'retired')
9
+ return 'retired';
10
+ const v = vet(m, o.policy, { discovered: false, strict: o.strict, now: o.now });
11
+ return v.ok ? undefined : v.reason === 'expiring' ? 'expiring' : 'refused';
12
+ }
13
+ const released = (m) => { const t = Date.parse(m.created_at ?? ''); return Number.isFinite(t) ? t : 0; };
14
+ /** The list to use, and what was changed in it: a gone model is dropped, and with `replaceRetired` swapped
15
+ * for the newest model of the same line (never one the app already named, never a deprecated or retired
16
+ * one, and only one the trust policy accepts and that can read images when the tier needs it). */
17
+ export function healProviderModels(curated, catalog, o) {
4
18
  if (!catalog || catalog.length === 0)
5
- return curated; // nothing known about what exists: use what the app named
19
+ return { models: curated, replaced: [], unreplaced: [] }; // nothing known about what exists: use what the app named
6
20
  const max = o.max ?? 3;
7
21
  const byId = new Map(catalog.map((m) => [m.id, m]));
8
- const kept = curated.filter((id) => { const m = byId.get(id); return m !== undefined && vet(m, o.policy, { discovered: false, strict: o.strict, now: o.now }).ok; });
22
+ const kept = [];
23
+ const replaced = [];
24
+ const unreplaced = [];
25
+ for (const id of curated) {
26
+ const why = gone(id, byId, o);
27
+ if (why === undefined) {
28
+ kept.push(id);
29
+ continue;
30
+ }
31
+ if (why === 'refused')
32
+ continue;
33
+ if (!o.replaceRetired)
34
+ continue;
35
+ const line = byId.get(id)?.line;
36
+ if (!line) {
37
+ unreplaced.push({ id, reason: why, note: byId.has(id) ? 'the provider names no line for it' : 'no longer listed, so its line is unknown' });
38
+ continue;
39
+ }
40
+ const to = catalog
41
+ .filter((m) => m.line === line && (m.lifecycle === undefined || m.lifecycle === 'active') && !curated.includes(m.id) && !kept.includes(m.id) && !replaced.some((r) => r.to === m.id)
42
+ && (o.tier !== 'vision' || takesImages(m)) && vet(m, o.policy, { discovered: true, strict: o.strict, now: o.now }).ok)
43
+ .sort((a, b) => released(b) - released(a) || a.id.localeCompare(b.id))[0];
44
+ if (!to) {
45
+ unreplaced.push({ id, reason: why, note: `no current ${line} model to take its place` });
46
+ continue;
47
+ }
48
+ kept.push(to.id);
49
+ replaced.push({ from: id, to: to.id, reason: why });
50
+ }
9
51
  if (!o.fill || kept.length >= max)
10
- return kept;
52
+ return { models: kept, replaced, unreplaced };
11
53
  const fill = eligible(catalog, o.tier, { promptUsdPerMTok: o.fill.maxPromptUsdPerMTok, completionUsdPerMTok: o.fill.maxCompletionUsdPerMTok })
12
- .filter((m) => !kept.includes(m.id) && vet(m, o.policy, { discovered: true, strict: o.strict, now: o.now }).ok)
54
+ .filter((m) => !kept.includes(m.id) && m.lifecycle !== 'retired' && vet(m, o.policy, { discovered: true, strict: o.strict, now: o.now }).ok)
13
55
  .sort((a, b) => rank(o.tier, a, b));
14
56
  for (const m of fill) {
15
57
  if (kept.length >= max)
16
58
  break;
17
59
  kept.push(m.id);
18
60
  }
19
- return kept;
61
+ return { models: kept, replaced, unreplaced };
20
62
  }
63
+ export const resolveProviderModels = (curated, catalog, o) => healProviderModels(curated, catalog, o).models;
package/dist/errors.d.ts CHANGED
@@ -26,7 +26,7 @@ export type FailureKind =
26
26
  | 'bad_output'
27
27
  /** The caller stopped the call (it passed a signal that was aborted). */
28
28
  | 'aborted'
29
- /** The platform's spend limit has been reached, so no paid model was used. */
29
+ /** A spend limit has been reached, so no paid model was used. `detail.scope` says whose: the app's pool, this caller's share of it, or the cap they set on their own key. */
30
30
  | 'budget'
31
31
  /** This caller has made too many calls in the time allowed (see `rateLimit`). `detail.retryAfterMs` says when to try again. */
32
32
  | 'throttled'
@@ -42,7 +42,8 @@ export declare class LodestoneError extends Error {
42
42
  ownKey?: boolean;
43
43
  needsCredit?: boolean;
44
44
  providerMessage?: string;
45
- retryAfterMs?: number;
45
+ retryAfterMs?: number; /** For 'budget': which limit stopped the call. */
46
+ scope?: 'app' | 'caller' | 'own-key';
46
47
  };
47
48
  constructor(kind: FailureKind, detail?: {
48
49
  status?: number;
@@ -50,6 +51,7 @@ export declare class LodestoneError extends Error {
50
51
  ownKey?: boolean;
51
52
  needsCredit?: boolean;
52
53
  providerMessage?: string;
53
- retryAfterMs?: number;
54
+ retryAfterMs?: number; /** For 'budget': which limit stopped the call. */
55
+ scope?: 'app' | 'caller' | 'own-key';
54
56
  });
55
57
  }
package/dist/explain.js CHANGED
@@ -45,6 +45,10 @@ export function explain(err, opts = {}) {
45
45
  case 'aborted':
46
46
  return 'Cancelled.';
47
47
  case 'budget':
48
+ if (detail.scope === 'caller')
49
+ return `You've used your share of ${product}'s AI for now, and the free models are busy. Try again later${opts.settingsHint ? `, or add your own OpenRouter key ${opts.settingsHint} to keep going` : ''}.`;
50
+ if (detail.scope === 'own-key')
51
+ return `Your own key has reached the spending limit you set${opts.settingsHint ? ` ${opts.settingsHint}` : ''}, and the free models are busy. Raise the limit or try again later.`;
48
52
  return `${product}'s AI budget for now is used up, and the free models are busy. Try again later${opts.settingsHint ? `, or add your own OpenRouter key ${opts.settingsHint.replace(/^in /, 'in ')} to keep going` : ''}.`;
49
53
  case 'throttled': {
50
54
  const mins = Math.max(1, Math.ceil((detail.retryAfterMs ?? 60_000) / 60_000));
package/dist/index.d.ts CHANGED
@@ -1,5 +1,5 @@
1
1
  export { createLodestone } from './lodestone.js';
2
- export type { Lodestone, LodestoneConfig, CompleteOptions, CompleteResult, SafeParser, Profile, FallbackConfig, AttemptEvent, RouteStep, SpendControl, SpendInfo, CheckResult } from './lodestone.js';
2
+ export type { Lodestone, LodestoneConfig, CompleteOptions, CompleteResult, SafeParser, Profile, FallbackConfig, AttemptEvent, RouteStep, SpendControl, SpendInfo, SpendCap, SpendScope, CheckResult } from './lodestone.js';
3
3
  export { LodestoneError, ProviderError, messageFrom } from './errors.js';
4
4
  export type { FailureKind } from './errors.js';
5
5
  export { explain } from './explain.js';
@@ -16,7 +16,7 @@ export type { SpendLimitOptions, SpendReport } from './spend.js';
16
16
  export { createRateLimiter } from './limiter.js';
17
17
  export type { RateLimiter, RateLimiterOptions, RateLimitResult } from './limiter.js';
18
18
  export { openrouter } from './providers/openrouter.js';
19
- export { anthropic } from './providers/anthropic.js';
19
+ export { anthropic, normalizeAnthropic } from './providers/anthropic.js';
20
20
  export { google } from './providers/google.js';
21
21
  export { mock } from './providers/mock.js';
22
22
  export type { Tier, Message, ContentPart, Provider, ProviderRequest, ProviderReply, KeyValueStore, AtomicStore, Usage } from './types.js';
@@ -28,12 +28,12 @@ export { CANDIDATES_SCHEMA } from './candidates.js';
28
28
  export type { Candidates, Candidate } from './candidates.js';
29
29
  export { createHealth } from './health.js';
30
30
  export type { Health, HealthOptions, ModelHealth } from './health.js';
31
- export { openaiCompatible, requesty, llmapi, groq, cerebras, openai, mistral, together, ollama, normalizeRequesty, normalizeLlmapi, normalizeGroq } from './providers/openaiCompatible.js';
31
+ export { openaiCompatible, requesty, llmapi, groq, cerebras, omniakey, openai, mistral, together, ollama, normalizeRequesty, normalizeLlmapi, normalizeGroq } from './providers/openaiCompatible.js';
32
32
  export type { OpenAICompatibleOptions, PresetOptions } from './providers/openaiCompatible.js';
33
33
  export { vet, publisherOf, DEFAULT_PUBLISHERS } from './trust.js';
34
34
  export type { TrustPolicy, Verdict } from './trust.js';
35
- export { resolveProviderModels } from './discover.js';
36
- export type { FillOptions } from './discover.js';
35
+ export { resolveProviderModels, healProviderModels } from './discover.js';
36
+ export type { FillOptions, Healed, GoneReason } from './discover.js';
37
37
  export type { CatalogModel as ModelInfo } from './catalog.js';
38
38
  export { netlifyBlobsStore, cloudflareKvStore, upstashStore, denoKvStore, safeKey } from './stores/index.js';
39
39
  export type { NetlifyBlobsLike, KVNamespaceLike, UpstashOptions, DenoKvLike } from './stores/index.js';
package/dist/index.js CHANGED
@@ -9,14 +9,14 @@ export { memoryStore } from './store.js';
9
9
  export { createSpendLimit } from './spend.js';
10
10
  export { createRateLimiter } from './limiter.js';
11
11
  export { openrouter } from './providers/openrouter.js';
12
- export { anthropic } from './providers/anthropic.js';
12
+ export { anthropic, normalizeAnthropic } from './providers/anthropic.js';
13
13
  export { google } from './providers/google.js';
14
14
  export { mock } from './providers/mock.js';
15
15
  export { SCORECARD_SCHEMA, ScorecardError, parseScorecard, fetchScorecard, rankModels, scoreFor, baseModelId } from './scorecard.js';
16
16
  export { createStats } from './stats.js';
17
17
  export { CANDIDATES_SCHEMA } from './candidates.js';
18
18
  export { createHealth } from './health.js';
19
- export { openaiCompatible, requesty, llmapi, groq, cerebras, openai, mistral, together, ollama, normalizeRequesty, normalizeLlmapi, normalizeGroq } from './providers/openaiCompatible.js';
19
+ export { openaiCompatible, requesty, llmapi, groq, cerebras, omniakey, openai, mistral, together, ollama, normalizeRequesty, normalizeLlmapi, normalizeGroq } from './providers/openaiCompatible.js';
20
20
  export { vet, publisherOf, DEFAULT_PUBLISHERS } from './trust.js';
21
- export { resolveProviderModels } from './discover.js';
21
+ export { resolveProviderModels, healProviderModels } from './discover.js';
22
22
  export { netlifyBlobsStore, cloudflareKvStore, upstashStore, denoKvStore, safeKey } from './stores/index.js';
@@ -21,10 +21,11 @@ export interface FallbackConfig {
21
21
  models: string[];
22
22
  /** This provider costs nothing (a free tier, a local server): its calls are not counted against, or stopped by, the spend limit. */
23
23
  free?: boolean;
24
- /** Keep this provider's list alive from its own catalogue (needs a provider that can list its models): models it no longer offers, or that are retiring or refused by `trust`, are dropped, and with `fill` a short list is topped up with trusted models within a price limit, never above `max` (default 3). */
24
+ /** Keep this provider's list alive from its own catalogue (needs a provider that can list its models): models it no longer offers, or that are retiring or refused by `trust`, are dropped, and with `fill` a short list is topped up with trusted models within a price limit, never above `max` (default 3). With `replaceRetired`, a model you named that has gone (retired, no longer listed, or about to be) is swapped for the provider's newest current model of the same line instead of just dropped (Anthropic: another Sonnet for a retired Sonnet). It is off by default because a replacement can cost more or behave differently, and `checkModels()` reports every swap. */
25
25
  discover?: {
26
26
  fill?: FillOptions;
27
27
  max?: number;
28
+ replaceRetired?: boolean;
28
29
  };
29
30
  /** Also use this provider when the caller supplied their own OpenRouter key. Default false: a fallback has the platform's key, and a caller's own key must never lead to the platform's money being spent. */
30
31
  allowWithCallerKey?: boolean;
@@ -48,7 +49,20 @@ export interface SpendInfo {
48
49
  provider: string;
49
50
  model: string;
50
51
  phase: 'paid' | 'fallback';
52
+ /** Who is making the call (`caller` on the call), when it says. */
53
+ caller?: string;
54
+ /** The call is on the caller's own key: their money. Only asked about when they set a `spendCap`. */
55
+ ownKey?: boolean;
56
+ /** The limit the caller chose for themselves (`spendCap` on the call). */
57
+ cap?: SpendCap;
58
+ }
59
+ /** A limit in US dollars. Both parts are optional. */
60
+ export interface SpendCap {
61
+ dailyUsd?: number;
62
+ monthlyUsd?: number;
51
63
  }
64
+ /** Which limit stopped a call: the app's own pool, this caller's share of it, or the cap the caller set on their own key. */
65
+ export type SpendScope = 'app' | 'caller' | 'own-key';
52
66
  export interface SpendControl {
53
67
  /** Asked before each such attempt. Return false once the limit is reached and that model is not used. Many calls can be in flight at once, so a limit that only compares a total with a number lets a burst through: take a hold under `info.id` here (createSpendLimit does). */
54
68
  allow(info: SpendInfo): boolean | Promise<boolean>;
@@ -59,6 +73,10 @@ export interface SpendControl {
59
73
  }): void | Promise<void>;
60
74
  /** Told when an attempt that allow() let through did not answer, so a hold taken in allow() can be given back. */
61
75
  release?(info: SpendInfo): void | Promise<void>;
76
+ /** True if this limit keeps a share for each `caller` and a cap of the caller's own on their own key (createSpendLimit does). A `spendCap` on a call needs it. A limit without it is never asked about a caller's own key, because that money is theirs. */
77
+ handlesOwnKey?: boolean;
78
+ /** Which limit refused the attempt that `allow()` just turned down, so the error can say whether it was the app's pool or the caller's share. */
79
+ refusal?(id: string): SpendScope | undefined;
62
80
  }
63
81
  export interface LodestoneConfig {
64
82
  openrouter?: {
@@ -153,8 +171,10 @@ export interface CompleteOptions {
153
171
  accept?: (text: string) => boolean;
154
172
  /** The prompt must not be kept or trained on (a CV, say). OpenRouter is told to use only providers that do not. Another provider is used only if its own model list says the model is not trained on (Requesty does), or you vouched for it with `privacy: 'no-training'` on the fallback. Many free models cannot meet this, and the paid ones are used instead. A private call is never saved to the cache, whatever `cacheKey` says. A call with an explicit `route` is your own choice of provider: Lodestone passes the request to it as told. */
155
173
  private?: boolean;
156
- /** Who is making this call (a user id, an IP address), for `rateLimit`. Only a hash is stored. */
174
+ /** Who is making this call (a user id, an IP address), for `rateLimit` and for a per-caller `spend` allowance (`perCaller`). Only a hash is stored. */
157
175
  caller?: string;
176
+ /** A limit the person chose for themselves, in dollars. On the app's money it can only lower their allowance (`perCaller`), never raise it. On their own key (`apiKey`) it is a cap on what that key spends through this app, off unless you pass one. Needs `caller` and a limit made by createSpendLimit(). The provider's own limit on the key is the one that really holds: Lodestone's count is best-effort. */
177
+ spendCap?: SpendCap;
158
178
  /** Stops the call when the caller no longer wants the answer (the person left the page). */
159
179
  signal?: AbortSignal;
160
180
  /** Same key, same answer: the answer is saved and reused. Only for pure, repeatable tasks, and the key must stand for the whole input (use a hash of it). As a safeguard an answer is also tied to a fingerprint of the system prompt, messages and tier, so a key that is too coarse gives a miss, never another person's answer. Ignored when `private` is set. */
@@ -181,7 +201,17 @@ export interface CheckResult {
181
201
  providers?: Record<string, {
182
202
  models: number;
183
203
  withdrawn: string[];
184
- expiring: string[];
204
+ expiring: string[]; /** Models swapped for a current one of the same line (needs `discover.replaceRetired`). Fix your list: this is Lodestone telling you, not asking. */
205
+ replaced?: {
206
+ from: string;
207
+ to: string;
208
+ reason: string;
209
+ }[]; /** Named models that have gone and found no replacement. */
210
+ unreplaced?: {
211
+ id: string;
212
+ reason: string;
213
+ note?: string;
214
+ }[];
185
215
  error?: string;
186
216
  }>;
187
217
  }
package/dist/lodestone.js CHANGED
@@ -1,5 +1,5 @@
1
1
  import { createCatalog, pickModels, ROUTER } from './catalog.js';
2
- import { resolveProviderModels } from './discover.js';
2
+ import { healProviderModels, resolveProviderModels } from './discover.js';
3
3
  import { CANDIDATES_SCHEMA } from './candidates.js';
4
4
  import { LodestoneError, messageFrom, ProviderError } from './errors.js';
5
5
  import { tryParseJson } from './json.js';
@@ -70,7 +70,7 @@ export function createLodestone(config = {}) {
70
70
  if (strict && !vouched && !listed?.length)
71
71
  return []; // no list, so nothing to check the models against
72
72
  const p = strict && !vouched ? { ...policy, requireNoTraining: true } : policy;
73
- return resolveProviderModels(fb.models, listed, { tier, policy: p, strict: strict && !vouched, fill: fb.discover?.fill, max: fb.discover?.max, now: now() });
73
+ return resolveProviderModels(fb.models, listed, { tier, policy: p, strict: strict && !vouched, fill: fb.discover?.fill, max: fb.discover?.max, replaceRetired: fb.discover?.replaceRetired, now: now() });
74
74
  }
75
75
  catch {
76
76
  return strict && !vouched ? [] : fb.models;
@@ -100,6 +100,16 @@ export function createLodestone(config = {}) {
100
100
  return Number.isFinite(inP) && Number.isFinite(outP) ? usage.inputTokens * inP + usage.outputTokens * outP : undefined;
101
101
  }
102
102
  async function complete(opts) {
103
+ // A limit a person asked for must be kept or refused, never quietly ignored.
104
+ if (opts.spendCap) {
105
+ const c = opts.spendCap;
106
+ if (!opts.caller)
107
+ throw new TypeError('spendCap needs a caller: it is counted per caller');
108
+ if (!config.spend?.handlesOwnKey)
109
+ throw new TypeError('spendCap needs a spend limit made by createSpendLimit()');
110
+ if ((c.dailyUsd !== undefined && !(Number.isFinite(c.dailyUsd) && c.dailyUsd >= 0)) || (c.monthlyUsd !== undefined && !(Number.isFinite(c.monthlyUsd) && c.monthlyUsd >= 0)))
111
+ throw new TypeError('spendCap must be zero or more dollars');
112
+ }
103
113
  const messages = opts.messages ?? (opts.prompt !== undefined ? [{ role: 'user', content: opts.prompt }] : []);
104
114
  if (messages.length === 0)
105
115
  throw new Error('complete() needs a prompt or messages');
@@ -152,7 +162,7 @@ export function createLodestone(config = {}) {
152
162
  let rateLimited = null;
153
163
  let authFailed = false;
154
164
  let needsCredit = false;
155
- let spendBlocked = false;
165
+ let spendScope = null;
156
166
  const callerKey = Boolean(opts.apiKey);
157
167
  /** Drops the models the breaker has down, saying so. */
158
168
  const healthy = async (provider, list, phase) => {
@@ -170,21 +180,24 @@ export function createLodestone(config = {}) {
170
180
  /** May the platform's money be spent on this attempt? A caller's own key is their money, not ours. */
171
181
  let held = null;
172
182
  const mayUse = async (provider, model, phase, free = false) => {
173
- if (callerKey || free)
183
+ if (free)
184
+ return true;
185
+ // A caller's own key is their money. It is only counted when they set a cap of their own.
186
+ if (callerKey && !(phase === 'paid' && opts.spendCap && config.spend))
174
187
  return true;
175
188
  // A caller over their allowance (with 'free-only') costs the platform nothing.
176
- if (throttled) {
189
+ if (throttled && !callerKey) {
177
190
  config.onAttempt?.({ provider: provider.name, model, phase, outcome: 'skipped', ms: 0 });
178
191
  return false;
179
192
  }
180
193
  if (!config.spend)
181
194
  return true;
182
- const info = { id: crypto.randomUUID(), provider: provider.name, model, phase };
195
+ const info = { id: crypto.randomUUID(), provider: provider.name, model, phase, ...(opts.caller ? { caller: opts.caller } : {}), ...(callerKey ? { ownKey: true } : {}), ...(opts.spendCap ? { cap: opts.spendCap } : {}) };
183
196
  if (await config.spend.allow(info)) {
184
197
  held = info;
185
198
  return true;
186
199
  }
187
- spendBlocked = true;
200
+ spendScope = config.spend.refusal?.(info.id) ?? (callerKey ? 'own-key' : 'app');
188
201
  config.onAttempt?.({ provider: provider.name, model, phase, outcome: 'skipped', ms: 0 });
189
202
  return false;
190
203
  };
@@ -239,14 +252,14 @@ export function createLodestone(config = {}) {
239
252
  };
240
253
  const finish = async (provider, model, reply, phase, free = false) => {
241
254
  const result = { text: reply.text, provider: provider.name, model, usage: reply.usage };
242
- if (phase && !callerKey && !free && config.spend) {
255
+ if (phase && !free && config.spend && (!callerKey || (phase === 'paid' && opts.spendCap))) {
243
256
  const costUsd = reply.usage?.costUsd ?? (await estimateCost(model, reply.usage));
244
257
  const hold = held;
245
258
  const id = hold && hold.provider === provider.name && hold.model === model ? hold.id : crypto.randomUUID();
246
259
  held = null;
247
260
  // A ledger that fails must never fail an answer the person is waiting for.
248
261
  try {
249
- await config.spend.record({ id, provider: provider.name, model, phase, usage: reply.usage, costUsd });
262
+ await config.spend.record({ id, provider: provider.name, model, phase, usage: reply.usage, costUsd, ...(opts.caller ? { caller: opts.caller } : {}), ...(callerKey ? { ownKey: true } : {}), ...(opts.spendCap ? { cap: opts.spendCap } : {}) });
250
263
  }
251
264
  catch { /* not counted */ }
252
265
  }
@@ -385,8 +398,8 @@ export function createLodestone(config = {}) {
385
398
  const ownKey = Boolean(opts.apiKey);
386
399
  if (throttled)
387
400
  throw new LodestoneError('throttled', throttled);
388
- if (spendBlocked)
389
- throw new LodestoneError('budget');
401
+ if (spendScope)
402
+ throw new LodestoneError('budget', { scope: spendScope });
390
403
  if (!attempted)
391
404
  throw new LodestoneError('not_configured');
392
405
  const lastFail = last;
@@ -428,6 +441,11 @@ export function createLodestone(config = {}) {
428
441
  const done = parse(result.text);
429
442
  return { ...result, data: (done.ok ? done.data : tryParseJson(result.text)) };
430
443
  }
444
+ /** What the daily check says about a provider's own list: swaps made and models that could not be swapped. */
445
+ function healed(fb, listed) {
446
+ const h = healProviderModels(fb.models, listed, { tier: 'strong', policy, replaceRetired: fb.discover?.replaceRetired, now: now() });
447
+ return { ...(h.replaced.length ? { replaced: h.replaced } : {}), ...(h.unreplaced.length ? { unreplaced: h.unreplaced } : {}) };
448
+ }
431
449
  /** Fetches and saves OpenRouter's current catalogue. Run it daily. */
432
450
  const refreshCatalog = () => catalog.refresh();
433
451
  /** The daily health check: saves the catalogue, and says which preferred models have
@@ -458,6 +476,7 @@ export function createLodestone(config = {}) {
458
476
  models: listed.length,
459
477
  withdrawn: fb.models.filter((m) => !have.has(m)),
460
478
  expiring: fb.models.filter((m) => { const d = have.get(m)?.expiration_date; return d !== undefined && Date.parse(d) - now() < horizon; }),
479
+ ...healed(fb, listed),
461
480
  };
462
481
  }
463
482
  return { ok: shortTiers.length === 0, catalogSize: models.length, withdrawn, expiring, shortTiers, ...(Object.keys(providers).length ? { providers } : {}) };
@@ -1,6 +1,11 @@
1
+ import type { CatalogModel } from '../catalog.js';
1
2
  import type { Provider } from '../types.js';
2
3
  export interface AnthropicOptions {
3
4
  fetch?: typeof fetch;
5
+ /** Send the request's temperature. Default false. Anthropic's current models reject a temperature: Claude 4.7 and later answer 400 to a non-default one, and API version 2026-10-21 removes the parameter for every model. Lodestone's calls default to 0.2, so by default it is left out and the model's own setting applies. Turn this on only for an older model that still accepts one. */
6
+ sendTemperature?: boolean;
4
7
  }
8
+ /** One row of Anthropic's model list as a CatalogModel. The retirement date becomes `expiration_date`, so a model about to go is left out like any other. */
9
+ export declare function normalizeAnthropic(r: unknown): CatalogModel | null;
5
10
  /** Anthropic's own API, for use as a fallback with a separate key from OpenRouter. */
6
11
  export declare function anthropic(opts?: AnthropicOptions): Provider;
@@ -1,4 +1,28 @@
1
1
  import { ProviderError } from '../errors.js';
2
+ import { isRecord } from '../util.js';
3
+ const VERSION = '2023-06-01';
4
+ const LIFECYCLES = ['active', 'deprecated', 'retired'];
5
+ const string = (v) => (typeof v === 'string' && v ? v : undefined);
6
+ /** One row of Anthropic's model list as a CatalogModel. The retirement date becomes `expiration_date`, so a model about to go is left out like any other. */
7
+ export function normalizeAnthropic(r) {
8
+ if (!isRecord(r) || typeof r.id !== 'string' || !r.id)
9
+ return null;
10
+ const caps = isRecord(r.capabilities) ? r.capabilities : {};
11
+ const image = isRecord(caps.image_input) && caps.image_input.supported === true;
12
+ const lifecycle = LIFECYCLES.find((l) => l === r.lifecycle);
13
+ const retires = lifecycle === 'retired' ? undefined : string(r.retires_at);
14
+ const context = typeof r.max_input_tokens === 'number' && r.max_input_tokens > 0 ? r.max_input_tokens : undefined;
15
+ return {
16
+ id: r.id,
17
+ publisher: 'anthropic',
18
+ architecture: { input_modalities: image ? ['text', 'image'] : ['text'], output_modalities: ['text'] },
19
+ ...(context ? { context_length: context } : {}),
20
+ ...(lifecycle ? { lifecycle } : {}),
21
+ ...(string(r.line) ? { line: string(r.line) } : {}),
22
+ ...(string(r.created_at) ? { created_at: string(r.created_at) } : {}),
23
+ ...(retires ? { expiration_date: retires } : {}),
24
+ };
25
+ }
2
26
  const IMAGE = /^data:(image\/(?:jpeg|png|gif|webp));base64,(.+)$/s;
3
27
  function part(p) {
4
28
  if (p.type === 'text')
@@ -12,17 +36,33 @@ const wire = (m) => ({ role: m.role, content: typeof m.content === 'string' ? m.
12
36
  /** Anthropic's own API, for use as a fallback with a separate key from OpenRouter. */
13
37
  export function anthropic(opts = {}) {
14
38
  const f = opts.fetch ?? fetch;
39
+ /** Every page of the list for one query, newest first; null if the list cannot be read. */
40
+ async function pages(query, apiKey, signal) {
41
+ const rows = [];
42
+ let after = '';
43
+ for (let page = 0; page < 5; page++) {
44
+ const res = await f(`https://api.anthropic.com/v1/models?limit=1000${query}${after}`, { headers: { 'x-api-key': apiKey, 'anthropic-version': VERSION }, signal });
45
+ if (!res.ok)
46
+ return null;
47
+ const body = (await res.json());
48
+ rows.push(...(body.data ?? []));
49
+ if (!body.has_more || !body.last_id)
50
+ break;
51
+ after = `&after_id=${encodeURIComponent(body.last_id)}`;
52
+ }
53
+ return rows;
54
+ }
15
55
  return {
16
56
  name: 'anthropic',
17
57
  vision: true,
18
58
  async call(req) {
19
59
  const res = await f('https://api.anthropic.com/v1/messages', {
20
60
  method: 'POST',
21
- headers: { 'Content-Type': 'application/json', 'x-api-key': req.apiKey, 'anthropic-version': '2023-06-01' },
61
+ headers: { 'Content-Type': 'application/json', 'x-api-key': req.apiKey, 'anthropic-version': VERSION },
22
62
  body: JSON.stringify({
23
63
  model: req.model,
24
64
  max_tokens: req.maxTokens,
25
- ...(req.temperature === undefined ? {} : { temperature: req.temperature }),
65
+ ...(opts.sendTemperature && req.temperature !== undefined ? { temperature: req.temperature } : {}),
26
66
  ...(req.system ? { system: req.system } : {}),
27
67
  messages: req.messages.map(wire),
28
68
  }),
@@ -36,5 +76,20 @@ export function anthropic(opts = {}) {
36
76
  usage: data.usage ? { inputTokens: data.usage.input_tokens ?? 0, outputTokens: data.usage.output_tokens ?? 0 } : undefined,
37
77
  };
38
78
  },
79
+ /** What Anthropic offers now. The default list has the active and deprecated models, so a retired id is simply missing; a second read asks for the retired ones too, so Lodestone can tell which line a retired id belonged to. If that second read fails, the first still stands. */
80
+ async listModels(apiKey, signal) {
81
+ const s = signal ?? AbortSignal.timeout(10_000);
82
+ const live = await pages('', apiKey, s);
83
+ if (!live)
84
+ return null;
85
+ const retired = await pages('&lifecycle=retired', apiKey, s).catch(() => null);
86
+ const out = new Map();
87
+ for (const row of [...live, ...(retired ?? [])]) {
88
+ const m = normalizeAnthropic(row);
89
+ if (m && !out.has(m.id))
90
+ out.set(m.id, m);
91
+ }
92
+ return out.size ? [...out.values()] : null;
93
+ },
39
94
  };
40
95
  }
@@ -38,6 +38,8 @@ export declare const requesty: (o?: PresetOptions & {
38
38
  export declare const llmapi: (o?: PresetOptions) => Provider;
39
39
  /** https://groq.com: fast inference with a free developer tier. */
40
40
  export declare const groq: (o?: PresetOptions) => Provider;
41
+ /** https://omniakey.com: a prepaid gateway to text, image, video and music models, with an OpenAI-compatible text endpoint (api.omniakey.com/v1). Its model list needs a key (401 without one), and its site says it does not switch a request to a different model by itself, which is what Lodestone does on top of it. Not yet called live. */
42
+ export declare const omniakey: (o?: PresetOptions) => Provider;
41
43
  /** https://cerebras.ai: fast inference with a free tier. */
42
44
  export declare const cerebras: (o?: PresetOptions) => Provider;
43
45
  export declare const openai: (o?: PresetOptions) => Provider;
@@ -89,6 +89,8 @@ export const requesty = (o = {}) => openaiCompatible({ name: 'requesty', baseUrl
89
89
  export const llmapi = (o = {}) => openaiCompatible({ name: 'llmapi', baseUrl: 'https://api.llmapi.ai/v1', normalize: normalizeLlmapi, ...o });
90
90
  /** https://groq.com: fast inference with a free developer tier. */
91
91
  export const groq = (o = {}) => openaiCompatible({ name: 'groq', baseUrl: 'https://api.groq.com/openai/v1', normalize: normalizeGroq, ...o });
92
+ /** https://omniakey.com: a prepaid gateway to text, image, video and music models, with an OpenAI-compatible text endpoint (api.omniakey.com/v1). Its model list needs a key (401 without one), and its site says it does not switch a request to a different model by itself, which is what Lodestone does on top of it. Not yet called live. */
93
+ export const omniakey = (o = {}) => openaiCompatible({ name: 'omniakey', baseUrl: 'https://api.omniakey.com/v1', ...o });
92
94
  /** https://cerebras.ai: fast inference with a free tier. */
93
95
  export const cerebras = (o = {}) => openaiCompatible({ name: 'cerebras', baseUrl: 'https://api.cerebras.ai/v1', ...o });
94
96
  export const openai = (o = {}) => openaiCompatible({ name: 'openai', baseUrl: 'https://api.openai.com/v1', ...o });
package/dist/spend.d.ts CHANGED
@@ -1,4 +1,4 @@
1
- import type { SpendControl } from './lodestone.js';
1
+ import type { SpendCap, SpendControl } from './lodestone.js';
2
2
  import type { KeyValueStore } from './types.js';
3
3
  /** A daily and a monthly spend limit that holds under load. A bare `allow()` then `record()` lets a burst of
4
4
  * requests all pass the check before any of them has been counted. This takes a hold for each paid attempt
@@ -8,11 +8,21 @@ import type { KeyValueStore } from './types.js';
8
8
  *
9
9
  * Exact inside one process. Between processes it is as good as the store: with a plain get/set store, two
10
10
  * functions allowing at the same moment can each take a hold the other did not see, so size `holdUsd` and
11
- * the limit with that in mind. If the store cannot be read, the attempt is refused (the money is yours). */
11
+ * the limit with that in mind. If the store cannot be read, the attempt is refused (the money is yours).
12
+ *
13
+ * Each caller can have an allowance of their own (`perCaller`), so one person cannot use up the pool for
14
+ * everyone. It needs the call to name the caller (`caller: userId`); a call without one is only held to the
15
+ * pool. A caller's own key is their money and never counts here, unless they asked for a limit of their own
16
+ * (`spendCap` on the call): then their paid calls on that key are counted separately, in a ledger that never
17
+ * touches yours, and stopped when they reach the number they chose.
18
+ *
19
+ * What is stored under a caller is a hash of the caller, never the caller. */
12
20
  export interface SpendLimitOptions {
13
21
  store: KeyValueStore;
14
22
  dailyUsd?: number;
15
23
  monthlyUsd?: number;
24
+ /** What each caller may spend of the pool's money, per day and per month. A call names its caller with `caller`. A call may carry a lower `spendCap` of its own (a person lowering their own allowance); it never raises this. */
25
+ perCaller?: SpendCap;
16
26
  /** Held for each paid attempt until it settles: roughly the most one call can cost. Default $0.01. */
17
27
  holdUsd?: number;
18
28
  /** A hold that is never settled goes away after this. Default 2 minutes. */
@@ -30,4 +40,8 @@ export interface SpendReport {
30
40
  }
31
41
  export declare function createSpendLimit(o: SpendLimitOptions): Required<SpendControl> & {
32
42
  report(): Promise<SpendReport>;
43
+ callerReport(caller: string, opts?: {
44
+ ownKey?: boolean;
45
+ cap?: SpendCap;
46
+ }): Promise<SpendReport>;
33
47
  };
package/dist/spend.js CHANGED
@@ -1,4 +1,5 @@
1
- import { createMutex, isNonNegative, isRecord } from './util.js';
1
+ import { createMutex, isNonNegative, isRecord, sha256Hex } from './util.js';
2
+ const lowest = (a, b) => (a === undefined ? b : b === undefined ? a : Math.min(a, b));
2
3
  const MAX_HOLDS = 500;
3
4
  export function createSpendLimit(o) {
4
5
  const now = o.now ?? Date.now;
@@ -6,12 +7,20 @@ export function createSpendLimit(o) {
6
7
  const hold = o.holdUsd ?? 0.01;
7
8
  const ttl = o.holdTtlMs ?? 120_000;
8
9
  const exclusive = createMutex();
9
- /** What is in the store, checked: a ledger somebody else wrote is not trusted to be the right shape. */
10
- async function load() {
10
+ /** Which limit refused an attempt, until Lodestone asks. */
11
+ const refusals = new Map();
12
+ const refuse = (id, scope) => {
13
+ if (refusals.size >= 100)
14
+ refusals.delete(refusals.keys().next().value);
15
+ refusals.set(id, scope);
16
+ return false;
17
+ };
18
+ /** What is in the store at `at`, checked: a ledger somebody else wrote is not trusted to be the right shape. */
19
+ async function load(at) {
11
20
  const t = new Date(now());
12
21
  const day = t.toISOString().slice(0, 10);
13
22
  const month = day.slice(0, 7);
14
- const raw = await o.store.get(key);
23
+ const raw = await o.store.get(at);
15
24
  const s = isRecord(raw) ? raw : {};
16
25
  const holds = {};
17
26
  if (isRecord(s.holds)) {
@@ -27,20 +36,53 @@ export function createSpendLimit(o) {
27
36
  };
28
37
  }
29
38
  const held = (l) => Object.values(l.holds).reduce((a, h) => a + h[0], 0);
30
- const save = (l) => o.store.set(key, l);
39
+ const save = (at, l) => o.store.set(at, l);
40
+ /** Would one more hold go over the day or month limit? */
41
+ const over = (l, limit) => {
42
+ const pending = held(l) + hold;
43
+ return (limit?.dailyUsd !== undefined && l.dayUsd + pending > limit.dailyUsd) || (limit?.monthlyUsd !== undefined && l.monthUsd + pending > limit.monthlyUsd);
44
+ };
45
+ const settle = (l, id, costUsd) => {
46
+ const taken = l.holds[id]?.[0] ?? hold;
47
+ delete l.holds[id];
48
+ // A cost nobody reported counts as the hold: better to over-count than to let a free ride through.
49
+ const cost = isNonNegative(costUsd) ? costUsd : taken;
50
+ l.dayUsd += cost;
51
+ l.monthUsd += cost;
52
+ };
53
+ /** Where this attempt is counted besides the pool: the caller's allowance, or, on their own key, the cap they chose. */
54
+ async function target(info) {
55
+ if (!info.caller)
56
+ return null;
57
+ if (info.ownKey)
58
+ return info.cap ? { at: `${key}:own:${await sha256Hex(info.caller)}`, limit: info.cap } : null;
59
+ if (!o.perCaller && !info.cap)
60
+ return null;
61
+ return { at: `${key}:caller:${await sha256Hex(info.caller)}`, limit: { dailyUsd: lowest(o.perCaller?.dailyUsd, info.cap?.dailyUsd), monthlyUsd: lowest(o.perCaller?.monthlyUsd, info.cap?.monthlyUsd) } };
62
+ }
31
63
  return {
64
+ handlesOwnKey: true,
65
+ refusal: (id) => { const s = refusals.get(id); refusals.delete(id); return s; },
32
66
  allow: (info) => exclusive(async () => {
33
67
  try {
34
- const l = await load();
35
- const pending = held(l) + hold;
36
- if (o.dailyUsd !== undefined && l.dayUsd + pending > o.dailyUsd)
37
- return false;
38
- if (o.monthlyUsd !== undefined && l.monthUsd + pending > o.monthlyUsd)
39
- return false;
40
- if (Object.keys(l.holds).length >= MAX_HOLDS)
41
- return false;
42
- l.holds[info.id] = [hold, now() + ttl];
43
- await save(l);
68
+ const mine = await target(info);
69
+ const taken = [];
70
+ if (!info.ownKey) {
71
+ const l = await load(key);
72
+ if (over(l, o) || Object.keys(l.holds).length >= MAX_HOLDS)
73
+ return refuse(info.id, 'app');
74
+ taken.push([key, l]);
75
+ }
76
+ if (mine) {
77
+ const c = await load(mine.at);
78
+ if (over(c, mine.limit) || Object.keys(c.holds).length >= MAX_HOLDS)
79
+ return refuse(info.id, info.ownKey ? 'own-key' : 'caller');
80
+ taken.push([mine.at, c]);
81
+ }
82
+ for (const [at, l] of taken) {
83
+ l.holds[info.id] = [hold, now() + ttl];
84
+ await save(at, l);
85
+ }
44
86
  return true;
45
87
  }
46
88
  catch {
@@ -48,25 +90,32 @@ export function createSpendLimit(o) {
48
90
  }
49
91
  }),
50
92
  record: (info) => exclusive(async () => {
51
- const l = await load();
52
- const taken = l.holds[info.id]?.[0] ?? hold;
53
- delete l.holds[info.id];
54
- // A cost nobody reported counts as the hold: better to over-count than to let a free ride through.
55
- const cost = isNonNegative(info.costUsd) ? info.costUsd : taken;
56
- l.dayUsd += cost;
57
- l.monthUsd += cost;
58
- await save(l);
93
+ const mine = await target(info);
94
+ for (const at of [...(info.ownKey ? [] : [key]), ...(mine ? [mine.at] : [])]) {
95
+ const l = await load(at);
96
+ settle(l, info.id, info.costUsd);
97
+ await save(at, l);
98
+ }
59
99
  }),
60
100
  release: (info) => exclusive(async () => {
61
- const l = await load();
62
- if (!(info.id in l.holds))
63
- return;
64
- delete l.holds[info.id];
65
- await save(l);
101
+ const mine = await target(info);
102
+ for (const at of [...(info.ownKey ? [] : [key]), ...(mine ? [mine.at] : [])]) {
103
+ const l = await load(at);
104
+ if (!(info.id in l.holds))
105
+ continue;
106
+ delete l.holds[info.id];
107
+ await save(at, l);
108
+ }
66
109
  }),
67
110
  report: () => exclusive(async () => {
68
- const l = await load();
111
+ const l = await load(key);
69
112
  return { todayUsd: l.dayUsd, monthUsd: l.monthUsd, heldUsd: held(l), dailyUsd: o.dailyUsd, monthlyUsd: o.monthlyUsd };
70
113
  }),
114
+ /** What one caller has spent and may spend: on the pool's money, or on their own key with `ownKey` and the `cap` they chose. For showing "$0.04 of $0.10 used today". */
115
+ callerReport: (caller, opts = {}) => exclusive(async () => {
116
+ const mine = await target({ caller, ownKey: opts.ownKey, cap: opts.cap });
117
+ const l = await load(mine?.at ?? `${key}:caller:${await sha256Hex(caller)}`);
118
+ return { todayUsd: l.dayUsd, monthUsd: l.monthUsd, heldUsd: held(l), dailyUsd: mine?.limit?.dailyUsd, monthlyUsd: mine?.limit?.monthlyUsd };
119
+ }),
71
120
  };
72
121
  }
@@ -0,0 +1,141 @@
1
+ # Free Model Index: spec (draft 1, 2 Nov 2026)
2
+
3
+ > **Status:** milestone 1 (harness, validity rule, outcome classification, daily rollup) is built and public at https://github.com/obasaolorunfemi-ai/free-model-index. The scheduler, `scorecard.json`, the page and the Naija Eval join are not.
4
+
5
+ A public, open, scheduled measurement of how reliably OpenRouter's free models answer a real request, published as an open dataset, a methodology page and a weekly update. It is the Lodestone version of what makes SemiAnalysis's InferenceX worth citing: every number comes from a public run anyone can inspect, and it is updated continuously.
6
+
7
+ ## What it is, and is not
8
+
9
+ - **Is:** for each free model, how often a probe got a usable answer inside a serverless time budget, how fast, and how it failed when it did not. One uniform probe, so the rates are comparable between models.
10
+ - **Is not:** a quality ranking. A fast, reliable model can still be a poor translator. Quality lives in Naija Eval (below) and, for other tasks, nowhere yet.
11
+ - **Is not:** a measurement of paid models, or of every region. The probes run from GitHub's runners, not from Lagos.
12
+ - **Differs from the "weather" feed:** weather counts what one app's traffic happened to try, so a model is tried only because earlier ones failed, and its rate says little. The index sends the same request to every model on a schedule, so a rate here means something.
13
+
14
+ ## Name, versions, cadence
15
+
16
+ - **Name:** Free Model Index. Avoid the "MAX" and "X" suffixes SemiAnalysis uses on its own named products.
17
+ - **Methodology versions:** `v1.0` now. Any change to the probe, the validity rule or the timeout bumps the version; every datapoint carries the version it was taken under, and rates are never compared across versions without saying so.
18
+ - **Weekly updates:** numbered, published Mondays (Update 1, 2, ...). Each is the JSON for the week, a short written summary, and the list of models added or withdrawn.
19
+ - **Announce versions, not the daily data.** `v1` is the first public week with at least 14 days of history.
20
+
21
+ ## Where it lives, and licences
22
+
23
+ - **Repo:** a new public repo, `free-model-index`, under the same account. Separate from the library so the index can be cited and forked on its own, and so the library's releases do not depend on it.
24
+ - **Code (probe harness, aggregation, site page):** Apache-2.0, like InferenceX's benchmark repo.
25
+ - **Data:** CC BY 4.0, so a citation is the price of reuse. Attribution is the point.
26
+ - **README rule, borrowed from InferenceX:** only this repo contains the official index. Forks and reruns are unofficial and must say so.
27
+ - **Branch layout:** `main` holds code and the methodology; an append-only `data` branch holds the probe records, so the code history is not buried in one commit per run.
28
+
29
+ ## The probe
30
+
31
+ - **One fixed request**, published in the repo. A short JSON extraction (a one-line job posting to `{title, remote}`), temperature 0, `max_tokens` about 200, no reasoning option set, no system prompt. This is the kind of call Lodestone is for, and a model that "thinks" for thousands of tokens will time out, which is a real result for a serverless tool.
32
+ - **Validity rule, in code:** the answer parses as JSON, `title` mentions the role, `remote` is true. Anything else is "answered, unusable".
33
+ - **Timing:** measured to the last byte of the body, never to the headers. (28 Oct 2026: headers arrive seconds early and made every model look faster.)
34
+ - **Time limits:** a hard timeout at 45 s (the hang length seen on 28 Oct) and a "serverless budget" line at 24 s (Lodestone's default).
35
+ - **Explicit model ids only.** The harness refuses any id that does not end in `:free` and whose catalogue price is not zero, so a probe can never spend money. It does not use OpenRouter's router model.
36
+
37
+ ## Outcomes
38
+
39
+ Every probe ends in exactly one outcome, using the same names as Lodestone's failure kinds where they exist:
40
+
41
+ | Outcome | Meaning |
42
+ |---|---|
43
+ | `usable` | valid answer within 24 s |
44
+ | `usable_late` | valid answer between 24 s and the 45 s timeout |
45
+ | `unusable` | answered, but empty or failed the validity rule |
46
+ | `timeout` | no complete body within 45 s |
47
+ | `rate_limited_upstream` / `rate_limited_other` | a 429 from the model's own provider, or one we could not tell apart (classified with `readRateLimit()`) |
48
+ | `withdrawn` | 404 |
49
+ | `unavailable` | 5xx |
50
+ | `other` | anything else, with the status |
51
+ | `restricted` | 403: the model refuses direct API calls (for example it is offered only through approved apps). Recorded with OpenRouter's message, listed, never rated: it says nothing about reliability |
52
+ | `probe_budget` | our own account's daily or per-minute limit. OpenRouter's "daily" and "minute" 429s apply to the whole account, so they say nothing about a model. Recorded, never counted for or against it |
53
+
54
+ Headline metric per model per window: **usable rate** = `usable` / all counted outcomes. Also published: `usable_late`, `unusable`, `timeout` and rate-limit shares, and p50 and p95 latency of answered probes.
55
+
56
+ ## Budget: the constraint that shapes everything
57
+
58
+ OpenRouter's documented limits for `:free` models are 20 requests a minute, and 50 a day with under 10 credits purchased or 1,000 a day with 10 or more. Limits are per account, and extra keys do not raise them.
59
+
60
+ - **Use a dedicated account and key.** Probes must not share a cap with KeyStone, Inkstone, TouchStone or Naija Eval, or they would eat those tools' allowance and the tools would corrupt the index.
61
+ - **Option A, no purchase (50 a day):** on 2 Nov 2026 OpenRouter's catalogue lists 14 free text models, of which 12 are general chat models (a safety classifier and a code-only model are left out, as Lodestone leaves them out). 12 models x 3 probes a day (every 8 hours) is 36 a day, inside the cap with room for new models. Each model reaches n >= 30 in about 10 days, so the published window is trailing 14 days.
62
+ - **Option B, one-time 10-credit purchase (1,000 a day):** up to about 40 models, one probe an hour each, about 960 a day. n is about 168 a week per model, so a weekly window works. The credit is not consumed by free calls. This is the recommendation, and it is the owner's decision to spend it.
63
+ - **If there are more free models than the budget covers:** probe the most-used first (the models in Lodestone's default lists and the ones KeyStone, Inkstone and TouchStone call), and the rest on a rota, labelled "sampled".
64
+ - **Back-off:** honour `Retry-After`. Never retry a failed probe within the same hour; a retry would measure our persistence, not the model.
65
+
66
+ ## Finding new models the day they appear
67
+
68
+ - An hourly job reads OpenRouter's catalogue (not a completion, so it does not use the free-request cap), keeps the `:free` models, and diffs against the last list.
69
+ - A new free model gets its first probe in the same run and is added to the schedule. The weekly update names it ("added Tuesday 14:00 UTC").
70
+ - A model that is withdrawn (404) or whose expiry date is within a week is marked, not deleted. Its history stays.
71
+
72
+ ## Schedule and storage
73
+
74
+ - **Scheduler:** GitHub Actions cron on a public repo (standard runners are free for public repos). GitHub documents that scheduled runs can start late and that it disables schedules in a public repo with no activity for 60 days; recording the real start time of every probe and committing the results keeps it active.
75
+ - **Records:** `data/probes/YYYY-MM-DD.jsonl`, one line per probe: `{ts, version, model, outcome, status, ms, answered_chars, account_remaining}`. No model answers are stored beyond a length and the validity result; a weekly audit sample of 20 raw answers is kept for checking.
76
+ - **Rollups:** `data/daily/YYYY-MM-DD.json` and `data/weekly/YYYY-WW.json`.
77
+ - **Public file:** `scorecard.json` in the repo's published output, in `model-scorecard/1`, so Lodestone can read it unchanged:
78
+ `source: "free-model-index"`, `task: "availability"`, `metric: "usable_rate"` (and `latency_p50_ms`, `latency_p95_ms`), `n`, `measuredAt`, `slice: "openrouter-free"`.
79
+ - **Stable URL on the Lodestone site**, for example `https://lodestone.extrafemi.com/data/free-model-index.json`, with `Access-Control-Allow-Origin: *` so anyone can read it and the site's own CSP does not need a new origin.
80
+
81
+ ## The page
82
+
83
+ A page on the Lodestone site, `/free-model-index/`, built the way the Runs on Lodestone page is: static, reading the published JSON.
84
+
85
+ - A sortable table: model, usable rate, p50 and p95, timeouts, rate limits, first seen, last probed, n. Models with n < 30 in the window are listed but not rated.
86
+ - A 14-day strip per model (one cell a day) so a change is visible.
87
+ - "Updated" time, the methodology version, a link to the data and the repo, and a copy-ready citation line.
88
+ - Plain-language note at the top: "A low rate is not a bad model. These are the free tiers, which are rationed."
89
+ - Emit `Dataset` structured data and add the page to `llms.txt` and the sitemap, so search engines and AI assistants can find and quote it.
90
+
91
+ ## How Naija Eval connects
92
+
93
+ Two halves of the same question, "can I build on this free model?": the index says whether it answers at all, Naija Eval says how good the answer is in Hausa, Igbo and Yoruba.
94
+
95
+ - **Same format.** Both publish `model-scorecard/1`. Lodestone already reads several scorecards and orders models with `quality: { task, metric, language }`. A Hausa task can sort by Naija Eval's `chrf++`; any other task by the index's `usable_rate`.
96
+ - **The index gates Naija Eval's measurements.** On 28 Oct 2026 Naija Eval's free-model translation run got 0 of 180 answers from gemma-4-31b and nemotron-3.5-lightning, 4 of 180 from nemotron-3.5-super-vl, and 180 of 180 from nemotron-3-ultra. It correctly refused to publish a row of zeros. With the index, Naija Eval can run a free model's quality test only when that model has recently been usable, and can say "not measured: answers too rarely" instead of nothing.
97
+ - **The index feeds candidates both ways.** Lodestone's `model-candidates/1` already tells Naija Eval which models are in use. The index adds which free models exist and are alive, so Naija Eval benchmarks live models and skips dead ones.
98
+ - **A joined view.** One extra table, "up and good": usable rate (index) beside average chrF++ for Hausa, Igbo and Yoruba (Naija Eval, n = 100 per direction, with its date). A quality number is shown only for a model whose usable rate is above a stated floor in the same window, and always with its own n and date. Nobody else publishes this for African languages.
99
+ - **One rule, stated once:** quality and reliability stay separate columns and are never blended into one score.
100
+ - **Account separation again.** Naija Eval's all-day free runs also draw on an OpenRouter cap. It must use its own key, not the index's.
101
+
102
+ ## Methodology and independence note (published with v1)
103
+
104
+ - What a probe is, the exact prompt, the validity rule, the timeouts, the outcome table, the window, the minimum n.
105
+ - Limits, stated up front: one prompt; GitHub's runners and region; free tiers are rationed and change without notice; ranks are not a judgement of the model.
106
+ - Independence: we sell nothing whose price or availability depends on a model's rank. We take no payment from any model provider or gateway.
107
+ - Disclosure: Extrafemi's own tools (KeyStone, Inkstone, TouchStone) and Naija Eval use these models, and Lodestone is the library that routes around the ones that fail. That is a reason we measure this and a reason to read it critically; the harness and data are open so anyone can check.
108
+ - Corrections policy: a wrong number is corrected in the open, with the old value kept in the history.
109
+ - Nothing is implied about OpenRouter's view of the index; it has not reviewed it.
110
+
111
+ ## Funnel (later, not part of v1)
112
+
113
+ Library and index free, then hosted alerts (tell me when a model in my chain goes down or is withdrawn, using the index plus my Lodestone config), then a managed store for the circuit breaker and rate limits, then Extrafemi integration work. None of these should change what the index says.
114
+
115
+ ## Success measures
116
+
117
+ Referrals to the page and the JSON, GitHub stars and forks of the repo, npm downloads of the library in the weeks after each update, AI-crawler and assistant citations of the page, and inbound messages that mention the index. Review after 8 weekly updates.
118
+
119
+ ## Milestones
120
+
121
+ 1. **Harness and data repo:** probe, validity rule, outcome classification, dedicated key, a manual run, the raw records and the daily rollup. About 2 days.
122
+ 2. **Scheduler and catalogue watcher:** hourly or 8-hourly runs on GitHub Actions, new-model detection, the `data` branch. About 1 day.
123
+ 3. **`scorecard.json` and the page:** the table, the strip, structured data, `llms.txt`. About 2 days.
124
+ 4. **Methodology page and the first 14 days of history,** then **Update 1** and the announcement post.
125
+ 5. **Naija Eval join:** the gate and the joined table, once Naija Eval is ready to read the scorecard.
126
+
127
+ ## Risks
128
+
129
+ - **The free tier changes.** Limits, model availability and rules can change overnight. The methodology says what was true when, and the version bumps.
130
+ - **"One prompt proves nothing."** True, and stated. A second probe type (a longer reasoning-light task) is a v1.1 candidate, not a v1 requirement.
131
+ - **Our probes distort the thing measured.** A dedicated account and a once-an-hour rate keep that small, and `probe_budget` outcomes are excluded.
132
+ - **Cloudflare or an upstream blocks the runners.** Recorded as `other` with the status. If it is persistent, move the scheduler to a Worker cron, which is also free.
133
+ - **Credibility.** The index is only as good as the care in the methodology. Publish the failures and the corrections.
134
+ - **Terms.** OpenRouter's limits page has no rule about benchmarking, only that heavy abuse is blocked and limits are global. Re-read it before launch.
135
+
136
+ ## Open decisions for the owner
137
+
138
+ 1. Spend the one-time 10 credits on a dedicated OpenRouter account (Option B), or start on Option A with 15 models?
139
+ 2. A separate `free-model-index` repo, or a folder inside the Lodestone repo?
140
+ 3. Data licence: CC BY 4.0 (recommended) or CC0?
141
+ 4. First scorecard consumer: ship the Lodestone site page first, or the Naija Eval join first?
@@ -14,6 +14,26 @@ Two small JSON formats let a model evaluator and Lodestone feed each other witho
14
14
 
15
15
  Nothing here is automatic or hidden. A scorecard changes the order models are *tried* in, for tasks you name, and nothing else. A model nobody measured is never dropped, added or moved.
16
16
 
17
+ ## Why Naija Eval is a good evaluator to start with
18
+
19
+ [Naija Eval](https://naija-eval.extrafemi.com) tests Nigeria's own language model, N-ATLaS, against the Llama model it was built from and four hosted models, on Hausa, Igbo and Yoruba. Four things make it a sound first source for Lodestone, and its limits are part of the answer.
20
+
21
+ - **It measures what a price list cannot tell you.** Whether a model can translate between English and Hausa, Igbo or Yoruba, or transcribe speech in those languages, is not something a provider's own claims settle. Naija Eval scores each model on six translation directions (chrF++) and three speech languages (word error rate), and also shows each score ignoring tone marks, so a missing mark can be told apart from a wrong word.
22
+ - **Anyone can re-run it.** It uses public test sets (Google's FLEURS, CC BY 4.0, and Meta's FLORES, CC BY-SA 4.0), one fixed prompt for every model and temperature 0. A script for each step, the results file (`summary.json`) and the scorecard (`scorecard.json`) are public, and repeating the four hosted-model runs cost $1.59. A number that can be repeated can also be checked, and corrected.
23
+ - **It states what it did not do.** Each score rests on 100 items per language and direction, and Naija Eval calls itself a first pass, not a leaderboard. The 4-bit local models may score a point or two below full precision. The test sets are public, so any model may have seen them in training. FLEURS is clean read speech, and street voice notes will score differently.
24
+ - **It publishes every number with what it rests on.** In `model-scorecard/1` each score carries its metric, which way is good, the number of items (`n`) and the day it was measured, so Lodestone never has to guess how much weight a figure deserves.
25
+
26
+ ## Why its results are reliable enough for Lodestone
27
+
28
+ "Reliable" here means something narrow. Lodestone uses a scorecard to put models in a better order for a task you name, and it is built so that a wrong number cannot do much harm:
29
+
30
+ - A scorecard only changes the *order* models are tried in. Nothing is added, dropped or called that you did not list.
31
+ - A score resting on fewer than 30 items, or measured more than a year ago, is ignored. Naija Eval's scores rest on 100 items each.
32
+ - A model nobody measured keeps its slot, an explicit `route` is never reordered, and if the scorecard cannot be fetched within 1.5 seconds the order is left alone.
33
+ - The worst a bad number can do is put a slightly worse model first for that task. A failure or a slow answer still moves the call to the next model, as it always does.
34
+
35
+ What it does not tell you: Naija Eval measures translation and speech in Hausa, Igbo and Yoruba, so use it only for tasks you tag that way. It says nothing about extracting a job posting or scoring a CV in English. And one disclosure: Naija Eval and Lodestone are both Extrafemi projects, so the evaluator is not independent of Lodestone's author. The protection is that its method and data are public, so anyone can check it.
36
+
17
37
  ## `model-scorecard/1`
18
38
 
19
39
  Measured results about models.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@extrafemi-ai/lodestone",
3
- "version": "0.7.0",
3
+ "version": "0.8.0",
4
4
  "description": "Calls AI models so your tool keeps working when a model is withdrawn, slow, rate-limited or down: failover across providers, self-updating model lists, spend limits.",
5
5
  "license": "MIT",
6
6
  "repository": {