@extrafemi-ai/lodestone 0.7.0 → 0.7.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +7 -0
- package/README.md +8 -7
- package/dist/index.d.ts +1 -1
- package/dist/index.js +1 -1
- package/dist/providers/anthropic.d.ts +2 -0
- package/dist/providers/anthropic.js +1 -1
- package/dist/providers/openaiCompatible.d.ts +2 -0
- package/dist/providers/openaiCompatible.js +2 -0
- package/docs/free-model-index-spec.md +141 -0
- package/docs/scorecards.md +20 -0
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,13 @@
|
|
|
2
2
|
|
|
3
3
|
Versions before 1.0 may change between minor releases; each change that could break a caller is marked **breaking**.
|
|
4
4
|
|
|
5
|
+
## 0.7.1
|
|
6
|
+
|
|
7
|
+
- **`omniakey()` preset** (TypeScript and Python) for [OmniaKey](https://omniakey.com), a prepaid gateway with an OpenAI-compatible text endpoint. Use it as a `fallbacks` entry. It is a preset only: a key was accepted and its model list read (98 entries, with image, video and music models mixed in and nothing to tell them apart, so name your models and do not use `discover` with it), but the first chat call was refused with `insufficient_user_quota`, so it has not been seen to answer, and the Providers page says so. Text and JSON calls only; OmniaKey's image, video and music models are outside what Lodestone does.
|
|
8
|
+
|
|
9
|
+
- **Fixed: the direct Anthropic adapter no longer sends a temperature.** Anthropic's current models reject one: Claude 4.7 and later answer 400 to a non-default value, and API version 2026-10-21 removes the parameter for every model. Lodestone's calls default to 0.2, so a direct call to a current Claude model would have failed. `anthropic()` now leaves it out and the model's own setting applies; `anthropic({ sendTemperature: true })` restores it for an older model. The Python port's `Anthropic(send_temperature=...)` works the same way. Calls through OpenRouter are unaffected. This was found by reading Anthropic's deprecations page, not from a failure: no live call to Anthropic has been made.
|
|
10
|
+
- README: the route example used `claude-sonnet-4-5`, which Anthropic has deprecated; it now uses `claude-sonnet-5-5`.
|
|
11
|
+
|
|
5
12
|
## 0.7.0
|
|
6
13
|
|
|
7
14
|
Security fixes, shared state that works on every platform, per-caller limits, and runtimes beyond Node. Found in a review of 0.6.0.
|
package/README.md
CHANGED
|
@@ -4,11 +4,11 @@
|
|
|
4
4
|
|
|
5
5
|
Calls AI models so your tool keeps working when a model is withdrawn, slow or rate-limited.
|
|
6
6
|
|
|
7
|
-
Free
|
|
7
|
+
Free models come and go. A hard-coded model id breaks without warning, one slow model can use your whole time limit, and "429" can mean four different things that need four different messages. Lodestone handles those so each tool doesn't have to.
|
|
8
8
|
|
|
9
9
|
- **Replaces withdrawn models by itself.** Your lists are preferences. They are checked against OpenRouter's live catalogue, and any model that has gone is swapped for a current one of the same kind.
|
|
10
10
|
- **Keeps inside a time budget.** Each model gets a slice of it. A slow or failing model is skipped for the next, so a function never hits its gateway limit with an unreadable error.
|
|
11
|
-
- **Falls back across providers.** OpenRouter's free models, then a
|
|
11
|
+
- **Falls back across providers.** OpenRouter's free models, then a paid model, then Anthropic or Google on their own keys.
|
|
12
12
|
- **Explains failures in plain words.** A daily limit, a per-minute limit, a busy upstream provider, a bad key and a slow model are different errors with different advice.
|
|
13
13
|
- **Checks the answer.** `json()` skips a model whose answer isn't valid JSON, or doesn't pass your validator.
|
|
14
14
|
- **Bring your own key.** A caller's key replaces the platform's and allows the paid fallback, without touching the platform's budget.
|
|
@@ -59,8 +59,8 @@ Tiers are for tools that don't care which model answers. A tool that picks its o
|
|
|
59
59
|
await ai.complete({
|
|
60
60
|
system, messages, maxTokens: 900,
|
|
61
61
|
route: [
|
|
62
|
-
{ provider: 'anthropic', model: 'claude-sonnet-
|
|
63
|
-
{ provider: 'openrouter', model: 'anthropic/claude-sonnet-
|
|
62
|
+
{ provider: 'anthropic', model: 'claude-sonnet-5-5', apiKey: customerKey },
|
|
63
|
+
{ provider: 'openrouter', model: 'anthropic/claude-sonnet-5.5', apiKey: backupKey },
|
|
64
64
|
],
|
|
65
65
|
});
|
|
66
66
|
```
|
|
@@ -78,7 +78,7 @@ Pass the customer's OpenRouter key as `apiKey` on a call. It replaces the platfo
|
|
|
78
78
|
For each call, in order, until one answers:
|
|
79
79
|
|
|
80
80
|
1. OpenRouter's free models for the tier (current ones, per the catalogue).
|
|
81
|
-
2. A
|
|
81
|
+
2. A paid OpenRouter model, only if the caller supplied their own key or you set `allowPaid`. With no credit OpenRouter refuses it and nothing is spent.
|
|
82
82
|
3. Each `fallbacks` provider, with its own key.
|
|
83
83
|
|
|
84
84
|
When a later phase exists, the free models get a short turn (9 s in all by default), so there is always time left for it. The whole call stays inside `budgetMs` (24 s by default, under a 30 s gateway limit).
|
|
@@ -98,7 +98,7 @@ const { ok, withdrawn, shortTiers } = await ai.checkModels();
|
|
|
98
98
|
|
|
99
99
|
### Free models that are not worth waiting for (`paidFirst`)
|
|
100
100
|
|
|
101
|
-
Free models can be down for hours (rate-limited, or holding a connection open for 45 seconds and never answering). On the platform's own key, `openrouter: { paidFirst: true }` tries the
|
|
101
|
+
Free models can be down for hours (rate-limited, or holding a connection open for 45 seconds and never answering). On the platform's own key, `openrouter: { paidFirst: true }` tries the paid model first and the free ones only if it fails or the spend limit is reached, so a request is not slowed by free models that are not going to answer. A caller's own key is not affected: the paid models are their money, so their free models still come first. Reasoning is set per model with `openrouter({ reasoning: (model) => ... })`, because models differ: one thinks for a thousand tokens unless told not to (measured: turning it off halved scoring time, 10 s to 5 s, with the same answers), another refuses to run with reasoning off.
|
|
102
102
|
|
|
103
103
|
### Not waiting on a model that is down (circuit breaker)
|
|
104
104
|
|
|
@@ -143,9 +143,10 @@ OpenRouter is the default, but one provider's outage should not stop your tool.
|
|
|
143
143
|
| `requesty()` | [Requesty](https://www.requesty.ai) | Gateway to 800+ models with failover. Publishes price, lab, whether a model is trained on and how long prompts are kept. `requesty({ region: 'eu' })` keeps prompts in the EU. No free models; a 5% markup. |
|
|
144
144
|
| `llmapi()` | [LLM API](https://llmapi.ai) | Gateway to 150+ chat models. Lists price; ignores its image, video and speech models. |
|
|
145
145
|
| `groq()`, `cerebras()` | Groq, Cerebras | Fast inference with free tiers. Use `free: true` so they are not counted against a spend limit. |
|
|
146
|
+
| `omniakey()` | [OmniaKey](https://omniakey.com) | A prepaid gateway for text, image, video and music models, with an OpenAI-compatible text endpoint. Text and JSON calls only here. Its model list needs a key and mixes chat with image, video and music models, so name your models. Not yet called live: the account's first chat call was refused for insufficient quota. |
|
|
146
147
|
| `openai()`, `mistral()`, `together()` | the providers themselves | |
|
|
147
148
|
| `ollama()` | a local server | Free and private. |
|
|
148
|
-
| `anthropic()`, `google()` | direct | Not OpenAI-style, so they have their own adapters. |
|
|
149
|
+
| `anthropic()`, `google()` | direct | Not OpenAI-style, so they have their own adapters. `anthropic()` leaves the temperature out, because Anthropic's current models reject one (Claude 4.7 and later answer 400 to a non-default value, and API version 2026-10-21 removes it); pass `anthropic({ sendTemperature: true })` for an older model that still accepts one. Model ids are retired on a schedule (`claude-sonnet-4-20250514` was retired on 15 June 2026): check [Anthropic's deprecations page](https://platform.claude.com/docs/en/about-claude/model-deprecations), and keep a second model or provider in the chain, so a retired id costs one attempt, not the call. |
|
|
149
150
|
|
|
150
151
|
```ts
|
|
151
152
|
fallbacks: [
|
package/dist/index.d.ts
CHANGED
|
@@ -28,7 +28,7 @@ export { CANDIDATES_SCHEMA } from './candidates.js';
|
|
|
28
28
|
export type { Candidates, Candidate } from './candidates.js';
|
|
29
29
|
export { createHealth } from './health.js';
|
|
30
30
|
export type { Health, HealthOptions, ModelHealth } from './health.js';
|
|
31
|
-
export { openaiCompatible, requesty, llmapi, groq, cerebras, openai, mistral, together, ollama, normalizeRequesty, normalizeLlmapi, normalizeGroq } from './providers/openaiCompatible.js';
|
|
31
|
+
export { openaiCompatible, requesty, llmapi, groq, cerebras, omniakey, openai, mistral, together, ollama, normalizeRequesty, normalizeLlmapi, normalizeGroq } from './providers/openaiCompatible.js';
|
|
32
32
|
export type { OpenAICompatibleOptions, PresetOptions } from './providers/openaiCompatible.js';
|
|
33
33
|
export { vet, publisherOf, DEFAULT_PUBLISHERS } from './trust.js';
|
|
34
34
|
export type { TrustPolicy, Verdict } from './trust.js';
|
package/dist/index.js
CHANGED
|
@@ -16,7 +16,7 @@ export { SCORECARD_SCHEMA, ScorecardError, parseScorecard, fetchScorecard, rankM
|
|
|
16
16
|
export { createStats } from './stats.js';
|
|
17
17
|
export { CANDIDATES_SCHEMA } from './candidates.js';
|
|
18
18
|
export { createHealth } from './health.js';
|
|
19
|
-
export { openaiCompatible, requesty, llmapi, groq, cerebras, openai, mistral, together, ollama, normalizeRequesty, normalizeLlmapi, normalizeGroq } from './providers/openaiCompatible.js';
|
|
19
|
+
export { openaiCompatible, requesty, llmapi, groq, cerebras, omniakey, openai, mistral, together, ollama, normalizeRequesty, normalizeLlmapi, normalizeGroq } from './providers/openaiCompatible.js';
|
|
20
20
|
export { vet, publisherOf, DEFAULT_PUBLISHERS } from './trust.js';
|
|
21
21
|
export { resolveProviderModels } from './discover.js';
|
|
22
22
|
export { netlifyBlobsStore, cloudflareKvStore, upstashStore, denoKvStore, safeKey } from './stores/index.js';
|
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
import type { Provider } from '../types.js';
|
|
2
2
|
export interface AnthropicOptions {
|
|
3
3
|
fetch?: typeof fetch;
|
|
4
|
+
/** Send the request's temperature. Default false. Anthropic's current models reject a temperature: Claude 4.7 and later answer 400 to a non-default one, and API version 2026-10-21 removes the parameter for every model. Lodestone's calls default to 0.2, so by default it is left out and the model's own setting applies. Turn this on only for an older model that still accepts one. */
|
|
5
|
+
sendTemperature?: boolean;
|
|
4
6
|
}
|
|
5
7
|
/** Anthropic's own API, for use as a fallback with a separate key from OpenRouter. */
|
|
6
8
|
export declare function anthropic(opts?: AnthropicOptions): Provider;
|
|
@@ -22,7 +22,7 @@ export function anthropic(opts = {}) {
|
|
|
22
22
|
body: JSON.stringify({
|
|
23
23
|
model: req.model,
|
|
24
24
|
max_tokens: req.maxTokens,
|
|
25
|
-
...(req.temperature
|
|
25
|
+
...(opts.sendTemperature && req.temperature !== undefined ? { temperature: req.temperature } : {}),
|
|
26
26
|
...(req.system ? { system: req.system } : {}),
|
|
27
27
|
messages: req.messages.map(wire),
|
|
28
28
|
}),
|
|
@@ -38,6 +38,8 @@ export declare const requesty: (o?: PresetOptions & {
|
|
|
38
38
|
export declare const llmapi: (o?: PresetOptions) => Provider;
|
|
39
39
|
/** https://groq.com: fast inference with a free developer tier. */
|
|
40
40
|
export declare const groq: (o?: PresetOptions) => Provider;
|
|
41
|
+
/** https://omniakey.com: a prepaid gateway to text, image, video and music models, with an OpenAI-compatible text endpoint (api.omniakey.com/v1). Its model list needs a key (401 without one), and its site says it does not switch a request to a different model by itself, which is what Lodestone does on top of it. Not yet called live. */
|
|
42
|
+
export declare const omniakey: (o?: PresetOptions) => Provider;
|
|
41
43
|
/** https://cerebras.ai: fast inference with a free tier. */
|
|
42
44
|
export declare const cerebras: (o?: PresetOptions) => Provider;
|
|
43
45
|
export declare const openai: (o?: PresetOptions) => Provider;
|
|
@@ -89,6 +89,8 @@ export const requesty = (o = {}) => openaiCompatible({ name: 'requesty', baseUrl
|
|
|
89
89
|
export const llmapi = (o = {}) => openaiCompatible({ name: 'llmapi', baseUrl: 'https://api.llmapi.ai/v1', normalize: normalizeLlmapi, ...o });
|
|
90
90
|
/** https://groq.com: fast inference with a free developer tier. */
|
|
91
91
|
export const groq = (o = {}) => openaiCompatible({ name: 'groq', baseUrl: 'https://api.groq.com/openai/v1', normalize: normalizeGroq, ...o });
|
|
92
|
+
/** https://omniakey.com: a prepaid gateway to text, image, video and music models, with an OpenAI-compatible text endpoint (api.omniakey.com/v1). Its model list needs a key (401 without one), and its site says it does not switch a request to a different model by itself, which is what Lodestone does on top of it. Not yet called live. */
|
|
93
|
+
export const omniakey = (o = {}) => openaiCompatible({ name: 'omniakey', baseUrl: 'https://api.omniakey.com/v1', ...o });
|
|
92
94
|
/** https://cerebras.ai: fast inference with a free tier. */
|
|
93
95
|
export const cerebras = (o = {}) => openaiCompatible({ name: 'cerebras', baseUrl: 'https://api.cerebras.ai/v1', ...o });
|
|
94
96
|
export const openai = (o = {}) => openaiCompatible({ name: 'openai', baseUrl: 'https://api.openai.com/v1', ...o });
|
|
@@ -0,0 +1,141 @@
|
|
|
1
|
+
# Free Model Index: spec (draft 1, 2 Nov 2026)
|
|
2
|
+
|
|
3
|
+
> **Status:** milestone 1 (harness, validity rule, outcome classification, daily rollup) is built and public at https://github.com/obasaolorunfemi-ai/free-model-index. The scheduler, `scorecard.json`, the page and the Naija Eval join are not.
|
|
4
|
+
|
|
5
|
+
A public, open, scheduled measurement of how reliably OpenRouter's free models answer a real request, published as an open dataset, a methodology page and a weekly update. It is the Lodestone version of what makes SemiAnalysis's InferenceX worth citing: every number comes from a public run anyone can inspect, and it is updated continuously.
|
|
6
|
+
|
|
7
|
+
## What it is, and is not
|
|
8
|
+
|
|
9
|
+
- **Is:** for each free model, how often a probe got a usable answer inside a serverless time budget, how fast, and how it failed when it did not. One uniform probe, so the rates are comparable between models.
|
|
10
|
+
- **Is not:** a quality ranking. A fast, reliable model can still be a poor translator. Quality lives in Naija Eval (below) and, for other tasks, nowhere yet.
|
|
11
|
+
- **Is not:** a measurement of paid models, or of every region. The probes run from GitHub's runners, not from Lagos.
|
|
12
|
+
- **Differs from the "weather" feed:** weather counts what one app's traffic happened to try, so a model is tried only because earlier ones failed, and its rate says little. The index sends the same request to every model on a schedule, so a rate here means something.
|
|
13
|
+
|
|
14
|
+
## Name, versions, cadence
|
|
15
|
+
|
|
16
|
+
- **Name:** Free Model Index. Avoid the "MAX" and "X" suffixes SemiAnalysis uses on its own named products.
|
|
17
|
+
- **Methodology versions:** `v1.0` now. Any change to the probe, the validity rule or the timeout bumps the version; every datapoint carries the version it was taken under, and rates are never compared across versions without saying so.
|
|
18
|
+
- **Weekly updates:** numbered, published Mondays (Update 1, 2, ...). Each is the JSON for the week, a short written summary, and the list of models added or withdrawn.
|
|
19
|
+
- **Announce versions, not the daily data.** `v1` is the first public week with at least 14 days of history.
|
|
20
|
+
|
|
21
|
+
## Where it lives, and licences
|
|
22
|
+
|
|
23
|
+
- **Repo:** a new public repo, `free-model-index`, under the same account. Separate from the library so the index can be cited and forked on its own, and so the library's releases do not depend on it.
|
|
24
|
+
- **Code (probe harness, aggregation, site page):** Apache-2.0, like InferenceX's benchmark repo.
|
|
25
|
+
- **Data:** CC BY 4.0, so a citation is the price of reuse. Attribution is the point.
|
|
26
|
+
- **README rule, borrowed from InferenceX:** only this repo contains the official index. Forks and reruns are unofficial and must say so.
|
|
27
|
+
- **Branch layout:** `main` holds code and the methodology; an append-only `data` branch holds the probe records, so the code history is not buried in one commit per run.
|
|
28
|
+
|
|
29
|
+
## The probe
|
|
30
|
+
|
|
31
|
+
- **One fixed request**, published in the repo. A short JSON extraction (a one-line job posting to `{title, remote}`), temperature 0, `max_tokens` about 200, no reasoning option set, no system prompt. This is the kind of call Lodestone is for, and a model that "thinks" for thousands of tokens will time out, which is a real result for a serverless tool.
|
|
32
|
+
- **Validity rule, in code:** the answer parses as JSON, `title` mentions the role, `remote` is true. Anything else is "answered, unusable".
|
|
33
|
+
- **Timing:** measured to the last byte of the body, never to the headers. (28 Oct 2026: headers arrive seconds early and made every model look faster.)
|
|
34
|
+
- **Time limits:** a hard timeout at 45 s (the hang length seen on 28 Oct) and a "serverless budget" line at 24 s (Lodestone's default).
|
|
35
|
+
- **Explicit model ids only.** The harness refuses any id that does not end in `:free` and whose catalogue price is not zero, so a probe can never spend money. It does not use OpenRouter's router model.
|
|
36
|
+
|
|
37
|
+
## Outcomes
|
|
38
|
+
|
|
39
|
+
Every probe ends in exactly one outcome, using the same names as Lodestone's failure kinds where they exist:
|
|
40
|
+
|
|
41
|
+
| Outcome | Meaning |
|
|
42
|
+
|---|---|
|
|
43
|
+
| `usable` | valid answer within 24 s |
|
|
44
|
+
| `usable_late` | valid answer between 24 s and the 45 s timeout |
|
|
45
|
+
| `unusable` | answered, but empty or failed the validity rule |
|
|
46
|
+
| `timeout` | no complete body within 45 s |
|
|
47
|
+
| `rate_limited_upstream` / `rate_limited_other` | a 429 from the model's own provider, or one we could not tell apart (classified with `readRateLimit()`) |
|
|
48
|
+
| `withdrawn` | 404 |
|
|
49
|
+
| `unavailable` | 5xx |
|
|
50
|
+
| `other` | anything else, with the status |
|
|
51
|
+
| `restricted` | 403: the model refuses direct API calls (for example it is offered only through approved apps). Recorded with OpenRouter's message, listed, never rated: it says nothing about reliability |
|
|
52
|
+
| `probe_budget` | our own account's daily or per-minute limit. OpenRouter's "daily" and "minute" 429s apply to the whole account, so they say nothing about a model. Recorded, never counted for or against it |
|
|
53
|
+
|
|
54
|
+
Headline metric per model per window: **usable rate** = `usable` / all counted outcomes. Also published: `usable_late`, `unusable`, `timeout` and rate-limit shares, and p50 and p95 latency of answered probes.
|
|
55
|
+
|
|
56
|
+
## Budget: the constraint that shapes everything
|
|
57
|
+
|
|
58
|
+
OpenRouter's documented limits for `:free` models are 20 requests a minute, and 50 a day with under 10 credits purchased or 1,000 a day with 10 or more. Limits are per account, and extra keys do not raise them.
|
|
59
|
+
|
|
60
|
+
- **Use a dedicated account and key.** Probes must not share a cap with KeyStone, Inkstone, TouchStone or Naija Eval, or they would eat those tools' allowance and the tools would corrupt the index.
|
|
61
|
+
- **Option A, no purchase (50 a day):** on 2 Nov 2026 OpenRouter's catalogue lists 14 free text models, of which 12 are general chat models (a safety classifier and a code-only model are left out, as Lodestone leaves them out). 12 models x 3 probes a day (every 8 hours) is 36 a day, inside the cap with room for new models. Each model reaches n >= 30 in about 10 days, so the published window is trailing 14 days.
|
|
62
|
+
- **Option B, one-time 10-credit purchase (1,000 a day):** up to about 40 models, one probe an hour each, about 960 a day. n is about 168 a week per model, so a weekly window works. The credit is not consumed by free calls. This is the recommendation, and it is the owner's decision to spend it.
|
|
63
|
+
- **If there are more free models than the budget covers:** probe the most-used first (the models in Lodestone's default lists and the ones KeyStone, Inkstone and TouchStone call), and the rest on a rota, labelled "sampled".
|
|
64
|
+
- **Back-off:** honour `Retry-After`. Never retry a failed probe within the same hour; a retry would measure our persistence, not the model.
|
|
65
|
+
|
|
66
|
+
## Finding new models the day they appear
|
|
67
|
+
|
|
68
|
+
- An hourly job reads OpenRouter's catalogue (not a completion, so it does not use the free-request cap), keeps the `:free` models, and diffs against the last list.
|
|
69
|
+
- A new free model gets its first probe in the same run and is added to the schedule. The weekly update names it ("added Tuesday 14:00 UTC").
|
|
70
|
+
- A model that is withdrawn (404) or whose expiry date is within a week is marked, not deleted. Its history stays.
|
|
71
|
+
|
|
72
|
+
## Schedule and storage
|
|
73
|
+
|
|
74
|
+
- **Scheduler:** GitHub Actions cron on a public repo (standard runners are free for public repos). GitHub documents that scheduled runs can start late and that it disables schedules in a public repo with no activity for 60 days; recording the real start time of every probe and committing the results keeps it active.
|
|
75
|
+
- **Records:** `data/probes/YYYY-MM-DD.jsonl`, one line per probe: `{ts, version, model, outcome, status, ms, answered_chars, account_remaining}`. No model answers are stored beyond a length and the validity result; a weekly audit sample of 20 raw answers is kept for checking.
|
|
76
|
+
- **Rollups:** `data/daily/YYYY-MM-DD.json` and `data/weekly/YYYY-WW.json`.
|
|
77
|
+
- **Public file:** `scorecard.json` in the repo's published output, in `model-scorecard/1`, so Lodestone can read it unchanged:
|
|
78
|
+
`source: "free-model-index"`, `task: "availability"`, `metric: "usable_rate"` (and `latency_p50_ms`, `latency_p95_ms`), `n`, `measuredAt`, `slice: "openrouter-free"`.
|
|
79
|
+
- **Stable URL on the Lodestone site**, for example `https://lodestone.extrafemi.com/data/free-model-index.json`, with `Access-Control-Allow-Origin: *` so anyone can read it and the site's own CSP does not need a new origin.
|
|
80
|
+
|
|
81
|
+
## The page
|
|
82
|
+
|
|
83
|
+
A page on the Lodestone site, `/free-model-index/`, built the way the Runs on Lodestone page is: static, reading the published JSON.
|
|
84
|
+
|
|
85
|
+
- A sortable table: model, usable rate, p50 and p95, timeouts, rate limits, first seen, last probed, n. Models with n < 30 in the window are listed but not rated.
|
|
86
|
+
- A 14-day strip per model (one cell a day) so a change is visible.
|
|
87
|
+
- "Updated" time, the methodology version, a link to the data and the repo, and a copy-ready citation line.
|
|
88
|
+
- Plain-language note at the top: "A low rate is not a bad model. These are the free tiers, which are rationed."
|
|
89
|
+
- Emit `Dataset` structured data and add the page to `llms.txt` and the sitemap, so search engines and AI assistants can find and quote it.
|
|
90
|
+
|
|
91
|
+
## How Naija Eval connects
|
|
92
|
+
|
|
93
|
+
Two halves of the same question, "can I build on this free model?": the index says whether it answers at all, Naija Eval says how good the answer is in Hausa, Igbo and Yoruba.
|
|
94
|
+
|
|
95
|
+
- **Same format.** Both publish `model-scorecard/1`. Lodestone already reads several scorecards and orders models with `quality: { task, metric, language }`. A Hausa task can sort by Naija Eval's `chrf++`; any other task by the index's `usable_rate`.
|
|
96
|
+
- **The index gates Naija Eval's measurements.** On 28 Oct 2026 Naija Eval's free-model translation run got 0 of 180 answers from gemma-4-31b and nemotron-3.5-lightning, 4 of 180 from nemotron-3.5-super-vl, and 180 of 180 from nemotron-3-ultra. It correctly refused to publish a row of zeros. With the index, Naija Eval can run a free model's quality test only when that model has recently been usable, and can say "not measured: answers too rarely" instead of nothing.
|
|
97
|
+
- **The index feeds candidates both ways.** Lodestone's `model-candidates/1` already tells Naija Eval which models are in use. The index adds which free models exist and are alive, so Naija Eval benchmarks live models and skips dead ones.
|
|
98
|
+
- **A joined view.** One extra table, "up and good": usable rate (index) beside average chrF++ for Hausa, Igbo and Yoruba (Naija Eval, n = 100 per direction, with its date). A quality number is shown only for a model whose usable rate is above a stated floor in the same window, and always with its own n and date. Nobody else publishes this for African languages.
|
|
99
|
+
- **One rule, stated once:** quality and reliability stay separate columns and are never blended into one score.
|
|
100
|
+
- **Account separation again.** Naija Eval's all-day free runs also draw on an OpenRouter cap. It must use its own key, not the index's.
|
|
101
|
+
|
|
102
|
+
## Methodology and independence note (published with v1)
|
|
103
|
+
|
|
104
|
+
- What a probe is, the exact prompt, the validity rule, the timeouts, the outcome table, the window, the minimum n.
|
|
105
|
+
- Limits, stated up front: one prompt; GitHub's runners and region; free tiers are rationed and change without notice; ranks are not a judgement of the model.
|
|
106
|
+
- Independence: we sell nothing whose price or availability depends on a model's rank. We take no payment from any model provider or gateway.
|
|
107
|
+
- Disclosure: Extrafemi's own tools (KeyStone, Inkstone, TouchStone) and Naija Eval use these models, and Lodestone is the library that routes around the ones that fail. That is a reason we measure this and a reason to read it critically; the harness and data are open so anyone can check.
|
|
108
|
+
- Corrections policy: a wrong number is corrected in the open, with the old value kept in the history.
|
|
109
|
+
- Nothing is implied about OpenRouter's view of the index; it has not reviewed it.
|
|
110
|
+
|
|
111
|
+
## Funnel (later, not part of v1)
|
|
112
|
+
|
|
113
|
+
Library and index free, then hosted alerts (tell me when a model in my chain goes down or is withdrawn, using the index plus my Lodestone config), then a managed store for the circuit breaker and rate limits, then Extrafemi integration work. None of these should change what the index says.
|
|
114
|
+
|
|
115
|
+
## Success measures
|
|
116
|
+
|
|
117
|
+
Referrals to the page and the JSON, GitHub stars and forks of the repo, npm downloads of the library in the weeks after each update, AI-crawler and assistant citations of the page, and inbound messages that mention the index. Review after 8 weekly updates.
|
|
118
|
+
|
|
119
|
+
## Milestones
|
|
120
|
+
|
|
121
|
+
1. **Harness and data repo:** probe, validity rule, outcome classification, dedicated key, a manual run, the raw records and the daily rollup. About 2 days.
|
|
122
|
+
2. **Scheduler and catalogue watcher:** hourly or 8-hourly runs on GitHub Actions, new-model detection, the `data` branch. About 1 day.
|
|
123
|
+
3. **`scorecard.json` and the page:** the table, the strip, structured data, `llms.txt`. About 2 days.
|
|
124
|
+
4. **Methodology page and the first 14 days of history,** then **Update 1** and the announcement post.
|
|
125
|
+
5. **Naija Eval join:** the gate and the joined table, once Naija Eval is ready to read the scorecard.
|
|
126
|
+
|
|
127
|
+
## Risks
|
|
128
|
+
|
|
129
|
+
- **The free tier changes.** Limits, model availability and rules can change overnight. The methodology says what was true when, and the version bumps.
|
|
130
|
+
- **"One prompt proves nothing."** True, and stated. A second probe type (a longer reasoning-light task) is a v1.1 candidate, not a v1 requirement.
|
|
131
|
+
- **Our probes distort the thing measured.** A dedicated account and a once-an-hour rate keep that small, and `probe_budget` outcomes are excluded.
|
|
132
|
+
- **Cloudflare or an upstream blocks the runners.** Recorded as `other` with the status. If it is persistent, move the scheduler to a Worker cron, which is also free.
|
|
133
|
+
- **Credibility.** The index is only as good as the care in the methodology. Publish the failures and the corrections.
|
|
134
|
+
- **Terms.** OpenRouter's limits page has no rule about benchmarking, only that heavy abuse is blocked and limits are global. Re-read it before launch.
|
|
135
|
+
|
|
136
|
+
## Open decisions for the owner
|
|
137
|
+
|
|
138
|
+
1. Spend the one-time 10 credits on a dedicated OpenRouter account (Option B), or start on Option A with 15 models?
|
|
139
|
+
2. A separate `free-model-index` repo, or a folder inside the Lodestone repo?
|
|
140
|
+
3. Data licence: CC BY 4.0 (recommended) or CC0?
|
|
141
|
+
4. First scorecard consumer: ship the Lodestone site page first, or the Naija Eval join first?
|
package/docs/scorecards.md
CHANGED
|
@@ -14,6 +14,26 @@ Two small JSON formats let a model evaluator and Lodestone feed each other witho
|
|
|
14
14
|
|
|
15
15
|
Nothing here is automatic or hidden. A scorecard changes the order models are *tried* in, for tasks you name, and nothing else. A model nobody measured is never dropped, added or moved.
|
|
16
16
|
|
|
17
|
+
## Why Naija Eval is a good evaluator to start with
|
|
18
|
+
|
|
19
|
+
[Naija Eval](https://naija-eval.extrafemi.com) tests Nigeria's own language model, N-ATLaS, against the Llama model it was built from and four hosted models, on Hausa, Igbo and Yoruba. Four things make it a sound first source for Lodestone, and its limits are part of the answer.
|
|
20
|
+
|
|
21
|
+
- **It measures what a price list cannot tell you.** Whether a model can translate between English and Hausa, Igbo or Yoruba, or transcribe speech in those languages, is not something a provider's own claims settle. Naija Eval scores each model on six translation directions (chrF++) and three speech languages (word error rate), and also shows each score ignoring tone marks, so a missing mark can be told apart from a wrong word.
|
|
22
|
+
- **Anyone can re-run it.** It uses public test sets (Google's FLEURS, CC BY 4.0, and Meta's FLORES, CC BY-SA 4.0), one fixed prompt for every model and temperature 0. A script for each step, the results file (`summary.json`) and the scorecard (`scorecard.json`) are public, and repeating the four hosted-model runs cost $1.59. A number that can be repeated can also be checked, and corrected.
|
|
23
|
+
- **It states what it did not do.** Each score rests on 100 items per language and direction, and Naija Eval calls itself a first pass, not a leaderboard. The 4-bit local models may score a point or two below full precision. The test sets are public, so any model may have seen them in training. FLEURS is clean read speech, and street voice notes will score differently.
|
|
24
|
+
- **It publishes every number with what it rests on.** In `model-scorecard/1` each score carries its metric, which way is good, the number of items (`n`) and the day it was measured, so Lodestone never has to guess how much weight a figure deserves.
|
|
25
|
+
|
|
26
|
+
## Why its results are reliable enough for Lodestone
|
|
27
|
+
|
|
28
|
+
"Reliable" here means something narrow. Lodestone uses a scorecard to put models in a better order for a task you name, and it is built so that a wrong number cannot do much harm:
|
|
29
|
+
|
|
30
|
+
- A scorecard only changes the *order* models are tried in. Nothing is added, dropped or called that you did not list.
|
|
31
|
+
- A score resting on fewer than 30 items, or measured more than a year ago, is ignored. Naija Eval's scores rest on 100 items each.
|
|
32
|
+
- A model nobody measured keeps its slot, an explicit `route` is never reordered, and if the scorecard cannot be fetched within 1.5 seconds the order is left alone.
|
|
33
|
+
- The worst a bad number can do is put a slightly worse model first for that task. A failure or a slow answer still moves the call to the next model, as it always does.
|
|
34
|
+
|
|
35
|
+
What it does not tell you: Naija Eval measures translation and speech in Hausa, Igbo and Yoruba, so use it only for tasks you tag that way. It says nothing about extracting a job posting or scoring a CV in English. And one disclosure: Naija Eval and Lodestone are both Extrafemi projects, so the evaluator is not independent of Lodestone's author. The protection is that its method and data are public, so anyone can check it.
|
|
36
|
+
|
|
17
37
|
## `model-scorecard/1`
|
|
18
38
|
|
|
19
39
|
Measured results about models.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@extrafemi-ai/lodestone",
|
|
3
|
-
"version": "0.7.
|
|
3
|
+
"version": "0.7.1",
|
|
4
4
|
"description": "Calls AI models so your tool keeps working when a model is withdrawn, slow, rate-limited or down: failover across providers, self-updating model lists, spend limits.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"repository": {
|