@extrafemi-ai/lodestone 0.3.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +34 -0
- package/dist/errors.d.ts +2 -0
- package/dist/explain.js +2 -0
- package/dist/health.d.ts +39 -0
- package/dist/health.js +73 -0
- package/dist/index.d.ts +3 -1
- package/dist/index.js +1 -0
- package/dist/lodestone.d.ts +30 -1
- package/dist/lodestone.js +119 -24
- package/dist/providers/openrouter.d.ts +2 -0
- package/dist/providers/openrouter.js +10 -2
- package/dist/stats.js +2 -0
- package/dist/types.d.ts +2 -0
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -60,6 +60,10 @@ A step with no key is skipped. A route skips the model lists and the catalogue,
|
|
|
60
60
|
|
|
61
61
|
Keep a customer's own key out of any fallback step: build the route from their key only.
|
|
62
62
|
|
|
63
|
+
### Your own key
|
|
64
|
+
|
|
65
|
+
Pass the customer's OpenRouter key as `apiKey` on a call. It replaces the platform's key, allows the paid fallback, and is never stored. The platform's `fallbacks` are **not** used on a call with a customer's own key (they carry the platform's money), unless a fallback sets `allowWithCallerKey`. Attempts that fail with 401, 402 or 403 are not counted in live stats, because they say nothing about the model.
|
|
66
|
+
|
|
63
67
|
### Phases
|
|
64
68
|
|
|
65
69
|
For each call, in order, until one answers:
|
|
@@ -83,6 +87,36 @@ const { ok, withdrawn, shortTiers } = await ai.checkModels();
|
|
|
83
87
|
|
|
84
88
|
`withdrawn` lists preferred models that have gone. That is harmless, they are already replaced. `ok` is false only when the catalogue can't be read or a tier is left with fewer than two real models. Without a daily run, a request that finds the saved catalogue over a day old refreshes it in the background.
|
|
85
89
|
|
|
90
|
+
### Free models that are not worth waiting for (`paidFirst`)
|
|
91
|
+
|
|
92
|
+
Free models can be down for hours (rate-limited, or holding a connection open for 45 seconds and never answering). On the platform's own key, `openrouter: { paidFirst: true }` tries the low-cost paid model first and the free ones only if it fails or the spend limit is reached, so a request is not slowed by free models that are not going to answer. A caller's own key is not affected: the paid models are their money, so their free models still come first. Reasoning is set per model with `openrouter({ reasoning: (model) => ... })`, because models differ: one thinks for a thousand tokens unless told not to (measured: turning it off halved scoring time, 10 s to 5 s, with the same answers), another refuses to run with reasoning off.
|
|
93
|
+
|
|
94
|
+
### Not waiting on a model that is down (circuit breaker)
|
|
95
|
+
|
|
96
|
+
Free models fail in bursts: rate-limited, stuck, or answering nothing. Without memory, every request pays for the same dead models again. `createHealth()` remembers:
|
|
97
|
+
|
|
98
|
+
```ts
|
|
99
|
+
import { createHealth } from '@extrafemi-ai/lodestone';
|
|
100
|
+
const health = createHealth({ store }); // the same store as the catalogue: every serverless function shares it
|
|
101
|
+
const ai = createLodestone({ /* ... */, health });
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
A model whose last three attempts (within ten minutes) all failed is skipped, so the next request goes straight to one that works. Every two minutes one attempt is let through to see whether it has recovered, and one success closes it again. A timeout, a rate limit, a server error and an unusable answer count as failures; a rejected key or missing credit (401, 402, 403) say nothing about the model and never do. If every free model looks down and a paid one can follow, the free ones are skipped entirely; if nothing can follow, they are still tried, because trying is better than not answering. `health.snapshot()` lists what it has seen, for a health page.
|
|
105
|
+
|
|
106
|
+
### Limiting what the platform spends
|
|
107
|
+
|
|
108
|
+
```ts
|
|
109
|
+
const ai = createLodestone({
|
|
110
|
+
/* ... */
|
|
111
|
+
spend: {
|
|
112
|
+
allow: async ({ model, phase }) => (await spentToday()) < DAILY_LIMIT_USD,
|
|
113
|
+
record: async ({ costUsd }) => addToday(costUsd ?? 0),
|
|
114
|
+
},
|
|
115
|
+
});
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
`allow` is asked before each attempt that spends the platform's money (the paid models and fallback providers on the platform's keys), and a `false` means that model is not used. `record` is told what each such call cost: the provider's own figure when it gives one (OpenRouter does), otherwise an estimate from the model's catalogue price. Free models and a caller's own key (their money, not yours) never go through either. When the limit stops every paid option the error is `kind: 'budget'`. A `record` that throws never fails the answer.
|
|
119
|
+
|
|
86
120
|
### Letting measurements decide the order
|
|
87
121
|
|
|
88
122
|
A scorecard (a small JSON file of measured results) can put the models that did best on your kind of task first, and Lodestone can publish its own live success rates and speeds in the same format. [Naija Eval](https://naija-eval.extrafemi.com) publishes translation and speech quality for Hausa, Igbo and Yoruba this way. It is opt-in, per task, and only changes the order models are tried in. See [docs/scorecards.md](docs/scorecards.md).
|
package/dist/errors.d.ts
CHANGED
|
@@ -22,6 +22,8 @@ export type FailureKind =
|
|
|
22
22
|
| 'unavailable'
|
|
23
23
|
/** A model answered but the answer was not usable (empty, not valid JSON, failed validation). */
|
|
24
24
|
| 'bad_output'
|
|
25
|
+
/** The platform's spend limit has been reached, so no paid model was used. */
|
|
26
|
+
| 'budget'
|
|
25
27
|
/** No provider has a key. */
|
|
26
28
|
| 'not_configured' | 'other';
|
|
27
29
|
/** Thrown when no model could answer. `kind` says why, so the caller can decide what
|
package/dist/explain.js
CHANGED
|
@@ -42,6 +42,8 @@ export function explain(err, opts = {}) {
|
|
|
42
42
|
return "The AI gave back something that couldn't be read. Try again; it usually works on a second attempt.";
|
|
43
43
|
case 'credit':
|
|
44
44
|
return 'The AI account has no credit for the backup model. Add a little credit and try again.';
|
|
45
|
+
case 'budget':
|
|
46
|
+
return `${product}'s AI budget for now is used up, and the free models are busy. Try again later${opts.settingsHint ? `, or add your own OpenRouter key ${opts.settingsHint.replace(/^in /, 'in ')} to keep going` : ''}.`;
|
|
45
47
|
case 'not_configured':
|
|
46
48
|
return `${product}'s AI isn't set up yet.`;
|
|
47
49
|
default:
|
package/dist/health.d.ts
ADDED
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
import type { KeyValueStore } from './types.js';
|
|
2
|
+
/** A circuit breaker: remembers which models have been failing, so a request does not spend its
|
|
3
|
+
* time on a model that is down. A model whose last few attempts all failed is skipped; every
|
|
4
|
+
* couple of minutes one attempt is let through to see whether it has recovered, and one success
|
|
5
|
+
* closes the breaker again. With a store, every instance (every serverless function) shares what
|
|
6
|
+
* it has learned, so one request's failure saves the next request's time.
|
|
7
|
+
*
|
|
8
|
+
* Counts a timeout, a rate limit, a server error and an unusable answer (empty, not JSON) as
|
|
9
|
+
* failures. A rejected key or missing credit (401, 402, 403) says nothing about the model and is
|
|
10
|
+
* never recorded. */
|
|
11
|
+
export interface HealthOptions {
|
|
12
|
+
/** Where to keep what it has learned, shared across instances. Default: memory only. */
|
|
13
|
+
store?: KeyValueStore;
|
|
14
|
+
key?: string;
|
|
15
|
+
/** A model is skipped when its last `failures` attempts all failed within `windowMs`. Default 3 and 10 minutes. */
|
|
16
|
+
failures?: number;
|
|
17
|
+
windowMs?: number;
|
|
18
|
+
/** While a model is skipped, one attempt is let through this often. Default 2 minutes. */
|
|
19
|
+
probeEveryMs?: number;
|
|
20
|
+
now?: () => number;
|
|
21
|
+
}
|
|
22
|
+
export interface ModelHealth {
|
|
23
|
+
provider: string;
|
|
24
|
+
model: string;
|
|
25
|
+
/** Skipped right now. */
|
|
26
|
+
open: boolean;
|
|
27
|
+
/** Failures among its last few attempts. */
|
|
28
|
+
recentFailures: number;
|
|
29
|
+
recentAttempts: number;
|
|
30
|
+
lastFailureAt: string | null;
|
|
31
|
+
}
|
|
32
|
+
export interface Health {
|
|
33
|
+
/** Should this model be skipped right now? */
|
|
34
|
+
shouldSkip(provider: string, model: string): Promise<boolean>;
|
|
35
|
+
record(provider: string, model: string, ok: boolean): Promise<void>;
|
|
36
|
+
/** Every model it has seen, for a health page. Models not seen in the window are left out. */
|
|
37
|
+
snapshot(): Promise<ModelHealth[]>;
|
|
38
|
+
}
|
|
39
|
+
export declare function createHealth(opts?: HealthOptions): Health;
|
package/dist/health.js
ADDED
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
const KEEP = 6;
|
|
2
|
+
const SHARE_MS = 15_000;
|
|
3
|
+
const id = (provider, model) => `${provider}\u0000${model}`;
|
|
4
|
+
export function createHealth(opts = {}) {
|
|
5
|
+
const need = opts.failures ?? 3;
|
|
6
|
+
const windowMs = opts.windowMs ?? 10 * 60_000;
|
|
7
|
+
const probeMs = opts.probeEveryMs ?? 2 * 60_000;
|
|
8
|
+
const now = opts.now ?? Date.now;
|
|
9
|
+
const key = opts.key ?? 'lodestone:health';
|
|
10
|
+
let state = {};
|
|
11
|
+
let loadedAt = -Infinity;
|
|
12
|
+
/** Brings in what other instances have saved, at most every 15 seconds. */
|
|
13
|
+
async function refresh() {
|
|
14
|
+
if (!opts.store || now() - loadedAt < SHARE_MS)
|
|
15
|
+
return;
|
|
16
|
+
loadedAt = now();
|
|
17
|
+
try {
|
|
18
|
+
state = merge(state, (await opts.store.get(key)) ?? {});
|
|
19
|
+
}
|
|
20
|
+
catch { /* an unreadable store is the same as an empty one */ }
|
|
21
|
+
}
|
|
22
|
+
const merge = (a, b) => {
|
|
23
|
+
const out = {};
|
|
24
|
+
for (const k of new Set([...Object.keys(a), ...Object.keys(b)])) {
|
|
25
|
+
const seen = new Set();
|
|
26
|
+
out[k] = [...(a[k] ?? []), ...(b[k] ?? [])].filter(([at, , sid]) => { const u = `${at}:${sid}`; return seen.has(u) ? false : (seen.add(u), true); }).sort((x, y) => x[0] - y[0]).slice(-KEEP);
|
|
27
|
+
}
|
|
28
|
+
return out;
|
|
29
|
+
};
|
|
30
|
+
const recent = (k) => (state[k] ?? []).filter(([at]) => now() - at <= windowMs);
|
|
31
|
+
const isOpen = (r) => {
|
|
32
|
+
const last = r.slice(-need);
|
|
33
|
+
if (last.length < need || last.some(([, ok]) => ok === 1))
|
|
34
|
+
return false;
|
|
35
|
+
return now() - last[last.length - 1][0] < probeMs; // after a while, let one attempt through to see if it has recovered
|
|
36
|
+
};
|
|
37
|
+
return {
|
|
38
|
+
async shouldSkip(provider, model) {
|
|
39
|
+
await refresh();
|
|
40
|
+
return isOpen(recent(id(provider, model)));
|
|
41
|
+
},
|
|
42
|
+
async record(provider, model, ok) {
|
|
43
|
+
const k = id(provider, model);
|
|
44
|
+
const before = recent(k);
|
|
45
|
+
const recovering = before.length > 0 && before[before.length - 1][1] === 0; // this success follows a failure
|
|
46
|
+
state[k] = [...(state[k] ?? []), [now(), ok ? 1 : 0, Math.random().toString(36).slice(2, 8)]].slice(-KEEP);
|
|
47
|
+
// Only failures, and a success right after a failure, change what anyone else needs to know;
|
|
48
|
+
// a model that is simply working costs nothing extra.
|
|
49
|
+
if (!opts.store || (ok && !recovering))
|
|
50
|
+
return;
|
|
51
|
+
try {
|
|
52
|
+
const theirs = (await opts.store.get(key)) ?? {};
|
|
53
|
+
state = merge(state, theirs);
|
|
54
|
+
await opts.store.set(key, state);
|
|
55
|
+
loadedAt = now();
|
|
56
|
+
}
|
|
57
|
+
catch { /* not saved: the next instance just has to learn it again */ }
|
|
58
|
+
},
|
|
59
|
+
async snapshot() {
|
|
60
|
+
await refresh();
|
|
61
|
+
const out = [];
|
|
62
|
+
for (const k of Object.keys(state)) {
|
|
63
|
+
const r = recent(k);
|
|
64
|
+
if (!r.length)
|
|
65
|
+
continue;
|
|
66
|
+
const [provider, model] = k.split('\u0000');
|
|
67
|
+
const fails = r.filter(([, ok]) => ok === 0);
|
|
68
|
+
out.push({ provider, model, open: isOpen(r), recentFailures: fails.length, recentAttempts: r.length, lastFailureAt: fails.length ? new Date(fails[fails.length - 1][0]).toISOString() : null });
|
|
69
|
+
}
|
|
70
|
+
return out;
|
|
71
|
+
},
|
|
72
|
+
};
|
|
73
|
+
}
|
package/dist/index.d.ts
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
export { createLodestone } from './lodestone.js';
|
|
2
|
-
export type { Lodestone, LodestoneConfig, CompleteOptions, CompleteResult, Profile, FallbackConfig, AttemptEvent, RouteStep } from './lodestone.js';
|
|
2
|
+
export type { Lodestone, LodestoneConfig, CompleteOptions, CompleteResult, Profile, FallbackConfig, AttemptEvent, RouteStep, SpendControl } from './lodestone.js';
|
|
3
3
|
export { LodestoneError, ProviderError, messageFrom } from './errors.js';
|
|
4
4
|
export type { FailureKind } from './errors.js';
|
|
5
5
|
export { explain } from './explain.js';
|
|
@@ -22,3 +22,5 @@ export { createStats } from './stats.js';
|
|
|
22
22
|
export type { Stats, StatsOptions, StatsState } from './stats.js';
|
|
23
23
|
export { CANDIDATES_SCHEMA } from './candidates.js';
|
|
24
24
|
export type { Candidates, Candidate } from './candidates.js';
|
|
25
|
+
export { createHealth } from './health.js';
|
|
26
|
+
export type { Health, HealthOptions, ModelHealth } from './health.js';
|
package/dist/index.js
CHANGED
|
@@ -13,3 +13,4 @@ export { mock } from './providers/mock.js';
|
|
|
13
13
|
export { SCORECARD_SCHEMA, ScorecardError, parseScorecard, fetchScorecard, rankModels, scoreFor, baseModelId } from './scorecard.js';
|
|
14
14
|
export { createStats } from './stats.js';
|
|
15
15
|
export { CANDIDATES_SCHEMA } from './candidates.js';
|
|
16
|
+
export { createHealth } from './health.js';
|
package/dist/lodestone.d.ts
CHANGED
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import { type CatalogModel } from './catalog.js';
|
|
2
2
|
import { type Candidates } from './candidates.js';
|
|
3
|
+
import type { Health } from './health.js';
|
|
3
4
|
import { type QualityQuery, type Scorecard } from './scorecard.js';
|
|
4
5
|
import type { Stats } from './stats.js';
|
|
5
6
|
import type { KeyValueStore, Message, Provider, Tier, Usage } from './types.js';
|
|
@@ -15,21 +16,44 @@ export interface FallbackConfig {
|
|
|
15
16
|
apiKey: string;
|
|
16
17
|
/** Tried in order. */
|
|
17
18
|
models: string[];
|
|
19
|
+
/** Also use this provider when the caller supplied their own OpenRouter key. Default false: a fallback has the platform's key, and a caller's own key must never lead to the platform's money being spent. */
|
|
20
|
+
allowWithCallerKey?: boolean;
|
|
18
21
|
}
|
|
19
22
|
export interface AttemptEvent {
|
|
20
23
|
provider: string;
|
|
21
24
|
model: string;
|
|
22
25
|
phase: 'free' | 'paid' | 'fallback' | 'route';
|
|
23
|
-
|
|
26
|
+
/** 'skipped': not tried, because the breaker had it down or the spend limit was reached. */
|
|
27
|
+
outcome: 'ok' | 'timeout' | 'failed' | 'skipped';
|
|
24
28
|
status?: number;
|
|
25
29
|
ms: number;
|
|
26
30
|
}
|
|
31
|
+
/** Where the platform's money is counted. Only attempts that spend it go through: the paid models and the
|
|
32
|
+
* fallback providers on the platform's keys. A call on the caller's own key never does. */
|
|
33
|
+
export interface SpendControl {
|
|
34
|
+
/** Asked before each such attempt. Return false once the limit is reached and that model is not used. */
|
|
35
|
+
allow(info: {
|
|
36
|
+
provider: string;
|
|
37
|
+
model: string;
|
|
38
|
+
phase: 'paid' | 'fallback';
|
|
39
|
+
}): boolean | Promise<boolean>;
|
|
40
|
+
/** Told after each such attempt that answered, with what it cost: the provider's own figure when it gives one (OpenRouter does), otherwise an estimate from the model's catalogue price, otherwise nothing. */
|
|
41
|
+
record(info: {
|
|
42
|
+
provider: string;
|
|
43
|
+
model: string;
|
|
44
|
+
phase: 'paid' | 'fallback';
|
|
45
|
+
usage?: Usage;
|
|
46
|
+
costUsd?: number;
|
|
47
|
+
}): void | Promise<void>;
|
|
48
|
+
}
|
|
27
49
|
export interface LodestoneConfig {
|
|
28
50
|
openrouter?: {
|
|
29
51
|
/** The platform's key. A caller's own key (`apiKey` on a call) takes priority. */
|
|
30
52
|
apiKey?: string;
|
|
31
53
|
/** Use the paid models after the free ones fail, on the platform's key. A caller's own key always allows it. */
|
|
32
54
|
allowPaid?: boolean;
|
|
55
|
+
/** On the platform's key, try the paid models first and the free ones only if they fail or the spend limit is reached. For when the free models are too unreliable to be worth waiting for. Not applied to a caller's own key: it is their money, so their free models are still tried first. */
|
|
56
|
+
paidFirst?: boolean;
|
|
33
57
|
models?: Partial<Record<Tier, string[]>>;
|
|
34
58
|
paidModels?: string[];
|
|
35
59
|
/** Replace the OpenRouter adapter (tests, proxies). */
|
|
@@ -44,6 +68,10 @@ export interface LodestoneConfig {
|
|
|
44
68
|
* nobody has measured keep their place. A function is called per request, so it can fetch
|
|
45
69
|
* and cache; if it fails or takes over 1.5 s the order is left alone. */
|
|
46
70
|
scorecards?: Scorecard[] | (() => Promise<Scorecard[]>);
|
|
71
|
+
/** Skips models that have been failing, so a request does not spend its time on them (see health.ts). */
|
|
72
|
+
health?: Health;
|
|
73
|
+
/** Counts and limits the platform's spend. Without it nothing is counted or limited. */
|
|
74
|
+
spend?: SpendControl;
|
|
47
75
|
/** Records every attempt so you can publish how each model does (stats.scorecard()). */
|
|
48
76
|
stats?: Stats;
|
|
49
77
|
/** Named tasks, so calls say what they are doing and not which model to use. */
|
|
@@ -58,6 +86,7 @@ export interface LodestoneConfig {
|
|
|
58
86
|
freeSliceWithLaterMs?: number;
|
|
59
87
|
freePhaseWithLaterMs?: number;
|
|
60
88
|
minSliceMs?: number;
|
|
89
|
+
paidSliceMs?: number;
|
|
61
90
|
};
|
|
62
91
|
fetch?: typeof fetch;
|
|
63
92
|
now?: () => number;
|
package/dist/lodestone.js
CHANGED
|
@@ -17,6 +17,7 @@ export function createLodestone(config = {}) {
|
|
|
17
17
|
freeSliceLater: config.timing?.freeSliceWithLaterMs ?? 5_000,
|
|
18
18
|
freePhaseLater: config.timing?.freePhaseWithLaterMs ?? 9_000,
|
|
19
19
|
minSlice: config.timing?.minSliceMs ?? 3_000,
|
|
20
|
+
paidSlice: config.timing?.paidSliceMs ?? 12_000,
|
|
20
21
|
};
|
|
21
22
|
const now = config.now ?? Date.now;
|
|
22
23
|
const store = config.store ?? memoryStore();
|
|
@@ -50,6 +51,15 @@ export function createLodestone(config = {}) {
|
|
|
50
51
|
return [];
|
|
51
52
|
}
|
|
52
53
|
}
|
|
54
|
+
/** What a call cost, estimated from the model's catalogue price, for providers that do not report it. */
|
|
55
|
+
async function estimateCost(model, usage) {
|
|
56
|
+
if (!usage)
|
|
57
|
+
return undefined;
|
|
58
|
+
const m = (await catalog.load().catch(() => null))?.find((x) => x.id === model);
|
|
59
|
+
const inP = Number(m?.pricing?.prompt);
|
|
60
|
+
const outP = Number(m?.pricing?.completion);
|
|
61
|
+
return Number.isFinite(inP) && Number.isFinite(outP) ? usage.inputTokens * inP + usage.outputTokens * outP : undefined;
|
|
62
|
+
}
|
|
53
63
|
async function complete(opts) {
|
|
54
64
|
const messages = opts.messages ?? (opts.prompt !== undefined ? [{ role: 'user', content: opts.prompt }] : []);
|
|
55
65
|
if (messages.length === 0)
|
|
@@ -76,7 +86,7 @@ export function createLodestone(config = {}) {
|
|
|
76
86
|
const left = () => budget - (now() - started);
|
|
77
87
|
const orKey = opts.apiKey || config.openrouter?.apiKey;
|
|
78
88
|
const paidAllowed = Boolean(opts.apiKey) || Boolean(config.openrouter?.allowPaid);
|
|
79
|
-
const fbs = fallbacks.filter((f) => f.apiKey && (!images || f.provider.vision));
|
|
89
|
+
const fbs = fallbacks.filter((f) => f.apiKey && (!images || f.provider.vision) && (!opts.apiKey || f.allowWithCallerKey));
|
|
80
90
|
const paidAvailable = Boolean(orKey) && paidAllowed && paidModels.length > 0;
|
|
81
91
|
const later = paidAvailable || fbs.length > 0;
|
|
82
92
|
const freeEnds = later ? t.freePhaseLater : budget;
|
|
@@ -86,6 +96,31 @@ export function createLodestone(config = {}) {
|
|
|
86
96
|
let rateLimited = null;
|
|
87
97
|
let authFailed = false;
|
|
88
98
|
let needsCredit = false;
|
|
99
|
+
let spendBlocked = false;
|
|
100
|
+
const callerKey = Boolean(opts.apiKey);
|
|
101
|
+
/** Drops the models the breaker has down, saying so. */
|
|
102
|
+
const healthy = async (provider, list, phase) => {
|
|
103
|
+
if (!config.health)
|
|
104
|
+
return list;
|
|
105
|
+
const keep = [];
|
|
106
|
+
for (const m of list) {
|
|
107
|
+
if (await config.health.shouldSkip(provider.name, m))
|
|
108
|
+
config.onAttempt?.({ provider: provider.name, model: m, phase, outcome: 'skipped', ms: 0 });
|
|
109
|
+
else
|
|
110
|
+
keep.push(m);
|
|
111
|
+
}
|
|
112
|
+
return keep;
|
|
113
|
+
};
|
|
114
|
+
/** May the platform's money be spent on this attempt? A caller's own key is their money, not ours. */
|
|
115
|
+
const mayUse = async (provider, model, phase) => {
|
|
116
|
+
if (callerKey || !config.spend)
|
|
117
|
+
return true;
|
|
118
|
+
if (await config.spend.allow({ provider: provider.name, model, phase }))
|
|
119
|
+
return true;
|
|
120
|
+
spendBlocked = true;
|
|
121
|
+
config.onAttempt?.({ provider: provider.name, model, phase, outcome: 'skipped', ms: 0 });
|
|
122
|
+
return false;
|
|
123
|
+
};
|
|
89
124
|
const attempt = async (provider, apiKey, model, phase, ms) => {
|
|
90
125
|
attempted = true;
|
|
91
126
|
const at = now();
|
|
@@ -109,13 +144,26 @@ export function createLodestone(config = {}) {
|
|
|
109
144
|
out = { kind: 'fail', status: 599, body: e.message };
|
|
110
145
|
}
|
|
111
146
|
const event = { provider: provider.name, model, phase, outcome: out.kind === 'ok' ? 'ok' : out.kind === 'timeout' ? 'timeout' : 'failed', status: out.kind === 'fail' ? out.status : undefined, ms: now() - at };
|
|
112
|
-
|
|
147
|
+
// A key or credit problem (401, 402, 403) says nothing about the model, so it is not counted against it.
|
|
148
|
+
const credential = out.kind === 'fail' && [401, 402, 403].includes(out.status);
|
|
149
|
+
if (!credential) {
|
|
150
|
+
config.stats?.record(event);
|
|
151
|
+
await config.health?.record(provider.name, model, out.kind === 'ok');
|
|
152
|
+
}
|
|
113
153
|
config.onAttempt?.(event);
|
|
114
154
|
last = out;
|
|
115
155
|
return out;
|
|
116
156
|
};
|
|
117
|
-
const finish = async (provider, model, reply) => {
|
|
157
|
+
const finish = async (provider, model, reply, phase) => {
|
|
118
158
|
const result = { text: reply.text, provider: provider.name, model, usage: reply.usage };
|
|
159
|
+
if (phase && !callerKey && config.spend) {
|
|
160
|
+
const costUsd = reply.usage?.costUsd ?? (await estimateCost(model, reply.usage));
|
|
161
|
+
// A ledger that fails must never fail an answer the person is waiting for.
|
|
162
|
+
try {
|
|
163
|
+
await config.spend.record({ provider: provider.name, model, phase, usage: reply.usage, costUsd });
|
|
164
|
+
}
|
|
165
|
+
catch { /* not counted */ }
|
|
166
|
+
}
|
|
119
167
|
if (cacheSlot)
|
|
120
168
|
await store.set(cacheSlot, { ...result, at: now() }).catch(() => { });
|
|
121
169
|
return result;
|
|
@@ -148,11 +196,19 @@ export function createLodestone(config = {}) {
|
|
|
148
196
|
throw new LodestoneError('slow');
|
|
149
197
|
throw classify(firstFail ?? (lastRoute?.kind === 'fail' ? lastRoute : null), Boolean(opts.apiKey), false, false);
|
|
150
198
|
}
|
|
151
|
-
//
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
199
|
+
// OpenRouter: the free models and the low-cost paid ones. Free first by default; paid first when the
|
|
200
|
+
// platform's key is used and the free models are not worth waiting for (see paidFirst).
|
|
201
|
+
const paidFirst = Boolean(config.openrouter?.paidFirst) && !callerKey && paidAvailable;
|
|
202
|
+
/** The free models. A withdrawn one has already been replaced by resolveModels. */
|
|
203
|
+
const runFree = async () => {
|
|
204
|
+
const all = await order(await resolveModels(tier));
|
|
205
|
+
let models = await healthy(router, all, 'free');
|
|
206
|
+
// With nothing after the free models to fall back on, a model that may be down is still better than no attempt.
|
|
207
|
+
if (models.length === 0 && !later && !paidFirst)
|
|
208
|
+
models = all;
|
|
209
|
+
const leftHere = () => (paidFirst ? left() : freeLeft());
|
|
210
|
+
for (let i = 0; i < models.length && leftHere() >= t.minSlice; i++) {
|
|
211
|
+
const slice = i >= models.length - 1 ? leftHere() : Math.min(later && !paidFirst ? t.freeSliceLater : t.freeSlice, leftHere());
|
|
156
212
|
const out = await attempt(router, orKey, models[i], 'free', slice);
|
|
157
213
|
if (out.kind === 'ok')
|
|
158
214
|
return finish(router, models[i], out.reply);
|
|
@@ -165,39 +221,78 @@ export function createLodestone(config = {}) {
|
|
|
165
221
|
rateLimited = out;
|
|
166
222
|
}
|
|
167
223
|
}
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
224
|
+
return null;
|
|
225
|
+
};
|
|
226
|
+
/** A low-cost paid model, which is not under the free-model limits. */
|
|
227
|
+
const runPaid = async () => {
|
|
228
|
+
const paidAll = await order(paidModels);
|
|
229
|
+
let paidList = await healthy(router, paidAll, 'paid');
|
|
230
|
+
if (paidList.length === 0)
|
|
231
|
+
paidList = paidAll; // never leave no paid attempt because of the breaker
|
|
232
|
+
for (let i = 0; i < paidList.length; i++) {
|
|
233
|
+
const m = paidList[i];
|
|
234
|
+
if (left() < t.minSlice)
|
|
235
|
+
break;
|
|
236
|
+
if (!(await mayUse(router, m, 'paid')))
|
|
237
|
+
continue;
|
|
238
|
+
// Going first, a paid model may not use all the time: the free models and the rest need some.
|
|
239
|
+
const ms = paidFirst && i < paidList.length - 1 ? Math.min(left(), t.paidSlice) : left();
|
|
240
|
+
const out = await attempt(router, orKey, m, 'paid', ms);
|
|
241
|
+
if (out.kind === 'ok')
|
|
242
|
+
return finish(router, m, out.reply, 'paid');
|
|
243
|
+
if (out.kind === 'fail') {
|
|
244
|
+
if (out.status === 402) {
|
|
245
|
+
needsCredit = true;
|
|
172
246
|
break;
|
|
173
|
-
const out = await attempt(router, orKey, m, 'paid', left());
|
|
174
|
-
if (out.kind === 'ok')
|
|
175
|
-
return finish(router, m, out.reply);
|
|
176
|
-
if (out.kind === 'fail') {
|
|
177
|
-
if (out.status === 402) {
|
|
178
|
-
needsCredit = true;
|
|
179
|
-
break;
|
|
180
|
-
}
|
|
181
|
-
if (out.status === 401)
|
|
182
|
-
break;
|
|
183
247
|
}
|
|
248
|
+
if (out.status === 401)
|
|
249
|
+
break;
|
|
250
|
+
}
|
|
251
|
+
}
|
|
252
|
+
return null;
|
|
253
|
+
};
|
|
254
|
+
if (orKey) {
|
|
255
|
+
if (paidFirst) {
|
|
256
|
+
const a = await runPaid();
|
|
257
|
+
if (a)
|
|
258
|
+
return a;
|
|
259
|
+
const b = authFailed ? null : await runFree();
|
|
260
|
+
if (b)
|
|
261
|
+
return b;
|
|
262
|
+
}
|
|
263
|
+
else {
|
|
264
|
+
const b = await runFree();
|
|
265
|
+
if (b)
|
|
266
|
+
return b;
|
|
267
|
+
if (paidAvailable && !authFailed) {
|
|
268
|
+
const a = await runPaid();
|
|
269
|
+
if (a)
|
|
270
|
+
return a;
|
|
184
271
|
}
|
|
185
272
|
}
|
|
186
273
|
}
|
|
187
274
|
// Phase 3: other providers, with their own keys.
|
|
188
275
|
for (const fb of fbs) {
|
|
189
|
-
|
|
276
|
+
const fbAll = await order(fb.models);
|
|
277
|
+
let fbList = await healthy(fb.provider, fbAll, 'fallback');
|
|
278
|
+
if (fbList.length === 0)
|
|
279
|
+
fbList = fbAll;
|
|
280
|
+
for (const m of fbList) {
|
|
190
281
|
if (left() < t.minSlice)
|
|
191
282
|
break;
|
|
283
|
+
if (!(await mayUse(fb.provider, m, 'fallback')))
|
|
284
|
+
continue;
|
|
192
285
|
const out = await attempt(fb.provider, fb.apiKey, m, 'fallback', left());
|
|
193
286
|
if (out.kind === 'ok')
|
|
194
|
-
return finish(fb.provider, m, out.reply);
|
|
287
|
+
return finish(fb.provider, m, out.reply, 'fallback');
|
|
195
288
|
if (out.kind === 'fail' && out.status === 401)
|
|
196
289
|
break; // this provider's key is wrong: skip its other models
|
|
197
290
|
}
|
|
198
291
|
}
|
|
199
292
|
// Nothing answered. Say why, using what the first useful failure told us.
|
|
200
293
|
const ownKey = Boolean(opts.apiKey);
|
|
294
|
+
if (spendBlocked)
|
|
295
|
+
throw new LodestoneError('budget');
|
|
201
296
|
if (!attempted)
|
|
202
297
|
throw new LodestoneError('not_configured');
|
|
203
298
|
const lastFail = last;
|
|
@@ -3,6 +3,8 @@ export interface OpenRouterOptions {
|
|
|
3
3
|
fetch?: typeof fetch;
|
|
4
4
|
/** Ask the model not to think at length. Default true: most tasks need an answer, not a long chain of thought, and thinking is what made some free models slower than the time allowed. */
|
|
5
5
|
lowReasoning?: boolean;
|
|
6
|
+
/** How much each model should think, by model id: an OpenRouter `reasoning` setting, or null to send none. Overrides `lowReasoning`. Models differ: some will not run with reasoning off ("mandatory for this endpoint"), some think for thousands of tokens unless told not to, so one setting for every model is wrong. For example `(m) => m.startsWith('anthropic/claude-haiku') ? { enabled: false } : { effort: 'low', exclude: true }`. */
|
|
7
|
+
reasoning?: (model: string) => Record<string, unknown> | null;
|
|
6
8
|
/** Sent as HTTP-Referer and X-Title, which OpenRouter uses to credit apps. */
|
|
7
9
|
referer?: string;
|
|
8
10
|
title?: string;
|
|
@@ -3,6 +3,13 @@ const part = (p) => (p.type === 'text' ? { type: 'text', text: p.text } : { type
|
|
|
3
3
|
const wire = (m) => ({ role: m.role, content: typeof m.content === 'string' ? m.content : m.content.map(part) });
|
|
4
4
|
export function openrouter(opts = {}) {
|
|
5
5
|
const f = opts.fetch ?? fetch;
|
|
6
|
+
const reasoningFor = (model) => {
|
|
7
|
+
if (opts.reasoning) {
|
|
8
|
+
const r = opts.reasoning(model);
|
|
9
|
+
return r ? { reasoning: r } : {};
|
|
10
|
+
}
|
|
11
|
+
return opts.lowReasoning === false ? {} : { reasoning: { effort: 'low', exclude: true } };
|
|
12
|
+
};
|
|
6
13
|
return {
|
|
7
14
|
name: 'openrouter',
|
|
8
15
|
vision: true,
|
|
@@ -17,9 +24,10 @@ export function openrouter(opts = {}) {
|
|
|
17
24
|
},
|
|
18
25
|
body: JSON.stringify({
|
|
19
26
|
model: req.model,
|
|
27
|
+
usage: { include: true }, // asks OpenRouter to report what the call cost
|
|
20
28
|
...(req.temperature === undefined ? {} : { temperature: req.temperature }),
|
|
21
29
|
max_tokens: req.maxTokens,
|
|
22
|
-
...(
|
|
30
|
+
...reasoningFor(req.model),
|
|
23
31
|
messages: [...(req.system ? [{ role: 'system', content: req.system }] : []), ...req.messages.map(wire)],
|
|
24
32
|
}),
|
|
25
33
|
signal: req.signal,
|
|
@@ -29,7 +37,7 @@ export function openrouter(opts = {}) {
|
|
|
29
37
|
const data = (await res.json());
|
|
30
38
|
return {
|
|
31
39
|
text: data.choices?.[0]?.message?.content ?? '',
|
|
32
|
-
usage: data.usage ? { inputTokens: data.usage.prompt_tokens ?? 0, outputTokens: data.usage.completion_tokens ?? 0 } : undefined,
|
|
40
|
+
usage: data.usage ? { inputTokens: data.usage.prompt_tokens ?? 0, outputTokens: data.usage.completion_tokens ?? 0, ...(typeof data.usage.cost === 'number' ? { costUsd: data.usage.cost } : {}) } : undefined,
|
|
33
41
|
};
|
|
34
42
|
},
|
|
35
43
|
};
|
package/dist/stats.js
CHANGED
|
@@ -8,6 +8,8 @@ export function createStats(opts = {}) {
|
|
|
8
8
|
let state = {};
|
|
9
9
|
return {
|
|
10
10
|
record(e) {
|
|
11
|
+
if (e.outcome === 'skipped')
|
|
12
|
+
return; // not tried, so nothing to count
|
|
11
13
|
const k = key(e.provider, e.model);
|
|
12
14
|
const list = (state[k] ??= []);
|
|
13
15
|
list.push({ ok: e.outcome === 'ok', ms: e.ms, at: now() });
|
package/dist/types.d.ts
CHANGED
|
@@ -13,6 +13,8 @@ export interface Message {
|
|
|
13
13
|
export interface Usage {
|
|
14
14
|
inputTokens: number;
|
|
15
15
|
outputTokens: number;
|
|
16
|
+
/** What the provider charged for this call, in US dollars, when it says (OpenRouter does). */
|
|
17
|
+
costUsd?: number;
|
|
16
18
|
}
|
|
17
19
|
/** What a provider adapter is asked to do: one model, one request. */
|
|
18
20
|
export interface ProviderRequest {
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@extrafemi-ai/lodestone",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.4.0",
|
|
4
4
|
"description": "Calls AI models so your tool keeps working when a model is withdrawn, slow or rate-limited.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"repository": {
|
|
@@ -40,7 +40,7 @@
|
|
|
40
40
|
"devDependencies": {
|
|
41
41
|
"@types/node": "^22.10.0",
|
|
42
42
|
"typescript": "^5.6.0",
|
|
43
|
-
"vitest": "^
|
|
43
|
+
"vitest": "^4.1.12"
|
|
44
44
|
},
|
|
45
45
|
"keywords": [
|
|
46
46
|
"ai",
|