@extrafemi-ai/lodestone 0.6.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +21 -0
- package/README.md +36 -14
- package/SECURITY.md +17 -0
- package/dist/catalog.js +11 -2
- package/dist/errors.d.ts +6 -0
- package/dist/errors.js +1 -2
- package/dist/explain.js +4 -0
- package/dist/health.js +19 -3
- package/dist/index.d.ts +9 -3
- package/dist/index.js +3 -0
- package/dist/limiter.d.ts +34 -0
- package/dist/limiter.js +35 -0
- package/dist/lodestone.d.ts +25 -12
- package/dist/lodestone.js +56 -10
- package/dist/scorecard.d.ts +9 -3
- package/dist/scorecard.js +45 -2
- package/dist/spend.d.ts +33 -0
- package/dist/spend.js +72 -0
- package/dist/store.d.ts +2 -2
- package/dist/store.js +19 -3
- package/dist/stores/cloudflare.d.ts +10 -0
- package/dist/stores/cloudflare.js +11 -0
- package/dist/stores/deno.d.ts +14 -0
- package/dist/stores/deno.js +8 -0
- package/dist/stores/index.d.ts +9 -0
- package/dist/stores/index.js +5 -0
- package/dist/stores/netlify.d.ts +12 -0
- package/dist/stores/netlify.js +9 -0
- package/dist/stores/safeKey.d.ts +10 -0
- package/dist/stores/safeKey.js +20 -0
- package/dist/stores/upstash.d.ts +14 -0
- package/dist/stores/upstash.js +50 -0
- package/dist/types.d.ts +15 -3
- package/dist/util.d.ts +7 -0
- package/dist/util.js +17 -0
- package/docs/protecting-your-budget.md +48 -0
- package/docs/runtimes.md +19 -0
- package/docs/stores.md +45 -0
- package/examples/netlify-function.ts +10 -7
- package/package.json +5 -2
|
@@ -0,0 +1,48 @@
|
|
|
1
|
+
# Protecting your budget on a public tool
|
|
2
|
+
|
|
3
|
+
A free public tool that calls a paid model has one big risk: somebody finds the endpoint and calls it in a loop. Lodestone gives you three layers, from the one that stops one person to the one that stops everyone.
|
|
4
|
+
|
|
5
|
+
## 1. Per caller: `rateLimit`
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
import { createLodestone, createRateLimiter, upstashStore } from '@extrafemi-ai/lodestone';
|
|
9
|
+
|
|
10
|
+
const store = upstashStore({ url: process.env.UPSTASH_URL!, token: process.env.UPSTASH_TOKEN! });
|
|
11
|
+
const ai = createLodestone({
|
|
12
|
+
/* ... */
|
|
13
|
+
rateLimit: {
|
|
14
|
+
limiter: createRateLimiter({ store, max: 20, windowMs: 3_600_000 }), // 20 calls an hour each
|
|
15
|
+
whenOver: 'free-only', // or 'reject' (the default)
|
|
16
|
+
},
|
|
17
|
+
});
|
|
18
|
+
|
|
19
|
+
await ai.complete({ prompt, caller: userId ?? clientIp });
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
- A call that names a `caller` counts against that caller's allowance. A call with no `caller`, a call on the caller's own key (their money) and an explicit `route` are not limited. **Pass `caller` on every public call.**
|
|
23
|
+
- Over the allowance, `whenOver: 'reject'` fails with `kind: 'throttled'` and `err.detail.retryAfterMs`; `explain(err)` says "Try again in about N minutes". `'free-only'` still answers, from free models only, so the abuser can never cost you anything.
|
|
24
|
+
- Only a SHA-256 hash of the caller is stored, never the caller, so an IP address does not end up in your store.
|
|
25
|
+
- Exactness depends on the store. With `upstashStore` (or `memoryStore` in one process) the count is atomic. With a plain get/set store it is exact inside one process and approximate between processes: two functions answering the same caller at the same moment can let a call or two through. For a tool that is attacked, use Upstash.
|
|
26
|
+
- If the store cannot be reached the limiter lets the call through by default (`onStoreError: 'allow'`) so a broken store does not take your tool down; the spend limit below is the backstop. Set `'deny'` to fail closed.
|
|
27
|
+
|
|
28
|
+
## 2. Everyone together: `createSpendLimit`
|
|
29
|
+
|
|
30
|
+
```ts
|
|
31
|
+
const spend = createSpendLimit({ store, dailyUsd: 2, monthlyUsd: 30, holdUsd: 0.01 });
|
|
32
|
+
const ai = createLodestone({ /* ... */, spend });
|
|
33
|
+
await spend.report(); // { todayUsd, monthUsd, heldUsd, ... }
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
A plain "is the total under the limit?" check lets a burst through: fifty requests all read the same total, all pass, all spend. `createSpendLimit` takes a **hold** (`holdUsd`, roughly the most one call can cost) for each paid attempt when it is allowed, and counts holds against the limit, so concurrent requests see each other. The hold is replaced by the real cost when the attempt answers and given back when it does not; a hold that is never settled (the function was killed) expires after two minutes.
|
|
37
|
+
|
|
38
|
+
Be honest about the limits of it: inside one process it is exact. Between processes, with a get/set store, two functions can each take a hold the other did not see, so leave headroom (set the limit a little under what you can afford). If the ledger cannot be read, the paid attempt is refused.
|
|
39
|
+
|
|
40
|
+
Bring your own: `spend` is just `{ allow, record, release? }`, and `allow`/`record`/`release` receive an `id` that names one attempt so you can do the same with a database that has transactions.
|
|
41
|
+
|
|
42
|
+
## 3. The ceiling on one call
|
|
43
|
+
|
|
44
|
+
`maxTokens` (3,500 by default) bounds what one call can produce, and `budgetMs` bounds how long it can run. Set `maxTokens` per task (`profiles`) to what the task needs.
|
|
45
|
+
|
|
46
|
+
## What this does not do
|
|
47
|
+
|
|
48
|
+
Lodestone cannot tell a real person from a script: `caller` is whatever you pass. If your callers are anonymous, an IP address is a weak identity (shared networks, rotating addresses). For a tool that is attacked, put a bot check (a CAPTCHA or a signed token) in front of the endpoint, and treat the layers above as the part that limits the damage when one gets through.
|
package/docs/runtimes.md
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
# Runtimes
|
|
2
|
+
|
|
3
|
+
Lodestone uses only web-standard APIs (`fetch`, `Response`, `AbortSignal`, `URL`, `crypto.subtle`, `TextEncoder`) and has no dependencies, so it is not tied to Node.
|
|
4
|
+
|
|
5
|
+
| Runtime | How it is checked | Where |
|
|
6
|
+
|---|---|---|
|
|
7
|
+
| Node 20, 22 and 24 | the full test suite and the smoke checks | `npm test`, `npm run smoke` |
|
|
8
|
+
| Cloudflare Workers | the smoke checks run inside `workerd` through Miniflare, **without** the `nodejs_compat` flag | `npm run smoke:workers` |
|
|
9
|
+
| Deno | the smoke checks | `deno run --allow-read scripts/smoke.mjs` |
|
|
10
|
+
| Bun | the smoke checks | `bun scripts/smoke.mjs` |
|
|
11
|
+
|
|
12
|
+
All four run in CI on every push. The smoke checks (`scripts/smoke-core.mjs`) are the same on every runtime: failover, cutting off a model that hangs, `json()`, `explain()`, the circuit breaker, the spend limit under a burst, per-caller rate limiting, private calls not being cached, and the Upstash, Workers KV and scorecard code paths. They use fake providers, so they need no network and no keys.
|
|
13
|
+
|
|
14
|
+
This shows the library's code runs on those runtimes. It does not show that each provider's API behaves the same from each platform's network, which only your own deployment can tell you.
|
|
15
|
+
|
|
16
|
+
## Things to know on Workers
|
|
17
|
+
|
|
18
|
+
- Use `cloudflareKvStore(env.KV)` for the catalogue and cached answers. KV is eventually consistent and about one write per second per key, so use `upstashStore` for rate-limit and spend counters.
|
|
19
|
+
- A Worker has CPU and subrequest limits. A call that tries three models and two providers makes up to five subrequests; keep `ACTIVE` small, or set `budgetMs` under your plan's wall-clock limit.
|
package/docs/stores.md
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# Stores
|
|
2
|
+
|
|
3
|
+
Lodestone keeps a little state: the model catalogue, which models are failing (the circuit breaker), saved answers, and, if you use them, rate-limit and spend counters. By default that state lives in memory, which is lost when a serverless function ends and is not shared between functions. For anything real, give Lodestone a store.
|
|
4
|
+
|
|
5
|
+
A store is two methods:
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
interface KeyValueStore {
|
|
9
|
+
get(key: string): Promise<unknown>;
|
|
10
|
+
set(key: string, value: unknown, opts?: { ttlMs?: number }): Promise<void>;
|
|
11
|
+
}
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
`ttlMs` says how long a value is worth keeping; a store that cannot expire values may ignore it. A store that can also count atomically (`incrBy`) makes the rate limiter exact across every function. Ready-made adapters need no client library: they take the binding or token you already have.
|
|
15
|
+
|
|
16
|
+
```ts
|
|
17
|
+
import { createHealth, createLodestone, netlifyBlobsStore } from '@extrafemi-ai/lodestone';
|
|
18
|
+
import { getStore } from '@netlify/blobs';
|
|
19
|
+
|
|
20
|
+
const store = netlifyBlobsStore(getStore('lodestone'));
|
|
21
|
+
const ai = createLodestone({ store, health: createHealth({ store }), /* ... */ });
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
| Adapter | Use it on | Expires values | Counts atomically | Notes |
|
|
25
|
+
|---|---|---|---|---|
|
|
26
|
+
| `netlifyBlobsStore(getStore('x'))` | Netlify Functions | by Lodestone (a value carries its own expiry) | no | Fine for the catalogue, cached answers and the breaker. No atomic add, and writes from separate functions can overwrite each other. |
|
|
27
|
+
| `cloudflareKvStore(env.KV)` | Cloudflare Workers | by KV | no | Eventually consistent, and one key can be written about once a second. Fine for the catalogue and cached answers; the breaker learns a little slower. Not for counters. |
|
|
28
|
+
| `upstashStore({ url, token })` | anywhere with `fetch`, including Vercel KV | by Redis | **yes** | The one to use for per-caller rate limits and spend counters. Plain `fetch`, so it runs on Workers, Deno, Bun and Node. |
|
|
29
|
+
| `denoKvStore(await Deno.openKv())` | Deno and Deno Deploy | by Deno KV | no | |
|
|
30
|
+
| `memoryStore()` | tests, one long-running process | yes | yes (in this process) | The default. |
|
|
31
|
+
|
|
32
|
+
## Which store for which job
|
|
33
|
+
|
|
34
|
+
| Job | A plain store (Blobs, KV, Deno KV) | Upstash |
|
|
35
|
+
|---|---|---|
|
|
36
|
+
| Model catalogue, cached answers | fine | fine |
|
|
37
|
+
| Circuit breaker | fine | fine |
|
|
38
|
+
| Per-caller rate limit | exact inside one process, approximate between processes | exact |
|
|
39
|
+
| Spend limit (`createSpendLimit`) | exact inside one process, approximate between processes | as good as a get/set ledger can be (see [protecting your budget](protecting-your-budget.md)) |
|
|
40
|
+
|
|
41
|
+
## Writing your own
|
|
42
|
+
|
|
43
|
+
Anything that gets and sets JSON by key will do. Keys contain colons (`lodestone:health`); if your store dislikes them, `safeKey(key)` returns a version it will accept (anything outside letters, digits, `.`, `_` and `-` becomes `~` and two hex digits, so two different keys never collide).
|
|
44
|
+
|
|
45
|
+
What comes back from a store is checked before it is used: a catalogue, breaker record or ledger of the wrong shape or an absurd size is ignored, so a store that someone else can write to cannot make Lodestone hide every model or grow without limit.
|
|
@@ -1,20 +1,22 @@
|
|
|
1
1
|
// A Netlify function that asks a model for JSON, written so it keeps working when a model is
|
|
2
2
|
// withdrawn, slow or rate-limited. Needs OPENROUTER_API_KEY (and, for a second route, GROQ_API_KEY).
|
|
3
|
-
import { createHealth, createLodestone, explain, groq, LodestoneError } from '@extrafemi-ai/lodestone';
|
|
3
|
+
import { createHealth, createLodestone, createRateLimiter, createSpendLimit, explain, groq, LodestoneError, netlifyBlobsStore, upstashStore } from '@extrafemi-ai/lodestone';
|
|
4
4
|
import { getStore } from '@netlify/blobs';
|
|
5
5
|
import type { Handler } from '@netlify/functions';
|
|
6
6
|
|
|
7
7
|
// Where Lodestone keeps what it learns: the model catalogue, which models are failing, saved answers.
|
|
8
|
-
//
|
|
9
|
-
|
|
10
|
-
const
|
|
11
|
-
|
|
8
|
+
// Netlify Blobs cannot count atomically, so the per-caller limit and the spend ledger go in Upstash when it is set up
|
|
9
|
+
// (UPSTASH_URL and UPSTASH_TOKEN); without it they are exact inside one function instance and approximate between instances.
|
|
10
|
+
export const store = netlifyBlobsStore(getStore('lodestone'));
|
|
11
|
+
const counters = process.env.UPSTASH_URL ? upstashStore({ url: process.env.UPSTASH_URL, token: process.env.UPSTASH_TOKEN! }) : store;
|
|
12
12
|
|
|
13
13
|
export const ai = createLodestone({
|
|
14
14
|
openrouter: { apiKey: process.env.OPENROUTER_API_KEY },
|
|
15
15
|
fallbacks: process.env.GROQ_API_KEY ? [{ provider: groq(), apiKey: process.env.GROQ_API_KEY, models: ['llama-3.3-70b-versatile'], free: true }] : [],
|
|
16
16
|
store,
|
|
17
17
|
health: createHealth({ store }), // a model that keeps failing is skipped by every function until it recovers
|
|
18
|
+
spend: createSpendLimit({ store: counters, dailyUsd: 2 }), // holds, so a burst cannot overshoot
|
|
19
|
+
rateLimit: { limiter: createRateLimiter({ store: counters, max: 20 }), whenOver: 'free-only' }, // one person cannot empty it
|
|
18
20
|
budgetMs: 24_000, // Netlify cuts a function off at 30 seconds
|
|
19
21
|
});
|
|
20
22
|
|
|
@@ -26,10 +28,11 @@ export const handler: Handler = async (event) => {
|
|
|
26
28
|
const { text } = JSON.parse(event.body ?? '{}') as { text?: string };
|
|
27
29
|
if (!text) return { statusCode: 400, body: JSON.stringify({ error: 'text is required' }) };
|
|
28
30
|
// A model whose answer is not valid JSON of this shape is skipped for the next one.
|
|
29
|
-
const
|
|
31
|
+
const caller = event.headers['x-nf-client-connection-ip'] ?? 'unknown'; // name who is calling, so each caller has their own allowance
|
|
32
|
+
const r = await ai.json<Summary>({ prompt: `Summarise as JSON {"title": string, "points": string[]}:\n\n${text}`, tier: 'fast', validate: isSummary, caller });
|
|
30
33
|
return { statusCode: 200, body: JSON.stringify(r.data) };
|
|
31
34
|
} catch (err) {
|
|
32
35
|
// A sentence a person can be shown, not a stack trace.
|
|
33
|
-
return { statusCode: err instanceof LodestoneError && err.kind === 'rate_limit' ? 429 : 502, body: JSON.stringify({ error: explain(err, { product: 'MyApp' }) }) };
|
|
36
|
+
return { statusCode: err instanceof LodestoneError && (err.kind === 'rate_limit' || err.kind === 'throttled') ? 429 : 502, body: JSON.stringify({ error: explain(err, { product: 'MyApp' }) }) };
|
|
34
37
|
}
|
|
35
38
|
};
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@extrafemi-ai/lodestone",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.7.0",
|
|
4
4
|
"description": "Calls AI models so your tool keeps working when a model is withdrawn, slow, rate-limited or down: failover across providers, self-updating model lists, spend limits.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"repository": {
|
|
@@ -40,10 +40,13 @@
|
|
|
40
40
|
"test": "vitest run",
|
|
41
41
|
"prepublishOnly": "npm run typecheck && npm run typecheck:examples && npm test && npm run build",
|
|
42
42
|
"typecheck:examples": "tsc -p tsconfig.examples.json",
|
|
43
|
-
"site": "npm run build && node site/build.mjs"
|
|
43
|
+
"site": "npm run build && node site/build.mjs",
|
|
44
|
+
"smoke": "npm run build && node scripts/smoke.mjs",
|
|
45
|
+
"smoke:workers": "node scripts/run-workers.mjs"
|
|
44
46
|
},
|
|
45
47
|
"devDependencies": {
|
|
46
48
|
"@types/node": "^22.10.0",
|
|
49
|
+
"miniflare": "^4.20260730.0",
|
|
47
50
|
"typescript": "^5.6.0",
|
|
48
51
|
"vitest": "^4.1.12"
|
|
49
52
|
},
|