agent402-tollbooth 0.5.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +97 -5
- package/index.js +106 -25
- package/package.json +6 -2
- package/pow.js +83 -8
- package/replay.js +118 -0
package/README.md
CHANGED
|
@@ -194,6 +194,7 @@ on abort. (PoW is checked first, so an agent without a wallet always has a free
|
|
|
194
194
|
| `resourceBaseUrl` | `""` | Absolute base used for the `resource` field / PoW binding |
|
|
195
195
|
| `observe` | `false` | Observe-only: classify and count, but never 402. For pre-launch traffic measurement. |
|
|
196
196
|
| `statsSink` | in-memory | Durable stats backend. Built-ins: `memorySink`, `kvStatsSink(kv)`, `httpStatsSink(url)`. |
|
|
197
|
+
| `replayStore` | **this process's memory** | Shared single-use record for solved proof-of-work tokens. Required for multi-worker / multi-instance / serverless. Built-ins: `sqliteReplayStore(db)`, `redisReplayStore(client)`. See below. |
|
|
197
198
|
|
|
198
199
|
### Environment variables
|
|
199
200
|
|
|
@@ -211,6 +212,7 @@ Read by the bundled proxy / Express entry point (`index.js`):
|
|
|
211
212
|
| `TOLLBOOTH_ADAPTIVE` | `false` | Raise PoW difficulty as charged-request load climbs |
|
|
212
213
|
| `TOLLBOOTH_ADAPTIVE_PER_BIT` | `300` | +1 difficulty bit per N charged requests/min |
|
|
213
214
|
| `TOLLBOOTH_SECRET` | random | HMAC secret binding PoW challenges (set it to survive restarts / run multiple instances) |
|
|
215
|
+
| `TOLLBOOTH_REPLAY_SQLITE` | – | Path to a SQLite file every process shares as the single-use PoW record. Set it whenever more than one process serves the same `TOLLBOOTH_SECRET`. Needs Node 22.5+ (built-in `node:sqlite`) or an installed `better-sqlite3`; refuses to start if neither can open the file |
|
|
214
216
|
| `TOLLBOOTH_RESOURCE_BASE` | `TOLLBOOTH_UPSTREAM` | Absolute base used for the `resource` field / PoW binding |
|
|
215
217
|
| `TOLLBOOTH_VERIFY_TIMEOUT_MS` | `10000` | Abort an x402 settlement check after this long |
|
|
216
218
|
| `TOLLBOOTH_OBSERVE` | `false` | Observe-only: classify and count, never 402 |
|
|
@@ -319,6 +321,88 @@ type StatsSink = {
|
|
|
319
321
|
};
|
|
320
322
|
```
|
|
321
323
|
|
|
324
|
+
## Single-use proof-of-work across workers and instances (`replayStore`)
|
|
325
|
+
|
|
326
|
+
**The default single-use record lives in ONE PROCESS'S MEMORY.** Read that
|
|
327
|
+
sentence twice if you run more than one process, because the two settings
|
|
328
|
+
interact:
|
|
329
|
+
|
|
330
|
+
- A stable `TOLLBOOTH_SECRET` is what makes a token minted by worker 1 verify on
|
|
331
|
+
worker 2. Without it, multi-process deploys reject every solution.
|
|
332
|
+
- With it, and with no shared replay store, each process keeps its own
|
|
333
|
+
"already used" list. One 18-bit solve is then redeemable **once per process**
|
|
334
|
+
inside the token's 5-minute TTL, and again after a worker recycles. Four
|
|
335
|
+
workers means four requests for one solve.
|
|
336
|
+
|
|
337
|
+
Pass a `replayStore` and every process claims against the same record:
|
|
338
|
+
|
|
339
|
+
```js
|
|
340
|
+
// Several Node workers on one host (cluster, pm2, one container with N processes).
|
|
341
|
+
import Database from "better-sqlite3"; // or node:sqlite on Node 22.5+
|
|
342
|
+
import { createTollbooth, sqliteReplayStore } from "agent402-tollbooth";
|
|
343
|
+
|
|
344
|
+
const db = new Database("/var/lib/tollbooth/replay.db");
|
|
345
|
+
db.pragma("journal_mode = WAL"); // concurrent writers
|
|
346
|
+
db.pragma("busy_timeout = 5000"); // wait for the write lock instead of throwing
|
|
347
|
+
|
|
348
|
+
app.use(createTollbooth({ replayStore: sqliteReplayStore(db) }));
|
|
349
|
+
```
|
|
350
|
+
|
|
351
|
+
```js
|
|
352
|
+
// Instances across hosts, or a serverless runtime that can reach Redis.
|
|
353
|
+
import { createClient } from "redis";
|
|
354
|
+
import { createTollbooth, redisReplayStore } from "agent402-tollbooth";
|
|
355
|
+
|
|
356
|
+
const redis = createClient({ url: process.env.REDIS_URL });
|
|
357
|
+
await redis.connect();
|
|
358
|
+
app.use(createTollbooth({ replayStore: redisReplayStore(redis) }));
|
|
359
|
+
```
|
|
360
|
+
|
|
361
|
+
`redisReplayStore` also accepts an ioredis client (their `SET` signatures differ;
|
|
362
|
+
the factory detects which one it was handed). Neither driver is a dependency of
|
|
363
|
+
this package: you pass a client you already opened, so the tollbooth still
|
|
364
|
+
installs with nothing but Express.
|
|
365
|
+
|
|
366
|
+
The bundled reverse proxy takes the SQLite path from the environment, so the CLI
|
|
367
|
+
needs no code:
|
|
368
|
+
|
|
369
|
+
```bash
|
|
370
|
+
TOLLBOOTH_SECRET=$(openssl rand -hex 32) \
|
|
371
|
+
TOLLBOOTH_REPLAY_SQLITE=/var/lib/tollbooth/replay.db \
|
|
372
|
+
TOLLBOOTH_UPSTREAM=https://your-origin.example \
|
|
373
|
+
npx agent402-tollbooth
|
|
374
|
+
```
|
|
375
|
+
|
|
376
|
+
Store interface (build your own against Postgres, DynamoDB, a Durable Object -
|
|
377
|
+
this is the same contract the edge gate's `store` option uses, so an
|
|
378
|
+
implementation ports between the two unchanged):
|
|
379
|
+
|
|
380
|
+
```ts
|
|
381
|
+
type ReplayStore = {
|
|
382
|
+
// MUST be atomic: true only the first time this token is seen, false after.
|
|
383
|
+
claim(token: string, expiresAtMs: number): boolean | Promise<boolean>;
|
|
384
|
+
};
|
|
385
|
+
```
|
|
386
|
+
|
|
387
|
+
Three things worth knowing before you write one:
|
|
388
|
+
|
|
389
|
+
- **It may be async.** `claim` can return a promise; `verify()` then returns a
|
|
390
|
+
promise too, and the gate awaits it. A synchronous store keeps the gate
|
|
391
|
+
synchronous, so existing single-process deploys are unaffected.
|
|
392
|
+
- **A throw is a refusal, not a pass.** If `claim` throws or rejects, the gate
|
|
393
|
+
answers `402` with `X-Pow-Error: replay store unavailable`. Let your store
|
|
394
|
+
throw when its backend is unreachable - guessing "probably unused" would hand
|
|
395
|
+
out exactly the free passes the store exists to stop.
|
|
396
|
+
- **Non-atomic is not enough.** A separate "read, then write" pair lets two
|
|
397
|
+
concurrent redemptions of one token both see "unseen" and both pass. SQLite's
|
|
398
|
+
`INSERT OR IGNORE` on a primary key and Redis `SET NX` are atomic; a plain
|
|
399
|
+
eventually-consistent key/value `get` + `put` is not.
|
|
400
|
+
|
|
401
|
+
Scope note: the SQLite store's guarantee is per **database file**, which covers
|
|
402
|
+
many processes sharing a disk. It does not cover hosts sharing a network
|
|
403
|
+
filesystem (SQLite locking is not reliable there) - use Redis or Postgres for
|
|
404
|
+
that shape.
|
|
405
|
+
|
|
322
406
|
## Edge analytics (Cloudflare Worker / Next.js)
|
|
323
407
|
|
|
324
408
|
The Cloudflare Worker entry (`worker.js`) auto-mounts both the dashboard and
|
|
@@ -341,10 +425,17 @@ as drop-in copyable snippets.
|
|
|
341
425
|
- **Set a stable `TOLLBOOTH_SECRET`.** Required for any multi-process/clustered
|
|
342
426
|
Node deploy and for all edge deploys - without it, proof-of-work tokens use a
|
|
343
427
|
random per-process secret and are rejected across restarts/workers/isolates.
|
|
344
|
-
- **
|
|
345
|
-
|
|
346
|
-
|
|
347
|
-
|
|
428
|
+
- **If more than one process shares that secret, supply a `replayStore`.** The
|
|
429
|
+
default single-use record is per process, so a stable secret plus no shared
|
|
430
|
+
store means one solve buys one free request per worker within its TTL. Node:
|
|
431
|
+
`replayStore: sqliteReplayStore(db)` / `redisReplayStore(client)`, or
|
|
432
|
+
`TOLLBOOTH_REPLAY_SQLITE=<path>` for the bundled proxy
|
|
433
|
+
([details](#single-use-proof-of-work-across-workers-and-instances-replaystore)).
|
|
434
|
+
- **For serverless/edge, supply a durable replay `store`** (bind a Durable Object
|
|
435
|
+
as `TOLLBOOTH_REPLAY` for atomic claims; KV is eventually consistent and the
|
|
436
|
+
Worker entry refuses to enforce on it without an explicit override). The
|
|
437
|
+
in-memory default is per-isolate, so a solved token could otherwise be reused
|
|
438
|
+
across isolates within its TTL.
|
|
348
439
|
- **The reverse proxy pins the host** to your configured upstream (a client can't
|
|
349
440
|
redirect it elsewhere) and **strips client-forged trust/forwarding headers**
|
|
350
441
|
(`X-Tollbooth-Paid`, `X-Forwarded-Host`, etc.) before forwarding.
|
|
@@ -357,7 +448,8 @@ as drop-in copyable snippets.
|
|
|
357
448
|
|
|
358
449
|
- Proof-of-work tokens are HMAC-signed, expiry-checked, single-use, and bound to
|
|
359
450
|
the exact resource (path + query, dots and all) - a solution for one URL can't
|
|
360
|
-
be replayed or reused on another.
|
|
451
|
+
be replayed or reused on another. Single-use is recorded **per process** unless
|
|
452
|
+
you pass a `replayStore`.
|
|
361
453
|
- MIT licensed. Part of [Agent402](https://github.com/MikeyPetrillo/Agent402).
|
|
362
454
|
|
|
363
455
|
## Charge in USDG on Robinhood Chain
|
package/index.js
CHANGED
|
@@ -21,6 +21,7 @@ import { memorySink } from "./sinks.js";
|
|
|
21
21
|
export { AI_BOTS, makeBotMatcher } from "./bots.js";
|
|
22
22
|
export { createPow, leadingZeroBits } from "./pow.js";
|
|
23
23
|
export { memorySink, kvStatsSink, httpStatsSink } from "./sinks.js";
|
|
24
|
+
export { sqliteReplayStore, redisReplayStore } from "./replay.js";
|
|
24
25
|
|
|
25
26
|
const VERIFY_TIMEOUT_MS = Number(process.env.TOLLBOOTH_VERIFY_TIMEOUT_MS) || 10_000;
|
|
26
27
|
|
|
@@ -87,6 +88,8 @@ const STRIP_INBOUND = new Set([
|
|
|
87
88
|
* @param {string} [config.asset="USDC"] stablecoin symbol in the quote (USDG on Robinhood Chain)
|
|
88
89
|
* @param {boolean} [config.pow=true] enable the free proof-of-work rail
|
|
89
90
|
* @param {number} [config.powDifficulty] PoW difficulty in leading zero bits
|
|
91
|
+
* @param {object} [config.replayStore] shared single-use PoW backend, `{ claim(token, expiresAtMs) => boolean|Promise<boolean> }`.
|
|
92
|
+
* Default is THIS PROCESS'S memory; required for multi-worker / multi-instance / serverless. See replay.js.
|
|
90
93
|
* @param {string[]} [config.botUserAgents] user-agents to charge (default: AI_BOTS)
|
|
91
94
|
* @param {(req)=>boolean} [config.charge] custom "should this client pay?" predicate
|
|
92
95
|
* @param {(req)=>boolean} [config.free] custom force-allow predicate (wins over charge)
|
|
@@ -108,6 +111,12 @@ export function createTollbooth(config = {}) {
|
|
|
108
111
|
pow = true,
|
|
109
112
|
powDifficulty,
|
|
110
113
|
powSecret,
|
|
114
|
+
// Shared single-use record for solved proof-of-work tokens. Unset means
|
|
115
|
+
// per-process memory, which is only single-use within ONE process: a stable
|
|
116
|
+
// TOLLBOOTH_SECRET (needed so tokens verify across workers at all) then makes
|
|
117
|
+
// one solve redeemable once per worker inside the token TTL. Pass a store
|
|
118
|
+
// from replay.js for any deploy with more than one process.
|
|
119
|
+
replayStore,
|
|
111
120
|
botUserAgents = AI_BOTS,
|
|
112
121
|
// Who pays. Default "bots" = the original behavior (charge AI crawler UAs).
|
|
113
122
|
// "all" — charge every client except a `free()` match (UA detection is
|
|
@@ -137,7 +146,7 @@ export function createTollbooth(config = {}) {
|
|
|
137
146
|
} = config;
|
|
138
147
|
|
|
139
148
|
const isBot = makeBotMatcher(botUserAgents);
|
|
140
|
-
const powEngine = pow ? createPow({ difficulty: powDifficulty, secret: powSecret }) : null;
|
|
149
|
+
const powEngine = pow ? createPow({ difficulty: powDifficulty, secret: powSecret, replayStore }) : null;
|
|
141
150
|
|
|
142
151
|
// Passive analytics — never affects request handling, just counts what happens.
|
|
143
152
|
// `mem` is an always-on in-process mirror so `.stats()` stays synchronous for
|
|
@@ -224,35 +233,56 @@ export function createTollbooth(config = {}) {
|
|
|
224
233
|
res.status(402).json(body);
|
|
225
234
|
};
|
|
226
235
|
|
|
236
|
+
// Paid rail: x402 (USDC). Settlement verification is operator-supplied so
|
|
237
|
+
// we reuse the standard, audited x402 stack rather than reinvent it. Reached
|
|
238
|
+
// either directly or after the free rail declined, so it lives in a function
|
|
239
|
+
// the (possibly async) proof-of-work branch can hand control back to.
|
|
240
|
+
const paidRail = () => {
|
|
241
|
+
const payHeader = req.headers["x-payment"] || req.headers["payment-signature"];
|
|
242
|
+
if (payTo && typeof verifyX402 === "function" && payHeader) {
|
|
243
|
+
// Bound verification time so a slow/hung verifier can't exhaust resources.
|
|
244
|
+
// F19: pass an AbortSignal and ABORT it on timeout, so a slow verifier that
|
|
245
|
+
// also SETTLES cannot move money after we have already returned 402 (a
|
|
246
|
+
// charged denial). A pure-verification callback may ignore the signal; a
|
|
247
|
+
// settling one MUST cancel/drain in-flight work on it before returning.
|
|
248
|
+
const ac = new AbortController();
|
|
249
|
+
const timeout = new Promise((resolve) => setTimeout(() => { ac.abort(); resolve(false); }, VERIFY_TIMEOUT_MS));
|
|
250
|
+
return Promise.race([Promise.resolve(verifyX402(req, { price, network, asset, payTo, resource, signal: ac.signal })), timeout])
|
|
251
|
+
.then((ok) => {
|
|
252
|
+
ac.abort(); // verification resolved: cancel anything still in flight
|
|
253
|
+
if (ok) { incr("x402Paid"); res.setHeader("X-Tollbooth-Paid", "x402"); return next(); }
|
|
254
|
+
send402();
|
|
255
|
+
})
|
|
256
|
+
.catch(() => { ac.abort(); res.setHeader("X-Tollbooth-Error", "x402-verify-failed"); send402(); });
|
|
257
|
+
}
|
|
258
|
+
|
|
259
|
+
return send402();
|
|
260
|
+
};
|
|
261
|
+
|
|
227
262
|
// Free rail: proof-of-work.
|
|
228
263
|
const powHeader = req.headers["x-pow-solution"];
|
|
229
264
|
if (powEngine && powHeader) {
|
|
265
|
+
const afterPow = (r) => {
|
|
266
|
+
if (r.ok) { incr("powSolved"); res.setHeader("X-Tollbooth-Paid", "pow"); return next(); }
|
|
267
|
+
res.setHeader("X-Pow-Error", r.reason);
|
|
268
|
+
return paidRail();
|
|
269
|
+
};
|
|
270
|
+
// pow.verify is synchronous with the default in-process replay store and
|
|
271
|
+
// returns a promise when a shared store (SQLite/Redis) answers with one.
|
|
272
|
+
// Branch on the shape instead of awaiting unconditionally: the default path
|
|
273
|
+
// must keep resolving inside this tick, because a synchronous caller (and
|
|
274
|
+
// our own single-process tests) would otherwise see the middleware return
|
|
275
|
+
// before it has decided anything.
|
|
230
276
|
const r = powEngine.verify(powHeader, powResource);
|
|
231
|
-
if (r
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
const payHeader = req.headers["x-payment"] || req.headers["payment-signature"];
|
|
238
|
-
if (payTo && typeof verifyX402 === "function" && payHeader) {
|
|
239
|
-
// Bound verification time so a slow/hung verifier can't exhaust resources.
|
|
240
|
-
// F19: pass an AbortSignal and ABORT it on timeout, so a slow verifier that
|
|
241
|
-
// also SETTLES cannot move money after we have already returned 402 (a
|
|
242
|
-
// charged denial). A pure-verification callback may ignore the signal; a
|
|
243
|
-
// settling one MUST cancel/drain in-flight work on it before returning.
|
|
244
|
-
const ac = new AbortController();
|
|
245
|
-
const timeout = new Promise((resolve) => setTimeout(() => { ac.abort(); resolve(false); }, VERIFY_TIMEOUT_MS));
|
|
246
|
-
return Promise.race([Promise.resolve(verifyX402(req, { price, network, asset, payTo, resource, signal: ac.signal })), timeout])
|
|
247
|
-
.then((ok) => {
|
|
248
|
-
ac.abort(); // verification resolved: cancel anything still in flight
|
|
249
|
-
if (ok) { incr("x402Paid"); res.setHeader("X-Tollbooth-Paid", "x402"); return next(); }
|
|
250
|
-
send402();
|
|
251
|
-
})
|
|
252
|
-
.catch(() => { ac.abort(); res.setHeader("X-Tollbooth-Error", "x402-verify-failed"); send402(); });
|
|
277
|
+
if (r && typeof r.then === "function") {
|
|
278
|
+
// A rejected verify would leave the request hanging with no response, so
|
|
279
|
+
// it degrades to the same refusal the store-throw path produces.
|
|
280
|
+
return r.then(afterPow, () => afterPow({ ok: false, reason: "replay store unavailable" }));
|
|
281
|
+
}
|
|
282
|
+
return afterPow(r);
|
|
253
283
|
}
|
|
254
284
|
|
|
255
|
-
return
|
|
285
|
+
return paidRail();
|
|
256
286
|
}
|
|
257
287
|
|
|
258
288
|
tollbooth.shouldCharge = shouldCharge;
|
|
@@ -335,9 +365,60 @@ async function startCli() {
|
|
|
335
365
|
console.error("✖ TOLLBOOTH_SECRET is the public placeholder from the deploy template. It is readable by anyone, so proof-of-work tokens can be forged and your gate bypassed for free. Refusing to start. Generate a real secret: openssl rand -hex 32");
|
|
336
366
|
process.exit(1);
|
|
337
367
|
}
|
|
368
|
+
// A stable secret makes one solved token verifiable by EVERY process sharing
|
|
369
|
+
// it, and the single-use record defaults to this process's memory, so behind a
|
|
370
|
+
// load balancer, or under `cluster`, one solve buys a free request per process
|
|
371
|
+
// for the token's lifetime. TOLLBOOTH_REPLAY_SQLITE points every process at one
|
|
372
|
+
// claim table and closes that. Opt-in, because a single-process proxy (the
|
|
373
|
+
// common case for this CLI) needs no file and no extra dependency.
|
|
374
|
+
let replayStore;
|
|
375
|
+
const replayPath = process.env.TOLLBOOTH_REPLAY_SQLITE;
|
|
376
|
+
if (replayPath) {
|
|
377
|
+
const { sqliteReplayStore } = await import("./replay.js");
|
|
378
|
+
// node:sqlite is built in (Node 22.5+); better-sqlite3 covers older Node and
|
|
379
|
+
// is what many operators already have installed. Neither is a dependency of
|
|
380
|
+
// this package: the store is opt-in, so its driver is too.
|
|
381
|
+
let db = null;
|
|
382
|
+
try {
|
|
383
|
+
const { DatabaseSync } = await import("node:sqlite");
|
|
384
|
+
db = new DatabaseSync(replayPath);
|
|
385
|
+
} catch {
|
|
386
|
+
try {
|
|
387
|
+
const { default: Database } = await import("better-sqlite3");
|
|
388
|
+
db = new Database(replayPath);
|
|
389
|
+
} catch { db = null; }
|
|
390
|
+
}
|
|
391
|
+
if (!db) {
|
|
392
|
+
// The operator asked for a shared store; starting without one would leave
|
|
393
|
+
// the replay hole open while the config says it is closed. Fail closed.
|
|
394
|
+
console.error(`✖ TOLLBOOTH_REPLAY_SQLITE=${replayPath} was set but no SQLite driver could be opened. Run Node 22.5+ (built-in node:sqlite) or install better-sqlite3. Refusing to start.`);
|
|
395
|
+
process.exit(1);
|
|
396
|
+
}
|
|
397
|
+
try {
|
|
398
|
+
// busy_timeout matters more than it looks: a claim that hits SQLITE_BUSY
|
|
399
|
+
// throws, pow.js fails closed on a throw, and the refusal lands on a
|
|
400
|
+
// crawler that did the work. WAL is best-effort because converting a
|
|
401
|
+
// brand-new file needs an exclusive lock a sibling worker may hold
|
|
402
|
+
// (measured: 2 of 160 simultaneous boots lost that lock); the winner writes
|
|
403
|
+
// the mode into the file, so the losers inherit it and nothing is lost.
|
|
404
|
+
db.exec("PRAGMA busy_timeout = 5000");
|
|
405
|
+
try { db.exec("PRAGMA journal_mode = WAL"); } catch { /* a sibling worker is setting it */ }
|
|
406
|
+
replayStore = sqliteReplayStore(db);
|
|
407
|
+
} catch (e) {
|
|
408
|
+
// Not retried: the failure this actually produces is a permanently broken
|
|
409
|
+
// file (for example an orphaned -wal/-shm left beside a deleted database),
|
|
410
|
+
// which no number of attempts clears. Refuse to start rather than serve
|
|
411
|
+
// with per-process-only replay protection while the config claims otherwise.
|
|
412
|
+
console.error(`✖ TOLLBOOTH_REPLAY_SQLITE=${replayPath} could not be initialised: ${e.message}. If a stale ${replayPath}-wal / -shm pair is present without its database, remove them. Refusing to start.`);
|
|
413
|
+
process.exit(1);
|
|
414
|
+
}
|
|
415
|
+
console.log(`shared proof-of-work replay store: sqlite ${replayPath}`);
|
|
416
|
+
} else if (_secret) {
|
|
417
|
+
console.warn("⚠ No shared proof-of-work replay store (TOLLBOOTH_REPLAY_SQLITE unset): solved tokens are single-use PER PROCESS only. Fine for one process; behind a load balancer or under `cluster`, one solve is redeemable once per process within its 5-minute TTL.");
|
|
418
|
+
}
|
|
338
419
|
const { default: express } = await import("express");
|
|
339
420
|
const app = express();
|
|
340
|
-
const gate = createTollbooth({ resourceBaseUrl: process.env.TOLLBOOTH_RESOURCE_BASE || upstream || "" });
|
|
421
|
+
const gate = createTollbooth({ resourceBaseUrl: process.env.TOLLBOOTH_RESOURCE_BASE || upstream || "", replayStore });
|
|
341
422
|
// Operator analytics — aggregate counts only (no per-request data), mounted
|
|
342
423
|
// before the gate so they're always reachable and never themselves charged.
|
|
343
424
|
// Two opt-in admin tokens (legacy `TOLLBOOTH_STATS_TOKEN` covers /stats only;
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "agent402-tollbooth",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.6.0",
|
|
4
4
|
"description": "Open-source, self-hostable x402 pay-per-crawl tollbooth: charge AI bots per request (USDC via x402 — or USDG on Robinhood Chain — or free proof-of-work) while humans browse free. Drop-in Express middleware or reverse proxy — no Cloudflare, no Stripe, no signup.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "index.js",
|
|
@@ -14,7 +14,9 @@
|
|
|
14
14
|
"./worker": "./worker.js",
|
|
15
15
|
"./worker.js": "./worker.js",
|
|
16
16
|
"./dashboard.js": "./dashboard.js",
|
|
17
|
-
"./sinks.js": "./sinks.js"
|
|
17
|
+
"./sinks.js": "./sinks.js",
|
|
18
|
+
"./replay": "./replay.js",
|
|
19
|
+
"./replay.js": "./replay.js"
|
|
18
20
|
},
|
|
19
21
|
"files": [
|
|
20
22
|
"index.js",
|
|
@@ -24,6 +26,7 @@
|
|
|
24
26
|
"pow.js",
|
|
25
27
|
"dashboard.js",
|
|
26
28
|
"sinks.js",
|
|
29
|
+
"replay.js",
|
|
27
30
|
"example.js",
|
|
28
31
|
"demo.js",
|
|
29
32
|
"README.md"
|
|
@@ -48,6 +51,7 @@
|
|
|
48
51
|
"http-402",
|
|
49
52
|
"402-payment-required",
|
|
50
53
|
"proof-of-work",
|
|
54
|
+
"replay-protection",
|
|
51
55
|
"usdc",
|
|
52
56
|
"reverse-proxy",
|
|
53
57
|
"express-middleware",
|
package/pow.js
CHANGED
|
@@ -6,6 +6,14 @@
|
|
|
6
6
|
// exact resource they were minted for, expiry-checked, and single-use (a solved
|
|
7
7
|
// token is remembered until it expires). Solving costs the caller CPU; issuing
|
|
8
8
|
// and verifying cost us one hash + one HMAC.
|
|
9
|
+
//
|
|
10
|
+
// Single-use is only as wide as the store that records it. The default store is
|
|
11
|
+
// THIS PROCESS'S memory, which is exactly right for a single-process deploy and
|
|
12
|
+
// insufficient for anything else: a stable TOLLBOOTH_SECRET makes one token
|
|
13
|
+
// verifiable by every worker/instance sharing it, so a single solve would buy
|
|
14
|
+
// one free request PER PROCESS within the token's TTL (and again after a worker
|
|
15
|
+
// recycle). Multi-worker, multi-instance and serverless operators must pass a
|
|
16
|
+
// shared `replayStore` (see replay.js and the README).
|
|
9
17
|
import { createHash, createHmac, randomBytes, timingSafeEqual } from "node:crypto";
|
|
10
18
|
|
|
11
19
|
const TTL_MS = 5 * 60 * 1000;
|
|
@@ -21,17 +29,52 @@ export function leadingZeroBits(buf) {
|
|
|
21
29
|
return bits;
|
|
22
30
|
}
|
|
23
31
|
|
|
32
|
+
/**
|
|
33
|
+
* @param {object} [opts]
|
|
34
|
+
* @param {string} [opts.secret] HMAC secret binding challenges to this operator
|
|
35
|
+
* @param {number} [opts.difficulty] leading zero bits a solution must carry
|
|
36
|
+
* @param {number} [opts.ttlMs] challenge lifetime
|
|
37
|
+
* @param {{claim:(token:string,expiresAtMs:number)=>boolean|Promise<boolean>}} [opts.replayStore]
|
|
38
|
+
* Shared single-use backend. `claim` must ATOMICALLY return true only the
|
|
39
|
+
* first time it sees a token, false on every later call. Mirrors the
|
|
40
|
+
* `store` contract of createEdgePow in edge.js (accepted here under that
|
|
41
|
+
* name too, so an edge snippet ports across unchanged). Defaults to an
|
|
42
|
+
* in-process Map: correct for one process, wrong for several.
|
|
43
|
+
*/
|
|
24
44
|
export function createPow({
|
|
25
45
|
secret = process.env.TOLLBOOTH_SECRET || randomBytes(32).toString("hex"),
|
|
26
46
|
difficulty = Number(process.env.TOLLBOOTH_POW_BITS) || 18,
|
|
27
47
|
ttlMs = TTL_MS,
|
|
48
|
+
replayStore,
|
|
49
|
+
store,
|
|
28
50
|
} = {}) {
|
|
29
|
-
const
|
|
30
|
-
|
|
51
|
+
const shared = replayStore || store;
|
|
52
|
+
if (shared && typeof shared.claim !== "function") {
|
|
53
|
+
// Loud at construction rather than silently falling back to per-process
|
|
54
|
+
// memory: an operator who passes a misshapen store believes they have
|
|
55
|
+
// cluster-wide single-use, and a silent fallback would leave the exact hole
|
|
56
|
+
// they configured the store to close.
|
|
57
|
+
throw new TypeError("createPow: replayStore must expose claim(token, expiresAtMs) => boolean | Promise<boolean>");
|
|
58
|
+
}
|
|
59
|
+
const used = new Map(); // token -> expiry(ms); only used by the default store
|
|
60
|
+
// Default replay backend, kept as the default so existing single-process
|
|
61
|
+
// deployments behave exactly as they did. The body is synchronous, so two
|
|
62
|
+
// concurrent claims of the same token in one process cannot both win, which a
|
|
63
|
+
// separate has()-then-set() would allow.
|
|
64
|
+
const memoryStore = {
|
|
65
|
+
claim: (token, expiresAtMs) => {
|
|
66
|
+
const seen = used.get(token);
|
|
67
|
+
if (seen !== undefined && seen >= Date.now()) return false; // a still-valid record => already spent
|
|
68
|
+
used.set(token, expiresAtMs);
|
|
69
|
+
return true;
|
|
70
|
+
},
|
|
71
|
+
};
|
|
72
|
+
const replay = shared || memoryStore;
|
|
73
|
+
const sweep = shared ? null : setInterval(() => {
|
|
31
74
|
const now = Date.now();
|
|
32
75
|
for (const [t, exp] of used) if (exp < now) used.delete(t);
|
|
33
76
|
}, 60_000);
|
|
34
|
-
sweep
|
|
77
|
+
sweep?.unref?.();
|
|
35
78
|
|
|
36
79
|
const sign = (payload) => createHmac("sha256", secret).update(payload).digest("base64url");
|
|
37
80
|
|
|
@@ -54,7 +97,37 @@ export function createPow({
|
|
|
54
97
|
};
|
|
55
98
|
}
|
|
56
99
|
|
|
57
|
-
|
|
100
|
+
// Claim single-use, translating the store's answer into a verify() result.
|
|
101
|
+
//
|
|
102
|
+
// FAIL CLOSED: a store that throws or rejects means we cannot know whether this
|
|
103
|
+
// token was already spent, so we refuse the request. Same posture as edge.js,
|
|
104
|
+
// where a claim-store exception used to propagate out of verify instead of
|
|
105
|
+
// becoming a denial. An outage on the replay store costs the operator a few
|
|
106
|
+
// refused crawls; the alternative costs them the paywall.
|
|
107
|
+
function claimSingleUse(token, expiresAtMs) {
|
|
108
|
+
let claimed;
|
|
109
|
+
try { claimed = replay.claim(token, expiresAtMs); }
|
|
110
|
+
catch { return { ok: false, reason: "replay store unavailable" }; }
|
|
111
|
+
if (claimed && typeof claimed.then === "function") {
|
|
112
|
+
return claimed.then(
|
|
113
|
+
(granted) => (granted === true ? { ok: true } : { ok: false, reason: "already used" }),
|
|
114
|
+
() => ({ ok: false, reason: "replay store unavailable" }),
|
|
115
|
+
);
|
|
116
|
+
}
|
|
117
|
+
return claimed === true ? { ok: true } : { ok: false, reason: "already used" };
|
|
118
|
+
}
|
|
119
|
+
|
|
120
|
+
/** Verify a "<token>:<nonce>" solution for a given resource.
|
|
121
|
+
*
|
|
122
|
+
* SYNC-OR-ASYNC BY DESIGN. Returns a plain result object whenever the replay
|
|
123
|
+
* store answers synchronously (the default in-process store always does), and
|
|
124
|
+
* a Promise of that object when a shared store answers with one. It is NOT
|
|
125
|
+
* unconditionally async because verify() has always been synchronous here:
|
|
126
|
+
* every existing caller reads `.ok` off the return value, and `.ok` on a
|
|
127
|
+
* Promise is `undefined`, so a blanket change would deny every honest solve
|
|
128
|
+
* while ALSO making the replay check a no-op for anyone who ignored the
|
|
129
|
+
* promise. Callers that may be configured with an async store must handle a
|
|
130
|
+
* thenable (index.js does; `await` works for both shapes). */
|
|
58
131
|
function verify(headerValue, resource) {
|
|
59
132
|
if (!headerValue || typeof headerValue !== "string") return { ok: false, reason: "missing solution" };
|
|
60
133
|
const cut = headerValue.lastIndexOf(":");
|
|
@@ -72,7 +145,6 @@ export function createPow({
|
|
|
72
145
|
if (a.length !== b.length || !timingSafeEqual(a, b)) return { ok: false, reason: "bad signature" };
|
|
73
146
|
if (res !== resource) return { ok: false, reason: "wrong resource" };
|
|
74
147
|
if (Date.now() > Number(expStr)) return { ok: false, reason: "expired" };
|
|
75
|
-
if (used.has(token)) return { ok: false, reason: "already used" };
|
|
76
148
|
// Defence in depth: the difficulty rides INSIDE the signed token, so a
|
|
77
149
|
// forged or downgraded token could claim difficulty 0 and pass with no work
|
|
78
150
|
// at all. Hold every solution to the configured difficulty as well, so a
|
|
@@ -82,9 +154,12 @@ export function createPow({
|
|
|
82
154
|
if (!Number.isFinite(claimed) || claimed < difficulty) return { ok: false, reason: "difficulty below policy" };
|
|
83
155
|
const hash = createHash("sha256").update(`${chal}:${nonce}`).digest();
|
|
84
156
|
if (leadingZeroBits(hash) < claimed) return { ok: false, reason: "insufficient work" };
|
|
85
|
-
|
|
86
|
-
|
|
157
|
+
// Claim LAST, once the solution is known good: a forged, expired or
|
|
158
|
+
// under-worked attempt must never consume the token an honest solver is
|
|
159
|
+
// still holding, and with a shared store this is the only step that can
|
|
160
|
+
// cost a round trip.
|
|
161
|
+
return claimSingleUse(token, Number(expStr));
|
|
87
162
|
}
|
|
88
163
|
|
|
89
|
-
return { challenge, verify, difficulty };
|
|
164
|
+
return { challenge, verify, difficulty, replayStore: replay };
|
|
90
165
|
}
|
package/replay.js
ADDED
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
// Shared proof-of-work replay stores for the Node gate.
|
|
2
|
+
//
|
|
3
|
+
// The gate's default single-use record lives in one process's memory. That is
|
|
4
|
+
// correct for a single-process deploy and a hole in every other shape: with a
|
|
5
|
+
// stable TOLLBOOTH_SECRET (which multi-worker deploys need, so tokens verify at
|
|
6
|
+
// all) one solved token is redeemable once per process inside its TTL, and again
|
|
7
|
+
// after each worker recycle. These factories give the `replayStore` option a
|
|
8
|
+
// backend every process shares.
|
|
9
|
+
//
|
|
10
|
+
// The contract is the one edge.js already uses, so a store written for the edge
|
|
11
|
+
// gate works here unchanged:
|
|
12
|
+
//
|
|
13
|
+
// { claim(token, expiresAtMs) => boolean | Promise<boolean> }
|
|
14
|
+
//
|
|
15
|
+
// `claim` must be ATOMIC: true only the very first time it sees a token, false
|
|
16
|
+
// for every later call. A get-then-put pair that is not atomic lets two
|
|
17
|
+
// concurrent redemptions of the same token both observe "unseen" and both pass.
|
|
18
|
+
// A store that throws makes the gate REFUSE the request (pow.js fails closed),
|
|
19
|
+
// so a claim implementation should throw rather than guess when its backend is
|
|
20
|
+
// unreachable.
|
|
21
|
+
//
|
|
22
|
+
// No new runtime dependency: every factory takes an ALREADY-OPEN client the
|
|
23
|
+
// operator constructed, so the tollbooth never pulls in a driver of its own.
|
|
24
|
+
|
|
25
|
+
/**
|
|
26
|
+
* SQLite-backed claim, for the common case of several Node workers on one host
|
|
27
|
+
* (`cluster`, pm2, a container with N processes).
|
|
28
|
+
*
|
|
29
|
+
* Pass an open better-sqlite3 `Database`, or a node:sqlite `DatabaseSync` (Node
|
|
30
|
+
* 22.5+). Both expose the same `exec()` / `prepare().run()` surface this uses.
|
|
31
|
+
*
|
|
32
|
+
* import Database from "better-sqlite3";
|
|
33
|
+
* const db = new Database("/var/lib/tollbooth/replay.db");
|
|
34
|
+
* db.pragma("journal_mode = WAL"); // concurrent writers
|
|
35
|
+
* db.pragma("busy_timeout = 5000"); // wait for the write lock, do not throw
|
|
36
|
+
* createTollbooth({ replayStore: sqliteReplayStore(db) });
|
|
37
|
+
*
|
|
38
|
+
* Set busy_timeout. A claim that hits SQLITE_BUSY throws, pow.js fails closed on
|
|
39
|
+
* a throw, and the refusal lands on an honest crawler that did the work. Waiting
|
|
40
|
+
* milliseconds for the write lock is strictly better than that.
|
|
41
|
+
*
|
|
42
|
+
* Atomicity comes from the PRIMARY KEY plus INSERT OR IGNORE: SQLite serializes
|
|
43
|
+
* writers on the database file, so exactly one process observes `changes === 1`
|
|
44
|
+
* for a given token. The scope of that guarantee is the FILE, which means this
|
|
45
|
+
* covers many processes sharing a disk and does NOT cover instances on different
|
|
46
|
+
* hosts (a network filesystem does not make SQLite locking reliable). Use Redis
|
|
47
|
+
* or Postgres across hosts.
|
|
48
|
+
*/
|
|
49
|
+
export function sqliteReplayStore(db, { table = "tollbooth_pow_replay", pruneEveryMs = 60_000 } = {}) {
|
|
50
|
+
if (!db || typeof db.prepare !== "function" || typeof db.exec !== "function") {
|
|
51
|
+
throw new TypeError("sqliteReplayStore(db): pass an open better-sqlite3 Database or node:sqlite DatabaseSync");
|
|
52
|
+
}
|
|
53
|
+
// Identifier, not a bindable value, so it is whitelisted rather than escaped.
|
|
54
|
+
if (!/^[A-Za-z_][A-Za-z0-9_]*$/.test(table)) throw new TypeError("sqliteReplayStore: table must be a plain identifier");
|
|
55
|
+
db.exec(`CREATE TABLE IF NOT EXISTS ${table} (token TEXT PRIMARY KEY, expires_ms INTEGER NOT NULL)`);
|
|
56
|
+
const insert = db.prepare(`INSERT OR IGNORE INTO ${table} (token, expires_ms) VALUES (?, ?)`);
|
|
57
|
+
const prune = db.prepare(`DELETE FROM ${table} WHERE expires_ms < ?`);
|
|
58
|
+
let prunedAt = 0;
|
|
59
|
+
|
|
60
|
+
return {
|
|
61
|
+
claim: (token, expiresAtMs) => {
|
|
62
|
+
const now = Date.now();
|
|
63
|
+
// Bounded growth without a background timer: the table only ever needs to
|
|
64
|
+
// hold unexpired tokens, and an expired row can be dropped because pow.js
|
|
65
|
+
// rejects an expired token before it ever claims. Time-gated so a busy
|
|
66
|
+
// gate does not run a DELETE on every request.
|
|
67
|
+
if (now - prunedAt > pruneEveryMs) { prunedAt = now; prune.run(now); }
|
|
68
|
+
const res = insert.run(String(token), Math.floor(Number(expiresAtMs) || now));
|
|
69
|
+
return Number(res.changes) === 1;
|
|
70
|
+
},
|
|
71
|
+
};
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
/**
|
|
75
|
+
* Redis-backed claim, for multiple instances across hosts (and the shape most
|
|
76
|
+
* serverless Node runtimes can reach).
|
|
77
|
+
*
|
|
78
|
+
* Pass an open client. Both major clients are supported because their SET
|
|
79
|
+
* signatures differ:
|
|
80
|
+
*
|
|
81
|
+
* node-redis v4+: createClient(...) -> set(k, v, { NX: true, PX: ttl })
|
|
82
|
+
* ioredis: new Redis(...) -> set(k, v, "PX", ttl, "NX")
|
|
83
|
+
*
|
|
84
|
+
* createTollbooth({ replayStore: redisReplayStore(redis, { prefix: "tb:pow:" }) });
|
|
85
|
+
*
|
|
86
|
+
* SET NX is atomic server-side across every client, and the PX expiry is the
|
|
87
|
+
* token's own remaining lifetime, so the keyspace self-cleans. `claim` returns a
|
|
88
|
+
* PROMISE, which is why pow.js verify() has to tolerate a thenable.
|
|
89
|
+
*
|
|
90
|
+
* Errors are deliberately NOT caught here: pow.js turns a throw into a refusal.
|
|
91
|
+
* Swallowing a Redis outage and returning true would silently restore the
|
|
92
|
+
* per-process hole this store exists to close.
|
|
93
|
+
*/
|
|
94
|
+
export function redisReplayStore(client, { prefix = "tollbooth:pow:" } = {}) {
|
|
95
|
+
if (!client || typeof client.set !== "function") {
|
|
96
|
+
throw new TypeError("redisReplayStore(client): pass an open node-redis or ioredis client");
|
|
97
|
+
}
|
|
98
|
+
// defineCommand is ioredis-only, and ioredis is the client that takes the
|
|
99
|
+
// positional SET form. Detect rather than send one shape and hope: the failure
|
|
100
|
+
// mode of guessing wrong is not an exception but a SET without the NX flag,
|
|
101
|
+
// which overwrites the record and grants every replay.
|
|
102
|
+
const isIoredis = typeof client.defineCommand === "function";
|
|
103
|
+
return {
|
|
104
|
+
claim: async (token, expiresAtMs) => {
|
|
105
|
+
const key = prefix + token;
|
|
106
|
+
// Floor of 1ms: Redis rejects a non-positive PX. pow.js has already
|
|
107
|
+
// refused anything actually expired, so a tiny window here just means the
|
|
108
|
+
// key is evicted almost immediately.
|
|
109
|
+
const px = Math.max(1, Math.floor(Number(expiresAtMs) - Date.now()));
|
|
110
|
+
const res = isIoredis
|
|
111
|
+
? await client.set(key, "1", "PX", px, "NX")
|
|
112
|
+
: await client.set(key, "1", { NX: true, PX: px });
|
|
113
|
+
// node-redis resolves null when NX declined; ioredis resolves null too,
|
|
114
|
+
// and "OK" on a win. Anything that is not a win is treated as already spent.
|
|
115
|
+
return res === "OK" || res === true;
|
|
116
|
+
},
|
|
117
|
+
};
|
|
118
|
+
}
|