toiljs 0.0.85 → 0.0.87
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +5 -0
- package/README.md +2 -2
- package/build/cli/.tsbuildinfo +1 -1
- package/build/cli/index.js +303 -293
- package/build/compiler/.tsbuildinfo +1 -1
- package/build/compiler/config.d.ts +2 -0
- package/build/compiler/config.js +1 -0
- package/build/compiler/docs.js +8 -24
- package/build/compiler/generate.js +4 -2
- package/build/compiler/index.d.ts +1 -1
- package/build/compiler/index.js +69 -6
- package/build/compiler/toil-docs.generated.js +64 -22
- package/build/devserver/.tsbuildinfo +1 -1
- package/build/devserver/analytics/index.js +7 -3
- package/build/devserver/db/database.d.ts +4 -0
- package/build/devserver/db/database.js +43 -1
- package/build/devserver/db/index.d.ts +1 -1
- package/build/devserver/db/index.js +1 -1
- package/build/devserver/db/types.d.ts +3 -0
- package/build/devserver/db/types.js +2 -0
- package/build/devserver/runtime/module.js +4 -1
- package/docs/README.md +104 -65
- package/docs/auth/README.md +102 -0
- package/docs/auth/configuration.md +94 -0
- package/docs/auth/extending.md +202 -0
- package/docs/auth/how-it-works.md +138 -0
- package/docs/auth/usage.md +188 -0
- package/docs/backend/README.md +143 -0
- package/docs/backend/data.md +351 -0
- package/docs/backend/rest.md +402 -0
- package/docs/backend/rpc.md +226 -0
- package/docs/background/README.md +114 -0
- package/docs/background/daemons.md +230 -0
- package/docs/background/derive.md +179 -0
- package/docs/cli/README.md +377 -0
- package/docs/concepts/config.md +416 -0
- package/docs/concepts/decorators.md +127 -0
- package/docs/concepts/security.md +108 -0
- package/docs/concepts/tiers.md +166 -0
- package/docs/concepts/types.md +216 -0
- package/docs/database/README.md +143 -0
- package/docs/database/capacity.md +350 -0
- package/docs/database/counters.md +174 -0
- package/docs/database/documents.md +255 -0
- package/docs/database/events.md +307 -0
- package/docs/database/membership.md +246 -0
- package/docs/database/setup.md +155 -0
- package/docs/database/unique.md +216 -0
- package/docs/database/views.md +246 -0
- package/docs/frontend/README.md +101 -0
- package/docs/frontend/data-fetching.md +243 -0
- package/docs/frontend/images.md +148 -0
- package/docs/frontend/metadata.md +344 -0
- package/docs/frontend/rendering.md +236 -0
- package/docs/frontend/routing.md +344 -0
- package/docs/frontend/scripts.md +118 -0
- package/docs/frontend/search.md +191 -0
- package/docs/frontend/styling.md +147 -0
- package/docs/getting-started/README.md +81 -0
- package/docs/getting-started/create-project.md +131 -0
- package/docs/getting-started/deploy.md +101 -0
- package/docs/getting-started/first-app.md +215 -0
- package/docs/getting-started/installation.md +106 -0
- package/docs/getting-started/migrating.md +125 -0
- package/docs/getting-started/project-structure.md +163 -0
- package/docs/introduction/README.md +41 -0
- package/docs/introduction/design-principles.md +55 -0
- package/docs/introduction/distributed.md +74 -0
- package/docs/introduction/how-it-works.md +74 -0
- package/docs/introduction/hyperscale.md +51 -0
- package/docs/introduction/modern-stack.md +36 -0
- package/docs/introduction/vs-other-frameworks.md +48 -0
- package/docs/introduction/why-toil.md +93 -0
- package/docs/llms.txt +90 -0
- package/docs/realtime/README.md +102 -0
- package/docs/realtime/channels.md +211 -0
- package/docs/realtime/streams.md +369 -0
- package/docs/services/README.md +60 -0
- package/docs/services/analytics.md +268 -0
- package/docs/services/caching.md +175 -0
- package/docs/services/cookies.md +235 -0
- package/docs/services/crypto.md +209 -0
- package/docs/services/email.md +289 -0
- package/docs/services/environment.md +141 -0
- package/docs/services/ratelimit.md +174 -0
- package/docs/services/time.md +85 -0
- package/examples/basic/client/routes/analytics.tsx +13 -12
- package/examples/basic/server/models/SiteAnalytics.ts +29 -17
- package/examples/basic/server/routes/Analytics.ts +15 -17
- package/examples/basic/server/routes/UserId.ts +22 -0
- package/package.json +15 -11
- package/scripts/gen-toil-docs.mjs +24 -35
- package/server/auth/AuthController.ts +336 -0
- package/server/auth/AuthUser.ts +23 -0
- package/server/auth/index.ts +16 -0
- package/server/globals/auth.ts +31 -0
- package/server/globals/userid.ts +107 -0
- package/src/compiler/config.ts +13 -0
- package/src/compiler/docs.ts +16 -33
- package/src/compiler/generate.ts +6 -2
- package/src/compiler/index.ts +114 -6
- package/src/compiler/toil-docs.generated.ts +64 -22
- package/src/devserver/analytics/index.ts +10 -4
- package/src/devserver/db/database.ts +67 -1
- package/src/devserver/db/index.ts +1 -0
- package/src/devserver/db/types.ts +13 -0
- package/src/devserver/runtime/module.ts +7 -0
- package/test/analytics-dev.test.ts +2 -1
- package/test/devserver-database.test.ts +113 -0
- package/docs/auth-todo.md +0 -149
- package/docs/auth.md +0 -322
- package/docs/caching.md +0 -115
- package/docs/cli.md +0 -17
- package/docs/client.md +0 -39
- package/docs/cookies.md +0 -457
- package/docs/crypto.md +0 -130
- package/docs/daemon.md +0 -123
- package/docs/data.md +0 -131
- package/docs/derive.md +0 -159
- package/docs/email.md +0 -326
- package/docs/environment.md +0 -97
- package/docs/getting-started.md +0 -128
- package/docs/index.md +0 -30
- package/docs/ratelimit.md +0 -95
- package/docs/routing.md +0 -259
- package/docs/rpc.md +0 -149
- package/docs/server.md +0 -61
- package/docs/ssr.md +0 -632
- package/docs/streams.md +0 -178
- package/docs/styling.md +0 -22
- package/docs/tiers.md +0 -133
- package/docs/time.md +0 -43
|
@@ -16,8 +16,9 @@ import type { MemoryRef } from '../runtime/host.js';
|
|
|
16
16
|
|
|
17
17
|
/** Frame version + counter count, kept in lockstep with the edge (`analytics.rs` / `metric_id.rs`). */
|
|
18
18
|
const FRAME_VERSION = 2;
|
|
19
|
-
const METRIC_COUNTERS =
|
|
20
|
-
/** MetricId indices we seed with sample data (others default 0). Mirrors the wire contract
|
|
19
|
+
const METRIC_COUNTERS = 43;
|
|
20
|
+
/** MetricId indices we seed with sample data (others default 0). Mirrors the wire contract
|
|
21
|
+
* (counter ids 0..=42; gauge ids ConnectedStreamsAvg=43, CommittedMemoryAvg=45). */
|
|
21
22
|
const MID = {
|
|
22
23
|
Requests: 0,
|
|
23
24
|
BytesOutL1: 1,
|
|
@@ -33,6 +34,8 @@ const MID = {
|
|
|
33
34
|
StreamBytesIn: 25,
|
|
34
35
|
StreamBytesOut: 26,
|
|
35
36
|
MemGrownBytes: 39,
|
|
37
|
+
CacheHits: 41,
|
|
38
|
+
CacheMisses: 42,
|
|
36
39
|
} as const;
|
|
37
40
|
|
|
38
41
|
interface DevTenantStats {
|
|
@@ -63,6 +66,8 @@ function devStats(): DevTenantStats {
|
|
|
63
66
|
life[MID.StreamBytesIn] = 2048n;
|
|
64
67
|
life[MID.StreamBytesOut] = 8192n;
|
|
65
68
|
life[MID.MemGrownBytes] = 262144n;
|
|
69
|
+
life[MID.CacheHits] = 900n;
|
|
70
|
+
life[MID.CacheMisses] = 100n;
|
|
66
71
|
return {
|
|
67
72
|
life,
|
|
68
73
|
connectedStreams: 3,
|
|
@@ -103,7 +108,7 @@ function encodeStats(s: DevTenantStats): Buffer {
|
|
|
103
108
|
|
|
104
109
|
/** The metric ids carried in the per-minute ring (mirrors the edge `MINUTE_RING_METRICS`). Only these get
|
|
105
110
|
* minute resolution on the 1h/6h ranges; every other metric falls back to the hour ring there. */
|
|
106
|
-
const MINUTE_RING_METRICS = new Set<number>([0, 2, 1, 25, 26, 12, 13, 39,
|
|
111
|
+
const MINUTE_RING_METRICS = new Set<number>([0, 2, 1, 25, 26, 12, 13, 39, 43, 45]);
|
|
107
112
|
|
|
108
113
|
/**
|
|
109
114
|
* Range id -> (bucketCount, bucketSecs), matching the edge `Range` + minute-ring fallback: 1h/6h are
|
|
@@ -206,7 +211,8 @@ export function buildAnalyticsImports(
|
|
|
206
211
|
range: number,
|
|
207
212
|
): number => {
|
|
208
213
|
if (domainLen > MAX_DOMAIN_LEN) return ABSENT;
|
|
209
|
-
|
|
214
|
+
// Valid metric ids are 0..=46 (counters 0..=42 + the 4 gauge avg/peak series), ranges 0..=7.
|
|
215
|
+
if (metricId < 0 || metricId > 46 || range < 0 || range > 7) return ABSENT;
|
|
210
216
|
if (domainLen > 0) {
|
|
211
217
|
if (!ref.memory) throw new Error('analytics_series called before memory was bound');
|
|
212
218
|
const m = Buffer.from(ref.memory.buffer);
|
|
@@ -208,6 +208,7 @@ enum DbOp {
|
|
|
208
208
|
ViewGet,
|
|
209
209
|
ViewPublish,
|
|
210
210
|
CapacitySetTotal,
|
|
211
|
+
EventsSince,
|
|
211
212
|
}
|
|
212
213
|
|
|
213
214
|
function isReadOp(op: DbOp): boolean {
|
|
@@ -221,12 +222,13 @@ function isReadOp(op: DbOp): boolean {
|
|
|
221
222
|
op === DbOp.MembershipList ||
|
|
222
223
|
op === DbOp.UniqueLookup ||
|
|
223
224
|
op === DbOp.Latest ||
|
|
225
|
+
op === DbOp.EventsSince ||
|
|
224
226
|
op === DbOp.CapacityAvailable
|
|
225
227
|
);
|
|
226
228
|
}
|
|
227
229
|
|
|
228
230
|
function isScanOp(op: DbOp): boolean {
|
|
229
|
-
return op === DbOp.Latest || op === DbOp.MembershipList;
|
|
231
|
+
return op === DbOp.Latest || op === DbOp.MembershipList || op === DbOp.EventsSince;
|
|
230
232
|
}
|
|
231
233
|
|
|
232
234
|
function kindAllows(kind: DbFunctionKind, op: DbOp): boolean {
|
|
@@ -303,6 +305,11 @@ export class DevDatabase {
|
|
|
303
305
|
private readonly counterIdem = new Map<string, bigint>();
|
|
304
306
|
/** Events family: `"collection\0key"` -> append-ordered event blobs (oldest first). */
|
|
305
307
|
private readonly events = new Map<string, Buffer[]>();
|
|
308
|
+
/** Durable resumable `events.since` cursors, keyed `"<deriveId>\0<storeKey>"` -> count of events folded.
|
|
309
|
+
* A derive run stages its advances in `db.pendingCheckpoints` and only commits them here after its
|
|
310
|
+
* `derive_run` returns (publish-then-persist, mirroring the edge), so a trapped fold never advances the
|
|
311
|
+
* cursor past unpublished events. Persisted in the snapshot so a restart resumes the fold. */
|
|
312
|
+
private readonly deriveCheckpoints = new Map<string, number>();
|
|
306
313
|
/** append_once dedup: `"collection\0key"` -> set of eventIds already appended. */
|
|
307
314
|
private readonly eventDedup = new Map<string, Set<string>>();
|
|
308
315
|
/** Capacity family: `"collection\0key"` -> an escrow ledger (ceiling + reservations). */
|
|
@@ -410,6 +417,7 @@ export class DevDatabase {
|
|
|
410
417
|
counterIdem: {},
|
|
411
418
|
events: {},
|
|
412
419
|
eventDedup: {},
|
|
420
|
+
deriveCheckpoints: {},
|
|
413
421
|
capacity: {},
|
|
414
422
|
};
|
|
415
423
|
for (const [k, v] of this.store)
|
|
@@ -436,6 +444,7 @@ export class DevDatabase {
|
|
|
436
444
|
snap.events[k] = log.map((b, i) => ({ v: b.toString('base64'), sv: ver[i] ?? 0 }));
|
|
437
445
|
}
|
|
438
446
|
for (const [k, s] of this.eventDedup) snap.eventDedup[k] = [...s];
|
|
447
|
+
for (const [k, v] of this.deriveCheckpoints) snap.deriveCheckpoints![k] = v;
|
|
439
448
|
for (const [k, l] of this.capacity)
|
|
440
449
|
snap.capacity[k] = {
|
|
441
450
|
total: l.total.toString(),
|
|
@@ -510,6 +519,8 @@ export class DevDatabase {
|
|
|
510
519
|
}
|
|
511
520
|
for (const [k, ids] of Object.entries(snap.eventDedup ?? {}))
|
|
512
521
|
this.eventDedup.set(k, new Set(ids));
|
|
522
|
+
for (const [k, v] of Object.entries(snap.deriveCheckpoints ?? {}))
|
|
523
|
+
this.deriveCheckpoints.set(k, v);
|
|
513
524
|
for (const [k, l] of Object.entries(snap.capacity ?? {})) {
|
|
514
525
|
const res = new Map<bigint, Reservation>();
|
|
515
526
|
for (const [id, r] of l.reservations)
|
|
@@ -539,6 +550,7 @@ export class DevDatabase {
|
|
|
539
550
|
this.eventVersions.clear();
|
|
540
551
|
this.events.clear();
|
|
541
552
|
this.eventDedup.clear();
|
|
553
|
+
this.deriveCheckpoints.clear();
|
|
542
554
|
this.capacity.clear();
|
|
543
555
|
}
|
|
544
556
|
|
|
@@ -1207,6 +1219,52 @@ export class DevDatabase {
|
|
|
1207
1219
|
return db.lastResult.length;
|
|
1208
1220
|
}
|
|
1209
1221
|
|
|
1222
|
+
// `events.since(handle, keyPtr, keyLen, limit)`: the DERIVE-path incremental read. Returns the next
|
|
1223
|
+
// batch of events for the key PAST this derive's checkpoint, OLDEST first, and advances the cursor. The
|
|
1224
|
+
// advance is STAGED in `db.pendingCheckpoints` (this run only) and committed to the durable
|
|
1225
|
+
// `deriveCheckpoints` after `derive_run` returns (see `commitDeriveCheckpoints`), so a trapped fold
|
|
1226
|
+
// re-reads the batch next run instead of skipping it. Same wire frame as `latest`; empty once caught up.
|
|
1227
|
+
since(
|
|
1228
|
+
ref: MemoryRef,
|
|
1229
|
+
db: DbDevState,
|
|
1230
|
+
handle: number,
|
|
1231
|
+
keyPtr: number,
|
|
1232
|
+
keyLen: number,
|
|
1233
|
+
limit: number,
|
|
1234
|
+
): number {
|
|
1235
|
+
const coll = collForOp(db, handle, DbOp.EventsSince, CollectionFamily.Events);
|
|
1236
|
+
if (typeof coll === 'number') return coll;
|
|
1237
|
+
const sk = storeKey(coll.name, readKey(ref, keyPtr, keyLen));
|
|
1238
|
+
const ckKey = `${db.deriveId}${sk}`;
|
|
1239
|
+
const log = this.events.get(sk) ?? [];
|
|
1240
|
+
const vers = this.eventVersions.get(sk) ?? [];
|
|
1241
|
+
// Resume from this run's staged cursor if `since` was already called for this key, else the durable
|
|
1242
|
+
// checkpoint, else the start.
|
|
1243
|
+
const cursor = db.pendingCheckpoints.get(ckKey) ?? this.deriveCheckpoints.get(ckKey) ?? 0;
|
|
1244
|
+
const n = Math.max(0, Math.min(limit, 0xffff));
|
|
1245
|
+
const end = Math.min(log.length, cursor + n);
|
|
1246
|
+
const batch = log.slice(cursor, end); // oldest-first
|
|
1247
|
+
const batchVers = vers.slice(cursor, end);
|
|
1248
|
+
db.pendingCheckpoints.set(ckKey, end); // staged; committed only on derive success
|
|
1249
|
+
const w = new DataWriter();
|
|
1250
|
+
w.writeU32(batch.length);
|
|
1251
|
+
for (let i = 0; i < batch.length; i++) {
|
|
1252
|
+
w.writeU32(batchVers[i] ?? 0).writeBytes(batch[i]);
|
|
1253
|
+
}
|
|
1254
|
+
db.lastResult = Buffer.from(w.toBytes());
|
|
1255
|
+
return db.lastResult.length;
|
|
1256
|
+
}
|
|
1257
|
+
|
|
1258
|
+
/** Commit the `events.since` cursor advances a derive run staged in `db.pendingCheckpoints` into the
|
|
1259
|
+
* durable map. Call ONLY after the fold's `derive_run` returned (its `view.publish`es have landed in
|
|
1260
|
+
* this.views), so a trapped fold leaves the durable cursor untouched and the next run re-reads the
|
|
1261
|
+
* batch (publish-then-persist, mirroring the edge `commit_derive_checkpoints`). No-op if none staged. */
|
|
1262
|
+
commitDeriveCheckpoints(db: DbDevState): void {
|
|
1263
|
+
if (db.pendingCheckpoints.size === 0) return;
|
|
1264
|
+
for (const [k, v] of db.pendingCheckpoints) this.deriveCheckpoints.set(k, v);
|
|
1265
|
+
db.pendingCheckpoints.clear();
|
|
1266
|
+
}
|
|
1267
|
+
|
|
1210
1268
|
// --- capacity family (escrow: set_total / available / reserve / confirm / cancel) ---
|
|
1211
1269
|
|
|
1212
1270
|
// Set the ceiling (restock / reduce). Job/derive only (kind-gated upstream).
|
|
@@ -1426,6 +1484,12 @@ export function persistDb(): void {
|
|
|
1426
1484
|
devDb.persist();
|
|
1427
1485
|
}
|
|
1428
1486
|
|
|
1487
|
+
/** Commit the `events.since` checkpoints a derive run staged in `db.pendingCheckpoints` (call after a
|
|
1488
|
+
* successful `derive_run`; a trapped fold must NOT reach this). No-op if the run staged none. */
|
|
1489
|
+
export function commitDeriveCheckpoints(db: DbDevState): void {
|
|
1490
|
+
devDb.commitDeriveCheckpoints(db);
|
|
1491
|
+
}
|
|
1492
|
+
|
|
1429
1493
|
/**
|
|
1430
1494
|
* Build the `data.*` host imports for one instance, delegating to the shared
|
|
1431
1495
|
* {@link devDb}. `db` is the per-request state (handles + result stash); bind a
|
|
@@ -1588,6 +1652,8 @@ export function buildDatabaseImports(
|
|
|
1588
1652
|
|
|
1589
1653
|
'data.latest': (handle: number, keyPtr: number, keyLen: number, limit: number): number =>
|
|
1590
1654
|
devDb.latest(ref, db, handle, keyPtr, keyLen, limit),
|
|
1655
|
+
'data.events_since': (handle: number, keyPtr: number, keyLen: number, limit: number): number =>
|
|
1656
|
+
devDb.since(ref, db, handle, keyPtr, keyLen, limit),
|
|
1591
1657
|
|
|
1592
1658
|
'data.capacity_set_total': (
|
|
1593
1659
|
handle: number,
|
|
@@ -55,6 +55,14 @@ export interface DbDevState {
|
|
|
55
55
|
* Only populated for non-Derive dispatches (a derive's own writes must not
|
|
56
56
|
* re-trigger it - see `database.ts` `recordWrite`). */
|
|
57
57
|
writtenCollections: Set<string>;
|
|
58
|
+
/** The derive index currently running (set by `runDerive`), so `events.since` keys its resumable
|
|
59
|
+
* checkpoint per (derive, source key). 0 outside a derive. */
|
|
60
|
+
deriveId: number;
|
|
61
|
+
/** Checkpoint advances `events.since` staged during THIS derive run, keyed `"<deriveId>\0<storeKey>"` ->
|
|
62
|
+
* events-folded count. Committed to the durable `deriveCheckpoints` only after the fold's `derive_run`
|
|
63
|
+
* returns (its `view.publish`es have landed): a trapped fold discards this overlay, so the next run
|
|
64
|
+
* re-reads the batch rather than skipping it (publish-then-persist, mirroring the edge). */
|
|
65
|
+
pendingCheckpoints: Map<string, number>;
|
|
58
66
|
}
|
|
59
67
|
|
|
60
68
|
export function freshDbState(): DbDevState {
|
|
@@ -64,6 +72,8 @@ export function freshDbState(): DbDevState {
|
|
|
64
72
|
lastResultVersion: -1,
|
|
65
73
|
functionKind: DbFunctionKind.Job,
|
|
66
74
|
writtenCollections: new Set<string>(),
|
|
75
|
+
deriveId: 0,
|
|
76
|
+
pendingCheckpoints: new Map<string, number>(),
|
|
67
77
|
};
|
|
68
78
|
}
|
|
69
79
|
|
|
@@ -95,6 +105,9 @@ export interface DbSnapshot {
|
|
|
95
105
|
counterIdem?: Record<string, string>;
|
|
96
106
|
events: Record<string, { v: string; sv: number }[]>;
|
|
97
107
|
eventDedup: Record<string, string[]>;
|
|
108
|
+
/** Durable `events.since` cursors, `"<deriveId>\0<storeKey>"` -> events-folded count, so a restart
|
|
109
|
+
* resumes an incremental `@derive` instead of re-folding the whole log onto the persisted view. */
|
|
110
|
+
deriveCheckpoints?: Record<string, number>;
|
|
98
111
|
capacity: Record<
|
|
99
112
|
string,
|
|
100
113
|
{
|
|
@@ -14,6 +14,7 @@
|
|
|
14
14
|
import fs from 'node:fs';
|
|
15
15
|
|
|
16
16
|
import {
|
|
17
|
+
commitDeriveCheckpoints,
|
|
17
18
|
DbFunctionKind,
|
|
18
19
|
type DeriveEntry,
|
|
19
20
|
derivesForWrites,
|
|
@@ -154,6 +155,7 @@ const PROVIDED_IMPORTS = new Set([
|
|
|
154
155
|
'data.append_once',
|
|
155
156
|
'data.enqueue',
|
|
156
157
|
'data.latest',
|
|
158
|
+
'data.events_since',
|
|
157
159
|
'data.capacity_set_total',
|
|
158
160
|
'data.capacity_available',
|
|
159
161
|
'data.capacity_reserve',
|
|
@@ -401,11 +403,16 @@ export class WasmServerModule {
|
|
|
401
403
|
const ref: MemoryRef = { memory: null };
|
|
402
404
|
const state = freshDispatchState();
|
|
403
405
|
state.db.functionKind = DbFunctionKind.Derive;
|
|
406
|
+
state.db.deriveId = deriveId;
|
|
404
407
|
const instance = new WebAssembly.Instance(this.module, buildHostImports(ref, state));
|
|
405
408
|
const exports = instance.exports as unknown as DeriveExports;
|
|
406
409
|
ref.memory = exports.memory;
|
|
407
410
|
if (typeof exports.derive_run !== 'function') return;
|
|
408
411
|
exports.derive_run(deriveId);
|
|
412
|
+
// Publish-then-persist: derive_run returned (its `view.publish`es landed synchronously in the dev
|
|
413
|
+
// store), so commit the `events.since` cursor advances staged this run. A trapped fold throws above
|
|
414
|
+
// and never reaches here, leaving the durable cursor for a re-read on the next trigger.
|
|
415
|
+
commitDeriveCheckpoints(state.db);
|
|
409
416
|
}
|
|
410
417
|
|
|
411
418
|
/** Fail instantiation up front, with names, when the guest needs imports we do not provide. */
|
|
@@ -32,12 +32,13 @@ describe('dev analytics stub v2 frame parity', () => {
|
|
|
32
32
|
expect(u16()).toBe(2); // FRAME_VERSION
|
|
33
33
|
u64(); // now_ms
|
|
34
34
|
const count = u32();
|
|
35
|
-
expect(count).toBe(
|
|
35
|
+
expect(count).toBe(43); // METRIC_COUNTERS
|
|
36
36
|
const life: bigint[] = [];
|
|
37
37
|
for (let i = 0; i < count; i++) life.push(i64());
|
|
38
38
|
expect(life[0]).toBe(42n); // Requests
|
|
39
39
|
expect(life[1]).toBe(12345n); // BytesOutL1
|
|
40
40
|
expect(life[12]).toBe(1_000_000n); // GasUsed
|
|
41
|
+
expect(life[41]).toBe(900n); // CacheHits
|
|
41
42
|
expect(i64()).toBe(3n); // connectedStreams
|
|
42
43
|
expect(i64()).toBe(65536n); // committedMemory
|
|
43
44
|
expect(i64()).toBe(5n); // reqMinuteUsed
|
|
@@ -10,6 +10,7 @@ import {
|
|
|
10
10
|
__resetDbForTests,
|
|
11
11
|
__setDbCatalogForTests,
|
|
12
12
|
buildDatabaseImports,
|
|
13
|
+
commitDeriveCheckpoints,
|
|
13
14
|
configureDbPersistence,
|
|
14
15
|
derivesForWrites,
|
|
15
16
|
freshDbState,
|
|
@@ -637,6 +638,118 @@ describe('toildb dev emulator (record family)', () => {
|
|
|
637
638
|
expect(buf.readUInt32LE(300)).toBe(0);
|
|
638
639
|
});
|
|
639
640
|
|
|
641
|
+
it('events.since: drains oldest-first, stages within a run, commits on success, resumes across runs', () => {
|
|
642
|
+
const { ref, imports, buf, db } = setup();
|
|
643
|
+
const h = resolve(imports, buf, 'App/feed');
|
|
644
|
+
const [kPtr, kLen] = put(buf, 32, 'room1');
|
|
645
|
+
for (let i = 0; i < 3; i++) {
|
|
646
|
+
const [eP, eL] = put(buf, 48, `e${String(i)}`);
|
|
647
|
+
expect(imports['data.append'](h, kPtr, kLen, eP, eL, 0)).toBe(0);
|
|
648
|
+
}
|
|
649
|
+
|
|
650
|
+
// Decode a framed since batch (u32 count, then per-event u32 ver + u32 len + bytes), oldest-first.
|
|
651
|
+
const drain = (
|
|
652
|
+
imps: Record<string, (...a: number[]) => number>,
|
|
653
|
+
total: number,
|
|
654
|
+
at: number,
|
|
655
|
+
): string[] => {
|
|
656
|
+
expect(imps['data.take_result'](at, 256)).toBe(total);
|
|
657
|
+
let off = at;
|
|
658
|
+
const count = buf.readUInt32LE(off);
|
|
659
|
+
off += 4;
|
|
660
|
+
const out: string[] = [];
|
|
661
|
+
for (let i = 0; i < count; i++) {
|
|
662
|
+
off += 4; // schema_version (0 in dev)
|
|
663
|
+
const len = buf.readUInt32LE(off);
|
|
664
|
+
off += 4;
|
|
665
|
+
out.push(buf.toString('utf8', off, off + len));
|
|
666
|
+
off += len;
|
|
667
|
+
}
|
|
668
|
+
return out;
|
|
669
|
+
};
|
|
670
|
+
|
|
671
|
+
// A derive run drains the log in bounded batches, resuming from the STAGED cursor within the run.
|
|
672
|
+
db.functionKind = DbFunctionKind.Derive;
|
|
673
|
+
db.deriveId = 0;
|
|
674
|
+
expect(drain(imports, imports['data.events_since'](h, kPtr, kLen, 2), 512)).toEqual(['e0', 'e1']);
|
|
675
|
+
expect(drain(imports, imports['data.events_since'](h, kPtr, kLen, 2), 768)).toEqual(['e2']);
|
|
676
|
+
expect(drain(imports, imports['data.events_since'](h, kPtr, kLen, 2), 1024)).toEqual([]); // caught up
|
|
677
|
+
|
|
678
|
+
// The advance is only STAGED until the fold's derive_run returns; commit persists it.
|
|
679
|
+
expect(db.pendingCheckpoints.size).toBeGreaterThan(0);
|
|
680
|
+
commitDeriveCheckpoints(db);
|
|
681
|
+
expect(db.pendingCheckpoints.size).toBe(0);
|
|
682
|
+
|
|
683
|
+
// A fresh derive run (new per-request state, same shared store) resumes PAST the committed cursor:
|
|
684
|
+
// it re-reads nothing, then folds only a newly appended event.
|
|
685
|
+
const db2 = freshDbState();
|
|
686
|
+
db2.functionKind = DbFunctionKind.Derive;
|
|
687
|
+
db2.deriveId = 0;
|
|
688
|
+
const imports2 = buildDatabaseImports(ref, db2) as Record<string, (...a: number[]) => number>;
|
|
689
|
+
const h2 = resolve(imports2, buf, 'App/feed');
|
|
690
|
+
expect(drain(imports2, imports2['data.events_since'](h2, kPtr, kLen, 10), 512)).toEqual([]);
|
|
691
|
+
const [e3P, e3L] = put(buf, 48, 'e3');
|
|
692
|
+
expect(imports2['data.append'](h2, kPtr, kLen, e3P, e3L, 0)).toBe(0);
|
|
693
|
+
expect(drain(imports2, imports2['data.events_since'](h2, kPtr, kLen, 10), 512)).toEqual(['e3']);
|
|
694
|
+
});
|
|
695
|
+
|
|
696
|
+
it('events.since: a derive run that does not commit leaves the durable cursor unmoved (trap-safety)', () => {
|
|
697
|
+
const { ref, imports, buf, db } = setup();
|
|
698
|
+
const h = resolve(imports, buf, 'App/feed');
|
|
699
|
+
const [kPtr, kLen] = put(buf, 32, 'room1');
|
|
700
|
+
for (let i = 0; i < 2; i++) {
|
|
701
|
+
const [eP, eL] = put(buf, 48, `e${String(i)}`);
|
|
702
|
+
expect(imports['data.append'](h, kPtr, kLen, eP, eL, 0)).toBe(0);
|
|
703
|
+
}
|
|
704
|
+
const count = (imps: Record<string, (...a: number[]) => number>, total: number, at: number): number => {
|
|
705
|
+
expect(imps['data.take_result'](at, 256)).toBe(total);
|
|
706
|
+
return buf.readUInt32LE(at);
|
|
707
|
+
};
|
|
708
|
+
|
|
709
|
+
// Run 1 drains both events but NEVER commits (simulating a fold that trapped before returning): the
|
|
710
|
+
// staged advance lives only on this discarded per-request state.
|
|
711
|
+
db.functionKind = DbFunctionKind.Derive;
|
|
712
|
+
db.deriveId = 0;
|
|
713
|
+
expect(count(imports, imports['data.events_since'](h, kPtr, kLen, 10), 512)).toBe(2);
|
|
714
|
+
|
|
715
|
+
// Run 2 (fresh per-request state) still sees BOTH events: the durable cursor never moved.
|
|
716
|
+
const db2 = freshDbState();
|
|
717
|
+
db2.functionKind = DbFunctionKind.Derive;
|
|
718
|
+
db2.deriveId = 0;
|
|
719
|
+
const imports2 = buildDatabaseImports(ref, db2) as Record<string, (...a: number[]) => number>;
|
|
720
|
+
const h2 = resolve(imports2, buf, 'App/feed');
|
|
721
|
+
expect(count(imports2, imports2['data.events_since'](h2, kPtr, kLen, 10), 512)).toBe(2);
|
|
722
|
+
});
|
|
723
|
+
|
|
724
|
+
it('__resetDbForTests clears events.since checkpoints (no cross-test cursor leak)', () => {
|
|
725
|
+
{
|
|
726
|
+
const { imports, buf, db } = setup();
|
|
727
|
+
const h = resolve(imports, buf, 'App/feed');
|
|
728
|
+
const [kPtr, kLen] = put(buf, 32, 'room1');
|
|
729
|
+
const [eP, eL] = put(buf, 48, 'e0');
|
|
730
|
+
expect(imports['data.append'](h, kPtr, kLen, eP, eL, 0)).toBe(0);
|
|
731
|
+
db.functionKind = DbFunctionKind.Derive;
|
|
732
|
+
db.deriveId = 0;
|
|
733
|
+
imports['data.events_since'](h, kPtr, kLen, 10);
|
|
734
|
+
commitDeriveCheckpoints(db); // durable cursor now at 1
|
|
735
|
+
}
|
|
736
|
+
__resetDbForTests();
|
|
737
|
+
{
|
|
738
|
+
// A brand-new store reusing the same (deriveId, key): the cursor must be back at 0, so a fresh
|
|
739
|
+
// append is read, not skipped by a stale checkpoint.
|
|
740
|
+
const { imports, buf, db } = setup();
|
|
741
|
+
const h = resolve(imports, buf, 'App/feed');
|
|
742
|
+
const [kPtr, kLen] = put(buf, 32, 'room1');
|
|
743
|
+
const [eP, eL] = put(buf, 48, 'fresh0');
|
|
744
|
+
expect(imports['data.append'](h, kPtr, kLen, eP, eL, 0)).toBe(0);
|
|
745
|
+
db.functionKind = DbFunctionKind.Derive;
|
|
746
|
+
db.deriveId = 0;
|
|
747
|
+
const total = imports['data.events_since'](h, kPtr, kLen, 10);
|
|
748
|
+
expect(imports['data.take_result'](512, 256)).toBe(total);
|
|
749
|
+
expect(buf.readUInt32LE(512)).toBe(1); // the fresh event, not skipped
|
|
750
|
+
}
|
|
751
|
+
});
|
|
752
|
+
|
|
640
753
|
it('collections are isolated (no key aliasing)', () => {
|
|
641
754
|
const { imports, buf } = setup();
|
|
642
755
|
const users = resolve(imports, buf, 'App/users');
|
package/docs/auth-todo.md
DELETED
|
@@ -1,149 +0,0 @@
|
|
|
1
|
-
# Auth: production TODO and toildb migration
|
|
2
|
-
|
|
3
|
-
Tracking doc for finishing the Toil PQ-Auth system (shipped experimental in
|
|
4
|
-
**v0.0.52**). The cryptographic core is in place; what is left is the deployment
|
|
5
|
-
substrate, protocol hardening, and account lifecycle.
|
|
6
|
-
|
|
7
|
-
> This system is a **hybrid, experimental** password-authenticated login and is **NOT a framework
|
|
8
|
-
> default until externally reviewed** (see Tier 2.8). It is opt-in: a tenant must wire
|
|
9
|
-
> it. The OPRF layer is classical ristretto255 (the one non-PQ piece); auth (ML-DSA-44)
|
|
10
|
-
> and key agreement (ML-KEM-768) are post-quantum.
|
|
11
|
-
|
|
12
|
-
## What ships today (v0.0.52)
|
|
13
|
-
|
|
14
|
-
- OPRF keyed salt (RFC 9497 ristretto255-SHA512, mode 0x00, per-user key) -> precomputation resistance.
|
|
15
|
-
- ML-DSA-44 client auth + registration proof-of-possession.
|
|
16
|
-
- ML-KEM-768 mutual auth: client encapsulates to a pinned server key, server returns a confirmation tag.
|
|
17
|
-
- Login enumeration fixed (deterministic per-user salt).
|
|
18
|
-
- Atomic single-use challenge consume (shape is correct; storage is the dev KV, see below).
|
|
19
|
-
|
|
20
|
-
Key files: `server/globals/auth.ts` (the `AuthService` global), `src/client/auth.ts`
|
|
21
|
-
(browser), `src/devserver/{crypto,kv}.ts` (dev host mocks + dev KV), the Rust edge host
|
|
22
|
-
imports in `toil-backend` (`crypto.mlkem_decapsulate`, `crypto.voprf_evaluate`),
|
|
23
|
-
`examples/basic/server/routes/Auth.ts` (the flow).
|
|
24
|
-
|
|
25
|
-
---
|
|
26
|
-
|
|
27
|
-
## Tier 1, blocks production (cannot run on the edge yet)
|
|
28
|
-
|
|
29
|
-
### 1.1 Storage on toildb (THE blocker) -- "when building toildb, consider this"
|
|
30
|
-
The accounts + login challenges live in the **DEV-ONLY** `kv.*` Map
|
|
31
|
-
(`src/devserver/kv.ts`), which is intentionally **not** on the production edge
|
|
32
|
-
(`toil-backend` `HOST_IMPORTS`), so a module importing `kv.*` will not instantiate in
|
|
33
|
-
prod. When toildb lands:
|
|
34
|
-
- **toildb MUST provide an atomic fetch-and-delete** (single-call read+remove). The
|
|
35
|
-
login challenge consume relies on it; a read-then-delete race makes a login replayable.
|
|
36
|
-
It also needs `get` and `put`.
|
|
37
|
-
- Migrate **two stores**: `Accounts` (username -> {salt, params, publicKey}) and
|
|
38
|
-
`Challenges` (cid -> {username, nonce, iat, exp}) in `examples/basic/server/routes/Auth.ts`.
|
|
39
|
-
- Add the toildb host imports to `toil-backend` `HOST_IMPORTS` + `define_*_imports`.
|
|
40
|
-
- Then **delete `src/devserver/kv.ts`**, the `kv.*` entries in `module.ts`
|
|
41
|
-
`PROVIDED_IMPORTS`, and the `buildKvImports` wiring in `host.ts`. Grep `REMOVE KV LATER`.
|
|
42
|
-
|
|
43
|
-
### 1.2 Real secrets (not dev placeholders)
|
|
44
|
-
All three must come from the managed env / `getSecure` store, per-deployment, identical
|
|
45
|
-
across edge instances, and rotatable:
|
|
46
|
-
- session HMAC secret (still the `...CHANGE-ME` default in `auth.ts`),
|
|
47
|
-
- OPRF master seed (currently `sha256Text('toil-demo-oprf-seed-v1')`),
|
|
48
|
-
- **server ML-KEM keypair** -- generate a real one per deployment and pin **its** public
|
|
49
|
-
key in the client (today a fixed demo key is pinned in `src/client/auth.ts`, secret in
|
|
50
|
-
the example `Auth.ts`).
|
|
51
|
-
|
|
52
|
-
### 1.3 Rate limiting on the auth routes
|
|
53
|
-
`@ratelimit` exists (`docs/ratelimit.md`) but is not wired onto register/login. Add
|
|
54
|
-
per-username + per-IP throttling with backoff/lockout. Offline resistance is good; online
|
|
55
|
-
guessing + registration spam are currently open.
|
|
56
|
-
|
|
57
|
-
### 1.4 Production Argon2id params
|
|
58
|
-
Demo uses 32 MiB / 2 iters for tab responsiveness. Raise to >= 256 MiB / >= 3 iters and
|
|
59
|
-
enforce a server-side floor.
|
|
60
|
-
|
|
61
|
-
### 1.5 Gas calibration (edge metering)
|
|
62
|
-
`MLKEM768_DECAPSULATE` and `VOPRF_EVALUATE` (`toil-backend src/config.rs`) are
|
|
63
|
-
**conservative 12,000,000 placeholders copied from `MLDSA44_VERIFY`, not benchmarked**.
|
|
64
|
-
They are charged correctly (before reads) and DoS-bounded by a test, and over-charge
|
|
65
|
-
(fail-closed safe), but should be calibrated: measure cyc/op via `crypto::bench`,
|
|
66
|
-
multiply by `GAS_PER_CYCLE`, round up ~30%, replace. Same TODO as `MLDSA44_VERIFY` itself.
|
|
67
|
-
|
|
68
|
-
### 1.6 Real-edge end-to-end
|
|
69
|
-
Never run: a full guest-dispatch through the **running** Rust edge exercising the two new
|
|
70
|
-
imports (only proven at unit + interop-vector level + structurally identical to the
|
|
71
|
-
in-prod `mldsa_verify` import). Do before trusting in prod.
|
|
72
|
-
|
|
73
|
-
---
|
|
74
|
-
|
|
75
|
-
## Tier 2, protocol hardening
|
|
76
|
-
|
|
77
|
-
### 2.6 A properly bound session key, DONE (on main, unreleased)
|
|
78
|
-
The session key is now `K = HMAC-SHA256(sharedSecret, SESSION_KEY_LABEL || H(M))` and the
|
|
79
|
-
mutual-auth tag is `HMAC-SHA256(K, SERVER_CONFIRM_LABEL || H(M))` (`AuthService.deriveSessionKey`
|
|
80
|
-
+ `serverConfirmTag`; client mirrors it with hash-wasm `createHMAC`). REMAINING: binding the
|
|
81
|
-
*session cookie* to the transport (so a stolen cookie is useless on another channel) needs
|
|
82
|
-
the TLS exporter, which the wasm guest cannot see, an edge/transport follow-up, not doable
|
|
83
|
-
purely in the guest.
|
|
84
|
-
|
|
85
|
-
### 2.7 Bind the KDF params + server key into the transcript, DONE (on main, unreleased)
|
|
86
|
-
The single `buildLoginMessage` now binds the ML-KEM ciphertext, the Argon2id params
|
|
87
|
-
(mem/iters/par), and `serverKemKeyId = SHA-256(serverKemPublicKey)`. Closes
|
|
88
|
-
key-substitution and param-downgrade confusion. (There is ONE login message format, no
|
|
89
|
-
versioned variants.)
|
|
90
|
-
|
|
91
|
-
### 2.8 External cryptographic review (the gate)
|
|
92
|
-
Hand-rolled KEM + signature + OPRF composition with no security proof. Transcript binding
|
|
93
|
-
especially needs a cryptographer. This is a human task; it cannot be self-served. Make it
|
|
94
|
-
tractable: write the wire spec (messages, what each signature/MAC covers, the KDF tree) as
|
|
95
|
-
a reviewable artifact.
|
|
96
|
-
|
|
97
|
-
---
|
|
98
|
-
|
|
99
|
-
## Tier 3, account lifecycle (missing today)
|
|
100
|
-
|
|
101
|
-
### 3.1 Password change / key rotation
|
|
102
|
-
Re-derive under a fresh salt while authenticated, re-register the new public key. None
|
|
103
|
-
exists today.
|
|
104
|
-
|
|
105
|
-
### 3.2 Recovery
|
|
106
|
-
"The password IS the key, no recovery" -> a forgotten password is a permanently dead
|
|
107
|
-
account. Needs recovery codes or a second factor (ties to the 2FA crate already on the
|
|
108
|
-
backend "left to do" list + the existing email support).
|
|
109
|
-
|
|
110
|
-
### 3.3 Revocation
|
|
111
|
-
No way to kill a compromised credential except deleting the account.
|
|
112
|
-
|
|
113
|
-
### 3.4 Tighten registration enumeration
|
|
114
|
-
`register/finish` still leaks "taken" via the generic fail after the PoP round.
|
|
115
|
-
|
|
116
|
-
---
|
|
117
|
-
|
|
118
|
-
## Design decisions (open)
|
|
119
|
-
|
|
120
|
-
### D1. User id = sha256(public key)? -- CAUTION: breaks under key rotation
|
|
121
|
-
Yes, the public key is still the verifier (`AuthService.verifyLogin` checks the ML-DSA
|
|
122
|
-
signature against the stored public key). `sha256(publicKey)` is a fine **content-addressed
|
|
123
|
-
credential fingerprint** (collision-resistant, re-derivable, good for dedup). BUT:
|
|
124
|
-
- It **cannot be the login lookup key**: at `login/start` the client presents an identifier
|
|
125
|
-
to fetch the salt + OPRF evaluation *before* it has derived its keypair (it needs the
|
|
126
|
-
OPRF response to derive the key -> to get the pubkey). So a human-meaningful **handle**
|
|
127
|
-
(username/email) is still required as the OPRF `info` + the `login/start` lookup key.
|
|
128
|
-
- It is **not stable across password change**: rotating the password changes the pubkey,
|
|
129
|
-
hence `sha256(pubkey)`. If the user id must survive a password change, it must be a
|
|
130
|
-
separate stable id (random UUID assigned at registration), with `sha256(pubkey)` treated
|
|
131
|
-
as the *current credential* fingerprint, not the permanent identity.
|
|
132
|
-
- **Recommendation:** handle (username/email) = login + OPRF info; permanent userId =
|
|
133
|
-
random id at registration; `sha256(pubkey)` = credential fingerprint (verifiable, for
|
|
134
|
-
dedup / "is this the same key"). Revisit only if password rotation is ruled out.
|
|
135
|
-
|
|
136
|
-
### D2. OPRF mode vs VOPRF/DLEQ
|
|
137
|
-
Plain OPRF mode by design (server auth comes from the ML-KEM layer). Add
|
|
138
|
-
DLEQ only if the client must verify OPRF-key consistency. Likely unnecessary.
|
|
139
|
-
|
|
140
|
-
### D3. Multi-device / WebAuthn-style per-device keys
|
|
141
|
-
One-key-per-user (deterministic from password) is fine unless hardware-backed or per-device
|
|
142
|
-
keys are wanted later; that needs a credential-set model.
|
|
143
|
-
|
|
144
|
-
---
|
|
145
|
-
|
|
146
|
-
## Suggested sequence
|
|
147
|
-
toildb storage (1.1) -> real secrets + real pinned KEM key (1.2) -> rate limiting (1.3) ->
|
|
148
|
-
HKDF session key (2.6) + transcript binding (2.7) -> external review (2.8) -> lifecycle
|
|
149
|
-
(3.1-3.3). 1.1-1.3 turn the demo into something deployable; 2.8 makes it trustworthy.
|