toiljs 0.0.85 → 0.0.87

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (132) hide show
  1. package/CHANGELOG.md +5 -0
  2. package/README.md +2 -2
  3. package/build/cli/.tsbuildinfo +1 -1
  4. package/build/cli/index.js +303 -293
  5. package/build/compiler/.tsbuildinfo +1 -1
  6. package/build/compiler/config.d.ts +2 -0
  7. package/build/compiler/config.js +1 -0
  8. package/build/compiler/docs.js +8 -24
  9. package/build/compiler/generate.js +4 -2
  10. package/build/compiler/index.d.ts +1 -1
  11. package/build/compiler/index.js +69 -6
  12. package/build/compiler/toil-docs.generated.js +64 -22
  13. package/build/devserver/.tsbuildinfo +1 -1
  14. package/build/devserver/analytics/index.js +7 -3
  15. package/build/devserver/db/database.d.ts +4 -0
  16. package/build/devserver/db/database.js +43 -1
  17. package/build/devserver/db/index.d.ts +1 -1
  18. package/build/devserver/db/index.js +1 -1
  19. package/build/devserver/db/types.d.ts +3 -0
  20. package/build/devserver/db/types.js +2 -0
  21. package/build/devserver/runtime/module.js +4 -1
  22. package/docs/README.md +104 -65
  23. package/docs/auth/README.md +102 -0
  24. package/docs/auth/configuration.md +94 -0
  25. package/docs/auth/extending.md +202 -0
  26. package/docs/auth/how-it-works.md +138 -0
  27. package/docs/auth/usage.md +188 -0
  28. package/docs/backend/README.md +143 -0
  29. package/docs/backend/data.md +351 -0
  30. package/docs/backend/rest.md +402 -0
  31. package/docs/backend/rpc.md +226 -0
  32. package/docs/background/README.md +114 -0
  33. package/docs/background/daemons.md +230 -0
  34. package/docs/background/derive.md +179 -0
  35. package/docs/cli/README.md +377 -0
  36. package/docs/concepts/config.md +416 -0
  37. package/docs/concepts/decorators.md +127 -0
  38. package/docs/concepts/security.md +108 -0
  39. package/docs/concepts/tiers.md +166 -0
  40. package/docs/concepts/types.md +216 -0
  41. package/docs/database/README.md +143 -0
  42. package/docs/database/capacity.md +350 -0
  43. package/docs/database/counters.md +174 -0
  44. package/docs/database/documents.md +255 -0
  45. package/docs/database/events.md +307 -0
  46. package/docs/database/membership.md +246 -0
  47. package/docs/database/setup.md +155 -0
  48. package/docs/database/unique.md +216 -0
  49. package/docs/database/views.md +246 -0
  50. package/docs/frontend/README.md +101 -0
  51. package/docs/frontend/data-fetching.md +243 -0
  52. package/docs/frontend/images.md +148 -0
  53. package/docs/frontend/metadata.md +344 -0
  54. package/docs/frontend/rendering.md +236 -0
  55. package/docs/frontend/routing.md +344 -0
  56. package/docs/frontend/scripts.md +118 -0
  57. package/docs/frontend/search.md +191 -0
  58. package/docs/frontend/styling.md +147 -0
  59. package/docs/getting-started/README.md +81 -0
  60. package/docs/getting-started/create-project.md +131 -0
  61. package/docs/getting-started/deploy.md +101 -0
  62. package/docs/getting-started/first-app.md +215 -0
  63. package/docs/getting-started/installation.md +106 -0
  64. package/docs/getting-started/migrating.md +125 -0
  65. package/docs/getting-started/project-structure.md +163 -0
  66. package/docs/introduction/README.md +41 -0
  67. package/docs/introduction/design-principles.md +55 -0
  68. package/docs/introduction/distributed.md +74 -0
  69. package/docs/introduction/how-it-works.md +74 -0
  70. package/docs/introduction/hyperscale.md +51 -0
  71. package/docs/introduction/modern-stack.md +36 -0
  72. package/docs/introduction/vs-other-frameworks.md +48 -0
  73. package/docs/introduction/why-toil.md +93 -0
  74. package/docs/llms.txt +90 -0
  75. package/docs/realtime/README.md +102 -0
  76. package/docs/realtime/channels.md +211 -0
  77. package/docs/realtime/streams.md +369 -0
  78. package/docs/services/README.md +60 -0
  79. package/docs/services/analytics.md +268 -0
  80. package/docs/services/caching.md +175 -0
  81. package/docs/services/cookies.md +235 -0
  82. package/docs/services/crypto.md +209 -0
  83. package/docs/services/email.md +289 -0
  84. package/docs/services/environment.md +141 -0
  85. package/docs/services/ratelimit.md +174 -0
  86. package/docs/services/time.md +85 -0
  87. package/examples/basic/client/routes/analytics.tsx +13 -12
  88. package/examples/basic/server/models/SiteAnalytics.ts +29 -17
  89. package/examples/basic/server/routes/Analytics.ts +15 -17
  90. package/examples/basic/server/routes/UserId.ts +22 -0
  91. package/package.json +15 -11
  92. package/scripts/gen-toil-docs.mjs +24 -35
  93. package/server/auth/AuthController.ts +336 -0
  94. package/server/auth/AuthUser.ts +23 -0
  95. package/server/auth/index.ts +16 -0
  96. package/server/globals/auth.ts +31 -0
  97. package/server/globals/userid.ts +107 -0
  98. package/src/compiler/config.ts +13 -0
  99. package/src/compiler/docs.ts +16 -33
  100. package/src/compiler/generate.ts +6 -2
  101. package/src/compiler/index.ts +114 -6
  102. package/src/compiler/toil-docs.generated.ts +64 -22
  103. package/src/devserver/analytics/index.ts +10 -4
  104. package/src/devserver/db/database.ts +67 -1
  105. package/src/devserver/db/index.ts +1 -0
  106. package/src/devserver/db/types.ts +13 -0
  107. package/src/devserver/runtime/module.ts +7 -0
  108. package/test/analytics-dev.test.ts +2 -1
  109. package/test/devserver-database.test.ts +113 -0
  110. package/docs/auth-todo.md +0 -149
  111. package/docs/auth.md +0 -322
  112. package/docs/caching.md +0 -115
  113. package/docs/cli.md +0 -17
  114. package/docs/client.md +0 -39
  115. package/docs/cookies.md +0 -457
  116. package/docs/crypto.md +0 -130
  117. package/docs/daemon.md +0 -123
  118. package/docs/data.md +0 -131
  119. package/docs/derive.md +0 -159
  120. package/docs/email.md +0 -326
  121. package/docs/environment.md +0 -97
  122. package/docs/getting-started.md +0 -128
  123. package/docs/index.md +0 -30
  124. package/docs/ratelimit.md +0 -95
  125. package/docs/routing.md +0 -259
  126. package/docs/rpc.md +0 -149
  127. package/docs/server.md +0 -61
  128. package/docs/ssr.md +0 -632
  129. package/docs/streams.md +0 -178
  130. package/docs/styling.md +0 -22
  131. package/docs/tiers.md +0 -133
  132. package/docs/time.md +0 -43
@@ -16,8 +16,9 @@ import type { MemoryRef } from '../runtime/host.js';
16
16
 
17
17
  /** Frame version + counter count, kept in lockstep with the edge (`analytics.rs` / `metric_id.rs`). */
18
18
  const FRAME_VERSION = 2;
19
- const METRIC_COUNTERS = 41;
20
- /** MetricId indices we seed with sample data (others default 0). Mirrors the wire contract. */
19
+ const METRIC_COUNTERS = 43;
20
+ /** MetricId indices we seed with sample data (others default 0). Mirrors the wire contract
21
+ * (counter ids 0..=42; gauge ids ConnectedStreamsAvg=43, CommittedMemoryAvg=45). */
21
22
  const MID = {
22
23
  Requests: 0,
23
24
  BytesOutL1: 1,
@@ -33,6 +34,8 @@ const MID = {
33
34
  StreamBytesIn: 25,
34
35
  StreamBytesOut: 26,
35
36
  MemGrownBytes: 39,
37
+ CacheHits: 41,
38
+ CacheMisses: 42,
36
39
  } as const;
37
40
 
38
41
  interface DevTenantStats {
@@ -63,6 +66,8 @@ function devStats(): DevTenantStats {
63
66
  life[MID.StreamBytesIn] = 2048n;
64
67
  life[MID.StreamBytesOut] = 8192n;
65
68
  life[MID.MemGrownBytes] = 262144n;
69
+ life[MID.CacheHits] = 900n;
70
+ life[MID.CacheMisses] = 100n;
66
71
  return {
67
72
  life,
68
73
  connectedStreams: 3,
@@ -103,7 +108,7 @@ function encodeStats(s: DevTenantStats): Buffer {
103
108
 
104
109
  /** The metric ids carried in the per-minute ring (mirrors the edge `MINUTE_RING_METRICS`). Only these get
105
110
  * minute resolution on the 1h/6h ranges; every other metric falls back to the hour ring there. */
106
- const MINUTE_RING_METRICS = new Set<number>([0, 2, 1, 25, 26, 12, 13, 39, 41, 43]);
111
+ const MINUTE_RING_METRICS = new Set<number>([0, 2, 1, 25, 26, 12, 13, 39, 43, 45]);
107
112
 
108
113
  /**
109
114
  * Range id -> (bucketCount, bucketSecs), matching the edge `Range` + minute-ring fallback: 1h/6h are
@@ -206,7 +211,8 @@ export function buildAnalyticsImports(
206
211
  range: number,
207
212
  ): number => {
208
213
  if (domainLen > MAX_DOMAIN_LEN) return ABSENT;
209
- if (metricId < 0 || metricId > 44 || range < 0 || range > 7) return ABSENT;
214
+ // Valid metric ids are 0..=46 (counters 0..=42 + the 4 gauge avg/peak series), ranges 0..=7.
215
+ if (metricId < 0 || metricId > 46 || range < 0 || range > 7) return ABSENT;
210
216
  if (domainLen > 0) {
211
217
  if (!ref.memory) throw new Error('analytics_series called before memory was bound');
212
218
  const m = Buffer.from(ref.memory.buffer);
@@ -208,6 +208,7 @@ enum DbOp {
208
208
  ViewGet,
209
209
  ViewPublish,
210
210
  CapacitySetTotal,
211
+ EventsSince,
211
212
  }
212
213
 
213
214
  function isReadOp(op: DbOp): boolean {
@@ -221,12 +222,13 @@ function isReadOp(op: DbOp): boolean {
221
222
  op === DbOp.MembershipList ||
222
223
  op === DbOp.UniqueLookup ||
223
224
  op === DbOp.Latest ||
225
+ op === DbOp.EventsSince ||
224
226
  op === DbOp.CapacityAvailable
225
227
  );
226
228
  }
227
229
 
228
230
  function isScanOp(op: DbOp): boolean {
229
- return op === DbOp.Latest || op === DbOp.MembershipList;
231
+ return op === DbOp.Latest || op === DbOp.MembershipList || op === DbOp.EventsSince;
230
232
  }
231
233
 
232
234
  function kindAllows(kind: DbFunctionKind, op: DbOp): boolean {
@@ -303,6 +305,11 @@ export class DevDatabase {
303
305
  private readonly counterIdem = new Map<string, bigint>();
304
306
  /** Events family: `"collection\0key"` -> append-ordered event blobs (oldest first). */
305
307
  private readonly events = new Map<string, Buffer[]>();
308
+ /** Durable resumable `events.since` cursors, keyed `"<deriveId>\0<storeKey>"` -> count of events folded.
309
+ * A derive run stages its advances in `db.pendingCheckpoints` and only commits them here after its
310
+ * `derive_run` returns (publish-then-persist, mirroring the edge), so a trapped fold never advances the
311
+ * cursor past unpublished events. Persisted in the snapshot so a restart resumes the fold. */
312
+ private readonly deriveCheckpoints = new Map<string, number>();
306
313
  /** append_once dedup: `"collection\0key"` -> set of eventIds already appended. */
307
314
  private readonly eventDedup = new Map<string, Set<string>>();
308
315
  /** Capacity family: `"collection\0key"` -> an escrow ledger (ceiling + reservations). */
@@ -410,6 +417,7 @@ export class DevDatabase {
410
417
  counterIdem: {},
411
418
  events: {},
412
419
  eventDedup: {},
420
+ deriveCheckpoints: {},
413
421
  capacity: {},
414
422
  };
415
423
  for (const [k, v] of this.store)
@@ -436,6 +444,7 @@ export class DevDatabase {
436
444
  snap.events[k] = log.map((b, i) => ({ v: b.toString('base64'), sv: ver[i] ?? 0 }));
437
445
  }
438
446
  for (const [k, s] of this.eventDedup) snap.eventDedup[k] = [...s];
447
+ for (const [k, v] of this.deriveCheckpoints) snap.deriveCheckpoints![k] = v;
439
448
  for (const [k, l] of this.capacity)
440
449
  snap.capacity[k] = {
441
450
  total: l.total.toString(),
@@ -510,6 +519,8 @@ export class DevDatabase {
510
519
  }
511
520
  for (const [k, ids] of Object.entries(snap.eventDedup ?? {}))
512
521
  this.eventDedup.set(k, new Set(ids));
522
+ for (const [k, v] of Object.entries(snap.deriveCheckpoints ?? {}))
523
+ this.deriveCheckpoints.set(k, v);
513
524
  for (const [k, l] of Object.entries(snap.capacity ?? {})) {
514
525
  const res = new Map<bigint, Reservation>();
515
526
  for (const [id, r] of l.reservations)
@@ -539,6 +550,7 @@ export class DevDatabase {
539
550
  this.eventVersions.clear();
540
551
  this.events.clear();
541
552
  this.eventDedup.clear();
553
+ this.deriveCheckpoints.clear();
542
554
  this.capacity.clear();
543
555
  }
544
556
 
@@ -1207,6 +1219,52 @@ export class DevDatabase {
1207
1219
  return db.lastResult.length;
1208
1220
  }
1209
1221
 
1222
+ // `events.since(handle, keyPtr, keyLen, limit)`: the DERIVE-path incremental read. Returns the next
1223
+ // batch of events for the key PAST this derive's checkpoint, OLDEST first, and advances the cursor. The
1224
+ // advance is STAGED in `db.pendingCheckpoints` (this run only) and committed to the durable
1225
+ // `deriveCheckpoints` after `derive_run` returns (see `commitDeriveCheckpoints`), so a trapped fold
1226
+ // re-reads the batch next run instead of skipping it. Same wire frame as `latest`; empty once caught up.
1227
+ since(
1228
+ ref: MemoryRef,
1229
+ db: DbDevState,
1230
+ handle: number,
1231
+ keyPtr: number,
1232
+ keyLen: number,
1233
+ limit: number,
1234
+ ): number {
1235
+ const coll = collForOp(db, handle, DbOp.EventsSince, CollectionFamily.Events);
1236
+ if (typeof coll === 'number') return coll;
1237
+ const sk = storeKey(coll.name, readKey(ref, keyPtr, keyLen));
1238
+ const ckKey = `${db.deriveId}${sk}`;
1239
+ const log = this.events.get(sk) ?? [];
1240
+ const vers = this.eventVersions.get(sk) ?? [];
1241
+ // Resume from this run's staged cursor if `since` was already called for this key, else the durable
1242
+ // checkpoint, else the start.
1243
+ const cursor = db.pendingCheckpoints.get(ckKey) ?? this.deriveCheckpoints.get(ckKey) ?? 0;
1244
+ const n = Math.max(0, Math.min(limit, 0xffff));
1245
+ const end = Math.min(log.length, cursor + n);
1246
+ const batch = log.slice(cursor, end); // oldest-first
1247
+ const batchVers = vers.slice(cursor, end);
1248
+ db.pendingCheckpoints.set(ckKey, end); // staged; committed only on derive success
1249
+ const w = new DataWriter();
1250
+ w.writeU32(batch.length);
1251
+ for (let i = 0; i < batch.length; i++) {
1252
+ w.writeU32(batchVers[i] ?? 0).writeBytes(batch[i]);
1253
+ }
1254
+ db.lastResult = Buffer.from(w.toBytes());
1255
+ return db.lastResult.length;
1256
+ }
1257
+
1258
+ /** Commit the `events.since` cursor advances a derive run staged in `db.pendingCheckpoints` into the
1259
+ * durable map. Call ONLY after the fold's `derive_run` returned (its `view.publish`es have landed in
1260
+ * this.views), so a trapped fold leaves the durable cursor untouched and the next run re-reads the
1261
+ * batch (publish-then-persist, mirroring the edge `commit_derive_checkpoints`). No-op if none staged. */
1262
+ commitDeriveCheckpoints(db: DbDevState): void {
1263
+ if (db.pendingCheckpoints.size === 0) return;
1264
+ for (const [k, v] of db.pendingCheckpoints) this.deriveCheckpoints.set(k, v);
1265
+ db.pendingCheckpoints.clear();
1266
+ }
1267
+
1210
1268
  // --- capacity family (escrow: set_total / available / reserve / confirm / cancel) ---
1211
1269
 
1212
1270
  // Set the ceiling (restock / reduce). Job/derive only (kind-gated upstream).
@@ -1426,6 +1484,12 @@ export function persistDb(): void {
1426
1484
  devDb.persist();
1427
1485
  }
1428
1486
 
1487
+ /** Commit the `events.since` checkpoints a derive run staged in `db.pendingCheckpoints` (call after a
1488
+ * successful `derive_run`; a trapped fold must NOT reach this). No-op if the run staged none. */
1489
+ export function commitDeriveCheckpoints(db: DbDevState): void {
1490
+ devDb.commitDeriveCheckpoints(db);
1491
+ }
1492
+
1429
1493
  /**
1430
1494
  * Build the `data.*` host imports for one instance, delegating to the shared
1431
1495
  * {@link devDb}. `db` is the per-request state (handles + result stash); bind a
@@ -1588,6 +1652,8 @@ export function buildDatabaseImports(
1588
1652
 
1589
1653
  'data.latest': (handle: number, keyPtr: number, keyLen: number, limit: number): number =>
1590
1654
  devDb.latest(ref, db, handle, keyPtr, keyLen, limit),
1655
+ 'data.events_since': (handle: number, keyPtr: number, keyLen: number, limit: number): number =>
1656
+ devDb.since(ref, db, handle, keyPtr, keyLen, limit),
1591
1657
 
1592
1658
  'data.capacity_set_total': (
1593
1659
  handle: number,
@@ -8,6 +8,7 @@ export {
8
8
  __resetDbForTests,
9
9
  __setDbCatalogForTests,
10
10
  buildDatabaseImports,
11
+ commitDeriveCheckpoints,
11
12
  configureDbPersistence,
12
13
  DevDatabase,
13
14
  devDb,
@@ -55,6 +55,14 @@ export interface DbDevState {
55
55
  * Only populated for non-Derive dispatches (a derive's own writes must not
56
56
  * re-trigger it - see `database.ts` `recordWrite`). */
57
57
  writtenCollections: Set<string>;
58
+ /** The derive index currently running (set by `runDerive`), so `events.since` keys its resumable
59
+ * checkpoint per (derive, source key). 0 outside a derive. */
60
+ deriveId: number;
61
+ /** Checkpoint advances `events.since` staged during THIS derive run, keyed `"<deriveId>\0<storeKey>"` ->
62
+ * events-folded count. Committed to the durable `deriveCheckpoints` only after the fold's `derive_run`
63
+ * returns (its `view.publish`es have landed): a trapped fold discards this overlay, so the next run
64
+ * re-reads the batch rather than skipping it (publish-then-persist, mirroring the edge). */
65
+ pendingCheckpoints: Map<string, number>;
58
66
  }
59
67
 
60
68
  export function freshDbState(): DbDevState {
@@ -64,6 +72,8 @@ export function freshDbState(): DbDevState {
64
72
  lastResultVersion: -1,
65
73
  functionKind: DbFunctionKind.Job,
66
74
  writtenCollections: new Set<string>(),
75
+ deriveId: 0,
76
+ pendingCheckpoints: new Map<string, number>(),
67
77
  };
68
78
  }
69
79
 
@@ -95,6 +105,9 @@ export interface DbSnapshot {
95
105
  counterIdem?: Record<string, string>;
96
106
  events: Record<string, { v: string; sv: number }[]>;
97
107
  eventDedup: Record<string, string[]>;
108
+ /** Durable `events.since` cursors, `"<deriveId>\0<storeKey>"` -> events-folded count, so a restart
109
+ * resumes an incremental `@derive` instead of re-folding the whole log onto the persisted view. */
110
+ deriveCheckpoints?: Record<string, number>;
98
111
  capacity: Record<
99
112
  string,
100
113
  {
@@ -14,6 +14,7 @@
14
14
  import fs from 'node:fs';
15
15
 
16
16
  import {
17
+ commitDeriveCheckpoints,
17
18
  DbFunctionKind,
18
19
  type DeriveEntry,
19
20
  derivesForWrites,
@@ -154,6 +155,7 @@ const PROVIDED_IMPORTS = new Set([
154
155
  'data.append_once',
155
156
  'data.enqueue',
156
157
  'data.latest',
158
+ 'data.events_since',
157
159
  'data.capacity_set_total',
158
160
  'data.capacity_available',
159
161
  'data.capacity_reserve',
@@ -401,11 +403,16 @@ export class WasmServerModule {
401
403
  const ref: MemoryRef = { memory: null };
402
404
  const state = freshDispatchState();
403
405
  state.db.functionKind = DbFunctionKind.Derive;
406
+ state.db.deriveId = deriveId;
404
407
  const instance = new WebAssembly.Instance(this.module, buildHostImports(ref, state));
405
408
  const exports = instance.exports as unknown as DeriveExports;
406
409
  ref.memory = exports.memory;
407
410
  if (typeof exports.derive_run !== 'function') return;
408
411
  exports.derive_run(deriveId);
412
+ // Publish-then-persist: derive_run returned (its `view.publish`es landed synchronously in the dev
413
+ // store), so commit the `events.since` cursor advances staged this run. A trapped fold throws above
414
+ // and never reaches here, leaving the durable cursor for a re-read on the next trigger.
415
+ commitDeriveCheckpoints(state.db);
409
416
  }
410
417
 
411
418
  /** Fail instantiation up front, with names, when the guest needs imports we do not provide. */
@@ -32,12 +32,13 @@ describe('dev analytics stub v2 frame parity', () => {
32
32
  expect(u16()).toBe(2); // FRAME_VERSION
33
33
  u64(); // now_ms
34
34
  const count = u32();
35
- expect(count).toBe(41); // METRIC_COUNTERS
35
+ expect(count).toBe(43); // METRIC_COUNTERS
36
36
  const life: bigint[] = [];
37
37
  for (let i = 0; i < count; i++) life.push(i64());
38
38
  expect(life[0]).toBe(42n); // Requests
39
39
  expect(life[1]).toBe(12345n); // BytesOutL1
40
40
  expect(life[12]).toBe(1_000_000n); // GasUsed
41
+ expect(life[41]).toBe(900n); // CacheHits
41
42
  expect(i64()).toBe(3n); // connectedStreams
42
43
  expect(i64()).toBe(65536n); // committedMemory
43
44
  expect(i64()).toBe(5n); // reqMinuteUsed
@@ -10,6 +10,7 @@ import {
10
10
  __resetDbForTests,
11
11
  __setDbCatalogForTests,
12
12
  buildDatabaseImports,
13
+ commitDeriveCheckpoints,
13
14
  configureDbPersistence,
14
15
  derivesForWrites,
15
16
  freshDbState,
@@ -637,6 +638,118 @@ describe('toildb dev emulator (record family)', () => {
637
638
  expect(buf.readUInt32LE(300)).toBe(0);
638
639
  });
639
640
 
641
+ it('events.since: drains oldest-first, stages within a run, commits on success, resumes across runs', () => {
642
+ const { ref, imports, buf, db } = setup();
643
+ const h = resolve(imports, buf, 'App/feed');
644
+ const [kPtr, kLen] = put(buf, 32, 'room1');
645
+ for (let i = 0; i < 3; i++) {
646
+ const [eP, eL] = put(buf, 48, `e${String(i)}`);
647
+ expect(imports['data.append'](h, kPtr, kLen, eP, eL, 0)).toBe(0);
648
+ }
649
+
650
+ // Decode a framed since batch (u32 count, then per-event u32 ver + u32 len + bytes), oldest-first.
651
+ const drain = (
652
+ imps: Record<string, (...a: number[]) => number>,
653
+ total: number,
654
+ at: number,
655
+ ): string[] => {
656
+ expect(imps['data.take_result'](at, 256)).toBe(total);
657
+ let off = at;
658
+ const count = buf.readUInt32LE(off);
659
+ off += 4;
660
+ const out: string[] = [];
661
+ for (let i = 0; i < count; i++) {
662
+ off += 4; // schema_version (0 in dev)
663
+ const len = buf.readUInt32LE(off);
664
+ off += 4;
665
+ out.push(buf.toString('utf8', off, off + len));
666
+ off += len;
667
+ }
668
+ return out;
669
+ };
670
+
671
+ // A derive run drains the log in bounded batches, resuming from the STAGED cursor within the run.
672
+ db.functionKind = DbFunctionKind.Derive;
673
+ db.deriveId = 0;
674
+ expect(drain(imports, imports['data.events_since'](h, kPtr, kLen, 2), 512)).toEqual(['e0', 'e1']);
675
+ expect(drain(imports, imports['data.events_since'](h, kPtr, kLen, 2), 768)).toEqual(['e2']);
676
+ expect(drain(imports, imports['data.events_since'](h, kPtr, kLen, 2), 1024)).toEqual([]); // caught up
677
+
678
+ // The advance is only STAGED until the fold's derive_run returns; commit persists it.
679
+ expect(db.pendingCheckpoints.size).toBeGreaterThan(0);
680
+ commitDeriveCheckpoints(db);
681
+ expect(db.pendingCheckpoints.size).toBe(0);
682
+
683
+ // A fresh derive run (new per-request state, same shared store) resumes PAST the committed cursor:
684
+ // it re-reads nothing, then folds only a newly appended event.
685
+ const db2 = freshDbState();
686
+ db2.functionKind = DbFunctionKind.Derive;
687
+ db2.deriveId = 0;
688
+ const imports2 = buildDatabaseImports(ref, db2) as Record<string, (...a: number[]) => number>;
689
+ const h2 = resolve(imports2, buf, 'App/feed');
690
+ expect(drain(imports2, imports2['data.events_since'](h2, kPtr, kLen, 10), 512)).toEqual([]);
691
+ const [e3P, e3L] = put(buf, 48, 'e3');
692
+ expect(imports2['data.append'](h2, kPtr, kLen, e3P, e3L, 0)).toBe(0);
693
+ expect(drain(imports2, imports2['data.events_since'](h2, kPtr, kLen, 10), 512)).toEqual(['e3']);
694
+ });
695
+
696
+ it('events.since: a derive run that does not commit leaves the durable cursor unmoved (trap-safety)', () => {
697
+ const { ref, imports, buf, db } = setup();
698
+ const h = resolve(imports, buf, 'App/feed');
699
+ const [kPtr, kLen] = put(buf, 32, 'room1');
700
+ for (let i = 0; i < 2; i++) {
701
+ const [eP, eL] = put(buf, 48, `e${String(i)}`);
702
+ expect(imports['data.append'](h, kPtr, kLen, eP, eL, 0)).toBe(0);
703
+ }
704
+ const count = (imps: Record<string, (...a: number[]) => number>, total: number, at: number): number => {
705
+ expect(imps['data.take_result'](at, 256)).toBe(total);
706
+ return buf.readUInt32LE(at);
707
+ };
708
+
709
+ // Run 1 drains both events but NEVER commits (simulating a fold that trapped before returning): the
710
+ // staged advance lives only on this discarded per-request state.
711
+ db.functionKind = DbFunctionKind.Derive;
712
+ db.deriveId = 0;
713
+ expect(count(imports, imports['data.events_since'](h, kPtr, kLen, 10), 512)).toBe(2);
714
+
715
+ // Run 2 (fresh per-request state) still sees BOTH events: the durable cursor never moved.
716
+ const db2 = freshDbState();
717
+ db2.functionKind = DbFunctionKind.Derive;
718
+ db2.deriveId = 0;
719
+ const imports2 = buildDatabaseImports(ref, db2) as Record<string, (...a: number[]) => number>;
720
+ const h2 = resolve(imports2, buf, 'App/feed');
721
+ expect(count(imports2, imports2['data.events_since'](h2, kPtr, kLen, 10), 512)).toBe(2);
722
+ });
723
+
724
+ it('__resetDbForTests clears events.since checkpoints (no cross-test cursor leak)', () => {
725
+ {
726
+ const { imports, buf, db } = setup();
727
+ const h = resolve(imports, buf, 'App/feed');
728
+ const [kPtr, kLen] = put(buf, 32, 'room1');
729
+ const [eP, eL] = put(buf, 48, 'e0');
730
+ expect(imports['data.append'](h, kPtr, kLen, eP, eL, 0)).toBe(0);
731
+ db.functionKind = DbFunctionKind.Derive;
732
+ db.deriveId = 0;
733
+ imports['data.events_since'](h, kPtr, kLen, 10);
734
+ commitDeriveCheckpoints(db); // durable cursor now at 1
735
+ }
736
+ __resetDbForTests();
737
+ {
738
+ // A brand-new store reusing the same (deriveId, key): the cursor must be back at 0, so a fresh
739
+ // append is read, not skipped by a stale checkpoint.
740
+ const { imports, buf, db } = setup();
741
+ const h = resolve(imports, buf, 'App/feed');
742
+ const [kPtr, kLen] = put(buf, 32, 'room1');
743
+ const [eP, eL] = put(buf, 48, 'fresh0');
744
+ expect(imports['data.append'](h, kPtr, kLen, eP, eL, 0)).toBe(0);
745
+ db.functionKind = DbFunctionKind.Derive;
746
+ db.deriveId = 0;
747
+ const total = imports['data.events_since'](h, kPtr, kLen, 10);
748
+ expect(imports['data.take_result'](512, 256)).toBe(total);
749
+ expect(buf.readUInt32LE(512)).toBe(1); // the fresh event, not skipped
750
+ }
751
+ });
752
+
640
753
  it('collections are isolated (no key aliasing)', () => {
641
754
  const { imports, buf } = setup();
642
755
  const users = resolve(imports, buf, 'App/users');
package/docs/auth-todo.md DELETED
@@ -1,149 +0,0 @@
1
- # Auth: production TODO and toildb migration
2
-
3
- Tracking doc for finishing the Toil PQ-Auth system (shipped experimental in
4
- **v0.0.52**). The cryptographic core is in place; what is left is the deployment
5
- substrate, protocol hardening, and account lifecycle.
6
-
7
- > This system is a **hybrid, experimental** password-authenticated login and is **NOT a framework
8
- > default until externally reviewed** (see Tier 2.8). It is opt-in: a tenant must wire
9
- > it. The OPRF layer is classical ristretto255 (the one non-PQ piece); auth (ML-DSA-44)
10
- > and key agreement (ML-KEM-768) are post-quantum.
11
-
12
- ## What ships today (v0.0.52)
13
-
14
- - OPRF keyed salt (RFC 9497 ristretto255-SHA512, mode 0x00, per-user key) -> precomputation resistance.
15
- - ML-DSA-44 client auth + registration proof-of-possession.
16
- - ML-KEM-768 mutual auth: client encapsulates to a pinned server key, server returns a confirmation tag.
17
- - Login enumeration fixed (deterministic per-user salt).
18
- - Atomic single-use challenge consume (shape is correct; storage is the dev KV, see below).
19
-
20
- Key files: `server/globals/auth.ts` (the `AuthService` global), `src/client/auth.ts`
21
- (browser), `src/devserver/{crypto,kv}.ts` (dev host mocks + dev KV), the Rust edge host
22
- imports in `toil-backend` (`crypto.mlkem_decapsulate`, `crypto.voprf_evaluate`),
23
- `examples/basic/server/routes/Auth.ts` (the flow).
24
-
25
- ---
26
-
27
- ## Tier 1, blocks production (cannot run on the edge yet)
28
-
29
- ### 1.1 Storage on toildb (THE blocker) -- "when building toildb, consider this"
30
- The accounts + login challenges live in the **DEV-ONLY** `kv.*` Map
31
- (`src/devserver/kv.ts`), which is intentionally **not** on the production edge
32
- (`toil-backend` `HOST_IMPORTS`), so a module importing `kv.*` will not instantiate in
33
- prod. When toildb lands:
34
- - **toildb MUST provide an atomic fetch-and-delete** (single-call read+remove). The
35
- login challenge consume relies on it; a read-then-delete race makes a login replayable.
36
- It also needs `get` and `put`.
37
- - Migrate **two stores**: `Accounts` (username -> {salt, params, publicKey}) and
38
- `Challenges` (cid -> {username, nonce, iat, exp}) in `examples/basic/server/routes/Auth.ts`.
39
- - Add the toildb host imports to `toil-backend` `HOST_IMPORTS` + `define_*_imports`.
40
- - Then **delete `src/devserver/kv.ts`**, the `kv.*` entries in `module.ts`
41
- `PROVIDED_IMPORTS`, and the `buildKvImports` wiring in `host.ts`. Grep `REMOVE KV LATER`.
42
-
43
- ### 1.2 Real secrets (not dev placeholders)
44
- All three must come from the managed env / `getSecure` store, per-deployment, identical
45
- across edge instances, and rotatable:
46
- - session HMAC secret (still the `...CHANGE-ME` default in `auth.ts`),
47
- - OPRF master seed (currently `sha256Text('toil-demo-oprf-seed-v1')`),
48
- - **server ML-KEM keypair** -- generate a real one per deployment and pin **its** public
49
- key in the client (today a fixed demo key is pinned in `src/client/auth.ts`, secret in
50
- the example `Auth.ts`).
51
-
52
- ### 1.3 Rate limiting on the auth routes
53
- `@ratelimit` exists (`docs/ratelimit.md`) but is not wired onto register/login. Add
54
- per-username + per-IP throttling with backoff/lockout. Offline resistance is good; online
55
- guessing + registration spam are currently open.
56
-
57
- ### 1.4 Production Argon2id params
58
- Demo uses 32 MiB / 2 iters for tab responsiveness. Raise to >= 256 MiB / >= 3 iters and
59
- enforce a server-side floor.
60
-
61
- ### 1.5 Gas calibration (edge metering)
62
- `MLKEM768_DECAPSULATE` and `VOPRF_EVALUATE` (`toil-backend src/config.rs`) are
63
- **conservative 12,000,000 placeholders copied from `MLDSA44_VERIFY`, not benchmarked**.
64
- They are charged correctly (before reads) and DoS-bounded by a test, and over-charge
65
- (fail-closed safe), but should be calibrated: measure cyc/op via `crypto::bench`,
66
- multiply by `GAS_PER_CYCLE`, round up ~30%, replace. Same TODO as `MLDSA44_VERIFY` itself.
67
-
68
- ### 1.6 Real-edge end-to-end
69
- Never run: a full guest-dispatch through the **running** Rust edge exercising the two new
70
- imports (only proven at unit + interop-vector level + structurally identical to the
71
- in-prod `mldsa_verify` import). Do before trusting in prod.
72
-
73
- ---
74
-
75
- ## Tier 2, protocol hardening
76
-
77
- ### 2.6 A properly bound session key, DONE (on main, unreleased)
78
- The session key is now `K = HMAC-SHA256(sharedSecret, SESSION_KEY_LABEL || H(M))` and the
79
- mutual-auth tag is `HMAC-SHA256(K, SERVER_CONFIRM_LABEL || H(M))` (`AuthService.deriveSessionKey`
80
- + `serverConfirmTag`; client mirrors it with hash-wasm `createHMAC`). REMAINING: binding the
81
- *session cookie* to the transport (so a stolen cookie is useless on another channel) needs
82
- the TLS exporter, which the wasm guest cannot see, an edge/transport follow-up, not doable
83
- purely in the guest.
84
-
85
- ### 2.7 Bind the KDF params + server key into the transcript, DONE (on main, unreleased)
86
- The single `buildLoginMessage` now binds the ML-KEM ciphertext, the Argon2id params
87
- (mem/iters/par), and `serverKemKeyId = SHA-256(serverKemPublicKey)`. Closes
88
- key-substitution and param-downgrade confusion. (There is ONE login message format, no
89
- versioned variants.)
90
-
91
- ### 2.8 External cryptographic review (the gate)
92
- Hand-rolled KEM + signature + OPRF composition with no security proof. Transcript binding
93
- especially needs a cryptographer. This is a human task; it cannot be self-served. Make it
94
- tractable: write the wire spec (messages, what each signature/MAC covers, the KDF tree) as
95
- a reviewable artifact.
96
-
97
- ---
98
-
99
- ## Tier 3, account lifecycle (missing today)
100
-
101
- ### 3.1 Password change / key rotation
102
- Re-derive under a fresh salt while authenticated, re-register the new public key. None
103
- exists today.
104
-
105
- ### 3.2 Recovery
106
- "The password IS the key, no recovery" -> a forgotten password is a permanently dead
107
- account. Needs recovery codes or a second factor (ties to the 2FA crate already on the
108
- backend "left to do" list + the existing email support).
109
-
110
- ### 3.3 Revocation
111
- No way to kill a compromised credential except deleting the account.
112
-
113
- ### 3.4 Tighten registration enumeration
114
- `register/finish` still leaks "taken" via the generic fail after the PoP round.
115
-
116
- ---
117
-
118
- ## Design decisions (open)
119
-
120
- ### D1. User id = sha256(public key)? -- CAUTION: breaks under key rotation
121
- Yes, the public key is still the verifier (`AuthService.verifyLogin` checks the ML-DSA
122
- signature against the stored public key). `sha256(publicKey)` is a fine **content-addressed
123
- credential fingerprint** (collision-resistant, re-derivable, good for dedup). BUT:
124
- - It **cannot be the login lookup key**: at `login/start` the client presents an identifier
125
- to fetch the salt + OPRF evaluation *before* it has derived its keypair (it needs the
126
- OPRF response to derive the key -> to get the pubkey). So a human-meaningful **handle**
127
- (username/email) is still required as the OPRF `info` + the `login/start` lookup key.
128
- - It is **not stable across password change**: rotating the password changes the pubkey,
129
- hence `sha256(pubkey)`. If the user id must survive a password change, it must be a
130
- separate stable id (random UUID assigned at registration), with `sha256(pubkey)` treated
131
- as the *current credential* fingerprint, not the permanent identity.
132
- - **Recommendation:** handle (username/email) = login + OPRF info; permanent userId =
133
- random id at registration; `sha256(pubkey)` = credential fingerprint (verifiable, for
134
- dedup / "is this the same key"). Revisit only if password rotation is ruled out.
135
-
136
- ### D2. OPRF mode vs VOPRF/DLEQ
137
- Plain OPRF mode by design (server auth comes from the ML-KEM layer). Add
138
- DLEQ only if the client must verify OPRF-key consistency. Likely unnecessary.
139
-
140
- ### D3. Multi-device / WebAuthn-style per-device keys
141
- One-key-per-user (deterministic from password) is fine unless hardware-backed or per-device
142
- keys are wanted later; that needs a credential-set model.
143
-
144
- ---
145
-
146
- ## Suggested sequence
147
- toildb storage (1.1) -> real secrets + real pinned KEM key (1.2) -> rate limiting (1.3) ->
148
- HKDF session key (2.6) + transcript binding (2.7) -> external review (2.8) -> lifecycle
149
- (3.1-3.3). 1.1-1.3 turn the demo into something deployable; 2.8 makes it trustworthy.