@hearthlog/ledger 0.10.2 → 0.11.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -192,7 +192,8 @@ Authenticated endpoints under `/logs/:logId`:
192
192
  - `GET /maintenance-preview`: the current snapshot/generation and prefix payload counts/bytes for review.
193
193
  - `POST /maintenance`: `{ requestId, kind: "snapshot" | "compact", expectedCheckpointId?, expectedGeneration? }`.
194
194
  Compaction requires the checkpoint ID and generation from the reviewed preview; stale previews fail with 409.
195
- - `POST /maintenance/step`: `{ requestId }`, resumable work using the same request identity.
195
+ - `POST /maintenance/step`: `{ requestId }`; runs one bounded step now (sharing any step already in flight) and restarts a
196
+ job that stopped after repeated failures. Normally unnecessary: the Ledger advances jobs by itself (see below).
196
197
  - `GET /checkpoint?page=N`: one checksum-verified canonical replay chunk and its entity claims.
197
198
 
198
199
  From the development site, the same operation can run without an open browser:
@@ -206,6 +207,17 @@ pnpm -C dev ledger:maintain --log <log-id> --resume
206
207
  Set `LEDGER_BASE_URL` and `LEDGER_AUTH_TOKEN` in the process environment; the CLI never prints credentials.
207
208
  Use `--request-id` to retry an explicit operation and `--once` for one bounded work step.
208
209
 
210
+ A started job is carried to completion by the application object's alarm, with no caller or open browser. Each step
211
+ works for a few seconds, folding source events into small per-entity replay slots (source versions only). The whole job's
212
+ slots live in one private R2 object that is read and written once per step, so a snapshot costs a handful of SQL writes
213
+ however many records the log holds. The retained events are then written as checksummed pages. Archiving rewrites each
214
+ archived event row once (its payload moves to R2); the archived prefix is tracked per log rather than indexed per event. A later snapshot starts from the previous one's retained facts instead of re-reading
215
+ archived history, unless a restore/removal marker has been added since. A failing step is retried with backoff; the
216
+ error and attempt count stay visible, and after five attempts the job stops as `failed` until it is explicitly retried
217
+ (the maintenance panel's Retry, `POST /maintenance/step`, or the CLI). A job that uses its daily budget is `paused`
218
+ with `pauseReason` and `resumeAt`, and resumes by itself at the next UTC day. Opening the panel re-arms a job whose
219
+ alarm was lost.
220
+
209
221
  Jobs persist every source batch and publish only verified snapshots. Canonical snapshots keep original source
210
222
  witnesses for raw field values, explicit nulls, timestamps and lifecycle transitions; dynamic/nonstandard facts
211
223
  are retained losslessly. Browser projection uses a separate baseline table plus accepted suffix and pending
@@ -217,9 +229,8 @@ payload placeholders. The SDK's server checkpoint reader lets Stash seed a cold
217
229
  `MAINTENANCE_DAILY_WRITES` defaults to `10000` per job (bounded to 128–100000), and
218
230
  `MAINTENANCE_DAILY_READS` defaults to `1000000` (bounded to 10000–1000000000). Jobs pause at either budget. Budgets use consumed SQL cursor
219
231
  counts and can overrun by one bounded step. Structured step logs include SQL rows, R2 operations/bytes, elapsed
220
- time and replay counts, without payloads or credentials. Resume with the same ID on a later UTC quota day;
221
- other applications consume account quota too. The browser
222
- can be closed safely and the operation resumed from the panel. No automatic snapshots or archive expiry run.
232
+ time and replay counts, without payloads or credentials. Other applications consume account quota too. The browser
233
+ can be closed safely; the job continues on the server. No automatic snapshots or archive expiry run.
223
234
  Deleting a whole log uses bounded background cleanup of that log's private archive objects, after normal
224
235
  fork/deletion checks. Individual archived-history deletion is a separate future feature.
225
236
 
@@ -228,7 +239,7 @@ payloads into the existing portable format. A SQLite-only export returns `409 ar
228
239
  external checkpoint/archive objects are required, avoiding a misleading incomplete backup. Rollback after
229
240
  compaction needs a compatible reader or a full logical restore first.
230
241
 
231
- The first format bounds a single entity's replay witnesses to 8 MB and stores page descriptors in SQLite.
242
+ The first format bounds a single entity's replay state to about 1.5 MB and a job's work state to 64 MB and stores page descriptors in SQLite.
232
243
  Exceeding a snapshot work/metadata bound fails the job while source history stays readable. Measure maintenance
233
244
  work and storage on representative data; retained identity metadata still grows, and physical SQLite file size
234
245
  need not shrink immediately when payload space becomes reusable.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@hearthlog/ledger",
3
- "version": "0.10.2",
3
+ "version": "0.11.1",
4
4
  "description": "Hearthlog's Ledger: the event store Worker (Elysia over SQLite Durable Objects), its contracts and tests, and the logical backup tooling",
5
5
  "license": "AGPL-3.0-only",
6
6
  "repository": {
@@ -30,7 +30,7 @@
30
30
  "dependencies": {
31
31
  "@sinclair/typebox": "0.34.52",
32
32
  "elysia": "1.4.30",
33
- "@hearthlog/contracts": "0.10.2"
33
+ "@hearthlog/contracts": "0.11.1"
34
34
  },
35
35
  "peerDependencies": {
36
36
  "@hearthlog/generated": "*"
@@ -45,6 +45,7 @@ async function main(): Promise<void> {
45
45
  await new Promise(resolve => setTimeout(resolve, 500));
46
46
  }
47
47
  if (state.job?.status === 'failed') throw new Error(state.job.error ?? 'Maintenance failed');
48
+ if (state.job?.status === 'paused') console.error(`Paused (${state.job.pauseReason ?? 'budget'}); the Ledger resumes at ${state.job.resumeAt ?? 'the next UTC day'}.`);
48
49
  } while (!values.once && state.job?.status !== 'complete' && state.job?.status !== 'paused');
49
50
  console.log(JSON.stringify({ logId: values.log, requestId, status: state.job?.status,
50
51
  checkpointId: state.checkpoint?.id ?? null, throughVersion: state.checkpoint?.throughVersion ?? 0,
package/src/ledger-app.ts CHANGED
@@ -148,14 +148,18 @@ export class LedgerApp extends DurableObject<Env> {
148
148
  private maintenance(assertWritable: () => void = () => undefined): LogMaintenanceStore {
149
149
  const appId = this.ctx.storage.sql.exec<{ app_id: string }>('SELECT app_id FROM app_metadata WHERE singleton = 1').one().app_id;
150
150
  const store = new LogMaintenanceStore(this.ctx.storage, this.env.LEDGER_ARCHIVE, appId,
151
- async (logId, after, through, limit, entityId) => {
151
+ async (logId, after, through, limit) => {
152
152
  const rows = store.observeSql(this.ctx.storage.sql.exec<EventRow>(`WITH candidates AS (
153
153
  SELECT *, SUM(COALESCE(payload_bytes, length(CAST(payload AS BLOB)))) OVER (ORDER BY version) AS byte_count,
154
154
  ROW_NUMBER() OVER (ORDER BY version) AS row_number
155
- FROM (SELECT * FROM log_events WHERE log_id = ? AND version > ? AND version <= ?
156
- ${entityId === undefined ? '' : 'AND entity_id = ?'} ORDER BY version LIMIT ?)
155
+ FROM (SELECT * FROM log_events WHERE log_id = ? AND version > ? AND version <= ? ORDER BY version LIMIT ?)
157
156
  ) SELECT * FROM candidates WHERE byte_count <= 1500000 OR row_number = 1`,
158
- logId, after, through, ...(entityId === undefined ? [] : [entityId]), limit)).toArray();
157
+ logId, after, through, limit)).toArray();
158
+ return store.hydrate(logId, rows.map(storedEvent));
159
+ }, async (logId, versions) => {
160
+ const rows = store.observeSql(this.ctx.storage.sql.exec<EventRow>(
161
+ 'SELECT * FROM log_events WHERE log_id = ? AND version IN (SELECT value FROM json_each(?)) ORDER BY version',
162
+ logId, JSON.stringify(versions))).toArray();
159
163
  return store.hydrate(logId, rows.map(storedEvent));
160
164
  }, assertWritable, Math.max(128, Math.min(100000, Number(this.env.MAINTENANCE_DAILY_WRITES) || 10000)),
161
165
  Math.max(10000, Math.min(1_000_000_000, Number(this.env.MAINTENANCE_DAILY_READS) || 1_000_000)));
@@ -163,10 +167,13 @@ export class LedgerApp extends DurableObject<Env> {
163
167
  }
164
168
 
165
169
  async logMaintenance(access: AccessClaims, logId: string): Promise<RpcResult<LogMaintenance>> {
166
- return rpcCall(() => {
170
+ return rpcCall(async () => {
167
171
  this.currentAccess(access);
168
172
  if (!this.log(parseUuid(logId, 'log id'))) throw validationError('Log not found');
169
- return rpcSuccess(this.maintenance().state(logId));
173
+ const store = this.maintenance();
174
+ // A job begun before alarms existed (or whose alarm was lost) is picked up again here.
175
+ if (store.hasActiveJob() && await this.ctx.storage.getAlarm() === null) await this.scheduleAlarm();
176
+ return rpcSuccess(store.state(logId));
170
177
  });
171
178
  }
172
179
 
@@ -179,9 +186,12 @@ export class LedgerApp extends DurableObject<Env> {
179
186
  }
180
187
 
181
188
  async startLogMaintenance(access: AccessClaims, logId: string, id: string, kind: 'snapshot' | 'compact', expectedCheckpointId?: string, expectedGeneration?: number): Promise<RpcResult<LogMaintenance>> {
182
- return rpcCall(() => {
189
+ return rpcCall(async () => {
183
190
  this.writableAccess(access);
184
- return rpcSuccess(this.maintenance().start(parseUuid(logId, 'log id'), parseUuid(id, 'request id'), kind, expectedCheckpointId, expectedGeneration));
191
+ const state = this.maintenance().start(parseUuid(logId, 'log id'), parseUuid(id, 'request id'), kind, expectedCheckpointId, expectedGeneration);
192
+ // The server carries the job to completion; the caller only needs to watch.
193
+ await this.scheduleAlarm();
194
+ return rpcSuccess(state);
185
195
  });
186
196
  }
187
197
 
@@ -194,6 +204,8 @@ export class LedgerApp extends DurableObject<Env> {
194
204
  () => this.maintenance(() => { this.writableAccess(access); }).step(parsedLog, parsedId),
195
205
  () => this.maintenance().state(parsedLog, parsedId));
196
206
  this.currentAccess(access);
207
+ // An explicit step also restarts a job that stopped after repeated failures.
208
+ await this.scheduleAlarm();
197
209
  return rpcSuccess(result);
198
210
  });
199
211
  }
@@ -213,21 +225,59 @@ export class LedgerApp extends DurableObject<Env> {
213
225
  });
214
226
  }
215
227
 
216
- /** Delete storage belonging to an explicitly deleted log, in bounded batches. */
228
+ /** Advance log maintenance and delete storage of deleted logs, each in bounded batches. */
217
229
  override async alarm(): Promise<void> {
230
+ let failed = false;
231
+ try { await this.deleteArchivedLogs(); }
232
+ catch (cause) { failed = true; console.error('ledger_archive_deletion_failed', normalizeError(cause).message); }
233
+ try { await this.advanceMaintenance(); }
234
+ catch (cause) { failed = true; console.error('ledger_maintenance_alarm_failed', normalizeError(cause).message); }
235
+ await this.scheduleAlarm({ deletionDelayMs: 60_000, failureDelayMs: failed ? 30_000 : undefined });
236
+ }
237
+
238
+ private async deleteArchivedLogs(): Promise<void> {
218
239
  const job = this.ctx.storage.sql.exec<{ log_id: string }>('SELECT log_id FROM log_archive_deletions LIMIT 1').toArray()[0];
219
240
  if (!job) return;
220
- try {
221
- const app = this.ctx.storage.sql.exec<{ app_id: string }>('SELECT app_id FROM app_metadata WHERE singleton = 1').one();
222
- const bucket = this.env.LEDGER_ARCHIVE;
223
- const objects = await bucket.list({ prefix: `v1/${app.app_id}/${job.log_id}/`, limit: 128 });
224
- if (objects.objects.length) await bucket.delete(objects.objects.map(object => object.key));
225
- if (!objects.truncated) this.ctx.storage.sql.exec('DELETE FROM log_archive_deletions WHERE log_id = ?', job.log_id);
226
- } finally {
227
- if (this.ctx.storage.sql.exec('SELECT 1 FROM log_archive_deletions LIMIT 1').toArray().length) {
228
- await this.ctx.storage.setAlarm(Date.now() + 60_000);
229
- }
241
+ const app = this.ctx.storage.sql.exec<{ app_id: string }>('SELECT app_id FROM app_metadata WHERE singleton = 1').one();
242
+ const bucket = this.env.LEDGER_ARCHIVE;
243
+ const objects = await bucket.list({ prefix: `v1/${app.app_id}/${job.log_id}/`, limit: 128 });
244
+ if (objects.objects.length) await bucket.delete(objects.objects.map(object => object.key));
245
+ if (!objects.truncated) this.ctx.storage.sql.exec('DELETE FROM log_archive_deletions WHERE log_id = ?', job.log_id);
246
+ }
247
+
248
+ /** Run due maintenance steps for a few seconds, so a job finishes without any caller. */
249
+ private async advanceMaintenance(): Promise<void> {
250
+ if (this.metadataOptional() === undefined) return;
251
+ const started = Date.now();
252
+ const assertWritable = () => this.assertApplicationWritable();
253
+ do {
254
+ const due = this.maintenance(assertWritable).dueJobs()[0];
255
+ if (!due) return;
256
+ await this.maintenanceSteps.run(`${due.logId}:${due.id}`, () => this.maintenance(assertWritable).step(due.logId, due.id));
257
+ } while (Date.now() - started < 5000);
258
+ }
259
+
260
+ /** Wake at the earliest time any pending work (maintenance step, retry, resume, deletion) is due. */
261
+ private async scheduleAlarm(options: { deletionDelayMs?: number; failureDelayMs?: number } = {}): Promise<void> {
262
+ const now = Date.now();
263
+ const wakes: number[] = [];
264
+ if (this.ctx.storage.sql.exec('SELECT 1 FROM log_archive_deletions LIMIT 1').toArray().length) {
265
+ wakes.push(now + (options.deletionDelayMs ?? 1000));
230
266
  }
267
+ if (this.metadataOptional() !== undefined) {
268
+ const wake = this.maintenance().nextWake();
269
+ if (wake !== null) wakes.push(Math.max(wake, now + (options.failureDelayMs ?? 50)));
270
+ }
271
+ if (wakes.length) await this.ctx.storage.setAlarm(Math.min(...wakes));
272
+ }
273
+
274
+ /** Background work has no caller to authorize it, but must still stop for revoked or fenced applications. */
275
+ private assertApplicationWritable(): void {
276
+ if (this.metadata().revoked_at !== null) {
277
+ throw new LedgerError(401, { error: "access token has been rotated or revoked", code: "unauthorized" });
278
+ }
279
+ const fence = this.fence();
280
+ if (fence) throw fenceError(fence);
231
281
  }
232
282
 
233
283
  override async fetch(request: Request): Promise<Response> {
@@ -435,7 +485,7 @@ export class LedgerApp extends DurableObject<Env> {
435
485
 
436
486
  async administratorExportSqlite(appId: string): Promise<Response> {
437
487
  this.administratorApplication(appId);
438
- if (this.ctx.storage.sql.exec('SELECT 1 FROM log_events WHERE archive_key IS NOT NULL LIMIT 1').toArray().length
488
+ if (this.ctx.storage.sql.exec('SELECT 1 FROM log_storage WHERE archived_through > 0 LIMIT 1').toArray().length
439
489
  || this.ctx.storage.sql.exec('SELECT 1 FROM log_storage WHERE checkpoint IS NOT NULL LIMIT 1').toArray().length) {
440
490
  return jsonError('archive_bundle_required', 'Use the complete logical export or application transfer; archived payloads are outside SQLite.', 409);
441
491
  }
@@ -710,7 +760,7 @@ export class LedgerApp extends DurableObject<Env> {
710
760
  return rpcCall(async () => {
711
761
  const claims = this.writableAccess(access);
712
762
  this.deleteLogValue(parseUuid(logId, "logId"), claims);
713
- await this.ctx.storage.setAlarm(Date.now() + 1000);
763
+ await this.scheduleAlarm();
714
764
  return rpcSuccess(null, 204);
715
765
  });
716
766
  }
@@ -860,7 +910,7 @@ export class LedgerApp extends DurableObject<Env> {
860
910
  throw transferError("not_importing", "Only an import in progress can be aborted");
861
911
  }
862
912
  this.ctx.storage.transactionSync(() => {
863
- for (const table of ["log_storage", "log_archive_pages", "log_maintenance_jobs", "gallery_photo_references", "log_fork_requests", "append_requests", "entity_claims", "log_events", "logs", "app_metadata"]) {
913
+ for (const table of ["log_checkpoint_entities", "log_storage", "log_archive_pages", "log_maintenance_jobs", "gallery_photo_references", "log_fork_requests", "append_requests", "entity_claims", "log_events", "logs", "app_metadata"]) {
864
914
  this.ctx.storage.sql.exec(`DELETE FROM ${table}`);
865
915
  }
866
916
  });
@@ -1114,6 +1164,12 @@ export class LedgerApp extends DurableObject<Env> {
1114
1164
  FROM log_events WHERE log_id = ? AND version <= ? ORDER BY version`,
1115
1165
  id, sourceLogId, targetVersion,
1116
1166
  );
1167
+ // A fork shares its source's archived prefix up to the fork point.
1168
+ this.ctx.storage.sql.exec(
1169
+ `INSERT INTO log_storage (log_id, compacted_through, generation, checkpoint, archived_through)
1170
+ SELECT ?, 0, 0, NULL, MIN(archived_through, ?) FROM log_storage WHERE log_id = ? AND archived_through > 0`,
1171
+ id, targetVersion, sourceLogId,
1172
+ );
1117
1173
  this.ctx.storage.sql.exec(
1118
1174
  `INSERT INTO entity_claims (log_id, entity_id, current_version, version_created, created_at, updated_at)
1119
1175
  SELECT ?, source.entity_id, source.entity_version, first_version.first_version, ?, ?
@@ -1606,7 +1662,7 @@ export class LedgerApp extends DurableObject<Env> {
1606
1662
  return;
1607
1663
  }
1608
1664
  const head = this.logHead(attachment.logId);
1609
- const archived = this.ctx.storage.sql.exec<{ version: number | null }>('SELECT MAX(version) AS version FROM log_events WHERE log_id = ? AND archive_key IS NOT NULL', attachment.logId).one().version ?? 0;
1665
+ const archived = this.ctx.storage.sql.exec<{ archived_through: number }>('SELECT archived_through FROM log_storage WHERE log_id = ?', attachment.logId).toArray()[0]?.archived_through ?? 0;
1610
1666
  if (attachment.cursor < archived) {
1611
1667
  this.sendFrame(ws, { type: 'resync_required', headVersion: head, reason: 'gap_too_large' });
1612
1668
  ws.close(1008, 'baseline resync required');
@@ -1,21 +1,47 @@
1
1
  import { EVENT_SCHEMAS } from '#generated/event-schemas';
2
- import { checkpointDigest, checkpointEntityEvents, checkpointSelection, type CheckpointChunk, type CheckpointEvent, type CheckpointManifest,
3
- type CheckpointPage, type LogMaintenance, type MaintenanceJob, type CompactionPreview } from '@hearthlog/contracts/history/checkpoint';
2
+ import { checkpointDigest, checkpointSelection, checkpointStateVersions, foldCheckpointEvent, newCheckpointEntityState,
3
+ type CheckpointChunk, type CheckpointEntityState, type CheckpointEvent, type CheckpointManifest, type CheckpointPage,
4
+ type CheckpointSelector, type LogMaintenance, type MaintenanceJob, type CompactionPreview } from '@hearthlog/contracts/history/checkpoint';
4
5
  import { LedgerError, validationError } from './errors';
5
6
 
7
+ /** Events archived per compaction step. */
6
8
  const PAGE = 128;
9
+ /** Source events per read while folding (also bounded by the byte window of the reader). */
10
+ const FOLD_EVENTS = 512;
11
+ /** Retained events read per query while writing snapshot pages. */
12
+ const READ_VERSIONS = 256;
13
+ const CLAIMS_PER_PAGE = 4096;
14
+ /** Wall time one snapshot step keeps working before it saves its state. */
15
+ const STEP_MS = 4000;
16
+ const MAX_WORK_BYTES = 64_000_000;
7
17
  const MAX_PAGE_BYTES = 1_500_000;
18
+ const MAX_STATE_BYTES = 1_500_000;
19
+ const MAX_MARKER_BYTES = 1_000_000;
20
+ /** Automatic attempts before a job stops and waits for an explicit retry. */
21
+ export const MAX_ATTEMPTS = 5;
8
22
  const DEFAULT_DAY_WRITE_BUDGET = 10_000;
9
23
  const standardTypes = new Set(Object.keys(EVENT_SCHEMAS));
10
24
  const directives = ['EVENT_TOMBSTONED', 'WORKSPACE_RESTORED', 'EVENTS_EXCISED'];
25
+ const SELECTORS: readonly CheckpointSelector[] = [
26
+ event => typeof event.payload.status === 'string' ? `status:${event.payload.status}` : null,
27
+ event => typeof event.payload.thumbnailRef === 'string' ? 'thumbnail:string' : null,
28
+ ];
11
29
  type Row = Record<string, SqlStorageValue>;
30
+ type Phase = 'seed' | 'fold' | 'pages';
12
31
  interface JobRow extends Row {
13
32
  id: string; log_id: string; kind: MaintenanceJob['kind']; through_version: number;
14
33
  status: MaintenanceJob['status']; cursor: number; entity_cursor: string; pages: string;
15
34
  error: string | null; updated_at: string; day: string; writes: number; reads: number;
35
+ phase: Phase | null; version_cursor: number; attempts: number; retry_at: string | null;
36
+ resume_at: string | null; pause_reason: 'writes' | 'reads' | null; paused_status: MaintenanceJob['status'] | null;
37
+ markers: string | null;
16
38
  }
17
39
  interface LayoutRow extends Row { log_id: string; compacted_through: number; generation: number; checkpoint: string | null; }
18
- type ReadEvents = (logId: string, after: number, through: number, limit: number, entityId?: string) => Promise<CheckpointEvent[]>;
40
+ /** Replay slots for one entity plus the claim it held at the last source event. */
41
+ interface WorkState { s: CheckpointEntityState; c: { currentVersion: number; logVersion: number }; }
42
+ type Marker = Pick<CheckpointEvent, 'id' | 'type' | 'version' | 'payload'>;
43
+ type ReadEvents = (logId: string, after: number, through: number, limit: number) => Promise<CheckpointEvent[]>;
44
+ type ReadVersions = (logId: string, versions: number[]) => Promise<CheckpointEvent[]>;
19
45
 
20
46
  /** Durable job state lives in SQLite; immutable payloads live in the private bucket. */
21
47
  export class LogMaintenanceStore {
@@ -28,8 +54,10 @@ export class LogMaintenanceStore {
28
54
  return cursor;
29
55
  }
30
56
 
31
- private exec<T extends Record<string, SqlStorageValue> = Record<string, SqlStorageValue>>(query: string, ...bindings: SqlStorageValue[]): SqlStorageCursor<T> {
32
- return this.observeSql(this.storage.sql.exec<T>(query, ...bindings));
57
+ /** Run a statement to completion so its row counts are final, and meter it. */
58
+ private query<T extends Record<string, SqlStorageValue> = Record<string, SqlStorageValue>>(statement: string, ...bindings: SqlStorageValue[]): T[] {
59
+ const cursor = this.observeSql(this.storage.sql.exec<T>(statement, ...bindings));
60
+ return cursor.toArray();
33
61
  }
34
62
 
35
63
  private async getObject(key: string): Promise<R2ObjectBody | null> {
@@ -45,12 +73,16 @@ export class LogMaintenanceStore {
45
73
  }
46
74
 
47
75
  constructor(private readonly storage: DurableObjectStorage, private readonly bucket: R2Bucket | undefined,
48
- private readonly appId: string, private readonly readEvents: ReadEvents, private readonly assertWritable: () => void = () => undefined,
76
+ private readonly appId: string, private readonly readEvents: ReadEvents, private readonly readVersions: ReadVersions,
77
+ private readonly assertWritable: () => void = () => undefined,
49
78
  private readonly dayWriteBudget = DEFAULT_DAY_WRITE_BUDGET, private readonly dayReadBudget = 1_000_000) {}
50
79
 
51
80
  state(logId: string, jobId?: string): LogMaintenance {
52
81
  const layout = this.layout(logId);
53
- const job = this.exec<JobRow>(`SELECT * FROM log_maintenance_jobs WHERE log_id = ? ${jobId ? 'AND id = ?' : ''} ORDER BY rowid DESC LIMIT 1`, logId, ...(jobId ? [jobId] : [])).toArray()[0];
82
+ // Polled often: leave out the large work columns (pages, markers).
83
+ const job = this.query<JobRow>(`SELECT id, log_id, kind, through_version, status, cursor, error, updated_at, attempts, retry_at,
84
+ resume_at, pause_reason FROM log_maintenance_jobs WHERE log_id = ? ${jobId ? 'AND id = ?' : ''} ORDER BY rowid DESC LIMIT 1`,
85
+ logId, ...(jobId ? [jobId] : []))[0];
54
86
  return { compactedThroughVersion: layout.compacted_through, storageGeneration: layout.generation,
55
87
  checkpoint: layout.checkpoint ? JSON.parse(layout.checkpoint) : null,
56
88
  job: job ? publicJob(job) : null, available: Boolean(this.bucket) };
@@ -58,9 +90,9 @@ export class LogMaintenanceStore {
58
90
 
59
91
  preview(logId: string): CompactionPreview {
60
92
  const state = this.state(logId);
61
- const totals = this.exec<{ count: number; bytes: number }>(`SELECT COUNT(*) AS count,
93
+ const totals = this.query<{ count: number; bytes: number }>(`SELECT COUNT(*) AS count,
62
94
  COALESCE(SUM(length(CAST(payload AS BLOB))), 0) AS bytes FROM log_events
63
- WHERE log_id = ? AND version <= ? AND archive_key IS NULL`, logId, state.checkpoint?.throughVersion ?? 0).one();
95
+ WHERE log_id = ? AND version <= ? AND archive_key IS NULL`, logId, state.checkpoint?.throughVersion ?? 0)[0]!;
64
96
  return { checkpointId: state.checkpoint?.id ?? null, storageGeneration: state.storageGeneration,
65
97
  throughVersion: state.checkpoint?.throughVersion ?? 0, activePayloadEvents: totals.count, activePayloadBytes: totals.bytes,
66
98
  snapshotBytes: state.checkpoint?.pages.reduce((bytes, page) => bytes + page.bytes, 0) ?? 0 };
@@ -70,16 +102,16 @@ export class LogMaintenanceStore {
70
102
  if (!this.bucket) throw new LedgerError(503, { error: 'Unavailable', code: 'archive_unconfigured', message: 'Configure private Ledger archive storage first.' });
71
103
  this.storage.transactionSync(() => {
72
104
  this.assertWritable();
73
- const prior = this.exec<JobRow>('SELECT * FROM log_maintenance_jobs WHERE id = ?', id).toArray()[0];
105
+ const prior = this.query<JobRow>('SELECT id, log_id, kind FROM log_maintenance_jobs WHERE id = ?', id)[0];
74
106
  if (prior) {
75
107
  if (prior.log_id !== logId || prior.kind !== kind) throw validationError('Maintenance request ID already used');
76
108
  return;
77
109
  }
78
- const active = this.exec<JobRow>("SELECT * FROM log_maintenance_jobs WHERE log_id = ? AND status NOT IN ('complete', 'failed') LIMIT 1", logId).toArray()[0];
110
+ const active = this.query<JobRow>("SELECT id FROM log_maintenance_jobs WHERE log_id = ? AND status NOT IN ('complete', 'failed') LIMIT 1", logId)[0];
79
111
  if (active) throw new LedgerError(409, { error: 'Conflict', code: 'maintenance_in_progress', message: 'Resume the existing maintenance operation.' });
80
- const log = this.exec<{ version: number; status: string }>('SELECT version, status FROM logs WHERE id = ?', logId).toArray()[0];
112
+ const log = this.query<{ version: number; status: string }>('SELECT version, status FROM logs WHERE id = ?', logId)[0];
81
113
  if (!log || log.status !== 'ready') throw validationError('Choose a ready log');
82
- const boundary = this.exec<{ action_index: number | null; action_size: number | null }>('SELECT action_index, action_size FROM log_events WHERE log_id = ? AND version = ?', logId, log.version).toArray()[0];
114
+ const boundary = this.query<{ action_index: number | null; action_size: number | null }>('SELECT action_index, action_size FROM log_events WHERE log_id = ? AND version = ?', logId, log.version)[0];
83
115
  if (boundary?.action_size && boundary.action_index !== boundary.action_size - 1) throw validationError('Incomplete action boundary');
84
116
  const checkpoint = this.state(logId).checkpoint;
85
117
  if (kind === 'compact' && (expectedCheckpointId !== checkpoint?.id || expectedGeneration !== this.layout(logId).generation)) {
@@ -88,9 +120,11 @@ export class LogMaintenanceStore {
88
120
  if (kind === 'compact' && (!checkpoint || checkpoint.throughVersion <= this.layout(logId).compacted_through)) {
89
121
  throw new LedgerError(409, { error: 'Conflict', code: 'snapshot_required', message: 'Create a newer verified snapshot before compacting.' });
90
122
  }
91
- this.exec(`INSERT INTO log_maintenance_jobs (id, log_id, kind, through_version, status, cursor, entity_cursor, pages, updated_at, day, writes)
92
- VALUES (?, ?, ?, ?, ?, 0, '', '[]', ?, ?, 0)`, id, logId, kind,
93
- kind === 'compact' ? checkpoint!.throughVersion : log.version, kind === 'compact' ? 'archiving' : 'building', new Date().toISOString(), day());
123
+ const start = kind === 'snapshot' ? this.initialBuild(logId, log.version) : { phase: null, cursor: 0 };
124
+ this.query(`INSERT INTO log_maintenance_jobs (id, log_id, kind, through_version, status, cursor, entity_cursor, pages, updated_at, day, writes, phase)
125
+ VALUES (?, ?, ?, ?, ?, ?, '', '[]', ?, ?, 0, ?)`, id, logId, kind,
126
+ kind === 'compact' ? checkpoint!.throughVersion : log.version, kind === 'compact' ? 'archiving' : 'building', start.cursor,
127
+ new Date().toISOString(), day(), start.phase);
94
128
  });
95
129
  return this.state(logId, id);
96
130
  }
@@ -101,10 +135,10 @@ export class LogMaintenanceStore {
101
135
  const started = Date.now();
102
136
  let result: LogMaintenance | undefined;
103
137
  try {
104
- const before = this.exec<JobRow>('SELECT * FROM log_maintenance_jobs WHERE id = ? AND log_id = ?', id, logId).toArray()[0];
138
+ const before = this.query<JobRow>('SELECT day, reads, writes FROM log_maintenance_jobs WHERE id = ? AND log_id = ?', id, logId)[0];
105
139
  result = await this.stepValue(logId, id);
106
140
  this.assertWritable();
107
- this.exec('UPDATE log_maintenance_jobs SET reads = ?, writes = ? WHERE id = ? AND log_id = ?',
141
+ this.query('UPDATE log_maintenance_jobs SET reads = ?, writes = ? WHERE id = ? AND log_id = ?',
108
142
  (before?.day === day() ? before.reads : 0) + this.cursors.reduce((count, cursor) => count + cursor.rowsRead, 0),
109
143
  (before?.day === day() ? before.writes : 0) + this.cursors.reduce((count, cursor) => count + cursor.rowsWritten, 0) + 1,
110
144
  id, logId);
@@ -118,34 +152,81 @@ export class LogMaintenanceStore {
118
152
  }
119
153
  }
120
154
 
155
+ /** Active jobs whose next step is due now, least recently advanced first. */
156
+ dueJobs(now = Date.now()): Array<{ logId: string; id: string }> {
157
+ return this.activeJobs().filter(job => job.wake <= now).sort((a, b) => a.updatedAt.localeCompare(b.updatedAt))
158
+ .map(job => ({ logId: job.logId, id: job.id }));
159
+ }
160
+
161
+ /** Earliest time any active job needs another step, or null when none does. */
162
+ nextWake(): number | null {
163
+ const wakes = this.activeJobs().map(job => job.wake);
164
+ return wakes.length ? Math.min(...wakes) : null;
165
+ }
166
+
167
+ hasActiveJob(): boolean {
168
+ return this.query("SELECT 1 FROM log_maintenance_jobs WHERE status NOT IN ('complete', 'failed') LIMIT 1").length > 0;
169
+ }
170
+
171
+ private activeJobs(): Array<{ id: string; logId: string; wake: number; updatedAt: string }> {
172
+ const now = Date.now();
173
+ return this.query<JobRow>(`SELECT id, log_id, status, day, updated_at, retry_at, resume_at FROM log_maintenance_jobs
174
+ WHERE status NOT IN ('complete', 'failed')`).map(job => {
175
+ let wake = now;
176
+ if (job.status === 'paused') wake = job.resume_at ? Date.parse(job.resume_at) : nextMidnight(job.day);
177
+ else if (job.retry_at) wake = Date.parse(job.retry_at);
178
+ return { id: job.id, logId: job.log_id, wake: Number.isFinite(wake) ? wake : now, updatedAt: job.updated_at };
179
+ });
180
+ }
181
+
121
182
  private async stepValue(logId: string, id: string): Promise<LogMaintenance> {
122
183
  if (!this.bucket) throw new Error('Archive storage unavailable');
123
184
  this.assertWritable();
124
- let job = this.exec<JobRow>('SELECT * FROM log_maintenance_jobs WHERE id = ? AND log_id = ?', id, logId).toArray()[0];
185
+ let job = this.query<JobRow>('SELECT * FROM log_maintenance_jobs WHERE id = ? AND log_id = ?', id, logId)[0];
125
186
  if (!job) throw validationError('Maintenance operation not found');
126
187
  if (job.status === 'complete') return this.state(logId, id);
127
188
  if (this.state(logId).job?.id !== id) throw new LedgerError(409, { error: 'Conflict', code: 'stale_maintenance', message: 'A newer maintenance operation replaced this one.' });
189
+ const resumeStatus = (row: JobRow): MaintenanceJob['status'] => row.paused_status && row.paused_status !== 'paused' && row.paused_status !== 'failed'
190
+ ? row.paused_status : row.kind === 'snapshot' ? 'building' : 'archiving';
128
191
  if (job.status === 'failed') {
129
- this.exec("UPDATE log_maintenance_jobs SET status = ?, error = NULL WHERE id = ?", job.kind === 'snapshot' ? 'building' : 'archiving', id);
130
- job = { ...job, status: job.kind === 'snapshot' ? 'building' : 'archiving', error: null };
192
+ // An explicit retry: start counting attempts again from the failed step.
193
+ this.query("UPDATE log_maintenance_jobs SET status = ?, error = NULL, attempts = 0, retry_at = NULL, paused_status = NULL WHERE id = ?", resumeStatus(job), id);
194
+ job = { ...job, status: resumeStatus(job), error: null, attempts: 0, retry_at: null, paused_status: null };
131
195
  }
132
196
  if (job.day !== day()) {
133
- this.exec('UPDATE log_maintenance_jobs SET day = ?, writes = 0, reads = 0, status = ? WHERE id = ?', day(), job.kind === 'snapshot' ? 'building' : 'archiving', id);
134
- job = { ...job, day: day(), writes: 0, reads: 0, status: job.kind === 'snapshot' ? 'building' : 'archiving' };
197
+ const status = job.status === 'paused' ? resumeStatus(job) : job.status;
198
+ this.query('UPDATE log_maintenance_jobs SET day = ?, writes = 0, reads = 0, status = ?, pause_reason = NULL, resume_at = NULL, paused_status = NULL WHERE id = ?', day(), status, id);
199
+ job = { ...job, day: day(), writes: 0, reads: 0, status, pause_reason: null, resume_at: null, paused_status: null };
135
200
  }
136
- if (job.writes >= this.dayWriteBudget || job.reads >= this.dayReadBudget) {
137
- this.exec("UPDATE log_maintenance_jobs SET status = 'paused' WHERE id = ?", id);
201
+ if (job.status === 'paused' || job.writes >= this.dayWriteBudget || job.reads >= this.dayReadBudget) {
202
+ if (job.status !== 'paused') {
203
+ const reason = job.writes >= this.dayWriteBudget ? 'writes' : 'reads';
204
+ this.query("UPDATE log_maintenance_jobs SET status = 'paused', paused_status = ?, pause_reason = ?, resume_at = ?, updated_at = ? WHERE id = ?",
205
+ job.status, reason, new Date(nextMidnight(job.day)).toISOString(), new Date().toISOString(), id);
206
+ }
138
207
  return this.state(logId, id);
139
208
  }
140
209
  try {
141
210
  if (job.status === 'building') await this.build(job);
142
211
  else if (job.status === 'cleaning') await this.clean(job);
143
212
  else if (job.status === 'archiving' || job.status === 'reclaiming') await this.archive(job);
213
+ if (job.attempts > 0 || job.error) {
214
+ this.query('UPDATE log_maintenance_jobs SET attempts = 0, error = NULL, retry_at = NULL WHERE id = ?', id);
215
+ }
144
216
  } catch (cause) {
145
- // Keep source rows and page cursors untouched when verification fails.
217
+ // Access problems are not job failures; source rows and cursors stay untouched.
146
218
  this.assertWritable();
147
- this.exec("UPDATE log_maintenance_jobs SET status = 'failed', error = ?, updated_at = ? WHERE id = ?",
148
- cause instanceof Error ? cause.message : 'Maintenance failed', new Date().toISOString(), id);
219
+ const attempts = job.attempts + 1;
220
+ const message = cause instanceof Error ? cause.message : 'Maintenance failed';
221
+ const now = new Date().toISOString();
222
+ if (attempts >= MAX_ATTEMPTS) {
223
+ this.query("UPDATE log_maintenance_jobs SET status = 'failed', paused_status = ?, error = ?, attempts = ?, retry_at = NULL, updated_at = ? WHERE id = ?",
224
+ job.status, message, attempts, now, id);
225
+ } else {
226
+ // Stay in the current stage and let the alarm retry with backoff; the error stays visible meanwhile.
227
+ this.query('UPDATE log_maintenance_jobs SET error = ?, attempts = ?, retry_at = ?, updated_at = ? WHERE id = ?',
228
+ message, attempts, new Date(Date.now() + Math.min(600_000, 15_000 * 2 ** (attempts - 1))).toISOString(), now, id);
229
+ }
149
230
  }
150
231
  return this.state(logId, id);
151
232
  }
@@ -164,16 +245,16 @@ export class LogMaintenanceStore {
164
245
  /** Bounded archive pages are verified before any accepted payload is reclaimed. */
165
246
  async hydrate(logId: string, events: CheckpointEvent[]): Promise<CheckpointEvent[]> {
166
247
  if (!events.length) return events;
167
- const references = this.exec<{ event_id: string; archive_key: string }>(
248
+ const references = this.query<{ event_id: string; archive_key: string }>(
168
249
  'SELECT event_id, archive_key FROM log_events WHERE log_id = ? AND event_id IN (SELECT value FROM json_each(?)) AND archive_key IS NOT NULL',
169
- logId, JSON.stringify(events.map(event => event.id))).toArray();
250
+ logId, JSON.stringify(events.map(event => event.id)));
170
251
  if (!references.length) return events;
171
252
  if (!this.bucket) throw archiveError('Archive storage unavailable');
172
253
  const wanted = new Set(events.map(event => event.id));
173
254
  const payloads = new Map<string, CheckpointEvent>();
174
255
  for (const key of new Set(references.map(row => row.archive_key))) {
175
256
  if (!key.startsWith(`v1/${this.appId}/`)) throw archiveError('Archive scope mismatch');
176
- const reference = this.exec<{ sha256: string }>('SELECT sha256 FROM log_archive_pages WHERE object_key = ? LIMIT 1', key).toArray()[0];
257
+ const reference = this.query<{ sha256: string }>('SELECT sha256 FROM log_archive_pages WHERE object_key = ? LIMIT 1', key)[0];
177
258
  if (!reference) throw archiveError('Archive manifest is missing');
178
259
  const object = await this.getObject(key);
179
260
  if (!object) throw archiveError('Archived history is missing');
@@ -192,154 +273,266 @@ export class LogMaintenanceStore {
192
273
  }
193
274
 
194
275
  private layout(logId: string): LayoutRow {
195
- return this.exec<LayoutRow>('SELECT * FROM log_storage WHERE log_id = ?', logId).toArray()[0]
276
+ return this.query<LayoutRow>('SELECT * FROM log_storage WHERE log_id = ?', logId)[0]
196
277
  ?? { log_id: logId, compacted_through: 0, generation: 0, checkpoint: null };
197
278
  }
198
279
 
199
280
  private async build(job: JobRow): Promise<void> {
200
- if (job.cursor < job.through_version) {
201
- // Scan the primary log index once. Work records are populated in
202
- // budgeted batches rather than building an index over old history
203
- // during a schema migration (which could exhaust free-tier writes).
204
- const events = await this.readEvents(job.log_id, job.cursor, job.through_version, 16);
205
- if (!events.length || events[0]!.version !== job.cursor + 1) throw new Error('Snapshot source contains a gap');
206
- const selected = this.selection(job.log_id, job.through_version);
207
- const groups = new Map<string, CheckpointEvent[]>();
208
- for (const event of events) {
209
- const group = groups.get(event.entityId) ?? [];
210
- group.push(event); groups.set(event.entityId, group);
211
- }
212
- for (const [entityId, source] of groups) {
213
- const prior = this.exec<{ object_key: string }>('SELECT object_key FROM log_checkpoint_entities WHERE job_id = ? AND entity_id = ?', job.id, entityId).toArray()[0];
214
- const previous = prior ? await this.readWork(prior.object_key) : { events: [], claims: [] };
215
- const last = source[source.length - 1]!;
216
- const witnesses = checkpointEntityEvents([...previous.events, ...source.filter(event => selected(event) && !directives.includes(event.type))], [
217
- event => typeof event.payload.status === 'string' ? `status:${event.payload.status}` : null,
218
- event => typeof event.payload.thumbnailRef === 'string' ? 'thumbnail:string' : null,
219
- ], standardTypes);
220
- const value = JSON.stringify({ events: witnesses, claims: [{ entityId, currentVersion: last.entityVersion, logVersion: last.version }] });
221
- if (new TextEncoder().encode(value).byteLength > 8_000_000) throw new Error('Entity replay state exceeds the snapshot work budget');
222
- const key = `v1/${this.appId}/${job.log_id}/work/${job.id}/${last.version}.json`;
223
- await this.putVerified(key, value);
224
- this.storage.transactionSync(() => {
225
- this.assertWritable();
226
- this.exec(`INSERT INTO log_checkpoint_entities (job_id, entity_id, object_key) VALUES (?, ?, ?)
227
- ON CONFLICT(job_id, entity_id) DO UPDATE SET object_key = excluded.object_key`, job.id, entityId, key);
228
- this.exec('UPDATE log_maintenance_jobs SET writes = writes + 2 WHERE id = ?', job.id);
229
- });
230
- if (prior && prior.object_key !== key) await this.deleteObject(prior.object_key);
231
- }
232
- this.assertWritable();
233
- this.exec('UPDATE log_maintenance_jobs SET cursor = ?, writes = writes + 1, updated_at = ? WHERE id = ?',
234
- events[events.length - 1]!.version, new Date().toISOString(), job.id);
235
- return;
281
+ if (!job.phase || !['seed', 'fold', 'pages'].includes(job.phase)) return this.reset(job);
282
+ if (job.phase === 'pages') return this.pages(job);
283
+ return this.fold(job);
284
+ }
285
+
286
+ /** Where a new snapshot starts: from the previous one's retained facts when no later replay directive can reach behind it. */
287
+ private initialBuild(logId: string, through: number): { phase: Phase; cursor: number } {
288
+ const checkpoint = this.state(logId).checkpoint;
289
+ if (checkpoint && checkpoint.throughVersion > 0 && checkpoint.throughVersion <= through
290
+ && !this.query(`SELECT 1 FROM log_events WHERE log_id = ? AND version > ? AND event_type IN ('EVENT_TOMBSTONED', 'WORKSPACE_RESTORED', 'EVENTS_EXCISED') LIMIT 1`,
291
+ logId, checkpoint.throughVersion).length) {
292
+ return { phase: 'seed', cursor: checkpoint.throughVersion };
236
293
  }
237
- const pages = JSON.parse(job.pages) as CheckpointPage[];
238
- const entities = this.exec<{ entity_id: string; object_key: string }>(`SELECT entity_id, object_key FROM log_checkpoint_entities
239
- WHERE job_id = ? AND entity_id > ? ORDER BY entity_id LIMIT 16`, job.id, job.entity_cursor).toArray();
240
- if (!entities.length) {
241
- const manifest: CheckpointManifest = { format: 1, appId: this.appId, logId: job.log_id, id: job.id,
242
- throughVersion: job.through_version, createdAt: new Date().toISOString(), pages,
243
- eventCount: pages.reduce((count, page) => count + page.count, 0), sourceEventCount: job.through_version,
244
- sha256: await checkpointDigest(JSON.stringify(pages)) };
245
- this.storage.transactionSync(() => {
294
+ return { phase: 'fold', cursor: 0 };
295
+ }
296
+
297
+ /** Discard partial work (including work files of earlier builders) and start the build again. */
298
+ private async reset(job: JobRow): Promise<void> {
299
+ const legacy = this.query<{ entity_id: string; object_key: string }>('SELECT entity_id, object_key FROM log_checkpoint_entities WHERE job_id = ? LIMIT 16', job.id);
300
+ if (legacy.length) {
301
+ for (const row of legacy) {
302
+ if (row.object_key) await this.deleteObject(row.object_key);
246
303
  this.assertWritable();
247
- this.exec(`INSERT INTO log_storage (log_id, compacted_through, generation, checkpoint) VALUES (?, 0, 0, ?)
248
- ON CONFLICT(log_id) DO UPDATE SET checkpoint = excluded.checkpoint`, job.log_id, JSON.stringify(manifest));
249
- this.exec("UPDATE log_maintenance_jobs SET status = 'cleaning', updated_at = ? WHERE id = ?", new Date().toISOString(), job.id);
250
- });
304
+ this.query('DELETE FROM log_checkpoint_entities WHERE job_id = ? AND entity_id = ?', job.id, row.entity_id);
305
+ }
251
306
  return;
252
307
  }
253
- const chunk: CheckpointChunk = { events: [], claims: [] };
254
- let last = job.entity_cursor;
255
- for (const entity of entities) {
256
- const part = await this.readWork(entity.object_key);
257
- chunk.events.push(...part.events); chunk.claims.push(...part.claims);
258
- last = entity.entity_id;
259
- if (new TextEncoder().encode(JSON.stringify(chunk)).byteLength >= MAX_PAGE_BYTES) break;
308
+ const reference = workReference(job.entity_cursor);
309
+ if (reference) await this.deleteObject(reference.key);
310
+ const start = this.initialBuild(job.log_id, job.through_version);
311
+ this.assertWritable();
312
+ this.query(`UPDATE log_maintenance_jobs SET phase = ?, cursor = ?, version_cursor = 0, entity_cursor = '', pages = '[]', markers = NULL, updated_at = ?
313
+ WHERE id = ?`, start.phase, start.cursor, new Date().toISOString(), job.id);
314
+ }
315
+
316
+ /**
317
+ * The whole job's replay slots live in one private object, read and written once per step,
318
+ * so SQL writes grow with steps rather than with entities. null when it was lost (the build restarts).
319
+ */
320
+ private async loadState(job: JobRow): Promise<Map<string, WorkState> | null> {
321
+ const reference = workReference(job.entity_cursor);
322
+ if (!reference) return new Map();
323
+ const object = await this.getObject(reference.key);
324
+ const value = object ? await object.text() : null;
325
+ if (value === null || await checkpointDigest(value) !== reference.sha256) {
326
+ console.warn('ledger_maintenance_state_lost', { appId: this.appId, logId: job.log_id, jobId: job.id });
327
+ this.assertWritable();
328
+ this.query("UPDATE log_maintenance_jobs SET entity_cursor = '', phase = NULL WHERE id = ?", job.id);
329
+ return null;
330
+ }
331
+ return new Map(Object.entries(JSON.parse(value) as Record<string, WorkState>));
332
+ }
333
+
334
+ private async saveState(job: JobRow, states: Map<string, WorkState>, next: { phase: Phase; cursor: number; versionCursor: number }): Promise<void> {
335
+ let value = '{';
336
+ let first = true;
337
+ for (const [entityId, state] of states) {
338
+ const encoded = JSON.stringify(state);
339
+ if (encoded.length > MAX_STATE_BYTES) throw new Error('Entity replay state exceeds the snapshot work budget');
340
+ value += `${first ? '' : ','}${JSON.stringify(entityId)}:${encoded}`;
341
+ first = false;
260
342
  }
261
- chunk.events.sort((a, b) => a.version - b.version);
262
- const value = JSON.stringify(chunk);
263
- const key = `v1/${this.appId}/${job.log_id}/snapshots/${job.id}/${pages.length}.json`;
264
- const sha256 = await this.putVerified(key, value);
265
- pages.push({ key, sha256, count: chunk.events.length, bytes: new TextEncoder().encode(value).byteLength });
343
+ value += '}';
344
+ if (value.length > MAX_WORK_BYTES) throw new Error('Snapshot work state exceeds the snapshot work budget');
345
+ const key = `v1/${this.appId}/${job.log_id}/work/${job.id}/${next.phase}-${next.cursor}-${next.versionCursor}.json`;
346
+ const sha256 = await this.put(key, value);
266
347
  this.assertWritable();
267
- this.exec('UPDATE log_maintenance_jobs SET entity_cursor = ?, pages = ?, writes = writes + 1, updated_at = ? WHERE id = ?',
268
- last, JSON.stringify(pages), new Date().toISOString(), job.id);
348
+ this.query('UPDATE log_maintenance_jobs SET phase = ?, cursor = ?, version_cursor = ?, entity_cursor = ?, updated_at = ? WHERE id = ?',
349
+ next.phase, next.cursor, next.versionCursor, `${key}#${sha256}`, new Date().toISOString(), job.id);
350
+ const previous = workReference(job.entity_cursor);
351
+ if (previous && previous.key !== key) await this.deleteObject(previous.key);
269
352
  }
270
353
 
271
- private async readWork(key: string): Promise<CheckpointChunk> {
272
- const object = await this.getObject(key);
273
- if (!object) throw new Error('Snapshot work page is missing');
274
- const encoded = await object.text();
275
- if (await checkpointDigest(encoded) !== object.customMetadata?.sha256) throw new Error('Snapshot work checksum mismatch');
276
- return JSON.parse(encoded) as CheckpointChunk;
354
+ /** Seed from the previous snapshot, then fold source events, for as long as one step allows. */
355
+ private async fold(job: JobRow): Promise<void> {
356
+ const states = await this.loadState(job);
357
+ if (!states) return;
358
+ const started = Date.now();
359
+ const work = (entityId: string): WorkState => {
360
+ let state = states.get(entityId);
361
+ if (!state) { state = { s: newCheckpointEntityState(), c: { currentVersion: 0, logVersion: 0 } }; states.set(entityId, state); }
362
+ return state;
363
+ };
364
+ let phase = job.phase as Phase, cursor = job.cursor, versionCursor = job.version_cursor;
365
+ if (phase === 'seed') {
366
+ const layout = this.layout(job.log_id);
367
+ const checkpoint = layout.checkpoint ? JSON.parse(layout.checkpoint) as CheckpointManifest : null;
368
+ if (!checkpoint || checkpoint.throughVersion !== job.cursor) throw new Error('The previous snapshot changed while seeding');
369
+ while (versionCursor < checkpoint.pages.length && Date.now() - started < STEP_MS) {
370
+ const chunk = await this.chunk(job.log_id, versionCursor);
371
+ for (const event of [...chunk.events].sort((a, b) => a.version - b.version)) foldCheckpointEvent(work(event.entityId).s, event, SELECTORS, standardTypes);
372
+ for (const claim of chunk.claims) work(claim.entityId).c = { currentVersion: claim.currentVersion, logVersion: claim.logVersion };
373
+ versionCursor++;
374
+ }
375
+ if (versionCursor >= checkpoint.pages.length) { phase = 'fold'; versionCursor = 0; }
376
+ }
377
+ if (phase === 'fold') {
378
+ const selected = cursor < job.through_version ? this.selection(job) : () => true;
379
+ while (cursor < job.through_version && Date.now() - started < STEP_MS) {
380
+ const events = await this.readEvents(job.log_id, cursor, job.through_version, FOLD_EVENTS);
381
+ if (!events.length || events[0]!.version !== cursor + 1 || events[events.length - 1]!.version - events[0]!.version !== events.length - 1) {
382
+ throw new Error('Snapshot source contains a gap');
383
+ }
384
+ for (const event of events) {
385
+ const state = work(event.entityId);
386
+ if (selected(event) && !directives.includes(event.type)) foldCheckpointEvent(state.s, event, SELECTORS, standardTypes);
387
+ state.c = { currentVersion: event.entityVersion, logVersion: event.version };
388
+ }
389
+ cursor = events[events.length - 1]!.version;
390
+ }
391
+ if (cursor >= job.through_version) { phase = 'pages'; versionCursor = 0; }
392
+ }
393
+ await this.saveState(job, states, { phase, cursor, versionCursor });
277
394
  }
278
395
 
279
- private async clean(job: JobRow): Promise<void> {
280
- const rows = this.exec<{ entity_id: string; object_key: string }>('SELECT entity_id, object_key FROM log_checkpoint_entities WHERE job_id = ? LIMIT 16', job.id).toArray();
281
- for (const row of rows) {
282
- await this.deleteObject(row.object_key);
283
- this.assertWritable();
284
- this.exec('DELETE FROM log_checkpoint_entities WHERE job_id = ? AND entity_id = ?', job.id, row.entity_id);
396
+ /** Write claim pages and the retained source events in version order, then publish the manifest. */
397
+ private async pages(job: JobRow): Promise<void> {
398
+ const states = await this.loadState(job);
399
+ if (!states) return;
400
+ const started = Date.now();
401
+ const pages = JSON.parse(job.pages) as CheckpointPage[];
402
+ const pageKey = () => `v1/${this.appId}/${job.log_id}/snapshots/${job.id}/${pages.length}.json`;
403
+ const write = async (chunk: CheckpointChunk) => {
404
+ const value = JSON.stringify(chunk);
405
+ const key = pageKey();
406
+ pages.push({ key, sha256: await this.put(key, value), count: chunk.events.length, bytes: new TextEncoder().encode(value).byteLength });
407
+ };
408
+ if (!pages.length && job.version_cursor === 0) {
409
+ const claims = [...states].filter(([, state]) => state.c.logVersion > 0).sort(([a], [b]) => a < b ? -1 : a > b ? 1 : 0)
410
+ .map(([entityId, state]) => ({ entityId, currentVersion: state.c.currentVersion, logVersion: state.c.logVersion }));
411
+ for (let offset = 0; offset < claims.length; offset += CLAIMS_PER_PAGE) await write({ events: [], claims: claims.slice(offset, offset + CLAIMS_PER_PAGE) });
412
+ }
413
+ const retained = new Set<number>();
414
+ for (const state of states.values()) for (const version of checkpointStateVersions(state.s)) retained.add(version);
415
+ const versions = [...retained].filter(version => version > job.version_cursor).sort((a, b) => a - b);
416
+ let versionCursor = job.version_cursor;
417
+ let chunk: CheckpointChunk = { events: [], claims: [] };
418
+ let bytes = 2;
419
+ let offset = 0;
420
+ while (offset < versions.length && Date.now() - started < STEP_MS) {
421
+ const wanted = versions.slice(offset, offset + READ_VERSIONS);
422
+ const events = await this.readVersions(job.log_id, wanted);
423
+ if (events.length !== wanted.length || events.some((event, index) => event.version !== wanted[index])) throw new Error('Snapshot source is missing retained events');
424
+ for (const event of events) {
425
+ const size = new TextEncoder().encode(JSON.stringify(event)).byteLength + 1;
426
+ if (chunk.events.length && bytes + size >= MAX_PAGE_BYTES) {
427
+ await write(chunk);
428
+ versionCursor = chunk.events[chunk.events.length - 1]!.version;
429
+ chunk = { events: [], claims: [] }; bytes = 2;
430
+ }
431
+ chunk.events.push(event); bytes += size;
432
+ }
433
+ offset += wanted.length;
434
+ }
435
+ if (chunk.events.length) {
436
+ await write(chunk);
437
+ versionCursor = chunk.events[chunk.events.length - 1]!.version;
438
+ }
439
+ this.assertWritable();
440
+ if (offset < versions.length) {
441
+ this.query('UPDATE log_maintenance_jobs SET version_cursor = ?, pages = ?, updated_at = ? WHERE id = ?',
442
+ versionCursor, JSON.stringify(pages), new Date().toISOString(), job.id);
443
+ return;
285
444
  }
286
- this.exec('UPDATE log_maintenance_jobs SET status = ?, writes = writes + ?, updated_at = ? WHERE id = ?',
287
- rows.length ? 'cleaning' : 'complete', rows.length + 1, new Date().toISOString(), job.id);
445
+ const manifest: CheckpointManifest = { format: 1, appId: this.appId, logId: job.log_id, id: job.id,
446
+ throughVersion: job.through_version, createdAt: new Date().toISOString(), pages,
447
+ eventCount: pages.reduce((count, page) => count + page.count, 0), sourceEventCount: job.through_version,
448
+ sha256: await checkpointDigest(JSON.stringify(pages)) };
449
+ this.storage.transactionSync(() => {
450
+ this.assertWritable();
451
+ this.query(`INSERT INTO log_storage (log_id, compacted_through, generation, checkpoint) VALUES (?, 0, 0, ?)
452
+ ON CONFLICT(log_id) DO UPDATE SET checkpoint = excluded.checkpoint`, job.log_id, JSON.stringify(manifest));
453
+ this.query("UPDATE log_maintenance_jobs SET status = 'cleaning', version_cursor = ?, pages = ?, updated_at = ? WHERE id = ?",
454
+ versionCursor, JSON.stringify(pages), new Date().toISOString(), job.id);
455
+ });
456
+ }
457
+
458
+ private async clean(job: JobRow): Promise<void> {
459
+ const reference = workReference(job.entity_cursor);
460
+ if (reference) await this.deleteObject(reference.key);
461
+ this.assertWritable();
462
+ this.query("UPDATE log_maintenance_jobs SET status = 'complete', entity_cursor = '', updated_at = ? WHERE id = ?", new Date().toISOString(), job.id);
288
463
  }
289
464
 
290
- private async putVerified(key: string, value: string): Promise<string> {
465
+ /** R2 rejects an upload whose bytes do not match the digest, so no read-back is needed. */
466
+ private async put(key: string, value: string): Promise<string> {
291
467
  const sha256 = await checkpointDigest(value);
292
468
  this.io.r2Writes++;
293
469
  this.io.r2WrittenBytes += new TextEncoder().encode(value).byteLength;
294
- await this.bucket!.put(key, value, { customMetadata: { sha256 }, httpMetadata: { contentType: 'application/json' } });
295
- const check = await this.getObject(key);
296
- if (!check || await checkpointDigest(await check.text()) !== sha256) throw new Error('Snapshot verification failed');
470
+ await this.bucket!.put(key, value, { sha256, customMetadata: { sha256 }, httpMetadata: { contentType: 'application/json' } });
297
471
  return sha256;
298
472
  }
299
473
 
300
474
  private async archive(job: JobRow): Promise<void> {
301
475
  // Metadata stays as the compact identity/action registry. Only payloads
302
476
  // move, so historical boundaries and event/request uniqueness survive.
303
- const rows = this.exec<{ version: number }>(`SELECT version FROM log_events WHERE log_id = ?
304
- AND version <= ? AND version > ? AND archive_key IS NULL ORDER BY version LIMIT ?`, job.log_id, job.through_version, job.cursor, PAGE).toArray();
477
+ const rows = this.query<{ version: number }>(`SELECT version FROM log_events WHERE log_id = ?
478
+ AND version <= ? AND version > ? AND archive_key IS NULL ORDER BY version LIMIT ?`, job.log_id, job.through_version, job.cursor, PAGE);
305
479
  if (!rows.length) {
306
480
  this.storage.transactionSync(() => {
307
- this.assertWritable();
308
- this.exec('UPDATE log_storage SET compacted_through = MAX(compacted_through, ?), generation = generation + 1 WHERE log_id = ?', job.through_version, job.log_id);
309
- this.exec("UPDATE log_maintenance_jobs SET status = 'complete', cursor = ?, updated_at = ? WHERE id = ?", job.through_version, new Date().toISOString(), job.id);
481
+ this.assertWritable();
482
+ this.query('UPDATE log_storage SET compacted_through = MAX(compacted_through, ?), generation = generation + 1 WHERE log_id = ?', job.through_version, job.log_id);
483
+ this.query("UPDATE log_maintenance_jobs SET status = 'complete', cursor = ?, updated_at = ? WHERE id = ?", job.through_version, new Date().toISOString(), job.id);
310
484
  });
311
485
  return;
312
486
  }
313
487
  const first = rows[0]!.version;
314
488
  const events = await this.readEvents(job.log_id, first - 1, rows[rows.length - 1]!.version, PAGE);
315
489
  const value = JSON.stringify(events);
316
- const sha256 = await checkpointDigest(value);
317
490
  const key = `v1/${this.appId}/${job.log_id}/archive/${job.id}/${first}.json`;
318
- await this.putVerified(key, value);
491
+ const sha256 = await this.put(key, value);
319
492
  this.storage.transactionSync(() => {
320
493
  this.assertWritable();
321
- this.exec(`INSERT OR IGNORE INTO log_archive_pages (log_id, first_version, last_version, object_key, sha256) VALUES (?, ?, ?, ?, ?)`,
494
+ this.query(`INSERT OR IGNORE INTO log_archive_pages (log_id, first_version, last_version, object_key, sha256) VALUES (?, ?, ?, ?, ?)`,
322
495
  job.log_id, first, events[events.length - 1]!.version, key, sha256);
323
- const reclaimed = this.exec(`UPDATE log_events SET archive_key = ?, payload_bytes = COALESCE(payload_bytes, length(CAST(payload AS BLOB))), payload = CASE WHEN event_type IN
496
+ this.query(`UPDATE log_events SET archive_key = ?, payload_bytes = COALESCE(payload_bytes, length(CAST(payload AS BLOB))), payload = CASE WHEN event_type IN
324
497
  ('EVENT_TOMBSTONED', 'WORKSPACE_RESTORED', 'EVENTS_EXCISED') THEN payload ELSE '{}' END
325
498
  WHERE log_id = ? AND version >= ? AND version <= ? AND archive_key IS NULL`, key, job.log_id, first, events[events.length - 1]!.version);
326
- reclaimed.toArray();
327
- this.exec("UPDATE log_maintenance_jobs SET status = 'archiving', cursor = ?, writes = writes + ?, updated_at = ? WHERE id = ?",
328
- events[events.length - 1]!.version, reclaimed.rowsWritten + 2, new Date().toISOString(), job.id);
499
+ this.query('UPDATE log_storage SET archived_through = MAX(archived_through, ?) WHERE log_id = ?', events[events.length - 1]!.version, job.log_id);
500
+ this.query("UPDATE log_maintenance_jobs SET status = 'archiving', cursor = ?, updated_at = ? WHERE id = ?",
501
+ events[events.length - 1]!.version, new Date().toISOString(), job.id);
329
502
  });
330
503
  }
331
504
 
332
- private selection(logId: string, through: number): (event: CheckpointEvent) => boolean {
333
- const rows = this.exec<{ version: number; event_id: string; event_type: string; payload: string }>(`SELECT version, event_id, event_type, payload FROM log_events
334
- WHERE log_id = ? AND version <= ? AND event_type IN ('EVENT_TOMBSTONED', 'WORKSPACE_RESTORED', 'EVENTS_EXCISED') ORDER BY version DESC`, logId, through).toArray();
335
- return checkpointSelection(rows.map(event => ({ id: event.event_id, type: event.event_type,
336
- version: event.version, payload: JSON.parse(event.payload) as Record<string, unknown> })), through);
505
+ /** Replay directives up to the snapshot boundary, resolved once per job. */
506
+ private selection(job: JobRow): (event: CheckpointEvent) => boolean {
507
+ let markers: Marker[];
508
+ if (job.markers) {
509
+ markers = JSON.parse(job.markers) as Marker[];
510
+ } else {
511
+ markers = this.query<{ version: number; event_id: string; event_type: string; payload: string }>(`SELECT version, event_id, event_type, payload FROM log_events
512
+ WHERE log_id = ? AND version <= ? AND event_type IN ('EVENT_TOMBSTONED', 'WORKSPACE_RESTORED', 'EVENTS_EXCISED') ORDER BY version DESC`,
513
+ job.log_id, job.through_version).map(event => ({ id: event.event_id, type: event.event_type,
514
+ version: event.version, payload: JSON.parse(event.payload) as Record<string, unknown> }));
515
+ const encoded = JSON.stringify(markers);
516
+ if (encoded.length < MAX_MARKER_BYTES) {
517
+ this.assertWritable();
518
+ this.query('UPDATE log_maintenance_jobs SET markers = ? WHERE id = ?', encoded, job.id);
519
+ }
520
+ }
521
+ return checkpointSelection(markers, job.through_version);
337
522
  }
338
523
  }
339
524
 
525
+ /** The job's current work-state object, stored as `key#sha256` in the job row. */
526
+ function workReference(value: string): { key: string; sha256: string } | null {
527
+ const separator = value.lastIndexOf('#');
528
+ return separator > 0 ? { key: value.slice(0, separator), sha256: value.slice(separator + 1) } : null;
529
+ }
340
530
  function day(): string { return new Date().toISOString().slice(0, 10); }
531
+ /** Start of the UTC day after `dayValue` (YYYY-MM-DD). */
532
+ function nextMidnight(dayValue: string): number { return Date.parse(`${dayValue}T00:00:00.000Z`) + 86_400_000; }
341
533
  function archiveError(message: string): LedgerError { return new LedgerError(503, { error: 'Unavailable', code: 'archive_unavailable', message }); }
342
534
  function publicJob(row: JobRow): MaintenanceJob {
343
535
  return { id: row.id, logId: row.log_id, kind: row.kind, throughVersion: row.through_version, status: row.status,
344
- cursor: row.cursor, error: row.error, updatedAt: row.updated_at };
536
+ cursor: row.cursor, error: row.error, updatedAt: row.updated_at, attempts: row.attempts, retryAt: row.retry_at,
537
+ pauseReason: row.pause_reason, resumeAt: row.resume_at };
345
538
  }
@@ -1,4 +1,4 @@
1
- /** One in-flight step per operation, with a bounded wait for HTTP/RPC callers.
1
+ /** One in-flight step per operation (alarm and callers share it), with a bounded wait for HTTP/RPC callers.
2
2
  * Losing a response must not enqueue another copy of the same running step.
3
3
  */
4
4
  export class MaintenanceSteps<T> {
@@ -7,7 +7,8 @@ export class MaintenanceSteps<T> {
7
7
 
8
8
  constructor(private readonly keepAlive: (work: Promise<unknown>) => void, private readonly responseWaitMs = 1000) {}
9
9
 
10
- async advance(key: string, work: () => Promise<T>, current: () => T): Promise<T> {
10
+ /** Queue one step behind earlier ones (or join the identical step already in flight) and resolve with its real result. */
11
+ run(key: string, work: () => Promise<T>): Promise<T> {
11
12
  let run = this.running.get(key);
12
13
  if (!run) {
13
14
  run = this.queue.then(work);
@@ -18,6 +19,11 @@ export class MaintenanceSteps<T> {
18
19
  this.queue = settled;
19
20
  this.keepAlive(settled);
20
21
  }
22
+ return run;
23
+ }
24
+
25
+ async advance(key: string, work: () => Promise<T>, current: () => T): Promise<T> {
26
+ const run = this.run(key, work);
21
27
  let timer: ReturnType<typeof setTimeout> | undefined;
22
28
  try {
23
29
  return await Promise.race([run, new Promise<T>((resolve, reject) => {
package/src/schema.ts CHANGED
@@ -215,6 +215,46 @@ const LEDGER_APP_MIGRATIONS: readonly Migration[] = [
215
215
  );
216
216
  CREATE INDEX log_archive_object ON log_archive_pages(object_key);
217
217
  ` },
218
+ // Snapshot work state now lives in SQLite as per-entity replay slots (no R2
219
+ // work files). Jobs are advanced by an alarm and record retry/pause details.
220
+ { id: 15, sql: `
221
+ CREATE TABLE log_checkpoint_work (
222
+ job_id TEXT NOT NULL, entity_id TEXT NOT NULL, state TEXT NOT NULL,
223
+ PRIMARY KEY(job_id, entity_id)
224
+ );
225
+ CREATE TABLE log_checkpoint_versions (
226
+ job_id TEXT NOT NULL, version INTEGER NOT NULL, PRIMARY KEY(job_id, version)
227
+ );
228
+ ALTER TABLE log_maintenance_jobs ADD COLUMN phase TEXT;
229
+ ALTER TABLE log_maintenance_jobs ADD COLUMN version_cursor INTEGER NOT NULL DEFAULT 0;
230
+ ALTER TABLE log_maintenance_jobs ADD COLUMN attempts INTEGER NOT NULL DEFAULT 0;
231
+ ALTER TABLE log_maintenance_jobs ADD COLUMN retry_at TEXT;
232
+ ALTER TABLE log_maintenance_jobs ADD COLUMN resume_at TEXT;
233
+ ALTER TABLE log_maintenance_jobs ADD COLUMN pause_reason TEXT;
234
+ ALTER TABLE log_maintenance_jobs ADD COLUMN paused_status TEXT;
235
+ ALTER TABLE log_maintenance_jobs ADD COLUMN markers TEXT;
236
+ ` },
237
+ // Snapshot work state moves to one R2 object per job (SQL writes per step, not per entity), and
238
+ // archiving no longer writes an index entry per event: the archived prefix is tracked per log.
239
+ { id: 16, sql: `
240
+ ALTER TABLE log_storage ADD COLUMN archived_through INTEGER NOT NULL DEFAULT 0;
241
+ UPDATE log_storage SET archived_through = COALESCE((SELECT MAX(version) FROM log_events
242
+ WHERE log_events.log_id = log_storage.log_id AND archive_key IS NOT NULL), 0);
243
+ INSERT INTO log_storage (log_id, compacted_through, generation, checkpoint, archived_through)
244
+ SELECT log_id, 0, 0, NULL, MAX(version) FROM log_events
245
+ WHERE archive_key IS NOT NULL AND log_id NOT IN (SELECT log_id FROM log_storage) GROUP BY log_id;
246
+ DROP INDEX log_events_archived_cursor;
247
+ DROP TABLE log_checkpoint_work;
248
+ DROP TABLE log_checkpoint_versions;
249
+ -- Builds in progress restart with the new builder. Their counters measured the old one, so a
250
+ -- build that paused on them may continue today.
251
+ UPDATE log_maintenance_jobs SET phase = NULL, entity_cursor = '', version_cursor = 0, pages = '[]', markers = NULL,
252
+ writes = 0, reads = 0, attempts = 0, error = NULL, retry_at = NULL, status = 'building',
253
+ pause_reason = NULL, resume_at = NULL, paused_status = NULL
254
+ WHERE kind = 'snapshot' AND (status = 'building' OR (status = 'paused' AND COALESCE(paused_status, 'building') = 'building'));
255
+ UPDATE log_maintenance_jobs SET entity_cursor = '', status = 'cleaning', pause_reason = NULL, resume_at = NULL, paused_status = NULL
256
+ WHERE kind = 'snapshot' AND (status = 'cleaning' OR (status = 'paused' AND paused_status = 'cleaning'));
257
+ ` },
218
258
  ];
219
259
 
220
260
  const APP_REGISTRY_MIGRATIONS: readonly Migration[] = [