@hearthlog/ledger 0.10.2 → 0.11.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +16 -5
- package/package.json +2 -2
- package/scripts/maintenance.ts +1 -0
- package/src/ledger-app.ts +79 -23
- package/src/log-maintenance.ts +324 -131
- package/src/maintenance-steps.ts +8 -2
- package/src/schema.ts +40 -0
package/README.md
CHANGED
|
@@ -192,7 +192,8 @@ Authenticated endpoints under `/logs/:logId`:
|
|
|
192
192
|
- `GET /maintenance-preview`: the current snapshot/generation and prefix payload counts/bytes for review.
|
|
193
193
|
- `POST /maintenance`: `{ requestId, kind: "snapshot" | "compact", expectedCheckpointId?, expectedGeneration? }`.
|
|
194
194
|
Compaction requires the checkpoint ID and generation from the reviewed preview; stale previews fail with 409.
|
|
195
|
-
- `POST /maintenance/step`: `{ requestId }
|
|
195
|
+
- `POST /maintenance/step`: `{ requestId }`; runs one bounded step now (sharing any step already in flight) and restarts a
|
|
196
|
+
job that stopped after repeated failures. Normally unnecessary: the Ledger advances jobs by itself (see below).
|
|
196
197
|
- `GET /checkpoint?page=N`: one checksum-verified canonical replay chunk and its entity claims.
|
|
197
198
|
|
|
198
199
|
From the development site, the same operation can run without an open browser:
|
|
@@ -206,6 +207,17 @@ pnpm -C dev ledger:maintain --log <log-id> --resume
|
|
|
206
207
|
Set `LEDGER_BASE_URL` and `LEDGER_AUTH_TOKEN` in the process environment; the CLI never prints credentials.
|
|
207
208
|
Use `--request-id` to retry an explicit operation and `--once` for one bounded work step.
|
|
208
209
|
|
|
210
|
+
A started job is carried to completion by the application object's alarm, with no caller or open browser. Each step
|
|
211
|
+
works for a few seconds, folding source events into small per-entity replay slots (source versions only). The whole job's
|
|
212
|
+
slots live in one private R2 object that is read and written once per step, so a snapshot costs a handful of SQL writes
|
|
213
|
+
however many records the log holds. The retained events are then written as checksummed pages. Archiving rewrites each
|
|
214
|
+
archived event row once (its payload moves to R2); the archived prefix is tracked per log rather than indexed per event. A later snapshot starts from the previous one's retained facts instead of re-reading
|
|
215
|
+
archived history, unless a restore/removal marker has been added since. A failing step is retried with backoff; the
|
|
216
|
+
error and attempt count stay visible, and after five attempts the job stops as `failed` until it is explicitly retried
|
|
217
|
+
(the maintenance panel's Retry, `POST /maintenance/step`, or the CLI). A job that uses its daily budget is `paused`
|
|
218
|
+
with `pauseReason` and `resumeAt`, and resumes by itself at the next UTC day. Opening the panel re-arms a job whose
|
|
219
|
+
alarm was lost.
|
|
220
|
+
|
|
209
221
|
Jobs persist every source batch and publish only verified snapshots. Canonical snapshots keep original source
|
|
210
222
|
witnesses for raw field values, explicit nulls, timestamps and lifecycle transitions; dynamic/nonstandard facts
|
|
211
223
|
are retained losslessly. Browser projection uses a separate baseline table plus accepted suffix and pending
|
|
@@ -217,9 +229,8 @@ payload placeholders. The SDK's server checkpoint reader lets Stash seed a cold
|
|
|
217
229
|
`MAINTENANCE_DAILY_WRITES` defaults to `10000` per job (bounded to 128–100000), and
|
|
218
230
|
`MAINTENANCE_DAILY_READS` defaults to `1000000` (bounded to 10000–1000000000). Jobs pause at either budget. Budgets use consumed SQL cursor
|
|
219
231
|
counts and can overrun by one bounded step. Structured step logs include SQL rows, R2 operations/bytes, elapsed
|
|
220
|
-
time and replay counts, without payloads or credentials.
|
|
221
|
-
|
|
222
|
-
can be closed safely and the operation resumed from the panel. No automatic snapshots or archive expiry run.
|
|
232
|
+
time and replay counts, without payloads or credentials. Other applications consume account quota too. The browser
|
|
233
|
+
can be closed safely; the job continues on the server. No automatic snapshots or archive expiry run.
|
|
223
234
|
Deleting a whole log uses bounded background cleanup of that log's private archive objects, after normal
|
|
224
235
|
fork/deletion checks. Individual archived-history deletion is a separate future feature.
|
|
225
236
|
|
|
@@ -228,7 +239,7 @@ payloads into the existing portable format. A SQLite-only export returns `409 ar
|
|
|
228
239
|
external checkpoint/archive objects are required, avoiding a misleading incomplete backup. Rollback after
|
|
229
240
|
compaction needs a compatible reader or a full logical restore first.
|
|
230
241
|
|
|
231
|
-
The first format bounds a single entity's replay
|
|
242
|
+
The first format bounds a single entity's replay state to about 1.5 MB and a job's work state to 64 MB and stores page descriptors in SQLite.
|
|
232
243
|
Exceeding a snapshot work/metadata bound fails the job while source history stays readable. Measure maintenance
|
|
233
244
|
work and storage on representative data; retained identity metadata still grows, and physical SQLite file size
|
|
234
245
|
need not shrink immediately when payload space becomes reusable.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@hearthlog/ledger",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.11.1",
|
|
4
4
|
"description": "Hearthlog's Ledger: the event store Worker (Elysia over SQLite Durable Objects), its contracts and tests, and the logical backup tooling",
|
|
5
5
|
"license": "AGPL-3.0-only",
|
|
6
6
|
"repository": {
|
|
@@ -30,7 +30,7 @@
|
|
|
30
30
|
"dependencies": {
|
|
31
31
|
"@sinclair/typebox": "0.34.52",
|
|
32
32
|
"elysia": "1.4.30",
|
|
33
|
-
"@hearthlog/contracts": "0.
|
|
33
|
+
"@hearthlog/contracts": "0.11.1"
|
|
34
34
|
},
|
|
35
35
|
"peerDependencies": {
|
|
36
36
|
"@hearthlog/generated": "*"
|
package/scripts/maintenance.ts
CHANGED
|
@@ -45,6 +45,7 @@ async function main(): Promise<void> {
|
|
|
45
45
|
await new Promise(resolve => setTimeout(resolve, 500));
|
|
46
46
|
}
|
|
47
47
|
if (state.job?.status === 'failed') throw new Error(state.job.error ?? 'Maintenance failed');
|
|
48
|
+
if (state.job?.status === 'paused') console.error(`Paused (${state.job.pauseReason ?? 'budget'}); the Ledger resumes at ${state.job.resumeAt ?? 'the next UTC day'}.`);
|
|
48
49
|
} while (!values.once && state.job?.status !== 'complete' && state.job?.status !== 'paused');
|
|
49
50
|
console.log(JSON.stringify({ logId: values.log, requestId, status: state.job?.status,
|
|
50
51
|
checkpointId: state.checkpoint?.id ?? null, throughVersion: state.checkpoint?.throughVersion ?? 0,
|
package/src/ledger-app.ts
CHANGED
|
@@ -148,14 +148,18 @@ export class LedgerApp extends DurableObject<Env> {
|
|
|
148
148
|
private maintenance(assertWritable: () => void = () => undefined): LogMaintenanceStore {
|
|
149
149
|
const appId = this.ctx.storage.sql.exec<{ app_id: string }>('SELECT app_id FROM app_metadata WHERE singleton = 1').one().app_id;
|
|
150
150
|
const store = new LogMaintenanceStore(this.ctx.storage, this.env.LEDGER_ARCHIVE, appId,
|
|
151
|
-
async (logId, after, through, limit
|
|
151
|
+
async (logId, after, through, limit) => {
|
|
152
152
|
const rows = store.observeSql(this.ctx.storage.sql.exec<EventRow>(`WITH candidates AS (
|
|
153
153
|
SELECT *, SUM(COALESCE(payload_bytes, length(CAST(payload AS BLOB)))) OVER (ORDER BY version) AS byte_count,
|
|
154
154
|
ROW_NUMBER() OVER (ORDER BY version) AS row_number
|
|
155
|
-
FROM (SELECT * FROM log_events WHERE log_id = ? AND version > ? AND version <= ?
|
|
156
|
-
${entityId === undefined ? '' : 'AND entity_id = ?'} ORDER BY version LIMIT ?)
|
|
155
|
+
FROM (SELECT * FROM log_events WHERE log_id = ? AND version > ? AND version <= ? ORDER BY version LIMIT ?)
|
|
157
156
|
) SELECT * FROM candidates WHERE byte_count <= 1500000 OR row_number = 1`,
|
|
158
|
-
logId, after, through,
|
|
157
|
+
logId, after, through, limit)).toArray();
|
|
158
|
+
return store.hydrate(logId, rows.map(storedEvent));
|
|
159
|
+
}, async (logId, versions) => {
|
|
160
|
+
const rows = store.observeSql(this.ctx.storage.sql.exec<EventRow>(
|
|
161
|
+
'SELECT * FROM log_events WHERE log_id = ? AND version IN (SELECT value FROM json_each(?)) ORDER BY version',
|
|
162
|
+
logId, JSON.stringify(versions))).toArray();
|
|
159
163
|
return store.hydrate(logId, rows.map(storedEvent));
|
|
160
164
|
}, assertWritable, Math.max(128, Math.min(100000, Number(this.env.MAINTENANCE_DAILY_WRITES) || 10000)),
|
|
161
165
|
Math.max(10000, Math.min(1_000_000_000, Number(this.env.MAINTENANCE_DAILY_READS) || 1_000_000)));
|
|
@@ -163,10 +167,13 @@ export class LedgerApp extends DurableObject<Env> {
|
|
|
163
167
|
}
|
|
164
168
|
|
|
165
169
|
async logMaintenance(access: AccessClaims, logId: string): Promise<RpcResult<LogMaintenance>> {
|
|
166
|
-
return rpcCall(() => {
|
|
170
|
+
return rpcCall(async () => {
|
|
167
171
|
this.currentAccess(access);
|
|
168
172
|
if (!this.log(parseUuid(logId, 'log id'))) throw validationError('Log not found');
|
|
169
|
-
|
|
173
|
+
const store = this.maintenance();
|
|
174
|
+
// A job begun before alarms existed (or whose alarm was lost) is picked up again here.
|
|
175
|
+
if (store.hasActiveJob() && await this.ctx.storage.getAlarm() === null) await this.scheduleAlarm();
|
|
176
|
+
return rpcSuccess(store.state(logId));
|
|
170
177
|
});
|
|
171
178
|
}
|
|
172
179
|
|
|
@@ -179,9 +186,12 @@ export class LedgerApp extends DurableObject<Env> {
|
|
|
179
186
|
}
|
|
180
187
|
|
|
181
188
|
async startLogMaintenance(access: AccessClaims, logId: string, id: string, kind: 'snapshot' | 'compact', expectedCheckpointId?: string, expectedGeneration?: number): Promise<RpcResult<LogMaintenance>> {
|
|
182
|
-
return rpcCall(() => {
|
|
189
|
+
return rpcCall(async () => {
|
|
183
190
|
this.writableAccess(access);
|
|
184
|
-
|
|
191
|
+
const state = this.maintenance().start(parseUuid(logId, 'log id'), parseUuid(id, 'request id'), kind, expectedCheckpointId, expectedGeneration);
|
|
192
|
+
// The server carries the job to completion; the caller only needs to watch.
|
|
193
|
+
await this.scheduleAlarm();
|
|
194
|
+
return rpcSuccess(state);
|
|
185
195
|
});
|
|
186
196
|
}
|
|
187
197
|
|
|
@@ -194,6 +204,8 @@ export class LedgerApp extends DurableObject<Env> {
|
|
|
194
204
|
() => this.maintenance(() => { this.writableAccess(access); }).step(parsedLog, parsedId),
|
|
195
205
|
() => this.maintenance().state(parsedLog, parsedId));
|
|
196
206
|
this.currentAccess(access);
|
|
207
|
+
// An explicit step also restarts a job that stopped after repeated failures.
|
|
208
|
+
await this.scheduleAlarm();
|
|
197
209
|
return rpcSuccess(result);
|
|
198
210
|
});
|
|
199
211
|
}
|
|
@@ -213,21 +225,59 @@ export class LedgerApp extends DurableObject<Env> {
|
|
|
213
225
|
});
|
|
214
226
|
}
|
|
215
227
|
|
|
216
|
-
/**
|
|
228
|
+
/** Advance log maintenance and delete storage of deleted logs, each in bounded batches. */
|
|
217
229
|
override async alarm(): Promise<void> {
|
|
230
|
+
let failed = false;
|
|
231
|
+
try { await this.deleteArchivedLogs(); }
|
|
232
|
+
catch (cause) { failed = true; console.error('ledger_archive_deletion_failed', normalizeError(cause).message); }
|
|
233
|
+
try { await this.advanceMaintenance(); }
|
|
234
|
+
catch (cause) { failed = true; console.error('ledger_maintenance_alarm_failed', normalizeError(cause).message); }
|
|
235
|
+
await this.scheduleAlarm({ deletionDelayMs: 60_000, failureDelayMs: failed ? 30_000 : undefined });
|
|
236
|
+
}
|
|
237
|
+
|
|
238
|
+
private async deleteArchivedLogs(): Promise<void> {
|
|
218
239
|
const job = this.ctx.storage.sql.exec<{ log_id: string }>('SELECT log_id FROM log_archive_deletions LIMIT 1').toArray()[0];
|
|
219
240
|
if (!job) return;
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
241
|
+
const app = this.ctx.storage.sql.exec<{ app_id: string }>('SELECT app_id FROM app_metadata WHERE singleton = 1').one();
|
|
242
|
+
const bucket = this.env.LEDGER_ARCHIVE;
|
|
243
|
+
const objects = await bucket.list({ prefix: `v1/${app.app_id}/${job.log_id}/`, limit: 128 });
|
|
244
|
+
if (objects.objects.length) await bucket.delete(objects.objects.map(object => object.key));
|
|
245
|
+
if (!objects.truncated) this.ctx.storage.sql.exec('DELETE FROM log_archive_deletions WHERE log_id = ?', job.log_id);
|
|
246
|
+
}
|
|
247
|
+
|
|
248
|
+
/** Run due maintenance steps for a few seconds, so a job finishes without any caller. */
|
|
249
|
+
private async advanceMaintenance(): Promise<void> {
|
|
250
|
+
if (this.metadataOptional() === undefined) return;
|
|
251
|
+
const started = Date.now();
|
|
252
|
+
const assertWritable = () => this.assertApplicationWritable();
|
|
253
|
+
do {
|
|
254
|
+
const due = this.maintenance(assertWritable).dueJobs()[0];
|
|
255
|
+
if (!due) return;
|
|
256
|
+
await this.maintenanceSteps.run(`${due.logId}:${due.id}`, () => this.maintenance(assertWritable).step(due.logId, due.id));
|
|
257
|
+
} while (Date.now() - started < 5000);
|
|
258
|
+
}
|
|
259
|
+
|
|
260
|
+
/** Wake at the earliest time any pending work (maintenance step, retry, resume, deletion) is due. */
|
|
261
|
+
private async scheduleAlarm(options: { deletionDelayMs?: number; failureDelayMs?: number } = {}): Promise<void> {
|
|
262
|
+
const now = Date.now();
|
|
263
|
+
const wakes: number[] = [];
|
|
264
|
+
if (this.ctx.storage.sql.exec('SELECT 1 FROM log_archive_deletions LIMIT 1').toArray().length) {
|
|
265
|
+
wakes.push(now + (options.deletionDelayMs ?? 1000));
|
|
230
266
|
}
|
|
267
|
+
if (this.metadataOptional() !== undefined) {
|
|
268
|
+
const wake = this.maintenance().nextWake();
|
|
269
|
+
if (wake !== null) wakes.push(Math.max(wake, now + (options.failureDelayMs ?? 50)));
|
|
270
|
+
}
|
|
271
|
+
if (wakes.length) await this.ctx.storage.setAlarm(Math.min(...wakes));
|
|
272
|
+
}
|
|
273
|
+
|
|
274
|
+
/** Background work has no caller to authorize it, but must still stop for revoked or fenced applications. */
|
|
275
|
+
private assertApplicationWritable(): void {
|
|
276
|
+
if (this.metadata().revoked_at !== null) {
|
|
277
|
+
throw new LedgerError(401, { error: "access token has been rotated or revoked", code: "unauthorized" });
|
|
278
|
+
}
|
|
279
|
+
const fence = this.fence();
|
|
280
|
+
if (fence) throw fenceError(fence);
|
|
231
281
|
}
|
|
232
282
|
|
|
233
283
|
override async fetch(request: Request): Promise<Response> {
|
|
@@ -435,7 +485,7 @@ export class LedgerApp extends DurableObject<Env> {
|
|
|
435
485
|
|
|
436
486
|
async administratorExportSqlite(appId: string): Promise<Response> {
|
|
437
487
|
this.administratorApplication(appId);
|
|
438
|
-
if (this.ctx.storage.sql.exec('SELECT 1 FROM
|
|
488
|
+
if (this.ctx.storage.sql.exec('SELECT 1 FROM log_storage WHERE archived_through > 0 LIMIT 1').toArray().length
|
|
439
489
|
|| this.ctx.storage.sql.exec('SELECT 1 FROM log_storage WHERE checkpoint IS NOT NULL LIMIT 1').toArray().length) {
|
|
440
490
|
return jsonError('archive_bundle_required', 'Use the complete logical export or application transfer; archived payloads are outside SQLite.', 409);
|
|
441
491
|
}
|
|
@@ -710,7 +760,7 @@ export class LedgerApp extends DurableObject<Env> {
|
|
|
710
760
|
return rpcCall(async () => {
|
|
711
761
|
const claims = this.writableAccess(access);
|
|
712
762
|
this.deleteLogValue(parseUuid(logId, "logId"), claims);
|
|
713
|
-
await this.
|
|
763
|
+
await this.scheduleAlarm();
|
|
714
764
|
return rpcSuccess(null, 204);
|
|
715
765
|
});
|
|
716
766
|
}
|
|
@@ -860,7 +910,7 @@ export class LedgerApp extends DurableObject<Env> {
|
|
|
860
910
|
throw transferError("not_importing", "Only an import in progress can be aborted");
|
|
861
911
|
}
|
|
862
912
|
this.ctx.storage.transactionSync(() => {
|
|
863
|
-
for (const table of ["log_storage", "log_archive_pages", "log_maintenance_jobs", "gallery_photo_references", "log_fork_requests", "append_requests", "entity_claims", "log_events", "logs", "app_metadata"]) {
|
|
913
|
+
for (const table of ["log_checkpoint_entities", "log_storage", "log_archive_pages", "log_maintenance_jobs", "gallery_photo_references", "log_fork_requests", "append_requests", "entity_claims", "log_events", "logs", "app_metadata"]) {
|
|
864
914
|
this.ctx.storage.sql.exec(`DELETE FROM ${table}`);
|
|
865
915
|
}
|
|
866
916
|
});
|
|
@@ -1114,6 +1164,12 @@ export class LedgerApp extends DurableObject<Env> {
|
|
|
1114
1164
|
FROM log_events WHERE log_id = ? AND version <= ? ORDER BY version`,
|
|
1115
1165
|
id, sourceLogId, targetVersion,
|
|
1116
1166
|
);
|
|
1167
|
+
// A fork shares its source's archived prefix up to the fork point.
|
|
1168
|
+
this.ctx.storage.sql.exec(
|
|
1169
|
+
`INSERT INTO log_storage (log_id, compacted_through, generation, checkpoint, archived_through)
|
|
1170
|
+
SELECT ?, 0, 0, NULL, MIN(archived_through, ?) FROM log_storage WHERE log_id = ? AND archived_through > 0`,
|
|
1171
|
+
id, targetVersion, sourceLogId,
|
|
1172
|
+
);
|
|
1117
1173
|
this.ctx.storage.sql.exec(
|
|
1118
1174
|
`INSERT INTO entity_claims (log_id, entity_id, current_version, version_created, created_at, updated_at)
|
|
1119
1175
|
SELECT ?, source.entity_id, source.entity_version, first_version.first_version, ?, ?
|
|
@@ -1606,7 +1662,7 @@ export class LedgerApp extends DurableObject<Env> {
|
|
|
1606
1662
|
return;
|
|
1607
1663
|
}
|
|
1608
1664
|
const head = this.logHead(attachment.logId);
|
|
1609
|
-
const archived = this.ctx.storage.sql.exec<{
|
|
1665
|
+
const archived = this.ctx.storage.sql.exec<{ archived_through: number }>('SELECT archived_through FROM log_storage WHERE log_id = ?', attachment.logId).toArray()[0]?.archived_through ?? 0;
|
|
1610
1666
|
if (attachment.cursor < archived) {
|
|
1611
1667
|
this.sendFrame(ws, { type: 'resync_required', headVersion: head, reason: 'gap_too_large' });
|
|
1612
1668
|
ws.close(1008, 'baseline resync required');
|
package/src/log-maintenance.ts
CHANGED
|
@@ -1,21 +1,47 @@
|
|
|
1
1
|
import { EVENT_SCHEMAS } from '#generated/event-schemas';
|
|
2
|
-
import { checkpointDigest,
|
|
3
|
-
type
|
|
2
|
+
import { checkpointDigest, checkpointSelection, checkpointStateVersions, foldCheckpointEvent, newCheckpointEntityState,
|
|
3
|
+
type CheckpointChunk, type CheckpointEntityState, type CheckpointEvent, type CheckpointManifest, type CheckpointPage,
|
|
4
|
+
type CheckpointSelector, type LogMaintenance, type MaintenanceJob, type CompactionPreview } from '@hearthlog/contracts/history/checkpoint';
|
|
4
5
|
import { LedgerError, validationError } from './errors';
|
|
5
6
|
|
|
7
|
+
/** Events archived per compaction step. */
|
|
6
8
|
const PAGE = 128;
|
|
9
|
+
/** Source events per read while folding (also bounded by the byte window of the reader). */
|
|
10
|
+
const FOLD_EVENTS = 512;
|
|
11
|
+
/** Retained events read per query while writing snapshot pages. */
|
|
12
|
+
const READ_VERSIONS = 256;
|
|
13
|
+
const CLAIMS_PER_PAGE = 4096;
|
|
14
|
+
/** Wall time one snapshot step keeps working before it saves its state. */
|
|
15
|
+
const STEP_MS = 4000;
|
|
16
|
+
const MAX_WORK_BYTES = 64_000_000;
|
|
7
17
|
const MAX_PAGE_BYTES = 1_500_000;
|
|
18
|
+
const MAX_STATE_BYTES = 1_500_000;
|
|
19
|
+
const MAX_MARKER_BYTES = 1_000_000;
|
|
20
|
+
/** Automatic attempts before a job stops and waits for an explicit retry. */
|
|
21
|
+
export const MAX_ATTEMPTS = 5;
|
|
8
22
|
const DEFAULT_DAY_WRITE_BUDGET = 10_000;
|
|
9
23
|
const standardTypes = new Set(Object.keys(EVENT_SCHEMAS));
|
|
10
24
|
const directives = ['EVENT_TOMBSTONED', 'WORKSPACE_RESTORED', 'EVENTS_EXCISED'];
|
|
25
|
+
const SELECTORS: readonly CheckpointSelector[] = [
|
|
26
|
+
event => typeof event.payload.status === 'string' ? `status:${event.payload.status}` : null,
|
|
27
|
+
event => typeof event.payload.thumbnailRef === 'string' ? 'thumbnail:string' : null,
|
|
28
|
+
];
|
|
11
29
|
type Row = Record<string, SqlStorageValue>;
|
|
30
|
+
type Phase = 'seed' | 'fold' | 'pages';
|
|
12
31
|
interface JobRow extends Row {
|
|
13
32
|
id: string; log_id: string; kind: MaintenanceJob['kind']; through_version: number;
|
|
14
33
|
status: MaintenanceJob['status']; cursor: number; entity_cursor: string; pages: string;
|
|
15
34
|
error: string | null; updated_at: string; day: string; writes: number; reads: number;
|
|
35
|
+
phase: Phase | null; version_cursor: number; attempts: number; retry_at: string | null;
|
|
36
|
+
resume_at: string | null; pause_reason: 'writes' | 'reads' | null; paused_status: MaintenanceJob['status'] | null;
|
|
37
|
+
markers: string | null;
|
|
16
38
|
}
|
|
17
39
|
interface LayoutRow extends Row { log_id: string; compacted_through: number; generation: number; checkpoint: string | null; }
|
|
18
|
-
|
|
40
|
+
/** Replay slots for one entity plus the claim it held at the last source event. */
|
|
41
|
+
interface WorkState { s: CheckpointEntityState; c: { currentVersion: number; logVersion: number }; }
|
|
42
|
+
type Marker = Pick<CheckpointEvent, 'id' | 'type' | 'version' | 'payload'>;
|
|
43
|
+
type ReadEvents = (logId: string, after: number, through: number, limit: number) => Promise<CheckpointEvent[]>;
|
|
44
|
+
type ReadVersions = (logId: string, versions: number[]) => Promise<CheckpointEvent[]>;
|
|
19
45
|
|
|
20
46
|
/** Durable job state lives in SQLite; immutable payloads live in the private bucket. */
|
|
21
47
|
export class LogMaintenanceStore {
|
|
@@ -28,8 +54,10 @@ export class LogMaintenanceStore {
|
|
|
28
54
|
return cursor;
|
|
29
55
|
}
|
|
30
56
|
|
|
31
|
-
|
|
32
|
-
|
|
57
|
+
/** Run a statement to completion so its row counts are final, and meter it. */
|
|
58
|
+
private query<T extends Record<string, SqlStorageValue> = Record<string, SqlStorageValue>>(statement: string, ...bindings: SqlStorageValue[]): T[] {
|
|
59
|
+
const cursor = this.observeSql(this.storage.sql.exec<T>(statement, ...bindings));
|
|
60
|
+
return cursor.toArray();
|
|
33
61
|
}
|
|
34
62
|
|
|
35
63
|
private async getObject(key: string): Promise<R2ObjectBody | null> {
|
|
@@ -45,12 +73,16 @@ export class LogMaintenanceStore {
|
|
|
45
73
|
}
|
|
46
74
|
|
|
47
75
|
constructor(private readonly storage: DurableObjectStorage, private readonly bucket: R2Bucket | undefined,
|
|
48
|
-
private readonly appId: string, private readonly readEvents: ReadEvents, private readonly
|
|
76
|
+
private readonly appId: string, private readonly readEvents: ReadEvents, private readonly readVersions: ReadVersions,
|
|
77
|
+
private readonly assertWritable: () => void = () => undefined,
|
|
49
78
|
private readonly dayWriteBudget = DEFAULT_DAY_WRITE_BUDGET, private readonly dayReadBudget = 1_000_000) {}
|
|
50
79
|
|
|
51
80
|
state(logId: string, jobId?: string): LogMaintenance {
|
|
52
81
|
const layout = this.layout(logId);
|
|
53
|
-
|
|
82
|
+
// Polled often: leave out the large work columns (pages, markers).
|
|
83
|
+
const job = this.query<JobRow>(`SELECT id, log_id, kind, through_version, status, cursor, error, updated_at, attempts, retry_at,
|
|
84
|
+
resume_at, pause_reason FROM log_maintenance_jobs WHERE log_id = ? ${jobId ? 'AND id = ?' : ''} ORDER BY rowid DESC LIMIT 1`,
|
|
85
|
+
logId, ...(jobId ? [jobId] : []))[0];
|
|
54
86
|
return { compactedThroughVersion: layout.compacted_through, storageGeneration: layout.generation,
|
|
55
87
|
checkpoint: layout.checkpoint ? JSON.parse(layout.checkpoint) : null,
|
|
56
88
|
job: job ? publicJob(job) : null, available: Boolean(this.bucket) };
|
|
@@ -58,9 +90,9 @@ export class LogMaintenanceStore {
|
|
|
58
90
|
|
|
59
91
|
preview(logId: string): CompactionPreview {
|
|
60
92
|
const state = this.state(logId);
|
|
61
|
-
const totals = this.
|
|
93
|
+
const totals = this.query<{ count: number; bytes: number }>(`SELECT COUNT(*) AS count,
|
|
62
94
|
COALESCE(SUM(length(CAST(payload AS BLOB))), 0) AS bytes FROM log_events
|
|
63
|
-
WHERE log_id = ? AND version <= ? AND archive_key IS NULL`, logId, state.checkpoint?.throughVersion ?? 0)
|
|
95
|
+
WHERE log_id = ? AND version <= ? AND archive_key IS NULL`, logId, state.checkpoint?.throughVersion ?? 0)[0]!;
|
|
64
96
|
return { checkpointId: state.checkpoint?.id ?? null, storageGeneration: state.storageGeneration,
|
|
65
97
|
throughVersion: state.checkpoint?.throughVersion ?? 0, activePayloadEvents: totals.count, activePayloadBytes: totals.bytes,
|
|
66
98
|
snapshotBytes: state.checkpoint?.pages.reduce((bytes, page) => bytes + page.bytes, 0) ?? 0 };
|
|
@@ -70,16 +102,16 @@ export class LogMaintenanceStore {
|
|
|
70
102
|
if (!this.bucket) throw new LedgerError(503, { error: 'Unavailable', code: 'archive_unconfigured', message: 'Configure private Ledger archive storage first.' });
|
|
71
103
|
this.storage.transactionSync(() => {
|
|
72
104
|
this.assertWritable();
|
|
73
|
-
const prior = this.
|
|
105
|
+
const prior = this.query<JobRow>('SELECT id, log_id, kind FROM log_maintenance_jobs WHERE id = ?', id)[0];
|
|
74
106
|
if (prior) {
|
|
75
107
|
if (prior.log_id !== logId || prior.kind !== kind) throw validationError('Maintenance request ID already used');
|
|
76
108
|
return;
|
|
77
109
|
}
|
|
78
|
-
const active = this.
|
|
110
|
+
const active = this.query<JobRow>("SELECT id FROM log_maintenance_jobs WHERE log_id = ? AND status NOT IN ('complete', 'failed') LIMIT 1", logId)[0];
|
|
79
111
|
if (active) throw new LedgerError(409, { error: 'Conflict', code: 'maintenance_in_progress', message: 'Resume the existing maintenance operation.' });
|
|
80
|
-
const log = this.
|
|
112
|
+
const log = this.query<{ version: number; status: string }>('SELECT version, status FROM logs WHERE id = ?', logId)[0];
|
|
81
113
|
if (!log || log.status !== 'ready') throw validationError('Choose a ready log');
|
|
82
|
-
const boundary = this.
|
|
114
|
+
const boundary = this.query<{ action_index: number | null; action_size: number | null }>('SELECT action_index, action_size FROM log_events WHERE log_id = ? AND version = ?', logId, log.version)[0];
|
|
83
115
|
if (boundary?.action_size && boundary.action_index !== boundary.action_size - 1) throw validationError('Incomplete action boundary');
|
|
84
116
|
const checkpoint = this.state(logId).checkpoint;
|
|
85
117
|
if (kind === 'compact' && (expectedCheckpointId !== checkpoint?.id || expectedGeneration !== this.layout(logId).generation)) {
|
|
@@ -88,9 +120,11 @@ export class LogMaintenanceStore {
|
|
|
88
120
|
if (kind === 'compact' && (!checkpoint || checkpoint.throughVersion <= this.layout(logId).compacted_through)) {
|
|
89
121
|
throw new LedgerError(409, { error: 'Conflict', code: 'snapshot_required', message: 'Create a newer verified snapshot before compacting.' });
|
|
90
122
|
}
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
123
|
+
const start = kind === 'snapshot' ? this.initialBuild(logId, log.version) : { phase: null, cursor: 0 };
|
|
124
|
+
this.query(`INSERT INTO log_maintenance_jobs (id, log_id, kind, through_version, status, cursor, entity_cursor, pages, updated_at, day, writes, phase)
|
|
125
|
+
VALUES (?, ?, ?, ?, ?, ?, '', '[]', ?, ?, 0, ?)`, id, logId, kind,
|
|
126
|
+
kind === 'compact' ? checkpoint!.throughVersion : log.version, kind === 'compact' ? 'archiving' : 'building', start.cursor,
|
|
127
|
+
new Date().toISOString(), day(), start.phase);
|
|
94
128
|
});
|
|
95
129
|
return this.state(logId, id);
|
|
96
130
|
}
|
|
@@ -101,10 +135,10 @@ export class LogMaintenanceStore {
|
|
|
101
135
|
const started = Date.now();
|
|
102
136
|
let result: LogMaintenance | undefined;
|
|
103
137
|
try {
|
|
104
|
-
const before = this.
|
|
138
|
+
const before = this.query<JobRow>('SELECT day, reads, writes FROM log_maintenance_jobs WHERE id = ? AND log_id = ?', id, logId)[0];
|
|
105
139
|
result = await this.stepValue(logId, id);
|
|
106
140
|
this.assertWritable();
|
|
107
|
-
this.
|
|
141
|
+
this.query('UPDATE log_maintenance_jobs SET reads = ?, writes = ? WHERE id = ? AND log_id = ?',
|
|
108
142
|
(before?.day === day() ? before.reads : 0) + this.cursors.reduce((count, cursor) => count + cursor.rowsRead, 0),
|
|
109
143
|
(before?.day === day() ? before.writes : 0) + this.cursors.reduce((count, cursor) => count + cursor.rowsWritten, 0) + 1,
|
|
110
144
|
id, logId);
|
|
@@ -118,34 +152,81 @@ export class LogMaintenanceStore {
|
|
|
118
152
|
}
|
|
119
153
|
}
|
|
120
154
|
|
|
155
|
+
/** Active jobs whose next step is due now, least recently advanced first. */
|
|
156
|
+
dueJobs(now = Date.now()): Array<{ logId: string; id: string }> {
|
|
157
|
+
return this.activeJobs().filter(job => job.wake <= now).sort((a, b) => a.updatedAt.localeCompare(b.updatedAt))
|
|
158
|
+
.map(job => ({ logId: job.logId, id: job.id }));
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
/** Earliest time any active job needs another step, or null when none does. */
|
|
162
|
+
nextWake(): number | null {
|
|
163
|
+
const wakes = this.activeJobs().map(job => job.wake);
|
|
164
|
+
return wakes.length ? Math.min(...wakes) : null;
|
|
165
|
+
}
|
|
166
|
+
|
|
167
|
+
hasActiveJob(): boolean {
|
|
168
|
+
return this.query("SELECT 1 FROM log_maintenance_jobs WHERE status NOT IN ('complete', 'failed') LIMIT 1").length > 0;
|
|
169
|
+
}
|
|
170
|
+
|
|
171
|
+
private activeJobs(): Array<{ id: string; logId: string; wake: number; updatedAt: string }> {
|
|
172
|
+
const now = Date.now();
|
|
173
|
+
return this.query<JobRow>(`SELECT id, log_id, status, day, updated_at, retry_at, resume_at FROM log_maintenance_jobs
|
|
174
|
+
WHERE status NOT IN ('complete', 'failed')`).map(job => {
|
|
175
|
+
let wake = now;
|
|
176
|
+
if (job.status === 'paused') wake = job.resume_at ? Date.parse(job.resume_at) : nextMidnight(job.day);
|
|
177
|
+
else if (job.retry_at) wake = Date.parse(job.retry_at);
|
|
178
|
+
return { id: job.id, logId: job.log_id, wake: Number.isFinite(wake) ? wake : now, updatedAt: job.updated_at };
|
|
179
|
+
});
|
|
180
|
+
}
|
|
181
|
+
|
|
121
182
|
private async stepValue(logId: string, id: string): Promise<LogMaintenance> {
|
|
122
183
|
if (!this.bucket) throw new Error('Archive storage unavailable');
|
|
123
184
|
this.assertWritable();
|
|
124
|
-
let job = this.
|
|
185
|
+
let job = this.query<JobRow>('SELECT * FROM log_maintenance_jobs WHERE id = ? AND log_id = ?', id, logId)[0];
|
|
125
186
|
if (!job) throw validationError('Maintenance operation not found');
|
|
126
187
|
if (job.status === 'complete') return this.state(logId, id);
|
|
127
188
|
if (this.state(logId).job?.id !== id) throw new LedgerError(409, { error: 'Conflict', code: 'stale_maintenance', message: 'A newer maintenance operation replaced this one.' });
|
|
189
|
+
const resumeStatus = (row: JobRow): MaintenanceJob['status'] => row.paused_status && row.paused_status !== 'paused' && row.paused_status !== 'failed'
|
|
190
|
+
? row.paused_status : row.kind === 'snapshot' ? 'building' : 'archiving';
|
|
128
191
|
if (job.status === 'failed') {
|
|
129
|
-
|
|
130
|
-
|
|
192
|
+
// An explicit retry: start counting attempts again from the failed step.
|
|
193
|
+
this.query("UPDATE log_maintenance_jobs SET status = ?, error = NULL, attempts = 0, retry_at = NULL, paused_status = NULL WHERE id = ?", resumeStatus(job), id);
|
|
194
|
+
job = { ...job, status: resumeStatus(job), error: null, attempts: 0, retry_at: null, paused_status: null };
|
|
131
195
|
}
|
|
132
196
|
if (job.day !== day()) {
|
|
133
|
-
|
|
134
|
-
|
|
197
|
+
const status = job.status === 'paused' ? resumeStatus(job) : job.status;
|
|
198
|
+
this.query('UPDATE log_maintenance_jobs SET day = ?, writes = 0, reads = 0, status = ?, pause_reason = NULL, resume_at = NULL, paused_status = NULL WHERE id = ?', day(), status, id);
|
|
199
|
+
job = { ...job, day: day(), writes: 0, reads: 0, status, pause_reason: null, resume_at: null, paused_status: null };
|
|
135
200
|
}
|
|
136
|
-
if (job.writes >= this.dayWriteBudget || job.reads >= this.dayReadBudget) {
|
|
137
|
-
|
|
201
|
+
if (job.status === 'paused' || job.writes >= this.dayWriteBudget || job.reads >= this.dayReadBudget) {
|
|
202
|
+
if (job.status !== 'paused') {
|
|
203
|
+
const reason = job.writes >= this.dayWriteBudget ? 'writes' : 'reads';
|
|
204
|
+
this.query("UPDATE log_maintenance_jobs SET status = 'paused', paused_status = ?, pause_reason = ?, resume_at = ?, updated_at = ? WHERE id = ?",
|
|
205
|
+
job.status, reason, new Date(nextMidnight(job.day)).toISOString(), new Date().toISOString(), id);
|
|
206
|
+
}
|
|
138
207
|
return this.state(logId, id);
|
|
139
208
|
}
|
|
140
209
|
try {
|
|
141
210
|
if (job.status === 'building') await this.build(job);
|
|
142
211
|
else if (job.status === 'cleaning') await this.clean(job);
|
|
143
212
|
else if (job.status === 'archiving' || job.status === 'reclaiming') await this.archive(job);
|
|
213
|
+
if (job.attempts > 0 || job.error) {
|
|
214
|
+
this.query('UPDATE log_maintenance_jobs SET attempts = 0, error = NULL, retry_at = NULL WHERE id = ?', id);
|
|
215
|
+
}
|
|
144
216
|
} catch (cause) {
|
|
145
|
-
//
|
|
217
|
+
// Access problems are not job failures; source rows and cursors stay untouched.
|
|
146
218
|
this.assertWritable();
|
|
147
|
-
|
|
148
|
-
|
|
219
|
+
const attempts = job.attempts + 1;
|
|
220
|
+
const message = cause instanceof Error ? cause.message : 'Maintenance failed';
|
|
221
|
+
const now = new Date().toISOString();
|
|
222
|
+
if (attempts >= MAX_ATTEMPTS) {
|
|
223
|
+
this.query("UPDATE log_maintenance_jobs SET status = 'failed', paused_status = ?, error = ?, attempts = ?, retry_at = NULL, updated_at = ? WHERE id = ?",
|
|
224
|
+
job.status, message, attempts, now, id);
|
|
225
|
+
} else {
|
|
226
|
+
// Stay in the current stage and let the alarm retry with backoff; the error stays visible meanwhile.
|
|
227
|
+
this.query('UPDATE log_maintenance_jobs SET error = ?, attempts = ?, retry_at = ?, updated_at = ? WHERE id = ?',
|
|
228
|
+
message, attempts, new Date(Date.now() + Math.min(600_000, 15_000 * 2 ** (attempts - 1))).toISOString(), now, id);
|
|
229
|
+
}
|
|
149
230
|
}
|
|
150
231
|
return this.state(logId, id);
|
|
151
232
|
}
|
|
@@ -164,16 +245,16 @@ export class LogMaintenanceStore {
|
|
|
164
245
|
/** Bounded archive pages are verified before any accepted payload is reclaimed. */
|
|
165
246
|
async hydrate(logId: string, events: CheckpointEvent[]): Promise<CheckpointEvent[]> {
|
|
166
247
|
if (!events.length) return events;
|
|
167
|
-
const references = this.
|
|
248
|
+
const references = this.query<{ event_id: string; archive_key: string }>(
|
|
168
249
|
'SELECT event_id, archive_key FROM log_events WHERE log_id = ? AND event_id IN (SELECT value FROM json_each(?)) AND archive_key IS NOT NULL',
|
|
169
|
-
logId, JSON.stringify(events.map(event => event.id)))
|
|
250
|
+
logId, JSON.stringify(events.map(event => event.id)));
|
|
170
251
|
if (!references.length) return events;
|
|
171
252
|
if (!this.bucket) throw archiveError('Archive storage unavailable');
|
|
172
253
|
const wanted = new Set(events.map(event => event.id));
|
|
173
254
|
const payloads = new Map<string, CheckpointEvent>();
|
|
174
255
|
for (const key of new Set(references.map(row => row.archive_key))) {
|
|
175
256
|
if (!key.startsWith(`v1/${this.appId}/`)) throw archiveError('Archive scope mismatch');
|
|
176
|
-
const reference = this.
|
|
257
|
+
const reference = this.query<{ sha256: string }>('SELECT sha256 FROM log_archive_pages WHERE object_key = ? LIMIT 1', key)[0];
|
|
177
258
|
if (!reference) throw archiveError('Archive manifest is missing');
|
|
178
259
|
const object = await this.getObject(key);
|
|
179
260
|
if (!object) throw archiveError('Archived history is missing');
|
|
@@ -192,154 +273,266 @@ export class LogMaintenanceStore {
|
|
|
192
273
|
}
|
|
193
274
|
|
|
194
275
|
private layout(logId: string): LayoutRow {
|
|
195
|
-
return this.
|
|
276
|
+
return this.query<LayoutRow>('SELECT * FROM log_storage WHERE log_id = ?', logId)[0]
|
|
196
277
|
?? { log_id: logId, compacted_through: 0, generation: 0, checkpoint: null };
|
|
197
278
|
}
|
|
198
279
|
|
|
199
280
|
private async build(job: JobRow): Promise<void> {
|
|
200
|
-
if (job.
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
}
|
|
212
|
-
for (const [entityId, source] of groups) {
|
|
213
|
-
const prior = this.exec<{ object_key: string }>('SELECT object_key FROM log_checkpoint_entities WHERE job_id = ? AND entity_id = ?', job.id, entityId).toArray()[0];
|
|
214
|
-
const previous = prior ? await this.readWork(prior.object_key) : { events: [], claims: [] };
|
|
215
|
-
const last = source[source.length - 1]!;
|
|
216
|
-
const witnesses = checkpointEntityEvents([...previous.events, ...source.filter(event => selected(event) && !directives.includes(event.type))], [
|
|
217
|
-
event => typeof event.payload.status === 'string' ? `status:${event.payload.status}` : null,
|
|
218
|
-
event => typeof event.payload.thumbnailRef === 'string' ? 'thumbnail:string' : null,
|
|
219
|
-
], standardTypes);
|
|
220
|
-
const value = JSON.stringify({ events: witnesses, claims: [{ entityId, currentVersion: last.entityVersion, logVersion: last.version }] });
|
|
221
|
-
if (new TextEncoder().encode(value).byteLength > 8_000_000) throw new Error('Entity replay state exceeds the snapshot work budget');
|
|
222
|
-
const key = `v1/${this.appId}/${job.log_id}/work/${job.id}/${last.version}.json`;
|
|
223
|
-
await this.putVerified(key, value);
|
|
224
|
-
this.storage.transactionSync(() => {
|
|
225
|
-
this.assertWritable();
|
|
226
|
-
this.exec(`INSERT INTO log_checkpoint_entities (job_id, entity_id, object_key) VALUES (?, ?, ?)
|
|
227
|
-
ON CONFLICT(job_id, entity_id) DO UPDATE SET object_key = excluded.object_key`, job.id, entityId, key);
|
|
228
|
-
this.exec('UPDATE log_maintenance_jobs SET writes = writes + 2 WHERE id = ?', job.id);
|
|
229
|
-
});
|
|
230
|
-
if (prior && prior.object_key !== key) await this.deleteObject(prior.object_key);
|
|
231
|
-
}
|
|
232
|
-
this.assertWritable();
|
|
233
|
-
this.exec('UPDATE log_maintenance_jobs SET cursor = ?, writes = writes + 1, updated_at = ? WHERE id = ?',
|
|
234
|
-
events[events.length - 1]!.version, new Date().toISOString(), job.id);
|
|
235
|
-
return;
|
|
281
|
+
if (!job.phase || !['seed', 'fold', 'pages'].includes(job.phase)) return this.reset(job);
|
|
282
|
+
if (job.phase === 'pages') return this.pages(job);
|
|
283
|
+
return this.fold(job);
|
|
284
|
+
}
|
|
285
|
+
|
|
286
|
+
/** Where a new snapshot starts: from the previous one's retained facts when no later replay directive can reach behind it. */
|
|
287
|
+
private initialBuild(logId: string, through: number): { phase: Phase; cursor: number } {
|
|
288
|
+
const checkpoint = this.state(logId).checkpoint;
|
|
289
|
+
if (checkpoint && checkpoint.throughVersion > 0 && checkpoint.throughVersion <= through
|
|
290
|
+
&& !this.query(`SELECT 1 FROM log_events WHERE log_id = ? AND version > ? AND event_type IN ('EVENT_TOMBSTONED', 'WORKSPACE_RESTORED', 'EVENTS_EXCISED') LIMIT 1`,
|
|
291
|
+
logId, checkpoint.throughVersion).length) {
|
|
292
|
+
return { phase: 'seed', cursor: checkpoint.throughVersion };
|
|
236
293
|
}
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
294
|
+
return { phase: 'fold', cursor: 0 };
|
|
295
|
+
}
|
|
296
|
+
|
|
297
|
+
/** Discard partial work (including work files of earlier builders) and start the build again. */
|
|
298
|
+
private async reset(job: JobRow): Promise<void> {
|
|
299
|
+
const legacy = this.query<{ entity_id: string; object_key: string }>('SELECT entity_id, object_key FROM log_checkpoint_entities WHERE job_id = ? LIMIT 16', job.id);
|
|
300
|
+
if (legacy.length) {
|
|
301
|
+
for (const row of legacy) {
|
|
302
|
+
if (row.object_key) await this.deleteObject(row.object_key);
|
|
246
303
|
this.assertWritable();
|
|
247
|
-
this.
|
|
248
|
-
|
|
249
|
-
this.exec("UPDATE log_maintenance_jobs SET status = 'cleaning', updated_at = ? WHERE id = ?", new Date().toISOString(), job.id);
|
|
250
|
-
});
|
|
304
|
+
this.query('DELETE FROM log_checkpoint_entities WHERE job_id = ? AND entity_id = ?', job.id, row.entity_id);
|
|
305
|
+
}
|
|
251
306
|
return;
|
|
252
307
|
}
|
|
253
|
-
const
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
308
|
+
const reference = workReference(job.entity_cursor);
|
|
309
|
+
if (reference) await this.deleteObject(reference.key);
|
|
310
|
+
const start = this.initialBuild(job.log_id, job.through_version);
|
|
311
|
+
this.assertWritable();
|
|
312
|
+
this.query(`UPDATE log_maintenance_jobs SET phase = ?, cursor = ?, version_cursor = 0, entity_cursor = '', pages = '[]', markers = NULL, updated_at = ?
|
|
313
|
+
WHERE id = ?`, start.phase, start.cursor, new Date().toISOString(), job.id);
|
|
314
|
+
}
|
|
315
|
+
|
|
316
|
+
/**
|
|
317
|
+
* The whole job's replay slots live in one private object, read and written once per step,
|
|
318
|
+
* so SQL writes grow with steps rather than with entities. null when it was lost (the build restarts).
|
|
319
|
+
*/
|
|
320
|
+
private async loadState(job: JobRow): Promise<Map<string, WorkState> | null> {
|
|
321
|
+
const reference = workReference(job.entity_cursor);
|
|
322
|
+
if (!reference) return new Map();
|
|
323
|
+
const object = await this.getObject(reference.key);
|
|
324
|
+
const value = object ? await object.text() : null;
|
|
325
|
+
if (value === null || await checkpointDigest(value) !== reference.sha256) {
|
|
326
|
+
console.warn('ledger_maintenance_state_lost', { appId: this.appId, logId: job.log_id, jobId: job.id });
|
|
327
|
+
this.assertWritable();
|
|
328
|
+
this.query("UPDATE log_maintenance_jobs SET entity_cursor = '', phase = NULL WHERE id = ?", job.id);
|
|
329
|
+
return null;
|
|
330
|
+
}
|
|
331
|
+
return new Map(Object.entries(JSON.parse(value) as Record<string, WorkState>));
|
|
332
|
+
}
|
|
333
|
+
|
|
334
|
+
private async saveState(job: JobRow, states: Map<string, WorkState>, next: { phase: Phase; cursor: number; versionCursor: number }): Promise<void> {
|
|
335
|
+
let value = '{';
|
|
336
|
+
let first = true;
|
|
337
|
+
for (const [entityId, state] of states) {
|
|
338
|
+
const encoded = JSON.stringify(state);
|
|
339
|
+
if (encoded.length > MAX_STATE_BYTES) throw new Error('Entity replay state exceeds the snapshot work budget');
|
|
340
|
+
value += `${first ? '' : ','}${JSON.stringify(entityId)}:${encoded}`;
|
|
341
|
+
first = false;
|
|
260
342
|
}
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
const key = `v1/${this.appId}/${job.log_id}/
|
|
264
|
-
const sha256 = await this.
|
|
265
|
-
pages.push({ key, sha256, count: chunk.events.length, bytes: new TextEncoder().encode(value).byteLength });
|
|
343
|
+
value += '}';
|
|
344
|
+
if (value.length > MAX_WORK_BYTES) throw new Error('Snapshot work state exceeds the snapshot work budget');
|
|
345
|
+
const key = `v1/${this.appId}/${job.log_id}/work/${job.id}/${next.phase}-${next.cursor}-${next.versionCursor}.json`;
|
|
346
|
+
const sha256 = await this.put(key, value);
|
|
266
347
|
this.assertWritable();
|
|
267
|
-
this.
|
|
268
|
-
|
|
348
|
+
this.query('UPDATE log_maintenance_jobs SET phase = ?, cursor = ?, version_cursor = ?, entity_cursor = ?, updated_at = ? WHERE id = ?',
|
|
349
|
+
next.phase, next.cursor, next.versionCursor, `${key}#${sha256}`, new Date().toISOString(), job.id);
|
|
350
|
+
const previous = workReference(job.entity_cursor);
|
|
351
|
+
if (previous && previous.key !== key) await this.deleteObject(previous.key);
|
|
269
352
|
}
|
|
270
353
|
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
354
|
+
/** Seed from the previous snapshot, then fold source events, for as long as one step allows. */
|
|
355
|
+
private async fold(job: JobRow): Promise<void> {
|
|
356
|
+
const states = await this.loadState(job);
|
|
357
|
+
if (!states) return;
|
|
358
|
+
const started = Date.now();
|
|
359
|
+
const work = (entityId: string): WorkState => {
|
|
360
|
+
let state = states.get(entityId);
|
|
361
|
+
if (!state) { state = { s: newCheckpointEntityState(), c: { currentVersion: 0, logVersion: 0 } }; states.set(entityId, state); }
|
|
362
|
+
return state;
|
|
363
|
+
};
|
|
364
|
+
let phase = job.phase as Phase, cursor = job.cursor, versionCursor = job.version_cursor;
|
|
365
|
+
if (phase === 'seed') {
|
|
366
|
+
const layout = this.layout(job.log_id);
|
|
367
|
+
const checkpoint = layout.checkpoint ? JSON.parse(layout.checkpoint) as CheckpointManifest : null;
|
|
368
|
+
if (!checkpoint || checkpoint.throughVersion !== job.cursor) throw new Error('The previous snapshot changed while seeding');
|
|
369
|
+
while (versionCursor < checkpoint.pages.length && Date.now() - started < STEP_MS) {
|
|
370
|
+
const chunk = await this.chunk(job.log_id, versionCursor);
|
|
371
|
+
for (const event of [...chunk.events].sort((a, b) => a.version - b.version)) foldCheckpointEvent(work(event.entityId).s, event, SELECTORS, standardTypes);
|
|
372
|
+
for (const claim of chunk.claims) work(claim.entityId).c = { currentVersion: claim.currentVersion, logVersion: claim.logVersion };
|
|
373
|
+
versionCursor++;
|
|
374
|
+
}
|
|
375
|
+
if (versionCursor >= checkpoint.pages.length) { phase = 'fold'; versionCursor = 0; }
|
|
376
|
+
}
|
|
377
|
+
if (phase === 'fold') {
|
|
378
|
+
const selected = cursor < job.through_version ? this.selection(job) : () => true;
|
|
379
|
+
while (cursor < job.through_version && Date.now() - started < STEP_MS) {
|
|
380
|
+
const events = await this.readEvents(job.log_id, cursor, job.through_version, FOLD_EVENTS);
|
|
381
|
+
if (!events.length || events[0]!.version !== cursor + 1 || events[events.length - 1]!.version - events[0]!.version !== events.length - 1) {
|
|
382
|
+
throw new Error('Snapshot source contains a gap');
|
|
383
|
+
}
|
|
384
|
+
for (const event of events) {
|
|
385
|
+
const state = work(event.entityId);
|
|
386
|
+
if (selected(event) && !directives.includes(event.type)) foldCheckpointEvent(state.s, event, SELECTORS, standardTypes);
|
|
387
|
+
state.c = { currentVersion: event.entityVersion, logVersion: event.version };
|
|
388
|
+
}
|
|
389
|
+
cursor = events[events.length - 1]!.version;
|
|
390
|
+
}
|
|
391
|
+
if (cursor >= job.through_version) { phase = 'pages'; versionCursor = 0; }
|
|
392
|
+
}
|
|
393
|
+
await this.saveState(job, states, { phase, cursor, versionCursor });
|
|
277
394
|
}
|
|
278
395
|
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
396
|
+
/** Write claim pages and the retained source events in version order, then publish the manifest. */
|
|
397
|
+
private async pages(job: JobRow): Promise<void> {
|
|
398
|
+
const states = await this.loadState(job);
|
|
399
|
+
if (!states) return;
|
|
400
|
+
const started = Date.now();
|
|
401
|
+
const pages = JSON.parse(job.pages) as CheckpointPage[];
|
|
402
|
+
const pageKey = () => `v1/${this.appId}/${job.log_id}/snapshots/${job.id}/${pages.length}.json`;
|
|
403
|
+
const write = async (chunk: CheckpointChunk) => {
|
|
404
|
+
const value = JSON.stringify(chunk);
|
|
405
|
+
const key = pageKey();
|
|
406
|
+
pages.push({ key, sha256: await this.put(key, value), count: chunk.events.length, bytes: new TextEncoder().encode(value).byteLength });
|
|
407
|
+
};
|
|
408
|
+
if (!pages.length && job.version_cursor === 0) {
|
|
409
|
+
const claims = [...states].filter(([, state]) => state.c.logVersion > 0).sort(([a], [b]) => a < b ? -1 : a > b ? 1 : 0)
|
|
410
|
+
.map(([entityId, state]) => ({ entityId, currentVersion: state.c.currentVersion, logVersion: state.c.logVersion }));
|
|
411
|
+
for (let offset = 0; offset < claims.length; offset += CLAIMS_PER_PAGE) await write({ events: [], claims: claims.slice(offset, offset + CLAIMS_PER_PAGE) });
|
|
412
|
+
}
|
|
413
|
+
const retained = new Set<number>();
|
|
414
|
+
for (const state of states.values()) for (const version of checkpointStateVersions(state.s)) retained.add(version);
|
|
415
|
+
const versions = [...retained].filter(version => version > job.version_cursor).sort((a, b) => a - b);
|
|
416
|
+
let versionCursor = job.version_cursor;
|
|
417
|
+
let chunk: CheckpointChunk = { events: [], claims: [] };
|
|
418
|
+
let bytes = 2;
|
|
419
|
+
let offset = 0;
|
|
420
|
+
while (offset < versions.length && Date.now() - started < STEP_MS) {
|
|
421
|
+
const wanted = versions.slice(offset, offset + READ_VERSIONS);
|
|
422
|
+
const events = await this.readVersions(job.log_id, wanted);
|
|
423
|
+
if (events.length !== wanted.length || events.some((event, index) => event.version !== wanted[index])) throw new Error('Snapshot source is missing retained events');
|
|
424
|
+
for (const event of events) {
|
|
425
|
+
const size = new TextEncoder().encode(JSON.stringify(event)).byteLength + 1;
|
|
426
|
+
if (chunk.events.length && bytes + size >= MAX_PAGE_BYTES) {
|
|
427
|
+
await write(chunk);
|
|
428
|
+
versionCursor = chunk.events[chunk.events.length - 1]!.version;
|
|
429
|
+
chunk = { events: [], claims: [] }; bytes = 2;
|
|
430
|
+
}
|
|
431
|
+
chunk.events.push(event); bytes += size;
|
|
432
|
+
}
|
|
433
|
+
offset += wanted.length;
|
|
434
|
+
}
|
|
435
|
+
if (chunk.events.length) {
|
|
436
|
+
await write(chunk);
|
|
437
|
+
versionCursor = chunk.events[chunk.events.length - 1]!.version;
|
|
438
|
+
}
|
|
439
|
+
this.assertWritable();
|
|
440
|
+
if (offset < versions.length) {
|
|
441
|
+
this.query('UPDATE log_maintenance_jobs SET version_cursor = ?, pages = ?, updated_at = ? WHERE id = ?',
|
|
442
|
+
versionCursor, JSON.stringify(pages), new Date().toISOString(), job.id);
|
|
443
|
+
return;
|
|
285
444
|
}
|
|
286
|
-
|
|
287
|
-
|
|
445
|
+
const manifest: CheckpointManifest = { format: 1, appId: this.appId, logId: job.log_id, id: job.id,
|
|
446
|
+
throughVersion: job.through_version, createdAt: new Date().toISOString(), pages,
|
|
447
|
+
eventCount: pages.reduce((count, page) => count + page.count, 0), sourceEventCount: job.through_version,
|
|
448
|
+
sha256: await checkpointDigest(JSON.stringify(pages)) };
|
|
449
|
+
this.storage.transactionSync(() => {
|
|
450
|
+
this.assertWritable();
|
|
451
|
+
this.query(`INSERT INTO log_storage (log_id, compacted_through, generation, checkpoint) VALUES (?, 0, 0, ?)
|
|
452
|
+
ON CONFLICT(log_id) DO UPDATE SET checkpoint = excluded.checkpoint`, job.log_id, JSON.stringify(manifest));
|
|
453
|
+
this.query("UPDATE log_maintenance_jobs SET status = 'cleaning', version_cursor = ?, pages = ?, updated_at = ? WHERE id = ?",
|
|
454
|
+
versionCursor, JSON.stringify(pages), new Date().toISOString(), job.id);
|
|
455
|
+
});
|
|
456
|
+
}
|
|
457
|
+
|
|
458
|
+
private async clean(job: JobRow): Promise<void> {
|
|
459
|
+
const reference = workReference(job.entity_cursor);
|
|
460
|
+
if (reference) await this.deleteObject(reference.key);
|
|
461
|
+
this.assertWritable();
|
|
462
|
+
this.query("UPDATE log_maintenance_jobs SET status = 'complete', entity_cursor = '', updated_at = ? WHERE id = ?", new Date().toISOString(), job.id);
|
|
288
463
|
}
|
|
289
464
|
|
|
290
|
-
|
|
465
|
+
/** R2 rejects an upload whose bytes do not match the digest, so no read-back is needed. */
|
|
466
|
+
private async put(key: string, value: string): Promise<string> {
|
|
291
467
|
const sha256 = await checkpointDigest(value);
|
|
292
468
|
this.io.r2Writes++;
|
|
293
469
|
this.io.r2WrittenBytes += new TextEncoder().encode(value).byteLength;
|
|
294
|
-
await this.bucket!.put(key, value, { customMetadata: { sha256 }, httpMetadata: { contentType: 'application/json' } });
|
|
295
|
-
const check = await this.getObject(key);
|
|
296
|
-
if (!check || await checkpointDigest(await check.text()) !== sha256) throw new Error('Snapshot verification failed');
|
|
470
|
+
await this.bucket!.put(key, value, { sha256, customMetadata: { sha256 }, httpMetadata: { contentType: 'application/json' } });
|
|
297
471
|
return sha256;
|
|
298
472
|
}
|
|
299
473
|
|
|
300
474
|
private async archive(job: JobRow): Promise<void> {
|
|
301
475
|
// Metadata stays as the compact identity/action registry. Only payloads
|
|
302
476
|
// move, so historical boundaries and event/request uniqueness survive.
|
|
303
|
-
const rows = this.
|
|
304
|
-
AND version <= ? AND version > ? AND archive_key IS NULL ORDER BY version LIMIT ?`, job.log_id, job.through_version, job.cursor, PAGE)
|
|
477
|
+
const rows = this.query<{ version: number }>(`SELECT version FROM log_events WHERE log_id = ?
|
|
478
|
+
AND version <= ? AND version > ? AND archive_key IS NULL ORDER BY version LIMIT ?`, job.log_id, job.through_version, job.cursor, PAGE);
|
|
305
479
|
if (!rows.length) {
|
|
306
480
|
this.storage.transactionSync(() => {
|
|
307
|
-
|
|
308
|
-
this.
|
|
309
|
-
this.
|
|
481
|
+
this.assertWritable();
|
|
482
|
+
this.query('UPDATE log_storage SET compacted_through = MAX(compacted_through, ?), generation = generation + 1 WHERE log_id = ?', job.through_version, job.log_id);
|
|
483
|
+
this.query("UPDATE log_maintenance_jobs SET status = 'complete', cursor = ?, updated_at = ? WHERE id = ?", job.through_version, new Date().toISOString(), job.id);
|
|
310
484
|
});
|
|
311
485
|
return;
|
|
312
486
|
}
|
|
313
487
|
const first = rows[0]!.version;
|
|
314
488
|
const events = await this.readEvents(job.log_id, first - 1, rows[rows.length - 1]!.version, PAGE);
|
|
315
489
|
const value = JSON.stringify(events);
|
|
316
|
-
const sha256 = await checkpointDigest(value);
|
|
317
490
|
const key = `v1/${this.appId}/${job.log_id}/archive/${job.id}/${first}.json`;
|
|
318
|
-
await this.
|
|
491
|
+
const sha256 = await this.put(key, value);
|
|
319
492
|
this.storage.transactionSync(() => {
|
|
320
493
|
this.assertWritable();
|
|
321
|
-
this.
|
|
494
|
+
this.query(`INSERT OR IGNORE INTO log_archive_pages (log_id, first_version, last_version, object_key, sha256) VALUES (?, ?, ?, ?, ?)`,
|
|
322
495
|
job.log_id, first, events[events.length - 1]!.version, key, sha256);
|
|
323
|
-
|
|
496
|
+
this.query(`UPDATE log_events SET archive_key = ?, payload_bytes = COALESCE(payload_bytes, length(CAST(payload AS BLOB))), payload = CASE WHEN event_type IN
|
|
324
497
|
('EVENT_TOMBSTONED', 'WORKSPACE_RESTORED', 'EVENTS_EXCISED') THEN payload ELSE '{}' END
|
|
325
498
|
WHERE log_id = ? AND version >= ? AND version <= ? AND archive_key IS NULL`, key, job.log_id, first, events[events.length - 1]!.version);
|
|
326
|
-
|
|
327
|
-
this.
|
|
328
|
-
events[events.length - 1]!.version,
|
|
499
|
+
this.query('UPDATE log_storage SET archived_through = MAX(archived_through, ?) WHERE log_id = ?', events[events.length - 1]!.version, job.log_id);
|
|
500
|
+
this.query("UPDATE log_maintenance_jobs SET status = 'archiving', cursor = ?, updated_at = ? WHERE id = ?",
|
|
501
|
+
events[events.length - 1]!.version, new Date().toISOString(), job.id);
|
|
329
502
|
});
|
|
330
503
|
}
|
|
331
504
|
|
|
332
|
-
|
|
333
|
-
|
|
334
|
-
|
|
335
|
-
|
|
336
|
-
|
|
505
|
+
/** Replay directives up to the snapshot boundary, resolved once per job. */
|
|
506
|
+
private selection(job: JobRow): (event: CheckpointEvent) => boolean {
|
|
507
|
+
let markers: Marker[];
|
|
508
|
+
if (job.markers) {
|
|
509
|
+
markers = JSON.parse(job.markers) as Marker[];
|
|
510
|
+
} else {
|
|
511
|
+
markers = this.query<{ version: number; event_id: string; event_type: string; payload: string }>(`SELECT version, event_id, event_type, payload FROM log_events
|
|
512
|
+
WHERE log_id = ? AND version <= ? AND event_type IN ('EVENT_TOMBSTONED', 'WORKSPACE_RESTORED', 'EVENTS_EXCISED') ORDER BY version DESC`,
|
|
513
|
+
job.log_id, job.through_version).map(event => ({ id: event.event_id, type: event.event_type,
|
|
514
|
+
version: event.version, payload: JSON.parse(event.payload) as Record<string, unknown> }));
|
|
515
|
+
const encoded = JSON.stringify(markers);
|
|
516
|
+
if (encoded.length < MAX_MARKER_BYTES) {
|
|
517
|
+
this.assertWritable();
|
|
518
|
+
this.query('UPDATE log_maintenance_jobs SET markers = ? WHERE id = ?', encoded, job.id);
|
|
519
|
+
}
|
|
520
|
+
}
|
|
521
|
+
return checkpointSelection(markers, job.through_version);
|
|
337
522
|
}
|
|
338
523
|
}
|
|
339
524
|
|
|
525
|
+
/** The job's current work-state object, stored as `key#sha256` in the job row. */
|
|
526
|
+
function workReference(value: string): { key: string; sha256: string } | null {
|
|
527
|
+
const separator = value.lastIndexOf('#');
|
|
528
|
+
return separator > 0 ? { key: value.slice(0, separator), sha256: value.slice(separator + 1) } : null;
|
|
529
|
+
}
|
|
340
530
|
function day(): string { return new Date().toISOString().slice(0, 10); }
|
|
531
|
+
/** Start of the UTC day after `dayValue` (YYYY-MM-DD). */
|
|
532
|
+
function nextMidnight(dayValue: string): number { return Date.parse(`${dayValue}T00:00:00.000Z`) + 86_400_000; }
|
|
341
533
|
function archiveError(message: string): LedgerError { return new LedgerError(503, { error: 'Unavailable', code: 'archive_unavailable', message }); }
|
|
342
534
|
function publicJob(row: JobRow): MaintenanceJob {
|
|
343
535
|
return { id: row.id, logId: row.log_id, kind: row.kind, throughVersion: row.through_version, status: row.status,
|
|
344
|
-
cursor: row.cursor, error: row.error, updatedAt: row.updated_at
|
|
536
|
+
cursor: row.cursor, error: row.error, updatedAt: row.updated_at, attempts: row.attempts, retryAt: row.retry_at,
|
|
537
|
+
pauseReason: row.pause_reason, resumeAt: row.resume_at };
|
|
345
538
|
}
|
package/src/maintenance-steps.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
/** One in-flight step per operation, with a bounded wait for HTTP/RPC callers.
|
|
1
|
+
/** One in-flight step per operation (alarm and callers share it), with a bounded wait for HTTP/RPC callers.
|
|
2
2
|
* Losing a response must not enqueue another copy of the same running step.
|
|
3
3
|
*/
|
|
4
4
|
export class MaintenanceSteps<T> {
|
|
@@ -7,7 +7,8 @@ export class MaintenanceSteps<T> {
|
|
|
7
7
|
|
|
8
8
|
constructor(private readonly keepAlive: (work: Promise<unknown>) => void, private readonly responseWaitMs = 1000) {}
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
/** Queue one step behind earlier ones (or join the identical step already in flight) and resolve with its real result. */
|
|
11
|
+
run(key: string, work: () => Promise<T>): Promise<T> {
|
|
11
12
|
let run = this.running.get(key);
|
|
12
13
|
if (!run) {
|
|
13
14
|
run = this.queue.then(work);
|
|
@@ -18,6 +19,11 @@ export class MaintenanceSteps<T> {
|
|
|
18
19
|
this.queue = settled;
|
|
19
20
|
this.keepAlive(settled);
|
|
20
21
|
}
|
|
22
|
+
return run;
|
|
23
|
+
}
|
|
24
|
+
|
|
25
|
+
async advance(key: string, work: () => Promise<T>, current: () => T): Promise<T> {
|
|
26
|
+
const run = this.run(key, work);
|
|
21
27
|
let timer: ReturnType<typeof setTimeout> | undefined;
|
|
22
28
|
try {
|
|
23
29
|
return await Promise.race([run, new Promise<T>((resolve, reject) => {
|
package/src/schema.ts
CHANGED
|
@@ -215,6 +215,46 @@ const LEDGER_APP_MIGRATIONS: readonly Migration[] = [
|
|
|
215
215
|
);
|
|
216
216
|
CREATE INDEX log_archive_object ON log_archive_pages(object_key);
|
|
217
217
|
` },
|
|
218
|
+
// Snapshot work state now lives in SQLite as per-entity replay slots (no R2
|
|
219
|
+
// work files). Jobs are advanced by an alarm and record retry/pause details.
|
|
220
|
+
{ id: 15, sql: `
|
|
221
|
+
CREATE TABLE log_checkpoint_work (
|
|
222
|
+
job_id TEXT NOT NULL, entity_id TEXT NOT NULL, state TEXT NOT NULL,
|
|
223
|
+
PRIMARY KEY(job_id, entity_id)
|
|
224
|
+
);
|
|
225
|
+
CREATE TABLE log_checkpoint_versions (
|
|
226
|
+
job_id TEXT NOT NULL, version INTEGER NOT NULL, PRIMARY KEY(job_id, version)
|
|
227
|
+
);
|
|
228
|
+
ALTER TABLE log_maintenance_jobs ADD COLUMN phase TEXT;
|
|
229
|
+
ALTER TABLE log_maintenance_jobs ADD COLUMN version_cursor INTEGER NOT NULL DEFAULT 0;
|
|
230
|
+
ALTER TABLE log_maintenance_jobs ADD COLUMN attempts INTEGER NOT NULL DEFAULT 0;
|
|
231
|
+
ALTER TABLE log_maintenance_jobs ADD COLUMN retry_at TEXT;
|
|
232
|
+
ALTER TABLE log_maintenance_jobs ADD COLUMN resume_at TEXT;
|
|
233
|
+
ALTER TABLE log_maintenance_jobs ADD COLUMN pause_reason TEXT;
|
|
234
|
+
ALTER TABLE log_maintenance_jobs ADD COLUMN paused_status TEXT;
|
|
235
|
+
ALTER TABLE log_maintenance_jobs ADD COLUMN markers TEXT;
|
|
236
|
+
` },
|
|
237
|
+
// Snapshot work state moves to one R2 object per job (SQL writes per step, not per entity), and
|
|
238
|
+
// archiving no longer writes an index entry per event: the archived prefix is tracked per log.
|
|
239
|
+
{ id: 16, sql: `
|
|
240
|
+
ALTER TABLE log_storage ADD COLUMN archived_through INTEGER NOT NULL DEFAULT 0;
|
|
241
|
+
UPDATE log_storage SET archived_through = COALESCE((SELECT MAX(version) FROM log_events
|
|
242
|
+
WHERE log_events.log_id = log_storage.log_id AND archive_key IS NOT NULL), 0);
|
|
243
|
+
INSERT INTO log_storage (log_id, compacted_through, generation, checkpoint, archived_through)
|
|
244
|
+
SELECT log_id, 0, 0, NULL, MAX(version) FROM log_events
|
|
245
|
+
WHERE archive_key IS NOT NULL AND log_id NOT IN (SELECT log_id FROM log_storage) GROUP BY log_id;
|
|
246
|
+
DROP INDEX log_events_archived_cursor;
|
|
247
|
+
DROP TABLE log_checkpoint_work;
|
|
248
|
+
DROP TABLE log_checkpoint_versions;
|
|
249
|
+
-- Builds in progress restart with the new builder. Their counters measured the old one, so a
|
|
250
|
+
-- build that paused on them may continue today.
|
|
251
|
+
UPDATE log_maintenance_jobs SET phase = NULL, entity_cursor = '', version_cursor = 0, pages = '[]', markers = NULL,
|
|
252
|
+
writes = 0, reads = 0, attempts = 0, error = NULL, retry_at = NULL, status = 'building',
|
|
253
|
+
pause_reason = NULL, resume_at = NULL, paused_status = NULL
|
|
254
|
+
WHERE kind = 'snapshot' AND (status = 'building' OR (status = 'paused' AND COALESCE(paused_status, 'building') = 'building'));
|
|
255
|
+
UPDATE log_maintenance_jobs SET entity_cursor = '', status = 'cleaning', pause_reason = NULL, resume_at = NULL, paused_status = NULL
|
|
256
|
+
WHERE kind = 'snapshot' AND (status = 'cleaning' OR (status = 'paused' AND paused_status = 'cleaning'));
|
|
257
|
+
` },
|
|
218
258
|
];
|
|
219
259
|
|
|
220
260
|
const APP_REGISTRY_MIGRATIONS: readonly Migration[] = [
|