@rig-c/review 2.4.1 → 2.4.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/REVIEW_PACKAGE.md +3 -1
- package/docs/STABILITY.md +27 -0
- package/package.json +1 -1
- package/src/job-contract.ts +1 -0
- package/src/job-front.ts +88 -87
- package/src/job-session.ts +149 -20
- package/src/job-source.ts +41 -5
- package/src/job-store.ts +131 -3
- package/src/reference.ts +1 -1
- package/web/catalog-en.ts +1 -0
- package/web/catalog-ko.ts +1 -0
package/docs/REVIEW_PACKAGE.md
CHANGED
|
@@ -1076,7 +1076,9 @@ the host-owned integration example are in
|
|
|
1076
1076
|
|
|
1077
1077
|
- **Mutating requests and CAS.** Agent mutations (`publish`, `ask`, `close` in `JobMutation`) require `expectedGeneration` (a nonnegative safe integer) and `idempotencyKey` alongside `schema` (`rig-review/job-api@2`), `jobId` and `requestId`. Stale generations refuse with `JOB_GENERATION_STALE` and conflicting idempotency payloads refuse with `JOB_IDEMPOTENCY_CONFLICT`.
|
|
1078
1078
|
- **Explicit producer-held freeze.** `JobPublishRequest` designates a root-relative `result` (`path` and nullable `selector`, never a command) and requires `freeze` with `kind` set to `'producer-held'`, `holdId` and `record` (`JobFileIdentity` for the exact record used for selection, or `null` for a declared non-Create root) held until acknowledgment. Review validates retained bytes, detects a moved or incomplete record (`JOB_FREEZE_REQUIRED`, `JOB_RECORD_INCOMPLETE`, `JOB_RECORD_MOVED`, `JOB_SOURCE_MISMATCH`, `JOB_SOURCE_UNRETAINABLE`, `JOB_ROOT_REPLACED`) and commits local CAS. The freeze assertion is a caller obligation on the publishing producer or agent, not a lock Review can impose on independent writers.
|
|
1079
|
+
- **Managed-state privacy boundary.** Managed state stays behind the host ports. The canonical work root and configured managed-state directory must be disjoint. A root equal to, inside or containing that state directory refuses `JOB_SOURCE_UNRETAINABLE` before job admission, including directory aliases resolved through symlink ancestors. Existing root identity failures, including a directly symlinked source root, retain `JOB_ROOT_REPLACED`. Trusted publication handles follow the origin job's lifetime, and publishing callers must maintain their producer-held freeze.
|
|
1079
1080
|
- **Explicit close lifecycle.** Review offers no watched-root or automatic-close guarantee. Only an explicit `close` request (`JobCloseRequest` with `reason`) ends an active job (`status` transitions from `'active'` to `'closed'`, and subsequent mutations or fetches refuse with `JOB_CLOSED`).
|
|
1081
|
+
- **Snapshot preservation lifecycle.** Every open-job publication, pending ask, page and trusted publication handle pins its snapshot. Explicit close invalidates pending asks, pages and trusted handles, whose subsequent operations refuse `JOB_CLOSED`. Every finalized act permanently preserves its bound source bytes, hashes and retained reproduction evidence. Unfinalized snapshots become reclaimable only after close and explicitly counted asynchronous file reads have finished or cancelled. Event/replay/previous-publication identities remain readable without pinning bytes. No age-based deletion or consumer retain/release API exists. Reclaimed snapshots retain an immutable private publication ID, snapshot hash, reclamation time and `closed` or `superseded` reason, and references reaching the reclaimed-snapshot gate refuse `JOB_SNAPSHOT_RECLAIMED`. The private hash is SHA-256 of UTF-8 `JSON.stringify(orderedSeals.map(({path,sha256}) => ({path,sha256})))`, retaining order, strings and fixed keys. Existing public source-hash formulas remain the source binding. Close and capability revocation use additive durable records outside the bounded JSON document so they succeed at its exact 16 MiB boundary while preserving historical document/replay bytes. Unknown legacy ownership and interrupted-reader counts preserve bytes until ownership or explicit completion is proved. See [Public API stability](STABILITY.md#managed-owner-host).
|
|
1080
1082
|
- **Finalized acts and cursor replay.** `JobNextRequest` (`verb` set to `'next'`, `after` cursor or `null`, and daemon-bounded `waitMs`) and `subscribeJob` replay complete durable `JobEvent` items (`published`, `pending`, `act`, `closed`) in order, returning `JobNextValue` (`events`, `cursor`, and `status` as `'events'`, `'timeout'` or `'closed'`). Waiting is abortable and never consumes an act. Invalid cursors refuse with `JOB_CURSOR_INVALID`.
|
|
1081
1083
|
|
|
1082
1084
|
### Exact displayed source and local draft retention
|
|
@@ -1134,4 +1136,4 @@ The manifest and complete retained image rows enter the composite source hash. L
|
|
|
1134
1136
|
- **Sibling binding ownership.** `JOB_CONSUMER_AUTH_MODE` is `'writer-owned-sibling-fetch-capability'` (`JobConsumerGrant` with `jobId`, `operation` set to `'fetch-act'`, `actKinds` and `lifetime` set to `'until-explicit-close'`). [Create #184](https://github.com/rig-c/create/issues/184) owns creation and path of a private writer-owned sibling binding outside `project.json` whose versioned record carries only the semantic job locator (Create's production reader/writer readback integration remains pending [Create #184](https://github.com/rig-c/create/issues/184)). The binding is never tracked in version control, never present in unstaged project output and never shipped in templates, public examples or fixtures.
|
|
1135
1137
|
- **Agent isolation and capability lifecycle.** The consumer writer reads its own sibling binding directly. Agents never construct, parse or carry Review session, picks, verdict, applied-record or sibling-binding files. Actual separate-UID isolation remains required for Review's distinct state unreadability test (`JOB_AGENT_ISOLATION_HOLE`), and no same-user forger-resistance claim is made. The random per-job bearer capability grants finalized act fetch (`fetch-act`) only, via exact loopback routes and `Authorization`. Neither `next` nor `status` returns it, and human metadata alone never authenticates a fetch. The capability dies at explicit `close`, old project history cannot restore it, each new job incarnation receives a new grant, invalid grants refuse with `JOB_CAPABILITY_REQUIRED` or `JOB_AUTHORITY_REFUSED` and closed-job fetches refuse with `JOB_CLOSED`. No self-service grant API exists.
|
|
1136
1138
|
- **Consumer deduplication and future in-process mesh consumption.** Consumer writers deduplicate `actId` before current-source checks, then validate each new source and atomically commit the writer result plus replay identity. Replaying an applied `actId` returns the existing record, while a new stale act refuses before mutation. Verdict, signoff and stage transition remain three separate consumer writes. Once `mesh-edit` is jointly frozen with [Parts #1159](https://github.com/rig-c/parts/issues/1159), Create will fetch a finalized mesh act and pass its typed value in process to Parts' public exported API owned by [Parts #1159](https://github.com/rig-c/parts/issues/1159), with Parts reading neither Review nor Create storage and taking no Create-project argument, while standalone `--delta` remains legacy outside this cycle.
|
|
1137
|
-
- **Error codes.** `JOB_ERROR_CODES` enumerates the closed refusal set (`JOB_REQUEST_INVALID`, `JOB_BODY_LIMIT`, `JOB_LOOPBACK_REQUIRED`, `JOB_SAME_ORIGIN_REQUIRED`, `JOB_CAPABILITY_REQUIRED`, `JOB_AUTHORITY_REFUSED`, `JOB_UNKNOWN`, `JOB_CLOSED`, `JOB_GENERATION_STALE`, `JOB_IDEMPOTENCY_CONFLICT`, `JOB_FREEZE_REQUIRED`, `JOB_RECORD_INCOMPLETE`, `JOB_RECORD_MOVED`, `JOB_SOURCE_MISMATCH`, `JOB_SOURCE_UNRETAINABLE`, `JOB_ROOT_REPLACED`, `JOB_PUBLICATION_UNKNOWN`, `JOB_IMAGE_SET_UNSUPPORTED`, `JOB_IMAGE_SET_INVALID`, `JOB_IMAGE_SET_DUPLICATE_NAME`, `JOB_IMAGE_SET_DUPLICATE_PATH`, `JOB_IMAGE_SET_ROLE_INVALID`, `JOB_IMAGE_SET_PATH_REFUSED`, `JOB_ASK_INVALID`, `JOB_ASK_SUPERSEDED`, `JOB_ACT_UNKNOWN`, `JOB_ACT_INVALID`, `JOB_ACT_KIND_DISABLED`, `JOB_ACT_SOURCE_STALE`, `JOB_DISPLAY_MISMATCH`, `JOB_CURSOR_INVALID`, `JOB_STORAGE_FAILED`, `JOB_AGENT_ISOLATION_HOLE`), each returning bounded safe detail with no arbitrary worker text or private paths.
|
|
1139
|
+
- **Error codes.** `JOB_ERROR_CODES` enumerates the closed refusal set (`JOB_REQUEST_INVALID`, `JOB_BODY_LIMIT`, `JOB_LOOPBACK_REQUIRED`, `JOB_SAME_ORIGIN_REQUIRED`, `JOB_CAPABILITY_REQUIRED`, `JOB_AUTHORITY_REFUSED`, `JOB_UNKNOWN`, `JOB_CLOSED`, `JOB_GENERATION_STALE`, `JOB_IDEMPOTENCY_CONFLICT`, `JOB_FREEZE_REQUIRED`, `JOB_RECORD_INCOMPLETE`, `JOB_RECORD_MOVED`, `JOB_SOURCE_MISMATCH`, `JOB_SOURCE_UNRETAINABLE`, `JOB_ROOT_REPLACED`, `JOB_PUBLICATION_UNKNOWN`, `JOB_SNAPSHOT_RECLAIMED`, `JOB_IMAGE_SET_UNSUPPORTED`, `JOB_IMAGE_SET_INVALID`, `JOB_IMAGE_SET_DUPLICATE_NAME`, `JOB_IMAGE_SET_DUPLICATE_PATH`, `JOB_IMAGE_SET_ROLE_INVALID`, `JOB_IMAGE_SET_PATH_REFUSED`, `JOB_ASK_INVALID`, `JOB_ASK_SUPERSEDED`, `JOB_ACT_UNKNOWN`, `JOB_ACT_INVALID`, `JOB_ACT_KIND_DISABLED`, `JOB_ACT_SOURCE_STALE`, `JOB_DISPLAY_MISMATCH`, `JOB_CURSOR_INVALID`, `JOB_STORAGE_FAILED`, `JOB_AGENT_ISOLATION_HOLE`), each returning bounded safe detail with no arbitrary worker text or private paths.
|
package/docs/STABILITY.md
CHANGED
|
@@ -153,6 +153,33 @@ credentials through their respective channels. Only explicit semantic `close`
|
|
|
153
153
|
ends a job and revokes its writer grant. `stop()` disposes this host handle and
|
|
154
154
|
stops its owned listener. A reused handle disposes its local session.
|
|
155
155
|
|
|
156
|
+
Managed state is private storage accessed through the host ports. The canonical
|
|
157
|
+
work root and configured managed-state directory must be disjoint. Before job
|
|
158
|
+
admission, a root equal to, inside or containing that state directory refuses
|
|
159
|
+
`JOB_SOURCE_UNRETAINABLE`. Directory aliases resolve before this comparison.
|
|
160
|
+
Existing root identity failures, including a directly symlinked source root,
|
|
161
|
+
retain `JOB_ROOT_REPLACED`. Trusted publication handles follow the origin job's
|
|
162
|
+
lifetime, and publishing callers must maintain their producer-held freeze.
|
|
163
|
+
|
|
164
|
+
All publications, pending asks, pages and trusted publication handles pin their
|
|
165
|
+
snapshots while the job is open. Explicit close invalidates pending asks, pages
|
|
166
|
+
and trusted handles. Subsequent trusted publication reads, UI operations and
|
|
167
|
+
consumer fetches refuse `JOB_CLOSED`. Finalized acts permanently preserve their
|
|
168
|
+
bound source bytes, hashes and retained reproduction evidence. Consumer fetch
|
|
169
|
+
adds no separate retention protocol.
|
|
170
|
+
|
|
171
|
+
After close, unfinalized snapshots can be reclaimed only after explicitly counted
|
|
172
|
+
file reads finish or cancel. Counts cover asynchronous preparation and complete
|
|
173
|
+
response streams across host sessions. Time never releases a count. Historical
|
|
174
|
+
events, replay receipts and previous-publication links preserve identities without
|
|
175
|
+
pinning those bytes. Reclamation leaves an immutable private tombstone containing
|
|
176
|
+
publication ID, snapshot hash, reclamation time and reason. A reference reaching
|
|
177
|
+
the reclaimed-snapshot gate refuses `JOB_SNAPSHOT_RECLAIMED`. Close and grant
|
|
178
|
+
revocation commit outside the bounded historical document, including at its exact
|
|
179
|
+
16 MiB limit. The original document and replay values remain intact. Legacy
|
|
180
|
+
snapshots without recorded creation ownership and interrupted readers remain
|
|
181
|
+
preserved until ownership or explicit completion can be proved.
|
|
182
|
+
|
|
156
183
|
The runtime entry starts the full owner host with retained publication routes
|
|
157
184
|
and shipped HTML, JavaScript and CSS. Raw Requests and the existing A/B ports
|
|
158
185
|
preserve capability, origin, size, cancellation and replay checks.
|
package/package.json
CHANGED
package/src/job-contract.ts
CHANGED
package/src/job-front.ts
CHANGED
|
@@ -195,96 +195,97 @@ function retainedPublicationRoutes(session: JobSession) {
|
|
|
195
195
|
return async (jobId: string, publicationId: string, key: string, path: string, request: Request): Promise<Response> => {
|
|
196
196
|
if (jobId !== session.jobId) return failure('JOB_UNKNOWN', 404);
|
|
197
197
|
try {
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
198
|
+
return await session.readPublicationResponse(publicationId, async retained => {
|
|
199
|
+
const display = session.displayPublication(publicationId);
|
|
200
|
+
if (key !== display.publication.instanceKey) return failure('JOB_DISPLAY_MISMATCH', 409);
|
|
201
|
+
let pending = views.get(publicationId);
|
|
202
|
+
if (!pending) {
|
|
203
|
+
pending = (async () => {
|
|
204
|
+
const review = await loadReview(retained.manifestPath);
|
|
205
|
+
const parts = retained.source.parts
|
|
206
|
+
? await loadPartsView(dirname(join(retained.snapshotRoot, retained.source.parts.manifest.path)))
|
|
207
|
+
: null;
|
|
208
|
+
if (parts) {
|
|
209
|
+
if (parts.items.length !== retained.partIds.length || parts.items.some(item => !retained.partIds.includes(item.name))) {
|
|
210
|
+
throw new JobError('JOB_SOURCE_MISMATCH');
|
|
211
|
+
}
|
|
212
|
+
for (const file of [...parts.files, ...parts.flagFiles]) {
|
|
213
|
+
const path = relative(retained.snapshotRoot, file.abs).split('\\').join('/');
|
|
214
|
+
if (!retained.seals.some(seal => seal.path === path && seal.sha256 === file.sha256)) {
|
|
215
|
+
throw new JobError('JOB_SOURCE_UNRETAINABLE');
|
|
216
|
+
}
|
|
216
217
|
}
|
|
217
218
|
}
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
headers: {...headers, 'Content-Type': 'application/json'},
|
|
227
|
-
});
|
|
228
|
-
const sealedFile = async (relativePath: string) => {
|
|
229
|
-
const seal = retained.seals.find(file => file.path === relativePath);
|
|
230
|
-
if (!seal) return failure('JOB_SOURCE_UNRETAINABLE', 404);
|
|
231
|
-
const file = await safeFile(retained.snapshotRoot, relativePath);
|
|
232
|
-
return serveReference(request, file, seal.sha256, headers);
|
|
233
|
-
};
|
|
234
|
-
if (path === 'api/review') {
|
|
235
|
-
return json({
|
|
236
|
-
manifest: review.manifest,
|
|
237
|
-
sourceHash: retained.source.sourceHash,
|
|
238
|
-
reviewedSource: review.reviewedSource,
|
|
239
|
-
initialAnimation: null,
|
|
240
|
-
setsHash: review.setsHash,
|
|
241
|
-
direction: review.direction,
|
|
242
|
-
referenceHashes: review.referenceHashes,
|
|
243
|
-
sourcePainting: review.sourcePainting && {...review.sourcePainting, url: '/review-painting'},
|
|
244
|
-
feedbackCatalog: review.feedbackCatalog,
|
|
245
|
-
assetsBase: display.routes.assetsBase,
|
|
246
|
-
capabilities: {playback: !!review.manifest.model, interview: retained.questions.length > 0,
|
|
247
|
-
preRig: retained.source.selector?.kind === 'draft', modeling: false, motion: false, feedback: true, audio: false},
|
|
219
|
+
return {review, parts};
|
|
220
|
+
})();
|
|
221
|
+
views.set(publicationId, pending);
|
|
222
|
+
}
|
|
223
|
+
const {review, parts} = await pending;
|
|
224
|
+
await assertReviewUnchanged(review);
|
|
225
|
+
const json = (value: unknown) => new Response(request.method === 'HEAD' ? null : JSON.stringify(value), {
|
|
226
|
+
headers: {...headers, 'Content-Type': 'application/json'},
|
|
248
227
|
});
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
}
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
269
|
-
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
const
|
|
284
|
-
if (
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
228
|
+
const sealedFile = async (relativePath: string) => {
|
|
229
|
+
const seal = retained.seals.find(file => file.path === relativePath);
|
|
230
|
+
if (!seal) return failure('JOB_SOURCE_UNRETAINABLE', 404);
|
|
231
|
+
const file = await safeFile(retained.snapshotRoot, relativePath);
|
|
232
|
+
return serveReference(request, file, seal.sha256, headers);
|
|
233
|
+
};
|
|
234
|
+
if (path === 'api/review') {
|
|
235
|
+
return json({
|
|
236
|
+
manifest: review.manifest,
|
|
237
|
+
sourceHash: retained.source.sourceHash,
|
|
238
|
+
reviewedSource: review.reviewedSource,
|
|
239
|
+
initialAnimation: null,
|
|
240
|
+
setsHash: review.setsHash,
|
|
241
|
+
direction: review.direction,
|
|
242
|
+
referenceHashes: review.referenceHashes,
|
|
243
|
+
sourcePainting: review.sourcePainting && {...review.sourcePainting, url: '/review-painting'},
|
|
244
|
+
feedbackCatalog: review.feedbackCatalog,
|
|
245
|
+
assetsBase: display.routes.assetsBase,
|
|
246
|
+
capabilities: {playback: !!review.manifest.model, interview: retained.questions.length > 0,
|
|
247
|
+
preRig: retained.source.selector?.kind === 'draft', modeling: false, motion: false, feedback: true, audio: false},
|
|
248
|
+
});
|
|
249
|
+
}
|
|
250
|
+
if (path === 'api/interview') {
|
|
251
|
+
return json({questions: retained.questions.map(question => ({...question,
|
|
252
|
+
images: question.images.map((image, index) => ({name: image.name,
|
|
253
|
+
url: `/api/interview/image/${question.ref.sha256}/${index}`})),
|
|
254
|
+
}))});
|
|
255
|
+
}
|
|
256
|
+
const interview = /^api\/interview\/image\/([a-f0-9]{64})\/(0|[1-9]\d*)$/.exec(path);
|
|
257
|
+
if (interview) {
|
|
258
|
+
const image = retained.questions.find(question => question.ref.sha256 === interview[1])?.images[Number(interview[2])];
|
|
259
|
+
if (!image) return failure('JOB_SOURCE_UNRETAINABLE', 404);
|
|
260
|
+
return await sealedFile(image.ref.path);
|
|
261
|
+
}
|
|
262
|
+
const imageSetImage = /^api\/image-set\/(0|[1-9]\d*)$/.exec(path);
|
|
263
|
+
if (imageSetImage && retained.source.subject === 'image-set') {
|
|
264
|
+
const image = retained.source.imageSet.images[Number(imageSetImage[1])];
|
|
265
|
+
if (!image) return failure('JOB_SOURCE_UNRETAINABLE', 404);
|
|
266
|
+
return await sealedFile(image.path);
|
|
267
|
+
}
|
|
268
|
+
if (path === 'api/stage-parts') {
|
|
269
|
+
if (!parts) return failure('JOB_SOURCE_UNRETAINABLE', 409);
|
|
270
|
+
return json({...partsViewJson(parts, {verdict: null, stale: null}), sourceHash: retained.source.sourceHash,
|
|
271
|
+
inspectionSourceHash: retained.source.sourceHash, readOnly: true});
|
|
272
|
+
}
|
|
273
|
+
const part = /^api\/stage-parts\/(image|page)\/(0|[1-9]\d*)$/.exec(path);
|
|
274
|
+
if (part && parts) {
|
|
275
|
+
const file = servedImage(parts, part[1] as 'image' | 'page', Number(part[2]));
|
|
276
|
+
if (!file) return failure('JOB_SOURCE_UNRETAINABLE', 404);
|
|
277
|
+
return await sealedFile(relative(retained.snapshotRoot, file).split('\\').join('/'));
|
|
278
|
+
}
|
|
279
|
+
if (path === 'review-painting' && review.sourcePainting && review.handoff?.source) {
|
|
280
|
+
return await sealedFile(relative(retained.snapshotRoot, review.handoff.source).split('\\').join('/'));
|
|
281
|
+
}
|
|
282
|
+
if (path.startsWith('assets/')) {
|
|
283
|
+
const asset = packagePath(decodeURIComponent(path.slice('assets/'.length)));
|
|
284
|
+
if (!review.files.has(asset)) return failure('JOB_SOURCE_UNRETAINABLE', 404);
|
|
285
|
+
return await sealedFile(relative(retained.snapshotRoot, join(review.root, asset)).split('\\').join('/'));
|
|
286
|
+
}
|
|
287
|
+
return failure('JOB_REQUEST_INVALID', 404);
|
|
288
|
+
});
|
|
288
289
|
} catch (error) {
|
|
289
290
|
return failure(error instanceof JobError ? error.code : 'JOB_SOURCE_MISMATCH', 409);
|
|
290
291
|
}
|
package/src/job-session.ts
CHANGED
|
@@ -1,5 +1,4 @@
|
|
|
1
1
|
import { randomBytes, randomUUID } from 'node:crypto';
|
|
2
|
-
import { rm } from 'node:fs/promises';
|
|
3
2
|
import { isAbsolute, relative, sep } from 'node:path';
|
|
4
3
|
import {
|
|
5
4
|
JOB_API_SCHEMA, JOB_ACT_SCHEMA, JOB_CAPABILITIES, JOB_ENABLED_ACT_KINDS, type JobPartsEnumerator, type JobAct, type JobAgentRequest, type JobDisplayAcknowledgment,
|
|
@@ -9,10 +8,10 @@ import {
|
|
|
9
8
|
import { bindJobAsk, finalizeJobAct, parseUISubmission } from './job-act.ts';
|
|
10
9
|
import { jobHash, jobPartsEnumeratorVersion, jobWaitLimitMs, requestDigest, sameCapability } from './job-api.ts';
|
|
11
10
|
import {
|
|
12
|
-
assertJobRoot, assertJobSourceCurrent, captureJobRoot, prepareJobSource,
|
|
11
|
+
assertJobRoot, assertJobSourceCurrent, captureJobRoot, prepareJobSource, jobSnapshotHash, removeOwnedJobSnapshot,
|
|
13
12
|
type JobPublicationReader, type JobRetainedSource, type JobRootIdentity,
|
|
14
13
|
} from './job-source.ts';
|
|
15
|
-
import { JobError, JobStore, type StoredJob } from './job-store.ts';
|
|
14
|
+
import { JobError, JobStore, type JobSnapshotOwner, type StoredJob } from './job-store.ts';
|
|
16
15
|
import { canonicalStatePath } from './state-dir.ts';
|
|
17
16
|
|
|
18
17
|
interface StoredPublication {
|
|
@@ -78,10 +77,16 @@ export class JobSession {
|
|
|
78
77
|
private readonly store: JobStore<JobData>;
|
|
79
78
|
private readonly listeners = new Set<() => void>();
|
|
80
79
|
private operations = 0;
|
|
80
|
+
private disposed = false;
|
|
81
81
|
|
|
82
82
|
constructor(private readonly options: JobSessionOptions) {
|
|
83
83
|
const root = captureJobRoot(options.root), state = canonicalStatePath(options.stateDir);
|
|
84
|
-
if (within(root.path, state))
|
|
84
|
+
if (within(root.path, state)) {
|
|
85
|
+
throw new JobError('JOB_SOURCE_UNRETAINABLE', 'Job state must be outside the designated work root');
|
|
86
|
+
}
|
|
87
|
+
if (within(state, root.path)) {
|
|
88
|
+
throw new JobError('JOB_SOURCE_UNRETAINABLE', 'The designated work root must be outside job state');
|
|
89
|
+
}
|
|
85
90
|
if (!options.reviewer.id || !['owner', 'delegate'].includes(options.reviewer.role)) throw new JobError('JOB_AUTHORITY_REFUSED');
|
|
86
91
|
if (options.partsEnumerator) jobPartsEnumeratorVersion(options.partsEnumerator.version);
|
|
87
92
|
this.store = new JobStore(state);
|
|
@@ -110,6 +115,18 @@ export class JobSession {
|
|
|
110
115
|
private current(): StoredJob<JobData> {
|
|
111
116
|
const row = this.store.get(this.jobId);
|
|
112
117
|
assertStoredSchemas(row.data);
|
|
118
|
+
if (!row.active) {
|
|
119
|
+
row.data.fetchCapability = null;
|
|
120
|
+
row.data.pages = {};
|
|
121
|
+
for (const ask of Object.values(row.data.asks)) {
|
|
122
|
+
if (ask.status === 'pending') row.data.asks[ask.askId] = {...ask, status: 'superseded'};
|
|
123
|
+
}
|
|
124
|
+
}
|
|
125
|
+
if (row.closure) {
|
|
126
|
+
row.data.generation = row.closure.generation;
|
|
127
|
+
row.data.events.push(row.closure.event);
|
|
128
|
+
row.data.replays[row.closure.replayKey] = row.closure.replay;
|
|
129
|
+
}
|
|
113
130
|
return row;
|
|
114
131
|
}
|
|
115
132
|
|
|
@@ -150,18 +167,102 @@ export class JobSession {
|
|
|
150
167
|
}
|
|
151
168
|
|
|
152
169
|
retainedSource(publicationId: string): JobRetainedSource {
|
|
153
|
-
const
|
|
170
|
+
const current = this.current();
|
|
171
|
+
if (!current.active) throw new JobError('JOB_CLOSED');
|
|
172
|
+
const stored = current.data.publications[publicationId];
|
|
154
173
|
if (!stored) throw new JobError('JOB_PUBLICATION_UNKNOWN');
|
|
155
174
|
return stored.retained;
|
|
156
175
|
}
|
|
157
176
|
|
|
158
177
|
displayPublication(publicationId: string): JobDisplayPublication {
|
|
159
|
-
const
|
|
178
|
+
const current = this.current();
|
|
179
|
+
if (!current.active) throw new JobError('JOB_CLOSED');
|
|
180
|
+
const stored = current.data.publications[publicationId];
|
|
160
181
|
if (!stored) throw new JobError('JOB_PUBLICATION_UNKNOWN');
|
|
161
182
|
const base = `/jobs/${this.jobId}/publications/${publicationId}/instances/${stored.publication.instanceKey}/`;
|
|
162
183
|
return { publication: stored.publication, routes: { entry: base, apiBase: base + 'api/', assetsBase: base + 'assets/' } };
|
|
163
184
|
}
|
|
164
185
|
|
|
186
|
+
/** Internal front seam. A count covers preparation and the entire file stream. */
|
|
187
|
+
async readPublicationResponse(publicationId: string,
|
|
188
|
+
read: (retained: JobRetainedSource) => Promise<Response>): Promise<Response> {
|
|
189
|
+
const retained = this.retainedSource(publicationId);
|
|
190
|
+
const readId = this.store.beginSnapshotRead(this.jobId, publicationId,
|
|
191
|
+
jobSnapshotHash(retained.seals), retained.snapshotRoot);
|
|
192
|
+
this.operations++;
|
|
193
|
+
let operationEnded = false;
|
|
194
|
+
const endOperation = () => {
|
|
195
|
+
if (operationEnded) return;
|
|
196
|
+
operationEnded = true;
|
|
197
|
+
this.finishOperation();
|
|
198
|
+
};
|
|
199
|
+
let finishing: Promise<void> | undefined;
|
|
200
|
+
const finish = () => finishing ??= (async () => {
|
|
201
|
+
try {
|
|
202
|
+
this.store.endSnapshotRead(readId);
|
|
203
|
+
await this.reclaimClosedSnapshots(publicationId);
|
|
204
|
+
} finally { endOperation(); }
|
|
205
|
+
})();
|
|
206
|
+
try {
|
|
207
|
+
const response = await read(retained);
|
|
208
|
+
if (!response.body) { await finish(); return response; }
|
|
209
|
+
const reader = response.body.getReader();
|
|
210
|
+
let cancellation: Promise<void> | undefined;
|
|
211
|
+
const cancel = (reason?: unknown) => cancellation ??= Promise.resolve()
|
|
212
|
+
.then(() => reader.cancel(reason))
|
|
213
|
+
.catch(error => {endOperation(); throw error;});
|
|
214
|
+
const body = new ReadableStream<Uint8Array>({
|
|
215
|
+
async pull(controller) {
|
|
216
|
+
try {
|
|
217
|
+
const next = await reader.read();
|
|
218
|
+
if (next.done) { await cancellation; await finish(); controller.close(); }
|
|
219
|
+
else controller.enqueue(next.value);
|
|
220
|
+
} catch (error) {
|
|
221
|
+
try { await cancel(error); await finish(); }
|
|
222
|
+
catch (cleanupError) {controller.error(cleanupError); return;}
|
|
223
|
+
controller.error(error);
|
|
224
|
+
}
|
|
225
|
+
},
|
|
226
|
+
async cancel(reason) {
|
|
227
|
+
await cancel(reason);
|
|
228
|
+
await finish();
|
|
229
|
+
},
|
|
230
|
+
}, {highWaterMark: 0});
|
|
231
|
+
return new Response(body, {status: response.status, statusText: response.statusText, headers: response.headers});
|
|
232
|
+
} catch (error) {
|
|
233
|
+
// Failed preparation or rejected cancellation cannot prove file closure.
|
|
234
|
+
endOperation();
|
|
235
|
+
throw error;
|
|
236
|
+
}
|
|
237
|
+
}
|
|
238
|
+
|
|
239
|
+
/** Internal maintenance, never an agent or consumer operation. */
|
|
240
|
+
async reclaimSnapshot(publicationId: string, snapshotHash: string): Promise<void> {
|
|
241
|
+
const owner = this.store.markSnapshotReclaimed(this.jobId, publicationId, snapshotHash,
|
|
242
|
+
data => Object.values(data.acts).some(act => act.publicationId === publicationId));
|
|
243
|
+
if (!owner) throw new JobError('JOB_SOURCE_UNRETAINABLE', 'JOB_SNAPSHOT_PINNED');
|
|
244
|
+
await removeOwnedJobSnapshot(this.store.root, owner);
|
|
245
|
+
}
|
|
246
|
+
|
|
247
|
+
private async reclaimClosedSnapshots(onlyPublicationId?: string): Promise<void> {
|
|
248
|
+
const current = this.current();
|
|
249
|
+
if (current.active) return;
|
|
250
|
+
const publications = onlyPublicationId === undefined ? Object.keys(current.data.publications) : [onlyPublicationId];
|
|
251
|
+
for (const publicationId of publications) {
|
|
252
|
+
// Legacy snapshots without an explicit creation identity stay preserved.
|
|
253
|
+
const owner = this.store.snapshotOwner(this.jobId, publicationId);
|
|
254
|
+
if (!owner) continue;
|
|
255
|
+
const reclaimed = this.store.markSnapshotReclaimed(this.jobId, publicationId, owner.snapshotHash,
|
|
256
|
+
data => Object.values(data.acts).some(act => act.publicationId === publicationId));
|
|
257
|
+
if (reclaimed) await removeOwnedJobSnapshot(this.store.root, reclaimed);
|
|
258
|
+
}
|
|
259
|
+
}
|
|
260
|
+
|
|
261
|
+
private finishOperation(): void {
|
|
262
|
+
this.operations--;
|
|
263
|
+
if (this.disposed && !this.busy) this.store.close();
|
|
264
|
+
}
|
|
265
|
+
|
|
165
266
|
private replay(data: JobData, key: string, digest: string): Replay | null {
|
|
166
267
|
const previous = data.replays[key];
|
|
167
268
|
if (previous && previous.digest !== digest) throw new JobError('JOB_IDEMPOTENCY_CONFLICT');
|
|
@@ -198,11 +299,39 @@ export class JobSession {
|
|
|
198
299
|
const digest = requestDigest(semantic), key = 'agent:' + input.idempotencyKey;
|
|
199
300
|
let current = this.current();
|
|
200
301
|
const previous = this.replay(current.data, key, digest);
|
|
201
|
-
if (previous)
|
|
302
|
+
if (previous) {
|
|
303
|
+
this.operations++;
|
|
304
|
+
try {
|
|
305
|
+
if (input.verb === 'close') await this.reclaimClosedSnapshots();
|
|
306
|
+
return previous.value;
|
|
307
|
+
} finally { this.finishOperation(); }
|
|
308
|
+
}
|
|
202
309
|
this.active(current, input.expectedGeneration);
|
|
203
310
|
if (signal.aborted) throw new JobError('JOB_REQUEST_INVALID', 'Request cancelled');
|
|
311
|
+
if (input.verb === 'close') {
|
|
312
|
+
this.operations++;
|
|
313
|
+
try {
|
|
314
|
+
const closure = this.store.sealClose(this.jobId, current.revision, fresh => {
|
|
315
|
+
if (fresh.closure?.replayKey === key) {
|
|
316
|
+
if (fresh.closure.replay.digest !== digest) throw new JobError('JOB_IDEMPOTENCY_CONFLICT');
|
|
317
|
+
return fresh.closure;
|
|
318
|
+
}
|
|
319
|
+
const previous = this.replay(fresh.data, key, digest);
|
|
320
|
+
return previous ? {replay: previous} : null;
|
|
321
|
+
}, fresh => {
|
|
322
|
+
this.active(fresh, input.expectedGeneration);
|
|
323
|
+
fresh.data.generation++;
|
|
324
|
+
this.event(fresh.data, {type: 'closed', reason: input.reason} as Omit<JobEvent, 'cursor' | 'jobId' | 'generation'>);
|
|
325
|
+
const event = fresh.data.events.at(-1)!;
|
|
326
|
+
const value = {status: 'closed', cursor: event.cursor};
|
|
327
|
+
return {generation: fresh.data.generation, event, replayKey: key, replay: {digest, value}};
|
|
328
|
+
});
|
|
329
|
+
this.notify();
|
|
330
|
+
await this.reclaimClosedSnapshots();
|
|
331
|
+
return closure.replay.value;
|
|
332
|
+
} finally { this.finishOperation(); }
|
|
333
|
+
}
|
|
204
334
|
let candidate: JobRetainedSource | null = null;
|
|
205
|
-
let committed = false;
|
|
206
335
|
this.operations++;
|
|
207
336
|
try {
|
|
208
337
|
if (input.verb === 'publish') {
|
|
@@ -219,6 +348,7 @@ export class JobSession {
|
|
|
219
348
|
if (raced) { result = raced.value; return { data: fresh.data }; }
|
|
220
349
|
this.active(fresh, input.expectedGeneration);
|
|
221
350
|
const data = fresh.data;
|
|
351
|
+
let snapshot: JobSnapshotOwner | undefined;
|
|
222
352
|
data.generation++;
|
|
223
353
|
switch (input.verb) {
|
|
224
354
|
case 'publish': {
|
|
@@ -231,6 +361,8 @@ export class JobSession {
|
|
|
231
361
|
if (ask.status === 'pending') data.asks[ask.askId] = { ...ask, status: 'superseded' };
|
|
232
362
|
}
|
|
233
363
|
data.publications[publication.publicationId] = { publication, retained: candidate! };
|
|
364
|
+
snapshot = {publicationId: publication.publicationId, snapshotHash: jobSnapshotHash(candidate!.seals),
|
|
365
|
+
snapshotRoot: candidate!.snapshotRoot, ...candidate!.snapshotIdentity!};
|
|
234
366
|
data.head = publication.publicationId;
|
|
235
367
|
this.event(data, { type: 'published', publication } as Omit<JobEvent, 'cursor' | 'jobId' | 'generation'>);
|
|
236
368
|
result = publication;
|
|
@@ -247,23 +379,18 @@ export class JobSession {
|
|
|
247
379
|
result = ask;
|
|
248
380
|
break;
|
|
249
381
|
}
|
|
250
|
-
case 'close':
|
|
251
|
-
data.fetchCapability = null;
|
|
252
|
-
data.pages = {};
|
|
253
|
-
this.event(data, { type: 'closed', reason: input.reason } as Omit<JobEvent, 'cursor' | 'jobId' | 'generation'>);
|
|
254
|
-
result = { status: 'closed', cursor: data.events.at(-1)!.cursor };
|
|
255
|
-
break;
|
|
256
382
|
}
|
|
257
383
|
data.replays[key] = { digest, value: result };
|
|
258
|
-
return { data,
|
|
384
|
+
return { data, ...(snapshot ? {snapshot} : {}) };
|
|
259
385
|
});
|
|
260
|
-
committed = candidate === null || Object.values(this.current().data.publications)
|
|
261
|
-
.some(stored => stored.retained.snapshotRoot === candidate!.snapshotRoot);
|
|
262
386
|
this.notify();
|
|
263
387
|
return result;
|
|
264
388
|
} finally {
|
|
265
|
-
|
|
266
|
-
|
|
389
|
+
try {
|
|
390
|
+
if (candidate?.snapshotIdentity && !this.store.snapshotReferenced(candidate.snapshotRoot)) {
|
|
391
|
+
await removeOwnedJobSnapshot(this.store.root, {snapshotRoot: candidate.snapshotRoot, ...candidate.snapshotIdentity});
|
|
392
|
+
}
|
|
393
|
+
} finally { this.finishOperation(); }
|
|
267
394
|
}
|
|
268
395
|
}
|
|
269
396
|
|
|
@@ -366,7 +493,9 @@ export class JobSession {
|
|
|
366
493
|
}
|
|
367
494
|
|
|
368
495
|
dispose(): void {
|
|
496
|
+
if (this.disposed) return;
|
|
369
497
|
this.notify();
|
|
370
|
-
this.
|
|
498
|
+
this.disposed = true;
|
|
499
|
+
if (!this.busy) this.store.close();
|
|
371
500
|
}
|
|
372
501
|
}
|
package/src/job-source.ts
CHANGED
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
import { randomUUID } from 'node:crypto';
|
|
2
|
-
import { lstatSync, realpathSync } from 'node:fs';
|
|
2
|
+
import { lstatSync, realpathSync, type Stats } from 'node:fs';
|
|
3
3
|
import { mkdir, open, rm, writeFile } from 'node:fs/promises';
|
|
4
4
|
import { dirname, isAbsolute, join, posix, relative, resolve, sep } from 'node:path';
|
|
5
5
|
import { PNG } from 'pngjs';
|
|
@@ -13,7 +13,7 @@ import { loadReview } from './manifest.ts';
|
|
|
13
13
|
import { openKey } from './open.ts';
|
|
14
14
|
import { assertReviewUnchanged } from './reference.ts';
|
|
15
15
|
import { jobHash, jobId, jobPartsEnumeratorVersion, jobObject, jobPath, jobSha, jobText, requestDigest } from './job-api.ts';
|
|
16
|
-
import { JobError } from './job-store.ts';
|
|
16
|
+
import { JobError, type JobSnapshotOwner } from './job-store.ts';
|
|
17
17
|
import { prepareJobImageSet } from './job-image-set.ts';
|
|
18
18
|
|
|
19
19
|
export interface JobRootIdentity {
|
|
@@ -48,6 +48,7 @@ export interface JobRetainedSource {
|
|
|
48
48
|
readonly record: JobFileIdentity | null;
|
|
49
49
|
readonly seals: readonly JobFileIdentity[];
|
|
50
50
|
readonly snapshotRoot: string;
|
|
51
|
+
readonly snapshotIdentity?: { readonly dev: number; readonly ino: number };
|
|
51
52
|
readonly manifestPath: string;
|
|
52
53
|
readonly source: JobSourceBinding;
|
|
53
54
|
readonly instanceKey: string;
|
|
@@ -62,6 +63,37 @@ const maxSourceBytes = 256 * 1024 * 1024;
|
|
|
62
63
|
const maxSourceFiles = 4096;
|
|
63
64
|
const maxFileBytes = 64 * 1024 * 1024;
|
|
64
65
|
|
|
66
|
+
/** Private preservation identity. Keep the supplied seal order and fixed keys. */
|
|
67
|
+
export function jobSnapshotHash(seals: readonly JobFileIdentity[]): string {
|
|
68
|
+
return jobHash(JSON.stringify(seals.map(({path, sha256}) => ({path, sha256}))));
|
|
69
|
+
}
|
|
70
|
+
|
|
71
|
+
export async function removeOwnedJobSnapshot(stateRoot: string,
|
|
72
|
+
snapshot: Pick<JobSnapshotOwner, 'snapshotRoot' | 'dev' | 'ino'>): Promise<void> {
|
|
73
|
+
const parent = join(stateRoot, 'snapshots'), path = snapshot.snapshotRoot;
|
|
74
|
+
if (dirname(path) !== parent || !/^[a-f0-9-]{36}$/.test(posix.basename(path))) {
|
|
75
|
+
throw new JobError('JOB_SOURCE_UNRETAINABLE', 'Snapshot ownership path differs');
|
|
76
|
+
}
|
|
77
|
+
let info: Stats;
|
|
78
|
+
try { info = lstatSync(path); }
|
|
79
|
+
catch (error) { if ((error as NodeJS.ErrnoException).code === 'ENOENT') return; throw error; }
|
|
80
|
+
if (!info.isDirectory() || info.dev !== snapshot.dev || info.ino !== snapshot.ino || (info.mode & 0o077) !== 0 ||
|
|
81
|
+
typeof process.getuid === 'function' && info.uid !== process.getuid()) {
|
|
82
|
+
throw new JobError('JOB_SOURCE_UNRETAINABLE', 'Snapshot ownership differs');
|
|
83
|
+
}
|
|
84
|
+
if (realpathSync(parent) !== parent) throw new JobError('JOB_SOURCE_UNRETAINABLE', 'Snapshot ownership differs');
|
|
85
|
+
try {
|
|
86
|
+
if (realpathSync(path) !== path) throw new JobError('JOB_SOURCE_UNRETAINABLE', 'Snapshot ownership differs');
|
|
87
|
+
} catch (error) {
|
|
88
|
+
if ((error as NodeJS.ErrnoException).code !== 'ENOENT') throw error;
|
|
89
|
+
// A dangling replacement still exists and must refuse. Only absence drains.
|
|
90
|
+
try { lstatSync(path); }
|
|
91
|
+
catch (missing) {if ((missing as NodeJS.ErrnoException).code === 'ENOENT') return; throw missing;}
|
|
92
|
+
throw new JobError('JOB_SOURCE_UNRETAINABLE', 'Snapshot ownership differs');
|
|
93
|
+
}
|
|
94
|
+
await rm(path, {recursive: true, force: true});
|
|
95
|
+
}
|
|
96
|
+
|
|
65
97
|
export function captureJobRoot(path: string): JobRootIdentity {
|
|
66
98
|
try {
|
|
67
99
|
const canonical = realpathSync(path), info = lstatSync(path);
|
|
@@ -330,8 +362,12 @@ export async function prepareJobSource(options: {
|
|
|
330
362
|
if (wanted.size > maxSourceFiles) throw new JobError('JOB_SOURCE_UNRETAINABLE');
|
|
331
363
|
const snapshotRoot = join(options.stateRoot, 'snapshots', randomUUID());
|
|
332
364
|
const seals: JobFileIdentity[] = [];
|
|
365
|
+
let snapshotIdentity: {dev: number; ino: number} | null = null;
|
|
333
366
|
try {
|
|
334
|
-
await mkdir(snapshotRoot, {
|
|
367
|
+
await mkdir(dirname(snapshotRoot), {recursive: true, mode: 0o700});
|
|
368
|
+
await mkdir(snapshotRoot, {mode: 0o700});
|
|
369
|
+
const info = lstatSync(snapshotRoot);
|
|
370
|
+
snapshotIdentity = {dev: info.dev, ino: info.ino};
|
|
335
371
|
const imageSet = context.imageSet === undefined ? null : prepareJobImageSet(context.imageSet,
|
|
336
372
|
path => sourceBytes(root.path, path));
|
|
337
373
|
for (const image of imageSet?.imageSet.images ?? []) add(image.path, image.sha256);
|
|
@@ -393,7 +429,7 @@ export async function prepareJobSource(options: {
|
|
|
393
429
|
}
|
|
394
430
|
const boundReview = { ...review, sourceHash: source.sourceHash };
|
|
395
431
|
const prepared: JobRetainedSource = {
|
|
396
|
-
root, record, seals, snapshotRoot,
|
|
432
|
+
root, record, seals, snapshotRoot, snapshotIdentity,
|
|
397
433
|
manifestPath: retained.manifestPath,
|
|
398
434
|
source,
|
|
399
435
|
instanceKey: openKey({ mode: 'single', p: boundReview, inputs: { input: review.manifestPath },
|
|
@@ -412,7 +448,7 @@ export async function prepareJobSource(options: {
|
|
|
412
448
|
assertJobSourceCurrent(prepared);
|
|
413
449
|
return prepared;
|
|
414
450
|
} catch (error) {
|
|
415
|
-
await
|
|
451
|
+
if (snapshotIdentity) await removeOwnedJobSnapshot(options.stateRoot, {snapshotRoot, ...snapshotIdentity});
|
|
416
452
|
if (error instanceof JobError) throw error;
|
|
417
453
|
throw new JobError('JOB_SOURCE_MISMATCH');
|
|
418
454
|
}
|
package/src/job-store.ts
CHANGED
|
@@ -2,7 +2,7 @@ import { Database } from 'bun:sqlite';
|
|
|
2
2
|
import { randomUUID } from 'node:crypto';
|
|
3
3
|
import { chmodSync, lstatSync, mkdirSync, realpathSync } from 'node:fs';
|
|
4
4
|
import { join } from 'node:path';
|
|
5
|
-
import { JOB_API_SCHEMA, JOB_ACT_SCHEMA, type JobErrorCode } from './job-contract.ts';
|
|
5
|
+
import { JOB_API_SCHEMA, JOB_ACT_SCHEMA, type JobErrorCode, type JobEvent } from './job-contract.ts';
|
|
6
6
|
|
|
7
7
|
export class JobError extends Error {
|
|
8
8
|
constructor(readonly code: JobErrorCode, message: string = code) {
|
|
@@ -17,6 +17,29 @@ export interface StoredJob<Data> {
|
|
|
17
17
|
readonly revision: number;
|
|
18
18
|
readonly active: boolean;
|
|
19
19
|
readonly data: Data;
|
|
20
|
+
readonly closure?: JobClosure;
|
|
21
|
+
}
|
|
22
|
+
|
|
23
|
+
export interface JobClosure {
|
|
24
|
+
readonly generation: number;
|
|
25
|
+
readonly event: JobEvent;
|
|
26
|
+
readonly replayKey: string;
|
|
27
|
+
readonly replay: { readonly digest: string; readonly value: unknown };
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
export interface JobSnapshotOwner {
|
|
31
|
+
readonly publicationId: string;
|
|
32
|
+
readonly snapshotHash: string;
|
|
33
|
+
readonly snapshotRoot: string;
|
|
34
|
+
readonly dev: number;
|
|
35
|
+
readonly ino: number;
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
export interface JobSnapshotTombstone {
|
|
39
|
+
readonly publicationId: string;
|
|
40
|
+
readonly snapshotHash: string;
|
|
41
|
+
readonly reclaimedAt: string;
|
|
42
|
+
readonly reason: 'closed' | 'superseded';
|
|
20
43
|
}
|
|
21
44
|
|
|
22
45
|
interface JobRow {
|
|
@@ -25,6 +48,7 @@ interface JobRow {
|
|
|
25
48
|
revision: number;
|
|
26
49
|
active: number;
|
|
27
50
|
document: string;
|
|
51
|
+
closure?: string | null;
|
|
28
52
|
}
|
|
29
53
|
|
|
30
54
|
const maximumDocumentBytes = 16 * 1024 * 1024;
|
|
@@ -57,6 +81,7 @@ function decode<Data>(row: JobRow): StoredJob<Data> {
|
|
|
57
81
|
revision: row.revision,
|
|
58
82
|
active: row.active === 1,
|
|
59
83
|
data: JSON.parse(row.document) as Data,
|
|
84
|
+
...(row.closure ? { closure: JSON.parse(row.closure) as JobClosure } : {}),
|
|
60
85
|
};
|
|
61
86
|
} catch {
|
|
62
87
|
throw new JobError('JOB_STORAGE_FAILED', 'Stored job state is invalid');
|
|
@@ -102,7 +127,23 @@ export class JobStore<Data> {
|
|
|
102
127
|
);
|
|
103
128
|
CREATE UNIQUE INDEX IF NOT EXISTS jobs_active_root ON jobs(root_key) WHERE active=1;
|
|
104
129
|
CREATE INDEX IF NOT EXISTS jobs_active_canonical_root
|
|
105
|
-
ON jobs(json_extract(document, '$.root.path')) WHERE active=1
|
|
130
|
+
ON jobs(json_extract(document, '$.root.path')) WHERE active=1;
|
|
131
|
+
CREATE TABLE IF NOT EXISTS job_closures (
|
|
132
|
+
job_id TEXT PRIMARY KEY, document TEXT NOT NULL
|
|
133
|
+
);
|
|
134
|
+
CREATE TABLE IF NOT EXISTS job_snapshot_owners (
|
|
135
|
+
job_id TEXT NOT NULL, publication_id TEXT NOT NULL, snapshot_hash TEXT NOT NULL,
|
|
136
|
+
snapshot_root TEXT NOT NULL, dev INTEGER NOT NULL, ino INTEGER NOT NULL,
|
|
137
|
+
PRIMARY KEY(job_id, publication_id)
|
|
138
|
+
);
|
|
139
|
+
CREATE TABLE IF NOT EXISTS job_snapshot_reads (
|
|
140
|
+
read_id TEXT PRIMARY KEY, job_id TEXT NOT NULL, snapshot_root TEXT NOT NULL
|
|
141
|
+
);
|
|
142
|
+
CREATE TABLE IF NOT EXISTS job_snapshot_tombstones (
|
|
143
|
+
job_id TEXT NOT NULL, publication_id TEXT NOT NULL, snapshot_hash TEXT NOT NULL,
|
|
144
|
+
reclaimed_at TEXT NOT NULL, reason TEXT NOT NULL CHECK(reason IN ('closed','superseded')),
|
|
145
|
+
PRIMARY KEY(job_id, publication_id)
|
|
146
|
+
);`);
|
|
106
147
|
} catch {
|
|
107
148
|
throw new JobError('JOB_STORAGE_FAILED', 'Private job storage could not be opened');
|
|
108
149
|
}
|
|
@@ -120,7 +161,8 @@ export class JobStore<Data> {
|
|
|
120
161
|
}
|
|
121
162
|
|
|
122
163
|
get(jobId: string): StoredJob<Data> {
|
|
123
|
-
const row = this.db.query<JobRow, [string]>(
|
|
164
|
+
const row = this.db.query<JobRow, [string]>(`SELECT jobs.*, job_closures.document AS closure
|
|
165
|
+
FROM jobs LEFT JOIN job_closures USING(job_id) WHERE job_id=?`).get(jobId);
|
|
124
166
|
if (!row) throw new JobError('JOB_UNKNOWN');
|
|
125
167
|
return decode<Data>(row);
|
|
126
168
|
}
|
|
@@ -162,6 +204,7 @@ export class JobStore<Data> {
|
|
|
162
204
|
update(jobId: string, expectedRevision: number, operation: (current: StoredJob<Data>) => {
|
|
163
205
|
readonly data: Data;
|
|
164
206
|
readonly active?: boolean;
|
|
207
|
+
readonly snapshot?: JobSnapshotOwner;
|
|
165
208
|
}): StoredJob<Data> {
|
|
166
209
|
return this.transaction(() => {
|
|
167
210
|
const current = this.get(jobId);
|
|
@@ -169,12 +212,97 @@ export class JobStore<Data> {
|
|
|
169
212
|
const next = operation(current);
|
|
170
213
|
const active = next.active ?? current.active;
|
|
171
214
|
const document = encode(next.data);
|
|
215
|
+
if (next.snapshot) {
|
|
216
|
+
const snapshot = next.snapshot;
|
|
217
|
+
this.db.query(`INSERT INTO job_snapshot_owners
|
|
218
|
+
(job_id,publication_id,snapshot_hash,snapshot_root,dev,ino) VALUES(?,?,?,?,?,?)`)
|
|
219
|
+
.run(jobId, snapshot.publicationId, snapshot.snapshotHash, snapshot.snapshotRoot, snapshot.dev, snapshot.ino);
|
|
220
|
+
}
|
|
172
221
|
this.db.query('UPDATE jobs SET revision=?,active=?,document=? WHERE job_id=?')
|
|
173
222
|
.run(current.revision + 1, active ? 1 : 0, document, jobId);
|
|
174
223
|
return this.get(jobId);
|
|
175
224
|
});
|
|
176
225
|
}
|
|
177
226
|
|
|
227
|
+
/** Close never grows or rewrites the bounded historical document. */
|
|
228
|
+
sealClose(jobId: string, expectedRevision: number,
|
|
229
|
+
replay: (current: StoredJob<Data>) => Pick<JobClosure, 'replay'> | null,
|
|
230
|
+
prepare: (current: StoredJob<Data>) => JobClosure): Pick<JobClosure, 'replay'> {
|
|
231
|
+
return this.transaction(() => {
|
|
232
|
+
const current = this.get(jobId);
|
|
233
|
+
const previous = replay(current);
|
|
234
|
+
if (previous) return previous;
|
|
235
|
+
if (current.revision !== expectedRevision) throw new JobError('JOB_GENERATION_STALE');
|
|
236
|
+
if (!current.active) throw new JobError('JOB_CLOSED');
|
|
237
|
+
const closure = prepare(current), document = encode(closure);
|
|
238
|
+
this.db.query('INSERT INTO job_closures(job_id,document) VALUES(?,?)').run(jobId, document);
|
|
239
|
+
this.db.query('UPDATE jobs SET revision=revision+1,active=0 WHERE job_id=?').run(jobId);
|
|
240
|
+
return closure;
|
|
241
|
+
});
|
|
242
|
+
}
|
|
243
|
+
|
|
244
|
+
snapshotOwner(jobId: string, publicationId: string): JobSnapshotOwner | null {
|
|
245
|
+
return this.db.query<JobSnapshotOwner, [string, string]>(`SELECT publication_id AS publicationId,
|
|
246
|
+
snapshot_hash AS snapshotHash, snapshot_root AS snapshotRoot, dev, ino
|
|
247
|
+
FROM job_snapshot_owners WHERE job_id=? AND publication_id=?`).get(jobId, publicationId);
|
|
248
|
+
}
|
|
249
|
+
|
|
250
|
+
snapshotTombstone(jobId: string, publicationId: string): JobSnapshotTombstone | null {
|
|
251
|
+
return this.db.query<JobSnapshotTombstone, [string, string]>(`SELECT publication_id AS publicationId,
|
|
252
|
+
snapshot_hash AS snapshotHash, reclaimed_at AS reclaimedAt, reason
|
|
253
|
+
FROM job_snapshot_tombstones WHERE job_id=? AND publication_id=?`).get(jobId, publicationId);
|
|
254
|
+
}
|
|
255
|
+
|
|
256
|
+
snapshotReferenced(snapshotRoot: string, except?: { jobId: string; publicationId: string }): boolean {
|
|
257
|
+
return this.db.query<{ present: number }, [string, string, string]>(`SELECT 1 AS present FROM jobs,
|
|
258
|
+
json_each(jobs.document, '$.publications') AS publication
|
|
259
|
+
WHERE json_extract(publication.value, '$.retained.snapshotRoot')=?
|
|
260
|
+
AND NOT (jobs.job_id=? AND publication.key=?) LIMIT 1`)
|
|
261
|
+
.get(snapshotRoot, except?.jobId ?? '', except?.publicationId ?? '') !== null;
|
|
262
|
+
}
|
|
263
|
+
|
|
264
|
+
/** These are internal file-I/O counts, never a consumer grant or a lease.
|
|
265
|
+
* Only explicit completion removes a count. A crashed reader pins its bytes.
|
|
266
|
+
*/
|
|
267
|
+
beginSnapshotRead(jobId: string, publicationId: string, snapshotHash: string,
|
|
268
|
+
snapshotRoot: string): string {
|
|
269
|
+
return this.transaction(() => {
|
|
270
|
+
if (this.snapshotTombstone(jobId, publicationId)) throw new JobError('JOB_SNAPSHOT_RECLAIMED');
|
|
271
|
+
if (!this.get(jobId).active) throw new JobError('JOB_CLOSED');
|
|
272
|
+
const owner = this.snapshotOwner(jobId, publicationId);
|
|
273
|
+
if (owner && (owner.snapshotHash !== snapshotHash || owner.snapshotRoot !== snapshotRoot)) {
|
|
274
|
+
throw new JobError('JOB_SOURCE_MISMATCH');
|
|
275
|
+
}
|
|
276
|
+
const readId = randomUUID();
|
|
277
|
+
this.db.query('INSERT INTO job_snapshot_reads(read_id,job_id,snapshot_root) VALUES(?,?,?)')
|
|
278
|
+
.run(readId, jobId, snapshotRoot);
|
|
279
|
+
return readId;
|
|
280
|
+
});
|
|
281
|
+
}
|
|
282
|
+
|
|
283
|
+
endSnapshotRead(readId: string): void {
|
|
284
|
+
this.transaction(() => {
|
|
285
|
+
this.db.query('DELETE FROM job_snapshot_reads WHERE read_id=?').run(readId);
|
|
286
|
+
});
|
|
287
|
+
}
|
|
288
|
+
|
|
289
|
+
markSnapshotReclaimed(jobId: string, publicationId: string, snapshotHash: string,
|
|
290
|
+
finalized: (data: Data) => boolean): JobSnapshotOwner | null {
|
|
291
|
+
return this.transaction(() => {
|
|
292
|
+
const current = this.get(jobId), owner = this.snapshotOwner(jobId, publicationId);
|
|
293
|
+
if (!owner) throw new JobError('JOB_SOURCE_UNRETAINABLE', 'Snapshot ownership is unavailable');
|
|
294
|
+
if (owner.snapshotHash !== snapshotHash) throw new JobError('JOB_SOURCE_MISMATCH');
|
|
295
|
+
const reading = this.db.query<{ present: number }, [string]>(
|
|
296
|
+
'SELECT 1 AS present FROM job_snapshot_reads WHERE snapshot_root=? LIMIT 1').get(owner.snapshotRoot);
|
|
297
|
+
if (current.active || finalized(current.data) || reading ||
|
|
298
|
+
this.snapshotReferenced(owner.snapshotRoot, {jobId, publicationId})) return null;
|
|
299
|
+
this.db.query(`INSERT OR IGNORE INTO job_snapshot_tombstones
|
|
300
|
+
(job_id,publication_id,snapshot_hash,reclaimed_at,reason) VALUES(?,?,?,?,?)`)
|
|
301
|
+
.run(jobId, publicationId, snapshotHash, new Date().toISOString(), 'closed');
|
|
302
|
+
return owner;
|
|
303
|
+
});
|
|
304
|
+
}
|
|
305
|
+
|
|
178
306
|
close(): void {
|
|
179
307
|
this.db.close();
|
|
180
308
|
}
|
package/src/reference.ts
CHANGED
|
@@ -50,7 +50,7 @@ export async function safeSelectedReference(p:ReviewPackage):Promise<string>{
|
|
|
50
50
|
/** Hash and serve the same opened file, bounded to 64 KiB buffers; never reopen a checked path. */
|
|
51
51
|
export async function serveReference(req:Request,path:string,sha256:string,base:Record<string,string>):Promise<Response>{
|
|
52
52
|
const file=await open(path,constants.O_RDONLY|constants.O_NOFOLLOW).catch(()=>{throw new Error('Reference inputs changed or unavailable. Restart review to bind current video bytes.');});
|
|
53
|
-
let
|
|
53
|
+
let closing:Promise<void>|undefined;const close=()=>closing??=file.close();
|
|
54
54
|
try{
|
|
55
55
|
const before=await file.stat();if(!before.isFile())throw new Error('Reference inputs changed: expected regular video');
|
|
56
56
|
const hash=createHash('sha256'),buffer=Buffer.alloc(65536);let position=0;
|
package/web/catalog-en.ts
CHANGED
|
@@ -3377,6 +3377,7 @@ export const enCatalog: Readonly<Record<string, string>> = {
|
|
|
3377
3377
|
"JOB_PRELOAD_CANCELLED": "JOB_PRELOAD_CANCELLED",
|
|
3378
3378
|
"JOB_PRELOAD_FAILED": "JOB_PRELOAD_FAILED",
|
|
3379
3379
|
"JOB_PUBLICATION_UNKNOWN": "JOB_PUBLICATION_UNKNOWN",
|
|
3380
|
+
"JOB_SNAPSHOT_RECLAIMED": "JOB_SNAPSHOT_RECLAIMED",
|
|
3380
3381
|
"JOB_RECORD_INCOMPLETE": "JOB_RECORD_INCOMPLETE",
|
|
3381
3382
|
"JOB_RECORD_MOVED": "JOB_RECORD_MOVED",
|
|
3382
3383
|
"JOB_REQUEST_INVALID": "JOB_REQUEST_INVALID",
|
package/web/catalog-ko.ts
CHANGED
|
@@ -3377,6 +3377,7 @@ export const koCatalog: Readonly<Record<string, string>> = {
|
|
|
3377
3377
|
"JOB_PRELOAD_CANCELLED": "JOB_PRELOAD_CANCELLED",
|
|
3378
3378
|
"JOB_PRELOAD_FAILED": "JOB_PRELOAD_FAILED",
|
|
3379
3379
|
"JOB_PUBLICATION_UNKNOWN": "JOB_PUBLICATION_UNKNOWN",
|
|
3380
|
+
"JOB_SNAPSHOT_RECLAIMED": "JOB_SNAPSHOT_RECLAIMED",
|
|
3380
3381
|
"JOB_RECORD_INCOMPLETE": "JOB_RECORD_INCOMPLETE",
|
|
3381
3382
|
"JOB_RECORD_MOVED": "JOB_RECORD_MOVED",
|
|
3382
3383
|
"JOB_REQUEST_INVALID": "JOB_REQUEST_INVALID",
|