@hraness/sponge 0.1.2 → 0.2.1

package/README.md

@@ -1,30 +1,177 @@
 # Sponge
 
-Sponge is being rebuilt as a native app. The app is not available yet.
+Sponge is a local library for articles, images, PDFs and media transcripts. Capturing runs on your machine through Ghostget; saved
+content stays in a private SQLite database and adjacent hashed asset files.
 
-This repository contains the [sponge.computer](https://sponge.computer) website
-and an intentionally inactive `sponge` command in `@hraness/sponge`. Running
-the command exits successfully without output or side effects.
+Version 0.2.0 introduces the local-library CLI; earlier 0.1.x commands are inactive.
+The native desktop app and cloud sync are not available yet.
+The website is [sponge.computer](https://sponge.computer). Previous research lives
+at [`hraness/sponge-research`](https://github.com/hraness/sponge-research).
 
-The previous research project continues at
-[`hraness/sponge-research`](https://github.com/hraness/sponge-research).
+## CLI
 
-## Development
+Node.js 24.20 or newer is required. Capture currently targets Apple Silicon
+macOS; read and queue interfaces remain portable.
+
+Install the released package with `npm install -g @hraness/sponge`.
+For development from this checkout:
 
 ```sh
-cd site
 npm ci
-npm run dev
+npm run build
+node bin/sponge.js init
+node bin/sponge.js setup --capture
+node bin/sponge.js add https://example.com/article --wait
+node bin/sponge.js list
+node bin/sponge.js show ITEM
+node bin/sponge.js search 'your query'
 ```
 
-The package has no runtime dependencies. Run `node bin/sponge.js` to check its
-current behavior. Run `npm run build` in `site/` to verify the website.
+The default library is `~/.local/share/sponge`. Use `--library /absolute/path`
+for another private library. `add` without `--wait` queues a link; `run --drain`
+processes queued links sequentially. `refresh ITEM --wait` attempts a new
+revision and preserves the old one if capture fails. `--json` provides structured
+output. Errors are categorical JSON on stderr; exit codes are 0 success,
+1 failure, 2 invalid input, 3 partial content, 4 action required, and 130 cancelled.
+
+Capture setup explicitly downloads pinned Ghostget, Wordcell and Bun artifacts
+plus frozen dependencies. Read/search never download runtimes or contact source
+websites. Setup reports disk requirements, a monitored 2 GiB temporary installation
+limit, a 1 GiB retained runtime limit and a 30-minute dependency installation
+deadline. Temporary dependency caches are removed
+before the runtime is promoted.
+`doctor` reports runtime integrity and pending cleanup. Private-network capture
+requires `--allow-private-network` with `--wait` or `run --drain`.
+
+Images use `sponge-asset:SHA256` references. `asset ITEM SHA256 > image.png`
+retrieves verified bytes offline; `asset ITEM SHA256 --json` returns its metadata.
+Unresolved remote images become unavailable markers, and captures retain partial
+status. `delete ITEM` hides content and removes it from search. `queue status`,
+`queue cancel JOB`, and `queue retry JOB` manage work. After an interruption,
+`queue recover` verifies that the old worker and its subprocesses have stopped
+before making work available again. Uncertain cleanup remains blocked.
+
+### PDFs and media
+
+```sh
+node bin/sponge.js add /absolute/path/paper.pdf --kind pdf --wait
+node bin/sponge.js add https://example.com/episode --kind media --wait
+```
+
+PDF capture requires Poppler (`pdfinfo` and `pdftohtml`). Tesseract enables
+optional local OCR. Provider media acquisition requires yt-dlp; audio validation
+requires ffprobe from FFmpeg. Install these separately, for example with
+`brew install poppler tesseract yt-dlp ffmpeg` on macOS. Capture setup does not
+install system tools.
+
+PDF capture retains the original and readable Markdown with page references.
+Wordcell owns extraction and reports uncertain layout. Media capture prefers
+available captions and retains timestamped transcripts, metadata and thumbnails.
+Temporary audio is deleted after transcription and verified cleanup.
+
+Local inference requires Apple Silicon and macOS 26 or newer. Release packages
+include a manifest for optional native downloads; a source checkout without that
+manifest reports `runtime-missing`. Installation never downloads models.
+
+Native helpers are distributed from the private `hraness/sponge` repository.
+For explicit native setup, provide `GH_TOKEN` (or `GITHUB_TOKEN`) with read access
+to that repository's contents. Sponge never saves the token and sends it only to
+GitHub's API; public model downloads do not receive it. With an authorized GitHub
+CLI session, for example: `GH_TOKEN="$(gh auth token)" sponge setup --asr`.
+Repeat with `--embedding` for search. Already verified caches work without a token.
+
+```sh
+sponge setup --asr
+sponge add https://example.com/episode --kind media --wait --language en
+sponge refresh ITEM --wait --language en --force-local
+sponge setup --embedding
+sponge index create
+sponge index run
+sponge index status GENERATION
+sponge search 'your query'
+```
+
+Local ASR currently supports English. `--force-local` bypasses provider captions.
+Language choices apply to the current `--wait`, `run --drain`, or `worker serve`
+invocation; queued-only submissions reject these flags. `worker install --language en`
+saves the English ASR choice for background jobs. Search combines local
+full-text and semantic results when a compatible index and model are available,
+and reports when semantic search is unavailable. `--mode lexical` uses only
+full-text search. Use `index create --profile NAME` and `search QUERY --profile NAME`
+to keep separate search profiles. Model changes create separate index generations; a new one
+becomes active only after it covers the current library. `index rollback
+GENERATION` requires a complete generation compatible with the current revisions.
+
+### Export, recovery and background work
+
+```sh
+sponge export ITEM --output /absolute/new/export-directory
+sponge backup --output /absolute/new/library.sponge
+sponge restore /absolute/library.sponge --into /absolute/new/library
+sponge gc --dry-run
+sponge gc --apply /absolute/path/to/report.json
+sponge worker install --language en
+sponge worker status
+sponge worker uninstall
+```
+
+Exports contain Markdown, images and saved notes. Backups preserve revisions,
+assets, reading state and notes. Restore requires a new directory, verifies every
+file and leaves connections unresolved; runtimes and credentials are excluded.
+Garbage collection removes only unreferenced blobs after rechecking the report.
+
+The background worker is opt-in and processes work sequentially. Its LaunchAgent
+pins the installed command; after an upgrade, reinstall the worker to update that
+binding. Uninstall stops the worker and preserves the library. In-session startup
+and cleanup are checked; fresh-login startup remains unqualified. See
+[importing existing captures](docs/importing.md) for a preview-first import flow.
+
+### Selected authenticated connections
+
+The initial development interface accepts private `connections.json` in the
+library (mode `0600`). It contains references to existing Ghostget accounts,
+never cookies or tokens. A future setup phase adds connection-management commands.
+
+```json
+{"schemaVersion":1,"connections":[{"id":"personal","kind":"account","stateHome":"/absolute/ghostget-auth","accountId":"personal"}]}
+```
+
+Use `add URL --connection personal --wait`. A public capture never silently
+selects your browser or account. For articles, attached browser connections use
+`kind: "cdp"` and an explicitly selected local `endpoint`; attached browsers remain
+open. PDF and media capture support selected Ghostget accounts, not CDP. Remote
+PDF acquisition requires public network addresses; `--allow-private-network` is
+not supported for PDF capture. Live authenticated remote-PDF qualification remains
+pending; failures never fall back to a different account or a public request.
+Changing connection identity creates a distinct private source realm.
+
+## Design and validation
+
+See the [design](docs/local-library-design.md),
+[phase plan](plans/local-library-cli.md), and
+[evidence inventory](docs/local-library-evidence.md). PDF improvements belong
+upstream in Wordcell; Sponge owns library ingestion and offline reading.
+
+```sh
+npm run typecheck
+npm test
+npm run build
+npm run verify:package
+```
+
+Use the repository's host scheduler for required local resource lanes. The
+installed capture qualification also requires a separately started controlled
+fixture server, its allocated `PORT`, and an installed `SPONGE_INSTALLED_CLI`:
+`npm run verify:steel-thread -- --installed`. It tests public/authenticated
+capture, restart, offline read/search/assets and failed refresh preservation.
+
+The website lives in `site/`: `npm ci`, `npm run dev`, and `npm run build` there.
 
 ## Releases
 
-After a version bump in `package.json` reaches `main` with green CI, the
-release tagger creates an annotated `v<version>` tag. The release workflow
-checks that tag against the exact `main` commit, packs and installs the CLI,
-then publishes immutable GitHub Release assets. The same verified tarball is
-published to npm through a trusted publisher tied to this repository's
-`Release` workflow. The repository is private, so npm provenance is disabled.
+After a version bump reaches protected `main` with green CI, the release tagger
+creates an annotated version tag. Release CI builds, packs and installs the
+exact artifact, then publishes immutable GitHub Release assets. The same verified
+tarball is mirrored to npm through trusted publishing. No runtime installation
+runs as an npm lifecycle hook. The repository is private, so npm provenance is
+disabled. The website stays out of the package.

package/bin/sponge.js

@@ -1,3 +1,10 @@
 #!/usr/bin/env node
-
-// Reserved for the future native app. Intentionally no output or effects.
+import { createRequire } from 'node:module';
+const [major,minor]=process.versions.node.split('.').map(Number);
+if(major<24||(major===24&&minor<20)){
+  process.stderr.write('Sponge requires Node.js 24.20 or newer.\n');process.exitCode=2;
+}else{
+  const {version}=createRequire(import.meta.url)('../package.json');
+  const {main}=await import('../dist/src/cli/main.js');
+  process.exitCode=await main(process.argv.slice(2),version);
+}

package/dist/contracts/index.d.ts

@@ -0,0 +1,164 @@
+/** Version 1 public boundaries, frozen at G00. No browser credentials cross these interfaces. */
+export type Sha256 = string;
+export interface Envelope {
+    schemaVersion: 1;
+    requestId: string;
+}
+export declare const exitCodes: {
+    readonly success: 0;
+    readonly failure: 1;
+    readonly 'invalid-input': 2;
+    readonly partial: 3;
+    readonly 'action-required': 4;
+    readonly cancelled: 130;
+};
+export type ErrorCode = 'grant-permission' | 'repair-auth' | 'unsupported' | 'unavailable' | 'rate-limited' | 'partial' | 'cleanup-required' | 'cancelled' | 'invalid-input' | 'unclassified' | 'integrity' | 'limit-exceeded' | 'runtime-missing' | 'schema-newer' | 'owner-busy';
+export interface ErrorEnvelope extends Envelope {
+    ok: false;
+    code: ErrorCode;
+    retryable: boolean;
+    retryAfterMs?: number;
+    /** Categorical, never upstream prose, source content, or credentials. */
+    phase?: 'setup' | 'acquisition' | 'verification' | 'storage' | 'inference' | 'cleanup';
+}
+export type MetadataWarning = 'input-limit' | 'claim-limit' | 'value-limit' | 'json-limit' | 'json-invalid';
+export interface MetadataClaim {
+    field: string;
+    value: string | string[];
+    source: 'html' | 'opengraph' | 'twitter' | 'jsonld' | 'pdf' | 'media';
+    observedAt: string;
+    rule?: string;
+    timezoneKnown?: boolean;
+    observed?: string | string[];
+    locator?: string;
+    selected?: boolean;
+    extractionVersion?: string;
+    datePrecision?: 'date' | 'datetime' | 'invalid';
+}
+export interface Asset {
+    sha256: Sha256;
+    relativePath: string;
+    mediaType: string;
+    bytes: number;
+}
+export interface CaptureResult extends Envelope {
+    kind: 'article' | 'pdf';
+    status: 'complete' | 'partial';
+    sourceUrl: string;
+    markdown: Asset;
+    assets: Asset[];
+    sourcePdf?: Asset;
+    metadata: MetadataClaim[];
+    metadataWarnings?: MetadataWarning[];
+    warnings: ErrorCode[];
+    extractor: {
+        name: string;
+        version: string;
+    };
+    manifestSha256: Sha256;
+    completeness: {
+        text: 'complete' | 'partial';
+        images: 'complete' | 'partial' | 'not-requested';
+    };
+}
+export interface Transcript extends Envelope {
+    source: 'human-captions' | 'automatic-captions' | 'local-asr';
+    language: string;
+    modelFingerprint?: Sha256;
+    segments: {
+        startMs: number;
+        endMs: number;
+        text: string;
+    }[];
+    cleanup: 'settled' | 'pending';
+}
+export interface ModelManifest extends Envelope {
+    id: string;
+    version: string;
+    license: string;
+    runtime: {
+        name: string;
+        version: string;
+        sha256: Sha256;
+    };
+    artifacts: {
+        url: string;
+        sha256: Sha256;
+        bytes: number;
+    }[];
+    platforms: {
+        os: string;
+        arch: string;
+        minimumOs: string;
+    }[];
+}
+export interface EmbeddingIdentity {
+    modelSha256: Sha256;
+    runtimeSha256: Sha256;
+    tokenizerSha256: Sha256;
+    dimensions: number;
+    modalities: ('text' | 'image')[];
+    normalized: boolean;
+    metric: 'cosine';
+    pooling: string;
+    quantization: string;
+    backend: string;
+    queryTemplateSha256: Sha256;
+    documentTemplateSha256: Sha256;
+    preprocessingSha256: Sha256;
+}
+export interface EmbeddingSpace extends Envelope, EmbeddingIdentity {
+    fingerprint: Sha256;
+}
+export interface JobCheckpoint extends Envelope {
+    jobId: string;
+    attemptId: string;
+    ownerIncarnation: string;
+    fence: number;
+    stage: 'queued' | 'acquiring' | 'extracting' | 'indexing' | 'promoting' | 'cleanup' | 'settled';
+    status: 'pending' | 'running' | 'action-required' | 'cancelled' | 'failed' | 'complete';
+    artifacts: Asset[];
+    cleanup: 'settled' | 'pending';
+}
+export interface HelperBounds extends Envelope {
+    deadlineMs: number;
+    maxInputBytes: number;
+    maxOutputBytes: number;
+}
+/** Data-only file reference. The parent separately proves ownership and byte identity. */
+export interface HelperFile {
+    ownedPath: string;
+    sha256: Sha256;
+    bytes: number;
+}
+export type HelperRequest = HelperBounds & ({
+    operation: 'transcribe';
+    input: HelperFile;
+} | {
+    operation: 'embed-image';
+    input: HelperFile;
+} | {
+    operation: 'embed-text';
+    input: {
+        text: string;
+    };
+} | {
+    operation: 'cancel';
+    targetRequestId: string;
+});
+export declare const limits: Readonly<{
+    pdfBytes: number;
+    imageBytes: number;
+    imagePixels: 40000000;
+    imagesPerItem: 200;
+    mediaDurationMs: 3600000;
+    temporaryMediaBytes: number;
+    stagingBytes: number;
+    freeReserveBytes: number;
+    metadataBytes: number;
+    metadataClaims: 512;
+    markdownBytes: number;
+    captureDeadlineMs: 120000;
+    helperFrameBytes: number;
+    helperOutputBytes: number;
+}>;

package/dist/contracts/index.js

@@ -0,0 +1,6 @@
+export const exitCodes = { success: 0, failure: 1, 'invalid-input': 2, partial: 3, 'action-required': 4, cancelled: 130 };
+export const limits = Object.freeze({ pdfBytes: 100 * 1024 ** 2, imageBytes: 20 * 1024 ** 2,
+    imagePixels: 40_000_000, imagesPerItem: 200, mediaDurationMs: 3_600_000,
+    temporaryMediaBytes: 1024 ** 3, stagingBytes: 2 * 1024 ** 3, freeReserveBytes: 2 * 1024 ** 3,
+    metadataBytes: 1024 ** 2, metadataClaims: 512, markdownBytes: 16 * 1024 ** 2,
+    captureDeadlineMs: 120_000, helperFrameBytes: 64 * 1024, helperOutputBytes: 16 * 1024 ** 2 });

package/dist/contracts/validate.d.ts

@@ -0,0 +1,14 @@
+import type { Asset, MetadataClaim, MetadataWarning, CaptureResult, EmbeddingIdentity, ErrorEnvelope, ModelManifest, JobCheckpoint, HelperRequest } from './index.js';
+/** Lexical confinement only. Consumers MUST additionally no-follow open/hash the bytes. */
+export declare function validateAsset(value: unknown): asserts value is Asset;
+export declare function validateCapture(value: unknown): asserts value is CaptureResult;
+export declare function validateMetadataWarnings(value: unknown): asserts value is MetadataWarning[];
+export declare function validateMetadataClaim(claim: unknown): asserts claim is MetadataClaim;
+export declare function validateError(value: unknown): asserts value is ErrorEnvelope;
+/** Fixed ordered tuple avoids key-order or irrelevant envelope fields changing a space. */
+export declare function embeddingFingerprint(identity: EmbeddingIdentity): string;
+/** Manifest validation is admission metadata, not proof of downloaded bytes. */
+export declare function validateModelManifest(value: unknown): asserts value is ModelManifest;
+/** Structural validation; storage separately enforces current-owner/fence transitions. */
+export declare function validateJobCheckpoint(value: unknown): asserts value is JobCheckpoint;
+export declare function validateHelperRequest(value: unknown): asserts value is HelperRequest;

package/dist/contracts/validate.js

@@ -0,0 +1,187 @@
+import { isAbsolute } from 'node:path';
+import { createHash } from 'node:crypto';
+import { limits } from './index.js';
+const digest = /^[a-f0-9]{64}$/;
+function requireValue(ok) { if (!ok)
+    throw new Error('invalid-contract'); }
+function record(value) { requireValue(value !== null && typeof value === 'object' && !Array.isArray(value)); }
+function boundedString(value, max = 4096) { requireValue(typeof value === 'string' && value.length > 0 && value.length <= max && !value.includes('\0')); }
+function onlyKeys(value, keys) { requireValue(Object.keys(value).every(key => keys.includes(key))); }
+function envelope(value) { requireValue(value.schemaVersion === 1); boundedString(value.requestId, 128); }
+/** Lexical confinement only. Consumers MUST additionally no-follow open/hash the bytes. */
+export function validateAsset(value) {
+    record(value);
+    requireValue(typeof value.sha256 === 'string' && digest.test(value.sha256));
+    boundedString(value.relativePath);
+    requireValue(!value.relativePath.startsWith('/') && !value.relativePath.includes('\\') && !value.relativePath.includes(':') && value.relativePath.split('/').every(p => p !== '' && p !== '.' && p !== '..'));
+    boundedString(value.mediaType, 128);
+    requireValue(Number.isSafeInteger(value.bytes) && Number(value.bytes) >= 0);
+}
+export function validateCapture(value) {
+    record(value);
+    envelope(value);
+    requireValue(value.kind === 'article' || value.kind === 'pdf');
+    requireValue(value.status === 'complete' || value.status === 'partial');
+    boundedString(value.sourceUrl, 16384);
+    requireValue(typeof value.manifestSha256 === 'string' && digest.test(value.manifestSha256));
+    validateAsset(value.markdown);
+    requireValue(value.markdown.bytes <= limits.markdownBytes && value.markdown.mediaType === 'text/markdown');
+    requireValue(Array.isArray(value.assets) && value.assets.length <= limits.imagesPerItem);
+    value.assets.forEach(asset => { validateAsset(asset); requireValue(asset.mediaType.startsWith('image/') && asset.bytes <= limits.imageBytes); });
+    if (value.sourcePdf !== undefined) {
+        validateAsset(value.sourcePdf);
+        requireValue(value.sourcePdf.mediaType === 'application/pdf' && value.sourcePdf.bytes <= limits.pdfBytes);
+    }
+    requireValue(value.kind !== 'pdf' || value.sourcePdf !== undefined);
+    record(value.completeness);
+    requireValue(typeof value.completeness.text === 'string' && ['complete', 'partial'].includes(String(value.completeness.text)));
+    requireValue(typeof value.completeness.images === 'string' && ['complete', 'partial', 'not-requested'].includes(String(value.completeness.images)));
+    requireValue(value.status !== 'complete' || (value.completeness.text === 'complete' && value.completeness.images !== 'partial'));
+    record(value.extractor);
+    boundedString(value.extractor.name, 128);
+    boundedString(value.extractor.version, 128);
+    requireValue(Array.isArray(value.warnings) && value.warnings.length <= 128);
+    value.warnings.forEach(v => requireValue(errorCodes.has(v)));
+    requireValue(Array.isArray(value.metadata) && value.metadata.length <= limits.metadataClaims && Buffer.byteLength(JSON.stringify(value.metadata)) <= limits.metadataBytes);
+    value.metadata.forEach(validateMetadataClaim);
+    if (value.metadataWarnings !== undefined)
+        validateMetadataWarnings(value.metadataWarnings);
+}
+export function validateMetadataWarnings(value) {
+    requireValue(Array.isArray(value) && value.length <= 5 && new Set(value).size === value.length && value.every(v => typeof v === 'string' && ['input-limit', 'claim-limit', 'value-limit', 'json-limit', 'json-invalid'].includes(v)));
+}
+export function validateMetadataClaim(claim) {
+    record(claim);
+    boundedString(claim.field, 128);
+    boundedString(claim.observedAt, 64);
+    requireValue(Number.isFinite(Date.parse(claim.observedAt)));
+    requireValue(typeof claim.source === 'string' && ['html', 'opengraph', 'twitter', 'jsonld', 'pdf', 'media'].includes(String(claim.source)));
+    if (Array.isArray(claim.value)) {
+        requireValue(claim.value.length <= 128);
+        claim.value.forEach(v => boundedString(v, 16384));
+    }
+    else
+        boundedString(claim.value, 16384);
+    if (claim.rule !== undefined)
+        boundedString(claim.rule, 256);
+    if (claim.timezoneKnown !== undefined)
+        requireValue(typeof claim.timezoneKnown === 'boolean');
+    if (claim.observed !== undefined) {
+        if (Array.isArray(claim.observed)) {
+            requireValue(claim.observed.length <= 128);
+            claim.observed.forEach(v => boundedString(v, 16384));
+        }
+        else
+            boundedString(claim.observed, 16384);
+    }
+    if (claim.locator !== undefined)
+        boundedString(claim.locator, 512);
+    if (claim.extractionVersion !== undefined)
+        boundedString(claim.extractionVersion, 128);
+    if (claim.selected !== undefined)
+        requireValue(typeof claim.selected === 'boolean');
+    if (claim.datePrecision !== undefined)
+        requireValue(typeof claim.datePrecision === 'string' && ['date', 'datetime', 'invalid'].includes(claim.datePrecision));
+}
+const errorCodes = new Set(['grant-permission', 'repair-auth', 'unsupported', 'unavailable', 'rate-limited', 'partial', 'cleanup-required', 'cancelled', 'invalid-input', 'unclassified', 'integrity', 'limit-exceeded', 'runtime-missing', 'schema-newer', 'owner-busy']);
+export function validateError(value) {
+    record(value);
+    envelope(value);
+    onlyKeys(value, ['schemaVersion', 'requestId', 'ok', 'code', 'retryable', 'retryAfterMs', 'phase']);
+    requireValue(value.ok === false && typeof value.code === 'string' && errorCodes.has(value.code) && typeof value.retryable === 'boolean');
+    requireValue(!value.retryable || value.code === 'unavailable' || value.code === 'rate-limited');
+    if (value.phase !== undefined)
+        requireValue(typeof value.phase === 'string' && ['setup', 'acquisition', 'verification', 'storage', 'inference', 'cleanup'].includes(String(value.phase)));
+    if (value.retryAfterMs !== undefined)
+        requireValue(Number.isSafeInteger(value.retryAfterMs) && Number(value.retryAfterMs) >= 0 && Number(value.retryAfterMs) <= 86_400_000);
+}
+/** Fixed ordered tuple avoids key-order or irrelevant envelope fields changing a space. */
+export function embeddingFingerprint(identity) {
+    const hashes = [identity.modelSha256, identity.runtimeSha256, identity.tokenizerSha256, identity.queryTemplateSha256, identity.documentTemplateSha256, identity.preprocessingSha256];
+    requireValue(hashes.every(h => typeof h === 'string' && digest.test(h)));
+    requireValue(Number.isSafeInteger(identity.dimensions) && identity.dimensions > 0 && identity.dimensions <= 16384);
+    requireValue(identity.metric === 'cosine' && typeof identity.normalized === 'boolean');
+    requireValue(Array.isArray(identity.modalities) && identity.modalities.length > 0 && identity.modalities.every(m => m === 'text' || m === 'image') && new Set(identity.modalities).size === identity.modalities.length);
+    [identity.pooling, identity.quantization, identity.backend].forEach(v => boundedString(v, 256));
+    return createHash('sha256').update(JSON.stringify(['sponge-space-v1', ...hashes, identity.dimensions, [...identity.modalities].sort(), identity.normalized, identity.metric, identity.pooling, identity.quantization, identity.backend])).digest('hex');
+}
+/** Manifest validation is admission metadata, not proof of downloaded bytes. */
+export function validateModelManifest(value) {
+    record(value);
+    envelope(value);
+    boundedString(value.id, 128);
+    boundedString(value.version, 128);
+    boundedString(value.license, 256);
+    record(value.runtime);
+    boundedString(value.runtime.name, 128);
+    boundedString(value.runtime.version, 128);
+    requireValue(typeof value.runtime.sha256 === 'string' && digest.test(value.runtime.sha256));
+    requireValue(Array.isArray(value.artifacts) && value.artifacts.length > 0 && value.artifacts.length <= 128);
+    const urls = new Set();
+    let bytes = 0;
+    for (const artifact of value.artifacts) {
+        record(artifact);
+        boundedString(artifact.url, 16384);
+        const url = new URL(artifact.url);
+        requireValue(url.protocol === 'https:' && !url.username && !url.password && !url.hash && !urls.has(url.href));
+        urls.add(url.href);
+        requireValue(typeof artifact.sha256 === 'string' && digest.test(artifact.sha256));
+        requireValue(Number.isSafeInteger(artifact.bytes) && Number(artifact.bytes) > 0);
+        bytes += Number(artifact.bytes);
+        requireValue(Number.isSafeInteger(bytes));
+    }
+    requireValue(Array.isArray(value.platforms) && value.platforms.length > 0 && value.platforms.length <= 32);
+    for (const platform of value.platforms) {
+        record(platform);
+        boundedString(platform.os, 64);
+        boundedString(platform.arch, 64);
+        boundedString(platform.minimumOs, 64);
+    }
+}
+/** Structural validation; storage separately enforces current-owner/fence transitions. */
+export function validateJobCheckpoint(value) {
+    record(value);
+    envelope(value);
+    for (const key of ['jobId', 'attemptId', 'ownerIncarnation'])
+        boundedString(value[key], 128);
+    requireValue(Number.isSafeInteger(value.fence) && Number(value.fence) > 0);
+    requireValue(typeof value.stage === 'string' && ['queued', 'acquiring', 'extracting', 'indexing', 'promoting', 'cleanup', 'settled'].includes(String(value.stage)));
+    requireValue(typeof value.status === 'string' && ['pending', 'running', 'action-required', 'cancelled', 'failed', 'complete'].includes(String(value.status)));
+    requireValue(value.cleanup === 'settled' || value.cleanup === 'pending');
+    requireValue(value.stage !== 'settled' || value.cleanup === 'settled');
+    requireValue(value.status !== 'complete' || value.stage === 'settled');
+    requireValue(value.stage !== 'queued' || value.status === 'pending');
+    requireValue(Array.isArray(value.artifacts) && value.artifacts.length <= 4096);
+    value.artifacts.forEach(validateAsset);
+}
+export function validateHelperRequest(value) {
+    record(value);
+    envelope(value);
+    requireValue(Buffer.byteLength(JSON.stringify(value)) <= limits.helperFrameBytes);
+    requireValue(Number.isSafeInteger(value.deadlineMs) && Number(value.deadlineMs) > 0 && Number(value.deadlineMs) <= 86_400_000);
+    requireValue(Number.isSafeInteger(value.maxInputBytes) && Number(value.maxInputBytes) > 0 && Number(value.maxInputBytes) <= limits.temporaryMediaBytes);
+    requireValue(Number.isSafeInteger(value.maxOutputBytes) && Number(value.maxOutputBytes) > 0 && Number(value.maxOutputBytes) <= limits.helperOutputBytes);
+    const keys = ['schemaVersion', 'requestId', 'operation', 'deadlineMs', 'maxInputBytes', 'maxOutputBytes'];
+    if (value.operation === 'cancel') {
+        onlyKeys(value, [...keys, 'targetRequestId']);
+        boundedString(value.targetRequestId, 128);
+        requireValue(value.targetRequestId !== value.requestId);
+        return;
+    }
+    onlyKeys(value, [...keys, 'input']);
+    record(value.input);
+    if (value.operation === 'embed-text') {
+        onlyKeys(value.input, ['text']);
+        boundedString(value.input.text, limits.markdownBytes);
+        requireValue(Buffer.byteLength(value.input.text) <= Number(value.maxInputBytes));
+        return;
+    }
+    requireValue(value.operation === 'transcribe' || value.operation === 'embed-image');
+    onlyKeys(value.input, ['ownedPath', 'sha256', 'bytes']);
+    boundedString(value.input.ownedPath);
+    requireValue(isAbsolute(value.input.ownedPath));
+    requireValue(typeof value.input.sha256 === 'string' && digest.test(value.input.sha256));
+    requireValue(Number.isSafeInteger(value.input.bytes) && Number(value.input.bytes) > 0 && Number(value.input.bytes) <= Number(value.maxInputBytes));
+    if (value.operation === 'embed-image')
+        requireValue(Number(value.input.bytes) <= limits.imageBytes);
+}

package/dist/migrations/001.d.ts

@@ -0,0 +1,3 @@
+/** Immutable schema migration. Add a new numbered migration after this ships. */
+export declare const version = 1;
+export declare const sql = "\nCREATE TABLE schema_migrations(version INTEGER PRIMARY KEY, checksum TEXT NOT NULL, applied_at TEXT NOT NULL) STRICT;\nCREATE TABLE library_info(id INTEGER PRIMARY KEY CHECK(id=1), uuid TEXT NOT NULL, schema_version INTEGER NOT NULL, created_by TEXT NOT NULL, revision_epoch INTEGER NOT NULL DEFAULT 0 CHECK(revision_epoch>=0)) STRICT;\nCREATE TABLE items(id TEXT PRIMARY KEY, kind TEXT NOT NULL CHECK(kind IN ('article','pdf','media')), title TEXT NOT NULL DEFAULT '', created_at TEXT NOT NULL, current_revision TEXT REFERENCES revisions(id), deleted_at TEXT) STRICT;\nCREATE TABLE source_refs(id TEXT PRIMARY KEY, item_id TEXT NOT NULL REFERENCES items(id), source_key TEXT NOT NULL, realm_key TEXT NOT NULL DEFAULT '', locator TEXT NOT NULL, final_url TEXT, canonical_claim TEXT, provider_id TEXT, UNIQUE(source_key,realm_key)) STRICT;\nCREATE TABLE jobs(id TEXT PRIMARY KEY, item_id TEXT NOT NULL REFERENCES items(id), operation TEXT NOT NULL DEFAULT 'capture', state TEXT NOT NULL CHECK(state IN ('queued','running','delayed','needs-permission','needs-auth','unsupported','failed','succeeded','cancelled','cleanup-pending')), priority INTEGER NOT NULL DEFAULT 0, not_before TEXT, attempt_count INTEGER NOT NULL DEFAULT 0, cancel_requested INTEGER NOT NULL DEFAULT 0 CHECK(cancel_requested IN (0,1)), reason TEXT, created_at TEXT NOT NULL) STRICT;\nCREATE INDEX jobs_ready ON jobs(state,not_before,priority);\nCREATE TABLE submissions(id TEXT PRIMARY KEY, idempotency_key TEXT NOT NULL UNIQUE, source_id TEXT NOT NULL REFERENCES source_refs(id), item_id TEXT NOT NULL REFERENCES items(id), job_id TEXT NOT NULL REFERENCES jobs(id), created_at TEXT NOT NULL, intent TEXT NOT NULL) STRICT;\nCREATE TABLE worker_owner(id INTEGER PRIMARY KEY CHECK(id=1), incarnation TEXT, fence INTEGER NOT NULL DEFAULT 0 CHECK(fence>=0)) STRICT;\nINSERT INTO worker_owner(id) VALUES(1);\nCREATE TABLE attempts(id TEXT PRIMARY KEY, job_id TEXT NOT NULL REFERENCES jobs(id), owner_incarnation TEXT NOT NULL, fence INTEGER NOT NULL, stage TEXT NOT NULL, cleanup TEXT NOT NULL CHECK(cleanup IN ('pending','settled')), created_at TEXT NOT NULL, finished_at TEXT) STRICT;\nCREATE INDEX attempts_job ON attempts(job_id);\nCREATE TABLE staging_pins(attempt_id TEXT NOT NULL REFERENCES attempts(id), sha256 TEXT NOT NULL CHECK(length(sha256)=64), bytes INTEGER NOT NULL CHECK(bytes>=0), state TEXT NOT NULL CHECK(state IN ('staging','published')), PRIMARY KEY(attempt_id,sha256)) STRICT;\nCREATE TABLE revisions(id TEXT PRIMARY KEY, item_id TEXT NOT NULL REFERENCES items(id), parent_id TEXT REFERENCES revisions(id), bundle_hash TEXT NOT NULL CHECK(length(bundle_hash)=64), extractor_name TEXT NOT NULL, extractor_version TEXT NOT NULL, captured_at TEXT NOT NULL, status TEXT NOT NULL CHECK(status IN ('complete','partial')), warnings_json TEXT NOT NULL, payload_hash TEXT NOT NULL, UNIQUE(item_id,bundle_hash)) STRICT;\nCREATE TRIGGER immutable_revision BEFORE UPDATE ON revisions BEGIN SELECT RAISE(ABORT,'immutable-revision'); END;\nCREATE TABLE assets(sha256 TEXT PRIMARY KEY CHECK(length(sha256)=64), bytes INTEGER NOT NULL CHECK(bytes>=0), media_type TEXT NOT NULL, relative_path TEXT NOT NULL UNIQUE) STRICT;\nCREATE TABLE revision_assets(revision_id TEXT NOT NULL REFERENCES revisions(id), sha256 TEXT NOT NULL REFERENCES assets(sha256), ordinal INTEGER NOT NULL CHECK(ordinal>=0), relationship TEXT NOT NULL, original_locator TEXT, alt TEXT, caption TEXT, PRIMARY KEY(revision_id,ordinal)) STRICT;\nCREATE TABLE representations(id TEXT PRIMARY KEY, revision_id TEXT NOT NULL REFERENCES revisions(id), kind TEXT NOT NULL, mime_type TEXT NOT NULL, sha256 TEXT NOT NULL, bytes INTEGER NOT NULL CHECK(bytes>=0), markdown TEXT, blob_sha256 TEXT REFERENCES assets(sha256), extractor_version TEXT NOT NULL, ancestry_json TEXT NOT NULL DEFAULT '[]', CHECK((markdown IS NOT NULL AND blob_sha256 IS NULL) OR (markdown IS NULL AND blob_sha256 IS NOT NULL))) STRICT;\nCREATE TABLE metadata_claims(id TEXT PRIMARY KEY, revision_id TEXT NOT NULL REFERENCES revisions(id), field TEXT NOT NULL, value_json TEXT NOT NULL, source TEXT NOT NULL, observed_at TEXT NOT NULL, rule TEXT, timezone_known INTEGER CHECK(timezone_known IN (0,1))) STRICT;\nCREATE TABLE blocks(id TEXT PRIMARY KEY, revision_id TEXT NOT NULL REFERENCES revisions(id), ordinal INTEGER NOT NULL, kind TEXT NOT NULL, text TEXT NOT NULL, heading_path_json TEXT NOT NULL DEFAULT '[]', page INTEGER, bounds_json TEXT, start_ms INTEGER, end_ms INTEGER, asset_sha256 TEXT REFERENCES assets(sha256), UNIQUE(revision_id,ordinal)) STRICT;\nCREATE VIRTUAL TABLE readable_fts USING fts5(item_id UNINDEXED,revision_id UNINDEXED,title,markdown,tokenize='unicode61');\nCREATE TABLE model_artifacts(id TEXT PRIMARY KEY, manifest_json TEXT NOT NULL, sha256 TEXT NOT NULL, bytes INTEGER NOT NULL CHECK(bytes>=0), license TEXT NOT NULL, available INTEGER NOT NULL DEFAULT 0 CHECK(available IN (0,1))) STRICT;\nCREATE TABLE embedding_spaces(fingerprint TEXT PRIMARY KEY, identity_json TEXT NOT NULL, dimensions INTEGER NOT NULL CHECK(dimensions>0), normalization TEXT NOT NULL, metric TEXT NOT NULL) STRICT;\nCREATE TABLE index_generations(id TEXT PRIMARY KEY, fingerprint TEXT NOT NULL REFERENCES embedding_spaces(fingerprint), profile_json TEXT NOT NULL, state TEXT NOT NULL, revision_epoch INTEGER NOT NULL, coverage_json TEXT NOT NULL DEFAULT '{}') STRICT;\nCREATE TABLE vectors(generation_id TEXT NOT NULL REFERENCES index_generations(id), source_id TEXT NOT NULL, revision_id TEXT NOT NULL REFERENCES revisions(id), asset_sha256 TEXT REFERENCES assets(sha256), derivation_hash TEXT NOT NULL, vector BLOB NOT NULL, PRIMARY KEY(generation_id,source_id)) STRICT;\nCREATE TABLE active_indexes(profile_id TEXT PRIMARY KEY, generation_id TEXT NOT NULL REFERENCES index_generations(id)) STRICT;\nCREATE TABLE reading_state(item_id TEXT PRIMARY KEY REFERENCES items(id), state_json TEXT NOT NULL, updated_at TEXT NOT NULL) STRICT;\nCREATE TABLE annotations(id TEXT PRIMARY KEY, item_id TEXT NOT NULL REFERENCES items(id), revision_id TEXT REFERENCES revisions(id), anchor_json TEXT NOT NULL, body TEXT NOT NULL, created_at TEXT NOT NULL) STRICT;\nCREATE TABLE import_receipts(namespace TEXT NOT NULL, source_key TEXT NOT NULL, source_hash TEXT NOT NULL, importer_version TEXT NOT NULL, item_id TEXT REFERENCES items(id), outcome TEXT NOT NULL, PRIMARY KEY(namespace,source_key,source_hash)) STRICT;\nCREATE TABLE backup_fences(id TEXT PRIMARY KEY, revision_epoch INTEGER NOT NULL, state TEXT NOT NULL CHECK(state IN ('active','complete')), created_at TEXT NOT NULL) STRICT;\n";

package/dist/migrations/001.js

@@ -0,0 +1,33 @@
+/** Immutable schema migration. Add a new numbered migration after this ships. */
+export const version = 1;
+export const sql = `
+CREATE TABLE schema_migrations(version INTEGER PRIMARY KEY, checksum TEXT NOT NULL, applied_at TEXT NOT NULL) STRICT;
+CREATE TABLE library_info(id INTEGER PRIMARY KEY CHECK(id=1), uuid TEXT NOT NULL, schema_version INTEGER NOT NULL, created_by TEXT NOT NULL, revision_epoch INTEGER NOT NULL DEFAULT 0 CHECK(revision_epoch>=0)) STRICT;
+CREATE TABLE items(id TEXT PRIMARY KEY, kind TEXT NOT NULL CHECK(kind IN ('article','pdf','media')), title TEXT NOT NULL DEFAULT '', created_at TEXT NOT NULL, current_revision TEXT REFERENCES revisions(id), deleted_at TEXT) STRICT;
+CREATE TABLE source_refs(id TEXT PRIMARY KEY, item_id TEXT NOT NULL REFERENCES items(id), source_key TEXT NOT NULL, realm_key TEXT NOT NULL DEFAULT '', locator TEXT NOT NULL, final_url TEXT, canonical_claim TEXT, provider_id TEXT, UNIQUE(source_key,realm_key)) STRICT;
+CREATE TABLE jobs(id TEXT PRIMARY KEY, item_id TEXT NOT NULL REFERENCES items(id), operation TEXT NOT NULL DEFAULT 'capture', state TEXT NOT NULL CHECK(state IN ('queued','running','delayed','needs-permission','needs-auth','unsupported','failed','succeeded','cancelled','cleanup-pending')), priority INTEGER NOT NULL DEFAULT 0, not_before TEXT, attempt_count INTEGER NOT NULL DEFAULT 0, cancel_requested INTEGER NOT NULL DEFAULT 0 CHECK(cancel_requested IN (0,1)), reason TEXT, created_at TEXT NOT NULL) STRICT;
+CREATE INDEX jobs_ready ON jobs(state,not_before,priority);
+CREATE TABLE submissions(id TEXT PRIMARY KEY, idempotency_key TEXT NOT NULL UNIQUE, source_id TEXT NOT NULL REFERENCES source_refs(id), item_id TEXT NOT NULL REFERENCES items(id), job_id TEXT NOT NULL REFERENCES jobs(id), created_at TEXT NOT NULL, intent TEXT NOT NULL) STRICT;
+CREATE TABLE worker_owner(id INTEGER PRIMARY KEY CHECK(id=1), incarnation TEXT, fence INTEGER NOT NULL DEFAULT 0 CHECK(fence>=0)) STRICT;
+INSERT INTO worker_owner(id) VALUES(1);
+CREATE TABLE attempts(id TEXT PRIMARY KEY, job_id TEXT NOT NULL REFERENCES jobs(id), owner_incarnation TEXT NOT NULL, fence INTEGER NOT NULL, stage TEXT NOT NULL, cleanup TEXT NOT NULL CHECK(cleanup IN ('pending','settled')), created_at TEXT NOT NULL, finished_at TEXT) STRICT;
+CREATE INDEX attempts_job ON attempts(job_id);
+CREATE TABLE staging_pins(attempt_id TEXT NOT NULL REFERENCES attempts(id), sha256 TEXT NOT NULL CHECK(length(sha256)=64), bytes INTEGER NOT NULL CHECK(bytes>=0), state TEXT NOT NULL CHECK(state IN ('staging','published')), PRIMARY KEY(attempt_id,sha256)) STRICT;
+CREATE TABLE revisions(id TEXT PRIMARY KEY, item_id TEXT NOT NULL REFERENCES items(id), parent_id TEXT REFERENCES revisions(id), bundle_hash TEXT NOT NULL CHECK(length(bundle_hash)=64), extractor_name TEXT NOT NULL, extractor_version TEXT NOT NULL, captured_at TEXT NOT NULL, status TEXT NOT NULL CHECK(status IN ('complete','partial')), warnings_json TEXT NOT NULL, payload_hash TEXT NOT NULL, UNIQUE(item_id,bundle_hash)) STRICT;
+CREATE TRIGGER immutable_revision BEFORE UPDATE ON revisions BEGIN SELECT RAISE(ABORT,'immutable-revision'); END;
+CREATE TABLE assets(sha256 TEXT PRIMARY KEY CHECK(length(sha256)=64), bytes INTEGER NOT NULL CHECK(bytes>=0), media_type TEXT NOT NULL, relative_path TEXT NOT NULL UNIQUE) STRICT;
+CREATE TABLE revision_assets(revision_id TEXT NOT NULL REFERENCES revisions(id), sha256 TEXT NOT NULL REFERENCES assets(sha256), ordinal INTEGER NOT NULL CHECK(ordinal>=0), relationship TEXT NOT NULL, original_locator TEXT, alt TEXT, caption TEXT, PRIMARY KEY(revision_id,ordinal)) STRICT;
+CREATE TABLE representations(id TEXT PRIMARY KEY, revision_id TEXT NOT NULL REFERENCES revisions(id), kind TEXT NOT NULL, mime_type TEXT NOT NULL, sha256 TEXT NOT NULL, bytes INTEGER NOT NULL CHECK(bytes>=0), markdown TEXT, blob_sha256 TEXT REFERENCES assets(sha256), extractor_version TEXT NOT NULL, ancestry_json TEXT NOT NULL DEFAULT '[]', CHECK((markdown IS NOT NULL AND blob_sha256 IS NULL) OR (markdown IS NULL AND blob_sha256 IS NOT NULL))) STRICT;
+CREATE TABLE metadata_claims(id TEXT PRIMARY KEY, revision_id TEXT NOT NULL REFERENCES revisions(id), field TEXT NOT NULL, value_json TEXT NOT NULL, source TEXT NOT NULL, observed_at TEXT NOT NULL, rule TEXT, timezone_known INTEGER CHECK(timezone_known IN (0,1))) STRICT;
+CREATE TABLE blocks(id TEXT PRIMARY KEY, revision_id TEXT NOT NULL REFERENCES revisions(id), ordinal INTEGER NOT NULL, kind TEXT NOT NULL, text TEXT NOT NULL, heading_path_json TEXT NOT NULL DEFAULT '[]', page INTEGER, bounds_json TEXT, start_ms INTEGER, end_ms INTEGER, asset_sha256 TEXT REFERENCES assets(sha256), UNIQUE(revision_id,ordinal)) STRICT;
+CREATE VIRTUAL TABLE readable_fts USING fts5(item_id UNINDEXED,revision_id UNINDEXED,title,markdown,tokenize='unicode61');
+CREATE TABLE model_artifacts(id TEXT PRIMARY KEY, manifest_json TEXT NOT NULL, sha256 TEXT NOT NULL, bytes INTEGER NOT NULL CHECK(bytes>=0), license TEXT NOT NULL, available INTEGER NOT NULL DEFAULT 0 CHECK(available IN (0,1))) STRICT;
+CREATE TABLE embedding_spaces(fingerprint TEXT PRIMARY KEY, identity_json TEXT NOT NULL, dimensions INTEGER NOT NULL CHECK(dimensions>0), normalization TEXT NOT NULL, metric TEXT NOT NULL) STRICT;
+CREATE TABLE index_generations(id TEXT PRIMARY KEY, fingerprint TEXT NOT NULL REFERENCES embedding_spaces(fingerprint), profile_json TEXT NOT NULL, state TEXT NOT NULL, revision_epoch INTEGER NOT NULL, coverage_json TEXT NOT NULL DEFAULT '{}') STRICT;
+CREATE TABLE vectors(generation_id TEXT NOT NULL REFERENCES index_generations(id), source_id TEXT NOT NULL, revision_id TEXT NOT NULL REFERENCES revisions(id), asset_sha256 TEXT REFERENCES assets(sha256), derivation_hash TEXT NOT NULL, vector BLOB NOT NULL, PRIMARY KEY(generation_id,source_id)) STRICT;
+CREATE TABLE active_indexes(profile_id TEXT PRIMARY KEY, generation_id TEXT NOT NULL REFERENCES index_generations(id)) STRICT;
+CREATE TABLE reading_state(item_id TEXT PRIMARY KEY REFERENCES items(id), state_json TEXT NOT NULL, updated_at TEXT NOT NULL) STRICT;
+CREATE TABLE annotations(id TEXT PRIMARY KEY, item_id TEXT NOT NULL REFERENCES items(id), revision_id TEXT REFERENCES revisions(id), anchor_json TEXT NOT NULL, body TEXT NOT NULL, created_at TEXT NOT NULL) STRICT;
+CREATE TABLE import_receipts(namespace TEXT NOT NULL, source_key TEXT NOT NULL, source_hash TEXT NOT NULL, importer_version TEXT NOT NULL, item_id TEXT REFERENCES items(id), outcome TEXT NOT NULL, PRIMARY KEY(namespace,source_key,source_hash)) STRICT;
+CREATE TABLE backup_fences(id TEXT PRIMARY KEY, revision_epoch INTEGER NOT NULL, state TEXT NOT NULL CHECK(state IN ('active','complete')), created_at TEXT NOT NULL) STRICT;
+`;

package/dist/migrations/002.d.ts

@@ -0,0 +1,3 @@
+/** Additive retry evidence; schema 1 and its checksum remain immutable. */
+export declare const version = 2;
+export declare const sql = "\nALTER TABLE jobs ADD COLUMN retryable INTEGER NOT NULL DEFAULT 0 CHECK(retryable IN (0,1));\nALTER TABLE jobs ADD COLUMN retry_after_ms INTEGER CHECK(retry_after_ms>=0 AND retry_after_ms<=86400000);\n";