videobook-engine 2.0.1 → 4.2.0

package/docs/engine-layout.md

@@ -1,8 +1,9 @@
 # Videobook engine layout
 
 This document is the column-by-column layout of the engine-owned data model.
-It describes schema version 4 as implemented today, then identifies the
-additional structures needed for a full non-linear video editor.
+It describes the original schema version 4 layout, notes where later schema
+versions changed the picture (currently v18), and identifies the additional
+structures needed for a full non-linear video editor.
 
 The executable source of truth remains
 [`src/schema.ts`](../src/schema.ts). If this document and the DDL disagree, the
@@ -41,7 +42,7 @@ flowchart LR
   OUTBOX --> STAGE
   STAGE --> REV[Dolt main revision]
 
-  FILES[Artifact file APIs] --> CAS[Immutable SHA-256 objects]
+  FILES[Artifact file APIs] --> CAS[SHA-256 objects, forgettable]
   FILES --> MAP[Versioned artifact_files mapping]
   CAS --> WS[Disposable artifact workspace]
   MAP --> WS
@@ -52,15 +53,24 @@ flowchart LR
 
 ## Global invariants
 
-- One engine catalog contains exactly one `book` row and one matching
-  `timeline` row. There is no project scope.
+- One engine catalog contains exactly one `book` row and one primary
+  `sequences` row. There is no project scope.
 - The only supported live Dolt branch is `main`.
 - Semantic mutations are SQL transactions followed by forward-only Dolt
   commits. A restore creates a new commit; it never rewinds the live branch.
-- `SEMANTIC_TABLES` is the staging allowlist. Every mutation also records an
-  `operations` row.
-- `runtime_%` and `sqlite_sequence` are ignored by Dolt. Runtime state can be
-  rebuilt, expired, or invalidated without changing semantic history.
+- `SEMANTIC_TABLES` is the staging allowlist. Every operation declares the
+  semantic tables it may write (`OperationInput.tables`, including ON DELETE
+  CASCADE targets); a commit probes exactly that set with row-level diffs,
+  stages the truly dirty tables, asserts them clean afterwards, and is
+  skipped when only runtime bookkeeping changed. A full-catalog sweep runs
+  once per open and faults on rows no operation attributed (doltStatus
+  itself over-reports and is never trusted for dirtiness). The commit itself
+  is the provenance record: the operation, its parameters, write set, and
+  base revision ride in a structured commit message under the configured
+  identity as `--author`.
+- `runtime_%`, `job_runs`, and `sqlite_sequence` are ignored by Dolt. Runtime
+  state can be rebuilt, expired, or invalidated without changing semantic
+  history.
 - Engine-generated surrogate identities are stable UUIDv7 strings; the SQL
   columns are `TEXT`, so UUID form is enforced by engine APIs rather than a
   database check. Content identity is a lowercase, 64-character SHA-256.
@@ -68,8 +78,16 @@ flowchart LR
   currently modeled use integer frames.
 - JSON is stored as canonical text with recursively sorted object keys.
 - Deletes are hard deletes. Owned rows cascade; live artifact/entity
-  references restrict deletion; immutable objects and prior Dolt revisions
-  remain available.
+  references restrict deletion; prior Dolt revisions remain available.
+- Objects are content-immutable but forgettable. `engine.storage.deleteObject`
+  forgets one object (refusing `IN_USE` references at HEAD unless forced);
+  `engine.storage.gc` sweeps every object nothing references at HEAD.
+  Forgetting is a semantic commit that sets `objects.forgotten_at`; the row
+  is never deleted, so it remains as the tombstone (hash + size + forgotten
+  timestamp) for historical references, and reads of forgotten content
+  surface `OBJECT_UNAVAILABLE` rather than failing or lying. Published
+  history is append-only, so bulk forgettable text lives behind CAS hashes
+  (see "Forgettable data and raw-text audit" below).
 
 ## Dolt-versioned semantic schema
 
@@ -81,16 +99,16 @@ Notation used below:
 - `→` names the referenced column.
 - Defaults and checks are shown inline.
 
-There are 25 allowlisted semantic tables.
+There are 34 allowlisted semantic tables.
 
 ### Catalog, artifacts, and content
 
 | Table | Columns | Keys, constraints, and purpose |
 | --- | --- | --- |
-| `engine_schema` | `singleton INTEGER`<br>`version INTEGER`<br>`created_at INTEGER` | `singleton PK CHECK(singleton = 1)`. Records the clean-break catalog version. Version 4 rejects older catalogs rather than migrating them. |
+| `engine_schema` | `singleton INTEGER`<br>`version INTEGER`<br>`created_at INTEGER` | `singleton PK CHECK(singleton = 1)`. Records the clean-break catalog version. The recorded version rejects older catalogs rather than migrating them. |
 | `book` | `book_id TEXT`<br>`slug TEXT`<br>`created_at INTEGER` | `book_id PK`; `slug UQ`. Exactly one row per engine root. |
 | `artifacts` | `artifact_id TEXT`<br>`slug TEXT`<br>`kind TEXT`<br>`created_at INTEGER` | `artifact_id PK`; `slug UQ`; `kind CHECK IN (video, image, audio, script, character, prompt, scene, final)`. The artifact is the stable identity for source media, generated media, documents, and final renders. |
-| `objects` | `object_hash TEXT`<br>`size_bytes INTEGER`<br>`created_at INTEGER` | `object_hash PK`; `size_bytes CHECK >= 0`. This is versioned metadata for immutable bytes stored outside the database. |
+| `objects` | `object_hash TEXT`<br>`size_bytes INTEGER`<br>`created_at INTEGER`<br>`forgotten_at INTEGER?` | `object_hash PK`; `size_bytes CHECK >= 0`. Versioned metadata for bytes stored outside the database. Rows are append-only: `forgotten_at` marks a tombstone whose bytes were deleted by `deleteObject`/`gc`. |
 | `artifact_files` | `artifact_id TEXT`<br>`path TEXT`<br>`object_hash TEXT`<br>`mtime_ms INTEGER`<br>`created_at INTEGER` | `(artifact_id, path) PK`; `artifact_id FK → artifacts.artifact_id ON DELETE CASCADE`; `object_hash FK → objects.object_hash ON DELETE RESTRICT`. Maps a logical artifact path to immutable content. |
 | `book_metadata` | `key TEXT`<br>`value_json TEXT` | `key PK`. Singleton-book key/value metadata; no redundant `book_id`. |
 | `artifact_metadata` | `artifact_id TEXT`<br>`key TEXT`<br>`value_json TEXT` | `(artifact_id, key) PK`; `artifact_id FK → artifacts.artifact_id ON DELETE CASCADE`. Extensible artifact metadata that participates in revisions. |
@@ -100,36 +118,47 @@ There are 25 allowlisted semantic tables.
 | Table | Columns | Keys, constraints, and purpose |
 | --- | --- | --- |
 | `entities` | `entity_id TEXT`<br>`type TEXT`<br>`name TEXT`<br>`description TEXT?`<br>`prompt TEXT?`<br>`data_json TEXT DEFAULT '{}'`<br>`created_at INTEGER` | `entity_id PK`; `type CHECK IN (prompt, character, scene)`. Normalized reusable creative concepts. |
-| `notebooks` | `notebook_id TEXT`<br>`name TEXT`<br>`properties_json TEXT DEFAULT '{}'`<br>`created_at INTEGER` | `notebook_id PK`. Owns a generation/authoring graph. |
-| `cells` | `notebook_id TEXT`<br>`cell_id TEXT`<br>`type TEXT`<br>`title TEXT`<br>`position_x REAL`<br>`position_y REAL`<br>`entity_id TEXT?`<br>`prompt TEXT?`<br>`model TEXT?`<br>`inputs_json TEXT DEFAULT '{}'`<br>`output_artifact_id TEXT?` | `(notebook_id, cell_id) PK`; notebook `FK → notebooks ON DELETE CASCADE`; entity `FK → entities ON DELETE RESTRICT`; output `FK → artifacts ON DELETE RESTRICT`; `type CHECK IN (prompt, character, scene, asset, image, video)`. |
+| `notebooks` | `notebook_id TEXT`<br>`name TEXT`<br>`created_at INTEGER` | `notebook_id PK`. Owns a generation/authoring graph. Schema v18 removed the monolithic `properties_json` cell. |
+| `notebook_fields` | `notebook_id TEXT`<br>`field TEXT`<br>`value_json TEXT` | `(notebook_id, field) PK`; notebook `FK → notebooks ON DELETE CASCADE`; `field` is restricted to the typed public notebook fields. Stores optional notebook-level workflow values independently. |
+| `cells` | `notebook_id TEXT`<br>`cell_id TEXT`<br>`type TEXT`<br>`slug TEXT`<br>`grid_row INTEGER`<br>`grid_column INTEGER`<br>`output_entity_id TEXT?`<br>`prompt TEXT?`<br>`provider TEXT?`<br>`model TEXT?`<br>`operation TEXT?`<br>`tool TEXT?`<br>`inputs_json TEXT DEFAULT '{}'`<br>`output_artifact_id TEXT?` | `(notebook_id, cell_id) PK`; `(notebook_id, slug) UQ`; notebook `FK → notebooks ON DELETE CASCADE`; entity and output artifacts use `ON DELETE RESTRICT`. Cell types and kind-specific slug prefixes are checked by the schema. |
+| `notebook_cell_executions` | `notebook_id TEXT`<br>`cell_id TEXT`<br>fingerprint/status/output/provider/run/timestamp/tool/error fields<br>`stale INTEGER`<br>`fixture_baseline INTEGER` | `(notebook_id, cell_id) PK` and composite cell FK with cascade. Gives each cell's execution and staleness state its own merge boundary. |
+| `notebook_generation_plans` | `notebook_id TEXT`<br>`plan_id TEXT`<br>`cell_id TEXT`<br>`status TEXT`<br>`plan_json TEXT`<br>output/error/timestamp fields | `(notebook_id, plan_id) PK`; composite cell FK with cascade. |
+| `notebook_run_plans` | `notebook_id TEXT`<br>`plan_id TEXT`<br>`status TEXT`<br>plan/cost/fingerprint/output fields<br>timestamps | `(notebook_id, plan_id) PK`. Stores approval and execution plans as independently mergeable rows. |
+| `notebook_transcript_edits` | `notebook_id TEXT`<br>`action_id TEXT`<br>`kind TEXT`<br>`restored INTEGER`<br>`payload_json TEXT` | `(notebook_id, action_id) PK`; notebook FK with cascade. |
+| `notebook_transcript_attachments` | `notebook_id TEXT`<br>`attachment_id TEXT`<br>`payload_json TEXT` | `(notebook_id, attachment_id) PK`; notebook FK with cascade. |
 | `edges` | `notebook_id TEXT`<br>`edge_id TEXT`<br>`source_cell_id TEXT`<br>`target_cell_id TEXT`<br>`target_input TEXT` | `(notebook_id, edge_id) PK`; notebook `FK → notebooks ON DELETE CASCADE`; composite source and target FKs reference cells in the same notebook and cascade on cell deletion. |
 | `runs` | `run_id TEXT`<br>`notebook_id TEXT`<br>`status TEXT`<br>`started_at INTEGER`<br>`completed_at INTEGER`<br>`cell_order_json TEXT`<br>`outputs_json TEXT`<br>`error TEXT?` | `run_id PK`; notebook `FK → notebooks ON DELETE CASCADE`; `status CHECK IN (completed, failed, aborted)`. Terminal, versioned notebook execution records. |
 
-### Current timeline and media editing state
+### Sequence timeline and media editing state
+
+Sequences are the single timeline model. `sequences`, `sequence_tracks`,
+`sequence_clips`, `clip_links`, `clip_transforms`, `transitions`, and
+`caption_cues` hold the edit; `engine.sequences` and `engine.edits` are the
+only timeline APIs. The legacy schema-v4 `timeline`, `timeline_slots`, and
+`timeline_audio` tables were removed in v17 — schema-v4 imports convert
+still-image slots into clips on the primary sequence instead. See
+[`src/schema.ts`](../src/schema.ts) for the full sequence DDL.
 
 | Table | Columns | Keys, constraints, and purpose |
 | --- | --- | --- |
-| `timeline` | `book_id TEXT`<br>`render TEXT DEFAULT 'landscape'` | `book_id PK FK → book.book_id ON DELETE CASCADE`; `render CHECK IN (landscape, portrait, square)`. One sequence-level orientation row. |
-| `timeline_slots` | `slot_id TEXT`<br>`artifact_id TEXT`<br>`ordinal INTEGER`<br>`volume REAL?`<br>`audio_fade_in REAL?`<br>`audio_fade_out REAL?` | `slot_id PK`; artifact `FK → artifacts ON DELETE RESTRICT`; `ordinal CHECK >= 0`; volume and fades are null or `>= 0`. A single ordered visual lane; ownership by the singleton timeline is implicit. |
-| `timeline_audio` | `audio_id TEXT`<br>`artifact_id TEXT`<br>`ordinal INTEGER`<br>`start_frame INTEGER`<br>`duration_frames INTEGER`<br>`volume REAL?`<br>`fade_in REAL?`<br>`fade_out REAL?` | `audio_id PK`; artifact `FK → artifacts ON DELETE RESTRICT`; ordinal/start `CHECK >= 0`; duration `CHECK > 0`; volume and fades are null or `>= 0`. Ordered overlays with explicit timeline timing. |
 | `audio_waveforms` | `artifact_id TEXT`<br>`peaks_json TEXT` | `artifact_id PK FK → artifacts.artifact_id ON DELETE CASCADE`. Versioned waveform peaks used by editing UI. |
 
 The normalized current relationship is:
 
 ```mermaid
 erDiagram
-  BOOK ||--|| TIMELINE : owns
-  ARTIFACTS ||--o{ TIMELINE_SLOTS : supplies
-  ARTIFACTS ||--o{ TIMELINE_AUDIO : supplies
+  BOOK ||--o{ SEQUENCES : owns
+  SEQUENCES ||--o{ SEQUENCE_TRACKS : orders
+  SEQUENCE_TRACKS ||--o{ SEQUENCE_CLIPS : places
+  ARTIFACTS ||--o{ SEQUENCE_CLIPS : supplies
   ARTIFACTS ||--o| AUDIO_WAVEFORMS : has
   ARTIFACTS ||--o{ ARTIFACT_FILES : maps
   OBJECTS ||--o{ ARTIFACT_FILES : backs
 ```
 
-`timeline_slots` order is defined by `(ordinal, slot_id)`. It does not
-currently store timeline start, source in/out, duration, track, transform,
-speed, opacity, effects, or transitions. `timeline_audio` does store start and
-duration frames, but there is no catalog-level frame-rate/timebase row yet.
+Sequence timing is integer frames against each sequence's rational frame
+rate; timed clip sources use rational timebase ticks. Track and clip
+ordering uses fractional order keys with the row UUID as tie-breaker.
 
 ### Prompts and messages
 
@@ -138,25 +167,39 @@ duration frames, but there is no catalog-level frame-rate/timebase row yet.
 | `prompt_entries` | `prompt_id TEXT`<br>`surface TEXT`<br>`prompt TEXT`<br>`context_json TEXT DEFAULT '{}'`<br>`created_at INTEGER` | `prompt_id PK`. Semantic prompt history grouped by UI or agent surface. |
 | `messages` | `message_id TEXT`<br>`role TEXT`<br>`body_json TEXT`<br>`created_at INTEGER` | `message_id PK`. Structured semantic conversation history. |
 
-### Operations, action graph, and terminal jobs
+### Provenance and terminal jobs
 
-| Table | Columns | Keys, constraints, and purpose |
-| --- | --- | --- |
-| `operations` | `operation_id TEXT`<br>`operation TEXT`<br>`artifact_id TEXT?`<br>`details_json TEXT DEFAULT '{}'`<br>`write_set_json TEXT DEFAULT '[]'`<br>`base_revision TEXT?`<br>`created_at INTEGER`<br>`author TEXT` | `operation_id PK`. One low-level provenance row is written with every semantic mutation. Artifact and revision references are deliberately loose so audit history survives deletion. |
-| `actions` | `action_id TEXT`<br>`operation TEXT`<br>`scope TEXT`<br>`actor TEXT`<br>`lane TEXT`<br>`phase TEXT`<br>`base_revision TEXT?`<br>`target_artifact_id TEXT?`<br>`target_action_id TEXT?`<br>`layout_json TEXT?`<br>`details_json TEXT DEFAULT '{}'`<br>`created_at INTEGER` | `action_id PK`; scope `CHECK IN (book, artifact, layout, external, system)`; phase `CHECK IN (requested, started, completed, failed, cancelled, conflicted)`. High-level workflow/action projection. Targets are loose audit references. |
-| `action_events` | `event_id TEXT`<br>`action_id TEXT`<br>`operation_id TEXT`<br>`phase TEXT`<br>`details_json TEXT DEFAULT '{}'`<br>`created_at INTEGER` | `event_id PK`; action `FK → actions ON DELETE CASCADE`; same phase check as `actions`. `operation_id` is a loose provenance link. |
-| `action_parents` | `action_id TEXT`<br>`parent_action_id TEXT` | `(action_id, parent_action_id) PK`; child `FK → actions ON DELETE CASCADE`; parent `FK → actions ON DELETE RESTRICT`. Forms the action DAG. |
-| `action_artifacts` | `action_id TEXT`<br>`artifact_id TEXT`<br>`direction TEXT` | `(action_id, artifact_id, direction) PK`; action `FK → actions ON DELETE CASCADE`; `direction CHECK IN (input, output)`. Artifact ID is deliberately loose for durable lineage. |
-| `action_write_set` | `action_id TEXT`<br>`resource TEXT` | `(action_id, resource) PK`; action `FK → actions ON DELETE CASCADE`. Normalized resources used for overlap/conflict reasoning. |
-| `job_runs` | `run_id TEXT`<br>`artifact_id TEXT?`<br>`job_type TEXT`<br>`state TEXT`<br>`payload_json TEXT`<br>`result_json TEXT?`<br>`error_json TEXT?`<br>`started_at INTEGER?`<br>`finished_at INTEGER` | `run_id PK`; `state CHECK IN (done, failed, aborted)`. Terminal job audit. Artifact references are loose so completed history survives artifact deletion. |
-
-`actions.layout_json` currently maps to the public
-`HistoryLayout { stage: number; column: number }`. It is workflow layout
-metadata, not spatial video composition.
+Provenance is not a set of tables; it is the Dolt commit log itself. Every
+semantic commit message is structured and machine-parseable:
+
+```
+<operation>[ artifact:<artifactId>]
+
+op-id: <uuidv7>
+base-revision: <commit hash>      (when the mutation declared one)
+actor: <operation actor>          (when the mutation declared one)
+write-set: <canonical JSON array> (when non-empty)
+details: <canonical JSON object>  (when non-empty)
+```
+
+History listings, per-artifact history, stale write-set conflict checks, and
+edit restore are all derived from `dolt_log`, `dolt_diff`, and
+`dolt_at_<table>(revision)` projections — there is no parallel `operations`,
+`actions`, or `edit_batches` record to merge. doltlite rejects commit
+messages of 65536 bytes or more, so an oversized `details` or `write-set`
+payload is dropped in favor of a `details-omitted` / `write-set-omitted`
+trailer that records the payload size; projections treat omitted trailers as
+empty.
+
+Terminal job audit rows live in `job_runs` (`run_id PK`; `state CHECK IN
+(done, failed, aborted)`), an ignored runtime table rather than a semantic
+one: job payload blobs are rebuildable bookkeeping, so recording a terminal
+job never mints a commit. Artifact references are loose so completed history
+survives artifact deletion.
 
 ### Committed Dolt policy table
 
-`dolt_ignore` is created and staged separately from the 25-table allowlist:
+`dolt_ignore` is created and staged separately from the 28-table allowlist:
 
 | Column | Constraint |
 | --- | --- |
@@ -168,8 +211,147 @@ Its committed rows are:
 | Pattern | Ignored | Meaning |
 | --- | ---: | --- |
 | `runtime_%` | `1` | Never version any engine runtime table. |
+| `job_runs` | `1` | Never version terminal job audit blobs. |
 | `sqlite_sequence` | `1` | Never version local AUTOINCREMENT counters. |
 
+### Merge policy per constraint class
+
+Merges run through `mergeWithPolicy` (`src/merge-policy.ts`), which encodes
+one rule per constraint class. doltlite verifies the merged working set
+against UNIQUE, CHECK, and foreign-key constraints and rolls a violating
+merge back atomically ("working set with constraint violations"); there is
+no `dolt_verify_constraints()` in doltlite, so the merge itself plus
+post-merge scans (`PRAGMA foreign_key_check`, duplicate-singleton scans)
+are the constraint-verification primitives.
+
+- **Precondition: same schema version.** Both sides must carry the same
+  `engine_schema.version`; a mismatch is refused with
+  `SCHEMA_INCOMPATIBLE` before any merge is attempted
+  (`assertSameSchemaVersion`). This was previously only implied by the
+  engine's open-time version gate.
+- **`artifacts.slug` (globally unique) → user-facing conflict.** Two forks
+  minting the same slug for different artifacts is detected before the
+  merge by a three-way (merge-base/ours/theirs) simulation over the
+  artifacts slug projection (`findSlugConflicts`, using `dolt_merge_base`
+  and `dolt_at_artifacts`), and refused with `MERGE_CONFLICT` listing the
+  slug and both artifact ids. doltlite's own verification is the backstop:
+  refusals it reports are re-diagnosed and surface as `MERGE_CONFLICT`
+  when attributable to slugs. Row-level same-row edits surface as
+  `MERGE_CONFLICT` as well. At create time, slug dedup runs inside the
+  serialized write chain (not as a read-then-write outside it), so
+  concurrent creates cannot race onto the same slug.
+- **RESTRICT foreign keys → verification-surfaced typed violation.** A
+  fork that deletes a row another fork newly references is caught by
+  doltlite's merge-time working-set verification, which refuses and rolls
+  back; the policy maps that refusal to `MERGE_VIOLATION` (never a raw
+  `IO_ERROR`) and re-verifies referential integrity after every successful
+  merge (`verifyConstraintHealth`). Delete-time `IN_USE` pre-checks cover
+  every RESTRICT referencing table of `artifacts` (cells, streams,
+  transcripts, clips, pinned search results) so single-branch deletes also
+  fail with typed errors.
+- **`transcripts.state='current'` / `sequences.is_primary` → derived
+  singletons.** A merge of forks that each crowned a different row yields
+  duplicates. Reads resolve a deterministic winner and the post-merge
+  reconcile (`reconcileSingletonFlags`) rewrites losers: transcripts keep
+  the latest `created_at` (ties: lowest `transcript_id`), sequences keep
+  the earliest `created_at` — the original primary (ties: lowest
+  `sequence_id`). The reconcile is Dolt-committed so the working set is
+  clean for the next merge.
+- **Grid slots and order keys** need no resolution: `(grid_row,
+  grid_column)` collisions and identical fractional order keys are both
+  resolved at read time by the stable row-UUID tie-break (see the order
+  rules above). Writers self-repair: `moveTrack` runs `reconcileOrderKeys`
+  over a duplicate-key sibling group before computing a between-key, so the
+  position between two merge-minted duplicates stays reachable.
+- **`objects.forgotten_at` → forget wins, earliest stamp.** The same
+  takedown applied independently on fork and upstream produces a same-row
+  different-value cell (wall clocks differ). The projection merge resolves
+  it instead of conflicting: when both sides agree on everything except
+  `forgotten_at`, a set value beats NULL (deleted bytes stay deleted on
+  both lineages) and two set values keep the earlier timestamp
+  (`resolveObjectsRow` in `src/fork.ts`).
+
+ve-wsu: doltlite currently corrupts secondary UNIQUE indexes on
+`dolt_checkout` once a working set has three or more tables, corrupts full
+engine catalogs on checkout and `dolt_clone` (the cloned file's schema does
+not even parse: `invalid rootpage` on a secondary autoindex), misfires its
+"uncommitted changes" merge guard on the full 28-table catalog (every
+table reports a phantom `modified` status with zero row diffs), and — when
+the guard is bypassed by committing the phantom dirt — dies in schema
+loading on true merges (`schema conflict on table 'sqlite_autoindex_*'`).
+The dedicated merge-back flow therefore runs this policy around a
+projection-level three-way merge instead of `dolt_merge`; see "Forks and
+merge-back integration" below. `mergeWithPolicy` remains the drop-in merge
+mechanism once the upstream bugs are fixed, exercised against the real
+semantic DDL in `tests/merge-policy.test.ts`.
+
+### Forks and merge-back integration
+
+A fork of a public book is, from the engine's point of view:
+
+1. **A platform fork.** Creating the hosted copy of a catalog and giving
+   it a URL is the hosting layer's job; it is out of engine scope.
+2. **A clone of the catalog into a local engine root** (`bootstrapFork` in
+   [`src/fork.ts`](../src/fork.ts)). Because `dolt_clone` corrupts full
+   catalogs (ve-wsu), bootstrap takes a byte snapshot of a healthy upstream
+   `videobook.db` (captured while the upstream engine is closed) and opens
+   it as a normal engine — no `initialBookSlug`, the singleton book row
+   comes along with the snapshot. A URL bootstrap path attempts
+   `dolt_clone` and health-validates the result, surfacing a typed
+   `FEATURE_UNAVAILABLE` while the upstream bug stands; it starts working
+   unchanged once doltlite is fixed.
+3. **A public-read object store keyed by SHA-256.** `ContentStore` stays
+   the abstraction; the existing `ensureLocal` lazy fetch in
+   [`src/cas.ts`](../src/cas.ts) downloads any object the fork lacks on
+   first touch, so upstream objects of a public book are readable to
+   forkers without a bulk copy. Whether the fork's store proxies to
+   upstream's is a hosting concern — an engine root only ever sees one
+   `ContentStore`.
+
+The fork is then a full citizen: it commits on its own `main` and backs up
+to its own catalog remote and object store.
+
+**The live-engine never-pulls rule stands.** An open engine never fetches,
+pulls, or merges, and `main` remains the only supported live branch.
+Integration is a dedicated flow, `mergeBack` in
+[`src/fork.ts`](../src/fork.ts), separate from any live engine:
+
+1. Copy a healthy upstream `videobook.db` (engine closed) into a throwaway
+   temp workspace. The flow never requires or mutates the user's open
+   catalog — the source file is only read.
+2. Register/fetch the fork remote (`dolt_remote`, `dolt_fetch`) and
+   resolve heads and the merge base (`dolt_merge_base`; commit hashes via
+   `dolt_log`/`dolt_branches` — `doltHashOf` returns content hashes, not
+   commit hashes; the fetched remote-tracking ref gets a local branch
+   pointer, a ref-only write that is safe under ve-wsu).
+3. Run the merge policy: same-schema precondition, pre-merge slug-conflict
+   detection (`MERGE_CONFLICT`), then a projection-level three-way row
+   merge over `dolt_at_<table>` snapshots of base/ours/theirs (row
+   semantics mirror Dolt: one-sided changes win, identical changes
+   resolve, incompatible changes abort with `MERGE_CONFLICT`), deterministic
+   singleton-flag reconcile, and post-merge constraint verification
+   (`MERGE_VIOLATION`). The working-set rewrite uses the restore idiom —
+   delete all semantic tables in reverse, reinsert parent-before-child.
+4. Upload the fork's new objects (rows in `dolt_at_objects(theirs)` not at
+   ours, tombstones excluded) from the fork's object store to upstream's
+   BEFORE the catalog ref moves — the same objects-before-push ordering as
+   `engine.storage.backup`.
+5. Land one forward integration commit on `main` and `dolt_push` it.
+
+ve-wsu makes a true two-parent `dolt_merge` commit impossible on full
+catalogs today, so the integration commit is single-parent and records the
+integrated fork head in a `merged-revision` commit-message trailer (plus
+`base-revision`, per the structured-message convention). Re-running the
+flow is a no-op when the fork's net changes are already on `main`. The
+projection merge in `mergeRefs` is the single swap point: when doltlite is
+fixed, `mergeWithPolicy` + `dolt_merge` replace it and the commit becomes
+a true merge commit.
+
+When a plain `engine.storage.backup()` push is rejected because upstream
+moved, the backup surfaces `DIVERGED` with guidance into this flow —
+someone with a healthy upstream catalog runs `mergeBack`; the fork never
+pulls to catch up.
+
 ### Semantic indexes
 
 Primary keys and unique declarations create their own backing indexes. The
@@ -186,15 +368,8 @@ schema additionally defines every index below.
 | `edges_source` | `edges(notebook_id, source_cell_id)` |
 | `edges_target` | `edges(notebook_id, target_cell_id)` |
 | `runs_notebook_completed` | `runs(notebook_id, completed_at, run_id)` |
-| `timeline_slots_order` | `timeline_slots(ordinal, slot_id)` |
-| `timeline_slots_artifact` | `timeline_slots(artifact_id)` |
-| `timeline_audio_order` | `timeline_audio(ordinal, audio_id)` |
-| `timeline_audio_artifact` | `timeline_audio(artifact_id)` |
 | `prompt_entries_lookup` | `prompt_entries(surface, created_at, prompt_id)` |
 | `messages_created` | `messages(created_at, message_id)` |
-| `operations_created` | `operations(created_at, operation_id)` |
-| `operations_artifact_created` | `operations(artifact_id, created_at, operation_id)` |
-| `action_events_action_created` | `action_events(action_id, created_at, event_id)` |
 
 ## Local-only runtime schema
 
@@ -252,57 +427,106 @@ to recover safely:
 
 1. Start `BEGIN IMMEDIATE`.
 2. Change the requested semantic rows.
-3. Insert the mutation's `operations` row.
-4. Insert `runtime_commit_outbox(operation_id, tables_json, message, created_at)`.
-5. Commit the SQL transaction.
-6. Stage only changed names from `SEMANTIC_TABLES`.
-7. Create a Dolt commit on `main`.
-8. Delete the runtime outbox row in a runtime transaction.
-
-On reopen, the engine drains any surviving outbox record. Runtime staging is
-checked before writes and after commits. Historical reads use
+3. Insert `runtime_commit_outbox(operation_id, tables_json, message, created_at)`;
+   `tables_json` carries the operation's declared write set alongside the
+   allow-empty flag, so the declaration commits atomically with the mutation
+   and recovery stages the same tables.
+4. Commit the SQL transaction.
+5. Probe each declared table with a row-level diff (`dolt_diff_<table>`,
+   which also sees staged-but-uncommitted rows during recovery) and stage
+   the dirty ones one `dolt_add` at a time — doltlite's multi-argument
+   `dolt_add` over-stages, and its `doltStatus` phantom `modified` entries
+   are why status is never consulted for dirtiness.
+6. Create a Dolt commit on `main` with the configured identity as `--author`;
+   when nothing semantic changed, clear the outbox row instead of minting an
+   empty commit.
+7. Assert the committed tables are no longer dirty, then delete the runtime
+   outbox row.
+
+On reopen, the engine drains any surviving outbox record (legacy rows
+without a declared table list fall back to probing the full allowlist) and
+then sweeps the whole catalog: any semantic table with unattributed
+uncommitted rows faults the open with `STORAGE_ERROR`. Staged tables are
+asserted against the allowlist before every commit. Opening a catalog never creates a
+commit: terminal-job reconciliation writes the ignored `job_runs` table
+through a runtime transaction. Historical reads use
 `dolt_at_<semantic_table>(revision)`; history and conflict projections use
-Dolt log, status, and diff APIs. Remote catalog support is push-backup only.
+Dolt log, status, and diff APIs. A whole-book restore reloads **every** table
+in `SEMANTIC_TABLES` from its `dolt_at_*` projection at the target revision —
+deleting in reverse and reinserting in forward, parent-before-child order,
+with columns taken from `PRAGMA table_info` — so the restored state is
+exactly the recorded state and no hand-maintained table list can drift.
+Remote catalog support is push-backup only.
+
+## Forgettable data and raw-text audit
+
+Published history is append-only: once a catalog has been pushed anywhere
+someone could fork it, its commits are permanent. Forgetting therefore means
+deleting content-addressed objects, never rewriting history. History squash
+tooling would only be legal before the first push (before anyone can fork);
+the engine deliberately does not implement it.
+
+### Object deletion and GC
+
+- `engine.storage.deleteObject(hash, { force?, remote? })` forgets one
+  object. It refuses with `IN_USE` (listing the HEAD references) when a HEAD
+  row still names the hash, unless `force` is given — the takedown path.
+- `engine.storage.gc({ dryRun?, remote?, doltGc? })` sweeps every object
+  nothing references at HEAD, plus stray local files that never got an
+  `objects` row. A hash is referenced at HEAD when a HEAD row names it in a
+  first-class `object_hash`/`payload_hash` column (`artifact_files`,
+  `artifact_streams`, `pinned_search_results`, `sequence_clips`,
+  `transcripts`) or embeds it as an `objectHash` in a `cell_references`
+  snapshot. Historical revisions are not consulted.
+- Both record forgetting as a semantic commit (`delete_object` /
+  `gc_objects`) that sets `objects.forgotten_at`. The row is never deleted:
+  it stays as the tombstone (hash + size + forgotten timestamp) for every
+  historical row that named the object, and backup never tries to publish a
+  forgotten object again. Re-importing the same bytes resurrects the row
+  (`forgotten_at` is cleared when the object is re-linked).
+- Restoring an old revision whose object was forgotten relinks the
+  tombstone: the forward restore commit stands, and reads of the missing
+  bytes surface `OBJECT_UNAVAILABLE` through the existing error path instead
+  of crashing. Deleting a referenced object with `force` behaves the same
+  way at HEAD.
+- With `remote: true`, deletion also unpublishes via
+  `ContentStore.delete(key)` and clears the `runtime_object_publications`
+  marker so a later re-import is published again.
+- doltlite exposes `dolt_gc()` as a SQL function (verified: it returns a
+  `"N chunks removed, M chunks kept"` summary). `gc({ doltGc: true })` runs
+  it after collecting to physically reclaim chunks left behind by dropped
+  table data in the versioned catalog.
+- Run `deleteObject` and `gc` only while no imports are in flight; CAS puts
+  happen outside the serialized write chain, so a concurrent import could
+  race the sweep.
+
+### Raw-text audit
+
+| Table / column | Decision | Rationale |
+| --- | --- | --- |
+| `transcript_segments.text`, `transcript_words.text` | **Moved behind CAS** (schema v16) | Full transcripts are the bulk-text case. Segment/word text lives in one CAS object named by `transcripts.payload_hash`; the versioned rows keep only structure (IDs, ordinals, ticks, speaker, confidence, kind), so `transcripts.selectionRange` and caption-cue word references keep working after the payload is forgotten. `transcripts.delete` removes the rows; the payload then becomes GC-collectable. |
+| `caption_cues.text` | Accepted permanent | Short editorial cue text needed to render without CAS access; a deliberate editorial snapshot that can diverge from the transcript. |
+| `prompt_entries.prompt`, `cells.prompt`, `entities.prompt`/`description` | Accepted permanent | Small user-authored creative working data; part of the semantic record users expect to persist, like commit messages. |
+| `messages.body_json` | Accepted permanent | Small per-row authored conversation record. |
+| `pinned_search_results.query_json`/`signals_json`/`representative_json`, `cell_references.snapshot_json` | Accepted permanent | Small selection snapshots; may embed short excerpts. Snapshot `objectHash` values are honored as loose GC roots. |
+| `runs.cell_order_json`/`outputs_json`/`error`, `book_metadata`/`artifact_metadata.value_json` | Accepted permanent | Structural or small key/value data. |
+| `runtime_segment_text.text`, `runtime_text_similarity_chunks.chunk_text` | Out of scope | `runtime_*` tables are never versioned or pushed, so they never enter the public record; rebuild or drop them locally at will. |
 
 ## Video-editing structures exposed today
 
 The SQL schema is only one part of the editing model. These structures and
 conventions are also important.
 
-### Timeline contract
-
-```ts
-type TimelineRender = "landscape" | "portrait" | "square";
-
-interface Timeline {
-  bookId: string;
-  render: TimelineRender;
-  slots: TimelineSlot[];
-  audio: TimelineAudio[];
-}
-
-interface TimelineSlot {
-  id: string;                 // UUIDv7
-  artifactId: string;
-  volume?: number;
-  audioFadeIn?: number;
-  audioFadeOut?: number;
-}
-
-interface TimelineAudio {
-  id: string;                 // UUIDv7
-  artifactId: string;
-  startFrame: number;
-  durationFrames: number;
-  volume?: number;
-  fadeIn?: number;
-  fadeOut?: number;
-}
-```
+### Sequence contract
 
-`engine.timeline.set()` replaces/synchronizes the normalized rows while
-preserving caller-supplied stable IDs. `getAtRevision()` projects the three
-timeline tables at a Dolt revision.
+The timeline is a `Sequence` projection: rational frame rate and pixel
+aspect, an audio sample rate and channel layout, ordered video/audio/caption
+tracks, clips with source ranges and transforms, transitions, and caption
+cues. `engine.sequences` reads and mutates sequence structure (tracks,
+names, primary sequence); `engine.edits` applies transactional clip,
+transition, and caption operations. `getAtRevision()` projects the sequence
+tables at a Dolt revision. The full contract types live in
+[`src/mvp-contracts.ts`](../src/mvp-contracts.ts).
 
 ### Artifact manifest and content
 
@@ -360,12 +584,10 @@ status, and error. Entities provide structured prompts, characters, and scenes.
 
 ### Provenance and concurrency
 
-- `Revision` projects Dolt commit hash, operation, artifact, details, and file
-  changes.
-- `HistoryAction` projects the action DAG, lifecycle events, input/output
-  artifact lineage, workflow layout, and write set.
-- `base_revision` plus normalized write sets detect overlapping semantic
-  actions.
+- `Revision` projects the Dolt commit hash, operation, artifact, and
+  parameters parsed from the structured commit message.
+- `base_revision` plus the write sets carried in commit messages detect
+  overlapping semantic changes.
 - Runtime jobs, resource leases, artifact views, owner IDs, expiry times, and
   monotonic fences coordinate processors without polluting Dolt history.
 
@@ -381,10 +603,15 @@ status, and error. Entities provide structured prompts, characters, and scenes.
 
 ## Full NLE structures not yet implemented
 
-Schema v4 can assemble one ordered visual lane plus timed audio overlays. It is
-not yet a complete non-linear editor model. The following is a concrete
-candidate layout for a future clean-break schema. Every name in this section
-is proposed and does **not** exist in schema v4.
+The core sequence model proposed below has since landed (`sequences`,
+`sequence_tracks`, `sequence_clips`, `clip_links`, `clip_transforms`,
+`transitions`, `caption_cues`) and is now the only timeline model; the
+legacy v4 lane described here was removed in v17. The remaining proposals —
+effects, keyframes, audio buses, markers, derivatives — are still open.
+Schema v4 assembled one ordered visual lane plus timed audio overlays. The
+following is a concrete candidate layout for the remaining pieces. Every
+name in this section is proposed and, unless noted above, does **not** exist
+in the catalog.
 
 ### Core sequence model
 
@@ -460,7 +687,16 @@ must reproduce a render.
 - Preserve loose references only in audit/lineage records that must outlive the
   target.
 - Keep order as an integer plus stable ID tie-breaker, or use a documented
-  fractional ordering key if frequent insertion warrants it.
+  fractional ordering key if frequent insertion warrants it. The engine uses
+  fractional/lexicographic order keys (base-62, `"a0"`-style midpoint keys,
+  see `src/order-keys.ts`) for engine-maintained orderings — sequence tracks
+  and, before the legacy triple was removed in v17, timeline slots and
+  timeline audio — with the row UUID as tie-breaker, so inserting or moving
+  one row never renumbers its neighbors. Notebook cells
+  keep their explicit integer `(grid_row, grid_column)` slot as the ordering
+  key with the cell UUID as tie-breaker; slot uniqueness is no longer
+  enforced by the schema and collisions left by a merge are repaired on the
+  next write instead.
 - Put user intent and accepted edit decisions in Dolt. Put playhead position,
   selections, UI panels, decode caches, thumbnails, temporary renders, active
   jobs, locks, and presence in ignored runtime tables.
@@ -478,12 +714,16 @@ must reproduce a render.
 | Concern | Implementation |
 | --- | --- |
 | DDL, table allowlists, indexes | [`src/schema.ts`](../src/schema.ts) |
+| Fractional order keys and minimal rekeying | [`src/order-keys.ts`](../src/order-keys.ts) |
 | SQL transactions, Dolt staging, commits, outbox recovery | [`src/store.ts`](../src/store.ts) |
+| Merge policy per constraint class | [`src/merge-policy.ts`](../src/merge-policy.ts) |
+| Fork bootstrap and merge-back integration | [`src/fork.ts`](../src/fork.ts) |
 | Engine paths and row projections | [`src/context.ts`](../src/context.ts) |
-| Immutable object layout and remote keys | [`src/cas.ts`](../src/cas.ts) |
+| Content-addressed object layout, remote keys, deletion | [`src/cas.ts`](../src/cas.ts) |
+| Object publication, deletion, GC, and catalog backup | [`src/storage.ts`](../src/storage.ts) |
 | Artifact mappings and workspace materialization | [`src/files.ts`](../src/files.ts) |
-| Current timeline API and row synchronization | [`src/timeline.ts`](../src/timeline.ts) |
-| Public timeline, manifest, job, status, and similarity types | [`src/engine-types.ts`](../src/engine-types.ts) |
+| Sequence reads/structure and transactional edits | [`src/sequences.ts`](../src/sequences.ts), [`src/edits.ts`](../src/edits.ts) |
+| Public manifest, job, status, and similarity types | [`src/engine-types.ts`](../src/engine-types.ts) |
 | Entity/notebook graph types | [`src/notebook/types.ts`](../src/notebook/types.ts) |
 | Revision and action projections/restores | [`src/history.ts`](../src/history.ts), [`src/history-types.ts`](../src/history-types.ts) |
 | Media naming and discovery conventions | [`src/media.ts`](../src/media.ts) |