@abloatai/ablo 0.35.0 → 0.37.0

package/docs/internal/commit-identifiers.md

@@ -0,0 +1,91 @@
+# Commit identifiers: the two axes
+
+Maintainer reference. A commit carries several ids, and they are easy to
+conflate because they travel together and are all "some number attached to a
+write." They are not interchangeable. Each answers a different question, and
+they split cleanly along **one line**: does this id help decide whether the
+write *wins*, or does it only help *identify* the write after the fact?
+
+Keeping the two axes separate is what lets each id be reasoned about — and
+audited — on its own. This doc is the single place they sit side by side.
+
+## The line
+
+| axis | the question it answers | when it acts | if it's absent |
+|---|---|---|---|
+| **Conflict resolution** | *should this write land, given what else happened?* | at the commit chokepoint, before the row is written | the write is unguarded (last-writer-wins) |
+| **Correlation / audit** | *which write is this, and have I seen it before?* | on receipt (dedup) and after the fact (attribution) | the write still lands; you just can't dedup or trace it as precisely |
+
+A conflict-resolution id can **reject** a commit. A correlation id never does —
+at most it makes a retried commit a no-op (idempotency). Never reach for one to
+do the other's job: a correlation id can't fence a stale write, and a fence
+can't dedup a retry.
+
+## Axis 1: conflict resolution (does this write win?)
+
+Evaluated inside `executeCommit`'s transaction, atomic with the delta write.
+Three independent fences, each catching what the others can't; the full
+narrative is [ADR 0009 §6](../../../../docs/decisions/0009-claim-durability-two-reclaim-clocks.md)
+and the [coordination reference](../coordination.md).
+
+| id | wire field | persisted to | what it asserts | rejects when |
+|---|---|---|---|---|
+| **read basis** | per-op `readAt` | `sync_deltas.read_at_sync_id` | "the state I reasoned **from**": a version watermark | the row moved since `readAt` (version-CAS), under `onStale: 'reject'` |
+| **fencing token** | per-op `fenceToken` | `sync_deltas.fence_token` **and** `claim_fence_watermark.fence_token` | "the lease generation I was authorized **at**": a monotonic per-entity high-water | the token is below the entity's persisted high-water: a lapsed holder writing after its successor already claimed, wrote, and released |
+| **claim / lease** | `claimId`, `heldBy` on the `WireClaim` | the coordination store (Redis), not `sync_deltas` | "I hold this row right now": live mutual exclusion | a non-holder writes a row another participant holds |
+
+`onStale` (`notify` / `reject` / `overwrite`) is **not** an id — it's the
+disposition that decides what a stale `readAt` *does*. It rides with the read
+basis but is policy, not evidence, so it isn't persisted.
+
+Why the token is a distinct id from `readAt`, and not just reused `sync_id`:
+`readAt` advances on every **write** and asserts *from what data*; the token
+advances on every **grant** and asserts *at what lease generation*. Their events
+differ, so one can't stand in for the other — a lapsed holder that skips
+version-CAS (no `readAt`, a blind write) is invisible to the read basis but
+still carries a stale token. That is precisely fence (c) closing what (a) can't.
+The reasoning in full lives in
+[the fencing-token scope doc](../../../../docs/plans/claim-fencing-token-option-b-scope.md).
+
+## Axis 2: correlation / audit (which write is this?)
+
+Never decides a conflict. These are how a write is recognized — as a duplicate,
+as your own echo, or as one row in a signed history.
+
+| id | wire field | persisted to | purpose |
+|---|---|---|---|
+| **idempotency key** | batch `clientTxId` (public alias `idempotencyKey`) | dedup ledger keyed by it | a retried batch commits **once**: the second attempt is recognized and folded to a no-op, not re-applied |
+| **per-op transaction id** | per-op `transactionId` | `sync_deltas.transaction_id` | echo detection: the broadcast delta arrives at the originating client carrying the **same** id its queue marked pending, so it reconciles its optimistic write instead of double-applying |
+| **sync id** | assigned server-side (`next_sync_id`) | `sync_deltas.id` | the monotonic total order: the serialization order every reader tails and every `readAt` names. It is *assigned*, never client-supplied |
+| **attribution** | actor / capability / delegation on the frame | `sync_deltas` actor columns + the signed audit chain | who acted, on whose behalf, under which key: the [audit log](../audit.md)'s who/when |
+
+The batch key and the per-op id are deliberately separate: a multi-row commit is
+**one** idempotent unit (one `clientTxId`) made of **many** individually-echoable
+ops (each its own `transactionId`). Collapsing them would make echo detection
+batch-coarse and break optimistic reconciliation for multi-op commits.
+
+## The evidence tuple on a `sync_deltas` row
+
+Both axes leave their mark on the delta, which is what makes a delta a complete,
+self-describing audit record — you can reconstruct the full justification of a
+write from the row alone, never from a live lease that has since vanished:
+
+- **who:** `actor_id` / `capability_id` (+ the signed chain)
+- **what:** `data` / `previous_data`
+- **when:** `id` (`sync_id`) / `created_at`
+- **from what known state:** `read_at_sync_id` (the read basis)
+- **at what lease generation:** `fence_token` (the token the commit fenced)
+- **as which client operation:** `transaction_id` (echo identity)
+
+`read_at_sync_id` and `fence_token` are companions: the first records the data
+version the write reasoned against, the second the lease generation it was
+authorized at. Both are `NULL` when the write carried none (an unclaimed write, a
+human `user` committer exempt under Law 7, or a legacy row) — **never fabricated
+server-side**. The evidence derives only from what the write actually presented,
+so the audit row can't drift from what the fence enforced.
+
+## One-line test for "which id is this?"
+
+> If removing it could turn an accepted commit into a rejected one, it's
+> **Axis 1**. If removing it only costs you dedup, echo reconciliation, or
+> traceability — while the write still lands — it's **Axis 2**.

package/docs/internal/concurrency-open-decisions.md

@@ -0,0 +1,37 @@
+# Concurrency Convention: Open Decisions
+
+Maintainer notes for [`concurrency-convention.md`](../concurrency-convention.md).
+These are decisions the team has deliberately not made yet. They change public
+behaviour, so they are tracked here rather than in the public contract, where an
+unmade decision reads as an unsettled guarantee.
+
+## Default disposition for agents
+
+Should an agent-participant guarded write default to `notify` (philosophy
+aligned: surface, do not overwrite) instead of `reject` (back-compat)?
+
+The trade-off is alignment against a behaviour change for existing agent
+callers. Today a guarded write with `readAt` but no `onStale` defaults to
+`reject` for every participant kind.
+
+## Batch premises through the policy seam
+
+Should premise conflicts also pass through `ConflictPolicy`, or stay on the
+direct `onStale` mapping?
+
+Routing them through the seam requires a group-aware conflict shape, because a
+batch premise can name a sync group rather than a row. Custom `ConflictPolicy`
+functions currently see write-target conflicts only (`stale_context` /
+`claim_held`); batch-premise conflicts resolve directly through each entry's
+`onStale`.
+
+## The serializability floor
+
+A batch premise is a sound check, not a full precedence-graph guarantee: it
+catches only what the caller declared. A caller that declares nothing gets no
+check at all, because write-target checking needs a `readAt` to check against,
+and a plain write is last-writer-wins.
+
+The floor is therefore zero, and closing that gap is the subject of ADR 0018.
+The public page states the resulting behaviour as a limit; the framing of it as
+a gap to be closed belongs here.

package/docs/internal/data-source-reverse-channel.md

@@ -0,0 +1,150 @@
+# Data Source Reverse Channel (local-dev parity)
+
+Maintainer scoping doc. Closes the one real day-one DX gap in Data Source
+mode: the `commit`/`load`/`list` legs are inbound webhooks (Ablo → your
+endpoint), so on `localhost` they need a tunnel (ngrok/cloudflared). This
+scopes a built-in reverse channel so Data Source works on localhost the way
+managed mode already does — the Stripe-CLI `stripe listen` pattern.
+
+## The gap, precisely
+
+Data Source has two directions today:
+
+| Leg | Direction | Transport(s) today | localhost-friendly? |
+|---|---|---|---|
+| `events` (external writes → Ablo) | customer → Ablo | **poll** (`events` handler, Ablo calls you) **+ push** (`createPushQueue` → `POST /api/source/events`, you call Ablo) | ✅ yes, via push |
+| `commit` / `load` / `list` | Ablo → customer | **inbound webhook only** (`dataSource()` route) | ❌ no: needs public URL |
+
+The asymmetry is the whole bug. `events` already ships an outbound transport
+(`src/source/pushQueue.ts`), so external writes reach Ablo from localhost
+without a tunnel. The `commit`/`load`/`list` leg never got one, so Ablo Cloud
+has no way to reach a `localhost:3000` dev server.
+
+Inbound is exactly what localhost cannot receive. Managed mode has no inbound
+leg (the browser/SDK opens the only connection, outbound to Ablo), which is why
+managed mode "just works" locally and Data Source doesn't.
+
+## Prior art
+
+- **Stripe CLI `stripe listen`:** the canonical fix. The CLI opens an
+  *outbound* WebSocket to Stripe; Stripe drains webhook events down it and the
+  CLI forwards them to `localhost`. No public URL, no tunnel. We want the same
+  for the `commit`/`load`/`list` leg.
+- **Our own `createPushQueue`:** already proves the outbound-from-customer
+  pattern for the `events` leg. The reverse channel is the symmetric primitive
+  for the other direction.
+
+## Design
+
+A customer-run **source connector** dials out to Ablo Cloud and serves
+`commit`/`load`/`list` over the open connection, instead of Ablo making inbound
+HTTP calls.
+
+```
+LOCAL DEV (no public URL)
+
+  ablo client ──ws──▶ Ablo Cloud ──┐
+                                   │  (no inbound HTTP to localhost)
+  customer connector ──ws (dial-out)──▶ Ablo Cloud
+       │  drains pending commit/load/list for this source
+       ▼
+  dataSource(options)  ← UNCHANGED handler, fed a synthesized Request
+       ▼
+  local Postgres
+       │  signed response posted back up the same ws
+       └──────────────────────────────────────▶ Ablo Cloud ──ws──▶ ablo client
+```
+
+### Why it's small
+
+`dataSource(options)` is already `(request: Request) => Promise<Response>` in
+`src/source/factory.ts`. The connector does not reimplement any handler logic —
+it:
+
+1. Opens a WS to a new Ablo Cloud endpoint (e.g. `/v1/source/listen`),
+   authenticating with the project API key.
+2. Registers which source/org it serves. Ablo Cloud routes that source's
+   `commit`/`load`/`list` requests to this socket instead of the configured
+   webhook URL (when a live connector is attached).
+3. For each drained request frame: synthesize a `Request` with the same signed
+   headers Ablo would have sent, call the customer's existing `dataSource`
+   handler, and post the `Response` back up the socket.
+
+Customer-side surface is one wrapper around the handler they already wrote:
+
+```ts
+// dev only — same handler object as the deployed route
+import { dataSource, createSourceConnector } from '@abloatai/ablo';
+import { sourceOptions } from './ablo.source'; // shared with route.ts
+
+const connector = createSourceConnector({
+  apiKey: process.env.ABLO_API_KEY!,      // sk_test_*
+  handler: dataSource(sourceOptions),     // the unchanged (Request)=>Response
+});
+await connector.run(abortSignal);
+```
+
+`route.ts` (deployed) and the connector (local) share the same
+`sourceOptions` — zero handler drift.
+
+### Server side (sync-server)
+
+- New WS endpoint `/v1/source/listen`. Auth: project API key → resolves the
+  source. Reject if the key isn't `sk_test_*` unless the source explicitly
+  opts into reverse-channel for production (see "Production" below).
+- Per-source request queue. When a `commit`/`load`/`list` needs the customer
+  and a connector is attached, enqueue + drain down the socket instead of
+  POSTing the webhook URL. Reuse the same signed-envelope shape so the
+  customer handler verifies identically (`verifyAbloSourceRequest` unchanged).
+- Fallback: no connector attached → existing inbound webhook path. The
+  reverse channel is purely additive; nothing changes for deployed apps.
+
+### Signature / security
+
+- The drained frames carry the **same** Standard Webhooks signature
+  (`webhook-id`/`webhook-timestamp`/`webhook-signature`) computed with the
+  project key, so the connector verifies them through the existing
+  `verifyAbloSourceRequest` with no special-casing. The transport changes; the
+  trust model does not.
+- Gate to `sk_test_*` by default. The DB still stays canonical in the
+  customer's process; nothing here gives Ablo the `DATABASE_URL`.
+
+## Test-mode interplay
+
+`SourceRequestContext.mode` (`src/source/types.ts`) already distinguishes
+`test`/`live`. The reverse channel is the natural home for `mode: 'test'`
+traffic: a local connector attached with an `sk_test_*` key receives the
+source's test commits, runs them against the customer's test DB, and the SDK
+sees confirmed rows + fan-out exactly as in production. This is the missing
+piece that makes `sk_test_*` a complete local loop rather than just a data
+namespace.
+
+## Production stance
+
+Keep the inbound webhook as the default deployed transport — it's lower
+latency (no long-lived socket to babysit) and stateless. The reverse channel
+is primarily the **dev** affordance. A secondary, opt-in use is a
+"no-public-URL deploy" mode for customers who cannot expose an inbound
+endpoint at all (locked-down VPCs); that's a follow-on, not the initial scope.
+
+## Scope boundary (what this is NOT)
+
+- Not a generic tunnel — it forwards only signed Ablo source frames for one
+  source, not arbitrary traffic.
+- Not a change to the handler contract — `dataSource` is untouched; the
+  connector wraps its handler.
+- Not a managed-mode change — managed mode has no inbound leg and is unaffected.
+
+## Touch list (when built)
+
+- `packages/transaction/src/source/connector.ts` — `createSourceConnector`
+  (dial-out WS client; synthesize Request → existing handler → post Response).
+- `packages/transaction` export surface — expose `createSourceConnector` next to
+  `createPushQueue`.
+- `apps/sync-server` — `/v1/source/listen` WS endpoint + per-source request
+  queue + "drain to connector if attached, else webhook" branch in the source
+  dispatch path.
+- `docs/data-sources.md` — document the local-dev loop (the current docs only
+  describe the public-HTTPS webhook).
+- Tests: connector round-trip (drained commit → handler → response), signature
+  parity with the webhook path, fallback-to-webhook when no connector attached.

package/docs/internal/per-field-conflict-detection.md

@@ -0,0 +1,165 @@
+# Per-Field Conflict Detection (Track A)
+
+Maintainer decision doc. Scopes the move from entity-level to field-level stale
+detection in `executeCommit`. Library-free; restores Linear parity for the
+disjoint-field case while keeping the agent-specific `readAt` reject.
+
+## Problem
+
+`executeCommit` Step 0 (`apps/sync-server/src/mutators/commit.ts`) detects stale
+writes at **entity granularity**:
+
+```sql
+SELECT MAX(id) FROM sync_deltas WHERE model_name = ? AND model_id = ?
+```
+
+If `observed > op.readAt`, the op conflicts. This means two writers touching
+**different fields** of the same row collide falsely: agent A sets
+`report.status`, human B sets `report.reviewer`, B carried a `readAt` from before A's
+write → B is rejected with `AbloStaleContextError`, even though the edits never
+overlapped.
+
+This is an over-rejection. It is also stricter than the system we
+reverse-engineered from (Linear), which never had this problem.
+
+## Why Linear says this is right
+
+The model is reverse-engineered from Linear's sync engine. Linear's design
+confirms every load-bearing choice here:
+
+- **Transactions are property-level.** Linear's `UpdateTransaction` records
+  "the name of the changed property and its previous value" — it carries only
+  the changed properties, not a whole-object snapshot. Our `changed_fields`
+  column re-derives exactly that.
+- **Resolution is last-writer-wins, per property, by total order:** `syncId`
+  (our `sync_id_seq`) is the total order. A partial transaction applies only its
+  properties, so two clients editing **different** properties both win — LWW
+  only bites on the **same** property.
+- **CRDT is used only for issue descriptions.** Linear keeps LWW-per-property
+  for structured fields and reserves a CRDT for the one rich-text body. That is
+  the same Track A / Track B line we draw: this doc is Track A; rich-text bodies
+  (TipTap `content_json`) are out of scope and belong to a separate CRDT track.
+
+So Track A is not a new feature — it **restores Linear parity** at the property
+level we had flattened to entity level.
+
+Sources: [reverse-linear-sync-engine (CTO-endorsed)](https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/SUMMARY.md),
+[Architectures for Central Server Collaboration — Weidner](https://mattweidner.com/2024/06/04/server-architectures.html).
+
+## The constraint that shapes the design
+
+`sync_deltas.data` stores the **full post-update row**, not the changed columns.
+This was a deliberate change (see `feedback_partial_update_delta_ui_drift`) so
+the live-pool update path fires MobX reactivity for nested fields. Consequence:
+we **cannot** recover "which fields did this delta change" from `data` — a
+full-row snapshot does not tell you what moved.
+
+We do have the changed set for free at write time: `Object.keys(snakeInput)` at
+`commit.ts` UPDATE branch, after the unknown-column strip and before the
+`updated_at` injection. So this is a write-side capture + a read-side
+intersection, not a diff-the-snapshots problem.
+
+## Design
+
+### 1. Schema: `sync_deltas.changed_fields text[]` (nullable)
+
+- Populate **only for UPDATE** with the real changed columns
+  (`Object.keys(snakeInput)` after strip, before `updated_at`).
+- Leave `null` for CREATE / DELETE / ARCHIVE / UNARCHIVE.
+
+`null` is semantically "whole-entity change" and always conflicts. This gives a
+**safe migration**: every pre-migration delta is `null`, so detection falls back
+to exactly today's entity-level behavior. Field granularity phases in only as
+new deltas land — no risky backfill.
+
+We store **field names only**, not previous values. LWW needs no value
+comparison (latest `sync_id` wins); names are sufficient for the overlap check
+that drives the optional reject. Prev-values would only matter for
+"same field, same value ⇒ not a conflict" tie-breaking — defer it.
+
+### 2. Detection rewrite: Step 0 (`commit.ts`)
+
+Replace the scalar `MAX(id)` with a field-aware scan:
+
+```ts
+// op's own field set (snake-cased, framework cols excluded)
+const opFields = new Set(Object.keys(op.input ?? {}).map(toSnakeCase));
+
+const rows = await tx.unsafe(
+  `SELECT id, changed_fields FROM sync_deltas
+    WHERE model_name = $1 AND model_id = $2 AND id > $3
+    ORDER BY id DESC`,
+  [mapping.modelName, op.id, op.readAt] as never[],
+);
+
+// Conflict iff a newer delta touched a field this op also writes,
+// OR a newer delta is whole-entity (changed_fields IS NULL → CREATE/DELETE).
+const overlap = rows.find(
+  (r) => r.changedFields === null || r.changedFields.some((f) => opFields.has(f)),
+);
+if (overlap) {
+  conflicts.push({
+    /* ...existing fields... */
+    observedSyncId: overlap.id,
+    conflictingFields: intersect(overlap.changedFields, opFields),
+  });
+}
+```
+
+Disjoint-field concurrent writes now produce **no conflict** — they both apply.
+That is LWW-per-field achieved by *not rejecting*; no merge code.
+
+### 3. Policy type: additive (`packages/transaction/src/policy/types.ts`)
+
+Extend `StaleContextConflict` with:
+
+```ts
+readonly conflictingFields?: readonly string[];
+```
+
+Pure addition. `defaultPolicy` still rejects; existing policies compile
+unchanged. A policy can now reason at field granularity, e.g. allow when the
+only conflicting field is cosmetic.
+
+### 4. Scope boundary (honest)
+
+Granularity is **column-level**, not JSON-path. Two writers editing different
+keys *inside* one `content_json` column still conflict — that is the rich-text
+case (Track B / CRDT), not this. JSON Merge Patch (RFC 7386) sub-column
+granularity is a later refinement on the same column; v1 stops at columns.
+
+## Relationship to the existing `readAt` reject
+
+Linear is pure LWW-per-property with no stale check. We keep `readAt` /
+`onStale: 'reject'` as an **opt-in** for the agent-reasoned-against-stale-state
+case (an LLM that read a stale value and reasoned on it is a real failure mode
+humans rarely hit). After Track A:
+
+- **No `readAt`** → LWW-per-field, Linear parity: disjoint fields never conflict.
+- **`readAt` set** → reject only if a newer delta touched a field this op also
+  writes. The `changed_fields` column makes that intersection computable.
+- **`onStale: 'overwrite'`** → unchanged; still skips detection entirely.
+
+## Index
+
+Verify a `(model_name, model_id, id)` index exists on `sync_deltas` (the old
+`MAX(id)` relied on it too). The new query is a bounded range scan (`id >
+readAt`, usually a small recent window) on the same index prefix.
+
+## Tests (vitest, sync-server)
+
+- Disjoint fields (A: `status`, B: `assignee`, same row, both stale `readAt`) →
+  **both commit, no `AbloStaleContextError`** — the regression that proves the win.
+- Same field, both stale → still rejects (default policy unchanged).
+- DELETE after `readAt` → conflicts regardless of op fields (`null`).
+- Pre-migration delta (`null`) in the window → conflicts (back-compat).
+- `onStale: 'overwrite'` → still skips detection.
+
+## Touch list
+
+- `sync_deltas` migration — add `changed_fields text[]`.
+- `commit.ts` — Step 0 detect rewrite + UPDATE write path populates
+  `changed_fields` + `deltaInfos` shape.
+- `apps/sync-server/src/db/deltas.ts` — insert path carries `changed_fields`.
+- `packages/transaction/src/policy/types.ts` — additive `conflictingFields`.
+- vitest in sync-server.

package/docs/internal/postgres-replication.md

@@ -0,0 +1,64 @@
+# Postgres replication: internal architecture
+
+> **Status: wired and covered by unit and real-Postgres journeys.** This is server-internal code under `apps/sync-server/src/replication/postgres/`; it is not an SDK surface.
+
+## Why this exists
+
+Ablo observes a customer's Postgres through a publication and logical-replication slot. The customer owns the schema and write path. Ablo decodes committed changes, appends them to its control-plane log, and serves sync from that log. The low-level decoder and lifecycle draw on Zero and PowerSync patterns; ADR 0002 governs the product boundary.
+
+## The one job
+
+**Postgres `pgoutput` messages → `PreparedDelta[]` + a confirmed LSN.** The consumer writes through `appendExternalDeltas`; deltas land in the control-plane `sync_deltas` log and use the normal fan-out pipeline.
+
+## Module map (`apps/sync-server/src/replication/postgres/`)
+
+| file                                                     | role                                                       | provenance                    |
+| -------------------------------------------------------- | ---------------------------------------------------------- | ----------------------------- |
+| `binaryReader.ts`                                        | big-endian protocol reader                                 | modeled on Zero               |
+| `pgoutputTypes.ts`, `pgoutput.ts`                        | typed `pgoutput` messages and decoder                      | modeled on Zero               |
+| `lsn.ts`                                                 | `LSN` string ↔ `bigint` (`toBigInt`/`fromBigInt`)          | ported subset ← Zero `lsn.ts` |
+| `connection.ts`, `stream.ts`, `streamAdapter.ts`         | dedicated query/replication connections and stream adapter | Zero/PowerSync patterns       |
+| `assembler.ts`                                           | buffers a transaction and maps changes to `PreparedDelta`  | Ablo adapter                  |
+| `consumer.ts`                                            | persist-before-ack consume loop with retry                 | PowerSync/Ablo patterns       |
+| `slot.ts`, `slotLease.ts`, `backfill.ts`, `watermark.ts` | slot ownership, initial snapshot, and durable progress     | Ablo                          |
+| `sources.ts`, `fleet.ts`, `start.ts`                     | registry resolution, reconciliation, start/stop lifecycle  | Ablo                          |
+| `preflight.ts`, `readiness.ts`, `publicationDrift.ts`    | registration checks and runtime diagnostics                | Ablo                          |
+
+## Data flow
+
+```
+registered Postgres source
+  → replication slot + initial snapshot
+  → pgoutput stream
+  → TransactionAssembler
+  → WalConsumer
+  → appendExternalDeltas(controlSql, deltas, context)
+  → persist watermark
+  → acknowledge commit LSN
+```
+
+### Mapping (in `TransactionAssembler`, mirrors `events.ts:eventsToDeltas`)
+
+- `actionType`: `insert→'I'`, `update→'U'`, `delete→'D'` (the 1:1 pgoutput↔Ablo coincidence).
+- `modelName`: `mapping.tableToModel(schema, table)` — `null` skips the change.
+- `modelId`: `mapping.rowToModelId(key)` over the replica-identity key.
+- `data`: the row bound as an **object, never pre-stringified** (the jsonb double-encode trap at `deltaAppend.ts`).
+- `transactionId`: `String(xid)`.
+
+## Load-bearing invariants
+
+- **Persist-before-ack** (`WalConsumer`): `appendExternalDeltas` and the watermark transaction resolve before `ack(commitLsn)`. A crash between them replays work instead of losing it.
+- **Keepalive watermark** (`streamAdapter.ts`, `stream.ts`): every reply carries the last confirmed LSN, never the server's live position — the timed status update included.
+- **Liveness off the socket** (`stream.ts`): a status update goes out every 75% of the upstream's `wal_sender_timeout` whether or not the consumer is reading, so backpressure cannot get the connection terminated for silence; inbound silence on a stream we are reading for twice that long destroys it and falls into the per-source backoff. A `wal_sender_timeout` of 0 runs untimed.
+- **Failover-capable slot** (`slot.ts`): from PostgreSQL 17 the slot is created with `FAILOVER true`, so a customer failover leaves our position intact instead of forcing the re-snapshot path. It only takes effect where the standby has `sync_replication_slots = on`, which the preflight recommends and never requires.
+- **Fresh subscription per retry** (`WalConsumer`): every backoff iteration opens a new subscription so a half-dead socket / stale relation cache never carries into the retry.
+- **Per-source isolation** (`start.ts`, `fleet.ts`): one broken source reports and retries without stopping healthy sources.
+- **Runtime reconciliation** (`fleet.ts`): registrations, removals, schema changes, and secret rotations converge without a server restart.
+
+## Tests
+
+Unit tests live beside the implementation in `replication/postgres/__tests__`. Real-Postgres coverage is grouped under `src/__journeys__/postgres-replication-*.journey.test.ts`: registration, registry migration, backfill, live streaming, source changes, bootstrap, query serving, read cutover, and customer-database isolation.
+
+## Operations
+
+Registration is the enable signal. `startPostgresReplication` starts the fleet after the server begins listening; `postgresReplicationReady` is drained during graceful shutdown. Use `docs/runbooks/connect-customer-database-postgres-replication.md` for source setup and live verification.

package/docs/internal/serializable-schema.md

@@ -0,0 +1,119 @@
+# A `Schema` is serializable
+
+A `Schema` (output of `defineSchema`) is JSON-serializable except for two
+things, both of which are client-only:
+
+- **Zod validators:** `model().schema` / `.shape`, `Schema.validators`. Used
+  by the client for type inference + validation. The server never reads them
+  (it checks `information_schema.columns` and does no field-shape validation in
+  the commit path).
+
+Everything the server reads — `typename`, `tableName`, `mutable`, `load`, the
+canonical `tenancy` descriptor (the `policy` authoring option is normalized away
+at build), bootstrap hints, `relations` (`foreignKeyColumn`), field names, and
+`identityRoles` — is plain data.
+
+## Identity roles are pure data
+
+```ts
+interface IdentityRole {
+  kind: string;
+  template: string;            // 'org:{id}'
+  source: IdentityRoleSource;  // { field: 'organizationId', multi: false }
+}
+```
+
+The runtime behaviour lives in `extractIdentityIds(identity, source)`, a pure
+function `composeIdentitySyncGroups` calls once per role. `identityRole({ kind,
+template, source, multi? })` is the factory. Absent/falsy fields yield `[]`, so
+a role whose field isn't present (a user with no `teamIds`) is a silent no-op —
+org-only, org+user, and org+team are just different `identityRoles` arrays, not
+different code paths. The engine ships zero prefixes; `org:`/`user:`/`team:`
+live only in `ablo.schema.ts`.
+
+## Why this matters
+
+Because a `Schema` carries no closures, the same object works in-process and,
+for a hosted multi-tenant server, after being reconstructed from JSON over the
+control plane (the GraphQL `printSchema` / `buildSchema` model). One type,
+both places — no separate server-side schema type.
+
+`apps/sync-server` reads the live `schema` directly today
+(`buildModelMap(schema)`, `composeIdentitySyncGroups` via the `@ablo/schema`
+wrapper).
+
+## Trust boundary
+
+Never trust a client-connection schema for authz (Zero/Convex/Instant). A
+client connection may carry only the schema **version** for compatibility
+gating; the authoritative `Schema` arrives over an authenticated control-plane
+path. The identity passed to `composeIdentitySyncGroups` is server-resolved
+trusted claims.
+
+## Wire form (`serialize.ts`)
+
+`serializeSchema(schema): string` / `parseSchema(json): Schema` are the
+control-plane transport — the GraphQL `printSchema`/`buildSchema` model. The
+JSON (`SchemaJSON`, envelope `{ v, models, identityRoles }`) carries every
+model's routing/scoping metadata, relations (incl. resolved
+`foreignKeyColumn`), field metadata, and identity roles. `parseSchema` rebuilds
+each model's Zod permissively from `FieldMeta` (the server does no field-shape
+validation) and drops `computed` closures. `schemaHash(schema)` is the stable
+FNV-1a content hash used for connect-time gating. Round-trip tested in
+`__tests__/serialize.test.ts`.
+
+## Storage + runtime resolution (`apps/sync-server/src/schema/`): built
+
+- **`ablo_schemas` table** (`packages/database/prisma/models/sync.prisma`,
+  `SchemaArtifact`) — `(organizationId, version, schemaJson, schemaHash, state,
+  error, createdBy, createdAt, activatedAt)`, unique `(orgId, version)`. State
+  `pending|validated|active|overwritten|failed`, ≤1 active per tenant (Convex
+  `_schemas` machine; Zero's "row in the operational DB"). *Migration written,
+  not applied — 0 users, Neon direct-endpoint rule.*
+- **`pgSchemaStore` / `memorySchemaStore`** (`schemaStore.ts`) — mirrors
+  `pgApiKeyStore`. `insertPending` assigns `MAX(version)+1`; `activate` is a
+  transaction that demotes the current active → `overwritten` then promotes the
+  target. State-machine invariants tested.
+- **`createSchemaRegistry(store)`** (`schemaRegistry.ts`) — `load(orgId)` parses
+  the active artifact's `schemaJson` to a `Schema` and caches it (shared
+  in-flight promise across concurrent cold loads); `invalidate(orgId)` busts it
+  on activation (Convex `schema_registry`). This is the seam that turns the
+  boot-time `import { schema }` into per-tenant runtime resolution.
+
+## Push route (`apps/sync-server/src/routes/schema.ts`): built
+
+`POST /api/schema`, mounted in `index.ts` (`schemaRoutes({ provider, store,
+registry })`). Auth: secret `sk_` key carrying the `schema:push` scope —
+`Identity.scopes` was added and `apiKeyProvider` now populates it from the key
+row's `scopes` column (restricted `rk_` keys get no `scopes`, so they're
+excluded). Tenant comes from `identity.organizationId`, never the body. Flow:
+read `{ schema, force? }` → validate via `parseSchema` (throws → 400) →
+authoritative hash via `schemaHash(parsed)` → reject removed-model changes (409)
+unless `force` → no-op fast path on identical hash (200) → `insertPending` →
+`activate` → `registry.invalidate(org)` → 201 `{ schemaId, version, hash }`. 6
+route tests.
+
+## CLI (`packages/ablo-cli`): built
+
+`ablo push` (`src/push.ts`, dispatched from `index.ts`). Imports the
+user's `sync/schema.ts` at runtime via tsx's `tsImport` (the real object —
+`migrate`'s regex parse can't produce a faithful AST), then `serializeSchema`
++ `schemaHash` and POSTs `{ schema, force, renames }` to `POST /api/schema`
+with `Authorization: Bearer $ABLO_API_KEY`. Flags: `--schema`, `--export`,
+`--url` (`$ABLO_API_URL`, default `https://api.abloatai.com`), `--force`,
+`--rename old:new` (repeatable). The route honors `renames` so a renamed model
+isn't flagged as a removed-model incompatibility. `parsePushArgs` unit-tested.
+
+## Schema drift is advisory
+
+Bootstrap includes the tenant's active schema hash. The client compares it to
+its built-in hash and warns once when they differ. Hash drift never closes the
+WebSocket: an additive rollout must allow old and new clients to overlap while
+data is expanded, dual-read/written, backfilled, verified, and finally
+contracted. Breaking wire shapes use the protocol-version codec registry
+instead.
+
+## Not built yet
+- **Switch the hot paths to per-tenant `registry.load(org)`:** boot still does
+  the single-tenant `import { schema }`; `buildModelMap`/bootstrap/commit
+  reading the registry per request is the final multi-tenant wiring.

package/docs/internal/structure.md

@@ -0,0 +1,32 @@
+# Repository Structure
+
+The public repository preserves the same ownership boundaries as the main
+monorepo. `@abloatai/ablo` is the product package; the packages beneath it are
+implementation owners and first-party extension surfaces.
+
+| Workspace | Responsibility |
+| --- | --- |
+| `packages/ablo` | Branded SDK, public entrypoints, docs, examples, release assets |
+| `packages/transaction` | Headless HTTP client, canonical contracts, reads, commits, settlement, claims, durable observation |
+| `packages/humans` | Reactive materializer, WebSocket transport, presence, browser persistence, React |
+| `packages/agent` | Agent behavior, perception, and coordination helpers |
+| `packages/cli` | Project setup, database connection, schema operations, and diagnostics |
+| `packages/tsconfig` | Private shared compiler configuration |
+
+Applications install and import `@abloatai/ablo`. The root entrypoint is the
+headless HTTP API. Human-facing reactive behavior is explicit:
+
+```ts
+import Ablo from '@abloatai/ablo';
+import ReactiveAblo from '@abloatai/ablo/client';
+import { AbloProvider, useAblo } from '@abloatai/ablo/react';
+```
+
+The backend implementation remains in `apps/sync-server` in the private
+monorepo. It consumes transaction contracts but is not part of the public SDK
+repository.
+
+Internal packages must not import the branded facade. Dependencies point from
+the facade to the owners, from humans and agents to transaction, and never back
+upward. The public mirror copies these workspaces as workspaces; it does not
+flatten them or generate compatibility source.