@abloatai/ablo 0.48.0 → 0.50.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +2 -2
- package/CHANGELOG.md +144 -2
- package/README.md +12 -12
- package/dist/ai-sdk.d.ts +11 -0
- package/dist/ai-sdk.d.ts.map +1 -1
- package/dist/ai-sdk.js +15 -0
- package/dist/ai-sdk.js.map +1 -1
- package/dist/batching.d.ts +7 -0
- package/dist/batching.d.ts.map +1 -0
- package/dist/batching.js +7 -0
- package/dist/batching.js.map +1 -0
- package/dist/client.d.ts +12 -0
- package/dist/client.d.ts.map +1 -1
- package/dist/client.js +9 -0
- package/dist/client.js.map +1 -1
- package/dist/context/await.d.ts +10 -0
- package/dist/context/await.d.ts.map +1 -0
- package/dist/context/await.js +33 -0
- package/dist/context/await.js.map +1 -0
- package/dist/context/evidence.d.ts +11 -0
- package/dist/context/evidence.d.ts.map +1 -0
- package/dist/context/evidence.js +53 -0
- package/dist/context/evidence.js.map +1 -0
- package/dist/context/sources.d.ts +21 -0
- package/dist/context/sources.d.ts.map +1 -0
- package/dist/context/sources.js +36 -0
- package/dist/context/sources.js.map +1 -0
- package/dist/context.d.ts +22 -0
- package/dist/context.d.ts.map +1 -0
- package/dist/context.js +33 -0
- package/dist/context.js.map +1 -0
- package/dist/coordination.d.ts +8 -0
- package/dist/coordination.d.ts.map +1 -1
- package/dist/index.d.ts +8 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +7 -0
- package/dist/index.js.map +1 -1
- package/docs/agent-messaging.md +2 -2
- package/docs/api.md +2 -2
- package/docs/concurrency-convention.md +82 -268
- package/docs/context.md +170 -0
- package/docs/coordination.md +178 -905
- package/docs/data-sources.md +32 -1
- package/docs/examples/ai-sdk-tool.md +4 -0
- package/docs/guarantees.md +29 -21
- package/docs/how-it-works.md +10 -40
- package/docs/identity.md +48 -0
- package/docs/index.md +2 -2
- package/docs/integrations.md +4 -0
- package/docs/migration.md +46 -443
- package/docs/operating-on-your-database.md +29 -36
- package/examples/README.md +24 -0
- package/examples/agent-turn.ts +37 -0
- package/examples/data-source/ablo-driver.ts +4 -4
- package/examples/data-source/customer-server.ts +9 -4
- package/examples/data-source/schema.ts +1 -1
- package/examples/expensive-agent-turn.ts +76 -0
- package/examples/quickstart.ts +9 -7
- package/package.json +19 -5
- package/docs/internal/README.md +0 -18
- package/docs/internal/agent-fleet-coordination-design.md +0 -171
- package/docs/internal/commit-identifiers.md +0 -91
- package/docs/internal/concurrency-open-decisions.md +0 -37
- package/docs/internal/data-source-reverse-channel.md +0 -147
- package/docs/internal/per-field-conflict-detection.md +0 -165
- package/docs/internal/postgres-replication.md +0 -64
- package/docs/internal/serializable-schema.md +0 -119
- package/docs/internal/structure.md +0 -36
|
@@ -1,147 +0,0 @@
|
|
|
1
|
-
# Data Source Reverse Channel (local-dev parity)
|
|
2
|
-
|
|
3
|
-
Maintainer scoping doc. Closes the one real day-one DX gap in Data Source
|
|
4
|
-
mode: the `commit`/`load`/`list` legs are inbound webhooks (Ablo → your
|
|
5
|
-
endpoint), so on `localhost` they need a tunnel (ngrok/cloudflared). This
|
|
6
|
-
scopes a built-in reverse channel so Data Source works on localhost without a
|
|
7
|
-
public tunnel.
|
|
8
|
-
|
|
9
|
-
## The gap, precisely
|
|
10
|
-
|
|
11
|
-
Data Source has two directions today:
|
|
12
|
-
|
|
13
|
-
| Leg | Direction | Transport(s) today | localhost-friendly? |
|
|
14
|
-
|---|---|---|---|
|
|
15
|
-
| `events` (external writes → Ablo) | customer → Ablo | **poll** (`events` handler, Ablo calls you) **+ push** (`createPushQueue` → `POST /api/source/events`, you call Ablo) | ✅ yes, via push |
|
|
16
|
-
| `commit` / `load` / `list` | Ablo → customer | **inbound webhook only** (`dataSource()` route) | ❌ no: needs public URL |
|
|
17
|
-
|
|
18
|
-
The asymmetry is the whole bug. `events` already ships an outbound transport
|
|
19
|
-
(`src/source/pushQueue.ts`), so external writes reach Ablo from localhost
|
|
20
|
-
without a tunnel. The `commit`/`load`/`list` leg never got one, so Ablo Cloud
|
|
21
|
-
has no way to reach a `localhost:3000` dev server.
|
|
22
|
-
|
|
23
|
-
Inbound is exactly what localhost cannot receive. Managed mode has no inbound
|
|
24
|
-
leg (the browser/SDK opens the only connection, outbound to Ablo), which is why
|
|
25
|
-
managed mode "just works" locally and Data Source doesn't.
|
|
26
|
-
|
|
27
|
-
## Prior art
|
|
28
|
-
|
|
29
|
-
- **Outbound relay:** the CLI opens a WebSocket to the hosted service, receives
|
|
30
|
-
events over that connection, and forwards them to `localhost`. No public URL
|
|
31
|
-
or tunnel is required. We want the same for the `commit`/`load`/`list` leg.
|
|
32
|
-
- **Our own `createPushQueue`:** already proves the outbound-from-customer
|
|
33
|
-
pattern for the `events` leg. The reverse channel is the symmetric primitive
|
|
34
|
-
for the other direction.
|
|
35
|
-
|
|
36
|
-
## Design
|
|
37
|
-
|
|
38
|
-
A customer-run **source connector** dials out to Ablo Cloud and serves
|
|
39
|
-
`commit`/`load`/`list` over the open connection, instead of Ablo making inbound
|
|
40
|
-
HTTP calls.
|
|
41
|
-
|
|
42
|
-
```
|
|
43
|
-
LOCAL DEV (no public URL)
|
|
44
|
-
|
|
45
|
-
ablo client ──ws──▶ Ablo Cloud ──┐
|
|
46
|
-
│ (no inbound HTTP to localhost)
|
|
47
|
-
customer connector ──ws (dial-out)──▶ Ablo Cloud
|
|
48
|
-
│ drains pending commit/load/list for this source
|
|
49
|
-
▼
|
|
50
|
-
dataSource(options) ← UNCHANGED handler, fed a synthesized Request
|
|
51
|
-
▼
|
|
52
|
-
local Postgres
|
|
53
|
-
│ signed response posted back up the same ws
|
|
54
|
-
└──────────────────────────────────────▶ Ablo Cloud ──ws──▶ ablo client
|
|
55
|
-
```
|
|
56
|
-
|
|
57
|
-
### Why it's small
|
|
58
|
-
|
|
59
|
-
`dataSource(options)` is already `(request: Request) => Promise<Response>` in
|
|
60
|
-
`src/source/factory.ts`. The connector does not reimplement any handler logic —
|
|
61
|
-
it:
|
|
62
|
-
|
|
63
|
-
1. Opens a WS to a new Ablo Cloud endpoint (e.g. `/v1/source/listen`),
|
|
64
|
-
authenticating with the project API key.
|
|
65
|
-
2. Registers which source/org it serves. Ablo Cloud routes that source's
|
|
66
|
-
`commit`/`load`/`list` requests to this socket instead of the configured
|
|
67
|
-
webhook URL (when a live connector is attached).
|
|
68
|
-
3. For each drained request frame: synthesize a `Request` with the same signed
|
|
69
|
-
headers Ablo would have sent, call the customer's existing `dataSource`
|
|
70
|
-
handler, and post the `Response` back up the socket.
|
|
71
|
-
|
|
72
|
-
Customer-side surface is one wrapper around the handler they already wrote:
|
|
73
|
-
|
|
74
|
-
```ts
|
|
75
|
-
// dev only — same handler object as the deployed route
|
|
76
|
-
import { dataSource, createSourceConnector } from '@abloatai/ablo';
|
|
77
|
-
import { sourceOptions } from './ablo.source'; // shared with route.ts
|
|
78
|
-
|
|
79
|
-
const connector = createSourceConnector({
|
|
80
|
-
apiKey: process.env.ABLO_API_KEY!, // sk_* bound to a child branch
|
|
81
|
-
handler: dataSource(sourceOptions), // the unchanged (Request)=>Response
|
|
82
|
-
});
|
|
83
|
-
await connector.run(abortSignal);
|
|
84
|
-
```
|
|
85
|
-
|
|
86
|
-
`route.ts` (deployed) and the connector (local) share the same
|
|
87
|
-
`sourceOptions` — zero handler drift.
|
|
88
|
-
|
|
89
|
-
### Server side (sync-server)
|
|
90
|
-
|
|
91
|
-
- New WS endpoint `/v1/source/listen`. Auth: project API key → resolves the
|
|
92
|
-
source. Reject if the persisted key binding is the root branch unless the
|
|
93
|
-
source explicitly opts into a production reverse channel (see below).
|
|
94
|
-
- Per-source request queue. When a `commit`/`load`/`list` needs the customer
|
|
95
|
-
and a connector is attached, enqueue + drain down the socket instead of
|
|
96
|
-
POSTing the webhook URL. Reuse the same signed-envelope shape so the
|
|
97
|
-
customer handler verifies identically (`verifyAbloSourceRequest` unchanged).
|
|
98
|
-
- Fallback: no connector attached → existing inbound webhook path. The
|
|
99
|
-
reverse channel is purely additive; nothing changes for deployed apps.
|
|
100
|
-
|
|
101
|
-
### Signature / security
|
|
102
|
-
|
|
103
|
-
- The drained frames carry the **same** Standard Webhooks signature
|
|
104
|
-
(`webhook-id`/`webhook-timestamp`/`webhook-signature`) computed with the
|
|
105
|
-
project key, so the connector verifies them through the existing
|
|
106
|
-
`verifyAbloSourceRequest` with no special-casing. The transport changes; the
|
|
107
|
-
trust model does not.
|
|
108
|
-
- Gate to non-root branch bindings by default. The DB still stays canonical in
|
|
109
|
-
the customer's process; nothing here gives Ablo the `DATABASE_URL`.
|
|
110
|
-
|
|
111
|
-
## Development-branch interplay
|
|
112
|
-
|
|
113
|
-
The reverse channel is the natural home for child-branch traffic: a local
|
|
114
|
-
connector attached with a child-bound `sk_*` receives that branch's commits,
|
|
115
|
-
runs them against the customer's development DB, and the SDK sees confirmed
|
|
116
|
-
rows and fan-out exactly as on the root. The server resolves child versus root
|
|
117
|
-
from the persisted branch binding, never from the key spelling.
|
|
118
|
-
|
|
119
|
-
## Production stance
|
|
120
|
-
|
|
121
|
-
Keep the inbound webhook as the default deployed transport — it's lower
|
|
122
|
-
latency (no long-lived socket to babysit) and stateless. The reverse channel
|
|
123
|
-
is primarily the **dev** affordance. A secondary, opt-in use is a
|
|
124
|
-
"no-public-URL deploy" mode for customers who cannot expose an inbound
|
|
125
|
-
endpoint at all (locked-down VPCs); that's a follow-on, not the initial scope.
|
|
126
|
-
|
|
127
|
-
## Scope boundary (what this is NOT)
|
|
128
|
-
|
|
129
|
-
- Not a generic tunnel — it forwards only signed Ablo source frames for one
|
|
130
|
-
source, not arbitrary traffic.
|
|
131
|
-
- Not a change to the handler contract — `dataSource` is untouched; the
|
|
132
|
-
connector wraps its handler.
|
|
133
|
-
- Not a managed-mode change — managed mode has no inbound leg and is unaffected.
|
|
134
|
-
|
|
135
|
-
## Touch list (when built)
|
|
136
|
-
|
|
137
|
-
- `packages/transaction/src/source/connector.ts` — `createSourceConnector`
|
|
138
|
-
(dial-out WS client; synthesize Request → existing handler → post Response).
|
|
139
|
-
- `packages/transaction` export surface — expose `createSourceConnector` next to
|
|
140
|
-
`createPushQueue`.
|
|
141
|
-
- `apps/sync-server` — `/v1/source/listen` WS endpoint + per-source request
|
|
142
|
-
queue + "drain to connector if attached, else webhook" branch in the source
|
|
143
|
-
dispatch path.
|
|
144
|
-
- `docs/data-sources.md` — document the local-dev loop (the current docs only
|
|
145
|
-
describe the public-HTTPS webhook).
|
|
146
|
-
- Tests: connector round-trip (drained commit → handler → response), signature
|
|
147
|
-
parity with the webhook path, fallback-to-webhook when no connector attached.
|
|
@@ -1,165 +0,0 @@
|
|
|
1
|
-
# Per-Field Conflict Detection (Track A)
|
|
2
|
-
|
|
3
|
-
Maintainer decision doc. Scopes the move from entity-level to field-level stale
|
|
4
|
-
detection in `executeCommit`. Library-free; restores Linear parity for the
|
|
5
|
-
disjoint-field case while keeping the agent-specific `readAt` reject.
|
|
6
|
-
|
|
7
|
-
## Problem
|
|
8
|
-
|
|
9
|
-
`executeCommit` Step 0 (`apps/sync-server/src/mutators/commit.ts`) detects stale
|
|
10
|
-
writes at **entity granularity**:
|
|
11
|
-
|
|
12
|
-
```sql
|
|
13
|
-
SELECT MAX(id) FROM sync_deltas WHERE model_name = ? AND model_id = ?
|
|
14
|
-
```
|
|
15
|
-
|
|
16
|
-
If `observed > op.readAt`, the op conflicts. This means two writers touching
|
|
17
|
-
**different fields** of the same row collide falsely: agent A sets
|
|
18
|
-
`report.status`, human B sets `report.reviewer`, B carried a `readAt` from before A's
|
|
19
|
-
write → B is rejected with `AbloStaleContextError`, even though the edits never
|
|
20
|
-
overlapped.
|
|
21
|
-
|
|
22
|
-
This is an over-rejection. It is also stricter than the system we
|
|
23
|
-
reverse-engineered from (Linear), which never had this problem.
|
|
24
|
-
|
|
25
|
-
## Why Linear says this is right
|
|
26
|
-
|
|
27
|
-
The model is reverse-engineered from Linear's sync engine. Linear's design
|
|
28
|
-
confirms every load-bearing choice here:
|
|
29
|
-
|
|
30
|
-
- **Transactions are property-level.** Linear's `UpdateTransaction` records
|
|
31
|
-
"the name of the changed property and its previous value" — it carries only
|
|
32
|
-
the changed properties, not a whole-object snapshot. Our `changed_fields`
|
|
33
|
-
column re-derives exactly that.
|
|
34
|
-
- **Resolution is last-writer-wins, per property, by total order:** `syncId`
|
|
35
|
-
(our `sync_id_seq`) is the total order. A partial transaction applies only its
|
|
36
|
-
properties, so two clients editing **different** properties both win — LWW
|
|
37
|
-
only bites on the **same** property.
|
|
38
|
-
- **CRDT is used only for issue descriptions.** Linear keeps LWW-per-property
|
|
39
|
-
for structured fields and reserves a CRDT for the one rich-text body. That is
|
|
40
|
-
the same Track A / Track B line we draw: this doc is Track A; rich-text bodies
|
|
41
|
-
(TipTap `content_json`) are out of scope and belong to a separate CRDT track.
|
|
42
|
-
|
|
43
|
-
So Track A is not a new feature — it **restores Linear parity** at the property
|
|
44
|
-
level we had flattened to entity level.
|
|
45
|
-
|
|
46
|
-
Sources: [reverse-linear-sync-engine (CTO-endorsed)](https://github.com/wzhudev/reverse-linear-sync-engine/blob/main/SUMMARY.md),
|
|
47
|
-
[Architectures for Central Server Collaboration — Weidner](https://mattweidner.com/2024/06/04/server-architectures.html).
|
|
48
|
-
|
|
49
|
-
## The constraint that shapes the design
|
|
50
|
-
|
|
51
|
-
`sync_deltas.data` stores the **full post-update row**, not the changed columns.
|
|
52
|
-
This was a deliberate change (see `feedback_partial_update_delta_ui_drift`) so
|
|
53
|
-
the live-pool update path fires MobX reactivity for nested fields. Consequence:
|
|
54
|
-
we **cannot** recover "which fields did this delta change" from `data` — a
|
|
55
|
-
full-row snapshot does not tell you what moved.
|
|
56
|
-
|
|
57
|
-
We do have the changed set for free at write time: `Object.keys(snakeInput)` at
|
|
58
|
-
`commit.ts` UPDATE branch, after the unknown-column strip and before the
|
|
59
|
-
`updated_at` injection. So this is a write-side capture + a read-side
|
|
60
|
-
intersection, not a diff-the-snapshots problem.
|
|
61
|
-
|
|
62
|
-
## Design
|
|
63
|
-
|
|
64
|
-
### 1. Schema: `sync_deltas.changed_fields text[]` (nullable)
|
|
65
|
-
|
|
66
|
-
- Populate **only for UPDATE** with the real changed columns
|
|
67
|
-
(`Object.keys(snakeInput)` after strip, before `updated_at`).
|
|
68
|
-
- Leave `null` for CREATE / DELETE / ARCHIVE / UNARCHIVE.
|
|
69
|
-
|
|
70
|
-
`null` is semantically "whole-entity change" and always conflicts. This gives a
|
|
71
|
-
**safe migration**: every pre-migration delta is `null`, so detection falls back
|
|
72
|
-
to exactly today's entity-level behavior. Field granularity phases in only as
|
|
73
|
-
new deltas land — no risky backfill.
|
|
74
|
-
|
|
75
|
-
We store **field names only**, not previous values. LWW needs no value
|
|
76
|
-
comparison (latest `sync_id` wins); names are sufficient for the overlap check
|
|
77
|
-
that drives the optional reject. Prev-values would only matter for
|
|
78
|
-
"same field, same value ⇒ not a conflict" tie-breaking — defer it.
|
|
79
|
-
|
|
80
|
-
### 2. Detection rewrite: Step 0 (`commit.ts`)
|
|
81
|
-
|
|
82
|
-
Replace the scalar `MAX(id)` with a field-aware scan:
|
|
83
|
-
|
|
84
|
-
```ts
|
|
85
|
-
// op's own field set (snake-cased, framework cols excluded)
|
|
86
|
-
const opFields = new Set(Object.keys(op.input ?? {}).map(toSnakeCase));
|
|
87
|
-
|
|
88
|
-
const rows = await tx.unsafe(
|
|
89
|
-
`SELECT id, changed_fields FROM sync_deltas
|
|
90
|
-
WHERE model_name = $1 AND model_id = $2 AND id > $3
|
|
91
|
-
ORDER BY id DESC`,
|
|
92
|
-
[mapping.modelName, op.id, op.readAt] as never[],
|
|
93
|
-
);
|
|
94
|
-
|
|
95
|
-
// Conflict iff a newer delta touched a field this op also writes,
|
|
96
|
-
// OR a newer delta is whole-entity (changed_fields IS NULL → CREATE/DELETE).
|
|
97
|
-
const overlap = rows.find(
|
|
98
|
-
(r) => r.changedFields === null || r.changedFields.some((f) => opFields.has(f)),
|
|
99
|
-
);
|
|
100
|
-
if (overlap) {
|
|
101
|
-
conflicts.push({
|
|
102
|
-
/* ...existing fields... */
|
|
103
|
-
observedSyncId: overlap.id,
|
|
104
|
-
conflictingFields: intersect(overlap.changedFields, opFields),
|
|
105
|
-
});
|
|
106
|
-
}
|
|
107
|
-
```
|
|
108
|
-
|
|
109
|
-
Disjoint-field concurrent writes now produce **no conflict** — they both apply.
|
|
110
|
-
That is LWW-per-field achieved by *not rejecting*; no merge code.
|
|
111
|
-
|
|
112
|
-
### 3. Policy type: additive (`packages/transaction/src/policy/types.ts`)
|
|
113
|
-
|
|
114
|
-
Extend `StaleContextConflict` with:
|
|
115
|
-
|
|
116
|
-
```ts
|
|
117
|
-
readonly conflictingFields?: readonly string[];
|
|
118
|
-
```
|
|
119
|
-
|
|
120
|
-
Pure addition. `defaultPolicy` still rejects; existing policies compile
|
|
121
|
-
unchanged. A policy can now reason at field granularity, e.g. allow when the
|
|
122
|
-
only conflicting field is cosmetic.
|
|
123
|
-
|
|
124
|
-
### 4. Scope boundary (honest)
|
|
125
|
-
|
|
126
|
-
Granularity is **column-level**, not JSON-path. Two writers editing different
|
|
127
|
-
keys *inside* one `content_json` column still conflict — that is the rich-text
|
|
128
|
-
case (Track B / CRDT), not this. JSON Merge Patch (RFC 7386) sub-column
|
|
129
|
-
granularity is a later refinement on the same column; v1 stops at columns.
|
|
130
|
-
|
|
131
|
-
## Relationship to the existing `readAt` reject
|
|
132
|
-
|
|
133
|
-
Linear is pure LWW-per-property with no stale check. We keep `readAt` /
|
|
134
|
-
`onStale: 'reject'` as an **opt-in** for the agent-reasoned-against-stale-state
|
|
135
|
-
case (an LLM that read a stale value and reasoned on it is a real failure mode
|
|
136
|
-
humans rarely hit). After Track A:
|
|
137
|
-
|
|
138
|
-
- **No `readAt`** → LWW-per-field, Linear parity: disjoint fields never conflict.
|
|
139
|
-
- **`readAt` set** → reject only if a newer delta touched a field this op also
|
|
140
|
-
writes. The `changed_fields` column makes that intersection computable.
|
|
141
|
-
- **`onStale: 'overwrite'`** → unchanged; still skips detection entirely.
|
|
142
|
-
|
|
143
|
-
## Index
|
|
144
|
-
|
|
145
|
-
Verify a `(model_name, model_id, id)` index exists on `sync_deltas` (the old
|
|
146
|
-
`MAX(id)` relied on it too). The new query is a bounded range scan (`id >
|
|
147
|
-
readAt`, usually a small recent window) on the same index prefix.
|
|
148
|
-
|
|
149
|
-
## Tests (vitest, sync-server)
|
|
150
|
-
|
|
151
|
-
- Disjoint fields (A: `status`, B: `assignee`, same row, both stale `readAt`) →
|
|
152
|
-
**both commit, no `AbloStaleContextError`** — the regression that proves the win.
|
|
153
|
-
- Same field, both stale → still rejects (default policy unchanged).
|
|
154
|
-
- DELETE after `readAt` → conflicts regardless of op fields (`null`).
|
|
155
|
-
- Pre-migration delta (`null`) in the window → conflicts (back-compat).
|
|
156
|
-
- `onStale: 'overwrite'` → still skips detection.
|
|
157
|
-
|
|
158
|
-
## Touch list
|
|
159
|
-
|
|
160
|
-
- `sync_deltas` migration — add `changed_fields text[]`.
|
|
161
|
-
- `commit.ts` — Step 0 detect rewrite + UPDATE write path populates
|
|
162
|
-
`changed_fields` + `deltaInfos` shape.
|
|
163
|
-
- `apps/sync-server/src/db/deltas.ts` — insert path carries `changed_fields`.
|
|
164
|
-
- `packages/transaction/src/policy/types.ts` — additive `conflictingFields`.
|
|
165
|
-
- vitest in sync-server.
|
|
@@ -1,64 +0,0 @@
|
|
|
1
|
-
# Postgres replication: internal architecture
|
|
2
|
-
|
|
3
|
-
> **Status: wired and covered by unit and real-Postgres journeys.** This is server-internal code under `apps/sync-server/src/replication/postgres/`; it is not an SDK surface.
|
|
4
|
-
|
|
5
|
-
## Why this exists
|
|
6
|
-
|
|
7
|
-
Ablo observes a customer's Postgres through a publication and logical-replication slot. The customer owns the schema and write path. Ablo decodes committed changes, appends them to its control-plane log, and serves sync from that log. The low-level decoder and lifecycle draw on Zero and PowerSync patterns; ADR 0002 governs the product boundary.
|
|
8
|
-
|
|
9
|
-
## The one job
|
|
10
|
-
|
|
11
|
-
**Postgres `pgoutput` messages → `PreparedDelta[]` + a confirmed LSN.** The consumer writes through `appendExternalDeltas`; deltas land in the control-plane `sync_deltas` log and use the normal fan-out pipeline.
|
|
12
|
-
|
|
13
|
-
## Module map (`apps/sync-server/src/replication/postgres/`)
|
|
14
|
-
|
|
15
|
-
| file | role | provenance |
|
|
16
|
-
| -------------------------------------------------------- | ---------------------------------------------------------- | ----------------------------- |
|
|
17
|
-
| `binaryReader.ts` | big-endian protocol reader | modeled on Zero |
|
|
18
|
-
| `pgoutputTypes.ts`, `pgoutput.ts` | typed `pgoutput` messages and decoder | modeled on Zero |
|
|
19
|
-
| `lsn.ts` | `LSN` string ↔ `bigint` (`toBigInt`/`fromBigInt`) | ported subset ← Zero `lsn.ts` |
|
|
20
|
-
| `connection.ts`, `stream.ts`, `streamAdapter.ts` | dedicated query/replication connections and stream adapter | Zero/PowerSync patterns |
|
|
21
|
-
| `assembler.ts` | buffers a transaction and maps changes to `PreparedDelta` | Ablo adapter |
|
|
22
|
-
| `consumer.ts` | persist-before-ack consume loop with retry | PowerSync/Ablo patterns |
|
|
23
|
-
| `slot.ts`, `slotLease.ts`, `backfill.ts`, `watermark.ts` | slot ownership, initial snapshot, and durable progress | Ablo |
|
|
24
|
-
| `sources.ts`, `fleet.ts`, `start.ts` | registry resolution, reconciliation, start/stop lifecycle | Ablo |
|
|
25
|
-
| `preflight.ts`, `readiness.ts`, `publicationDrift.ts` | registration checks and runtime diagnostics | Ablo |
|
|
26
|
-
|
|
27
|
-
## Data flow
|
|
28
|
-
|
|
29
|
-
```
|
|
30
|
-
registered Postgres source
|
|
31
|
-
→ replication slot + initial snapshot
|
|
32
|
-
→ pgoutput stream
|
|
33
|
-
→ TransactionAssembler
|
|
34
|
-
→ WalConsumer
|
|
35
|
-
→ appendExternalDeltas(controlSql, deltas, context)
|
|
36
|
-
→ persist watermark
|
|
37
|
-
→ acknowledge commit LSN
|
|
38
|
-
```
|
|
39
|
-
|
|
40
|
-
### Mapping (in `TransactionAssembler`, mirrors `events.ts:eventsToDeltas`)
|
|
41
|
-
|
|
42
|
-
- `actionType`: `insert→'I'`, `update→'U'`, `delete→'D'` (the 1:1 pgoutput↔Ablo coincidence).
|
|
43
|
-
- `modelName`: `mapping.tableToModel(schema, table)` — `null` skips the change.
|
|
44
|
-
- `modelId`: `mapping.rowToModelId(key)` over the replica-identity key.
|
|
45
|
-
- `data`: the row bound as an **object, never pre-stringified** (the jsonb double-encode trap at `deltaAppend.ts`).
|
|
46
|
-
- `transactionId`: `String(xid)`.
|
|
47
|
-
|
|
48
|
-
## Load-bearing invariants
|
|
49
|
-
|
|
50
|
-
- **Persist-before-ack** (`WalConsumer`): `appendExternalDeltas` and the watermark transaction resolve before `ack(commitLsn)`. A crash between them replays work instead of losing it.
|
|
51
|
-
- **Keepalive watermark** (`streamAdapter.ts`, `stream.ts`): every reply carries the last confirmed LSN, never the server's live position — the timed status update included.
|
|
52
|
-
- **Liveness off the socket** (`stream.ts`): a status update goes out every 75% of the upstream's `wal_sender_timeout` whether or not the consumer is reading, so backpressure cannot get the connection terminated for silence; inbound silence on a stream we are reading for twice that long destroys it and falls into the per-source backoff. A `wal_sender_timeout` of 0 runs untimed.
|
|
53
|
-
- **Failover-capable slot** (`slot.ts`): from PostgreSQL 17 the slot is created with `FAILOVER true`, so a customer failover leaves our position intact instead of forcing the re-snapshot path. It only takes effect where the standby has `sync_replication_slots = on`, which the preflight recommends and never requires.
|
|
54
|
-
- **Fresh subscription per retry** (`WalConsumer`): every backoff iteration opens a new subscription so a half-dead socket / stale relation cache never carries into the retry.
|
|
55
|
-
- **Per-source isolation** (`start.ts`, `fleet.ts`): one broken source reports and retries without stopping healthy sources.
|
|
56
|
-
- **Runtime reconciliation** (`fleet.ts`): registrations, removals, schema changes, and secret rotations converge without a server restart.
|
|
57
|
-
|
|
58
|
-
## Tests
|
|
59
|
-
|
|
60
|
-
Unit tests live beside the implementation in `replication/postgres/__tests__`. Real-Postgres coverage is grouped under `src/__journeys__/postgres-replication-*.journey.test.ts`: registration, registry migration, backfill, live streaming, source changes, bootstrap, query serving, read cutover, and customer-database isolation.
|
|
61
|
-
|
|
62
|
-
## Operations
|
|
63
|
-
|
|
64
|
-
Registration is the enable signal. `startPostgresReplication` starts the fleet after the server begins listening; `postgresReplicationReady` is drained during graceful shutdown. Use `docs/runbooks/connect-customer-database-postgres-replication.md` for source setup and live verification.
|
|
@@ -1,119 +0,0 @@
|
|
|
1
|
-
# A `Schema` is serializable
|
|
2
|
-
|
|
3
|
-
A `Schema` (output of `defineSchema`) is JSON-serializable except for two
|
|
4
|
-
things, both of which are client-only:
|
|
5
|
-
|
|
6
|
-
- **Zod validators:** `model().schema` / `.shape`, `Schema.validators`. Used
|
|
7
|
-
by the client for type inference + validation. The server never reads them
|
|
8
|
-
(it checks `information_schema.columns` and does no field-shape validation in
|
|
9
|
-
the commit path).
|
|
10
|
-
|
|
11
|
-
Everything the server reads — `typename`, `tableName`, `mutable`, `load`, the
|
|
12
|
-
canonical `tenancy` descriptor (the `policy` authoring option is normalized away
|
|
13
|
-
at build), bootstrap hints, `relations` (`foreignKeyColumn`), field names, and
|
|
14
|
-
`identityRoles` — is plain data.
|
|
15
|
-
|
|
16
|
-
## Identity roles are pure data
|
|
17
|
-
|
|
18
|
-
```ts
|
|
19
|
-
interface IdentityRole {
|
|
20
|
-
kind: string;
|
|
21
|
-
template: string; // 'org:{id}'
|
|
22
|
-
source: IdentityRoleSource; // { field: 'organizationId', multi: false }
|
|
23
|
-
}
|
|
24
|
-
```
|
|
25
|
-
|
|
26
|
-
The runtime behaviour lives in `extractIdentityIds(identity, source)`, a pure
|
|
27
|
-
function `composeIdentitySyncGroups` calls once per role. `identityRole({ kind,
|
|
28
|
-
template, source, multi? })` is the factory. Absent/falsy fields yield `[]`, so
|
|
29
|
-
a role whose field isn't present (a user with no `teamIds`) is a silent no-op —
|
|
30
|
-
org-only, org+user, and org+team are just different `identityRoles` arrays, not
|
|
31
|
-
different code paths. The engine ships zero prefixes; `org:`/`user:`/`team:`
|
|
32
|
-
live only in `ablo.schema.ts`.
|
|
33
|
-
|
|
34
|
-
## Why this matters
|
|
35
|
-
|
|
36
|
-
Because a `Schema` carries no closures, the same object works in-process and,
|
|
37
|
-
for a hosted multi-tenant server, after being reconstructed from JSON over the
|
|
38
|
-
control plane (the GraphQL `printSchema` / `buildSchema` model). One type,
|
|
39
|
-
both places — no separate server-side schema type.
|
|
40
|
-
|
|
41
|
-
`apps/sync-server` reads the live `schema` directly today
|
|
42
|
-
(`buildModelMap(schema)`, `composeIdentitySyncGroups` via the `@ablo/schema`
|
|
43
|
-
wrapper).
|
|
44
|
-
|
|
45
|
-
## Trust boundary
|
|
46
|
-
|
|
47
|
-
Never trust a client-connection schema for authz (Zero/Convex/Instant). A
|
|
48
|
-
client connection may carry only the schema **version** for compatibility
|
|
49
|
-
gating; the authoritative `Schema` arrives over an authenticated control-plane
|
|
50
|
-
path. The identity passed to `composeIdentitySyncGroups` is server-resolved
|
|
51
|
-
trusted claims.
|
|
52
|
-
|
|
53
|
-
## Wire form (`serialize.ts`)
|
|
54
|
-
|
|
55
|
-
`serializeSchema(schema): string` / `parseSchema(json): Schema` are the
|
|
56
|
-
control-plane transport — the GraphQL `printSchema`/`buildSchema` model. The
|
|
57
|
-
JSON (`SchemaJSON`, envelope `{ v, models, identityRoles }`) carries every
|
|
58
|
-
model's routing/scoping metadata, relations (incl. resolved
|
|
59
|
-
`foreignKeyColumn`), field metadata, and identity roles. `parseSchema` rebuilds
|
|
60
|
-
each model's Zod permissively from `FieldMeta` (the server does no field-shape
|
|
61
|
-
validation) and drops `computed` closures. `schemaHash(schema)` is the stable
|
|
62
|
-
FNV-1a content hash used for connect-time gating. Round-trip tested in
|
|
63
|
-
`__tests__/serialize.test.ts`.
|
|
64
|
-
|
|
65
|
-
## Storage + runtime resolution (`apps/sync-server/src/schema/`): built
|
|
66
|
-
|
|
67
|
-
- **`ablo_schemas` table** (`packages/database/prisma/models/sync.prisma`,
|
|
68
|
-
`SchemaArtifact`) — `(organizationId, version, schemaJson, schemaHash, state,
|
|
69
|
-
error, createdBy, createdAt, activatedAt)`, unique `(orgId, version)`. State
|
|
70
|
-
`pending|validated|active|overwritten|failed`, ≤1 active per tenant (Convex
|
|
71
|
-
`_schemas` machine; Zero's "row in the operational DB"). *Migration written,
|
|
72
|
-
not applied — 0 users at the time.*
|
|
73
|
-
- **`pgSchemaStore` / `memorySchemaStore`** (`schemaStore.ts`) — mirrors
|
|
74
|
-
`pgApiKeyStore`. `insertPending` assigns `MAX(version)+1`; `activate` is a
|
|
75
|
-
transaction that demotes the current active → `overwritten` then promotes the
|
|
76
|
-
target. State-machine invariants tested.
|
|
77
|
-
- **`createSchemaRegistry(store)`** (`schemaRegistry.ts`) — `load(orgId)` parses
|
|
78
|
-
the active artifact's `schemaJson` to a `Schema` and caches it (shared
|
|
79
|
-
in-flight promise across concurrent cold loads); `invalidate(orgId)` busts it
|
|
80
|
-
on activation (Convex `schema_registry`). This is the seam that turns the
|
|
81
|
-
boot-time `import { schema }` into per-tenant runtime resolution.
|
|
82
|
-
|
|
83
|
-
## Push route (`apps/sync-server/src/routes/schema.ts`): built
|
|
84
|
-
|
|
85
|
-
`POST /api/schema`, mounted in `index.ts` (`schemaRoutes({ provider, store,
|
|
86
|
-
registry })`). Auth: secret `sk_` key carrying the `schema:push` scope —
|
|
87
|
-
`Identity.scopes` was added and `apiKeyProvider` now populates it from the key
|
|
88
|
-
row's `scopes` column (restricted `rk_` keys get no `scopes`, so they're
|
|
89
|
-
excluded). Tenant comes from `identity.organizationId`, never the body. Flow:
|
|
90
|
-
read `{ schema, force? }` → validate via `parseSchema` (throws → 400) →
|
|
91
|
-
authoritative hash via `schemaHash(parsed)` → reject removed-model changes (409)
|
|
92
|
-
unless `force` → no-op fast path on identical hash (200) → `insertPending` →
|
|
93
|
-
`activate` → `registry.invalidate(org)` → 201 `{ schemaId, version, hash }`. 6
|
|
94
|
-
route tests.
|
|
95
|
-
|
|
96
|
-
## CLI (`packages/ablo-cli`): built
|
|
97
|
-
|
|
98
|
-
`ablo push` (`src/push.ts`, dispatched from `index.ts`). Imports the
|
|
99
|
-
user's `sync/schema.ts` at runtime via tsx's `tsImport` (the real object —
|
|
100
|
-
`migrate`'s regex parse can't produce a faithful AST), then `serializeSchema`
|
|
101
|
-
+ `schemaHash` and POSTs `{ schema, force, renames }` to `POST /api/schema`
|
|
102
|
-
with `Authorization: Bearer $ABLO_API_KEY`. Flags: `--schema`, `--export`,
|
|
103
|
-
`--url` (`$ABLO_API_URL`, default `https://api.abloatai.com`), `--force`,
|
|
104
|
-
`--rename old:new` (repeatable). The route honors `renames` so a renamed model
|
|
105
|
-
isn't flagged as a removed-model incompatibility. `parsePushArgs` unit-tested.
|
|
106
|
-
|
|
107
|
-
## Schema drift is advisory
|
|
108
|
-
|
|
109
|
-
Bootstrap includes the tenant's active schema hash. The client compares it to
|
|
110
|
-
its built-in hash and warns once when they differ. Hash drift never closes the
|
|
111
|
-
WebSocket: an additive rollout must allow old and new clients to overlap while
|
|
112
|
-
data is expanded, dual-read/written, backfilled, verified, and finally
|
|
113
|
-
contracted. Breaking wire shapes use the protocol-version codec registry
|
|
114
|
-
instead.
|
|
115
|
-
|
|
116
|
-
## Not built yet
|
|
117
|
-
- **Switch the hot paths to per-tenant `registry.load(org)`:** boot still does
|
|
118
|
-
the single-tenant `import { schema }`; `buildModelMap`/bootstrap/commit
|
|
119
|
-
reading the registry per request is the final multi-tenant wiring.
|
|
@@ -1,36 +0,0 @@
|
|
|
1
|
-
# Repository Structure
|
|
2
|
-
|
|
3
|
-
For a verb-by-verb guide to declarations and implementations—including exactly
|
|
4
|
-
where `create`, `update`, `delete`, and `claim` live—read the public
|
|
5
|
-
[`CODEMAP.md`](../../CODEMAP.md).
|
|
6
|
-
|
|
7
|
-
The public repository preserves the same ownership boundaries as the main
|
|
8
|
-
monorepo. `@abloatai/ablo` is the product package; the packages beneath it are
|
|
9
|
-
implementation owners and first-party extension surfaces.
|
|
10
|
-
|
|
11
|
-
| Workspace | Responsibility |
|
|
12
|
-
| --- | --- |
|
|
13
|
-
| `packages/ablo` | Branded SDK, public entrypoints, docs, examples, release assets |
|
|
14
|
-
| `packages/transaction` | Headless HTTP client, canonical contracts, reads, commits, settlement, claims, durable observation |
|
|
15
|
-
| `packages/humans` | Reactive materializer, WebSocket transport, presence, browser persistence, React |
|
|
16
|
-
| `packages/agent` | Agent behavior, perception, and coordination helpers |
|
|
17
|
-
| `packages/cli` | Project setup, database connection, schema operations, and diagnostics |
|
|
18
|
-
| `packages/tsconfig` | Private shared compiler configuration |
|
|
19
|
-
|
|
20
|
-
Applications install and import `@abloatai/ablo`. The root entrypoint is the
|
|
21
|
-
headless HTTP API. Human-facing reactive behavior is explicit:
|
|
22
|
-
|
|
23
|
-
```ts
|
|
24
|
-
import Ablo from '@abloatai/ablo';
|
|
25
|
-
import ReactiveAblo from '@abloatai/ablo/client';
|
|
26
|
-
import { AbloProvider, useAblo } from '@abloatai/ablo/react';
|
|
27
|
-
```
|
|
28
|
-
|
|
29
|
-
The backend implementation remains in `apps/sync-server` in the private
|
|
30
|
-
monorepo. It consumes transaction contracts but is not part of the public SDK
|
|
31
|
-
repository.
|
|
32
|
-
|
|
33
|
-
Internal packages must not import the branded facade. Dependencies point from
|
|
34
|
-
the facade to the owners, from humans and agents to transaction, and never back
|
|
35
|
-
upward. The public mirror copies these workspaces as workspaces; it does not
|
|
36
|
-
flatten them or generate compatibility source.
|