@interocitor/core 0.0.0-beta.10
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +706 -0
- package/dist/adapters/cloudflare.d.ts +78 -0
- package/dist/adapters/cloudflare.d.ts.map +1 -0
- package/dist/adapters/cloudflare.js +325 -0
- package/dist/adapters/google-drive.d.ts +64 -0
- package/dist/adapters/google-drive.d.ts.map +1 -0
- package/dist/adapters/google-drive.js +339 -0
- package/dist/adapters/memory.d.ts +53 -0
- package/dist/adapters/memory.d.ts.map +1 -0
- package/dist/adapters/memory.js +182 -0
- package/dist/adapters/webdav.d.ts +70 -0
- package/dist/adapters/webdav.d.ts.map +1 -0
- package/dist/adapters/webdav.js +323 -0
- package/dist/core/codec.d.ts +20 -0
- package/dist/core/codec.d.ts.map +1 -0
- package/dist/core/codec.js +102 -0
- package/dist/core/compaction.d.ts +45 -0
- package/dist/core/compaction.d.ts.map +1 -0
- package/dist/core/compaction.js +190 -0
- package/dist/core/connected-stores.d.ts +77 -0
- package/dist/core/connected-stores.d.ts.map +1 -0
- package/dist/core/connected-stores.js +76 -0
- package/dist/core/crdt.d.ts +36 -0
- package/dist/core/crdt.d.ts.map +1 -0
- package/dist/core/crdt.js +174 -0
- package/dist/core/errors.d.ts +47 -0
- package/dist/core/errors.d.ts.map +1 -0
- package/dist/core/errors.js +61 -0
- package/dist/core/flush.d.ts +9 -0
- package/dist/core/flush.d.ts.map +1 -0
- package/dist/core/flush.js +98 -0
- package/dist/core/hlc.d.ts +25 -0
- package/dist/core/hlc.d.ts.map +1 -0
- package/dist/core/hlc.js +75 -0
- package/dist/core/ids.d.ts +49 -0
- package/dist/core/ids.d.ts.map +1 -0
- package/dist/core/ids.js +132 -0
- package/dist/core/internals.d.ts +33 -0
- package/dist/core/internals.d.ts.map +1 -0
- package/dist/core/internals.js +72 -0
- package/dist/core/manifest.d.ts +56 -0
- package/dist/core/manifest.d.ts.map +1 -0
- package/dist/core/manifest.js +203 -0
- package/dist/core/pull.d.ts +26 -0
- package/dist/core/pull.d.ts.map +1 -0
- package/dist/core/pull.js +113 -0
- package/dist/core/row-id.d.ts +12 -0
- package/dist/core/row-id.d.ts.map +1 -0
- package/dist/core/row-id.js +11 -0
- package/dist/core/schema-types.d.ts +26 -0
- package/dist/core/schema-types.d.ts.map +1 -0
- package/dist/core/schema-types.js +31 -0
- package/dist/core/schema-types.type-test.d.ts +2 -0
- package/dist/core/schema-types.type-test.d.ts.map +1 -0
- package/dist/core/schema-types.type-test.js +224 -0
- package/dist/core/sync-engine.d.ts +364 -0
- package/dist/core/sync-engine.d.ts.map +1 -0
- package/dist/core/sync-engine.js +2475 -0
- package/dist/core/table.d.ts +260 -0
- package/dist/core/table.d.ts.map +1 -0
- package/dist/core/table.js +461 -0
- package/dist/core/types.d.ts +952 -0
- package/dist/core/types.d.ts.map +1 -0
- package/dist/core/types.js +6 -0
- package/dist/crypto/encryption.d.ts +61 -0
- package/dist/crypto/encryption.d.ts.map +1 -0
- package/dist/crypto/encryption.js +216 -0
- package/dist/crypto/keys.d.ts +48 -0
- package/dist/crypto/keys.d.ts.map +1 -0
- package/dist/crypto/keys.js +54 -0
- package/dist/handshake/channel.d.ts +117 -0
- package/dist/handshake/channel.d.ts.map +1 -0
- package/dist/handshake/channel.js +245 -0
- package/dist/handshake/index.d.ts +216 -0
- package/dist/handshake/index.d.ts.map +1 -0
- package/dist/handshake/index.js +199 -0
- package/dist/handshake/qr-public.d.ts +3 -0
- package/dist/handshake/qr-public.d.ts.map +1 -0
- package/dist/handshake/qr-public.js +1 -0
- package/dist/handshake/qr.d.ts +100 -0
- package/dist/handshake/qr.d.ts.map +1 -0
- package/dist/handshake/qr.js +102 -0
- package/dist/index.d.ts +50 -0
- package/dist/index.d.ts.map +1 -0
- package/dist/index.js +50 -0
- package/dist/storage/credential-store.d.ts +122 -0
- package/dist/storage/credential-store.d.ts.map +1 -0
- package/dist/storage/credential-store.js +356 -0
- package/dist/storage/local-store.d.ts +64 -0
- package/dist/storage/local-store.d.ts.map +1 -0
- package/dist/storage/local-store.js +490 -0
- package/dist/storage/reset.d.ts +10 -0
- package/dist/storage/reset.d.ts.map +1 -0
- package/dist/storage/reset.js +18 -0
- package/package.json +76 -0
package/README.md
ADDED
|
@@ -0,0 +1,706 @@
|
|
|
1
|
+
<p align="center">
|
|
2
|
+
<a href="https://github.com/TheUiTeam/interocitor">
|
|
3
|
+
<img src="../../docs/assets/hero.svg" alt="Interocitor" width="640" />
|
|
4
|
+
</a>
|
|
5
|
+
</p>
|
|
6
|
+
|
|
7
|
+
<p align="center">
|
|
8
|
+
<em>Encrypted local-first CRDT database and durable file store for browser apps.</em>
|
|
9
|
+
</p>
|
|
10
|
+
|
|
11
|
+
# @interocitor/core
|
|
12
|
+
|
|
13
|
+
End‑to‑end encrypted, local‑first app data over storage you already own
|
|
14
|
+
(Google Drive, WebDAV, Cloudflare R2, your own server). Interocitor gives
|
|
15
|
+
your app both structured CRDT rows and path-addressed files/images. The
|
|
16
|
+
cloud is a mailbox; merge and decryption happen on the device.
|
|
17
|
+
|
|
18
|
+
## What it is
|
|
19
|
+
|
|
20
|
+
- A client-side CRDT database for structured state. Local reads/writes go
|
|
21
|
+
through an embedded store (IndexedDB in the browser). The engine ships
|
|
22
|
+
diffs, not queries.
|
|
23
|
+
- A durable file store for blobs/images that belong to the same mesh.
|
|
24
|
+
Files are encrypted, path-addressed, and overwritten/deleted explicitly;
|
|
25
|
+
they do not merge or compact.
|
|
26
|
+
- A CRDT over per‑column HLC values. Every device converges to the same
|
|
27
|
+
row state without a central merge authority.
|
|
28
|
+
- An end‑to‑end encryption layer. The remote sees ciphertext blobs and
|
|
29
|
+
enough metadata to route them; nothing else.
|
|
30
|
+
- A pluggable transport. Backends provide object storage operations for
|
|
31
|
+
sync objects and, optionally, first-class durable file operations. See
|
|
32
|
+
`docs/adapter-contract.md`.
|
|
33
|
+
|
|
34
|
+
## Why
|
|
35
|
+
|
|
36
|
+
- **Offline‑first by construction.** Every read and write hits a local
|
|
37
|
+
store. Network is needed only to share state with other devices.
|
|
38
|
+
- **No server you have to operate.** The remote is dumb storage. You can
|
|
39
|
+
run the engine against a user's own Google Drive and ship zero
|
|
40
|
+
backend.
|
|
41
|
+
- **End‑to‑end encryption by default.** The default config encrypts
|
|
42
|
+
every change file and snapshot before it leaves the device.
|
|
43
|
+
- **Small, embeddable runtime.** No workers, no background services
|
|
44
|
+
required.
|
|
45
|
+
|
|
46
|
+
## Quick start
|
|
47
|
+
|
|
48
|
+
```ts
|
|
49
|
+
import { Interocitor } from '@interocitor/core';
|
|
50
|
+
import { GoogleDriveAdapter } from '@interocitor/core/adapters/google-drive';
|
|
51
|
+
|
|
52
|
+
const adapter = new GoogleDriveAdapter({ clientId: 'YOUR_GOOGLE_CLIENT_ID' });
|
|
53
|
+
|
|
54
|
+
const db = new Interocitor(adapter, {
|
|
55
|
+
dbName: 'my-app', // local DB name; keep stable across reloads
|
|
56
|
+
appName: 'My App', // shown in biometric prompts / OS keychain
|
|
57
|
+
remotePath: '/MyApp', // mesh-scoped folder on the remote
|
|
58
|
+
encrypted: true, // E2E encryption (default)
|
|
59
|
+
});
|
|
60
|
+
|
|
61
|
+
await db.init();
|
|
62
|
+
await db.connect();
|
|
63
|
+
|
|
64
|
+
// Write — local-first, no network required
|
|
65
|
+
await db.table('todos').add(
|
|
66
|
+
{ text: 'Ship privacy-first sync', done: false },
|
|
67
|
+
{ prefix: 'todo' },
|
|
68
|
+
);
|
|
69
|
+
|
|
70
|
+
// Share with another device — see "New device / restore" below
|
|
71
|
+
console.log('Passphrase:', db.getPassphrase());
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
> The constructor accepts either `(adapter, config)` or `(config)` alone.
|
|
75
|
+
> Use the `(config)` form for local‑only mode (no remote). When you
|
|
76
|
+
> attach a remote later, call `setRemoteStorage(adapter)` and then
|
|
77
|
+
> `connect()`.
|
|
78
|
+
|
|
79
|
+
## Mental model
|
|
80
|
+
|
|
81
|
+
```mermaid
|
|
82
|
+
flowchart LR
|
|
83
|
+
A[Browser app] --> B[Interocitor engine]
|
|
84
|
+
B --> C[Local store]
|
|
85
|
+
B --> D[Encrypt locally]
|
|
86
|
+
D --> E[Adapter]
|
|
87
|
+
E --> F[Remote mailbox<br/>ciphertext only]
|
|
88
|
+
F --> E
|
|
89
|
+
E --> G[Decrypt locally]
|
|
90
|
+
G --> B
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
The remote is a mailbox. For rows, the engine puts encrypted change
|
|
94
|
+
files into it and pulls down change files from peers. For files/images,
|
|
95
|
+
the engine puts encrypted durable objects under `files/` and reads or
|
|
96
|
+
deletes them by app path. Row merging happens on the device. There is no
|
|
97
|
+
server-side compute required for correctness.
|
|
98
|
+
|
|
99
|
+
Four artifact families live on the remote:
|
|
100
|
+
|
|
101
|
+
- **change files** — one per write batch, named `<HLC>-chg_<id>.json`;
|
|
102
|
+
- **snapshots** — periodic full row state, written by compaction;
|
|
103
|
+
- **durable files** — app blobs/images under `files/`, addressed by app path;
|
|
104
|
+
- **a manifest** — pointer to the current generation, plus mesh metadata.
|
|
105
|
+
|
|
106
|
+
See `docs/adapter-contract.md` for the full layout.
|
|
107
|
+
|
|
108
|
+
## Security model
|
|
109
|
+
|
|
110
|
+
Encryption is on by default (`encrypted: true`). The engine uses
|
|
111
|
+
**AES‑GCM 256** with a key derived from a base58 passphrase.
|
|
112
|
+
|
|
113
|
+
What it protects:
|
|
114
|
+
|
|
115
|
+
- Row contents (field names, field values, table names) inside change
|
|
116
|
+
files and snapshots.
|
|
117
|
+
- Integrity of each entry (AES‑GCM authentication tag).
|
|
118
|
+
|
|
119
|
+
What it does **not** protect:
|
|
120
|
+
|
|
121
|
+
- File names (they encode the HLC and a device id).
|
|
122
|
+
- Manifest contents (mesh id, schema version, epoch, encrypted flag,
|
|
123
|
+
serverId).
|
|
124
|
+
- Sizes, write timing, device count.
|
|
125
|
+
- The local store on the device (IndexedDB / SQLite stores plaintext).
|
|
126
|
+
|
|
127
|
+
What a malicious remote can still do:
|
|
128
|
+
|
|
129
|
+
- Drop, withhold, replay, or roll back files.
|
|
130
|
+
- Observe activity timing and device identities.
|
|
131
|
+
|
|
132
|
+
For the threat model, the metadata table, and full mitigations see
|
|
133
|
+
`docs/security-model.md`.
|
|
134
|
+
|
|
135
|
+
## Offline guarantees
|
|
136
|
+
|
|
137
|
+
| Operation | Network? |
|
|
138
|
+
| --- | --- |
|
|
139
|
+
| `new Interocitor()` | No |
|
|
140
|
+
| `init()` | No |
|
|
141
|
+
| `table.add()` / `patch()` / `replace()` / `delete()` | No |
|
|
142
|
+
| `table.row()` / `table.query()` / `table.where()` | No |
|
|
143
|
+
| `connect()` | Yes (if adapter configured) |
|
|
144
|
+
| `flush()` / `pull()` / `compact()` | Yes |
|
|
145
|
+
|
|
146
|
+
Writes are persisted locally and queued in an outbox. They survive
|
|
147
|
+
reload, crash, and offline periods. On reconnect the engine drains the
|
|
148
|
+
outbox to the remote.
|
|
149
|
+
|
|
150
|
+
## Sync guarantees
|
|
151
|
+
|
|
152
|
+
- **Eventual convergence.** Two devices that have seen the same set of
|
|
153
|
+
change files (in any order) reach byte‑identical local state.
|
|
154
|
+
- **Per‑column merge.** Conflicts are resolved field‑by‑field, not at
|
|
155
|
+
the row level. Default strategy is `'remote-wins'` — see
|
|
156
|
+
"Conflict resolution".
|
|
157
|
+
- **Idempotent merge.** Replaying an already‑applied change is a no‑op.
|
|
158
|
+
Safe to re‑pull, safe to re‑process the same change file twice.
|
|
159
|
+
- **Per‑device HLC monotonicity.** A single device's HLCs strictly
|
|
160
|
+
increase. Cross‑device order is total but only as wall‑clocks allow.
|
|
161
|
+
- **Restore via snapshot.** A device that joins late (or rehydrates after
|
|
162
|
+
a long offline) loads the latest snapshot, then applies any change
|
|
163
|
+
files newer than the snapshot watermark.
|
|
164
|
+
|
|
165
|
+
What we do **not** guarantee:
|
|
166
|
+
|
|
167
|
+
- Cross‑device atomicity. A `db.batch()` is one atomic remote file, but
|
|
168
|
+
two unrelated batches from different devices are independent.
|
|
169
|
+
- Real‑time delivery. The default poll interval is 30 s; some adapters
|
|
170
|
+
(Cloudflare) layer push notifications on top.
|
|
171
|
+
- Recovery if the remote silently lies (drops writes, rolls back the
|
|
172
|
+
manifest). See `docs/security-model.md`.
|
|
173
|
+
- Recovery if the passphrase is lost — see "New device / restore".
|
|
174
|
+
|
|
175
|
+
### Sync cadence
|
|
176
|
+
|
|
177
|
+
The engine adapts how often it polls the remote so it does useful work
|
|
178
|
+
when there is data to sync, and stays out of the way otherwise. This is
|
|
179
|
+
all automatic — there is no configuration knob.
|
|
180
|
+
|
|
181
|
+
- **Adaptive backoff.** Polling starts at `pollInterval` (30 s by
|
|
182
|
+
default). After every poll that merges zero entries the interval
|
|
183
|
+
doubles, capped at 60 s. Any poll that merges ≥ 1 entry resets the
|
|
184
|
+
interval back to base. The intent is to absorb idle bursts of clients
|
|
185
|
+
without hammering the remote, while still recovering immediately when
|
|
186
|
+
data starts flowing.
|
|
187
|
+
- **Tab visibility.** When the host page is hidden
|
|
188
|
+
(`document.visibilityState === 'hidden'`) the current poll interval is
|
|
189
|
+
multiplied by 10 — backgrounded tabs poll lazily. When the tab becomes
|
|
190
|
+
visible again the interval is reset to base **and** an immediate
|
|
191
|
+
`pull()` is fired so foregrounded data jumps in without waiting for
|
|
192
|
+
the next tick. This is a standard SWR‑style refresh‑on‑focus pattern.
|
|
193
|
+
The listener is wired during `connect()` and torn down by
|
|
194
|
+
`disconnect()`; environments without `document` (e.g. Node) skip it.
|
|
195
|
+
- **Push fallback.** Adapters that push invalidations (Cloudflare relay)
|
|
196
|
+
trigger an immediate pull and bypass the polling cadence entirely.
|
|
197
|
+
Polling is the safety net when the push channel is unavailable; the
|
|
198
|
+
WS connection itself uses bounded retries with a long cooldown after
|
|
199
|
+
repeated failed upgrades, so a misbehaving relay cannot DDoS itself.
|
|
200
|
+
|
|
201
|
+
## Setup lifecycle
|
|
202
|
+
|
|
203
|
+
```
|
|
204
|
+
new Interocitor(adapter?, config)
|
|
205
|
+
│
|
|
206
|
+
▼
|
|
207
|
+
configureMesh({...}) ← optional; pin meshId / passphrase upfront
|
|
208
|
+
│
|
|
209
|
+
▼
|
|
210
|
+
init() ← opens local store, restores credentials
|
|
211
|
+
│
|
|
212
|
+
▼
|
|
213
|
+
setRemoteStorage(adapter) ← only if no adapter passed to ctor
|
|
214
|
+
│
|
|
215
|
+
▼
|
|
216
|
+
connect() ← loads/creates manifest, pulls, flushes, polls
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
`connect()` will auto‑`init()` if needed. App code should still treat
|
|
220
|
+
init as explicit so credential restore happens before any writes.
|
|
221
|
+
|
|
222
|
+
The `encrypted` flag is **pinned at mesh bootstrap** and cannot change
|
|
223
|
+
between sessions for the same `dbName`. Reconnecting with a different
|
|
224
|
+
mode throws a typed `MeshEncryptionMismatchError`.
|
|
225
|
+
|
|
226
|
+
```ts
|
|
227
|
+
import { MeshEncryptionMismatchError } from '@interocitor/core';
|
|
228
|
+
|
|
229
|
+
try {
|
|
230
|
+
await db.connect();
|
|
231
|
+
} catch (err) {
|
|
232
|
+
if (err instanceof MeshEncryptionMismatchError) {
|
|
233
|
+
// err.expectedMode is the mode the remote was bootstrapped with
|
|
234
|
+
}
|
|
235
|
+
throw err;
|
|
236
|
+
}
|
|
237
|
+
```
|
|
238
|
+
|
|
239
|
+
## New device / restore
|
|
240
|
+
|
|
241
|
+
Joining a new device to an existing mesh requires three things:
|
|
242
|
+
|
|
243
|
+
1. The **`meshId`** of the existing mesh.
|
|
244
|
+
2. The **passphrase** that decrypts the mesh.
|
|
245
|
+
3. Access to the same **remote mailbox** (the same `remotePath` on a
|
|
246
|
+
storage backend the new device can reach).
|
|
247
|
+
|
|
248
|
+
A production app should keep CRUD, sync lifecycle, and pairing separate:
|
|
249
|
+
|
|
250
|
+
```text
|
|
251
|
+
lib/interocitor-db.ts engine, schema, local repository, credential primitives
|
|
252
|
+
lib/interocitor-sync.ts mesh id lifecycle, adapter, connect/disconnect, recovery
|
|
253
|
+
lib/interocitor-pairing.ts QR handshake only
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
The pairing flow ships credentials over an ECDH relay handshake. The handshake relay base and sync adapter base are different concepts: a relay base is the temporary handshake-file path, such as `/Taska`; the Cloudflare adapter base URL is the concrete Worker route, such as `/sync/io/{meshId}`. The engine `remotePath` is still the mesh folder path used inside that adapter, such as `/Taska`.
|
|
257
|
+
|
|
258
|
+
### Pairing intents
|
|
259
|
+
|
|
260
|
+
**Join QR** is for a device that does not have credentials yet:
|
|
261
|
+
|
|
262
|
+
1. Joiner mints a fresh mesh id.
|
|
263
|
+
2. Joiner creates a Cloudflare adapter with base URL `/sync/io/{meshId}`.
|
|
264
|
+
3. Joiner calls `generateJoinQR()`.
|
|
265
|
+
4. Existing paired device scans and pushes credentials.
|
|
266
|
+
5. Joiner receives credentials from `credentials` on the result.
|
|
267
|
+
6. Joiner applies the passphrase and connects to the minted mesh.
|
|
268
|
+
|
|
269
|
+
**Share QR** is for an existing mesh member inviting a new device:
|
|
270
|
+
|
|
271
|
+
1. Existing device connects to the active mesh.
|
|
272
|
+
2. Existing device calls `generateShareQR()` with `remotePath` and `passphrase`.
|
|
273
|
+
3. New device scans and receives credentials from `handleScannedQR()`.
|
|
274
|
+
4. New device applies credentials and connects using the adapter config from the payload.
|
|
275
|
+
|
|
276
|
+
`handleScannedQR()` returns `null` for join intent because the scanner pushed its own credentials. It returns credentials for share intent because the scanner received credentials. Always handle the return value:
|
|
277
|
+
|
|
278
|
+
```ts
|
|
279
|
+
import { decodeQRPayload, handleScannedQR, parseQRFromUrl } from '@interocitor/core';
|
|
280
|
+
// Raw QR payload decoder is also available from '@interocitor/core/handshake/qr'.
|
|
281
|
+
|
|
282
|
+
const payload = parseQRFromUrl(location.hash) ?? decodeQRPayload(rawPastedPayload);
|
|
283
|
+
const received = await handleScannedQR({ adapter, relayBase: '/Taska', payload });
|
|
284
|
+
|
|
285
|
+
if (received) {
|
|
286
|
+
if (received.passphrase) db.setPassphrase(received.passphrase);
|
|
287
|
+
await connectFromPayload(received.remotePath);
|
|
288
|
+
}
|
|
289
|
+
```
|
|
290
|
+
|
|
291
|
+
After the handshake the new device:
|
|
292
|
+
|
|
293
|
+
```ts
|
|
294
|
+
const db = new Interocitor(adapter, {
|
|
295
|
+
dbName: 'my-app',
|
|
296
|
+
appName: 'My App',
|
|
297
|
+
remotePath: '/Taska',
|
|
298
|
+
passphrase: 'base58-from-handshake',
|
|
299
|
+
encrypted: true,
|
|
300
|
+
});
|
|
301
|
+
|
|
302
|
+
await db.init();
|
|
303
|
+
await db.connect(); // pulls the manifest, rehydrates from snapshot, then catches up
|
|
304
|
+
```
|
|
305
|
+
|
|
306
|
+
On `connect()` the engine:
|
|
307
|
+
|
|
308
|
+
1. Reads `manifest.json` from the remote.
|
|
309
|
+
2. Compares stored `meshId` (from the credential store) against the live
|
|
310
|
+
one. Mismatch → `MeshCredentialMismatchError`.
|
|
311
|
+
3. If local epoch < remote epoch, calls `rehydrate()` to load the
|
|
312
|
+
latest snapshot.
|
|
313
|
+
4. Pulls any change files newer than the snapshot watermark.
|
|
314
|
+
5. Starts polling.
|
|
315
|
+
|
|
316
|
+
> **If the passphrase is lost and no other device holds it, the mesh is
|
|
317
|
+
> unreadable.** The engine has no recovery path — the data is end‑to‑end
|
|
318
|
+
> encrypted and the key is the passphrase. Treat the passphrase as the
|
|
319
|
+
> only thing that matters; back it up out of band (1Password, paper,
|
|
320
|
+
> another device's `WebAuthnCredentialStore`).
|
|
321
|
+
|
|
322
|
+
For the credential store details (records, anchors, biometric paths)
|
|
323
|
+
see `docs/credential-store.md`.
|
|
324
|
+
|
|
325
|
+
## Core API
|
|
326
|
+
|
|
327
|
+
```ts
|
|
328
|
+
const db = new Interocitor(adapter, {
|
|
329
|
+
dbName: 'my-app',
|
|
330
|
+
appName: 'My App',
|
|
331
|
+
remotePath: '/MyApp',
|
|
332
|
+
schema,
|
|
333
|
+
});
|
|
334
|
+
await db.init();
|
|
335
|
+
|
|
336
|
+
await db.table('tasks').add({ title: 'Ship it', done: false }, { prefix: 'task' });
|
|
337
|
+
await db.table('tasks').patch(taskId, { done: true });
|
|
338
|
+
await db.table('tasks').replace(taskId, fullTask);
|
|
339
|
+
await db.table('tasks').delete(taskId);
|
|
340
|
+
|
|
341
|
+
await db.table('tasks').row(taskId); // single row
|
|
342
|
+
await db.table('tasks').query(); // all rows
|
|
343
|
+
await db.table('tasks').where('done').equals(false).orderBy('title');
|
|
344
|
+
|
|
345
|
+
await db.connect(); // attach + sync
|
|
346
|
+
await db.flush(); // force outbox drain
|
|
347
|
+
await db.pull(); // force pull
|
|
348
|
+
await db.compact(); // see docs/compaction.md
|
|
349
|
+
await db.disconnect(); // tear down
|
|
350
|
+
|
|
351
|
+
// Durable files — encrypted, not compacted, not merged
|
|
352
|
+
await db.putFile('receipts/may.pdf', pdfBytes, 'application/pdf');
|
|
353
|
+
const pdf = await db.getFile('receipts/may.pdf');
|
|
354
|
+
const fileMeta = await db.getFileMetadata('receipts/may.pdf');
|
|
355
|
+
await db.deleteFile('receipts/may.pdf');
|
|
356
|
+
|
|
357
|
+
// Images — first-class file helpers
|
|
358
|
+
await db.putImage('avatars/me.png', fileOrBlob);
|
|
359
|
+
const image = await db.getImage('avatars/me.png'); // { data, blob, metadata }
|
|
360
|
+
const blobUrl = await db.getImageBlobUrl('avatars/me.png'); // { url, revoke }
|
|
361
|
+
blobUrl.revoke();
|
|
362
|
+
|
|
363
|
+
// Credentials
|
|
364
|
+
db.getPassphrase();
|
|
365
|
+
db.setPassphrase(passphrase);
|
|
366
|
+
await db.secureWithBiometrics();
|
|
367
|
+
await db.restoreWithBiometrics();
|
|
368
|
+
await db.clearCredentials();
|
|
369
|
+
|
|
370
|
+
// Batched writes — one ChangeEntry per batch
|
|
371
|
+
await db.batch(async () => {
|
|
372
|
+
await db.table('todos').add({ title: 'a' });
|
|
373
|
+
await db.table('todos').patch(otherId, { done: true });
|
|
374
|
+
});
|
|
375
|
+
```
|
|
376
|
+
|
|
377
|
+
### Files and images
|
|
378
|
+
|
|
379
|
+
Durable files are application objects stored under the mesh `files/` namespace. They are encrypted with the same mesh key when encryption is enabled, but they are not CRDT rows: no merge, no replay, no compaction, no snapshot membership. A file at a path simply exists until overwritten or deleted.
|
|
380
|
+
|
|
381
|
+
```ts
|
|
382
|
+
await db.putFile('attachments/report.pdf', bytes, 'application/pdf');
|
|
383
|
+
|
|
384
|
+
const bytes = await db.getFile('attachments/report.pdf');
|
|
385
|
+
const meta = await db.getFileMetadata('attachments/report.pdf');
|
|
386
|
+
|
|
387
|
+
await db.deleteFile('attachments/report.pdf');
|
|
388
|
+
```
|
|
389
|
+
|
|
390
|
+
`StoredFileMetadata` includes path, size, content type, uploader device, upload time, stored/plaintext byte sizes, and backend-tracked access counters when available.
|
|
391
|
+
|
|
392
|
+
Images are convenience wrappers over files:
|
|
393
|
+
|
|
394
|
+
```ts
|
|
395
|
+
await db.putImage('avatars/me.png', file); // File, Blob, ArrayBuffer, Uint8Array, data URL, or SVG string
|
|
396
|
+
|
|
397
|
+
const image = await db.getImage('avatars/me.png');
|
|
398
|
+
// image.data: Uint8Array
|
|
399
|
+
// image.blob: Blob
|
|
400
|
+
// image.contentType: image/png, image/jpeg, ...
|
|
401
|
+
|
|
402
|
+
const view = await db.getImageBlobUrl('avatars/me.png');
|
|
403
|
+
img.src = view.url;
|
|
404
|
+
view.revoke();
|
|
405
|
+
```
|
|
406
|
+
|
|
407
|
+
### Schema typing
|
|
408
|
+
|
|
409
|
+
```ts
|
|
410
|
+
import { types } from '@interocitor/core';
|
|
411
|
+
import type { DatabaseSchemaDefinition } from '@interocitor/core';
|
|
412
|
+
|
|
413
|
+
const schema = {
|
|
414
|
+
tables: {
|
|
415
|
+
todos: {
|
|
416
|
+
fields: {
|
|
417
|
+
text: types.string,
|
|
418
|
+
done: types.boolean,
|
|
419
|
+
createdAt: types.index(types.date),
|
|
420
|
+
note: types.string.optional,
|
|
421
|
+
items: types.typed<TodoItem[]>('json'),
|
|
422
|
+
},
|
|
423
|
+
},
|
|
424
|
+
},
|
|
425
|
+
} satisfies DatabaseSchemaDefinition;
|
|
426
|
+
|
|
427
|
+
// Inferred row type:
|
|
428
|
+
// { text: string; done: boolean; createdAt: Date; note?: string; items: TodoItem[] }
|
|
429
|
+
```
|
|
430
|
+
|
|
431
|
+
`.optional` makes the field optional in the inferred type. Indexed and
|
|
432
|
+
unique fields cannot be optional. `types.index(...)` marks a field for
|
|
433
|
+
efficient `where`/`orderBy`.
|
|
434
|
+
|
|
435
|
+
### Conflict resolution
|
|
436
|
+
|
|
437
|
+
Per‑column CRDT with HLC. Default merge strategy: **`'remote-wins'`**.
|
|
438
|
+
|
|
439
|
+
> "Remote‑wins" is per‑column, not per‑row. When two devices write the
|
|
440
|
+
> same column on the same row, the merge keeps the value with the higher
|
|
441
|
+
> HLC. Because HLCs are timestamp‑first, this is "later wall‑clock
|
|
442
|
+
> wins, ties broken by device id". Calling it "remote‑wins" is a
|
|
443
|
+
> historical accident of where the merge runs (during pull); both sides
|
|
444
|
+
> apply the same rule and reach the same answer. Override per database,
|
|
445
|
+
> table, or field if you need `'lww'`, `'local-wins'`, or a custom
|
|
446
|
+
> `MergeFunction`.
|
|
447
|
+
|
|
448
|
+
Available strategies:
|
|
449
|
+
|
|
450
|
+
- `'lww'` — last‑write‑wins by HLC. Functionally identical to
|
|
451
|
+
`'remote-wins'` for this CRDT but keeps semantics explicit.
|
|
452
|
+
- `'remote-wins'` (default).
|
|
453
|
+
- `'local-wins'` — keep local on tie; remote still wins on a strictly
|
|
454
|
+
greater HLC.
|
|
455
|
+
- `MergeFunction` — `(local, remote, ctx) => result` for custom logic.
|
|
456
|
+
|
|
457
|
+
### Deletion semantics
|
|
458
|
+
|
|
459
|
+
`delete()` writes a **tombstone**, not an immediate unlink. Tombstones:
|
|
460
|
+
|
|
461
|
+
- carry `deletedHlc` like any other write;
|
|
462
|
+
- hide the row from public reads and queries;
|
|
463
|
+
- keep an empty payload, so deleted user data does not linger inside the
|
|
464
|
+
tombstone;
|
|
465
|
+
- are needed so that slower devices cannot replay an older `add()` and
|
|
466
|
+
resurrect the row.
|
|
467
|
+
|
|
468
|
+
Re-inserting the same row id after a delete starts a fresh row
|
|
469
|
+
incarnation. Columns from the old incarnation are not carried forward;
|
|
470
|
+
a partial insert only contains the new columns.
|
|
471
|
+
|
|
472
|
+
Compaction can later hard-delete tombstones from snapshots. The engine
|
|
473
|
+
tracks per-device acknowledgements in `devices/<deviceId>.json`; once
|
|
474
|
+
all active devices have observed a manifest watermark, compaction
|
|
475
|
+
publishes `manifest.gcFloorHlc` as a point of no return. Tombstones with
|
|
476
|
+
`deletedHlc <= gcFloorHlc` are omitted from the next snapshot.
|
|
477
|
+
|
|
478
|
+
Devices not seen within `offlineGraceMs` (default seven days) are
|
|
479
|
+
excluded from GC consensus. If one wakes up with local outbox entries at
|
|
480
|
+
or before `gcFloorHlc`, the engine refuses to flush those entries and
|
|
481
|
+
rehydrates from the canonical snapshot instead.
|
|
482
|
+
|
|
483
|
+
### Local store
|
|
484
|
+
|
|
485
|
+
The local store is a pluggable `LocalStoreAdapter`. Browser default is
|
|
486
|
+
IndexedDB. The Swift package ships SQLite. Tests use an in‑memory
|
|
487
|
+
implementation. Reads, writes, queries, and the outbox all go through
|
|
488
|
+
this interface.
|
|
489
|
+
|
|
490
|
+
## Adapters
|
|
491
|
+
|
|
492
|
+
| Adapter | Use when | Notes |
|
|
493
|
+
| --- | --- | --- |
|
|
494
|
+
| `GoogleDriveAdapter` | You want zero infrastructure | OAuth in the browser; the user owns the data |
|
|
495
|
+
| `WebDAVAdapter` | Self‑hosted (Nextcloud, OwnCloud, custom WebDAV) | Easy to inspect remotely |
|
|
496
|
+
| `CloudflareAdapter` | You operate a worker; want push invalidations | Experimental |
|
|
497
|
+
| `MemoryAdapter` | Tests and demos | No persistence |
|
|
498
|
+
|
|
499
|
+
Implementing your own adapter: see `docs/adapter-contract.md` for
|
|
500
|
+
required semantics, consistency assumptions, and the contract test
|
|
501
|
+
suite.
|
|
502
|
+
|
|
503
|
+
## Remote mailbox layout
|
|
504
|
+
|
|
505
|
+
For `remotePath: '/MyApp'`:
|
|
506
|
+
|
|
507
|
+
```
|
|
508
|
+
/MyApp/
|
|
509
|
+
├── manifest.json # pointer { currentGeneration, file }
|
|
510
|
+
├── manifest-1.json # generation 1
|
|
511
|
+
├── manifest-2.json # ...
|
|
512
|
+
├── changes/
|
|
513
|
+
│ ├── head.json # { latestHlc } — pull fast path
|
|
514
|
+
│ └── <HLC>-chg_<id>.json # encrypted change entries
|
|
515
|
+
├── devices/
|
|
516
|
+
│ └── <deviceId>.json # device metadata
|
|
517
|
+
├── files/
|
|
518
|
+
│ └── <app path> # durable encrypted app files
|
|
519
|
+
└── mainline/
|
|
520
|
+
└── snapshot-<epoch>-<serverId>.json # encrypted snapshot
|
|
521
|
+
```
|
|
522
|
+
|
|
523
|
+
`remotePath` is **mesh‑scoped**. One folder = one mesh = one logical
|
|
524
|
+
database. Two meshes sharing the same folder will fight over the
|
|
525
|
+
manifest and poison the remote.
|
|
526
|
+
|
|
527
|
+
## Maintenance / compaction
|
|
528
|
+
|
|
529
|
+
Compaction collapses the change log into a snapshot, bumps the manifest,
|
|
530
|
+
prunes old change files, and advances the tombstone GC floor when active
|
|
531
|
+
devices have acknowledged the prior watermark. Sync works without it;
|
|
532
|
+
the remote folder just grows until *some* device compacts.
|
|
533
|
+
|
|
534
|
+
Two paths run automatically:
|
|
535
|
+
|
|
536
|
+
- **Immediate sampled** — after a flush of ≥ `compactAutoThreshold`
|
|
537
|
+
(default 50) ops, with probability ≈
|
|
538
|
+
`compactAutoSampleNumerator / compactAutoDeviceCount`.
|
|
539
|
+
- **Delayed two‑phase** — per‑write timer (10 ± 5 min) → check that
|
|
540
|
+
remote change files exceed `compactRemoteChangeThreshold` (default 2)
|
|
541
|
+
→ second timer (15 ± 5 min) → run.
|
|
542
|
+
|
|
543
|
+
Both paths are deduped by a single in‑flight guard. You can also call
|
|
544
|
+
`db.compact()` manually.
|
|
545
|
+
|
|
546
|
+
> **The auto defaults and the recommended manual policy are different
|
|
547
|
+
> things.** The manual policy ("idle > 1 min, churn > 20") is what to
|
|
548
|
+
> gate a "Sync now" button on. The auto defaults are what runs without
|
|
549
|
+
> any button. See `docs/compaction.md`.
|
|
550
|
+
|
|
551
|
+
> **Compaction is not race‑safe across devices.** The adapter contract
|
|
552
|
+
> has no CAS/ETag write, so two simultaneous compactors can both
|
|
553
|
+
> overwrite the manifest pointer. Mitigations: rely on probabilistic
|
|
554
|
+
> avoidance for small meshes, or run with `serverManaged: true` and a
|
|
555
|
+
> single authorized writer.
|
|
556
|
+
|
|
557
|
+
Full protocol, events, lock story, device acknowledgement / GC-floor
|
|
558
|
+
rules, prune invariants, and tuning checklist: **`docs/compaction.md`**.
|
|
559
|
+
|
|
560
|
+
## Connected stores
|
|
561
|
+
|
|
562
|
+
`engine.connectedStores` is a small **credential vault** for sub-stores
|
|
563
|
+
that conceptually belong to this Interocitor (for example a "reviews"
|
|
564
|
+
store derived from a "family planner" store). It does **not** construct
|
|
565
|
+
or run child engines — it only stores credentials so apps don't have to
|
|
566
|
+
reinvent that storage themselves.
|
|
567
|
+
|
|
568
|
+
Direction is one-way by construction: the parent stores credentials for
|
|
569
|
+
sub-stores, and sub-stores have no awareness of the parent. Anyone with
|
|
570
|
+
read access to the parent inherits read access to the sub-store
|
|
571
|
+
credentials persisted here.
|
|
572
|
+
|
|
573
|
+
```ts
|
|
574
|
+
import type { ConnectedStoreCredentials } from '@interocitor/core';
|
|
575
|
+
|
|
576
|
+
await db.connectedStores.put({
|
|
577
|
+
id: 'reviews',
|
|
578
|
+
alias: 'family-reviews',
|
|
579
|
+
remotePath: '/family/reviews',
|
|
580
|
+
passphrase: 'review-pass',
|
|
581
|
+
encrypted: true,
|
|
582
|
+
dbName: 'reviews-db',
|
|
583
|
+
adapter: { kind: 'memory' },
|
|
584
|
+
metadata: { icon: 'star' },
|
|
585
|
+
});
|
|
586
|
+
|
|
587
|
+
const all: ConnectedStoreCredentials[] = await db.connectedStores.list();
|
|
588
|
+
const reviews = await db.connectedStores.get('reviews');
|
|
589
|
+
await db.connectedStores.remove('reviews');
|
|
590
|
+
```
|
|
591
|
+
|
|
592
|
+
Notes:
|
|
593
|
+
|
|
594
|
+
- Credentials are persisted as a single JSON list under a dedicated meta
|
|
595
|
+
key inside the parent's local store. Sub-stores never appear as parent
|
|
596
|
+
tables or rows.
|
|
597
|
+
- `put(creds)` upserts by `id`, preserves `createdAt`, and refreshes
|
|
598
|
+
`updatedAt` automatically.
|
|
599
|
+
- The engine never reads `passphrase`/`adapter`/`remotePath` from these
|
|
600
|
+
records — apps construct their own `Interocitor` instances from them.
|
|
601
|
+
|
|
602
|
+
## Schema migration
|
|
603
|
+
|
|
604
|
+
Interocitor now manages **local cache/index upgrades automatically**.
|
|
605
|
+
Adding or removing `types.index(...)` fields no longer requires bumping a
|
|
606
|
+
public schema version just to keep IndexedDB in sync. The local store
|
|
607
|
+
computes its own cache fingerprint, repairs missing indexes on open, and
|
|
608
|
+
falls back to scans if a stale cache slips through.
|
|
609
|
+
|
|
610
|
+
`schema.version` is now **optional** and only matters if you want an
|
|
611
|
+
explicit logical compatibility gate in the remote manifest. If you set
|
|
612
|
+
it, the engine writes it to `manifest.schema` and will reject manifests
|
|
613
|
+
written under a different logical version.
|
|
614
|
+
|
|
615
|
+
Use `schema.version` only for app-level data meaning changes such as:
|
|
616
|
+
|
|
617
|
+
1. row shapes that old clients cannot safely read,
|
|
618
|
+
2. `onInit` migrations that rewrite logical data,
|
|
619
|
+
3. staged rollouts where you want explicit manifest compatibility checks.
|
|
620
|
+
|
|
621
|
+
Recommended pattern for logical migrations:
|
|
622
|
+
|
|
623
|
+
1. Bump `schema.version` in your code.
|
|
624
|
+
2. Implement a one-shot `onInit` migration that reads old rows from the
|
|
625
|
+
local store and writes the new shape back. Use `db.batch(...)` to
|
|
626
|
+
keep it atomic per row group.
|
|
627
|
+
3. Trigger compaction after the migration so the snapshot is written
|
|
628
|
+
under the new logical version and old change files are pruned.
|
|
629
|
+
4. During rollout, make sure old clients either tolerate both shapes or
|
|
630
|
+
are blocked by the manifest version mismatch on connect.
|
|
631
|
+
|
|
632
|
+
For breaking changes that cannot be rolled out gradually, the
|
|
633
|
+
heavier path is to bootstrap a fresh mesh, replicate data over, and
|
|
634
|
+
retire the old mesh. The engine does not automate this.
|
|
635
|
+
|
|
636
|
+
## Events
|
|
637
|
+
|
|
638
|
+
```ts
|
|
639
|
+
db.on(event => {
|
|
640
|
+
switch (event.type) {
|
|
641
|
+
case 'sync:start': /* pull began */ break;
|
|
642
|
+
case 'sync:complete': /* event.entriesMerged */ break;
|
|
643
|
+
case 'change': /* event.table, event.rowId, event.row */ break;
|
|
644
|
+
case 'delete': /* event.table, event.rowId */ break;
|
|
645
|
+
case 'flush:start': /* event.entryCount */ break;
|
|
646
|
+
case 'flush:complete': break;
|
|
647
|
+
case 'flush:error': /* event.error */ break;
|
|
648
|
+
case 'remote:poisoned': /* unrecoverable; see security-model.md */ break;
|
|
649
|
+
case 'credentials:meshMismatch': /* stored meshId != live; offer clearCredentials() */ break;
|
|
650
|
+
case 'compact:warning': /* outbox is large */ break;
|
|
651
|
+
// compact:auto:start / complete / skip / error / delayed:* — see docs/compaction.md
|
|
652
|
+
}
|
|
653
|
+
});
|
|
654
|
+
```
|
|
655
|
+
|
|
656
|
+
## What this is not
|
|
657
|
+
|
|
658
|
+
- not Firebase
|
|
659
|
+
- not Fireproof
|
|
660
|
+
- not PowerSync
|
|
661
|
+
- not Replicache
|
|
662
|
+
- not a hosted backend
|
|
663
|
+
- not a query engine over the cloud
|
|
664
|
+
- not a server‑trusted merge layer
|
|
665
|
+
|
|
666
|
+
## What can go wrong
|
|
667
|
+
|
|
668
|
+
A short field guide. Detailed mitigations in the linked docs.
|
|
669
|
+
|
|
670
|
+
| Symptom | Likely cause | Where to look |
|
|
671
|
+
| --- | --- | --- |
|
|
672
|
+
| `MeshCredentialMismatchError` on connect | Same `dbName`, new mesh; stale credential record | `engine.clearCredentials()` then reconnect; `docs/credential-store.md` |
|
|
673
|
+
| `MeshEncryptionMismatchError` on connect | App flipped `encrypted` between sessions | Pin `encrypted` per `dbName`, never change |
|
|
674
|
+
| `remote:poisoned` event | Decode failure on a manifest, change file, or snapshot | `docs/security-model.md` — usually wrong key, schema drift, or remote tampering |
|
|
675
|
+
| Writes never appear on peer | Peer never compacted, or peer's poll interval is long, or remote dropped writes | Check `flush:complete` events; check remote folder by hand |
|
|
676
|
+
| Local store has rows that are "old" after re‑pair | Engine kept local data when you re‑paired with a fresh mesh | Either delete local DB on re‑pair, or accept the merge |
|
|
677
|
+
| Lost passphrase | No recovery | Passphrase is the key. Back it up out of band |
|
|
678
|
+
| Long‑offline device "lost" recent edits | Rehydrate replaced local state with the snapshot | Local writes already in the outbox survive; in‑flight uncommitted UI state does not |
|
|
679
|
+
| Compaction never runs | `autoCompact: false`, or no remote, or `compactAutoThreshold` never reached | `docs/compaction.md` — subscribe to `compact:auto:skip` |
|
|
680
|
+
| Two compactors race | No CAS in the adapter; small probability in small meshes | Use `serverManaged: true` for large meshes |
|
|
681
|
+
|
|
682
|
+
## Tests
|
|
683
|
+
|
|
684
|
+
```bash
|
|
685
|
+
yarn workspace @interocitor/core test
|
|
686
|
+
```
|
|
687
|
+
|
|
688
|
+
Adapter contract tests (run for every adapter):
|
|
689
|
+
|
|
690
|
+
```bash
|
|
691
|
+
yarn workspace @interocitor/core test webdav.adapter.contract
|
|
692
|
+
```
|
|
693
|
+
|
|
694
|
+
## Package context
|
|
695
|
+
|
|
696
|
+
Part of the Interocitor monorepo. See:
|
|
697
|
+
|
|
698
|
+
- root `README.md` — monorepo overview
|
|
699
|
+
- `docs/security-model.md` — threat model
|
|
700
|
+
- `docs/adapter-contract.md` — adapter requirements
|
|
701
|
+
- `docs/compaction.md` — compaction protocol & tuning
|
|
702
|
+
- `docs/credential-store.md` — credential persistence
|
|
703
|
+
|
|
704
|
+
## License
|
|
705
|
+
|
|
706
|
+
MIT
|