@minca/core 0.1.16 → 0.1.18
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +10 -9
- package/dist/elements/auth-styles.d.ts +1 -1
- package/dist/elements/auth-styles.d.ts.map +1 -1
- package/dist/elements/auth-styles.js +14 -5
- package/dist/elements/auth-styles.js.map +1 -1
- package/dist/elements/minca-auth.d.ts.map +1 -1
- package/dist/elements/minca-auth.js +18 -15
- package/dist/elements/minca-auth.js.map +1 -1
- package/dist/files/space-files.d.ts +2 -2
- package/dist/files/space-files.d.ts.map +1 -1
- package/dist/files/space-files.js +7 -5
- package/dist/files/space-files.js.map +1 -1
- package/dist/inbox/names.d.ts +1 -1
- package/dist/inbox/names.js +1 -1
- package/dist/index.d.ts +2 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +2 -0
- package/dist/index.js.map +1 -1
- package/dist/network/ws-transport.d.ts.map +1 -1
- package/dist/network/ws-transport.js +3 -0
- package/dist/network/ws-transport.js.map +1 -1
- package/dist/node/actions.d.ts +5 -0
- package/dist/node/actions.d.ts.map +1 -1
- package/dist/node/actions.js +39 -20
- package/dist/node/actions.js.map +1 -1
- package/dist/node/api.d.ts +2 -0
- package/dist/node/api.d.ts.map +1 -1
- package/dist/node/api.js +2 -0
- package/dist/node/api.js.map +1 -1
- package/dist/node/carrier.js +1 -1
- package/dist/node/carrier.js.map +1 -1
- package/dist/node/node-carriers.d.ts.map +1 -1
- package/dist/node/node-carriers.js +6 -1
- package/dist/node/node-carriers.js.map +1 -1
- package/dist/node/node-contacts.d.ts.map +1 -1
- package/dist/node/node-contacts.js +1 -0
- package/dist/node/node-contacts.js.map +1 -1
- package/dist/node/node-hosting.d.ts.map +1 -1
- package/dist/node/node-hosting.js +11 -4
- package/dist/node/node-hosting.js.map +1 -1
- package/dist/node/node-records.d.ts.map +1 -1
- package/dist/node/node-records.js +6 -1
- package/dist/node/node-records.js.map +1 -1
- package/dist/node/node.d.ts.map +1 -1
- package/dist/node/node.js +14 -1
- package/dist/node/node.js.map +1 -1
- package/dist/node/remote.d.ts.map +1 -1
- package/dist/node/remote.js +7 -1
- package/dist/node/remote.js.map +1 -1
- package/dist/node/space-holding.d.ts +7 -3
- package/dist/node/space-holding.d.ts.map +1 -1
- package/dist/node/space-holding.js +90 -18
- package/dist/node/space-holding.js.map +1 -1
- package/dist/node/space-peers.d.ts.map +1 -1
- package/dist/node/space-peers.js +58 -24
- package/dist/node/space-peers.js.map +1 -1
- package/dist/node/space-runtime.d.ts +11 -3
- package/dist/node/space-runtime.d.ts.map +1 -1
- package/dist/node/space-runtime.js +22 -1
- package/dist/node/space-runtime.js.map +1 -1
- package/dist/node/space-writing.d.ts +2 -0
- package/dist/node/space-writing.d.ts.map +1 -1
- package/dist/node/space-writing.js +2 -0
- package/dist/node/space-writing.js.map +1 -1
- package/dist/node/types.d.ts +130 -36
- package/dist/node/types.d.ts.map +1 -1
- package/dist/query/types.d.ts +1 -0
- package/dist/query/types.d.ts.map +1 -1
- package/dist/query/types.js.map +1 -1
- package/dist/records/contract-builder.d.ts +22 -6
- package/dist/records/contract-builder.d.ts.map +1 -1
- package/dist/records/contract-builder.js.map +1 -1
- package/dist/schemas/automations.d.ts +6 -2
- package/dist/schemas/automations.d.ts.map +1 -1
- package/dist/schemas/automations.js.map +1 -1
- package/dist/schemas/connectors.d.ts +4 -3
- package/dist/schemas/connectors.d.ts.map +1 -1
- package/dist/schemas/connectors.js +4 -3
- package/dist/schemas/connectors.js.map +1 -1
- package/dist/schemas/library/media.js +1 -1
- package/dist/schemas/library/media.js.map +1 -1
- package/dist/schemas/library/social.d.ts +258 -0
- package/dist/schemas/library/social.d.ts.map +1 -1
- package/dist/schemas/library/social.js +28 -1
- package/dist/schemas/library/social.js.map +1 -1
- package/dist/session/auth.d.ts +7 -3
- package/dist/session/auth.d.ts.map +1 -1
- package/dist/session/auth.js.map +1 -1
- package/dist/session/hosting.d.ts.map +1 -1
- package/dist/session/hosting.js +8 -2
- package/dist/session/hosting.js.map +1 -1
- package/dist/space/presets.d.ts +2 -2
- package/dist/space/presets.d.ts.map +1 -1
- package/dist/space/presets.js +3 -2
- package/dist/space/presets.js.map +1 -1
- package/dist/storage/blob-store.d.ts +3 -0
- package/dist/storage/blob-store.d.ts.map +1 -1
- package/dist/types.d.ts +11 -2
- package/dist/types.d.ts.map +1 -1
- package/dist/types.js.map +1 -1
- package/docs/README.md +86 -93
- package/docs/actions.md +212 -144
- package/docs/agents.md +198 -155
- package/docs/automations.md +296 -204
- package/docs/blueprints-and-guides.md +405 -295
- package/docs/bots.md +208 -134
- package/docs/building-an-app.md +74 -163
- package/docs/collections.md +248 -328
- package/docs/computed.md +112 -89
- package/docs/contacts.md +151 -112
- package/docs/contracts.md +299 -120
- package/docs/direct-messages.md +80 -61
- package/docs/inbox.md +138 -93
- package/docs/node.md +1072 -972
- package/docs/permissions.md +142 -0
- package/docs/proposing-collections.md +74 -55
- package/docs/query-format.md +150 -126
- package/docs/records-and-queries.md +205 -126
- package/docs/sign-in.md +496 -418
- package/docs/spaces.md +261 -245
- package/docs/standard-library.md +473 -504
- package/docs/storage.md +325 -348
- package/docs/usernames.md +164 -97
- package/docs/warrants.md +100 -67
- package/package.json +1 -2
package/docs/storage.md
CHANGED
|
@@ -1,389 +1,366 @@
|
|
|
1
1
|
# Sync and storage in the library
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
Where a node keeps its data, and how it syncs it. You rarely need this page
|
|
4
|
+
to build an app. Read it to attach files to records, to know what a node keeps
|
|
5
|
+
on a device, or to plug in storage of your own.
|
|
5
6
|
|
|
6
|
-
|
|
7
|
-
> implementation may do it differently and still interoperate. What peers must
|
|
8
|
-
> agree on (the sync messages, the store's entries, the data folder, sealing,
|
|
9
|
-
> segments) is in [spec 05](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md).
|
|
7
|
+
## Files
|
|
10
8
|
|
|
11
|
-
|
|
9
|
+
Records hold small things. A photo or a document goes in a file, and a record
|
|
10
|
+
points to it:
|
|
11
|
+
|
|
12
|
+
```ts
|
|
13
|
+
declare const file: File;
|
|
14
|
+
// ---cut---
|
|
15
|
+
const bytes = new Uint8Array(await file.arrayBuffer());
|
|
16
|
+
const photo = await node.records.addFile(space.id, bytes, { mime: file.type, name: file.name });
|
|
17
|
+
await node.records.put(space.id, 'app.post', { text: 'Look', photo });
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
`addFile` encrypts the bytes under a fresh key and gives back a
|
|
21
|
+
[`FileRef`](https://minca.dev/api/core/interfaces/FileRef/). Put it anywhere
|
|
22
|
+
in a body. When the body is written, every file in it is named on the record,
|
|
23
|
+
at most 16.
|
|
24
|
+
|
|
25
|
+
To show it, read the bytes back:
|
|
26
|
+
|
|
27
|
+
```ts
|
|
28
|
+
import type { FileRef, NodeRecord } from '@minca/core';
|
|
29
|
+
declare const post: NodeRecord<{ text: string; photo: FileRef }>;
|
|
30
|
+
declare const img: HTMLImageElement;
|
|
31
|
+
// ---cut---
|
|
32
|
+
if (post.body) {
|
|
33
|
+
const { photo } = post.body;
|
|
34
|
+
const bytes = await node.records.readFile(space.id, photo);
|
|
35
|
+
img.src = URL.createObjectURL(new Blob([bytes.slice()], { type: photo.mime }));
|
|
36
|
+
}
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
`readFile` gives the bytes if they're here. Otherwise it fetches them from
|
|
40
|
+
connected peers, one after another, then from mirrors, checking every chunk.
|
|
41
|
+
When nobody has them, it waits. It asks again whenever records change or a
|
|
42
|
+
peer connects, and every 15 s, and resolves once someone does.
|
|
43
|
+
|
|
44
|
+
It rejects for a file no record here names, for one that won't open as named,
|
|
45
|
+
and when the space closes.
|
|
46
|
+
|
|
47
|
+
### What a device keeps
|
|
48
|
+
|
|
49
|
+
Files follow records. About 2 s after records change, a node fetches the
|
|
50
|
+
files they name, while the writer is likely still connected. A file nobody
|
|
51
|
+
had is not asked of everyone again for 10 minutes, but is asked at once of
|
|
52
|
+
each peer that connects.
|
|
53
|
+
|
|
54
|
+
A device limits the files it didn't add, per space:
|
|
55
|
+
|
|
56
|
+
- It fetches unread only files up to 5 MiB. So photos and documents come
|
|
57
|
+
ahead of time, and video when played.
|
|
58
|
+
- Past 256 MiB of files in a space, it drops those fetched or opened longest
|
|
59
|
+
ago, down to 90%. A file never fetched whole goes first.
|
|
60
|
+
- It never drops a file added here, or one being fetched or read. A file
|
|
61
|
+
larger than the limit is dropped after it is read, and fetched again at the
|
|
62
|
+
next read.
|
|
63
|
+
- A dropped file that was whole is not fetched again unread while a record
|
|
64
|
+
names it. Reading it fetches it.
|
|
65
|
+
|
|
66
|
+
Because the limit is per space, a device in many spaces holds more. A node
|
|
67
|
+
with `fetchFiles: 'all'`, such as a carrier or host, fetches and keeps every
|
|
68
|
+
file.
|
|
69
|
+
|
|
70
|
+
A person can choose other limits for their device, such as fewer downloads on
|
|
71
|
+
a phone. Give them to the node each time it starts, since it keeps none:
|
|
72
|
+
|
|
73
|
+
```ts
|
|
74
|
+
// Only thumbnails and other small files come ahead; up to 1 GiB kept per space.
|
|
75
|
+
await node.setFileLimits({ ahead: 256 * 1024, keep: 1024 * 1024 * 1024 });
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
`Infinity` fetches or keeps everything. New limits take effect in every
|
|
79
|
+
open space within a few seconds: past `keep`, files go at once, and raising
|
|
80
|
+
`ahead` fetches what now fits. A file dropped earlier still waits until it is
|
|
81
|
+
read. Pictures and videos carry a small `thumbnail` (see the
|
|
82
|
+
[standard library](standard-library.md)) so that a device fetching only
|
|
83
|
+
small files still shows them.
|
|
84
|
+
|
|
85
|
+
About 2 s after records change, files no record names are dropped. A file
|
|
86
|
+
added here in the last 24 hours, and not yet named, is kept. Files of
|
|
87
|
+
records that don't stand are not kept.
|
|
88
|
+
|
|
89
|
+
### How files are served and stored
|
|
90
|
+
|
|
91
|
+
- A peer answers a request for a file with at most 4 chunks at a time, so a
|
|
92
|
+
large file never floods a connection.
|
|
93
|
+
- Chunks are entries in the space's own store: `f/<id>/list`, `f/<id>/<n>`,
|
|
94
|
+
`f/<id>/own` (when it was added here, 0 once a record names it) and
|
|
95
|
+
`f/<id>/used` (when it was last fetched or read). A data folder keeps these
|
|
96
|
+
in this device's own stores, not the folder.
|
|
97
|
+
- Each file a node holds whole goes to its mirrors, with its chunk list,
|
|
98
|
+
under `files/<space>/<store writer>/`. Files no longer named are deleted
|
|
99
|
+
there.
|
|
100
|
+
|
|
101
|
+
What peers must agree on, from chunks to encryption, is in
|
|
102
|
+
[spec 05 §16.6](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md).
|
|
12
103
|
|
|
13
|
-
|
|
14
|
-
node speaks, and how much it keeps in flight, is its own:
|
|
15
|
-
|
|
16
|
-
| What | Value | Why |
|
|
17
|
-
| ------------------------------ | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
18
|
-
| Heartbeat | a `hello` to every peer every 30 s | Finds what a lost push or a dropped `want` missed. |
|
|
19
|
-
| Hello after taking in | ≈100 ms | After something new arrives from a peer, a folder or a mirror, so a version passed along does not wait a heartbeat at every hop. |
|
|
20
|
-
| `want`s in flight | 4 per peer (`MAX_WANTS_IN_FLIGHT`) | Asking goes through a queue per peer; when a `want` is answered, the next ones go. |
|
|
21
|
-
| Rounds per session | 64 | Only the initiator counts rounds. Past 64 the session is abandoned as runaway. |
|
|
22
|
-
| Session and `want` stale after | 30 s | A session that has heard nothing, or a `want` unanswered, for 30 s is dropped at the next heartbeat, and the next queued ones go. What a dropped `want` asked for is found again on the next round. |
|
|
23
|
-
| Versions waiting | 1,000 | Versions the gatekeeper answers _later_ wait in memory; past 1,000 the oldest give way. |
|
|
24
|
-
| Refusals remembered | 10,000 | Refusals are kept per peer and id, so a refused version is not asked of that peer again; past 10,000 the oldest are forgotten. |
|
|
25
|
-
|
|
26
|
-
The queue of ids to ask for puts the space's own records (`sys.*`) and the
|
|
27
|
-
first or previous versions a waiting version names at the front; everything
|
|
28
|
-
else joins the back. An id already queued or asked for is not queued again.
|
|
29
|
-
|
|
30
|
-
If a session for a collection with a peer is already running and has heard
|
|
31
|
-
something in the last 30 s, a new difference does not start another: the
|
|
32
|
-
running one is marked to run again once it ends.
|
|
33
|
-
|
|
34
|
-
The node reports `synced` and `level` with each peer as events inside the
|
|
35
|
-
space runtime; holding part of a space ([node](node.md#holding-part-of-a-space))
|
|
36
|
-
uses them, together with whether the peer holds `"all"`. It tells the page that
|
|
37
|
-
records changed once per message taken in, not once per version.
|
|
38
|
-
|
|
39
|
-
_Source: `packages/core/src/sync/sync-engine.ts` (`MAX_WANTS_IN_FLIGHT`,
|
|
40
|
-
`MAX_ROUNDS`, `STALE_MS`, `MAX_WAITING`, `MAX_REFUSED`, `want`, `pump`,
|
|
41
|
-
`sweep`), `packages/core/src/node/space-peers.ts` (heartbeat,
|
|
42
|
-
`announceSoon`). Tests: `packages/core/tests/reconcile.test.ts` ("a want whose
|
|
43
|
-
answer is lost is given up, and the peer is synced again"),
|
|
44
|
-
`packages/core/tests/sync.test.ts`._
|
|
104
|
+
### Planned: video and pictures
|
|
45
105
|
|
|
46
|
-
|
|
106
|
+
> **Planned:** not normative.
|
|
107
|
+
>
|
|
108
|
+
> - **Video.** Play before the file is whole, through a service worker that
|
|
109
|
+
> answers the browser's range requests from the chunks read so far. MP4 with
|
|
110
|
+
> its index first, or WebM. A poster is a second, small image.
|
|
111
|
+
> - **Several peers at once.** Ask chunks of several peers in parallel.
|
|
112
|
+
> - **Mirrors by range.** Read one chunk from a mirror without the whole file.
|
|
113
|
+
> - **Pictures.** `std.image` gains `width`, `height` and a tiny placeholder
|
|
114
|
+
> shown before the picture arrives. `std.profile`'s `avatar` is resized in
|
|
115
|
+
> the browser before it is added.
|
|
47
116
|
|
|
48
|
-
|
|
49
|
-
running sum in memory, updated by every change made through it. So are the
|
|
50
|
-
`r/` and `g/` entries it read or wrote, and up to 50,000 version bodies (the
|
|
51
|
-
oldest let go first): a body never changes under its id. An entry read while
|
|
52
|
-
a change was landing is not kept.
|
|
53
|
-
|
|
54
|
-
When another writer changed the same store, the node re-reads all of it
|
|
55
|
-
(`invalidate`). Other tabs of one browser are told through a
|
|
56
|
-
`BroadcastChannel` named `minca-node:<root DID>:<space id>`. Where a browser
|
|
57
|
-
has shared workers that work, the apps in this repository run one node for all
|
|
58
|
-
of a site's tabs in a `SharedWorker` (one per account and key), so those tabs share one store and have
|
|
59
|
-
nothing to tell each other; the channel remains for the rest (Chrome on
|
|
60
|
-
Android has none; WebKit drops a message to one that holds a `CryptoKey`),
|
|
61
|
-
where each tab's node runs in a worker of its own
|
|
62
|
-
(`packages/core/src/node/worker.ts`). Writers on a shared folder are found by
|
|
63
|
-
re-reading it ([data folders](#data-folders)).
|
|
64
|
-
|
|
65
|
-
The store's overall fingerprint, shown in `spaces.status` and used by tests,
|
|
66
|
-
is the Negentropy fingerprint of the sum of every collection's sum, as hex.
|
|
67
|
-
|
|
68
|
-
_Source: `packages/core/src/storage/storage-provider.ts` (`MAX_BODIES`,
|
|
69
|
-
`invalidate`), `packages/core/src/node/worker.ts`. Tests:
|
|
70
|
-
`packages/core/tests/reconcile.test.ts` ("a store written by someone else is
|
|
71
|
-
read again once told")._
|
|
117
|
+
## IndexedDB
|
|
72
118
|
|
|
73
|
-
|
|
119
|
+
In the browser, a node keeps each store in its own IndexedDB database,
|
|
120
|
+
through `indexedDBStores(prefix)`. The database is named
|
|
121
|
+
`<prefix>:<path>`, with `/` in the path turned into `:`. Only this site can
|
|
122
|
+
read them.
|
|
74
123
|
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
|
79
|
-
|
|
|
80
|
-
| `get(key)` | Entry bytes, or null |
|
|
81
|
-
| `put(key, bytes)` | Set an entry |
|
|
82
|
-
| `delete(key)` | Remove an entry; absent is fine |
|
|
83
|
-
| `list(prefix?)` | Every entry key starting with `prefix`, in no guaranteed order |
|
|
84
|
-
| `batch(ops)` | Apply `{type:'put',key,value}` / `{type:'delete',key}` ops; should be atomic |
|
|
85
|
-
| `putVersion(v)` | Store a version body by its `id` |
|
|
86
|
-
| `getVersion(id)` | A version body, or null |
|
|
87
|
-
| `deleteVersion(id)` | Remove a version body |
|
|
88
|
-
| `close()` | Release it |
|
|
89
|
-
| `entries(prefix)` | _Optional._ Every entry under `prefix` with its bytes, in one read. Without it, `list` then `get` each |
|
|
90
|
-
| `getVersions(ids)` | _Optional._ Version bodies by id, null where absent, in the order asked, in one read. Without it, `getVersion` each |
|
|
91
|
-
| `commit({ store, ops, remove })` | _Optional._ Stores these bodies, applies these entry ops and deletes these bodies, atomically. Without it, `putVersion` each, `batch`, `deleteVersion` each |
|
|
92
|
-
|
|
93
|
-
The optional methods only save round trips: a store gives the same answers
|
|
94
|
-
with or without them. With `commit`, one placement of a version is one atomic
|
|
95
|
-
write; without, versions go first, then the entries in one batch, then the
|
|
96
|
-
deletes, so no entry ever names a version that is not there.
|
|
97
|
-
|
|
98
|
-
Entry keys are strings; values are bytes. The adapters are IndexedDB (below),
|
|
99
|
-
a directory (`directoryStores`: the data folder's registry, and a node's
|
|
100
|
-
stores on disk), the sealing wrapper, and an in-memory one for tests.
|
|
101
|
-
|
|
102
|
-
_Source: `packages/core/src/types.ts` (`StorageAdapter`, `BatchOp`). Tests:
|
|
103
|
-
`packages/core/tests/folder-adapter.test.ts`,
|
|
104
|
-
`packages/core/tests/helpers/memory-adapter.ts`._
|
|
124
|
+
| Who | Prefix |
|
|
125
|
+
| ----------------------------------- | ----------------------------------------------------------------------- |
|
|
126
|
+
| An account signed in in the browser | `minca:<dataPath>`, as `minca:accounts:k3x9q2:stores:spaces:<space id>` |
|
|
127
|
+
| An app connected to an account home | `minca-app:<account DID>` |
|
|
128
|
+
| A data folder's own stores | `minca-pod:<dataPath>` |
|
|
105
129
|
|
|
106
|
-
|
|
130
|
+
A data folder uses `minca-pod:`, not `minca:`, because `minca:` is deleted
|
|
131
|
+
when the person forgets an account's browser copy.
|
|
132
|
+
|
|
133
|
+
Each database is version 4. Opening one made at an earlier version empties
|
|
134
|
+
it. Space records come back by syncing, but the registry is lost with it:
|
|
135
|
+
space keys, unused invites, and anything written here and not yet synced.
|
|
136
|
+
That is accepted before v1. After it, a change must keep the registry.
|
|
107
137
|
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
`
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
- is version **4**. Opening one made at an earlier version deletes its object
|
|
117
|
-
stores and starts empty. Space records are rebuilt by syncing, but the
|
|
118
|
-
registry is lost with it: space keys, unused invite secrets, and versions
|
|
119
|
-
written here but not yet synced. Before v1 that is accepted; after it, a
|
|
120
|
-
schema change must keep the registry.
|
|
121
|
-
- has object store `kv`: out-of-line string keys, values `ArrayBuffer`s (the
|
|
122
|
-
entry bytes). `list(prefix)` is a key-range cursor over
|
|
123
|
-
`[prefix, prefix + ""]`.
|
|
124
|
-
- has object store `versions`: key path `id`, no indexes. Values are the
|
|
125
|
-
version objects.
|
|
126
|
-
- writes `batch` as one `readwrite` transaction over `kv`. Bodies are written
|
|
127
|
-
in their own transactions, except through `commit`: one `readwrite`
|
|
128
|
-
transaction over `kv` and `versions`.
|
|
129
|
-
- reads `entries(prefix)` with `getAllKeys` and `getAll` over the same key
|
|
130
|
-
range in one `readonly` transaction; `getVersions` is one `readonly`
|
|
138
|
+
Inside each database:
|
|
139
|
+
|
|
140
|
+
- Object store `kv` holds entries: string keys, `ArrayBuffer` values.
|
|
141
|
+
`list(prefix)` is a key-range cursor over `[prefix, prefix + ""]`.
|
|
142
|
+
- Object store `versions` holds version bodies, keyed by `id`.
|
|
143
|
+
- `batch` is one `readwrite` transaction over `kv`. Bodies get their own
|
|
144
|
+
transactions, except through `commit`, which is one transaction over both.
|
|
145
|
+
- `entries(prefix)` and `getVersions` each read in one `readonly`
|
|
131
146
|
transaction.
|
|
132
147
|
|
|
133
|
-
|
|
134
|
-
`minca-pod:<dataPath with / replaced by :>`: not `minca:`, which is deleted
|
|
135
|
-
when the person forgets an account's browser copy.
|
|
148
|
+
Other databases:
|
|
136
149
|
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
150
|
+
| Database | Object store | Holds |
|
|
151
|
+
| ------------------- | ------------ | ------------------------------------------------------------------- |
|
|
152
|
+
| `minca-accounts` | `accounts` | Accounts kept in the browser ([sign-in](sign-in.md#account-stores)) |
|
|
153
|
+
| `minca-folder` | `handles` | The remembered data folder, under key `data-folder` |
|
|
154
|
+
| `minca-device-keys` | `keys` | Device keys ([sign-in](sign-in.md#device-keys-and-passkeys)) |
|
|
140
155
|
|
|
141
156
|
A host keeps its subscriptions in its own store, path `host`, under keys
|
|
142
|
-
`subscription:<id>`
|
|
143
|
-
|
|
144
|
-
_Source: `packages/core/src/storage/indexeddb-adapter.ts`,
|
|
145
|
-
`packages/core/src/storage/directory-access.ts`,
|
|
146
|
-
`packages/core/src/node/stores.ts`, `packages/core/src/session/places.ts`
|
|
147
|
-
(`storesFor`), `packages/core/src/node/host.ts`. Tests: none directly (the
|
|
148
|
-
adapter needs a browser)._
|
|
157
|
+
`subscription:<id>`
|
|
158
|
+
([spec 06](https://github.com/leifriksheim/minca/blob/main/spec/06-nodes-and-sessions.md)).
|
|
149
159
|
|
|
150
160
|
## Data folders
|
|
151
161
|
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
`folderStores(directory, { basePath, localKey, local })`
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
([mirrors](#mirrors))
|
|
163
|
-
`watchIntervalMs` (default 2 s, `0`
|
|
164
|
-
change and on close.
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
162
|
+
A data folder, or pod, is the whole account in a folder. Copy it to a USB
|
|
163
|
+
stick and you have all of it. Put it in iCloud, Dropbox or Syncthing, and
|
|
164
|
+
several devices converge with no relay, since records only ever add up.
|
|
165
|
+
|
|
166
|
+
`folderStores(directory, { basePath, localKey, local })` gives a node its
|
|
167
|
+
stores for one:
|
|
168
|
+
|
|
169
|
+
- The `registry` lives in the folder, one file per entry, sealed under
|
|
170
|
+
`localKey`.
|
|
171
|
+
- Each space lives in the folder as segments, under `<basePath>/spaces/`.
|
|
172
|
+
The node mirrors to them ([mirrors](#mirrors)). It pulls when a space opens
|
|
173
|
+
and every `watchIntervalMs` (default 2 s, `0` turns it off), and pushes 1 s
|
|
174
|
+
after a change and on close.
|
|
175
|
+
- Everything else goes in `local`, this device's own stores.
|
|
176
|
+
|
|
177
|
+
Each device adds a folder of segments per space, about one file per 256 KiB
|
|
178
|
+
of records. It compacts them once it has 32, and twice what the last
|
|
179
|
+
compaction left.
|
|
180
|
+
|
|
181
|
+
### This device's stores
|
|
182
|
+
|
|
183
|
+
`local` is where:
|
|
184
|
+
|
|
185
|
+
- **In the browser**, IndexedDB under `minca-pod:<dataPath>`, by default.
|
|
186
|
+
- **The CLI** keeps it outside the folder, since the folder may be one a
|
|
187
|
+
browser or a sync service also uses. It is `$MINCA_LOCAL`, or
|
|
188
|
+
`~/.local/state/minca`, then `<first 16 hex of SHA-256 of the home's
|
|
189
|
+
path>/<account id>/stores`. A
|
|
190
|
+
command and the daemon share it, and the daemon finds the command's writes
|
|
191
|
+
(`reconcileFolder`). Writer keys are beside it in `writer-keys/`. Older
|
|
192
|
+
slots left in the home, and `writers/`, are not read, so two versions of
|
|
193
|
+
minca never sign with one key. `minca agent` and the commands an agent runs
|
|
194
|
+
keep each session's key and stores in a slot of `<home>/agent`
|
|
180
195
|
([actions](actions.md#the-agent-on-a-computer)).
|
|
181
|
-
- **Elsewhere**
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
A space kept in an older layout is not read
|
|
186
|
-
([spec 05 §14.8](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md));
|
|
187
|
-
peers bring it back.
|
|
188
|
-
|
|
189
|
-
Getting the folder:
|
|
190
|
-
|
|
191
|
-
- **In the browser** the folder comes from `showDirectoryPicker` with
|
|
192
|
-
`id: "minca-pod"`, `mode: "readwrite"`, `startIn: "documents"`, which needs a
|
|
193
|
-
user gesture. The handle is remembered in IndexedDB (above); on a later visit
|
|
194
|
-
`readwrite` permission is queried, and requested again only from a gesture.
|
|
195
|
-
Browsers without the API (Firefox, Safari, mobile) keep data in IndexedDB
|
|
196
|
-
instead. The File System Access API replaces a file atomically on `close()`.
|
|
197
|
-
- **The CLI** (Node and Bun) writes `<name>.<uuid>.tmp` with mode `0600` and
|
|
198
|
-
renames it into place. Directories are created with mode `0700`.
|
|
199
|
-
|
|
200
|
-
Two folders holding the same account (a copy, a restored backup) share this
|
|
201
|
-
device's local stores, which are named by the account's id: a version known
|
|
202
|
-
to be in one is not pushed to the other. Open one copy at a time, or let the
|
|
203
|
-
other fill in from peers.
|
|
204
|
-
|
|
205
|
-
A folder is the account: copy it to a USB stick and it is all of it, or put
|
|
206
|
-
it in iCloud, Dropbox or Syncthing and several devices converge with no relay,
|
|
207
|
-
since the set of versions only grows. Being readable by several origins is the
|
|
208
|
-
point, which is why OPFS (origin-private) is not used for it. Each device
|
|
209
|
-
adds a folder of segments per space, about one file per 256 KiB of records,
|
|
210
|
-
compacted once it has 32 and twice what the last compaction left.
|
|
211
|
-
|
|
212
|
-
_Source: `packages/core/src/node/stores.ts` (`folderStores`),
|
|
213
|
-
`packages/core/src/storage/blob/directory.ts`,
|
|
214
|
-
`packages/core/src/storage/directory-access.ts`,
|
|
215
|
-
`packages/cli/src/fs-directory.ts`, `packages/cli/src/home.ts` (`unlock`),
|
|
216
|
-
`packages/core/src/node/space-mirrors.ts` (`openMirrors`, `watch`). Tests: `packages/core/tests/folder-adapter.test.ts`,
|
|
217
|
-
`packages/cli/tests/path-safety.test.ts`._
|
|
196
|
+
- **Elsewhere** you must pass one. `folderStores` refuses to start without
|
|
197
|
+
it, since a key's state kept only in memory would be reused after a
|
|
198
|
+
restart.
|
|
218
199
|
|
|
219
|
-
|
|
200
|
+
Two folders holding the same account, such as a copy or a restored backup,
|
|
201
|
+
share this device's local stores. So a version known to be in one is not
|
|
202
|
+
pushed to the other. Open one copy at a time, or let the other fill in from
|
|
203
|
+
peers.
|
|
204
|
+
|
|
205
|
+
### Getting the folder
|
|
220
206
|
|
|
221
|
-
|
|
222
|
-
|
|
207
|
+
- **In the browser** it comes from `showDirectoryPicker` with
|
|
208
|
+
`id: "minca-pod"`, `mode: "readwrite"` and `startIn: "documents"`, which
|
|
209
|
+
needs a click. The handle is remembered in IndexedDB. On a later visit the
|
|
210
|
+
permission is checked, and asked for again only from a click. Firefox,
|
|
211
|
+
Safari and mobile browsers don't have the API, and keep data in IndexedDB
|
|
212
|
+
instead. A file is replaced atomically on `close()`.
|
|
213
|
+
- **The CLI**, in Node and Bun, writes `<name>.<uuid>.tmp` with mode `0600`
|
|
214
|
+
and renames it into place. Directories get mode `0700`.
|
|
223
215
|
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
| `get(key)` | bytes, or null when absent |
|
|
227
|
-
| `put(key, bytes)` | create or replace |
|
|
228
|
-
| `delete(key)` | remove; absent is fine |
|
|
229
|
-
| `list(prefix)` | every key under `prefix`, any order |
|
|
230
|
-
| `changes?(prefix, cursor)` | optional: keys added or removed since `cursor`; no mirror uses it yet |
|
|
216
|
+
A folder is meant to be read from several sites, so it can't be OPFS, which
|
|
217
|
+
belongs to one.
|
|
231
218
|
|
|
232
|
-
|
|
219
|
+
A space kept in an older layout is not read. Peers bring it back. The
|
|
220
|
+
folder's layout, and how several writers converge, are in
|
|
221
|
+
[spec 05 §14](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md).
|
|
222
|
+
|
|
223
|
+
## Blob stores
|
|
224
|
+
|
|
225
|
+
A blob store keeps bytes by name. It is assumed slow, eventually consistent
|
|
226
|
+
and without atomic operations. A
|
|
227
|
+
[`BlobStore`](https://minca.dev/api/core/interfaces/BlobStore/) only gets, puts, deletes and
|
|
228
|
+
lists keys under a prefix.
|
|
229
|
+
|
|
230
|
+
The library has three:
|
|
233
231
|
|
|
234
232
|
- **Memory**, for tests.
|
|
235
|
-
- **Directory
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
`packages/core/src/storage/blob/memory.ts`,
|
|
250
|
-
`packages/core/src/storage/blob/directory.ts`,
|
|
251
|
-
`packages/core/src/storage/blob/s3.ts`. Tests:
|
|
252
|
-
`packages/core/tests/mirror.test.ts`._
|
|
233
|
+
- **Directory** (`createDirectoryBlobStore`), over a folder handle or a
|
|
234
|
+
folder on disk. Each `/`-separated part of a key is a folder, and the last
|
|
235
|
+
a file, named as a data folder names its files. A folder goes with the
|
|
236
|
+
last key in it. This is what a data folder keeps its spaces in.
|
|
237
|
+
- **S3-compatible**: R2, B2, MinIO, Wasabi and AWS.
|
|
238
|
+
- Path-style URLs, `<endpoint>/<bucket>/<prefix/><key>`, each part
|
|
239
|
+
`encodeURIComponent`-ed.
|
|
240
|
+
- Signed with SigV4 (`aws4fetch`). Region defaults to `auto`.
|
|
241
|
+
- `list` is `ListObjectsV2` (`list-type=2`), following continuation
|
|
242
|
+
tokens.
|
|
243
|
+
- A 404 on `get` is null, and on `delete` is success.
|
|
244
|
+
- 429 and 5xx are tried up to 5 times. It waits `Retry-After` seconds if
|
|
245
|
+
given, else `min(200·2^attempt, 5000)` ms times a random factor in
|
|
246
|
+
[0.75, 1.25).
|
|
253
247
|
|
|
254
248
|
## Mirrors
|
|
255
249
|
|
|
256
|
-
A mirror
|
|
257
|
-
|
|
258
|
-
Today data folders, carriers and hosts mirror.
|
|
250
|
+
A mirror copies a space into a blob store, so devices that are never online
|
|
251
|
+
together can still meet. Today data folders, carriers and hosts mirror.
|
|
259
252
|
|
|
260
|
-
A store writer
|
|
261
|
-
the
|
|
253
|
+
A store writer pushes 5 s after the first change, 1 s for a data folder, and
|
|
254
|
+
on close. It packs what the store doesn't have yet into segments of about
|
|
255
|
+
256 KiB. Once it has 32 or more segments of its own, and twice as many as its
|
|
256
|
+
last compaction left, it compacts them, keeping only what its store holds.
|
|
262
257
|
|
|
263
|
-
|
|
264
|
-
|
|
265
|
-
| `mirror:writer` | the writer id |
|
|
266
|
-
| `mirror:counter` | the segment counter, decimal |
|
|
267
|
-
| `mirror:read:<segment key>` | `1`, for each segment read or written |
|
|
268
|
-
| `mirror:known:<version id>` | `1`, for each version known to be in the store |
|
|
258
|
+
It keeps its state in the store `mirrors/<space id>/<n>`, where `n` is the
|
|
259
|
+
mirror's index:
|
|
269
260
|
|
|
270
|
-
|
|
271
|
-
|
|
261
|
+
| Key | Value |
|
|
262
|
+
| --------------------------- | --------------------------------------- |
|
|
263
|
+
| `mirror:writer` | The writer id |
|
|
264
|
+
| `mirror:counter` | The segment counter, in decimal |
|
|
265
|
+
| `mirror:read:<segment key>` | `1`, for each segment read or written |
|
|
266
|
+
| `mirror:known:<version id>` | `1`, for each version known to be there |
|
|
272
267
|
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
entries once, when it starts. Once it has 32 or more segments of its own,
|
|
277
|
-
and twice as many as its last compaction left, it compacts them, keeping
|
|
278
|
-
only what its store holds.
|
|
268
|
+
"Known" is everything it uploaded or read, so nobody uploads everyone
|
|
269
|
+
else's versions again. It reads the `mirror:known:` entries once, when it
|
|
270
|
+
starts.
|
|
279
271
|
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
`packages/core/src/node/host.ts`. Tests: `packages/core/tests/mirror.test.ts`._
|
|
272
|
+
Segments, who may write where, pulling and compacting are protocol, in
|
|
273
|
+
[spec 05 §16](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md).
|
|
283
274
|
|
|
284
275
|
### Planned: mirrors in your own storage
|
|
285
276
|
|
|
286
|
-
> **Planned
|
|
287
|
-
>
|
|
288
|
-
>
|
|
289
|
-
|
|
290
|
-
- **Any node, several mirrors.** A node
|
|
291
|
-
(`NodeConfig.mirrors`),
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
298
|
-
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
-
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
updated on every `put` and invalidated on 404. Drive allows duplicate
|
|
307
|
-
names, so a `put` updates an existing file id rather than creating
|
|
308
|
-
another. Tokens come from a callback (`getAccessToken`); service accounts
|
|
309
|
-
do not work (they have no storage of their own).
|
|
310
|
-
- _Dropbox_ and _OneDrive_, in their app folders, the narrowest grant each
|
|
311
|
-
offers. Dropbox first.
|
|
312
|
-
- Not iCloud: it has no usable web API; iCloud users keep a data folder
|
|
313
|
-
instead.
|
|
314
|
-
- **Budget.** A store like Drive allows about 1,000 requests per 100 seconds.
|
|
315
|
-
The segment, never the single version, is the unit fetched, and syncing
|
|
316
|
-
1,000 records must fit that budget.
|
|
277
|
+
> **Planned:** mirrors on any node, in storage the person already has. The
|
|
278
|
+
> protocol side is planned in
|
|
279
|
+
> [spec 05 §16.5](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md).
|
|
280
|
+
|
|
281
|
+
- **Any node, several mirrors.** A node takes a list of blob stores
|
|
282
|
+
(`NodeConfig.mirrors`), such as the person's Dropbox and a host's bucket at
|
|
283
|
+
once. It pulls when a space opens and on each change, and pushes after
|
|
284
|
+
writes and on close.
|
|
285
|
+
- **Change feeds.** Where the service has one, pulling uses
|
|
286
|
+
`BlobStore.changes` instead of listing: Dropbox `list_folder/continue`,
|
|
287
|
+
Drive `changes.list`.
|
|
288
|
+
- **More drivers**, each passing one shared test suite, with shared backoff
|
|
289
|
+
on 429 and 5xx. None ever logs a token or signed URL.
|
|
290
|
+
- _Google Drive_, in one folder the app made, with `drive.file` scope only.
|
|
291
|
+
Drive finds files by id, not name, so the driver keeps a map from names
|
|
292
|
+
to ids. Service accounts don't work: they have no storage of their own.
|
|
293
|
+
- _Dropbox_ and _OneDrive_, in their app folders. Dropbox first.
|
|
294
|
+
- Not iCloud, which has no usable web API. iCloud users keep a data folder.
|
|
295
|
+
- **Budget.** Drive allows about 1,000 requests per 100 seconds. Segments,
|
|
296
|
+
never single versions, are fetched, and syncing 1,000 records must fit.
|
|
317
297
|
|
|
318
|
-
##
|
|
298
|
+
## Storage adapters
|
|
319
299
|
|
|
320
|
-
|
|
321
|
-
|
|
322
|
-
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
- **Adding.** `records.addFile(space, bytes, { mime, name? })` seals the
|
|
326
|
-
bytes under a fresh key and salt, keeps them, and gives back a `FileRef`:
|
|
327
|
-
`{ hash, size, key, salt, mime, name? }`. Put it anywhere in a body;
|
|
328
|
-
writing that body names every reference in it on the version
|
|
329
|
-
(`fileRefsIn`, at most 16), so the envelope and the body always agree.
|
|
330
|
-
- **Reading.** `records.readFile(space, ref)` gives the bytes: held here, or
|
|
331
|
-
fetched from connected peers, one after another, then from mirrors,
|
|
332
|
-
checking every chunk. When none has them it waits, asking everyone again
|
|
333
|
-
whenever records change or a peer connects, and every 15 s, and resolves
|
|
334
|
-
once one does. It rejects for a file no version this node keeps whole
|
|
335
|
-
names, one whose id is here whole yet won't open as named, and when the
|
|
336
|
-
space closes. For a picture, `URL.createObjectURL(new Blob([bytes], {
|
|
337
|
-
type: ref.mime }))` shows it.
|
|
338
|
-
- **Where chunks live.** Entries in the space's own store: `f/<id>/list`,
|
|
339
|
-
`f/<id>/<n>`, `f/<id>/own` with the time a file was added here (0 once a
|
|
340
|
-
version names it), and `f/<id>/used` with when it was last fetched or
|
|
341
|
-
read. A data folder keeps these in this device's own stores, not the
|
|
342
|
-
folder.
|
|
343
|
-
- **What a node fetches.** Files follow records. A node fetches every file
|
|
344
|
-
named by a version it keeps whole about 2 s after its records change,
|
|
345
|
-
while the writer is likely still connected. A file this sweep found
|
|
346
|
-
nobody had is not asked of everyone again for 10 minutes, but is asked at
|
|
347
|
-
once of each peer that connects meanwhile.
|
|
348
|
-
- **A device's limits.** A node limits the files it did not add, in every
|
|
349
|
-
space: it fetches unread only files of up to 5 MiB, so photos and
|
|
350
|
-
documents come ahead and video when played. Past 256 MiB of them in a
|
|
351
|
-
space, counting each at the size its list proves, or every chunk held of
|
|
352
|
-
one never whole, it drops those fetched or opened longest ago, down to
|
|
353
|
-
90%, at the start of each pass and as it fetches. One never whole goes
|
|
354
|
-
first. A file added here is never dropped, nor one being fetched or read;
|
|
355
|
-
one larger than the limit is dropped on the pass after it is read, and
|
|
356
|
-
fetched again at the next read. A dropped file that was whole keeps its
|
|
357
|
-
`used` entry, so it is not fetched unread again while a version names it;
|
|
358
|
-
reading it fetches it. The limit is per space, so a device in many spaces
|
|
359
|
-
holds more. A node with `fetchFiles: 'all'`, a carrier or host, fetches
|
|
360
|
-
and keeps every file.
|
|
361
|
-
- **Serving.** A `want-file` is answered with at most 4 chunks; the asker
|
|
362
|
-
asks again for the rest, so a large file never floods a connection.
|
|
363
|
-
- **Keeping.** About 2 s after records change, files no version kept whole
|
|
364
|
-
names are dropped (files of versions that don't stand are not kept),
|
|
365
|
-
except one added here in the last 24 hours and not yet named.
|
|
366
|
-
- **Mirrors.** The same pass uploads each named file this node holds whole,
|
|
367
|
-
with its chunk list, to `files/<space>/<store writer>/`, and deletes this
|
|
368
|
-
writer's copies of files no longer named.
|
|
369
|
-
|
|
370
|
-
_Source: `packages/core/src/files/space-files.ts`, `packages/core/src/files/file-id.ts`,
|
|
371
|
-
`packages/core/src/files/seal.ts`, `packages/core/src/node/space-runtime.ts` (`filesChanged`).
|
|
372
|
-
Tests: `packages/core/tests/files.test.ts`._
|
|
300
|
+
A store runs over a [`StorageAdapter`](https://minca.dev/api/core/interfaces/StorageAdapter/).
|
|
301
|
+
Write one to keep a node's data somewhere new. Every method is async. It keeps
|
|
302
|
+
entries, string keys to bytes, and version bodies by id. A few methods are
|
|
303
|
+
optional.
|
|
373
304
|
|
|
374
|
-
|
|
305
|
+
The optional methods only save round trips. A store gives the same answers
|
|
306
|
+
without them, using `list` and `get`, `getVersion`, or `putVersion`, `batch`
|
|
307
|
+
and `deleteVersion` instead.
|
|
308
|
+
|
|
309
|
+
With `commit`, placing a version is one atomic write. Without it, versions go
|
|
310
|
+
first, then the entries in one batch, then the deletes. So no entry ever
|
|
311
|
+
names a version that isn't there.
|
|
312
|
+
|
|
313
|
+
The library's adapters are IndexedDB ([above](#indexeddb)), a directory
|
|
314
|
+
(`directoryStores`, for a data folder's registry and a node's stores on
|
|
315
|
+
disk), a sealing wrapper, and one in memory for tests.
|
|
316
|
+
|
|
317
|
+
## The store in memory
|
|
318
|
+
|
|
319
|
+
A store reads its entries once, then keeps what it needs in memory: each
|
|
320
|
+
collection's set of records, and the entries it read or wrote. It also keeps
|
|
321
|
+
up to 50,000 version bodies, letting the oldest go first. A body never
|
|
322
|
+
changes, so this is always safe. An entry read while a change was landing is
|
|
323
|
+
not kept.
|
|
324
|
+
|
|
325
|
+
When another writer changes the same store, the node reads all of it again
|
|
326
|
+
(`invalidate`):
|
|
327
|
+
|
|
328
|
+
- **Tabs in one browser.** Where shared workers work, the apps in this
|
|
329
|
+
repository run one node for all of a site's tabs in a `SharedWorker`, one
|
|
330
|
+
per account and key. Then there's nothing to tell.
|
|
331
|
+
- **Elsewhere** each tab runs its node in a worker of its own. Tabs tell each other through a
|
|
332
|
+
`BroadcastChannel` named `minca-node:<root DID>:<space id>`. Chrome on
|
|
333
|
+
Android has no shared workers, and WebKit drops a message to one that
|
|
334
|
+
holds a `CryptoKey`.
|
|
335
|
+
- **Writers on a shared folder** are found by reading it again
|
|
336
|
+
([data folders](#data-folders)).
|
|
337
|
+
|
|
338
|
+
The store's fingerprint, in `spaces.status`, is the Negentropy fingerprint
|
|
339
|
+
of every collection together, as hex. Two nodes with the same fingerprint
|
|
340
|
+
hold the same data.
|
|
341
|
+
|
|
342
|
+
## Pacing sync
|
|
375
343
|
|
|
376
|
-
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
|
|
382
|
-
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
|
|
386
|
-
|
|
387
|
-
|
|
388
|
-
|
|
389
|
-
|
|
344
|
+
The spec says which sync messages go, and when a peer must answer. How often
|
|
345
|
+
a node speaks, and how much it asks for at once, is its own choice:
|
|
346
|
+
|
|
347
|
+
| What | Value | Why |
|
|
348
|
+
| --------------------- | ---------------------------------- | -------------------------------------------------------------------------------------- |
|
|
349
|
+
| Heartbeat | a `hello` to every peer every 30 s | Finds what a lost message missed |
|
|
350
|
+
| Hello after taking in | about 100 ms | So a new version passed along doesn't wait a heartbeat at every hop |
|
|
351
|
+
| `want`s in flight | 4 per peer (`MAX_WANTS_IN_FLIGHT`) | When one is answered, the next goes |
|
|
352
|
+
| Rounds per session | 64 | Counted by the side that started. Past 64 the session is given up |
|
|
353
|
+
| Stale after | 30 s | A silent session, or an unanswered `want`, is dropped and found again later |
|
|
354
|
+
| Versions waiting | 1,000 | Versions held back for later wait in memory. Past 1,000 the oldest go |
|
|
355
|
+
| Refusals remembered | 10,000 | A refused version isn't asked of that peer again. Past 10,000 the oldest are forgotten |
|
|
356
|
+
|
|
357
|
+
The node asks first for the space's own records (`sys.*`), and for earlier
|
|
358
|
+
versions a waiting version needs. Everything else waits its turn. An id
|
|
359
|
+
already asked for is not asked again.
|
|
360
|
+
|
|
361
|
+
A session with a peer that is running and has heard something in the last
|
|
362
|
+
30 s isn't started again. It runs once more when it ends.
|
|
363
|
+
|
|
364
|
+
Each space reports `synced` and `level` with each peer, which
|
|
365
|
+
[holding part of a space](node.md#holding-part-of-a-space) uses. The page is
|
|
366
|
+
told records changed once per message taken in, not once per version.
|