@minca/core 0.1.16 → 0.1.18

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (125) hide show
  1. package/README.md +10 -9
  2. package/dist/elements/auth-styles.d.ts +1 -1
  3. package/dist/elements/auth-styles.d.ts.map +1 -1
  4. package/dist/elements/auth-styles.js +14 -5
  5. package/dist/elements/auth-styles.js.map +1 -1
  6. package/dist/elements/minca-auth.d.ts.map +1 -1
  7. package/dist/elements/minca-auth.js +18 -15
  8. package/dist/elements/minca-auth.js.map +1 -1
  9. package/dist/files/space-files.d.ts +2 -2
  10. package/dist/files/space-files.d.ts.map +1 -1
  11. package/dist/files/space-files.js +7 -5
  12. package/dist/files/space-files.js.map +1 -1
  13. package/dist/inbox/names.d.ts +1 -1
  14. package/dist/inbox/names.js +1 -1
  15. package/dist/index.d.ts +2 -0
  16. package/dist/index.d.ts.map +1 -1
  17. package/dist/index.js +2 -0
  18. package/dist/index.js.map +1 -1
  19. package/dist/network/ws-transport.d.ts.map +1 -1
  20. package/dist/network/ws-transport.js +3 -0
  21. package/dist/network/ws-transport.js.map +1 -1
  22. package/dist/node/actions.d.ts +5 -0
  23. package/dist/node/actions.d.ts.map +1 -1
  24. package/dist/node/actions.js +39 -20
  25. package/dist/node/actions.js.map +1 -1
  26. package/dist/node/api.d.ts +2 -0
  27. package/dist/node/api.d.ts.map +1 -1
  28. package/dist/node/api.js +2 -0
  29. package/dist/node/api.js.map +1 -1
  30. package/dist/node/carrier.js +1 -1
  31. package/dist/node/carrier.js.map +1 -1
  32. package/dist/node/node-carriers.d.ts.map +1 -1
  33. package/dist/node/node-carriers.js +6 -1
  34. package/dist/node/node-carriers.js.map +1 -1
  35. package/dist/node/node-contacts.d.ts.map +1 -1
  36. package/dist/node/node-contacts.js +1 -0
  37. package/dist/node/node-contacts.js.map +1 -1
  38. package/dist/node/node-hosting.d.ts.map +1 -1
  39. package/dist/node/node-hosting.js +11 -4
  40. package/dist/node/node-hosting.js.map +1 -1
  41. package/dist/node/node-records.d.ts.map +1 -1
  42. package/dist/node/node-records.js +6 -1
  43. package/dist/node/node-records.js.map +1 -1
  44. package/dist/node/node.d.ts.map +1 -1
  45. package/dist/node/node.js +14 -1
  46. package/dist/node/node.js.map +1 -1
  47. package/dist/node/remote.d.ts.map +1 -1
  48. package/dist/node/remote.js +7 -1
  49. package/dist/node/remote.js.map +1 -1
  50. package/dist/node/space-holding.d.ts +7 -3
  51. package/dist/node/space-holding.d.ts.map +1 -1
  52. package/dist/node/space-holding.js +90 -18
  53. package/dist/node/space-holding.js.map +1 -1
  54. package/dist/node/space-peers.d.ts.map +1 -1
  55. package/dist/node/space-peers.js +58 -24
  56. package/dist/node/space-peers.js.map +1 -1
  57. package/dist/node/space-runtime.d.ts +11 -3
  58. package/dist/node/space-runtime.d.ts.map +1 -1
  59. package/dist/node/space-runtime.js +22 -1
  60. package/dist/node/space-runtime.js.map +1 -1
  61. package/dist/node/space-writing.d.ts +2 -0
  62. package/dist/node/space-writing.d.ts.map +1 -1
  63. package/dist/node/space-writing.js +2 -0
  64. package/dist/node/space-writing.js.map +1 -1
  65. package/dist/node/types.d.ts +130 -36
  66. package/dist/node/types.d.ts.map +1 -1
  67. package/dist/query/types.d.ts +1 -0
  68. package/dist/query/types.d.ts.map +1 -1
  69. package/dist/query/types.js.map +1 -1
  70. package/dist/records/contract-builder.d.ts +22 -6
  71. package/dist/records/contract-builder.d.ts.map +1 -1
  72. package/dist/records/contract-builder.js.map +1 -1
  73. package/dist/schemas/automations.d.ts +6 -2
  74. package/dist/schemas/automations.d.ts.map +1 -1
  75. package/dist/schemas/automations.js.map +1 -1
  76. package/dist/schemas/connectors.d.ts +4 -3
  77. package/dist/schemas/connectors.d.ts.map +1 -1
  78. package/dist/schemas/connectors.js +4 -3
  79. package/dist/schemas/connectors.js.map +1 -1
  80. package/dist/schemas/library/media.js +1 -1
  81. package/dist/schemas/library/media.js.map +1 -1
  82. package/dist/schemas/library/social.d.ts +258 -0
  83. package/dist/schemas/library/social.d.ts.map +1 -1
  84. package/dist/schemas/library/social.js +28 -1
  85. package/dist/schemas/library/social.js.map +1 -1
  86. package/dist/session/auth.d.ts +7 -3
  87. package/dist/session/auth.d.ts.map +1 -1
  88. package/dist/session/auth.js.map +1 -1
  89. package/dist/session/hosting.d.ts.map +1 -1
  90. package/dist/session/hosting.js +8 -2
  91. package/dist/session/hosting.js.map +1 -1
  92. package/dist/space/presets.d.ts +2 -2
  93. package/dist/space/presets.d.ts.map +1 -1
  94. package/dist/space/presets.js +3 -2
  95. package/dist/space/presets.js.map +1 -1
  96. package/dist/storage/blob-store.d.ts +3 -0
  97. package/dist/storage/blob-store.d.ts.map +1 -1
  98. package/dist/types.d.ts +11 -2
  99. package/dist/types.d.ts.map +1 -1
  100. package/dist/types.js.map +1 -1
  101. package/docs/README.md +86 -93
  102. package/docs/actions.md +212 -144
  103. package/docs/agents.md +198 -155
  104. package/docs/automations.md +296 -204
  105. package/docs/blueprints-and-guides.md +405 -295
  106. package/docs/bots.md +208 -134
  107. package/docs/building-an-app.md +74 -163
  108. package/docs/collections.md +248 -328
  109. package/docs/computed.md +112 -89
  110. package/docs/contacts.md +151 -112
  111. package/docs/contracts.md +299 -120
  112. package/docs/direct-messages.md +80 -61
  113. package/docs/inbox.md +138 -93
  114. package/docs/node.md +1072 -972
  115. package/docs/permissions.md +142 -0
  116. package/docs/proposing-collections.md +74 -55
  117. package/docs/query-format.md +150 -126
  118. package/docs/records-and-queries.md +205 -126
  119. package/docs/sign-in.md +496 -418
  120. package/docs/spaces.md +261 -245
  121. package/docs/standard-library.md +473 -504
  122. package/docs/storage.md +325 -348
  123. package/docs/usernames.md +164 -97
  124. package/docs/warrants.md +100 -67
  125. package/package.json +1 -2
package/docs/storage.md CHANGED
@@ -1,389 +1,366 @@
1
1
  # Sync and storage in the library
2
2
 
3
- How the library paces sync, keeps a store in memory, talks to IndexedDB, a
4
- data folder, a file on disk and a blob store, and mirrors a space into one.
3
+ Where a node keeps its data, and how it syncs it. You rarely need this page
4
+ to build an app. Read it to attach files to records, to know what a node keeps
5
+ on a device, or to plug in storage of your own.
5
6
 
6
- > Not protocol. This page describes the reference library, and another
7
- > implementation may do it differently and still interoperate. What peers must
8
- > agree on (the sync messages, the store's entries, the data folder, sealing,
9
- > segments) is in [spec 05](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md).
7
+ ## Files
10
8
 
11
- ## Pacing sync
9
+ Records hold small things. A photo or a document goes in a file, and a record
10
+ points to it:
11
+
12
+ ```ts
13
+ declare const file: File;
14
+ // ---cut---
15
+ const bytes = new Uint8Array(await file.arrayBuffer());
16
+ const photo = await node.records.addFile(space.id, bytes, { mime: file.type, name: file.name });
17
+ await node.records.put(space.id, 'app.post', { text: 'Look', photo });
18
+ ```
19
+
20
+ `addFile` encrypts the bytes under a fresh key and gives back a
21
+ [`FileRef`](https://minca.dev/api/core/interfaces/FileRef/). Put it anywhere
22
+ in a body. When the body is written, every file in it is named on the record,
23
+ at most 16.
24
+
25
+ To show it, read the bytes back:
26
+
27
+ ```ts
28
+ import type { FileRef, NodeRecord } from '@minca/core';
29
+ declare const post: NodeRecord<{ text: string; photo: FileRef }>;
30
+ declare const img: HTMLImageElement;
31
+ // ---cut---
32
+ if (post.body) {
33
+ const { photo } = post.body;
34
+ const bytes = await node.records.readFile(space.id, photo);
35
+ img.src = URL.createObjectURL(new Blob([bytes.slice()], { type: photo.mime }));
36
+ }
37
+ ```
38
+
39
+ `readFile` gives the bytes if they're here. Otherwise it fetches them from
40
+ connected peers, one after another, then from mirrors, checking every chunk.
41
+ When nobody has them, it waits. It asks again whenever records change or a
42
+ peer connects, and every 15 s, and resolves once someone does.
43
+
44
+ It rejects for a file no record here names, for one that won't open as named,
45
+ and when the space closes.
46
+
47
+ ### What a device keeps
48
+
49
+ Files follow records. About 2 s after records change, a node fetches the
50
+ files they name, while the writer is likely still connected. A file nobody
51
+ had is not asked of everyone again for 10 minutes, but is asked at once of
52
+ each peer that connects.
53
+
54
+ A device limits the files it didn't add, per space:
55
+
56
+ - It fetches unread only files up to 5 MiB. So photos and documents come
57
+ ahead of time, and video when played.
58
+ - Past 256 MiB of files in a space, it drops those fetched or opened longest
59
+ ago, down to 90%. A file never fetched whole goes first.
60
+ - It never drops a file added here, or one being fetched or read. A file
61
+ larger than the limit is dropped after it is read, and fetched again at the
62
+ next read.
63
+ - A dropped file that was whole is not fetched again unread while a record
64
+ names it. Reading it fetches it.
65
+
66
+ Because the limit is per space, a device in many spaces holds more. A node
67
+ with `fetchFiles: 'all'`, such as a carrier or host, fetches and keeps every
68
+ file.
69
+
70
+ A person can choose other limits for their device, such as fewer downloads on
71
+ a phone. Give them to the node each time it starts, since it keeps none:
72
+
73
+ ```ts
74
+ // Only thumbnails and other small files come ahead; up to 1 GiB kept per space.
75
+ await node.setFileLimits({ ahead: 256 * 1024, keep: 1024 * 1024 * 1024 });
76
+ ```
77
+
78
+ `Infinity` fetches or keeps everything. New limits take effect in every
79
+ open space within a few seconds: past `keep`, files go at once, and raising
80
+ `ahead` fetches what now fits. A file dropped earlier still waits until it is
81
+ read. Pictures and videos carry a small `thumbnail` (see the
82
+ [standard library](standard-library.md)) so that a device fetching only
83
+ small files still shows them.
84
+
85
+ About 2 s after records change, files no record names are dropped. A file
86
+ added here in the last 24 hours, and not yet named, is kept. Files of
87
+ records that don't stand are not kept.
88
+
89
+ ### How files are served and stored
90
+
91
+ - A peer answers a request for a file with at most 4 chunks at a time, so a
92
+ large file never floods a connection.
93
+ - Chunks are entries in the space's own store: `f/<id>/list`, `f/<id>/<n>`,
94
+ `f/<id>/own` (when it was added here, 0 once a record names it) and
95
+ `f/<id>/used` (when it was last fetched or read). A data folder keeps these
96
+ in this device's own stores, not the folder.
97
+ - Each file a node holds whole goes to its mirrors, with its chunk list,
98
+ under `files/<space>/<store writer>/`. Files no longer named are deleted
99
+ there.
100
+
101
+ What peers must agree on, from chunks to encryption, is in
102
+ [spec 05 §16.6](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md).
12
103
 
13
- The spec says which sync messages go and when a peer must answer. How often a
14
- node speaks, and how much it keeps in flight, is its own:
15
-
16
- | What | Value | Why |
17
- | ------------------------------ | ---------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
18
- | Heartbeat | a `hello` to every peer every 30 s | Finds what a lost push or a dropped `want` missed. |
19
- | Hello after taking in | ≈100 ms | After something new arrives from a peer, a folder or a mirror, so a version passed along does not wait a heartbeat at every hop. |
20
- | `want`s in flight | 4 per peer (`MAX_WANTS_IN_FLIGHT`) | Asking goes through a queue per peer; when a `want` is answered, the next ones go. |
21
- | Rounds per session | 64 | Only the initiator counts rounds. Past 64 the session is abandoned as runaway. |
22
- | Session and `want` stale after | 30 s | A session that has heard nothing, or a `want` unanswered, for 30 s is dropped at the next heartbeat, and the next queued ones go. What a dropped `want` asked for is found again on the next round. |
23
- | Versions waiting | 1,000 | Versions the gatekeeper answers _later_ wait in memory; past 1,000 the oldest give way. |
24
- | Refusals remembered | 10,000 | Refusals are kept per peer and id, so a refused version is not asked of that peer again; past 10,000 the oldest are forgotten. |
25
-
26
- The queue of ids to ask for puts the space's own records (`sys.*`) and the
27
- first or previous versions a waiting version names at the front; everything
28
- else joins the back. An id already queued or asked for is not queued again.
29
-
30
- If a session for a collection with a peer is already running and has heard
31
- something in the last 30 s, a new difference does not start another: the
32
- running one is marked to run again once it ends.
33
-
34
- The node reports `synced` and `level` with each peer as events inside the
35
- space runtime; holding part of a space ([node](node.md#holding-part-of-a-space))
36
- uses them, together with whether the peer holds `"all"`. It tells the page that
37
- records changed once per message taken in, not once per version.
38
-
39
- _Source: `packages/core/src/sync/sync-engine.ts` (`MAX_WANTS_IN_FLIGHT`,
40
- `MAX_ROUNDS`, `STALE_MS`, `MAX_WAITING`, `MAX_REFUSED`, `want`, `pump`,
41
- `sweep`), `packages/core/src/node/space-peers.ts` (heartbeat,
42
- `announceSoon`). Tests: `packages/core/tests/reconcile.test.ts` ("a want whose
43
- answer is lost is given up, and the peer is synced again"),
44
- `packages/core/tests/sync.test.ts`._
104
+ ### Planned: video and pictures
45
105
 
46
- ## The store in memory
106
+ > **Planned:** not normative.
107
+ >
108
+ > - **Video.** Play before the file is whole, through a service worker that
109
+ > answers the browser's range requests from the chunks read so far. MP4 with
110
+ > its index first, or WebM. A poster is a second, small image.
111
+ > - **Several peers at once.** Ask chunks of several peers in parallel.
112
+ > - **Mirrors by range.** Read one chunk from a mirror without the whole file.
113
+ > - **Pictures.** `std.image` gains `width`, `height` and a tiny placeholder
114
+ > shown before the picture arrives. `std.profile`'s `avatar` is resized in
115
+ > the browser before it is added.
47
116
 
48
- A store reads its `i/` entries once, and keeps each collection's item set and
49
- running sum in memory, updated by every change made through it. So are the
50
- `r/` and `g/` entries it read or wrote, and up to 50,000 version bodies (the
51
- oldest let go first): a body never changes under its id. An entry read while
52
- a change was landing is not kept.
53
-
54
- When another writer changed the same store, the node re-reads all of it
55
- (`invalidate`). Other tabs of one browser are told through a
56
- `BroadcastChannel` named `minca-node:<root DID>:<space id>`. Where a browser
57
- has shared workers that work, the apps in this repository run one node for all
58
- of a site's tabs in a `SharedWorker` (one per account and key), so those tabs share one store and have
59
- nothing to tell each other; the channel remains for the rest (Chrome on
60
- Android has none; WebKit drops a message to one that holds a `CryptoKey`),
61
- where each tab's node runs in a worker of its own
62
- (`packages/core/src/node/worker.ts`). Writers on a shared folder are found by
63
- re-reading it ([data folders](#data-folders)).
64
-
65
- The store's overall fingerprint, shown in `spaces.status` and used by tests,
66
- is the Negentropy fingerprint of the sum of every collection's sum, as hex.
67
-
68
- _Source: `packages/core/src/storage/storage-provider.ts` (`MAX_BODIES`,
69
- `invalidate`), `packages/core/src/node/worker.ts`. Tests:
70
- `packages/core/tests/reconcile.test.ts` ("a store written by someone else is
71
- read again once told")._
117
+ ## IndexedDB
72
118
 
73
- ## Storage adapters
119
+ In the browser, a node keeps each store in its own IndexedDB database,
120
+ through `indexedDBStores(prefix)`. The database is named
121
+ `<prefix>:<path>`, with `/` in the path turned into `:`. Only this site can
122
+ read them.
74
123
 
75
- A store runs over an adapter with this contract (`StorageAdapter`, all
76
- methods async):
77
-
78
- | Method | Contract |
79
- | -------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
80
- | `get(key)` | Entry bytes, or null |
81
- | `put(key, bytes)` | Set an entry |
82
- | `delete(key)` | Remove an entry; absent is fine |
83
- | `list(prefix?)` | Every entry key starting with `prefix`, in no guaranteed order |
84
- | `batch(ops)` | Apply `{type:'put',key,value}` / `{type:'delete',key}` ops; should be atomic |
85
- | `putVersion(v)` | Store a version body by its `id` |
86
- | `getVersion(id)` | A version body, or null |
87
- | `deleteVersion(id)` | Remove a version body |
88
- | `close()` | Release it |
89
- | `entries(prefix)` | _Optional._ Every entry under `prefix` with its bytes, in one read. Without it, `list` then `get` each |
90
- | `getVersions(ids)` | _Optional._ Version bodies by id, null where absent, in the order asked, in one read. Without it, `getVersion` each |
91
- | `commit({ store, ops, remove })` | _Optional._ Stores these bodies, applies these entry ops and deletes these bodies, atomically. Without it, `putVersion` each, `batch`, `deleteVersion` each |
92
-
93
- The optional methods only save round trips: a store gives the same answers
94
- with or without them. With `commit`, one placement of a version is one atomic
95
- write; without, versions go first, then the entries in one batch, then the
96
- deletes, so no entry ever names a version that is not there.
97
-
98
- Entry keys are strings; values are bytes. The adapters are IndexedDB (below),
99
- a directory (`directoryStores`: the data folder's registry, and a node's
100
- stores on disk), the sealing wrapper, and an in-memory one for tests.
101
-
102
- _Source: `packages/core/src/types.ts` (`StorageAdapter`, `BatchOp`). Tests:
103
- `packages/core/tests/folder-adapter.test.ts`,
104
- `packages/core/tests/helpers/memory-adapter.ts`._
124
+ | Who | Prefix |
125
+ | ----------------------------------- | ----------------------------------------------------------------------- |
126
+ | An account signed in in the browser | `minca:<dataPath>`, as `minca:accounts:k3x9q2:stores:spaces:<space id>` |
127
+ | An app connected to an account home | `minca-app:<account DID>` |
128
+ | A data folder's own stores | `minca-pod:<dataPath>` |
105
129
 
106
- ## IndexedDB
130
+ A data folder uses `minca-pod:`, not `minca:`, because `minca:` is deleted
131
+ when the person forgets an account's browser copy.
132
+
133
+ Each database is version 4. Opening one made at an earlier version empties
134
+ it. Space records come back by syncing, but the registry is lost with it:
135
+ space keys, unused invites, and anything written here and not yet synced.
136
+ That is accepted before v1. After it, a change must keep the registry.
107
137
 
108
- `indexedDBStores(prefix)` opens one database per store path, named
109
- `<prefix>:<path with / replaced by :>`. An account signed in in the browser
110
- uses the prefix `minca:<dataPath with / replaced by :>`, for example
111
- `minca:accounts:k3x9q2:stores:spaces:<space id>`; an app connected to an
112
- account home uses `minca-app:<account DID>` (the grant's `did`). Only this origin reads them.
113
-
114
- Each database:
115
-
116
- - is version **4**. Opening one made at an earlier version deletes its object
117
- stores and starts empty. Space records are rebuilt by syncing, but the
118
- registry is lost with it: space keys, unused invite secrets, and versions
119
- written here but not yet synced. Before v1 that is accepted; after it, a
120
- schema change must keep the registry.
121
- - has object store `kv`: out-of-line string keys, values `ArrayBuffer`s (the
122
- entry bytes). `list(prefix)` is a key-range cursor over
123
- `[prefix, prefix + "￿"]`.
124
- - has object store `versions`: key path `id`, no indexes. Values are the
125
- version objects.
126
- - writes `batch` as one `readwrite` transaction over `kv`. Bodies are written
127
- in their own transactions, except through `commit`: one `readwrite`
128
- transaction over `kv` and `versions`.
129
- - reads `entries(prefix)` with `getAllKeys` and `getAll` over the same key
130
- range in one `readonly` transaction; `getVersions` is one `readonly`
138
+ Inside each database:
139
+
140
+ - Object store `kv` holds entries: string keys, `ArrayBuffer` values.
141
+ `list(prefix)` is a key-range cursor over `[prefix, prefix + "￿"]`.
142
+ - Object store `versions` holds version bodies, keyed by `id`.
143
+ - `batch` is one `readwrite` transaction over `kv`. Bodies get their own
144
+ transactions, except through `commit`, which is one transaction over both.
145
+ - `entries(prefix)` and `getVersions` each read in one `readonly`
131
146
  transaction.
132
147
 
133
- A data folder's own stores ([below](#data-folders)) use the prefix
134
- `minca-pod:<dataPath with / replaced by :>`: not `minca:`, which is deleted
135
- when the person forgets an account's browser copy.
148
+ Other databases:
136
149
 
137
- Accounts kept in a browser live in database `minca-accounts`, object store
138
- `accounts`. A remembered data-folder handle lives in database `minca-folder`,
139
- object store `handles`, key `data-folder`.
150
+ | Database | Object store | Holds |
151
+ | ------------------- | ------------ | ------------------------------------------------------------------- |
152
+ | `minca-accounts` | `accounts` | Accounts kept in the browser ([sign-in](sign-in.md#account-stores)) |
153
+ | `minca-folder` | `handles` | The remembered data folder, under key `data-folder` |
154
+ | `minca-device-keys` | `keys` | Device keys ([sign-in](sign-in.md#device-keys-and-passkeys)) |
140
155
 
141
156
  A host keeps its subscriptions in its own store, path `host`, under keys
142
- `subscription:<id>` ([spec 06](https://github.com/leifriksheim/minca/blob/main/spec/06-nodes-and-sessions.md)).
143
-
144
- _Source: `packages/core/src/storage/indexeddb-adapter.ts`,
145
- `packages/core/src/storage/directory-access.ts`,
146
- `packages/core/src/node/stores.ts`, `packages/core/src/session/places.ts`
147
- (`storesFor`), `packages/core/src/node/host.ts`. Tests: none directly (the
148
- adapter needs a browser)._
157
+ `subscription:<id>`
158
+ ([spec 06](https://github.com/leifriksheim/minca/blob/main/spec/06-nodes-and-sessions.md)).
149
159
 
150
160
  ## Data folders
151
161
 
152
- The folder's layout, file names and the way several writers converge are
153
- protocol ([spec 05 §14](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md)).
154
- How this library keeps and writes one is not.
155
-
156
- `folderStores(directory, { basePath, localKey, local })` keeps the
157
- `registry` in the folder, one file per entry, sealed under `localKey`, and
158
- opens every other path in `local`: a space's entries and bodies, what the
159
- node keeps beside them, and the `folder/<space id>` store writer state. The
160
- factory names the folder's segments (`<basePath>/spaces/`, a directory blob
161
- store) as `folder`, and every space runtime opened from it mirrors to them
162
- ([mirrors](#mirrors)): it pulls when the space opens and every
163
- `watchIntervalMs` (default 2 s, `0` to turn it off), and pushes 1 s after a
164
- change and on close.
165
-
166
- `local` is this device's:
167
-
168
- - **In the browser**, IndexedDB under `minca-pod:<dataPath>` (the default).
169
- - **The CLI** keeps it outside the home, which may be the folder a browser
170
- uses or one a sync service copies: `$MINCA_LOCAL` or `~/.local/state/minca`,
171
- then `<first 16 hex of SHA-256 of the home's path>/<account id>/stores`, a
172
- directory store (`localPath` in `packages/cli/src/home.ts`). A command and
173
- the daemon share it, and the daemon finds the command's writes as it finds
174
- any directory store's (`reconcileFolder`). Writer keys are beside it in
175
- `writer-keys/`; a slot left in the home from before is not read, nor
176
- `writers/`, where a minca from before the numbered lock keeps its keys, so
177
- the two never sign with one key. `minca agent`
178
- and `minca mcp` keep each session's key and stores together in a slot of
179
- `<home>/agent`, beside the data folder the sessions share
162
+ A data folder, or pod, is the whole account in a folder. Copy it to a USB
163
+ stick and you have all of it. Put it in iCloud, Dropbox or Syncthing, and
164
+ several devices converge with no relay, since records only ever add up.
165
+
166
+ `folderStores(directory, { basePath, localKey, local })` gives a node its
167
+ stores for one:
168
+
169
+ - The `registry` lives in the folder, one file per entry, sealed under
170
+ `localKey`.
171
+ - Each space lives in the folder as segments, under `<basePath>/spaces/`.
172
+ The node mirrors to them ([mirrors](#mirrors)). It pulls when a space opens
173
+ and every `watchIntervalMs` (default 2 s, `0` turns it off), and pushes 1 s
174
+ after a change and on close.
175
+ - Everything else goes in `local`, this device's own stores.
176
+
177
+ Each device adds a folder of segments per space, about one file per 256 KiB
178
+ of records. It compacts them once it has 32, and twice what the last
179
+ compaction left.
180
+
181
+ ### This device's stores
182
+
183
+ `local` is where:
184
+
185
+ - **In the browser**, IndexedDB under `minca-pod:<dataPath>`, by default.
186
+ - **The CLI** keeps it outside the folder, since the folder may be one a
187
+ browser or a sync service also uses. It is `$MINCA_LOCAL`, or
188
+ `~/.local/state/minca`, then `<first 16 hex of SHA-256 of the home's
189
+ path>/<account id>/stores`. A
190
+ command and the daemon share it, and the daemon finds the command's writes
191
+ (`reconcileFolder`). Writer keys are beside it in `writer-keys/`. Older
192
+ slots left in the home, and `writers/`, are not read, so two versions of
193
+ minca never sign with one key. `minca agent` and the commands an agent runs
194
+ keep each session's key and stores in a slot of `<home>/agent`
180
195
  ([actions](actions.md#the-agent-on-a-computer)).
181
- - **Elsewhere** it must be given: `folderStores` refuses to start without
182
- one, since a lasting key's places kept in memory would be handed out again
183
- after a restart.
184
-
185
- A space kept in an older layout is not read
186
- ([spec 05 §14.8](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md));
187
- peers bring it back.
188
-
189
- Getting the folder:
190
-
191
- - **In the browser** the folder comes from `showDirectoryPicker` with
192
- `id: "minca-pod"`, `mode: "readwrite"`, `startIn: "documents"`, which needs a
193
- user gesture. The handle is remembered in IndexedDB (above); on a later visit
194
- `readwrite` permission is queried, and requested again only from a gesture.
195
- Browsers without the API (Firefox, Safari, mobile) keep data in IndexedDB
196
- instead. The File System Access API replaces a file atomically on `close()`.
197
- - **The CLI** (Node and Bun) writes `<name>.<uuid>.tmp` with mode `0600` and
198
- renames it into place. Directories are created with mode `0700`.
199
-
200
- Two folders holding the same account (a copy, a restored backup) share this
201
- device's local stores, which are named by the account's id: a version known
202
- to be in one is not pushed to the other. Open one copy at a time, or let the
203
- other fill in from peers.
204
-
205
- A folder is the account: copy it to a USB stick and it is all of it, or put
206
- it in iCloud, Dropbox or Syncthing and several devices converge with no relay,
207
- since the set of versions only grows. Being readable by several origins is the
208
- point, which is why OPFS (origin-private) is not used for it. Each device
209
- adds a folder of segments per space, about one file per 256 KiB of records,
210
- compacted once it has 32 and twice what the last compaction left.
211
-
212
- _Source: `packages/core/src/node/stores.ts` (`folderStores`),
213
- `packages/core/src/storage/blob/directory.ts`,
214
- `packages/core/src/storage/directory-access.ts`,
215
- `packages/cli/src/fs-directory.ts`, `packages/cli/src/home.ts` (`unlock`),
216
- `packages/core/src/node/space-mirrors.ts` (`openMirrors`, `watch`). Tests: `packages/core/tests/folder-adapter.test.ts`,
217
- `packages/cli/tests/path-safety.test.ts`._
196
+ - **Elsewhere** you must pass one. `folderStores` refuses to start without
197
+ it, since a key's state kept only in memory would be reused after a
198
+ restart.
218
199
 
219
- ## Blob stores
200
+ Two folders holding the same account, such as a copy or a restored backup,
201
+ share this device's local stores. So a version known to be in one is not
202
+ pushed to the other. Open one copy at a time, or let the other fill in from
203
+ peers.
204
+
205
+ ### Getting the folder
220
206
 
221
- A blob store keeps bytes by name and is assumed slow, eventually consistent,
222
- and without atomic operations (`BlobStore`):
207
+ - **In the browser** it comes from `showDirectoryPicker` with
208
+ `id: "minca-pod"`, `mode: "readwrite"` and `startIn: "documents"`, which
209
+ needs a click. The handle is remembered in IndexedDB. On a later visit the
210
+ permission is checked, and asked for again only from a click. Firefox,
211
+ Safari and mobile browsers don't have the API, and keep data in IndexedDB
212
+ instead. A file is replaced atomically on `close()`.
213
+ - **The CLI**, in Node and Bun, writes `<name>.<uuid>.tmp` with mode `0600`
214
+ and renames it into place. Directories get mode `0700`.
223
215
 
224
- | Method | Contract |
225
- | -------------------------- | --------------------------------------------------------------------- |
226
- | `get(key)` | bytes, or null when absent |
227
- | `put(key, bytes)` | create or replace |
228
- | `delete(key)` | remove; absent is fine |
229
- | `list(prefix)` | every key under `prefix`, any order |
230
- | `changes?(prefix, cursor)` | optional: keys added or removed since `cursor`; no mirror uses it yet |
216
+ A folder is meant to be read from several sites, so it can't be OPFS, which
217
+ belongs to one.
231
218
 
232
- Drivers:
219
+ A space kept in an older layout is not read. Peers bring it back. The
220
+ folder's layout, and how several writers converge, are in
221
+ [spec 05 §14](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md).
222
+
223
+ ## Blob stores
224
+
225
+ A blob store keeps bytes by name. It is assumed slow, eventually consistent
226
+ and without atomic operations. A
227
+ [`BlobStore`](https://minca.dev/api/core/interfaces/BlobStore/) only gets, puts, deletes and
228
+ lists keys under a prefix.
229
+
230
+ The library has three:
233
231
 
234
232
  - **Memory**, for tests.
235
- - **Directory**, over a directory handle or a directory on disk
236
- (`createDirectoryBlobStore`): each `/`-separated part of a key a
237
- directory, the last a file, every part named as a data folder names its
238
- files; a directory is removed with the last key in it. What a data folder
239
- keeps its spaces in.
240
- - **S3-compatible** (R2, B2, MinIO, Wasabi, AWS): path-style URLs
241
- `<endpoint>/<bucket>/<prefix/><key>`, each `/`-separated part
242
- `encodeURIComponent`-ed; SigV4-signed (`aws4fetch`), region default
243
- `auto`; `list` is `ListObjectsV2` (`list-type=2`) following continuation
244
- tokens; `get` 404 is null; `delete` 404 is success; 429 and 5xx are retried
245
- up to 5 attempts, waiting `Retry-After` seconds if given, else
246
- `min(200·2^attempt, 5000) ms` times a random factor in [0.75, 1.25).
247
-
248
- _Source: `packages/core/src/storage/blob-store.ts`,
249
- `packages/core/src/storage/blob/memory.ts`,
250
- `packages/core/src/storage/blob/directory.ts`,
251
- `packages/core/src/storage/blob/s3.ts`. Tests:
252
- `packages/core/tests/mirror.test.ts`._
233
+ - **Directory** (`createDirectoryBlobStore`), over a folder handle or a
234
+ folder on disk. Each `/`-separated part of a key is a folder, and the last
235
+ a file, named as a data folder names its files. A folder goes with the
236
+ last key in it. This is what a data folder keeps its spaces in.
237
+ - **S3-compatible**: R2, B2, MinIO, Wasabi and AWS.
238
+ - Path-style URLs, `<endpoint>/<bucket>/<prefix/><key>`, each part
239
+ `encodeURIComponent`-ed.
240
+ - Signed with SigV4 (`aws4fetch`). Region defaults to `auto`.
241
+ - `list` is `ListObjectsV2` (`list-type=2`), following continuation
242
+ tokens.
243
+ - A 404 on `get` is null, and on `delete` is success.
244
+ - 429 and 5xx are tried up to 5 times. It waits `Retry-After` seconds if
245
+ given, else `min(200·2^attempt, 5000)` ms times a random factor in
246
+ [0.75, 1.25).
253
247
 
254
248
  ## Mirrors
255
249
 
256
- A mirror's segments, where a store writer may write and how it pulls and compacts
257
- are protocol ([spec 05 §16](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md)).
258
- Today data folders, carriers and hosts mirror.
250
+ A mirror copies a space into a blob store, so devices that are never online
251
+ together can still meet. Today data folders, carriers and hosts mirror.
259
252
 
260
- A store writer keeps its state in the store `mirrors/<space id>/<n>`, where `n` is
261
- the mirror's index:
253
+ A store writer pushes 5 s after the first change, 1 s for a data folder, and
254
+ on close. It packs what the store doesn't have yet into segments of about
255
+ 256 KiB. Once it has 32 or more segments of its own, and twice as many as its
256
+ last compaction left, it compacts them, keeping only what its store holds.
262
257
 
263
- | Key | Value |
264
- | --------------------------- | ---------------------------------------------- |
265
- | `mirror:writer` | the writer id |
266
- | `mirror:counter` | the segment counter, decimal |
267
- | `mirror:read:<segment key>` | `1`, for each segment read or written |
268
- | `mirror:known:<version id>` | `1`, for each version known to be in the store |
258
+ It keeps its state in the store `mirrors/<space id>/<n>`, where `n` is the
259
+ mirror's index:
269
260
 
270
- "Known" is everything this store writer uploaded **or read**, so nobody re-uploads
271
- everyone else's versions.
261
+ | Key | Value |
262
+ | --------------------------- | --------------------------------------- |
263
+ | `mirror:writer` | The writer id |
264
+ | `mirror:counter` | The segment counter, in decimal |
265
+ | `mirror:read:<segment key>` | `1`, for each segment read or written |
266
+ | `mirror:known:<version id>` | `1`, for each version known to be there |
272
267
 
273
- A store writer pushes 5 s after the first change (1 s for a data folder) and
274
- on close, packing what is not yet known into segments of about 256 KiB (a
275
- segment is closed once it reaches that size). It reads the `mirror:known:`
276
- entries once, when it starts. Once it has 32 or more segments of its own,
277
- and twice as many as its last compaction left, it compacts them, keeping
278
- only what its store holds.
268
+ "Known" is everything it uploaded or read, so nobody uploads everyone
269
+ else's versions again. It reads the `mirror:known:` entries once, when it
270
+ starts.
279
271
 
280
- _Source: `packages/core/src/storage/mirror.ts`,
281
- `packages/core/src/node/space-mirrors.ts` (`openMirrors`),
282
- `packages/core/src/node/host.ts`. Tests: `packages/core/tests/mirror.test.ts`._
272
+ Segments, who may write where, pulling and compacting are protocol, in
273
+ [spec 05 §16](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md).
283
274
 
284
275
  ### Planned: mirrors in your own storage
285
276
 
286
- > **Planned.** Mirrors on any node, in storage the person already has. What
287
- > a mirror holds, and absorbing a store writer that went quiet, are protocol and
288
- > planned in [spec 05 §16.5](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md).
289
-
290
- - **Any node, several mirrors.** A node will take a list of blob stores
291
- (`NodeConfig.mirrors`), one per space each, since a person may have their
292
- own Dropbox and a host's bucket at once. It pulls when a space opens and on
293
- each change notice, and flushes after local writes and on close. Two devices
294
- never online together then meet through the store.
295
- - **Change feeds.** Where the service has one, pull uses `BlobStore.changes`
296
- instead of listing: Dropbox `list_folder/continue` with long polling, Drive
297
- `changes.list`. S3 and directories list.
298
- - **More drivers**, each passing one contract suite (round-trips, absent is
299
- null, a second delete is fine, `list` over several pages, keys with slashes
300
- and unicode, a 1 MB blob), with shared backoff on 429 and 5xx that honours
301
- `Retry-After`, never retries a delete that returned 404, and never logs a
302
- token or signed URL:
303
- - _Google Drive_: everything in one app-created folder, `drive.file` scope
304
- only (the app sees only its own files). Drive addresses files by id, not
305
- name, so the driver keeps a name → file id map from one `files.list`,
306
- updated on every `put` and invalidated on 404. Drive allows duplicate
307
- names, so a `put` updates an existing file id rather than creating
308
- another. Tokens come from a callback (`getAccessToken`); service accounts
309
- do not work (they have no storage of their own).
310
- - _Dropbox_ and _OneDrive_, in their app folders, the narrowest grant each
311
- offers. Dropbox first.
312
- - Not iCloud: it has no usable web API; iCloud users keep a data folder
313
- instead.
314
- - **Budget.** A store like Drive allows about 1,000 requests per 100 seconds.
315
- The segment, never the single version, is the unit fetched, and syncing
316
- 1,000 records must fit that budget.
277
+ > **Planned:** mirrors on any node, in storage the person already has. The
278
+ > protocol side is planned in
279
+ > [spec 05 §16.5](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md).
280
+
281
+ - **Any node, several mirrors.** A node takes a list of blob stores
282
+ (`NodeConfig.mirrors`), such as the person's Dropbox and a host's bucket at
283
+ once. It pulls when a space opens and on each change, and pushes after
284
+ writes and on close.
285
+ - **Change feeds.** Where the service has one, pulling uses
286
+ `BlobStore.changes` instead of listing: Dropbox `list_folder/continue`,
287
+ Drive `changes.list`.
288
+ - **More drivers**, each passing one shared test suite, with shared backoff
289
+ on 429 and 5xx. None ever logs a token or signed URL.
290
+ - _Google Drive_, in one folder the app made, with `drive.file` scope only.
291
+ Drive finds files by id, not name, so the driver keeps a map from names
292
+ to ids. Service accounts don't work: they have no storage of their own.
293
+ - _Dropbox_ and _OneDrive_, in their app folders. Dropbox first.
294
+ - Not iCloud, which has no usable web API. iCloud users keep a data folder.
295
+ - **Budget.** Drive allows about 1,000 requests per 100 seconds. Segments,
296
+ never single versions, are fetched, and syncing 1,000 records must fit.
317
297
 
318
- ## Files
298
+ ## Storage adapters
319
299
 
320
- What peers must agree on (the `files` member, chunks and the id, RFC 8188
321
- records, the three messages, keeping a file with its body) is in
322
- [spec 05 §16.6](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md).
323
- How this library does the rest:
324
-
325
- - **Adding.** `records.addFile(space, bytes, { mime, name? })` seals the
326
- bytes under a fresh key and salt, keeps them, and gives back a `FileRef`:
327
- `{ hash, size, key, salt, mime, name? }`. Put it anywhere in a body;
328
- writing that body names every reference in it on the version
329
- (`fileRefsIn`, at most 16), so the envelope and the body always agree.
330
- - **Reading.** `records.readFile(space, ref)` gives the bytes: held here, or
331
- fetched from connected peers, one after another, then from mirrors,
332
- checking every chunk. When none has them it waits, asking everyone again
333
- whenever records change or a peer connects, and every 15 s, and resolves
334
- once one does. It rejects for a file no version this node keeps whole
335
- names, one whose id is here whole yet won't open as named, and when the
336
- space closes. For a picture, `URL.createObjectURL(new Blob([bytes], {
337
- type: ref.mime }))` shows it.
338
- - **Where chunks live.** Entries in the space's own store: `f/<id>/list`,
339
- `f/<id>/<n>`, `f/<id>/own` with the time a file was added here (0 once a
340
- version names it), and `f/<id>/used` with when it was last fetched or
341
- read. A data folder keeps these in this device's own stores, not the
342
- folder.
343
- - **What a node fetches.** Files follow records. A node fetches every file
344
- named by a version it keeps whole about 2 s after its records change,
345
- while the writer is likely still connected. A file this sweep found
346
- nobody had is not asked of everyone again for 10 minutes, but is asked at
347
- once of each peer that connects meanwhile.
348
- - **A device's limits.** A node limits the files it did not add, in every
349
- space: it fetches unread only files of up to 5 MiB, so photos and
350
- documents come ahead and video when played. Past 256 MiB of them in a
351
- space, counting each at the size its list proves, or every chunk held of
352
- one never whole, it drops those fetched or opened longest ago, down to
353
- 90%, at the start of each pass and as it fetches. One never whole goes
354
- first. A file added here is never dropped, nor one being fetched or read;
355
- one larger than the limit is dropped on the pass after it is read, and
356
- fetched again at the next read. A dropped file that was whole keeps its
357
- `used` entry, so it is not fetched unread again while a version names it;
358
- reading it fetches it. The limit is per space, so a device in many spaces
359
- holds more. A node with `fetchFiles: 'all'`, a carrier or host, fetches
360
- and keeps every file.
361
- - **Serving.** A `want-file` is answered with at most 4 chunks; the asker
362
- asks again for the rest, so a large file never floods a connection.
363
- - **Keeping.** About 2 s after records change, files no version kept whole
364
- names are dropped (files of versions that don't stand are not kept),
365
- except one added here in the last 24 hours and not yet named.
366
- - **Mirrors.** The same pass uploads each named file this node holds whole,
367
- with its chunk list, to `files/<space>/<store writer>/`, and deletes this
368
- writer's copies of files no longer named.
369
-
370
- _Source: `packages/core/src/files/space-files.ts`, `packages/core/src/files/file-id.ts`,
371
- `packages/core/src/files/seal.ts`, `packages/core/src/node/space-runtime.ts` (`filesChanged`).
372
- Tests: `packages/core/tests/files.test.ts`._
300
+ A store runs over a [`StorageAdapter`](https://minca.dev/api/core/interfaces/StorageAdapter/).
301
+ Write one to keep a node's data somewhere new. Every method is async. It keeps
302
+ entries, string keys to bytes, and version bodies by id. A few methods are
303
+ optional.
373
304
 
374
- ### Planned: video and pictures
305
+ The optional methods only save round trips. A store gives the same answers
306
+ without them, using `list` and `get`, `getVersion`, or `putVersion`, `batch`
307
+ and `deleteVersion` instead.
308
+
309
+ With `commit`, placing a version is one atomic write. Without it, versions go
310
+ first, then the entries in one batch, then the deletes. So no entry ever
311
+ names a version that isn't there.
312
+
313
+ The library's adapters are IndexedDB ([above](#indexeddb)), a directory
314
+ (`directoryStores`, for a data folder's registry and a node's stores on
315
+ disk), a sealing wrapper, and one in memory for tests.
316
+
317
+ ## The store in memory
318
+
319
+ A store reads its entries once, then keeps what it needs in memory: each
320
+ collection's set of records, and the entries it read or wrote. It also keeps
321
+ up to 50,000 version bodies, letting the oldest go first. A body never
322
+ changes, so this is always safe. An entry read while a change was landing is
323
+ not kept.
324
+
325
+ When another writer changes the same store, the node reads all of it again
326
+ (`invalidate`):
327
+
328
+ - **Tabs in one browser.** Where shared workers work, the apps in this
329
+ repository run one node for all of a site's tabs in a `SharedWorker`, one
330
+ per account and key. Then there's nothing to tell.
331
+ - **Elsewhere** each tab runs its node in a worker of its own. Tabs tell each other through a
332
+ `BroadcastChannel` named `minca-node:<root DID>:<space id>`. Chrome on
333
+ Android has no shared workers, and WebKit drops a message to one that
334
+ holds a `CryptoKey`.
335
+ - **Writers on a shared folder** are found by reading it again
336
+ ([data folders](#data-folders)).
337
+
338
+ The store's fingerprint, in `spaces.status`, is the Negentropy fingerprint
339
+ of every collection together, as hex. Two nodes with the same fingerprint
340
+ hold the same data.
341
+
342
+ ## Pacing sync
375
343
 
376
- > **Planned.** Not normative.
377
-
378
- - **Video.** `readFile` fetches a whole file. Playing before it is whole
379
- needs a URL a service worker serves, answering the browser's range
380
- requests from only the chunks read, opened one at a time (`fileOpener`).
381
- MP4 with its index first, or WebM; no transcoding; a poster is a second,
382
- small image file.
383
- - **Several peers at once.** Chunks asked of several peers in parallel, and
384
- of a slow one again elsewhere.
385
- - **Mirrors as sources by range.** `BlobStore.getRange`: a `Range` header on
386
- S3, `File.slice` in a data folder, so a chunk is read without the whole.
387
- - **Pictures.** `std.image` gains `width`, `height` and a tiny WebP
388
- placeholder as a `data:` URL, shown before a chunk arrives; and
389
- `std.profile`'s `avatar`, resized in the browser before it is added.
344
+ The spec says which sync messages go, and when a peer must answer. How often
345
+ a node speaks, and how much it asks for at once, is its own choice:
346
+
347
+ | What | Value | Why |
348
+ | --------------------- | ---------------------------------- | -------------------------------------------------------------------------------------- |
349
+ | Heartbeat | a `hello` to every peer every 30 s | Finds what a lost message missed |
350
+ | Hello after taking in | about 100 ms | So a new version passed along doesn't wait a heartbeat at every hop |
351
+ | `want`s in flight | 4 per peer (`MAX_WANTS_IN_FLIGHT`) | When one is answered, the next goes |
352
+ | Rounds per session | 64 | Counted by the side that started. Past 64 the session is given up |
353
+ | Stale after | 30 s | A silent session, or an unanswered `want`, is dropped and found again later |
354
+ | Versions waiting | 1,000 | Versions held back for later wait in memory. Past 1,000 the oldest go |
355
+ | Refusals remembered | 10,000 | A refused version isn't asked of that peer again. Past 10,000 the oldest are forgotten |
356
+
357
+ The node asks first for the space's own records (`sys.*`), and for earlier
358
+ versions a waiting version needs. Everything else waits its turn. An id
359
+ already asked for is not asked again.
360
+
361
+ A session with a peer that is running and has heard something in the last
362
+ 30 s isn't started again. It runs once more when it ends.
363
+
364
+ Each space reports `synced` and `level` with each peer, which
365
+ [holding part of a space](node.md#holding-part-of-a-space) uses. The page is
366
+ told records changed once per message taken in, not once per version.