@minca/core 0.1.8 → 0.1.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/files/file-id.d.ts +51 -0
- package/dist/files/file-id.d.ts.map +1 -0
- package/dist/files/file-id.js +97 -0
- package/dist/files/file-id.js.map +1 -0
- package/dist/files/seal.d.ts +20 -0
- package/dist/files/seal.d.ts.map +1 -0
- package/dist/files/seal.js +77 -0
- package/dist/files/seal.js.map +1 -0
- package/dist/files/space-files.d.ts +51 -0
- package/dist/files/space-files.d.ts.map +1 -0
- package/dist/files/space-files.js +476 -0
- package/dist/files/space-files.js.map +1 -0
- package/dist/index.d.ts +5 -4
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +3 -2
- package/dist/index.js.map +1 -1
- package/dist/node/api.d.ts +5 -1
- package/dist/node/api.d.ts.map +1 -1
- package/dist/node/api.js +10 -2
- package/dist/node/api.js.map +1 -1
- package/dist/node/carrier.d.ts +17 -2
- package/dist/node/carrier.d.ts.map +1 -1
- package/dist/node/carrier.js +26 -4
- package/dist/node/carrier.js.map +1 -1
- package/dist/node/copy.d.ts +0 -4
- package/dist/node/copy.d.ts.map +1 -1
- package/dist/node/copy.js +26 -0
- package/dist/node/copy.js.map +1 -1
- package/dist/node/host.d.ts +66 -1
- package/dist/node/host.d.ts.map +1 -1
- package/dist/node/host.js +571 -5
- package/dist/node/host.js.map +1 -1
- package/dist/node/index.d.ts +2 -2
- package/dist/node/index.d.ts.map +1 -1
- package/dist/node/index.js +1 -1
- package/dist/node/index.js.map +1 -1
- package/dist/node/node.d.ts.map +1 -1
- package/dist/node/node.js +100 -18
- package/dist/node/node.js.map +1 -1
- package/dist/node/space-runtime.d.ts +9 -0
- package/dist/node/space-runtime.d.ts.map +1 -1
- package/dist/node/space-runtime.js +125 -16
- package/dist/node/space-runtime.js.map +1 -1
- package/dist/node/stores.d.ts +44 -8
- package/dist/node/stores.d.ts.map +1 -1
- package/dist/node/stores.js +44 -7
- package/dist/node/stores.js.map +1 -1
- package/dist/node/types.d.ts +65 -6
- package/dist/node/types.d.ts.map +1 -1
- package/dist/node/usage.d.ts +54 -0
- package/dist/node/usage.d.ts.map +1 -0
- package/dist/node/usage.js +201 -0
- package/dist/node/usage.js.map +1 -0
- package/dist/schema/version.d.ts +5 -0
- package/dist/schema/version.d.ts.map +1 -1
- package/dist/schema/version.js +1 -0
- package/dist/schema/version.js.map +1 -1
- package/dist/schemas/fragments.d.ts +23 -2
- package/dist/schemas/fragments.d.ts.map +1 -1
- package/dist/schemas/fragments.js +5 -2
- package/dist/schemas/fragments.js.map +1 -1
- package/dist/schemas/library/annotations.d.ts +24 -0
- package/dist/schemas/library/annotations.d.ts.map +1 -1
- package/dist/schemas/library/community.d.ts +24 -0
- package/dist/schemas/library/community.d.ts.map +1 -1
- package/dist/schemas/library/life.d.ts +72 -0
- package/dist/schemas/library/life.d.ts.map +1 -1
- package/dist/schemas/library/media.d.ts +144 -0
- package/dist/schemas/library/media.d.ts.map +1 -1
- package/dist/schemas/library/money.d.ts +24 -0
- package/dist/schemas/library/money.d.ts.map +1 -1
- package/dist/schemas/library/publishing.d.ts +96 -0
- package/dist/schemas/library/publishing.d.ts.map +1 -1
- package/dist/schemas/library/social.d.ts +72 -0
- package/dist/schemas/library/social.d.ts.map +1 -1
- package/dist/session/auth.d.ts.map +1 -1
- package/dist/session/auth.js +8 -1
- package/dist/session/auth.js.map +1 -1
- package/dist/session/hosting.d.ts +73 -28
- package/dist/session/hosting.d.ts.map +1 -1
- package/dist/session/hosting.js +85 -5
- package/dist/session/hosting.js.map +1 -1
- package/dist/session/places.d.ts +2 -0
- package/dist/session/places.d.ts.map +1 -1
- package/dist/session/places.js +8 -2
- package/dist/session/places.js.map +1 -1
- package/dist/space/account-registry.d.ts +13 -0
- package/dist/space/account-registry.d.ts.map +1 -1
- package/dist/space/account-registry.js +13 -0
- package/dist/space/account-registry.js.map +1 -1
- package/dist/storage/blob/directory.d.ts +14 -0
- package/dist/storage/blob/directory.d.ts.map +1 -0
- package/dist/storage/blob/directory.js +80 -0
- package/dist/storage/blob/directory.js.map +1 -0
- package/dist/storage/folder-adapter.d.ts +22 -0
- package/dist/storage/folder-adapter.d.ts.map +1 -1
- package/dist/storage/folder-adapter.js +5 -5
- package/dist/storage/folder-adapter.js.map +1 -1
- package/dist/storage/mirror.d.ts +9 -2
- package/dist/storage/mirror.d.ts.map +1 -1
- package/dist/storage/mirror.js +60 -35
- package/dist/storage/mirror.js.map +1 -1
- package/dist/storage/segment.d.ts +7 -2
- package/dist/storage/segment.d.ts.map +1 -1
- package/dist/storage/segment.js +25 -3
- package/dist/storage/segment.js.map +1 -1
- package/dist/sync/sync-engine.d.ts +2 -0
- package/dist/sync/sync-engine.d.ts.map +1 -1
- package/dist/sync/sync-engine.js +5 -0
- package/dist/sync/sync-engine.js.map +1 -1
- package/dist/sync/sync-messages.d.ts +24 -0
- package/dist/sync/sync-messages.d.ts.map +1 -1
- package/dist/sync/sync-messages.js.map +1 -1
- package/docs/node.md +39 -22
- package/docs/spaces.md +18 -10
- package/docs/standard-library.md +16 -20
- package/docs/storage.md +120 -15
- package/package.json +1 -1
package/docs/node.md
CHANGED
|
@@ -27,7 +27,7 @@ conveniences.
|
|
|
27
27
|
| `sessionTtlSeconds` | number, optional | Lifetime of each session note. Default `3600`. |
|
|
28
28
|
| `sessionKey` | `CryptoKeyPair`, optional | Sign with this key instead of a fresh one. Used when the signer's note names a specific key (an app's key, an agent's key, [spec 06 §2](https://github.com/leifriksheim/minca/blob/main/spec/06-nodes-and-sessions.md), [spec 06 §3](https://github.com/leifriksheim/minca/blob/main/spec/06-nodes-and-sessions.md)). |
|
|
29
29
|
| `writerKey` | `WriterKeySource`, optional | Keep a signing key across runs, so this device keeps one log in each space ([writer keys](#writer-keys)). Ignored with `sessionKey`. |
|
|
30
|
-
| `watchIntervalMs` | number, optional | How often to look for writes another process made
|
|
30
|
+
| `watchIntervalMs` | number, optional | How often to look for writes another app, device or process made: a data folder's segments, a store file another process shares. `0` disables. Default `2000`. _Implementation detail._ |
|
|
31
31
|
| `cache` | `CacheConfig`, optional | Hold only the collections used, in spaces that name a keeper ([holding part of a space](#holding-part-of-a-space)). |
|
|
32
32
|
| `mailbox` | `MailboxClient`, optional | How doors reach relays' mailboxes ([07](https://github.com/leifriksheim/minca/blob/main/spec/07-doors.md)). Default: a WebSocket to each relay. |
|
|
33
33
|
|
|
@@ -165,9 +165,13 @@ What a path means is the factory's choice:
|
|
|
165
165
|
- `indexedDBStores(prefix)` — one IndexedDB database per path, named
|
|
166
166
|
`<prefix>:<path with / replaced by :>`. Never sealed: the browser profile
|
|
167
167
|
already guards it.
|
|
168
|
-
- `folderStores(directory, { basePath?, localKey? })` — a
|
|
169
|
-
|
|
170
|
-
when one
|
|
168
|
+
- `folderStores(directory, { basePath?, localKey?, local? })` — a data folder
|
|
169
|
+
([sync and storage](storage.md#data-folders)): the `registry` in the folder
|
|
170
|
+
under `basePath`, sealed under `localKey` (`deriveLocalKey(seed)`) when one
|
|
171
|
+
is given; every other path in `local`; and each space mirrored to the
|
|
172
|
+
folder's segments, which the factory names as `folder`.
|
|
173
|
+
- `directoryStores(directory, { localKey? })` — a directory on this device,
|
|
174
|
+
one file per entry and per version; the `registry` sealed as above.
|
|
171
175
|
|
|
172
176
|
Every version a node signs in a space takes the next place in its key's line
|
|
173
177
|
for that collection there ([spec 02 §4.10](https://github.com/leifriksheim/minca/blob/main/spec/02-records.md)).
|
|
@@ -224,7 +228,7 @@ node makes a key for this run as it would without a source.
|
|
|
224
228
|
write fails until the page reloads, since signing on would fork the line
|
|
225
229
|
and withdraw what the key wrote. A slot whose connection the browser
|
|
226
230
|
closed opens again only while it still holds the key.
|
|
227
|
-
- The CLI keeps them in
|
|
231
|
+
- The CLI keeps them outside its home, in `<local>/writers/slot-<n>/` (`localPath` in `packages/cli/src/home.ts`, [sync and storage](storage.md#data-folders)): the key as
|
|
228
232
|
JWK in an envelope (spec 01 §9.1) under the account's local key, with label
|
|
229
233
|
`minca/writer-key/v1` and bound `{ did: <account DID>, slot: <n> }`
|
|
230
234
|
(`key.sealed`, mode 600), so a copy of the files can't sign, and a key file
|
|
@@ -242,7 +246,8 @@ by default. These names are _implementation details_ ([sync and storage](storage
|
|
|
242
246
|
a pod's layout is in
|
|
243
247
|
[05](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md).
|
|
244
248
|
A node also opens `mirrors/<spaceId>/<n>` for a mirror's writer state
|
|
245
|
-
([mirrors](storage.md#mirrors)),
|
|
249
|
+
([mirrors](storage.md#mirrors)), `folder/<spaceId>` for a data folder's, and
|
|
250
|
+
a host `host` for its subscriptions.
|
|
246
251
|
|
|
247
252
|
`copyAccountData({ from, to, did, accountKey? })` copies every space an
|
|
248
253
|
account holds, with its key and records, from one set of stores to another —
|
|
@@ -586,7 +591,8 @@ successors, mine, heads, key, relays, keepers, copies, forks, flagged }`
|
|
|
586
591
|
with at most one more queued.
|
|
587
592
|
- `node.account` — `profile()`, `setName(name)` (writes `sys.profile`, key
|
|
588
593
|
`profile`, in the account registry, then republishes the profile in every
|
|
589
|
-
open space), `
|
|
594
|
+
open space), `setAvatar({ bytes, mime } | null, { name? })` and `picture()`
|
|
595
|
+
(the account's picture, likewise; see [spaces](spaces.md)), `revoke(key, { waitMs? })` (one of the account's keys, by DID, in the account registry). All need the account
|
|
590
596
|
key.
|
|
591
597
|
- `spaces.revoke(space, key, { waitMs? })` and `account.revoke` resolve once the revoke is written; with
|
|
592
598
|
`waitMs`, once a peer says it has taken it in, or that long has passed (`true` when one did). A node
|
|
@@ -849,18 +855,18 @@ holds. Nothing shows these yet, the browser extension included.
|
|
|
849
855
|
|
|
850
856
|
**Hosting.** `node.hosting`:
|
|
851
857
|
|
|
852
|
-
| Call | Does
|
|
853
|
-
| -------------------------------------------------------------------------------------- |
|
|
854
|
-
| `use(url)` | Reads the host's description, makes the subscription key and writes the `sys.hosting` record.
|
|
855
|
-
| `list()` | Asks each host for its status and hands over the carry space when paid and not carrying, making the carrier first if needed (named after the host's address). At most once a minute per host; a second look waits for a handover in flight.
|
|
856
|
-
| `pay(url,
|
|
857
|
-
| `
|
|
858
|
-
| `
|
|
859
|
-
| `
|
|
860
|
-
| `
|
|
861
|
-
| `payForSpace(id, url, { amount, method, monthly? })`, `remindForSpace(id, url, email)` | Adds to a space's fund at a host it names (dollars, by `checkout` or `request`, monthly by card), and asks for reminders before it runs out. Anyone in it may. `space(id)` gives each host's `fund` offer, and in its status the `balance`, `daily` spend and `bots`.
|
|
862
|
-
| `startBot(id, url, { name, role? })` | Asks a host the space names to run a bot there ([spec 08 §3](https://github.com/leifriksheim/minca/blob/main/spec/08-hosting.md)): makes an invite for `role` (by default as `spaces.invite` picks one) and hands it over. The host holds the bot's keys, so an app says so first.
|
|
863
|
-
| `stopForSpace(id, url)` | Stops a space using a host it names: deletes its `std.host` record, removes the host's bots and changes the key (below).
|
|
858
|
+
| Call | Does |
|
|
859
|
+
| -------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
860
|
+
| `use(url)` | Reads the host's description, makes the subscription key and writes the `sys.hosting` record. |
|
|
861
|
+
| `list()` | Asks each host for its status and hands over the carry space when paid and not carrying, making the carrier first if needed (named after the host's address). At most once a minute per host; a second look waits for a handover in flight. |
|
|
862
|
+
| `pay(url, { amount, method, monthly? })` | Adds to the account's balance at the host (dollars, by `checkout` or `request`, monthly by card, as its `HostingView.fund` says): a pay answer, `checkout` (a page to open in a new tab with no opener) or `request` (a payment for a wallet, shown by the app). Call `list()` until the balance moves. |
|
|
863
|
+
| `remind(url, email)` | Asks the host to email a reminder before paid time runs out; it mails a link to confirm first. |
|
|
864
|
+
| `stop(url)` | Sends `DELETE …/carry` (ignoring failure), removes the carrier and deletes the `sys.hosting` record. |
|
|
865
|
+
| `space(id)` | The hosts a space names in `std.host`, each asked how the space's own subscription stands, with how its fund is paid into (below). |
|
|
866
|
+
| `over(id)` | The account's hosts whose last status says the space is over their free line, each `{ url, name, over }` (`over.until`, `over.reasons`), from the receipts in `sys.hosting`: asks no host. An agent may call it for a space it was given. |
|
|
867
|
+
| `payForSpace(id, url, { amount, method, monthly? })`, `remindForSpace(id, url, email)` | Adds to a space's fund at a host it names (dollars, by `checkout` or `request`, monthly by card), and asks for reminders before it runs out. Anyone in it may. `space(id)` gives each host's `fund` offer, and in its status the `balance`, `daily` spend and `bots`. |
|
|
868
|
+
| `startBot(id, url, { name, role? })` | Asks a host the space names to run a bot there ([spec 08 §3](https://github.com/leifriksheim/minca/blob/main/spec/08-hosting.md)): makes an invite for `role` (by default as `spaces.invite` picks one) and hands it over. The host holds the bot's keys, so an app says so first. |
|
|
869
|
+
| `stopForSpace(id, url)` | Stops a space using a host it names: deletes its `std.host` record, removes the host's bots and changes the key (below). |
|
|
864
870
|
|
|
865
871
|
Every device runs `list()` after each reconciliation ([following the account](#following-the-account)).
|
|
866
872
|
A device reaches a host at the socket its description names (`peer`,
|
|
@@ -882,13 +888,24 @@ The reference host (`minca host`, `packages/cli/src/host.ts`) keeps a lapsed
|
|
|
882
888
|
subscription for `graceDays` (default 30) after `paidUntil`, and drops lapsed
|
|
883
889
|
ones in a sweep every hour (`sweepMs`). It measures each carried space's
|
|
884
890
|
folder at most every ten minutes, and each status gives the sum as `bytes`,
|
|
885
|
-
with `quotaBytes` as `quota` (`MINCA_HOST_QUOTA_GB
|
|
886
|
-
|
|
891
|
+
with `quotaBytes` as `quota` (`MINCA_HOST_QUOTA_GB`, none unless set, at a
|
|
892
|
+
host without limits). At its quota, an account's carry space takes no new space
|
|
887
893
|
(`full`, in `createCarryCore`): what it carries stays and keeps syncing, and a
|
|
888
894
|
space that waited is taken once a later measure finds room. All of this is the
|
|
889
895
|
host's own policy.
|
|
890
896
|
|
|
891
|
-
|
|
897
|
+
With `limits` ([spec 08 §1.1](https://github.com/leifriksheim/minca/blob/main/spec/08-hosting.md),
|
|
898
|
+
`MINCA_HOST_FREE_GB`), an account's subscription is active with nothing paid,
|
|
899
|
+
its `plan` `free` until it pays, and its quota by plan; a device hands such a
|
|
900
|
+
host its carry space as it would a free one. A space its own fund pays for
|
|
901
|
+
takes none of its members' room. The host counts each carried space's writers
|
|
902
|
+
as versions arrive (`onArrived`, in `createCarryCore`) and its readers as
|
|
903
|
+
sockets prove themselves (`onPeer`, `packages/cli/src/serve.ts`), and once an
|
|
904
|
+
hour (`evaluate`) holds each to the free line. Past a space's `until`, every
|
|
905
|
+
account's carry space stops carrying it (`skip`, in `createCarryCore`), while
|
|
906
|
+
its own subscription still may. Each change is a line in the host's log.
|
|
907
|
+
|
|
908
|
+
_Source: `packages/core/src/node/node.ts` (`carriers`, `notifications`, `hosting`), `packages/core/src/node/carrier.ts` (`arrived`, `skip`, `onArrived`), `packages/core/src/node/host.ts` (`evaluate`), `packages/core/src/session/hosting.ts`, `packages/cli/src/host.ts`. Tests: `packages/core/tests/carrier.test.ts`, `packages/core/tests/host-limits.test.ts`, `packages/cli/tests/host.test.ts`._
|
|
892
909
|
|
|
893
910
|
### A space paying for itself
|
|
894
911
|
|
package/docs/spaces.md
CHANGED
|
@@ -196,17 +196,29 @@ whose root's `hex40` is not the record key:
|
|
|
196
196
|
empty means no profile), `did` (its root) and `updatedAt` (its `createdAt`);
|
|
197
197
|
- `contactKey` comes from the newest remaining version that carries a valid
|
|
198
198
|
P-256 point, so a newer version written without one does not hide it.
|
|
199
|
+
- `avatar` comes from the newest version only (one without it has taken it
|
|
200
|
+
away): a file reference (`records.addFile`) whose `mime` is an image.
|
|
199
201
|
|
|
200
202
|
**Writing.** Profiles are written with `retain`, so older versions stay. A
|
|
201
203
|
writer that does not hold the contact key must carry forward the
|
|
202
204
|
`contactKey` of its current profile.
|
|
203
205
|
|
|
204
|
-
**The account's name.** The account keeps its name in the
|
|
205
|
-
as `sys.profile` at key `profile` with body
|
|
206
|
-
wins. `node.account.setName` writes
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
206
|
+
**The account's name and picture.** The account keeps its name in the
|
|
207
|
+
account registry, as `sys.profile` at key `profile` with body
|
|
208
|
+
`{ name, avatar? }`; the newest version wins. `node.account.setName` writes
|
|
209
|
+
the name, `setAvatar({ bytes, mime })` the picture (a file in the registry;
|
|
210
|
+
the app resizes it first), and `picture()` reads it back. The node publishes
|
|
211
|
+
both into every space it opens, and again on a change. A space's members
|
|
212
|
+
can't read the registry, so the picture is copied into each space once, as a
|
|
213
|
+
file of that space, and its reference names the registry file it copies
|
|
214
|
+
(`source`), so opening the space again makes no second copy. A picture that
|
|
215
|
+
can't be copied now (its bytes on a device that is away) leaves the space's
|
|
216
|
+
copy as it was, and never holds back the name. Profiles are retained, so a
|
|
217
|
+
picture once shared stays with the members who hold it: taking it away
|
|
218
|
+
shows none from then on, and withdraws nothing. It publishes
|
|
219
|
+
in every space but the registry and the contacts space, and not at all from
|
|
220
|
+
a node whose note lacks `/account/write` (an agent's); a non-member
|
|
221
|
+
publishes nothing
|
|
210
222
|
([spec 06 §1.3](https://github.com/leifriksheim/minca/blob/main/spec/06-nodes-and-sessions.md)).
|
|
211
223
|
|
|
212
224
|
> **Planned:** profiles, round two.
|
|
@@ -215,10 +227,6 @@ non-member publishes nothing
|
|
|
215
227
|
> per space; what is missing is a way to say the name was chosen for this
|
|
216
228
|
> space, so publishing the account name on the next rename does not
|
|
217
229
|
> overwrite it.
|
|
218
|
-
> - **An avatar**, as a reference to an image on the profile. Depends on a
|
|
219
|
-
> way for a record to carry a file, which is not specified yet (the blob
|
|
220
|
-
> stores of [spec 05](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md) hold mirrors, not files records
|
|
221
|
-
> point to).
|
|
222
230
|
> - **Private nicknames for others**: a name you give someone, seen only by
|
|
223
231
|
> you. It belongs in the account's own spaces, not in the shared one;
|
|
224
232
|
> `std.contact`'s `name` already does this for contacts.
|
package/docs/standard-library.md
CHANGED
|
@@ -57,23 +57,24 @@ _Source: `packages/core/src/schemas/blueprints.ts` (`standardCollections`), `pac
|
|
|
57
57
|
out in each definition, built by the library's `fragments` so they are
|
|
58
58
|
identical everywhere:
|
|
59
59
|
|
|
60
|
-
| Fragment | Shape
|
|
61
|
-
| ---------- |
|
|
62
|
-
| `when` | string 10–64: RFC 3339 date-time (`2026-03-01T18:30:00+01:00`), or `YYYY-MM-DD` for a whole day
|
|
63
|
-
| `day` | string of 10: `YYYY-MM-DD`
|
|
64
|
-
| `timeZone` | string 1–64: an IANA zone, `Europe/Oslo`
|
|
65
|
-
| DID | string 1–256: an account DID (built by `person`)
|
|
66
|
-
| `money` | `{ amount, currency }`, both required: `amount` a decimal string 1–32 (`"12.50"`), never a number; `currency` ISO 4217, 3 characters
|
|
67
|
-
| `address` | `{ street, locality, region, postcode, country }`, each optional; `country` ISO 3166-1 alpha-2
|
|
68
|
-
| `place` | `{ name, address, lat, lon, cells }` (built by `placeRef`), each optional; `lat` −90–90, `lon` −180–180, WGS 84; `cells` string 1–12[] (≤ 8), the geohash cells of `lat` and `lon` (see Places and areas)
|
|
69
|
-
| `blob` | `{ hash, size, mime, name? }`: `
|
|
70
|
-
| `image` | `{ blob, alt? }`: `alt` ≤ 2000, what it shows
|
|
71
|
-
| `position` | string 1–200 that sorts where a record goes (Positions, below)
|
|
60
|
+
| Fragment | Shape |
|
|
61
|
+
| ---------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
62
|
+
| `when` | string 10–64: RFC 3339 date-time (`2026-03-01T18:30:00+01:00`), or `YYYY-MM-DD` for a whole day |
|
|
63
|
+
| `day` | string of 10: `YYYY-MM-DD` |
|
|
64
|
+
| `timeZone` | string 1–64: an IANA zone, `Europe/Oslo` |
|
|
65
|
+
| DID | string 1–256: an account DID (built by `person`) |
|
|
66
|
+
| `money` | `{ amount, currency }`, both required: `amount` a decimal string 1–32 (`"12.50"`), never a number; `currency` ISO 4217, 3 characters |
|
|
67
|
+
| `address` | `{ street, locality, region, postcode, country }`, each optional; `country` ISO 3166-1 alpha-2 |
|
|
68
|
+
| `place` | `{ name, address, lat, lon, cells }` (built by `placeRef`), each optional; `lat` −90–90, `lon` −180–180, WGS 84; `cells` string 1–12[] (≤ 8), the geohash cells of `lat` and `lon` (see Places and areas) |
|
|
69
|
+
| `blob` | `{ hash, size, mime, name?, key?, salt? }`: what `records.addFile` gives: `hash` the file's id (64 hex characters), `size` bytes as stored, `mime` required; `key` and `salt` open it (22 base64url characters each) |
|
|
70
|
+
| `image` | `{ blob, alt? }`: `alt` ≤ 2000, what it shows |
|
|
71
|
+
| `position` | string 1–200 that sorts where a record goes (Positions, below) |
|
|
72
72
|
|
|
73
73
|
Until the dialect has `format` ([spec 02 §6.2](https://github.com/leifriksheim/minca/blob/main/spec/02-records.md)), `when` and `day` are checked only for
|
|
74
|
-
length.
|
|
75
|
-
([
|
|
76
|
-
the
|
|
74
|
+
length. A `blob` is what `records.addFile` gives
|
|
75
|
+
([storage](storage.md#files)): its `hash` is the file's id, and it carries
|
|
76
|
+
the `key` and `salt` that open it
|
|
77
|
+
([spec 05 §16.6](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md)).
|
|
77
78
|
|
|
78
79
|
## The definitions
|
|
79
80
|
|
|
@@ -485,11 +486,6 @@ _Source: `packages/core/src/schemas/geo.ts` (`geohash`, `cellsOf`, `cellsAround`
|
|
|
485
486
|
|
|
486
487
|
> **Planned.** Not normative. Each item names its own issue.
|
|
487
488
|
>
|
|
488
|
-
> - **File bytes** ([spec 05 §16.6](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md),
|
|
489
|
-
> [#37](https://github.com/leifriksheim/minca/issues/37)). The `blob`
|
|
490
|
-
> fragment names bytes by hash, but a space cannot yet store or send them,
|
|
491
|
-
> so `std.file`, `std.photo`, `std.video` and the images elsewhere point at
|
|
492
|
-
> bytes a reader may not be able to fetch.
|
|
493
489
|
> - **References to other spaces** ([spec 02 §5.3](https://github.com/leifriksheim/minca/blob/main/spec/02-records.md),
|
|
494
490
|
> [#38](https://github.com/leifriksheim/minca/issues/38)). `std.follow`'s
|
|
495
491
|
> `space` is a plain space id, and a repost or a list item can point only
|
package/docs/storage.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
# Sync and storage in the library
|
|
2
2
|
|
|
3
3
|
How the library paces sync, keeps a store in memory, talks to IndexedDB, a
|
|
4
|
-
data folder and a blob store, and mirrors a space into one.
|
|
4
|
+
data folder, a file on disk and a blob store, and mirrors a space into one.
|
|
5
5
|
|
|
6
6
|
> Not protocol. This page describes the reference library, and another
|
|
7
7
|
> implementation may do it differently and still interoperate. What peers must
|
|
@@ -96,7 +96,8 @@ write; without, versions go first, then the entries in one batch, then the
|
|
|
96
96
|
deletes, so no entry ever names a version that is not there.
|
|
97
97
|
|
|
98
98
|
Entry keys are strings; values are bytes. The adapters are IndexedDB (below),
|
|
99
|
-
the data folder
|
|
99
|
+
a directory (`directoryStores`: the data folder's registry, and a node's
|
|
100
|
+
stores on disk), the sealing wrapper, and an in-memory one for tests.
|
|
100
101
|
|
|
101
102
|
_Source: `packages/core/src/types.ts` (`StorageAdapter`, `BatchOp`). Tests:
|
|
102
103
|
`packages/core/tests/folder-adapter.test.ts`,
|
|
@@ -129,6 +130,10 @@ Each database:
|
|
|
129
130
|
range in one `readonly` transaction; `getVersions` is one `readonly`
|
|
130
131
|
transaction.
|
|
131
132
|
|
|
133
|
+
A data folder's own stores ([below](#data-folders)) use the prefix
|
|
134
|
+
`minca-pod:<dataPath with / replaced by :>`: not `minca:`, which is deleted
|
|
135
|
+
when the person forgets an account's browser copy.
|
|
136
|
+
|
|
132
137
|
Accounts kept in a browser live in database `minca-accounts`, object store
|
|
133
138
|
`accounts`. A remembered data-folder handle lives in database `minca-folder`,
|
|
134
139
|
object store `handles`, key `data-folder`.
|
|
@@ -146,7 +151,38 @@ adapter needs a browser)._
|
|
|
146
151
|
|
|
147
152
|
The folder's layout, file names and the way several writers converge are
|
|
148
153
|
protocol ([spec 05 §14](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md)).
|
|
149
|
-
How this library
|
|
154
|
+
How this library keeps and writes one is not.
|
|
155
|
+
|
|
156
|
+
`folderStores(directory, { basePath, localKey, local })` keeps the
|
|
157
|
+
`registry` in the folder, one file per entry, sealed under `localKey`, and
|
|
158
|
+
opens every other path in `local`: a space's entries and bodies, what the
|
|
159
|
+
node keeps beside them, and the `folder/<space id>` store writer state. The
|
|
160
|
+
factory names the folder's segments (`<basePath>/spaces/`, a directory blob
|
|
161
|
+
store) as `folder`, and every space runtime opened from it mirrors to them
|
|
162
|
+
([mirrors](#mirrors)): it pulls when the space opens and every
|
|
163
|
+
`watchIntervalMs` (default 2 s, `0` to turn it off), and pushes 1 s after a
|
|
164
|
+
change and on close.
|
|
165
|
+
|
|
166
|
+
`local` is this device's:
|
|
167
|
+
|
|
168
|
+
- **In the browser**, IndexedDB under `minca-pod:<dataPath>` (the default).
|
|
169
|
+
- **The CLI** keeps it outside the home, which may be the folder a browser
|
|
170
|
+
uses or one a sync service copies: `$MINCA_LOCAL` or `~/.local/state/minca`,
|
|
171
|
+
then `<first 16 hex of SHA-256 of the home's path>/<account id>/stores`, a
|
|
172
|
+
directory store (`localPath` in `packages/cli/src/home.ts`). A command and
|
|
173
|
+
the daemon share it, and the daemon finds the command's writes as it finds
|
|
174
|
+
any directory store's (`reconcileFolder`). Writer keys are beside it in
|
|
175
|
+
`writers/`; a slot left in the home from before is not read. `minca agent`
|
|
176
|
+
still keeps its key and stores together in `<home>/agent`.
|
|
177
|
+
- **Elsewhere** it must be given: `folderStores` refuses to start without
|
|
178
|
+
one, since a lasting key's places kept in memory would be handed out again
|
|
179
|
+
after a restart.
|
|
180
|
+
|
|
181
|
+
A space kept in an older layout is not read
|
|
182
|
+
([spec 05 §14.8](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md));
|
|
183
|
+
peers bring it back.
|
|
184
|
+
|
|
185
|
+
Getting the folder:
|
|
150
186
|
|
|
151
187
|
- **In the browser** the folder comes from `showDirectoryPicker` with
|
|
152
188
|
`id: "minca-pod"`, `mode: "readwrite"`, `startIn: "documents"`, which needs a
|
|
@@ -156,18 +192,25 @@ How this library gets and writes one is not:
|
|
|
156
192
|
instead. The File System Access API replaces a file atomically on `close()`.
|
|
157
193
|
- **The CLI** (Node and Bun) writes `<name>.<uuid>.tmp` with mode `0600` and
|
|
158
194
|
renames it into place. Directories are created with mode `0700`.
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
195
|
+
|
|
196
|
+
Two folders holding the same account (a copy, a restored backup) share this
|
|
197
|
+
device's local stores, which are named by the account's id: a version known
|
|
198
|
+
to be in one is not pushed to the other. Open one copy at a time, or let the
|
|
199
|
+
other fill in from peers.
|
|
162
200
|
|
|
163
201
|
A folder is the account: copy it to a USB stick and it is all of it, or put
|
|
164
202
|
it in iCloud, Dropbox or Syncthing and several devices converge with no relay,
|
|
165
203
|
since the set of versions only grows. Being readable by several origins is the
|
|
166
|
-
point, which is why OPFS (origin-private) is not used for it.
|
|
204
|
+
point, which is why OPFS (origin-private) is not used for it. Each device
|
|
205
|
+
adds a folder of segments per space, about one file per 256 KiB of records,
|
|
206
|
+
compacted once it has 32 and twice what the last compaction left.
|
|
167
207
|
|
|
168
|
-
_Source: `packages/core/src/
|
|
169
|
-
`packages/
|
|
170
|
-
|
|
208
|
+
_Source: `packages/core/src/node/stores.ts` (`folderStores`),
|
|
209
|
+
`packages/core/src/storage/blob/directory.ts`,
|
|
210
|
+
`packages/core/src/storage/directory-access.ts`,
|
|
211
|
+
`packages/cli/src/fs-directory.ts`, `packages/cli/src/home.ts` (`unlock`),
|
|
212
|
+
`packages/core/src/node/space-runtime.ts` ("Mirrors", "Writers outside this
|
|
213
|
+
process"). Tests: `packages/core/tests/folder-adapter.test.ts`,
|
|
171
214
|
`packages/cli/tests/path-safety.test.ts`._
|
|
172
215
|
|
|
173
216
|
## Blob stores
|
|
@@ -186,6 +229,11 @@ and without atomic operations (`BlobStore`):
|
|
|
186
229
|
Drivers:
|
|
187
230
|
|
|
188
231
|
- **Memory**, for tests.
|
|
232
|
+
- **Directory**, over a directory handle or a directory on disk
|
|
233
|
+
(`createDirectoryBlobStore`): each `/`-separated part of a key a
|
|
234
|
+
directory, the last a file, every part named as a data folder names its
|
|
235
|
+
files; a directory is removed with the last key in it. What a data folder
|
|
236
|
+
keeps its spaces in.
|
|
189
237
|
- **S3-compatible** (R2, B2, MinIO, Wasabi, AWS): path-style URLs
|
|
190
238
|
`<endpoint>/<bucket>/<prefix/><key>`, each `/`-separated part
|
|
191
239
|
`encodeURIComponent`-ed; SigV4-signed (`aws4fetch`), region default
|
|
@@ -196,6 +244,7 @@ Drivers:
|
|
|
196
244
|
|
|
197
245
|
_Source: `packages/core/src/storage/blob-store.ts`,
|
|
198
246
|
`packages/core/src/storage/blob/memory.ts`,
|
|
247
|
+
`packages/core/src/storage/blob/directory.ts`,
|
|
199
248
|
`packages/core/src/storage/blob/s3.ts`. Tests:
|
|
200
249
|
`packages/core/tests/mirror.test.ts`._
|
|
201
250
|
|
|
@@ -203,7 +252,7 @@ _Source: `packages/core/src/storage/blob-store.ts`,
|
|
|
203
252
|
|
|
204
253
|
A mirror's segments, where a store writer may write and how it pulls and compacts
|
|
205
254
|
are protocol ([spec 05 §16](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md)).
|
|
206
|
-
Today carriers and hosts mirror.
|
|
255
|
+
Today data folders, carriers and hosts mirror.
|
|
207
256
|
|
|
208
257
|
A store writer keeps its state in the store `mirrors/<space id>/<n>`, where `n` is
|
|
209
258
|
the mirror's index:
|
|
@@ -218,9 +267,12 @@ the mirror's index:
|
|
|
218
267
|
"Known" is everything this store writer uploaded **or read**, so nobody re-uploads
|
|
219
268
|
everyone else's versions.
|
|
220
269
|
|
|
221
|
-
A store writer pushes 5 s after the first change
|
|
222
|
-
yet known into segments of about 256 KiB (a
|
|
223
|
-
|
|
270
|
+
A store writer pushes 5 s after the first change (1 s for a data folder) and
|
|
271
|
+
on close, packing what is not yet known into segments of about 256 KiB (a
|
|
272
|
+
segment is closed once it reaches that size). It reads the `mirror:known:`
|
|
273
|
+
entries once, when it starts. Once it has 32 or more segments of its own,
|
|
274
|
+
and twice as many as its last compaction left, it compacts them, keeping
|
|
275
|
+
only what its store holds.
|
|
224
276
|
|
|
225
277
|
_Source: `packages/core/src/storage/mirror.ts`,
|
|
226
278
|
`packages/core/src/node/space-runtime.ts` ("Mirrors"),
|
|
@@ -245,7 +297,6 @@ _Source: `packages/core/src/storage/mirror.ts`,
|
|
|
245
297
|
and unicode, a 1 MB blob), with shared backoff on 429 and 5xx that honours
|
|
246
298
|
`Retry-After`, never retries a delete that returned 404, and never logs a
|
|
247
299
|
token or signed URL:
|
|
248
|
-
- _Directory_: a directory handle or a directory on disk.
|
|
249
300
|
- _Google Drive_: everything in one app-created folder, `drive.file` scope
|
|
250
301
|
only (the app sees only its own files). Drive addresses files by id, not
|
|
251
302
|
name, so the driver keeps a name → file id map from one `files.list`,
|
|
@@ -260,3 +311,57 @@ _Source: `packages/core/src/storage/mirror.ts`,
|
|
|
260
311
|
- **Budget.** A store like Drive allows about 1,000 requests per 100 seconds.
|
|
261
312
|
The segment, never the single version, is the unit fetched, and syncing
|
|
262
313
|
1,000 records must fit that budget.
|
|
314
|
+
|
|
315
|
+
## Files
|
|
316
|
+
|
|
317
|
+
What peers must agree on (the `files` member, chunks and the id, RFC 8188
|
|
318
|
+
records, the three messages, keeping a file with its body) is in
|
|
319
|
+
[spec 05 §16.6](https://github.com/leifriksheim/minca/blob/main/spec/05-sync-and-storage.md).
|
|
320
|
+
How this library does the rest:
|
|
321
|
+
|
|
322
|
+
- **Adding.** `records.addFile(space, bytes, { mime, name? })` seals the
|
|
323
|
+
bytes under a fresh key and salt, keeps them, and gives back a `FileRef`:
|
|
324
|
+
`{ hash, size, key, salt, mime, name? }`. Put it anywhere in a body;
|
|
325
|
+
writing that body names every reference in it on the version
|
|
326
|
+
(`fileRefsIn`, at most 16), so the envelope and the body always agree.
|
|
327
|
+
- **Reading.** `records.readFile(space, ref)` gives the bytes: held here, or
|
|
328
|
+
fetched from connected peers, one after another, then from mirrors,
|
|
329
|
+
checking every chunk. It asks only for files a version this node keeps
|
|
330
|
+
whole names. For a picture, `URL.createObjectURL(new Blob([bytes], { type:
|
|
331
|
+
ref.mime }))` shows it.
|
|
332
|
+
- **Where chunks live.** Entries in the space's own store: `f/<id>/list`,
|
|
333
|
+
`f/<id>/<n>`, and `f/<id>/own` with the time a file was added and not yet
|
|
334
|
+
named. A data folder keeps these in this device's own stores, not the
|
|
335
|
+
folder.
|
|
336
|
+
- **What a node fetches.** A node with `fetchFiles: 'all'` (carriers and
|
|
337
|
+
hosts) fetches every named file about 2 s after its records change. Any
|
|
338
|
+
other node fetches a file when it is read.
|
|
339
|
+
- **Serving.** A `want-file` is answered with at most 4 chunks; the asker
|
|
340
|
+
asks again for the rest, so a large file never floods a connection.
|
|
341
|
+
- **Keeping.** About 2 s after records change, files no version kept whole
|
|
342
|
+
names are dropped (files of versions that don't stand are not kept),
|
|
343
|
+
except one added here in the last 24 hours and not yet named.
|
|
344
|
+
- **Mirrors.** The same pass uploads each named file this node holds whole,
|
|
345
|
+
with its chunk list, to `files/<space>/<store writer>/`, and deletes this
|
|
346
|
+
writer's copies of files no longer named.
|
|
347
|
+
|
|
348
|
+
_Source: `packages/core/src/files/space-files.ts`, `packages/core/src/files/file-id.ts`,
|
|
349
|
+
`packages/core/src/files/seal.ts`, `packages/core/src/node/space-runtime.ts` ("Files").
|
|
350
|
+
Tests: `packages/core/tests/files.test.ts`._
|
|
351
|
+
|
|
352
|
+
### Planned: video and pictures
|
|
353
|
+
|
|
354
|
+
> **Planned.** Not normative.
|
|
355
|
+
|
|
356
|
+
- **Video.** `readFile` fetches a whole file. Playing before it is whole
|
|
357
|
+
needs a URL a service worker serves, answering the browser's range
|
|
358
|
+
requests from only the chunks read, opened one at a time (`fileOpener`).
|
|
359
|
+
MP4 with its index first, or WebM; no transcoding; a poster is a second,
|
|
360
|
+
small image file.
|
|
361
|
+
- **Several peers at once.** Chunks asked of several peers in parallel, and
|
|
362
|
+
of a slow one again elsewhere.
|
|
363
|
+
- **Mirrors as sources by range.** `BlobStore.getRange`: a `Range` header on
|
|
364
|
+
S3, `File.slice` in a data folder, so a chunk is read without the whole.
|
|
365
|
+
- **Pictures.** `std.image` gains `width`, `height` and a tiny WebP
|
|
366
|
+
placeholder as a `data:` URL, shown before a chunk arrives; and
|
|
367
|
+
`std.profile`'s `avatar`, resized in the browser before it is added.
|