@readymadeit/mcp 0.0.0-stage → 0.1.0

package/bundle/docs/content/concepts/remotes.md

@@ -0,0 +1,423 @@
+# Remotes
+
+How a board says which runtime server a runtime belongs on without writing down an address, and how those servers come to belong to a deployed board: they connect to the coordinator, which dials nothing.
+
+---
+
+## The problem
+
+A board used to record **how to reach** a runtime — `"url": "http://127.0.0.1:8080"` —
+and an address is only true from where it is dialled. That loopback address is a
+different machine for everyone who opens the board, and for a coordinator in a
+datacentre it is the coordinator itself.
+
+Two things followed, and they are the two halves of this page:
+
+- A **shared board** carried a fact about one person's network as though it were
+  a fact about the runtime.
+- A **deployed board** asked its coordinator to dial addresses out of a board —
+  untrusted input — from inside whatever network the coordinator ran in. That
+  needed an allowlist, a private-range policy, and still could not reach a
+  laptop behind NAT.
+
+---
+
+## Naming a remote
+
+A **remote** is a runtime server a client knows by name. Every host keeps a list
+of them — the Add-runtime picker shows it, *Manage runtime servers* edits it —
+and a board may name one instead of giving its address.
+
+A runtime says where it runs in exactly **one** of two ways, both authored:
+
+```json
+{ "id": "node", "name": "Node", "type": "rest", "url": "http://127.0.0.1:8080" }
+{ "id": "node", "name": "Node", "type": "rest", "remote": "Laptop" }
+```
+
+| Field | Says | Resolved |
+|---|---|---|
+| `url` | an address the person wrote | used as written. Every board from before remotes; valid forever |
+| `remote` | a name | looked up among this client's remotes, by any name one of them answers to |
+
+There is no third way that says only *what kind* of server would do and leaves
+the client to pick one. A runtime may reference credentials, and those are
+released to the server it lands on — so where it lands has to be something the
+board said or the person can read off it, never a choice made on their behalf
+among whatever servers they happen to keep.
+
+### Both is an error, not a preference
+
+No fallback and no precedence. A name resolves differently for each person, so
+anyone who can put a board in front of you controls whether its name resolves.
+If an unresolvable name fell back to a `url`, they would get two attempts at
+making your client dial an address and need only the second. A runtime carrying
+both is refused before anything is dialled.
+
+### The runtime the app embeds: `embedded`
+
+One name is the same for everybody. The Readymade apps each embed an hkp-rt and
+keep it among their remotes, and each used to have a name of its own for it —
+`meander-cpp` on the desktop, `meander-ios` on a phone — which is as local a
+fact as an address. So every host that embeds a runtime answers to
+**`embedded`** for it, and the desktop app lists it under that name, in
+*Settings → Connections* and in the Add-runtime picker:
+
+```json
+{ "id": "rt", "name": "hkp-rt", "type": "rest", "remote": "embedded" }
+```
+
+A board that should run on the app that opens it says this, and nothing else
+about where: no address, so no port that has to be free and that the board and
+the app have to agree on; no name that works in one app and not the next. It is
+what every shipped board meant for the embedded runtime uses.
+
+Inside an app the name is that app's runtime, whatever else is kept under it —
+a board asking for the app's own runtime must not land on a server somebody
+happened to call the same. A host that embeds nothing, a browser tab, has no
+such runtime; there `embedded` is a name like any other, which a person may
+give a server of theirs to open such a board from the playground, and
+otherwise the board says what it wanted and stops.
+
+An app's older name for its runtime is still answered to, so a board that says
+`hkp://remotes/meander-cpp` opens in the desktop app as it did. And no kept
+runtime server may take either name there.
+
+`hkp://remotes/<name>` — the spelling boards used for the app's embedded runtime
+before `remote` existed — is a name in a url's clothing. It **counts as a name**
+(so it cannot sit beside a `remote`), and is still resolved by the host that
+serves the `hkp:` scheme, which refuses a name it does not hold.
+
+### A runtime server of a kind: `node` and `python`
+
+`embedded` settles where a board runs when the app that opens it brings the
+runtime along. hkp-node and hkp-python come with no app: somebody starts them,
+on a port of their choosing, on this machine or on another. A board that wrote
+down `http://127.0.0.1:8080` for one recorded where its author's happened to
+listen — and since both servers default to that port, recorded it for
+whichever of the two it meant, so a board needing both had to move one and say
+so in its description.
+
+So a board that needs a server of a kind, and does not care whose, names the
+kind and nothing else:
+
+```json
+{ "id": "reader", "name": "Reader", "type": "rest", "remote": "node" }
+{ "id": "speech", "name": "Speech", "type": "rest", "remote": "python" }
+```
+
+| Name | The runtime server kept under it |
+|---|---|
+| `node` | an hkp-node |
+| `python` | an hkp-python |
+
+Whoever opens such a board keeps their own server under that name, once, with
+the address it runs at for them — as the name it is listed under, or as
+another name of a server they already keep as something else (below). Every
+board that says the name then opens there, and none of them has to be edited
+to move between people. It is what every shipped board with an hkp-node or
+hkp-python runtime uses.
+
+Unlike `embedded`, nothing about these names is enforced: they are **agreed**.
+No host answers to one unasked, a client that keeps no server under it does not
+resolve it, and nothing checks what kind of server it finds there — a server
+that lacks a board's services says so when they are created. A client that
+keeps nothing under the name asks, as it does for any other, and adds what the
+name stands for (`SHARED_REMOTE_NAMES`).
+
+This is not the third way ruled out above. Nothing is picked: the person said
+which of their servers is `node`, and a board that says `node` lands there and
+nowhere else.
+
+A board that needs one *particular* machine — the one with the model files, the
+one inside a network — is not served by a kind. It names that server for what
+it is (`"remote": "Studio"`), or keeps its `url`.
+
+### More than one name: aliases
+
+A remote answers to the name it is listed under and to any number of
+**aliases**:
+
+```json
+{ "name": "Laptop", "url": "http://127.0.0.1:8080", "aliases": ["node", "NodeJS"] }
+```
+
+Two things need this, and both come from the same fact — a board holds a name
+for as long as the board exists, and the person who keeps the server does not
+hold the board:
+
+- **One server, several roles.** The hkp-node on a laptop is `Laptop` to its
+  owner and `node` to every shipped board. Without aliases it could be one or
+  the other.
+- **Renaming.** A server that is renamed keeps the name it had as an alias, so
+  the boards that say the old one still open. The edit form puts it there as
+  the new name is typed, where it can be seen and taken out again — retiring a
+  name is a decision, not a side effect.
+
+The name a server is listed under comes before anybody's alias, so a second
+name for one server never takes a board away from the server actually called
+that. Two servers may not answer to one name; the form refuses it. And in an
+app, no kept server may take a name the embedded runtime goes by, as an alias
+no more than as its own.
+
+Aliases are the client's, like the rest of a remote: they are kept wherever
+the host keeps its servers and never enter a board.
+
+### What a board is given when a runtime is added
+
+Putting a runtime on a server from the Add-runtime picker gives the board that
+server's **name**, not its address — the address stays on the live runtime
+beside it and a save writes the name alone. So a board built in the playground
+is as portable as one written by hand.
+
+Which name, when the server has several: the most portable one it answers to
+(`remoteNameForBoard`).
+
+| The server | The board says |
+|---|---|
+| the runtime the host embeds | `embedded`, whatever the app lists it as |
+| answers to a name boards share for its kind | that name — `node`, `python` |
+| anything else | the name it is listed under |
+
+A runtime put on an address this client keeps nothing under says the address.
+
+### Moving a runtime to another server
+
+A runtime already on a board is moved from its menu, **Move to …**, which
+opens the runtime servers this client keeps, each with its address and the
+one the runtime is on marked — names alone do not say which server is this
+machine. The runtime is built on
+the new server with the state its services have at that moment, read live
+from the server it is leaving, and only then removed from that one, so a move
+the new server refuses leaves it where it was. The board then names the new
+server by the same rule as above; when that name would lead to a different
+server on opening, it gets the address instead.
+
+The runtime keeps its id, so whatever the rest of the board says about it —
+a `hkp-mount://` reference among them — still holds. What moves is the
+services' state. What a server keeps for a board beside it — tables, records,
+files — is that server's and stays there; on the new server the runtime finds
+whatever that server keeps under the same names.
+
+This is how a board built against a local server is put on the one a
+coordinator runs before it is deployed: a board's runtimes run where the
+board names them, wherever the coordinator is.
+
+This makes a kept server and the boards that use it depend on each other in a
+way they did not when a board held the address: remove the server, and a board
+that names it no longer resolves. Renaming does not do that, because of
+aliases, and removing leads to the question below rather than to a dead board.
+
+### Resolution is the person's client's, and is never written back
+
+A name is resolved **only on the person's side**, against the remotes their
+client keeps, when the board loads there. Nothing else resolves one: a
+coordinator holds no name-to-address table, and a name in a deployed board is a
+label for people.
+
+- A name this client does not hold is **put to the person**: *which runtime
+  server is `node`?* They answer with one of the servers they keep, which then
+  answers to that name as well, or with the address of one they do not keep
+  yet, which is kept under it. Either way the answer is kept, so it is asked
+  once, and the board carries on opening where it was
+  (`core/remotePrompt.ts`, `UnknownRemoteDialog`).
+- That is a question, not a choice made for them. Nothing matches the name to
+  a server that looks suitable, and what the dialog hands back counts only if
+  it is a remote that answers to the name. Every address is resolved before
+  any runtime is built, so a board the person declines to place has built
+  nothing anywhere.
+- Unanswered — dismissed, or where there is nobody to ask — the name **fails
+  the load**, naming the remote. There is nothing to fall back to, and a board
+  quietly missing a runtime is worse than one that says which runtime server
+  it wanted.
+- A remote runtime that names **no** server — neither a `url` nor a `remote` —
+  fails the load the same way. Nothing is chosen for it.
+- The resolved address lives on the **live** runtime, beside the name it came
+  from. A save writes the name and drops the address
+  (`authoredAddressing`). That is what keeps the one-way rule enforceable: if
+  resolution wrote into `url`, every resolved board would be in the state the
+  rule forbids.
+- **Export bakes.** Handing one concrete runtime to somewhere this client's
+  names mean nothing substitutes the address and drops the name
+  (`bakeAddressing`) — still exactly one way.
+
+Nothing ever moves a runtime onto a machine other than the one it named: the
+files, models, devices and credentials behind a runtime are not in the board, so
+a copy elsewhere would deploy, report success, and be wrong.
+
+The vocabulary and the lookup live in
+`hkp-frontend/src/runtime/board/remote.ts`; `restoreBoard`
+(`core/boardPersistence.ts`) resolves on load.
+
+---
+
+## Connecting to a coordinator
+
+A coordinator **never dials** a runtime server. Every participant in a deployed
+board connects *to it*: runtime servers on a laptop, runtime servers in a
+datacentre, and the browsers that open the board.
+
+One rule, and what falls out of it:
+
+- **No confused deputy.** The coordinator makes no outbound connection on a
+  board's say-so, so there is no address policy to maintain and nothing for a
+  crafted board to aim it at.
+- **No reachability requirement.** NAT and loopback are not special cases. The
+  only thing that has to be reachable is the coordinator.
+- **No address for the coordinator to hold.** It is told none and resolves none.
+
+### Deploy is an introduction
+
+The browser is the only party that reaches both sides — it holds the person's
+session with the coordinator *and* with each of their runtime servers. So
+handing a board over is an introduction, made by `deployBoard()`
+(`hkp-frontend/src/core/deploy.ts`):
+
+1. It asks the coordinator for a **ticket** per remote runtime
+   (`POST /coordinator/users/<sub>/boards/<board>/tickets`).
+2. It tells each runtime's server — at the address the name resolved to —
+   *connect to this coordinator, with this ticket*
+   (`POST <server>/coordinator-links`), over the session it already creates
+   runtimes there with.
+3. Each server opens a WebSocket to the coordinator's `/coordinator/join`,
+   presenting the ticket, and says what it is: its kind, its registry, and
+   whether that runtime is already running there.
+4. Only once they have all connected is the board registered.
+
+### What a ticket is
+
+A bearer credential, and deliberately a narrow one: it speaks for **one runtime
+of one board of one person**. The connection made with it can do exactly the
+things a coordinator does to that runtime — build it, describe it, configure a
+service, change what it logs, drive it, remove it — as the person who
+introduced it, and nothing else.
+
+| | |
+|---|---|
+| Who keeps it | the runtime server, beside its mount secret (`~/.hkp/node-<instance>/` or `~/.hkp/python/`, `coordinator-links.json`, `0600`). The coordinator keeps only a **hash** |
+| How long it lasts | until replaced or revoked. It is what a runtime server reconnects with after a restart or a dropped connection, with nobody present |
+| Replaced | by deploying again — and only once that deploy went through. A ticket asked for while the runtime has one is **pending** beside it: its holder is welcomed and waits, and the server holding the first stays the board's and can still reconnect. Registering the board settles it: a pending ticket a server is waiting with becomes the one that counts, the one before it is forgotten and its holder gives way; a pending ticket nobody connected with is dropped |
+| Taken back | when a deploy fails. The client gives up the tickets it asked for; a server waiting with one is let go, and nothing about the running board changed |
+| The same server again | a runtime server that is already the board's keeps its connection and its ticket when it is introduced again, and takes only the secrets. The new ticket goes unused |
+| A runtime with no server | the one case a pending ticket takes over before registration: its holder arrives and nothing is connected for that runtime. Nothing is running there to protect, and it is how a runtime whose server left is given one again without deploying the board |
+| Revoked | when the board is deleted, or deployed without that runtime. The connection is closed |
+| On the holder | a ticket the coordinator no longer holds ends the link for good: the runtime server drops it **and the runtime it was for**, which was the coordinator's and is now nobody's |
+
+There is **one connection per runtime of a board**. Two boards on one machine are
+two tickets and two connections, which keeps one board's traffic off another's
+socket and keeps lifecycle and failure per board. A runtime server keys a link
+by board as well as runtime id, and keeps the runtime it builds for it in that
+board's own space, so two boards that both call a runtime `node` do not meet
+(`concepts/cloud-boards.md`). Leaving a board is
+`DELETE <server>/coordinator-links/<board>/<runtimeId>`.
+
+A board's runtime on a server belongs to **one coordinator at a time**. A link
+does not say which coordinator it is for beyond its address, and the space a
+board's runtimes live in is named by the board alone, so the same board name
+deployed to a second coordinator through the same server would take the first
+one's runtime. The server refuses that introduction instead, naming the
+coordinator the board is already deployed by. A link whose coordinator is not
+connected is replaced.
+
+### Required and transient
+
+A cloud board already contains a browser runtime, run by whichever browser is
+attached, and browsers come and go constantly. So "a participant is missing"
+does not mean one thing:
+
+| | Which runtimes | When it is away |
+|---|---|---|
+| **Required** | remote (`rest`) | the board is in `error`, **naming it** |
+| **Transient** | browser | data arriving for it stops there, exactly as a `null` stops a pipeline, and the board stays `running` |
+
+`error` is **not terminal**. A required participant that reconnects is taken
+back: if it still has the runtime this board built — only the connection
+dropped — the runtime is picked up as it is, live state and all. If it does not
+— the server restarted — the runtime is rebuilt from the board's config. What
+does not come back is the run.
+
+A runtime server that reconnects to a board that has since been stopped has its
+leftover runtime released, so stopping a board while a server is away does not
+leave an orphan behind.
+
+### The same thing on both restarts
+
+Both sides keep exactly one thing about the connection — the ticket — so both
+restarts end the same way:
+
+- **A runtime server restarts.** It reconnects with its ticket; the coordinator
+  rebuilds the runtime from the board's config.
+- **A coordinator restarts.** A board that was running comes back in `error`,
+  naming every runtime it is waiting for, and builds each as its server
+  reconnects. A board that was stopped stays stopped.
+
+No user token is involved in either.
+
+---
+
+## Credentials
+
+Values flow from the person's client to the runtime server **they chose**, and
+not through the coordinator:
+
+- The introduction (step 2 above) carries the values for the references that
+  runtime's services hold. Consent is asked for the **address the name resolved
+  to**, never for the name — a remote's name is board-controlled and resolves
+  differently for each person, so a grant keyed on it would follow the board
+  wherever it pointed.
+- The runtime server holds them with the link, in memory, and hands them to the
+  runtime when the coordinator builds it. They are not sent to the coordinator.
+- A runtime built without a value it references says so:
+  `Runtime "node": needs configuration — its runtime server holds no value for
+  imap.password`. That is what a board looks like after its runtime server
+  restarted — the ticket survived, the values did not — and deploying again
+  supplies them.
+
+→ `plans/TODO-SECRETS.md` for the coordinator-side vault this does not replace.
+
+---
+
+## Which runtime servers can join
+
+| Runtime server | Joins a coordinator |
+|---|---|
+| hkp-node | yes |
+| hkp-python | yes |
+| hkp-rt (C++), standalone or in a container | yes |
+| hkp-rt, embedded in the desktop app | yes |
+| hkp-rt, embedded in the iOS or Android app | **no**, deliberately — a phone suspends the app at will, the link would drop, and the board would go to `error`. The deploy preflight says so beforehand |
+
+A server says whether it can in `GET /runtimes` (`coordinatorLinks: true`),
+beside its kind and registry. On hkp-rt that is the host's decision: the library
+connects to a coordinator only once its host has given it somewhere to keep
+tickets (`Server::enableCoordinatorLinks`).
+
+hkp-rt is single-tenant — one id space, an email allow-list — so its links have
+no owner: whoever may create a runtime there may link one. It keeps its tickets
+in `~/.hkp/cpp/coordinator-links.json`.
+
+---
+
+## Rough index into the source
+
+| Concern | Where |
+|---|---|
+| `url` / `remote`: vocabulary, lookup, aliases, what a save keeps | `hkp-frontend/src/runtime/board/remote.ts` |
+| Resolving on load | `hkp-frontend/src/core/boardPersistence.ts` (`resolveForRestore`) |
+| Asking about a name nobody is kept under | `hkp-frontend/src/core/remotePrompt.ts`, `ui-components/UnknownRemoteDialog.tsx` |
+| The name a board is given when a runtime is added | `hkp-frontend/src/core/runtimeOperations.ts` (`addRuntime`) |
+| Editing a server's names | `hkp-frontend/src/ui-components/connections/EditServerForm.tsx` |
+| Where each host keeps its servers | web: `localStorage` (`available-remote-runtimes`); desktop: `meander/backend/settings.h`, `remoteRoute.h`; iOS: `remotes.json` in the app's documents |
+| Preflight, per runtime | `hkp-frontend/src/core/deployPreflight.ts` |
+| The introduction | `hkp-frontend/src/core/deploy.ts`, `components/Toolbar/DeployDialog.tsx` |
+| Tickets and accepted connections | `hkp-node/src/coordinator/participants.ts`, `join.ts` |
+| The protocol over a connection | `hkp-node/src/coordinator/participantProtocol.ts` |
+| A board over its participants | `hkp-node/src/coordinator/session.ts` |
+| The runtime server's end | `hkp-node/src/coordinatorLinks.ts`, `hkp-python/src/hkp/coordinator_links.py`, `hkp-rt/lib/src/coordinator_links.cpp` (over `common/link_socket.cpp`) |
+| Tests | `hkp-node/tests/coordinator-participants.test.ts`, `coordinator-session.test.ts`, `coordinator-links.test.ts`, `coordinator-restart.test.ts`, `coordinator-python.test.ts`, `coordinator-rt.test.ts`; `hkp-python/tests/test_coordinator_links.py`; `hkp-rt/tests/coordinator_links.test.cpp`, `link_socket.test.cpp`; `hkp-frontend/src/runtime/board/tests/remote.test.ts`, `core/tests/deploy*.test.ts`, `remote-resolution.test.ts`; `e2e/tests/cloud/deploy.spec.ts` |
+
+---
+
+See also: **Cloud boards** (`concepts/cloud-boards.md`) for deploying as a whole,
+**Coordinator** (`concepts/coordinator.md`) for the role, and **Mounts**
+(`concepts/mounts.md`) for the other thing a board names rather than addresses.