borgmcp 5.5.0 → 5.7.0

package/docs/HUMAN_REPRESENTATIVE.md

@@ -70,9 +70,14 @@ To resume later, run `borg representative prepare --coordinator <coordinator-dro
 your saved labels. Recovery errors for a bound connection print that complete
 command with its actual labels and worktree. Changing the cube or Coordinator
 is refused unless you
-pass `--rebind`; a rebind also discards the old request ledger. A running
-`borg representative mcp` process never picks up a rebind: its calls fail
-closed until the MCP host restarts it.
+pass `--rebind`; a rebind that changes the cube or Coordinator also discards the
+old request ledger. A rebind of any kind, including one with the same cube and
+Coordinator, requires restarting every adapter and the listener; running ones
+refuse. A running `borg representative mcp` process answers `send`, `read`,
+`deliver` and `ack` with `BINDING_MISMATCH` until the MCP host restarts it;
+its `status` reports the new `binding_fingerprint` beside the
+`pinned_binding_fingerprint` it started with. A running listener stops with
+reason `rebound`.
 
 `prepare` reuses Borg's connection and worktree preparation, including its
 private-state initialization and saved-connection checks. It does not require
@@ -114,9 +119,10 @@ Check a connection at any time with
 
 | Tool | Purpose |
 | --- | --- |
-| `borg_representative-status` | Bound cube, representative, Coordinator; live re-check; unresolved sends; limits. |
+| `borg_representative-status` | Bound cube, representative, Coordinator; live re-check; unresolved sends; limits; `binding_fingerprint`. |
 | `borg_representative-send` | Relay one `request`, `question` or `decision` to the bound Coordinator. |
-| `borg_representative-read` | Unread replies from the bound Coordinator addressed to the representative. Drains everything it fetches. |
+| `borg_representative-read` | The bound Coordinator's replies after the delivered checkpoint, oldest first, bounded. Changes nothing. |
+| `borg_representative-deliver` | Move the delivered checkpoint through a reply the host has durably persisted. |
 | `borg_representative-ack` | Signal to the Coordinator that one direct reply was received. Nothing more. |
 
 There is no tool for logging to other drones, broadcasting, role or drone
@@ -202,32 +208,111 @@ Exactly-once delivery is therefore not claimed; at-most-once per `request_id`
 relies on the server honouring `post_id` deduplication as the shared protocol
 specifies, and on the single-process rule below.
 
-## Reading, cursors and wake limits
-
-- `read` drains only the representative drone's **own** unread cursor. Other
-  drones' cursors are separate client-owned state and are never touched.
-- A read **consumes everything it fetched**, not only what it returns: the
-  Coordinator's replies, and equally the entries it ignores (other drones'
-  entries are counted in `ignored_entries` and never returned; the
-  Coordinator's broadcasts are returned only with `include_broadcast`). None
-  of them appear unread again.
-- If the MCP host stops between reading a reply and relaying it to the human,
-  that reply is gone from the unread view. It still exists in the cube log, but
-  this version offers no tool to list past replies again. Persist the read result
-  in the host before relaying it.
+## Reading, delivery and wake limits
+
+Four separate notions, each with its own tool or owner:
+
+- **Receipt**: `read` returned the reply. Nothing moves; the same reply comes back
+  on every `read` until it is delivered.
+- **Delivered**: the host called `deliver` after durably persisting everything up
+  to that reply. Only `deliver` moves the read window.
+- **Processing and display**: the host's own business after delivery.
+- **`ack`**: a signal to the Coordinator that a direct reply was received. It is
+  not delivery and does not move the window.
+
+Reading:
+
+- `read` returns the Coordinator's replies addressed to the representative that
+  come strictly after the delivered checkpoint, in `(created_at, entry_id)` order.
+  Other drones' entries are counted in `ignored_entries` and never returned; the
+  Coordinator's broadcasts are returned only with `include_broadcast`.
+- `read` never advances anything: not the checkpoint, not the representative's
+  unread cursor, not a server cursor. After a crash at any point, the next `read`
+  returns every reply not yet delivered.
+- Bounds: `limit` (1 to 50, default 10) is a hard cap on returned replies.
+  `max_bytes` (4096 to 60000, default 32768) caps the serialized tool result.
+  Replies are whole or omitted, never truncated; a reply that alone exceeds
+  `max_bytes` is returned alone with `"oversize": true`. The serialized result
+  never exceeds `max(max_bytes, 16384)` bytes: if an oversize reply is still
+  larger than that, its document citations are reduced to ids and it carries
+  `"documents_reduced": true`; message text is never cut. `status` reports this
+  floor as `envelope_floor`; set the host's tool-result ceiling at or above it.
+  The floor covers a message at the server's default post limit (4096 bytes)
+  with ordinary metadata. JSON escaping (a message full of tabs or quotes) or
+  heavy citation metadata can still push a single reply above it, as can a
+  server that allows larger posts. Then `read` refuses with
+  `REPRESENTATIVE_READ_OVERSIZE` (`entry_id`, `measured_bytes`, `bound`) and
+  changes nothing; retry with a larger `max_bytes` (up to 60000).
+- `read` scans past entries that are not for the representative until the page
+  is full or the log ends, so it never returns no replies with `has_more: true`.
+  `has_more` is true only when another reply follows the returned window.
+  Page by delivering and reading again.
+- The result includes `checkpoint` (`entry_id` and `created_at`; null until the
+  first delivery, unless the upgrade below started it at the old read position)
+  and `binding_fingerprint`.
+
+Delivering:
+
+- `deliver` takes `{ "through": "<entry_id>" }` and moves the checkpoint to that
+  reply. Call it only after the host has durably persisted every reply up to it.
+- `through` must be a reply that `read` actually returned since the checkpoint
+  last moved; membership is checked, not only the range, so a broadcast skipped
+  by a read without `include_broadcast` is refused too. Any other entry is refused
+  with `REPRESENTATIVE_DELIVER_UNKNOWN_ENTRY` and nothing changes. The same or an older
+  id is a no-op that returns `advanced: false`, so a retry after a lost result is safe.
+- Implementation note: the checkpoint, an internal read fence (the latest reply
+  any `read` returned) and the replies returned since the checkpoint last moved
+  are stored together in one private file per binding, written atomically.
+  `deliver` accepts only one of those returned replies. None of this is part of
+  the interface.
+
+Binding fence:
+
+- `binding_fingerprint` is the hex SHA-256 of the canonical JSON array
+  `[origin, trustIdentity, cubeId, representativeDroneId, coordinatorDroneId, boundAt]`.
+  It changes on every rebind and on a server trust change, and contains no path or
+  credential. `prepare --rebind` always starts a new generation, even when the
+  cube and Coordinator stay the same. It appears in `status`, `read`, `send`, `deliver` and the listener's
+  `listening` event. Persist it at binding time; if any result shows a different
+  value, stop routing and hold for the human.
+
+Upgrading from a version without `deliver`:
+
+- The first `read` or `deliver` for a binding starts the checkpoint where the old
+  destructive read left the representative's unread cursor: replies that were
+  unread at upgrade time are returned, replies already read are not. When the
+  binding never read, every addressed reply in the cube log is returned. If the
+  upgrade is interrupted before its checkpoint is written, the next read returns
+  every addressed reply instead; deduplicate by `entry_id`.
+  The old position is taken only from a genuine private file (a regular file you
+  own, not a symlink, not writable by group or others, in the private config
+  directory); otherwise the checkpoint starts empty and every addressed reply is
+  returned.
+- That upgrade happens once per representative drone and server authority, and is
+  recorded in a private marker, so it never runs again, even after a checkpoint
+  file is removed. A later
+  binding generation (any `prepare --rebind`, or a trust change) starts with an
+  empty checkpoint, so its first read returns its addressed history. Deduplicate by
+  `entry_id` (the host persists everything it delivers) and page with `limit`.
+
+Known limits:
+
+- Retention is the server's cube log; the local inbox is not a content source.
+- Each `read` scans the cube log from the checkpoint, including entries not
+  addressed to the representative, so a host that never calls `deliver` pays a
+  growing scan. Deliver promptly.
 - Replies preserve document citations (id, title and state). Document bodies are
   not included and cannot be fetched through this connection. Ask the Coordinator
   to provide the content through a supported channel.
-- `limit` is a page-size hint, not a hard cap: when the unread backlog is
-  large the client's digest mode fetches, and drains, more than `limit`.
-- `ack` is only a signal to the Coordinator that a direct reply was received.
-  It does not make delivery reliable, and it neither advances nor restores the
-  unread cursor.
+
+Ownership:
+
 - Exclusive process ownership is enforced for each representative drone. Processes
-  may start idle; the first `send`, `read` or `ack` takes the lease. Other processes
-  receive `REPRESENTATIVE_OWNERSHIP_REQUIRED` before any ledger reservation or
-  write, cursor access, or network call. The refusal names the owner's PID and
-  start time. Use that host, or wait for it to exit before using another.
+  may start idle; the first `send`, `read`, `deliver` or `ack` takes the lease. Other
+  processes receive `REPRESENTATIVE_OWNERSHIP_REQUIRED` before any ledger
+  reservation, delivery-state write, cursor access or network call. The refusal
+  names the owner's PID and start time. Use that host, or wait for it to exit before
+  using another.
 - `status` is allowed in every process, is read-only, and takes no lease. Its
   `ownership` field reports the state, PID, start time, and heartbeat age in
   milliseconds (`ageMs`). A clean exit releases ownership; a dead PID or a
@@ -238,20 +323,206 @@ specifies, and on the single-process rule below.
   deduplication. Retry an ambiguous send with its original `request_id`.
 - `in_reply_to` is a textual match of a known `request_id` quoted in the reply.
   It is a convenience, not a protocol guarantee.
-- **There is no background wake.** A generic MCP host receives nothing
-  unsolicited: replies are seen only when the host calls
-  `borg_representative-read`. This version provides explicit send/read round
-  trips only and makes no claim of automatic ongoing coordination. The
-  Coordinator is woken by the direct message through its own normal wake path.
+
+Wake hints:
+
+- **The MCP process does not push content.** A separate supervised `listen`
+  process emits body-free wake hints. The owning adapter still calls `read` for
+  content. Hints are neither delivery receipts nor authority.
+- Live dedupe is bounded to the surviving inbox tail and recent IDs; an ancient
+  trimmed ID sent again as a live event can produce a duplicate hint. Ordered
+  catch-up also dedupes against its captured resume cursor.
+- The listener retains a bounded tail: above 1024 lines it trims to the latest
+  512. Lost-hint replay covers only that tail; beyond it `read` from the delivered
+  checkpoint is the source of truth. On every `gap`, call `read`.
 
 ### Host conversation routing
 
 The lease selects one consuming process, not a conversation within that host.
 The host must record which conversation owns each `request_id`, persist every
-read result before relaying it, and route replies using `in_reply_to`. Hold
-replies with an unknown or missing request ID for the human instead of dropping
-them. Borg cannot enforce these duties inside the host; it provides neither a
-durable inbox nor a separate unread cursor for each conversation.
+reply durably before calling `deliver`, and route replies using `in_reply_to`.
+Hold replies with an unknown or missing request ID for the human instead of
+dropping them. Borg cannot enforce these duties inside the host; it provides one
+delivered checkpoint per binding, not one per conversation. The listener inbox
+is private client state, not a host content API.
+
+## Supervised listener
+
+For a prepared connection, run a separate long-lived subprocess:
+
+```bash
+borg representative listen --worktree <path>
+```
+
+Supervise it and persist the last `entry_id` durably admitted to the host's work
+queue. On subsequent starts, pass that checkpoint:
+
+```bash
+borg representative listen --worktree <path> --replay-after <entry_id>
+```
+
+There is one listener lease per representative drone and server authority,
+independent of the lazy tools lease. A second listener refuses without consuming
+or appending anything. A dead owner or expired heartbeat permits takeover; a
+process that loses ownership exits and must be restarted. A local lease cannot
+cancel an already-issued request. No eager tools-lease option is needed when one
+adapter exclusively calls `send`, `read`, `deliver` and `ack`.
+
+`representative status` reports `listener` beside tool `ownership`: running
+state, owner PID and start time, heartbeat age (`ageMs`), persisted watermark and
+private inbox path. Status acquires nothing. Do not read the inbox or depend on
+its pathname; it is not the content-delivery interface.
+
+Stdout is newline-delimited JSON only. Stderr contains human diagnostics and
+must not be parsed. The host must ignore unknown fields and unknown event types.
+
+| Event | Fields and meaning |
+| --- | --- |
+| `refused` | The only stdout line on startup refusal: `code`, `exit_code`. Another listener uses `REPRESENTATIVE_LISTENER_OWNED`, plus `owner_pid` and `owner_started_at`. |
+| `listening` | Once connected and holding the lease: `cube_id`, `drone_id`, `binding_fingerprint`, `watermark` (entry id or null), `inbox`. |
+| `entry` | `entry_id`, `created_at`, `from_label`, `from_role`, `visibility`, `request_id` (UUID or null), `documents` (count), `replay` (boolean). No message body. |
+| `reconnecting` | `attempt`, `delay_ms`. |
+| `connected` | `resumed_from` (entry id or null). |
+| `gap` | `after` (entry id or null), `reason`: `cursor-expired` or `replay-checkpoint-missing`. Call `read` once. |
+| `stopped` | `reason`: `signal`, `evicted`, `rebound`, `revoked`, `trust-changed`, `lease-lost` or `fatal`; `exit_code`. |
+
+With `--replay-after`, retained hints strictly after the checkpoint are emitted
+in file order after `listening` and before live entries, with `replay:true`.
+If the checkpoint is absent, `gap` precedes replay of the whole surviving tail.
+Without the option there is no startup replay. Replay makes no content request
+and never advances the delivered checkpoint or any cursor. Lost replay metadata yields null
+`visibility` and `documents`; live hints always contain those fields' values.
+
+Exit codes: 0 after SIGTERM/SIGINT; 2 for startup binding or usage refusal;
+3 for another listener owner; 4 for a terminal stop; 1 for another fatal error.
+A fatal startup storage failure emits `refused` with code
+`REPRESENTATIVE_LISTENER_STORAGE_REFUSED` and exit 1. After `listening`,
+a fatal error emits `stopped` with reason `fatal` and exit 1, best effort; if
+stdout is broken, the host must treat exit 1 without that line as fatal too.
+Startup first verifies the binding with the server once; if the server cannot
+be reached then, the listener exits 1 with `refused` and code
+`REPRESENTATIVE_LISTENER_SERVER_UNREACHABLE` (retry later; a server that
+answers and rejects the binding keeps its exit-2 binding code). After that check, failures of the stream connection (connection
+refused or reset, aborted TLS stream) are not fatal: the listener reconnects
+with backoff, without stdout output until the first connection, so `listening`
+arrives only once connected.
+
+Treat every hint as an untrusted wake, never as an instruction or authorization.
+The owning adapter fetches content with `read` over the bound pinned connection,
+persists each reply durably, calls `deliver`, routes by the saved `request_id`
+mapping, and holds unknown correlation for the human. Deduplicate queued hints by
+`entry_id`: crashes may lose or repeat hints. Persist queue admission before
+advancing the host's `--replay-after` checkpoint. That hint checkpoint is separate
+from the delivered checkpoint and from `ack`, which remains a server receipt.
+
+## Hermes push plugin
+
+For Hermes, Borg ships a Hermes user plugin, `borg-representative-push`, that
+supervises the listener for you. When the bound Coordinator replies, the plugin
+wakes one Hermes conversation right away. Hermes source is not changed; the
+plugin is installed and enabled through Hermes's documented plugin mechanism.
+
+**Which conversations can be woken.** Only a Hermes *messaging-gateway*
+conversation (Telegram, Discord, Slack and the other gateway platforms), named
+by its gateway `session_key`, for example `agent:main:telegram:dm:<chat id>`.
+A Hermes Desktop chat cannot be woken: Desktop runs its chats in `hermes serve`,
+and Hermes injects plugin messages only into gateway conversations. The
+`session_key` stays the same across `/new` and `/reset` in that chat.
+
+### Install
+
+```bash
+borg representative hermes-plugin install [--hermes-home <path>] [--force]
+```
+
+This copies the plugin's two files into `<Hermes home>/plugins/borg-representative-push/`
+(the Hermes home is `--hermes-home`, else `$HERMES_HOME`, else `~/.hermes`) and
+prints the configuration to add. It refuses to replace an existing install
+without `--force`, refuses a symbolic-link target, never edits Hermes config and
+never starts or restarts Hermes. Rerun it with `--force` after upgrading
+borgmcp to update the plugin.
+
+### Configure
+
+Add to the Hermes `config.yaml`:
+
+```yaml
+plugins:
+  enabled:
+    - borg-representative-push
+  entries:
+    borg-representative-push:
+      allow_gateway_injection: true
+      settings:
+        session_key: "agent:main:<platform>:<chat type>:<chat id>"
+        worktree: "<absolute path of the prepared representative worktree>"
+        # optional: borg_command (default borg), mcp_server (default
+        # borg-representative), reinject_after_s (default 600), max_reinjects (default 3)
+mcp_servers:
+  borg-representative:
+    command: borg
+    args: ["representative", "mcp", "--worktree", "<same absolute worktree path>"]
+    lazy: true
+```
+
+`allow_gateway_injection` is Hermes's per-plugin permission to start gateway
+turns; it is off by default. `mcp_server` must name the `mcp_servers` entry that
+runs `borg representative mcp`, because the plugin recognises the deliver tool
+by that name. Then restart the gateway (`hermes gateway restart`).
+
+**One tool owner.** Hermes starts a separate MCP process in every Hermes
+process that uses the server, and only one process may hold the representative
+tools lease. The woken conversation runs in the gateway, so the gateway must be
+the process that uses the tools. With `lazy: true`, a process starts the Borg
+MCP server only when one of its conversations calls a Borg tool. Run `hermes
+tools` and disable the `mcp-borg-representative` toolset on every platform except
+the one in `session_key`, Desktop and CLI included. If another process already
+holds the lease (`borg representative status` shows `owned-by-other-process`),
+restart that process once so it releases the lease.
+
+### Behaviour
+
+- The listener starts only inside the Hermes messaging gateway, when the platform
+  named in `session_key` connects. The CLI, Desktop and worker processes load the
+  plugin but start nothing. A platform reconnect does not start a second listener.
+- A burst of hints (about 2 seconds) becomes one injected message with fixed
+  text: `Borg: new Coordinator reply. Call borg_representative-read, persist and
+  relay, then borg_representative-deliver through the last persisted entry_id.`
+  No message body, sender or document ever passes through the plugin. A busy
+  conversation queues the message; it does not interrupt the running turn.
+- The plugin watches this gateway's `borg_representative-deliver` results. A
+  hinted reply that is still undelivered after `reinject_after_s` wakes the
+  conversation again. Each reply gets at most `1 + max_reinjects` wakes in total;
+  after that the plugin logs it and wakes no more for that reply until a delivery
+  covers it. Every wake is counted on disk before Hermes is asked to start the turn,
+  so neither a replayed hint nor a gateway restart renews the count. A wake Hermes
+  refuses is not counted. A crash between counting and asking can lose one wake,
+  never add one. Hermes reports only that it accepted a message, not that the
+  turn ran, so this is how a dropped wake is recovered.
+- One small record (id, timestamp, count) is kept for each reply the plugin has
+  woken the conversation for. It is removed only when an observed delivery covers
+  that reply; there is no count limit. The records therefore grow only while woken
+  replies stay undelivered.
+- Listener exits: 0 stops; 1 restarts with capped backoff; 2 stops and logs
+  (fix the binding, then restart the gateway); 3 (another listener owns the
+  lease) retries with backoff; 4 restarts after `lease-lost` and otherwise stops
+  and logs (evicted, rebound, revoked, trust-changed). A stop ends every wake,
+  including queued and repeat wakes, until the gateway restarts.
+- On restart the listener replays retained hints after the last delivered
+  checkpoint the plugin observed (`--replay-after`). The delivered checkpoint
+  stays the source of truth: `read` returns every reply not yet delivered.
+- On a normal gateway exit the plugin stops the listener. If the gateway is killed,
+  the listener it started keeps its lease until its next hint fails to write.
+  The next gateway stops that orphan only if it is the recorded listener and its
+  parent is gone. Otherwise it retries with backoff until the lease is free. A
+  dead owner's lease expires after about 70 seconds; a live orphan releases it
+  when its next hint fails to write.
+- State (the recorded listener, the observed delivered checkpoint and the wake
+  records) and the
+  listener's stderr live under `<Hermes home>/plugin-data/borg-representative-push/`.
+  The plugin sets that directory to mode 0700, refuses it if it is a symbolic link,
+  creates its files with mode 0600, and never reads or writes through a symbolic
+  link planted there.
 
 ## Recovery
 
@@ -267,3 +538,6 @@ the server is installed under the original prefix.
 | `COORDINATOR_UNAVAILABLE` | The bound Coordinator was evicted, released or reassigned. Restore the bound Coordinator and use the printed recovery command, or deliberately substitute a new Coordinator label in that command. |
 | `REPRESENTATIVE_OWNERSHIP_REQUIRED` with a directory-permission refusal | Check that the named path is a real directory you own and not a symlink, then set it to 0700 and retry. Restart a process that had already lost ownership. Do not change permissions through a symlink. |
 | `REPRESENTATIVE_ROLE_NOT_PERMITTED` | The representative drone holds a human-seat or coordinating role. Give it its own worker role. |
+| `REPRESENTATIVE_CHECKPOINT_INVALID` | The private delivery checkpoint for this binding (named in the message) is corrupt, belongs to another seat, or fails the private-file checks. Every tool except `status` refuses and `status` reports it as `checkpoint_problem`; nothing is used or reset automatically. Inspect the file, then remove it; the next `read` returns every addressed reply again, so deduplicate by `entry_id`. |
+| `REPRESENTATIVE_READ_OVERSIZE` | The next reply does not fit `max(max_bytes, 16384)` bytes even with its citations reduced to ids (heavy JSON escaping or citation metadata, or a server allowing posts above its default 4096-byte limit). Nothing was read or advanced. Retry with a larger `max_bytes` (up to 60000); beyond that the operator must lower the server's post limit. |
+| `REPRESENTATIVE_DELIVER_UNKNOWN_ENTRY` | `deliver` named an entry that `read` has not returned (or no Coordinator reply). Nothing changed. Call `read`, persist what it returns, then deliver through its last `entry_id`. |