@bom0792/herdgate 0.9.28 → 0.9.31
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/herdr/SKILL.md +81 -0
- package/.agents/skills/herdr/herdr_pi.py +272 -0
- package/.agents/skills/herdr/profiles.example.toml +18 -0
- package/.agents/skills/herdr-advisor/SKILL.md +12 -0
- package/.agents/skills/herdr-committee/SKILL.md +13 -0
- package/.agents/skills/herdr-handoff/SKILL.md +19 -0
- package/.agents/skills/herdr-help/SKILL.md +14 -0
- package/.agents/skills/herdr-plugin/SKILL.md +21 -0
- package/DEPLOY.md +341 -0
- package/DESIGN.md +216 -0
- package/README.md +143 -52
- package/bin/herdgate.mjs +14 -2
- package/bin/home-tui.mjs +2 -0
- package/bin/pi-skills.mjs +99 -0
- package/dist-readme.md +27 -6
- package/package.json +7 -2
- package/plugin/server/ssh.mjs +12 -4
- package/scripts/bench-mobile-tui.mjs +246 -0
- package/scripts/install-pi-skills.mjs +4 -0
- package/src/host/mobile-tui.ts +11 -1
- package/windows/install-cli.mjs +84 -18
- package/windows/setup-tui.mjs +19 -2
package/DEPLOY.md
ADDED
|
@@ -0,0 +1,341 @@
|
|
|
1
|
+
# Deploying herdgate publicly (cloudflared, this machine)
|
|
2
|
+
|
|
3
|
+
The cloud binds `127.0.0.1` and is published through a Cloudflare tunnel. Two shapes:
|
|
4
|
+
|
|
5
|
+
## A. Quick tunnel (no DNS setup, throwaway URL)
|
|
6
|
+
|
|
7
|
+
```sh
|
|
8
|
+
# terminal 1 — the cloud
|
|
9
|
+
cd /home/bom/source/herdgate
|
|
10
|
+
export HERDGATE_ACCOUNT_TOKEN=$(openssl rand -base64 32)
|
|
11
|
+
export HERDGATE_STATE=$HOME/.herdgate/state.json
|
|
12
|
+
npm start
|
|
13
|
+
|
|
14
|
+
# terminal 2 — a trycloudflare URL (random, changes every run)
|
|
15
|
+
cloudflared tunnel --url http://127.0.0.1:8787
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
Use the printed `https://<random>.trycloudflare.com` as the public URL. Nothing to configure; fine for testing.
|
|
19
|
+
|
|
20
|
+
## B. Named tunnel + your domain (stable hostname)
|
|
21
|
+
|
|
22
|
+
```sh
|
|
23
|
+
cloudflared tunnel login # once, picks a zone
|
|
24
|
+
cloudflared tunnel create herdgate
|
|
25
|
+
cloudflared tunnel route dns herdgate herd.example.com
|
|
26
|
+
|
|
27
|
+
cat > ~/.cloudflared/herdgate.yml <<'YML'
|
|
28
|
+
tunnel: herdgate
|
|
29
|
+
credentials-file: /home/bom/.cloudflared/<TUNNEL-ID>.json
|
|
30
|
+
ingress:
|
|
31
|
+
- hostname: herd.example.com
|
|
32
|
+
service: http://127.0.0.1:8787
|
|
33
|
+
- service: http_status:404
|
|
34
|
+
YML
|
|
35
|
+
|
|
36
|
+
cloudflared tunnel run --config ~/.cloudflared/herdgate.yml
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
Then start the cloud with the hostname pinned:
|
|
40
|
+
|
|
41
|
+
```sh
|
|
42
|
+
export HERDGATE_ACCOUNT_TOKEN=$(openssl rand -base64 32)
|
|
43
|
+
export HERDGATE_STATE=$HOME/.herdgate/state.json
|
|
44
|
+
export HERDGATE_PUBLIC_HOST=herd.example.com
|
|
45
|
+
export HERDGATE_PUBLIC_URL=https://herd.example.com
|
|
46
|
+
npm start
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
`HERDGATE_PUBLIC_HOST` makes the gateway answer **only** that Host (421 otherwise); `HERDGATE_PUBLIC_URL` builds
|
|
50
|
+
the pairing links shown to hosts.
|
|
51
|
+
|
|
52
|
+
### Funnel vs Tunnel
|
|
53
|
+
|
|
54
|
+
- **Tunnel** (above): the hostname resolves for everyone, but traffic only flows for this tunnel — which is
|
|
55
|
+
what we want, because herdgate authenticates every request and socket itself (hello-first tokens). Use this.
|
|
56
|
+
- **Funnel** (`cloudflared tunnel --url … --hello-world` variants / funnel ingress) would additionally let
|
|
57
|
+
arbitrary internet clients terminate at your origin without a CF account — herdgate's gate is the only
|
|
58
|
+
protection then. It works with the same code (the gate never trusts CF headers), but adds no value and more
|
|
59
|
+
exposure. Not recommended.
|
|
60
|
+
|
|
61
|
+
## Rehearse before going live (edge-sim)
|
|
62
|
+
|
|
63
|
+
`npm run edge` runs a local Cloudflare-edge simulator: TLS termination + `Host` rewrite +
|
|
64
|
+
WebSocket Upgrade splice — byte-for-byte the origin-side job cloudflared does. It lets you
|
|
65
|
+
exercise the public contract (`expectedHost` pinning → 421 on wrong host, the PWA over
|
|
66
|
+
TLS, both wss legs) on this machine, with a throwaway dev cert:
|
|
67
|
+
|
|
68
|
+
```sh
|
|
69
|
+
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 \
|
|
70
|
+
-keyout /tmp/edge-key.pem -out /tmp/edge-cert.pem -days 2 -nodes -subj "/CN=127.0.0.1"
|
|
71
|
+
|
|
72
|
+
HERDGATE_PUBLIC_HOST=gate.local HERDGATE_SSH_PORT=0 npm start # terminal 1
|
|
73
|
+
HERDGATE_TARGET_PORT=8787 HERDGATE_PINNED_HOST=gate.local \
|
|
74
|
+
HERDGATE_EDGE_CERT=/tmp/edge-cert.pem HERDGATE_EDGE_KEY=/tmp/edge-key.pem \
|
|
75
|
+
npm run edge # terminal 2
|
|
76
|
+
# → https://127.0.0.1:<port> (trust the cert or pass --ignore-certificate-errors)
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
Verified by `test/edge.test.ts` and a full headless-Chrome run: sign in → `online` →
|
|
80
|
+
machine online → pair read → live Herdr view + SSH hint, all over TLS with both wss legs
|
|
81
|
+
through the splice.
|
|
82
|
+
|
|
83
|
+
### A note on `trycloudflare.com` quick tunnels
|
|
84
|
+
|
|
85
|
+
On this machine, quick tunnels **register** but the edge answers **404** for the
|
|
86
|
+
trycloudflare hostname from both IPv4 and IPv6 — the free-for-all hostname space is
|
|
87
|
+
blocked somewhere between this machine and Cloudflare. **Named tunnels on your own zone
|
|
88
|
+
do not have this problem**: `gate.phuongnb.io.vn` (tunnel `herdgate`, id `dfc578ef`) is
|
|
89
|
+
live — PWA + CSP through the edge, owner + host wss legs through the edge, pairing and
|
|
90
|
+
the live Herdr view all verified in headless Chrome with the real Cloudflare cert (no
|
|
91
|
+
`--ignore-certificate-errors`). The live recipe on this machine:
|
|
92
|
+
|
|
93
|
+
```sh
|
|
94
|
+
cloudflared tunnel create herdgate
|
|
95
|
+
cloudflared tunnel route dns -f <tunnel-uuid> gate.phuongnb.io.vn # -f overwrites; the
|
|
96
|
+
# bare form once logged the WRONG tunnel id (fell back to config.yml) — always pass
|
|
97
|
+
# the UUID and check the log line says your tunnel id
|
|
98
|
+
# ~/.cloudflared/herdgate.yml → ingress gate.phuongnb.io.vn → http://127.0.0.1:8833
|
|
99
|
+
HERDGATE_PUBLIC_HOST=gate.phuongnb.io.vn HERDGATE_PORT=8833 HERDGATE_SSH_PORT=2299 npm start
|
|
100
|
+
cloudflared --config ~/.cloudflared/herdgate.yml tunnel run herdgate
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Gotchas hit along the way: port 8787 was already taken by the hermes webui (the old
|
|
104
|
+
`config.yml` routes `hermes.phuongnb.io.vn` there — an ingress clash would silently serve
|
|
105
|
+
the wrong app through our hostname), and running the gateway script without `&` in a
|
|
106
|
+
shell helper blocks forever. Local `curl http://127.0.0.1:8833/` returns **421** by
|
|
107
|
+
design — the gateway pins `Host: gate.phuongnb.io.vn`; that is the pinning contract
|
|
108
|
+
working, not a bug.
|
|
109
|
+
|
|
110
|
+
### The SSH leg — passwordless logins (publickey)
|
|
111
|
+
|
|
112
|
+
`hc machine add` shells out to the system `ssh`, which prefers keys and the agent and
|
|
113
|
+
refuses to answer prompts when it has no terminal. A device whose machine should add
|
|
114
|
+
it non-interactively registers that machine's public key first:
|
|
115
|
+
|
|
116
|
+
1. In the PWA, open the host's room → **SSH for the Herdr app**, select the exact device ID,
|
|
117
|
+
and add the controlling machine's public key. For Herdr Mobile, add the phone's public key to
|
|
118
|
+
its device ID instead. Do not edit the publisher's `authorized_keys` or copy a private key.
|
|
119
|
+
2. Confirm the fingerprint presented by the client matches the Gate's known SSH host fingerprint.
|
|
120
|
+
Pin that key; the TCP edge does not present the VPS admin SSH key.
|
|
121
|
+
3. Use the room's device ID as the SSH username. For example, a Core client can run
|
|
122
|
+
`hc machine add d_device@ssh.example.com:2299`. On Mobile, use the reachable TCP edge
|
|
123
|
+
hostname or IPv4 address and port in the machine settings. Password auth with the paired
|
|
124
|
+
device secret also works.
|
|
125
|
+
|
|
126
|
+
The REST surface (same two identities as `/v1/device/push`: the device's own secret,
|
|
127
|
+
or the account token): `GET/POST/DELETE /v1/device/sshkey`. Ed25519, RSA and ECDSA
|
|
128
|
+
(nistp256/384/521) keys are accepted; signatures are verified per RFC 4252 §7.
|
|
129
|
+
|
|
130
|
+
## The SSH leg is not a Cloudflare-TCP leg — three ways in
|
|
131
|
+
|
|
132
|
+
Phones speak plain SSH to `host:port`; Cloudflare Tunnel only carries TCP for clients
|
|
133
|
+
running `cloudflared access`, which the phone does not. Reachability options, in order
|
|
134
|
+
of setup cost:
|
|
135
|
+
|
|
136
|
+
1. **LAN / same network** — `HERDGATE_SSH_BIND=0.0.0.0`, phones dial the machine.
|
|
137
|
+
2. **IPv6 AAAA** (this machine's setup) — a DNS-only `AAAA` record to the gate
|
|
138
|
+
machine's global IPv6; `~/.local/bin/herdgate-ssh-ddns` + the `herdgate-ddns`
|
|
139
|
+
timer keep it current (needs a Cloudflare API token + `ufw allow`).
|
|
140
|
+
3. **Tokenless TCP edge (recommended for a VPS)** — one TCP connection from Herdr
|
|
141
|
+
Mobile maps to one WSS connection to the public gate. The edge never terminates
|
|
142
|
+
SSH. Use the existing Herdgate SSH host key on the phone, not the VPS admin key.
|
|
143
|
+
|
|
144
|
+
### Deploy the tokenless TCP edge on a VPS
|
|
145
|
+
|
|
146
|
+
Use a public IPv4 VPS with Node 22+ installed system-wide. Build the standalone
|
|
147
|
+
JavaScript on a machine with TypeScript-enabled Node 22.18+ before copying it to
|
|
148
|
+
the VPS (Ubuntu's packaged Node may not support running `.ts` directly):
|
|
149
|
+
|
|
150
|
+
```sh
|
|
151
|
+
node --input-type=module -e 'import { readFileSync } from "node:fs"; import { stripTypeScriptTypes } from "node:module"; process.stdout.write(stripTypeScriptTypes(readFileSync("src/cloud/ssh-tcp-edge.ts", "utf8")))' > /tmp/ssh-tcp-edge.mjs
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
Copy the output to `/opt/herdgate-tcp-edge/ssh-tcp-edge.mjs` and install only the
|
|
155
|
+
locked `ws` version there (`npm install --save-exact --omit=dev --ignore-scripts --prefix /opt/herdgate-tcp-edge ws@8.23.0`).
|
|
156
|
+
Install `templates/herdgate-tcp-edge.service` into `/etc/systemd/system/` (adjust
|
|
157
|
+
`ExecStart` if Node is not at `/usr/bin/node`), and put this in
|
|
158
|
+
`/etc/herdgate-tcp-edge.env`:
|
|
159
|
+
|
|
160
|
+
```dotenv
|
|
161
|
+
HERDGATE_URL=https://gate.phuongnb.io.vn
|
|
162
|
+
HERDGATE_TCP_EDGE_BIND=0.0.0.0
|
|
163
|
+
HERDGATE_TCP_EDGE_PORT=2299
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
Do **not** copy any Herdgate credential or SSH host key to the VPS. Enable the unit
|
|
167
|
+
with `sudo systemctl enable --now herdgate-tcp-edge` and allow TCP 2299 in the VPS
|
|
168
|
+
provider and OS firewalls. Restrict the VPS's separate admin SSH access. The edge
|
|
169
|
+
caps concurrent sessions at 3 per IP and 6 total by default; the gate's own tunnel
|
|
170
|
+
caps still apply, and the gate may see several clients as one source behind a proxy.
|
|
171
|
+
|
|
172
|
+
Test `VPS_IP:2299` with the existing mobile username and key over cellular before
|
|
173
|
+
changing DNS. Confirm the host key matches the gate's pinned fingerprint. To switch
|
|
174
|
+
`ssh.phuongnb.io.vn`, add a DNS-only `A` record for the VPS and remove the old `AAAA`
|
|
175
|
+
unless it also points to a verified edge. Disable the old home AAAA DDNS timer if
|
|
176
|
+
using that hostname for the VPS. Keep the HTTPS hostname unchanged.
|
|
177
|
+
|
|
178
|
+
### Legacy authenticated SSH bridge
|
|
179
|
+
|
|
180
|
+
The older SSH bridge is also available, but requires an account owner token on
|
|
181
|
+
its VPS and shares one upstream connection:
|
|
182
|
+
|
|
183
|
+
```sh
|
|
184
|
+
# on the VPS (Node ≥ 22, one file, no dependencies):
|
|
185
|
+
node ssh-bridge.ts # env: HERDGATE_URL, HERDGATE_BRIDGE_TOKEN (the account token),
|
|
186
|
+
# HERDGATE_BRIDGE_BIND/PORT (default 0.0.0.0:2299),
|
|
187
|
+
# HERDGATE_BRIDGE_PUBLIC_HOST/PORT — advertise what phones dial
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
The bridge dials the cloud over an OUTBOUND WebSocket and pipes raw SSH bytes both
|
|
191
|
+
ways; the cloud feeds them into its sshd through ssh2's `injectSocket`. Device-key
|
|
192
|
+
auth, act-scope enforcement, the read-only allowlist and brute-force strikes all stay
|
|
193
|
+
in the cloud. The legacy bridge credential is the account (owner) token. While a bridge with an advertised address is connected, the
|
|
194
|
+
device hello's `ssh {host, port}` points phones at it; disconnect it and the hello
|
|
195
|
+
falls back to the gate's own address. Verified end to end through the real Cloudflare
|
|
196
|
+
tunnel: pair over wss → SSH the bridge → `hello` (32 methods), `server.status`, and
|
|
197
|
+
`terminal.send` refused with the act-scope error.
|
|
198
|
+
|
|
199
|
+
The public HTTP/WSS surface continues to go through the tunnel.
|
|
200
|
+
|
|
201
|
+
### When the publisher and gateway run on the same machine
|
|
202
|
+
|
|
203
|
+
Set `HERDGATE_AGENT_LOOPBACK_PORT=8833` in the publisher's `herdgate.env`
|
|
204
|
+
(use the gateway's actual port). Keep `HERDGATE_URL` set to the canonical public
|
|
205
|
+
URL. The publisher dials `127.0.0.1` with the public `Host` header, so gateway host
|
|
206
|
+
validation and room-token authentication still apply. This opt-in mode uses the
|
|
207
|
+
existing `ws` dependency from the npm installation; the standalone plugin uses
|
|
208
|
+
Node's native WebSocket by default. Restart only the publisher. Remove the line
|
|
209
|
+
to restore the public transport. No client SSH configuration changes are needed.
|
|
210
|
+
|
|
211
|
+
For the standalone Linux Mobile TUI publisher (`src/host/mobile-tui.ts`),
|
|
212
|
+
set `HERDGATE_AGENT_LOOPBACK_PORT` in its **process environment** (for systemd,
|
|
213
|
+
a drop-in with `[Service]` and `Environment=HERDGATE_AGENT_LOOPBACK_PORT=8833`,
|
|
214
|
+
using the actual local port). Keep the public `HERDGATE_URL`, the existing room
|
|
215
|
+
and the device scope. Only enable this when the Gate is on the same machine.
|
|
216
|
+
Restart that Mobile publisher alone in a maintenance window and verify its
|
|
217
|
+
journal reports `cloud hello: … via loopback 127.0.0.1:…`. Reconnect Mobile
|
|
218
|
+
and test the existing entry. Remove the drop-in setting and restart the same
|
|
219
|
+
publisher to roll back. This does not change the public WSS leg on the VPS,
|
|
220
|
+
the SSH host key, or the Herdr server.
|
|
221
|
+
|
|
222
|
+
## Measure Mobile latency
|
|
223
|
+
|
|
224
|
+
For a local baseline with an isolated Herdr 0.9.5 server, pane, Gate and TCP
|
|
225
|
+
edge (no live rooms or keys), run:
|
|
226
|
+
|
|
227
|
+
```sh
|
|
228
|
+
TEST_HERDR_BIN="$(command -v herdr)" node scripts/bench-mobile-tui.mjs 25
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
Compare native Herdr, the Core adapter over local pipes, and the full Mobile
|
|
232
|
+
transport over loopback. The script reports key-to-ANSI, Send-to-ACK and
|
|
233
|
+
Send-to-ANSI separately; attach is measured separately from steady-state
|
|
234
|
+
input. This does not include cellular RTT or the official app's rendering.
|
|
235
|
+
An observe/read-only TUI attachment instead polls every 1.25 seconds; its
|
|
236
|
+
refresh interval is not comparable to the native interactive stream.
|
|
237
|
+
|
|
238
|
+
## Keep it running
|
|
239
|
+
|
|
240
|
+
On this machine (Linux + systemd) a user unit keeps the gate up; the tunnel is a second unit:
|
|
241
|
+
|
|
242
|
+
```sh
|
|
243
|
+
mkdir -p ~/.config/systemd/user
|
|
244
|
+
cat > ~/.config/systemd/user/herdgate.service <<'UNIT'
|
|
245
|
+
[Unit]
|
|
246
|
+
Description=herdgate cloud relay hub
|
|
247
|
+
After=network-online.target
|
|
248
|
+
|
|
249
|
+
[Service]
|
|
250
|
+
ExecStart=/home/bom/.nvm/versions/node/v22.23.1/bin/node /home/bom/source/herdgate/src/cloud/main.ts
|
|
251
|
+
Environment=HERDGATE_ACCOUNT_TOKEN=change-me
|
|
252
|
+
Environment=HERDGATE_PUBLIC_HOST=gate.phuongnb.io.vn
|
|
253
|
+
Environment=HERDGATE_PORT=8833
|
|
254
|
+
Environment=HERDGATE_STATE=/home/bom/.local/state/herdgate/state.json
|
|
255
|
+
WorkingDirectory=/home/bom/source/herdgate
|
|
256
|
+
Restart=always
|
|
257
|
+
RestartSec=3
|
|
258
|
+
|
|
259
|
+
[Install]
|
|
260
|
+
WantedBy=default.target
|
|
261
|
+
UNIT
|
|
262
|
+
systemctl --user daemon-reload && systemctl --user enable --now herdgate
|
|
263
|
+
# tunnel: cloudflared --config ~/.cloudflared/herdgate.yml tunnel run herdgate
|
|
264
|
+
# (a second unit wrapping that line, same shape)
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
Other platforms:
|
|
268
|
+
|
|
269
|
+
```sh
|
|
270
|
+
# launchd agent sketch (adjust paths); or use tmux/systemd-in-wsl as you prefer
|
|
271
|
+
cat > ~/Library/LaunchAgents/dev.herdgate.cloud.plist <<'PLIST'
|
|
272
|
+
<?xml version="1.0" encoding="UTF-8"?>
|
|
273
|
+
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
|
274
|
+
<plist version="1.0"><dict>
|
|
275
|
+
<key>Label</key><string>dev.herdgate.cloud</string>
|
|
276
|
+
<key>ProgramArguments</key><array>
|
|
277
|
+
<string>/Users/bom/.nvm/versions/node/v22.23.1/bin/node</string>
|
|
278
|
+
<string>/home/bom/source/herdgate/src/cloud/main.ts</string>
|
|
279
|
+
</array>
|
|
280
|
+
<key>RunAtLoad</key><true/>
|
|
281
|
+
<key>KeepAlive</key><true/>
|
|
282
|
+
<key>EnvironmentVariables</key><dict>
|
|
283
|
+
<key>HERDGATE_ACCOUNT_TOKEN</key><string>…</string>
|
|
284
|
+
<key>HERDGATE_STATE</key><string>/Users/bom/.herdgate/state.json</string>
|
|
285
|
+
<key>HERDGATE_PUBLIC_HOST</key><string>herd.example.com</string>
|
|
286
|
+
<key>HERDGATE_PUBLIC_URL</key><string>https://herd.example.com</string>
|
|
287
|
+
</dict>
|
|
288
|
+
</dict></plist>
|
|
289
|
+
PLIST
|
|
290
|
+
launchctl load ~/Library/LaunchAgents/dev.herdgate.cloud.plist
|
|
291
|
+
```
|
|
292
|
+
|
|
293
|
+
(The plist above is a sketch — fill the token, keep `node` on your PATH or use the absolute binary.)
|
|
294
|
+
|
|
295
|
+
## Migrate the Linux TUI publisher
|
|
296
|
+
|
|
297
|
+
On the host, use Node 22+ and run as the user who owns the running Herdr TUI
|
|
298
|
+
and the systemd user unit. Verify publisher/remote access before starting and
|
|
299
|
+
pick a window when a short remote disconnection is acceptable:
|
|
300
|
+
|
|
301
|
+
```sh
|
|
302
|
+
systemctl --user is-active herdgate-home-tui.service
|
|
303
|
+
herdr status server --json
|
|
304
|
+
npx --yes @bom0792/herdgate@latest migrate
|
|
305
|
+
npx --yes @bom0792/herdgate@latest status
|
|
306
|
+
```
|
|
307
|
+
|
|
308
|
+
For a new host with no previous publisher unit, use `herdgate setup` after
|
|
309
|
+
pairing instead. Migration only adopts the known legacy Herdgate home unit.
|
|
310
|
+
It stages the publisher under `${XDG_DATA_HOME:-$HOME/.local/share}/herdgate/home-tui/releases`,
|
|
311
|
+
reuses the existing room config, and checks for a Gate hello from the new
|
|
312
|
+
systemd invocation. If the new publisher cannot connect, it attempts to
|
|
313
|
+
restore the previous service. It does not stop or update the Herdr server.
|
|
314
|
+
If Herdr's executable or socket is not at the default path, pass
|
|
315
|
+
`HERDR_BIN_PATH` and/or `HERDR_SOCKET_PATH` when running setup or update.
|
|
316
|
+
|
|
317
|
+
After success, reconnect from the remote Herdr TUI and confirm the existing
|
|
318
|
+
session is reachable. If migration fails, check
|
|
319
|
+
`journalctl --user -u herdgate-home-tui.service` locally; do not re-pair or
|
|
320
|
+
share tokens or keys. Use `herdgate update` for later main-publisher updates.
|
|
321
|
+
|
|
322
|
+
### Upgrade boundaries
|
|
323
|
+
|
|
324
|
+
- The publisher CLI updates the main TUI service only. It does not deploy
|
|
325
|
+
the Gate, web/PWA, Cloudflare tunnel, VPS TCP edge, or independent Mobile
|
|
326
|
+
read/control services. Maintain and verify those separately.
|
|
327
|
+
- Herdr Mobile Core, Mobile TUI read and Mobile TUI control are different
|
|
328
|
+
targets. Preserve the room, SSH key and scope of each entry. Do not turn a
|
|
329
|
+
read device into an interactive device just to resolve a connection issue.
|
|
330
|
+
- If several phone entries use the same public IP and a new one cannot
|
|
331
|
+
connect, close unused entries and retry. Check the TCP edge logs before
|
|
332
|
+
attributing the failure to the connection limits.
|
|
333
|
+
|
|
334
|
+
## Operational notes
|
|
335
|
+
|
|
336
|
+
- Preserve the Gate state and SSH host key in protected backups. Deleting
|
|
337
|
+
state loses room/device registrations; replacing the host key invalidates
|
|
338
|
+
clients' pins.
|
|
339
|
+
- **Rotate the account token** by restarting with a new `HERDGATE_ACCOUNT_TOKEN` (devices must update it).
|
|
340
|
+
- **Logs** go to stderr (agent connect/pair events). Payloads are never logged — keep it that way.
|
|
341
|
+
- **Watch the exposure**: `curl -s https://host/healthz` public; everything else needs tokens.
|
package/DESIGN.md
ADDED
|
@@ -0,0 +1,216 @@
|
|
|
1
|
+
# herdgate — design
|
|
2
|
+
|
|
3
|
+
Architecture and trust boundaries. For installation and upgrade commands, see [README.md](README.md) and
|
|
4
|
+
[DEPLOY.md](DEPLOY.md).
|
|
5
|
+
|
|
6
|
+
## 1. What herdgate is
|
|
7
|
+
|
|
8
|
+
A relay hub between **host apps** and **devices** (browsers, PWAs). The first host target is official
|
|
9
|
+
[Herdr](https://herdr.dev). A herdgate-owned local adapter will use a verified upstream interface; the adapter
|
|
10
|
+
runs on the host and the agent connects outbound to the gateway. No fork, third-party plugin, or relay project
|
|
11
|
+
is required. The hub handles identity, routing, and presence; content and policy stay in the host adapter. Payloads crossing the cloud are **opaque by rule**: the cloud never parses, logs or
|
|
12
|
+
persists them.
|
|
13
|
+
|
|
14
|
+
Three deployment properties:
|
|
15
|
+
|
|
16
|
+
1. Hosts and devices both connect **outbound**; only the cloud is publicly reachable (via cloudflared on the
|
|
17
|
+
machine that runs it).
|
|
18
|
+
2. The cloud is the **identity provider**. No Cloudflare Access in front, no trusted proxy headers. If
|
|
19
|
+
`HERDGATE_PUBLIC_HOST` is set, the Host header is enforced (anti host-confusion) but nothing else about the
|
|
20
|
+
edge is trusted.
|
|
21
|
+
3. Single-owner v0: accounts come from config (env tokens). Multi-account signup is roadmap, not code.
|
|
22
|
+
|
|
23
|
+
## 2. Identity: accounts, rooms, devices
|
|
24
|
+
|
|
25
|
+
| Layer | Who | Credential | Minted by |
|
|
26
|
+
|---|---|---|---|
|
|
27
|
+
| account | a person | config token (v0) | operator (env) |
|
|
28
|
+
| room | one host app | room token (random 32B, stored **hashed**) | device-flow approval |
|
|
29
|
+
| device | one room-scoped browser or SSH client | device ID + device secret, or registered SSH public key for SSH auth | `pair.begin` and key registration |
|
|
30
|
+
|
|
31
|
+
- **Host provisioning (device flow).** The host mints a code (`POST /v1/agent/code`), a human approves it from a
|
|
32
|
+
logged-in device (`POST /v1/agent/approve`), the host polls `GET /v1/agent/code/:code/status` and receives the
|
|
33
|
+
room token exactly once. The host stores it; re-provisioning rotates it.
|
|
34
|
+
- **Device pairing.** A device hellos with its account token, then `pair.begin {roomId, name, scope}`.
|
|
35
|
+
`read` pairs immediately. `act` triggers `pair.request` **down to the host** — the human approves on the host
|
|
36
|
+
(approve-on-host), the cloud grants on `pair.response`. Timeout 60s → denied.
|
|
37
|
+
- **Scopes.** `read` = view; `act` = may cause effects. The cloud knows scopes and reports them
|
|
38
|
+
(`room.update`, `hello.devices`), but **enforcement of what an `act` payload may do is host-side** — the cloud
|
|
39
|
+
cannot see payloads. This split is deliberate (see §5).
|
|
40
|
+
|
|
41
|
+
## 3. Wire protocol (v1)
|
|
42
|
+
|
|
43
|
+
One JSON object per WS text frame: `{v:1, t:"…", …}`. Unknown `v` → close. Legs:
|
|
44
|
+
|
|
45
|
+
- `GET /v1/agent/ws` — host leg. `hello {roomToken, info}` first (10s deadline, else close 4001). Then
|
|
46
|
+
`frame {device, payload}` ⇄, `presence`, `pair.request`/`pair.response`, `ping`/`pong`, `room.update`.
|
|
47
|
+
One live agent per room: a second connection supersedes the first (4000).
|
|
48
|
+
- `GET /v1/device/ws` — device leg. `hello {accountToken | device+deviceSecret, resume?}` first. Then `frame {payload}`
|
|
49
|
+
(routed to the paired room's host), `pair.begin` → `pair.granted`/`pair.denied`, `presence`, `ping`/`pong`.
|
|
50
|
+
|
|
51
|
+
### Room streams and resume (v0.6.0)
|
|
52
|
+
|
|
53
|
+
Every host→device message that matters while a phone is away (frames, presence) carries a
|
|
54
|
+
room-scoped `seq` and is buffered briefly (last 256 KiB / 120 s per room). Device hello
|
|
55
|
+
responses carry a `stream {gen, seq}` header; a reconnecting device adds
|
|
56
|
+
`resume {room, gen, seq}` to its hello and receives `resumed:true` plus the buffered tail
|
|
57
|
+
in order. `gen` changes on every gateway process start, so stale bookkeeping from a
|
|
58
|
+
previous life invalidates in one comparison — the device gets a fresh hello (`resumed:false`)
|
|
59
|
+
and re-requests state. A resume from before what was kept answers `stream.gap:true`
|
|
60
|
+
(replay what survived; the device re-requests its view). Frames with an `ssh.*` op are
|
|
61
|
+
**ephemeral**: sent raw, unstamped, never buffered — they belong to an SSH channel that
|
|
62
|
+
died with the drop, and replaying them into a new SSH session would corrupt its wire
|
|
63
|
+
protocol. `pair.granted` (carries the one-time device secret) is likewise never replayed.
|
|
64
|
+
|
|
65
|
+
HTTP: `GET /healthz`, `POST /v1/agent/code`, `POST /v1/agent/approve` (bearer account token),
|
|
66
|
+
`GET /v1/agent/code/:code/status`.
|
|
67
|
+
|
|
68
|
+
Limits: frames 256 KiB (`maxPayload`), 60 msg/s per socket (token bucket), hello within 10s, failed-hello
|
|
69
|
+
strikes per IP (5/min → 60s block), and WebSocket heartbeat pings (8s by default) with termination on silence.
|
|
70
|
+
|
|
71
|
+
## 4. State
|
|
72
|
+
|
|
73
|
+
`state.json` (atomic tmp+rename, 0600): rooms (token hashed) + devices. In-memory pending pairing codes never
|
|
74
|
+
touch disk. v0 loses open sockets on restart; the room stream resumes devices after a drop (§3), and the `gen`
|
|
75
|
+
header makes a restart a clean re-sync rather than a wrong replay.
|
|
76
|
+
|
|
77
|
+
## 5. Threat model — what the cloud promises
|
|
78
|
+
|
|
79
|
+
| Threat | Mitigation (v0) |
|
|
80
|
+
|---|---|
|
|
81
|
+
| Guessing a room token / account token | 256-bit random, hashed at rest, constant-time compare, failed-hello strikes |
|
|
82
|
+
| Brute-forcing pairing codes | 8-char no-lookalike alphabet, 10-min TTL, single retrieval, rate-limited mint per IP |
|
|
83
|
+
| A stolen `act` device | approve-on-host at pair time; host re-confirmation of dangerous ops is host policy |
|
|
84
|
+
| Hostile room claiming | codes bound to the approving account; rooms never cross accounts |
|
|
85
|
+
| DoS | per-IP blocks, per-socket rate, frame caps, supersede rule, unbounded fan-out is a bug |
|
|
86
|
+
| Cloud reading content | Gate routes application frames without interpreting them; only a negotiated Noise session protects that payload from the Gate. SSH terminates at the Gate. |
|
|
87
|
+
| Host confusion via cloudflared | optional `expectedHost` pin; nothing else about the edge is trusted |
|
|
88
|
+
|
|
89
|
+
Per-device secrets and Noise_XX payload sessions now exist. Noise applies to its own application payload
|
|
90
|
+
channel, not to SSH, which terminates at the Gate. Do not infer that all payloads use Noise or that the
|
|
91
|
+
single-owner deployment model provides multi-tenant isolation.
|
|
92
|
+
|
|
93
|
+
## 6. Roadmap
|
|
94
|
+
|
|
95
|
+
1. ~~**Per-device secrets**~~ — **done (v0.5.0)**: `pair.granted` returns a device secret once; hello uses it;
|
|
96
|
+
the account token stays in the dashboard only (and is never an SSH credential).
|
|
97
|
+
2. ~~**Resume**~~ — **done (v0.6.0)**: room streams with `seq`/`gen`, a bounded replay buffer, gap declaration;
|
|
98
|
+
see §3. Ciphertext-friendly by construction (the buffer stores opaque envelopes).
|
|
99
|
+
3. ~~**E2E payloads**~~ — **done (v0.7.0)**: Noise_XX (P-256/SHA-256/AES-256-GCM, WebCrypto-only so the
|
|
100
|
+
plugin and the PWA run one identical module) over the payload channel; the device initiates after
|
|
101
|
+
pairing, the host acks with `e2e.ready` (nothing encrypts before the ack — a frame racing the
|
|
102
|
+
handshake's last step would desync the counters forever), and a decrypt failure drops the session for
|
|
103
|
+
renegotiation. Host static key is TOFU: the device pins its fingerprint and screams if it changes.
|
|
104
|
+
See DESIGN §3 and `plugin/server/e2e.mjs`.
|
|
105
|
+
4. ~~**Official Herdr adapter**~~ — **done (v0.1.0)**, read-only. `src/host/herdr.ts` runs
|
|
106
|
+
`herdr api snapshot` (verified on Herdr 0.9.5, protocol 22) and maps `workspaces[]`, `tabs[]`, `panes[]`,
|
|
107
|
+
`agents[]` and the focused ids into a versioned view. Device ops are `view` and `ping`; every other op is
|
|
108
|
+
refused on the host with `read-only`. Next in this line: an `act` op set once per-op policy and audit exist.
|
|
109
|
+
5. **Device PWA** — shipped: pairing, multi-room dashboard, per-device SSH key management and read-only pane views.
|
|
110
|
+
5b. **SSH through the Gate** — shipped: per-device public keys, TUI bridge, real Core and the scoped Mobile
|
|
111
|
+
TUI adapter (see §8).
|
|
112
|
+
5c. **SSH without inbound publisher ports** — shipped: SSH over WSS for desktop clients and a tokenless
|
|
113
|
+
TCP edge for Mobile. The old owner-token SSH bridge is a separate legacy deployment.
|
|
114
|
+
6. ~~**Push wake**~~ — **done (v0.8.0)**: VAPID keys beside the state file; a paired device stores its web-push
|
|
115
|
+
subscription (`/v1/device/push`); the host sends `wake {device}` and the cloud delivers a silent,
|
|
116
|
+
payload-less push — but only when the device is not connected (a connected one is already live). The
|
|
117
|
+
PWA's service worker raises a notification only when no window is open.
|
|
118
|
+
7. Multi-account signup, quotas, Postgres state, audit log.
|
|
119
|
+
|
|
120
|
+
## 7. Host payload contract (herdgate.herdr.v1)
|
|
121
|
+
|
|
122
|
+
The cloud routes `payload` blind, so the host and its devices own their own schema. Current ops:
|
|
123
|
+
|
|
124
|
+
| Direction | Message | Reply |
|
|
125
|
+
|---|---|---|
|
|
126
|
+
| device → host | `{op:"view"}` | `{op:"view", view}` (lazy read if none yet) |
|
|
127
|
+
| device → host | `{op:"ping", n?}` | `{op:"pong", n, at}` |
|
|
128
|
+
| host → device | `{op:"view", view}` | pushed when the view changes, and to a newly paired device |
|
|
129
|
+
| host → device | `{op:"error", code, message}` | `read-only` for any other op; `host-unavailable` when Herdr cannot be read |
|
|
130
|
+
|
|
131
|
+
`view` = `{v:1, source:"herdr", herdrVersion, protocol, takenAt, focused{workspaceId,tabId,paneId},
|
|
132
|
+
workspaces[{id,label,number,tabs,panes,agentStatus,focused}], agents[{name,kind,status,workspaceId,tabId,paneId,cwd,title}]}`.
|
|
133
|
+
A new Herdr version that adds keys keeps working (every field is read defensively); a missing key degrades to an
|
|
134
|
+
empty/honest value rather than throwing.
|
|
135
|
+
|
|
136
|
+
The table describes the standalone metadata adapter. The TUI publisher also exposes read-only
|
|
137
|
+
`term.list` and `term.read` for the PWA. Browser payload operations do not grant terminal input;
|
|
138
|
+
SSH access is a separate channel with its own authorization.
|
|
139
|
+
|
|
140
|
+
## 8. How herdgate attaches to Herdr (official surface only)
|
|
141
|
+
|
|
142
|
+
Herdr is not forked, patched, or re-implemented. herdgate is a **Herdr plugin**: a directory with
|
|
143
|
+
`herdr-plugin.toml` whose commands Herdr validates, launches, and logs — the surface Herdr documents for
|
|
144
|
+
extensions (`docs/plugins.mdx`) and lists in the marketplace (`herdr-plugin` GitHub topic, `/tui/plugins/`).
|
|
145
|
+
|
|
146
|
+
| Manifest entry | Command | Why |
|
|
147
|
+
|---|---|---|
|
|
148
|
+
| `[[startup]]` | `node server/index.mjs serve` | Herdr runs it once after the socket API is ready; it spawns the long-lived publisher and exits, so a failed publisher cannot fail the Herdr server |
|
|
149
|
+
| `[[actions]]` | `pair`, `status`, `stop` | the human-facing operations, invoked as `herdr plugin action invoke dev.herdgate.host.pair` |
|
|
150
|
+
| `[[panes]]` | `node server/index.mjs pane` | a `split` pane inside Herdr showing honest publish state (light 10s poll) |
|
|
151
|
+
|
|
152
|
+
Two official surfaces are used, one per program: the **plugin** (§ above, `herdr-plugin.toml`) for the TUI, and
|
|
153
|
+
the **app mod** (§ below, `mods/<name>/`) for the desktop app.
|
|
154
|
+
|
|
155
|
+
The plugin uses Herdr's CLI for metadata and pane reads, and fixed Herdr bridge commands for remote TUI
|
|
156
|
+
connections. The interactive Mobile adapter additionally uses Herdr's native terminal session control.
|
|
157
|
+
These integrations use the running TUI server's configured executable and socket, not a second Herdr
|
|
158
|
+
server. Nothing patches Herdr.
|
|
159
|
+
|
|
160
|
+
Credentials go in `HERDR_PLUGIN_CONFIG_DIR`, not the managed plugin checkout; runtime state goes in
|
|
161
|
+
`HERDR_PLUGIN_STATE_DIR`. The standalone plugin uses Node's native `WebSocket`; the opt-in Linux Gate
|
|
162
|
+
loopback transport uses the bundled `ws` dependency.
|
|
163
|
+
|
|
164
|
+
Note on Herdr's moving parts: the **TUI** (`herdr`) exposes workspaces/tabs/panes/agents; **herdr-core** (`hc`)
|
|
165
|
+
is the separate runtime under the Herdr app and its iPhone/iPad/Android app, with a JSON-RPC API of its own.
|
|
166
|
+
The plugin attaches to the TUI's documented CLI.
|
|
167
|
+
|
|
168
|
+
### The Herdr app mod (`app-mod/herdgate/`)
|
|
169
|
+
|
|
170
|
+
The desktop app extends through **mods**, a different official surface (`docs/mods.md`,
|
|
171
|
+
`defaults/AGENTS.md`): `mods/<name>/` with `mod.json` + `main.js`, written in gpui-kit. A mod is trusted code
|
|
172
|
+
with `fetch`, `exec`, and unrestricted network — but **no Node globals** (`process`, `require`, `setTimeout`
|
|
173
|
+
are absent). Verified in a real window: `herdr-app check` passes, `mod.open` reports `overlays:["herdgate"]`
|
|
174
|
+
with `errors: []`, and a probe mod reached a live herdgate cloud with `fetch` (HTTP 200).
|
|
175
|
+
|
|
176
|
+
The mod and the TUI plugin share one publisher: the mod owns the pairing conversation (mint a code, poll,
|
|
177
|
+
write the token 0600) and shells out to `plugin/server/index.mjs serve` for the long-lived connection. One
|
|
178
|
+
implementation, two surfaces.
|
|
179
|
+
|
|
180
|
+
### SSH clients: TUI, real Core and the Mobile TUI adapter
|
|
181
|
+
|
|
182
|
+
The Gate (`src/cloud/sshd.ts`) terminates SSH and authenticates the device ID with its registered public
|
|
183
|
+
key or device secret, never with the owner token. Each exec is an ephemeral channel bound to the
|
|
184
|
+
authenticated device and room. The publisher dispatches supported commands to fixed argv, not a shell.
|
|
185
|
+
A VPS TCP edge, when used, carries raw SSH over WSS and holds no Gate credentials; the client verifies
|
|
186
|
+
and pins the Gate's SSH host key. SSH channels are not replayed or resumed after a disconnect.
|
|
187
|
+
|
|
188
|
+
| Target | Host scope enforcement |
|
|
189
|
+
|---|---|
|
|
190
|
+
| Existing Herdr TUI via the desktop TUI bridge | `act` only; the native two-way bridge is not a read filter. |
|
|
191
|
+
| Real `herdr-core` | A `read` device gets filtered plain JSON-RPC; framed hello is downgraded. Approved `act` gets byte-transparent Core RPC. |
|
|
192
|
+
| Existing TUI panes via the Mobile adapter | Read/observe attaches show a text snapshot; no input or resize. An explicitly configured device with matching `act` scope can use Send, keys, Interrupt and native attach with ANSI, input and resize. |
|
|
193
|
+
|
|
194
|
+
The Mobile adapter speaks the Core protocol subset needed for those operations and rejects unsupported
|
|
195
|
+
RPCs. It does not implement Core task lifecycle, templates or file operations. See
|
|
196
|
+
`plugin/server/mobile-tui-core.mjs` and `plugin/server/mobile-tui-stream.mjs`.
|
|
197
|
+
|
|
198
|
+
### Publisher releases and migration
|
|
199
|
+
|
|
200
|
+
The platform-specific publisher runtime is installed outside npm's cache. On Windows `herdgate install`
|
|
201
|
+
installs a per-user, Node-pinned command from npm. First setup also provisions it. Its `command` junction
|
|
202
|
+
is separate from the publisher's `current` junction, so publisher rollback does not remove the command.
|
|
203
|
+
User PATH registration is optional and never changes the machine-wide PATH. Migration stages and validates
|
|
204
|
+
first, then saves the existing device mode only after the new publisher connects; failure triggers a
|
|
205
|
+
rollback attempt. On Linux the CLI installs a user systemd unit, recognizes only the known legacy home
|
|
206
|
+
unit for adoption, and checks the new invocation's Gate hello. Both preserve the room config and the
|
|
207
|
+
running Herdr server. The Gate, TCP edge, and standalone Linux Mobile publishers have independent
|
|
208
|
+
lifecycles.
|
|
209
|
+
|
|
210
|
+
## 9. Herdr-relay comparison (why a new project)
|
|
211
|
+
|
|
212
|
+
herdr-relay: Mac-local bridge, Tailscale identity, renders Herdr's mods in sandboxed processes, diffs trees
|
|
213
|
+
bridge-side. herdgate: the cloud in the middle, own identity, host renders, cloud routes opaque frames. The
|
|
214
|
+
ideas kept: hello-first auth, hashed tokens, scope-aware routing, single-use artifacts, rate limits everywhere.
|
|
215
|
+
The parts dropped: mod sandboxing (the host is trusted code on its own machine), tree recording (the host owns
|
|
216
|
+
its UI), Tailscale (public via cloudflared).
|