@delorenj/pjangler 1.2.21 → 1.2.25

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (63) hide show
  1. package/dist/index.js +1852 -368
  2. package/dist/mcp-server.js +1782 -296
  3. package/package.json +14 -3
  4. package/templates/commonproject/copier.yml +3 -0
  5. package/templates/commonproject/template/.mise/scripts/provision-bmad-skills.py +480 -0
  6. package/templates/commonproject/template/.mise/scripts/sync-skills.py +449 -0
  7. package/templates/commonproject/template/mise.toml.jinja +9 -2
  8. package/templates/commonproject/tests/test_provision_bmad_skills.py +203 -0
  9. package/templates/commonproject/tests/test_sync_skills_topology.py +120 -0
  10. package/templates/hermes-agent/README.md +70 -29
  11. package/templates/hermes-agent/copier.yml +9 -6
  12. package/templates/hermes-agent/docs/architecture.md +75 -51
  13. package/templates/hermes-agent/docs/fleet-control-plane/architecture.md +11 -6
  14. package/templates/hermes-agent/docs/fleet-control-plane/epics-and-stories.md +2 -2
  15. package/templates/hermes-agent/docs/fleet-control-plane/prd.md +2 -3
  16. package/templates/hermes-agent/docs/operations.md +179 -81
  17. package/templates/hermes-agent/docs/runbooks/runtime-checkpoint-repair.md +31 -171
  18. package/templates/hermes-agent/docs/sentinel/README.md +10 -10
  19. package/templates/hermes-agent/docs/sentinel/architecture.md +6 -8
  20. package/templates/hermes-agent/docs/sentinel/development.md +12 -15
  21. package/templates/hermes-agent/docs/sentinel.md +1 -1
  22. package/templates/hermes-agent/install-local.sh +41 -4
  23. package/templates/hermes-agent/runtime-scaffold/README.md +4 -2
  24. package/templates/hermes-agent/scripts/degit-runtime.sh +192 -0
  25. package/templates/hermes-agent/scripts/fleet-prune-debris.sh +128 -0
  26. package/templates/hermes-agent/scripts/fleet-sync.sh +164 -7
  27. package/templates/hermes-agent/scripts/hermes-runtime-templatize.py +321 -0
  28. package/templates/hermes-agent/scripts/migrate-profile-scoped-chat-secrets.py +136 -0
  29. package/templates/hermes-agent/scripts/momo-unify-agent.py +224 -0
  30. package/templates/hermes-agent/scripts/repair-runtime-checkpoint.sh +7 -119
  31. package/templates/hermes-agent/template/.gitignore.jinja +5 -2
  32. package/templates/hermes-agent/template/.runtime-scaffold/README.md +4 -2
  33. package/templates/hermes-agent/template/.scripts/00-banner.sh +1 -1
  34. package/templates/hermes-agent/template/.scripts/01-config.sh +3 -0
  35. package/templates/hermes-agent/template/.scripts/10-hermes-profile.sh +12 -1
  36. package/templates/hermes-agent/template/.scripts/20-runtime-repo.sh +55 -129
  37. package/templates/hermes-agent/template/.scripts/30-telegram.sh +276 -22
  38. package/templates/hermes-agent/template/.scripts/31-slack.sh +362 -0
  39. package/templates/hermes-agent/template/.scripts/60-bloodbank.sh +8 -66
  40. package/templates/hermes-agent/template/.scripts/70-systemd.sh +44 -30
  41. package/templates/hermes-agent/template/.scripts/80-registry.sh +93 -13
  42. package/templates/hermes-agent/template/.scripts/99-summary.sh +1 -4
  43. package/templates/hermes-agent/template/.scripts/_lib.sh +66 -1
  44. package/templates/hermes-agent/template/.scripts/checkpoint.sh +3 -2
  45. package/templates/hermes-agent/template/.scripts/config.example.toml +10 -10
  46. package/templates/hermes-agent/template/.scripts/heartbeat.sh +1 -1
  47. package/templates/hermes-agent/template/SOUL.md.jinja +2 -2
  48. package/templates/hermes-agent/template/hermes.jinja +6 -3
  49. package/templates/hermes-agent/template/role.yaml.jinja +24 -13
  50. package/templates/hermes-agent/tests/test_bloodbank_consumer_contract.py +429 -130
  51. package/templates/hermes-agent/tests/test_fleet_upgrade_contract.py +176 -0
  52. package/templates/hermes-agent/tests/test_launcher_chat_secret_isolation.py +85 -0
  53. package/templates/hermes-agent/tests/test_local_runtime_provisioning.py +130 -0
  54. package/templates/hermes-agent/tests/test_operator_docs_local_runtime_contract.py +132 -0
  55. package/templates/hermes-agent/tests/test_profile_env_detach_contract.py +14 -0
  56. package/templates/hermes-agent/tests/test_profile_scoped_chat_secrets.py +89 -0
  57. package/templates/hermes-agent/tests/test_repository_privacy_contract.py +32 -0
  58. package/templates/hermes-agent/tests/test_slack_provisioning.py +344 -0
  59. package/templates/hermes-agent/tests/test_telegram_provisioning.py +475 -0
  60. package/templates/hermes-agent/.codegraph/daemon.pid +0 -6
  61. package/templates/hermes-agent/.omo/run-continuation/ses_0e5b28303ffeXxL53hZjKggXDW.json +0 -10
  62. package/templates/hermes-agent/runtime-scaffold/bloodbank-consumer.py +0 -181
  63. package/templates/hermes-agent/template/.runtime-scaffold/bloodbank-consumer.py +0 -181
@@ -27,11 +27,12 @@ invocations).
27
27
  | 01 config | Seed `~/.config/hermes-agent-template/config.toml` from the shipped example if absent (see [Configuration](#configuration)) | n/a |
28
28
  | 05 fleet env | Ensure `~/.hermes/fleet.env` exists (shared Hermes binary/repo/registry source-of-truth), populated from `config.toml` | n/a |
29
29
  | 10 hermes profile | `hermes profile create <repo>-<role> --clone --no-alias` + mirror skills/plugins/hooks from default + symlink canonical runtime skills (`delonet-conventions`, `delonet-dotenv`, `hermes-pm-template-maintenance`, `hindsight`, `subagent-driven-development`) from `/home/delorenj/.agents/skills`; PM roles also seed `VOX_URL` in profile `.env` | n/a |
30
- | 20 runtime repo | Create gh:delorenj/agent-hm-<repo>-<role> (private), push scaffold from role-local `.runtime-scaffold/`, submodule-add into ./runtime/, symlink ~/.hermes/profiles/<id> → runtime; PM roles also link the Voxxy plugin and set `tts.provider: voxxy` | `SKIP_RUNTIME_REPO=1` |
31
- | 30 telegram | Capture BotFather token, write to runtime/.env, enable hermes-telegram toolset | `SKIP_TELEGRAM=1` |
30
+ | 20 local runtime | Populate missing files from role-local `.runtime-scaffold/` into ignored `./runtime/`, symlink `~/.hermes/profiles/<id>` → runtime, and refuse stale project gitlinks/mappings; PM roles also link the Voxxy plugin and set `tts.provider: voxxy` | `SKIP_RUNTIME_REPO=1` |
31
+ | 30 telegram | Verify an invocation-supplied, profile-dedicated BotFather token; reject fleet reuse; write only to runtime/.env | `SKIP_TELEGRAM=1` |
32
+ | 31 slack | Disabled/deferred by default; verify a dedicated app+bot pair with `auth.test` and write it only to runtime/.env when explicitly enabled | `SKIP_SLACK=1` |
32
33
  | 40 plane | Create Plane project in 33god workspace (1:1 with agent), patch identifier into role.yaml | `SKIP_PLANE=1` |
33
- | 60 bloodbank | Install consumer (renders from scaffold w/ agent values), health-check NATS, install nats-py via uv if missing | `SKIP_BLOODBANK=1` |
34
- | 70 systemd | Install user units: gateway, consumer, heartbeat timer (board-reconciliation sentinel pass + gated runtime checkpoint, one tick) | `SKIP_SYSTEMD=1` |
34
+ | 60 bloodbank | Compatibility checkpoint for fleet-shared routing; installs no files, dependencies, or services | `SKIP_BLOODBANK=1` remains a no-op |
35
+ | 70 systemd | Install user units: profile gateway and board-reconciliation heartbeat timer | `SKIP_SYSTEMD=1` |
35
36
  | 80 registry | Append entry to ~/.hermes/agents-registry.yaml | n/a |
36
37
  | 99 summary | Print summary | n/a |
37
38
 
@@ -54,28 +55,23 @@ different machine/user, edit this one file:
54
55
  [fleet]
55
56
  hermes_bin = "/path/to/hermes-agent/.venv/bin/hermes"
56
57
  hermes_repo = "/path/to/hermes-agent"
58
+ hermes_git_url = "https://github.com/delorenj/hermes-agent.git"
59
+ hermes_git_ref = "feature/PJAN-19-routing-publication"
60
+ hermes_git_sha = "113e1b182b6d72a7dd02a191f134a41668ceaf0e"
57
61
  oauth_file = "~/.hermes/auth.json"
58
62
  codex_home = "~/.codex"
59
63
  canonical_skills_dir = "/path/to/.agents/skills"
60
64
  voxxy_plugin_dir = "~/code/voxxy/plugins/tts/voxxy"
61
65
  vox_url = "https://vox.delo.sh"
62
66
 
63
- [github]
64
- runtime_repo_owner = "your-gh-owner"
65
-
66
67
  [plane]
67
68
  base = "https://plane.example.com"
68
69
  workspace = "your-workspace"
69
-
70
- [bloodbank]
71
- nats_host = "127.0.0.1"
72
- nats_port = 4222
73
70
  ```
74
71
 
75
- `role.yaml` stores an empty `runtime.github_owner` and `plane.workspace` for
76
- freshly provisioned agents; the shell layer fills them from `config.toml` at
77
- runtime (older manifests that baked `owner/name` into `runtime.github_repo`
78
- still work unchanged).
72
+ `role.yaml` stores compatibility metadata for older generated roles, but the
73
+ current local-runtime provisioner does not use it for storage. Plane workspace
74
+ defaults are still filled from `config.toml`.
79
75
 
80
76
  ## Fleet source-of-truth
81
77
 
@@ -93,6 +89,13 @@ For Codex auth, run `hermes auth add openai-codex` through any generated agent
93
89
  launcher once; all agents using the same fleet env read the same Hermes OAuth
94
90
  store afterward.
95
91
 
92
+ The reviewed runtime source is the `delorenj/hermes-agent` fork, publication
93
+ ref `feature/PJAN-19-routing-publication`, pinned at
94
+ `113e1b182b6d72a7dd02a191f134a41668ceaf0e`. `install-local.sh` performs a
95
+ single-branch clone, verifies the pin is on that ref, passes both `--branch` and
96
+ `--commit` to the checked-out installer, and refuses an existing checkout with
97
+ a different origin. It never fetches or writes upstream `main`.
98
+
96
99
  To retrofit older provisioned agents onto this model, run:
97
100
 
98
101
  ```bash
@@ -100,15 +103,38 @@ cd /home/delorenj/code/hermes-agent-template
100
103
  ./scripts/backfill-fleet-sot.sh
101
104
  ```
102
105
 
106
+ Then audit legacy consumers before relying on the fleet-shared Bloodbank
107
+ gateway:
108
+
109
+ ```bash
110
+ ./scripts/fleet-sync.sh
111
+ ./scripts/fleet-sync.sh --apply
112
+ ```
113
+
114
+ Any active, enabled, installed, or registry-declared
115
+ `hermes-<agent>-consumer.service` is unhealthy drift. Apply mode disables and
116
+ stops it, proves the unit is explicitly `inactive` and `disabled`, then removes
117
+ the unit, reloads user systemd, and removes the legacy registry metadata. A
118
+ user-manager/query error, failed disable, or ambiguous post-disable state fails
119
+ closed and preserves both the unit file and registry metadata. The same cleanup
120
+ runs in step 70 before its done marker is honored.
121
+
122
+ Fleet registry and chat identity changes share `${registry_file}.lock`. `flock`
123
+ owns the lock in the kernel (an on-disk lock file surviving a crash is safe),
124
+ while atomic replace prevents partial YAML. Registry and lock files are mode
125
+ `0600`; symlink targets are refused. Registry and profile-local credential
126
+ replacements also sync the containing directory where the platform supports it.
127
+ Unexpected directory-sync errors are reported as failures without marking the
128
+ step complete, so rerunning the idempotent provisioning step is safe.
129
+
103
130
  ## Start the daemons for an agent
104
131
 
105
132
  ```bash
106
133
  AGENT=bloodbank-pm
107
- systemctl --user start hermes-${AGENT}-consumer.service
108
134
  systemctl --user start hermes-${AGENT}-heartbeat.timer
109
135
 
110
- # Gateway will fail to start until Telegram is wired up (no other platforms
111
- # configured). After running .scripts/30-telegram.sh:
136
+ # Gateway will fail to start until at least one messaging platform is wired.
137
+ # After running .scripts/30-telegram.sh or .scripts/31-slack.sh:
112
138
  systemctl --user start hermes-${AGENT}-gateway.service
113
139
  ```
114
140
 
@@ -117,8 +143,9 @@ systemctl --user start hermes-${AGENT}-gateway.service
117
143
  | Channel | How |
118
144
  | --- | --- |
119
145
  | Telegram | DM `@<repo>_<role>_bot` (once Telegram is wired) |
146
+ | Slack | DM or mention the verified per-agent Slack bot (once Slack is wired) |
120
147
  | Local CLI | `./agents/hermes/<role>/hermes chat "..."` |
121
- | Bloodbank | Publish to `bloodbank.cmd.v1.agent.task.assign` with `data.target_agent_id = <agent_id>` |
148
+ | Bloodbank | Publish to `bloodbank.cmd.v1.agent.invocation.start` with `data.target_agent_id = <agent_id>` |
122
149
 
123
150
  ## Inspect fleet state
124
151
 
@@ -134,8 +161,8 @@ EOF
134
161
  systemctl --user list-units --state=active 'hermes-*'
135
162
  systemctl --user list-timers 'hermes-*'
136
163
 
137
- # Tail the consumer for live bloodbank events
138
- journalctl --user -fu hermes-<agent-id>-consumer.service
164
+ # Bloodbank routing identity consumed by the fleet-shared gateway
165
+ python3 -c "import yaml,pathlib; print(yaml.safe_load(pathlib.Path.home().joinpath('.hermes/agents-registry.yaml').read_text())['agents']['<agent-id>']['bloodbank'])"
139
166
  ```
140
167
 
141
168
  ## Deferred manual steps (one-time per agent)
@@ -150,65 +177,130 @@ journalctl --user -fu hermes-<agent-id>-consumer.service
150
177
  ```bash
151
178
  cd <project>/agents/hermes/<role>
152
179
  rm .scripts/.done-30-telegram
153
- SKIP_TELEGRAM=0 ./.scripts/30-telegram.sh
180
+ TELEGRAM_BOT_TOKEN='<bot-id>:<secret>' \
181
+ TELEGRAM_ALLOWED_USERS='<your-user-id>' \
182
+ SKIP_TELEGRAM=0 ./.scripts/30-telegram.sh
154
183
  systemctl --user restart hermes-<agent-id>-gateway.service
155
184
  ```
156
185
 
157
- ## Restore an agent on a new machine
186
+ The token is captured before shared fleet configuration is loaded, verified
187
+ through Telegram `getMe`, checked against local token and bot-identity owners,
188
+ and atomically written only to the profile's gitignored `runtime/.env` with
189
+ mode `0600`. `~/.hermes/fleet.env` and `~/.hermes/.env` must not contain
190
+ `TELEGRAM_BOT_TOKEN`; the manifest and registry store only verified bot
191
+ identity metadata. `TELEGRAM_ALLOWED_USERS` is non-secret and may be shared.
192
+
193
+ ### Slack app and bot (opt-in)
194
+
195
+ Slack remains deferred unless `ENABLE_SLACK=1` (also accepts
196
+ `WIRE_SLACK=1`) is set or both credentials are supplied. Each enabled agent
197
+ must have its own Slack app-level Socket Mode token and bot token; neither may
198
+ be reused by another profile.
158
199
 
159
200
  ```bash
160
- cd /path/to/the/project-repo
161
- git submodule update --init --recursive
162
- git -C agents/hermes/<role>/runtime lfs pull
163
-
164
- # Restore secrets that were excluded from git:
165
- op read 'op://DeLoSecrets/agent-hm-<repo>-<role>/.env' > agents/hermes/<role>/runtime/.env
166
- # (or copy from a backup machine)
167
-
168
- # Symlink the profile dir
169
- ln -sfn $PWD/agents/hermes/<role>/runtime ~/.hermes/profiles/<repo>-<role>
170
-
171
- # Re-enable systemd units
172
- systemctl --user enable hermes-<repo>-<role>-{gateway,consumer}.service
173
- systemctl --user enable hermes-<repo>-<role>-heartbeat.timer
174
- systemctl --user start hermes-<repo>-<role>-{consumer}.service
175
- systemctl --user start hermes-<repo>-<role>-heartbeat.timer
201
+ cd <project>/agents/hermes/<role>
202
+ ENABLE_SLACK=1 \
203
+ SLACK_BOT_TOKEN='xoxb-...' \
204
+ SLACK_APP_TOKEN='xapp-...' \
205
+ SLACK_ALLOWED_USERS='U01ABC2DEF3' \
206
+ ./.scripts/31-slack.sh
207
+ systemctl --user restart hermes-<agent-id>-gateway.service
176
208
  ```
177
209
 
178
- ## Retire an agent (manual, until v1.1 ships retire.sh)
210
+ The step calls Slack's read-only `auth.test` endpoint for the bot token, checks
211
+ the local fleet for token or bot-identity reuse, and records only the verified
212
+ workspace/bot identity in `role.yaml` and the fleet registry. Tokens are
213
+ atomically written to the agent's gitignored `runtime/.env` with mode `0600`.
214
+ They must never be placed in `~/.hermes/.env` or `~/.hermes/fleet.env`.
215
+
216
+ `SLACK_ALLOWED_USERS` is non-secret and may instead be set in `fleet.env` as a
217
+ shared policy. An empty allow-list is safe but denies all inbound Slack users.
218
+
219
+ ## Back up and restore an agent
220
+
221
+ The template does **not** ship automatic backup for the ignored
222
+ `agents/hermes/<role>/runtime/` directory. A project clone recreates the
223
+ tracked role and its empty scaffold, not accumulated local state. Configure an
224
+ encrypted filesystem backup or snapshot that includes the exact runtime path
225
+ before treating the agent as recoverable.
226
+
227
+ For a manual transfer, create a private archive, copy it to operator-managed
228
+ encrypted storage, and verify both the checksum and readable member list:
179
229
 
180
230
  ```bash
181
- AGENT=bloodbank-dev
182
- # 1. Stop daemons
183
- systemctl --user disable --now hermes-${AGENT}-{gateway,consumer}.service
184
- systemctl --user disable --now hermes-${AGENT}-heartbeat.timer
185
-
186
- # 2. Delete hermes profile (cascades to symlinked runtime — make sure
187
- # that's what you want!)
188
- hermes profile delete ${AGENT}
189
-
190
- # 3. Archive Plane project (Plane UI or API)
191
- PROJECT_ID=$(python3 -c "import yaml,pathlib; print(yaml.safe_load(pathlib.Path.home().joinpath('.hermes/agents-registry.yaml').read_text())['agents']['${AGENT}']['plane']['project_id'])")
192
- curl -X POST "https://plane.delo.sh/api/v1/workspaces/33god/projects/${PROJECT_ID}/archive/" \
193
- -H "X-API-Key: ${PLANE_33GOD_API_KEY}"
194
-
195
- # 4. BotFather: /deletebot @<repo>_<role>_bot
196
- # 5. Archive runtime repo (GitHub UI; we don't have delete_repo scope by default)
197
- # 6. Remove registry entry
198
- python3 -c "
199
- import yaml, pathlib
200
- p = pathlib.Path.home() / '.hermes' / 'agents-registry.yaml'
201
- d = yaml.safe_load(p.read_text()); d['agents'].pop('${AGENT}', None)
202
- p.write_text(yaml.safe_dump(d))"
203
-
204
- # 7. In the project repo, remove the submodule
205
- cd /path/to/project
206
- git submodule deinit -f agents/hermes/<role>/runtime
207
- git rm -f agents/hermes/<role>/runtime
208
- rm -rf .git/modules/agents/hermes/<role>/runtime
209
- rm -rf agents/hermes/<role>
231
+ PROJECT=/absolute/path/to/project
232
+ ROLE=pm
233
+ BACKUP_DIR="$HOME/.local/state/hermes-runtime-backups"
234
+ mkdir -p -m 0700 "$BACKUP_DIR"
235
+ BACKUP="$BACKUP_DIR/$(basename "$PROJECT")-${ROLE}-$(date -u +%Y%m%dT%H%M%SZ).tar.gz"
236
+ tar -C "$PROJECT" -czf "$BACKUP" "agents/hermes/${ROLE}/runtime"
237
+ chmod 0600 "$BACKUP"
238
+ sha256sum "$BACKUP" > "${BACKUP}.sha256"
239
+ sha256sum -c "${BACKUP}.sha256"
240
+ tar -tzf "$BACKUP" >/dev/null
210
241
  ```
211
242
 
243
+ That archive may contain credentials and private conversations. It is not an
244
+ off-host backup until it has been copied to encrypted storage controlled by the
245
+ operator and independently verified there.
246
+
247
+ Recovery sources are intentionally distinct:
248
+
249
+ - The verified filesystem backup is the only complete source for local config,
250
+ sessions, databases, skills, and other runtime files.
251
+ - Hindsight can restore only memories/events that were previously written to
252
+ its remote bank. It is not a backup of the runtime directory.
253
+ - The secret manager can restore only credentials deliberately stored there;
254
+ it does not contain memories, sessions, or local configuration by default.
255
+ - Re-running provisioning restores the scaffold and service definitions, not
256
+ learned state.
257
+
258
+ Restore the project and provision the role first. With its services stopped,
259
+ extract the verified archive at the project root, confirm the profile symlink
260
+ targets the restored runtime, then enable the services.
261
+
262
+ ## Retire an agent (preserves runtime by default)
263
+
264
+ Retirement stops external behavior and detaches the profile while preserving
265
+ the role directory and every runtime byte. Do not use a profile command whose
266
+ deletion behavior is unknown.
267
+
268
+ ```bash
269
+ PROJECT=/absolute/path/to/project
270
+ ROLE=pm
271
+ AGENT=bloodbank-pm
272
+ RUNTIME="$PROJECT/agents/hermes/$ROLE/runtime"
273
+ PROFILE="$HOME/.hermes/profiles/$AGENT"
274
+
275
+ systemctl --user disable --now "hermes-${AGENT}-gateway.service"
276
+ systemctl --user disable --now "hermes-${AGENT}-heartbeat.timer"
277
+
278
+ # Detach only the expected symlink. Never follow it into the runtime.
279
+ test -L "$PROFILE"
280
+ test "$(readlink -f -- "$PROFILE")" = "$(readlink -f -- "$RUNTIME")"
281
+ unlink -- "$PROFILE"
282
+
283
+ # Archive the Plane project and retire Telegram/Slack identities through their
284
+ # administrative UIs, then remove the fleet registry entry under its lock.
285
+ # The runtime directory remains in place.
286
+ test -d "$RUNTIME"
287
+ ```
288
+
289
+ ### Runtime retention after retirement
290
+
291
+ Profile and service retirement always preserves the local runtime. This
292
+ release intentionally provides no automated runtime purge, and this operations
293
+ guide supplies no deletion recipe.
294
+
295
+ Any purge is a separate future operator-retention process. It requires a
296
+ separately reviewed, path-safe tool that canonicalizes both the repository root
297
+ and role path, refuses ambiguous or linked targets, and verifies that the
298
+ backup archive contains the expected runtime members before it can remove any
299
+ data. Until such a tool is reviewed and shipped, preserve the runtime.
300
+
301
+ Removing the tracked role scaffold is a different project change and is not
302
+ part of profile or service retirement.
303
+
212
304
  ## Troubleshooting
213
305
 
214
306
  ### Gateway service fails immediately
@@ -216,22 +308,28 @@ rm -rf agents/hermes/<role>
216
308
  - `journalctl --user -u hermes-<agent>-gateway.service`
217
309
  - If "all configured messaging platforms failed to connect" — Telegram step wasn't run yet. Run `.scripts/30-telegram.sh` first.
218
310
 
219
- ### Consumer not seeing events
220
- - Verify NATS is up: `docker compose -f ~/code/33GOD/bloodbank/compose/docker-compose.yml ps`
221
- - Tail consumer: `journalctl --user -fu hermes-<agent>-consumer.service`
222
- - Make sure something is publishing canonical repo events such as
223
- `bloodbank.evt.v1.repo.issue.updated` with `data.repo = <repo>`
311
+ ### Bloodbank command not reaching an agent
312
+ - Confirm the fleet registry entry has `bloodbank.gateway_scope: fleet`
313
+ - Confirm its `bloodbank.target_agent_id` exactly matches the command's
314
+ `data.target_agent_id`
315
+ - Inspect the fleet-shared Bloodbank gateway; there is intentionally no
316
+ `hermes-<agent>-consumer.service` or runtime inbox to repair
317
+
318
+ ### Runtime changes are not appearing in Hindsight or backups
224
319
 
225
- ### Heartbeat not checkpointing (runtime not pushing)
226
- The checkpoint runs inside the heartbeat tick (after the board-reconciliation
227
- sentinel pass), gated to at most once an hour.
320
+ Pure-local runtime changes are not synchronized by the heartbeat.
228
321
  - Look at the most recent heartbeat log: `tail <role>/runtime/logs/heartbeat.log`
229
- - Verify the submodule's git remote is reachable: `cd ...runtime && git push origin HEAD`
230
- - If LFS items fail to push: `git lfs push origin HEAD --all`
231
- - Force a checkpoint out-of-band: `bash agents/hermes/<role>/.scripts/checkpoint.sh`
322
+ - Verify the configured filesystem backup includes the exact runtime path and
323
+ successfully restore-test its latest snapshot.
324
+ - Query Hindsight separately for the agent bank; only events already written
325
+ there are recoverable from Hindsight.
326
+ - Check the secret manager separately for the profile credentials you chose to
327
+ store there.
232
328
 
233
329
  ### Profile dir contains nested `profiles/profiles/...`
234
- - That was a `--clone-all` bug; we switched to `--clone`. If you see it, just `rm -rf` the nested tree. The template's 10-hermes-profile.sh also has a belt-and-suspenders rm.
330
+ - That was a `--clone-all` bug; current provisioning uses `--clone`. Preserve
331
+ the profile and runtime, inspect the unexpected nesting, and move only the
332
+ confirmed redundant entries to a quarantine directory for review.
235
333
 
236
334
  ### `hermes` launcher complains about HERMES_BIN
237
335
  - Check the launcher script: `./agents/hermes/<role>/hermes` falls back to `$HOME/.hermes/hermes-agent/.venv/bin/hermes` (after `$HERMES_BIN`, `fleet.env`, and config.toml). Override with `HERMES_BIN=/path/to/hermes ./agents/hermes/pm/hermes status`.
@@ -1,171 +1,31 @@
1
- # Runbook: repair a failing runtime-checkpoint service
2
-
3
- Status: Active operations runbook
4
- Applies to: every Hermes agent whose `runtime/` is a git submodule checkpointed
5
- by its `hermes-<agent>-heartbeat.service` systemd unit (the heartbeat runner
6
- calls `.scripts/checkpoint.sh` to commit+push the runtime).
7
-
8
- This runbook fixes the failure where the runtime checkpoint dies with
9
- **exit 128** and the agent's runtime "brain" stops being backed up. The
10
- checkpoint now runs inside the fused heartbeat tick (board-reconciliation
11
- sentinel pass + gated checkpoint), so the failing unit is the agent's
12
- `heartbeat.service`. It is written so any agent or operator can copy it to
13
- another repo and apply it safely. First captured 2026-06-01 fixing the
14
- Drumjangler PM checkpoint; the same fault was present on other agents in the
15
- fleet.
16
-
17
- ## Symptom
18
-
19
- ```bash
20
- systemctl --user status hermes-<agent>-heartbeat.service
21
- # Active: failed (Result: exit-code) ... status=128/n/a
22
- # the heartbeat invokes .../agents/hermes/<role>/.scripts/checkpoint.sh
23
- ```
24
-
25
- `checkpoint.sh` is just `cd runtime && git add -A && commit && push`. Run it by
26
- hand and you see the real error:
27
-
28
- ```text
29
- fatal: in unpopulated submodule 'agents/hermes/<role>/runtime'
30
- ```
31
-
32
- ## Root cause
33
-
34
- `agents/hermes/<role>/runtime` is registered as a **git submodule** in the
35
- project repo (`.gitmodules` has its URL; the parent index holds a `160000`
36
- gitlink), **but the submodule is not populated** — there is no `runtime/.git`,
37
- and `.git/modules/<path>` is gone. So any git command run *inside* `runtime/`
38
- resolves **up to the parent repo**, which treats `runtime/` as an unpopulated
39
- submodule path and makes `git add -A` fatal (exit 128).
40
-
41
- Confirm it:
42
-
43
- ```bash
44
- RT=agents/hermes/<role>/runtime
45
- git -C "$RT" rev-parse --show-toplevel # prints the PARENT repo, not RT -> red flag
46
- ls "$RT/.git" # missing -> unpopulated
47
- git ls-files -s "$RT" | head -1 # mode 160000 <sha> ... -> gitlink
48
- git config -f .gitmodules --get-regexp "$RT" # submodule.<path>.url git@...
49
- ls .git/modules/"$RT" 2>/dev/null || echo "objects gone -> must re-fetch from remote"
50
- ```
51
-
52
- Consequence: the agent's durable brain (`SOUL.md`, `memories/`, `profiles/`,
53
- `skills/`, config) lives on disk (the running agent still reads it) but is **not
54
- committed or pushed anywhere** — no backup until this is repaired.
55
-
56
- ## The fix (safe, in-place, preserves all on-disk state)
57
-
58
- The key safety property: `git reset --mixed` moves `HEAD` + the index only and
59
- **never rewrites working-tree files**, so the live brain (and a multi-hundred-MB
60
- `state.db`) is untouched. Never use `git submodule update`/`git checkout <tree>`
61
- here — those would overwrite live, uncommitted edits with the older committed
62
- versions.
63
-
64
- ```bash
65
- RT=agents/hermes/<role>/runtime
66
- URL=$(git config -f .gitmodules --get submodule."$RT".url)
67
-
68
- # 0. remote must be reachable (read-only)
69
- GIT_SSH_COMMAND='ssh -o BatchMode=yes -o ConnectTimeout=15' git ls-remote "$URL" >/dev/null
70
-
71
- cd "$RT"
72
- # 1. re-init git in place on branch 'main' (creates .git/ only; no file changes)
73
- git init -q
74
- git symbolic-ref HEAD refs/heads/main
75
- git remote add origin "$URL"
76
- git fetch -q origin
77
- # 2. re-base history onto the remote tip WITHOUT touching the working tree
78
- git reset --mixed -q origin/main
79
- git branch --set-upstream-to=origin/main main 2>/dev/null || true
80
- # 3. the repo uses git-lfs for *.db/images (.gitattributes); register filters
81
- git lfs install --local
82
- # 4. restore tracked config that may be missing on disk (avoids spurious deletes)
83
- git checkout origin/main -- .gitattributes README.md 2>/dev/null || true
84
- ```
85
-
86
- ### Make it sustainable: back up the brain, not volatile state
87
-
88
- Even though `.gitattributes` routes `*.db` through LFS, committing a live,
89
- constantly-growing `state.db` (often hundreds of MB) every hour is not
90
- sustainable, and `lsp/` (node_modules), `checkpoints/` (a nested object store),
91
- and `sessions/` (ephemeral dumps) must never be committed. The checkpoint repo's
92
- job is the **durable brain**. Ensure `runtime/.gitignore` contains:
93
-
94
- ```gitignore
95
- # Runtime working state — NOT part of the durable brain backup.
96
- *.db
97
- *.db-wal
98
- *.db-shm
99
- *.sqlite
100
- *.sqlite3
101
- *.lock
102
- .update_check
103
- lsp/
104
- checkpoints/
105
- sessions/
106
- cron/
107
- *_cache.json
108
- ```
109
-
110
- Then untrack anything volatile that is currently tracked (keeps the files on
111
- disk):
112
-
113
- ```bash
114
- git rm -r --cached --ignore-unmatch state.db state.db-wal state.db-shm cron
115
- ```
116
-
117
- ### Verify BEFORE committing (mandatory safety gate)
118
-
119
- ```bash
120
- git add -A --dry-run | awk '{print $2}' | while read -r f; do
121
- [ -f "$f" ] && s=$(stat -c%s "$f") && [ "$s" -gt 1048576 ] && echo "BIG: $((s/1048576))MB $f"; done
122
- # ^ must print nothing. Also confirm no state.db / lsp/ / sessions/ / checkpoints/,
123
- # and no secrets (.env, auth.json, *.key, *token*) in the staged set.
124
- ```
125
-
126
- ### Commit, push, and bring the service green
127
-
128
- ```bash
129
- bash agents/hermes/<role>/.scripts/checkpoint.sh # exits 0, pushes brain
130
- systemctl --user reset-failed hermes-<agent>-heartbeat.service
131
- systemctl --user start hermes-<agent>-heartbeat.service
132
- systemctl --user show hermes-<agent>-heartbeat.service -p Result -p ExecMainStatus
133
- # Result=success ExecMainStatus=0
134
- ```
135
-
136
- ## One-shot
137
-
138
- `scripts/repair-runtime-checkpoint.sh` automates all of the above. It is
139
- **dry-run by default** (diagnoses and prints the plan) and only mutates with
140
- `--apply`. Run it from a project repo root:
141
-
142
- ```bash
143
- # diagnose
144
- hermes-agent-template/scripts/repair-runtime-checkpoint.sh agents/hermes/pm/runtime
145
- # apply the in-place re-attach + .gitignore policy + untrack (no commit)
146
- hermes-agent-template/scripts/repair-runtime-checkpoint.sh --apply agents/hermes/pm/runtime
147
- # then validate the service
148
- systemctl --user reset-failed <svc> && systemctl --user start <svc>
149
- ```
150
-
151
- ## Fleet sweep — this is rarely just one repo
152
-
153
- ```bash
154
- systemctl --user list-units --all '*-heartbeat.service' | grep -i failed
155
- # for each failing agent, locate its runtime and run the repair script.
156
- ```
157
-
158
- ## Lessons learned (why this happened / what to remember)
159
-
160
- - An **unpopulated submodule** silently redirects in-directory git commands to
161
- the parent repo; `git add -A` then fatals with exit 128. The fix is to restore
162
- the submodule's own `.git` (objects from the remote), not to touch the parent.
163
- - `git reset --mixed <ref>` is the safe way to re-attach history to a populated
164
- working directory: it preserves every on-disk file.
165
- - The checkpoint must **exclude volatile working state** (`state.db*`, `lsp/`,
166
- `checkpoints/`, `sessions/`, caches, locks). Back up identity + memory + skills.
167
- - Core dumps from the agent will **not** appear in `coredumpctl`/`/var/crash` on
168
- Ubuntu: apport discards crashes from unpackaged venv-Python binaries. Use
169
- `PYTHONFAULTHANDLER=1` or a private `core_pattern` to capture a backtrace.
170
- - Always run the fleet sweep — provisioning copies this layout to every agent,
171
- so a structural fault tends to be fleet-wide.
1
+ # Historical runbook: retired checkpoint architecture
2
+
3
+ Status: Historical evidence only — **do not execute**
4
+
5
+ This page formerly described recovery for an older per-agent persistence
6
+ design. That design is not the current Hermes template contract, and its
7
+ commands have intentionally been removed so an operator cannot mistake them
8
+ for a supported repair procedure.
9
+
10
+ Current agents use an ignored, pure-local
11
+ `agents/hermes/<role>/runtime/` directory. Provisioning refuses stale project
12
+ gitlinks or mappings at that exact path and preserves all existing runtime
13
+ bytes. The heartbeat performs board reconciliation; it is not the runtime
14
+ backup mechanism.
15
+
16
+ For current recovery and retirement procedures, use
17
+ [Operations](../operations.md#back-up-and-restore-an-agent). In particular:
18
+
19
+ - configure and verify an encrypted filesystem backup for the exact runtime
20
+ path;
21
+ - treat Hindsight as recovery for only the memories/events previously written
22
+ to its bank;
23
+ - treat the secret manager as recovery for only credentials deliberately
24
+ stored there;
25
+ - retire services and profile links without removing runtime data; and
26
+ - preserve retired runtime data because this release intentionally ships no
27
+ automated purge; any future purge requires a separately reviewed path-safe
28
+ tool and a verified off-host backup.
29
+
30
+ The earlier procedure remains recoverable from repository history for incident
31
+ forensics. Repository history is evidence, not an active operator interface.
@@ -20,9 +20,9 @@ and emits a decision event. A real finding is `held` and the ticket goes back to
20
20
  active.
21
21
 
22
22
  There is no longer a separate `scrum-master` role; the sentinel folded into the
23
- PM and now runs as part of the PM's **heartbeat** — a fused systemd-timer tick
24
- that does the board-reconciliation pass and then a gated runtime checkpoint
25
- (`.scripts/heartbeat.sh`). The engine itself lives under the PM at
23
+ PM and now runs as the PM's **heartbeat** systemd-timer tick
24
+ (`.scripts/heartbeat.sh`). Under the pure-local runtime contract this timer
25
+ performs board reconciliation only. The engine itself lives under the PM at
26
26
  `.scripts/sentinel/`.
27
27
 
28
28
  It talks to the ticket board through a pluggable adapter, so the same engine
@@ -50,8 +50,8 @@ The following is true as of June 1, 2026.
50
50
 
51
51
  ## Quick local install
52
52
 
53
- For a local, single-machine install (no GitHub runtime repo, no Telegram, no
54
- NATS), use the one-command bootstrap. From inside the target project:
53
+ For a local, single-machine install (no Telegram and no NATS), use the
54
+ one-command bootstrap. From inside the target project:
55
55
 
56
56
  ```bash
57
57
  export PLANE_API_KEY=<key> # or LINEAR_API_KEY / TRELLO_KEY + TRELLO_TOKEN
@@ -65,9 +65,9 @@ macOS). See [Development guide: local install](development.md#local-install-one-
65
65
 
66
66
  <!-- prettier-ignore -->
67
67
  > [!IMPORTANT]
68
- > Full (non-local) provisioning is outward-facing. It can create a GitHub
69
- > runtime repo, a Telegram bot, and a Plane project, and the Telegram step is
70
- > interactive. Use `install-local.sh` or the `SKIP_*` flags described in
68
+ > Full provisioning is outward-facing. It can configure a Telegram bot and a
69
+ > Plane project, and the Telegram step is interactive. Use `install-local.sh`
70
+ > or the `SKIP_*` flags described in
71
71
  > [Development guide:
72
72
  > provisioning](development.md#provisioning-the-pm-manual) for local or lean
73
73
  > installs.
@@ -83,9 +83,9 @@ All paths are relative to the repository root.
83
83
  | `template/.scripts/lib/ticket-provider.sh` | The adapter dispatcher (`tp`). The engine's only seam to a ticket system. |
84
84
  | `template/.scripts/providers/{linear,plane,trello}.sh` | The provider adapters. |
85
85
  | `template/.scripts/42-ticket-provider.sh` | Provisioning step that resolves or creates the board. |
86
- | `template/.scripts/70-systemd.sh` | Provisioning step that installs the gateway, consumer, and the fused `heartbeat` timer (board-reconciliation sentinel pass + gated runtime checkpoint). |
86
+ | `template/.scripts/70-systemd.sh` | Provisioning step that installs the profile gateway and board-reconciliation `heartbeat` timer. |
87
87
  | `install-local.sh` | One-command local install (no cloud, macOS + Linux). |
88
- | `template/.scripts/heartbeat.sh` | The heartbeat runner: the sentinel full-pass dispatch (with its own cooldown/lock) plus the gated runtime checkpoint, fused into one tick. |
88
+ | `template/.scripts/heartbeat.sh` | The heartbeat runner: sentinel full-pass dispatch with its own cooldown and lock. |
89
89
  | `template/.scripts/sentinel.prompt.md.jinja` | The prompt the runner feeds to Hermes for a full reconciliation pass (rendered to `.scripts/sentinel.prompt.md`). |
90
90
  | `template/.scripts/sentinel/bin/` | Enforcement tools: `issue-autonomous-review.sh`, `issue-close-gate.sh`, `emit-event.py`. |
91
91
  | `template/.scripts/sentinel/docs/` | Runtime protocol docs shipped to each deployment. |
@@ -22,12 +22,11 @@ provisioning](development.md#provisioning-the-pm-manual).
22
22
  ## The heartbeat loop
23
23
 
24
24
  A scheduler (`systemd` timer on Linux, `launchd` agent on macOS) fires the
25
- runner, `template/.scripts/heartbeat.sh`, about once a minute. Each tick fuses
26
- two jobs: a board-reconciliation **sentinel pass** and a **gated runtime
27
- checkpoint**. The sentinel pass is a cheap heartbeat that decides whether a
28
- full, LLM-backed reconciliation pass is worth running; the checkpoint
29
- (commit+push of the runtime submodule) is gated to at most once an hour. This
30
- keeps cost low while staying responsive.
25
+ runner, `template/.scripts/heartbeat.sh`, about once a minute. For a pure-local
26
+ runtime, each tick performs a board-reconciliation **sentinel pass**. The cheap
27
+ heartbeat decides whether a full, LLM-backed reconciliation pass is worth
28
+ running. This keeps cost low while staying responsive; runtime backup remains
29
+ an operator-managed filesystem concern outside the sentinel.
31
30
 
32
31
  The sentinel reads the work-state file,
33
32
  `runtime/continuous-ticket-sentinel-state.json`, and chooses one of these
@@ -193,8 +192,7 @@ A full pass moves through these components in order:
193
192
  which render an `accepted` or `held` verdict and emit a decision event; the
194
193
  loop acts on the verdict (treat as done and unblock dependents, or send the
195
194
  ticket back to active) without waiting on the operator.
196
- 6. The runner writes the outcome to the state file for the next tick, then
197
- opportunistically checkpoints the runtime (gated to ~hourly) before exiting.
195
+ 6. The runner writes the outcome to the state file for the next tick and exits.
198
196
 
199
197
  ## Read next
200
198