@turmind/halo 2.1.6 → 2.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/admin-out/404.html +1 -1
- package/admin-out/_next/static/chunks/1607-bf227fc5e4db45b7.js +1 -0
- package/admin-out/_next/static/chunks/1989-3b2faa2731d973c4.js +1 -0
- package/admin-out/_next/static/chunks/5287.7ea08918388fa804.js +1 -0
- package/admin-out/_next/static/chunks/app/page-10a4afac9cbc27a2.js +1 -0
- package/admin-out/_next/static/chunks/{webpack-7f8156dbbdb54868.js → webpack-9532ec7f1fa9a757.js} +1 -1
- package/admin-out/_next/static/css/8fc43fce83e5ce5d.css +1 -0
- package/admin-out/index.html +1 -1
- package/admin-out/index.txt +5 -5
- package/bundled-docs/design/a2a.md +3 -3
- package/bundled-docs/design/agentcore.md +1 -1
- package/bundled-docs/design/architecture.md +3 -3
- package/bundled-docs/design/background-dispatch.md +2 -2
- package/bundled-docs/design/canvas-extensions.md +2 -2
- package/bundled-docs/design/cron.md +77 -53
- package/bundled-docs/design/evolution.md +111 -39
- package/bundled-docs/design/node.md +5 -12
- package/bundled-docs/design/observability.md +11 -11
- package/bundled-docs/design/prompt-system.md +1 -1
- package/bundled-docs/design/relay.md +2 -2
- package/bundled-docs/design/session.md +11 -9
- package/bundled-docs/design/slack.md +1 -1
- package/bundled-docs/design/storage.md +14 -11
- package/bundled-docs/design/telegram.md +1 -1
- package/bundled-docs/design/ws.md +9 -7
- package/bundled-docs/dev/api.md +17 -14
- package/bundled-docs/dev/env.md +2 -2
- package/bundled-docs/dev/logging.md +1 -1
- package/bundled-docs/dev/tools.md +2 -2
- package/bundled-docs/guide/agents.md +1 -1
- package/bundled-docs/guide/cli.md +3 -3
- package/bundled-docs/guide/delegation-and-access.md +1 -1
- package/bundled-docs/guide/workspace.md +1 -1
- package/bundled-docs/requirements/agent.md +1 -1
- package/bundled-docs/requirements/command.md +1 -1
- package/bundled-docs/requirements/settings.md +2 -2
- package/bundled-docs/requirements/workspace.md +4 -4
- package/dist/evo-wrapper.js +913 -201
- package/dist/index.js +2680 -1768
- package/package.json +1 -1
- package/templates/agents/__apply_agent__/AGENT.md +10 -6
- package/templates/agents/__evo_agent__/AGENT.md +15 -4
- package/templates/agents/__score__/AGENT.md +4 -0
- package/templates/skills/cron/SKILL.md +68 -16
- package/templates/skills/cron/manage-cron.py +91 -19
- package/admin-out/_next/static/chunks/1101-d2f818895a3aa32e.js +0 -1
- package/admin-out/_next/static/chunks/1989-c24411a5e971b8b9.js +0 -1
- package/admin-out/_next/static/chunks/5058.8514aa6d6a947cce.js +0 -1
- package/admin-out/_next/static/chunks/app/page-84ab8d21c70823c0.js +0 -1
- package/admin-out/_next/static/css/6da86a077e119a6c.css +0 -1
- /package/admin-out/_next/static/{xg-EvYXc8NEUDwTlmbWvy → j4QFRrUtg33DNMUIgg4ij}/_buildManifest.js +0 -0
- /package/admin-out/_next/static/{xg-EvYXc8NEUDwTlmbWvy → j4QFRrUtg33DNMUIgg4ij}/_ssgManifest.js +0 -0
|
@@ -24,20 +24,24 @@ Let agents draft and validate improvements to their own prompt surface by learni
|
|
|
24
24
|
│ (Phase A'/B'/12 for apply: merge+regress+sync) │
|
|
25
25
|
└────────────────────────────────────────────────┘
|
|
26
26
|
↓
|
|
27
|
-
|
|
28
|
-
|
|
27
|
+
L2/L3: three gates (source full · change category · score)
|
|
28
|
+
↓ pass ↓ fail / below L2
|
|
29
|
+
wrapper queues the apply itself evolution_runs: id → 'awaiting_review'
|
|
30
|
+
(run → 'approved', approved_by=auto) ↓ (admin approves)
|
|
31
|
+
↓ ↓
|
|
29
32
|
evolution_applies: id → 'pending'
|
|
30
33
|
↓ (ticker spawns apply wrapper)
|
|
31
34
|
wrapper merges patches, regress-tests
|
|
32
|
-
↓ (phase 12 sync to main)
|
|
35
|
+
↓ (phase 12: auto re-check, sync to main)
|
|
33
36
|
evolution_applies: id → 'applied'
|
|
34
37
|
evolution_runs: id → 'applied'
|
|
35
|
-
↓
|
|
38
|
+
↓ (auto: one chat:system line in the source session)
|
|
39
|
+
↓ (either: one-click rollback → 'rolled_back')
|
|
36
40
|
SessionManager cold-start picks up new prompts
|
|
37
41
|
next user message uses patched rules
|
|
38
42
|
```
|
|
39
43
|
|
|
40
|
-
All coordination via global db (`~/.halo/global/evo.db`). Wrapper heartbeats every 60s; ticker requeues a stale `running` row (→ `timeout` once `max_attempts` is spent). Detached process model: server restart doesn't kill in-flight wrappers.
|
|
44
|
+
All coordination via global db (`~/.halo/global/evo.db`). Wrapper heartbeats every 60s, only on its own claim (status + attempt); a heartbeat that finds the row re-claimed stops writing, SIGTERMs the sub-cli in flight (its harness.stop ends the shell_exec groups it started), SIGKILLs what is left after 10s and exits the wrapper, so it can't write on into the run / apply dir the new owner shares; ticker requeues a stale `running` row (→ `timeout` once `max_attempts` is spent). Detached process model: server restart doesn't kill in-flight wrappers.
|
|
41
45
|
|
|
42
46
|
## Trigger and Enqueue Flow
|
|
43
47
|
|
|
@@ -58,6 +62,7 @@ Server flow (synchronous, ~10ms):
|
|
|
58
62
|
- Write `source-snapshot.json` (frozen `session.agent.messages`)
|
|
59
63
|
- Write `tool-flow.md` (tool_result-stripped Markdown view for skimming)
|
|
60
64
|
- Write `meta.json` (metadata: id, triggerKind, sourceSession, userHint, workspacePath, createdAt)
|
|
65
|
+
- Record the source session's persisted access level in `evolution_runs.source_access` (null → `full`, the session-agent-builder default) — the L2/L3 source gate reads it. A session that was ever handed a cron webhook event (a `cron_events` row with its `session_id`, any state) records `webhook` instead, at any access level
|
|
61
66
|
- Write `evo-context.json` (snapshot of prompt surface at trigger time: assembled system prompt, all prompt files, agent/skill listings)
|
|
62
67
|
4. INSERT `evolution_runs` row with `status='pending'`, `trigger_kind='note'` (internal trigger id keeps the old name; only the user-visible command was renamed to `/evo`).
|
|
63
68
|
5. Reply via `chat:system`: "📝 Queued for evaluation".
|
|
@@ -68,7 +73,7 @@ Server flow (synchronous, ~10ms):
|
|
|
68
73
|
|
|
69
74
|
Compaction only sets a per-session `compactedThisTurn` flag; `SessionManager.enqueueEvoForCompactedTurn()` consumes it once the compact is done — at the end of `runSession` for auto-compact (mid-turn or turn-end), right after the summary lands for manual `/compact`. A turn that compacts several times therefore enqueues one run.
|
|
70
75
|
|
|
71
|
-
1. Checks: `evolution.level`
|
|
76
|
+
1. Checks: `evolution.level` ≥ L1, `evolution.triggers.pre_compact` on, root session (`parentId` null), and the session's `reply_to` is not `{cron}` — a cron run's own turn (a stored job prompt plus what the run fetched, nobody in the loop) teaches nothing and would otherwise pass the source gate as a full session. The flag is still consumed. `/evo` on that session is not affected.
|
|
72
77
|
2. `enqueueEvoRun()` with `trigger='pre-compact'` snapshots the session's current `rawMessages`. Because the enqueue is deferred, that is the already-compacted history (summary + kept tail), not the pre-compact one.
|
|
73
78
|
3. INSERT `evolution_runs` row.
|
|
74
79
|
|
|
@@ -78,7 +83,7 @@ Located: `/packages/server/src/evolution/ticker.ts`
|
|
|
78
83
|
|
|
79
84
|
Four jobs per pass:
|
|
80
85
|
|
|
81
|
-
1. **Mark timeouts**: `running` rows (applies: also `syncing`) with heartbeat older than `runTimeoutMinutes` (`applyTimeoutMinutes` for applies) → back to `pending` while
|
|
86
|
+
1. **Mark timeouts**: `running` rows (applies: also `syncing`) with heartbeat older than `runTimeoutMinutes` (`applyTimeoutMinutes` for applies) → back to `pending` while the attempts used are below `maxAttempts` (a run counts `attempts - attempts_base`, this round's; an apply has no retry, so `attempts`); exhausted rows → terminal `timeout` (a `syncing` apply → `failed`).
|
|
82
87
|
2. **Start runs**: count current `running`, claim `pending` runs (oldest first, up to slot budget), spawn `evo-wrapper.js --mode=run --id=<X>`.
|
|
83
88
|
3. **Start applies**: same shape, but with per-workspace mutex (only one apply per workspace at a time, since apply reads **current** main to build sandbox).
|
|
84
89
|
4. **Broadcast**: diff in-flight run/apply statuses against the previous tick and push `evolution:run_changed` / `evolution:apply_changed` — wrappers are detached processes and can't reach the server's WS.
|
|
@@ -208,19 +213,64 @@ For each source_run_id:
|
|
|
208
213
|
1. Read `testScenario` from source run's `patch.md`.
|
|
209
214
|
2. Run `halo cli -a <agentId> -n -w <applyDir>/sandbox --access workspace` (testMessage on stdin, same 2700s timer) against merged sandbox.
|
|
210
215
|
3. Spawn `__score__` with regress-mode brief, write `regress/<runId>/score.json`.
|
|
211
|
-
4. Any score with `lint < 50` or `behavior < 50` (or a missing / malformed `score.json`) → regression, abort with `failed`.
|
|
216
|
+
4. Any score with `lint < 50` or `behavior < 50` (or a missing / malformed `score.json`) → regression, abort with `failed`. A previous attempt's `score.json` is deleted before the scorer spawns (run-mode phase C too), so a scorer that dies without writing can't be read as the old attempt's score.
|
|
212
217
|
|
|
213
218
|
### Phase 12 — Final Sync (checkpointed)
|
|
214
219
|
|
|
215
220
|
Four steps, with a checkpoint before the dangerous one:
|
|
216
221
|
|
|
217
|
-
1. **Preflight** (idempotent): walk sandbox, diff against main, snapshot pre-apply files to `history/apply-<id>/` with `MANIFEST.json`.
|
|
218
|
-
|
|
222
|
+
1. **Preflight** (idempotent): walk sandbox, diff against main, snapshot pre-apply files to `history/apply-<id>/` with `MANIFEST.json`. A fresh claim (`running`) first confirms it still owns the row (heartbeat on `running` at its attempt; refused → exits, nothing written), then rewrites `history/apply-<id>/` from scratch: the row never reached `syncing`, so main is still pre-apply, and anything there came from an earlier attempt or a stale wrapper. A resume (`syncing`) keeps the original.
|
|
223
|
+
- **Auto re-check** (`approved_by='auto'`, not on resume): the files preflight will publish are classified again against the live surface, and the level is read again (the user may have lowered it since). Blocked → apply `failed` with `auto_recheck:<reason>`, main untouched. See [Auto-apply](#auto-apply-l2--l3).
|
|
224
|
+
2. **Checkpoint** (`markSyncing`): set `evolution_applies.status='syncing'` (a resume only refreshes the heartbeat). From here on, if wrapper crashes, ticker recovery resumes at step 3 (skip A'/B'/preflight to avoid re-running LLM phases).
|
|
219
225
|
3. **Publish** (dangerous): cp each changed file from sandbox to main.
|
|
220
|
-
4.
|
|
226
|
+
4. `finalizeApplied`, one transaction: apply row `applied`, its still-`approved` source runs `applied`.
|
|
227
|
+
|
|
228
|
+
A wrapper's terminal writes (run `finalize` / `markObservation`, apply `finalizeApply`) only land while the row is still `running` at the attempt that wrapper was spawned on: a re-claimed row belongs to the newer wrapper. Phase 12's own writes hold the same condition: the checkpoint lands only on `running` (resume: `syncing`) at this attempt, and refused, the wrapper exits without touching main; `finalizeApplied` lands only on `syncing` at this attempt. A resume whose preflight fails, or a publish that throws, releases the `syncing` row at once (clears started / heartbeat, records the reason in `failure_reason`) for the ticker, which re-resumes it and, once retries run out, fails it with that reason and keeps the run `approved`. A new resume claim clears the reason.
|
|
221
229
|
|
|
222
230
|
Per-workspace mutex in ticker prevents two applies from racing on the same workspace — the assumption phase 12 leans on.
|
|
223
231
|
|
|
232
|
+
## Auto-apply (L2 / L3)
|
|
233
|
+
|
|
234
|
+
Code: `evolution/auto-apply.ts` (pure, no agent-loader import — it runs in the detached wrapper), hook `maybeAutoApply` in `evo-wrapper.ts`, shared enqueue `queue-apply.ts`.
|
|
235
|
+
|
|
236
|
+
**Where.** At L2+, right after phase C the run-mode wrapper decides and writes `runs/<id>/auto-apply.json`. A pass calls `queueApply({ mode: 'auto', attempts })`: one transaction flips the run `running → approved` (only while it is still `running` at this wrapper's attempt) and inserts a `pending` apply with `approved_by='auto'`. A pass never goes through `awaiting_review`, so it can't race a reviewer. A fail (or below L2) finalizes `awaiting_review` as before. The ticker needs no change — its existing broadcast diff pushes `running → approved` and the new apply.
|
|
237
|
+
|
|
238
|
+
**Three gates**, checked in this order; the first that fails is the recorded `reason`:
|
|
239
|
+
|
|
240
|
+
1. **Source access** — the run's `source_access` must be `full`. A `workspace` / `readonly` session's conversation (a channel guest's, say) must not rewrite prompts every session runs on without a reviewer. Rows from before the column read null → never auto-applied. Reason `source_access`. A run recorded `webhook` (its session took cron webhook events — payloads from outside, see enqueue above) never auto-applies either, reason `source_webhook`, whichever trigger enqueued it.
|
|
241
|
+
2. **Change category** — the sandbox is diffed against the live pre-patch surface (workspace file, else global for `agents/` / `skills/` / `prompts/` dirs the workspace doesn't override; byte-equal copies of a global folder drop out). Each changed file is `prompt`, `capability` or `manual`; the strictest wins. Prompt needs L2, capability L3, manual never. Reasons `level`, `no_effective_change`, `category_manual:<rel>(<why>)`.
|
|
242
|
+
3. **Score** — `avg ≥ auto_apply_threshold`, `confidence: high`, `behavior ≥ 70`, `lint = 100`, all four `checks` passed (`patchInSandbox: pass`, `usedInDryRun: yes`, `probe: valid`, `probeLanguage: same`). The behavior floor matters: 100/50/100 averages 83. Reasons `score_missing`, `below_threshold`, `confidence`, `behavior_floor`, `lint_floor`, `checks:<name>`.
|
|
243
|
+
|
|
244
|
+
| Change | Category |
|
|
245
|
+
|---|---|
|
|
246
|
+
| `INSTRUCTIONS.md`, `INDEX.md`, `agents/<id>/AGENT.md`, `prompts/{bootstrap,all,root}/*.md` | prompt |
|
|
247
|
+
| `SKILL.md` body / `name` / `description` only | prompt |
|
|
248
|
+
| `SKILL.md` `disable-model-invocation` / `user-invocable`; `verbs` with no `requiresAccess` change | capability |
|
|
249
|
+
| `agent.yaml` `description` / `system_prompt` / `context`; `tools` removed or a non-egress tool added; `skills` removed; `model.id` / `maxTokens` / `promptCaching` / `thinking` / `verbosity` | capability |
|
|
250
|
+
| `agent.yaml` `priority` (routing), `team` / `internal` / `name` / unknown keys, `model.provider` / `model.endpoint`, `tools` adding `shell_exec` / `web_fetch` / `relay_send` / `a2a_send` / `node_*`, `skills` adding one | manual |
|
|
251
|
+
| `SKILL.md` `command` (routing: verb gates only apply to object commands `SUBCOMMAND_ROUTES` knows, so a rename drops them), `requiresAccess`, any verb's effective `requiresAccess` (own or object-level fallback; a verb added or removed counts), other frontmatter keys | manual |
|
|
252
|
+
| `USER.md`; any other file in a skill / agent folder; a new agent / skill dir; a deleted file; an internal agent (live `agent.yaml` `internal: true`, or a `__` id); unrecognized paths | manual |
|
|
253
|
+
|
|
254
|
+
Egress tools and `relay_send` / `node_*` are manual because the dry-run runs `--access workspace`, where they aren't even injected — a score is no evidence about them.
|
|
255
|
+
|
|
256
|
+
**Phase-12 re-check.** An auto apply is gated again before `markSyncing`: the files preflight will publish are classified against the live surface and the level is re-read (the apply agent may touch files the run didn't; the user may have lowered the level). Publish never deletes, so a workspace file the merge dropped is not a change here; a new override dir that would hide a global file the sandbox lacks still is. The score gate isn't repeated — phase B' already held the merge to the regression floor. Blocked → `auto_recheck:<reason>`, main untouched.
|
|
257
|
+
|
|
258
|
+
**Failure.** Any failure of an auto apply (merge, regress, re-check, timeout) fails the apply and sends its run back to `awaiting_review` in the same transaction (`failApply`), so a reviewer can approve it by hand. Reviewer applies keep their run `approved`.
|
|
259
|
+
|
|
260
|
+
**Notice.** Each tick, `notifyAutoApplied` (`evolution/notify.ts`) finds `approved_by='auto' AND status='applied' AND notified_at IS NULL` (db, so restarts neither resend nor miss) and sends one `chat:system` line to the source session: the effectively changed files (MANIFEST minus `unchanged`, > 3 summarised), score, and an observation preview. `notified_at` is stamped even when the session is gone.
|
|
261
|
+
|
|
262
|
+
## Rollback
|
|
263
|
+
|
|
264
|
+
`POST /api/evolution/applies/:id/rollback { force? }` (`evolution/rollback.ts`), for reviewer and auto applies alike. Synchronous fs with no await, so the in-process ticker can't interleave. Source of truth is `history/apply-<id>/` on disk.
|
|
265
|
+
|
|
266
|
+
1. Preconditions (409): apply `applied` (`not_applied`), no `pending` / `running` / `syncing` apply in the workspace (`apply_in_flight`), a readable MANIFEST whose paths stay inside `.halo/` with every `overwrite` backup present (`no_snapshot`).
|
|
267
|
+
2. Per-file state against main now: `clean` (hash = `after`), `reverted` (already back: hash = `before`, or a `new` file that is gone — skipped, not a conflict), `modified`, `missing`, `unverified` (old manifest, no hashes). Any of the last three without `force` → 409 `conflict` with `conflicts[{rel, reason}]`.
|
|
268
|
+
3. Then (conflicts cleared or forced): any `overwrite` backup that doesn't hash to `before` → 409 `backup_mismatch` with `files`, `force` or not — restoring it would write something else. Nothing has been written at this point.
|
|
269
|
+
4. Copy what will be overwritten / removed to `rollback-pre/`, restore `overwrite` files, remove `new` ones, then `rmdir` each `createdDirs` entry deepest first. An emptied `agents/<id>/` (`skills/`, `prompts/`) dir must go: it would override the global one wholesale. A dir someone has since put files in is kept and reported in `keptDirs`; the UI lists them after the rollback, since each still overrides the global agent / skill / prompts of that name.
|
|
270
|
+
5. Write `ROLLBACK.json`; in one transaction the apply and its `applied` source runs become `rolled_back`; broadcast.
|
|
271
|
+
|
|
272
|
+
`200 { ok, restored, removed, removedDirs, keptDirs }`. A second call answers `not_applied`. Limits: an apply row is archived 14 days after it is terminal and purged 30 days later — no rollback entry after that; deleting the run deletes the snapshot too; rollback restores files, not session behaviour already shown.
|
|
273
|
+
|
|
224
274
|
## Data on Disk
|
|
225
275
|
|
|
226
276
|
```
|
|
@@ -234,6 +284,8 @@ Per-workspace mutex in ticker prevents two applies from racing on the same works
|
|
|
234
284
|
patch.md # __evo_agent__ writes (has testScenario frontmatter)
|
|
235
285
|
.skip.md # evo wrote this → no patch proposed
|
|
236
286
|
score.json # __score__ writes (lint/behavior/scope/avg + checks)
|
|
287
|
+
auto-apply.json # L2+ verdict (gates, files, score, observation);
|
|
288
|
+
# not an input file, so a retry clears it
|
|
237
289
|
dry-run-output.txt # stdout from phase B dry-run
|
|
238
290
|
dry-run-fail-<n>.log # failure logs from dry-run attempts
|
|
239
291
|
sub-cli.log # tee'd stdout+stderr of every halo cli spawn
|
|
@@ -249,8 +301,10 @@ Per-workspace mutex in ticker prevents two applies from racing on the same works
|
|
|
249
301
|
dry-run-fail.log # only when the regress dry-run failed
|
|
250
302
|
score.json
|
|
251
303
|
history/apply-<id>/ # rollback snapshot (NOT archived)
|
|
252
|
-
MANIFEST.json #
|
|
253
|
-
<files> # pre-apply content
|
|
304
|
+
MANIFEST.json # files[{rel, kind, before, after, unchanged?}] + createdDirs
|
|
305
|
+
<files> # pre-apply content of `overwrite` files
|
|
306
|
+
rollback-pre/<files> # main's content just before a rollback overwrote / removed it
|
|
307
|
+
ROLLBACK.json # what the rollback did (per file action, dirs removed / kept)
|
|
254
308
|
archive/
|
|
255
309
|
run-<id>.zip # zipped after 14 days in terminal state
|
|
256
310
|
apply-<id>.zip
|
|
@@ -288,13 +342,28 @@ literals.
|
|
|
288
342
|
|
|
289
343
|
Skill content NOT inlined (too large at scale); only listings. Both agents and skills include a `scope` tag (`workspace` | `global` | `builtin`).
|
|
290
344
|
|
|
345
|
+
`MANIFEST.json` format (written by a fresh claim's preflight, when the sandbox is final and main still pre-apply):
|
|
346
|
+
|
|
347
|
+
```json
|
|
348
|
+
{
|
|
349
|
+
"applyId": "…", "sourceRunIds": ["…"], "capturedAt": "<ISO>",
|
|
350
|
+
"files": [
|
|
351
|
+
{ "rel": "INSTRUCTIONS.md", "kind": "overwrite", "before": "<sha256>", "after": "<sha256>" },
|
|
352
|
+
{ "rel": "agents/helper/agent.yaml", "kind": "new", "before": null, "after": "<sha256>", "unchanged": true }
|
|
353
|
+
],
|
|
354
|
+
"createdDirs": ["agents/helper", "agents"]
|
|
355
|
+
}
|
|
356
|
+
```
|
|
357
|
+
|
|
358
|
+
`rel` uses native separators. `before` / `after` are sha256 of main before and of what was published; `unchanged` marks a copy byte-equal to the global file it now overrides (completing a new workspace override dir — not an effective change, left out of notices and the admin file list); `createdDirs` are the dirs publish created, deepest first. Manifests from before this format have no hashes / `createdDirs`.
|
|
359
|
+
|
|
291
360
|
## Archive and Retention
|
|
292
361
|
|
|
293
362
|
Located: `/packages/server/src/evolution/archive.ts`
|
|
294
363
|
|
|
295
364
|
Two-stage lifecycle (both orthogonal to `status`):
|
|
296
365
|
|
|
297
|
-
- **14 days after terminal status** (applied/rejected/skipped/failed/timeout): zip the run/apply dir to `archive/{run|apply}-<id>.zip`, delete original, set `archived_at = now()`. DB row stays.
|
|
366
|
+
- **14 days after terminal status** (applied/rejected/skipped/failed/timeout/rolled_back): zip the run/apply dir to `archive/{run|apply}-<id>.zip`, delete original, set `archived_at = now()`. DB row stays.
|
|
298
367
|
- **30 days after `archived_at`**: delete zip and DB row outright.
|
|
299
368
|
|
|
300
369
|
Active rows (pending/running/awaiting_review/approved/syncing) never archived. `history/apply-<id>/` (rollback tree) never archived — kept cheap and discoverable.
|
|
@@ -318,6 +387,10 @@ After apply publishes to main, **no explicit session-release step needed**. Sess
|
|
|
318
387
|
| `/packages/server/src/evolution/spawn.ts` | Real spawner: detached Node child running `evo-wrapper.js` from its own module's directory (`dist/evo-wrapper.js` in the npm bundle — build-bundle's second entry). stdin/stdout ignored, stderr appended to the run / apply log; a non-zero exit logs an `[EvoSpawn]` error. `spawnEvoWrapper(path, mode, id)` takes the wrapper path for tests; the ticker's spawner is still override-able via `setEvoSpawner`. |
|
|
319
388
|
| `/packages/server/src/evolution/evo-wrapper.ts` | Wrapper orchestrator. 3-phase run mode (draft/dry-run/score). Apply mode (merge/regress/publish). Heartbeat every 60s. Handles Windows command-line limits via stdin briefs. |
|
|
320
389
|
| `/packages/server/src/evolution/archive.ts` | Archive job: 14d → zip (pure-JS jszip) + delete, 30d → purge. Runs daily + at boot. Idempotent. |
|
|
390
|
+
| `/packages/server/src/evolution/auto-apply.ts` | L2/L3 decision: live-baseline diff (`collectChanges` / `publishChanges`), per-file classification, the source / category / score gates. Pure; loaded by the wrapper. |
|
|
391
|
+
| `/packages/server/src/evolution/queue-apply.ts` | `queueApply` (reviewer approve + auto, one transaction) and `failApply` (auto failure → run back to `awaiting_review`). |
|
|
392
|
+
| `/packages/server/src/evolution/rollback.ts` | `inspectRollback` / `rollbackApply` from the history MANIFEST; `readPublishedFiles`. |
|
|
393
|
+
| `/packages/server/src/evolution/notify.ts` | `notifyAutoApplied`: one `chat:system` line per landed auto apply, driven by `notified_at`. |
|
|
321
394
|
|
|
322
395
|
## Sandbox Model
|
|
323
396
|
|
|
@@ -337,7 +410,8 @@ Session rows, db, logs, memory — all intentionally excluded from sandbox. Avoi
|
|
|
337
410
|
general:
|
|
338
411
|
language: en-US # BCP-47; also used for evo/score output language
|
|
339
412
|
evolution:
|
|
340
|
-
level: L0 # L0
|
|
413
|
+
level: L0 # L0..L3, see below
|
|
414
|
+
auto_apply_threshold: 80 # 0–100, L2/L3 only; out of range → 400 on save
|
|
341
415
|
max_concurrent_run: 1
|
|
342
416
|
max_concurrent_apply: 1
|
|
343
417
|
run_timeout_minutes: 12
|
|
@@ -347,18 +421,21 @@ general:
|
|
|
347
421
|
pre_compact: true
|
|
348
422
|
```
|
|
349
423
|
|
|
350
|
-
L0 = manual drafting only (`/evo`); L1 also enables pre-compact triggering. Defaults conservative; tune if machine handles more.
|
|
424
|
+
L0 = manual drafting only (`/evo`); L1 also enables pre-compact triggering; L2 also auto-applies prompt-only changes that pass the gates; L3 also auto-applies capability changes. The level only decides how runs are enqueued and whether the wrapper queues an apply itself — runs already queued always finish. All evolution keys are global-only. Defaults conservative; tune if machine handles more.
|
|
351
425
|
|
|
352
426
|
## Admin UI Integration
|
|
353
427
|
|
|
354
428
|
Top-level "Evolution" tab shows:
|
|
355
429
|
|
|
356
|
-
- **List**: evolution_runs + latest apply per run, newest first (cursor-paginated by `created_at`, up to 300 loaded). Filterable by status, plus an `archived` view.
|
|
357
|
-
- **Detail**: patch.md (raw), score.json, the `.skip.md` reason, a snapshot summary (message count, first user / assistant message), wrapper log + sub-cli log.
|
|
430
|
+
- **List**: evolution_runs + latest apply per run, newest first (cursor-paginated by `created_at`, up to 300 loaded). Filterable by status (incl. `rolled_back`), plus an `archived` view and two client-side filters: `auto` (latest apply `approved_by='auto'`) and `observation` (`has_observation`). Rows carry an AUTO tag, a lightbulb when the draft has an observation, and an "apply failed" mark.
|
|
431
|
+
- **Detail**: patch.md (raw), score.json, the `.skip.md` reason, a snapshot summary (message count, first user / assistant message), wrapper log + sub-cli log. At L2+: the auto-applied banner (level, score, the files the apply effectively changed — `publishedFiles`) or the translated reason it wasn't auto-applied. The Larger observation block (see below), and a **Rollback** button while the latest apply is `applied`.
|
|
358
432
|
- **Approve**: Opens an inline panel, optional reviewer_hint. Flips the run to `approved` **and** inserts the pending `evolution_applies` row **inside one `db.transaction`** — it's a single reviewer decision, and half of it (run flipped, apply missing) strands the run in a status the ticker never advances.
|
|
359
433
|
- **Reject**: Sets status='rejected'.
|
|
360
|
-
- **Retry**: Resets row to pending, requires new hint.
|
|
361
|
-
- **
|
|
434
|
+
- **Retry**: Resets row to pending, requires new hint. `attempts` is not cleared (wrapper claims compare it, so a zeroed counter could hand a stale wrapper its claim back); `attempts_base = attempts` restarts the retry budget, and the API's `attempts` is `attempts - attempts_base`, this round's count. The route refuses `pending` / `running` / `applied` (roll back first) and an `approved` run unless its latest apply failed / timed out (409); the UI offers it on `awaiting_review`, `failed` / `timeout` / `skipped` / `rejected` / `rolled_back`, and on an `approved` run whose apply failed.
|
|
435
|
+
- **Rollback**: see [Rollback](#rollback). On `conflict` the UI lists the files and asks again before retrying with `force`; `backup_mismatch` is shown as a refusal.
|
|
436
|
+
- **Manual Delete**: Removes artifacts (live dir + archive zip, whichever exists) + DB row immediately. Blocked on `pending` / `running` / `approved` runs; also deletes the applies the run produced — and with them the rollback snapshot (the confirm says so).
|
|
437
|
+
|
|
438
|
+
**Larger observation.** A draft may end with a `## Larger observation` section (patch.md body or `.skip.md`): something bigger than one prompt patch. The wrapper flags the run `has_observation` after phase A and lifts the section (≤ 2000 chars) into `auto-apply.json`; the detail pane shows it with two prefilled GitHub-issue links — the workspace's github.com origin (if any) and `turmind/halo-agent`. The link only opens github.com's form (nothing is sent until the user submits); the body lists run id, ≤ 20 published files, score and Halo version, cut on grapheme boundaries to fit 8000 URL chars, and a hint warns the text may quote workspace files or the conversation.
|
|
362
439
|
|
|
363
440
|
Realtime updates via WebSocket (`evolution:run_changed`, `evolution:apply_changed` events). REST mutations broadcast directly; wrapper state changes broadcast at each ticker tick (up to 30s latency).
|
|
364
441
|
|
|
@@ -366,9 +443,9 @@ Realtime updates via WebSocket (`evolution:run_changed`, `evolution:apply_change
|
|
|
366
443
|
|
|
367
444
|
**Server restart**: Ticker re-evaluates db on first pass. Old `running` rows timeout naturally; `pending` rows continue to be picked up. Applies in `syncing` state are treated as resume candidates — ticker clears heartbeat and spawns wrapper which sees `status='syncing'` on entry and skips LLM phases.
|
|
368
445
|
|
|
369
|
-
**Wrapper crash**: Heartbeat expires after `runTimeoutMinutes` (default 12min). If
|
|
446
|
+
**Wrapper crash**: Heartbeat expires after `runTimeoutMinutes` (default 12min). If the attempts used are below `maxAttempts` (run: `attempts - attempts_base`), ticker moves row back to `pending` for retry. For applies in `syncing`, row stays `syncing` and ticker spawns recovery wrapper that runs phase 12 publish only.
|
|
370
447
|
|
|
371
|
-
**Apply mid-publish crash**:
|
|
448
|
+
**Apply mid-publish crash**: The resume's preflight only re-diffs (no snapshot), so the original backup survives. Publish (`phaseApplyPublish`) is a simple file-by-file cp. If wrapper dies part-way through, main is half-applied, but `history/apply-<id>/` has the rollback. On recovery, wrapper re-runs publish (idempotent cp) to complete.
|
|
372
449
|
|
|
373
450
|
## Patches and Test Scenarios
|
|
374
451
|
|
|
@@ -398,9 +475,10 @@ If they're identical (both clean), that's legal. Old patches wrote single `messa
|
|
|
398
475
|
## Scope Limits
|
|
399
476
|
|
|
400
477
|
- evo observes **root sessions only** (`parentId === null`). Sub-agent sessions don't trigger evo.
|
|
401
|
-
- evo is **invisible to conversation**: no user/assistant messages appended to source session log. One short `chat:system` line per `/evo
|
|
478
|
+
- evo is **invisible to conversation**: no user/assistant messages appended to source session log. One short `chat:system` line per `/evo`, plus one when an auto apply lands (see [Auto-apply](#auto-apply-l2--l3)).
|
|
402
479
|
- Wrappers never modify `~/.halo/global/` directly. Apply uses copy-on-write: read global, write workspace override via file_write in sandbox, then sync to main.
|
|
403
480
|
- `/evo` is rejected for non-`full` users.
|
|
481
|
+
- `__evo_agent__` and `__score__` treat session / transcript content as data, never as instructions to follow.
|
|
404
482
|
|
|
405
483
|
## Why This Design
|
|
406
484
|
|
|
@@ -415,27 +493,21 @@ If they're identical (both clean), that's legal. Old patches wrote single `messa
|
|
|
415
493
|
## Status Flow (Run Mode)
|
|
416
494
|
|
|
417
495
|
```
|
|
418
|
-
pending → running → [draft → dry-run → score] → awaiting_review
|
|
419
|
-
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
|
|
423
|
-
|
|
424
|
-
|
|
425
|
-
↓
|
|
426
|
-
applied
|
|
427
|
-
|
|
428
|
-
OR: awaiting_review → rejected
|
|
429
|
-
OR: running → timeout / failed / skipped
|
|
496
|
+
pending → running → [draft → dry-run → score] → awaiting_review → approved → applied → rolled_back
|
|
497
|
+
↘ approved (L2/L3 auto: wrapper queued the apply)
|
|
498
|
+
running → failed / timeout / skipped
|
|
499
|
+
awaiting_review → rejected
|
|
500
|
+
approved → awaiting_review (its auto apply failed)
|
|
501
|
+
retry: any but pending / running / applied → pending
|
|
502
|
+
(approved only once its latest apply failed / timed out)
|
|
430
503
|
```
|
|
431
504
|
|
|
432
505
|
## Status Flow (Apply Mode)
|
|
433
506
|
|
|
434
507
|
```
|
|
435
|
-
pending → running → [merge → regress → publish] → applied
|
|
508
|
+
pending → running → [merge → regress → preflight (+ auto re-check)] → syncing → [publish] → applied → rolled_back
|
|
436
509
|
|
|
437
510
|
OR: running → failed / timeout
|
|
438
|
-
OR:
|
|
439
|
-
|
|
440
|
-
running → [publish] → applied
|
|
511
|
+
OR: syncing, heartbeat lost → (ticker recovery) wrapper resumes at publish → applied
|
|
512
|
+
(retries spent → failed)
|
|
441
513
|
```
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# Linked machines (`halo node`)
|
|
2
2
|
|
|
3
|
-
> User guide: [guide/node.md](../guide/node.md). Tool reference: [dev/tools.md → Node tools](../dev/tools.md#node-tools). API: [dev/api.md → Linked machines](../dev/api.md#linked-machines-halo-node). Code: `packages/server/src/nodes/` (wired in `index.ts`, `session-agent-builder.ts
|
|
3
|
+
> User guide: [guide/node.md](../guide/node.md). Tool reference: [dev/tools.md → Node tools](../dev/tools.md#node-tools). API: [dev/api.md → Linked machines](../dev/api.md#linked-machines-halo-node). Code: `packages/server/src/nodes/` (wired in `index.ts`, `session-agent-builder.ts`), CLI `packages/cli/src/index.ts` (`cmdNode`), Go client `packages/node-go/`, admin `features/nodes/`.
|
|
4
4
|
|
|
5
5
|
## Problem
|
|
6
6
|
|
|
@@ -12,8 +12,8 @@ The server's agents work on the server's filesystem. Users also want them to tou
|
|
|
12
12
|
|---|---|---|
|
|
13
13
|
| Accounts | `nodes/accounts.ts` | Pairing tokens in `~/.halo/secrets/channels/channels.db` `channel_accounts` (`channel_type='node'`, `label` = machine name, `config` = `{ token }`, `access_level` = `'full'`). Name `^[a-z0-9][a-z0-9-]{0,31}$`, unique per workspace (realpath compare) → 409 |
|
|
14
14
|
| Hub | `nodes/hub.ts` | Server side. `noServer` `WebSocketServer` on `/api/nodes/connect`; one live connection per account; request/response correlation, timeouts, cancel, heartbeat; `nodes:changed` broadcast. Process singleton via `setNodeHub` / `getNodeHub` |
|
|
15
|
-
| Tools | `nodes/tools.ts` | `buildNodeTools(workspaceRoot)` → the fixed 8 tools. Dispatch: hub (in-process) →
|
|
16
|
-
| Routes | `nodes/routes.ts` | Admin CRUD (`/api/nodes/accounts`)
|
|
15
|
+
| Tools | `nodes/tools.ts` | `buildNodeTools(workspaceRoot)` → the fixed 8 tools. Dispatch: hub (in-process) → `NODES_UNAVAILABLE` |
|
|
16
|
+
| Routes | `nodes/routes.ts` | Admin CRUD (`/api/nodes/accounts`) |
|
|
17
17
|
| Client | `nodes/node-client.ts` | `runNode()` — the `halo node` process: connect, reconnect with backoff, execute requests |
|
|
18
18
|
| Executor | `nodes/node-exec.ts` + `node-path.ts` | Machine side. Wraps the **same** `createWorkspaceTools(base, { accessLevel: 'full' })` the server's agents use; adds the root check |
|
|
19
19
|
| Go client | `packages/node-go/` | `halo-node`: a static Go binary (linux amd64 / arm64 / armv7, no cgo, ~7 MB) speaking the same protocol v1 with the same flags. It ports the client, the executor and the parts of `workspace-tools.ts` / `sandbox.ts` it needs (the seven ops, rm guard, process-group kill), with JS value, UTF-16 and V8 regex semantics (`internal/jsv`, `internal/jsregex` over regexp2). Ships as release assets via `.github/workflows/node-go.yml`, not npm |
|
|
@@ -100,16 +100,9 @@ When `halo node` itself exits — Ctrl-C (`stop`) or a fatal refusal such as a r
|
|
|
100
100
|
|
|
101
101
|
Offline vs unknown are distinct errors: `machine "<n>" is offline (…). Do not retry in a loop — tell the user to start it.` vs `unknown machine "<n>" — machines linked to this workspace: a, b (see node_list)`.
|
|
102
102
|
|
|
103
|
-
## Cron
|
|
103
|
+
## Cron runs
|
|
104
104
|
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
- `HALO_NODE_RELAY_URL` = `http://127.0.0.1:<port>/api/nodes/call`
|
|
108
|
-
- `HALO_NODE_RELAY_TOKEN` = a 32-byte hex secret generated per server boot (never persisted)
|
|
109
|
-
|
|
110
|
-
The child's node tools POST `{ workspace, node, op, args }` there (`op: 'nodes'` = node_list) with header `x-halo-node-relay`. The route is in `PUBLIC_PATHS` and gated by loopback source **and** a `timingSafeEqual` match on the secret — the secret is the real gate (a same-host reverse proxy is loopback too). The secret is **server-wide, not bound to a workspace**: the body's `workspace` picks the machines, so whoever holds it can reach every workspace's machines. `child-env.ts` strips `HALO_NODE_RELAY_URL` / `HALO_NODE_RELAY_TOKEN` / `HALO_NODE_TOKEN` from every other child (shell_exec and the rest of `cleanChildEnv`'s users), so they don't appear in agent subprocess env. That is env hygiene, not a boundary: a full-level command runs unsandboxed (no bwrap) as the same user and can read that user's processes and files — the cron child's environment included — anyway.
|
|
111
|
-
|
|
112
|
-
The relay request carries its own timeout (`max(shell timeout, 120 s) + 30 s`, node:http — fetch's fixed 300 s headers timeout would cut a long node_run first); the hub's per-call timeout normally answers well before it.
|
|
105
|
+
Cron runs execute in the server process since 2.2 ([cron.md](cron.md#execution-runjob)), so their node tools reach the hub directly like any other server-side session. The loopback relay the old `halo cli` child used (`POST /api/nodes/call`, a per-boot secret in `HALO_NODE_RELAY_URL` / `HALO_NODE_RELAY_TOKEN`) is gone. `child-env.ts` still strips `HALO_NODE_TOKEN` from every child it builds, so a `halo node`'s own `node_run` commands don't see its pairing token — env hygiene, not a boundary: a full-level command runs unsandboxed as the same user and can read that user's processes and files anyway.
|
|
113
106
|
|
|
114
107
|
## Prompt-cache invariant
|
|
115
108
|
|
|
@@ -2,20 +2,20 @@
|
|
|
2
2
|
|
|
3
3
|
## Overview
|
|
4
4
|
|
|
5
|
-
A vendor-neutral observability layer built on the official OpenTelemetry JS SDK. One setting, `general.observability.endpoint`, turns on OTLP http/protobuf export of traces, metrics and logs to whatever collector the operator points at — empty means fully off. It applies to every process that runs the agent loop: the server, `halo cli` and `halo tui` — and so the
|
|
5
|
+
A vendor-neutral observability layer built on the official OpenTelemetry JS SDK. One setting, `general.observability.endpoint`, turns on OTLP http/protobuf export of traces, metrics and logs to whatever collector the operator points at — empty means fully off. It applies to every process that runs the agent loop: the server (cron jobs included — they run in-process since 2.2), `halo cli` and `halo tui` — and so the evolution runs, which are spawned `halo cli` children. Principle: zero AWS (or any vendor) code in the server — SigV4 signing, backend endpoints, and resource enrichment all live in the collector, not in halo.
|
|
6
6
|
|
|
7
7
|
## Configuration
|
|
8
8
|
|
|
9
9
|
| key | type | default | description |
|
|
10
10
|
|---|---|---|---|
|
|
11
|
-
| `general.observability.endpoint` | string | `''` | OTLP base URL of an OpenTelemetry collector (e.g. `http://localhost:4318`). Traces, metrics and logs all exported over OTLP http/protobuf — `http://` or `https://`, on the collector's OTLP/HTTP port (4318 by default); gRPC (4317) not supported. Empty = off. Server: restart required;
|
|
12
|
-
| `general.observability.service_name` | string | `halo` | OTel resource `service.name`. Server: restart required;
|
|
13
|
-
| `general.observability.headers` | string, **secret** | `''` | Extra headers for every OTLP request, comma-separated `k=v` (e.g. `authorization=Bearer …`). Server: restart required;
|
|
14
|
-
| `general.observability.capture_content` | boolean | `false` | Put prompt / completion / tool argument+result text on spans (`gen_ai.*.messages` etc.). Off = metadata only (model, tokens, latency, tool names). Env `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` overrides it (below). Server: restart required;
|
|
11
|
+
| `general.observability.endpoint` | string | `''` | OTLP base URL of an OpenTelemetry collector (e.g. `http://localhost:4318`). Traces, metrics and logs all exported over OTLP http/protobuf — `http://` or `https://`, on the collector's OTLP/HTTP port (4318 by default); gRPC (4317) not supported. Empty = off. Server (cron runs included): restart required; evolution runs: next run. |
|
|
12
|
+
| `general.observability.service_name` | string | `halo` | OTel resource `service.name`. Server (cron runs included): restart required; evolution runs: next run. |
|
|
13
|
+
| `general.observability.headers` | string, **secret** | `''` | Extra headers for every OTLP request, comma-separated `k=v` (e.g. `authorization=Bearer …`). Server (cron runs included): restart required; evolution runs: next run. |
|
|
14
|
+
| `general.observability.capture_content` | boolean | `false` | Put prompt / completion / tool argument+result text on spans (`gen_ai.*.messages` etc.). Off = metadata only (model, tokens, latency, tool names). Env `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` overrides it (below). Server (cron runs included): restart required; evolution runs: next run. |
|
|
15
15
|
|
|
16
|
-
All four keys are `globalOnly`. The server's own exporter switches only on restart — `otel.ts` reads them once at boot, before anything else touches the OTel API. A `halo cli` / `halo tui` process reads them the same way at its own start, so a change reaches the next run — including the
|
|
16
|
+
All four keys are `globalOnly`. The server's own exporter switches only on restart — `otel.ts` reads them once at boot, before anything else touches the OTel API. A `halo cli` / `halo tui` process reads them the same way at its own start, so a change reaches the next run — including the evolution children the server spawns, which don't inherit the server's boot-time values (see the child-env rule below). A changed or cleared endpoint therefore reaches their next run without a server restart.
|
|
17
17
|
|
|
18
|
-
**Which process exported.** Every process shares `service.name`, so the resource also carries `halo.process` = `server` \| `cli` \| `tui` (`HaloProcess` in `otel.ts`; `initObservability()` defaults to `server`).
|
|
18
|
+
**Which process exported.** Every process shares `service.name`, so the resource also carries `halo.process` = `server` \| `cli` \| `tui` (`HaloProcess` in `otel.ts`; `initObservability()` defaults to `server`). Evolution children are `halo cli` runs, so they report `cli`; their `session.id` (the `__evo_agent__`-style agent name) tells them apart from a hand-run cli. Cron runs execute in the server since 2.2 and report `server`. The startup log line names it: `[Observability] OTLP export → … (service.name=…, halo.process=cli, …)`.
|
|
19
19
|
|
|
20
20
|
**Env-var mapping.** `initObservability()` maps settings onto the standard `OTEL_*` env vars with `??=`, so an operator who already exports them (or `--require`s an external distro) wins:
|
|
21
21
|
|
|
@@ -24,11 +24,11 @@ All four keys are `globalOnly`. The server's own exporter switches only on resta
|
|
|
24
24
|
- `service_name` → `OTEL_SERVICE_NAME`
|
|
25
25
|
- always `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` (declarative — the `*-otlp-proto` exporters are hard-wired to it; this just makes an externally-loaded SDK agree on wire protocol)
|
|
26
26
|
|
|
27
|
-
`capture_content` goes the other way: it isn't mapped onto an env var; instead `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` (the OTel GenAI instrumentation's own switch), when exported as `true` / `false` / `1` / `0` (any case), beats the setting in both directions (`config.ts` `envBool`); unset or any other value → the setting. Read once at process start, like the rest. This is what lets a container (an AgentCore runtime) turn content capture on or off from its env alone, with no settings edit. A server that exports it passes it on to its
|
|
27
|
+
`capture_content` goes the other way: it isn't mapped onto an env var; instead `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` (the OTel GenAI instrumentation's own switch), when exported as `true` / `false` / `1` / `0` (any case), beats the setting in both directions (`config.ts` `envBool`); unset or any other value → the setting. Read once at process start, like the rest. This is what lets a container (an AgentCore runtime) turn content capture on or off from its env alone, with no settings edit. A server that exports it passes it on to its evolution children (it isn't in `settingsOtelEnvKeys`).
|
|
28
28
|
|
|
29
29
|
No `OTEL_EXPORTER_OTLP_ENDPOINT` after the mapping → `enabled=false`, `initObservability()` returns without registering anything.
|
|
30
30
|
|
|
31
|
-
**Child env.** Every key the mapping actually set (one that was unset before) is recorded in `settingsOtelEnvKeys` (`child-env.ts`), and `cleanChildEnv()` strips those keys from every child it builds. Inherited, the server's boot-time values would win
|
|
31
|
+
**Child env.** Every key the mapping actually set (one that was unset before) is recorded in `settingsOtelEnvKeys` (`child-env.ts`), and `cleanChildEnv()` strips those keys from every child it builds. Inherited, the server's boot-time values would win an evolution `halo cli` child's own `??=`, so a changed endpoint would never arrive and a cleared one would keep exporting until a server restart. Operator-exported `OTEL_*` keys aren't recorded and still pass through ("operator wins" holds in children too). Side effect: terminals, sandbox shells and the Jupyter server — also built from `cleanChildEnv()` — no longer inherit Halo's settings-derived collector config.
|
|
32
32
|
|
|
33
33
|
**External-SDK detection.** If an external SDK is already registered (e.g. via `node --require` of a vendor distro) — detected by comparing `ProxyTracerProvider.getDelegate()` against a fresh proxy's `NoopTracerProvider` delegate — halo logs `[Observability] external OpenTelemetry SDK already registered — using it, skipping built-in exporters` and skips its own exporters. The `??=` env mapping above still applies first, so the external SDK picks up halo's settings unless it already set its own.
|
|
34
34
|
|
|
@@ -54,10 +54,10 @@ No `OTEL_EXPORTER_OTLP_ENDPOINT` after the mapping → `enabled=false`, `initObs
|
|
|
54
54
|
|
|
55
55
|
**Why the SDK is loaded by dynamic import.** The SDK module graph costs ~120ms, which every unconfigured `halo` start (`--version` included) would otherwise pay. `otel.ts` only `import()`s `otel-sdk.ts` after confirming an endpoint is configured, and `otel-sdk.ts` itself `import()`s the SDK / exporter packages inside `registerSdk` rather than at the top: the npm bundle (esbuild) inlines `otel-sdk.ts`, which would hoist top-level imports of those external packages into the bundle's static imports and load them on every start. So the unconfigured path (server, `halo cli`, `halo tui`) never loads the SDK.
|
|
56
56
|
|
|
57
|
-
**Flush on exit (cli / tui).** A short-lived process must push its batched spans / metrics / logs before it goes. `shutdownObservability()` force-flushes and shuts down every provider, is idempotent (a signal can land while the end-of-run flush is in flight), and is **capped at 3 s** (`SHUTDOWN_CAP_MS`) so a dead collector delays an exit but never hangs it; it resolves `false` when the cap fired, and is instant when observability is off. The 3 s sits inside the
|
|
57
|
+
**Flush on exit (cli / tui).** A short-lived process must push its batched spans / metrics / logs before it goes. `shutdownObservability()` force-flushes and shuts down every provider, is idempotent (a signal can land while the end-of-run flush is in flight), and is **capped at 3 s** (`SHUTDOWN_CAP_MS`) so a dead collector delays an exit but never hangs it; it resolves `false` when the cap fired, and is instant when observability is off. The 3 s sits inside the evolution wrapper's 10 s SIGTERM→SIGKILL grace. Three exit paths:
|
|
58
58
|
- **natural end** (one-shot run done, TUI quit, a thrown error): a `beforeExit` hook registered by `initRuntime` flushes; when the cap fired the exporters' retry timers still hold the event loop, so the hook then calls `process.exit()` itself (stdout has drained by then). `beforeExit` never fires on `process.exit()`.
|
|
59
59
|
- **SIGINT / SIGTERM**: the handlers stop the harness first (ending the turn, so its spans are closed and in the batch), then `exitAfterFlush(130 | 143)` (`harness.ts`) flushes and exits. A second SIGINT still hard-exits at once.
|
|
60
|
-
- Any other `process.exit()` skips the flush.
|
|
60
|
+
- Any other `process.exit()` skips the flush.
|
|
61
61
|
|
|
62
62
|
**Why the meter is lazy.** The `@opentelemetry/api` tracer and logger are proxies — they stay no-op until a real provider is registered, then transparently start forwarding, so `tracer` / `otelLogger` can be module-level consts. The metrics API has **no proxy**: a `Meter` fetched via `metrics.getMeter()` before `setGlobalMeterProvider()` runs is a permanent no-op forever after, even once a provider registers later. `getMeter()` in `otel.ts` is therefore a function, not a const, and `genai-spans.ts` calls it lazily on first metric emission (`instruments()`, cached after).
|
|
63
63
|
|
|
@@ -388,7 +388,7 @@ Schema declared by each package + values stored centrally is the same model VSCo
|
|
|
388
388
|
## working_dir behaviour
|
|
389
389
|
|
|
390
390
|
- `start_session` accepts a `working_dir` argument (absolute or workspace-relative)
|
|
391
|
-
- A root session takes one too: `halo cli` / `halo tui --dir <path
|
|
391
|
+
- A root session takes one too: `halo cli` / `halo tui --dir <path>`, or a cron job's folder (the runner passes it to `createSession` / `setWorkingDir` directly), same checks; on a resumed session it re-points the stored value via `SessionManager.setWorkingDir` (`--dir .` = back to the root), refused while the session is running
|
|
392
392
|
- `/session new` (server-side `execNew`) gives the new session the folder of the issuing session's root, re-validated (gone → root)
|
|
393
393
|
- Default: `workspaceRoot`
|
|
394
394
|
- Enforced inside workspace (path traversal is rejected)
|
|
@@ -30,7 +30,7 @@ Relay rides the same "deliver a session's wrap-up to some *other* session when i
|
|
|
30
30
|
- **Nested trees report once.** The quiet gate is the same one `tryReportToParent` uses, so a target that fans out three sub-agents delivers exactly one relay report, after the last child bubbles up — not one per child turn end.
|
|
31
31
|
- **Full access only.** `session-agent-builder` injects the set only when `nameSet.has('relay_send') && accessLevel === null`. A sandboxed (`workspace` / `readonly`) session can't be given a tool that opens other workspaces' sqlite and session trees; listing the name in such an agent's yaml is silently ignored.
|
|
32
32
|
- **Name-gated bundle.** Only `relay_send` is recognised in `tools:`; the other four (`relay_interrupt` / `relay_stop` / `relay_read` / `relay_list`) ride along. Prevents a half-configured agent that can send but never read back / stop. The admin picker mirrors this with one chip (`GET /agent-configs/tools` builds it from `buildRelayTools` against a dummy target and rewrites the description to name the full set).
|
|
33
|
-
- **Server only.** `setRelayRegistry` is called once in `index.ts`; the CLI / TUI never call it, so `getRelayRegistry()` is `null` there — every tool returns `{"code": 1, "error": RELAY_UNAVAILABLE}` (`relay.ts`: "relay is not available in the CLI / TUI — it only runs inside halo server", permanent for this runtime, do not retry; send from the admin UI / an IM or Web channel, or open the target with `halo tui -w <workspace path>`) and `deliverRelayReport` logs a warning and returns. No CLI code path knows relay exists. Note the builder does **not** check the registry before injecting, so a CLI / TUI session
|
|
33
|
+
- **Server only.** `setRelayRegistry` is called once in `index.ts`; the CLI / TUI never call it, so `getRelayRegistry()` is `null` there — every tool returns `{"code": 1, "error": RELAY_UNAVAILABLE}` (`relay.ts`: "relay is not available in the CLI / TUI — it only runs inside halo server", permanent for this runtime, do not retry; send from the admin UI / an IM or Web channel, or open the target with `halo tui -w <workspace path>`) and `deliverRelayReport` logs a warning and returns. No CLI code path knows relay exists. Note the builder does **not** check the registry before injecting, so a CLI / TUI session still *sees* the relay tools when its agent lists `relay_send` — the error is permanent there, not a transient outage; route cross-workspace dispatch through a server-side channel (admin / Web / IM / `halo acp`) instead.
|
|
34
34
|
- **Target-side transcript is normal.** The department workspace's session looks like any other root session in its Sessions tab (user message, agent reply); the only trace of relay is the `reply_to` row while a dispatch is pending. Nothing in the target's system prompt changes.
|
|
35
35
|
- **Same workspace is allowed — except self and ancestors.** `workspace` is only checked for existence + `.halo/`; passing the caller's own path makes `getOrCreate` return the caller's own `SessionManager`, and everything else is unchanged, with one exception: there `relay_send` / `relay_interrupt` / `relay_stop` refuse a `session_id` that is the caller itself or one of its ancestors (`isSelfOrAncestor` from `session-tools.ts`; the tools compare the resolved manager with their own host, so the same id in another workspace is unaffected). A send there stamped `reply_to` with a session inside the target's own tree — a self-report, or an ancestor whose report goes back to its own descendant — and a stop awaited the very turn making the call. In effect relay is a `query_session` without the own-tree scoping — it reaches any **root** session in any workspace, including new ones. Not a hole: the tool is full-access only, and a full-access agent could already `shell_exec` its way into any session file.
|
|
36
36
|
- **Reports come back from roots only.** `deliverRelayReport` fires for `parentId === null`; a sub-session's turn end goes to `tryReportToParent` instead. Dispatching to a `parent>child` id delivers the message but never a report — target roots when you want to hear back. Dispatching to yourself is refused (see above).
|
|
@@ -61,7 +61,7 @@ No quiet gate (the session is by definition not done) and **`reply_to` is kept**
|
|
|
61
61
|
|
|
62
62
|
## What it deliberately doesn't do
|
|
63
63
|
|
|
64
|
-
- **No cross-server reach.** Same `SessionManagerRegistry`, same process. Different server → [A2A](a2a.md), which reuses the same `reply_to` column (`{ a2a: taskId }`), quiet gate and interim doors; so does AgentCore's `POST /invocations` (`{ invocation: id }`, [agentcore.md](agentcore.md#post-invocations)). A2A and relay both write theirs via `takeReplyTo`, which supersedes a pending `{ invocation }` job first.
|
|
64
|
+
- **No cross-server reach.** Same `SessionManagerRegistry`, same process. Different server → [A2A](a2a.md), which reuses the same `reply_to` column (`{ a2a: taskId }`), quiet gate and interim doors; so does AgentCore's `POST /invocations` (`{ invocation: id }`, [agentcore.md](agentcore.md#post-invocations)) and an in-process cron run (`{ cron: runId }`, final only, [cron.md](cron.md#execution-runjob)). A2A and relay both write theirs via `takeReplyTo`, which supersedes a pending `{ invocation }` job or `{ cron }` run first.
|
|
65
65
|
- **No session discovery beyond a flat root list.** `relay_list` returns a workspace's root sessions (id / agent / title / status, newest 100) so the secretary can reuse an existing conversation; it doesn't walk trees or search transcripts — that's the admin's job.
|
|
66
66
|
- **No broadcast / fan-out helper.** The secretary calls `relay_send` once per department; the reports come back individually with the workspace + session id in the header, so it can tell them apart without extra bookkeeping.
|
|
67
67
|
- **No ACL beyond full access.** A full-access secretary can reach every workspace on the box — that's the deployment's trust boundary already (it can `shell_exec` into them anyway).
|