@turmind/halo 1.6.3 → 1.6.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +66 -67
- package/admin-out/404.html +1 -1
- package/admin-out/_next/static/chunks/87-6a2716c98a819c4e.js +1 -0
- package/admin-out/_next/static/chunks/989-066a874961bd519f.js +1 -0
- package/admin-out/_next/static/chunks/app/page-812bcbb46db628b6.js +1 -0
- package/admin-out/_next/static/css/e890aa0d13e94290.css +1 -0
- package/admin-out/index.html +1 -1
- package/admin-out/index.txt +5 -5
- package/bin/halo.js +4 -0
- package/bundled-docs/design/architecture.md +1 -1
- package/bundled-docs/design/canvas-extensions.md +40 -9
- package/bundled-docs/design/cron.md +12 -4
- package/bundled-docs/design/halo-city.md +14 -149
- package/bundled-docs/design/observability.md +7 -5
- package/bundled-docs/design/storage.md +14 -7
- package/bundled-docs/design/stt-local.md +106 -0
- package/bundled-docs/design/web.md +1 -1
- package/bundled-docs/design/wechat.md +1 -1
- package/bundled-docs/dev/add-model-provider.md +2 -1
- package/bundled-docs/dev/api.md +13 -9
- package/bundled-docs/dev/deploy.md +114 -26
- package/bundled-docs/dev/desktop-packaging.md +50 -10
- package/bundled-docs/dev/env.md +4 -4
- package/bundled-docs/dev/logging.md +13 -2
- package/bundled-docs/dev/tools.md +2 -2
- package/bundled-docs/guide/agents.md +2 -2
- package/bundled-docs/guide/cli.md +20 -8
- package/bundled-docs/guide/getting-started.md +13 -5
- package/bundled-docs/guide/workspace.md +2 -0
- package/bundled-docs/requirements/agent.md +1 -1
- package/bundled-docs/requirements/chat.md +8 -0
- package/bundled-docs/requirements/explorer.md +14 -2
- package/bundled-docs/requirements/settings.md +2 -2
- package/bundled-docs/requirements/workspace.md +4 -0
- package/dist/index.js +3645 -1725
- package/package.json +4 -2
- package/templates/models/gemini.yaml +93 -0
- package/templates/skills/cron/SKILL.md +2 -1
- package/templates/skills/cron/manage-cron.py +2 -1
- package/admin-out/_next/static/chunks/147-d4200c70445aabb6.js +0 -1
- package/admin-out/_next/static/chunks/989-ddba0eb72566da23.js +0 -1
- package/admin-out/_next/static/chunks/app/page-7ffcbddb49dc56c4.js +0 -1
- package/admin-out/_next/static/css/3b4dcd0ac6b8eae8.css +0 -1
- /package/admin-out/_next/static/{k5zuJrivSdbaPifBatwdr → 8v1hCXbUqJhyMuK7LVeet}/_buildManifest.js +0 -0
- /package/admin-out/_next/static/{k5zuJrivSdbaPifBatwdr → 8v1hCXbUqJhyMuK7LVeet}/_ssgManifest.js +0 -0
package/bundled-docs/dev/api.md
CHANGED
|
@@ -36,6 +36,7 @@ File: `packages/server/src/routes/files.ts`
|
|
|
36
36
|
| GET | `/api/files?path=&projectId=` | Read file content (max 10 MB) |
|
|
37
37
|
| GET | `/api/files/stat?path=&projectId=` | Lightweight mtime + size |
|
|
38
38
|
| GET | `/api/files/download?path=&projectId=&inline=` | Download or inline-preview — streams the file (no full read into memory), supports `Range` (206 Partial Content) so `<video>`/`<audio>` can seek + partial-load; aborts the read if the client disconnects. Missing file → 404 `{error:'File not found'}` |
|
|
39
|
+
| GET | `/api/files/download-folder?path=&projectId=[&include=excluded][&estimate=1]` | Folder → one zip, built by yazl while it streams (zip64, UTF-8 names; already-compressed extensions stored, the rest deflated; one file open at a time, paced by backpressure; never staged on disk or in memory). `path` must be a directory inside the project, `''` = workspace root; it may itself be a folder symlink inside the workspace (resolved, then checked like any target); no `/tmp` exception. `Content-Disposition: attachment` with RFC 5987 `filename*` = `<folder name>.zip` (root → workspace dir name); no `Content-Length`, no Range. **Excluded by default**: any *folder* whose name is in `IGNORED_SEGMENTS` (`ws/file-watcher.ts`); `include=excluded` brings them back. **Always excluded**: `isHiddenHostPath` / `isHiddenWorkspacePath` (also against a nested `.halo/`), plus common credential stores (~/.kube, cloud CLI configs, shell histories — a zip-only list in `folder-zip.ts`, matched relative to the home dir), `.halo/settings.yaml`, `.env` / `.env.*`. Names a zip can't hold — any containing `\`, or a top-level `X:` prefix — are skipped as `unreadable`. Directory symlinks never followed; file symlinks only when the realpath stays inside the workspace and passes those checks. `estimate=1` → `{files, bytes, excluded: {<reason>: n}, ignoredNames: {<folder>: n}}` with reasons `ignored` / `blocked` / `symlink` / `unreadable`, same rules, no content read. More than 200,000 files or a walk over 10 s → **413** `{error}` (both modes, before any header). A blocked target → 403. Anything left out → `_halo-excluded.txt` at the zip root (grouped by reason, a pruned dir listed once, 1000 paths per reason + a count). Mid-stream (status already sent): a file that fails to open is skipped, a read error partway through truncates that entry — both are listed there. A client disconnect stops the walk and destroys the zip stream |
|
|
39
40
|
| PUT | `/api/files` | Save file (body: `{path, content, projectId}`) |
|
|
40
41
|
| PUT | `/api/files/raw?path=&projectId=[&expectMtime=][&create=1][&append=1][&root=]` | Replace an **existing** file's bytes with the raw request body (binary-safe; `PUT /api/files` is utf-8 text only). Used by canvas preview extensions' save path. 404 if the file doesn't exist (save never creates) unless `create=1`; 400 for a directory. `expectMtime` is the `modifiedAt` the client loaded from `/api/files/stat` (float ms — compared after `Math.round`): when the on-disk mtime differs → **409** `{error:'conflict', mtime, size}` so the client can offer overwrite (retry without / with the new `expectMtime`). `create=1` creates a missing file (parent dirs `mkdir -p`); `append=1` appends instead of overwriting (`expectMtime` ignored; with `create=1` a missing file is created first) — the bundle extensions' `fs` write / append. `root=<dir>` (workspace-relative; the bundle host always sends its bundle path): `<dir>` must be an existing directory and `path` strictly inside it, else **404** `{error:'Bundle root not found'}` with nothing created. Success → `{ok, path, mtime, size}` |
|
|
41
42
|
| POST | `/api/files/new` | Create empty file |
|
|
@@ -46,7 +47,7 @@ File: `packages/server/src/routes/files.ts`
|
|
|
46
47
|
|
|
47
48
|
**Inline Content-Type** for images comes from core's shared extension→MIME table (`imageMimeFromExt`, `packages/core/src/media/mime.ts`) on both `/api/files/download?inline=1` and `/api/web/file`; each route keeps only its own non-image entries (pdf / html / video / audio) and falls back to `application/octet-stream`. That table is why `.bmp` now serves as `image/bmp` on both — the two routes used to carry their own copies and `/api/web/file` returned `application/octet-stream` for it. `.svg` / `.ico` / `.avif` get a real image Content-Type when served, but are deliberately not "photos" (they're outside `IMAGE_EXTS`) so they never reach a vision model or a channel's sendPhoto.
|
|
48
49
|
|
|
49
|
-
All file operations validate the path stays inside the project root (prevents path traversal). The check (`validatePath` in `routes/workspace-path.ts`, shared with `data-preview.ts` and `git.ts`) realpaths both sides — longest existing ancestor plus the re-appended tail — so a symlink inside the workspace pointing outside it is rejected too, not just lexical `../`. One exception: `GET /api/files/download` also accepts absolute paths under `/tmp/` so that agent-produced working files (e.g. Playwright screenshots) can be inline-previewed from chat media chips.
|
|
50
|
+
All file operations validate the path stays inside the project root (prevents path traversal). The check (`validatePath` in `routes/workspace-path.ts`, shared with `data-preview.ts` and `git.ts`) realpaths both sides — longest existing ancestor plus the re-appended tail — so a symlink inside the workspace pointing outside it is rejected too, not just lexical `../`. One exception: `GET /api/files/download` (not `download-folder`) also accepts absolute paths under `/tmp/` so that agent-produced working files (e.g. Playwright screenshots) can be inline-previewed from chat media chips.
|
|
50
51
|
|
|
51
52
|
### Filesystem browse (not project-scoped — for the workspace picker)
|
|
52
53
|
|
|
@@ -95,6 +96,7 @@ Installed extensions are directories under `~/.halo/global/extensions/<id>/` wit
|
|
|
95
96
|
| DELETE | `/api/extensions/:id` | cookie | Remove the directory. 400 for an invalid id shape, 404 if not installed. Returns `{ok, id}` |
|
|
96
97
|
| GET | `/api/extensions/:id/:version/:token/<asset>` | **path token, no cookie** | Static file from the extension directory. 401 unless `verifyScopedToken(token,'ext')`; 404 if `version` ≠ installed version or the asset escapes the directory. Response headers: `Access-Control-Allow-Origin: *`, `Referrer-Policy: no-referrer`, `Cache-Control: public, max-age=31536000, immutable` (the version segment is the cache key) |
|
|
97
98
|
| WS | `/api/transcribe/stream?ext=<id>&lang=<auto\|xx-XX>` | cookie | Streaming transcription proxy to Amazon Transcribe (`routes/transcribe-ws.ts`; not mounted in AgentCore mode). Handshake 401 without the login cookie, 403 unless `ext` is installed and declares capability `transcribe`. Client → binary PCM s16le mono 16 kHz, text `{"type":"end"}` (flush, server closes 1000). Server → `{"type":"ready"}` (on connect; stream audio right after it), `{"type":"partial"\|"final",start,end,text,lang}` (seconds from the stream's first audio byte), `{"type":"error",code,message}` then close; `code` ∈ `credentials` \| `denied` \| `limit` \| `bad-request` \| `io`. Region / languages / credentials from the extension's `ext-<id>` settings (default chain when no keys) — see [design](../design/canvas-extensions.md#streaming-transcription-proxy) |
|
|
99
|
+
| WS | `/api/stt/stream?ext=<id>` | cookie | On-device speech-to-text — SenseVoice + Silero VAD via sherpa-onnx, no cloud (`routes/stt-ws.ts`; not mounted in AgentCore mode). Handshake 401 without the login cookie, 403 unless `ext` is installed and declares capability `stt-local`. Server → `{"type":"model",state,received?,total?,message?}` first (`state` ∈ `missing` \| `downloading` \| `ready` \| `error`; a missing model is downloaded from here, progress pushed ≤ 2/s), `{"type":"ready"}` once model + engine are loaded (binary sent before it is dropped), `{"type":"final",start,end,text,lang}` per VAD segment (seconds from the connection's first accepted sample; `lang` ∈ `zh` \| `en` \| `yue` \| `ja` \| `ko`), `{"type":"error",code,message}` then close 1011; `code` ∈ `model` \| `engine` \| `busy` \| `bad-request` \| `io`. Client → binary PCM s16le mono 16 kHz, text `{"type":"end"}` (remaining finals, then close 1000). No `lang` parameter — see [design/stt-local.md](../design/stt-local.md#ws-apisttstreamextid) |
|
|
98
100
|
| GET | `/api/jupyter/status` | cookie | Notebook kernels (`routes/jupyter.ts`; not mounted in AgentCore mode). `{ enabled, available, version?, error?, hint? }` — never starts the server; disabled → `enabled:false, available:false`; Jupyter missing → `available:false` + `error` + `hint: 'pip install jupyter_server ipykernel'` |
|
|
99
101
|
| GET | `/api/jupyter/kernelspecs` | cookie | `{ default, specs: [{ name, displayName, language }] }`. Starts the local jupyter server on demand; 503 `{error, hint?}` when disabled / unavailable / it fails to start |
|
|
100
102
|
| POST | `/api/jupyter/kernels` | cookie | Body `{ projectId, path, name }` (`path` = workspace-relative notebook path, as in `init.file.path`). 400 for a missing field, unknown project, path outside the workspace or a missing directory. Starts kernel `name` with cwd = the notebook's directory → `{ id, name }` |
|
|
@@ -102,7 +104,7 @@ Installed extensions are directories under `~/.halo/global/extensions/<id>/` wit
|
|
|
102
104
|
| DELETE | `/api/jupyter/kernels/:id` | cookie | Shut the kernel down → `{ ok: true }`; 404 for an unknown id |
|
|
103
105
|
| WS | `/api/jupyter/channels?kernel=<id>[&session_id=<s>]` | cookie | Kernel channels: frames piped as-is both ways to the upstream `/api/kernels/<id>/channels` — legacy JSON text protocol (`channel` field), no subprotocol. `session_id` (`^[\w-]{1,128}$`, 1.6.3+) is forwarded, so a reconnect with the same one gets what jupyter buffered while the session had no socket. Handshake 401 without the cookie, 404 for an unknown kernel, 400 for a malformed `session_id`. The last socket closing starts a 60 s grace, then the kernel is shut down — see [design](../design/canvas-extensions.md#notebook-kernels-jupyter-proxy) |
|
|
104
106
|
|
|
105
|
-
Extension capabilities the host understands (`ExtensionCapability`): `save`, `media`, `transcribe`, `fs-read`; an unknown value makes the manifest an error. `fs-read` adds no extension-specific route: the host serves the extension's scoped `fs` frames and `pick` file picker from the [Filesystem browse](#filesystem-browse-not-project-scoped--for-the-workspace-picker) routes above (`scope: 'system'`) and the ordinary `/api/files/*` routes (`scope: 'workspace'`) — see [design/canvas-extensions.md](../design/canvas-extensions.md#scoped-file-access-and-the-picker-fs-read).
|
|
107
|
+
Extension capabilities the host understands (`ExtensionCapability`): `save`, `media`, `transcribe`, `fs-read`, `stt-local` (may open `/api/stt/stream`; an installed one also starts the model download — [design/stt-local.md](../design/stt-local.md)); an unknown value makes the manifest an error. An optional manifest `activity` object (`icon`, `title[_zh]`, `description[_zh]`, `order`) arrives as `ExtensionInfo.activity` and gives the extension an admin activity-bar page; an optional `chatbar` object (`icon`, `title[_zh]`, `height`, `order`) arrives as `ExtensionInfo.chatbar` and gives it a chat-input toolbar button whose panel may `insert-text` into the draft and report `busy` to get a `close-request` before a user close unmounts it (no route of its own); with either, `extensions` may be `[]` (the icon is a normal asset served by the route above, `.svg` as `image/svg+xml`) — rules in [design/canvas-extensions.md](../design/canvas-extensions.md#manifest-halo-extensionjson). `fs-read` adds no extension-specific route: the host serves the extension's scoped `fs` frames and `pick` file picker from the [Filesystem browse](#filesystem-browse-not-project-scoped--for-the-workspace-picker) routes above (`scope: 'system'`) and the ordinary `/api/files/*` routes (`scope: 'workspace'`) — see [design/canvas-extensions.md](../design/canvas-extensions.md#scoped-file-access-and-the-picker-fs-read).
|
|
106
108
|
|
|
107
109
|
**Why the asset route isn't cookie-authed.** The admin first hosted extensions in `<iframe sandbox="allow-scripts">` without `allow-same-origin` — an opaque origin. The document navigation still carries the cookie, but every subresource it loads (classic/module scripts, img, css, fetch, wasm, dynamic import, Worker) is a cross-site request with **no cookie**, and module/fetch/wasm need `Access-Control-Allow-Origin` too. A `?token=` query would be dropped when the document resolves `./viewer.js`-style relative URLs, so the token is a **path segment** that every relative URL inherits. The host now grants `allow-same-origin` (a cookie-auth proxy in front of halo — CloudFront + midway, oauth2-proxy — needs its *own* cookie on those requests, which only a same-origin iframe sends; see [design/canvas-extensions.md](../design/canvas-extensions.md#asset-serving-and-the-scoped-token)), so the admin cookie does travel again, but the route keeps the path token as its single auth path. `authMiddleware` lets exactly that path shape (`EXTENSION_ASSET_PATH = /^\/api\/extensions\/[^/]+\/[^/]+\/[^/]+\/./`) through without a cookie and the route verifies the token itself. The token is good for nothing else: `validateToken` (admin cookie / WS upgrade) refuses any JWT payload that carries a `scope`, so an asset token replayed as `halo_token` is a 401.
|
|
108
110
|
|
|
@@ -217,7 +219,7 @@ File: `packages/server/src/routes/cron.ts`. Admin cookie auth. REST CRUD over `c
|
|
|
217
219
|
| GET | `/api/cron/jobs?limit=&before=` | Cursor-paginated newest-first by `createdAt`. Default `limit` 20, max 100; `before` is a `createdAt` cursor. Returns `{jobs, hasMore, nextCursor}`. Each job has `targets` decoded, a live `nextRunAt` (computed from schedule/runAt, inside the active window), `sessionId` (`null` = runs in the job's own session), `effectiveSessionId` (the session the cli actually runs in: `sessionId`, else the job's own — same id as the job, `cron-<id>` for a custom id without that prefix; older jobs were pinned to their original `cron-cron-…` session by a cron.db migration), `activeFrom` / `activeUntil` (`null` = unbounded), `workingDir` (workspace-relative folder the run focuses on; `null` = the workspace root), `stateless` (boolean; `true` = a fresh session every run), and `sessionAgentId` (the agent of the effective session once it exists — the job's agent is locked to it; `null` for a stateless job, a session not created yet, or a session that can't be read) |
|
|
218
220
|
| POST | `/api/cron/jobs` | Create a job. Body: `{label?, workspacePath, agentId, userPrompt, schedule, runAt?, timezone?, timeoutSec?, sessionId?, workingDir?, stateless?, targets?, enabled?}`. Exactly one of `schedule` (5-field cron) or `runAt` (epoch ms, future) is required; mutually exclusive. `timeoutSec` must be an integer 60–21600 (else 400); unset = runner default 3600. `sessionId` picks an existing (or to-be-created) **root** session of `workspacePath` for the cli to run in — `[A-Za-z0-9_:-]{1,200}`, no `>` sub-session path (else 400); unset / `null` / `''` = the job's own session (id = the job id). A picked session that already exists keeps its own access level, and its agent must equal `agentId` (agent lock, below); see [design/cron.md](../design/cron.md#execution-runjob) for the shared-session guards. `activeFrom` / `activeUntil` (epoch ms or `null`, recurring only): the schedule fires only inside `[activeFrom, activeUntil)`; with `runAt` → 400; `activeUntil` must be in the future and after `activeFrom` (else 400); `activeFrom` may be in the past. `workingDir` (workspace-relative or absolute; unset / `null` / `''` = the root) must be an existing directory inside `workspacePath` (else 400 `workingDir: …`); stored workspace-relative. `stateless` (boolean, else 400) runs every fire in a fresh session; with `sessionId` → 400 `stateless (new session every run) and sessionId are mutually exclusive`. Agent lock: when the effective session already exists and runs another agent → 400 `session <id> already runs agent <x>; …`; a session db that can't be read → 400 `could not read session <id> from <ws>: …`. Returns `{ok, id}` |
|
|
219
221
|
| PUT | `/api/cron/jobs/:id` | Partial update over the same fields. Re-validates `schedule` / `runAt` and enforces trigger-mode exclusivity on the **merged** row: both in one body → 400 `schedule and runAt are mutually exclusive`; supplying one side is a mode switch that implicitly clears the other; a body that clears the active side without supplying a replacement → 400 `schedule or runAt required`. `runAt` must be a finite future epoch-ms number or `null`. `timeoutSec` follows the same partial-body contract: omitted = unchanged, `null` = clear back to the default, a number is range-validated (60–21600) same as create. `sessionId` likewise: omitted = unchanged, `null` / `''` = back to the job's own session (an older job pinned to `cron-cron-…` then starts a fresh one), a string is validated same as create. `activeFrom` / `activeUntil` likewise (omitted = unchanged, `null` = clear); the window rules are checked on the **merged** row, so switching to `runAt` while a window is stored needs both bounds cleared in the same body, and a newly supplied `activeUntil` must be in the future (a stored past one doesn't block editing other fields). `workingDir` likewise (omitted = unchanged, `null` / `''` = back to the root); a stored folder is re-checked when `workspacePath` changes. `stateless` and the `sessionId` exclusivity are checked on the **merged** row. The agent-lock / read-error 400s (see POST) apply only when the body changes the stored agent, session, `stateless` or `workspacePath` — so an unrelated edit such as `{enabled:false}` always succeeds, even for a job whose session already runs another agent. Flips schedule/unschedule on `enabled`. Returns `{ok}` |
|
|
220
|
-
| DELETE | `/api/cron/jobs/:id` | Unschedule, delete each stateless run's own session (except running / user-continued ones) and
|
|
222
|
+
| DELETE | `/api/cron/jobs/:id` | Unschedule, delete each stateless run's own session (except running / user-continued ones) and the job's `cron_runs` rows, then delete the job. A run still in progress keeps its row and session until it finalizes, then both go ([cron.md → Retention](../design/cron.md#retention)). Returns `{ok}` |
|
|
221
223
|
| POST | `/api/cron/jobs/:id/run-now` | Fire the job immediately (fire-and-forget). Returns `{ok}` |
|
|
222
224
|
| POST | `/api/cron/reload` | Re-read every job from db and rebuild schedules. Returns `{ok}` |
|
|
223
225
|
|
|
@@ -381,20 +383,22 @@ The push is a doorbell: only the task id is read from the body. The receiver the
|
|
|
381
383
|
|
|
382
384
|
## Show (world snapshot)
|
|
383
385
|
|
|
384
|
-
File: `packages/server/src/routes/halo-city.ts`.
|
|
385
|
-
the Web channel — same shared `resolveTokenAuth` core, same JSON error shape,
|
|
386
|
-
same brute-force lockout bucket). Read-only, cross-workspace
|
|
386
|
+
File: `packages/server/src/routes/halo-city.ts`. Read-only, cross-workspace
|
|
387
387
|
snapshot powering the `halo-city` pixel visualizer — one call returns the whole
|
|
388
388
|
runtime so the frontend can render rooms (workspaces) + characters (sessions).
|
|
389
389
|
|
|
390
|
+
**Auth** — either credential; both paths stay in `PUBLIC_PATHS` and the router authenticates itself:
|
|
391
|
+
- **Token** (`x-token` header or `?token=`, same as the Web channel — the shared `resolveTokenAuth` core, same JSON error shape, same brute-force lockout bucket). A request that carries a token is judged by the token alone, even when an admin cookie is also present (bad token + good cookie → 401 `invalid token`, counted as a strike).
|
|
392
|
+
- **Admin login cookie** (`halo_token`, the same check as the WS upgrade), used only when no token is sent: treated as `full` — every workspace, any session, `accessLevel: 'full'` in `/show/state`. This is how the halo-city extension page inside the admin calls these routes without a token. An invalid / expired cookie is just "no credential" (401 `token required`) and never counts as a strike.
|
|
393
|
+
|
|
390
394
|
| Method | Path | Purpose |
|
|
391
395
|
|---|---|---|
|
|
392
|
-
| GET | `/api/show/state` | `full` or `observer` token → every known workspace; otherwise the account's own. Returns `{ serverTime, uptime, accessLevel, skills[], workspaces[] }` |
|
|
393
|
-
| GET | `/api/show/session?ws=&id=` | Inspector-panel detail for a single session. Trimmed message log (last 40, content/tool I/O capped) plus `totalMessages` / `contextTokens` / `outputTokens` / `maxContextTokens` / `isRunning`.
|
|
396
|
+
| GET | `/api/show/state` | Admin cookie, `full` or `observer` token → every known workspace; otherwise the account's own. Returns `{ serverTime, uptime, accessLevel, skills[], workspaces[] }` |
|
|
397
|
+
| GET | `/api/show/session?ws=&id=` | Inspector-panel detail for a single session. Trimmed message log (last 40, content/tool I/O capped) plus `totalMessages` / `contextTokens` / `outputTokens` / `maxContextTokens` / `isRunning`. Tokens other than `full`/`observer` may only read their own workspace, and within it only sessions their own account minted (`web_<accountId>_*`); anything else is 403. The admin cookie reads any workspace. |
|
|
394
398
|
|
|
395
399
|
`observer` is more than an aggregate-counts role: past `/show/state`, it can also call `/show/session` to read **any workspace's** session transcript (last 40 messages, content truncated to 600 chars, tool input to 200 chars). Mint it knowing it grants cross-workspace transcript read access, not just dashboard counters.
|
|
396
400
|
|
|
397
|
-
Each `workspace` = `{ path, key, label, counts{running,idle,stopped}, totalSessions, skills[], sessions[] }`; each `session` = `{ id, parentId, depth, agentName, description, status, lastTool, activeSkill, contextTokens, outputTokens, messageCount, updatedAt }`. `messageCount` here is the session file's raw UI-log length (an activity signal for the citizen animation) — deliberately **not** the admin list's lifetime `exchangeCount`; it shrinks when a compact archives history. `lastTool` / `activeSkill` come from the live in-memory UI log (empty when the session isn't loaded). Sessions per room are capped (`totalSessions` reports the true total). Frontend: `halo-city
|
|
401
|
+
Each `workspace` = `{ path, key, label, counts{running,idle,stopped}, totalSessions, skills[], sessions[] }`; each `session` = `{ id, parentId, depth, agentName, description, status, lastTool, activeSkill, contextTokens, outputTokens, messageCount, updatedAt }`. `messageCount` here is the session file's raw UI-log length (an activity signal for the citizen animation) — deliberately **not** the admin list's lifetime `exchangeCount`; it shrinks when a compact archives history. `lastTool` / `activeSkill` come from the live in-memory UI log (empty when the session isn't loaded). Sessions per room are capped (`totalSessions` reports the true total). Frontend: the halo-hub extension [`halo-city`](https://github.com/turmind/halo-hub/tree/main/extensions/halo-city).
|
|
398
402
|
|
|
399
403
|
**Read-only by construction — degraded snapshots.** Both endpoints resolve a workspace's runtime via `registry.peek()` — in-memory lookup only, **never** `getOrCreate`: constructing a SessionManager has write side effects (boot orphan-reconciliation batch-stops live sub-session rows; `.halo/` scaffolding), so a pure visualization poll must not trigger it. When no live SessionManager exists in this process, the handler opens that workspace's `.halo/halo.db` **read-only** (cached connection; missing db/tables silently degrade to an empty room) and projects rows / the persisted session file into the same wire shape. Sessions not in memory therefore return a **persisted snapshot**: `status` / token counts / `messageCount` may lag behind the last persistence point, and the live-only signals (`lastTool` / `activeSkill`) are empty by definition. The degraded `/api/show/session` path reports `maxContextTokens: 0` (the real cap needs a live agent config; the city inspector hides the meter when the cap is unknown).
|
|
400
404
|
|
|
@@ -16,8 +16,8 @@ set -e
|
|
|
16
16
|
version='x.y.z' # exact requested version
|
|
17
17
|
curl -sfI "https://registry.npmjs.org/@turmind/halo/-/halo-$version.tgz" >/dev/null # tarball replicated? (a fresh publish can 404 for ~15 min)
|
|
18
18
|
sudo sh -c 'umask 022; npm install -g --prefix /usr --no-audit --no-fund "@turmind/halo@$1"' sh "$version"
|
|
19
|
-
installed=$(sudo -u ubuntu env HOME=/home/ubuntu /usr/bin/halo --version
|
|
20
|
-
test "$installed" = "halo $version" #
|
|
19
|
+
installed=$(sudo -u ubuntu env HOME=/home/ubuntu /usr/bin/halo --version)
|
|
20
|
+
test "$installed" = "halo $version" # mismatch or launch failure stops before restart
|
|
21
21
|
sudo systemctl restart halo.service
|
|
22
22
|
curl --fail --retry 10 --retry-connrefused --retry-delay 1 --max-time 3 http://127.0.0.1:9527/api/health # status ok, requested version
|
|
23
23
|
sudo journalctl -u halo.service --since '2 minutes ago' -n 40 --no-pager
|
|
@@ -49,9 +49,40 @@ Nginx is **not required**. Use it only for domain routing, SSL termination, or w
|
|
|
49
49
|
|
|
50
50
|
## Prerequisites
|
|
51
51
|
|
|
52
|
-
- Node.js ≥ 22
|
|
53
|
-
- pnpm 11 (pinned by the root `packageManager`; `corepack enable` picks it up)
|
|
54
|
-
- AWS credentials
|
|
52
|
+
- Node.js ≥ 22 — Ubuntu 24.04's apt `nodejs` is 18, too old; use NodeSource (`curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash - && sudo apt-get install -y nodejs`), `brew install node@22`, or nvm / fnm. On an older Node, `halo` exits 1 at launch with the version it found and these install hints instead of crashing. On a just-booted cloud VM the first `apt-get install` can fail on the dpkg lock (`Could not get lock …`, exit 100) while cloud-init / the SSM agent is still installing packages — wait a minute and rerun
|
|
53
|
+
- pnpm 11 — source builds only (pinned by the root `packageManager`; `corepack enable` picks it up)
|
|
54
|
+
- AWS credentials — optional, only for the Bedrock providers (`~/.aws/credentials`, env vars, or an instance role)
|
|
55
|
+
- bubblewrap (Linux) — needed for the `workspace` / `readonly` access levels: `sudo apt install bubblewrap`; on Ubuntu 23.10+ / 24.04 also the [AppArmor profile](#bubblewrap-on-ubuntu-2310-and-2404-apparmor)
|
|
56
|
+
|
|
57
|
+
### bubblewrap on Ubuntu 23.10 and 24.04 (AppArmor)
|
|
58
|
+
|
|
59
|
+
Ubuntu 23.10+ ships `kernel.apparmor_restrict_unprivileged_userns = 1`, and the `bubblewrap` package brings no AppArmor profile, so `apt install bubblewrap` alone isn't enough. Check with:
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
bwrap --ro-bind / / --die-with-parent -- /bin/true
|
|
63
|
+
# bwrap: setting up uid map: Permission denied ← blocked by AppArmor
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Halo's boot probe sees the same failure. The server warns once at startup, `[Server] OS sandbox unavailable (bwrap cannot create namespaces: bwrap: setting up uid map: Permission denied) — workspace/readonly sessions get no shell_exec. Fix: …` with a link to this section, and then runs without an OS sandbox. `halo cli` / `tui` print the same line, tagged `[cli]`, but only for `--access workspace|readonly`; without bubblewrap installed, the fix it names is `sudo apt install bubblewrap`. Fix it with a profile that lets only `bwrap` create user namespaces:
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
sudo apt-get install -y bubblewrap
|
|
70
|
+
sudo tee /etc/apparmor.d/bwrap >/dev/null <<'EOF'
|
|
71
|
+
abi <abi/4.0>,
|
|
72
|
+
include <tunables/global>
|
|
73
|
+
profile bwrap /usr/bin/bwrap flags=(unconfined) {
|
|
74
|
+
userns,
|
|
75
|
+
include if exists <local/bwrap>
|
|
76
|
+
}
|
|
77
|
+
EOF
|
|
78
|
+
sudo apparmor_parser -r /etc/apparmor.d/bwrap
|
|
79
|
+
halo server restart && curl -s --retry 15 --retry-connrefused --retry-delay 1 localhost:9527/api/health # "sandbox":"bwrap"
|
|
80
|
+
# a unit-run server is restarted through systemctl (Option C: may need the printed `sudo systemctl restart halo`)
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
Alternative: `sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0` (persist it in `/etc/sysctl.d/`). That also works, but it lifts the restriction for **every** program, not just bwrap.
|
|
84
|
+
|
|
85
|
+
Check: `GET /api/health` reports `"sandbox":"bwrap"`; `"sandbox":null` means non-full sessions still have no OS sandbox (and no `shell_exec`).
|
|
55
86
|
|
|
56
87
|
## 1. Install dependencies and build
|
|
57
88
|
|
|
@@ -78,7 +109,7 @@ halo setup # interactive — picks up arrow-key UI on TTYs
|
|
|
78
109
|
|
|
79
110
|
Re-run any time to change the password, refresh model keys, or toggle optional skills. Built-in agent / skill files are force-overwritten on every run; user-created agents and skills are left alone.
|
|
80
111
|
|
|
81
|
-
For Docker / CI builds where stdin isn't a TTY, see
|
|
112
|
+
For Docker / CI builds where stdin isn't a TTY, see [Non-interactive (Docker / CI) details](#non-interactive-docker--ci-details) below.
|
|
82
113
|
|
|
83
114
|
## 4. Environment variables (optional)
|
|
84
115
|
|
|
@@ -93,7 +124,9 @@ When `HALO_PASSWORD` is set, the password chosen via `halo setup` is ignored at
|
|
|
93
124
|
|
|
94
125
|
## 5. Start the server
|
|
95
126
|
|
|
96
|
-
>
|
|
127
|
+
> **Single-instance lock**: on start the server writes `~/.halo/global/server.lock` (flock when the `flock` command exists, otherwise a pid probe) and removes it on exit. If the process recorded in the lock is still alive, a new server refuses to start and prints a `kill <pid>` hint. The reason: the WeChat long-poll loop is decoupled from the HTTP server, so several processes side by side would fan the same WeChat message out to several sessions. A stale lock (process gone) is detected and cleared automatically. To restart by hand, kill the old one first: `kill $(cat ~/.halo/global/server.lock)`.
|
|
128
|
+
|
|
129
|
+
Options A and B below run a **source checkout**; for the npm install see [systemd for the npm install](#option-c-systemd-for-the-npm-install).
|
|
97
130
|
|
|
98
131
|
### Option A: quick start
|
|
99
132
|
|
|
@@ -128,6 +161,39 @@ sudo systemctl daemon-reload
|
|
|
128
161
|
sudo systemctl enable --now halo
|
|
129
162
|
```
|
|
130
163
|
|
|
164
|
+
### Option C: systemd for the npm install
|
|
165
|
+
|
|
166
|
+
With `@turmind/halo` installed via npm (see [§8](#8-install-via-npm-recommended)) on Linux with systemd, the simple path is `halo server start -d`. It writes and enables the user unit `~/.config/systemd/user/halo.service` (`Restart=on-failure`, `RestartSec=5`), turns on linger so it starts at boot, and saves your shell's environment to `~/.halo/secrets/server.env` for it. If linger can't be enabled it still starts, and warns `sudo loginctl enable-linger <user>`.
|
|
167
|
+
|
|
168
|
+
Write the system unit below when you want a system-level unit instead. If `start -d` already made the user unit, `systemctl --user disable --now halo` first, or both start at boot. Once a unit in `/etc/systemd/system/` runs `halo server start` as you (`User=` you, same HOME), `start -d` starts it instead of creating a user unit. The system unit runs the `halo` binary itself — no `WorkingDirectory`, and no `HALO_PASSWORD` once `halo setup` has stored the password hash (run setup as the same user):
|
|
169
|
+
|
|
170
|
+
```bash
|
|
171
|
+
sudo tee /etc/systemd/system/halo.service <<'EOF'
|
|
172
|
+
[Unit]
|
|
173
|
+
Description=Halo Server
|
|
174
|
+
After=network.target
|
|
175
|
+
|
|
176
|
+
[Service]
|
|
177
|
+
Type=simple
|
|
178
|
+
User=YOUR_USER
|
|
179
|
+
ExecStart=/usr/bin/halo server start
|
|
180
|
+
Restart=on-failure
|
|
181
|
+
RestartSec=5
|
|
182
|
+
|
|
183
|
+
[Install]
|
|
184
|
+
WantedBy=multi-user.target
|
|
185
|
+
EOF
|
|
186
|
+
|
|
187
|
+
sudo systemctl daemon-reload
|
|
188
|
+
sudo systemctl enable --now halo
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
Use `which halo` for the `ExecStart` path (`/usr/bin/halo` with apt / NodeSource Node). Each start prints `[Server] Halo <version> listening on port 9527 on all network interfaces — local: http://localhost:9527` and a network-exposure line to stderr, whatever the log level. For a system unit, look in the journal (`journalctl -u halo`).
|
|
192
|
+
|
|
193
|
+
Under either unit kind, `halo server stop|restart|status` find the unit that owns the server (the lock pid's cgroup) and go through `systemctl` (`--user` for the user unit). A system unit may need sudo: when polkit refuses, the command prints the exact `sudo systemctl restart|stop halo` line to run. Running that directly is equally valid. `stop` only stops the unit; it starts again at boot until you disable it (`systemctl --user disable halo`, or `sudo systemctl disable halo`). Upgrade with `halo upgrade && halo server restart`.
|
|
194
|
+
|
|
195
|
+
**Known limits**: the user unit pins the Node binary that ran `start -d` (`process.execPath`), so after an nvm Node upgrade run `halo server restart` once to rewrite it.
|
|
196
|
+
|
|
131
197
|
### Crash semantics — why `Restart=on-failure` is load-bearing
|
|
132
198
|
|
|
133
199
|
The server **exits 1 on any uncaught exception** (`process.on('uncaughtException')`
|
|
@@ -137,18 +203,22 @@ half-taken locks and half-written files, and since the db is the source of truth
|
|
|
137
203
|
— every daemon (cron / evo ticker / channels) reconciles from it on boot — a
|
|
138
204
|
restart is a full recovery while a zombie process silently corrupts state.
|
|
139
205
|
|
|
140
|
-
So the supervisor is what turns a crash into a ~5s blip: the
|
|
141
|
-
`Restart=on-failure` (fires
|
|
142
|
-
|
|
206
|
+
So the supervisor is what turns a crash into a ~5s blip: the units above — and
|
|
207
|
+
the user unit `halo server start -d` writes — pair `Restart=on-failure` (fires
|
|
208
|
+
exactly on non-zero exit) with `RestartSec=5`. Docker deployments want an
|
|
209
|
+
equivalent `restart:` policy. Under a bare `nohup
|
|
143
210
|
node dist/index.js` / foreground `halo server start` there is still no
|
|
144
211
|
supervisor — a crash is a hard stop, with the stack in
|
|
145
212
|
`~/.halo/global/logs/server.log` for whoever restarts it.
|
|
146
213
|
`unhandledRejection` is only logged, not fatal.
|
|
147
214
|
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
(
|
|
151
|
-
|
|
215
|
+
**Off systemd, `halo server start -d` brings its own bounded supervisor** —
|
|
216
|
+
on macOS, Windows, Linux without systemd, a HOME that isn't your passwd home,
|
|
217
|
+
or a user manager the shell can't reach (on Linux `-d` prints a one-line note
|
|
218
|
+
saying which). The detached process `-d` spawns is not the server: it re-execs
|
|
219
|
+
`halo server start` (foreground) with `HALO_SUPERVISE=1`, and *that* process
|
|
220
|
+
supervises the real server as its child (`packages/cli/src/server-supervisor.ts`).
|
|
221
|
+
Policy:
|
|
152
222
|
|
|
153
223
|
- **Restarts only on failure** — non-zero exit, or death by any signal other
|
|
154
224
|
than `SIGTERM` / `SIGINT`. `SIGKILL` (OOM killer) and `SIGSEGV` count as
|
|
@@ -159,7 +229,9 @@ server as its child (`packages/cli/src/server-supervisor.ts`). Policy:
|
|
|
159
229
|
server start -d`.
|
|
160
230
|
- **2s pause between attempts**, so an instant-crash loop can't spin the CPU.
|
|
161
231
|
- Every decision lands in the daemon log as `[respawn] …` — the stdout/stderr
|
|
162
|
-
capture `~/.halo/logs/server.log`, which `halo server logs` tails.
|
|
232
|
+
capture `~/.halo/logs/server.log`, which `halo server logs` tails. Only this
|
|
233
|
+
supervisor writes them; under the user unit a crash restart is systemd's
|
|
234
|
+
`RestartSec=5` (`systemctl --user status halo` shows it).
|
|
163
235
|
|
|
164
236
|
`halo server stop` (and `--force`) is not mistaken for a crash: it stamps the
|
|
165
237
|
server pid into a marker file `~/.halo/global/server.stop` *before* signalling,
|
|
@@ -172,9 +244,11 @@ one's marker.
|
|
|
172
244
|
|
|
173
245
|
Pid ownership is unchanged: `~/.halo/global/server.lock` is still written and
|
|
174
246
|
flock'd by the **server** process itself, so `halo server stop|restart|status`
|
|
175
|
-
|
|
176
|
-
its own; killing it directly leaves the server
|
|
177
|
-
state as the pre-supervisor `-d`), and `halo
|
|
247
|
+
target the server's own pid (or, for a unit-run server, its unit). The
|
|
248
|
+
supervisor has no pidfile of its own; killing it directly leaves the server
|
|
249
|
+
running as an orphan (same end state as the pre-supervisor `-d`), and `halo
|
|
250
|
+
server stop` still stops it. On a systemd host, `halo server restart` moves a
|
|
251
|
+
supervised `-d` server under the user unit.
|
|
178
252
|
|
|
179
253
|
## 6. Verify
|
|
180
254
|
|
|
@@ -196,7 +270,7 @@ cd /path/to/halo
|
|
|
196
270
|
cd packages/admin && npx next build --no-lint && node scripts/copy-monaco.mjs && cd ../..
|
|
197
271
|
pnpm --filter @turmind/halo-server build
|
|
198
272
|
|
|
199
|
-
#
|
|
273
|
+
# use the pid in the lock file so the old process really exits (an orphan would keep long-polling)
|
|
200
274
|
kill $(cat ~/.halo/global/server.lock) 2>/dev/null
|
|
201
275
|
sleep 2
|
|
202
276
|
cd packages/server && HALO_PASSWORD=your_password nohup node dist/index.js > /tmp/server.log 2>&1 &
|
|
@@ -217,8 +291,8 @@ This installs the `halo` binary on `$PATH`. Subcommands available:
|
|
|
217
291
|
| `halo setup` | Interactive password / port / model keys / optional skills setup |
|
|
218
292
|
| `halo setup --non-interactive` (alias `-y`) | Skip every prompt — seed templates only, supply password via `HALO_PASSWORD` env. Use in Dockerfiles / CI. |
|
|
219
293
|
| `halo upgrade` | Bump the npm install in place. Compares the bundled version against `npm view @turmind/halo version`; no-op if already latest, otherwise runs `npm install -g @turmind/halo@latest` and prints a server-restart hint. On EACCES, suggests retrying with `sudo`. |
|
|
220
|
-
| `halo server start` | Launch HTTP/WS server (foreground). Add `-d` for daemon. |
|
|
221
|
-
| `halo server stop` / `restart` / `status` / `logs` | Server lifecycle |
|
|
294
|
+
| `halo server start` | Launch HTTP/WS server (foreground). Add `-d` for daemon — on Linux with systemd, a user unit that starts at boot ([Option C](#option-c-systemd-for-the-npm-install)). |
|
|
295
|
+
| `halo server stop` / `restart` / `status` / `logs` | Server lifecycle; a unit-run server goes through `systemctl` |
|
|
222
296
|
| `halo tui` | Interactive TUI client |
|
|
223
297
|
| `halo cli "<prompt>"` | One-shot prompt → reply, exit |
|
|
224
298
|
| `halo agents` / `halo sessions` | List agents / sessions |
|
|
@@ -228,7 +302,7 @@ This installs the `halo` binary on `$PATH`. Subcommands available:
|
|
|
228
302
|
### Upgrade flow
|
|
229
303
|
|
|
230
304
|
1. `halo upgrade` — bumps the on-disk npm package
|
|
231
|
-
2. `halo server restart` — server's startup check sees `~/.halo/global/.template-version` is behind the new bundled `TEMPLATE_VERSION`, runs `ensureHaloHome` automatically, then starts (`halo cli` / `halo tui` run the same check on start, so a CLI-only install refreshes without a server). Refreshes `docs/`, built-in agents, built-in skills, system prompts, and the model registry. User-owned files (USER.md, custom agents/skills, INSTRUCTIONS.md overrides) are left alone. See `init.ts` for the per-category overwrite policy.
|
|
305
|
+
2. `halo server restart` (under a user or system unit alike; `sudo systemctl restart halo` also works for a system unit) — server's startup check sees `~/.halo/global/.template-version` is behind the new bundled `TEMPLATE_VERSION`, runs `ensureHaloHome` automatically, then starts (`halo cli` / `halo tui` run the same check on start, so a CLI-only install refreshes without a server). Refreshes `docs/`, built-in agents, built-in skills, system prompts, and the model registry. User-owned files (USER.md, custom agents/skills, INSTRUCTIONS.md overrides) are left alone. See `init.ts` for the per-category overwrite policy.
|
|
232
306
|
|
|
233
307
|
### Public release command reference
|
|
234
308
|
|
|
@@ -244,7 +318,9 @@ Use only for a new public release; the retained details here are not extra prepa
|
|
|
244
318
|
|
|
245
319
|
**npm token gotcha**: `npm publish` on this package needs a granular access token created with **"Bypass 2FA"** checked — scope / permission alone yields `403 Two-factor authentication or granular access token with bypass 2fa enabled is required`. `npm whoami` and `npm token list` succeed with a non-bypass token, so neither is a valid pre-flight; check `GET https://registry.npmjs.org/-/npm/v1/tokens` (with the token as bearer) and look for `"bypass_2fa": true` on the token in use before starting a release.
|
|
246
320
|
|
|
247
|
-
The published package contains a single bundled JS entry (~1.4 MB), all built-in templates (agents / skills / prompts / models), bundled platform docs, and the admin Web UI static export. Total install footprint
|
|
321
|
+
The published package contains a single bundled JS entry (~1.4 MB), all built-in templates (agents / skills / prompts / models), bundled platform docs, and the admin Web UI static export. Total install footprint is about 280 MB on disk (`du`; mostly `node_modules`).
|
|
322
|
+
|
|
323
|
+
Two log files: `~/.halo/logs/server.log` is the stdout/stderr capture of `halo server start -d`, whether the user unit or the off-systemd supervisor runs it (what `halo server logs` tails; the supervisor's `[respawn]` lines appear only off systemd; a system unit logs to the journal instead); `~/.halo/global/logs/server.log` is the server's structured, rotated log (level `general.logging.level`, default `warn`).
|
|
248
324
|
|
|
249
325
|
### Non-interactive (Docker / CI) details
|
|
250
326
|
|
|
@@ -252,8 +328,20 @@ The published package contains a single bundled JS entry (~1.4 MB), all built-in
|
|
|
252
328
|
|
|
253
329
|
- Seeds / refreshes `~/.halo/global/` from the bundled templates (per the per-category overwrite policy in `init.ts`)
|
|
254
330
|
- Generates `server.jwt_secret` if missing
|
|
255
|
-
- Binds
|
|
331
|
+
- Binds every built-in agent (default / executor / deep-executor plus the internal ones) to the provider named by `HALO_DEFAULT_PROVIDER`, when set (an unknown id only warns)
|
|
332
|
+
- For that provider, writes each env-backed key into `~/.halo/secrets/settings.yaml` as a `<<NAME>>` placeholder when `NAME` is set in setup's env (e.g. `DEEPSEEK_API_KEY` → `deepseek.secrets.api_key: "<<DEEPSEEK_API_KEY>>"`) — never the key itself; an unset `NAME` only warns. Bedrock is skipped: the AWS credential chain reads `AWS_*` on its own
|
|
256
333
|
- **Does not** set a password — supply one via `HALO_PASSWORD` env
|
|
334
|
+
- **Does not** store any literal model key — `HALO_PASSWORD` is the admin password only
|
|
335
|
+
|
|
336
|
+
The placeholder is expanded on read (see [env.md](env.md)). Without `HALO_DEFAULT_PROVIDER`, or for another provider, write it yourself; e.g. for DeepSeek:
|
|
337
|
+
|
|
338
|
+
```yaml
|
|
339
|
+
deepseek:
|
|
340
|
+
secrets:
|
|
341
|
+
api_key: "<<DEEPSEEK_API_KEY>>"
|
|
342
|
+
```
|
|
343
|
+
|
|
344
|
+
The variable must be set wherever the file is read — the server and `halo cli` / `tui` alike; under systemd add `Environment=DEEPSEEK_API_KEY=…` or an `EnvironmentFile=` to the unit. If you create the file yourself instead of editing the one `halo setup` wrote, `chmod 600` it.
|
|
257
345
|
|
|
258
346
|
Minimal Dockerfile:
|
|
259
347
|
|
|
@@ -261,10 +349,10 @@ Minimal Dockerfile:
|
|
|
261
349
|
FROM node:22-slim
|
|
262
350
|
RUN apt-get update && apt-get install -y --no-install-recommends bubblewrap && rm -rf /var/lib/apt/lists/*
|
|
263
351
|
RUN npm install -g @turmind/halo
|
|
264
|
-
ENV HALO_PASSWORD=changeme
|
|
352
|
+
ENV HALO_PASSWORD=changeme HALO_DEFAULT_PROVIDER=deepseek
|
|
265
353
|
ENV HALO_PORT=9527
|
|
266
354
|
EXPOSE 9527
|
|
267
|
-
CMD halo setup -y && halo server start
|
|
355
|
+
CMD halo setup -y && halo server start # key from `docker run -e DEEPSEEK_API_KEY=…`; setup writes its placeholder
|
|
268
356
|
```
|
|
269
357
|
|
|
270
358
|
## 9. Observability (OpenTelemetry collector)
|
|
@@ -104,8 +104,8 @@ HALO_STAGE_FULL=1 CI=true pnpm dist:win
|
|
|
104
104
|
```
|
|
105
105
|
|
|
106
106
|
`HALO_STAGE_FULL=1` is **mandatory** here: a cross-stage must re-run the
|
|
107
|
-
per-target native fixups (`@parcel/watcher-win32-x64` swap, `
|
|
108
|
-
win prebuild), which the fast path skips. `wine` must be installed (electron-
|
|
107
|
+
per-target native fixups (`@parcel/watcher-win32-x64` swap, `sherpa-onnx-win-x64`
|
|
108
|
+
swap, `better-sqlite3` win prebuild), which the fast path skips. `wine` must be installed (electron-
|
|
109
109
|
builder runs the nsis maker through it).
|
|
110
110
|
|
|
111
111
|
Verified on the v0.1.9 build (Linux x86_64 host → win-x64):
|
|
@@ -114,7 +114,9 @@ Verified on the v0.1.9 build (Linux x86_64 host → win-x64):
|
|
|
114
114
|
<ver>.exe` + `.exe.blockmap` + `win-unpacked/`. The "discard the stray
|
|
115
115
|
arm64 dmg" gotcha below is mac-host-only.
|
|
116
116
|
- **Native arch check** — every staged `.node` must be PE32+ x86-64, no ELF:
|
|
117
|
-
`find dist/win-unpacked -path '*watcher-*' -name '*.node' | xargs file
|
|
117
|
+
`find dist/win-unpacked -path '*watcher-*' -name '*.node' | xargs file`,
|
|
118
|
+
`find dist/win-unpacked -path '*sherpa-onnx-*' \( -name '*.node' -o -name '*.dll' \) | xargs file`
|
|
119
|
+
(only `sherpa-onnx-win-x64` dirs, one per runtime, no `linux-x64`) and
|
|
118
120
|
`file dist/win-unpacked/resources/node.exe` (should read `PE32+ ... x86-64`).
|
|
119
121
|
- **Two benign warnings**, both expected: `file source doesn't exist
|
|
120
122
|
from=.../resources/node` (win bundles `node.exe`, not the Unix `node` — the
|
|
@@ -204,7 +206,9 @@ Still good practice: build server + admin **before** `pnpm dist:arm64`.
|
|
|
204
206
|
3. **Native fixup** — re-fetch `better-sqlite3` prebuilds via `prebuild-install
|
|
205
207
|
--target=$NODE_VERSION_BARE` **unconditionally** (pins the ABI to the bundled
|
|
206
208
|
node regardless of the build host's node — see the ABI gotcha below);
|
|
207
|
-
`@parcel/watcher`'s per-platform binary is swapped only when cross-staging
|
|
209
|
+
`@parcel/watcher`'s per-platform binary is swapped only when cross-staging;
|
|
210
|
+
`sherpa-onnx`'s per-platform package is swapped to the target on every full
|
|
211
|
+
stage (server-runtime and cli-runtime — see the sherpa gotcha below).
|
|
208
212
|
4. **Trim bloat** — drop non-target `node-pty` prebuilds, `better-sqlite3`'s
|
|
209
213
|
C source, and the server's `src/`/`tests/`.
|
|
210
214
|
5. **Copy `admin-out/`** — the Next.js static export.
|
|
@@ -212,9 +216,10 @@ Still good practice: build server + admin **before** `pnpm dist:arm64`.
|
|
|
212
216
|
nodejs.org, extract the `node` binary into `resources/`.
|
|
213
217
|
7. **codesign (ad-hoc)** — `codesign --force --sign -` on the node binary for
|
|
214
218
|
Gatekeeper. **We deliberately do NOT `strip`** — see gotcha below.
|
|
215
|
-
8. **Native-addon smoke test** — load the staged better-sqlite3
|
|
216
|
-
bundled node binary; abort the build if
|
|
217
|
-
runs on full and fast paths). See the
|
|
219
|
+
8. **Native-addon smoke test** — load the staged better-sqlite3 and
|
|
220
|
+
sherpa-onnx with the bundled node binary; abort the build if either can't
|
|
221
|
+
`dlopen` (runs on full and fast paths, skipped when cross-staging). See the
|
|
222
|
+
ABI gotcha below.
|
|
218
223
|
|
|
219
224
|
## Gotchas (learned the hard way)
|
|
220
225
|
|
|
@@ -456,8 +461,8 @@ Still good practice: build server + admin **before** `pnpm dist:arm64`.
|
|
|
456
461
|
`node_modules` differs from what's on disk, run with `HALO_STAGE_FULL=1`. The
|
|
457
462
|
fast path only re-syncs compiled outputs and **reuses the existing
|
|
458
463
|
`node_modules`**, so it never re-runs the per-target native fixups
|
|
459
|
-
(`@parcel/watcher`
|
|
460
|
-
forgetting: the win/x64 dmg still carries the host's arm64 binaries.
|
|
464
|
+
(`@parcel/watcher` / `sherpa-onnx` swaps, `better-sqlite3` prebuild-install).
|
|
465
|
+
Symptom of forgetting: the win/x64 dmg still carries the host's arm64 binaries.
|
|
461
466
|
|
|
462
467
|
- **`@parcel/watcher` binary doesn't cross-stage via `npm install`.** Its
|
|
463
468
|
per-platform binary package (`@parcel/watcher-win32-x64` etc.) declares
|
|
@@ -474,6 +479,29 @@ Still good practice: build server + admin **before** `pnpm dist:arm64`.
|
|
|
474
479
|
(`better-sqlite3` and `node-pty` don't hit this — they use `prebuild-install`
|
|
475
480
|
and bundled multi-platform prebuilds respectively.)
|
|
476
481
|
|
|
482
|
+
- **`sherpa-onnx-node` (local STT) has the same os/cpu trap — and its Windows
|
|
483
|
+
package is `win-x64`, not `win32-x64`.** The main package is pure JS; the
|
|
484
|
+
N-API addon + onnxruntime libs ship in `sherpa-onnx-<plat>-<arch>` optional
|
|
485
|
+
deps (`darwin-arm64`, `win-x64`, `linux-x64`, …), and npm / pnpm install only
|
|
486
|
+
the host's, so a Linux → win stage ends up with just `sherpa-onnx-linux-x64`.
|
|
487
|
+
`installSherpaBinary()` `npm pack`s the target's package at the exact
|
|
488
|
+
installed `sherpa-onnx-node` version, extracts it as a sibling of
|
|
489
|
+
`sherpa-onnx-node` (its loader requires `../sherpa-onnx-<plat>-<arch>/sherpa-onnx.node`)
|
|
490
|
+
in both server-runtime and cli-runtime, and deletes every other platform's
|
|
491
|
+
package. Runs on every full stage, cross or not. The server loads it
|
|
492
|
+
lazily (`await import`), so a broken binary never blocks boot — it only
|
|
493
|
+
surfaces as dead transcription; on a native stage `smokeTestNativeAddons()`
|
|
494
|
+
`require`s it with the bundled node to catch that early.
|
|
495
|
+
- **Windows: must be loaded by a process named `node.exe`.** The win
|
|
496
|
+
`sherpa-onnx.node` delay-loads `node.exe` for its N-API imports, so
|
|
497
|
+
`require`-ing it inside `Halo.exe` (Electron main / `ELECTRON_RUN_AS_NODE`)
|
|
498
|
+
fails. Fine today: the server and the `halo.cmd` CLI both run under the
|
|
499
|
+
bundled `resources\node.exe`. Keep it out of the Electron process.
|
|
500
|
+
- **macOS: minimum OS for the arm64 `.node` is 14.0 (`LC_BUILD_VERSION`
|
|
501
|
+
minos; the dylibs say 11.0), unverified on older macOS.** Whether 12/13 can
|
|
502
|
+
load it hasn't been tested on real hardware. Dylibs resolve through
|
|
503
|
+
`@loader_path` rpath, so no `DYLD_LIBRARY_PATH` is needed.
|
|
504
|
+
|
|
477
505
|
- **`ENOTEMPTY` / `directory not empty` mid-stage or in electron-builder.** A
|
|
478
506
|
prior stage's `resources/{cli,server}-runtime/node_modules` (or
|
|
479
507
|
`dist/win-unpacked`) can fail to delete cleanly on macOS (open handles, deep
|
|
@@ -522,7 +550,19 @@ Still good practice: build server + admin **before** `pnpm dist:arm64`.
|
|
|
522
550
|
close. Fixed by matching `Get-CimInstance Win32_Process` `.ExecutablePath`,
|
|
523
551
|
which is readable from either bitness. Rule: **any PowerShell the installer
|
|
524
552
|
runs is 32-bit** — never filter processes by `Get-Process .Path` /
|
|
525
|
-
`.MainModule` there.
|
|
553
|
+
`.MainModule` there. The match itself was a second trap: `$INSTDIR` used to
|
|
554
|
+
be spliced into a single-quoted PS literal with `-like`, so a `'` in the
|
|
555
|
+
install path ended the string (parse error, nothing killed) and `[` / `]`
|
|
556
|
+
read as wildcards (could match a sibling dir). The macro now puts `$INSTDIR`
|
|
557
|
+
in env var `HALO_INSTDIR` (`System::Call
|
|
558
|
+
'kernel32::SetEnvironmentVariable(…)'`, inherited by the `nsExec` child) and
|
|
559
|
+
matches `ExecutablePath` with a literal, case-insensitive
|
|
560
|
+
`.StartsWith($env:HALO_INSTDIR + '\')` — rule: never splice a path into the
|
|
561
|
+
PS command line. A live `jupyter server` (ipynb kernels) could hold the
|
|
562
|
+
install dir open too, since it inherited the server's cwd inside it; both
|
|
563
|
+
jupyter spawns now pass `cwd: os.homedir()` (see
|
|
564
|
+
[canvas-extensions.md](../design/canvas-extensions.md#notebook-kernels-jupyter-proxy)).
|
|
565
|
+
To tell the two causes apart on a user's box, list
|
|
526
566
|
what still runs from the install dir (64-bit PowerShell): `Get-CimInstance
|
|
527
567
|
Win32_Process | ? ExecutablePath -like "$env:LOCALAPPDATA\Programs\Halo\*"`
|
|
528
568
|
— non-empty = this cause (append `| % { Stop-Process -Id $_.ProcessId
|
package/bundled-docs/dev/env.md
CHANGED
|
@@ -52,8 +52,8 @@ nohup node dist/index.js >> /dev/null 2>&1 & # HALO_PASSWORD read from env/con
|
|
|
52
52
|
|
|
53
53
|
- Node.js ≥ 22 (better-sqlite3 native binding is bound to v22)
|
|
54
54
|
- pnpm 11 (pinned by the root `packageManager`; `corepack enable` picks it up)
|
|
55
|
-
- AWS credentials
|
|
56
|
-
- bubblewrap (`sudo apt install bubblewrap`) — OS-level sandbox for non-full access levels on Linux (macOS uses the built-in Seatbelt). Without one, only app-level path validation is active and `shell_exec` is blocked for non-full sessions
|
|
55
|
+
- AWS credentials (`~/.aws/credentials`, env vars or an instance role) — only for the Bedrock providers; any other provider just needs its API key
|
|
56
|
+
- bubblewrap (`sudo apt install bubblewrap`) — OS-level sandbox for non-full access levels on Linux (macOS uses the built-in Seatbelt). Without one, only app-level path validation is active and `shell_exec` is blocked for non-full sessions. On Ubuntu 23.10+ / 24.04 AppArmor blocks it (`bwrap: setting up uid map: Permission denied`) until you add a profile — see [deploy.md](deploy.md#bubblewrap-on-ubuntu-2310-and-2404-apparmor)
|
|
57
57
|
|
|
58
58
|
### nvm PATH
|
|
59
59
|
|
|
@@ -99,7 +99,7 @@ Source: `packages/server/src/config.ts`
|
|
|
99
99
|
|---|---|---|---|
|
|
100
100
|
| `HALO_PORT` | `9527` | `config.yaml server.port` | Hono listen port |
|
|
101
101
|
| `HALO_PASSWORD` | (none) | `config.yaml server.password` (scrypt hash) | Plaintext login password. When set, takes precedence over the stored hash and bypasses scrypt entirely — intended for Docker / systemd / CI. It also satisfies the **startup gate**: the server refuses to boot with no password configured, and env plaintext counts as a first-class credential there, so an env-only deployment (`halo setup -y && HALO_PASSWORD=... halo server start`, no stored hash) boots fine. The hash is set by `halo setup` for interactive installs. |
|
|
102
|
-
| `HALO_CORS_ORIGINS` | empty (reflect any origin) | `config.yaml server.cors_origins` | CORS allowlist (comma-separated, exact origins). Empty = reflect any incoming Origin **without** `Allow-Credentials` (header-token callers like `x-token` work; cookie-carrying cross-origin requests don't). A non-empty list = only those origins get CORS, with `Allow-Credentials` (e.g. a frontend behind an SSO that must send its cookie). Read at startup — a change needs a restart. Prefer a same-origin reverse proxy over listing origins (see
|
|
102
|
+
| `HALO_CORS_ORIGINS` | empty (reflect any origin) | `config.yaml server.cors_origins` | CORS allowlist (comma-separated, exact origins). Empty = reflect any incoming Origin **without** `Allow-Credentials` (header-token callers like `x-token` work; cookie-carrying cross-origin requests don't). A non-empty list = only those origins get CORS, with `Allow-Credentials` (e.g. a frontend behind an SSO that must send its cookie). Read at startup — a change needs a restart. Prefer a same-origin reverse proxy over listing origins (see the halo-city extension's [README](https://github.com/turmind/halo-hub/blob/main/extensions/halo-city/README.md#和-halo-同源)). |
|
|
103
103
|
| `HALO_FRONTEND_DIR` | auto (bundled `admin-out`, else `packages/admin/out`) | — | Static frontend dir; resolved against the current working directory |
|
|
104
104
|
| `HALO_MAX_CONTEXT_TOKENS` | `272000` | — | Model max context |
|
|
105
105
|
| `HALO_SHELL_TIMEOUT` | `120000` | `config.yaml timeout.shell_exec` | Shell command timeout (ms) |
|
|
@@ -118,7 +118,7 @@ Source: `packages/server/src/config.ts`
|
|
|
118
118
|
| `HALO_A2A_ROOT` | the user's home | — | Base dir A2A exposes workspaces under: a workspace `<base>/A/B` with `.halo/agent-card.json` is served at `/a2a/A/B` (see [design/a2a.md](../design/a2a.md)); workspaces outside it can't be exposed |
|
|
119
119
|
| `HALO_A2A_PUBLIC_URL` | (none) | `settings.yaml general.a2a.public_url` | This server's A2A origin as peers see it (no trailing slash), used for the agent card's interface URL and as the default push-webhook base. In agentcore-a2a mode: the runtime's invoke URL (`…/runtimes/<escaped ARN>/invocations`) |
|
|
120
120
|
| `HALO_A2A_ACCESS` | `workspace` | — | agentcore-a2a mode only: access level of every A2A session — `workspace` \| `full` \| `readonly`; anything else is a boot error |
|
|
121
|
-
| `HALO_DEFAULT_PROVIDER` | (none) | — | Read by `halo setup --non-interactive`: binds
|
|
121
|
+
| `HALO_DEFAULT_PROVIDER` | (none) | — | Read by `halo setup --non-interactive`: binds every built-in agent to this provider id (and its default model) and, for each of its env-backed keys whose variable is set, writes the `<<NAME>>` placeholder into `settings.yaml` (Bedrock excepted). An unknown id only prints a warning |
|
|
122
122
|
| `HALO_CLI` | `halo` (`halo.cmd` on Windows) | — | `halo` cli executable that cron runs and the evolution wrapper spawn — a dev override |
|
|
123
123
|
| `HALO_VERSION` | `dev` | — | Version string reported by `GET /api/health` and the cli; stamped by the bundle / desktop launcher, so normally not set by hand |
|
|
124
124
|
| `HALO_PROMPT` | (none) | — | `plain` makes `halo setup` use numbered-menu prompts instead of the raw-mode TUI |
|
|
@@ -15,18 +15,29 @@ Any `?token=…` value in a message is replaced with `?token=<redacted>` before
|
|
|
15
15
|
|
|
16
16
|
## Log locations
|
|
17
17
|
|
|
18
|
-
One file for the whole server: `~/.halo/global/logs/server.log`. A single server process serves many workspaces, so there is no per-workspace log. Cron runs and evolution runs keep their own logs under `~/.halo/global/logs/{cron,evo}/`. `halo server start -d` additionally captures the process's stdout/stderr in `~/.halo/logs/server.log`
|
|
18
|
+
One file for the whole server: `~/.halo/global/logs/server.log`. A single server process serves many workspaces, so there is no per-workspace log. Cron runs and evolution runs keep their own logs under `~/.halo/global/logs/{cron,evo}/`. `halo server start -d` additionally captures the process's stdout/stderr in `~/.halo/logs/server.log` — via the systemd user unit's `StandardOutput=append:` on Linux with systemd, otherwise via the built-in supervisor, which adds its `[respawn]` lines — that is the file `halo server logs` tails. A system unit's stdout goes to the journal (`journalctl -u <unit>`).
|
|
19
19
|
|
|
20
20
|
## Log format
|
|
21
21
|
|
|
22
22
|
```
|
|
23
|
-
time=2026-04-19T10:30:00.000Z level=info [Server]
|
|
23
|
+
time=2026-04-19T10:30:00.000Z level=info [Server] Halo 1.6.3 listening on port 9527 on all network interfaces — local: http://localhost:9527
|
|
24
24
|
time=2026-04-19T10:30:01.234Z level=error [WS] Chat error: ThrottlingException
|
|
25
25
|
time=2026-04-19T10:30:02.567Z level=warn [SessionManager] Context overflow, auto-compacting...
|
|
26
26
|
```
|
|
27
27
|
|
|
28
28
|
Every line is logfmt-flavored: `time=<ISO timestamp> level=<debug|info|warn|error> <message>`; the message keeps its `[Module]` prefix.
|
|
29
29
|
|
|
30
|
+
### Always-shown lines
|
|
31
|
+
|
|
32
|
+
The startup banner bypasses the level gate: `logAlways()` (`logger.ts`) writes it straight to stderr whatever `general.logging.level` is, so a foreground `halo server start` at the default `warn` still shows that it came up. It is also written to the log file / OTel as an `info` line, but only when `info` passes the gate. The banner names the real bound address (`formatListenBanner()` in `listen-banner.ts`). The server passes no hostname, so it binds every interface, and the banner says so in two lines:
|
|
33
|
+
|
|
34
|
+
```
|
|
35
|
+
[Server] Halo 1.6.3 listening on port 9527 on all network interfaces — local: http://localhost:9527
|
|
36
|
+
[Server] Reachable from the network: allow the port only from trusted sources, or put a TLS reverse proxy in front
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
A specific bound address prints one line, `[Server] Halo <version> listening on http://<addr>:<port>` (IPv6 bracketed). AgentCore mode prints only the first line. Lines printed before `initLogger()` (setup / template-refresh notices, the source-run-only `monorepo root … not found` warning) aren't filtered either, and they never reach the log file.
|
|
40
|
+
|
|
30
41
|
## Rotation
|
|
31
42
|
|
|
32
43
|
When the log file exceeds the cap:
|
|
@@ -132,11 +132,11 @@ Each session carries `accessLevel: 'readonly' | 'workspace' | null` (persisted i
|
|
|
132
132
|
|
|
133
133
|
`view_image` is additionally gated on `capabilities.image` (see its section above). Models without vision support get the same lists minus `view_image` — so a non-vision model on `readonly` ends up with 4 tools, not 5.
|
|
134
134
|
|
|
135
|
-
OS sandbox backend per platform — `getSandboxBackend()` in `sandbox.ts` returns `'bwrap' | 'seatbelt' | null` after the boot probe (`initBwrapCheck()`), logged at startup as `[Server] OS sandbox: …` and exposed as `sandbox` on `GET /api/health`:
|
|
135
|
+
OS sandbox backend per platform — `getSandboxBackend()` in `sandbox.ts` returns `'bwrap' | 'seatbelt' | null` after the boot probe (`initBwrapCheck()`), logged at startup as `[Server] OS sandbox: …` (`info` level, so hidden at the default `warn`). When the probe found bwrap missing (ENOENT) or unable to create namespaces, the server instead warns once at boot: `[Server] OS sandbox unavailable (<reason>) — workspace/readonly sessions get no shell_exec. Fix: <hint>` (`sudo apt install bubblewrap`, or the AppArmor profile in [deploy.md](deploy.md#bubblewrap-on-ubuntu-2310-and-2404-apparmor)). `halo cli` / `tui` print the same line (`[cli]` tag) only for a `--access workspace|readonly` session; the probe itself never prints (`getSandboxUnavailableReason()` / `sandboxUnavailableWarning()` in `sandbox.ts`). Also exposed as `sandbox` on `GET /api/health`:
|
|
136
136
|
|
|
137
137
|
| Platform | `shell_exec` | File tools | Dependency |
|
|
138
138
|
|---|---|---|---|
|
|
139
|
-
| Linux | bwrap | bwrap | `bubblewrap` (`apt install bubblewrap`); the probe runs a real sandboxed no-op, so an install that can't create namespaces (Ubuntu 24.04 AppArmor userns restriction) counts as unavailable |
|
|
139
|
+
| Linux | bwrap | bwrap | `bubblewrap` (`apt install bubblewrap`); the probe runs a real sandboxed no-op, so an install that can't create namespaces (Ubuntu 23.10+ / 24.04 AppArmor userns restriction: `bwrap: setting up uid map: Permission denied`) counts as unavailable — fix in [deploy.md](deploy.md#bubblewrap-on-ubuntu-2310-and-2404-apparmor) |
|
|
140
140
|
| macOS | Seatbelt (`/usr/bin/sandbox-exec -p <profile>`) | in-process, behind `assertPathAllowed` | none — ships with macOS (Apple marks it deprecated, but it's present on every release). The probe runs `sandbox-exec -p '(version 1)(allow default)' /usr/bin/true` |
|
|
141
141
|
| Windows | none | none | — see below |
|
|
142
142
|
|
|
@@ -6,7 +6,7 @@ The agent is Halo's core abstraction: personality (AGENT.md) + config (agent.yam
|
|
|
6
6
|
|
|
7
7
|
| Kind | Description |
|
|
8
8
|
|---|---|
|
|
9
|
-
| Built-in | Server-shipped agents (`default`, `executor`, `deep-executor` + the internal
|
|
9
|
+
| Built-in | Server-shipped agents (`default`, `executor`, `deep-executor` + the internal ones such as `__evo_agent__` / `__score__` / `__apply_agent__`). Live under `~/.halo/global/agents/<id>/`. **Force-overwritten on every server startup** — local edits to these files will be lost on the next start. To customize: copy into the workspace scope (workspace replaces global) and edit there. |
|
|
10
10
|
| Global | Any other agent under `~/.halo/global/agents/<id>/`, e.g. one you created via the admin UI. Shared across projects. Never overwritten by the server. |
|
|
11
11
|
| Workspace | `<project>/.halo/agents/<id>/`, private to the current project; same-id workspace wins over global. Override is **whole-folder**: the workspace agent folder replaces the global one entirely, so copy *both* `agent.yaml` and `AGENT.md` when customizing — a folder with only `AGENT.md` has no model config and won't load. |
|
|
12
12
|
|
|
@@ -113,7 +113,7 @@ Top-right `Test` button:
|
|
|
113
113
|
|
|
114
114
|
Better than the old built-in test chat — it has full workspace tools + session persistence.
|
|
115
115
|
|
|
116
|
-
Internal agents (`internal: true`, e.g. self-evolution agents) have no Test button and never appear in the chat agent selector, the Cron agent picker or `/session new`'s default pick — they're delegated to by other agents, never driven directly. They remain editable in the management sidebar's collapsed **Internal** group.
|
|
116
|
+
Internal agents (`internal: true`, e.g. self-evolution agents) have no Test button and never appear in the chat agent selector, the Cron agent picker or `/session new`'s default pick — they're delegated to by other agents, never driven directly. They remain editable in the management sidebar's collapsed **Internal** group.
|
|
117
117
|
|
|
118
118
|
## Delete
|
|
119
119
|
|