@turmind/halo 0.1.9 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +1 -5
- package/admin-out/404.html +1 -1
- package/admin-out/_next/static/chunks/298-4fac7d44a7949eec.js +1 -0
- package/admin-out/_next/static/chunks/{503-7c6d9935df6e0cea.js → 307-d7a99f9a4a436ec3.js} +1 -1
- package/admin-out/_next/static/chunks/app/page-b12a2e8e48fe4ea0.js +1 -0
- package/admin-out/_next/static/css/{8fce4e1a54dac12a.css → 372e3dbd4a0d1c25.css} +1 -1
- package/admin-out/index.html +1 -1
- package/admin-out/index.txt +4 -4
- package/bundled-docs/dev/acp-adapter.md +5 -4
- package/bundled-docs/dev/desktop-packaging.md +94 -8
- package/bundled-docs/guide/getting-started.md +3 -2
- package/bundled-docs/requirements/explorer.md +11 -0
- package/dist/index.js +13373 -13246
- package/package.json +1 -1
- package/templates/agents/executor/agent.yaml +2 -2
- package/templates/models/aws-bedrock-claude-invoke.yaml +26 -0
- package/templates/skills/acp/SKILL.md +8 -1
- package/templates/skills/acp/templates/SKILL.md.tmpl +2 -1
- package/templates/skills/acp/templates/ask.py +2 -1
- package/templates/skills/acp/templates/config.yaml.tmpl +5 -0
- package/admin-out/_next/static/chunks/298-d6b224b569da4de0.js +0 -1
- package/admin-out/_next/static/chunks/app/page-cc8043335282d745.js +0 -1
- package/bundled-docs/plans/cron-tasks.md +0 -362
- package/bundled-docs/plans/design-suggestions.md +0 -203
- package/bundled-docs/plans/issue-session-deep-nesting.md +0 -208
- package/bundled-docs/plans/launch-plan.md +0 -111
- package/bundled-docs/plans/md-and-agent-refactor.md +0 -197
- package/bundled-docs/plans/remote-mac-worker.md +0 -196
- package/bundled-docs/plans/roadmap.md +0 -40
- package/bundled-docs/plans/self-evolution.md +0 -1379
- package/bundled-docs/plans/todo.md +0 -32
- package/bundled-docs/plans/weixin-channel.md +0 -255
- package/bundled-docs/test/docs-seeding.md +0 -101
- package/bundled-docs/test/session.md +0 -374
- /package/admin-out/_next/static/{IBP_mwBDf-cPwZSZZGpi8 → ep7BWneNhTuIPwJfFFh72}/_buildManifest.js +0 -0
- /package/admin-out/_next/static/{IBP_mwBDf-cPwZSZZGpi8 → ep7BWneNhTuIPwJfFFh72}/_ssgManifest.js +0 -0
|
@@ -1,362 +0,0 @@
|
|
|
1
|
-
# Cron Tasks
|
|
2
|
-
|
|
3
|
-
Global, cross-workspace scheduled agent runs. The user defines a job
|
|
4
|
-
(workspace + agent + prompt + schedule + channel targets); the server
|
|
5
|
-
schedules it via croner; on fire, a `halo cli` child runs the prompt;
|
|
6
|
-
the captured stdout is fanned out to the bound channels and the run is
|
|
7
|
-
recorded in an audit log.
|
|
8
|
-
|
|
9
|
-
Two trigger modes:
|
|
10
|
-
|
|
11
|
-
- **Recurring** — standard 5-field cron expression; fires on every match.
|
|
12
|
-
- **One-shot (at-mode)** — exact instant via `runAt` (epoch ms); fires
|
|
13
|
-
once at that time, then auto-disables (the row is kept for history /
|
|
14
|
-
rerun, but `enabled=0` so reload doesn't re-instantiate the schedule).
|
|
15
|
-
Same shape as `at(1)`. Picked when the user says "remind me at 3pm",
|
|
16
|
-
"send the report next Monday" — anything that doesn't repeat.
|
|
17
|
-
|
|
18
|
-
## Goal
|
|
19
|
-
|
|
20
|
-
Enable scheduled agent work — daily reports, periodic monitoring digests,
|
|
21
|
-
one-off reminders, scheduled data refreshes — without leaving a session
|
|
22
|
-
open or wiring external schedulers (cron, systemd timers, etc.).
|
|
23
|
-
|
|
24
|
-
Non-goals:
|
|
25
|
-
|
|
26
|
-
- **One-off ad-hoc invocations bound to "fresh state" per fire.** Each
|
|
27
|
-
job has a stable `cron-<jobId>` session id; first fire creates it,
|
|
28
|
-
subsequent fires resume it. So history accumulates over time and the
|
|
29
|
-
user can review the full conversation in the admin Sessions tab.
|
|
30
|
-
Threading via `-s cron-<jobId>` lets the agent remember what it told
|
|
31
|
-
the channel last week without each prompt needing to bootstrap from
|
|
32
|
-
`.halo/memory/`.
|
|
33
|
-
- **Conditional / event-driven triggers.** Only time-based scheduling.
|
|
34
|
-
- **Cross-job orchestration.** Each job is independent.
|
|
35
|
-
|
|
36
|
-
## Components
|
|
37
|
-
|
|
38
|
-
### Storage
|
|
39
|
-
|
|
40
|
-
`~/.halo/global/cron.db` (sqlite + drizzle, parallel to `evo.db`).
|
|
41
|
-
|
|
42
|
-
```sql
|
|
43
|
-
cron_jobs(
|
|
44
|
-
id TEXT PRIMARY KEY,
|
|
45
|
-
label TEXT, -- nullable
|
|
46
|
-
workspace_path TEXT NOT NULL,
|
|
47
|
-
agent_id TEXT NOT NULL,
|
|
48
|
-
user_prompt TEXT NOT NULL,
|
|
49
|
-
schedule TEXT NOT NULL, -- 5-field cron; '' for at-mode
|
|
50
|
-
run_at INTEGER, -- one-shot fire time (epoch ms);
|
|
51
|
-
-- mutually exclusive with `schedule`
|
|
52
|
-
timezone TEXT, -- IANA zone, null = host TZ
|
|
53
|
-
targets TEXT NOT NULL -- JSON array of
|
|
54
|
-
-- {channelType, accountId, chatId?}
|
|
55
|
-
DEFAULT '[]',
|
|
56
|
-
enabled INTEGER NOT NULL DEFAULT 1,
|
|
57
|
-
last_run_status TEXT, -- runner-managed cache
|
|
58
|
-
last_run_at INTEGER,
|
|
59
|
-
last_run_id TEXT,
|
|
60
|
-
created_at INTEGER NOT NULL,
|
|
61
|
-
updated_at INTEGER NOT NULL
|
|
62
|
-
)
|
|
63
|
-
|
|
64
|
-
cron_runs(
|
|
65
|
-
id TEXT PRIMARY KEY,
|
|
66
|
-
job_id TEXT NOT NULL,
|
|
67
|
-
trigger_kind TEXT NOT NULL, -- 'scheduled' | 'manual'
|
|
68
|
-
status TEXT NOT NULL, -- running | succeeded | failed | timeout
|
|
69
|
-
started_at INTEGER NOT NULL,
|
|
70
|
-
completed_at INTEGER,
|
|
71
|
-
output TEXT, -- final cli stdout
|
|
72
|
-
exit_code INTEGER,
|
|
73
|
-
failure_reason TEXT,
|
|
74
|
-
log_path TEXT, -- absolute path to per-run log file
|
|
75
|
-
dispatch_results TEXT -- JSON array of per-recipient outcomes
|
|
76
|
-
-- {channelType, accountId, chatId?, ok, error?}
|
|
77
|
-
)
|
|
78
|
-
```
|
|
79
|
-
|
|
80
|
-
`run_at` is added in-place to pre-existing dbs via a `PRAGMA table_info`
|
|
81
|
-
guard in `createCronDb` (idempotent ALTER) — no manual migration step.
|
|
82
|
-
|
|
83
|
-
Per-target `chatId` pins delivery to a specific conversation. For
|
|
84
|
-
**telegram / slack / feishu** it is required — the dispatchers reject
|
|
85
|
-
targets without one (the old "fan out to allowedUsers" / "fall back to
|
|
86
|
-
lastActiveChatId" behaviour was removed because in practice the most-
|
|
87
|
-
recent inbound was rarely the cron creator, and the result kept landing
|
|
88
|
-
in a stranger's chat). For **wechat** it stays optional — wechat has a
|
|
89
|
-
QR-bound owner so single-recipient delivery has an obvious target.
|
|
90
|
-
|
|
91
|
-
Cron jobs created from inside a chat via the `manage-cron-jobs` skill auto-pin
|
|
92
|
-
the originating `chatId` (the channel context exposes it as
|
|
93
|
-
`{{channel.chat_id}}`). Admin-UI cron jobs that don't specify a chatId
|
|
94
|
-
for tg/slack/feishu run silently — the result shows in the cron log,
|
|
95
|
-
nothing is pushed.
|
|
96
|
-
|
|
97
|
-
Why a separate db: same reasoning as `evo.db` — global, cross-workspace
|
|
98
|
-
state shouldn't live inside any one workspace's `.halo/`. The runner
|
|
99
|
-
itself is workspace-agnostic.
|
|
100
|
-
|
|
101
|
-
### Runner — `packages/server/src/cron/runner.ts`
|
|
102
|
-
|
|
103
|
-
Boot path:
|
|
104
|
-
1. `startCronDaemon()` is called from server `index.ts` after
|
|
105
|
-
`setCronDb(...)`.
|
|
106
|
-
2. Reads every `enabled=1` row, instantiates a croner schedule per job.
|
|
107
|
-
- Recurring rows: `new Cron(job.schedule, { timezone }, fn)`.
|
|
108
|
-
- One-shot rows (`runAt` set): `new Cron(new Date(job.runAt), …, fn)`.
|
|
109
|
-
croner accepts a `Date` and self-stops after firing once. Past-due
|
|
110
|
-
`runAt` is logged and skipped — the user can still fire via run-now.
|
|
111
|
-
3. Starts two timers:
|
|
112
|
-
- `pruneOldLogs` — daily, drops log files past 30-day retention and
|
|
113
|
-
`cron_runs` rows past 60 days.
|
|
114
|
-
- `reconcileFromDb` — every 10 seconds, syncs the in-memory schedule
|
|
115
|
-
map against db rows. Detects adds / removes / schedule / runAt /
|
|
116
|
-
timezone / enabled changes and patches the croner instances
|
|
117
|
-
accordingly. Cheap (small table) and covers out-of-band edits (the
|
|
118
|
-
`manage-cron-jobs` skill writes via `sqlite3` rather than the REST
|
|
119
|
-
API). Per-row fingerprint = `enabled|schedule|runAt|timezone`.
|
|
120
|
-
|
|
121
|
-
When a fire happens:
|
|
122
|
-
1. `runJob(jobId, 'scheduled')` writes a `cron_runs` row with
|
|
123
|
-
`status='running'`.
|
|
124
|
-
2. Spawns `halo cli -a <agentId> -s cron-<jobId> -w <workspacePath>` with
|
|
125
|
-
the prompt piped on **stdin** (not argv — a long prompt would overflow the
|
|
126
|
-
Windows command-line limit). A Node timer kills the child after
|
|
127
|
-
`CLI_TIMEOUT_SEC` and reports exit 124 — same contract as the Linux-only
|
|
128
|
-
`timeout(1)` binary, but cross-platform.
|
|
129
|
-
On Windows `resolveHaloCli()` returns `halo.cmd` (not bare `halo`):
|
|
130
|
-
the desktop installer drops `halo.cmd` and `Halo.exe` into the same
|
|
131
|
-
dir on PATH, and PATHEXT ranks `.EXE` above `.CMD`, so a bare `halo`
|
|
132
|
-
would launch the GUI instead of the cli. Spawning a `.cmd` directly is
|
|
133
|
-
rejected by Node ≥21.7 (EINVAL), so the child is routed through
|
|
134
|
-
`cmd.exe /c` on Windows.
|
|
135
|
-
On **macOS** bare `halo` resolves via PATH to the `/usr/local/bin`
|
|
136
|
-
launcher, but a Finder/Dock-launched desktop app has only launchd's
|
|
137
|
-
minimal PATH — so the Electron main process prepends the standard CLI
|
|
138
|
-
dirs to the server child's `env.PATH` (see `dev/desktop-packaging.md`),
|
|
139
|
-
otherwise this spawn would `ENOENT`.
|
|
140
|
-
The `-s` flag uses **create-on-missing semantics** (added alongside
|
|
141
|
-
cron): if the session id already exists in `agent_sessions`, cli
|
|
142
|
-
resumes it; if not, cli creates a new row with that exact id. So the
|
|
143
|
-
first fire bootstraps the session, every subsequent fire continues
|
|
144
|
-
the same conversation, and the user can review history in the admin
|
|
145
|
-
Sessions tab. The `cron-<jobId>` prefix avoids collision with normal
|
|
146
|
-
`cli_*` / `tg_*` / `wx_*` ids.
|
|
147
|
-
Same shape as the evo wrapper (`spawn` + tee'd stdout/stderr to
|
|
148
|
-
`~/.halo/global/logs/cron/<runId>.log`).
|
|
149
|
-
3. On exit:
|
|
150
|
-
- Exit 0 + non-empty stdout → `succeeded`, dispatch the trimmed
|
|
151
|
-
stdout to every target.
|
|
152
|
-
- Exit 124 → `timeout`.
|
|
153
|
-
- Anything else → `failed` with stderr tail as `failure_reason`.
|
|
154
|
-
- Empty output on exit 0 → `failed: cli produced no output`.
|
|
155
|
-
4. If every target's dispatch failed, downgrade the run status to
|
|
156
|
-
`failed` so the UI surfaces the issue (and so a job that always
|
|
157
|
-
times-out on telegram doesn't keep showing green).
|
|
158
|
-
5. **At-mode auto-disable.** `finalize` looks up the row; when `runAt`
|
|
159
|
-
is set, it patches `enabled=0` alongside the lastRunStatus update.
|
|
160
|
-
Reconcile then tears down the croner instance on its next pass.
|
|
161
|
-
The row is kept (so the admin UI shows it under run history; user
|
|
162
|
-
can still hit run-now to refire).
|
|
163
|
-
|
|
164
|
-
### Dispatcher — `packages/server/src/cron/dispatcher.ts` (registry)
|
|
165
|
-
|
|
166
|
-
The dispatcher is **channel-agnostic**. It exposes a tiny registry
|
|
167
|
-
(`registerCronDispatcher`) and a single fan-out entry-point
|
|
168
|
-
(`dispatchToTargets`); each channel module supplies its own
|
|
169
|
-
`CronChannelDispatcher` and registers it at server boot. There is no
|
|
170
|
-
switch-on-channelType anywhere in `cron/`.
|
|
171
|
-
|
|
172
|
-
```ts
|
|
173
|
-
interface CronChannelDispatcher {
|
|
174
|
-
channelType: string
|
|
175
|
-
dispatch: (accountId: string, text: string, chatId?: string) => Promise<DispatchResult[]>
|
|
176
|
-
listTargets?: () => CronTargetOption[] // for admin UI dropdown
|
|
177
|
-
}
|
|
178
|
-
```
|
|
179
|
-
|
|
180
|
-
`dispatch` returns one `DispatchResult` row per recipient. Today every
|
|
181
|
-
channel is single-recipient (one explicit chatId → one row), but the
|
|
182
|
-
array shape is preserved so a future channel that supports broadcast
|
|
183
|
-
could return N. Throwing wraps into a single failed-result.
|
|
184
|
-
|
|
185
|
-
Channel modules:
|
|
186
|
-
|
|
187
|
-
- `packages/server/src/channels/telegram/cron-dispatcher.ts` —
|
|
188
|
-
`registerTelegramCronDispatcher()`. Requires explicit numeric `chatId`;
|
|
189
|
-
errors otherwise. (Previous build fanned out to `allowedUsers` and fell
|
|
190
|
-
back to `lastActiveChatId`; both removed because cron output kept
|
|
191
|
-
landing in unrelated chats.)
|
|
192
|
-
- `packages/server/src/channels/wechat/cron-dispatcher.ts` —
|
|
193
|
-
`registerWechatCronDispatcher()`. Single-recipient with chained
|
|
194
|
-
fallback: explicit `chatId` → account `userId` (QR-bind owner) →
|
|
195
|
-
cached `lastActiveChatId`. WeChat keeps the fallback because the
|
|
196
|
-
QR-bound owner is a stable, unambiguous target.
|
|
197
|
-
- `packages/server/src/channels/slack/cron-dispatcher.ts` —
|
|
198
|
-
`registerSlackCronDispatcher()`. Requires explicit `chatId` shaped as
|
|
199
|
-
`D…` (DM), `C…` (channel top-level), or `C…:<thread_ts>` (anchored
|
|
200
|
-
inside a thread). Posts via `postMessage` with `thread_ts` set when
|
|
201
|
-
present.
|
|
202
|
-
- `packages/server/src/channels/feishu/cron-dispatcher.ts` —
|
|
203
|
-
`registerFeishuCronDispatcher()`. Requires explicit `chatId` shaped as
|
|
204
|
-
`oc_…`. Posts via the v1 messages API with `receive_id_type=chat_id`.
|
|
205
|
-
(A `:rootId` suffix is accepted in the storage shape for forward-compat
|
|
206
|
-
but currently ignored — Feishu's open API has no thread-anchored
|
|
207
|
-
sendMessage path analogous to Slack's `thread_ts`.)
|
|
208
|
-
|
|
209
|
-
Both register at server boot in `index.ts`. **Adding a new channel = ship
|
|
210
|
-
a `cron-dispatcher.ts` next to its handler + register it once.** Nothing
|
|
211
|
-
in `cron/` or `routes/cron.ts` needs to change.
|
|
212
|
-
|
|
213
|
-
`listTargets()` per channel feeds the admin UI's create-form dropdown
|
|
214
|
-
via `listAllCronTargets()` (aggregator). `ready` flag: wechat = QR-bind
|
|
215
|
-
userId OR cached lastActiveChatId; telegram / slack / feishu always
|
|
216
|
-
report `ready=true` (readiness depends on the per-target `chatId` the
|
|
217
|
-
admin picks, not on prior inbound history). The admin form has a
|
|
218
|
-
per-target chatId input + search dropdown for slack/feishu, backed by
|
|
219
|
-
`users.conversations` / `/im/v1/chats`. Channels with no UI-pickable
|
|
220
|
-
surface (a hypothetical webhook channel) can omit `listTargets`.
|
|
221
|
-
|
|
222
|
-
`DispatchResult` shape:
|
|
223
|
-
|
|
224
|
-
```ts
|
|
225
|
-
{ channelType, accountId, chatId?, ok, error? }
|
|
226
|
-
```
|
|
227
|
-
|
|
228
|
-
`chatId` is recorded so the admin UI can render per-recipient outcomes
|
|
229
|
-
("slack:T01234/D012345 ✓ · feishu:appA/oc_xxx ✗ (no permission)") when a
|
|
230
|
-
job has multiple targets, instead of just a single ok/fail roll-up.
|
|
231
|
-
|
|
232
|
-
### Realtime updates — WS broadcast (no client polling)
|
|
233
|
-
|
|
234
|
-
The admin UI subscribes to `cron:job_changed` and `cron:run_changed`
|
|
235
|
-
events on the existing `/ws` channel rather than polling on a timer.
|
|
236
|
-
Server emits them at every state-change point:
|
|
237
|
-
|
|
238
|
-
- `runner.runJob` insert of the `running` cron_runs row →
|
|
239
|
-
`cron:run_changed status='running'`
|
|
240
|
-
- `runner.finalize` (success/failure/timeout) →
|
|
241
|
-
`cron:run_changed status='succeeded'|'failed'|'timeout'` plus a
|
|
242
|
-
`cron:job_changed` so the list's "last run" badge updates
|
|
243
|
-
- REST routes (POST/PUT/DELETE on `/cron/jobs[/:id]`) →
|
|
244
|
-
`cron:job_changed kind='created'|'updated'|'deleted'`
|
|
245
|
-
|
|
246
|
-
Broadcast is fan-out via `ws/broadcast.ts` to every connected admin
|
|
247
|
-
socket. Idle traffic is ~zero.
|
|
248
|
-
|
|
249
|
-
### REST routes — `packages/server/src/routes/cron.ts`
|
|
250
|
-
|
|
251
|
-
Standard CRUD over `cron_jobs` plus:
|
|
252
|
-
|
|
253
|
-
- `POST /api/cron/jobs/:id/run-now` — fire-and-forget; returns
|
|
254
|
-
immediately, run status flips broadcast as cron:run_changed events.
|
|
255
|
-
- `GET /api/cron/jobs/:id/runs?limit=N&before=<runId>` — cursor-paged
|
|
256
|
-
history. `runId` is `<isoTimestamp>-<slug>`, so its lexicographic
|
|
257
|
-
order matches time — `id < cursor` gives "older than" without a
|
|
258
|
-
numeric sequence column. Response: `{ runs, hasMore, nextCursor }`.
|
|
259
|
-
- `GET /api/cron/runs/:runId/log` — raw log content (or null if past
|
|
260
|
-
retention).
|
|
261
|
-
- `GET /api/cron/channel-targets` — the create-form's list of bindable
|
|
262
|
-
channel accounts. Aggregated from the registry via
|
|
263
|
-
`listAllCronTargets()` — routes/cron.ts is channel-agnostic. `ready`
|
|
264
|
-
flag is computed inside each channel's `listTargets()` (see
|
|
265
|
-
"Dispatcher" above for the per-channel rules).
|
|
266
|
-
- `POST /api/cron/reload` — force re-read all schedules. Rarely needed
|
|
267
|
-
(reconcile already polls); useful after manual db edits if you don't
|
|
268
|
-
want to wait 10 seconds.
|
|
269
|
-
|
|
270
|
-
All routes go through the standard auth middleware — same cookie/token
|
|
271
|
-
as the rest of `/api`.
|
|
272
|
-
|
|
273
|
-
### Agent skill — `templates/skills/manage-cron-jobs/`
|
|
274
|
-
|
|
275
|
-
Registered in `init.ts`'s `BUILTIN_SKILL_IDS`. Frontmatter has
|
|
276
|
-
`requiresAccess: full`, so readonly / workspace channels (like a
|
|
277
|
-
public-facing telegram bot) can't see or invoke this skill — only
|
|
278
|
-
admin-shell agents can. The skill ships with a `manage-cron.py` helper
|
|
279
|
-
that the agent calls via `shell_exec`; the script handles id
|
|
280
|
-
generation, target JSON encoding, and timestamps so the agent isn't
|
|
281
|
-
hand-crafting SQL.
|
|
282
|
-
|
|
283
|
-
The runner's reconcile loop picks up db changes within ~10 seconds and
|
|
284
|
-
broadcasts `cron:job_changed { kind: 'reconciled' | 'deleted' }` so
|
|
285
|
-
admin UIs in another tab/browser see the same edit. The skill doesn't
|
|
286
|
-
need REST/auth — it's filesystem + sqlite, with the in-process runner
|
|
287
|
-
catching up on its own pace.
|
|
288
|
-
|
|
289
|
-
The skill walks the agent through:
|
|
290
|
-
1. Identifying user intent (list / create / edit / delete).
|
|
291
|
-
2. Defaulting workspace (current), agent (`default`), targets (current
|
|
292
|
-
chat when invoked from a channel — see channel context below).
|
|
293
|
-
3. Translating spoken-time → 5-field cron (recurring) **or** ISO-8601
|
|
294
|
-
instant (`--run-at`, one-shot at-mode).
|
|
295
|
-
4. Querying `channels.db` for available targets and surfacing
|
|
296
|
-
readiness warnings (currently only wechat can be `ready=false`).
|
|
297
|
-
5. Writing the appropriate INSERT/UPDATE/DELETE.
|
|
298
|
-
|
|
299
|
-
**Channel context at skill invocation.** When the user triggers the
|
|
300
|
-
`manage-cron-jobs` skill from inside a telegram / wechat / slack / feishu
|
|
301
|
-
chat, the channel handler injects structured origin into the skill's
|
|
302
|
-
render context as built-in placeholders: `{{channel.type}}`,
|
|
303
|
-
`{{channel.account_id}}`, `{{channel.chat_id}}`. The skill body uses
|
|
304
|
-
these to default the `--targets` flag to `<type>:<account_id>:<chat_id>`
|
|
305
|
-
so the schedule keeps replying in the same conversation. Admin/WS
|
|
306
|
-
invocations leave the placeholders empty — the skill then either asks
|
|
307
|
-
for an explicit chatId or accepts a silent (no-target) cron.
|
|
308
|
-
|
|
309
|
-
**Reverse-lookup by chatId.** Inside a chat the user doesn't know the
|
|
310
|
-
cron `id` ("delete the daily digest I set up here"). The helper script
|
|
311
|
-
exposes `list --chat-id <id>` that filters `cron_jobs` to those whose
|
|
312
|
-
`targets[].chatId` matches. The skill uses
|
|
313
|
-
`list --chat-id {{channel.chat_id}}` to find candidate jobs, confirms
|
|
314
|
-
with the user, then deletes by id. Different channels' chatId formats
|
|
315
|
-
don't collide (telegram numeric, slack `D…/C…`, feishu `oc_…`,
|
|
316
|
-
wechat `o…`), so a single chatId match is unambiguous across channels.
|
|
317
|
-
|
|
318
|
-
### Admin UI — `packages/admin/src/features/cron/cron-main.tsx`
|
|
319
|
-
|
|
320
|
-
Activity-bar entry next to Evolution. Two-pane layout matching
|
|
321
|
-
Evolution's: list of jobs on the left, detail panel on the right with
|
|
322
|
-
run history and edit/run-now/delete actions. Form supports both create
|
|
323
|
-
and edit. Form pulls workspace + agent + channel-target options from
|
|
324
|
-
existing APIs (`/api/agent-configs`, `/api/cron/channel-targets`).
|
|
325
|
-
|
|
326
|
-
## Boot
|
|
327
|
-
|
|
328
|
-
`packages/server/src/index.ts`:
|
|
329
|
-
```ts
|
|
330
|
-
setChannelDb(channelDb) // for dispatcher's getChannelDb()
|
|
331
|
-
setCronDb(createCronDb(path.join(HALO_HOME, 'global')))
|
|
332
|
-
// `bootChannels` walks `defaultChannelDescriptors` and calls each
|
|
333
|
-
// descriptor's `registerCronDispatcher` (alongside its handler/route
|
|
334
|
-
// setup). Order is irrelevant — the registry is keyed by channelType.
|
|
335
|
-
bootChannels(app, defaultChannelDescriptors, { registry, db: channelDb })
|
|
336
|
-
// Daemon starts after channels boot so the registry is fully populated
|
|
337
|
-
// before the first scheduled fire.
|
|
338
|
-
startCronDaemon()
|
|
339
|
-
```
|
|
340
|
-
|
|
341
|
-
`packages/server/src/init.ts` `BUILTIN_SKILL_IDS` includes
|
|
342
|
-
`manage-cron-jobs` so `halo setup` deploys the skill.
|
|
343
|
-
|
|
344
|
-
## Known limits
|
|
345
|
-
|
|
346
|
-
- **No mid-run heartbeat.** If the server crashes between the
|
|
347
|
-
`cron_runs.insert(status='running')` and the cli exit handler, the row
|
|
348
|
-
is stranded in `running`. (Evo handles this via heartbeat + ticker
|
|
349
|
-
recovery.) For cron, runs are short (<10 minutes by `timeout`) and
|
|
350
|
-
infrequent; we accept the rough edge for now. Fix: add a daily
|
|
351
|
-
scanner that flips stale-running rows to `failed: server crashed`.
|
|
352
|
-
- **Web channel can't deliver.** SSE channel is reactive-only; cron
|
|
353
|
-
records a clear failure rather than swallowing.
|
|
354
|
-
- **No retry.** Failed runs stay failed; user sees them in history and
|
|
355
|
-
decides whether to investigate or wait for the next fire.
|
|
356
|
-
- **Single-process scheduler.** Two server instances against the same
|
|
357
|
-
global db would both schedule the same jobs. Today there's only ever
|
|
358
|
-
one server per host (pid lock); document & enforce when that
|
|
359
|
-
assumption ever breaks.
|
|
360
|
-
- **`run-now` ignores enabled flag.** A disabled job can still be
|
|
361
|
-
fired manually from the UI for testing — the schedule is paused, the
|
|
362
|
-
runner code path isn't.
|
|
@@ -1,203 +0,0 @@
|
|
|
1
|
-
# Session persistence and orchestration — design suggestions
|
|
2
|
-
|
|
3
|
-
Captured while debugging the "third message freeze" bug (`text content blocks must be non-empty`) — documents the structural issues found and proposes improvements over the current state. Everything revolves around **session message state**: multiple write paths, multiple sources of truth, and lossy backward inferences — the shared root of that bug.
|
|
4
|
-
|
|
5
|
-
---
|
|
6
|
-
|
|
7
|
-
## 1. Make message persistence one-way — `rawMessages` as the single source of truth
|
|
8
|
-
|
|
9
|
-
### Current state
|
|
10
|
-
|
|
11
|
-
Each session maintains two copies:
|
|
12
|
-
|
|
13
|
-
- **`rawMessages`**: Strands Agent's internal `agent.messages`, Bedrock's native format including full `tool_use` / `tool_result` pairs. Persisted by `saveAgentState` at turn end.
|
|
14
|
-
- **`messages`**: the UI display log, assembled by `event-processor` from streaming events (assistant text, tool_call cards, usage badges, etc.). Rendered directly on refresh.
|
|
15
|
-
|
|
16
|
-
The two paths write independently; consistency relies on order-of-ops going right.
|
|
17
|
-
|
|
18
|
-
### Problem
|
|
19
|
-
|
|
20
|
-
This bug came from a third path — `handler.ts`'s `restoreHistory(sid, client.messageLog)` — that tried to reverse the UI log back into `rawMessages`. Tool-only assistant turns in the UI log have empty-string content (tool cards don't live in the content field), so the inverse maps them to `{type: 'text', text: ''}` and Bedrock rejects it.
|
|
21
|
-
|
|
22
|
-
As long as the "UI log → rawMessages" direction exists, "restore agent state from UI" will keep producing bugs like this.
|
|
23
|
-
|
|
24
|
-
### Suggestion
|
|
25
|
-
|
|
26
|
-
- **`rawMessages` is the sole source of truth**; the UI log is derived.
|
|
27
|
-
- **Never infer `rawMessages` from the UI log.**
|
|
28
|
-
- On refresh / reconnect: UI log is loaded directly from disk (for rendering), `rawMessages` is lazily loaded by `ensureSession` (for the next Bedrock call). The two paths run **in parallel and don't rebuild each other.**
|
|
29
|
-
- Compact should also operate on `rawMessages` — summarise, replace the old slice, then derive the new UI log from the event stream. Not "edit UI log then reverse-derive".
|
|
30
|
-
|
|
31
|
-
---
|
|
32
|
-
|
|
33
|
-
## 2. Run repair once before every Bedrock call
|
|
34
|
-
|
|
35
|
-
### Current state
|
|
36
|
-
|
|
37
|
-
`repairConversationMessages` currently runs:
|
|
38
|
-
|
|
39
|
-
- At turn end (end of `runAgentTurn`)
|
|
40
|
-
- After interrupt / stop
|
|
41
|
-
- After compact
|
|
42
|
-
|
|
43
|
-
But **not before `sendUserMessage` hands over to Bedrock**.
|
|
44
|
-
|
|
45
|
-
### Problem
|
|
46
|
-
|
|
47
|
-
On-disk `rawMessages` may be unhealthy due to older bugs, external edits, version upgrades, etc. Today, once `ensureSession` loads bad state from disk, it gets sent to Bedrock and triggers the same kind of crash we just hit.
|
|
48
|
-
|
|
49
|
-
I've added "drop empty text blocks" as a safety net in `conversation-repair.ts`; still, moving the defensive step forward catches future unknown bad data.
|
|
50
|
-
|
|
51
|
-
### Suggestion
|
|
52
|
-
|
|
53
|
-
- In `sendUserMessage`, before hitting `handleUserTurn`, call `repairConversationMessages(session.agent.messages)`.
|
|
54
|
-
- Repair becomes the "last gate into Bedrock", so wherever rawMessages came from (new / disk restore / older-version file), it's cleansed.
|
|
55
|
-
- Performance is fine: repair is O(n) and n is not large.
|
|
56
|
-
|
|
57
|
-
---
|
|
58
|
-
|
|
59
|
-
## 3. Sub-session log persistence: move "merge-save" out of event-processor, into SessionManager
|
|
60
|
-
|
|
61
|
-
### Current state
|
|
62
|
-
|
|
63
|
-
`event-processor.ts`'s `saveSubSession`:
|
|
64
|
-
|
|
65
|
-
```
|
|
66
|
-
On every query_session / start_session
|
|
67
|
-
→ subSessionLogs initialised in memory
|
|
68
|
-
→ event stream writes into it
|
|
69
|
-
→ on agent_done: read from disk → merge by message id → write back
|
|
70
|
-
```
|
|
71
|
-
|
|
72
|
-
`subSessionLogs` is client-connection-level (WebSocket), rebuilt per query and holding only this turn's events. Hence the merge.
|
|
73
|
-
|
|
74
|
-
### Problem
|
|
75
|
-
|
|
76
|
-
- **merge-by-id depends on "ids are consistent and unique"**. Any id jitter, reconnect-leftover state, or duplicate `initSubSessionLog` call produces lost or duplicate messages.
|
|
77
|
-
- **Persistence is coupled to the network layer**. event-processor should just forward events to the WS; here it also takes on "cross-connection disk merge". Violates CLAUDE.md's "persistence must not depend on in-memory transient state" — `subSessionLogs` is in-memory transient state.
|
|
78
|
-
- **Reconnect / refresh is error-prone**: a new connection creates a new `subSessionLogs`; unflushed events from the old one are lost; detach creates yet another independent copy in `backgroundHandler`, and reconciling both is a hazard.
|
|
79
|
-
|
|
80
|
-
### Event loss risk (extended)
|
|
81
|
-
|
|
82
|
-
This is a separate risk beyond the merge mechanism — an **eye-vs-disk** inconsistency where **events actually happened, but the log doesn't reflect them**. Scenarios:
|
|
83
|
-
|
|
84
|
-
1. **Stream buffer not flushed before a crash**. `stream` events aggregate into in-memory `turnContentBlocks` / `streamBuffer`; only `usage` / `agent_done` / `complete` trigger `flushAssistantMessage` + `saveSubSession`. Crash beforehand (OOM, uncaught exception, kill) loses every piece of streaming text and tool_call for the turn — disk has no trace the agent said anything.
|
|
85
|
-
|
|
86
|
-
2. **WS-disconnect brief window**. Between disconnect and `backgroundHandler` takeover is a window where `sendJson` fails and the event is silently dropped; grace expiry triggers a single `save()`, and a crash in between loses the whole stretch.
|
|
87
|
-
|
|
88
|
-
3. **Merge id jitter losing messages**. `saveSubSession` dedupes by `m.id`, but `genId()`'s counter is process-scoped — process restart resets the counter. A restart between two flushes can produce same-id-different-message entries; merge overwrites them.
|
|
89
|
-
|
|
90
|
-
4. **Event semantics vs disk state out of sync**. Frontend / logs already saw a `tool_call` fire (went through `sendJson`), but it's only in the in-memory `messageLog`; the next `saveSubSession` writes it. Checking disk in between makes it look like nothing happened. Fatal for audit-log use cases.
|
|
91
|
-
|
|
92
|
-
### Suggestion
|
|
93
|
-
|
|
94
|
-
**Short-term (bundle with the merge issue)**:
|
|
95
|
-
- SessionManager **owns** the sub-session UI log (persisted alongside `rawMessages`), append-only, single-process concurrency-safe.
|
|
96
|
-
- event-processor reverts to **pure forwarding**: receive event → push to WS; no persistence duty.
|
|
97
|
-
- Tighten flush granularity from "key events" to "every tool_call / tool_result / every N seconds" to shrink the crash window.
|
|
98
|
-
|
|
99
|
-
**Long-term (fully eliminate event loss)**:
|
|
100
|
-
- Switch to an **append-only JSONL event log**: every agent event lands as a new line in `.halo/sessions/{agentId}/{sessionId}.jsonl` the moment it fires.
|
|
101
|
-
- Renderers aggregate to `SessionMessage[]` on load (event-sourcing projection).
|
|
102
|
-
- Cost: higher disk I/O (one-write-per-stream-delta is too much). Compromise: "every tool_call / tool_result / end-of-assistant-paragraph / every 500 ms batch append".
|
|
103
|
-
- Benefit: crash window shrinks from "whole turn" to "last flush interval"; disk is a faithful log of events, no code-path skipping `save()` loses a whole stretch.
|
|
104
|
-
|
|
105
|
-
---
|
|
106
|
-
|
|
107
|
-
## 4. `query_session` to an IDLE target: `onComplete` vs `releaseSession` race
|
|
108
|
-
|
|
109
|
-
### Current state
|
|
110
|
-
|
|
111
|
-
```typescript
|
|
112
|
-
// session-manager.ts :1166
|
|
113
|
-
this.runSession(targetSessionId, prefix + message).then((result) => {
|
|
114
|
-
onComplete?.(result)
|
|
115
|
-
})
|
|
116
|
-
return `Message sent to session ${targetSessionId}...`
|
|
117
|
-
```
|
|
118
|
-
|
|
119
|
-
`runSession` is fire-and-forget with `onComplete` in `.then()`. But `runSession`'s own `finally` (:1030) immediately calls `releaseSession`.
|
|
120
|
-
|
|
121
|
-
### Problem
|
|
122
|
-
|
|
123
|
-
JavaScript microtask scheduling usually keeps `finally` before the `.then()` callback, but the race exists:
|
|
124
|
-
|
|
125
|
-
- `finally`'s `releaseSession` drops the session from memory.
|
|
126
|
-
- Immediately after, `.then(onComplete)` fires; `onComplete` itself calls `querySession(sessionId, childSessionId, result)` (auto-report to source), and that `querySession` does `ensureSession(childSessionId)` — which has to go through the disk restore path since memory is gone.
|
|
127
|
-
- If the disk save hasn't fully flushed (`saveAgentState` is synchronous but can be interleaved by the Node event loop), or something else is writing concurrently, `ensureSession` might read stale state.
|
|
128
|
-
|
|
129
|
-
Not guaranteed, but the window exists. E3 (query_session to an idle sub-agent) can likely reproduce under concurrency / repetition.
|
|
130
|
-
|
|
131
|
-
### Suggestion
|
|
132
|
-
|
|
133
|
-
- Chain `onComplete` onto `session.promise` itself, so "`runSession` fully finishes (including `releaseSession`) → `onComplete` → next `querySession`" runs strictly in sequence.
|
|
134
|
-
- Or simpler: have `querySession` itself `await runSession` and call `onComplete` synchronously, avoiding `.then()` entirely.
|
|
135
|
-
- Add a "just-released session stays in memory for N ms" cache window in `ensureSession` to damp rapid release/restore cycles.
|
|
136
|
-
|
|
137
|
-
---
|
|
138
|
-
|
|
139
|
-
## 5. Post-disconnect event replay needs a barrier on reconnect
|
|
140
|
-
|
|
141
|
-
### Current state
|
|
142
|
-
|
|
143
|
-
WS disconnects:
|
|
144
|
-
- Session marked "detached", `backgroundHandler` registered
|
|
145
|
-
- Events flow into `detached.pendingEvents`
|
|
146
|
-
|
|
147
|
-
WS reconnects:
|
|
148
|
-
- `handleSubscribe` detects the detached session → replays `pendingEvents` one by one via `sendJson`
|
|
149
|
-
- Meanwhile switches the listener back to the new WS's live callback
|
|
150
|
-
|
|
151
|
-
### Problem
|
|
152
|
-
|
|
153
|
-
There's no barrier between listener switch and `pendingEvents` replay. If SessionManager is currently emitting a new event:
|
|
154
|
-
|
|
155
|
-
- The new event (B) goes through the live listener to the new WS
|
|
156
|
-
- Older `pendingEvents` (A) are still being sent one-by-one
|
|
157
|
-
- Frontend sees B before A — out of order
|
|
158
|
-
|
|
159
|
-
F2 ("disconnect → reconnect") + "steady events during disconnect + fresh events at the moment of reconnect" can reproduce it.
|
|
160
|
-
|
|
161
|
-
### Suggestion
|
|
162
|
-
|
|
163
|
-
- On reconnect, hold a lock: `pendingEvents` keeps accepting but doesn't **drain**; the listener is still on the background handler.
|
|
164
|
-
- Atomic switchover: `listener replace + pendingEvents drain + subSessionLogs restore` is one transaction; new events during this keep flowing to `pendingEvents`.
|
|
165
|
-
- After the transaction, flush `pendingEvents` and switch the listener to the new WS.
|
|
166
|
-
- Add an integration test: F2 scenario + forced concurrent events; watch the sequence numbers arriving on the frontend are monotonically increasing.
|
|
167
|
-
|
|
168
|
-
---
|
|
169
|
-
|
|
170
|
-
## 6. Clean up the deprecated `destroySessions`
|
|
171
|
-
|
|
172
|
-
### Current state
|
|
173
|
-
|
|
174
|
-
`session-manager.ts` has a `@deprecated`-tagged `destroySessions` method with a doc saying "Use deleteSession + deleteSessionLogs in Phase 3".
|
|
175
|
-
|
|
176
|
-
### Problem
|
|
177
|
-
|
|
178
|
-
- It's debt — deprecated methods can still be called accidentally, and they raise eyebrows in review.
|
|
179
|
-
- New readers get confused: which one do I use?
|
|
180
|
-
- Keeping two deletion paths means every future change has to consider both.
|
|
181
|
-
|
|
182
|
-
### Suggestion
|
|
183
|
-
|
|
184
|
-
- Just delete `destroySessions`.
|
|
185
|
-
- Grep-verify no callers (from `.halo/docs/` through `packages/`).
|
|
186
|
-
- If stragglers exist, migrate them to `deleteSession` + `deleteSessionLogs`, then delete.
|
|
187
|
-
|
|
188
|
-
---
|
|
189
|
-
|
|
190
|
-
## Priority
|
|
191
|
-
|
|
192
|
-
By fix-value vs change-footprint:
|
|
193
|
-
|
|
194
|
-
| # | Suggestion | Priority | Scope |
|
|
195
|
-
|---|---|---|---|
|
|
196
|
-
| 2 | Repair before sendUserMessage | High | one line |
|
|
197
|
-
| 6 | Delete destroySessions | High | one method |
|
|
198
|
-
| 1 | One-way (forbid UI→raw inverse) | High | architectural principle |
|
|
199
|
-
| 4 | Fix onComplete race | Medium | one callback |
|
|
200
|
-
| 5 | Reconnect event barrier | Medium | one subsystem |
|
|
201
|
-
| 3 | Sink sub-session persistence | Low, long-term | larger refactor |
|
|
202
|
-
|
|
203
|
-
1, 2, 6 can be done right away; 4, 5 deserve integration tests first. 3 is the long-term direction — not urgent, but inevitable.
|