discoclaw 1.2.3 → 1.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. package/.context/voice.md +30 -2
  2. package/.env.example +6 -0
  3. package/dist/cli/dashboard.js +7 -1
  4. package/dist/config.js +7 -0
  5. package/dist/cron/executor.js +72 -1
  6. package/dist/dashboard/api/metrics.js +7 -0
  7. package/dist/dashboard/api/metrics.test.js +16 -0
  8. package/dist/dashboard/api/traces.js +14 -0
  9. package/dist/dashboard/api/traces.test.js +40 -0
  10. package/dist/dashboard/page.js +187 -8
  11. package/dist/dashboard/server.js +81 -14
  12. package/dist/dashboard/server.test.js +120 -4
  13. package/dist/discord/deferred-runner.js +306 -219
  14. package/dist/discord/message-coordinator.js +1 -28
  15. package/dist/discord/reaction-handler.js +81 -3
  16. package/dist/index.js +15 -1
  17. package/dist/observability/trace-store.js +56 -0
  18. package/dist/observability/trace-utils.js +31 -0
  19. package/dist/runtime/codex-cli.js +3 -2
  20. package/dist/runtime/codex-cli.test.js +33 -0
  21. package/dist/runtime/model-tiers.js +1 -1
  22. package/dist/runtime/model-tiers.test.js +9 -0
  23. package/dist/runtime/openai-tool-schemas.js +17 -0
  24. package/dist/voice/audio-pipeline.js +246 -6
  25. package/dist/voice/audio-pipeline.test.js +481 -0
  26. package/dist/voice/audio-receiver.js +8 -0
  27. package/dist/voice/audio-receiver.test.js +16 -0
  28. package/dist/voice/conversation-buffer.js +16 -6
  29. package/dist/voice/providers/gemini-live-provider.js +481 -0
  30. package/dist/voice/providers/gemini-live-provider.test.js +834 -0
  31. package/dist/voice/providers/gemini-live-responder.js +267 -0
  32. package/dist/voice/providers/gemini-live-responder.test.js +615 -0
  33. package/dist/voice/providers/gemini-live-token-estimator.js +100 -0
  34. package/dist/voice/providers/gemini-live-token-estimator.test.js +160 -0
  35. package/dist/voice/providers/gemini-live-types.js +32 -0
  36. package/dist/voice/providers/gemini-tool-mapper.js +91 -0
  37. package/dist/voice/providers/gemini-tool-mapper.test.js +253 -0
  38. package/dist/voice/providers/index.js +3 -0
  39. package/dist/voice/types.test.js +6 -0
  40. package/dist/voice/voice-prompt-builder.js +26 -17
  41. package/dist/voice/voice-prompt-builder.test.js +16 -1
  42. package/package.json +1 -1
  43. package/templates/instructions/SYSTEM_DEFAULTS.md +8 -0
package/.context/voice.md CHANGED
@@ -29,10 +29,16 @@ Two native npm packages power the Discord voice integration:
29
29
  | `src/voice/transcript-mirror.ts` | Posts user transcriptions and bot responses to a text channel |
30
30
  | `src/voice/voice-action-flags.ts` | Restricted action subset for voice invocations (messaging + tasks + memory only) |
31
31
  | `src/voice/conversation-buffer.ts` | Per-guild conversation ring buffer (10 turns) — stores user/model exchanges in memory; backfills from voice-log channel on join |
32
+ | `src/voice/providers/gemini-live-types.ts` | TypeScript interfaces for Gemini Live: `GeminiLiveOpts`, `GeminiLiveEvent`, `GeminiLiveState` |
33
+ | `src/voice/providers/gemini-live-provider.ts` | Bidirectional WebSocket session wrapper for the Gemini Multimodal Live API — connect/disconnect, audio send/receive, reconnect with exponential backoff |
34
+ | `src/voice/providers/gemini-live-responder.ts` | Bridges `GeminiLiveProvider` audio/text events to Discord `AudioPlayer` playback and `TranscriptMirror` logging |
35
+ | `src/voice/providers/index.ts` | Barrel re-export for Gemini Live provider modules |
32
36
  | `src/discord/actions-voice.ts` | Discord action types: `voiceJoin`, `voiceLeave`, `voiceStatus`, `voiceMute`, `voiceDeafen` |
33
37
 
34
38
  ## Audio Data Flow
35
39
 
40
+ ### Default pipeline (`voiceProvider: 'pipeline'`)
41
+
36
42
  ```
37
43
  User speaks in Discord voice channel
38
44
  → @discordjs/voice receiver emits Opus packets per user
@@ -47,6 +53,23 @@ User speaks in Discord voice channel
47
53
  → AudioPlayer → Discord voice connection
48
54
  ```
49
55
 
56
+ ### Gemini Live (`voiceProvider: 'gemini-live'`)
57
+
58
+ Bypasses separate STT/TTS/AI stages — Gemini handles speech recognition, reasoning, and speech synthesis in a single bidirectional WebSocket session.
59
+
60
+ ```
61
+ User speaks in Discord voice channel
62
+ → @discordjs/voice receiver emits Opus packets per user
63
+ → AudioReceiver: allowlist gate → OpusDecoder (48 kHz stereo PCM)
64
+ → downsample to 16 kHz mono
65
+ → SttProvider shim → GeminiLiveProvider.sendAudio() (WebSocket)
66
+ → Gemini Live: STT + reasoning + TTS (server-side)
67
+ ← audio events (24 kHz mono PCM) + text events
68
+ → GeminiLiveResponder: upsampleToDiscord (48 kHz stereo)
69
+ → AudioPlayer → Discord voice connection
70
+ → onBotResponse callback → TranscriptMirror (text channel)
71
+ ```
72
+
50
73
  ## Key Patterns
51
74
 
52
75
  - **Allowlist gating** — `AudioReceiver` only subscribes to users in `DISCORD_ALLOW_USER_IDS`. Empty allowlist = ignore everyone (fail-closed).
@@ -56,6 +79,8 @@ User speaks in Discord voice channel
56
79
  - **Generation-based cancellation** — `VoiceResponder` increments a generation counter on each new transcription. If a newer transcription arrives mid-pipeline, the older one is silently abandoned.
57
80
  - **Barge-in** — Gated on a non-empty STT transcription result, not the raw VAD `speaking.start` event. Echo from the bot's own TTS leaking through the user's mic produces empty transcriptions and is ignored. Only when `VoiceResponder.handleTranscription()` receives a non-empty transcript while the player is active does it stop playback and advance the generation counter. This eliminates false positives from echo without relying on a static grace-period timeout.
58
81
  - **Conversation ring buffer** — `ConversationBuffer` maintains a per-guild 10-turn ring buffer of user/model exchanges that gets injected into the voice prompt as formatted conversation history. Turns are appended live during a session. On voice join, the buffer backfills from recent voice-log channel messages so context carries across disconnects. The buffer is cleared when the bot leaves the voice channel.
82
+ - **`SttProvider` shim for Gemini Live** — In `gemini-live` mode, the pipeline still uses `AudioReceiver` for Opus decode and downsampling, but replaces the real STT provider with a lightweight shim object that implements the `SttProvider` interface. The shim's `feedAudio()` forwards PCM frames directly to `GeminiLiveProvider.sendAudio()`, while its `start()`/`stop()`/`onTranscription()` are no-ops. This reuses the existing audio-receive path without duplicating Opus decode or downsample logic.
83
+ - **Session rotation timer** — `GeminiLiveProvider` starts a timer on each successful connection that fires at `DISCOCLAW_GEMINI_SESSION_ROTATION_MS` (default 13 min), proactively triggering a graceful reconnect before Gemini's ~15 min server-side session limit. The timer reuses the existing reconnect-with-resume-handle path (ws-039), so audio gap is minimal. The timer is cleared on disconnect and reset on each reconnect. Set to `0` to disable rotation (the server will eventually kill the session).
59
84
  - **Re-entrancy guard** — `AudioPipelineManager.startPipeline` uses a `starting` set because `VoiceConnection.subscribe()` synchronously fires a Ready state change.
60
85
  - **Error containment** — `VoiceConnectionManager` catches connection errors and destroys the connection to prevent process crashes (e.g. DAVE handshake failures).
61
86
  - **Deepgram TTS 2000-char limit** — Deepgram Aura REST TTS returns HTTP 413 (silent failure) for inputs exceeding ~2000 characters. `tts-deepgram.ts` truncates the input to 2000 chars before sending to prevent silent audio dropouts. If the AI response is unexpectedly long (e.g. from a missing `VOICE_STYLE_INSTRUCTION`), the user will still hear a truncated response rather than silence.
@@ -78,8 +103,9 @@ When `voiceEnabled=true`, the post-connect block in `src/index.ts` initializes t
78
103
  | `DISCOCLAW_VOICE_ENABLED` | `0` | Master switch |
79
104
  | `DISCOCLAW_DISCORD_ACTIONS_VOICE` | `0` | Enable voice action types |
80
105
  | `DISCOCLAW_VOICE_AUTO_JOIN` | `0` | Auto-join when allowlisted user enters |
81
- | `DISCOCLAW_STT_PROVIDER` | `deepgram` | STT backend |
82
- | `DISCOCLAW_TTS_PROVIDER` | `cartesia` | TTS backend (`cartesia`, `deepgram`, `openai`, `kokoro`) |
106
+ | `DISCOCLAW_VOICE_PIPELINE_PROVIDER` | `pipeline` | Voice pipeline mode: `pipeline` (separate STT/AI/TTS stages) or `gemini-live` (single bidirectional Gemini WebSocket). Requires `GEMINI_API_KEY` when set to `gemini-live`. |
107
+ | `DISCOCLAW_STT_PROVIDER` | `deepgram` | STT backend (used in `pipeline` mode only; ignored in `gemini-live` mode) |
108
+ | `DISCOCLAW_TTS_PROVIDER` | `cartesia` | TTS backend (`cartesia`, `deepgram`, `openai`, `kokoro`) (used in `pipeline` mode only; ignored in `gemini-live` mode) |
83
109
  | `DISCOCLAW_VOICE_HOME_CHANNEL` | — | Voice audio channel name/ID used for prompt context (not transcript mirroring) |
84
110
  | `DISCOCLAW_VOICE_LOG_CHANNEL` | — | Text channel name/ID where `TranscriptMirror` posts user transcriptions and bot responses; falls back to bootstrap-provided `voiceLogChannelId` if unset |
85
111
  | `DISCOCLAW_VOICE_MODEL` | `capable` | AI model tier for voice responses |
@@ -89,5 +115,7 @@ When `voiceEnabled=true`, the post-connect block in `src/index.ts` initializes t
89
115
  | `DEEPGRAM_TTS_VOICE` | `aura-2-asteria-en` | Deepgram TTS voice name |
90
116
  | `DEEPGRAM_TTS_SPEED` | `1.3` | Deepgram TTS playback speed (range 0.5–1.5) |
91
117
  | `CARTESIA_API_KEY` | — | Required for cartesia TTS |
118
+ | `DISCOCLAW_GEMINI_SESSION_ROTATION_MS` | `780000` (13 min) | Time before proactive session rotation in `gemini-live` mode. Must be less than Gemini's ~15 min server-side limit. Set to `0` to disable. |
119
+ | `GEMINI_API_KEY` | — | Required when `DISCOCLAW_VOICE_PIPELINE_PROVIDER=gemini-live`. Authenticates the Gemini Multimodal Live WebSocket session. Also used by the `gemini-api` runtime adapter (see `runtime.md`). |
92
120
  | `ANTHROPIC_API_KEY` | — | Enables the Anthropic REST adapter; when set and voice is enabled, voice auto-wires to the direct Messages API path (zero CLI cold-start). See `runtime.md § Anthropic REST Runtime`. |
93
121
  | *(built-in)* | — | Telegraphic style instruction hardcoded into every voice AI invocation — front-loads the answer, strips preambles/markdown/filler, keeps responses short for TTS latency. Not an env var; not overridable by `DISCOCLAW_VOICE_SYSTEM_PROMPT`. |
package/.env.example CHANGED
@@ -193,6 +193,12 @@ DISCORD_GUILD_ID=
193
193
  # Run `pnpm setup` or `discoclaw init` to enable voice interactively,
194
194
  # or set these vars manually to enable voice chat (STT/TTS via Deepgram).
195
195
  #DISCOCLAW_VOICE_ENABLED=0
196
+ # Voice pipeline provider: pipeline (default, Deepgram STT/TTS) or gemini-live
197
+ # (Gemini Live WebSocket — requires GEMINI_API_KEY).
198
+ #DISCOCLAW_VOICE_PIPELINE_PROVIDER=pipeline
199
+ # Gemini Live session rotation threshold (ms). The provider proactively reconnects
200
+ # before Gemini's ~15 min session limit to minimize audio gap. Default: 780000 (13 min).
201
+ #DISCOCLAW_GEMINI_SESSION_ROTATION_MS=780000
196
202
  # Text channel used for voice prompt context and actions (e.g. posting action results,
197
203
  # reading pinned notes). Required for full voice functionality when voice is enabled.
198
204
  #DISCOCLAW_VOICE_HOME_CHANNEL= # e.g. "voice"
@@ -146,7 +146,13 @@ function normalizeRuntimeName(value) {
146
146
  const trimmed = value?.trim().toLowerCase();
147
147
  if (!trimmed)
148
148
  return undefined;
149
- const normalized = trimmed === 'claude_code' ? 'claude' : trimmed;
149
+ let normalized = trimmed === 'claude_code' ? 'claude' : trimmed;
150
+ if (normalized === 'claude-cli')
151
+ normalized = 'claude';
152
+ if (normalized === 'codex-cli')
153
+ normalized = 'codex';
154
+ if (normalized === 'claude' || normalized === 'codex')
155
+ return normalized;
150
156
  return KNOWN_RUNTIMES.has(normalized) ? normalized : undefined;
151
157
  }
152
158
  function trimEnvValue(value) {
package/dist/config.js CHANGED
@@ -512,6 +512,8 @@ export function parseConfig(env) {
512
512
  const voiceAutoJoin = parseBoolean(env, 'DISCOCLAW_VOICE_AUTO_JOIN', false);
513
513
  const voiceSttProvider = parseEnum(env, 'DISCOCLAW_STT_PROVIDER', ['deepgram', 'whisper', 'openai'], 'deepgram');
514
514
  const voiceTtsProvider = parseEnum(env, 'DISCOCLAW_TTS_PROVIDER', ['cartesia', 'deepgram', 'kokoro', 'openai'], 'cartesia');
515
+ const voicePipelineProvider = parseEnum(env, 'DISCOCLAW_VOICE_PIPELINE_PROVIDER', ['pipeline', 'gemini-live'], 'pipeline');
516
+ const geminiSessionRotationMs = parseNonNegativeInt(env, 'DISCOCLAW_GEMINI_SESSION_ROTATION_MS', 780_000);
515
517
  let voiceHomeChannel = parseTrimmedString(env, 'DISCOCLAW_VOICE_HOME_CHANNEL');
516
518
  if (!voiceHomeChannel) {
517
519
  const legacy = parseTrimmedString(env, 'DISCOCLAW_VOICE_TRANSCRIPT_CHANNEL');
@@ -563,6 +565,9 @@ export function parseConfig(env) {
563
565
  if (voiceEnabled && !voiceHomeChannel) {
564
566
  warnings.push('DISCOCLAW_VOICE_ENABLED=1 but DISCOCLAW_VOICE_HOME_CHANNEL is not set; voice actions will be disabled (no target channel for action execution).');
565
567
  }
568
+ if (voiceEnabled && voicePipelineProvider === 'gemini-live' && !geminiApiKey) {
569
+ warnings.push('DISCOCLAW_VOICE_PIPELINE_PROVIDER=gemini-live but GEMINI_API_KEY is not set; voice pipeline will fail at runtime.');
570
+ }
566
571
  const coldStorageEnabled = parseBoolean(env, 'DISCOCLAW_COLD_STORAGE_ENABLED', false);
567
572
  const coldStorageApiKey = parseTrimmedString(env, 'COLD_STORAGE_API_KEY') ?? openaiApiKey;
568
573
  const coldStorageProvider = parseEnum(env, 'COLD_STORAGE_PROVIDER', ['openai', 'openai-compat'], 'openai');
@@ -743,6 +748,8 @@ export function parseConfig(env) {
743
748
  voiceSystemPrompt,
744
749
  voiceSttProvider,
745
750
  voiceTtsProvider,
751
+ voicePipelineProvider,
752
+ geminiSessionRotationMs,
746
753
  voiceHomeChannel,
747
754
  voiceLogChannel,
748
755
  deepgramApiKey,
@@ -1,3 +1,4 @@
1
+ import { randomUUID } from 'node:crypto';
1
2
  import { execa } from 'execa';
2
3
  import { resolveDefaultModel as resolveImagegenDefaultModel } from '../discord/actions-imagegen.js';
3
4
  import { acquireCronLock, releaseCronLock } from './job-lock.js';
@@ -9,6 +10,7 @@ import { sendChunks, appendUnavailableActionTypesNotice, appendParseFailureNotic
9
10
  import { buildPromptPreamble, loadWorkspacePaFiles, inlineContextFiles, resolveEffectiveTools } from '../discord/prompt-common.js';
10
11
  import { ensureStatusMessage } from './discord-sync.js';
11
12
  import { globalMetrics } from '../observability/metrics.js';
13
+ import { globalTraceStore } from '../observability/trace-store.js';
12
14
  import { mapRuntimeErrorToUserMessage } from '../discord/user-errors.js';
13
15
  import { resolveModel } from '../runtime/model-tiers.js';
14
16
  import { cliExecaEnv, stripAnsi } from '../runtime/cli-shared.js';
@@ -223,6 +225,10 @@ export async function executeCronJob(job, ctx) {
223
225
  return;
224
226
  }
225
227
  }
228
+ const traceId = `cron_${randomUUID()}`;
229
+ const sessionKey = `cron:${job.cronId || job.id}`;
230
+ let traceOutcome = 'success';
231
+ globalTraceStore.startTrace(traceId, sessionKey, 'cron', undefined);
226
232
  job.running = true;
227
233
  activeCronRunKeys.add(runKey);
228
234
  ctx.runControl?.register(job.id, requestCancel);
@@ -244,6 +250,13 @@ export async function executeCronJob(job, ctx) {
244
250
  const guild = ctx.client.guilds.cache.get(job.guildId);
245
251
  if (!guild) {
246
252
  ctx.log?.error({ jobId: job.id, guildId: job.guildId }, 'cron:exec guild not found');
253
+ traceOutcome = 'error';
254
+ globalTraceStore.addEvent(traceId, {
255
+ type: 'error',
256
+ at: Date.now(),
257
+ message: `guild ${job.guildId} not found`,
258
+ stage: 'cron_setup',
259
+ });
247
260
  await ctx.status?.runtimeError({ sessionKey: `cron:${job.id}` }, `Cron "${job.name}": guild ${job.guildId} not found`);
248
261
  await recordError(ctx, job, `guild ${job.guildId} not found`);
249
262
  return;
@@ -251,6 +264,13 @@ export async function executeCronJob(job, ctx) {
251
264
  const targetChannel = resolveChannel(guild, job.def.channel);
252
265
  if (!targetChannel) {
253
266
  ctx.log?.error({ jobId: job.id, channel: job.def.channel }, 'cron:exec target channel not found');
267
+ traceOutcome = 'error';
268
+ globalTraceStore.addEvent(traceId, {
269
+ type: 'error',
270
+ at: Date.now(),
271
+ message: `target channel "${job.def.channel}" not found`,
272
+ stage: 'cron_setup',
273
+ });
254
274
  await ctx.status?.runtimeError({ sessionKey: `cron:${job.id}`, channelName: job.def.channel }, `Cron "${job.name}": target channel "${job.def.channel}" not found`);
255
275
  await recordError(ctx, job, `target channel "${job.def.channel}" not found`);
256
276
  return;
@@ -264,6 +284,13 @@ export async function executeCronJob(job, ctx) {
264
284
  (parentId && ctx.allowChannelIds.has(parentId));
265
285
  if (!allowed) {
266
286
  ctx.log?.error({ jobId: job.id, channel: job.def.channel }, 'cron:exec target channel not allowlisted');
287
+ traceOutcome = 'error';
288
+ globalTraceStore.addEvent(traceId, {
289
+ type: 'error',
290
+ at: Date.now(),
291
+ message: `target channel "${job.def.channel}" not allowlisted`,
292
+ stage: 'cron_setup',
293
+ });
267
294
  await ctx.status?.runtimeError({ sessionKey: `cron:${job.id}`, channelName: job.def.channel }, `Cron "${job.name}": target channel "${job.def.channel}" is not allowlisted`);
268
295
  await recordError(ctx, job, `target channel "${job.def.channel}" not allowlisted`);
269
296
  return;
@@ -367,6 +394,12 @@ export async function executeCronJob(job, ctx) {
367
394
  }
368
395
  }
369
396
  metrics.recordInvokeStart('cron');
397
+ globalTraceStore.addEvent(traceId, {
398
+ type: 'invoke_start',
399
+ at: Date.now(),
400
+ summary: `cron job "${job.name}"`,
401
+ promptPreview: prompt.slice(0, 220),
402
+ });
370
403
  ctx.log?.info({ flow: 'cron', jobId: job.id, cronId: job.cronId }, 'obs.invoke.start');
371
404
  let finalText = '';
372
405
  let deltaText = '';
@@ -401,6 +434,13 @@ export async function executeCronJob(job, ctx) {
401
434
  collectedImages.push(evt.image);
402
435
  }
403
436
  else if (evt.type === 'error') {
437
+ traceOutcome = 'error';
438
+ globalTraceStore.addEvent(traceId, {
439
+ type: 'error',
440
+ at: Date.now(),
441
+ message: evt.message,
442
+ stage: 'runtime',
443
+ });
404
444
  metrics.recordInvokeResult('cron', Date.now() - t0, false, evt.message);
405
445
  metrics.increment('cron.run.error');
406
446
  ctx.log?.error({ jobId: job.id, error: evt.message }, 'cron:exec runtime error');
@@ -425,11 +465,24 @@ export async function executeCronJob(job, ctx) {
425
465
  if (runtimeIterator?.return) {
426
466
  await runtimeIterator.return();
427
467
  }
468
+ traceOutcome = 'canceled';
469
+ globalTraceStore.addEvent(traceId, {
470
+ type: 'error',
471
+ at: Date.now(),
472
+ message: cancelReason,
473
+ stage: 'runtime',
474
+ });
428
475
  metrics.increment('cron.run.canceled');
429
476
  ctx.log?.warn({ jobId: job.id, cronId: job.cronId }, 'cron:exec canceled');
430
477
  await recordError(ctx, job, cancelReason);
431
478
  return;
432
479
  }
480
+ globalTraceStore.addEvent(traceId, {
481
+ type: 'invoke_end',
482
+ at: Date.now(),
483
+ ok: true,
484
+ summary: `completed in ${Date.now() - t0}ms`,
485
+ });
433
486
  metrics.recordInvokeResult('cron', Date.now() - t0, true);
434
487
  ctx.log?.info({ flow: 'cron', jobId: job.id, ms: Date.now() - t0, ok: true }, 'obs.invoke.end');
435
488
  let output = finalText || deltaText;
@@ -518,8 +571,16 @@ export async function executeCronJob(job, ctx) {
518
571
  imagegenCtx: ctx.imagegenCtx,
519
572
  voiceCtx: ctx.voiceCtx,
520
573
  });
521
- for (const result of results) {
574
+ for (let i = 0; i < results.length; i++) {
575
+ const result = results[i];
522
576
  metrics.recordActionResult(result.ok);
577
+ globalTraceStore.addEvent(traceId, {
578
+ type: 'action_result',
579
+ at: Date.now(),
580
+ action: actions[i].type,
581
+ ok: result.ok,
582
+ detail: result.ok ? undefined : ('error' in result ? result.error : undefined),
583
+ });
523
584
  ctx.log?.info({ flow: 'cron', jobId: job.id, ok: result.ok }, 'obs.action.result');
524
585
  }
525
586
  const anyActionSucceeded = results.some((r) => r.ok);
@@ -604,6 +665,15 @@ export async function executeCronJob(job, ctx) {
604
665
  }
605
666
  catch (err) {
606
667
  const msg = err instanceof Error ? err.message : String(err);
668
+ traceOutcome = 'error';
669
+ globalTraceStore.addEvent(traceId, {
670
+ type: 'error',
671
+ at: Date.now(),
672
+ message: msg,
673
+ name: err instanceof Error ? err.name : undefined,
674
+ stage: 'cron_flow',
675
+ stack: err instanceof Error ? err.stack?.slice(0, 400) : undefined,
676
+ });
607
677
  metrics.increment('cron.run.error');
608
678
  ctx.log?.error({ err, jobId: job.id }, 'cron:exec failed');
609
679
  await ctx.status?.runtimeError({ sessionKey: `cron:${job.id}`, channelName: job.def.channel }, `Cron "${job.name}": ${msg}`);
@@ -623,6 +693,7 @@ export async function executeCronJob(job, ctx) {
623
693
  await recordError(ctx, job, msg);
624
694
  }
625
695
  finally {
696
+ globalTraceStore.endTrace(traceId, traceOutcome);
626
697
  const shouldRerun = queuedCronRerunKeys.delete(runKey);
627
698
  if (lockToken && ctx.lockDir && job.cronId) {
628
699
  await releaseCronLock(ctx.lockDir, job.cronId, lockToken).catch((err) => {
@@ -0,0 +1,7 @@
1
+ import { globalMetrics } from '../../observability/metrics.js';
2
+ export function buildMetricsResponse() {
3
+ return {
4
+ ok: true,
5
+ metrics: globalMetrics.snapshot(),
6
+ };
7
+ }
@@ -0,0 +1,16 @@
1
+ import { describe, expect, it } from 'vitest';
2
+ import { buildMetricsResponse } from './metrics.js';
3
+ describe('buildMetricsResponse', () => {
4
+ it('returns ok with a metrics snapshot', () => {
5
+ const response = buildMetricsResponse();
6
+ expect(response.ok).toBe(true);
7
+ expect(response.metrics).toBeDefined();
8
+ expect(typeof response.metrics.startedAt).toBe('number');
9
+ expect(response.metrics.counters).toBeDefined();
10
+ expect(response.metrics.latencies).toBeDefined();
11
+ expect(response.metrics.latencies).toHaveProperty('message');
12
+ expect(response.metrics.latencies).toHaveProperty('reaction');
13
+ expect(response.metrics.latencies).toHaveProperty('cron');
14
+ expect(response.metrics.latencies).toHaveProperty('defer');
15
+ });
16
+ });
@@ -0,0 +1,14 @@
1
+ import { globalTraceStore } from '../../observability/trace-store.js';
2
+ const DEFAULT_LIMIT = 50;
3
+ const MAX_LIMIT = 200;
4
+ export function buildTracesResponse(limitParam) {
5
+ const parsed = limitParam !== null ? Math.floor(Number(limitParam)) : DEFAULT_LIMIT;
6
+ const limit = Number.isFinite(parsed)
7
+ ? Math.max(1, Math.min(MAX_LIMIT, parsed))
8
+ : DEFAULT_LIMIT;
9
+ return {
10
+ ok: true,
11
+ summary: globalTraceStore.summary(),
12
+ recentTraces: globalTraceStore.listRecent(limit),
13
+ };
14
+ }
@@ -0,0 +1,40 @@
1
+ import { describe, expect, it } from 'vitest';
2
+ import { buildTracesResponse } from './traces.js';
3
+ import { globalTraceStore } from '../../observability/trace-store.js';
4
+ describe('buildTracesResponse', () => {
5
+ it('returns ok with summary and recent traces', () => {
6
+ const response = buildTracesResponse(null);
7
+ expect(response.ok).toBe(true);
8
+ expect(response.summary).toBeDefined();
9
+ expect(typeof response.summary.total).toBe('number');
10
+ expect(response.summary.byFlow).toBeDefined();
11
+ expect(Array.isArray(response.recentTraces)).toBe(true);
12
+ });
13
+ it('uses default limit of 50 when param is null', () => {
14
+ const response = buildTracesResponse(null);
15
+ expect(response.ok).toBe(true);
16
+ // With an empty store, recentTraces should be empty
17
+ expect(response.recentTraces.length).toBeLessThanOrEqual(50);
18
+ });
19
+ it('respects a custom limit param', () => {
20
+ // Seed a few traces
21
+ globalTraceStore.startTrace('t1', 'user:ch', 'message');
22
+ globalTraceStore.endTrace('t1', 'success');
23
+ globalTraceStore.startTrace('t2', 'user:ch', 'cron');
24
+ globalTraceStore.endTrace('t2', 'success');
25
+ globalTraceStore.startTrace('t3', 'user:ch', 'reaction');
26
+ globalTraceStore.endTrace('t3', 'success');
27
+ const response = buildTracesResponse('2');
28
+ expect(response.ok).toBe(true);
29
+ expect(response.recentTraces.length).toBeLessThanOrEqual(2);
30
+ });
31
+ it('clamps limit to max of 200', () => {
32
+ const response = buildTracesResponse('999');
33
+ expect(response.ok).toBe(true);
34
+ // Just verify it doesn't throw — the limit is clamped internally
35
+ });
36
+ it('falls back to default for non-numeric limit', () => {
37
+ const response = buildTracesResponse('abc');
38
+ expect(response.ok).toBe(true);
39
+ });
40
+ });
@@ -697,7 +697,7 @@ export function renderDashboardPage() {
697
697
  </label>
698
698
  </div>
699
699
  <div class="actions">
700
- <button id="chat-runtime-submit-btn" type="submit">Apply Runtime</button>
700
+ <button id="chat-runtime-submit-btn" type="submit">Apply Runtime + Save</button>
701
701
  <button id="chat-auth-btn" class="secondary" type="button">Check Auth</button>
702
702
  </div>
703
703
  </form>
@@ -709,9 +709,9 @@ export function renderDashboardPage() {
709
709
  <select id="chat-model-select" name="model" required></select>
710
710
  </label>
711
711
  </div>
712
- <div class="field-note">Tier options double as the practical thinking profile on runtimes that support explicit effort.</div>
712
+ <div class="field-note">Tier options double as the practical thinking profile on runtimes that support explicit effort. These chat controls also save the next-start default.</div>
713
713
  <div class="actions">
714
- <button id="chat-model-submit-btn" type="submit">Apply Model</button>
714
+ <button id="chat-model-submit-btn" type="submit">Apply Model + Save</button>
715
715
  </div>
716
716
  </form>
717
717
  </div>
@@ -839,6 +839,52 @@ export function renderDashboardPage() {
839
839
  </details>
840
840
  </section>
841
841
 
842
+ <section class="card span-12">
843
+ <div class="card-header">
844
+ <div>
845
+ <h2>Observability</h2>
846
+ </div>
847
+ <div class="actions">
848
+ <button id="traces-btn" class="secondary" type="button">Refresh Traces</button>
849
+ </div>
850
+ </div>
851
+ <div id="traces-summary" class="metrics"></div>
852
+ <details>
853
+ <summary>Runtime Metrics</summary>
854
+ <div class="details-body">
855
+ <div id="metrics-counters" class="metrics"></div>
856
+ <div id="metrics-latencies" class="metrics"></div>
857
+ <div id="metrics-memory" class="metrics"></div>
858
+ </div>
859
+ </details>
860
+ <details>
861
+ <summary>Recent Traces</summary>
862
+ <div class="details-body">
863
+ <div class="table-wrap">
864
+ <table>
865
+ <thead>
866
+ <tr>
867
+ <th>Flow</th>
868
+ <th>Outcome</th>
869
+ <th>Duration</th>
870
+ <th>Started</th>
871
+ <th>Events</th>
872
+ </tr>
873
+ </thead>
874
+ <tbody id="traces-body"></tbody>
875
+ </table>
876
+ </div>
877
+ </div>
878
+ </details>
879
+ <details>
880
+ <summary>Recent Errors</summary>
881
+ <div class="details-body">
882
+ <div id="traces-errors" class="checklist"></div>
883
+ </div>
884
+ </details>
885
+ <div id="traces-status" class="status"></div>
886
+ </section>
887
+
842
888
  <section class="card span-12">
843
889
  <div class="card-header">
844
890
  <div>
@@ -955,6 +1001,13 @@ export function renderDashboardPage() {
955
1001
  const secretValueInput = document.getElementById('secret-value-input');
956
1002
  const settingsContainer = document.getElementById('settings-container');
957
1003
  const settingsStatus = document.getElementById('settings-status');
1004
+ const tracesSummary = document.getElementById('traces-summary');
1005
+ const tracesBody = document.getElementById('traces-body');
1006
+ const tracesErrors = document.getElementById('traces-errors');
1007
+ const tracesStatus = document.getElementById('traces-status');
1008
+ const metricsCounters = document.getElementById('metrics-counters');
1009
+ const metricsLatencies = document.getElementById('metrics-latencies');
1010
+ const metricsMemory = document.getElementById('metrics-memory');
958
1011
  const ROLE_LABELS = {
959
1012
  chat: 'Chat',
960
1013
  'plan-run': 'Plan Run',
@@ -1265,7 +1318,9 @@ export function renderDashboardPage() {
1265
1318
  chatRuntimeSelect.value = live.chatRuntime || snapshot.primaryRuntime;
1266
1319
 
1267
1320
  clearNode(chatModelSelect);
1268
- (snapshot.modelOptions.chat || []).forEach(function (model) {
1321
+ (snapshot.modelOptions.chat || []).filter(function (model) {
1322
+ return model !== 'default';
1323
+ }).forEach(function (model) {
1269
1324
  appendSelectOption(chatModelSelect, model, formatModelOptionLabel('chat', model));
1270
1325
  });
1271
1326
  if ((snapshot.modelOptions.chat || []).indexOf(live.chatModel) >= 0) {
@@ -1303,6 +1358,121 @@ export function renderDashboardPage() {
1303
1358
  secretKeySelect.value = recommendSecretKey(snapshot);
1304
1359
  }
1305
1360
 
1361
+ function renderTraces(data) {
1362
+ var summary = data.summary || {};
1363
+ var recentTraces = data.recentTraces || [];
1364
+ var byFlow = summary.byFlow || {};
1365
+ clearNode(tracesSummary);
1366
+ appendMetric(tracesSummary, 'total traces', String(summary.total || 0));
1367
+ appendMetric(tracesSummary, 'in progress', String(summary.inProgress || 0));
1368
+ var flows = ['message', 'reaction', 'cron', 'defer'];
1369
+ flows.forEach(function (flow) {
1370
+ var fs = byFlow[flow];
1371
+ if (!fs || fs.total === 0) return;
1372
+ var avg = fs.avgDurationMs > 0 ? ' avg ' + fs.avgDurationMs + 'ms' : '';
1373
+ appendMetric(tracesSummary, flow, fs.succeeded + ' ok / ' + fs.failed + ' err / ' + fs.inProgress + ' running' + avg);
1374
+ });
1375
+
1376
+ clearNode(tracesBody);
1377
+ recentTraces.forEach(function (trace) {
1378
+ var tr = document.createElement('tr');
1379
+ var flowCell = document.createElement('td');
1380
+ flowCell.textContent = trace.flow;
1381
+ var outcomeCell = document.createElement('td');
1382
+ outcomeCell.textContent = trace.outcome;
1383
+ if (trace.outcome === 'success') outcomeCell.style.color = 'var(--green)';
1384
+ else if (trace.outcome === 'in_progress') outcomeCell.style.color = 'var(--amber)';
1385
+ else if (trace.outcome !== 'success') outcomeCell.style.color = 'var(--red)';
1386
+ var durationCell = document.createElement('td');
1387
+ durationCell.textContent = trace.outcome === 'in_progress' ? '\u2014' : trace.durationMs + 'ms';
1388
+ var startedCell = document.createElement('td');
1389
+ startedCell.textContent = new Date(trace.startedAt).toLocaleTimeString();
1390
+ var eventsCell = document.createElement('td');
1391
+ eventsCell.textContent = String((trace.events || []).length);
1392
+ tr.append(flowCell, outcomeCell, durationCell, startedCell, eventsCell);
1393
+ tracesBody.append(tr);
1394
+ });
1395
+
1396
+ clearNode(tracesErrors);
1397
+ var recentErrors = summary.recentErrors || [];
1398
+ if (recentErrors.length === 0) {
1399
+ var noErrors = document.createElement('div');
1400
+ noErrors.className = 'card-copy';
1401
+ noErrors.textContent = 'No recent errors.';
1402
+ tracesErrors.append(noErrors);
1403
+ } else {
1404
+ recentErrors.forEach(function (err) {
1405
+ var item = document.createElement('div');
1406
+ item.className = 'checklist-item';
1407
+ var top = document.createElement('div');
1408
+ top.className = 'checklist-top';
1409
+ var dot = document.createElement('div');
1410
+ dot.className = 'status-dot error';
1411
+ var label = document.createElement('div');
1412
+ label.className = 'checklist-label';
1413
+ label.textContent = err.flow + ': ' + err.message;
1414
+ top.append(dot, label);
1415
+ var body = document.createElement('div');
1416
+ body.className = 'checklist-body';
1417
+ body.textContent = new Date(err.at).toLocaleString();
1418
+ item.append(top, body);
1419
+ tracesErrors.append(item);
1420
+ });
1421
+ }
1422
+ }
1423
+
1424
+ async function refreshTraces() {
1425
+ var response = await fetchJson('/api/traces');
1426
+ renderTraces(response);
1427
+ }
1428
+
1429
+ function renderMetrics(data) {
1430
+ var m = data.metrics || {};
1431
+ var counters = m.counters || {};
1432
+ var latencies = m.latencies || {};
1433
+ var memory = m.memory;
1434
+
1435
+ clearNode(metricsCounters);
1436
+ var upSince = m.startedAt ? new Date(m.startedAt).toLocaleString() : 'unknown';
1437
+ appendMetric(metricsCounters, 'up since', upSince);
1438
+ var counterKeys = Object.keys(counters).sort();
1439
+ counterKeys.forEach(function (key) {
1440
+ appendMetric(metricsCounters, key, String(counters[key]));
1441
+ });
1442
+ if (counterKeys.length === 0) {
1443
+ appendMetric(metricsCounters, 'counters', 'none recorded yet');
1444
+ }
1445
+
1446
+ clearNode(metricsLatencies);
1447
+ var flows = ['message', 'reaction', 'cron', 'defer'];
1448
+ flows.forEach(function (flow) {
1449
+ var lat = latencies[flow];
1450
+ if (!lat || lat.count === 0) return;
1451
+ appendMetric(metricsLatencies, flow + ' latency',
1452
+ 'p50=' + lat.p50Ms + 'ms p95=' + lat.p95Ms + 'ms max=' + lat.maxMs + 'ms (n=' + lat.count + ')');
1453
+ });
1454
+ if (metricsLatencies.children.length === 0) {
1455
+ appendMetric(metricsLatencies, 'latencies', 'no samples yet');
1456
+ }
1457
+
1458
+ clearNode(metricsMemory);
1459
+ if (memory) {
1460
+ function fmtMB(bytes) { return bytes ? (bytes / 1048576).toFixed(1) + ' MB' : 'n/a'; }
1461
+ appendMetric(metricsMemory, 'rss', fmtMB(memory.rssBytes) + ' (hwm ' + fmtMB(memory.rssHwmBytes) + ')');
1462
+ appendMetric(metricsMemory, 'heap used', fmtMB(memory.heapUsedBytes) + ' (hwm ' + fmtMB(memory.heapUsedHwmBytes) + ')');
1463
+ appendMetric(metricsMemory, 'heap total', fmtMB(memory.heapTotalBytes));
1464
+ appendMetric(metricsMemory, 'external', fmtMB(memory.externalBytes));
1465
+ appendMetric(metricsMemory, 'samples', String(memory.sampleCount || 0));
1466
+ } else {
1467
+ appendMetric(metricsMemory, 'memory', 'sampler not active');
1468
+ }
1469
+ }
1470
+
1471
+ async function refreshMetrics() {
1472
+ var response = await fetchJson('/api/metrics');
1473
+ renderMetrics(response);
1474
+ }
1475
+
1306
1476
  function renderSnapshot(snapshot) {
1307
1477
  if (!snapshot.live) snapshot.live = {};
1308
1478
  const selectedRole = roleSelect.value;
@@ -1560,13 +1730,22 @@ export function renderDashboardPage() {
1560
1730
 
1561
1731
  document.getElementById('refresh-btn').addEventListener('click', async function () {
1562
1732
  try {
1563
- await Promise.all([refreshSnapshot(false), refreshDoctor(false)]);
1733
+ await Promise.all([refreshSnapshot(false), refreshDoctor(false), refreshTraces(), refreshMetrics()]);
1564
1734
  setStatus(heroStatus, 'Dashboard refreshed.', 'ok');
1565
1735
  } catch (error) {
1566
1736
  setStatus(heroStatus, String(error), 'error');
1567
1737
  }
1568
1738
  });
1569
1739
 
1740
+ document.getElementById('traces-btn').addEventListener('click', async function () {
1741
+ try {
1742
+ await Promise.all([refreshTraces(), refreshMetrics()]);
1743
+ setStatus(tracesStatus, 'Traces refreshed.', 'ok');
1744
+ } catch (error) {
1745
+ setStatus(tracesStatus, String(error), 'error');
1746
+ }
1747
+ });
1748
+
1570
1749
  document.getElementById('status-btn').addEventListener('click', async function () {
1571
1750
  try {
1572
1751
  const response = await fetchJson('/api/status');
@@ -1639,7 +1818,7 @@ export function renderDashboardPage() {
1639
1818
  const response = await fetchJson('/api/live-model', {
1640
1819
  method: 'POST',
1641
1820
  headers: { 'Content-Type': 'application/json' },
1642
- body: JSON.stringify({ role: 'chat', model: chatRuntimeSelect.value })
1821
+ body: JSON.stringify({ role: 'chat', model: chatRuntimeSelect.value, persist: true })
1643
1822
  });
1644
1823
  renderSnapshot(response.snapshot);
1645
1824
  setStatus(chatStatus, response.message, 'ok');
@@ -1655,7 +1834,7 @@ export function renderDashboardPage() {
1655
1834
  const response = await fetchJson('/api/live-model', {
1656
1835
  method: 'POST',
1657
1836
  headers: { 'Content-Type': 'application/json' },
1658
- body: JSON.stringify({ role: 'chat', model: chatModelSelect.value })
1837
+ body: JSON.stringify({ role: 'chat', model: chatModelSelect.value, persist: true })
1659
1838
  });
1660
1839
  renderSnapshot(response.snapshot);
1661
1840
  setStatus(chatStatus, response.message, 'ok');
@@ -1762,7 +1941,7 @@ export function renderDashboardPage() {
1762
1941
  syncSecondaryModelOptions(roleSelect.value, '');
1763
1942
  });
1764
1943
 
1765
- Promise.all([refreshSnapshot(false), refreshDoctor(false), loadSettings()]).then(function () {
1944
+ Promise.all([refreshSnapshot(false), refreshDoctor(false), loadSettings(), refreshTraces(), refreshMetrics()]).then(function () {
1766
1945
  setStatus(heroStatus, 'Dashboard ready.', 'ok');
1767
1946
  if (lastSnapshot) {
1768
1947
  populateSecondaryRoleForm('', '');