@vellumai/assistant 0.12.5 → 0.12.6-dev.202609292116.41f7caa
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.env.example +2 -0
- package/ARCHITECTURE.md +26 -3
- package/docs/automatic-done.md +32 -0
- package/docs/desktop-browser-cli.md +3 -2
- package/node_modules/@vellumai/ces-client/node_modules/@vellumai/service-contracts/package.json +2 -1
- package/node_modules/@vellumai/ces-client/node_modules/@vellumai/service-contracts/src/__tests__/mcp-oauth.test.ts +53 -0
- package/node_modules/@vellumai/ces-client/node_modules/@vellumai/service-contracts/src/index.ts +1 -0
- package/node_modules/@vellumai/ces-client/node_modules/@vellumai/service-contracts/src/mcp-oauth.ts +60 -0
- package/node_modules/@vellumai/environments/src/shell.test.ts +13 -1
- package/node_modules/@vellumai/environments/src/shell.ts +22 -1
- package/node_modules/@vellumai/gateway-client/node_modules/@vellumai/service-contracts/package.json +2 -1
- package/node_modules/@vellumai/gateway-client/node_modules/@vellumai/service-contracts/src/__tests__/mcp-oauth.test.ts +53 -0
- package/node_modules/@vellumai/gateway-client/node_modules/@vellumai/service-contracts/src/index.ts +1 -0
- package/node_modules/@vellumai/gateway-client/node_modules/@vellumai/service-contracts/src/mcp-oauth.ts +60 -0
- package/node_modules/@vellumai/gateway-client/src/__tests__/status-contracts.test.ts +39 -0
- package/node_modules/@vellumai/gateway-client/src/gateway-ipc-contracts.ts +25 -0
- package/node_modules/@vellumai/service-contracts/package.json +2 -1
- package/node_modules/@vellumai/service-contracts/src/__tests__/mcp-oauth.test.ts +53 -0
- package/node_modules/@vellumai/service-contracts/src/index.ts +1 -0
- package/node_modules/@vellumai/service-contracts/src/mcp-oauth.ts +60 -0
- package/openapi.yaml +270 -10
- package/package.json +1 -1
- package/src/__tests__/agent-loop.test.ts +76 -0
- package/src/__tests__/anthropic-provider.test.ts +2 -0
- package/src/__tests__/byok-default-profile-ensure.test.ts +7 -0
- package/src/__tests__/config-get-vision-flag.test.ts +1 -1
- package/src/__tests__/config-loader-validation-reset-notice.test.ts +2 -2
- package/src/__tests__/conversation-agent-loop-overflow.test.ts +160 -154
- package/src/__tests__/conversation-evictor.test.ts +54 -0
- package/src/__tests__/conversation-pairing.test.ts +112 -0
- package/src/__tests__/document-append-idempotency.test.ts +13 -0
- package/src/__tests__/document-create-dedupe.test.ts +13 -0
- package/src/__tests__/document-find-replace.test.ts +13 -0
- package/src/__tests__/document-revision-history.test.ts +390 -0
- package/src/__tests__/document-revision.test.ts +425 -0
- package/src/__tests__/document-tool-security.test.ts +13 -0
- package/src/__tests__/document-update-default-surface.test.ts +13 -0
- package/src/__tests__/emit-signal-routing-intent.test.ts +270 -0
- package/src/__tests__/gateway-flag-listener.test.ts +59 -6
- package/src/__tests__/headless-browser-navigate.test.ts +20 -30
- package/src/__tests__/host-cu-proxy.test.ts +79 -0
- package/src/__tests__/host-cu-routes-targeted.test.ts +14 -5
- package/src/__tests__/host-shell-tool.test.ts +6 -2
- package/src/__tests__/identity-routes.test.ts +4 -3
- package/src/__tests__/list-messages-client-message-id.test.ts +43 -0
- package/src/__tests__/llm-catalog-parity.test.ts +4 -4
- package/src/__tests__/llm-schema.test.ts +1 -1
- package/src/__tests__/notification-platform-adapter.test.ts +83 -1
- package/src/__tests__/oauth2-gateway-transport.test.ts +2 -0
- package/src/__tests__/openai-provider.test.ts +38 -11
- package/src/__tests__/pricing.test.ts +11 -0
- package/src/__tests__/public-ingress-urls.test.ts +26 -0
- package/src/__tests__/retry-thinking-tool-choice.test.ts +47 -0
- package/src/__tests__/skill-execute-input.test.ts +11 -0
- package/src/__tests__/tool-approval-handler.test.ts +4 -0
- package/src/__tests__/tool-preview-lifecycle.test.ts +26 -0
- package/src/__tests__/usage-attribution.test.ts +31 -0
- package/src/__tests__/workspace-migration-133-collapse-provider-connections.test.ts +3 -3
- package/src/__tests__/workspace-migration-160-repair-retired-fireworks-model-ids.test.ts +262 -0
- package/src/__tests__/workspace-migration-161-repair-retired-fireworks-glm-5p2-model-id.test.ts +233 -0
- package/src/acp/agent-process.ts +6 -1
- package/src/acp/client-handler.ts +6 -1
- package/src/agent/loop-unparseable-tool-retry.test.ts +333 -0
- package/src/agent/loop.ts +161 -17
- package/src/api/events/document-editor-show.ts +5 -0
- package/src/api/events/document-editor-update.ts +10 -0
- package/src/api/events/tool-result.ts +1 -0
- package/src/calls/__tests__/call-controller.test.ts +273 -1
- package/src/calls/__tests__/media-stream-server-integration.test.ts +40 -0
- package/src/calls/__tests__/media-stream-stt-session.test.ts +33 -0
- package/src/calls/__tests__/voice-session-bridge.test.ts +54 -0
- package/src/calls/call-constants.ts +10 -0
- package/src/calls/call-controller.ts +359 -25
- package/src/calls/channel-admission-reader.ts +15 -9
- package/src/calls/media-stream-server.ts +18 -0
- package/src/calls/media-stream-stt-session.ts +30 -9
- package/src/calls/twilio-provider.ts +14 -31
- package/src/calls/voice-session-bridge.ts +18 -1
- package/src/cli/commands/browser.ts +1 -2
- package/src/cli/lib/__tests__/plugin-artifact.test.ts +23 -0
- package/src/cli/lib/__tests__/plugin-catalog-visibility.test.ts +2 -0
- package/src/cli/lib/__tests__/plugin-details.test.ts +16 -0
- package/src/cli/lib/bundled-marketplace.json +54 -2
- package/src/cli/lib/inspect-plugin.ts +15 -15
- package/src/cli/lib/list-installed-plugins.ts +29 -28
- package/src/cli/lib/plugin-artifact.ts +74 -63
- package/src/cli/lib/plugin-catalog-visibility.ts +1 -0
- package/src/cli/lib/plugin-details.ts +27 -42
- package/src/cli/lib/plugin-marketplace.ts +6 -8
- package/src/cli/lib/plugin-surfaces.ts +8 -7
- package/src/cli.ts +1 -3
- package/src/config/__tests__/chat-settings.test.ts +121 -0
- package/src/config/assistant-feature-flags.ts +28 -23
- package/src/config/bundled-skills/computer-use/SKILL.md +20 -0
- package/src/config/bundled-skills/computer-use/TOOLS.json +11 -11
- package/src/config/chat-settings.ts +100 -0
- package/src/config/default-profile-catalog.ts +1 -1
- package/src/config/feature-flag-cache.ts +7 -25
- package/src/config/feature-flag-registry.json +9 -9
- package/src/config/loader.ts +5 -1
- package/src/config/profile-text-generation.ts +25 -21
- package/src/config/schemas/conversations.ts +14 -0
- package/src/config/schemas/llm.ts +1 -0
- package/src/config/schemas/notifications.ts +8 -1
- package/src/contacts/gateway-channel-read.ts +1 -1
- package/src/conversations/__tests__/auto-archive-activity.test.ts +136 -0
- package/src/conversations/__tests__/auto-archive.test.ts +304 -0
- package/src/conversations/auto-archive-activity.ts +61 -0
- package/src/conversations/auto-archive.ts +201 -0
- package/src/daemon/computer-use-observation.test.ts +81 -0
- package/src/daemon/computer-use-observation.ts +23 -0
- package/src/daemon/conversation-agent-loop-handlers.ts +18 -2
- package/src/daemon/conversation-agent-loop.ts +14 -6
- package/src/daemon/conversation-evictor.ts +50 -16
- package/src/daemon/conversation-messaging.ts +4 -0
- package/src/daemon/conversation-process.ts +15 -0
- package/src/daemon/conversation-queue-manager.ts +2 -0
- package/src/daemon/conversation.ts +2 -0
- package/src/daemon/doordash-steps.ts +1 -3
- package/src/daemon/handlers/__tests__/config-a2a-accept.test.ts +43 -1
- package/src/daemon/handlers/__tests__/config-channels.test.ts +1 -1
- package/src/daemon/handlers/config-a2a.ts +75 -2
- package/src/daemon/handlers/shared.ts +13 -2
- package/src/daemon/host-cu-proxy.ts +34 -3
- package/src/daemon/lifecycle.ts +15 -2
- package/src/daemon/process-message.ts +5 -0
- package/src/daemon/shutdown-handlers.ts +3 -0
- package/src/daemon/turn-finalization.ts +4 -0
- package/src/desktop/desktop-browser-endpoint.ts +11 -2
- package/src/desktop/desktop-chrome-policy.test.ts +1 -0
- package/src/desktop/desktop-chrome-policy.ts +7 -1
- package/src/desktop/desktop-computer-use.test.ts +14 -4
- package/src/desktop/desktop-session-manager.ts +5 -3
- package/src/documents/document-revisions-store.ts +153 -0
- package/src/documents/document-store.ts +471 -133
- package/src/followups/followup-store.ts +0 -4
- package/src/i18n/__tests__/i18n.test.ts +25 -0
- package/src/i18n/assistant-text.ts +18 -0
- package/src/i18n/index.ts +1 -0
- package/src/i18n/messages.ts +11 -0
- package/src/inbound/__tests__/oauth-callback-url.test.ts +25 -2
- package/src/inbound/oauth-callback-url.ts +30 -1
- package/src/inbound/public-ingress-urls.ts +12 -2
- package/src/ipc/gateway-client.test.ts +40 -0
- package/src/ipc/gateway-client.ts +12 -29
- package/src/ipc/gateway-flag-listener.ts +7 -10
- package/src/live-voice/__tests__/live-voice-agent-turn.test.ts +71 -0
- package/src/live-voice/__tests__/live-voice-flux-turn-end.test.ts +83 -0
- package/src/live-voice/__tests__/protocol.test.ts +77 -0
- package/src/live-voice/__tests__/share-targets.test.ts +163 -0
- package/src/live-voice/live-voice-session.ts +23 -6
- package/src/live-voice/protocol.ts +22 -0
- package/src/live-voice/share-targets.ts +225 -0
- package/src/mcp/__tests__/mcp-oauth-client-registration.test.ts +144 -3
- package/src/mcp/__tests__/mcp-oauth-provider.test.ts +2 -0
- package/src/mcp/client.ts +1 -19
- package/src/mcp/mcp-oauth-provider.ts +66 -49
- package/src/media/openrouter-image-service.ts +39 -22
- package/src/monitoring/__tests__/oom-inheritance-sweep.test.ts +86 -0
- package/src/monitoring/__tests__/oom-kill-reporter.test.ts +374 -0
- package/src/monitoring/control.ts +2 -1
- package/src/monitoring/oom-inheritance-sweep.ts +96 -0
- package/src/monitoring/oom-kill-reporter.ts +469 -0
- package/src/monitoring/resource-sampler.ts +30 -0
- package/src/notifications/README.md +53 -4
- package/src/notifications/__tests__/assistant-initiated-thread.test.ts +254 -0
- package/src/notifications/__tests__/broadcaster.test.ts +558 -6
- package/src/notifications/__tests__/completion-policy.test.ts +37 -0
- package/src/notifications/__tests__/from-me-alert-integration.test.ts +192 -0
- package/src/notifications/adapters/platform.ts +14 -0
- package/src/notifications/assistant-initiated-thread.ts +101 -0
- package/src/notifications/broadcaster.ts +94 -16
- package/src/notifications/chat-reply-policy.ts +11 -0
- package/src/notifications/completion-policy.ts +12 -2
- package/src/notifications/conversation-pairing.ts +15 -70
- package/src/notifications/emit-signal.ts +34 -3
- package/src/notifications/types.ts +2 -0
- package/src/oauth/connect-orchestrator.ts +1 -2
- package/src/persistence/__tests__/conversation-auto-archive.test.ts +330 -0
- package/src/persistence/conversation-auto-archive.ts +170 -0
- package/src/persistence/conversation-crud.ts +2 -1
- package/src/persistence/conversation-queries.ts +15 -8
- package/src/persistence/embeddings/embed.test.ts +96 -0
- package/src/persistence/embeddings/embed.ts +20 -2
- package/src/persistence/embeddings/embedding-gemini.test.ts +39 -3
- package/src/persistence/embeddings/embedding-gemini.ts +39 -1
- package/src/persistence/embeddings/embedding-local.ts +6 -0
- package/src/persistence/embeddings/qdrant-manager.ts +5 -0
- package/src/persistence/job-utils.ts +10 -0
- package/src/persistence/migrations/384-add-document-revision.test.ts +83 -0
- package/src/persistence/migrations/384-add-document-revision.ts +21 -0
- package/src/persistence/migrations/385-create-document-revisions.test.ts +67 -0
- package/src/persistence/migrations/385-create-document-revisions.ts +28 -0
- package/src/persistence/migrations/386-conversations-last-reopened-at.ts +10 -0
- package/src/persistence/migrations/__tests__/386-conversations-last-reopened-at.test.ts +32 -0
- package/src/persistence/schema/conversations.ts +1 -0
- package/src/persistence/schema/documents.ts +20 -0
- package/src/persistence/steps.ts +6 -0
- package/src/platform/managed-speech.ts +42 -46
- package/src/plugin-api/vision-support.test.ts +21 -21
- package/src/plugins/defaults/empty-response/hooks/post-model-call.ts +11 -6
- package/src/providers/__tests__/dispatch-connection-routing.test.ts +1 -1
- package/src/providers/__tests__/search-provider-catalog.test.ts +8 -0
- package/src/providers/__tests__/unparseable-tool-args.test.ts +25 -0
- package/src/providers/anthropic/client.ts +1 -1
- package/src/providers/fireworks/client.test.ts +168 -0
- package/src/providers/fireworks/client.ts +24 -0
- package/src/providers/inference/adapter-factory.ts +6 -0
- package/src/providers/ionet/client.test.ts +143 -0
- package/src/providers/ionet/client.ts +37 -0
- package/src/providers/model-catalog.ts +141 -84
- package/src/providers/model-intents.ts +4 -4
- package/src/providers/openai/chat-completions-provider.ts +33 -2
- package/src/providers/retry.ts +56 -4
- package/src/providers/search-provider-catalog.ts +10 -0
- package/src/providers/types.ts +10 -2
- package/src/providers/unparseable-tool-args.ts +29 -5
- package/src/providers/usage-tracking.ts +5 -3
- package/src/runtime/assistant-event-hub.ts +23 -0
- package/src/runtime/auth/__tests__/guard-tests.test.ts +0 -7
- package/src/runtime/auth/__tests__/route-policy.test.ts +1 -1
- package/src/runtime/channel-reply-delivery.ts +2 -0
- package/src/runtime/channel-retry-sweep.ts +1 -2
- package/src/runtime/http-types.ts +3 -0
- package/src/runtime/migrations/__tests__/vbundle-streaming-validator.test.ts +35 -10
- package/src/runtime/migrations/__tests__/vbundle-validator-v1-schema.test.ts +48 -0
- package/src/runtime/migrations/vbundle-builder.ts +19 -3
- package/src/runtime/migrations/vbundle-streaming-importer.ts +14 -19
- package/src/runtime/migrations/vbundle-streaming-validator.ts +14 -7
- package/src/runtime/migrations/vbundle-validator.ts +54 -1
- package/src/runtime/pending-interactions.ts +20 -2
- package/src/runtime/routes/__tests__/contact-routes.test.ts +5 -3
- package/src/runtime/routes/__tests__/conversation-list-routes.test.ts +36 -0
- package/src/runtime/routes/__tests__/conversation-query-routes.test.ts +375 -2
- package/src/runtime/routes/__tests__/inference-send-routes.test.ts +1 -1
- package/src/runtime/routes/__tests__/user-route-dispatcher.test.ts +12 -0
- package/src/runtime/routes/contact-routes.ts +2 -2
- package/src/runtime/routes/conversation-list-routes.ts +10 -2
- package/src/runtime/routes/conversation-query-routes.ts +48 -10
- package/src/runtime/routes/conversation-routes.ts +7 -0
- package/src/runtime/routes/documents-routes.ts +185 -47
- package/src/runtime/routes/events-routes.ts +1 -3
- package/src/runtime/routes/host-app-control-routes.ts +1 -0
- package/src/runtime/routes/host-bash-routes.ts +1 -0
- package/src/runtime/routes/host-browser-routes.ts +15 -0
- package/src/runtime/routes/host-cu-routes.ts +4 -1
- package/src/runtime/routes/host-file-routes.ts +1 -0
- package/src/runtime/routes/host-proxy-result-binding.test.ts +83 -0
- package/src/runtime/routes/host-proxy-result-binding.ts +25 -0
- package/src/runtime/routes/host-transfer-routes.ts +1 -0
- package/src/runtime/routes/identity-routes.ts +3 -3
- package/src/runtime/routes/inbound-stages/background-dispatch.ts +4 -0
- package/src/runtime/routes/integrations/a2a.ts +4 -1
- package/src/runtime/routes/migration-routes.ts +3 -1
- package/src/runtime/routes/notification-routes.ts +1 -2
- package/src/runtime/routes/ui-snapshot-routes.test.ts +1 -0
- package/src/runtime/routes/user-route-dispatcher.ts +5 -1
- package/src/runtime/verification-templates.ts +1 -2
- package/src/sequence/analytics.ts +0 -4
- package/src/sequence/engine.ts +1 -11
- package/src/sequence/guardrails.ts +1 -6
- package/src/sequence/importer.ts +0 -8
- package/src/sequence/store.ts +0 -10
- package/src/signals/user-message.ts +0 -6
- package/src/skills/catalog-install.ts +52 -23
- package/src/tools/browser/browser-execution.ts +20 -23
- package/src/tools/browser/cdp-client/cdp-inspect/ws-transport.ts +1 -3
- package/src/tools/document/document-tool.ts +13 -11
- package/src/tools/executor.ts +1 -2
- package/src/tools/host-terminal/host-shell.ts +4 -2
- package/src/tools/network/__tests__/web-search.test.ts +105 -0
- package/src/tools/network/web-fetch.ts +2 -4
- package/src/tools/network/web-search.ts +239 -12
- package/src/tools/shared/filesystem/format-diff.ts +1 -2
- package/src/tools/shared/shell-output.test.ts +42 -0
- package/src/tools/shared/shell-output.ts +27 -4
- package/src/tools/skills/execute.ts +10 -7
- package/src/tools/terminal/shell.ts +3 -3
- package/src/tools/tool-approval-handler.ts +9 -1
- package/src/tools/ui-surface/visual-validation.ts +1 -2
- package/src/usage/attribution.ts +18 -1
- package/src/util/__tests__/oom-priority.test.ts +55 -0
- package/src/util/__tests__/worker-memory.test.ts +16 -0
- package/src/util/debounce.ts +0 -4
- package/src/util/oom-priority.ts +79 -0
- package/src/util/platform.ts +0 -2
- package/src/util/plugin-manifest.ts +5 -7
- package/src/util/pricing.ts +25 -0
- package/src/util/worker-memory.ts +9 -3
- package/src/util/worker-process.ts +8 -0
- package/src/workspace/byok-default-profile-ensure.ts +11 -0
- package/src/workspace/hatched-date.ts +11 -4
- package/src/workspace/migrations/160-repair-retired-fireworks-model-ids.ts +209 -0
- package/src/workspace/migrations/161-repair-retired-fireworks-glm-5p2-model-id.ts +197 -0
- package/src/workspace/migrations/registry.ts +4 -0
|
@@ -0,0 +1,469 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Reports kernel OOM kills in this container as `oom_kill` watchdog events.
|
|
3
|
+
*
|
|
4
|
+
* The log comes from `dmesg`, not `/dev/kmsg`: the device is not mounted in
|
|
5
|
+
* the container, while `dmesg` uses the syslog syscall the container is
|
|
6
|
+
* allowed. The log is per kernel, not per container. A kill is this
|
|
7
|
+
* container's when the kernel's `oom-kill:` header names a cgroup carrying
|
|
8
|
+
* this container's id, which the container learns from its own mount table
|
|
9
|
+
* since `/proc/self/cgroup` reads `/` inside a cgroup namespace. The kernel
|
|
10
|
+
* rate-limits that header in a burst, so header-less kills fall back to the
|
|
11
|
+
* cgroup `memory.events` counter.
|
|
12
|
+
*/
|
|
13
|
+
|
|
14
|
+
import { mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
|
15
|
+
import { dirname, join } from "node:path";
|
|
16
|
+
|
|
17
|
+
import { getRawShareAnalytics } from "../platform/consent-cache.js";
|
|
18
|
+
import { recordWatchdogEvent } from "../telemetry/watchdog-events-store.js";
|
|
19
|
+
import { getContainerMemoryEvents } from "../util/cgroup-memory.js";
|
|
20
|
+
import { getLogger } from "../util/logger.js";
|
|
21
|
+
import { DAEMON_OOM_SCORE_ADJ } from "../util/oom-priority.js";
|
|
22
|
+
import { getMonitoringDataDir } from "../util/platform.js";
|
|
23
|
+
import type { ResourceSample } from "./resource-sample-types.js";
|
|
24
|
+
|
|
25
|
+
const log = getLogger("oom-kill-reporter");
|
|
26
|
+
|
|
27
|
+
export const OOM_KILL_CHECK_NAME = "oom_kill";
|
|
28
|
+
const CURSOR_FILENAME = "oom-kill-cursor.json";
|
|
29
|
+
/** Cadence for hosts without `memory.events`. */
|
|
30
|
+
const FALLBACK_SCAN_INTERVAL_MS = 60_000;
|
|
31
|
+
/** The detail bag is capped at 4 KiB server-side. */
|
|
32
|
+
const MAX_VICTIMS_IN_DETAIL = 10;
|
|
33
|
+
const DMESG_TIMEOUT_MS = 5_000;
|
|
34
|
+
/** Mounts whose source path the container runtime names after the container. */
|
|
35
|
+
const CONTAINER_ID_MOUNTPOINTS = new Set([
|
|
36
|
+
"/etc/hosts",
|
|
37
|
+
"/etc/hostname",
|
|
38
|
+
"/etc/resolv.conf",
|
|
39
|
+
"/dev/termination-log",
|
|
40
|
+
]);
|
|
41
|
+
|
|
42
|
+
export interface KernelLogEntry {
|
|
43
|
+
/** Seconds since boot, monotonic for the life of the kernel. */
|
|
44
|
+
time: number;
|
|
45
|
+
msg: string;
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
export interface OomKill {
|
|
49
|
+
time: number;
|
|
50
|
+
pid: number;
|
|
51
|
+
comm: string;
|
|
52
|
+
oomScoreAdj: number | null;
|
|
53
|
+
anonRssKb: number | null;
|
|
54
|
+
totalVmKb: number | null;
|
|
55
|
+
/** Victim's cgroup from the `oom-kill:` header; null when the header was rate-limited away. */
|
|
56
|
+
memcg: string | null;
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
type Attribution = "cgroup" | "cgroup_counter" | "kernel_log";
|
|
60
|
+
|
|
61
|
+
/** `dmesg --json`: `{"dmesg":[{"pri":3,"time":44170.542,"msg":"..."}]}`. */
|
|
62
|
+
export function parseDmesgJson(text: string): KernelLogEntry[] | null {
|
|
63
|
+
try {
|
|
64
|
+
const parsed: unknown = JSON.parse(text);
|
|
65
|
+
const rows = (parsed as { dmesg?: unknown }).dmesg;
|
|
66
|
+
if (!Array.isArray(rows)) {
|
|
67
|
+
return null;
|
|
68
|
+
}
|
|
69
|
+
const entries: KernelLogEntry[] = [];
|
|
70
|
+
for (const row of rows) {
|
|
71
|
+
const { time, msg } = row as { time?: unknown; msg?: unknown };
|
|
72
|
+
if (typeof time === "number" && typeof msg === "string") {
|
|
73
|
+
entries.push({ time, msg });
|
|
74
|
+
}
|
|
75
|
+
}
|
|
76
|
+
return entries;
|
|
77
|
+
} catch {
|
|
78
|
+
return null;
|
|
79
|
+
}
|
|
80
|
+
}
|
|
81
|
+
|
|
82
|
+
/** Plain `dmesg`: `[ 44170.542000] message` per line. */
|
|
83
|
+
export function parseDmesgText(text: string): KernelLogEntry[] {
|
|
84
|
+
const entries: KernelLogEntry[] = [];
|
|
85
|
+
for (const line of text.split("\n")) {
|
|
86
|
+
const match = /^\[\s*(\d+\.\d+)\]\s?(.*)$/.exec(line);
|
|
87
|
+
if (match) {
|
|
88
|
+
entries.push({ time: parseFloat(match[1]), msg: match[2] });
|
|
89
|
+
}
|
|
90
|
+
}
|
|
91
|
+
return entries;
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
const OOM_KILL_RE = /[Oo]ut of memory: Killed process (\d+) \(([^)]*)\)(.*)$/;
|
|
95
|
+
const OOM_HEADER_RE = /^oom-kill:.*task_memcg=([^,]*),task=[^,]*,pid=(\d+)/;
|
|
96
|
+
|
|
97
|
+
function kb(fields: string, key: string): number | null {
|
|
98
|
+
const match = new RegExp(`${key}:(\\d+)kB`).exec(fields);
|
|
99
|
+
return match ? parseInt(match[1], 10) : null;
|
|
100
|
+
}
|
|
101
|
+
|
|
102
|
+
export function oomKillsFromEntries(entries: KernelLogEntry[]): OomKill[] {
|
|
103
|
+
const sorted = [...entries].sort((a, b) => a.time - b.time);
|
|
104
|
+
// The header precedes its "Killed process" line and names the same pid.
|
|
105
|
+
const memcgByPid = new Map<number, string>();
|
|
106
|
+
const kills: OomKill[] = [];
|
|
107
|
+
for (const entry of sorted) {
|
|
108
|
+
const header = OOM_HEADER_RE.exec(entry.msg);
|
|
109
|
+
if (header) {
|
|
110
|
+
memcgByPid.set(parseInt(header[2], 10), header[1]);
|
|
111
|
+
continue;
|
|
112
|
+
}
|
|
113
|
+
const kill = OOM_KILL_RE.exec(entry.msg);
|
|
114
|
+
if (!kill) {
|
|
115
|
+
continue;
|
|
116
|
+
}
|
|
117
|
+
const pid = parseInt(kill[1], 10);
|
|
118
|
+
const fields = kill[3];
|
|
119
|
+
const adj = /oom_score_adj:(-?\d+)/.exec(fields);
|
|
120
|
+
kills.push({
|
|
121
|
+
time: entry.time,
|
|
122
|
+
pid,
|
|
123
|
+
comm: kill[2],
|
|
124
|
+
oomScoreAdj: adj ? parseInt(adj[1], 10) : null,
|
|
125
|
+
anonRssKb: kb(fields, "anon-rss"),
|
|
126
|
+
totalVmKb: kb(fields, "total-vm"),
|
|
127
|
+
memcg: memcgByPid.get(pid) ?? null,
|
|
128
|
+
});
|
|
129
|
+
memcgByPid.delete(pid);
|
|
130
|
+
}
|
|
131
|
+
return kills;
|
|
132
|
+
}
|
|
133
|
+
|
|
134
|
+
/** Null when `dmesg` is absent or lacks CAP_SYSLOG. */
|
|
135
|
+
export async function readKernelLog(): Promise<KernelLogEntry[] | null> {
|
|
136
|
+
if (process.platform !== "linux") {
|
|
137
|
+
return null;
|
|
138
|
+
}
|
|
139
|
+
const run = async (args: string[]): Promise<string | null> => {
|
|
140
|
+
try {
|
|
141
|
+
const proc = Bun.spawn(["dmesg", ...args], {
|
|
142
|
+
stdout: "pipe",
|
|
143
|
+
stderr: "ignore",
|
|
144
|
+
windowsHide: true,
|
|
145
|
+
timeout: DMESG_TIMEOUT_MS,
|
|
146
|
+
});
|
|
147
|
+
const text = await new Response(proc.stdout).text();
|
|
148
|
+
return (await proc.exited) === 0 ? text : null;
|
|
149
|
+
} catch {
|
|
150
|
+
return null;
|
|
151
|
+
}
|
|
152
|
+
};
|
|
153
|
+
const json = await run(["--json"]);
|
|
154
|
+
const fromJson = json != null ? parseDmesgJson(json) : null;
|
|
155
|
+
if (fromJson) {
|
|
156
|
+
return fromJson;
|
|
157
|
+
}
|
|
158
|
+
const text = await run([]);
|
|
159
|
+
return text != null ? parseDmesgText(text) : null;
|
|
160
|
+
}
|
|
161
|
+
|
|
162
|
+
/**
|
|
163
|
+
* Container ids from `/proc/self/mountinfo`: the runtime names the source of
|
|
164
|
+
* the hosts, hostname, resolv.conf and termination-log mounts after the
|
|
165
|
+
* container (Kata: `<id>-<hash>-hosts`; Docker: `.../containers/<id>/hosts`).
|
|
166
|
+
*/
|
|
167
|
+
export function parseOwnContainerIds(mountinfo: string): Set<string> {
|
|
168
|
+
const ids = new Set<string>();
|
|
169
|
+
for (const line of mountinfo.split("\n")) {
|
|
170
|
+
const fields = line.split(" ");
|
|
171
|
+
if (fields.length < 5 || !CONTAINER_ID_MOUNTPOINTS.has(fields[4])) {
|
|
172
|
+
continue;
|
|
173
|
+
}
|
|
174
|
+
for (const id of line.match(/[0-9a-f]{64}/g) ?? []) {
|
|
175
|
+
ids.add(id);
|
|
176
|
+
}
|
|
177
|
+
}
|
|
178
|
+
return ids;
|
|
179
|
+
}
|
|
180
|
+
|
|
181
|
+
function readOwnContainerIds(): Set<string> {
|
|
182
|
+
try {
|
|
183
|
+
return parseOwnContainerIds(readFileSync("/proc/self/mountinfo", "utf-8"));
|
|
184
|
+
} catch {
|
|
185
|
+
return new Set();
|
|
186
|
+
}
|
|
187
|
+
}
|
|
188
|
+
|
|
189
|
+
function readBootId(): string | null {
|
|
190
|
+
try {
|
|
191
|
+
return readFileSync("/proc/sys/kernel/random/boot_id", "utf-8").trim();
|
|
192
|
+
} catch {
|
|
193
|
+
return null;
|
|
194
|
+
}
|
|
195
|
+
}
|
|
196
|
+
|
|
197
|
+
interface Cursor {
|
|
198
|
+
bootId: string | null;
|
|
199
|
+
lastTime: number;
|
|
200
|
+
}
|
|
201
|
+
|
|
202
|
+
function readCursor(path: string): Cursor | null {
|
|
203
|
+
try {
|
|
204
|
+
const parsed = JSON.parse(readFileSync(path, "utf-8")) as Partial<Cursor>;
|
|
205
|
+
if (typeof parsed.lastTime === "number") {
|
|
206
|
+
return { bootId: parsed.bootId ?? null, lastTime: parsed.lastTime };
|
|
207
|
+
}
|
|
208
|
+
} catch {
|
|
209
|
+
// Missing or unreadable.
|
|
210
|
+
}
|
|
211
|
+
return null;
|
|
212
|
+
}
|
|
213
|
+
|
|
214
|
+
function writeCursor(path: string, cursor: Cursor): void {
|
|
215
|
+
mkdirSync(dirname(path), { recursive: true });
|
|
216
|
+
writeFileSync(path, JSON.stringify(cursor));
|
|
217
|
+
}
|
|
218
|
+
|
|
219
|
+
function memcgIsOurs(memcg: string, ids: ReadonlySet<string>): boolean {
|
|
220
|
+
for (const id of ids) {
|
|
221
|
+
if (memcg.includes(id)) {
|
|
222
|
+
return true;
|
|
223
|
+
}
|
|
224
|
+
}
|
|
225
|
+
return false;
|
|
226
|
+
}
|
|
227
|
+
|
|
228
|
+
export interface OomKillReporterOptions {
|
|
229
|
+
readKernelLog?: () => Promise<KernelLogEntry[] | null>;
|
|
230
|
+
/** Cumulative cgroup oom_kill count, read after the log so it covers every logged kill. */
|
|
231
|
+
readKillCounter?: () => number | null;
|
|
232
|
+
readOwnContainerIds?: () => ReadonlySet<string>;
|
|
233
|
+
readBootId?: () => string | null;
|
|
234
|
+
shareAnalytics?: () => boolean | null;
|
|
235
|
+
cursorPath?: string;
|
|
236
|
+
record?: typeof recordWatchdogEvent;
|
|
237
|
+
}
|
|
238
|
+
|
|
239
|
+
export interface OomKillReport {
|
|
240
|
+
attribution: Attribution;
|
|
241
|
+
victims: OomKill[];
|
|
242
|
+
/** Kills the counter saw that the log could not name. */
|
|
243
|
+
unnamed: number;
|
|
244
|
+
}
|
|
245
|
+
|
|
246
|
+
export interface OomKillReporter {
|
|
247
|
+
/** Returns what was queued, for tests. */
|
|
248
|
+
check(
|
|
249
|
+
sample: ResourceSample,
|
|
250
|
+
now: number,
|
|
251
|
+
daemonPid: number | null,
|
|
252
|
+
): Promise<OomKillReport | null>;
|
|
253
|
+
}
|
|
254
|
+
|
|
255
|
+
export function createOomKillReporter(
|
|
256
|
+
options: OomKillReporterOptions = {},
|
|
257
|
+
): OomKillReporter {
|
|
258
|
+
const read = options.readKernelLog ?? readKernelLog;
|
|
259
|
+
const readKillCounter =
|
|
260
|
+
options.readKillCounter ??
|
|
261
|
+
(() => getContainerMemoryEvents()?.oomKill ?? null);
|
|
262
|
+
const ownIds = options.readOwnContainerIds ?? readOwnContainerIds;
|
|
263
|
+
const bootId = options.readBootId ?? readBootId;
|
|
264
|
+
const shareAnalytics = options.shareAnalytics ?? getRawShareAnalytics;
|
|
265
|
+
const cursorPath =
|
|
266
|
+
options.cursorPath ?? join(getMonitoringDataDir(), CURSOR_FILENAME);
|
|
267
|
+
const record = options.record ?? recordWatchdogEvent;
|
|
268
|
+
|
|
269
|
+
let firstScan = true;
|
|
270
|
+
let lastScanAt = 0;
|
|
271
|
+
let lastKillCounter: number | null = null;
|
|
272
|
+
/**
|
|
273
|
+
* Counter credit with no log line yet: the kernel prints the kill line
|
|
274
|
+
* before it increments the counter, so a kill between the log read and the
|
|
275
|
+
* counter read is counted but not yet visible. Credit is spent on the next
|
|
276
|
+
* scan and reported as unnamed if still unspent, so a kill that never logs
|
|
277
|
+
* a line cannot claim a neighbour's later.
|
|
278
|
+
*/
|
|
279
|
+
let owed = 0;
|
|
280
|
+
/** A scan was skipped mid-way; the next tick scans regardless of the sampler. */
|
|
281
|
+
let scanDue = false;
|
|
282
|
+
/** A report the telemetry store refused; retried before the next scan. */
|
|
283
|
+
let pending: {
|
|
284
|
+
report: OomKillReport;
|
|
285
|
+
sample: ResourceSample;
|
|
286
|
+
daemonPid: number | null;
|
|
287
|
+
cursor: Cursor;
|
|
288
|
+
} | null = null;
|
|
289
|
+
|
|
290
|
+
const queue = (
|
|
291
|
+
report: OomKillReport,
|
|
292
|
+
sample: ResourceSample,
|
|
293
|
+
daemonPid: number | null,
|
|
294
|
+
cursor: Cursor,
|
|
295
|
+
): boolean => {
|
|
296
|
+
if (shareAnalytics() === false) {
|
|
297
|
+
writeCursor(cursorPath, cursor);
|
|
298
|
+
return true;
|
|
299
|
+
}
|
|
300
|
+
const killedDaemon = report.victims.some(
|
|
301
|
+
(kill) =>
|
|
302
|
+
kill.oomScoreAdj === DAEMON_OOM_SCORE_ADJ ||
|
|
303
|
+
(daemonPid != null && kill.pid === daemonPid),
|
|
304
|
+
);
|
|
305
|
+
const detail = {
|
|
306
|
+
attribution: report.attribution,
|
|
307
|
+
victim_count: report.victims.length + report.unnamed,
|
|
308
|
+
unnamed_victims: report.unnamed,
|
|
309
|
+
victims: report.victims.slice(-MAX_VICTIMS_IN_DETAIL).map((kill) => ({
|
|
310
|
+
pid: kill.pid,
|
|
311
|
+
comm: kill.comm,
|
|
312
|
+
oom_score_adj: kill.oomScoreAdj,
|
|
313
|
+
anon_rss_kb: kill.anonRssKb,
|
|
314
|
+
total_vm_kb: kill.totalVmKb,
|
|
315
|
+
})),
|
|
316
|
+
daemon_pid: daemonPid,
|
|
317
|
+
killed_daemon: killedDaemon,
|
|
318
|
+
memory_current_bytes: sample.memory?.currentBytes ?? null,
|
|
319
|
+
memory_limit_bytes: sample.memory?.limitBytes ?? null,
|
|
320
|
+
memory_peak_bytes: sample.memory?.peakBytes ?? null,
|
|
321
|
+
};
|
|
322
|
+
const queued = record({
|
|
323
|
+
checkName: OOM_KILL_CHECK_NAME,
|
|
324
|
+
value: detail.victim_count,
|
|
325
|
+
detail,
|
|
326
|
+
});
|
|
327
|
+
if (!queued) {
|
|
328
|
+
return false;
|
|
329
|
+
}
|
|
330
|
+
writeCursor(cursorPath, cursor);
|
|
331
|
+
log.warn(detail, "Kernel OOM kill in the assistant container");
|
|
332
|
+
return true;
|
|
333
|
+
};
|
|
334
|
+
|
|
335
|
+
return {
|
|
336
|
+
async check(sample, now, daemonPid) {
|
|
337
|
+
if (pending) {
|
|
338
|
+
if (
|
|
339
|
+
!queue(
|
|
340
|
+
pending.report,
|
|
341
|
+
pending.sample,
|
|
342
|
+
pending.daemonPid,
|
|
343
|
+
pending.cursor,
|
|
344
|
+
)
|
|
345
|
+
) {
|
|
346
|
+
return null;
|
|
347
|
+
}
|
|
348
|
+
pending = null;
|
|
349
|
+
}
|
|
350
|
+
|
|
351
|
+
const sampleDelta = sample.deltas?.events?.oomKill ?? 0;
|
|
352
|
+
if (
|
|
353
|
+
!firstScan &&
|
|
354
|
+
!scanDue &&
|
|
355
|
+
sampleDelta === 0 &&
|
|
356
|
+
now - lastScanAt < FALLBACK_SCAN_INTERVAL_MS
|
|
357
|
+
) {
|
|
358
|
+
return null;
|
|
359
|
+
}
|
|
360
|
+
scanDue = false;
|
|
361
|
+
|
|
362
|
+
const entries = await read();
|
|
363
|
+
const currentBootId = bootId();
|
|
364
|
+
if (entries == null) {
|
|
365
|
+
firstScan = false;
|
|
366
|
+
lastScanAt = now;
|
|
367
|
+
lastKillCounter = readKillCounter() ?? lastKillCounter;
|
|
368
|
+
if (sampleDelta === 0) {
|
|
369
|
+
return null;
|
|
370
|
+
}
|
|
371
|
+
const report: OomKillReport = {
|
|
372
|
+
attribution: "cgroup_counter",
|
|
373
|
+
victims: [],
|
|
374
|
+
unnamed: sampleDelta,
|
|
375
|
+
};
|
|
376
|
+
const cursor = { bootId: currentBootId, lastTime: 0 };
|
|
377
|
+
if (!queue(report, sample, daemonPid, cursor)) {
|
|
378
|
+
pending = { report, sample, daemonPid, cursor };
|
|
379
|
+
}
|
|
380
|
+
return report;
|
|
381
|
+
}
|
|
382
|
+
|
|
383
|
+
// Read after the log so the count covers every line the log holds.
|
|
384
|
+
const killCounter = readKillCounter();
|
|
385
|
+
if (killCounter == null && sample.events != null) {
|
|
386
|
+
// One failed read of a counter that exists: try again next tick
|
|
387
|
+
// rather than treating every kill on the kernel as ours.
|
|
388
|
+
scanDue = true;
|
|
389
|
+
return null;
|
|
390
|
+
}
|
|
391
|
+
const isFirstScan = firstScan;
|
|
392
|
+
firstScan = false;
|
|
393
|
+
lastScanAt = now;
|
|
394
|
+
const counterDelta =
|
|
395
|
+
killCounter != null && lastKillCounter != null
|
|
396
|
+
? Math.max(0, killCounter - lastKillCounter)
|
|
397
|
+
: null;
|
|
398
|
+
lastKillCounter = killCounter ?? lastKillCounter;
|
|
399
|
+
|
|
400
|
+
const all = oomKillsFromEntries(entries);
|
|
401
|
+
const maxTime = all.length > 0 ? all[all.length - 1].time : 0;
|
|
402
|
+
const stored = readCursor(cursorPath);
|
|
403
|
+
// A different boot id, or a log that has not yet reached the stored
|
|
404
|
+
// timestamp, means the kernel restarted and the cursor is stale.
|
|
405
|
+
const sameBoot =
|
|
406
|
+
stored != null &&
|
|
407
|
+
stored.bootId === currentBootId &&
|
|
408
|
+
maxTime >= stored.lastTime;
|
|
409
|
+
const sinceTime = sameBoot ? stored.lastTime : -1;
|
|
410
|
+
const fresh = all.filter((kill) => kill.time > sinceTime);
|
|
411
|
+
const cursor: Cursor = {
|
|
412
|
+
bootId: currentBootId,
|
|
413
|
+
lastTime: Math.max(sinceTime, maxTime),
|
|
414
|
+
};
|
|
415
|
+
const ids = ownIds();
|
|
416
|
+
|
|
417
|
+
let report: OomKillReport | null = null;
|
|
418
|
+
if (counterDelta == null || isFirstScan) {
|
|
419
|
+
// No counter to lean on: a kill that took the daemon down restarted
|
|
420
|
+
// this monitor with it, and the new container's counter starts at 0.
|
|
421
|
+
const ours = fresh.filter(
|
|
422
|
+
(kill) => kill.memcg == null || memcgIsOurs(kill.memcg, ids),
|
|
423
|
+
);
|
|
424
|
+
if (ours.length > 0) {
|
|
425
|
+
report = { attribution: "kernel_log", victims: ours, unnamed: 0 };
|
|
426
|
+
}
|
|
427
|
+
owed = 0;
|
|
428
|
+
} else {
|
|
429
|
+
const byHeader = fresh.filter(
|
|
430
|
+
(kill) => kill.memcg != null && memcgIsOurs(kill.memcg, ids),
|
|
431
|
+
);
|
|
432
|
+
const unknown = fresh.filter((kill) => kill.memcg == null);
|
|
433
|
+
// Header-attributed kills spend counter credit first; what is left
|
|
434
|
+
// names header-less kills oldest first. Header-less kills beyond the
|
|
435
|
+
// credit are neighbours', since our counter would have covered ours.
|
|
436
|
+
const carry = owed;
|
|
437
|
+
let credit = counterDelta + carry - byHeader.length;
|
|
438
|
+
const byCounter = unknown.slice(0, Math.max(0, credit));
|
|
439
|
+
credit = Math.max(0, credit - byCounter.length);
|
|
440
|
+
owed = Math.min(credit, counterDelta);
|
|
441
|
+
const unnamed = credit - owed;
|
|
442
|
+
const victims = [...byHeader, ...byCounter].sort(
|
|
443
|
+
(a, b) => a.time - b.time,
|
|
444
|
+
);
|
|
445
|
+
if (victims.length > 0 || unnamed > 0) {
|
|
446
|
+
report = {
|
|
447
|
+
attribution:
|
|
448
|
+
byCounter.length === 0 && unnamed === 0
|
|
449
|
+
? "cgroup"
|
|
450
|
+
: "cgroup_counter",
|
|
451
|
+
victims,
|
|
452
|
+
unnamed,
|
|
453
|
+
};
|
|
454
|
+
}
|
|
455
|
+
}
|
|
456
|
+
|
|
457
|
+
if (report == null) {
|
|
458
|
+
if (!sameBoot || cursor.lastTime !== stored?.lastTime) {
|
|
459
|
+
writeCursor(cursorPath, cursor);
|
|
460
|
+
}
|
|
461
|
+
return null;
|
|
462
|
+
}
|
|
463
|
+
if (!queue(report, sample, daemonPid, cursor)) {
|
|
464
|
+
pending = { report, sample, daemonPid, cursor };
|
|
465
|
+
}
|
|
466
|
+
return report;
|
|
467
|
+
},
|
|
468
|
+
};
|
|
469
|
+
}
|
|
@@ -46,6 +46,8 @@ import {
|
|
|
46
46
|
import { readActiveConversations } from "./active-conversations.js";
|
|
47
47
|
import { readDaemonHeartbeat } from "./daemon-heartbeat.js";
|
|
48
48
|
import { topProcessesByFd } from "./file-descriptors.js";
|
|
49
|
+
import { sweepInheritedOomProtection } from "./oom-inheritance-sweep.js";
|
|
50
|
+
import { createOomKillReporter } from "./oom-kill-reporter.js";
|
|
49
51
|
import { getTrackedDataFiles, readFileResidency } from "./page-cache.js";
|
|
50
52
|
import { topProcessesByMemory } from "./process-memory.js";
|
|
51
53
|
import {
|
|
@@ -70,6 +72,8 @@ const SNAPSHOTS_DIR = "snapshots";
|
|
|
70
72
|
const MAX_SNAPSHOTS = 20;
|
|
71
73
|
/** Cap on retained baseline snapshots (4h of history at the 10min default). */
|
|
72
74
|
const MAX_BASELINE_SNAPSHOTS = 24;
|
|
75
|
+
/** How often the daemon's descendants are checked for inherited OOM protection. */
|
|
76
|
+
const OOM_SWEEP_INTERVAL_MS = 5_000;
|
|
73
77
|
|
|
74
78
|
/** Snapshot kinds: filename prefix + retention are per-kind so periodic
|
|
75
79
|
* baselines can never evict high-memory forensics. */
|
|
@@ -264,6 +268,8 @@ export function startResourceSampler(
|
|
|
264
268
|
// thread's kernel state mid-stall when the heartbeat goes stale.
|
|
265
269
|
const stallCapture = createStallCaptureMonitor(dataDir);
|
|
266
270
|
const processUsage = createProcessUsageTracker();
|
|
271
|
+
const oomKills = createOomKillReporter();
|
|
272
|
+
let lastOomSweepAt = 0;
|
|
267
273
|
|
|
268
274
|
// Skip ticks while a sample is in flight: the disk measurement can take
|
|
269
275
|
// seconds (du over the workspace), and overlapping ticks would all delta
|
|
@@ -302,6 +308,30 @@ export function startResourceSampler(
|
|
|
302
308
|
log.warn({ err }, "Daemon stall check failed");
|
|
303
309
|
}
|
|
304
310
|
|
|
311
|
+
try {
|
|
312
|
+
await oomKills.check(sample, now, readDaemonHeartbeat(now)?.pid ?? null);
|
|
313
|
+
} catch (err) {
|
|
314
|
+
log.warn({ err }, "OOM kill report failed");
|
|
315
|
+
}
|
|
316
|
+
|
|
317
|
+
if (now - lastOomSweepAt >= OOM_SWEEP_INTERVAL_MS) {
|
|
318
|
+
lastOomSweepAt = now;
|
|
319
|
+
const daemonPid = readDaemonHeartbeat(now)?.pid;
|
|
320
|
+
if (daemonPid != null) {
|
|
321
|
+
try {
|
|
322
|
+
const resets = sweepInheritedOomProtection({
|
|
323
|
+
daemonPid,
|
|
324
|
+
keepPids: new Set([process.pid]),
|
|
325
|
+
});
|
|
326
|
+
for (const reset of resets) {
|
|
327
|
+
log.warn(reset, "Reset OOM protection inherited from the daemon");
|
|
328
|
+
}
|
|
329
|
+
} catch (err) {
|
|
330
|
+
log.warn({ err }, "OOM inheritance sweep failed");
|
|
331
|
+
}
|
|
332
|
+
}
|
|
333
|
+
}
|
|
334
|
+
|
|
305
335
|
const ratio = sample.memory?.ratio;
|
|
306
336
|
if (
|
|
307
337
|
ratio != null &&
|
|
@@ -2,17 +2,66 @@
|
|
|
2
2
|
|
|
3
3
|
Signal-driven notification architecture where producers emit free-form events and an LLM-backed decision engine determines whether, where, and how to notify the user.
|
|
4
4
|
|
|
5
|
+
## From me chat alerts
|
|
6
|
+
|
|
7
|
+
With `assistant-initiated-threads` enabled, an `assistant.share` from a background,
|
|
8
|
+
scheduled, or unresolved source can create a new chat in From me. The pipeline
|
|
9
|
+
resolves that promotion once and applies it only to vellum conversation pairing.
|
|
10
|
+
Ordinary shares retain their authored content and original urgency.
|
|
11
|
+
|
|
12
|
+
A new chat qualifies for an alert only after its conversation and seed message
|
|
13
|
+
are persisted and it belongs in From me: an assistant-initiated, visible,
|
|
14
|
+
ungrouped chat. Reusing an existing chat, filing it elsewhere, pinning it, or
|
|
15
|
+
failing to persist the seed does not enable this additional alert. An invalid
|
|
16
|
+
reuse target that falls back to a newly created eligible chat does qualify.
|
|
17
|
+
|
|
18
|
+
The successful creation makes the local intent non-silent. When vellum is
|
|
19
|
+
selected and platform credentials are configured, routing can also add the
|
|
20
|
+
existing mobile push channel. Explicit quiet intent, suppression, exclusive
|
|
21
|
+
channel allowlists, and single-channel routing retain precedence. A platform
|
|
22
|
+
route added solely for this feature is skipped if creation does not qualify;
|
|
23
|
+
independently selected urgent or explicit delivery retains its existing policy.
|
|
24
|
+
|
|
25
|
+
Both paths target the persisted new conversation and seed message. Replayed
|
|
26
|
+
decisions reuse the original delivery audit and destination without creating
|
|
27
|
+
another chat or seed. Shares retain their existing transcript, Home feed, and
|
|
28
|
+
notification bell behavior; they are not classified as completion signals.
|
|
29
|
+
|
|
30
|
+
Local delivery waits for the existing bounded platform outcome and preserves
|
|
31
|
+
per-platform remote/local duplicate suppression. Missing tokens, disabled remote
|
|
32
|
+
push, or failed platform delivery leave the local fallback available. Provider
|
|
33
|
+
acceptance does not prove that the OS displayed a banner.
|
|
34
|
+
|
|
35
|
+
iOS uses the existing APNs route and device registration. macOS uses the live
|
|
36
|
+
native notification bridge while the application is running, including hidden
|
|
37
|
+
or minimized windows; quitting the application ends that live-delivery path.
|
|
38
|
+
Android, other Electron clients, and permitted open browser tabs share these
|
|
39
|
+
channels and receive the same eligible events. Existing notification permission,
|
|
40
|
+
active-conversation suppression, tap navigation, and logout behavior apply.
|
|
41
|
+
Opening From me or reconnecting does not replay historical chats as alerts.
|
|
42
|
+
|
|
5
43
|
## Completion alerts
|
|
6
44
|
|
|
7
45
|
`chat.assistant_reply`, `schedule.result`, and explicitly user-facing
|
|
8
46
|
`activity.complete` signals use the available subset of `vellum` and
|
|
9
47
|
`platform`. A self-hosted assistant can deliver through `vellum` without
|
|
10
48
|
mobile push. `completion-policy.ts` owns local presentation: selected
|
|
11
|
-
completions can banner at medium urgency,
|
|
12
|
-
remain silent, and explicitly quiet work stays silent. The broadcaster resolves
|
|
13
|
-
the existing `silent` contract
|
|
49
|
+
completions can banner at medium urgency, and low/medium notifications without
|
|
50
|
+
a completion or new From me chat remain silent, and explicitly quiet work stays silent. The broadcaster resolves
|
|
51
|
+
the existing `silent` contract for the local intent and paired-conversation
|
|
14
52
|
event; clients continue honoring it.
|
|
15
53
|
|
|
54
|
+
`notifications.newMessageEnabled` defaults to true. Setting it false makes
|
|
55
|
+
only `chat.assistant_reply` local intents silent and suppresses their platform
|
|
56
|
+
push. Selected channels remain routing candidates throughout awaited broadcast
|
|
57
|
+
preparation, so re-enabling alerts before the send can still deliver a push.
|
|
58
|
+
`chat-reply-policy.ts` reads the current preference immediately before delivery,
|
|
59
|
+
including deferred local sends and each platform retry. Delivery audit rows
|
|
60
|
+
record a suppressed push as skipped; a push already accepted by a provider keeps
|
|
61
|
+
its acceptance metadata. Transcript pairing, unread state, feed policy,
|
|
62
|
+
external-channel delivery, scheduled results, and action-required alerts retain
|
|
63
|
+
their existing behavior.
|
|
64
|
+
|
|
16
65
|
Local completion delivery requires the active Vellum guardian principal from
|
|
17
66
|
the resolved destination. The broadcaster checks that identity before pairing,
|
|
18
67
|
and the adapter targets the event hub's `targetActorPrincipalId` before sending
|
|
@@ -189,7 +238,7 @@ Before an adapter sends a notification, `pairDeliveryWithConversation()` (in `co
|
|
|
189
238
|
|
|
190
239
|
- **Vellum (`start_new_conversation`)**:
|
|
191
240
|
- A signal that sets `requiresConversation` gets a conversation of its own, created with its seed message before the send (`standard` unless the producer overrides `conversationType`). An explicit `reuse_existing` action appends to a valid target instead, and falls back to a new conversation (`conversationFallbackUsed: true`) when the target is stale.
|
|
192
|
-
- A signal that does not set it creates no conversation, except that with `assistant-initiated-threads` on, an `assistant.share` with no `conversationMetadata.source` of its own is promoted to one when its producing conversation is a background or scheduled run, or does not resolve (`
|
|
241
|
+
- A signal that does not set it creates no conversation, except that with `assistant-initiated-threads` on, an `assistant.share` with no `conversationMetadata.source` of its own is promoted to one when its producing conversation is a background or scheduled run, or does not resolve (`resolveAssistantInitiatedThread()` in `assistant-initiated-thread.ts`). Otherwise its body is appended to the conversation that produced it (resolved from `sourceContextId`). A `chat.assistant_reply` appends nothing, because the full reply is already in that transcript.
|
|
193
242
|
- **External channels (`continue_existing_conversation`)**: only the chat's home conversation is resolved, and nothing is written. The broadcaster records the delivered post as an assistant row once the channel acknowledges it (`recordDeliveredChannelPost`), so a failed or pending delivery never reads as something the assistant said.
|
|
194
243
|
- **`push_only`** (platform) and **`not_deliverable`**: nothing is paired. Platform push deep-links through the vellum delivery's conversation.
|
|
195
244
|
- **Guardian-request deliveries to channels** pair nothing either: they are projections of a canonical request, and only their vellum delivery carries a conversation (see `notifications/AGENTS.md`).
|