@vellumai/assistant 0.11.3 → 0.11.4-dev.202608190019.b94dbf2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +9 -3
- package/ARCHITECTURE.md +32 -10
- package/docs/architecture/memory.md +43 -5
- package/docs/architecture/security.md +96 -152
- package/docs/architecture/turn-actor.md +70 -0
- package/docs/browser-use-architecture-phase2.md +128 -56
- package/docs/flux-turn-detection-spike.md +248 -0
- package/docs/guardian-request-flow.md +35 -0
- package/docs/stt-provider-onboarding.md +3 -1
- package/knip.json +3 -0
- package/node_modules/@vellumai/ces-client/node_modules/@vellumai/service-contracts/package.json +1 -0
- package/node_modules/@vellumai/ces-client/node_modules/@vellumai/service-contracts/src/__tests__/url-normalization.test.ts +135 -0
- package/node_modules/@vellumai/ces-client/node_modules/@vellumai/service-contracts/src/channels.ts +11 -0
- package/node_modules/@vellumai/ces-client/node_modules/@vellumai/service-contracts/src/index.ts +1 -0
- package/node_modules/@vellumai/ces-client/node_modules/@vellumai/service-contracts/src/remote-web-pairing.ts +60 -0
- package/node_modules/@vellumai/ces-client/node_modules/@vellumai/service-contracts/src/url-normalization.ts +107 -0
- package/node_modules/@vellumai/gateway-client/node_modules/@vellumai/service-contracts/package.json +1 -0
- package/node_modules/@vellumai/gateway-client/node_modules/@vellumai/service-contracts/src/__tests__/url-normalization.test.ts +135 -0
- package/node_modules/@vellumai/gateway-client/node_modules/@vellumai/service-contracts/src/channels.ts +11 -0
- package/node_modules/@vellumai/gateway-client/node_modules/@vellumai/service-contracts/src/index.ts +1 -0
- package/node_modules/@vellumai/gateway-client/node_modules/@vellumai/service-contracts/src/remote-web-pairing.ts +60 -0
- package/node_modules/@vellumai/gateway-client/node_modules/@vellumai/service-contracts/src/url-normalization.ts +107 -0
- package/node_modules/@vellumai/gateway-client/src/admission-policy-contract.ts +34 -0
- package/node_modules/@vellumai/gateway-client/src/gateway-ipc-contracts.ts +166 -1
- package/node_modules/@vellumai/gateway-client/src/index.ts +2 -0
- package/node_modules/@vellumai/service-contracts/package.json +1 -0
- package/node_modules/@vellumai/service-contracts/src/__tests__/url-normalization.test.ts +135 -0
- package/node_modules/@vellumai/service-contracts/src/channels.ts +11 -0
- package/node_modules/@vellumai/service-contracts/src/index.ts +1 -0
- package/node_modules/@vellumai/service-contracts/src/remote-web-pairing.ts +60 -0
- package/node_modules/@vellumai/service-contracts/src/url-normalization.ts +107 -0
- package/openapi.yaml +289 -113
- package/package.json +1 -1
- package/scripts/voice-ttft-spike.ts +3 -3
- package/scripts/write-plugin-api-shim.ts +10 -0
- package/src/__tests__/app-compiler.test.ts +38 -3
- package/src/__tests__/app-control-flow.test.ts +1 -0
- package/src/__tests__/approval-routes-http.test.ts +2 -2
- package/src/__tests__/assistant-feature-flag-guard.test.ts +25 -3
- package/src/__tests__/attachments-store.test.ts +21 -12
- package/src/__tests__/byok-default-profile-ensure.test.ts +17 -0
- package/src/__tests__/call-setup-flow-name-capture.test.ts +0 -1
- package/src/__tests__/call-site-routing-provider.test.ts +1 -1
- package/src/__tests__/channel-availability-routes.test.ts +14 -1
- package/src/__tests__/channel-capabilities-dedupe.test.ts +214 -0
- package/src/__tests__/channel-delivery-store.test.ts +14 -14
- package/src/__tests__/channel-inbound-disk-pressure.test.ts +4 -4
- package/src/__tests__/channel-setup-panel-ack.test.ts +1 -1
- package/src/__tests__/checker.test.ts +20 -314
- package/src/__tests__/cli-memory-v2-reembed-skills.test.ts +6 -2
- package/src/__tests__/compaction-events.test.ts +8 -10
- package/src/__tests__/config-loader-backfill.test.ts +3 -3
- package/src/__tests__/config-schema.test.ts +25 -10
- package/src/__tests__/context-search-memory-v2-source.test.ts +0 -1
- package/src/__tests__/conversation-agent-loop-inference-profile.test.ts +8 -11
- package/src/__tests__/conversation-agent-loop-overflow.test.ts +8 -11
- package/src/__tests__/conversation-agent-loop.test.ts +32 -21
- package/src/__tests__/conversation-attachments.test.ts +0 -1
- package/src/__tests__/conversation-attention-store.test.ts +63 -0
- package/src/__tests__/conversation-confirmation-signals.test.ts +112 -0
- package/src/__tests__/conversation-delete-schedule-cleanup.test.ts +0 -4
- package/src/__tests__/conversation-fork-crud.test.ts +69 -0
- package/src/__tests__/conversation-fork-referential.test.ts +67 -0
- package/src/__tests__/conversation-fork-retrospective.test.ts +24 -0
- package/src/__tests__/conversation-load-history-repair.test.ts +209 -0
- package/src/__tests__/conversation-notifiers-provenance.test.ts +59 -0
- package/src/__tests__/conversation-queue.test.ts +154 -4
- package/src/__tests__/conversation-routes-disk-view.test.ts +1 -1
- package/src/__tests__/conversation-routes-enabled-plugins.test.ts +1 -1
- package/src/__tests__/conversation-routes-guardian-reply.test.ts +9 -9
- package/src/__tests__/conversation-routes-hidden-queue.test.ts +1 -1
- package/src/__tests__/conversation-routes-slash-commands.test.ts +1 -1
- package/src/__tests__/conversation-runtime-assembly.test.ts +187 -102
- package/src/__tests__/conversation-runtime-workspace.test.ts +14 -10
- package/src/__tests__/conversation-slash-queue.test.ts +3 -0
- package/src/__tests__/conversation-surfaces-action-delivery.test.ts +1 -0
- package/src/__tests__/conversation-surfaces-activation-emit.test.ts +1 -0
- package/src/__tests__/conversation-surfaces-app-control.test.ts +1 -0
- package/src/__tests__/conversation-surfaces-app-open.test.ts +1 -1
- package/src/__tests__/conversation-surfaces-data-persist.test.ts +1 -1
- package/src/__tests__/conversation-surfaces-history-restored-completion.test.ts +21 -14
- package/src/__tests__/conversation-surfaces-queued-emit.test.ts +1 -0
- package/src/__tests__/conversation-surfaces-standalone-payloads.test.ts +1 -0
- package/src/__tests__/conversation-surfaces-standalone.test.ts +1 -0
- package/src/__tests__/conversation-surfaces-state-update.test.ts +1 -1
- package/src/__tests__/conversation-surfaces-table-action.test.ts +1 -1
- package/src/__tests__/conversation-surfaces-task-progress.test.ts +1 -1
- package/src/__tests__/conversation-tool-setup-app-refresh.test.ts +1 -1
- package/src/__tests__/conversation-tool-setup-attribution.test.ts +1 -1
- package/src/__tests__/credential-prompt-route.test.ts +7 -10
- package/src/__tests__/credential-security-invariants.test.ts +1 -0
- package/src/__tests__/cu-unified-flow.test.ts +1 -0
- package/src/__tests__/custom-profile-ensure.test.ts +5 -1
- package/src/__tests__/default-plugin-names-guard.test.ts +12 -1
- package/src/__tests__/discord-access-request-privacy.test.ts +5 -1
- package/src/__tests__/discord-requester-notice-privacy.test.ts +3 -3
- package/src/__tests__/document-append-idempotency.test.ts +233 -0
- package/src/__tests__/document-sync-tags.test.ts +0 -75
- package/src/__tests__/dynamic-skill-background-guard.test.ts +0 -2
- package/src/__tests__/edit-propagation.test.ts +107 -11
- package/src/__tests__/file-ops-service.test.ts +163 -30
- package/src/__tests__/filesystem-tools.test.ts +23 -24
- package/src/__tests__/gateway-only-guard.test.ts +2 -5
- package/src/__tests__/helpers/gateway-classify-mock.ts +27 -6
- package/src/__tests__/helpers/mock-actor-context.ts +49 -0
- package/src/__tests__/helpers/mock-conversation.ts +13 -1
- package/src/__tests__/host-file-read-tool.test.ts +16 -19
- package/src/__tests__/http-user-message-parity.test.ts +1 -1
- package/src/__tests__/init-feature-flag-overrides.test.ts +49 -0
- package/src/__tests__/injector-chain.test.ts +63 -41
- package/src/__tests__/injector-disk-pressure.test.ts +11 -23
- package/src/__tests__/inline-skill-load-permissions.test.ts +47 -36
- package/src/__tests__/llm-context-resolution.test.ts +73 -1
- package/src/__tests__/llm-schema.test.ts +5 -2
- package/src/__tests__/managed-skill-lifecycle.test.ts +7 -0
- package/src/__tests__/mcp-list-plugin-servers.test.ts +250 -0
- package/src/__tests__/media-generate-image.test.ts +131 -21
- package/src/__tests__/memory-retrieval-hook.test.ts +100 -7
- package/src/__tests__/messages-read-boundary-guard.test.ts +134 -0
- package/src/__tests__/mtime-cache.test.ts +3 -1
- package/src/__tests__/non-member-access-request.test.ts +0 -20
- package/src/__tests__/outbound-slack-persistence.test.ts +40 -1
- package/src/__tests__/platform-bash-auto-approve.test.ts +0 -4
- package/src/__tests__/plugin-api-store-credential.test.ts +261 -0
- package/src/__tests__/plugin-api-webhook-url.test.ts +10 -7
- package/src/__tests__/plugin-disabled-state.test.ts +2 -0
- package/src/__tests__/plugin-effective-enabled-set.test.ts +55 -3
- package/src/__tests__/plugin-execution-context.test.ts +73 -0
- package/src/__tests__/plugin-import-boundary-guard.test.ts +5 -1
- package/src/__tests__/plugin-import-boundary-reverse-guard.test.ts +7 -4
- package/src/__tests__/plugin-secret-pattern-contribution.test.ts +1 -1
- package/src/__tests__/post-compaction-reinjection-idempotency.test.ts +14 -7
- package/src/__tests__/provider-commit-message-generator.test.ts +20 -0
- package/src/__tests__/proxy-approval-callback.test.ts +1 -0
- package/src/__tests__/qdrant-manager.test.ts +14 -1
- package/src/__tests__/reaction-persistence.test.ts +200 -7
- package/src/__tests__/require-fresh-approval.test.ts +0 -4
- package/src/__tests__/risk-classification-boundary-guard.test.ts +115 -0
- package/src/__tests__/run-conversation-turn-persistence.test.ts +460 -105
- package/src/__tests__/run-due-schedules.test.ts +21 -0
- package/src/__tests__/scaffold-managed-skill-tool.test.ts +187 -18
- package/src/__tests__/schedule-routes.test.ts +23 -0
- package/src/__tests__/schedule-store.test.ts +17 -0
- package/src/__tests__/scoped-approval-grants.test.ts +11 -6
- package/src/__tests__/secret-ingress-channel.test.ts +0 -1
- package/src/__tests__/secret-ingress-http.test.ts +1 -1
- package/src/__tests__/send-endpoint-busy.test.ts +3 -3
- package/src/__tests__/skills.test.ts +32 -0
- package/src/__tests__/slack-edit-ordering-characterization.test.ts +0 -1
- package/src/__tests__/starter-task-flow.test.ts +5 -4
- package/src/__tests__/subagent-call-site-routing.test.ts +31 -19
- package/src/__tests__/subagent-fork-prompt-role.test.ts +1 -1
- package/src/__tests__/subagent-spawn-and-await.test.ts +18 -17
- package/src/__tests__/subagent-tool-gate-mode.test.ts +1 -1
- package/src/__tests__/subagent-tools.test.ts +81 -101
- package/src/__tests__/surface-completion-in-flight-snapshot.test.ts +1 -0
- package/src/__tests__/tool-execution-pipeline.benchmark.test.ts +1 -1
- package/src/__tests__/tool-executor-lifecycle-events.test.ts +132 -5
- package/src/__tests__/tool-executor.test.ts +68 -39
- package/src/__tests__/turn-events-store.test.ts +43 -0
- package/src/__tests__/ui-choice-copy-surfaces.test.ts +1 -1
- package/src/__tests__/ui-shape-teaching.test.ts +33 -0
- package/src/__tests__/ui-visual-surface.test.ts +1 -1
- package/src/__tests__/ui-voice-picker-surface.test.ts +128 -0
- package/src/__tests__/ui-work-result-surface.test.ts +1 -1
- package/src/__tests__/user-plugin-loader.test.ts +3 -1
- package/src/__tests__/verification-control-plane-policy.test.ts +0 -2
- package/src/__tests__/visible-app-context.test.ts +16 -9
- package/src/__tests__/voice-config-update.test.ts +40 -0
- package/src/__tests__/voice-scoped-grant-consumer.test.ts +5 -3
- package/src/__tests__/voice-session-bridge.test.ts +85 -29
- package/src/__tests__/worker-entrypoint-guards.test.ts +54 -0
- package/src/__tests__/worker-plugin-surface.test.ts +77 -0
- package/src/__tests__/workspace-migration-142-consolidate-voice-front-door.test.ts +158 -0
- package/src/__tests__/workspace-migration-143-repair-deprecated-codex-model-id.test.ts +133 -0
- package/src/__tests__/workspace-migration-144-convert-stranded-subscription-openai-profiles.test.ts +316 -0
- package/src/__tests__/workspace-migration-145-collapse-profile-bindings-to-entries.test.ts +325 -0
- package/src/__tests__/workspace-migration-146-repair-retired-fireworks-deepseek-flash-model-id.test.ts +235 -0
- package/src/acp/__tests__/acp-claude-oauth.test.ts +10 -2
- package/src/acp/__tests__/auth-required.test.ts +161 -0
- package/src/acp/acp-claude-oauth.ts +19 -2
- package/src/acp/agent-process.test.ts +100 -0
- package/src/acp/agent-process.ts +29 -26
- package/src/acp/auth-required.ts +102 -0
- package/src/acp/session-manager.test.ts +119 -0
- package/src/acp/session-manager.ts +76 -3
- package/src/api/events/acp-auth-required.ts +55 -0
- package/src/api/events/host-file.ts +2 -2
- package/src/api/index.ts +7 -0
- package/src/api/surfaces.ts +12 -3
- package/src/apps/app-store.ts +3 -0
- package/src/bundler/package-resolver.ts +2 -30
- package/src/calls/__tests__/voice-session-bridge.test.ts +194 -11
- package/src/calls/__tests__/voice-triage-escalate.test.ts +102 -0
- package/src/calls/call-controller.ts +19 -3
- package/src/calls/call-setup-flow.ts +0 -1
- package/src/calls/media-stream-stt-session.ts +15 -0
- package/src/calls/voice-session-bridge.ts +115 -41
- package/src/calls/voice-triage-escalate.ts +105 -2
- package/src/channels/__tests__/plugin-channel-declarations.test.ts +161 -0
- package/src/channels/config.ts +13 -0
- package/src/channels/plugin-channel-declarations.ts +108 -0
- package/src/channels/types.ts +30 -0
- package/src/cli/AGENTS.md +5 -2
- package/src/cli/bundled-modules.ts +29 -0
- package/src/cli/commands/credentials.help.ts +2 -2
- package/src/cli/commands/db/repair.ts +4 -8
- package/src/cli/commands/domain.ts +6 -3
- package/src/cli/commands/email.ts +6 -3
- package/src/cli/commands/inference-providers.ts +1 -1
- package/src/cli/commands/keys.ts +8 -3
- package/src/cli/commands/mcp.help.ts +13 -4
- package/src/cli/commands/mcp.ts +9 -0
- package/src/cli/commands/memory/__tests__/memory-v2.test.ts +57 -7
- package/src/cli/commands/memory/__tests__/memory-v3.test.ts +128 -5
- package/src/cli/commands/memory/index.help.ts +84 -28
- package/src/cli/commands/memory/index.ts +2 -0
- package/src/cli/commands/memory/memory-v2.ts +58 -54
- package/src/cli/commands/memory/memory-v3.ts +64 -0
- package/src/cli/commands/memory/memory-validate.ts +18 -0
- package/src/cli/commands/plugins.ts +85 -36
- package/src/cli/commands/schedules.ts +35 -1
- package/src/cli/commands/stt.help.ts +27 -2
- package/src/cli/commands/trust.ts +3 -14
- package/src/cli/lib/__tests__/upgrade-plugin.test.ts +39 -0
- package/src/cli/lib/bundled-marketplace.json +14 -1
- package/src/cli/lib/upgrade-plugin.ts +42 -0
- package/src/config/__tests__/balanced-model-experiment.test.ts +278 -0
- package/src/config/__tests__/default-profile-catalog.test.ts +34 -2
- package/src/config/__tests__/default-provider.test.ts +6 -1
- package/src/config/__tests__/profile-materialization.test.ts +75 -19
- package/src/config/assistant-feature-flags.ts +36 -15
- package/src/config/balanced-model-experiment.ts +35 -0
- package/src/config/bundled-skills/acp/SKILL.md +6 -7
- package/src/config/bundled-skills/document-editor/SKILL.md +2 -2
- package/src/config/bundled-skills/document-editor/TOOLS.json +2 -2
- package/src/config/bundled-skills/image-studio/SKILL.md +5 -4
- package/src/config/bundled-skills/image-studio/TOOLS.json +1 -1
- package/src/config/bundled-skills/image-studio/tools/media-generate-image.ts +101 -0
- package/src/config/bundled-skills/media-processing/services/preprocess.ts +14 -4
- package/src/config/bundled-skills/settings/TOOLS.json +3 -3
- package/src/config/bundled-skills/settings/tools/navigate-settings-tab.test.ts +65 -0
- package/src/config/bundled-skills/settings/tools/navigate-settings-tab.ts +7 -1
- package/src/config/bundled-skills/settings/tools/shared.ts +16 -0
- package/src/config/bundled-skills/settings/tools/voice-config-update.ts +19 -1
- package/src/config/bundled-skills/skill-management/TOOLS.json +9 -3
- package/src/config/bundled-skills/subagent/SKILL.md +17 -12
- package/src/config/bundled-skills/subagent/TOOLS.json +4 -4
- package/src/config/bundled-skills/transcribe/tools/transcribe-media.test.ts +22 -1
- package/src/config/bundled-skills/transcribe/tools/transcribe-media.ts +9 -2
- package/src/config/call-site-defaults.ts +11 -5
- package/src/config/default-profile-catalog.ts +179 -16
- package/src/config/default-profile-names.ts +4 -1
- package/src/config/default-provider-resolution.ts +4 -0
- package/src/config/feature-flag-registry.json +12 -12
- package/src/config/llm-context-resolution.ts +11 -3
- package/src/config/llm-resolver.ts +28 -1
- package/src/config/profile-materialization.ts +70 -22
- package/src/config/schemas/__tests__/live-voice.test.ts +107 -4
- package/src/config/schemas/call-site-catalog.ts +4 -4
- package/src/config/schemas/live-voice.ts +57 -23
- package/src/config/schemas/llm.ts +59 -32
- package/src/config/schemas/mcp.ts +23 -0
- package/src/config/schemas/plugin-updates.ts +6 -2
- package/src/config/schemas/stt.ts +1 -0
- package/src/config/skills.ts +9 -2
- package/src/context/outbound-sanitize.ts +96 -1
- package/src/daemon/__tests__/conversation-surfaces-launch.test.ts +1 -1
- package/src/daemon/__tests__/plugin-mcp-reconcile.test.ts +82 -0
- package/src/daemon/conversation-agent-loop-handlers.ts +15 -10
- package/src/daemon/conversation-agent-loop.ts +31 -19
- package/src/daemon/conversation-messaging.ts +5 -1
- package/src/daemon/conversation-notifiers.ts +20 -10
- package/src/daemon/conversation-process.ts +20 -6
- package/src/daemon/conversation-runtime-assembly.ts +12 -6
- package/src/daemon/conversation-store.ts +14 -4
- package/src/daemon/conversation-surfaces.ts +75 -26
- package/src/daemon/conversation-tool-setup.ts +3 -6
- package/src/daemon/conversation.ts +154 -51
- package/src/daemon/doordash-steps.ts +2 -2
- package/src/daemon/lifecycle.ts +14 -1
- package/src/daemon/mcp-reload-service.ts +36 -6
- package/src/daemon/process-message.ts +13 -26
- package/src/daemon/providers-setup.ts +6 -3
- package/src/daemon/trust-context-types.ts +29 -0
- package/src/daemon/trust-context.ts +12 -7
- package/src/daemon/wake-conversation-ops.ts +3 -2
- package/src/daemon/windows-compiled-entry.ts +4 -0
- package/src/documents/document-store.ts +143 -240
- package/src/hooks/hook-loader.ts +3 -3
- package/src/hooks/registry.ts +103 -45
- package/src/hooks/types.ts +5 -0
- package/src/inbound/__tests__/oauth-callback-url.test.ts +83 -0
- package/src/inbound/oauth-callback-url.ts +61 -0
- package/src/ipc/gateway-client.test.ts +1 -1
- package/src/ipc/gateway-client.ts +14 -21
- package/src/ipc/gateway-flag-listener.ts +17 -3
- package/src/live-voice/__tests__/live-voice-agent-turn.test.ts +1 -104
- package/src/live-voice/__tests__/live-voice-events.test.ts +7 -8
- package/src/live-voice/__tests__/live-voice-flux-turn-end.test.ts +1050 -0
- package/src/live-voice/__tests__/live-voice-metrics.test.ts +115 -8
- package/src/live-voice/__tests__/live-voice-photo.test.ts +100 -0
- package/src/live-voice/__tests__/live-voice-progress.test.ts +60 -194
- package/src/live-voice/__tests__/live-voice-stt.test.ts +14 -0
- package/src/live-voice/__tests__/live-voice-triage-escalate.test.ts +29 -0
- package/src/live-voice/__tests__/live-voice-tts-session.test.ts +0 -483
- package/src/live-voice/__tests__/live-voice-vad.test.ts +0 -16
- package/src/live-voice/__tests__/progress-narration.test.ts +214 -0
- package/src/live-voice/live-voice-archive.ts +2 -0
- package/src/live-voice/live-voice-manager.ts +16 -3
- package/src/live-voice/live-voice-metrics.ts +57 -32
- package/src/live-voice/live-voice-photo.ts +1 -2
- package/src/live-voice/live-voice-session.ts +598 -323
- package/src/live-voice/progress-narration.ts +277 -0
- package/src/live-voice/protocol.ts +21 -1
- package/src/live-voice/windows-compiled-live-voice.ts +4 -0
- package/src/mcp/__tests__/effective-config.test.ts +238 -0
- package/src/mcp/__tests__/mcp-auth-orchestrator.test.ts +0 -1
- package/src/mcp/__tests__/mcp-oauth-client-registration.test.ts +200 -0
- package/src/mcp/__tests__/mcp-oauth-provider.test.ts +9 -9
- package/src/mcp/__tests__/plugin-server-credential-isolation.test.ts +95 -0
- package/src/mcp/client.ts +16 -11
- package/src/mcp/effective-config.ts +113 -0
- package/src/mcp/manager.ts +11 -6
- package/src/mcp/mcp-auth-orchestrator.ts +13 -22
- package/src/mcp/mcp-oauth-provider.ts +205 -240
- package/src/monitoring/__tests__/plugin-auto-update.test.ts +166 -3
- package/src/monitoring/control.ts +1 -0
- package/src/monitoring/db-integrity-sample.ts +4 -5
- package/src/monitoring/plugin-auto-update.ts +128 -24
- package/src/notifications/AGENTS.md +2 -0
- package/src/notifications/approval-card-data.ts +33 -0
- package/src/notifications/signal.ts +1 -0
- package/src/permissions/AGENTS.md +16 -0
- package/src/permissions/checker.test.ts +75 -122
- package/src/permissions/checker.ts +44 -560
- package/src/permissions/confirmation-guardian-request.test.ts +15 -11
- package/src/permissions/confirmation-guardian-request.ts +2 -2
- package/src/permissions/prompter.ts +1 -5
- package/src/permissions/question-guardian-request.test.ts +14 -6
- package/src/permissions/question-guardian-request.ts +1 -2
- package/src/persistence/attachments-store.ts +8 -1
- package/src/persistence/bookmark-crud.ts +3 -7
- package/src/persistence/conversation-attention-store.ts +16 -45
- package/src/persistence/conversation-crud.ts +33 -4
- package/src/persistence/conversation-lineage.ts +9 -0
- package/src/persistence/conversation-queries.ts +182 -52
- package/src/persistence/delivery-crud.ts +150 -36
- package/src/persistence/embeddings/qdrant-manager.ts +84 -49
- package/src/persistence/external-conversation-store.ts +32 -4
- package/src/persistence/llm-request-log-store.ts +4 -10
- package/src/persistence/llm-usage-store.ts +8 -3
- package/src/persistence/message-reads.test.ts +197 -0
- package/src/persistence/message-reads.ts +211 -0
- package/src/persistence/migrations/360-add-document-workspace-path.ts +5 -14
- package/src/persistence/migrations/366-chatgpt-subscription-row-identity.test.ts +120 -0
- package/src/persistence/migrations/366-chatgpt-subscription-row-identity.ts +62 -0
- package/src/persistence/real-user-turn-filter.ts +27 -3
- package/src/persistence/schema/documents.ts +4 -4
- package/src/persistence/steps.ts +9 -0
- package/src/plugin-api/__tests__/oauth-callback-url-export.test.ts +29 -0
- package/src/plugin-api/constants.ts +8 -0
- package/src/plugin-api/conversation-turn.ts +195 -5
- package/src/plugin-api/index.ts +38 -6
- package/src/plugin-api/resolve-credential.ts +3 -2
- package/src/plugin-api/store-credential.ts +146 -0
- package/src/plugin-api/vision-support.test.ts +1 -1
- package/src/plugin-api/webhook-url.ts +13 -11
- package/src/plugins/__tests__/mcp-servers.test.ts +371 -0
- package/src/plugins/defaults/main.ts +21 -7
- package/src/plugins/defaults/memory/AGENTS.md +15 -3
- package/src/plugins/defaults/memory/__tests__/buffer-format.test.ts +204 -0
- package/src/plugins/defaults/memory/__tests__/memory-retrospective-accounting.test.ts +72 -0
- package/src/plugins/defaults/memory/__tests__/memory-retrospective-job.test.ts +4 -1
- package/src/plugins/defaults/memory/__tests__/memory-retrospective-provider-path.test.ts +4 -1
- package/src/plugins/defaults/memory/__tests__/memory-tier-boundary-guard.test.ts +0 -1
- package/src/plugins/defaults/memory/buffer-format.ts +165 -0
- package/src/plugins/defaults/memory/context-search/sources/conversations.ts +6 -0
- package/src/plugins/defaults/memory/graph/__tests__/conversation-graph-memory-v2-routing.test.ts +77 -0
- package/src/plugins/defaults/memory/graph/conversation-graph-memory.ts +24 -6
- package/src/plugins/defaults/memory/graph/image-ref-utils.ts +3 -0
- package/src/plugins/defaults/memory/graph/tool-handlers.ts +1 -30
- package/src/plugins/defaults/memory/graph-topology/pending-buffer.test.ts +34 -0
- package/src/plugins/defaults/memory/graph-topology/pending-buffer.ts +17 -24
- package/src/plugins/defaults/memory/hooks/post-compact.ts +1 -4
- package/src/plugins/defaults/memory/hooks/user-prompt-submit.ts +53 -5
- package/src/plugins/defaults/memory/indexer.ts +3 -1
- package/src/plugins/defaults/memory/memory-retrospective-accounting.ts +19 -7
- package/src/plugins/defaults/memory/memory-retrospective-job.ts +4 -4
- package/src/plugins/defaults/memory/src/__tests__/memory-v3-gate-stats.test.ts +281 -0
- package/src/plugins/defaults/memory/src/memory-v2-routes.ts +12 -10
- package/src/plugins/defaults/memory/src/memory-v3-routes.ts +207 -0
- package/src/plugins/defaults/memory/substrate/__tests__/consolidation-job.test.ts +147 -0
- package/src/plugins/defaults/memory/substrate/__tests__/consolidation-prompt-flag-gating-guard.test.ts +8 -0
- package/src/plugins/defaults/memory/substrate/__tests__/edge-index.test.ts +0 -30
- package/src/plugins/defaults/memory/substrate/__tests__/ingest.test.ts +73 -1
- package/src/plugins/defaults/memory/substrate/__tests__/page-index.test.ts +37 -0
- package/src/plugins/defaults/memory/substrate/__tests__/page-links.test.ts +208 -0
- package/src/plugins/defaults/memory/substrate/__tests__/prompts-consolidation.test.ts +158 -1
- package/src/plugins/defaults/memory/substrate/__tests__/static-context.test.ts +195 -58
- package/src/plugins/defaults/memory/substrate/consolidation-job.ts +93 -24
- package/src/plugins/defaults/memory/substrate/edge-index.ts +0 -31
- package/src/plugins/defaults/memory/substrate/ingest.ts +59 -0
- package/src/plugins/defaults/memory/substrate/page-index.ts +35 -8
- package/src/plugins/defaults/memory/substrate/page-links.ts +133 -0
- package/src/plugins/defaults/memory/substrate/page-store.ts +10 -0
- package/src/plugins/defaults/memory/substrate/prompts/consolidation.ts +124 -34
- package/src/plugins/defaults/memory/substrate/skill-content.ts +8 -1
- package/src/plugins/defaults/memory/substrate/static-context.ts +160 -33
- package/src/plugins/defaults/memory/substrate/sweep-job.ts +2 -4
- package/src/plugins/defaults/memory/v1/graph/extraction.ts +3 -1
- package/src/plugins/defaults/memory/v3/__tests__/injection.test.ts +18 -0
- package/src/plugins/defaults/memory/v3/__tests__/shadow-plugin.test.ts +66 -1
- package/src/plugins/defaults/memory/v3/card.ts +9 -9
- package/src/plugins/defaults/memory/v3/core-set.test.ts +7 -0
- package/src/plugins/defaults/memory/v3/core-set.ts +5 -1
- package/src/plugins/defaults/memory/v3/edge.ts +13 -57
- package/src/plugins/defaults/memory/v3/injector.ts +8 -0
- package/src/plugins/defaults/memory/v3/prune.ts +2 -0
- package/src/plugins/defaults/memory/v3/selection-log-store.ts +2 -0
- package/src/plugins/defaults/memory/v3/shadow-plugin.ts +23 -9
- package/src/plugins/defaults/memory/worker-control.ts +1 -0
- package/src/plugins/defaults/memory/worker.ts +6 -3
- package/src/plugins/defaults/worker-entrypoints.ts +3 -0
- package/src/plugins/external-plugin-loader.ts +47 -0
- package/src/plugins/mcp-servers.ts +361 -0
- package/src/plugins/mtime-cache.ts +37 -46
- package/src/plugins/pipeline.ts +14 -7
- package/src/plugins/plugin-execution-context.ts +35 -11
- package/src/plugins/worker-plugin-surface.ts +33 -0
- package/src/prompts/templates/system-sections.ts +0 -7
- package/src/providers/__tests__/connection-model-compat.test.ts +1 -1
- package/src/providers/__tests__/context-overflow-error.test.ts +24 -0
- package/src/providers/__tests__/dispatch-connection-routing.test.ts +214 -2
- package/src/providers/__tests__/preflight-resolved-config.test.ts +57 -0
- package/src/providers/__tests__/retry-callsite.test.ts +25 -2
- package/src/providers/anthropic/__tests__/pause-turn-continuation.test.ts +170 -0
- package/src/providers/anthropic/client.ts +332 -241
- package/src/providers/call-site-routing.ts +30 -3
- package/src/providers/connection-resolution.ts +194 -11
- package/src/providers/inference/__tests__/adapter-factory-openai-compatible.test.ts +4 -0
- package/src/providers/inference/adapter-factory.ts +3 -0
- package/src/providers/inference/auth.ts +6 -0
- package/src/providers/inference/connection-availability.ts +24 -2
- package/src/providers/inference/connections.ts +2 -0
- package/src/providers/model-catalog.ts +3 -3
- package/src/providers/model-intents.ts +28 -8
- package/src/providers/openai/__tests__/chat-completions-provider-reasoning.test.ts +163 -40
- package/src/providers/openai/__tests__/tool-choice-mapping.test.ts +125 -2
- package/src/providers/openai/chat-completions-provider.ts +133 -36
- package/src/providers/openai/codex-models.ts +2 -1
- package/src/providers/provider-send-message.ts +32 -3
- package/src/providers/speech-to-text/__tests__/deepgram-flux-frames.test.ts +433 -0
- package/src/providers/speech-to-text/__tests__/deepgram-flux-realtime.test.ts +625 -0
- package/src/providers/speech-to-text/__tests__/provider-catalog.test.ts +34 -0
- package/src/providers/speech-to-text/__tests__/resolve.test.ts +285 -6
- package/src/providers/speech-to-text/deepgram-flux-frames.ts +395 -0
- package/src/providers/speech-to-text/deepgram-flux-realtime.ts +679 -0
- package/src/providers/speech-to-text/provider-catalog.ts +99 -8
- package/src/providers/speech-to-text/resolve.ts +25 -2
- package/src/routes/control.ts +1 -0
- package/src/routes/route-host-client.ts +1 -0
- package/src/routes/route-host-protocol.ts +7 -0
- package/src/routes/worker.ts +48 -15
- package/src/runtime/AGENTS.md +16 -17
- package/src/runtime/access-request-helper.ts +9 -12
- package/src/runtime/agent-wake.ts +18 -15
- package/src/runtime/effective-capabilities.test.ts +43 -0
- package/src/runtime/effective-capabilities.ts +27 -0
- package/src/runtime/guardian-reply-router.ts +6 -2
- package/src/runtime/pre-first-message-gate.ts +4 -0
- package/src/runtime/routes/__tests__/acp-claude-auth-routes.test.ts +12 -4
- package/src/runtime/routes/__tests__/conversation-list-routes.test.ts +522 -2
- package/src/runtime/routes/__tests__/conversation-query-routes.test.ts +52 -0
- package/src/runtime/routes/__tests__/default-provider-routes.test.ts +61 -0
- package/src/runtime/routes/__tests__/inference-profiles-routes.test.ts +44 -0
- package/src/runtime/routes/__tests__/inference-provider-connection-routes.test.ts +102 -1
- package/src/runtime/routes/__tests__/plugins-routes.test.ts +132 -35
- package/src/runtime/routes/__tests__/schedule-routes-disarm-reason.test.ts +215 -0
- package/src/runtime/routes/__tests__/stt-routes.test.ts +25 -0
- package/src/runtime/routes/__tests__/surface-action-routes.test.ts +55 -6
- package/src/runtime/routes/__tests__/user-route-dispatcher-host.test.ts +52 -1
- package/src/runtime/routes/__tests__/user-route-dispatcher.test.ts +133 -1
- package/src/runtime/routes/channel-availability-routes.ts +32 -14
- package/src/runtime/routes/channel-route-shared.ts +0 -6
- package/src/runtime/routes/chatgpt-subscription-auth-routes.ts +6 -6
- package/src/runtime/routes/conversation-list-routes.ts +181 -23
- package/src/runtime/routes/conversation-management-routes.ts +17 -12
- package/src/runtime/routes/conversation-query-routes.ts +40 -27
- package/src/runtime/routes/conversation-routes.ts +43 -29
- package/src/runtime/routes/credential-prompt-routes.ts +4 -7
- package/src/runtime/routes/credential-routes.ts +26 -82
- package/src/runtime/routes/default-provider-routes.ts +15 -0
- package/src/runtime/routes/documents-routes.ts +3 -222
- package/src/runtime/routes/inbound-message-handler.ts +21 -45
- package/src/runtime/routes/inbound-stages/acl-enforcement.test.ts +0 -1
- package/src/runtime/routes/inbound-stages/acl-enforcement.ts +0 -9
- package/src/runtime/routes/inbound-stages/admission-policy.ts +1 -17
- package/src/runtime/routes/inbound-stages/bootstrap-intercept.test.ts +0 -1
- package/src/runtime/routes/inbound-stages/bootstrap-intercept.ts +2 -3
- package/src/runtime/routes/inbound-stages/edit-intercept.ts +101 -79
- package/src/runtime/routes/inbound-stages/guardian-reply-intercept.test.ts +0 -1
- package/src/runtime/routes/inbound-stages/guardian-reply-intercept.ts +12 -7
- package/src/runtime/routes/inbound-stages/reaction-intercept.test.ts +91 -8
- package/src/runtime/routes/inbound-stages/reaction-intercept.ts +72 -79
- package/src/runtime/routes/inbound-stages/secret-ingress-check.ts +2 -3
- package/src/runtime/routes/inference-profiles-routes.ts +20 -11
- package/src/runtime/routes/inference-provider-connection-routes.ts +77 -15
- package/src/runtime/routes/log-export-routes.ts +3 -0
- package/src/runtime/routes/mcp-auth-routes.ts +148 -57
- package/src/runtime/routes/playground/__tests__/inject-failures.test.ts +2 -0
- package/src/runtime/routes/playground/__tests__/reset-circuit.test.ts +3 -0
- package/src/runtime/routes/playground/inject-failures.ts +2 -2
- package/src/runtime/routes/playground/reset-circuit.ts +1 -1
- package/src/runtime/routes/plugins-routes.ts +38 -8
- package/src/runtime/routes/schedule-routes.ts +94 -7
- package/src/runtime/routes/settings-routes.ts +6 -6
- package/src/runtime/routes/stt-routes.ts +31 -25
- package/src/runtime/routes/surface-action-routes.ts +20 -19
- package/src/runtime/routes/surface-conversation-resolver.ts +3 -0
- package/src/runtime/routes/user-route-dispatcher.ts +69 -15
- package/src/runtime/routes/user-route-import.ts +108 -0
- package/src/runtime/routes/user-route-resolution.ts +18 -1
- package/src/runtime/routes/workspace-routes.ts +0 -9
- package/src/runtime/routes/workspace-utils.ts +3 -13
- package/src/runtime/services/conversation-serializer.ts +7 -2
- package/src/schedule/__tests__/plugin-schedule-declarations.test.ts +68 -5
- package/src/schedule/__tests__/plugin-schedule-reconciler.test.ts +81 -0
- package/src/schedule/plugin-schedule-availability.ts +58 -0
- package/src/schedule/plugin-schedule-declarations.ts +23 -27
- package/src/schedule/plugin-schedule-reconciler.ts +12 -3
- package/src/schedule/schedule-store.ts +5 -1
- package/src/schedule/scheduler.ts +9 -4
- package/src/schedule/worker-control.ts +1 -0
- package/src/schedule/worker.ts +6 -0
- package/src/security/oauth2.ts +6 -22
- package/src/stt/__tests__/daemon-batch-transcriber.test.ts +22 -0
- package/src/stt/__tests__/types.test.ts +94 -0
- package/src/stt/daemon-batch-transcriber.ts +10 -0
- package/src/stt/stt-stream-session.ts +8 -4
- package/src/stt/types.ts +103 -0
- package/src/subagent/__tests__/consult-prompt.test.ts +26 -15
- package/src/subagent/consult-context.ts +11 -11
- package/src/subagent/consult-prompt.ts +26 -35
- package/src/subagent/manager.ts +21 -40
- package/src/subagent/notify.ts +7 -1
- package/src/subagent/types.ts +15 -12
- package/src/tools/__tests__/tool-input-schemas.test.ts +7 -7
- package/src/tools/acp/spawn.test.ts +97 -0
- package/src/tools/acp/spawn.ts +38 -4
- package/src/tools/credentials/ref-parse.ts +35 -0
- package/src/tools/credentials/resolve.ts +4 -10
- package/src/tools/credentials/store.ts +168 -0
- package/src/tools/document/document-tool.ts +12 -3
- package/src/tools/executor.ts +53 -15
- package/src/tools/filesystem/read.ts +27 -10
- package/src/tools/host-filesystem/read.ts +27 -15
- package/src/tools/network/url-safety.ts +9 -16
- package/src/tools/permission-checker.ts +43 -52
- package/src/tools/registry.ts +2 -1
- package/src/tools/shared/filesystem/file-ops-service.ts +63 -35
- package/src/tools/shared/filesystem/legacy-read-args.ts +22 -0
- package/src/tools/shared/filesystem/types.ts +5 -5
- package/src/tools/skills/scaffold-managed.ts +25 -7
- package/src/tools/subagent/spawn.ts +28 -88
- package/src/tools/tool-types.ts +26 -13
- package/src/tools/types.ts +6 -6
- package/src/tools/ui-surface/surface-shape-docs.ts +12 -1
- package/src/tools/workflows/run-workflow.ts +1 -2
- package/src/tts/__tests__/reasoning-tag-filter.test.ts +78 -0
- package/src/tts/reasoning-tag-filter.ts +89 -0
- package/src/util/__tests__/worker-process-command.test.ts +37 -0
- package/src/util/logger.ts +16 -0
- package/src/util/think-tag-stream.ts +95 -0
- package/src/util/worker-process.ts +37 -4
- package/src/windows-compiled-cli.ts +32 -0
- package/src/windows-compiled-entry.ts +4 -0
- package/src/windows-compiled-logger.ts +6 -0
- package/src/windows-compiled-worker-entry.ts +29 -0
- package/src/workspace/byok-default-profile-ensure.ts +76 -24
- package/src/workspace/custom-profile-ensure.ts +4 -24
- package/src/workspace/migrations/142-consolidate-voice-front-door.ts +70 -0
- package/src/workspace/migrations/143-repair-deprecated-codex-model-id.ts +134 -0
- package/src/workspace/migrations/144-convert-stranded-subscription-openai-profiles.ts +265 -0
- package/src/workspace/migrations/145-collapse-profile-bindings-to-entries.ts +328 -0
- package/src/workspace/migrations/146-repair-retired-fireworks-deepseek-flash-model-id.ts +195 -0
- package/src/workspace/migrations/__tests__/141-stt-english-default-to-multilingual.test.ts +0 -10
- package/src/workspace/migrations/registry.ts +10 -0
- package/src/workspace/provider-commit-message-generator.ts +7 -5
- package/src/__tests__/document-workspace-file.test.ts +0 -467
- package/src/live-voice/__tests__/front-decision.test.ts +0 -645
- package/src/live-voice/front-decision.ts +0 -476
- package/src/permissions/ipc-risk-types.ts +0 -143
- package/src/permissions/risk-types.ts +0 -76
- package/src/subagent/__tests__/consult-transcript.test.ts +0 -184
- package/src/subagent/consult-transcript.ts +0 -90
|
@@ -4,6 +4,7 @@ import { DEEPGRAM_MULTI_LANGUAGE_CODES } from "../../providers/speech-to-text/de
|
|
|
4
4
|
import {
|
|
5
5
|
capEscalationBridge,
|
|
6
6
|
classifyFrontDoorLeading,
|
|
7
|
+
createFrontDoorStreamGate,
|
|
7
8
|
ESCALATE_VERDICT_TOKEN,
|
|
8
9
|
escalatedContinuationRule,
|
|
9
10
|
ESCALATION_CONTINUATION_CONTENT,
|
|
@@ -84,6 +85,14 @@ describe("front-door decision rule", () => {
|
|
|
84
85
|
expect(rule.toLowerCase()).toContain("tool");
|
|
85
86
|
});
|
|
86
87
|
|
|
88
|
+
test("escalates when an answer needs personal memory absent from context", () => {
|
|
89
|
+
expect(rule.toLowerCase()).toContain("saved personal fact");
|
|
90
|
+
expect(rule.toLowerCase()).toContain("escalate rather than guessing");
|
|
91
|
+
expect(rule.toLowerCase()).toContain(
|
|
92
|
+
"personal context that is already present is yours to use directly",
|
|
93
|
+
);
|
|
94
|
+
});
|
|
95
|
+
|
|
87
96
|
test("demands a silent decision — no narrated reasoning in spoken output", () => {
|
|
88
97
|
// Regression: a weak front-door model narrated its triage deliberation
|
|
89
98
|
// aloud ("Context is complete — Alex paused...") before the bridge.
|
|
@@ -439,3 +448,96 @@ describe("live-voice escalation orchestration (end-to-end — TODO)", () => {
|
|
|
439
448
|
() => {},
|
|
440
449
|
);
|
|
441
450
|
});
|
|
451
|
+
|
|
452
|
+
describe("createFrontDoorStreamGate", () => {
|
|
453
|
+
/** Feed `deltas` through a gate and return what it released, in order. */
|
|
454
|
+
function release(
|
|
455
|
+
deltas: string[],
|
|
456
|
+
opts?: { holdEnabled?: boolean; finish?: boolean },
|
|
457
|
+
): string[] {
|
|
458
|
+
const gate = createFrontDoorStreamGate(opts?.holdEnabled === true);
|
|
459
|
+
const out = deltas
|
|
460
|
+
.map((delta) => gate.push(delta))
|
|
461
|
+
.filter((chunk) => chunk.length > 0);
|
|
462
|
+
if (opts?.finish !== false) {
|
|
463
|
+
const trailing = gate.finish();
|
|
464
|
+
if (trailing.length > 0) {
|
|
465
|
+
out.push(trailing);
|
|
466
|
+
}
|
|
467
|
+
}
|
|
468
|
+
return out;
|
|
469
|
+
}
|
|
470
|
+
|
|
471
|
+
test("an answer passes through delta by delta", () => {
|
|
472
|
+
expect(release(["It is ", "Tuesday."])).toEqual(["It is ", "Tuesday."]);
|
|
473
|
+
});
|
|
474
|
+
|
|
475
|
+
test("leading text held while the verdict was pending is not lost", () => {
|
|
476
|
+
expect(release(["[", "A] is the option."])).toEqual(["[A] is the option."]);
|
|
477
|
+
});
|
|
478
|
+
|
|
479
|
+
test("an escalation releases the capped bridge and nothing else", () => {
|
|
480
|
+
expect(
|
|
481
|
+
release([
|
|
482
|
+
ESCALATE_VERDICT_TOKEN,
|
|
483
|
+
" Let me check your calendar.",
|
|
484
|
+
" rambling past the cap",
|
|
485
|
+
]),
|
|
486
|
+
).toEqual(["Let me check your calendar."]);
|
|
487
|
+
});
|
|
488
|
+
|
|
489
|
+
test("a verdict token split across deltas still never escapes", () => {
|
|
490
|
+
expect(release(["[", "1", "] One moment.", " more"])).toEqual([
|
|
491
|
+
"One moment.",
|
|
492
|
+
]);
|
|
493
|
+
});
|
|
494
|
+
|
|
495
|
+
test("a too-short bridge releases nothing, matching the audio-only fallback", () => {
|
|
496
|
+
// Under MIN_SPOKEN_BRIDGE_CHARS the session discards the model's bridge
|
|
497
|
+
// and plays a canned fallback with no caption and no persisted row, so
|
|
498
|
+
// releasing the capped text would show words the caller never heard.
|
|
499
|
+
expect(release([ESCALATE_VERDICT_TOKEN, " ok"])).toEqual([]);
|
|
500
|
+
expect(release([ESCALATE_VERDICT_TOKEN, " ."])).toEqual([]);
|
|
501
|
+
});
|
|
502
|
+
|
|
503
|
+
test("a too-short bridge releases nothing via finish either", () => {
|
|
504
|
+
const gate = createFrontDoorStreamGate(false);
|
|
505
|
+
expect(gate.push(`${ESCALATE_VERDICT_TOKEN} ok`)).toBe("");
|
|
506
|
+
expect(gate.finish()).toBe("");
|
|
507
|
+
});
|
|
508
|
+
|
|
509
|
+
test("a bridge at the spoken threshold is still released", () => {
|
|
510
|
+
// The boundary belongs to the spoken side: the session speaks anything
|
|
511
|
+
// that is not BELOW the threshold, so the gate must release it too.
|
|
512
|
+
expect(release([ESCALATE_VERDICT_TOKEN, " Sure."])).toEqual(["Sure."]);
|
|
513
|
+
});
|
|
514
|
+
|
|
515
|
+
test("a bridge with no terminator is released by finish", () => {
|
|
516
|
+
const gate = createFrontDoorStreamGate(false);
|
|
517
|
+
expect(gate.push("[1] Let me check")).toBe("");
|
|
518
|
+
expect(gate.finish()).toBe("Let me check");
|
|
519
|
+
});
|
|
520
|
+
|
|
521
|
+
test("a bridge is dropped when the leg never finishes (cancellation)", () => {
|
|
522
|
+
expect(release(["[1] Let me check"], { finish: false })).toEqual([]);
|
|
523
|
+
});
|
|
524
|
+
|
|
525
|
+
test("a hold verdict releases nothing", () => {
|
|
526
|
+
expect(release([HOLD_VERDICT_TOKEN], { holdEnabled: true })).toEqual([]);
|
|
527
|
+
});
|
|
528
|
+
|
|
529
|
+
test("the hold token is ordinary text on a leg that was never taught it", () => {
|
|
530
|
+
// A non-speculative leg's prompt has no hold branch, so its output must
|
|
531
|
+
// not be swallowed by a token it could only have parroted.
|
|
532
|
+
expect(release([`${HOLD_VERDICT_TOKEN} is the index.`])).toEqual([
|
|
533
|
+
`${HOLD_VERDICT_TOKEN} is the index.`,
|
|
534
|
+
]);
|
|
535
|
+
});
|
|
536
|
+
|
|
537
|
+
test("nothing more escapes after the bridge is released", () => {
|
|
538
|
+
const gate = createFrontDoorStreamGate(false);
|
|
539
|
+
expect(gate.push("[1] One moment.")).toBe("One moment.");
|
|
540
|
+
expect(gate.push(" and a leaked continuation")).toBe("");
|
|
541
|
+
expect(gate.finish()).toBe("");
|
|
542
|
+
});
|
|
543
|
+
});
|
|
@@ -21,6 +21,7 @@ import type { TrustContext } from "../daemon/trust-context-types.js";
|
|
|
21
21
|
import { DAEMON_INTERNAL_ASSISTANT_ID } from "../runtime/assistant-scope.js";
|
|
22
22
|
import { computeToolApprovalDigest } from "../security/tool-approval-digest.js";
|
|
23
23
|
import { getCatalogProvider } from "../tts/provider-catalog.js";
|
|
24
|
+
import { createReasoningTagFilter } from "../tts/reasoning-tag-filter.js";
|
|
24
25
|
import { extractSpeakableSegments } from "../tts/speakable-segments.js";
|
|
25
26
|
import {
|
|
26
27
|
type AudioStoreSink,
|
|
@@ -728,6 +729,12 @@ export class CallController {
|
|
|
728
729
|
// could be the start of a control marker.
|
|
729
730
|
let ttsBuffer = "";
|
|
730
731
|
let fullResponseText = "";
|
|
732
|
+
// Reasoning models can inline <think> spans in the content stream when a
|
|
733
|
+
// profile has not opted into parseThinkTags. Neither the spoken path nor
|
|
734
|
+
// the post-turn consumers of fullResponseText (transcripts,
|
|
735
|
+
// assistant_spoke, END_CALL/ASK_GUARDIAN detection) may see them: both
|
|
736
|
+
// are fed only filtered text.
|
|
737
|
+
const reasoningFilter = createReasoningTagFilter();
|
|
731
738
|
|
|
732
739
|
// Synthesized path: text is split at speakable boundaries as it streams
|
|
733
740
|
// and each segment is synthesized while the LLM keeps generating. The
|
|
@@ -930,8 +937,13 @@ export class CallController {
|
|
|
930
937
|
if (!this.isCurrentRun(runVersion)) {
|
|
931
938
|
return;
|
|
932
939
|
}
|
|
933
|
-
|
|
934
|
-
|
|
940
|
+
// One filter feeds both consumers: the spoken stream and the text
|
|
941
|
+
// used for transcripts, assistant_spoke, and END_CALL/ASK_GUARDIAN
|
|
942
|
+
// marker detection. A control marker inside a reasoning span must
|
|
943
|
+
// never trigger a real action the caller did not hear.
|
|
944
|
+
const speakable = reasoningFilter.push(text);
|
|
945
|
+
fullResponseText += speakable;
|
|
946
|
+
ttsBuffer += speakable;
|
|
935
947
|
ttsBuffer = stripInternalSpeechMarkers(ttsBuffer);
|
|
936
948
|
flushSafeText();
|
|
937
949
|
};
|
|
@@ -1010,7 +1022,11 @@ export class CallController {
|
|
|
1010
1022
|
return fullResponseText;
|
|
1011
1023
|
}
|
|
1012
1024
|
|
|
1013
|
-
// Final sweep:
|
|
1025
|
+
// Final sweep: release any held-back partial tag to both consumers,
|
|
1026
|
+
// then strip any remaining control markers from the buffer.
|
|
1027
|
+
const filterTail = reasoningFilter.flush();
|
|
1028
|
+
fullResponseText += filterTail;
|
|
1029
|
+
ttsBuffer += filterTail;
|
|
1014
1030
|
ttsBuffer = stripInternalSpeechMarkers(ttsBuffer);
|
|
1015
1031
|
if (ttsBuffer.length > 0) {
|
|
1016
1032
|
emitSafeChunk(ttsBuffer);
|
|
@@ -1393,7 +1393,6 @@ export class CallSetupFlow {
|
|
|
1393
1393
|
|
|
1394
1394
|
try {
|
|
1395
1395
|
const accessResult = await ndeps.notifyGuardianOfAccessRequest({
|
|
1396
|
-
canonicalAssistantId: accessRequest.assistantId,
|
|
1397
1396
|
sourceChannel: "phone",
|
|
1398
1397
|
conversationExternalId: accessRequest.fromNumber,
|
|
1399
1398
|
actorExternalId: accessRequest.fromNumber,
|
|
@@ -538,6 +538,17 @@ export class MediaStreamSttSession {
|
|
|
538
538
|
switch (event.type) {
|
|
539
539
|
case "partial":
|
|
540
540
|
return;
|
|
541
|
+
case "finalized":
|
|
542
|
+
return;
|
|
543
|
+
case "turn-start":
|
|
544
|
+
case "eager-turn-end":
|
|
545
|
+
case "turn-resumed":
|
|
546
|
+
case "turn-end":
|
|
547
|
+
// The local VAD and the provider's boundary finals own turn
|
|
548
|
+
// taking. Listed case-by-case rather than under `default` so the
|
|
549
|
+
// exhaustiveness check flags this site when turn detection is
|
|
550
|
+
// wired in.
|
|
551
|
+
return;
|
|
541
552
|
case "final": {
|
|
542
553
|
const durationMs = Math.round(this.utteranceAudioMs);
|
|
543
554
|
this.utteranceAudioMs = 0;
|
|
@@ -570,6 +581,10 @@ export class MediaStreamSttSession {
|
|
|
570
581
|
this.enterBatchMode();
|
|
571
582
|
}
|
|
572
583
|
return;
|
|
584
|
+
default: {
|
|
585
|
+
const _exhaustive: never = event;
|
|
586
|
+
return;
|
|
587
|
+
}
|
|
573
588
|
}
|
|
574
589
|
}
|
|
575
590
|
|
|
@@ -34,6 +34,7 @@ import {
|
|
|
34
34
|
recordConversationPersistedSeq,
|
|
35
35
|
updateMessageContent,
|
|
36
36
|
} from "../persistence/conversation-crud.js";
|
|
37
|
+
import { VOICE_ESCALATION_CONTINUATION_MESSAGE_KIND } from "../plugin-api/constants.js";
|
|
37
38
|
import { pinnedListeningLanguage } from "../providers/speech-to-text/provider-catalog.js";
|
|
38
39
|
import type { ContentBlock } from "../providers/types.js";
|
|
39
40
|
import { broadcastMessage } from "../runtime/assistant-event-hub.js";
|
|
@@ -55,6 +56,7 @@ import {
|
|
|
55
56
|
stripInternalSpeechMarkers,
|
|
56
57
|
} from "./voice-control-protocol.js";
|
|
57
58
|
import {
|
|
59
|
+
createFrontDoorStreamGate,
|
|
58
60
|
escalatedContinuationRule,
|
|
59
61
|
ESCALATION_CONTINUATION_CONTENT,
|
|
60
62
|
frontDoorCapabilityDigest,
|
|
@@ -407,18 +409,6 @@ export interface VoiceTurnHandle {
|
|
|
407
409
|
* on a live phone call" framing (the session system prompt already
|
|
408
410
|
* provides assistant identity) and guardian context (injected separately).
|
|
409
411
|
*/
|
|
410
|
-
/**
|
|
411
|
-
* Steering shared by every voice channel. A sign-in flow opens a browser
|
|
412
|
-
* window mid-call that the caller may be unable to see or complete, whether it
|
|
413
|
-
* is reached through a ui-surface tool or through shell and CLI tools (e.g.
|
|
414
|
-
* `assistant oauth connect`). Tell the model to speak the limitation and defer
|
|
415
|
-
* the flow to text chat instead.
|
|
416
|
-
*
|
|
417
|
-
* This outlives the ui-surface restriction it was written alongside: a
|
|
418
|
-
* live-voice call can now show surfaces, but a browser window handing control
|
|
419
|
-
* to a third party mid-call is a different problem, and one a minimized room
|
|
420
|
-
* does not solve.
|
|
421
|
-
*/
|
|
422
412
|
/**
|
|
423
413
|
* How long a live-voice call waits on an approval before deciding it itself.
|
|
424
414
|
*
|
|
@@ -429,8 +419,23 @@ export interface VoiceTurnHandle {
|
|
|
429
419
|
*/
|
|
430
420
|
const VOICE_APPROVAL_TIMEOUT_MS = 45_000;
|
|
431
421
|
|
|
432
|
-
|
|
433
|
-
|
|
422
|
+
/**
|
|
423
|
+
* Telephony-only steering. A sign-in flow opens a browser window on a screen
|
|
424
|
+
* the caller does not have in front of them, whether it is reached through a
|
|
425
|
+
* ui-surface tool or through shell and CLI tools (e.g. `assistant oauth
|
|
426
|
+
* connect`). Tell the model to speak the limitation and defer the flow to text
|
|
427
|
+
* chat instead.
|
|
428
|
+
*
|
|
429
|
+
* Scoped to the phone because the screen is what decides it. A phone call has
|
|
430
|
+
* no screen, so the `open_url` signal a CLI tool can reach lands somewhere the
|
|
431
|
+
* caller will never see, and that signal bus carries no capability or
|
|
432
|
+
* conversation context, so this rule is the only thing standing in front of it
|
|
433
|
+
* here. A live-voice call is the opposite case: the user is holding the screen,
|
|
434
|
+
* and the room minimizes itself to hand it back (see
|
|
435
|
+
* LIVE_VOICE_SETUP_FLOW_TEACHING).
|
|
436
|
+
*/
|
|
437
|
+
const PHONE_NO_SETUP_FLOWS_RULE =
|
|
438
|
+
"Never start account connections, OAuth or sign-in flows, or any other action that opens a browser window or needs the user's screen during this call, not even through shell or CLI tools. If the task needs one, say so briefly and offer to finish it in text chat after the call.";
|
|
434
439
|
|
|
435
440
|
/**
|
|
436
441
|
* The pre-speech tail of the speak-the-caller's-language rule. A monolingual
|
|
@@ -551,7 +556,7 @@ function buildVoiceCallControlPrompt(opts: {
|
|
|
551
556
|
'10. Do not make up information. If you are unsure, use [ASK_GUARDIAN: your question] to consult your guardian. For tool permission requests, use [ASK_GUARDIAN_APPROVAL: {"question":"...","toolName":"...","input":{...}}].',
|
|
552
557
|
`11. Your text is sent directly to a text-to-speech engine. Never use markdown formatting (asterisks, headers, backticks, links) or emojis in your spoken responses. Write plain conversational text only. Protocol markers like ${opts.isCallerGuardian ? "[END_CALL]" : "[ASK_GUARDIAN: ...] and [END_CALL]"} are not spoken text and should still be used normally.`,
|
|
553
558
|
`12. Speak the caller's language: reply in the language of the caller's most recent actual speech, and follow them if they switch languages mid-call. Synthetic user turns (parenthetical markers like the call-connected and verification-completed notices) are not caller speech and never set the language. Before the caller has spoken, such as on the opening greeting turn, ${preSpeechLanguageRuleFragment(config.services.stt.language, config.services.stt.provider)}.`,
|
|
554
|
-
`13. ${
|
|
559
|
+
`13. ${PHONE_NO_SETUP_FLOWS_RULE}`,
|
|
555
560
|
);
|
|
556
561
|
|
|
557
562
|
// Triage-and-escalate routing rules. The front-door leg decides and may
|
|
@@ -779,11 +784,13 @@ export async function startVoiceTurn(
|
|
|
779
784
|
// model wakes, but they are not user speech and must not render as a live
|
|
780
785
|
// user bubble. Their echo is suppressed below (parity with
|
|
781
786
|
// `isEchoSuppressedUserMessage` on the text path).
|
|
787
|
+
const isEscalationContinuation =
|
|
788
|
+
opts.content === ESCALATION_CONTINUATION_CONTENT;
|
|
782
789
|
const isSyntheticVoicePrompt =
|
|
783
790
|
opts.hiddenSyntheticPrompt === true ||
|
|
784
791
|
opts.content === CALL_OPENING_MARKER ||
|
|
785
792
|
opts.content === CALL_VERIFICATION_COMPLETE_MARKER ||
|
|
786
|
-
|
|
793
|
+
isEscalationContinuation;
|
|
787
794
|
|
|
788
795
|
// The escalation-continuation prompt is a pure internal instruction ("give
|
|
789
796
|
// the full answer now"), not a real utterance and not the sort of scaffolding
|
|
@@ -795,8 +802,7 @@ export async function startVoiceTurn(
|
|
|
795
802
|
// affects client display. A caller whose prompt text is not a fixed sentinel
|
|
796
803
|
// opts into the same treatment with `hiddenSyntheticPrompt`.
|
|
797
804
|
const isHiddenSyntheticPrompt =
|
|
798
|
-
opts.hiddenSyntheticPrompt === true ||
|
|
799
|
-
opts.content === ESCALATION_CONTINUATION_CONTENT;
|
|
805
|
+
opts.hiddenSyntheticPrompt === true || isEscalationContinuation;
|
|
800
806
|
|
|
801
807
|
// Build the call-control protocol prompt so the model knows how to emit
|
|
802
808
|
// control markers (ASK_GUARDIAN, END_CALL, etc.) and recognize opener turns.
|
|
@@ -930,10 +936,10 @@ export async function startVoiceTurn(
|
|
|
930
936
|
// failed while no concurrent turn held the lock). Paths where this turn
|
|
931
937
|
// LOST the conversation to a concurrent winner must use `restoreTurnState`
|
|
932
938
|
// instead — this reset-to-defaults would clobber the winner's live state.
|
|
933
|
-
// The
|
|
934
|
-
// (
|
|
935
|
-
//
|
|
936
|
-
let
|
|
939
|
+
// The approval observer is only detached when this turn installed it
|
|
940
|
+
// (`detachApprovalObserver` is set); otherwise cleanup would detach an
|
|
941
|
+
// observer installed by a prior turn.
|
|
942
|
+
let detachApprovalObserver: (() => void) | undefined;
|
|
937
943
|
/**
|
|
938
944
|
* Confirmations this voice turn left pending for the user to answer, and the
|
|
939
945
|
* timers that stop them waiting forever.
|
|
@@ -972,11 +978,10 @@ export async function startVoiceTurn(
|
|
|
972
978
|
conversation.setVoiceCallControlPrompt(null);
|
|
973
979
|
conversation.callSessionId = undefined;
|
|
974
980
|
conversation.forcePromptSideEffects = false;
|
|
975
|
-
|
|
976
|
-
|
|
977
|
-
|
|
978
|
-
|
|
979
|
-
}
|
|
981
|
+
// Detach so the stale closure doesn't act on events from future turns on
|
|
982
|
+
// the same conversation.
|
|
983
|
+
detachApprovalObserver?.();
|
|
984
|
+
detachApprovalObserver = undefined;
|
|
980
985
|
};
|
|
981
986
|
|
|
982
987
|
const requestId = uuidv7();
|
|
@@ -990,6 +995,9 @@ export async function startVoiceTurn(
|
|
|
990
995
|
// `isVoiceSessionUserMessage` for why the channel fields cannot carry it.
|
|
991
996
|
voiceSessionTurn: true,
|
|
992
997
|
...(isHiddenSyntheticPrompt ? { hidden: true } : {}),
|
|
998
|
+
...(isEscalationContinuation
|
|
999
|
+
? { messageKind: VOICE_ESCALATION_CONTINUATION_MESSAGE_KIND }
|
|
1000
|
+
: {}),
|
|
993
1001
|
...(opts.voiceTelemetry
|
|
994
1002
|
? {
|
|
995
1003
|
// Projected onto `TurnTelemetryEvent.client` by
|
|
@@ -1241,24 +1249,25 @@ export async function startVoiceTurn(
|
|
|
1241
1249
|
publishConversationMessagesChanged(opts.conversationId);
|
|
1242
1250
|
}
|
|
1243
1251
|
|
|
1244
|
-
//
|
|
1245
|
-
//
|
|
1252
|
+
// Observe confirmation_request and secret_request events so voice can
|
|
1253
|
+
// auto-deny/auto-allow/auto-resolve them (there is no interactive UI on a
|
|
1254
|
+
// call). Delivery is not this observer's job: the conversation's sink has
|
|
1255
|
+
// already put each event on the wire before the observer sees it, which is
|
|
1256
|
+
// what keeps a resolution's `interaction_resolved` after the request it
|
|
1257
|
+
// answers.
|
|
1246
1258
|
let lastError: string | null = null;
|
|
1247
|
-
|
|
1259
|
+
const handleVoiceEvent = async (msg: AssistantEvent): Promise<void> => {
|
|
1248
1260
|
// The user (or anything else) answered: stop the fallback from firing on a
|
|
1249
1261
|
// request that is already decided.
|
|
1250
1262
|
if (msg.type === "interaction_resolved") {
|
|
1251
1263
|
settleVoiceApproval(msg.requestId);
|
|
1252
1264
|
}
|
|
1253
1265
|
if (msg.type === "confirmation_request") {
|
|
1254
|
-
//
|
|
1255
|
-
//
|
|
1256
|
-
//
|
|
1257
|
-
//
|
|
1258
|
-
//
|
|
1259
|
-
// before `confirmation_request` on the wire, leaving an orphaned card
|
|
1260
|
-
// whose Allow/Deny buttons 404.
|
|
1261
|
-
broadcastMessage(msg);
|
|
1266
|
+
// The request is already on the wire (the sink delivered it before this
|
|
1267
|
+
// observer ran), so a resolution below lands after it: attached clients
|
|
1268
|
+
// (e.g. the web app behind a live-voice room) clear their approval card
|
|
1269
|
+
// only on `interaction_resolved`, and the reverse order would leave an
|
|
1270
|
+
// orphaned card whose Allow/Deny buttons 404.
|
|
1262
1271
|
if (!isGuardian) {
|
|
1263
1272
|
// Non-guardian voice callers have no interactive approval UI.
|
|
1264
1273
|
// The pre-exec gate (tool-approval-handler.ts) handles grant
|
|
@@ -1416,9 +1425,15 @@ export async function startVoiceTurn(
|
|
|
1416
1425
|
conversation.handleSecretResponse(msg.requestId, undefined, "store");
|
|
1417
1426
|
return;
|
|
1418
1427
|
}
|
|
1419
|
-
|
|
1428
|
+
};
|
|
1429
|
+
detachApprovalObserver = conversation.addEventObserver((msg) => {
|
|
1430
|
+
handleVoiceEvent(msg).catch((err) => {
|
|
1431
|
+
log.warn(
|
|
1432
|
+
{ turnId, eventType: msg.type, err },
|
|
1433
|
+
"Voice approval observer failed",
|
|
1434
|
+
);
|
|
1435
|
+
});
|
|
1420
1436
|
});
|
|
1421
|
-
clientCallbackInstalled = true;
|
|
1422
1437
|
|
|
1423
1438
|
// Registered before the agent loop starts so the NEXT turn on this
|
|
1424
1439
|
// conversation waits for this turn's `finally { cleanup() }` — not just
|
|
@@ -1447,6 +1462,34 @@ export async function startVoiceTurn(
|
|
|
1447
1462
|
// Set by the handle's discard(): the whole leg must leave no trace.
|
|
1448
1463
|
let discarded = false;
|
|
1449
1464
|
|
|
1465
|
+
// Verdict-first gate on the hub broadcast. A front-door leg's raw stream
|
|
1466
|
+
// carries its routing verdict, so hub subscribers (web, passive devices)
|
|
1467
|
+
// read it through the gate and see only the text the caller heard. Every
|
|
1468
|
+
// other leg, including the escalated continuation that answers for real,
|
|
1469
|
+
// broadcasts its deltas untouched.
|
|
1470
|
+
const frontDoorStreamGate =
|
|
1471
|
+
opts.routingLeg === "front-door"
|
|
1472
|
+
? createFrontDoorStreamGate(opts.unifiedVerdict === true)
|
|
1473
|
+
: null;
|
|
1474
|
+
|
|
1475
|
+
/**
|
|
1476
|
+
* Broadcast one agent-loop event to hub subscribers, holding a front-door
|
|
1477
|
+
* leg's control-plane text back at the boundary rather than emitting it and
|
|
1478
|
+
* repairing the transcript afterwards. Text released by the gate travels as
|
|
1479
|
+
* an ordinary delta on the leg's own reserved row, so a client that renders
|
|
1480
|
+
* the stream lands on the same text the teardown hygiene pass persists.
|
|
1481
|
+
*/
|
|
1482
|
+
const broadcastLegEvent = (msg: AssistantEvent): void => {
|
|
1483
|
+
if (frontDoorStreamGate === null || msg.type !== "assistant_text_delta") {
|
|
1484
|
+
broadcastMessage(msg);
|
|
1485
|
+
return;
|
|
1486
|
+
}
|
|
1487
|
+
const released = frontDoorStreamGate.push(msg.text);
|
|
1488
|
+
if (released.length > 0) {
|
|
1489
|
+
broadcastMessage({ ...msg, text: released });
|
|
1490
|
+
}
|
|
1491
|
+
};
|
|
1492
|
+
|
|
1450
1493
|
/**
|
|
1451
1494
|
* Teardown transcript hygiene. Runs after the agent loop has fully
|
|
1452
1495
|
* settled — including the stranded-content fold that finalizes an aborted
|
|
@@ -1474,6 +1517,12 @@ export async function startVoiceTurn(
|
|
|
1474
1517
|
* the escalated leg — blocked on this turn's teardown — snapshots it, so
|
|
1475
1518
|
* the quality model never sees the marker text either. Best-effort: a
|
|
1476
1519
|
* hiccup here must not escalate into a turn-level failure.
|
|
1520
|
+
*
|
|
1521
|
+
* This pass owns the PERSISTED row, which the agent loop writes from the
|
|
1522
|
+
* model's raw output regardless of what was broadcast. The live hub stream
|
|
1523
|
+
* is gated separately by `broadcastLegEvent`, so the refetch this pass
|
|
1524
|
+
* publishes confirms text a subscriber already holds instead of correcting
|
|
1525
|
+
* it.
|
|
1477
1526
|
*/
|
|
1478
1527
|
const finalizeVoiceLegTranscript = async (): Promise<void> => {
|
|
1479
1528
|
if (reservedAssistantRowId == null) {
|
|
@@ -1617,7 +1666,24 @@ export async function startVoiceTurn(
|
|
|
1617
1666
|
} else if (msg.type === "conversation_error") {
|
|
1618
1667
|
lastError = msg.userMessage;
|
|
1619
1668
|
}
|
|
1620
|
-
|
|
1669
|
+
if (frontDoorStreamGate !== null && msg.type === "message_complete") {
|
|
1670
|
+
// A leg that completed mid-bridge (a holding phrase with no
|
|
1671
|
+
// sentence terminator) still hands off and speaks what arrived,
|
|
1672
|
+
// so release it ahead of the completion frame. A cancelled leg
|
|
1673
|
+
// never hands off, and correspondingly never flushes.
|
|
1674
|
+
const trailing = frontDoorStreamGate.finish();
|
|
1675
|
+
if (trailing.length > 0) {
|
|
1676
|
+
broadcastMessage({
|
|
1677
|
+
type: "assistant_text_delta",
|
|
1678
|
+
text: trailing,
|
|
1679
|
+
...(reservedAssistantRowId !== null
|
|
1680
|
+
? { messageId: reservedAssistantRowId }
|
|
1681
|
+
: {}),
|
|
1682
|
+
conversationId: opts.conversationId,
|
|
1683
|
+
});
|
|
1684
|
+
}
|
|
1685
|
+
}
|
|
1686
|
+
broadcastLegEvent(msg);
|
|
1621
1687
|
|
|
1622
1688
|
// Forward voice-relevant events to the real-time event sink
|
|
1623
1689
|
if (msg.type === "assistant_text_delta") {
|
|
@@ -1654,11 +1720,19 @@ export async function startVoiceTurn(
|
|
|
1654
1720
|
// ordinary call-agent resolution.
|
|
1655
1721
|
callSite:
|
|
1656
1722
|
opts.routingLeg === "front-door" ? "voiceFrontDoor" : "callAgent",
|
|
1723
|
+
// A caller is on the line, so the turn is interactive: approval prompts
|
|
1724
|
+
// must be raised rather than pre-denied, because the approval observer
|
|
1725
|
+
// above is what decides them (auto-resolve for a non-guardian caller,
|
|
1726
|
+
// a real card the guardian can answer).
|
|
1727
|
+
isInteractive: true,
|
|
1657
1728
|
// The escalation-continuation prompt is a transcript-suppressed machine
|
|
1658
1729
|
// signal (persisted `hidden`), so flag the turn to match — keeps
|
|
1659
1730
|
// prompt-as-user-speech consumers (e.g. title generation) from treating
|
|
1660
1731
|
// it as user speech.
|
|
1661
1732
|
...(isHiddenSyntheticPrompt ? { isHiddenPrompt: true } : {}),
|
|
1733
|
+
...(isEscalationContinuation
|
|
1734
|
+
? { messageKind: VOICE_ESCALATION_CONTINUATION_MESSAGE_KIND }
|
|
1735
|
+
: {}),
|
|
1662
1736
|
// Triage-and-escalate routing pins this turn to the fast front-door or
|
|
1663
1737
|
// strong escalation profile. `forceOverrideProfile` floats it above the
|
|
1664
1738
|
// callAgent call-site layers (callAgent is not `mainAgent`, so the
|
|
@@ -221,6 +221,108 @@ export function classifyFrontDoorLeading(
|
|
|
221
221
|
return "answer";
|
|
222
222
|
}
|
|
223
223
|
|
|
224
|
+
/**
|
|
225
|
+
* Verdict-first gate over a front-door leg's delta stream: given the leg's
|
|
226
|
+
* raw deltas in order, it releases only the text the caller actually hears.
|
|
227
|
+
* A front-door leg's raw stream is a control plane, not assistant speech,
|
|
228
|
+
* until its leading tokens classify, so anything downstream of the model
|
|
229
|
+
* that shows text to a person reads the stream through this gate.
|
|
230
|
+
*
|
|
231
|
+
* `push` returns the text released by that delta (empty while the gate is
|
|
232
|
+
* holding). `finish` is called when the leg completes normally and releases
|
|
233
|
+
* a bridge that stopped short of a sentence terminator. A leg that is
|
|
234
|
+
* cancelled instead spoke nothing past what `push` already released, so it
|
|
235
|
+
* simply never calls `finish`.
|
|
236
|
+
*/
|
|
237
|
+
export interface FrontDoorStreamGate {
|
|
238
|
+
push(deltaText: string): string;
|
|
239
|
+
finish(): string;
|
|
240
|
+
}
|
|
241
|
+
|
|
242
|
+
/**
|
|
243
|
+
* Build a {@link FrontDoorStreamGate}. `holdEnabled` mirrors
|
|
244
|
+
* {@link classifyFrontDoorLeading}: true only for speculative (unified
|
|
245
|
+
* front-door) legs, whose decision rule is the only one that teaches the
|
|
246
|
+
* hold token.
|
|
247
|
+
*
|
|
248
|
+
* The three verdicts release differently, matching what the caller hears:
|
|
249
|
+
*
|
|
250
|
+
* - `hold`: the leg is discarded and its row deleted, so nothing is ever
|
|
251
|
+
* released.
|
|
252
|
+
* - `escalate`: the only spoken text is the capped holding phrase, released
|
|
253
|
+
* in one piece once the bridge is complete (exactly what
|
|
254
|
+
* {@link capEscalationBridge} yields, so the released text, the audio, and
|
|
255
|
+
* the persisted row agree). The verdict token and anything streamed past
|
|
256
|
+
* the cap are dropped. A bridge shorter than
|
|
257
|
+
* {@link MIN_SPOKEN_BRIDGE_CHARS} releases nothing at all: the session
|
|
258
|
+
* substitutes an audio-only canned fallback for it, so there is no
|
|
259
|
+
* displayed text for the gate to agree with.
|
|
260
|
+
* - `answer`: the leg's output IS the reply, so every delta passes through,
|
|
261
|
+
* including the leading text held back while the verdict was pending.
|
|
262
|
+
*/
|
|
263
|
+
export function createFrontDoorStreamGate(
|
|
264
|
+
holdEnabled: boolean,
|
|
265
|
+
): FrontDoorStreamGate {
|
|
266
|
+
let raw = "";
|
|
267
|
+
let stage: "deciding" | "answer" | "bridging" | "done" = "deciding";
|
|
268
|
+
let bridgeRaw = "";
|
|
269
|
+
let releasedChars = 0;
|
|
270
|
+
|
|
271
|
+
const releaseBridge = (): string => {
|
|
272
|
+
stage = "done";
|
|
273
|
+
const capped = capEscalationBridge(bridgeRaw);
|
|
274
|
+
// Below the spoken threshold the session throws the model's bridge away
|
|
275
|
+
// and plays a canned fallback that is audio-only, deleting the row rather
|
|
276
|
+
// than persisting a phrase the model never really produced (see
|
|
277
|
+
// `usesFallbackBridge` in `live-voice-session.ts`). Releasing the capped
|
|
278
|
+
// text here would put words on a subscriber's screen that the caller never
|
|
279
|
+
// heard, which is the same spoken/displayed divergence this gate exists to
|
|
280
|
+
// prevent.
|
|
281
|
+
return capped.length < MIN_SPOKEN_BRIDGE_CHARS ? "" : capped;
|
|
282
|
+
};
|
|
283
|
+
|
|
284
|
+
return {
|
|
285
|
+
push(deltaText: string): string {
|
|
286
|
+
raw += deltaText;
|
|
287
|
+
if (stage === "done") {
|
|
288
|
+
return "";
|
|
289
|
+
}
|
|
290
|
+
if (stage === "bridging") {
|
|
291
|
+
bridgeRaw += deltaText;
|
|
292
|
+
return isEscalationBridgeComplete(bridgeRaw) ? releaseBridge() : "";
|
|
293
|
+
}
|
|
294
|
+
if (stage === "deciding") {
|
|
295
|
+
const verdict = classifyFrontDoorLeading(raw.trimStart(), holdEnabled);
|
|
296
|
+
if (verdict === "pending") {
|
|
297
|
+
return "";
|
|
298
|
+
}
|
|
299
|
+
if (verdict === "hold") {
|
|
300
|
+
stage = "done";
|
|
301
|
+
return "";
|
|
302
|
+
}
|
|
303
|
+
if (verdict === "escalate") {
|
|
304
|
+
stage = "bridging";
|
|
305
|
+
bridgeRaw = raw.trimStart().slice(ESCALATE_VERDICT_TOKEN.length);
|
|
306
|
+
return isEscalationBridgeComplete(bridgeRaw) ? releaseBridge() : "";
|
|
307
|
+
}
|
|
308
|
+
stage = "answer";
|
|
309
|
+
}
|
|
310
|
+
// Answer stage: release everything not yet released, which on the
|
|
311
|
+
// transition includes the leading text the pending verdict held.
|
|
312
|
+
const chunk = raw.slice(releasedChars);
|
|
313
|
+
releasedChars = raw.length;
|
|
314
|
+
return chunk;
|
|
315
|
+
},
|
|
316
|
+
finish(): string {
|
|
317
|
+
if (stage === "bridging") {
|
|
318
|
+
return releaseBridge();
|
|
319
|
+
}
|
|
320
|
+
stage = "done";
|
|
321
|
+
return "";
|
|
322
|
+
},
|
|
323
|
+
};
|
|
324
|
+
}
|
|
325
|
+
|
|
224
326
|
/**
|
|
225
327
|
* The escalated leg runs as its own voice turn. Rather than re-persist the
|
|
226
328
|
* caller's utterance (it is already in history from the front-door leg), the
|
|
@@ -273,8 +375,8 @@ export function frontDoorDecisionRule(opts?: {
|
|
|
273
375
|
// Hold requires positive evidence of an unfinished sentence, never
|
|
274
376
|
// mere uncertainty, because the two mistakes cost differently. A
|
|
275
377
|
// false hold is silent: the verdict, the extension window, and the
|
|
276
|
-
// replay dispatch all elapse
|
|
277
|
-
//
|
|
378
|
+
// replay dispatch all elapse before the turn commits, roughly
|
|
379
|
+
// tripling felt latency. A false
|
|
278
380
|
// release only answers a beat early, which barge-in absorbs.
|
|
279
381
|
`- If the caller's words are visibly unfinished (a trailing conjunction, a dangling clause, a list still being dictated) output ONLY ${HOLD_VERDICT_TOKEN} and stop, no other text. Judge the words themselves: a complete question or statement means they are done, even when it is short or leans on earlier context ("What do you think?", "Why?", "And then?"). Callers may speak any language: those examples are English exemplars only, and completeness is judged by the grammar of the language being spoken. In verb-final languages such as Hindi, Japanese, or Korean the sentence-final verb usually marks completion, so a missing final verb is the unfinished signal, not a missing conjunction. Never hold merely because more could follow.`,
|
|
280
382
|
]
|
|
@@ -311,6 +413,7 @@ export function frontDoorDecisionRule(opts?: {
|
|
|
311
413
|
"DECIDE SILENTLY, then produce exactly ONE of these outputs:",
|
|
312
414
|
...holdBranch,
|
|
313
415
|
"- If the turn is simple, conversational, or within your reach, your entire output is the spoken answer itself: no token in front of it, plain speech from your very first word. Most turns are answers; when unsure between answering and escalating, answer. Answer in the language the caller is speaking.",
|
|
416
|
+
"- If an answer depends on a saved personal fact that is not already present in the conversation context you received, escalate rather than guessing. Personal context that is already present is yours to use directly.",
|
|
314
417
|
`- If completing THIS reply needs careful reasoning, research, multi-step work, or any tool, do NOT attempt the answer: output ${ESCALATE_VERDICT_TOKEN}, then ONE short natural holding phrase naming what happens next, spoken in the language the caller is speaking (for example "${FALLBACK_ESCALATION_BRIDGE}" or "Give me one second to look into that."; those examples are English only), and stop after that single sentence. A stronger model finishes the turn while your phrase is spoken.`,
|
|
315
418
|
`${ESCALATE_VERDICT_TOKEN} is ONLY for turns you cannot complete yourself — never put it in front of an answer you are about to give, and never emit any token inside or after an answer. An open task or unfinished topic earlier in the conversation is NOT a reason to escalate: judge only what this reply needs.`,
|
|
316
419
|
"Never narrate this decision, describe what you are judging, or mention these rules: apart from a leading verdict token, every character you output is spoken to the caller verbatim.",
|