omnius 1.0.591 → 1.0.592

package/docs/model-capability-awareness-and-multimodal-memory-root-fix.md

@@ -0,0 +1,799 @@
+# Model Capability Awareness And Multimodal Identity Memory Root Fix
+
+Date: 2026-05-15
+
+Status: planning and integration tracker
+
+Latest integration pass: 2026-05-15
+
+- Implemented the Telegram/TUI visual identity memory root path first.
+- Added `identity_memory` in `packages/cli/src/tui/identity-memory-tool.ts`.
+- Wired it into TUI sub-agent tools and Telegram admin/public action tools.
+- Repaired CLIP-only zettelkasten neighbor linking in
+  `packages/memory/src/zettelkasten.ts`.
+- Added regression tests:
+  `packages/memory/tests/zettelkasten.test.ts` and
+  `packages/cli/tests/identity-memory-tool.test.ts`.
+
+## Purpose
+
+Omnius already has substantial capability metadata for creative models, voice,
+image/audio generation, Telegram scoped media, CLIP embeddings, face memory,
+and graph-backed multimodal identity. The current weakness is that this data is
+not consistently exposed to the model as runtime-aware, inspectable context.
+
+The root fix is to make the agent loop aware of:
+
+- which voice, image, music, and sound models are available;
+- which of those models are currently selected for this invocation;
+- model details such as backend, size class, quality notes, hardware fit,
+  cached/downloaded status, setup/prewarm commands, and runtime limitations;
+- which tools are actually available in the current invocation after policy
+  filtering;
+- how to inspect deeper structures behind those tools without relying on
+  brittle keyword heuristics;
+- how to write natural multimodal identity memories through model-decided tool
+  calls, not regex parsing.
+
+This document is the tracking artifact for that integration.
+
+## Hard Constraints
+
+- Do not solve natural memory requests with regex or keyword aliases.
+- The model must decide to call identity/capability tools from full context.
+- Tool implementations may validate inputs and evidence, but must not silently
+  infer identity without explicit user-provided evidence.
+- Public Telegram scope must remain per-chat and policy filtered.
+- Private/admin/TUI/global context must not leak into public group context.
+- Capability awareness must describe the current invocation, not a stale static
+  list.
+- Shared services should feed TUI, Telegram, GUI/API, and future voice sessions.
+
+## Existing Anchors
+
+### Creative Model Registries
+
+Image model metadata already exists in:
+
+- `packages/execution/src/tools/image-generate.ts:24`
+  - `ImageGenerationPreset`
+  - fields: `id`, `label`, `backend`, `install`, `category`, `sizeClass`,
+    `quality`, `minVramGB`, `recommendedVramGB`, `deployment`, `steps`,
+    `guidance`, `width`, `height`, `fallbackFor`, `note`
+- `packages/execution/src/tools/image-generate.ts:59`
+  - `DEFAULT_DIFFUSERS_IMAGE_MODEL`
+- `packages/execution/src/tools/image-generate.ts:120`
+  - `IMAGE_GENERATION_MODEL_PRESETS`
+- `packages/execution/src/tools/image-generate.ts:727`
+  - `getImageGenerationPreset`
+- `packages/execution/src/tools/image-generate.ts:733`
+  - `imageGenerationQualityLadder`
+- `packages/execution/src/tools/image-generate.ts:739`
+  - `imageGenerationFallbackAlternates`
+- `packages/execution/src/tools/image-generate.ts:1145`
+  - `ImageGenerateTool`
+- `packages/execution/src/tools/image-generate.ts:1248`
+  - `ImageGenerateTool` supports `action=list_models`, but the schema still
+    marks `prompt` as required and output is shallow.
+
+Sound/music model metadata already exists in:
+
+- `packages/execution/src/tools/audio-generate.ts:27`
+  - `AudioGenerationPreset`
+  - fields: `id`, `label`, `kind`, `backend`, `install`, `category`,
+    `sizeClass`, `quality`, `output`, `bestUse`, `minVramGB`,
+    `recommendedVramGB`, `deployment`, `defaultDurationSec`, `defaultSteps`,
+    `note`
+- `packages/execution/src/tools/audio-generate.ts:79`
+  - `DEFAULT_SOUND_MODEL`
+- `packages/execution/src/tools/audio-generate.ts:80`
+  - `DEFAULT_MUSIC_MODEL`
+- `packages/execution/src/tools/audio-generate.ts:135`
+  - `AUDIO_GENERATION_MODEL_PRESETS`
+- `packages/execution/src/tools/audio-generate.ts:1420`
+  - `getAudioGenerationPreset`
+- `packages/execution/src/tools/audio-generate.ts:1426`
+  - `audioGenerationQualityLadder`
+- `packages/execution/src/tools/audio-generate.ts:1648`
+  - `AudioGenerateTool`
+- `packages/execution/src/tools/audio-generate.ts:1790`
+  - `AudioGenerateTool` supports `action=list_models`, but the schema does not
+    expose `action`.
+
+Voice model metadata exists in:
+
+- `packages/cli/src/tui/voice.ts:171`
+  - `VOICE_MODELS`
+- `packages/cli/src/tui/voice.ts:247`
+  - `listVoiceModels`
+- `packages/cli/src/tui/voice.ts:271`
+  - `getSupertonicVoiceOptions`
+- `packages/cli/src/api/serve.ts:5355`
+  - `GET /v1/voice/models`
+- `packages/cli/src/api/serve.ts:5359`
+  - `GET /v1/voice/supertonic-settings`
+- `packages/cli/src/api/serve.ts:5390`
+  - `POST /v1/voice/models/switch`
+- `packages/execution/src/tools/audio-playback.ts:610`
+  - `AudioPlaybackTool`
+- `packages/execution/src/tools/audio-playback.ts:797`
+  - `audio_playback action=list_voices`
+- `packages/execution/src/tools/audio-playback.ts:1096`
+  - `TtsGenerateTool`
+
+### Current Model Selections
+
+Settings schema:
+
+- `packages/cli/src/tui/omnius-directory.ts:171`
+  - `OmniusSettings`
+- `packages/cli/src/tui/omnius-directory.ts:182`
+  - `voiceModel`
+- `packages/cli/src/tui/omnius-directory.ts:216`
+  - `imageModel`
+- `packages/cli/src/tui/omnius-directory.ts:218`
+  - `imageBackend`
+- `packages/cli/src/tui/omnius-directory.ts:220`
+  - `soundModel`
+- `packages/cli/src/tui/omnius-directory.ts:222`
+  - `soundBackend`
+- `packages/cli/src/tui/omnius-directory.ts:224`
+  - `musicModel`
+- `packages/cli/src/tui/omnius-directory.ts:226`
+  - `musicBackend`
+- `packages/cli/src/tui/omnius-directory.ts:282`
+  - `resolveSettings`
+
+TUI configured defaults:
+
+- `packages/cli/src/tui/interactive.ts:982`
+  - `imageGenerationDefaultsForRepo`
+- `packages/cli/src/tui/interactive.ts:992`
+  - `createConfiguredImageGenerateTool`
+- `packages/cli/src/tui/interactive.ts:996`
+  - `audioGenerationDefaultsForRepo`
+- `packages/cli/src/tui/interactive.ts:1010`
+  - `createConfiguredAudioGenerateTool`
+
+Telegram configured defaults:
+
+- `packages/cli/src/tui/telegram-bridge.ts:5106`
+  - `imageGenerationDefaultsForRepo`
+- `packages/cli/src/tui/telegram-bridge.ts:5116`
+  - `audioGenerationDefaultsForRepo`
+- `packages/cli/src/tui/telegram-bridge.ts:4939`
+  - admin Telegram tools instantiate `ImageGenerateTool`
+- `packages/cli/src/tui/telegram-bridge.ts:4940`
+  - admin Telegram tools instantiate `AudioGenerateTool`
+- `packages/cli/src/tui/telegram-bridge.ts:4975`
+  - public Telegram appends scoped creative tools
+
+### Hardware Rating And Disk Status
+
+The richer model scoring is trapped inside TUI command rendering:
+
+- `packages/cli/src/tui/commands.ts:8689`
+  - `rateImagePresetForHardware`
+- `packages/cli/src/tui/commands.ts:8772`
+  - `renderImageModelList`
+- `packages/cli/src/tui/commands.ts:8876`
+  - `fetchOllamaModelSizes`
+- `packages/cli/src/tui/commands.ts:8904`
+  - `imageModelDiskStats`
+- `packages/cli/src/tui/commands.ts:8914`
+  - `audioModelDiskStats`
+- `packages/cli/src/tui/commands.ts:9193`
+  - `rateAudioPresetForHardware`
+- `packages/cli/src/tui/commands.ts:9240`
+  - `renderAudioModelList`
+
+This logic needs to move into a shared service that can be used by:
+
+- TUI slash commands;
+- Telegram context injection;
+- GUI/API endpoints;
+- agent-facing capability inspection tool.
+
+### Agent Tool Awareness
+
+Static discovery:
+
+- `packages/execution/src/tools/explore-tools.ts:20`
+  - static `TOOL_CATALOG`
+- `packages/execution/src/tools/explore-tools.ts:82`
+  - `ExploreToolsTool`
+
+Runtime-aware discovery:
+
+- `packages/orchestrator/src/agenticRunner.ts:1372`
+  - promoted/deferred tool tracking
+- `packages/orchestrator/src/agenticRunner.ts:17376`
+  - `buildToolDefinitions`
+- `packages/orchestrator/src/agenticRunner.ts:17630`
+  - `tool_search` pseudo-tool
+- `packages/orchestrator/src/agenticRunner.ts:17659`
+  - runtime handler for `tool_search`
+
+Post-compaction awareness:
+
+- `packages/orchestrator/src/agenticRunner.ts:15506`
+  - post-compaction skill and MCP tool re-injection
+
+Current gap:
+
+- `tool_search` reflects runtime registered tools, but only as deferred tool
+  one-liners.
+- `ExploreToolsTool` is static and can drift from the actual invocation.
+- Neither path exposes selected models, model detail, embedding status, scoped
+  media affordances, or deep inspection hooks.
+
+### Invocation Context Injection
+
+TUI dynamic context:
+
+- `packages/cli/src/tui/interactive.ts:2163`
+  - project context loaded
+- `packages/cli/src/tui/interactive.ts:2165`
+  - `dynamicContext`
+- `packages/cli/src/tui/interactive.ts:2188`
+  - `environmentProvider`
+- `packages/cli/src/tui/interactive.ts:2277`
+  - self-modify settings loaded
+- `packages/cli/src/tui/interactive.ts:2312`
+  - vision capability guidance injection
+- `packages/cli/src/tui/interactive.ts:2585`
+  - passed into `AgenticRunner`
+
+Telegram dynamic context:
+
+- `packages/cli/src/tui/telegram-bridge.ts:4123`
+  - quick-chat message construction
+- `packages/cli/src/tui/telegram-bridge.ts:4145`
+  - runtime context in quick-chat system prompt
+- `packages/cli/src/tui/telegram-bridge.ts:4248`
+  - action/chat sub-agent session context
+- `packages/cli/src/tui/telegram-bridge.ts:4266`
+  - `AgenticRunner` creation
+- `packages/cli/src/tui/telegram-bridge.ts:4276`
+  - Telegram `dynamicContext`
+- `packages/cli/src/tui/telegram-bridge.ts:3278`
+  - `buildTelegramSessionContext`
+- `packages/cli/src/tui/telegram-bridge.ts:3297`
+  - Telegram runtime context section
+- `packages/cli/src/tui/telegram-bridge.ts:3315`
+  - admin group safety/context section
+- `packages/cli/src/tui/telegram-bridge.ts:3317`
+  - public group safety/context section
+
+Telegram reply/media context anchors:
+
+- `packages/cli/src/tui/telegram-bridge.ts:204`
+  - `TelegramReplyContext`
+- `packages/cli/src/tui/telegram-bridge.ts:2110`
+  - `resolveTelegramReplyContext`
+- `packages/cli/src/tui/telegram-bridge.ts:2160`
+  - `buildTelegramCurrentReplyContext`
+- `packages/cli/src/tui/telegram-bridge.ts:2203`
+  - `formatTelegramCurrentMessageForPrompt`
+- `packages/cli/src/tui/telegram-bridge.ts:2793`
+  - `buildTelegramConversationContextStream`
+- `packages/cli/src/tui/telegram-bridge.ts:2853`
+  - recent media context section
+- `packages/cli/src/tui/telegram-bridge.ts:5048`
+  - `buildTelegramMediaRecentTool`
+- `packages/cli/src/tui/telegram-bridge.ts:5527`
+  - `processMediaContextForMessage`
+
+### Multimodal Memory And Identity
+
+Graph-backed multimodal identity:
+
+- `packages/memory/src/multimodalIdentity.ts:6`
+  - source/scope types
+- `packages/memory/src/multimodalIdentity.ts:35`
+  - media refs
+- `packages/memory/src/multimodalIdentity.ts:45`
+  - `MultimodalIdentityAssertion`
+- `packages/memory/src/multimodalIdentity.ts:53`
+  - `MultimodalEmbeddingBundle`
+- `packages/memory/src/multimodalIdentity.ts:167`
+  - `MultimodalIdentityService`
+- `packages/memory/src/multimodalIdentity.ts:178`
+  - `ingest`
+- `packages/memory/src/multimodalIdentity.ts:269`
+  - identity assertion application
+- `packages/memory/src/multimodalIdentity.ts:298`
+  - `applyIdentityAssertions`
+- `packages/memory/src/multimodalIdentity.ts:330`
+  - embedding persistence
+
+Knowledge graph relations:
+
+- `packages/memory/src/temporalGraph.ts:24`
+  - `KGRelation`
+- `packages/memory/src/temporalGraph.ts:27`
+  - reply/media identity relations: `authored_by`, `uploaded_by`,
+    `replied_to`, `depicts`, `named_as`
+- `packages/memory/src/temporalGraph.ts:28`
+  - `face_match`, `voice_sample_of`, `speaker_candidate`,
+    `same_person_candidate`
+
+Episode store:
+
+- `packages/memory/src/episodeStore.ts:69`
+  - `Episode`
+- `packages/memory/src/episodeStore.ts:80`
+  - native `embedding`
+- `packages/memory/src/episodeStore.ts:83`
+  - `clipEmbedding`
+- `packages/memory/src/episodeStore.ts:337`
+  - `clip_embedding` column
+- `packages/memory/src/episodeStore.ts:397`
+  - automatic zettelkasten linking on insert
+- `packages/memory/src/episodeStore.ts:625`
+  - `setEmbedding`
+- `packages/memory/src/episodeStore.ts:631`
+  - `setClipEmbedding`
+
+Zettelkasten:
+
+- `packages/memory/src/zettelkasten.ts:72`
+  - `findNeighbors`
+- `packages/memory/src/zettelkasten.ts:78`
+  - currently returns empty if the source episode has no native embedding
+- `packages/memory/src/zettelkasten.ts:90`
+  - CLIP fallback only runs when native embedding lengths mismatch
+- `packages/memory/src/zettelkasten.ts:114`
+  - `linkEpisode`
+- `packages/memory/src/zettelkasten.ts:177`
+  - `batchLink`
+
+Embedding workers:
+
+- `packages/cli/src/api/embedding-workers.ts:41`
+  - `startEmbeddingWorkers`
+- `packages/cli/src/api/embedding-workers.ts:64`
+  - scans visual/audio episodes missing native embeddings
+- `packages/cli/src/api/embedding-workers.ts:107`
+  - stores native embedding
+- `packages/cli/src/api/embedding-workers.ts:121`
+  - creates `EmbeddingAligner`
+- `packages/cli/src/api/embedding-workers.ts:143`
+  - stores aligned embedding as `clipEmbedding`
+- `packages/cli/src/api/embedding-workers.ts:148`
+  - coarse KG node insertion currently labels visual episodes as `person`
+- `packages/cli/src/api/embedding-workers.ts:165`
+  - stale `person_node_id` co-occurrence metadata path
+
+Embedding scripts:
+
+- `scripts/embed-image.py:23`
+  - OpenCLIP ViT-B-32 image embedding
+- `scripts/embed-text.py:14`
+  - OpenCLIP ViT-B-32 text embedding
+- `scripts/embed-audio.py:45`
+  - SpeechBrain ECAPA speaker embedding
+- `packages/cli/src/api/py-embed.ts:16`
+  - `ensureEmbedDeps`
+- `packages/cli/src/api/py-embed.ts:41`
+  - full embedding deps require `OMNIUS_INSTALL_FULL_EMBED_DEPS=1`
+- `packages/cli/src/api/py-embed.ts:55`
+  - `runEmbedImage`
+- `packages/cli/src/api/py-embed.ts:66`
+  - `runEmbedAudio`
+- `packages/cli/src/api/py-embed.ts:77`
+  - `runEmbedText`
+
+Older visual/multimodal memory tools:
+
+- `packages/execution/src/tools/visual-memory.ts:44`
+  - `VisualMemoryTool`
+- `packages/execution/src/tools/visual-memory.ts:58`
+  - actions: `detect`, `enroll`, `identify`, `teach`, `recognize`, `list`,
+    `forget`
+- `packages/execution/src/tools/visual-memory.ts:161`
+  - face enrollment
+- `packages/execution/src/tools/multimodal-memory.ts:74`
+  - `MultimodalMemoryTool`
+- `packages/execution/src/tools/multimodal-memory.ts:89`
+  - actions: `capture`, `meet`, `recall`, `timeline`
+- `packages/execution/src/tools/multimodal-memory.ts:639`
+  - stores visual CLIP into `EpisodeStore.setClipEmbedding`
+
+Current gap:
+
+- The new graph-backed identity path is central, but older face and
+  multimodal-memory stores are still separate silos.
+- Telegram media ingest writes scoped episodes, but there is no model-facing
+  identity tool for explicit user assertions such as "this person is Amy".
+- Zettelkasten linking does not fully exploit CLIP-only visual episodes.
+
+### Telegram Memory Ingest
+
+- `packages/cli/src/tui/telegram-bridge.ts:2227`
+  - sender normalization
+- `packages/cli/src/tui/telegram-bridge.ts:2247`
+  - memory scope
+- `packages/cli/src/tui/telegram-bridge.ts:2255`
+  - reply ref
+- `packages/cli/src/tui/telegram-bridge.ts:2272`
+  - `telegramMemoryIngestPayload`
+- `packages/cli/src/tui/telegram-bridge.ts:5477`
+  - image ingest POST to `/v1/memory/ingest`
+- `packages/cli/src/tui/telegram-bridge.ts:5506`
+  - audio/voice ingest POST to `/v1/memory/ingest`
+- `packages/cli/src/api/serve.ts:8324`
+  - `/v1/memory/ingest`
+- `packages/cli/src/api/serve.ts:8907`
+  - `handleMemoryIngest`
+- `packages/cli/src/api/serve.ts:8947`
+  - identity assertion extraction from API payload
+- `packages/cli/src/api/serve.ts:8997`
+  - `MultimodalIdentityService`
+
+## Root Fix Architecture
+
+### 1. Shared Runtime Capability Snapshot
+
+Create a shared capability module, likely under one of:
+
+- `packages/execution/src/model-capabilities.ts`
+- `packages/cli/src/tui/model-capabilities.ts`
+
+Preferred direction:
+
+- put pure registry/scoring logic in `@omnius/execution` if it does not need
+  TUI renderer state;
+- keep CLI-only disk probes and settings loaders in CLI helpers if necessary;
+- expose plain JSON structures, not formatted TUI lines.
+
+Proposed shape:
+
+```ts
+export type CapabilityModelKind = "voice" | "image" | "sound" | "music";
+
+export interface RuntimeModelSelection {
+  kind: CapabilityModelKind;
+  selectedModel?: string;
+  selectedBackend?: string;
+  source: "project" | "global" | "default" | "runtime";
+}
+
+export interface RuntimeModelFit {
+  score?: number;
+  label?: string;
+  note?: string;
+  minVramGB?: number;
+  recommendedVramGB?: number;
+}
+
+export interface RuntimeModelDiskState {
+  downloaded: boolean;
+  bytes: number;
+  paths: string[];
+}
+
+export interface RuntimeModelCapability {
+  kind: CapabilityModelKind;
+  id: string;
+  label: string;
+  backend: string;
+  category?: string;
+  sizeClass?: string;
+  quality?: string;
+  output?: string;
+  bestUse?: string;
+  deployment?: string;
+  install?: string;
+  defaults?: Record<string, unknown>;
+  fit?: RuntimeModelFit;
+  disk?: RuntimeModelDiskState;
+  selected?: boolean;
+  notes: string[];
+}
+
+export interface RuntimeCapabilitySnapshot {
+  generatedAt: string;
+  repoRoot: string;
+  hardware: {
+    totalRamGB: number;
+    availableRamGB: number;
+    gpuVramGB: number;
+    gpuName?: string;
+  };
+  selections: RuntimeModelSelection[];
+  models: RuntimeModelCapability[];
+  embedding: RuntimeEmbeddingStatus;
+  tools?: RuntimeToolAwareness;
+}
+```
+
+### 2. Agent-Facing Capability Tool
+
+Add a tool such as `model_capabilities` or `capability_inspect`.
+
+Actions:
+
+- `summary`
+  - current selected voice/image/sound/music plus compact fit.
+- `list`
+  - filter by `kind=voice|image|sound|music|embedding|tools`.
+- `inspect`
+  - inspect one model/tool/embedding provider in depth.
+- `current`
+  - exact current selections for this invocation.
+- `embedding_status`
+  - OpenCLIP, text CLIP, speaker embedding, transcript embedding, installed
+    status, vector spaces, counts.
+- `tool_context`
+  - actual registered tools in this invocation after policy filtering.
+
+This tool must be registered in:
+
+- TUI `buildTools` / sub-agent tool set.
+- Telegram admin and public contexts, with public output scoped and redacted.
+- GUI/API agent runs.
+
+### 3. Compact Capability Context Injection
+
+Every invocation should receive a compact, non-overwhelming block:
+
+```text
+<runtime-capabilities>
+Selected models:
+- voice: supertonic (backend supertonic)
+- image: stabilityai/sdxl-turbo (diffusers, fit 88/100 excellent)
+- sound: cvssp/audioldm-s-full-v2 (diffusers, fit 74/100 comfortable)
+- music: facebook/musicgen-small (transformers, fit 70/100 comfortable)
+
+Use model_capabilities(action="inspect", kind="image", id="...") for details.
+Use model_capabilities(action="list", kind="music") before switching models.
+</runtime-capabilities>
+```
+
+Injection anchors:
+
+- TUI: `packages/cli/src/tui/interactive.ts:2165`
+- TUI environment refresh: `packages/cli/src/tui/interactive.ts:2188`
+- Telegram quick chat: `packages/cli/src/tui/telegram-bridge.ts:4149`
+- Telegram action context: `packages/cli/src/tui/telegram-bridge.ts:3297`
+- API/GUI agent subprocess: agent launch path under `packages/cli/src/api/serve.ts`
+
+The context should remain compact. Deep detail belongs in the tool.
+
+### 4. Runtime Tool Awareness
+
+Improve agent self-awareness with the actual invocation tool set.
+
+Targets:
+
+- keep `tool_search` as the schema promotion mechanism;
+- replace or augment static `ExploreToolsTool` with a runtime-provided tool
+  index where possible;
+- expose policy-filtered tool names/categories in the capability snapshot;
+- after compaction, re-inject the compact tool/category summary alongside MCP
+  names.
+
+Important anchors:
+
+- `packages/execution/src/tools/explore-tools.ts:20`
+- `packages/orchestrator/src/agenticRunner.ts:17376`
+- `packages/orchestrator/src/agenticRunner.ts:17630`
+- `packages/orchestrator/src/agenticRunner.ts:15506`
+
+### 5. Natural Identity Memory Tool
+
+Add an agent-facing tool, likely `identity_memory`.
+
+Actions:
+
+- `assert_identity`
+  - user has explicitly provided identity information.
+- `enroll_face`
+  - enroll a face embedding using image evidence.
+- `identify`
+  - identify known faces/voices in supplied media.
+- `recall`
+  - retrieve identity evidence by name/person/media.
+- `inspect`
+  - show identity node, evidence, modalities, confidence, scope.
+
+Inputs:
+
+```ts
+interface IdentityMemoryArgs {
+  action: "assert_identity" | "enroll_face" | "identify" | "recall" | "inspect";
+  name?: string;
+  relation?: "depicts" | "speaker" | "named_as" | "same_person_candidate";
+  media?: "reply" | "latest" | string;
+  message_id?: string | number;
+  scope?: "current" | "private" | "group" | "global";
+  confidence?: number;
+  evidence_note?: string;
+}
+```
+
+Validation rules:
+
+- The model may call this naturally when the user asks to remember a face,
+  states a person name, identifies a speaker, or asks who/what is remembered.
+- The tool must require explicit `name` for identity assertion writes.
+- The tool must resolve media aliases through scoped media providers, not local
+  filesystem guessing.
+- If there is no usable image/voice evidence, store the textual identity
+  assertion but mark it as lacking biometric embedding evidence.
+- If a face is present, write through `MultimodalIdentityService` and mirror
+  enrollment into `VisualMemoryTool` or a shared face embedding helper.
+- In public Telegram, writes are scoped to the current chat.
+
+### 6. CLIP And Associative Memory Repair
+
+Required fixes:
+
+- Allow zettelkasten linking when an episode has `clipEmbedding` but no native
+  `embedding`.
+- Make `findNeighbors` consider candidates with compatible CLIP embeddings even
+  when native embeddings are absent.
+- Avoid labeling every visual episode node as `person` in embedding workers.
+- Replace stale `person_node_id` co-occurrence assumptions with graph relations
+  created by `MultimodalIdentityService`.
+- Add status/introspection for embedding providers and vector counts.
+
+Important anchors:
+
+- `packages/memory/src/zettelkasten.ts:78`
+- `packages/memory/src/zettelkasten.ts:90`
+- `packages/cli/src/api/embedding-workers.ts:148`
+- `packages/cli/src/api/embedding-workers.ts:165`
+- `packages/memory/src/multimodalIdentity.ts:330`
+
+## Implementation Checklist
+
+### Phase 0 - Tracking Document
+
+- [x] Create root-fix document with anchors and artifacts.
+- [x] Update this checklist as each implementation step lands.
+- [ ] Add links from related docs if the implementation spans multiple passes.
+
+### Phase 1 - Shared Model Capability Snapshot
+
+- [ ] Extract image hardware fit logic from TUI command rendering into a shared
+      pure function.
+- [ ] Extract sound/music hardware fit logic from TUI command rendering into a
+      shared pure function.
+- [ ] Add shared disk/cache probe helpers or a CLI wrapper around shared model
+      metadata.
+- [ ] Add `buildRuntimeCapabilitySnapshot(repoRoot, options)`.
+- [ ] Include current selected voice/image/sound/music settings.
+- [ ] Include defaults when no explicit setting is present.
+- [ ] Include voice model metadata from `listVoiceModels`.
+- [ ] Include image/music/sound model metadata and fit scores.
+- [ ] Include downloaded/cache size where available.
+- [ ] Add unit tests for snapshot shape and selected/default resolution.
+
+### Phase 2 - Agent-Facing Capability Tool
+
+- [ ] Add `ModelCapabilitiesTool` or `CapabilityInspectTool`.
+- [ ] Support `summary`, `current`, `list`, `inspect`, `embedding_status`,
+      and `tool_context`.
+- [ ] Register the tool in TUI main tool set.
+- [ ] Register the tool in TUI sub-agent tool set.
+- [ ] Register the tool in Telegram admin DM tools.
+- [ ] Register a scoped/redacted version in Telegram public tools.
+- [ ] Register the tool for GUI/API agent runs.
+- [ ] Add tests for public redaction and full admin output.
+
+### Phase 3 - Capability Context Injection
+
+- [ ] Add compact `<runtime-capabilities>` renderer.
+- [ ] Inject into TUI `dynamicContext`.
+- [ ] Inject into TUI `environmentProvider` if selections can change mid-run.
+- [ ] Inject into Telegram quick-chat system context.
+- [ ] Inject into Telegram action sub-agent dynamic context.
+- [ ] Inject into GUI/API agent subprocess context.
+- [ ] Add post-compaction reinjection for compact capability awareness.
+- [ ] Verify context remains compact for small/medium models.
+
+### Phase 4 - Existing Tool Schema Cleanup
+
+- [ ] Fix `ImageGenerateTool` schema so `action=list_models` does not require
+      `prompt`.
+- [ ] Expand `ImageGenerateTool` `list_models` output or delegate to the new
+      capability snapshot.
+- [ ] Add `action` to `AudioGenerateTool` schema.
+- [ ] Ensure `AudioGenerateTool action=list_models kind=music|sound` works
+      without `prompt`.
+- [ ] Ensure `generate_tts`/`audio_playback list_voices` points to the same
+      voice capability metadata.
+- [ ] Update `/help` and README model/tool descriptions after implementation.
+
+### Phase 5 - Natural Identity Memory Tool
+
+- [x] Add `IdentityMemoryTool`.
+- [x] Integrate with `MultimodalIdentityService`.
+- [x] Resolve scoped media aliases: `reply`, `latest`, and explicit
+      Telegram/media paths.
+- [x] Validate explicit name/evidence on identity assertion writes.
+- [x] Store scoped graph evidence for `depicts`, `named_as`,
+      `speaker_candidate`, and `same_person_candidate`.
+- [ ] Add voice embedding extraction so explicit speaker assertions can also
+      create `voice_sample_of` evidence.
+- [x] Mirror face enrollment into visual memory or a shared face embedding
+      backend.
+- [ ] Support recall/inspect by sender, media, voice, and scope. Person-name
+      recall/inspect is implemented.
+- [x] Add Telegram public scoped version.
+- [x] Add tests for no-regex behavior: tool only writes when called with
+      explicit structured args.
+
+Implemented anchors:
+
+- `packages/cli/src/tui/identity-memory-tool.ts`
+  - `IdentityMemoryTool`
+  - `formatIdentityMemoryContext`
+  - media resolver interface for Telegram, TUI, GUI, and future voice/session
+    surfaces.
+- `packages/cli/src/tui/telegram-bridge.ts`
+  - `buildTelegramIdentityMemoryTool`
+  - `resolveTelegramIdentityMedia`
+  - `formatIdentityMemoryContext` injection into Telegram action context.
+- `packages/cli/src/tui/interactive.ts`
+  - TUI main-agent and sub-agent registration of `IdentityMemoryTool`
+  - compact identity memory contract injection.
+
+### Phase 6 - CLIP/Zettelkasten Repair
+
+- [x] Update `findNeighbors` to handle CLIP-only source episodes.
+- [x] Update candidate filtering to include CLIP-compatible candidates.
+- [x] Ensure CLIP image/text embeddings use clearly named vector spaces.
+- [x] Add regression tests for visual episode with only `clipEmbedding`.
+- [ ] Remove or replace stale `person_node_id` coupling in embedding workers.
+- [ ] Stop labeling all visual embedding worker nodes as `person`.
+- [x] Link identity assertions to media/message/person nodes through the graph.
+
+### Phase 7 - GUI/API And Documentation
+
+- [ ] Add `/v1/capabilities` or `/v1/model-capabilities`.
+- [ ] Add `/v1/capabilities/current`.
+- [ ] Add `/v1/capabilities/models?kind=image|sound|music|voice`.
+- [ ] Add GUI display for selected/current models using the same endpoint.
+- [ ] Add README section for model capability awareness.
+- [ ] Add `/help` entries for the new capability and identity memory tools.
+- [ ] Update existing docs that mention embedding auto-installs to reflect
+      `OMNIUS_INSTALL_FULL_EMBED_DEPS=1`.
+
+### Phase 8 - Verification
+
+- [ ] `pnpm -r build`
+- [x] targeted memory package tests
+- [x] targeted CLI identity tool tests
+- [x] targeted CLI and memory builds
+- [ ] Telegram public scoped smoke test
+- [ ] Telegram admin DM smoke test
+- [ ] TUI smoke test for capability summary
+- [ ] GUI/API smoke test for capability endpoint
+- [ ] Manual test: ask "what image model are you using?"
+- [ ] Manual test: ask "list available music models and pick the best fit"
+- [ ] Manual test: reply to an image with "this is <name>" and confirm graph,
+      visual memory, and recall evidence
+
+## Expected End State
+
+The agent should be able to answer, without guessing:
+
+- "What voice model are you using right now?"
+- "Which image models can you use, and which fits this GPU best?"
+- "List music models by quality and size."
+- "Can you inspect your sound model setup?"
+- "Do you remember this face?"
+- "This person is Amy. Remember that."
+- "Who is in this image based on prior memory?"
+
+For those questions, the model should either:
+
+- answer from compact runtime capability context;
+- call the capability inspection tool for details;
+- call the identity memory tool with structured evidence;
+- ask a clarifying question if the identity evidence is ambiguous.
+
+No part of this should depend on regex phrase triggers.