omnius 1.0.591 → 1.0.592

package/docs/work-orders/WO-AM-GAPS.md

@@ -0,0 +1,638 @@
+# Associative Memory Gap Work Orders
+
+Generated: 2026-04-13
+Source: Deep audit of multimodal associative memory systems
+Status: READY FOR IMPLEMENTATION
+
+---
+
+## WO-AM-GAP-01: Modality-Aware Episode Creation in Orchestrator
+
+**Priority**: P0 (Critical — all downstream memory systems depend on correct modality tagging)
+**Effort**: Small (15 lines changed)
+**Risk**: Low (additive, no breaking changes)
+
+### Problem
+
+The orchestrator hardcodes `modality: "tool_result"` for ALL tool results at `agenticRunner.ts:2900`. Visual, audio, social, and spatial tool results are stored with the wrong modality, making them invisible to modality-filtered queries and causing incorrect importance/decay auto-assignment.
+
+### Root Cause
+
+Line 2900 in `packages/orchestrator/src/agenticRunner.ts`:
+```typescript
+modality: "tool_result",
+```
+
+This was written before the multimodal tools existed. Every tool — whether `file_read` or `vision` or `audio_capture` — gets the same modality tag.
+
+### Changes Required
+
+**File 1: `packages/orchestrator/src/agenticRunner.ts`**
+
+Location: Lines 2892-2902 (inside the post-tool-call episode insertion block)
+
+Replace:
+```typescript
+modality: "tool_result",
+```
+
+With:
+```typescript
+modality: inferEpisodeModality(tc.name),
+```
+
+Add helper function (place near line 2890, before the insertion block):
+```typescript
+function inferEpisodeModality(toolName: string): EpisodeModality {
+  // Visual tools
+  if (["vision", "camera_capture", "image_read", "screenshot", "ocr",
+       "ocr_image_advanced", "visual_memory", "desktop_describe"].includes(toolName)) {
+    return "visual";
+  }
+  // Audio tools
+  if (["audio_capture", "audio_analyze", "asr_listen", "transcribe_file",
+       "transcribe_url", "audio_playback"].includes(toolName)) {
+    return "audio";
+  }
+  // Social tools
+  if (["multimodal_memory", "send_message", "jibberlink"].includes(toolName)) {
+    return "social";
+  }
+  // Spatial tools
+  if (["gps_location", "bluetooth_scan", "wifi_control", "sdr_scan"].includes(toolName)) {
+    return "spatial";
+  }
+  // Code tools
+  if (["file_write", "file_edit", "file_patch", "batch_edit",
+       "code_sandbox", "repl_exec"].includes(toolName)) {
+    return "code";
+  }
+  return "tool_result";
+}
+```
+
+**File 2: `packages/orchestrator/src/agenticRunner.ts`** (imports)
+
+Add import at the top (near existing memory imports):
+```typescript
+import type { EpisodeModality } from "@omnius/memory";
+```
+
+### Upstream Dependencies
+- None. The `EpisodeModality` type already includes all needed values.
+
+### Downstream Effects
+- `autoImportance()` in `episodeStore.ts:115-152` will now correctly assign importance=7 for visual/audio episodes (currently gets 5 because modality is "tool_result")
+- `autoDecayClass()` in `episodeStore.ts:154-179` will correctly assign "daily" for visual/audio (currently gets "daily" anyway for "tool_result", but this makes it explicit)
+- The memory visualizer's episode view will now show correct modality icons
+- PPR retrieval will be able to filter by modality in future queries
+
+### Success Metrics
+- `pnpm test` in orchestrator package: all 387 tests pass
+- `npx tsc --noEmit`: zero errors
+- Manual test: call `vision(action="caption", path="test.png")` → verify episode has `modality: "visual"` in the database:
+  ```sql
+  SELECT modality FROM episodes ORDER BY timestamp DESC LIMIT 1;
+  -- Expected: "visual" (not "tool_result")
+  ```
+
+### Failure Modes to Avoid
+- Do NOT change the modality for `task_complete` — it must remain handled separately at line 4256 as `modality: "gist"`
+- Do NOT use the tool's category metadata — some tools are miscategorized. Use the explicit name list.
+- The `inferEpisodeModality` function must be O(1) — use a Set lookup if the list grows beyond 30 tools.
+
+---
+
+## WO-AM-GAP-02: Bridge Multimodal JSON Store to SQLite EpisodeStore
+
+**Priority**: P0 (Critical — multimodal episodes invisible to associative retrieval)
+**Effort**: Medium (60 lines added)
+**Risk**: Medium (touches two storage systems, must maintain consistency)
+
+### Problem
+
+`multimodal-memory.ts` saves rich cross-modal episodes (CLIP embeddings, face IDs, transcripts, GPS) to JSON files at `~/.omnius/multimodal-episodes/`. This data is completely disconnected from the SQLite `episodeStore` used by PPR retrieval, zettelkasten linking, and orchestrator context injection.
+
+Result: "Who did I meet yesterday?" only works via the multimodal_memory tool's `recall` action. The agent's automatic context injection (WO-AM-06 every 3 turns) never surfaces multimodal episodes.
+
+### Root Cause
+
+`multimodal-memory.ts` was built as a standalone tool before the episode store existed. Its `saveEpisode()` method (line ~293) writes only to JSON, not to SQLite.
+
+### Changes Required
+
+**File 1: `packages/execution/src/tools/multimodal-memory.ts`**
+
+Location: After `this.saveEpisode(episode)` call (line 293 in captureEpisode, line 381 in meetPerson)
+
+Add episode store bridge call:
+```typescript
+// Bridge to SQLite episode store for PPR retrieval + zettelkasten linking
+this.bridgeToEpisodeStore(episode);
+```
+
+Add the bridge method to the class:
+```typescript
+private bridgeToEpisodeStore(episode: MultiModalEpisode): void {
+  try {
+    // Dynamic import to avoid hard dependency on memory package in execution
+    const { EpisodeStore, TemporalGraph } = require("@omnius/memory");
+    const { join } = require("node:path");
+
+    // Find the project's .omnius_test directory (or .omnius/)
+    const omniusDirs = [
+      join(process.cwd(), ".omnius_test"),
+      join(process.cwd(), ".omnius"),
+    ];
+    const omniusDir = omniusDirs.find(d => require("node:fs").existsSync(d));
+    if (!omniusDir) return;
+
+    const es = new EpisodeStore(join(omniusDir, "memory.db"));
+    const tg = new TemporalGraph(join(omniusDir, "kg.db"));
+
+    // Build composite content from all modalities
+    const contentParts: string[] = [];
+    if (episode.social?.personName) contentParts.push(`Met ${episode.social.personName}`);
+    if (episode.audio?.transcript) contentParts.push(`Said: "${episode.audio.transcript}"`);
+    if (episode.audio?.soundClass) contentParts.push(`Sound: ${episode.audio.soundClass}`);
+    if (episode.visual?.faceNames?.length) contentParts.push(`Faces: ${episode.visual.faceNames.join(", ")}`);
+    if (episode.visual?.objects?.length) contentParts.push(`Objects: ${episode.visual.objects.join(", ")}`);
+    if (episode.spatial?.locationLabel) contentParts.push(`Location: ${episode.spatial.locationLabel}`);
+    const content = contentParts.join(". ") || episode.text?.content || "Multimodal capture";
+
+    // Determine primary modality
+    const modality = episode.social?.personName ? "social"
+      : episode.visual?.imagePath ? "visual"
+      : episode.audio?.recordingPath ? "audio"
+      : "text";
+
+    // Insert episode with CLIP embedding if available
+    const epId = es.insert({
+      sessionId: episode.sessionId,
+      modality,
+      toolName: "multimodal_memory",
+      content,
+      importance: episode.social?.personName ? 9 : 7,
+      decayClass: episode.social?.personName ? "procedural" : "daily",
+      metadata: {
+        multimodal_episode_id: episode.id,
+        has_face: (episode.visual?.faceIds?.length ?? 0) > 0,
+        has_audio: !!episode.audio?.recordingPath,
+        has_gps: !!episode.spatial?.gps,
+      },
+    });
+
+    // Set CLIP embedding if available (512d → Float32Array)
+    if (episode.visual?.clipEmbedding) {
+      const emb = new Float32Array(episode.visual.clipEmbedding);
+      es.setEmbedding(epId, emb);
+    }
+
+    // Create KG nodes for entities
+    if (episode.social?.personName) {
+      const personId = tg.upsertNode({ text: episode.social.personName, nodeType: "person" });
+      tg.addEdge({
+        srcId: personId, dstId: personId,
+        relation: "discovered_during",
+        fact: `Met ${episode.social.personName} via multimodal capture`,
+        edgeType: "triple",
+        sourceEpisodeId: epId,
+        modality,
+      });
+    }
+
+    if (episode.spatial?.locationLabel) {
+      const locId = tg.upsertNode({ text: episode.spatial.locationLabel, nodeType: "location" });
+      if (episode.social?.personName) {
+        const personId = tg.findNode(episode.social.personName, "person")?.id;
+        if (personId) {
+          tg.addEdge({
+            srcId: personId, dstId: locId,
+            relation: "appears_in",
+            fact: `${episode.social.personName} seen at ${episode.spatial.locationLabel}`,
+            sourceEpisodeId: epId,
+          });
+        }
+      }
+    }
+
+    es.close();
+    tg.close();
+  } catch {
+    // Non-critical — multimodal memory works without SQLite bridge
+  }
+}
+```
+
+### Upstream Dependencies
+- `@omnius/memory` must be available at runtime (it is — it's a workspace dependency)
+- The `.omnius_test/` or `.omnius/` directory must exist (created by the daemon on startup)
+
+### Downstream Effects
+- PPR retrieval (`pprRetrieval.ts`) will now find multimodal episodes via person/location nodes
+- Zettelkasten linking will create edges from multimodal episodes to text episodes
+- The memory visualizer will show multimodal episodes with correct modalities
+- Orchestrator context injection (every 3 turns) will surface "Met Alice" episodes
+
+### Success Metrics
+- Call `multimodal_memory(action="capture")` → verify episode appears in both:
+  - `~/.omnius/multimodal-episodes/{id}/episode.json` (JSON)
+  - `SELECT * FROM episodes WHERE tool_name='multimodal_memory' ORDER BY timestamp DESC LIMIT 1` (SQLite)
+- Call `multimodal_memory(action="meet", person_name="Test")` → verify KG node:
+  - `SELECT * FROM kg_nodes WHERE text='Test' AND node_type='person'`
+
+### Failure Modes to Avoid
+- Use `try/catch` around the entire bridge — multimodal_memory must work even if SQLite is unavailable
+- Use `require()` not static import — execution package should not hard-depend on memory package at build time
+- Do NOT duplicate the CLIP embedding computation — reuse the one already computed in captureEpisode
+- Close DB connections immediately after use (the tool is not a long-lived service)
+
+---
+
+## WO-AM-GAP-03: Cross-Modal Embedding Space Alignment
+
+**Priority**: P1 (Important — blocks zettelkasten linking across modalities)
+**Effort**: Large (100+ lines, architectural decision required)
+**Risk**: High (changes embedding semantics, may invalidate existing embeddings)
+
+### Problem
+
+The zettelkasten linker (`zettelkasten.ts`) links episodes by cosine similarity of their embeddings. But episodes from different modalities have incompatible embeddings:
+
+- Text episodes: nomic-embed-text 768d
+- Visual episodes: OpenCLIP ViT-B/32 512d
+- Audio episodes: ECAPA-TDNN 192d
+
+The linker silently skips pairs with mismatched dimensions (episodeStore.ts:306 — `if (ep.embedding.length === qEmb.length)`). Cross-modal linking never happens.
+
+### Root Cause
+
+The embedding pipeline was built per-modality without a unification strategy. Each modality uses a different model with different dimensions.
+
+### Solution Options (choose one)
+
+**Option A: Unified CLIP Space (recommended)**
+
+Use CLIP's text encoder for ALL episode embeddings. CLIP's text and image encoders share a 512d space, so text descriptions of visual content will be similar to actual visual embeddings.
+
+Changes:
+- `packages/memory/src/embeddings.ts`: Add `generateCLIPTextEmbedding()` function that calls OpenCLIP text encoder (512d) instead of nomic-embed-text (768d)
+- `packages/cli/src/api/embedding-workers.ts`: Route text episodes through CLIP text encoder when they're related to multimodal content
+- `scripts/embed-text.py`: Add CLIP text embedding mode
+
+Pros: All embeddings in same space, cross-modal cosine works naturally
+Cons: Requires re-embedding existing text episodes (migration), CLIP text embeddings are weaker for pure-text similarity than nomic-embed
+
+**Option B: Dual Embedding (store both)**
+
+Store two embeddings per episode: one in the modality's native space, one in a shared CLIP text space.
+
+Changes:
+- `packages/memory/src/episodeStore.ts`: Add `clip_embedding BLOB` column
+- `packages/memory/src/zettelkasten.ts`: Use `clip_embedding` for cross-modal linking, `embedding` for within-modality
+
+Pros: Best of both worlds, no migration needed for existing data
+Cons: Doubles embedding storage, more complex query logic
+
+**Option C: Projection Layer**
+
+Learn a linear projection from each modality's space to a shared 512d space.
+
+Cons: Requires training data, complex. Not recommended for now.
+
+### Implementation (Option A — Unified CLIP Space)
+
+**File 1: `scripts/embed-text.py`**
+
+Add CLIP text mode:
+```python
+def embed_clip_text(text):
+    """Embed text using CLIP text encoder for cross-modal matching."""
+    import open_clip
+    model, _, _ = open_clip.create_model_and_transforms('ViT-B-32', pretrained='laion2b_s34b_b79k')
+    tokenizer = open_clip.get_tokenizer('ViT-B-32')
+    tokens = tokenizer([text])
+    with torch.no_grad():
+        emb = model.encode_text(tokens)
+        emb = emb / emb.norm(dim=-1, keepdim=True)
+    return emb.squeeze(0).cpu().numpy().tolist()
+```
+
+**File 2: `packages/cli/src/api/py-embed.ts`**
+
+Add function:
+```typescript
+export function runEmbedTextCLIP(input: { text: string }): Float32Array | null {
+  // Calls embed-text.py with --mode clip
+}
+```
+
+**File 3: `packages/cli/src/api/embedding-workers.ts`**
+
+In the visual embedding worker (line ~105), after computing the CLIP image embedding, also compute CLIP text embedding for the episode content and store it. This ensures text content associated with visual episodes lives in the same embedding space.
+
+**File 4: `packages/memory/src/zettelkasten.ts`**
+
+In `findNeighbors()` (line ~60), add dimension check with fallback:
+```typescript
+// If dimensions mismatch, skip (different embedding spaces)
+if (a.length !== b.length) continue;
+```
+This already exists. With unified CLIP space, all cross-modal episodes will have matching 512d embeddings.
+
+### Success Metrics
+- Visual episode (512d CLIP image) and text episode (512d CLIP text) can be linked by zettelkasten
+- `cosineSimilarity(clipImageEmb, clipTextEmb)` > 0.2 for related content
+- Existing text-only episodes continue to work with nomic-embed (they keep their 768d embeddings and link to each other)
+
+### Failure Modes to Avoid
+- Do NOT replace nomic-embed for pure-text episodes — CLIP text is weaker for text-text similarity
+- Do NOT force re-embedding on startup — make it incremental (new episodes get CLIP, old ones keep nomic)
+- Handle the case where OpenCLIP is not installed gracefully (vision-ml-venv may not exist)
+
+---
+
+## WO-AM-GAP-04: Queryable Audio Features (FFT, Band Energy, Sound Class)
+
+**Priority**: P1 (Important — rich audio data captured but not searchable)
+**Effort**: Medium (40 lines)
+**Risk**: Low (additive, extends existing search)
+
+### Problem
+
+`audio-analyze.ts` computes FFT peak frequencies, 5-band energy distribution, RMS levels, and YAMNet sound classifications. These are stored in episode `metadata` as JSON but the episode store's `search()` function at `episodeStore.ts:260` only searches `content` text and `embedding` vectors. Metadata fields are opaque.
+
+### Root Cause
+
+The episode search was designed for text retrieval + embedding similarity. Structured metadata filtering was planned (WO-AM-03) but never implemented.
+
+### Changes Required
+
+**File 1: `packages/memory/src/episodeStore.ts`**
+
+Location: After the `EpisodeQuery` interface (line 62-72)
+
+Add metadata filter fields:
+```typescript
+export interface EpisodeQuery {
+  // ... existing fields ...
+  /** Filter by metadata key-value (exact match) */
+  metadataFilter?: Record<string, unknown>;
+  /** Filter by sound class (YAMNet classification) */
+  soundClass?: string;
+  /** Filter by RMS level range */
+  rmsRange?: { min?: number; max?: number };
+}
+```
+
+Location: In the `search()` method (line ~267-279), after the SQL WHERE clause builder
+
+Add metadata filtering in the post-fetch scoring phase:
+```typescript
+// Metadata filtering (post-fetch, JSON parsing)
+if (query.metadataFilter || query.soundClass || query.rmsRange) {
+  candidates = candidates.filter(ep => {
+    if (!ep.metadata) return false;
+    const meta = typeof ep.metadata === "string" ? JSON.parse(ep.metadata) : ep.metadata;
+
+    if (query.metadataFilter) {
+      for (const [k, v] of Object.entries(query.metadataFilter)) {
+        if (meta[k] !== v) return false;
+      }
+    }
+    if (query.soundClass && meta.sound_class !== query.soundClass) return false;
+    if (query.rmsRange) {
+      const rms = meta.rms_db ?? meta.rmsDb;
+      if (typeof rms !== "number") return false;
+      if (query.rmsRange.min !== undefined && rms < query.rmsRange.min) return false;
+      if (query.rmsRange.max !== undefined && rms > query.rmsRange.max) return false;
+    }
+    return true;
+  });
+}
+```
+
+**File 2: `packages/execution/src/tools/audio-analyze.ts`**
+
+Location: After the tool result is returned from each analysis action
+
+Ensure metadata is stored on the episode by including it in the tool result content:
+```typescript
+// In classifyAudio (line ~130):
+// Already returns JSON with classifications — content will be captured by orchestrator
+
+// In analyzeSpectrum (line ~230):
+// Already returns JSON with peak_frequencies, band_energy_db, rms_db
+```
+
+The orchestrator already stores the tool output as episode content (line 2894-2895). The metadata is embedded in the content string. For structured query, we need to also store key features in the `metadata` field.
+
+**File 3: `packages/orchestrator/src/agenticRunner.ts`**
+
+Location: Lines 2892-2910 (episode insertion block)
+
+After `const episodeContent = ...`, add metadata extraction for audio tools:
+```typescript
+let episodeMetadata: Record<string, unknown> = {
+  args_fingerprint: argsKey.slice(0, 200),
+  success: result.success,
+  duration_ms: performance.now() - toolStart,
+};
+
+// Extract structured audio features into metadata for queryable filtering
+if (result.success && ["audio_analyze", "audio_capture"].includes(tc.name)) {
+  try {
+    const parsed = JSON.parse(result.output ?? "{}");
+    if (parsed.rms_db !== undefined) episodeMetadata.rms_db = parsed.rms_db;
+    if (parsed.classifications) episodeMetadata.sound_class = parsed.classifications[0]?.class;
+    if (parsed.peak_frequencies) episodeMetadata.peak_frequencies = parsed.peak_frequencies.slice(0, 3);
+    if (parsed.band_energy_db) episodeMetadata.band_energy = parsed.band_energy_db;
+  } catch { /* tool output may not be JSON */ }
+}
+```
+
+### Success Metrics
+- `episodeStore.search({ soundClass: "Speech" })` returns only audio episodes classified as speech
+- `episodeStore.search({ rmsRange: { min: -30 } })` returns only loud audio episodes
+- Existing text search continues to work unchanged
+
+### Failure Modes to Avoid
+- Metadata filtering is post-fetch (JavaScript), not SQL WHERE — this is intentional. SQLite JSON operators are slow on large datasets. For <10K episodes this is fine.
+- Do NOT parse every episode's metadata on search — only parse if `metadataFilter`, `soundClass`, or `rmsRange` are specified in the query
+- Handle non-JSON metadata gracefully (some old episodes may have string metadata)
+
+---
+
+## WO-AM-GAP-05: Activate Ebbinghaus Strength in Scoring Formula
+
+**Priority**: P2 (Enhancement — strength is computed but unused)
+**Effort**: Small (5 lines changed)
+**Risk**: Low (additive scoring factor)
+
+### Problem
+
+The Ebbinghaus strength field is incremented on every retrieval (`episodeStore.ts:325-330`) but never used in the scoring formula at line 318. The score is `recency + importance + relevance` — strength is a dead field that grows but has no effect on ranking.
+
+### Root Cause
+
+Planned for WO-AM-02 (per code comments) but never implemented.
+
+### Changes Required
+
+**File 1: `packages/memory/src/episodeStore.ts`**
+
+Location: Line 318 (the scoring formula)
+
+Replace:
+```typescript
+const score = recency + importance + relevance;
+```
+
+With:
+```typescript
+// Ebbinghaus strength bonus: frequently retrieved episodes surface higher.
+// Log scale prevents runaway scores from heavily-retrieved episodes.
+// strength=1 (never retrieved) → bonus=0
+// strength=3 (retrieved twice) → bonus=0.48
+// strength=10 (retrieved 9 times) → bonus=1.0
+const strengthBonus = Math.min(1.0, Math.log2(Math.max(1, ep.strength)));
+const score = recency + importance + relevance + strengthBonus;
+```
+
+### Upstream Dependencies
+- None. `ep.strength` is already populated by `insert()` (default 1.0) and incremented by `search()` (line 325).
+
+### Downstream Effects
+- Episodes that are frequently retrieved will rank higher in future searches
+- The maximum score increases from 4.0 to 5.0 (recency 1.0 + importance 1.0 + relevance 2.0 + strength 1.0)
+- PPR retrieval uses its own scoring (PPR scores from PageRank), so this only affects direct episode search
+
+### Success Metrics
+- Insert episode A and B with same content/importance/timestamp
+- Search and retrieve A three times (strength goes to 4)
+- Search again — A should rank higher than B due to strength bonus
+- `strengthBonus(4) = log2(4) = 2.0` → capped to 1.0
+
+### Failure Modes to Avoid
+- Use `log2` not linear — linear strength would make old frequently-retrieved episodes dominate forever
+- Cap at 1.0 — strength should be a tiebreaker, not the dominant factor
+- Do NOT modify the PPR scoring in `pprRetrieval.ts` — that uses graph-based scores, not episode scores
+- Ensure `Math.max(1, ep.strength)` to handle any episodes where strength is 0 or null
+
+---
+
+## WO-AM-GAP-06: Modality-Aware Decay Classification
+
+**Priority**: P2 (Enhancement — more realistic memory dynamics)
+**Effort**: Small (15 lines changed)
+**Risk**: Low (changes defaults, existing episodes keep their assigned decay)
+
+### Problem
+
+`autoDecayClass()` in `episodeStore.ts:154-179` assigns `"daily"` (24h half-life) to visual and audio episodes. But:
+- Face memories should last weeks/months (humans remember faces for years)
+- Social associations (names) should be nearly permanent
+- Speech content decays faster than faces
+- Tool results (`file_read` output) decay fastest
+
+### Root Cause
+
+The decay classification was written with a focus on text tool results. When multimodal modalities were added, they all got `"daily"` as a reasonable default without deeper consideration.
+
+### Changes Required
+
+**File 1: `packages/memory/src/episodeStore.ts`**
+
+Location: Lines 154-179 (`autoDecayClass()` function)
+
+Replace the multimodal section:
+```typescript
+// Current (lines ~168-170):
+if (["visual", "audio", "social", "spatial"].includes(modality)) return "daily";
+```
+
+With modality-specific decay:
+```typescript
+// Modality-specific decay — matches human memory dynamics
+if (modality === "social") return "permanent";     // Names, relationships persist
+if (modality === "visual") return "procedural";    // Faces last weeks/months
+if (modality === "spatial") return "procedural";   // Locations persist
+if (modality === "audio") return "daily";          // Speech content fades in days
+```
+
+Also update `autoImportance()` at lines 134-135 to differentiate:
+```typescript
+// Current:
+if (modality === "visual" || modality === "audio") return 7;
+if (modality === "social") return 8;
+
+// Updated:
+if (modality === "social") return 9;    // People are most important
+if (modality === "visual") return 7;    // Sights are notable
+if (modality === "audio") return 6;     // Sounds are contextual
+if (modality === "spatial") return 5;   // Locations are background
+```
+
+### Downstream Effects
+- Social episodes ("Met Alice") will never decay — they'll always be retrievable
+- Visual episodes (face detections) will persist for 30 days before 50% decay
+- Audio episodes (speech, sounds) will decay in 24h (same as before)
+- The memory visualizer's episode view will show more variety in decay indicators
+
+### Success Metrics
+- `autoDecayClass("social")` returns `"permanent"`
+- `autoDecayClass("visual")` returns `"procedural"`
+- `autoDecayClass("audio")` returns `"daily"` (unchanged)
+- Social episode from 7 days ago still has recency > 0.95 (permanent, tau=infinity)
+- Visual episode from 7 days ago has recency ~0.85 (procedural, tau=30d)
+
+### Failure Modes to Avoid
+- Do NOT retroactively change existing episodes' decay class — only affect new inserts
+- Do NOT make tool_result episodes permanent — they must decay (session or daily)
+- Test that `pruneExpired()` still works — it only prunes `session` class episodes older than 3h
+
+---
+
+## Implementation Order
+
+| Order | Work Order | Depends On | Estimated Time |
+|-------|-----------|-----------|----------------|
+| 1 | WO-AM-GAP-01 (Modality tagging) | None | 30 min |
+| 2 | WO-AM-GAP-06 (Modality decay) | GAP-01 | 15 min |
+| 3 | WO-AM-GAP-05 (Ebbinghaus activation) | None | 15 min |
+| 4 | WO-AM-GAP-02 (Multimodal bridge) | GAP-01 | 1 hour |
+| 5 | WO-AM-GAP-04 (Audio features query) | GAP-01 | 45 min |
+| 6 | WO-AM-GAP-03 (Embedding alignment) | GAP-02 | 2-3 hours |
+
+GAP-01 is the foundation — all others build on correct modality tagging. GAP-03 is the largest and should be done last after validating the simpler fixes.
+
+---
+
+## Verification Checklist
+
+After all work orders are implemented:
+
+- [ ] `npx tsc --noEmit -p packages/orchestrator/tsconfig.json` — zero errors
+- [ ] `npx tsc --noEmit -p packages/memory/tsconfig.json` — zero errors (new types)
+- [ ] `pnpm test` in packages/orchestrator — all tests pass
+- [ ] `pnpm test` in packages/memory — all tests pass
+- [ ] Manual: `vision(action="caption")` → episode has `modality: "visual"` in SQLite
+- [ ] Manual: `audio_capture(duration=3)` → episode has `modality: "audio"` in SQLite
+- [ ] Manual: `multimodal_memory(action="meet", person_name="Test")` → episode in BOTH JSON and SQLite
+- [ ] Manual: `/memory episodes` → shows visual/audio/social episodes with correct icons
+- [ ] Manual: Episode search returns cross-modal results after GAP-03
+- [ ] Manual: Frequently retrieved episode ranks higher after GAP-05
+
+---
+
+## Research References
+
+- Generative Agents (2304.03442) — triple-factor retrieval
+- HippoRAG (2405.14831) — PPR over knowledge graphs
+- JARVIS-1 (2311.05997) — two-stage cross-modal retrieval
+- A-MEM (2502.12110) — retroactive memory evolution
+- MemoryOS (2506.06326) — 3-tier memory with heat-score eviction
+- ReadAgent (2402.09727) — gist compression (3.5-20x context extension)
+- Graphiti (getzep/graphiti) — temporal knowledge graph with fact supersession
+- MemoryBank — Ebbinghaus strength increments on retrieval
+- RGMem (2510.16392) — phase-transition threshold for abstraction