@memtensor/memos-local-openclaw-plugin 0.1.4 → 0.1.6

package/README.md

@@ -1,23 +1,62 @@
-# 🦞 MemOS Local — OpenClaw Memory Plugin
+# 🧠 MemOS Local — OpenClaw Memory Plugin
 
-Persistent local conversation memory for [OpenClaw](https://github.com/nicepkg/openclaw) AI Agents. Every conversation is automatically captured, semantically indexed, and instantly recallable.
+Persistent local conversation memory for [OpenClaw](https://github.com/nicepkg/openclaw) AI Agents. Every conversation is automatically captured, semantically indexed, and instantly recallable — with **smart task summarization** and **automatic skill evolution**.
 
-**Full-write | Hybrid Search (FTS5 + Vector) | RRF Fusion | MMR Diversity | Recency Decay**
+**Full-write | Hybrid Search | Task Summarization | Skill Evolution | Memory Viewer**
+
+## Why MemOS Local
+
+| Problem | Solution |
+|---------|----------|
+| Agent forgets everything between sessions | **Persistent memory** — every conversation auto-captured to local SQLite |
+| Fragmented memory chunks lack context | **Smart task summarization** — conversations organized into structured tasks with goals, steps, results |
+| Agent repeats past mistakes on similar tasks | **Skill evolution** — reusable skills auto-generated from real executions, continuously upgraded |
+| No visibility into what the agent remembers | **Memory Viewer** — full visualization of all memories, tasks, and skills |
+| Privacy concerns with cloud storage | **100% local** — zero cloud uploads, zero telemetry, password-protected |
 
 ## Features
 
-- **Auto-capture** — Stores user, assistant, and tool messages after each agent turn via `agent_end` event
-- **Content deduplication** — SHA-256 content hashing prevents duplicate chunks across repeated ingestion
-- **Task-level organization** — Conversations auto-grouped into tasks by topic shift (LLM-judged) or 2-hour idle timeout; each completed task gets a detailed LLM summary
-- **Task status** — `active` (in progress), `completed` (with LLM summary), `skipped` (too brief, excluded from search)
+### Memory Engine
+- **Auto-capture** — Stores user, assistant, and tool messages after each agent turn via `agent_end` event (consecutive assistant messages merged into one)
+- **Smart deduplication** — Exact content-hash skip; then Top-5 similar chunks (threshold 0.75) with LLM judge: DUPLICATE (skip), UPDATE (merge summary + append content), or NEW (create). Evolved chunks track merge history.
+- **Semantic chunking** — Splits by code blocks, function bodies, paragraphs; never cuts mid-function
 - **Hybrid retrieval** — FTS5 keyword + vector semantic dual-channel search with RRF fusion
-- **LLM summarization** — Detailed task summaries preserving code, commands, URLs, file paths; chunk-level short summaries for search indexing
+- **MMR diversity** — Maximal Marginal Relevance reranking prevents near-duplicate results
+- **Recency decay** — Configurable time-based decay (half-life: 14 days) biases recent memories
 - **Multi-provider embedding** — OpenAI-compatible, Gemini, Cohere, Voyage, Mistral, or local offline (Xenova/all-MiniLM-L6-v2)
-- **Multi-provider summarizer** — OpenAI-compatible, Anthropic, Gemini, AWS Bedrock
-- **Web viewer** — Built-in dashboard at `http://127.0.0.1:18799` with CRUD, search, analytics, task browser, chat-bubble chunk view
-- **Skill-based retrieval** — Agent learns when and how to use memory tools via a bundled Skill (no system prompt injection)
-- **i18n** — Memory Viewer supports Chinese / English with one-click toggle
-- **Privacy first** — All data in local SQLite, no cloud uploads, password-protected viewer
+
+### Task Summarization
+- **Auto task boundary detection** — LLM topic judgment + 2-hour idle timeout segments conversations into tasks
+- **Structured summaries** — LLM generates Goal, Key Steps, Result, Key Details for each completed task
+- **Key detail preservation** — Code, commands, URLs, file paths, error messages retained in summaries
+- **Quality filtering** — Tasks with too few chunks, too few turns, or trivial content are auto-skipped
+- **Task status** — `active` (in progress), `completed` (with LLM summary), `skipped` (too brief, excluded from search)
+
+### Skill Evolution
+- **Automatic evaluation** — After task completion, rule filter + LLM evaluates if the task is worth distilling into a skill
+- **Skill generation** — Multi-step LLM pipeline creates SKILL.md + scripts + references + evals from real execution records
+- **Skill upgrading** — When similar tasks appear, existing skills are auto-upgraded (refine / extend / fix)
+- **Quality scoring** — 0-10 quality assessment; scores below 6 marked as draft
+- **Version management** — Full version history with changelog, change summary, and upgrade type tracking
+- **Auto-install** — Generated skills can be auto-installed into the workspace for immediate use
+- **Dedicated model** — Optional separate LLM model for skill generation (e.g., Claude 4.6 for higher quality)
+
+### Memory Viewer
+- **6 management pages** — Memories, Tasks, Skills, Analytics, **Logs**, Settings
+- **Full CRUD** — Create, edit, delete, search memories; evolution badges and merge history on memory cards
+- **Task browser** — Status filters, chat-bubble chunk view, structured summaries, skill generation status
+- **Skill browser** — Version history, quality scores, one-click download as ZIP
+- **Analytics dashboard** — Daily read/write activity, memory breakdown charts
+- **Logs** — Tool call log (memory_search, auto_recall, memory_add, etc.) with input/output and duration; filter by tool, auto-refresh
+- **Online configuration** — Modify embedding, summarizer, skill evolution settings via web UI
+- **Security** — Password-protected, localhost-only (127.0.0.1), session cookies
+- **i18n** — Chinese / English toggle
+- **Themes** — Light / Dark mode
+
+### Privacy & Security
+- **100% on-device** — All data in local SQLite, no cloud uploads, no telemetry
+- **Viewer security** — Binds to 127.0.0.1 only, password-protected with session cookies
+- **Auto-recall + Skill** — Each turn, relevant memories are injected via `before_agent_start` hook (invisible to user). When nothing is recalled (e.g. long or unclear query), the agent is prompted to call `memory_search` with a self-generated short query. The bundled skill `memos-memory-guide` documents all tools and when to use them.
 
 ## Quick Start
 
@@ -110,6 +149,30 @@ Add the plugin config to `~/.openclaw/openclaw.json`:
 
 > **No summarizer config?** A rule-based fallback generates summaries from the first sentence + key entities. Good enough to start.
 
+#### Skill Evolution Configuration (Optional)
+
+You can optionally configure a dedicated model for skill generation (for higher quality skills):
+
+```jsonc
+{
+  "config": {
+    "skillSummarizer": {
+      "provider": "anthropic",
+      "apiKey": "sk-ant-xxx",
+      "model": "claude-sonnet-4-20250514",
+      "temperature": 0
+    },
+    "skillEvolution": {
+      "enabled": true,
+      "autoEvaluate": true,
+      "autoInstall": false
+    }
+  }
+}
+```
+
+If `skillSummarizer` is not configured, the plugin uses the regular `summarizer` model for skill generation.
+
 #### Environment Variable Support
 
 Use `${ENV_VAR}` placeholders in config to avoid hardcoding keys:
@@ -161,61 +224,127 @@ You: 你还记得我之前让你帮我处理过什么事情吗？
 Agent: (calls memory_search) 是的，我们之前讨论过...
 ```
 
-## Retrieval Strategy
+## How It Works
 
-The plugin uses a **Skill-based** approach to teach the agent when and how to use memory tools. A `SKILL.md` file is automatically installed into the workspace skills directory (`~/.openclaw/workspace/skills/memos-local/`).
+### Three Intelligent Pipelines
 
-The Skill instructs the agent to follow a **layered retrieval strategy**:
+MemOS Local operates through three interconnected pipelines that form a continuous learning loop:
 
-1. **`memory_search`** — Lightweight search returning hit summaries + IDs. Always the first step.
-2. **Sufficiency check** — If summaries are enough, answer directly without drilling down.
-3. **`memory_get`** — Read the full original text of a specific hit when the summary is truncated or insufficient.
-4. **`task_summary`** — Get the complete task-level summary when the answer may be spread across multiple turns of the same task.
-5. **`memory_timeline`** — Expand surrounding conversation context when cause-and-effect or chronological details are needed.
-6. **Ask user** — Only after exhausting search budget, ask the user a minimal follow-up question.
+```
+Conversation → Memory Write Pipeline → Task Generation Pipeline → Skill Evolution Pipeline
+                                                                          ↓
+                              Smart Retrieval Pipeline ← ← ← ← ← ← ← ← ←
+```
+
+### Pipeline 1: Memory Write (auto on every agent turn)
+
+```
+Conversation → Capture (filter roles, strip system prompts)
+→ Semantic chunking (code blocks, paragraphs, error stacks)
+→ Content hash dedup → LLM summarize each chunk
+→ Vector embedding → Store (SQLite + FTS5 + Vector)
+```
 
-The Skill triggers when:
-- The current context is insufficient to answer definitively
-- The agent is about to ask the user for information (search memory first)
-- The user references past conversations
-- User-specific information is needed (identity, preferences, project config, etc.)
+- System messages are skipped; tool results from the plugin's own tools are not re-stored
+- Evidence wrapper blocks (`[STORED_MEMORY]...[/STORED_MEMORY]`) are stripped to prevent feedback loops
+- Content hash (SHA-256, first 16 hex chars) prevents duplicate chunk ingestion within the same session+role
+
+### Pipeline 2: Task Generation (auto after memory write)
+
+```
+New chunks → Task boundary detection (LLM topic judge / 2h idle / session change)
+→ Boundary crossed? → Finalize previous task
+  → Chunks ≥ 4 & turns ≥ 2? → LLM structured summary → status = "completed"
+  → Otherwise → status = "skipped" (excluded from search)
+```
+
+**Why Tasks matter:**
+- Raw memory chunks are fragmented — a single conversation about "deploying Nginx" might span 20 chunks
+- Task summarization organizes these fragments into a structured record: Goal → Steps → Result → Key Details
+- When the agent searches memory, it can quickly locate the complete experience via `task_summary`, not just fragments
+- Task summaries preserve code, commands, URLs, configs, and error messages
+
+### Pipeline 3: Skill Evolution (auto after task completion)
+
+```
+Completed task → Rule filter (min chunks, non-trivial content)
+→ Search for related existing skills
+  → Related skill found (confidence ≥ 0.7)?
+    → Evaluate upgrade (refine/extend/fix) → Merge new experience → Version bump
+  → No related skill (or confidence < 0.3)?
+    → Evaluate create → Generate SKILL.md + scripts + evals
+    → Quality score (0-10) → Install if score ≥ 6
+```
+
+**Why Skills matter:**
+- Without skills, agents rediscover solutions every time they encounter similar problems
+- Skills crystallize successful executions into reusable guides with steps, pitfall warnings, and verification checks
+- Skills auto-upgrade when new tasks bring improved approaches — getting faster, more accurate, and more token-efficient
+- The evolution is automatic: task completes → evaluate → create/upgrade → install
+
+### Pipeline 4: Smart Retrieval
+
+**Auto-recall (every turn):** The plugin hooks `before_agent_start`, runs a memory search with the user's message, then uses an LLM to filter which candidates are relevant and whether they are sufficient to answer. The filtered memories are injected into the agent's system context (invisible to the user). If no memories are found or the query is long/unclear, the agent is prompted to call `memory_search` with a self-generated short query.
+
+**On-demand search (`memory_search`):**
+```
+Query → FTS5 + Vector dual recall → RRF Fusion → MMR Rerank
+→ Recency Decay → Score Filter → Top-K (e.g. 20)
+→ LLM relevance filter (minimum information) → Dedup by excerpt overlap
+→ Return excerpts + chunkId / task_id (no summaries)
+  → sufficient=false → suggest task_summary(taskId), skill_get(taskId), memory_timeline(chunkId)
+```
+
+- **RRF (Reciprocal Rank Fusion):** Merges FTS5 and vector search rankings into a unified score
+- **MMR (Maximal Marginal Relevance):** Re-ranks to balance relevance with diversity
+- **Recency Decay:** Recent memories get a boost (half-life: 14 days by default)
+- **LLM filter:** Only memories that are genuinely useful for the query are returned; sufficiency determines whether follow-up tool tips are appended
+
+## Retrieval Strategy
+
+1. **Auto-recall (hook)** — On every turn, the plugin runs a memory search using the user's message and injects LLM-filtered relevant memories into the agent's context (via `before_agent_start`). The agent sees this as system context; the user does not.
+2. **When nothing is recalled** — If the user's message is long, vague, or no matches are found, the plugin injects a short hint telling the agent to call **`memory_search`** with a **self-generated short query** (e.g. key topics or a rephrased question).
+3. **Bundled skill** — The plugin installs `memos-memory-guide` into `~/.openclaw/workspace/skills/memos-memory-guide/` and `~/.openclaw/skills/memos-memory-guide/`. This skill documents all memory tools, when to call them, and how to write good search queries. Add `skills.load.extraDirs: ["~/.openclaw/skills"]` in `openclaw.json` if you want the skill to appear in the OpenClaw skills dashboard.
+4. **Search results** — `memory_search` returns **excerpts** (original content snippets) and IDs (`chunkId`, `task_id`), not summaries. The agent uses `task_summary(taskId)`, `memory_timeline(chunkId)`, and `skill_get(skillId|taskId)` to drill down when needed. There is no `memory_get`; full context is obtained via `task_summary` or `memory_timeline`.
 
 ## Agent Tools
 
-The plugin registers 5 tools:
+The plugin registers **6 tools** and auto-installs the **memos-memory-guide** skill:
 
 | Tool | Purpose | When to Use |
-|---|---|---|
-| `memory_search` | Search memories by natural language query | When current context lacks needed facts; before asking the user for information |
-| `memory_get` | Get full original text of a memory chunk | When exact details (commands, code, errors) are needed from a specific hit |
-| `task_summary` | Get detailed summary of a completed task | When the answer may be spread across multiple turns of the same task |
-| `memory_timeline` | Get surrounding context for a search hit | When a search hit needs cause-and-effect or chronological context |
-| `memory_viewer` | Get the URL of the web dashboard | When user asks where to view/manage memories |
+|------|---------|-------------|
+| `memory_search` | Search memories; returns excerpts + `chunkId` / `task_id` | When auto-recall returned nothing or you need a different query; use a short, self-generated query if the user's message was long or unclear |
+| `task_summary` | Full structured summary of a completed task | When a hit has `task_id` and you need the full story (goal, steps, result) |
+| `memory_timeline` | Surrounding conversation around a chunk | When you need the exact dialogue before/after a hit; pass `chunkId` from the hit |
+| `skill_get` | Get skill content by `skillId` or `taskId` | When a hit has a linked task/skill and you want the reusable experience guide |
+| `skill_install` | Install a skill into the agent workspace | When the skill should be permanently available for future turns |
+| `memory_viewer` | Get the URL of the Memory Viewer web UI | When the user asks where to view or manage their memories |
+
+There is no `memory_get`; search returns excerpts and IDs; use `task_summary` or `memory_timeline` for deeper context.
 
 ### Search Parameters
 
 | Parameter | Default | Range | Description |
-|---|---|---|---|
-| `query` | — | — | Natural language search query |
-| `maxResults` | 6 | 1–20 | Maximum number of results |
+|-----------|---------|-------|-------------|
+| `query` | — | — | Natural language search query (keep it short and focused) |
+| `maxResults` | 20 | 1–20 | Maximum candidates before LLM filter |
 | `minScore` | 0.45 | 0.35–1.0 | Minimum relevance score |
-| `role` | — | `user` / `assistant` / `tool` | Filter by message role |
+| `role` | — | `user` / `assistant` / `tool` | Filter by message role (e.g. `user` to find what the user said) |
 
 ## Memory Viewer
 
 Open `http://127.0.0.1:18799` in your browser after starting the gateway.
 
-**Features:**
-- Browse all stored memories with timeline view
-- Semantic search powered by your embedding model
-- Create, edit, and delete memories (CRUD)
-- Task browser with status filters (active / completed / skipped)
-- Chat-bubble style chunk view in task details
-- Filter by session, role (user/assistant/tool), time range
-- Analytics dashboard — daily write/read activity charts, memory breakdown
-- Light / Dark theme toggle
-- Chinese / English language toggle
-- Pagination (30 per page)
+**Pages:**
+
+| Page | Features |
+|------|----------|
+| **Memories** | Timeline view, pagination, session/role/kind/date filters, CRUD, semantic search; evolution badges and merge history on cards |
+| **Tasks** | Task list with status filters (active/completed/skipped), chat-bubble chunk view, structured summaries, skill generation status |
+| **Skills** | Skill list with status badges, version history with changelogs, quality scores, related tasks, one-click ZIP download |
+| **Analytics** | Daily write/read activity charts, memory/task/skill totals, role breakdown |
+| **Logs** | Tool call log (memory_search, auto_recall, memory_add, etc.) with input/output, duration, and tool filter; auto-refresh |
+| **Settings** | Online configuration for embedding model, summarizer model, skill evolution settings, viewer port |
 
 **Viewer won't open?**
 
@@ -232,39 +361,6 @@ grep "password reset token:" ~/.openclaw/logs/gateway.log 2>/dev/null | tail -1
 
 Copy the 32-character hex string after `password reset token:`.
 
-## How It Works
-
-**Write path** (automatic on every agent turn via `agent_end` event):
-
-```
-Conversation → Capture (filter roles, strip system prompts)
-→ Content hash dedup check → Store chunk (SQLite + FTS5 + Embed)
-→ Associate to Task → Task boundary detected?
-  → Yes: Finalize previous task
-    → Chunks ≥ 4 & turns ≥ 2: LLM summarize → status = "completed"
-    → Otherwise: status = "skipped" (excluded from search)
-  → No: Continue accumulating in current task
-```
-
-Key details:
-- System messages are skipped; tool results from the plugin's own tools are not re-stored
-- Evidence wrapper blocks (`[STORED_MEMORY]...[/STORED_MEMORY]`) are stripped to prevent feedback loops
-- Content hash (SHA-256, first 16 hex chars) prevents duplicate chunk ingestion within the same session+role
-- Task boundaries are detected by LLM topic judgment or 2-hour idle timeout
-
-**Read path** (when agent calls `memory_search`):
-
-```
-Query → FTS5 + Vector dual recall → RRF Fusion → MMR Rerank
-→ Recency Decay → Score Filter → Top-K Results
-```
-
-Key details:
-- **RRF (Reciprocal Rank Fusion):** Merges FTS5 and vector search rankings into a unified score
-- **MMR (Maximal Marginal Relevance):** Re-ranks to balance relevance with diversity, preventing redundant results
-- **Recency Decay:** Recent memories get a boost (half-life: 14 days by default)
-- **Score Filter:** Relative threshold based on the top hit's score, preventing over-filtering with RRF's compressed score range
-
 ## Advanced Configuration
 
 All optional — shown with defaults:
@@ -282,7 +378,16 @@ All optional — shown with defaults:
       "recencyHalfLifeDays": 14   // Time decay half-life
     },
     "dedup": {
-      "similarityThreshold": 0.93 // Cosine similarity for dedup
+      "similarityThreshold": 0.75,  // Cosine similarity for smart-dedup candidates (Top-5)
+      "enableSmartMerge": true,     // LLM judge: DUPLICATE / UPDATE / NEW
+      "maxCandidates": 5            // Max similar chunks to send to LLM
+    },
+    "skillEvolution": {
+      "enabled": true,            // Enable skill evolution
+      "autoEvaluate": true,       // Auto-evaluate tasks for skill generation
+      "minChunksForEval": 6,      // Min chunks for a task to be evaluated
+      "minConfidence": 0.7,       // Min LLM confidence to create/upgrade skill
+      "autoInstall": false        // Auto-install generated skills
     },
     "viewerPort": 18799           // Memory Viewer port
   }
@@ -352,6 +457,11 @@ Then restart the gateway.
 
 6. **记忆和 OpenClaw 内置记忆冲突** — 如果看到 agent 同时调用了内置的 memory 搜索和插件的 `memory_search`，说明 `agents.defaults.memorySearch.enabled` 没有设为 `false`。
 
+7. **技能不生成** — 检查：
+   - `skillEvolution.enabled` 是否为 `true`
+   - 任务是否有足够内容（默认需要 ≥ 6 chunks）
+   - 查看日志中 `SkillEvolver` 相关输出
+
 ## Data Location
 
 | File | Path |
@@ -360,7 +470,9 @@ Then restart the gateway.
 | Viewer auth | `~/.openclaw/memos-local/viewer-auth.json` |
 | Gateway log | `~/.openclaw/logs/gateway.log` |
 | Plugin code | `~/.openclaw/extensions/memos-local/` |
-| Skill | `~/.openclaw/workspace/skills/memos-local/SKILL.md` |
+| Memory-guide skill | `~/.openclaw/workspace/skills/memos-memory-guide/SKILL.md` (and `~/.openclaw/skills/memos-memory-guide/`) |
+| Generated skills | `~/.openclaw/memos-local/skills-store/<skill-name>/` |
+| Installed skills | `~/.openclaw/workspace/skills/<skill-name>/` |
 
 ## Testing
 

package/dist/ingest/dedup.d.ts

@@ -8,4 +8,12 @@ import type { Logger } from "../types";
  * PRD §4.4: dedup threshold 0.92–0.95
  */
 export declare function findDuplicate(store: SqliteStore, newVec: number[], threshold: number, log: Logger): string | null;
+/**
+ * Find Top-N most similar chunks above a threshold.
+ * Used for smart dedup: retrieve candidates, then ask LLM to judge.
+ */
+export declare function findTopSimilar(store: SqliteStore, newVec: number[], threshold: number, topN: number, log: Logger): Array<{
+    chunkId: string;
+    score: number;
+}>;
 //# sourceMappingURL=dedup.d.ts.map

package/dist/ingest/dedup.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"dedup.d.ts","sourceRoot":"","sources":["../../src/ingest/dedup.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,mBAAmB,CAAC;AACrD,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,UAAU,CAAC;AAEvC;;;;;;GAMG;AACH,wBAAgB,aAAa,CAC3B,KAAK,EAAE,WAAW,EAClB,MAAM,EAAE,MAAM,EAAE,EAChB,SAAS,EAAE,MAAM,EACjB,GAAG,EAAE,MAAM,GACV,MAAM,GAAG,IAAI,CAoBf"}
+{"version":3,"file":"dedup.d.ts","sourceRoot":"","sources":["../../src/ingest/dedup.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,mBAAmB,CAAC;AACrD,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,UAAU,CAAC;AAEvC;;;;;;GAMG;AACH,wBAAgB,aAAa,CAC3B,KAAK,EAAE,WAAW,EAClB,MAAM,EAAE,MAAM,EAAE,EAChB,SAAS,EAAE,MAAM,EACjB,GAAG,EAAE,MAAM,GACV,MAAM,GAAG,IAAI,CAoBf;AAED;;;GAGG;AACH,wBAAgB,cAAc,CAC5B,KAAK,EAAE,WAAW,EAClB,MAAM,EAAE,MAAM,EAAE,EAChB,SAAS,EAAE,MAAM,EACjB,IAAI,EAAE,MAAM,EACZ,GAAG,EAAE,MAAM,GACV,KAAK,CAAC;IAAE,OAAO,EAAE,MAAM,CAAC;IAAC,KAAK,EAAE,MAAM,CAAA;CAAE,CAAC,CAiB3C"}

package/dist/ingest/dedup.js

@@ -1,6 +1,7 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.findDuplicate = findDuplicate;
+exports.findTopSimilar = findTopSimilar;
 const vector_1 = require("../storage/vector");
 /**
  * Check if a new summary embedding is a near-duplicate of any
@@ -26,4 +27,24 @@ function findDuplicate(store, newVec, threshold, log) {
     }
     return null;
 }
+/**
+ * Find Top-N most similar chunks above a threshold.
+ * Used for smart dedup: retrieve candidates, then ask LLM to judge.
+ */
+function findTopSimilar(store, newVec, threshold, topN, log) {
+    const all = store.getAllEmbeddings();
+    const scored = [];
+    for (const { chunkId, vector } of all) {
+        const sim = (0, vector_1.cosineSimilarity)(newVec, vector);
+        if (sim >= threshold) {
+            scored.push({ chunkId, score: sim });
+        }
+    }
+    scored.sort((a, b) => b.score - a.score);
+    const result = scored.slice(0, topN);
+    if (result.length > 0) {
+        log.debug(`findTopSimilar: found ${result.length} candidates above ${threshold} (best=${result[0].score.toFixed(4)})`);
+    }
+    return result;
+}
 //# sourceMappingURL=dedup.js.map

package/dist/ingest/dedup.js.map

@@ -1 +1 @@
-{"version":3,"file":"dedup.js","sourceRoot":"","sources":["../../src/ingest/dedup.ts"],"names":[],"mappings":";;AAWA,sCAyBC;AApCD,8CAAqD;AAIrD;;;;;;GAMG;AACH,SAAgB,aAAa,CAC3B,KAAkB,EAClB,MAAgB,EAChB,SAAiB,EACjB,GAAW;IAEX,MAAM,GAAG,GAAG,KAAK,CAAC,gBAAgB,EAAE,CAAC;IAErC,IAAI,MAAM,GAAkB,IAAI,CAAC;IACjC,IAAI,SAAS,GAAG,CAAC,CAAC;IAElB,KAAK,MAAM,EAAE,OAAO,EAAE,MAAM,EAAE,IAAI,GAAG,EAAE,CAAC;QACtC,MAAM,GAAG,GAAG,IAAA,yBAAgB,EAAC,MAAM,EAAE,MAAM,CAAC,CAAC;QAC7C,IAAI,GAAG,GAAG,SAAS,EAAE,CAAC;YACpB,SAAS,GAAG,GAAG,CAAC;YAChB,MAAM,GAAG,OAAO,CAAC;QACnB,CAAC;IACH,CAAC;IAED,IAAI,MAAM,IAAI,SAAS,IAAI,SAAS,EAAE,CAAC;QACrC,GAAG,CAAC,KAAK,CAAC,gCAAgC,MAAM,QAAQ,SAAS,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC;QAChF,OAAO,MAAM,CAAC;IAChB,CAAC;IAED,OAAO,IAAI,CAAC;AACd,CAAC"}
+{"version":3,"file":"dedup.js","sourceRoot":"","sources":["../../src/ingest/dedup.ts"],"names":[],"mappings":";;AAWA,sCAyBC;AAMD,wCAuBC;AAjED,8CAAqD;AAIrD;;;;;;GAMG;AACH,SAAgB,aAAa,CAC3B,KAAkB,EAClB,MAAgB,EAChB,SAAiB,EACjB,GAAW;IAEX,MAAM,GAAG,GAAG,KAAK,CAAC,gBAAgB,EAAE,CAAC;IAErC,IAAI,MAAM,GAAkB,IAAI,CAAC;IACjC,IAAI,SAAS,GAAG,CAAC,CAAC;IAElB,KAAK,MAAM,EAAE,OAAO,EAAE,MAAM,EAAE,IAAI,GAAG,EAAE,CAAC;QACtC,MAAM,GAAG,GAAG,IAAA,yBAAgB,EAAC,MAAM,EAAE,MAAM,CAAC,CAAC;QAC7C,IAAI,GAAG,GAAG,SAAS,EAAE,CAAC;YACpB,SAAS,GAAG,GAAG,CAAC;YAChB,MAAM,GAAG,OAAO,CAAC;QACnB,CAAC;IACH,CAAC;IAED,IAAI,MAAM,IAAI,SAAS,IAAI,SAAS,EAAE,CAAC;QACrC,GAAG,CAAC,KAAK,CAAC,gCAAgC,MAAM,QAAQ,SAAS,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC;QAChF,OAAO,MAAM,CAAC;IAChB,CAAC;IAED,OAAO,IAAI,CAAC;AACd,CAAC;AAED;;;GAGG;AACH,SAAgB,cAAc,CAC5B,KAAkB,EAClB,MAAgB,EAChB,SAAiB,EACjB,IAAY,EACZ,GAAW;IAEX,MAAM,GAAG,GAAG,KAAK,CAAC,gBAAgB,EAAE,CAAC;IACrC,MAAM,MAAM,GAA8C,EAAE,CAAC;IAE7D,KAAK,MAAM,EAAE,OAAO,EAAE,MAAM,EAAE,IAAI,GAAG,EAAE,CAAC;QACtC,MAAM,GAAG,GAAG,IAAA,yBAAgB,EAAC,MAAM,EAAE,MAAM,CAAC,CAAC;QAC7C,IAAI,GAAG,IAAI,SAAS,EAAE,CAAC;YACrB,MAAM,CAAC,IAAI,CAAC,EAAE,OAAO,EAAE,KAAK,EAAE,GAAG,EAAE,CAAC,CAAC;QACvC,CAAC;IACH,CAAC;IAED,MAAM,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,KAAK,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC;IACzC,MAAM,MAAM,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC,EAAE,IAAI,CAAC,CAAC;IACrC,IAAI,MAAM,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACtB,GAAG,CAAC,KAAK,CAAC,yBAAyB,MAAM,CAAC,MAAM,qBAAqB,SAAS,UAAU,MAAM,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC;IACzH,CAAC;IACD,OAAO,MAAM,CAAC;AAChB,CAAC"}

package/dist/ingest/providers/anthropic.d.ts

@@ -1,5 +1,19 @@
 import type { SummarizerConfig, Logger } from "../../types";
 export declare function summarizeTaskAnthropic(text: string, cfg: SummarizerConfig, log: Logger): Promise<string>;
 export declare function judgeNewTopicAnthropic(currentContext: string, newMessage: string, cfg: SummarizerConfig, log: Logger): Promise<boolean>;
+import type { FilterResult } from "./openai";
+export type { FilterResult } from "./openai";
+export declare function filterRelevantAnthropic(query: string, candidates: Array<{
+    index: number;
+    summary: string;
+    role: string;
+}>, cfg: SummarizerConfig, log: Logger): Promise<FilterResult>;
 export declare function summarizeAnthropic(text: string, cfg: SummarizerConfig, log: Logger): Promise<string>;
+import type { DedupResult } from "./openai";
+export type { DedupResult } from "./openai";
+export declare function judgeDedupAnthropic(newSummary: string, candidates: Array<{
+    index: number;
+    summary: string;
+    chunkId: string;
+}>, cfg: SummarizerConfig, log: Logger): Promise<DedupResult>;
 //# sourceMappingURL=anthropic.d.ts.map

package/dist/ingest/providers/anthropic.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"anthropic.d.ts","sourceRoot":"","sources":["../../../src/ingest/providers/anthropic.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAyC5D,wBAAsB,sBAAsB,CAC1C,IAAI,EAAE,MAAM,EACZ,GAAG,EAAE,gBAAgB,EACrB,GAAG,EAAE,MAAM,GACV,OAAO,CAAC,MAAM,CAAC,CA8BjB;AAeD,wBAAsB,sBAAsB,CAC1C,cAAc,EAAE,MAAM,EACtB,UAAU,EAAE,MAAM,EAClB,GAAG,EAAE,gBAAgB,EACrB,GAAG,EAAE,MAAM,GACV,OAAO,CAAC,OAAO,CAAC,CAkClB;AAED,wBAAsB,kBAAkB,CACtC,IAAI,EAAE,MAAM,EACZ,GAAG,EAAE,gBAAgB,EACrB,GAAG,EAAE,MAAM,GACV,OAAO,CAAC,MAAM,CAAC,CAgCjB"}
+{"version":3,"file":"anthropic.d.ts","sourceRoot":"","sources":["../../../src/ingest/providers/anthropic.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAyC5D,wBAAsB,sBAAsB,CAC1C,IAAI,EAAE,MAAM,EACZ,GAAG,EAAE,gBAAgB,EACrB,GAAG,EAAE,MAAM,GACV,OAAO,CAAC,MAAM,CAAC,CA8BjB;AAeD,wBAAsB,sBAAsB,CAC1C,cAAc,EAAE,MAAM,EACtB,UAAU,EAAE,MAAM,EAClB,GAAG,EAAE,gBAAgB,EACrB,GAAG,EAAE,MAAM,GACV,OAAO,CAAC,OAAO,CAAC,CAkClB;AAwBD,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,UAAU,CAAC;AAC7C,YAAY,EAAE,YAAY,EAAE,MAAM,UAAU,CAAC;AAE7C,wBAAsB,uBAAuB,CAC3C,KAAK,EAAE,MAAM,EACb,UAAU,EAAE,KAAK,CAAC;IAAE,KAAK,EAAE,MAAM,CAAC;IAAC,OAAO,EAAE,MAAM,CAAC;IAAC,IAAI,EAAE,MAAM,CAAA;CAAE,CAAC,EACnE,GAAG,EAAE,gBAAgB,EACrB,GAAG,EAAE,MAAM,GACV,OAAO,CAAC,YAAY,CAAC,CAmCvB;AAmBD,wBAAsB,kBAAkB,CACtC,IAAI,EAAE,MAAM,EACZ,GAAG,EAAE,gBAAgB,EACrB,GAAG,EAAE,MAAM,GACV,OAAO,CAAC,MAAM,CAAC,CAgCjB;AAKD,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,UAAU,CAAC;AAC5C,YAAY,EAAE,WAAW,EAAE,MAAM,UAAU,CAAC;AAE5C,wBAAsB,mBAAmB,CACvC,UAAU,EAAE,MAAM,EAClB,UAAU,EAAE,KAAK,CAAC;IAAE,KAAK,EAAE,MAAM,CAAC;IAAC,OAAO,EAAE,MAAM,CAAC;IAAC,OAAO,EAAE,MAAM,CAAA;CAAE,CAAC,EACtE,GAAG,EAAE,gBAAgB,EACrB,GAAG,EAAE,MAAM,GACV,OAAO,CAAC,WAAW,CAAC,CAiCtB"}

package/dist/ingest/providers/anthropic.js

@@ -2,7 +2,9 @@
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.summarizeTaskAnthropic = summarizeTaskAnthropic;
 exports.judgeNewTopicAnthropic = judgeNewTopicAnthropic;
+exports.filterRelevantAnthropic = filterRelevantAnthropic;
 exports.summarizeAnthropic = summarizeAnthropic;
+exports.judgeDedupAnthropic = judgeDedupAnthropic;
 const SYSTEM_PROMPT = `Summarize the text in ONE concise sentence (max 120 characters). IMPORTANT: Use the SAME language as the input text — if the input is Chinese, write Chinese; if English, write English. Preserve exact names, commands, error codes. No bullet points, no preamble — output only the sentence.`;
 const TASK_SUMMARY_PROMPT = `You create a DETAILED task summary from a multi-turn conversation. This summary will be the ONLY record of this conversation, so it must preserve ALL important information.
 
@@ -111,6 +113,76 @@ async function judgeNewTopicAnthropic(currentContext, newMessage, cfg, log) {
     log.debug(`Topic judge result: "${answer}"`);
     return answer.startsWith("NEW");
 }
+const FILTER_RELEVANT_PROMPT = `You are a memory relevance judge. Given a user's QUERY and a list of CANDIDATE memory summaries, do two things:
+
+1. Select ALL candidates that could be useful for answering the query. When in doubt, INCLUDE the candidate.
+   - For questions about lists, history, or "what/where/who" across multiple items (e.g. "which companies did I work at"), include ALL matching items — do NOT stop at the first match.
+   - For factual lookups (e.g. "what is the SSH port"), a single direct answer is enough.
+2. Judge whether the selected memories are SUFFICIENT to fully answer the query WITHOUT fetching additional context.
+
+IMPORTANT for "sufficient" judgment:
+- sufficient=true ONLY when the memories contain a concrete ANSWER, fact, decision, or actionable information that directly addresses the query.
+- sufficient=false when:
+  - The memories only repeat the same question the user asked before (echo, not answer).
+  - The memories show related topics but lack the specific detail needed.
+  - The memories contain partial information that would benefit from full task context, timeline, or related skills.
+
+Output a JSON object with exactly two fields:
+{"relevant":[1,3,5],"sufficient":true}
+
+- "relevant": array of candidate numbers that are useful. Empty array [] if none are relevant.
+- "sufficient": true ONLY if the memories contain a direct answer; false otherwise.
+
+Output ONLY the JSON object, nothing else.`;
+async function filterRelevantAnthropic(query, candidates, cfg, log) {
+    const endpoint = cfg.endpoint ?? "https://api.anthropic.com/v1/messages";
+    const model = cfg.model ?? "claude-3-haiku-20240307";
+    const headers = {
+        "Content-Type": "application/json",
+        "x-api-key": cfg.apiKey ?? "",
+        "anthropic-version": "2023-06-01",
+        ...cfg.headers,
+    };
+    const candidateText = candidates
+        .map((c) => `${c.index}. [${c.role}] ${c.summary}`)
+        .join("\n");
+    const resp = await fetch(endpoint, {
+        method: "POST",
+        headers,
+        body: JSON.stringify({
+            model,
+            max_tokens: 200,
+            temperature: 0,
+            system: FILTER_RELEVANT_PROMPT,
+            messages: [{ role: "user", content: `QUERY: ${query}\n\nCANDIDATES:\n${candidateText}` }],
+        }),
+        signal: AbortSignal.timeout(cfg.timeoutMs ?? 15_000),
+    });
+    if (!resp.ok) {
+        const body = await resp.text();
+        throw new Error(`Anthropic filter-relevant failed (${resp.status}): ${body}`);
+    }
+    const json = (await resp.json());
+    const raw = json.content.find((c) => c.type === "text")?.text?.trim() ?? "{}";
+    return parseFilterResult(raw, log);
+}
+function parseFilterResult(raw, log) {
+    try {
+        const match = raw.match(/\{[\s\S]*\}/);
+        if (match) {
+            const obj = JSON.parse(match[0]);
+            if (obj && Array.isArray(obj.relevant)) {
+                return {
+                    relevant: obj.relevant.filter((n) => typeof n === "number"),
+                    sufficient: obj.sufficient === true,
+                };
+            }
+        }
+    }
+    catch { }
+    log.warn(`filterRelevant: failed to parse LLM output: "${raw}", fallback to all+insufficient`);
+    return { relevant: [], sufficient: false };
+}
 async function summarizeAnthropic(text, cfg, log) {
     const endpoint = cfg.endpoint ?? "https://api.anthropic.com/v1/messages";
     const model = cfg.model ?? "claude-3-haiku-20240307";
@@ -139,4 +211,36 @@ async function summarizeAnthropic(text, cfg, log) {
     const json = (await resp.json());
     return json.content.find((c) => c.type === "text")?.text?.trim() ?? "";
 }
+// ─── Smart Dedup ───
+const openai_1 = require("./openai");
+async function judgeDedupAnthropic(newSummary, candidates, cfg, log) {
+    const endpoint = cfg.endpoint ?? "https://api.anthropic.com/v1/messages";
+    const model = cfg.model ?? "claude-3-haiku-20240307";
+    const headers = {
+        "Content-Type": "application/json",
+        "x-api-key": cfg.apiKey ?? "",
+        "anthropic-version": "2023-06-01",
+        ...cfg.headers,
+    };
+    const candidateText = candidates.map((c) => `${c.index}. ${c.summary}`).join("\n");
+    const resp = await fetch(endpoint, {
+        method: "POST",
+        headers,
+        body: JSON.stringify({
+            model,
+            max_tokens: 300,
+            temperature: 0,
+            system: openai_1.DEDUP_JUDGE_PROMPT,
+            messages: [{ role: "user", content: `NEW MEMORY:\n${newSummary}\n\nEXISTING MEMORIES:\n${candidateText}` }],
+        }),
+        signal: AbortSignal.timeout(cfg.timeoutMs ?? 15_000),
+    });
+    if (!resp.ok) {
+        const body = await resp.text();
+        throw new Error(`Anthropic dedup-judge failed (${resp.status}): ${body}`);
+    }
+    const json = (await resp.json());
+    const raw = json.content.find((c) => c.type === "text")?.text?.trim() ?? "{}";
+    return (0, openai_1.parseDedupResult)(raw, log);
+}
 //# sourceMappingURL=anthropic.js.map

package/dist/ingest/providers/anthropic.js.map

@@ -1 +1 @@
-{"version":3,"file":"anthropic.js","sourceRoot":"","sources":["../../../src/ingest/providers/anthropic.ts"],"names":[],"mappings":";;AAyCA,wDAkCC;AAeD,wDAuCC;AAED,gDAoCC;AArKD,MAAM,aAAa,GAAG,iSAAiS,CAAC;AAExT,MAAM,mBAAmB,GAAG;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;oCAmCQ,CAAC;AAE9B,KAAK,UAAU,sBAAsB,CAC1C,IAAY,EACZ,GAAqB,EACrB,GAAW;IAEX,MAAM,QAAQ,GAAG,GAAG,CAAC,QAAQ,IAAI,uCAAuC,CAAC;IACzE,MAAM,KAAK,GAAG,GAAG,CAAC,KAAK,IAAI,yBAAyB,CAAC;IACrD,MAAM,OAAO,GAA2B;QACtC,cAAc,EAAE,kBAAkB;QAClC,WAAW,EAAE,GAAG,CAAC,MAAM,IAAI,EAAE;QAC7B,mBAAmB,EAAE,YAAY;QACjC,GAAG,GAAG,CAAC,OAAO;KACf,CAAC;IAEF,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,QAAQ,EAAE;QACjC,MAAM,EAAE,MAAM;QACd,OAAO;QACP,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;YACnB,KAAK;YACL,UAAU,EAAE,IAAI;YAChB,WAAW,EAAE,GAAG,CAAC,WAAW,IAAI,GAAG;YACnC,MAAM,EAAE,mBAAmB;YAC3B,QAAQ,EAAE,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,IAAI,EAAE,CAAC;SAC5C,CAAC;QACF,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,IAAI,MAAM,CAAC;KACrD,CAAC,CAAC;IAEH,IAAI,CAAC,IAAI,CAAC,EAAE,EAAE,CAAC;QACb,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,IAAI,EAAE,CAAC;QAC/B,MAAM,IAAI,KAAK,CAAC,oCAAoC,IAAI,CAAC,MAAM,MAAM,IAAI,EAAE,CAAC,CAAC;IAC/E,CAAC;IAED,MAAM,IAAI,GAAG,CAAC,MAAM,IAAI,CAAC,IAAI,EAAE,CAAuD,CAAC;IACvF,OAAO,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,MAAM,CAAC,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC;AACzE,CAAC;AAED,MAAM,kBAAkB,GAAG;;;;;;;;;;;qCAWU,CAAC;AAE/B,KAAK,UAAU,sBAAsB,CAC1C,cAAsB,EACtB,UAAkB,EAClB,GAAqB,EACrB,GAAW;IAEX,MAAM,QAAQ,GAAG,GAAG,CAAC,QAAQ,IAAI,uCAAuC,CAAC;IACzE,MAAM,KAAK,GAAG,GAAG,CAAC,KAAK,IAAI,yBAAyB,CAAC;IACrD,MAAM,OAAO,GAA2B;QACtC,cAAc,EAAE,kBAAkB;QAClC,WAAW,EAAE,GAAG,CAAC,MAAM,IAAI,EAAE;QAC7B,mBAAmB,EAAE,YAAY;QACjC,GAAG,GAAG,CAAC,OAAO;KACf,CAAC;IAEF,MAAM,WAAW,GAAG,kCAAkC,cAAc,0BAA0B,UAAU,EAAE,CAAC;IAE3G,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,QAAQ,EAAE;QACjC,MAAM,EAAE,MAAM;QACd,OAAO;QACP,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;YACnB,KAAK;YACL,UAAU,EAAE,EAAE;YACd,WAAW,EAAE,CAAC;YACd,MAAM,EAAE,kBAAkB;YAC1B,QAAQ,EAAE,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,WAAW,EAAE,CAAC;SACnD,CAAC;QACF,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,IAAI,MAAM,CAAC;KACrD,CAAC,CAAC;IAEH,IAAI,CAAC,IAAI,CAAC,EAAE,EAAE,CAAC;QACb,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,IAAI,EAAE,CAAC;QAC/B,MAAM,IAAI,KAAK,CAAC,iCAAiC,IAAI,CAAC,MAAM,MAAM,IAAI,EAAE,CAAC,CAAC;IAC5E,CAAC;IAED,MAAM,IAAI,GAAG,CAAC,MAAM,IAAI,CAAC,IAAI,EAAE,CAAuD,CAAC;IACvF,MAAM,MAAM,GAAG,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,MAAM,CAAC,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC,WAAW,EAAE,IAAI,EAAE,CAAC;IAC7F,GAAG,CAAC,KAAK,CAAC,wBAAwB,MAAM,GAAG,CAAC,CAAC;IAC7C,OAAO,MAAM,CAAC,UAAU,CAAC,KAAK,CAAC,CAAC;AAClC,CAAC;AAEM,KAAK,UAAU,kBAAkB,CACtC,IAAY,EACZ,GAAqB,EACrB,GAAW;IAEX,MAAM,QAAQ,GAAG,GAAG,CAAC,QAAQ,IAAI,uCAAuC,CAAC;IACzE,MAAM,KAAK,GAAG,GAAG,CAAC,KAAK,IAAI,yBAAyB,CAAC;IACrD,MAAM,OAAO,GAA2B;QACtC,cAAc,EAAE,kBAAkB;QAClC,WAAW,EAAE,GAAG,CAAC,MAAM,IAAI,EAAE;QAC7B,mBAAmB,EAAE,YAAY;QACjC,GAAG,GAAG,CAAC,OAAO;KACf,CAAC;IAEF,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,QAAQ,EAAE;QACjC,MAAM,EAAE,MAAM;QACd,OAAO;QACP,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;YACnB,KAAK;YACL,UAAU,EAAE,GAAG;YACf,WAAW,EAAE,GAAG,CAAC,WAAW,IAAI,CAAC;YACjC,MAAM,EAAE,aAAa;YACrB,QAAQ,EAAE,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,IAAI,EAAE,CAAC;SAC5C,CAAC;QACF,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,IAAI,MAAM,CAAC;KACrD,CAAC,CAAC;IAEH,IAAI,CAAC,IAAI,CAAC,EAAE,EAAE,CAAC;QACb,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,IAAI,EAAE,CAAC;QAC/B,MAAM,IAAI,KAAK,CAAC,+BAA+B,IAAI,CAAC,MAAM,MAAM,IAAI,EAAE,CAAC,CAAC;IAC1E,CAAC;IAED,MAAM,IAAI,GAAG,CAAC,MAAM,IAAI,CAAC,IAAI,EAAE,CAE9B,CAAC;IACF,OAAO,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,MAAM,CAAC,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC;AACzE,CAAC"}
+{"version":3,"file":"anthropic.js","sourceRoot":"","sources":["../../../src/ingest/providers/anthropic.ts"],"names":[],"mappings":";;AAyCA,wDAkCC;AAeD,wDAuCC;AA2BD,0DAwCC;AAmBD,gDAoCC;AAQD,kDAsCC;AAvSD,MAAM,aAAa,GAAG,iSAAiS,CAAC;AAExT,MAAM,mBAAmB,GAAG;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;oCAmCQ,CAAC;AAE9B,KAAK,UAAU,sBAAsB,CAC1C,IAAY,EACZ,GAAqB,EACrB,GAAW;IAEX,MAAM,QAAQ,GAAG,GAAG,CAAC,QAAQ,IAAI,uCAAuC,CAAC;IACzE,MAAM,KAAK,GAAG,GAAG,CAAC,KAAK,IAAI,yBAAyB,CAAC;IACrD,MAAM,OAAO,GAA2B;QACtC,cAAc,EAAE,kBAAkB;QAClC,WAAW,EAAE,GAAG,CAAC,MAAM,IAAI,EAAE;QAC7B,mBAAmB,EAAE,YAAY;QACjC,GAAG,GAAG,CAAC,OAAO;KACf,CAAC;IAEF,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,QAAQ,EAAE;QACjC,MAAM,EAAE,MAAM;QACd,OAAO;QACP,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;YACnB,KAAK;YACL,UAAU,EAAE,IAAI;YAChB,WAAW,EAAE,GAAG,CAAC,WAAW,IAAI,GAAG;YACnC,MAAM,EAAE,mBAAmB;YAC3B,QAAQ,EAAE,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,IAAI,EAAE,CAAC;SAC5C,CAAC;QACF,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,IAAI,MAAM,CAAC;KACrD,CAAC,CAAC;IAEH,IAAI,CAAC,IAAI,CAAC,EAAE,EAAE,CAAC;QACb,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,IAAI,EAAE,CAAC;QAC/B,MAAM,IAAI,KAAK,CAAC,oCAAoC,IAAI,CAAC,MAAM,MAAM,IAAI,EAAE,CAAC,CAAC;IAC/E,CAAC;IAED,MAAM,IAAI,GAAG,CAAC,MAAM,IAAI,CAAC,IAAI,EAAE,CAAuD,CAAC;IACvF,OAAO,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,MAAM,CAAC,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC;AACzE,CAAC;AAED,MAAM,kBAAkB,GAAG;;;;;;;;;;;qCAWU,CAAC;AAE/B,KAAK,UAAU,sBAAsB,CAC1C,cAAsB,EACtB,UAAkB,EAClB,GAAqB,EACrB,GAAW;IAEX,MAAM,QAAQ,GAAG,GAAG,CAAC,QAAQ,IAAI,uCAAuC,CAAC;IACzE,MAAM,KAAK,GAAG,GAAG,CAAC,KAAK,IAAI,yBAAyB,CAAC;IACrD,MAAM,OAAO,GAA2B;QACtC,cAAc,EAAE,kBAAkB;QAClC,WAAW,EAAE,GAAG,CAAC,MAAM,IAAI,EAAE;QAC7B,mBAAmB,EAAE,YAAY;QACjC,GAAG,GAAG,CAAC,OAAO;KACf,CAAC;IAEF,MAAM,WAAW,GAAG,kCAAkC,cAAc,0BAA0B,UAAU,EAAE,CAAC;IAE3G,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,QAAQ,EAAE;QACjC,MAAM,EAAE,MAAM;QACd,OAAO;QACP,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;YACnB,KAAK;YACL,UAAU,EAAE,EAAE;YACd,WAAW,EAAE,CAAC;YACd,MAAM,EAAE,kBAAkB;YAC1B,QAAQ,EAAE,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,WAAW,EAAE,CAAC;SACnD,CAAC;QACF,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,IAAI,MAAM,CAAC;KACrD,CAAC,CAAC;IAEH,IAAI,CAAC,IAAI,CAAC,EAAE,EAAE,CAAC;QACb,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,IAAI,EAAE,CAAC;QAC/B,MAAM,IAAI,KAAK,CAAC,iCAAiC,IAAI,CAAC,MAAM,MAAM,IAAI,EAAE,CAAC,CAAC;IAC5E,CAAC;IAED,MAAM,IAAI,GAAG,CAAC,MAAM,IAAI,CAAC,IAAI,EAAE,CAAuD,CAAC;IACvF,MAAM,MAAM,GAAG,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,MAAM,CAAC,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC,WAAW,EAAE,IAAI,EAAE,CAAC;IAC7F,GAAG,CAAC,KAAK,CAAC,wBAAwB,MAAM,GAAG,CAAC,CAAC;IAC7C,OAAO,MAAM,CAAC,UAAU,CAAC,KAAK,CAAC,CAAC;AAClC,CAAC;AAED,MAAM,sBAAsB,GAAG;;;;;;;;;;;;;;;;;;;;2CAoBY,CAAC;AAKrC,KAAK,UAAU,uBAAuB,CAC3C,KAAa,EACb,UAAmE,EACnE,GAAqB,EACrB,GAAW;IAEX,MAAM,QAAQ,GAAG,GAAG,CAAC,QAAQ,IAAI,uCAAuC,CAAC;IACzE,MAAM,KAAK,GAAG,GAAG,CAAC,KAAK,IAAI,yBAAyB,CAAC;IACrD,MAAM,OAAO,GAA2B;QACtC,cAAc,EAAE,kBAAkB;QAClC,WAAW,EAAE,GAAG,CAAC,MAAM,IAAI,EAAE;QAC7B,mBAAmB,EAAE,YAAY;QACjC,GAAG,GAAG,CAAC,OAAO;KACf,CAAC;IAEF,MAAM,aAAa,GAAG,UAAU;SAC7B,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,GAAG,CAAC,CAAC,KAAK,MAAM,CAAC,CAAC,IAAI,KAAK,CAAC,CAAC,OAAO,EAAE,CAAC;SAClD,IAAI,CAAC,IAAI,CAAC,CAAC;IAEd,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,QAAQ,EAAE;QACjC,MAAM,EAAE,MAAM;QACd,OAAO;QACP,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;YACnB,KAAK;YACL,UAAU,EAAE,GAAG;YACf,WAAW,EAAE,CAAC;YACd,MAAM,EAAE,sBAAsB;YAC9B,QAAQ,EAAE,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,UAAU,KAAK,oBAAoB,aAAa,EAAE,EAAE,CAAC;SAC1F,CAAC;QACF,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,IAAI,MAAM,CAAC;KACrD,CAAC,CAAC;IAEH,IAAI,CAAC,IAAI,CAAC,EAAE,EAAE,CAAC;QACb,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,IAAI,EAAE,CAAC;QAC/B,MAAM,IAAI,KAAK,CAAC,qCAAqC,IAAI,CAAC,MAAM,MAAM,IAAI,EAAE,CAAC,CAAC;IAChF,CAAC;IAED,MAAM,IAAI,GAAG,CAAC,MAAM,IAAI,CAAC,IAAI,EAAE,CAAuD,CAAC;IACvF,MAAM,GAAG,GAAG,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,MAAM,CAAC,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,IAAI,CAAC;IAC9E,OAAO,iBAAiB,CAAC,GAAG,EAAE,GAAG,CAAC,CAAC;AACrC,CAAC;AAED,SAAS,iBAAiB,CAAC,GAAW,EAAE,GAAW;IACjD,IAAI,CAAC;QACH,MAAM,KAAK,GAAG,GAAG,CAAC,KAAK,CAAC,aAAa,CAAC,CAAC;QACvC,IAAI,KAAK,EAAE,CAAC;YACV,MAAM,GAAG,GAAG,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC;YACjC,IAAI,GAAG,IAAI,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,QAAQ,CAAC,EAAE,CAAC;gBACvC,OAAO;oBACL,QAAQ,EAAE,GAAG,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC,CAAM,EAAE,EAAE,CAAC,OAAO,CAAC,KAAK,QAAQ,CAAC;oBAChE,UAAU,EAAE,GAAG,CAAC,UAAU,KAAK,IAAI;iBACpC,CAAC;YACJ,CAAC;QACH,CAAC;IACH,CAAC;IAAC,MAAM,CAAC,CAAA,CAAC;IACV,GAAG,CAAC,IAAI,CAAC,gDAAgD,GAAG,iCAAiC,CAAC,CAAC;IAC/F,OAAO,EAAE,QAAQ,EAAE,EAAE,EAAE,UAAU,EAAE,KAAK,EAAE,CAAC;AAC7C,CAAC;AAEM,KAAK,UAAU,kBAAkB,CACtC,IAAY,EACZ,GAAqB,EACrB,GAAW;IAEX,MAAM,QAAQ,GAAG,GAAG,CAAC,QAAQ,IAAI,uCAAuC,CAAC;IACzE,MAAM,KAAK,GAAG,GAAG,CAAC,KAAK,IAAI,yBAAyB,CAAC;IACrD,MAAM,OAAO,GAA2B;QACtC,cAAc,EAAE,kBAAkB;QAClC,WAAW,EAAE,GAAG,CAAC,MAAM,IAAI,EAAE;QAC7B,mBAAmB,EAAE,YAAY;QACjC,GAAG,GAAG,CAAC,OAAO;KACf,CAAC;IAEF,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,QAAQ,EAAE;QACjC,MAAM,EAAE,MAAM;QACd,OAAO;QACP,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;YACnB,KAAK;YACL,UAAU,EAAE,GAAG;YACf,WAAW,EAAE,GAAG,CAAC,WAAW,IAAI,CAAC;YACjC,MAAM,EAAE,aAAa;YACrB,QAAQ,EAAE,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,IAAI,EAAE,CAAC;SAC5C,CAAC;QACF,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,IAAI,MAAM,CAAC;KACrD,CAAC,CAAC;IAEH,IAAI,CAAC,IAAI,CAAC,EAAE,EAAE,CAAC;QACb,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,IAAI,EAAE,CAAC;QAC/B,MAAM,IAAI,KAAK,CAAC,+BAA+B,IAAI,CAAC,MAAM,MAAM,IAAI,EAAE,CAAC,CAAC;IAC1E,CAAC;IAED,MAAM,IAAI,GAAG,CAAC,MAAM,IAAI,CAAC,IAAI,EAAE,CAE9B,CAAC;IACF,OAAO,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,MAAM,CAAC,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC;AACzE,CAAC;AAED,sBAAsB;AAEtB,qCAAgE;AAIzD,KAAK,UAAU,mBAAmB,CACvC,UAAkB,EAClB,UAAsE,EACtE,GAAqB,EACrB,GAAW;IAEX,MAAM,QAAQ,GAAG,GAAG,CAAC,QAAQ,IAAI,uCAAuC,CAAC;IACzE,MAAM,KAAK,GAAG,GAAG,CAAC,KAAK,IAAI,yBAAyB,CAAC;IACrD,MAAM,OAAO,GAA2B;QACtC,cAAc,EAAE,kBAAkB;QAClC,WAAW,EAAE,GAAG,CAAC,MAAM,IAAI,EAAE;QAC7B,mBAAmB,EAAE,YAAY;QACjC,GAAG,GAAG,CAAC,OAAO;KACf,CAAC;IAEF,MAAM,aAAa,GAAG,UAAU,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,GAAG,CAAC,CAAC,KAAK,KAAK,CAAC,CAAC,OAAO,EAAE,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IAEnF,MAAM,IAAI,GAAG,MAAM,KAAK,CAAC,QAAQ,EAAE;QACjC,MAAM,EAAE,MAAM;QACd,OAAO;QACP,IAAI,EAAE,IAAI,CAAC,SAAS,CAAC;YACnB,KAAK;YACL,UAAU,EAAE,GAAG;YACf,WAAW,EAAE,CAAC;YACd,MAAM,EAAE,2BAAkB;YAC1B,QAAQ,EAAE,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,gBAAgB,UAAU,2BAA2B,aAAa,EAAE,EAAE,CAAC;SAC5G,CAAC;QACF,MAAM,EAAE,WAAW,CAAC,OAAO,CAAC,GAAG,CAAC,SAAS,IAAI,MAAM,CAAC;KACrD,CAAC,CAAC;IAEH,IAAI,CAAC,IAAI,CAAC,EAAE,EAAE,CAAC;QACb,MAAM,IAAI,GAAG,MAAM,IAAI,CAAC,IAAI,EAAE,CAAC;QAC/B,MAAM,IAAI,KAAK,CAAC,iCAAiC,IAAI,CAAC,MAAM,MAAM,IAAI,EAAE,CAAC,CAAC;IAC5E,CAAC;IAED,MAAM,IAAI,GAAG,CAAC,MAAM,IAAI,CAAC,IAAI,EAAE,CAAuD,CAAC;IACvF,MAAM,GAAG,GAAG,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,KAAK,MAAM,CAAC,EAAE,IAAI,EAAE,IAAI,EAAE,IAAI,IAAI,CAAC;IAC9E,OAAO,IAAA,yBAAgB,EAAC,GAAG,EAAE,GAAG,CAAC,CAAC;AACpC,CAAC"}

package/dist/ingest/providers/bedrock.d.ts

@@ -1,5 +1,19 @@
 import type { SummarizerConfig, Logger } from "../../types";
 export declare function summarizeTaskBedrock(text: string, cfg: SummarizerConfig, log: Logger): Promise<string>;
 export declare function judgeNewTopicBedrock(currentContext: string, newMessage: string, cfg: SummarizerConfig, log: Logger): Promise<boolean>;
+import type { FilterResult } from "./openai";
+export type { FilterResult } from "./openai";
+export declare function filterRelevantBedrock(query: string, candidates: Array<{
+    index: number;
+    summary: string;
+    role: string;
+}>, cfg: SummarizerConfig, log: Logger): Promise<FilterResult>;
 export declare function summarizeBedrock(text: string, cfg: SummarizerConfig, log: Logger): Promise<string>;
+import type { DedupResult } from "./openai";
+export type { DedupResult } from "./openai";
+export declare function judgeDedupBedrock(newSummary: string, candidates: Array<{
+    index: number;
+    summary: string;
+    chunkId: string;
+}>, cfg: SummarizerConfig, log: Logger): Promise<DedupResult>;
 //# sourceMappingURL=bedrock.d.ts.map

package/dist/ingest/providers/bedrock.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"bedrock.d.ts","sourceRoot":"","sources":["../../../src/ingest/providers/bedrock.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAyC5D,wBAAsB,oBAAoB,CACxC,IAAI,EAAE,MAAM,EACZ,GAAG,EAAE,gBAAgB,EACrB,GAAG,EAAE,MAAM,GACV,OAAO,CAAC,MAAM,CAAC,CA+BjB;AAeD,wBAAsB,oBAAoB,CACxC,cAAc,EAAE,MAAM,EACtB,UAAU,EAAE,MAAM,EAClB,GAAG,EAAE,gBAAgB,EACrB,GAAG,EAAE,MAAM,GACV,OAAO,CAAC,OAAO,CAAC,CAmClB;AAED,wBAAsB,gBAAgB,CACpC,IAAI,EAAE,MAAM,EACZ,GAAG,EAAE,gBAAgB,EACrB,GAAG,EAAE,MAAM,GACV,OAAO,CAAC,MAAM,CAAC,CAoCjB"}
+{"version":3,"file":"bedrock.d.ts","sourceRoot":"","sources":["../../../src/ingest/providers/bedrock.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAyC5D,wBAAsB,oBAAoB,CACxC,IAAI,EAAE,MAAM,EACZ,GAAG,EAAE,gBAAgB,EACrB,GAAG,EAAE,MAAM,GACV,OAAO,CAAC,MAAM,CAAC,CA+BjB;AAeD,wBAAsB,oBAAoB,CACxC,cAAc,EAAE,MAAM,EACtB,UAAU,EAAE,MAAM,EAClB,GAAG,EAAE,gBAAgB,EACrB,GAAG,EAAE,MAAM,GACV,OAAO,CAAC,OAAO,CAAC,CAmClB;AAwBD,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,UAAU,CAAC;AAC7C,YAAY,EAAE,YAAY,EAAE,MAAM,UAAU,CAAC;AAE7C,wBAAsB,qBAAqB,CACzC,KAAK,EAAE,MAAM,EACb,UAAU,EAAE,KAAK,CAAC;IAAE,KAAK,EAAE,MAAM,CAAC;IAAC,OAAO,EAAE,MAAM,CAAC;IAAC,IAAI,EAAE,MAAM,CAAA;CAAE,CAAC,EACnE,GAAG,EAAE,gBAAgB,EACrB,GAAG,EAAE,MAAM,GACV,OAAO,CAAC,YAAY,CAAC,CAoCvB;AAmBD,wBAAsB,gBAAgB,CACpC,IAAI,EAAE,MAAM,EACZ,GAAG,EAAE,gBAAgB,EACrB,GAAG,EAAE,MAAM,GACV,OAAO,CAAC,MAAM,CAAC,CAoCjB;AAKD,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,UAAU,CAAC;AAC5C,YAAY,EAAE,WAAW,EAAE,MAAM,UAAU,CAAC;AAE5C,wBAAsB,iBAAiB,CACrC,UAAU,EAAE,MAAM,EAClB,UAAU,EAAE,KAAK,CAAC;IAAE,KAAK,EAAE,MAAM,CAAC;IAAC,OAAO,EAAE,MAAM,CAAC;IAAC,OAAO,EAAE,MAAM,CAAA;CAAE,CAAC,EACtE,GAAG,EAAE,gBAAgB,EACrB,GAAG,EAAE,MAAM,GACV,OAAO,CAAC,WAAW,CAAC,CA4BtB"}