recallnest 1.4.1 → 2.2.1

package/docs/superpowers/specs/2026-03-19-ingest-pipeline-upgrade-design.md

@@ -1,77 +0,0 @@
-# RecallNest Ingest Pipeline Upgrade: Conversation Fragments → Knowledge Extraction
-
-**Date**: 2026-03-19
-**Status**: Approved
-**Scope**: Route B (bulk distillation) + Route A (pipeline default change)
-
-## Problem
-
-29,984 total records; 23,453 (78.2%) are `fact` category — raw conversation slices (`[助手]`/`[用户]` prefixed) with no knowledge extraction. This is the root cause of 96.7% dead memory rate.
-
-## Decisions
-
-| Decision | Choice | Rationale |
-|----------|--------|-----------|
-| Disposition | New + Archive (B) | Preserve originals for traceability; ~5K new records is not bloat |
-| Extraction model | Reuse `smartExtractBatch()` (C) | Already has 6-category prompt + scoring; no new wheel |
-| Batch strategy | Dry-run then batch+checkpoint (C→B) | Verify quality first, then fault-tolerant batch |
-
-## Route B: Bulk Distillation
-
-### New file: `scripts/distill-facts.ts`
-
-**Modes**:
-- `--dry-run`: Process 1 session (~30-50 facts), print results for human review
-- Default: Process all sessions with checkpoint
-
-**Pipeline per batch (20 texts)**:
-1. `smartExtractBatch(texts)` → `SmartExtraction[]` (category, importance, l0, l1)
-2. `embedBatchPassage(extractedTexts)` → vectors
-3. Dedup against existing non-fact records (cosine > 0.80 → skip)
-4. Store new records: `boundary.layer = "durable"`, `boundary.authority = "distillation"`
-5. Archive originals: metadata += `{ archived: true, archivedAt, distilledTo: [ids] }`
-6. Write checkpoint: `data/distill-progress.json` (scope → done/skip)
-
-**Expected output**: 23K facts → ~3K-5K structured knowledge (5:1 to 8:1 compression)
-
-### Archive marking
-
-```json
-{
-  "archived": true,
-  "archivedAt": "2026-03-19T10:00:00Z",
-  "distilledTo": ["uuid-1", "uuid-2"],
-  "originalCategory": "fact"
-}
-```
-
-## Route A: Pipeline Default Change
-
-### Changes to `src/ingest.ts`
-
-1. `smartExtractBatch()` becomes **default** (was opt-in via `--llm`)
-2. New `--no-llm` flag for fallback mode
-3. LLM failure → don't store raw slice; queue to `data/pending-extraction.json`
-4. Next `lm ingest` auto-retries pending queue
-
-**Core principle**: Better to skip than to store garbage.
-
-### Changes to `src/retriever.ts`
-
-- Default filter: exclude `archived = true` from search results
-- New `includeArchived` parameter for traceability queries
-
-## Cost Estimate
-
-~1,173 LLM calls (20 texts/batch), ~4.1M input + ~1.2M output tokens.
-GPT-4o-mini: ~$1.3 | Gemini Flash: ~$0.7 | Claude Haiku: ~$8
-
-## Files to create/modify
-
-| File | Action |
-|------|--------|
-| `scripts/distill-facts.ts` | Create — bulk distillation script |
-| `src/ingest.ts` | Modify — flip smartExtract default |
-| `src/retriever.ts` | Modify — archived filter |
-| `data/distill-progress.json` | Auto-created at runtime |
-| `data/pending-extraction.json` | Auto-created at runtime |

package/docs/superpowers/specs/2026-03-23-pull-claude-desktop-design.md

@@ -1,95 +0,0 @@
-# Pull Claude Desktop Conversations into RecallNest
-
-Date: 2026-03-23
-
-## Goal
-
-One script to pull all historical Claude Desktop / claude.ai conversations and convert them into RecallNest's ingest format, so `lm ingest` can process them through the existing pipeline (noise filter, dedup, LLM extraction, 6-category classification).
-
-## Script
-
-`~/recallnest/scripts/pull-claude-desktop.ts` (Bun runtime)
-
-## Authentication
-
-- `CLAUDE_SESSION_KEY` env var (from browser DevTools → Application → Cookies → claude.ai → `sessionKey`)
-- Also accept `--session-key` CLI flag
-
-## Data Flow
-
-```
-sessionKey → GET /api/organizations → orgId
-           → GET /api/organizations/{orgId}/chat_conversations (paginated, 50/page)
-           → for each conversation:
-               GET .../chat_conversations/{id}?tree=True&rendering_mode=messages&render_all_tools=true
-               → parse branches → extract latest branch message chain
-               → convert to CC transcript .jsonl
-               → write to data/desktop-import/{conversationId}.jsonl
-```
-
-## Output Format
-
-Each conversation becomes one `.jsonl` file with entries matching CC transcript schema:
-
-```jsonl
-{"type":"user","message":{"role":"user","content":"text"},"sessionId":"conv-uuid","timestamp":"2025-01-15T10:00:00Z"}
-{"type":"assistant","message":{"role":"assistant","content":"reply"},"sessionId":"conv-uuid","timestamp":"2025-01-15T10:00:01Z"}
-```
-
-## Output Directory
-
-```
-data/desktop-import/
-  .sync-state.json
-  {conversation-id}.jsonl
-  ...
-```
-
-## Sync State (Resumable)
-
-`.sync-state.json` tracks:
-
-```json
-{
-  "lastSyncedAt": "ISO8601",
-  "conversations": {
-    "conv-id": {
-      "messageCount": 42,
-      "updatedAt": "ISO8601",
-      "pulledAt": "ISO8601"
-    }
-  }
-}
-```
-
-Re-run skips conversations whose `updatedAt` hasn't changed since last pull.
-
-## Branch Handling
-
-Claude web conversations support branching (edit/retry). The script extracts only the latest branch by walking `parent_message_uuid` chain from the most recent message backward.
-
-## Rate Limiting
-
-- 200ms delay between conversation detail requests
-- On 429 or 5xx: exponential backoff starting at 1s, max 3 retries
-- On auth failure (401/403): abort immediately with clear message
-
-## Error Handling
-
-- Single conversation failure does not abort the batch
-- Failed conversations logged and reported in final summary
-- Summary output: total conversations, pulled, skipped (already synced), failed
-
-## Ingest Integration
-
-After pulling, user runs:
-
-```bash
-lm ingest --source cc
-```
-
-Need to verify that ingest scans `data/desktop-import/` or that the path is configurable. If not, the script will symlink or copy files to the CC transcript directory.
-
-## Dependencies
-
-- Only Bun built-ins (fetch, fs, path) — no new npm packages

package/eval/continuity/results-2026-03-16.md

@@ -1,117 +0,0 @@
-# RecallNest Continuity Eval
-
-- Generated: 2026-03-16T03:35:14.079Z
-- Cases: 5
-- Passed: 5/5
-- Average score: 94.0%
-
-| Case | Profile | Score | Pass | Stable | Patterns | Cases | Checkpoint |
-|------|---------|-------|------|--------|----------|-------|------------|
-| writing_style_continuity | writing | 100% | yes | 3 | 1 | 0 | no |
-| visual_style_continuity | default | 100% | yes | 3 | 0 | 0 | no |
-| project_context_continuity | default | 100% | yes | 1 | 2 | 1 | no |
-| workflow_pattern_gap_check | default | 70% | yes | 3 | 3 | 1 | no |
-| recallnest_checkpoint_continuity | default | 100% | yes | 3 | 1 | 1 | yes |
-
-## Case Notes
-
-### writing_style_continuity
-- Task: 继续写我的AI小木屋的文章
-- Score: 100%
-- Pass: yes
-- Stable items: 3
-- Pattern items: 1
-- Case items: 0
-- Checkpoint present: no
-- Stable preview: Task focus: 小木屋的文章 | Pinned: 用户稳定偏好口语化、不端着、可以自嘲但不说教。表达应像朋友聊天，避免居高临下、用力过猛或碎嘴。 | Pinned: 用户常用视觉风格是手绘涂鸦风加高对比撞色；在内容包装和配图生成时，应优先沿用这一审美方向，除非任务目标明确要求其他风格。
-- Pattern preview: Workflow pattern: Promote recurring continuity workflow Use when: When the same startup or handoff workflow repeats a...
-- Case preview: -
-- Checkpoint summary: -
-- Matched stable any: 口语化, 不端着
-- Matched stable all: -
-- Matched patterns any: -
-- Matched cases any: -
-- Matched checkpoint any: -
-- Forbidden matches: -
-- Notes: Fresh windows should carry over stable writing-style preferences.
-
-### visual_style_continuity
-- Task: 给文章做封面和配图
-- Score: 100%
-- Pass: yes
-- Stable items: 3
-- Pattern items: 0
-- Case items: 0
-- Checkpoint present: no
-- Stable preview: Task focus: 给文章做封面和配图 | Pinned: 用户常用视觉风格是手绘涂鸦风加高对比撞色；在内容包装和配图生成时，应优先沿用这一审美方向，除非任务目标明确要求其他风格。 | Pinned: 用户稳定偏好口语化、不端着、可以自嘲但不说教。表达应像朋友聊天，避免居高临下、用力过猛或碎嘴。
-- Pattern preview: -
-- Case preview: -
-- Checkpoint summary: -
-- Matched stable any: 手绘涂鸦, 高对比撞色
-- Matched stable all: -
-- Matched patterns any: -
-- Matched cases any: -
-- Matched checkpoint any: -
-- Forbidden matches: -
-- Notes: Visual direction should survive without the user restating it.
-
-### project_context_continuity
-- Task: 一起继续排查 RecallNest 的连续性问题
-- Score: 100%
-- Pass: yes
-- Stable items: 1
-- Pattern items: 2
-- Case items: 1
-- Checkpoint present: no
-- Stable preview: Task focus: recallnest
-- Pattern preview: Workflow pattern: Cross-window continuity handoff Use when: When opening a fresh terminal window for the same project... | Workflow pattern: Promote recurring continuity workflow Use when: When the same startup or handoff workflow repeats a...
-- Case preview: 我先看真实召回密度，确认问题是不是出在 `scope: recallnest` 太窄。如果当前库里几乎没有这个 scope 的 durable memory，就该让 `resume_context` 更积极地退回到项目名驱动的 glo...
-- Checkpoint summary: -
-- Matched stable any: RecallNest
-- Matched stable all: -
-- Matched patterns any: -
-- Matched cases any: -
-- Matched checkpoint any: -
-- Forbidden matches: -
-- Notes: Fresh windows should at least recover the active project context.
-
-### workflow_pattern_gap_check
-- Task: 开个新窗口继续做同一个终端项目
-- Score: 70%
-- Pass: yes
-- Stable items: 3
-- Pattern items: 3
-- Case items: 1
-- Checkpoint present: no
-- Stable preview: Task focus: 开个新窗口继续做同一个终端项目 | Pinned: 用户稳定偏好口语化、不端着、可以自嘲但不说教。表达应像朋友聊天，避免居高临下、用力过猛或碎嘴。 | Pinned: 用户偏好先本地测通再上 Bot，避免烧配额；代码改动完成后要求 commit + push，避免本地与 GitHub 版本漂移。
-- Pattern preview: Workflow pattern: Cross-window continuity handoff Use when: When opening a fresh terminal window for the same project... | Workflow pattern: Promote recurring continuity workflow Use when: When the same startup or handoff workflow repeats a... | Workflow pattern: Checkpoint before switching windows Use when: When leaving an active coding window that will likely...
-- Case preview: Case: Three-terminal continuity trigger validation Problem: Claude Code, Codex, and Gemini CLI were configured with M...
-- Checkpoint summary: -
-- Matched stable any: -
-- Matched stable all: -
-- Matched patterns any: resume_context, checkpoint
-- Matched cases any: -
-- Matched checkpoint any: -
-- Forbidden matches: -
-- Notes: This is a stretch case. It should start passing as more workflow patterns are promoted into durable memory.
-
-### recallnest_checkpoint_continuity
-- Task: 继续 recallnest
-- Score: 100%
-- Pass: yes
-- Stable items: 3
-- Pattern items: 1
-- Case items: 1
-- Checkpoint present: yes
-- Stable preview: Checkpoint focus: RecallNest continuity layer — 状态交接给下个窗口 | Checkpoint summary: RecallNest Phase 3/4 并行推进中。核心三件套 store_memory/checkpoint_session/resume_context 已上线三端 MCP，但 resum... | Checkpoint decision: Phase 3/4 并行推进，continuity layer 是当前核心方向
-- Pattern preview: Workflow pattern: Promote recurring continuity workflow Use when: When the same startup or handoff workflow repeats a...
-- Case preview: Case: RecallNest sparse startup context cleanup Problem: resume_context was returning noisy transcript fragments and ...
-- Checkpoint summary: RecallNest Phase 3/4 并行推进中。核心三件套 store_memory/checkpoint_session/resume_context 已上线三端 MCP，但 resume_context compose 质量是最大短板——实测只返回 3 条 stable context、0 patter...
-- Matched stable any: -
-- Matched stable all: -
-- Matched patterns any: -
-- Matched cases any: -
-- Matched checkpoint any: Phase 3, resume_context, store_memory
-- Forbidden matches: -
-- Notes: When a project checkpoint exists, resume_context should prefer checkpoint-led continuity over noisy transcript fragments.
-