memory-layer-mcp 0.5.0__tar.gz

memory_layer_mcp-0.5.0/.claude/settings.local.json

@@ -0,0 +1,69 @@
+{
+  "hooks": {
+    "PreToolUse": [
+      {
+        "matcher": "Edit|Write",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "ROOT=$(git rev-parse --show-toplevel 2>/dev/null) && [ -n \"$ROOT\" ] && [ -f \"$ROOT/.hybrid-search/wiki/STALE.md\" ] && echo 'STALE wiki pages detected — update them BEFORE editing code:' && cat \"$ROOT/.hybrid-search/wiki/STALE.md\"; :"
+          }
+        ]
+      },
+      {
+        "matcher": "Grep|Read",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "/Users/ian/project/claude_project/hybrid-search-mcp/.venv/bin/python -m hybrid_search.cli qa-hook 2>/dev/null || true",
+            "timeout": 5
+          }
+        ]
+      }
+    ],
+    "PostToolUse": [
+      {
+        "matcher": "Edit|Write",
+        "hooks": [
+          {
+            "type": "command",
+            "command": "ROOT=$(git rev-parse --show-toplevel 2>/dev/null) && [ -n \"$ROOT\" ] && [ -f \"$ROOT/.hybrid-search/wiki-gaps.txt\" ] && echo 'Wiki gaps — new modules need wiki pages:' && cat \"$ROOT/.hybrid-search/wiki-gaps.txt\"; :"
+          }
+        ]
+      }
+    ],
+    "SessionStart": [
+      {
+        "hooks": [
+          {
+            "type": "command",
+            "command": "/Users/ian/project/claude_project/hybrid-search-mcp/.venv/bin/python -m hybrid_search.cli qa-hook 2>/dev/null || true",
+            "timeout": 5
+          }
+        ]
+      }
+    ],
+    "UserPromptSubmit": [
+      {
+        "hooks": [
+          {
+            "type": "command",
+            "command": "/Users/ian/project/claude_project/hybrid-search-mcp/.venv/bin/python -m hybrid_search.cli qa-hook 2>/dev/null || true",
+            "timeout": 10
+          }
+        ]
+      }
+    ],
+    "Stop": [
+      {
+        "hooks": [
+          {
+            "type": "command",
+            "command": "/Users/ian/project/claude_project/hybrid-search-mcp/.venv/bin/python -m hybrid_search.cli qa-hook 2>/dev/null || true",
+            "timeout": 5
+          }
+        ]
+      }
+    ]
+  }
+}

memory_layer_mcp-0.5.0/.github/workflows/tests.yml

@@ -0,0 +1,27 @@
+name: tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        python-version: ["3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+
+      - name: Install
+        run: pip install -e ".[dev]"
+
+      - name: Run tests
+        run: python -m pytest tests/ -q

memory_layer_mcp-0.5.0/.gitignore

@@ -0,0 +1,20 @@
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+dist/
+build/
+.venv/
+.env
+.env.*
+*.lock
+.ruff_cache/
+.pytest_cache/
+.mypy_cache/
+
+# Machine-local runtime state — never commit
+.hybrid-search/
+.codex/
+
+# PoC embedding cache — derived from conversation content, regenerated on run
+scripts/.poc_cache/

memory_layer_mcp-0.5.0/AGENTS.md

@@ -0,0 +1,5 @@
+<!-- hybrid-search-mcp:codex-routing -->
+## Hybrid Search Memory
+
+- Use `mcp__hybrid-search__hybrid_search` for project memory and code/docs search when context may already exist.
+- Codex hooks inject relevant hybrid-search memory before exploratory prompts and save completed turns after answers.

memory_layer_mcp-0.5.0/CHANGELOG.md

@@ -0,0 +1,301 @@
+# Changelog
+
+All notable changes to hybrid-search-mcp. Format loosely follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/);
+versions are [SemVer](https://semver.org/spec/v2.0.0.html).
+
+## Unreleased
+
+### Added
+
+- **Codex memory hooks**. New `codex-hook` entry handles Codex
+  `SessionStart`, `UserPromptSubmit`, and `Stop` payloads. Codex receives
+  hybrid-search context before exploratory prompts and completed turns are
+  saved as qa logs with `trigger: codex_stop_hook` and `client: codex`.
+- **Bounded answer excerpts for Stop-hook memory**. Claude and Codex completed
+  turns now persist a sanitized, capped `## Answer excerpt` plus
+  `answer_excerpt_chars`, improving memory quality without storing unbounded
+  transcripts.
+- **`install-codex-hook` CLI**. Writes Codex hooks to `.codex/hooks.json`
+  or `~/.codex/hooks.json`, enables `[features].hooks = true`, and
+  registers the MCP server using Codex TOML `[mcp_servers.hybrid-search]`.
+- Shared hook runtime for Claude Code and Codex prompt classification,
+  session context, prompt context, and completed-turn recording.
+- Status now reports Codex hook/config presence and warns about
+  `AGENTS.override.md` / near-limit `AGENTS.md` project docs.
+- **Memory cards**. New `memory-card create/list/show/grep` commands promote
+  qa logs into compact semantic memory under
+  `.hybrid-search/memory/cards/`, indexed as `node_type="memory_card"`.
+- **Memory compaction and graph-lite facts**. New `memory compact`,
+  `memory procedural review`, and `memory facts export/list` commands create
+  cards from qa logs, write reviewed procedural candidates, and export
+  lightweight temporal facts to `.hybrid-search/memory/facts.jsonl`.
+- **Router Phase 1 scanner noise filter**. Reindex now skips common content
+  binaries/media/archives and oversized Markdown under content roots by
+  default, with `[scanner.exclude]` overrides, `reindex --include-content`,
+  and a doctor excluded-paths summary.
+- **Router Phase 2 — quality signals**. Hybrid search responses now include
+  `top_score`, `score_gap`, confidence bands, weak-result fallback hints, and
+  a `recalibrate` CLI for project-specific thresholds.
+- **Router Phase 3 — heuristic prompt router**. User prompt hooks now surface
+  a bounded route suggestion for `hybrid_search`, `grep`, or `memory`, backed
+  by a hand-labeled router benchmark.
+- **In-flight dirty worktree visibility**. `hybrid_search(..., cwd=...)` now
+  overlays tracked uncommitted file changes as ephemeral `[in-flight]` results,
+  suppresses deleted dirty paths, and avoids persisting or embedding dirty
+  content.
+- `setup --dry-run`: preview CLAUDE.md/AGENTS.md changes without writing.
+- `setup --force`: recover from a corrupted routing block.
+
+### Changed
+
+- qa log frontmatter accepts an optional `client` field. Existing records
+  without it remain valid.
+- Memory-aware ranking now boosts curated `memory_card` chunks above raw
+  `qa_log` chunks for explicit recall queries.
+- Codex hook installation now migrates the deprecated
+  `[features].codex_hooks` flag to `[features].hooks`.
+- CLAUDE.md and AGENTS.md routing sections now use versioned sentinel
+  markers (`<!-- BEGIN/END hybrid-search-mcp routing v1 -->`). Existing
+  installs migrate automatically on the next `setup`. Idempotent —
+  re-running `setup` on a current install produces no diff.
+- Routing block now includes a self-justify rule (one-sentence tool
+  choice per call) and a weak-confidence fallback contract. G4 manual
+  replay (`benchmarks/router_replay_2026-05.md`) showed these rules
+  were already being followed via qa_log priming; the durable value is
+  pinning the contract in CLAUDE.md so it survives qa_log churn.
+
+## [0.4.0] — 2026-04-23
+
+**Memory integrity.** v0.3.0 guaranteed every turn persists; v0.4.0 keeps
+those persisted turns *useful* over time. Three deterministic passes run
+at every reindex — staleness, semantic dedup, archive TTL — so qa_log
+stops being an append-only dumpster and becomes actual memory that
+consolidates.
+
+### Added
+
+- **Staleness pruning** (`memory.integrity.detect_stale_qa`). qa_log
+  files whose ``## Top results`` paths are *all* absent from the store
+  DB are archived (not unlinked). Mirrors the v0.3.0 wiki orphan
+  detector.
+- **Semantic dedup** (`memory.integrity.detect_semantic_duplicates`).
+  For every pair of qa_log chunks, compares cosine similarity from the
+  already-indexed vectors. Pairs at or above threshold
+  (default **0.90**) cluster via union-find; the newest member is kept,
+  older members are archived. No re-embedding, no LLM cost.
+- **Archive tier**. Every prune (auto-prune, qa-prune, staleness,
+  dedup) moves files into ``<project>/.hybrid-search/qa-archive/
+  YYYY/MM/*.md`` instead of unlinking. Archive entries older than
+  ``archive_ttl_days`` (default **30**) are permanently removed on
+  subsequent reindexes — cheap insurance against regret.
+- **`qa-restore` CLI**. Brings an archived entry back into qa/. Accepts
+  the stem, hash prefix (≥4 hex chars), or friendly id from
+  ``qa-list``. Path-preserving restore.
+- **`integrity` CLI**. Runs the full pass on demand. `--dedup-threshold`
+  lets users tune sensitivity without editing config.
+- **`qa-stats` v2**. Surfaces ``active`` / ``archived`` / ``recent
+  archive (7d)`` / ``total ever`` counters alongside the existing
+  by-type / by-month breakdown. New ``by trigger`` line shows how many
+  of the active qa files came from each save path (``mcp_tool``,
+  ``stop_hook``, ``user_prompt_submit``).
+- **`[memory.integrity]` config block** — ``enabled`` (default true),
+  ``dedup_threshold`` (0.90), ``archive_ttl_days`` (30).
+- **`VectorEngine.get_vector(chunk_id)`** — surfaces the stored HNSW
+  vector to callers that need pairwise cosine without going through
+  ``search()``. Used by the dedup pass.
+- **Plan doc** `docs/plans/2026-04-23-v0.4.0-memory-integrity.md` with
+  architecture, stages, verification gates, and risk register.
+- Tests: 23 new in `test_memory_integrity.py` covering archive /
+  staleness / dedup / purge / restore / stats. Full suite:
+  **845 passing**.
+
+### Changed
+
+- Reindex tail order: `auto_prune` → `wiki_cleanup` → `integrity_pass`
+  → `archive purge`. Independent passes; each skips silently when
+  applicable sub-state is absent.
+- `_ensure_gitignore_entries` now auto-patches ``.hybrid-search/
+  qa-archive/`` so archived qa files never leak into git.
+
+### Verification
+
+Ran the full G-gate sweep:
+
+- G1 (staleness): unit suite + single-file smoke → archived correctly
+- G2 (dedup): live 3-identical-plants smoke → 1 kept, 2 archived
+- G3 (archive TTL): 45-day-old mtime-forge → purged on next pass
+- G4 (existing tests): 822 → 845 (+23, 100% pass)
+- G5 (new tests): 23 in `test_memory_integrity.py` alone
+- G6 (gitignore): ``.hybrid-search/qa-archive/`` auto-added on setup
+- G_integrity (valuein live): no new stale or dedup pairs on clean
+  run — environment already healthy post-v0.3.0 cleanup
+
+### Non-goals (unchanged from plan)
+
+- No LLM-based summarisation
+- No cross-project qa consolidation
+- Existing qa_log markdown format stays backward-compatible (additive
+  frontmatter only; v0.2.x files still parse cleanly)
+
+## [0.3.0] — 2026-04-23
+
+**Deterministic Memory Layer.** v0.2.0 left two leaks: qa_log save only
+fires when Claude chooses the MCP tool (stochastic), and wiki pages
+accumulate orphans as source files disappear from the index
+(gitignore drift, deletions). v0.3.0 closes both with two new hooks and
+automated wiki cleanup — every user–Claude turn persists regardless of
+tool choice, and every reindex purges orphan wiki pages.
+
+### Added
+
+- **`Stop` hook** (`hybrid_search.hooks._handle_stop`). Fires at the end
+  of every Claude turn; parses the transcript JSONL to find the last
+  user prompt + assistant activity, writes a qa_log entry tagged
+  ``trigger: stop_hook``. Respects ``stop_hook_active`` to avoid
+  continuation loops. Dedups against recent MCP-tool saves (5-second
+  window + query hash match) so turns that DID call
+  ``hybrid_search`` aren't double-persisted.
+- **`UserPromptSubmit` hook** (`_handle_user_prompt_submit`). Fires when
+  the user submits a prompt; classifies as exploratory via KO/EN
+  keyword heuristics (`어떤`/`어떻게`/`구조`/`how`/`explain`/...) plus
+  memory-intent markers (`지난번`/`previously`/...) that bypass the
+  length gate. Exploratory prompts trigger an on-the-fly
+  ``hybrid_search`` call; top-10 results are injected as
+  ``hookSpecificOutput.additionalContext`` and saved to qa_log with
+  ``trigger: user_prompt_submit``. Slash commands, bash pass-through,
+  `@file` references, and short single-token prompts short-circuit
+  before any search.
+- **Orphan wiki auto-cleanup** (`hybrid_search.wiki_cleanup`). At the
+  end of every reindex, compares each wiki page's ``## Files`` refs
+  against the store DB's ``relative_path`` set; pages whose references
+  are *all* absent from the DB are deleted. Protects against gitignore
+  drift (files excluded from the index but still on disk) and actual
+  file deletion. Available as a standalone CLI via
+  ``hybrid-search-mcp wiki-cleanup [--dry-run] [--verbose]``.
+- **qa_log format v2**: optional frontmatter fields ``trigger``,
+  ``tools_used``, ``answer_chars``. Backward-compatible — v0.2.x
+  records (without these fields) still parse cleanly.
+- **Stronger `CLAUDE.md` routing section**: imperative wording
+  ("반드시 이 순서로", "예외 없이") plus explicit
+  ``mcp__hybrid-search__hybrid_search`` tool name and trigger-word
+  tables per question category. Replaces the descriptive v0.2.x
+  "의도 기반 라우팅" section on re-setup.
+- **`install-memory-hook` upgrade**: now installs all four hook types
+  (PreToolUse, SessionStart, UserPromptSubmit, Stop). Existing
+  installs with stale paths are refreshed in place. Idempotent.
+- **`scripts/wiki_bloat_audit.py`** gains a `--delete` flag with a
+  DB-based orphan detector (previously disk-only, which undercounted
+  gitignore-drift zombies by an order of magnitude).
+- **Plan doc** at `docs/plans/v0.3.0-deterministic-memory.md` with
+  goals, design, stages, verification checklist, risk register.
+- Tests: 32 new (`test_memory_hook.py` +TestStopHook +TestUserPromptSubmitHook
+  +TestExploratoryClassifier, `test_wiki_orphan_cleanup.py`).
+  Full suite: **822 passing**.
+
+### Changed
+
+- `qa_log.record()` accepts a ``trigger`` argument (default
+  ``"mcp_tool"`` preserves prior behaviour). Adds ``record_turn()``
+  for non-MCP saves.
+- `install_memory_hook()` returns ``{added, updated, path, status}``
+  — ``updated`` counts stale-path refreshes that were rewritten in
+  place. CLI output distinguishes installed vs refreshed blocks.
+
+### Fixed
+
+- **valuein_homepage cleanup**: 765 orphan wiki pages removed
+  (2,584 → 1,819 pages, ≈30% reduction). Almost all were
+  ``mindvault-out/*`` or ``docs/valueinmath_docs/학습/*`` pages that
+  had been excluded from the index by `.gitignore` additions but were
+  never pruned.
+- Stop hook's transcript parser skips local-command stdout,
+  command-message envelopes, and system-reminder blocks — only
+  genuine user prompts survive the filter.
+
+### Notes
+
+- The bench (`benchmarks/run_compounding_bench.py`) drives the
+  orchestrator directly, so it does *not* exercise the v0.3.0
+  Stop / UserPromptSubmit guarantees (those only fire inside Claude
+  Code). Re-run numbers post-cleanup show Track A identity memory
+  surface unchanged at 80 %; Track B paraphrase dropped from 65 % to
+  50 % in this environment, attributable to the 30 % fewer wiki
+  chunks now in the index. The guaranteed-save property is verified
+  by unit tests (``TestStopHook`` suite) rather than the compounding
+  bench.
+- Non-goals preserved: no MCP tools added; existing qa_log files
+  remain readable; module_synth page-generation algorithm untouched.
+
+## [0.2.0] — 2026-04-22
+
+**Memory Layer, evidenced.** v0.1 introduced the qa_log compounding loop
+but made no measurable claim. v0.2 closes that gap with a benchmark, disk
+retention, and automatic memory consultation hooks — the three pieces that
+turn "it works on my machine" into a shippable product.
+
+### Added
+
+- **Compounding benchmark** (`benchmarks/run_compounding_bench.py`,
+  `benchmarks/compounding_pairs.json`). LongMemEval / LoCoMo-inspired
+  session-separated Q1a → Q1b methodology with leakage split. Two tracks:
+  identity re-query (upper bound) and paraphrased follow-up (realistic
+  case). Reports memory surface rate, answer-found rate, gold recall@10
+  regression guard, with cold/warm deltas per category. Safe to run on
+  a live project — the script backs up and restores `.hybrid-search/qa/`
+  around the experiment.
+  - First run on valuein_homepage (20 pairs, 1,307 files):
+    - Identity: answer_found **80% → 95%** (+15pp), memory surface
+      **0% → 80%** (+80pp), gold recall@10 flat within noise (−1.7pp).
+    - Paraphrase: answer_found **75% → 95%** (+20pp), memory surface
+      **0% → 65%** (+65pp).
+    - Non-leaky subset (15 pairs): answer_found **73% → 100%**.
+- **Auto-prune on reindex** (`[memory]` config section, default on).
+  journald-style two-ceiling retention (`retention_days=90`,
+  `max_files=2000`). First activation is a dry-run — the user confirms
+  via `qa-prune --confirm-first-run` before the policy actually deletes
+  anything. Opt-out via `auto_prune=false` in `config.toml`.
+  - New reader primitives: `reader.select_over_count`,
+    `reader.prune_keep_latest`, `reader.auto_prune`.
+- **PreToolUse + SessionStart memory hook** (`hybrid-search qa-hook`,
+  `hybrid-search install-memory-hook`). Matches Graphify's pattern: a
+  one-time SessionStart summary + per-Grep/Read contextual nudges, both
+  returning the v2.1.9 `hookSpecificOutput.additionalContext` JSON so
+  Claude sees the hint in-context before running the tool. Hooks are
+  silent when they have nothing to say (noise patterns, no matches),
+  capped at 800 chars per injection. `install-memory-hook` merges into
+  `.claude/settings.local.json` non-destructively — existing hooks are
+  preserved, re-running is idempotent.
+- **`MemoryConfig`** in `hybrid_search.config` — `auto_prune`,
+  `retention_days`, `max_files`, `require_first_run_confirm`.
+- Default `config.toml` template now includes a `[memory]` block with
+  inline documentation.
+- Tests: 37 new (`test_memory_hook.py`, `test_qa_reader::TestPruneKeepLatest`,
+  `::TestAutoPrune`). Full suite: **783 passing**.
+
+### Changed
+
+- `README.md` — hero claim ("search quality compounds with usage") now
+  cites measured numbers instead of asserting. New sections:
+  "Compounding benchmark (2026-04-22)", "Retention — Memory doesn't
+  balloon your disk", "Automatic memory consultation — PreToolUse +
+  SessionStart hooks". Package description in `pyproject.toml` updated
+  to lead with "Memory Layer MCP for Claude Code".
+- `cmd_reindex` gains a final-step call to `_run_auto_prune` (no-op when
+  `[memory].auto_prune = false` or the qa dir is absent).
+
+### Notes
+
+- No breaking changes. Existing `.hybrid-search/qa/` directories are
+  untouched on first reindex (dry-run gate); users must opt in once per
+  project.
+- Paraphrase track's non-leaky subset achieves 100% answer_found at the
+  cost of a modest top-1 redistribution (gold code/doc sometimes drops
+  one rank when a qa_log chunk surfaces). This is the expected trade-off
+  for memory: the user gets both signals.
+
+## [0.1.0] — 2026-04-21
+
+Initial Memory Layer release. `qa_log` persistence default-on, 30-day
+half-life decay, memory-intent boost (지난번에 / previously), secret
+filter, per-project scoping. Details in `HANDOFF.md`.

memory_layer_mcp-0.5.0/CLAUDE.md

@@ -0,0 +1,46 @@
+# Hybrid Search MCP
+
+BM25 + Vector 하이브리드 검색 MCP 서버.
+
+## 실행 환경
+
+```bash
+source .venv/bin/activate
+python -m pytest tests/ -x -q
+```
+
+<!-- BEGIN hybrid-search-mcp routing v1 -->
+## 검색 전략 — 반드시 이 순서로
+
+이 프로젝트는 `hybrid-search-mcp` Memory Layer가 설치돼 있다. **아래 규칙을 예외 없이 지킬 것.**
+
+| 질문 유형 | 신호 | **반드시 먼저 호출** | 보충 |
+|---|---|---|---|
+| **기능 탐색** | "어떤 기능", "관련 기능", "어떻게 구성", "흐름", "설명해줘", "정리해줘", "아키텍처" | `mcp__hybrid-search__hybrid_search` | Grep, Read |
+| **설계/맥락** | "왜 이렇게", "배경", "이유", "결정", "히스토리", "지난번" | `mcp__hybrid-search__hybrid_search` | Wiki |
+| **스키마/DB** | "테이블", "마이그레이션", "DDL", "스키마 변화" | `mcp__hybrid-search__hybrid_search` (file_pattern `*.sql`) | Grep |
+| **구조/관계** | "전체 그림", "누가 호출", "의존" | Wiki (`.hybrid-search/wiki/index.md`) | `mcp__hybrid-search__hybrid_search` |
+| **정밀 조회** | 정확한 심볼명 / 파일명 / 에러 문자열 | Grep | Read |
+
+**운영 규칙**:
+- **탐색형 질문에 Grep 먼저 호출 금지** — 반드시 `mcp__hybrid-search__hybrid_search` 먼저.
+- **쿼리는 사용자의 자연어 문장을 그대로** 쓸 것 — 키워드 뭉치로 재작성 금지.
+  (예: "우리 환불 기능에 대해 알려줘" ⭕ / "환불 퇴원 refund 워크플로우 정산" ❌)
+  자연어 문장이 벡터 매칭 품질이 더 좋고, 분류기가 가중치를 자동 조정한다.
+- 1차에서 답이 부족해도 도구를 **바꾸지 말고 같은 레인에서 보충** (hybrid→wiki MCP 레인, grep→read 텍스트 레인).
+- Wiki는 `.hybrid-search/wiki/index.md`에서 시작, `[[링크]]` 있으면 따라갈 것.
+
+**자동 동작 (수동 개입 불필요)**:
+- 질문 시작 시 관련 과거 Q&A 자동 컨텍스트 주입 (UserPromptSubmit)
+- 세션 시작 시 최근 Q&A 요약 주입 (SessionStart)
+- 답변 종료 시 `.hybrid-search/qa/`에 자동 저장 (Stop)
+- `git commit` 후 변경 파일만 재인덱싱 + 좀비 wiki 자동 삭제
+
+**자기 정당화 (Self-justify)**:
+- 모든 검색 호출 직전, **한 문장으로 어떤 도구를 골랐고 왜인지** 말할 것.
+- 예: "탐색형 질문이라 `mcp__hybrid-search__hybrid_search` 먼저 호출합니다."
+
+**Confidence 계약 (weak → fallback)**:
+- `hybrid_search` 응답의 `confidence: weak`이면 답하기 전에 `fallback_hint`에 적힌 대체 도구로 한 번 더 시도할 것.
+- `strong`/`mixed`면 그대로 진행.
+<!-- END hybrid-search-mcp routing v1 -->