lethe-memory 0.2.0__tar.gz

lethe_memory-0.2.0/.claude-plugin/marketplace.json

@@ -0,0 +1,21 @@
+{
+  "name": "lethe-plugins",
+  "owner": { "name": "Teimur Gasanov" },
+  "metadata": {
+    "description": "Self-improving memory for Claude Code — markdown-first, hybrid retrieval, RIF, optional LLM enrichment",
+    "version": "0.1.0"
+  },
+  "plugins": [
+    {
+      "name": "lethe",
+      "description": "Persistent memory across Claude Code sessions — hybrid BM25+dense retrieval, clustered retrieval-induced forgetting, optional Haiku enrichment.",
+      "version": "0.1.0",
+      "source": "./plugins/claude-code",
+      "author": { "name": "Teimur Gasanov" },
+      "homepage": "https://github.com/teimurjan/lethe",
+      "repository": "https://github.com/teimurjan/lethe",
+      "license": "MIT",
+      "keywords": ["memory", "rif", "hybrid-retrieval", "markdown", "longmemeval"]
+    }
+  ]
+}

lethe_memory-0.2.0/.github/workflows/publish-pypi.yml

@@ -0,0 +1,40 @@
+name: publish
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  build:
+    name: build sdist + wheel
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - name: Install build tooling
+        run: python -m pip install --upgrade build
+      - name: Build distributions
+        run: python -m build
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist/
+          if-no-files-found: error
+
+  publish:
+    name: upload to PyPI (trusted publishing)
+    needs: build
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/p/lethe-memory
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          name: dist
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@release/v1

lethe_memory-0.2.0/.github/workflows/release-please.yml

@@ -0,0 +1,41 @@
+name: release-please
+
+on:
+  push:
+    branches: [main]
+  workflow_dispatch:
+
+permissions:
+  contents: write
+  pull-requests: write
+  id-token: write
+
+jobs:
+  release-please:
+    runs-on: ubuntu-latest
+    outputs:
+      release_created: ${{ steps.release.outputs.release_created }}
+    steps:
+      - id: release
+        uses: googleapis/release-please-action@v4
+        with:
+          config-file: release-please-config.json
+          manifest-file: .release-please-manifest.json
+
+  publish:
+    needs: release-please
+    if: needs.release-please.outputs.release_created == 'true' || github.event_name == 'workflow_dispatch'
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/p/lethe-memory
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: python -m pip install --upgrade build
+      - run: python -m build
+      - uses: pypa/gh-action-pypi-publish@release/v1

lethe_memory-0.2.0/.gitignore

@@ -0,0 +1,16 @@
+results/
+data/
+__pycache__/
+*.egg-info/
+.venv/
+.mypy_cache/
+dist/
+*.pyc
+.DS_Store
+.coverage
+.pytest_cache/
+writeup/
+.memsearch/
+.claude/
+.env
+.lethe

lethe_memory-0.2.0/.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "0.2.0"
+}

lethe_memory-0.2.0/BENCHMARKS.md

@@ -0,0 +1,165 @@
+# Benchmarks
+
+Canonical results for `lethe`. For the narrative (why each experiment was run, what failed, how we got here), see [RESEARCH_JOURNEY.md](RESEARCH_JOURNEY.md).
+
+## Methodology
+
+### Dataset
+- **LongMemEval S variant** via HuggingFace `xiaowu0162/longmemeval-cleaned`.
+- **199,509 conversation turns** (the full "haystack") + **500 evaluation questions** with 1–3 relevant turns each, annotated via qrels.
+
+### Retrieval setup
+- Global corpus — all 200k turns indexed together, no per-query pre-filtering.
+- Turn-level retrieval (not session-level).
+- Return top-10 results per query.
+
+### Primary metric — NDCG@10
+Position-sensitive; penalizes putting the right turn at rank 8 vs rank 1. Ideal DCG computed from qrels.
+
+### Supporting metrics
+- **Recall@k** (k ∈ {5, 10, 30}) — fraction of qrels hits appearing in top-k.
+- **exact_episode** — top-1 ∈ qrels (precision@1 by another name).
+- **sibling_confusion** — top-1 is from an answer-session but not in qrels (right topic, wrong turn).
+- **wrong_family** — top-1 is from a session with no relevant turns (unrelated topic).
+- **stale_fact** (knowledge-update only) — top-1 from answer-session but not in qrels (proxy for older version of the fact).
+- **abstain@T** — fraction where top-1 xenc score < T (would trigger abstention).
+
+### Burn-in protocol for RIF results
+RIF is a learned mechanism — needs usage history to accumulate suppression. All RIF numbers below use a **5000-step burn-in** (query schedule: 70% hot queries drawn from a 20% subset, 30% cold queries) before the eval pass.
+
+### Integrity checks
+- All systems evaluated on identical queries with identical qrels.
+- All systems return exactly 10 results to the scorer.
+- No hyperparameters tuned on the eval set.
+- RRF vs our hybrid merge produce identical numbers on the baseline — confirmed the 0.3680 number is from the cross-encoder, not from the merge strategy (see checkpoint 10).
+
+---
+
+## Results
+
+### Retrieval baselines
+
+200-query eval sample, checkpoint 10 audit:
+
+| System | NDCG@10 | Recall@10 | Notes |
+|--------|---------|-----------|-------|
+| Vector only (MiniLM top-10) | 0.1376 | 0.2173 | Bi-encoder baseline |
+| BM25 only (top-10) | 0.2420 | 0.3264 | Sparse keyword baseline |
+| Hybrid BM25+vector RRF (memsearch style) | 0.2171 | 0.3334 | Rank fusion, no reranker |
+| Vector + cross-encoder rerank (k=30) | 0.2425 | 0.2892 | Dense + reranker |
+| **Hybrid + cross-encoder rerank (k=30)** | **0.3680** | **0.4694** | **Best without any learned component** |
+
+500-query full eval (the default for all Phase 2+ results):
+
+| System | NDCG@10 | Recall@30 |
+|--------|---------|-----------|
+| Baseline (hybrid + xenc, no RIF) | 0.2960 | 0.4103 |
+
+### Retrieval-induced forgetting (RIF)
+
+Checkpoints 11–13. All on 500-query full eval, 5000-step burn-in.
+
+| Config | NDCG@10 | Recall@30 | Notes |
+|--------|---------|-----------|-------|
+| Global RIF (original formula) | 0.2993 (+1.1%) | 0.4142 (+0.9%) | Cue-independent; conservative is optimal |
+| Global RIF (gap formula) | 0.3037 (+2.6%) | 0.4250 (+3.6%) | Gap formula alone is 2.4× better than original |
+| Clustered RIF 30 (original) | 0.3132 (+5.8%) | 0.4381 (+6.8%) | K-means 30 query clusters, per-cluster suppression |
+| **Clustered RIF 30 + gap formula** | **0.3152 (+6.5%)** | **0.4494 (+9.5%)** | **Best retrieval-only configuration** |
+
+Default config for the best row: `alpha=0.3`, `suppression_rate=0.1`, `reinforcement_rate=0.05`, `decay_lambda=0.005`, `n_clusters=30`, `use_rank_gap=True`.
+
+**Mechanism**: entries that make the candidate pool but lose to the cross-encoder accumulate a per-cluster suppression score. On subsequent retrievals in the same query cluster, suppression penalizes RRF scores before the cross-encoder sees them.
+- **Clustered**: `{entry_id → {cluster_id → float}}` instead of `{entry_id → float}`. An entry suppressed for "travel" queries stays available for "food" queries.
+- **Gap formula**: `max(0, xenc_rank − initial_rank) / pool × sigmoid(−xenc)`. Only penalizes entries that actually dropped in rank AND were actively rejected.
+
+### Extended behavior metrics (checkpoint 16)
+
+Decomposes the +6.5% NDCG gain into behavior-level changes. 500-query eval.
+
+| Metric | baseline | clustered+gap RIF | Δ |
+|--------|----------|-------------------|---|
+| exact_episode | 0.208 | 0.222 | +0.014 |
+| ndcg@10 | 0.293 | 0.316 | +0.023 |
+| **wrong_family** | **0.688** | **0.672** | **−0.016** |
+| sibling_confusion | 0.104 | 0.106 | +0.002 |
+| stale_fact (knowledge-update only) | 0.205 | 0.205 | 0.000 |
+| abstain@0 | 0.180 | 0.166 | −0.014 |
+| abstain@2 | 0.324 | 0.310 | −0.014 |
+| abstain@4 | 0.512 | 0.506 | −0.006 |
+
+**Read**: the NDCG gain comes primarily from **reducing cross-topic retrieval** (−1.6pp `wrong_family`). RIF has no effect on within-session discrimination (`sibling_confusion`) or temporal awareness (`stale_fact`).
+
+### LLM enrichment layer (checkpoint 17)
+
+Write-time structured extraction via `claude-haiku-4-5`: gist, 3 anticipated queries, entities, temporal markers. Indexed alongside original text in BM25/vector. Cross-encoder still scores original text.
+
+Coverage: 975 entries enriched (15% of queries have ≥1 enriched qrels entry). Partial-coverage result — overall numbers are diluted by the 85% of uncovered queries.
+
+**Overall (500 queries):**
+
+| Metric | baseline | RIF | RIF + enrichment | Δ(enr−RIF) |
+|--------|----------|-----|------------------|-----------|
+| exact_episode | 0.208 | 0.218 | 0.220 | +0.002 |
+| ndcg@10 | 0.293 | 0.312 | 0.324 | +0.012 |
+| wrong_family | 0.688 | 0.676 | 0.674 | −0.002 |
+
+**Covered bucket (75 queries — where the mechanism applies):**
+
+| Metric | baseline | RIF | RIF + enrichment | Δ(enr−RIF) |
+|--------|----------|-----|------------------|-----------|
+| exact_episode | 0.267 | 0.280 | **0.333** | **+0.053 (+19% rel)** |
+| ndcg@10 | 0.350 | 0.390 | **0.473** | **+0.083 (+21% rel)** |
+| wrong_family | 0.720 | 0.707 | **0.653** | **−0.053** |
+| sibling_confusion | 0.013 | 0.013 | 0.013 | 0 |
+| abstain@4 | 0.280 | 0.253 | 0.227 | −0.027 (more confident) |
+
+The covered-bucket +8.3pp NDCG gain is **3.6× larger than RIF's own contribution** — the largest single-lever improvement in the project. Mechanism: anticipated_queries reduces vocabulary mismatch, so correct sessions reach the candidate pool more often.
+
+**Cost**: $1.60 for 1000 entries on Haiku; system prompt is below the 4096-token cache minimum, so no prompt caching. Projected ~$16 for full coverage of the ~10k answer-relevant entries.
+
+### What doesn't work (negative results)
+
+Summarized here; full numbers in [RESEARCH_JOURNEY.md](RESEARCH_JOURNEY.md).
+
+| Approach | Phase | Result |
+|----------|-------|--------|
+| Embedding mutation (direct Gaussian, adapter, MLP) | 1 | No gain or negative on retrieval quality |
+| Text segmentation | 1 | −46% NDCG |
+| Co-relevance graph | 1 | −2.7% on LongMemEval (too sparse) |
+| GC routing index | 1 | −11% |
+| Exploration + rescue list (top-K injection) | 2 | −2.6pp at scale (looked positive on small eval, variance) |
+| Sparse Distributed Memory (binary LSH + cleanup) | 3 | FAISS wins outright on synthetic episodic data |
+| Iterative cleanup on SDM | 3 | Drifts to sibling prototypes — ≈3× precision drop |
+
+---
+
+## Benchmark methodology note (comparison with other tools)
+
+All numbers on this page measure **full-corpus NDCG@10 with 200k-turn needle-in-haystack retrieval**. This is deliberately close to production — in a real deployed memory, you don't know upfront which sessions are relevant to a query.
+
+Other memory-tool benchmarks commonly report **recall@5 over per-query fresh DBs of ~50 sessions at session granularity** — a ~2000× easier task (random baseline 10% vs 0.005%). Some implementations additionally leak ground-truth signal at indexing time (e.g. filtering assistant turns on `has_answer=true`, which is a dataset annotation). Published numbers in the 95-99% range on that methodology are state-of-the-art *for that methodology*; they are not comparable to numbers here.
+
+Running either side under a shared methodology is a separate experiment that hasn't been done.
+
+---
+
+## How to reproduce
+
+```bash
+# Prep data
+uv run python experiments/data_prep.py --dataset longmemeval
+
+# Retrieval-only
+uv run python benchmarks/run_benchmark.py          # checkpoint 10 baselines
+uv run python benchmarks/run_rif_benchmark.py      # checkpoint 11 (global RIF)
+uv run python benchmarks/run_rif_clustered.py      # checkpoint 12
+uv run python benchmarks/run_rif_gap.py            # checkpoint 13 (best retrieval-only)
+uv run python benchmarks/run_rif_extended_metrics.py  # checkpoint 16
+
+# LLM enrichment (needs ANTHROPIC_API_KEY)
+export ANTHROPIC_API_KEY=sk-ant-...
+uv run python experiments/enrich_longmemeval.py    # ~$16 for 10k entries
+uv run python benchmarks/run_rif_enriched.py       # checkpoint 17 (3-arm)
+```
+
+Raw per-run outputs from each benchmark script are checked into `benchmarks/results/` for auditability.

lethe_memory-0.2.0/CHANGELOG.md

@@ -0,0 +1,50 @@
+# Changelog
+
+## [0.2.0](https://github.com/teimurjan/lethe/compare/lethe-memory-v0.1.0...lethe-memory-v0.2.0) (2026-04-17)
+
+
+### Features
+
+* Add analysis script with plots and summary table ([b1e140b](https://github.com/teimurjan/lethe/commit/b1e140b93a4df4ebd32e483c779b7a8ea093fde5))
+* Add BM25 hybrid retrieval, +73% over static on LongMemEval ([7dd3320](https://github.com/teimurjan/lethe/commit/7dd33206804039d6076481e2b8aaff259a314caf))
+* Add data preparation script for NFCorpus ([a47ab4b](https://github.com/teimurjan/lethe/commit/a47ab4b793d62b613ef702e1e59207d70998d711))
+* Add health metrics (diversity, anchor drift, NDCG, recall) ([e0d13eb](https://github.com/teimurjan/lethe/commit/e0d13eb6a734a31b8760b937e4c621efc09076bd))
+* Add MemoryEntry dataclass and Tier enum ([c021bfb](https://github.com/teimurjan/lethe/commit/c021bfb7c6f6c42473c1cb80c7a5f4cf957fc942))
+* Add Static and RandomMutation baseline stores ([f20803f](https://github.com/teimurjan/lethe/commit/f20803f83598648db13e3799f0beeba34b3df421))
+* Clustered RIF, +5.8% NDCG — cue-dependent suppression 5x better than global ([cbfe8c3](https://github.com/teimurjan/lethe/commit/cbfe8c3e32b099eda3589656c638f09e66f37db1))
+* Enrichment layer validated on covered queries — +8.3pp NDCG over RIF ([eb4e551](https://github.com/teimurjan/lethe/commit/eb4e5513f6460aca8309b35ff94ff86ca908d86d))
+* Exploration + rescue list with smart top-K injection ([a828402](https://github.com/teimurjan/lethe/commit/a828402744f63e0bad8a64f56f261586873bef6d))
+* Implement experiment runner with circuit breakers ([4b6c4dd](https://github.com/teimurjan/lethe/commit/4b6c4dd289deb0832d8c803fde3c3df6b2e6f2cb))
+* Implement GCMemoryStore with FAISS retrieval and GC loop ([9fc0544](https://github.com/teimurjan/lethe/commit/9fc05440033d1a8729d6704aa95fba0695d82442))
+* Implement mutation and selection with unit tests ([a731564](https://github.com/teimurjan/lethe/commit/a731564f506adb761543fb0f922e65d1762f210f))
+* Lifecycle benchmark confirms GC tiers don't help retrieval ([2bc0df3](https://github.com/teimurjan/lethe/commit/2bc0df3c5845ec2638daa3914f59055464ff9bab))
+* LLM enrichment layer for checkpoint 13 (pipeline, not yet run) ([d9676ca](https://github.com/teimurjan/lethe/commit/d9676ca36193dc1fdfbd63e31ad1e1d4fe09d35a))
+* MLP with per-step delta cap shows first positive result ([fbd70fb](https://github.com/teimurjan/lethe/commit/fbd70fb9089e91eddc682bcbf222afccaa3208ae))
+* Production MemoryStore API with SQLite + FAISS + BM25 ([82f7076](https://github.com/teimurjan/lethe/commit/82f7076f23cc80e364f50def38340614a0ea90d3))
+* Rank-gap competition formula, clustered+gap hits +6.5% NDCG / +9.5% recall ([784bd59](https://github.com/teimurjan/lethe/commit/784bd597c81cbb936ad2e9c0b46c751d55bacca3))
+* Rescue index gives +12.9% hot NDCG on LongMemEval ([231e06c](https://github.com/teimurjan/lethe/commit/231e06c9eed988d6807452133450e79a82306941))
+* Retrieval-induced forgetting, first positive learned mechanism (+2.0% NDCG) ([3bb484f](https://github.com/teimurjan/lethe/commit/3bb484f8240bd8f356e79009c4e7385eb20af776))
+* Scaffold repo with pyproject.toml, config, and README ([f278435](https://github.com/teimurjan/lethe/commit/f27843593e8a6bcbd322da74a3ad1dd6fcf4aef2))
+* Ship lethe as a Claude Code plugin ([87f1a40](https://github.com/teimurjan/lethe/commit/87f1a403e10d2549d1380e9cef83c138a9eeb4bf))
+* Sparse Distributed Memory (SDM) research prototype ([7056b6c](https://github.com/teimurjan/lethe/commit/7056b6c2cb050fd590803e3f766b97bfc69c403a))
+* Stabilize MLP with lower promotion thresholds ([2798a4a](https://github.com/teimurjan/lethe/commit/2798a4afa044e98e566eac1d2bab41af18fca3ca))
+* Switch benchmark from NFCorpus/BEIR to LongMemEval ([b4a7194](https://github.com/teimurjan/lethe/commit/b4a7194e6cb924e6e6e82863097d7f5462fedd43))
+* V3 cross-encoder guided adapter mutation ([019bb1f](https://github.com/teimurjan/lethe/commit/019bb1f55ea1c6c62c0c8692b014c591d015c187))
+* V4 learned MLP adapter + segmentation mutation ([ef9a9c6](https://github.com/teimurjan/lethe/commit/ef9a9c649d8cf3a738ab06c1c093542adf34719f))
+* V4 results on both datasets + MLP selection fix ([aa4f333](https://github.com/teimurjan/lethe/commit/aa4f33377c60b526ad90de74c915821ec80e99ac))
+* Wire clustered RIF into production, add video demo ([d2b728b](https://github.com/teimurjan/lethe/commit/d2b728b6b90bbf27326e2e705cca0c14ce49d687))
+
+
+### Bug Fixes
+
+* Average NDCG@10 over all eval queries, not single sample ([413943e](https://github.com/teimurjan/lethe/commit/413943e7b9a567a765edc6729c8626d4ad35012d))
+* Enrichment rate-limiting — default concurrency 20→5, max_retries 4→8 ([2c04558](https://github.com/teimurjan/lethe/commit/2c045582ffe17c97d567d74fc12008d9353bfc00))
+
+
+### Documentation
+
+* Checkpoint 14 negative result — retrieval-only ceiling reached ([ca6fb06](https://github.com/teimurjan/lethe/commit/ca6fb063d8b4fdcc0d6385c84bc478b9a77510a4))
+* Checkpoint 15 — SDM prototype, negative result ([d657de4](https://github.com/teimurjan/lethe/commit/d657de4ee8a93e7c4be55698d2698171c207b7f8))
+* Checkpoints 16-17 + benchmark-methodology disclaimer ([082b0c6](https://github.com/teimurjan/lethe/commit/082b0c6380f872ac0e2a60399f58b817a41f96db))
+* Restructure — move result snapshots into benchmarks/results, rewrite BENCHMARKS and RESEARCH_JOURNEY ([1373264](https://github.com/teimurjan/lethe/commit/13732649ae27a8f42673d9c8573665f1002c3aa2))
+* Rewrite README for checkpoints 13+17 and methodology note ([7637ce2](https://github.com/teimurjan/lethe/commit/7637ce2a00c59d6fd1cbd883c154976877ea46bc))

lethe_memory-0.2.0/CLAUDE.md

@@ -0,0 +1,104 @@
+# lethe
+
+Self-improving memory store for LLM agents. BM25 + dense vector hybrid retrieval with cross-encoder reranking, clustered retrieval-induced forgetting, and optional LLM write-time enrichment.
+
+Ships as a **Claude Code plugin** (`plugins/claude-code/`) that drops a `.lethe/` directory into each project, writes daily markdown memory via hooks, and surfaces prior-session context through a `memory-recall` skill.
+
+## stack
+
+- Python 3.12, managed with `uv`
+- FAISS (faiss-cpu) for dense vector index
+- rank_bm25 for sparse keyword index
+- sentence-transformers: all-MiniLM-L6-v2 (bi-encoder), ms-marco-MiniLM-L-6-v2 (cross-encoder)
+- SQLite for metadata persistence
+- numpy for vectors, matplotlib for plots, pytest for tests
+- LongMemEval (S variant) + NFCorpus for benchmarks
+
+## layout
+
+```
+src/lethe/            # Production library (162 tests, ~95% coverage)
+├── memory_store.py   # Main API: MemoryStore
+├── markdown_store.py # Markdown chunker (##/### headings, content hashes)
+├── cli.py            # `lethe` CLI: index/search/expand/status/config/reset/enrich/projects
+├── union_store.py    # Read-only cross-project search via DuckDB ATTACH
+├── db.py             # DuckDB persistence (lethe.duckdb, unlimited ATTACH)
+├── vectors.py        # FAISS + BM25 index management
+├── reranker.py       # Cross-encoder + adaptive depth
+├── rif.py            # Retrieval-Induced Forgetting (clustered + gap)
+├── enrichment.py     # Optional LLM write-time enrichment (Anthropic SDK)
+├── dedup.py          # Hash + cosine deduplication
+├── encoders.py       # ONNX bi-encoder + cross-encoder adapters (fastembed)
+├── _lock.py          # fcntl-based per-project lock for concurrent CLI calls
+├── _registry.py      # ~/.lethe/projects.json for cross-project search
+└── entry.py          # MemoryEntry dataclass + Tier enum
+
+plugins/claude-code/  # Claude Code plugin
+├── hooks/            # SessionStart / UserPromptSubmit / Stop / SessionEnd
+├── scripts/          # transcript.py + derive-collection.sh helpers
+└── skills/memory-recall/
+
+.claude-plugin/       # Marketplace manifest (for `/plugin marketplace add`)
+
+benchmarks/           # Per-checkpoint benchmark scripts
+├── run_*.py
+├── _lib/             # Benchmark-only helpers (metrics, NDCG/recall)
+└── results/          # Raw per-run output markdowns
+
+scripts/              # Reproducibility utilities (dataset prep, enrichment runner)
+
+research/             # Experimental / non-production code
+├── gc_mutation/      # Germinal-center mutation thread (checkpoints 1-10)
+└── sdm/              # Sparse Distributed Memory prototype (checkpoint 15)
+
+tests/                # Production unit tests
+```
+
+## commands
+
+```bash
+uv venv --python 3.12 && uv pip install -e .
+uv run pytest tests/ -v
+uv run python experiments/data_prep.py --dataset longmemeval
+uv run python benchmarks/run_benchmark.py
+
+# CLI (exposed as console script `lethe`)
+lethe index                              # reindex .lethe/memory (auto-registers project)
+lethe search "query" --top-k 5           # single-project
+lethe search "query" --all --top-k 5     # all registered projects (DuckDB ATTACH)
+lethe projects list|add|remove|prune     # manage ~/.lethe/projects.json
+lethe expand <chunk-id>
+lethe status
+
+# Run CLI without local install
+uvx --from git+https://github.com/teimurjan/lethe lethe --version
+```
+
+## key architecture decisions
+
+- BM25 + FAISS hybrid retrieval (BM25 is the strongest single signal on conversation data)
+- Cross-encoder reranking on the merged candidate pool
+- Adaptive search depth: shallow k=30 for confident queries, deep k=200 when unsure
+- Cosine 0.95 dedup on add (removes 4.6% of LongMemEval, +6.5% NDCG)
+- Tier lifecycle: naive → gc → memory (with decay and apoptosis)
+- RIF: retrieval-induced forgetting suppresses chronic false positives at candidate selection stage
+- SQLite + .npz + FAISS for persistence (no external services)
+
+## benchmark results
+
+LongMemEval S (200k turns, 500 questions, 200-query eval sample):
+
+| System | NDCG@10 |
+|--------|---------|
+| Vector only | 0.1376 |
+| BM25 only | 0.2420 |
+| Hybrid RRF (memsearch style) | 0.2171 |
+| **Hybrid + cross-encoder rerank** | **0.3680** |
+
+The 0.3680 is from BM25+vector+cross-encoder reranking (standard IR). The GC mechanism does not improve this number. Verified with integrity checks.
+
+## research status
+
+17 checkpoints. Checkpoints 1-10 (GC mechanisms): all failed. Checkpoint 11 (global RIF): +1.1%. Checkpoint 12 (clustered RIF): +5.8%. Checkpoint 13 (clustered + rank-gap): +6.5% NDCG, +9.5% recall@30 — retrieval-only best. Checkpoint 14 (exploration + rescue list): negative at full scale. Checkpoint 15 (SDM research prototype in `sdm/`): negative. Checkpoint 16 (extended metrics for checkpoint 13): RIF's gain is primarily from reducing wrong_family (−1.6pp); sibling_confusion and stale_fact unchanged. Checkpoint 17 (LLM enrichment layer, Haiku write-time gist + anticipated queries + entities + temporal markers): **+8.3pp NDCG on covered queries (+21% rel over RIF), largest single-lever gain in the journey**. Overall diluted to +1.2pp by partial coverage (15% of queries). Full corpus enrichment expected ~$16.
+
+Full research journey: [RESEARCH_JOURNEY.md](RESEARCH_JOURNEY.md)

lethe_memory-0.2.0/PKG-INFO

@@ -0,0 +1,21 @@
+Metadata-Version: 2.4
+Name: lethe-memory
+Version: 0.2.0
+Summary: Self-improving memory store for LLM agents: hybrid retrieval, clustered retrieval-induced forgetting, optional LLM enrichment
+Requires-Python: >=3.11
+Requires-Dist: anthropic>=0.30
+Requires-Dist: duckdb>=0.10
+Requires-Dist: faiss-cpu>=1.7
+Requires-Dist: fastembed>=0.4
+Requires-Dist: numpy>=1.24
+Requires-Dist: rank-bm25>=0.2
+Provides-Extra: benchmarks
+Requires-Dist: beir>=2.0; extra == 'benchmarks'
+Requires-Dist: datasets>=2.0; extra == 'benchmarks'
+Requires-Dist: matplotlib>=3.7; extra == 'benchmarks'
+Requires-Dist: sentence-transformers>=2.0; extra == 'benchmarks'
+Requires-Dist: tqdm>=4.60; extra == 'benchmarks'
+Provides-Extra: dev
+Requires-Dist: mypy>=1.0; extra == 'dev'
+Requires-Dist: pytest-cov>=4.0; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'