@cachly-dev/mcp-server 0.10.62 → 0.10.65

package/BENCH.md

@@ -22,7 +22,8 @@ isolates *ranking quality*, not retrieval:
 
 | Ranker | What it does | Analogy |
 |---|---|---|
-| `baseline` | raw BM25+ keyword score (`src/search.ts`) | a text-only / flat-file memory |
+| `flatfile` | naive term-overlap, no IDF/quality, recency tiebreak | an LLM reading `/memories` files |
+| `baseline` | raw BM25+ keyword score (`src/search.ts`) | a decent text search engine |
 | `cachly` | BM25+ then quality-aware rerank (`src/rerank.ts`) | structured, trust-aware recall |
 
 Quality reranking boosts a lesson by five orthogonal signals:
@@ -56,30 +57,51 @@ weak baseline look bad. We are improving an already-good one on the cases that m
 
 ## Current results
 
+Three rankers, same corpus:
+
+- **flat-file** — naive term-overlap, no IDF, no length normalization, **no quality
+  signal**, ties broken by recency. This is an honest (in fact *charitable*) stand-in
+  for "an LLM reading its own `/memories` files": it has no ranking engine.
+- **baseline** — raw BM25+ keyword ranking.
+- **cachly** — BM25+ followed by quality-aware reranking (`rerank.ts`).
+
 ```
   corpus: 17 lessons · queries: 13
 
-  metric         baseline       cachly          lift
-  Precision@1       69.2%  →     84.6%     +22.2%
-  Precision@3       33.3%  →     33.3%      +0.0%
-  Recall@3          96.2%  →     96.2%      +0.0%
-  MRR               82.1%  →     91.0%     +10.9%
-  nDCG@5            85.5%  →     92.4%      +8.1%
+  metric       flatfile baseline   cachly    vs flat
+  Precision@1     76.9%    69.2%    84.6%     +10.0%
+  Precision@3     37.2%    33.3%    33.3%     -10.3%
+  Recall@3       100.0%    96.2%    96.2%      -3.8%
+  MRR             87.2%    82.1%    91.0%      +4.4%
+  nDCG@5          89.9%    85.5%    92.4%      +2.8%
+
+  vs BM25 baseline : MRR +10.9% · Precision@1 +22.2%
+  vs flat-file mem : MRR  +4.4% · Precision@1 +10.0%
 ```
 
-**Headline: Precision@1 +22.2%, MRR +10.9%, nDCG@5 +8.1%.**
+**Headline: Precision@1 +22.2% vs BM25, +10.0% vs a flat-file memory.**
+
+Interpretation — read the metrics that matter for an *agent*: it acts on the **top**
+result, so **Precision@1** and **MRR** are decisive. There, cachly wins against both
+baselines: it puts the single proven fix first most often. On the hardest cases — a
+symptom-dense failed attempt that beats the success on raw text, or two near-identical
+success lessons where only one was reviewed — quality reranking surfaces the lesson
+that actually solves the problem.
 
-Interpretation: when the agent asks a question, the *first* result is the proven
-fix 22% more often. On the hardest cases — a symptom-dense failed attempt beating
-the success on raw text, or two near-identical success lessons where only one was
-reviewed — reranking consistently surfaces the lesson that actually solves the problem.
+**Why is flat-file's Precision@3 / Recall@3 higher?** Honesty matters: the flat-file
+ranker scores the *entire* 17-lesson corpus, while cachly and BM25 rank the capped
+BM25 candidate set. On a tiny corpus, "return more" mechanically inflates recall-at-k
+without making the *top* answer better. We deliberately did **not** rig this away — and
+we **steelman** the flat-file with a recency tiebreak it wouldn't normally have. Even
+so, cachly wins the metrics an agent actually depends on.
 
 ## Honest limitations (read this)
 
 - **Small corpus.** 17 lessons, 13 queries. This proves the *mechanism* works for
   each quality signal and guards against regressions. It does **not** yet prove
-  real-world magnitude. The next step is a larger, externally-sourced corpus and a
-  head-to-head against an actual flat-file memory baseline on real agent traces.
+  real-world magnitude. The flat-file head-to-head is now built in (a naive,
+  quality-blind ranker), but the next step is a larger, externally-sourced corpus
+  and the same head-to-head on **real agent traces** with third-party labels.
 - **Ranking, not retrieval.** Both rankers see the same BM25 candidates, so this
   measures the quality layer only. CKG-traversal and semantic retrieval are
   evaluated separately via integration tests and are not in these numbers.

package/CHANGELOG.md

@@ -7,6 +7,55 @@
 
 ---
 
+## [0.10.65] – 2026-05-29
+
+### Flat-file head-to-head in the benchmark (the proof gets honest)
+
+- **Cachly-Bench now compares three rankers**, not two: a `flatfile` ranker
+  (naive term-overlap, no IDF, no length norm, **no quality signal**, recency
+  tiebreak) joins the BM25 `baseline` and `cachly`. The flat-file ranker is an
+  honest — in fact *charitable* — stand-in for "an LLM reading its own memory files."
+- **Result:** cachly wins the metrics an agent depends on — **Precision@1 +10.0 %,
+  MRR +4.4 % vs. flat-file** (and +22.2 % / +10.9 % vs. BM25). The report and BENCH.md
+  are honest about *why* flat-file's Precision@3 / Recall@3 look higher (it ranks the
+  whole corpus; we did not rig that away).
+- **CI-defended:** a new regression test asserts cachly ≥ flat-file on Precision@1
+  and MRR, and that the flat-file simulation isn't trivially weak (MRR > 0.5).
+
+## [0.10.64] – 2026-05-29
+
+### The three decisive metrics, now measurable
+
+- **`brain_metrics(instance_id)`** — new tool (101st). Reports the three metrics
+  that decide whether the Brain delivers value:
+  1. **Time-to-first-recall** — `born_at` (first learn) → `first_recall_at` (first
+     proven recall), both `SET NX` so only the first event wins.
+  2. **Recall-lift** — published Cachly-Bench headline (+22.2 % P@1), CI-defended.
+  3. **Team-knowledge-reuse** — % of proven recalls that used a *teammate's* lesson.
+- **Cross-author reuse tracking.** `smart_recall` gains an optional `author`. When
+  you recall a lesson written by someone else, cachly increments
+  `cross_author_recalls`, records the distinct reuse pair, and surfaces a
+  "👥 Team knowledge reuse" banner inline — the value only a shared brain delivers.
+
+## [0.10.63] – 2026-05-29
+
+### Graceful-degradation stability
+
+- New utilities: `withTimeout`, `scanKeys` (capped + timed-out scan), `normalizeGitPath`.
+- `skill_gaps` / `brain_coverage` / `team_expertise_map` / `brain_file_map` now scan
+  via `scanKeys` — a huge keyspace can no longer hang the agent turn (3 s cap + key cap).
+- `brain_from_git` normalizes git rename paths (`src/{old => new}/f.ts`) so file nodes
+  don't fragment on renames. Removed a dead person-node scan in `skill_gaps`.
+- +13 utils tests.
+
+## [0.10.62] – 2026-05-29
+
+### Hardened Phase 3 tools
+
+- `brain_who_knows`: reject empty/undefined `topic` (was crashing in `ckgSlug`); clamp `limit` to [1,50].
+- `brain_file_map`: filter empty path strings, handle non-array input gracefully.
+- +5 stability tests.
+
 ## [0.10.61] – 2026-05-29
 
 ### Phase 3C: 100 MCP tools milestone — zero-setup knowledge graph

package/PROGRESS.md

@@ -0,0 +1,141 @@
+# cachly-mcp — Progress Tracker
+
+> Lebendiges Dokument. Hält fest **was implementiert ist** und **was offen steht**.
+> Komplementär zu [STRATEGY.md](./STRATEGY.md) (das *Warum*) und
+> [VISION_10X.md](./VISION_10X.md) (das *Wohin*).
+>
+> **Stand:** v0.10.65 · 101 MCP-Tools · 425 Tests grün · 0 Lint-Warnings
+
+---
+
+## 0. Schnellüberblick
+
+| Dimension | Status |
+|---|---|
+| Version | `0.10.65` (npm `latest`) |
+| MCP-Tools | **101** |
+| Tests | **425** passing, 8 Suites |
+| Lint | **0 errors, 0 warnings** |
+| Build | sauber (`tsc`, Entry `dist/src/index.js`) |
+| Bench | Precision@1 **+22.2 %**, MRR **+10.9 %**, nDCG@5 **+8.1 %** vs. BM25 |
+
+---
+
+## 1. Erledigt ✅
+
+### Phase 2 — Beweisbar bessere Recall-Qualität (der Moat-Beweis)
+
+| Feature | Datei | Status |
+|---|---|---|
+| Quality-aware Reranking (5 Faktoren) | `src/rerank.ts` | ✅ live in `smart_recall` |
+| Cachly-Bench (IR-Metriken, CI-Wächter) | `src/bench/`, `src/__tests__/rerank.test.ts` | ✅ |
+| Hybrides Retrieval (BM25 + Semantik) | `src/handlers/brain.ts` | ✅ eine gerankte Liste |
+| CKG-Traversierung als 3. Signal (Layer 3) | `src/handlers/brain.ts` | ✅ 🕸️-Badge |
+| Contradiction-Resolution persistent | `cachly:contradictions:{topic}` | ✅ TTL'd, auditierbar |
+| "Brain saved you here"-Banner | `src/handlers/brain.ts` | ✅ Zeitersparnis inline |
+| Knowledge-Governance (`team_confirm`) | `src/handlers/team.ts` | ✅ 🛡️/✔️ Badges, 5. Rerank-Faktor |
+
+**5-Faktor-Rerank-Signal:** outcome × confidence × proven-ness × severity × governance
+(review-level × endorsements), geklammert auf [0.5, 2.5].
+
+### Phase 3 — Team-/Org-weiter Wissensgraph (der uneinholbare Moat)
+
+| Feature | Tool / Datei | Version | Status |
+|---|---|---|---|
+| Ambient Team-Propagation (author → team) | `learn_from_attempts` | 0.10.57 | ✅ default, nicht opt-in |
+| **Person-Nodes** im CKG | `ckgUpsertPersonNode`, `src/ckg.ts` | 0.10.59 | ✅ auto aus `author` |
+| **File-Nodes** im CKG | `ckgUpsertFileNode`, `src/ckg.ts` | 0.10.59 | ✅ auto aus `file_paths` |
+| `brain_who_knows(topic)` | `src/handlers/brain.ts` | 0.10.59 | ✅ Experten-Ranking 🥇🥈🥉 |
+| Author-Badge in `smart_recall` | `src/handlers/brain.ts` | 0.10.59 | ✅ 👤 inline |
+| `brain_file_map(file_paths)` | `src/handlers/brain.ts` | 0.10.60 | ✅ Experten + Lessons je Datei |
+| `team_expertise_map()` | `src/handlers/brain.ts` | 0.10.60 | ✅ Skills-Matrix als Tabelle |
+| **Visibility-Scopes** (`private`/`team`/`public`) | `learn_from_attempts` | 0.10.60 | ✅ private leakt nie in smart_recall |
+| `brain_from_git` → Person+File-Nodes | `src/handlers/fedbrain.ts` | 0.10.61 | ✅ Zero-Setup Graph retroaktiv |
+| `skill_gaps()` | `src/handlers/brain.ts` | 0.10.61 | ✅ 🔴/🟡/🔵 Blind-Spot-Report |
+| `brain_coverage()` | `src/handlers/brain.ts` | 0.10.61 | ✅ 0–100 Health-Score |
+
+### Stabilität & Hygiene
+
+| Thema | Status |
+|---|---|
+| Broken npm Entry-Point (0.10.50–52) | ✅ gefixt (`dist/src/index.js`) |
+| Version-Regression (52 < 56) | ✅ auf 0.10.57+ vorgezogen |
+| Lint: 70 Warnings → 0 | ✅ systematisch bereinigt |
+| Tool-Count-Konsistenz (80/89/95/96/98/100) | ✅ überall synchron |
+| Input-Guards für Phase-3-Tools | ✅ 0.10.62 (kein Crash bei leerem topic / non-array) |
+| `smart_recall` apiFetch null-guard | ✅ graceful degradation offline |
+| Scan-Timeout + Cap (`scanKeys`) überall | ✅ 0.10.63 — kein Hang bei großem Keyspace |
+| Git-Rename-Pfad-Normalisierung | ✅ 0.10.63 (`normalizeGitPath`) |
+| `withTimeout`-Utility (graceful degradation) | ✅ 0.10.63 |
+
+### Die drei Metriken — instrumentiert (0.10.64)
+
+| Metrik | Mechanismus | Status |
+|---|---|---|
+| **Time-to-first-recall** | `born_at` (erster Learn) → `first_recall_at` (erster proven Recall) | ✅ gemessen |
+| **Recall-Lift** | Cachly-Bench Headline, CI-verteidigt | ✅ +22.2 % P@1 |
+| **Team-Knowledge-Reuse** | Cross-Author-Recalls + distinct reuse-pairs | ✅ getrackt + inline sichtbar |
+| `brain_metrics()`-Tool | zeigt alle drei in einer Ansicht | ✅ 101. Tool |
+
+---
+
+## 2. Offen 🔲
+
+### Beweis (Phase-2-Rest) — der Glaubwürdigkeits-Beweis
+
+- ✅ **Flat-File Head-to-Head im Bench** (0.10.65) — naiver, quality-blinder Ranker
+  als ehrlicher Stellvertreter für "LLM liest Memory-Files". Cachly gewinnt auf den
+  entscheidenden Metriken: **P@1 +10.0 %, MRR +4.4 % vs. flat-file** · CI-verteidigt.
+- ✅ BENCH.md "Limitations" offen + ehrlich (warum flat-file bei P@3/Recall@3 vorn liegt).
+- 🔲 **Extern gelabelter Korpus** statt selbstgebautem Bench-Set (Dritt-Labels)
+- 🔲 Head-to-head auf **realen Agent-Traces** (nicht nur Fixture-Korpus)
+
+### Team-Graph (Phase-3-Rest)
+
+- 🔲 **Rollen-Modell** (admin / reviewer / contributor / viewer)
+- 🔲 **Sichtbarkeits-Scopes auf Team-Ebene** (heute nur lesson-level `private`)
+- 🔲 Service-/System-Nodes im Graph (heute: Konzept, Person, File)
+- 🔲 `brain_who_knows` über CKG-Kanten *zwischen* Personen (Kollaborations-Graph)
+- 🔲 Recall, das *wer-was-wo* gewichtet (personalisiertes Ranking)
+
+### Onboarding / Null-Reibung (Phase-1-Eintrittskarte)
+
+- 🔲 `npx @cachly-dev/init` Ein-Befehl-Setup <60 s, idempotent
+- 🟡 Circuit-Breaker / Timeouts: Scan-Tools abgesichert (`scanKeys`/`withTimeout`,
+  0.10.63); verbleibende Netzwerk-Calls (apiFetch) noch nicht überall timeout-gewrappt
+- 🔲 Self-Healing-Auth (kein "0 Recalls weil RAM-only")
+- ✅ Time-to-first-recall **messen** (0.10.64); auf <2 min *drücken* bleibt Onboarding-Arbeit 🔲
+
+### Enterprise / Reichweite (Phase-3/4)
+
+- 🔲 Self-Hosted-Tier mit SOC-2/ISO, BYOK by default
+- 🔲 Öffentliche/teilbare Brains (Domänen-Wissensbasen als Marktplatz)
+- 🔲 First-class-Support-Matrix je MCP-Client dokumentiert
+
+### Bekannte kleine Schulden
+
+- ✅ `brain_from_git`: Git-Rename-Pfade normalisiert (`normalizeGitPath`, 0.10.63).
+- ✅ README Tool-Count-Drift behoben (jetzt 101, überall synchron).
+- ✅ Scan-basierte Tools gegen großen Keyspace gehärtet (`scanKeys`, 0.10.63).
+- 🔲 `team_expertise_map`: Tabellen-Rendering bei *sehr* vielen Domains noch
+  nicht mit großem Datensatz lasttestbar (MockRedis deckt Funktionalität ab).
+
+---
+
+## 3. Die drei Metriken, die alles entscheiden (aus STRATEGY.md §6)
+
+| Metrik | Ziel | Heute |
+|---|---|---|
+| **Time-to-first-recall** | <2 min | ✅ gemessen via `brain_metrics` (born_at → first_recall_at) |
+| **Recall-Lift** (vs. BM25) | messbar >0, dann skalieren | ✅ +22.2 % P@1 (intern), in `brain_metrics` |
+| **Team-Knowledge-Reuse** | Lesson von A wird von B recallt | ✅ getrackt + inline + in `brain_metrics`; externes Dashboard 🔲 |
+
+> Alle drei sind jetzt in **einem Tool** sichtbar: `brain_metrics(instance_id)`.
+> Offen bleibt nur das *externe* Dashboard (server-side) und der *externe* Bench-Beweis (W1).
+
+---
+
+## 4. Changelog-Verweis
+
+Vollständige Release-Notes pro Version: [CHANGELOG.md](./CHANGELOG.md).
+Letzte Releases: 0.10.59 (Phase 3A) · 0.10.60 (Phase 3B) · 0.10.61 (Phase 3C, 100 Tools) · 0.10.62 (Härtung).