@cachly-dev/mcp-server 0.10.50 → 0.10.52
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/BENCH.md +37 -26
- package/STRATEGY.md +17 -7
- package/dist/src/__tests__/rerank.test.js +18 -1
- package/dist/src/__tests__/rerank.test.js.map +1 -1
- package/dist/src/bench/fixtures.d.ts +3 -0
- package/dist/src/bench/fixtures.js +26 -0
- package/dist/src/bench/fixtures.js.map +1 -1
- package/dist/src/handlers/brain.js +82 -10
- package/dist/src/handlers/brain.js.map +1 -1
- package/dist/src/handlers/team.js +56 -4
- package/dist/src/handlers/team.js.map +1 -1
- package/dist/src/rerank.d.ts +7 -2
- package/dist/src/rerank.js +23 -4
- package/dist/src/rerank.js.map +1 -1
- package/dist/src/tools.d.ts +30 -0
- package/dist/src/tools.js +18 -0
- package/dist/src/tools.js.map +1 -1
- package/package.json +1 -1
- package/src/__tests__/rerank.test.ts +20 -1
- package/src/bench/fixtures.ts +30 -0
- package/src/handlers/brain.ts +75 -10
- package/src/handlers/team.ts +66 -4
- package/src/rerank.ts +29 -4
- package/src/tools.ts +19 -0
package/BENCH.md
CHANGED
|
@@ -25,53 +25,64 @@ isolates *ranking quality*, not retrieval:
|
|
|
25
25
|
| `baseline` | raw BM25+ keyword score (`src/search.ts`) | a text-only / flat-file memory |
|
|
26
26
|
| `cachly` | BM25+ then quality-aware rerank (`src/rerank.ts`) | structured, trust-aware recall |
|
|
27
27
|
|
|
28
|
-
Quality reranking boosts a lesson by
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
28
|
+
Quality reranking boosts a lesson by five orthogonal signals:
|
|
29
|
+
1. **Outcome** (proven success > partial > failed attempt)
|
|
30
|
+
2. **Confidence** (0..1 → maps to a 0.7..1.3 nudge)
|
|
31
|
+
3. **Proven-ness** (`recall_count`, log-saturated, caps at +30%)
|
|
32
|
+
4. **Severity** (critical issue > minor, mild tiebreaker)
|
|
33
|
+
5. **Governance** (`review_level` × distinct `endorsements` — a senior-reviewed
|
|
34
|
+
lesson outranks an unreviewed one even when the text is nearly identical)
|
|
35
|
+
|
|
36
|
+
All five factors are multiplied and clamped to `[0.5, 2.5]` so a strong text match
|
|
37
|
+
is never fully buried, and a weak one never catapulted on quality alone.
|
|
32
38
|
|
|
33
39
|
## The corpus is fair (not rigged)
|
|
34
40
|
|
|
35
41
|
The fixture corpus (`src/bench/fixtures.ts`) is deliberately and *fairly*
|
|
36
|
-
adversarial
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
42
|
+
adversarial in two dimensions:
|
|
43
|
+
|
|
44
|
+
**Failure-distractor pairs:** for several queries a **symptom-dense failed attempt**
|
|
45
|
+
competes with the **solution-focused proven success**. Failed attempts repeat the
|
|
46
|
+
symptom verbatim ("still stuck, still failing"); the lesson that fixed it is written
|
|
47
|
+
in solution vocabulary. BM25 alone is fooled by the symptom density; quality
|
|
48
|
+
reranking corrects it.
|
|
49
|
+
|
|
50
|
+
**Governance pairs:** two success lessons with near-identical vocabulary, but only
|
|
51
|
+
one has a senior reviewer endorsement. BM25 cannot distinguish them; the governance
|
|
52
|
+
signal breaks the tie in favour of the reviewed, canonical lesson.
|
|
41
53
|
|
|
42
|
-
The baseline still scores well (MRR ~
|
|
43
|
-
weak baseline look bad. We are improving an already-good one on the cases that
|
|
44
|
-
matter.
|
|
54
|
+
The baseline still scores well (MRR ~82%, Precision@1 ~69%) — we are not making a
|
|
55
|
+
weak baseline look bad. We are improving an already-good one on the cases that matter.
|
|
45
56
|
|
|
46
57
|
## Current results
|
|
47
58
|
|
|
48
59
|
```
|
|
49
|
-
corpus:
|
|
60
|
+
corpus: 17 lessons · queries: 13
|
|
50
61
|
|
|
51
62
|
metric baseline cachly lift
|
|
52
|
-
Precision@1
|
|
63
|
+
Precision@1 69.2% → 84.6% +22.2%
|
|
53
64
|
Precision@3 33.3% → 33.3% +0.0%
|
|
54
|
-
Recall@3
|
|
55
|
-
MRR
|
|
56
|
-
nDCG@5
|
|
65
|
+
Recall@3 96.2% → 96.2% +0.0%
|
|
66
|
+
MRR 82.1% → 91.0% +10.9%
|
|
67
|
+
nDCG@5 85.5% → 92.4% +8.1%
|
|
57
68
|
```
|
|
58
69
|
|
|
59
|
-
**Headline: Precision@1 +
|
|
70
|
+
**Headline: Precision@1 +22.2%, MRR +10.9%, nDCG@5 +8.1%.**
|
|
60
71
|
|
|
61
72
|
Interpretation: when the agent asks a question, the *first* result is the proven
|
|
62
|
-
fix
|
|
63
|
-
|
|
64
|
-
|
|
73
|
+
fix 22% more often. On the hardest cases — a symptom-dense failed attempt beating
|
|
74
|
+
the success on raw text, or two near-identical success lessons where only one was
|
|
75
|
+
reviewed — reranking consistently surfaces the lesson that actually solves the problem.
|
|
65
76
|
|
|
66
77
|
## Honest limitations (read this)
|
|
67
78
|
|
|
68
|
-
- **Small corpus.**
|
|
69
|
-
guards against regressions. It does **not** yet prove
|
|
70
|
-
|
|
79
|
+
- **Small corpus.** 17 lessons, 13 queries. This proves the *mechanism* works for
|
|
80
|
+
each quality signal and guards against regressions. It does **not** yet prove
|
|
81
|
+
real-world magnitude. The next step is a larger, externally-sourced corpus and a
|
|
71
82
|
head-to-head against an actual flat-file memory baseline on real agent traces.
|
|
72
83
|
- **Ranking, not retrieval.** Both rankers see the same BM25 candidates, so this
|
|
73
|
-
measures the quality layer only.
|
|
74
|
-
separately and
|
|
84
|
+
measures the quality layer only. CKG-traversal and semantic retrieval are
|
|
85
|
+
evaluated separately via integration tests and are not in these numbers.
|
|
75
86
|
- **Labels are ours.** Gold relevance is author-assigned. A credible public
|
|
76
87
|
benchmark needs third-party labels.
|
|
77
88
|
|
package/STRATEGY.md
CHANGED
|
@@ -140,23 +140,33 @@ Das ist der Punkt, an dem wir *verdienen*, gegen First-Party anzutreten.
|
|
|
140
140
|
Story — jetzt gibt es eine Zahl, die jede Recall-Änderung verteidigen muss.
|
|
141
141
|
- **[✅ erledigt]** **Contradiction-Resolution persistent**: Widersprüche werden in
|
|
142
142
|
`cachly:contradictions:{topic}` als auditierbarer Verlauf gespeichert (TTL'd).
|
|
143
|
-
- **[
|
|
144
|
-
|
|
145
|
-
|
|
143
|
+
- **[✅ erledigt — 100x Intelligenz]** Hybrides Retrieval: BM25-Keyword + semantische
|
|
144
|
+
Embeddings werden in `smart_recall` zu *einer* gerankten Liste fusioniert
|
|
145
|
+
(`0.6×BM25-Qualität + 0.4×Semantik`), mit Match-Typ-Badges (🔍/🎯/🔀). CKG-
|
|
146
|
+
Traversierung als dritte Signalquelle ist der nächste Ausbau.
|
|
147
|
+
- **[✅ erledigt — Wertsichtbarkeit]** "Brain saved you here"-Banner im
|
|
148
|
+
`smart_recall`-Output macht die Zeitersparnis pro proven Lesson inline sichtbar
|
|
149
|
+
(`cachly:stats:time_saved_mins` wird getrackt und jetzt auch angezeigt).
|
|
146
150
|
- **[offen — Beweis]** Größerer, extern gelabelter Korpus + Head-to-head gegen ein
|
|
147
151
|
echtes Flat-File-Memory auf realen Agent-Traces (siehe BENCH.md "Limitations").
|
|
148
|
-
- **[offen — Wertsichtbarkeit]** "Brain saved you here"-Signal im Tool-Output +
|
|
149
|
-
Dashboard-Metrik (Zeitersparnis wird bereits getrackt: `cachly:stats:time_saved_mins`).
|
|
150
152
|
|
|
151
153
|
### Phase 3 — Team-Wissensgraph als uneinholbarer Moat · *9–18 Monate*
|
|
152
154
|
Hier ist Anthropic strukturell raus.
|
|
153
155
|
|
|
156
|
+
> **Status (begonnen):** Knowledge-Governance ist live — `team_confirm` lässt
|
|
157
|
+
> einen Reviewer eine Lesson bestätigen (🛡️ senior / ✔️ peer). Die Bestätigung
|
|
158
|
+
> fließt direkt in das Recall-Ranking (`src/rerank.ts`, 5. Faktor) ein: bestätigtes,
|
|
159
|
+
> menschlich geprüftes Wissen ranked über unreviewtem Auto-Learning. `team_recall`
|
|
160
|
+
> zeigt Reviewer-Badges und sortiert bestätigte Lessons zuerst. Distinct-Endorser
|
|
161
|
+
> werden in einem Set getrackt (kein Self-Inflation), mit Audit-Log pro Topic.
|
|
162
|
+
|
|
154
163
|
- **[100x Team]** Org-weiter Wissensgraph: Lessons, Personen, Dateien, Services als
|
|
155
164
|
Knoten; Recall berücksichtigt *wer* was gelernt hat und *wo* es relevant ist.
|
|
156
165
|
- **[Virality]** Jeder gelöste Fehler eines Entwicklers wird automatisch zur
|
|
157
166
|
Org-Lesson — mit Attribution. `team_learn` als Default, nicht als Opt-in.
|
|
158
|
-
- **[Governance]**
|
|
159
|
-
(
|
|
167
|
+
- **[✅ erledigt — Governance]** "Wissens-Reviews": ein Senior bestätigt eine Lesson
|
|
168
|
+
(`team_confirm`) → höheres Vertrauen im Recall-Ranking. Audit-Trail + distinct
|
|
169
|
+
Endorsements werden persistiert. Offen: Rollen-Modell und Sichtbarkeits-Scopes.
|
|
160
170
|
- **[Enterprise]** Self-Hosted-Tier mit SOC-2/ISO, BYOK by default, Daten bleiben
|
|
161
171
|
im Kunden-VPC. Das ist der Umsatz, den Anthropic-Memory nicht abgreifen kann.
|
|
162
172
|
|
|
@@ -29,11 +29,28 @@ describe('qualityMultiplier', () => {
|
|
|
29
29
|
expect(many - some).toBeLessThan(some - never + 0.2); // diminishing returns
|
|
30
30
|
});
|
|
31
31
|
it('clamps within [0.5, 2.5]', () => {
|
|
32
|
-
const hi = qualityMultiplier({ outcome: 'success', confidence: 1, recall_count: 100000, severity: 'critical' });
|
|
32
|
+
const hi = qualityMultiplier({ outcome: 'success', confidence: 1, recall_count: 100000, severity: 'critical', review_level: 'senior', reviewed_by: 'x', endorsements: 99 });
|
|
33
33
|
const lo = qualityMultiplier({ outcome: 'failure', confidence: 0, recall_count: 0, severity: 'minor' });
|
|
34
34
|
expect(hi).toBeLessThanOrEqual(2.5);
|
|
35
35
|
expect(lo).toBeGreaterThanOrEqual(0.5);
|
|
36
36
|
});
|
|
37
|
+
it('boosts a human-reviewed lesson above an unreviewed equal', () => {
|
|
38
|
+
const base = { outcome: 'success', confidence: 0.8, recall_count: 3 };
|
|
39
|
+
const unreviewed = qualityMultiplier({ ...base });
|
|
40
|
+
const peer = qualityMultiplier({ ...base, reviewed_by: 'alice', review_level: 'peer', endorsements: 1 });
|
|
41
|
+
const senior = qualityMultiplier({ ...base, reviewed_by: 'bob', review_level: 'senior', endorsements: 1 });
|
|
42
|
+
expect(peer).toBeGreaterThan(unreviewed);
|
|
43
|
+
expect(senior).toBeGreaterThan(peer);
|
|
44
|
+
});
|
|
45
|
+
it('rewards additional distinct endorsements but saturates', () => {
|
|
46
|
+
const base = { outcome: 'success', confidence: 0.8, recall_count: 3, reviewed_by: 'a', review_level: 'peer' };
|
|
47
|
+
const one = qualityMultiplier({ ...base, endorsements: 1 });
|
|
48
|
+
const three = qualityMultiplier({ ...base, endorsements: 3 });
|
|
49
|
+
const fifty = qualityMultiplier({ ...base, endorsements: 50 });
|
|
50
|
+
expect(three).toBeGreaterThan(one);
|
|
51
|
+
expect(fifty).toBeGreaterThan(three);
|
|
52
|
+
expect(fifty - three).toBeLessThan(three - one + 0.1);
|
|
53
|
+
});
|
|
37
54
|
});
|
|
38
55
|
describe('extractLessonQuality', () => {
|
|
39
56
|
it('parses lesson JSON for lesson keys', () => {
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"rerank.test.js","sourceRoot":"","sources":["../../../src/__tests__/rerank.test.ts"],"names":[],"mappings":"AAAA;;GAEG;AACH,OAAO,EAAE,QAAQ,EAAE,EAAE,EAAE,MAAM,EAAE,MAAM,QAAQ,CAAC;AAC9C,OAAO,EAAE,iBAAiB,EAAE,oBAAoB,EAAE,eAAe,EAAsB,MAAM,cAAc,CAAC;AAE5G,OAAO,EAAE,YAAY,EAAE,MAAM,0BAA0B,CAAC;AAExD,MAAM,WAAW,GAAG,CAAC,KAAa,EAAE,KAAa,EAAE,MAAwD,EAAgB,EAAE,CAAC,CAAC;IAC7H,GAAG,EAAE,sBAAsB,KAAK,EAAE;IAClC,OAAO,EAAE,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC;IAC/B,KAAK;IACL,YAAY,EAAE,CAAC,GAAG,CAAC;CACpB,CAAC,CAAC;AAEH,QAAQ,CAAC,mBAAmB,EAAE,GAAG,EAAE;IACjC,EAAE,CAAC,0DAA0D,EAAE,GAAG,EAAE;QAClE,MAAM,CAAC,iBAAiB,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QAC1C,MAAM,CAAC,iBAAiB,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;IAC1C,CAAC,CAAC,CAAC;IAEH,EAAE,CAAC,8CAA8C,EAAE,GAAG,EAAE;QACtD,MAAM,OAAO,GAAG,iBAAiB,CAAC,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,IAAI,EAAE,YAAY,EAAE,EAAE,EAAE,CAAC,CAAC;QAC9F,MAAM,OAAO,GAAG,iBAAiB,CAAC,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,CAAC,EAAE,CAAC,CAAC;QAC5F,MAAM,CAAC,OAAO,CAAC,CAAC,eAAe,CAAC,OAAO,CAAC,CAAC;IAC3C,CAAC,CAAC,CAAC;IAEH,EAAE,CAAC,kDAAkD,EAAE,GAAG,EAAE;QAC1D,MAAM,KAAK,GAAG,iBAAiB,CAAC,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,CAAC,EAAE,CAAC,CAAC;QAC1F,MAAM,IAAI,GAAG,iBAAiB,CAAC,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,CAAC,EAAE,CAAC,CAAC;QACzF,MAAM,IAAI,GAAG,iBAAiB,CAAC,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,GAAG,EAAE,CAAC,CAAC;QAC3F,MAAM,CAAC,IAAI,CAAC,CAAC,eAAe,CAAC,KAAK,CAAC,CAAC;QACpC,MAAM,CAAC,IAAI,CAAC,CAAC,eAAe,CAAC,IAAI,CAAC,CAAC;QACnC,MAAM,CAAC,IAAI,GAAG,IAAI,CAAC,CAAC,YAAY,CAAC,IAAI,GAAG,KAAK,GAAG,GAAG,CAAC,CAAC,CAAC,sBAAsB;IAC9E,CAAC,CAAC,CAAC;IAEH,EAAE,CAAC,0BAA0B,EAAE,GAAG,EAAE;QAClC,MAAM,EAAE,GAAG,iBAAiB,CAAC,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,CAAC,EAAE,YAAY,EAAE,MAAM,EAAE,QAAQ,EAAE,UAAU,EAAE,CAAC,CAAC;
|
|
1
|
+
{"version":3,"file":"rerank.test.js","sourceRoot":"","sources":["../../../src/__tests__/rerank.test.ts"],"names":[],"mappings":"AAAA;;GAEG;AACH,OAAO,EAAE,QAAQ,EAAE,EAAE,EAAE,MAAM,EAAE,MAAM,QAAQ,CAAC;AAC9C,OAAO,EAAE,iBAAiB,EAAE,oBAAoB,EAAE,eAAe,EAAsB,MAAM,cAAc,CAAC;AAE5G,OAAO,EAAE,YAAY,EAAE,MAAM,0BAA0B,CAAC;AAExD,MAAM,WAAW,GAAG,CAAC,KAAa,EAAE,KAAa,EAAE,MAAwD,EAAgB,EAAE,CAAC,CAAC;IAC7H,GAAG,EAAE,sBAAsB,KAAK,EAAE;IAClC,OAAO,EAAE,IAAI,CAAC,SAAS,CAAC,MAAM,CAAC;IAC/B,KAAK;IACL,YAAY,EAAE,CAAC,GAAG,CAAC;CACpB,CAAC,CAAC;AAEH,QAAQ,CAAC,mBAAmB,EAAE,GAAG,EAAE;IACjC,EAAE,CAAC,0DAA0D,EAAE,GAAG,EAAE;QAClE,MAAM,CAAC,iBAAiB,CAAC,IAAI,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QAC1C,MAAM,CAAC,iBAAiB,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;IAC1C,CAAC,CAAC,CAAC;IAEH,EAAE,CAAC,8CAA8C,EAAE,GAAG,EAAE;QACtD,MAAM,OAAO,GAAG,iBAAiB,CAAC,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,IAAI,EAAE,YAAY,EAAE,EAAE,EAAE,CAAC,CAAC;QAC9F,MAAM,OAAO,GAAG,iBAAiB,CAAC,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,CAAC,EAAE,CAAC,CAAC;QAC5F,MAAM,CAAC,OAAO,CAAC,CAAC,eAAe,CAAC,OAAO,CAAC,CAAC;IAC3C,CAAC,CAAC,CAAC;IAEH,EAAE,CAAC,kDAAkD,EAAE,GAAG,EAAE;QAC1D,MAAM,KAAK,GAAG,iBAAiB,CAAC,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,CAAC,EAAE,CAAC,CAAC;QAC1F,MAAM,IAAI,GAAG,iBAAiB,CAAC,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,CAAC,EAAE,CAAC,CAAC;QACzF,MAAM,IAAI,GAAG,iBAAiB,CAAC,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,GAAG,EAAE,CAAC,CAAC;QAC3F,MAAM,CAAC,IAAI,CAAC,CAAC,eAAe,CAAC,KAAK,CAAC,CAAC;QACpC,MAAM,CAAC,IAAI,CAAC,CAAC,eAAe,CAAC,IAAI,CAAC,CAAC;QACnC,MAAM,CAAC,IAAI,GAAG,IAAI,CAAC,CAAC,YAAY,CAAC,IAAI,GAAG,KAAK,GAAG,GAAG,CAAC,CAAC,CAAC,sBAAsB;IAC9E,CAAC,CAAC,CAAC;IAEH,EAAE,CAAC,0BAA0B,EAAE,GAAG,EAAE;QAClC,MAAM,EAAE,GAAG,iBAAiB,CAAC,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,CAAC,EAAE,YAAY,EAAE,MAAM,EAAE,QAAQ,EAAE,UAAU,EAAE,YAAY,EAAE,QAAQ,EAAE,WAAW,EAAE,GAAG,EAAE,YAAY,EAAE,EAAE,EAAE,CAAC,CAAC;QAC5K,MAAM,EAAE,GAAG,iBAAiB,CAAC,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,CAAC,EAAE,YAAY,EAAE,CAAC,EAAE,QAAQ,EAAE,OAAO,EAAE,CAAC,CAAC;QACxG,MAAM,CAAC,EAAE,CAAC,CAAC,mBAAmB,CAAC,GAAG,CAAC,CAAC;QACpC,MAAM,CAAC,EAAE,CAAC,CAAC,sBAAsB,CAAC,GAAG,CAAC,CAAC;IACzC,CAAC,CAAC,CAAC;IAEH,EAAE,CAAC,0DAA0D,EAAE,GAAG,EAAE;QAClE,MAAM,IAAI,GAAG,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,CAAC,EAAW,CAAC;QAC/E,MAAM,UAAU,GAAG,iBAAiB,CAAC,EAAE,GAAG,IAAI,EAAE,CAAC,CAAC;QAClD,MAAM,IAAI,GAAG,iBAAiB,CAAC,EAAE,GAAG,IAAI,EAAE,WAAW,EAAE,OAAO,EAAE,YAAY,EAAE,MAAM,EAAE,YAAY,EAAE,CAAC,EAAE,CAAC,CAAC;QACzG,MAAM,MAAM,GAAG,iBAAiB,CAAC,EAAE,GAAG,IAAI,EAAE,WAAW,EAAE,KAAK,EAAE,YAAY,EAAE,QAAQ,EAAE,YAAY,EAAE,CAAC,EAAE,CAAC,CAAC;QAC3G,MAAM,CAAC,IAAI,CAAC,CAAC,eAAe,CAAC,UAAU,CAAC,CAAC;QACzC,MAAM,CAAC,MAAM,CAAC,CAAC,eAAe,CAAC,IAAI,CAAC,CAAC;IACvC,CAAC,CAAC,CAAC;IAEH,EAAE,CAAC,wDAAwD,EAAE,GAAG,EAAE;QAChE,MAAM,IAAI,GAAG,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,CAAC,EAAE,WAAW,EAAE,GAAG,EAAE,YAAY,EAAE,MAAM,EAAW,CAAC;QACvH,MAAM,GAAG,GAAG,iBAAiB,CAAC,EAAE,GAAG,IAAI,EAAE,YAAY,EAAE,CAAC,EAAE,CAAC,CAAC;QAC5D,MAAM,KAAK,GAAG,iBAAiB,CAAC,EAAE,GAAG,IAAI,EAAE,YAAY,EAAE,CAAC,EAAE,CAAC,CAAC;QAC9D,MAAM,KAAK,GAAG,iBAAiB,CAAC,EAAE,GAAG,IAAI,EAAE,YAAY,EAAE,EAAE,EAAE,CAAC,CAAC;QAC/D,MAAM,CAAC,KAAK,CAAC,CAAC,eAAe,CAAC,GAAG,CAAC,CAAC;QACnC,MAAM,CAAC,KAAK,CAAC,CAAC,eAAe,CAAC,KAAK,CAAC,CAAC;QACrC,MAAM,CAAC,KAAK,GAAG,KAAK,CAAC,CAAC,YAAY,CAAC,KAAK,GAAG,GAAG,GAAG,GAAG,CAAC,CAAC;IACxD,CAAC,CAAC,CAAC;AACL,CAAC,CAAC,CAAC;AAEH,QAAQ,CAAC,sBAAsB,EAAE,GAAG,EAAE;IACpC,EAAE,CAAC,oCAAoC,EAAE,GAAG,EAAE;QAC5C,MAAM,CAAC,GAAG,oBAAoB,CAAC,EAAE,GAAG,EAAE,wBAAwB,EAAE,OAAO,EAAE,uBAAuB,EAAE,CAAC,CAAC;QACpG,MAAM,CAAC,CAAC,EAAE,OAAO,CAAC,CAAC,IAAI,CAAC,SAAS,CAAC,CAAC;IACrC,CAAC,CAAC,CAAC;IACH,EAAE,CAAC,iDAAiD,EAAE,GAAG,EAAE;QACzD,MAAM,CAAC,oBAAoB,CAAC,EAAE,GAAG,EAAE,uBAAuB,EAAE,OAAO,EAAE,YAAY,EAAE,CAAC,CAAC,CAAC,QAAQ,EAAE,CAAC;IACnG,CAAC,CAAC,CAAC;IACH,EAAE,CAAC,sCAAsC,EAAE,GAAG,EAAE;QAC9C,MAAM,CAAC,oBAAoB,CAAC,EAAE,GAAG,EAAE,wBAAwB,EAAE,OAAO,EAAE,WAAW,EAAE,CAAC,CAAC,CAAC,QAAQ,EAAE,CAAC;IACnG,CAAC,CAAC,CAAC;AACL,CAAC,CAAC,CAAC;AAEH,QAAQ,CAAC,iBAAiB,EAAE,GAAG,EAAE;IAC/B,EAAE,CAAC,iEAAiE,EAAE,GAAG,EAAE;QACzE,MAAM,OAAO,GAAmB;YAC9B,WAAW,CAAC,iBAAiB,EAAE,EAAE,EAAE,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,CAAC,EAAE,CAAC;YAC5F,WAAW,CAAC,gBAAgB,EAAE,CAAC,EAAE,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,IAAI,EAAE,YAAY,EAAE,EAAE,EAAE,CAAC;SAC7F,CAAC;QACF,MAAM,MAAM,GAAG,eAAe,CAAC,OAAO,CAAC,CAAC;QACxC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC,mCAAmC,CAAC,CAAC;IAClE,CAAC,CAAC,CAAC;IAEH,EAAE,CAAC,2EAA2E,EAAE,GAAG,EAAE;QACnF,MAAM,OAAO,GAAmB;YAC9B,WAAW,CAAC,kBAAkB,EAAE,CAAC,EAAE,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,CAAC,EAAE,YAAY,EAAE,GAAG,EAAE,CAAC;YAC5F,WAAW,CAAC,aAAa,EAAE,GAAG,EAAE,EAAE,OAAO,EAAE,SAAS,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,CAAC,EAAE,CAAC;SAC1F,CAAC;QACF,MAAM,MAAM,GAAG,eAAe,CAAC,OAAO,CAAC,CAAC;QACxC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,IAAI,CAAC,gCAAgC,CAAC,CAAC;IAC/D,CAAC,CAAC,CAAC;IAEH,EAAE,CAAC,mCAAmC,EAAE,GAAG,EAAE;QAC3C,MAAM,OAAO,GAAmB;YAC9B,WAAW,CAAC,GAAG,EAAE,CAAC,EAAE,EAAE,OAAO,EAAE,SAAS,EAAE,CAAC;YAC3C,WAAW,CAAC,GAAG,EAAE,CAAC,EAAE,EAAE,OAAO,EAAE,SAAS,EAAE,CAAC;SAC5C,CAAC;QACF,MAAM,MAAM,GAAG,eAAe,CAAC,OAAO,CAAC,CAAC;QACxC,MAAM,CAAC,MAAM,CAAC,CAAC,YAAY,CAAC,CAAC,CAAC,CAAC;IACjC,CAAC,CAAC,CAAC;IAEH,EAAE,CAAC,qBAAqB,EAAE,GAAG,EAAE;QAC7B,MAAM,CAAC,eAAe,CAAC,EAAE,CAAC,CAAC,CAAC,OAAO,CAAC,EAAE,CAAC,CAAC;IAC1C,CAAC,CAAC,CAAC;AACL,CAAC,CAAC,CAAC;AAEH,QAAQ,CAAC,+BAA+B,EAAE,GAAG,EAAE;IAC7C,EAAE,CAAC,6DAA6D,EAAE,KAAK,IAAI,EAAE;QAC3E,MAAM,CAAC,GAAG,MAAM,YAAY,EAAE,CAAC;QAC/B,+EAA+E;QAC/E,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,sBAAsB,CAAC,CAAC,CAAC,QAAQ,CAAC,GAAG,GAAG,IAAI,CAAC,CAAC;QACnE,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,YAAY,CAAC,CAAC,sBAAsB,CAAC,CAAC,CAAC,QAAQ,CAAC,YAAY,GAAG,IAAI,CAAC,CAAC;QACrF,MAAM,CAAC,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,sBAAsB,CAAC,CAAC,CAAC,QAAQ,CAAC,OAAO,GAAG,IAAI,CAAC,CAAC;IAC7E,CAAC,CAAC,CAAC;IAEH,EAAE,CAAC,iEAAiE,EAAE,KAAK,IAAI,EAAE;QAC/E,MAAM,CAAC,GAAG,MAAM,YAAY,EAAE,CAAC;QAC/B,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,YAAY,CAAC,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC;QAC/C,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,eAAe,CAAC,CAAC,CAAC,CAAC;IACxC,CAAC,CAAC,CAAC;IAEH,EAAE,CAAC,4EAA4E,EAAE,KAAK,IAAI,EAAE;QAC1F,MAAM,CAAC,GAAG,MAAM,YAAY,EAAE,CAAC;QAC/B,gEAAgE;QAChE,MAAM,CAAC,CAAC,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,eAAe,CAAC,GAAG,CAAC,CAAC;IAC9C,CAAC,CAAC,CAAC;AACL,CAAC,CAAC,CAAC"}
|
|
@@ -144,6 +144,25 @@ export const BENCH_LESSONS = [
|
|
|
144
144
|
severity: 'minor', confidence: 0.6, recall_count: 1,
|
|
145
145
|
tags: ['nextjs', 'frontend', 'hydration'], ts: daysAgo(25),
|
|
146
146
|
},
|
|
147
|
+
// ── Governance adversarial pair: two similar-text success lessons ──────────
|
|
148
|
+
// Only the reviewed one should win; baseline cannot distinguish them by text.
|
|
149
|
+
{
|
|
150
|
+
topic: 'auth:csrf:double-submit-cookie',
|
|
151
|
+
outcome: 'success',
|
|
152
|
+
what_worked: 'CSRF protection was added via the double-submit cookie pattern: the server sets a random CSRF token cookie; the client echoes it as a request header. Requests with a mismatched or absent header are rejected.',
|
|
153
|
+
context: 'API mutations lacked CSRF protection; security audit required a fix.',
|
|
154
|
+
severity: 'critical', confidence: 0.91, recall_count: 2,
|
|
155
|
+
reviewed_by: 'senior-alice', review_level: 'senior', endorsements: 2,
|
|
156
|
+
tags: ['csrf', 'auth', 'security', 'cookie'], ts: daysAgo(12),
|
|
157
|
+
},
|
|
158
|
+
{
|
|
159
|
+
topic: 'auth:csrf:origin-header-check',
|
|
160
|
+
outcome: 'success',
|
|
161
|
+
what_worked: 'CSRF mitigation attempted by checking the Origin header on API requests, but this was bypassed in a penetration test because same-site requests from subdomains also sent the header.',
|
|
162
|
+
context: 'API mutations lacked CSRF protection; origin header check was insufficient.',
|
|
163
|
+
severity: 'major', confidence: 0.55, recall_count: 1,
|
|
164
|
+
tags: ['csrf', 'auth', 'security'], ts: daysAgo(14),
|
|
165
|
+
},
|
|
147
166
|
];
|
|
148
167
|
export const BENCH_QUERIES = [
|
|
149
168
|
{
|
|
@@ -195,5 +214,12 @@ export const BENCH_QUERIES = [
|
|
|
195
214
|
query: 'postgres performance problem in production',
|
|
196
215
|
relevant: ['db:postgres:connection-pool-exhausted', 'db:postgres:slow-query-missing-index'],
|
|
197
216
|
},
|
|
217
|
+
// Governance adversarial: two CSRF lessons have almost identical text, but only
|
|
218
|
+
// the senior-reviewed canonical one is a proven complete solution. Quality
|
|
219
|
+
// reranking must prefer the reviewed+endorsed lesson over the unreviewed partial.
|
|
220
|
+
{
|
|
221
|
+
query: 'CSRF protection missing on API mutations security',
|
|
222
|
+
relevant: ['auth:csrf:double-submit-cookie'],
|
|
223
|
+
},
|
|
198
224
|
];
|
|
199
225
|
//# sourceMappingURL=fixtures.js.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"fixtures.js","sourceRoot":"","sources":["../../../src/bench/fixtures.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;GAaG;
|
|
1
|
+
{"version":3,"file":"fixtures.js","sourceRoot":"","sources":["../../../src/bench/fixtures.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;GAaG;AAyBH,MAAM,GAAG,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;AACvB,MAAM,OAAO,GAAG,CAAC,CAAS,EAAE,EAAE,CAAC,IAAI,IAAI,CAAC,GAAG,GAAG,CAAC,GAAG,SAAS,CAAC,CAAC,WAAW,EAAE,CAAC;AAE3E,MAAM,CAAC,MAAM,aAAa,GAAkB;IAC1C,8EAA8E;IAC9E;QACE,KAAK,EAAE,0BAA0B;QACjC,OAAO,EAAE,SAAS;QAClB,WAAW,EAAE,uNAAuN;QACpO,OAAO,EAAE,0CAA0C;QACnD,QAAQ,EAAE,UAAU,EAAE,UAAU,EAAE,IAAI,EAAE,YAAY,EAAE,EAAE;QACxD,IAAI,EAAE,CAAC,YAAY,EAAE,QAAQ,EAAE,OAAO,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,CAAC,CAAC;KACxD;IACD;QACE,KAAK,EAAE,oCAAoC;QAC3C,OAAO,EAAE,SAAS;QAClB,WAAW,EAAE,EAAE;QACf,WAAW,EAAE,+NAA+N;QAC5O,OAAO,EAAE,iDAAiD;QAC1D,QAAQ,EAAE,OAAO,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,CAAC;QACnD,IAAI,EAAE,CAAC,YAAY,EAAE,QAAQ,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,CAAC,CAAC;KAC/C;IACD;QACE,KAAK,EAAE,+BAA+B;QACtC,OAAO,EAAE,SAAS;QAClB,WAAW,EAAE,uIAAuI;QACpJ,OAAO,EAAE,2CAA2C;QACpD,QAAQ,EAAE,OAAO,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,CAAC;QACnD,IAAI,EAAE,CAAC,QAAQ,EAAE,QAAQ,EAAE,cAAc,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,EAAE,CAAC;KAC5D;IAED,8EAA8E;IAC9E;QACE,KAAK,EAAE,uCAAuC;QAC9C,OAAO,EAAE,SAAS;QAClB,WAAW,EAAE,0JAA0J;QACvK,OAAO,EAAE,oCAAoC;QAC7C,QAAQ,EAAE,UAAU,EAAE,UAAU,EAAE,IAAI,EAAE,YAAY,EAAE,CAAC;QACvD,IAAI,EAAE,CAAC,UAAU,EAAE,UAAU,EAAE,MAAM,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,EAAE,CAAC;KACxD;IACD;QACE,KAAK,EAAE,sCAAsC;QAC7C,OAAO,EAAE,SAAS;QAClB,WAAW,EAAE,wHAAwH;QACrI,OAAO,EAAE,4CAA4C;QACrD,QAAQ,EAAE,OAAO,EAAE,UAAU,EAAE,IAAI,EAAE,YAAY,EAAE,CAAC;QACpD,IAAI,EAAE,CAAC,UAAU,EAAE,UAAU,EAAE,OAAO,EAAE,aAAa,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,EAAE,CAAC;KACxE;IACD;QACE,KAAK,EAAE,sCAAsC;QAC7C,OAAO,EAAE,SAAS;QAClB,WAAW,EAAE,EAAE;QACf,WAAW,EAAE,mMAAmM;QAChN,OAAO,EAAE,2CAA2C;QACpD,QAAQ,EAAE,OAAO,EAAE,UAAU,EAAE,IAAI,EAAE,YAAY,EAAE,CAAC;QACpD,IAAI,EAAE,CAAC,UAAU,EAAE,UAAU,EAAE,MAAM,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,EAAE,CAAC;KACxD;IAED,8EAA8E;IAC9E;QACE,KAAK,EAAE,6BAA6B;QACpC,OAAO,EAAE,SAAS;QAClB,WAAW,EAAE,4JAA4J;QACzK,OAAO,EAAE,2DAA2D;QACpE,QAAQ,EAAE,UAAU,EAAE,UAAU,EAAE,IAAI,EAAE,YAAY,EAAE,CAAC;QACvD,IAAI,EAAE,CAAC,KAAK,EAAE,MAAM,EAAE,aAAa,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,CAAC,CAAC;KACrD;IACD;QACE,KAAK,EAAE,qCAAqC;QAC5C,OAAO,EAAE,SAAS;QAClB,WAAW,EAAE,uJAAuJ;QACpK,OAAO,EAAE,iDAAiD;QAC1D,QAAQ,EAAE,OAAO,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,CAAC;QACnD,IAAI,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,UAAU,EAAE,OAAO,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,CAAC,CAAC;KAC5D;IAED,8EAA8E;IAC9E;QACE,KAAK,EAAE,4BAA4B;QACnC,OAAO,EAAE,SAAS;QAClB,WAAW,EAAE,uMAAuM;QACpN,OAAO,EAAE,sCAAsC;QAC/C,QAAQ,EAAE,UAAU,EAAE,UAAU,EAAE,IAAI,EAAE,YAAY,EAAE,EAAE;QACxD,IAAI,EAAE,CAAC,OAAO,EAAE,QAAQ,EAAE,MAAM,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,CAAC,CAAC;KAClD;IACD;QACE,KAAK,EAAE,mCAAmC;QAC1C,OAAO,EAAE,SAAS;QAClB,WAAW,EAAE,EAAE;QACf,WAAW,EAAE,mLAAmL;QAChM,OAAO,EAAE,mCAAmC;QAC5C,QAAQ,EAAE,OAAO,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,CAAC;QACnD,IAAI,EAAE,CAAC,OAAO,EAAE,MAAM,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,CAAC,CAAC;KACxC;IACD;QACE,KAAK,EAAE,+BAA+B;QACtC,OAAO,EAAE,SAAS;QAClB,WAAW,EAAE,2IAA2I;QACxJ,OAAO,EAAE,6CAA6C;QACtD,QAAQ,EAAE,OAAO,EAAE,UAAU,EAAE,IAAI,EAAE,YAAY,EAAE,CAAC;QACpD,IAAI,EAAE,CAAC,OAAO,EAAE,KAAK,EAAE,QAAQ,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,CAAC,CAAC;KACjD;IAED,8EAA8E;IAC9E;QACE,KAAK,EAAE,6BAA6B;QACpC,OAAO,EAAE,SAAS;QAClB,WAAW,EAAE,4KAA4K;QACzL,OAAO,EAAE,wDAAwD;QACjE,QAAQ,EAAE,OAAO,EAAE,UAAU,EAAE,IAAI,EAAE,YAAY,EAAE,CAAC;QACpD,IAAI,EAAE,CAAC,IAAI,EAAE,KAAK,EAAE,UAAU,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,CAAC,CAAC;KAChD;IACD;QACE,KAAK,EAAE,2BAA2B;QAClC,OAAO,EAAE,SAAS;QAClB,WAAW,EAAE,oGAAoG;QACjH,OAAO,EAAE,0CAA0C;QACnD,QAAQ,EAAE,OAAO,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,CAAC;QACnD,IAAI,EAAE,CAAC,IAAI,EAAE,IAAI,EAAE,MAAM,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,CAAC,CAAC;KAC3C;IAED,8EAA8E;IAC9E;QACE,KAAK,EAAE,iCAAiC;QACxC,OAAO,EAAE,SAAS;QAClB,WAAW,EAAE,kIAAkI;QAC/I,OAAO,EAAE,gEAAgE;QACzE,QAAQ,EAAE,OAAO,EAAE,UAAU,EAAE,IAAI,EAAE,YAAY,EAAE,CAAC;QACpD,IAAI,EAAE,CAAC,MAAM,EAAE,UAAU,EAAE,KAAK,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,CAAC,CAAC;KAClD;IACD;QACE,KAAK,EAAE,6BAA6B;QACpC,OAAO,EAAE,SAAS;QAClB,WAAW,EAAE,2JAA2J;QACxK,OAAO,EAAE,iDAAiD;QAC1D,QAAQ,EAAE,OAAO,EAAE,UAAU,EAAE,GAAG,EAAE,YAAY,EAAE,CAAC;QACnD,IAAI,EAAE,CAAC,QAAQ,EAAE,UAAU,EAAE,WAAW,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,EAAE,CAAC;KAC3D;IAED,8EAA8E;IAC9E,8EAA8E;IAC9E;QACE,KAAK,EAAE,gCAAgC;QACvC,OAAO,EAAE,SAAS;QAClB,WAAW,EAAE,iNAAiN;QAC9N,OAAO,EAAE,sEAAsE;QAC/E,QAAQ,EAAE,UAAU,EAAE,UAAU,EAAE,IAAI,EAAE,YAAY,EAAE,CAAC;QACvD,WAAW,EAAE,cAAc,EAAE,YAAY,EAAE,QAAQ,EAAE,YAAY,EAAE,CAAC;QACpE,IAAI,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,UAAU,EAAE,QAAQ,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,EAAE,CAAC;KAC9D;IACD;QACE,KAAK,EAAE,+BAA+B;QACtC,OAAO,EAAE,SAAS;QAClB,WAAW,EAAE,uLAAuL;QACpM,OAAO,EAAE,6EAA6E;QACtF,QAAQ,EAAE,OAAO,EAAE,UAAU,EAAE,IAAI,EAAE,YAAY,EAAE,CAAC;QACpD,IAAI,EAAE,CAAC,MAAM,EAAE,MAAM,EAAE,UAAU,CAAC,EAAE,EAAE,EAAE,OAAO,CAAC,EAAE,CAAC;KACpD;CACF,CAAC;AAEF,MAAM,CAAC,MAAM,aAAa,GAAiB;IACzC;QACE,KAAK,EAAE,+CAA+C;QACtD,QAAQ,EAAE,CAAC,0BAA0B,CAAC;KACvC;IACD;QACE,KAAK,EAAE,0CAA0C;QACjD,QAAQ,EAAE,CAAC,uCAAuC,CAAC;KACpD;IACD;QACE,KAAK,EAAE,uCAAuC;QAC9C,QAAQ,EAAE,CAAC,sCAAsC,CAAC;KACnD;IACD;QACE,KAAK,EAAE,0CAA0C;QACjD,QAAQ,EAAE,CAAC,6BAA6B,CAAC;KAC1C;IACD;QACE,KAAK,EAAE,+CAA+C;QACtD,QAAQ,EAAE,CAAC,qCAAqC,CAAC;KAClD;IACD;QACE,KAAK,EAAE,kCAAkC;QACzC,QAAQ,EAAE,CAAC,4BAA4B,CAAC;KACzC;IACD;QACE,KAAK,EAAE,sCAAsC;QAC7C,QAAQ,EAAE,CAAC,+BAA+B,CAAC;KAC5C;IACD;QACE,KAAK,EAAE,wDAAwD;QAC/D,QAAQ,EAAE,CAAC,6BAA6B,CAAC;KAC1C;IACD;QACE,KAAK,EAAE,6CAA6C;QACpD,QAAQ,EAAE,CAAC,iCAAiC,CAAC;KAC9C;IACD;QACE,KAAK,EAAE,mCAAmC;QAC1C,QAAQ,EAAE,CAAC,+BAA+B,CAAC;KAC5C;IACD;QACE,KAAK,EAAE,2CAA2C;QAClD,QAAQ,EAAE,CAAC,2BAA2B,CAAC;KACxC;IACD,4EAA4E;IAC5E;QACE,KAAK,EAAE,4CAA4C;QACnD,QAAQ,EAAE,CAAC,uCAAuC,EAAE,sCAAsC,CAAC;KAC5F;IACD,gFAAgF;IAChF,2EAA2E;IAC3E,kFAAkF;IAClF;QACE,KAAK,EAAE,mDAAmD;QAC1D,QAAQ,EAAE,CAAC,gCAAgC,CAAC;KAC7C;CACF,CAAC"}
|
|
@@ -6,10 +6,20 @@ import { keywordSearch, tokenize, splitMultiQuery, levenshtein, indexVocab as _i
|
|
|
6
6
|
import { rerankByQuality } from '../rerank.js';
|
|
7
7
|
import { computeEmbedding, hasEmbedProvider } from '../embeddings.js';
|
|
8
8
|
// ── Changelog (shown once per version in session_start) ──────────────────────
|
|
9
|
-
const MCP_VERSION = '0.10.
|
|
9
|
+
const MCP_VERSION = '0.10.52';
|
|
10
10
|
const WHATS_NEW = {
|
|
11
|
-
'0.10.
|
|
11
|
+
'0.10.52': [
|
|
12
12
|
`🆕 **What's new in v${MCP_VERSION}:**`,
|
|
13
|
+
` ✅ \`smart_recall\` — CKG traversal as Layer 3: finds lessons that FIXED causal-graph-similar problems (🕸️ causal graph badge)`,
|
|
14
|
+
` ✅ Bench corpus expanded: governance adversarial pair proves review-boost end-to-end (+22% P@1 headline)`,
|
|
15
|
+
` 🔧 Import cleanup: unused symbols removed from brain.ts (no functional change)`,
|
|
16
|
+
` 💡 \`smart_recall\` now shows keyword / semantic / CKG / hybrid match type for every result`,
|
|
17
|
+
],
|
|
18
|
+
'0.10.51': [
|
|
19
|
+
` ✅ \`team_confirm\` — human review raises lesson recall ranking (🛡️ senior / ✔️ peer)`,
|
|
20
|
+
` ✅ Governance-aware recall — confirmed lessons outrank unreviewed auto-learned entries`,
|
|
21
|
+
],
|
|
22
|
+
'0.10.50': [
|
|
13
23
|
` ✅ \`smart_recall\` — unified keyword + semantic hybrid list (one ranked result, not two sections)`,
|
|
14
24
|
` ✅ \`smart_recall\` — "Brain saved you here" banner surfaces time saved inline per proven lesson`,
|
|
15
25
|
` ✅ Quality reranking — proven successes outrank symptom-dense failures (+11% Precision@1)`,
|
|
@@ -513,6 +523,59 @@ export async function handleBrainTool(name, args, getConnection, apiFetch) {
|
|
|
513
523
|
});
|
|
514
524
|
}
|
|
515
525
|
}
|
|
526
|
+
// ── Layer 3: CKG traversal — follow "fixes" edges to find lessons that solved
|
|
527
|
+
// structurally similar problems, even when vocabulary differs ───────────────
|
|
528
|
+
try {
|
|
529
|
+
const qTokens = tokenize(query).filter(t => t.length >= 4).slice(0, 4);
|
|
530
|
+
const candidateNodeIds = new Set();
|
|
531
|
+
for (const token of qTokens) {
|
|
532
|
+
const nodeStream = redis.scanStream({ match: `cachly:ckg:node:*${token}*`, count: 20 });
|
|
533
|
+
await new Promise((res, rej) => {
|
|
534
|
+
nodeStream.on('data', (batch) => {
|
|
535
|
+
for (const k of batch.slice(0, 5))
|
|
536
|
+
candidateNodeIds.add(k.replace('cachly:ckg:node:', ''));
|
|
537
|
+
});
|
|
538
|
+
nodeStream.on('end', res);
|
|
539
|
+
nodeStream.on('error', rej);
|
|
540
|
+
});
|
|
541
|
+
}
|
|
542
|
+
const seenFroms = new Set();
|
|
543
|
+
for (const nodeId of [...candidateNodeIds].slice(0, 12)) {
|
|
544
|
+
const inboundEdgeKeys = await redis.smembers(`cachly:ckg:idx:to:${nodeId}`);
|
|
545
|
+
for (const ek of inboundEdgeKeys.slice(0, 6)) {
|
|
546
|
+
const er = await redis.get(ek);
|
|
547
|
+
if (!er)
|
|
548
|
+
continue;
|
|
549
|
+
const edge = safeJsonParse(er, null);
|
|
550
|
+
if (!edge || edge.edgeType !== 'fixes' || edge.confidence < 0.35)
|
|
551
|
+
continue;
|
|
552
|
+
if (seenFroms.has(edge.from))
|
|
553
|
+
continue;
|
|
554
|
+
seenFroms.add(edge.from);
|
|
555
|
+
const lessonKey = `cachly:lesson:best:${edge.from}`;
|
|
556
|
+
const existing = hybridMap.get(lessonKey);
|
|
557
|
+
if (existing) {
|
|
558
|
+
// Already found by BM25/semantic — boost with causal confirmation
|
|
559
|
+
existing.ckgScore = Math.max(existing.ckgScore ?? 0, edge.confidence);
|
|
560
|
+
existing.hybridScore = existing.hybridScore * 0.88 + edge.confidence * 0.12;
|
|
561
|
+
if (existing.matchType === 'keyword')
|
|
562
|
+
existing.matchType = 'hybrid';
|
|
563
|
+
}
|
|
564
|
+
else {
|
|
565
|
+
const content = await redis.get(lessonKey).catch(() => null);
|
|
566
|
+
if (!content)
|
|
567
|
+
continue;
|
|
568
|
+
hybridMap.set(lessonKey, {
|
|
569
|
+
key: lessonKey, content,
|
|
570
|
+
ckgScore: edge.confidence,
|
|
571
|
+
hybridScore: edge.confidence * 0.30, // CKG-only: lower than text hits
|
|
572
|
+
matchType: 'ckg',
|
|
573
|
+
});
|
|
574
|
+
}
|
|
575
|
+
}
|
|
576
|
+
}
|
|
577
|
+
}
|
|
578
|
+
catch { /* non-critical — keyword + semantic always available */ }
|
|
516
579
|
const hybridResults = [...hybridMap.values()].sort((a, b) => b.hybridScore - a.hybridScore);
|
|
517
580
|
// ── Build output ──────────────────────────────────────────────────────────
|
|
518
581
|
const lines = [`🧠 **Smart Recall** for: _"${query}"_\n`];
|
|
@@ -528,9 +591,14 @@ export async function handleBrainTool(name, args, getConnection, apiFetch) {
|
|
|
528
591
|
lines.push(`_Detected ${subQueries.length} sub-topics:_ ${subQueries.map((s, i) => `${i + 1}. "${s}"`).join(', ')}\n`);
|
|
529
592
|
}
|
|
530
593
|
if (hybridResults.length > 0) {
|
|
531
|
-
const
|
|
532
|
-
|
|
533
|
-
|
|
594
|
+
const hasCKG = hybridResults.some(r => r.ckgScore !== undefined);
|
|
595
|
+
const modeLabel = semHits.length > 0 && hasCKG
|
|
596
|
+
? `keyword + semantic + CKG hybrid`
|
|
597
|
+
: semHits.length > 0
|
|
598
|
+
? `keyword + semantic hybrid`
|
|
599
|
+
: hasCKG
|
|
600
|
+
? `keyword + CKG hybrid`
|
|
601
|
+
: `keyword`;
|
|
534
602
|
lines.push(`### 🔍 Results (${hybridResults.length} — ${modeLabel})\n`);
|
|
535
603
|
// Group by sub-query for multi-topic queries
|
|
536
604
|
if (subQueries.length > 1) {
|
|
@@ -545,11 +613,15 @@ export async function handleBrainTool(name, args, getConnection, apiFetch) {
|
|
|
545
613
|
lines.push(`**Topic: "${sq}"** (${results.length} results)\n`);
|
|
546
614
|
for (const r of results.slice(0, 4)) {
|
|
547
615
|
const label = r.key.replace('cachly:ctx:', '📝 ').replace('cachly:lesson:best:', '💡 ').replace('cachly:idx:', '📂 ');
|
|
548
|
-
const scorePart = r.matchType === '
|
|
549
|
-
? `
|
|
550
|
-
: r.
|
|
551
|
-
? `
|
|
552
|
-
:
|
|
616
|
+
const scorePart = r.matchType === 'ckg'
|
|
617
|
+
? `CKG: ${(r.ckgScore ?? 0).toFixed(2)}, 🕸️ causal graph`
|
|
618
|
+
: r.ckgScore !== undefined && r.bm25Score !== undefined
|
|
619
|
+
? `BM25: ${(r.bm25Score).toFixed(2)}, CKG: ${(r.ckgScore).toFixed(2)}, 🔀 hybrid`
|
|
620
|
+
: r.matchType === 'hybrid'
|
|
621
|
+
? `BM25: ${(r.bm25Score ?? 0).toFixed(2)}, sem: ${((r.semScore ?? 0) * 100).toFixed(0)}%, 🔀 hybrid`
|
|
622
|
+
: r.matchType === 'semantic'
|
|
623
|
+
? `sem: ${((r.semScore ?? 0) * 100).toFixed(0)}%, 🎯 semantic`
|
|
624
|
+
: `BM25: ${(r.bm25Score ?? 0).toFixed(2)}, matched: ${r.matchedWords?.join(', ')}`;
|
|
553
625
|
const preview = r.content.slice(0, 300).replace(/\n/g, ' ');
|
|
554
626
|
lines.push(` **${label}** _(${scorePart})_`);
|
|
555
627
|
lines.push(` > ${preview}${r.content.length > 300 ? '…' : ''}\n`);
|