jeo-code 0.7.19 → 0.7.20
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +15 -0
- package/README.ja.md +1 -1
- package/README.ko.md +1 -1
- package/README.md +1 -1
- package/README.zh.md +1 -1
- package/package.json +1 -1
- package/src/agent/AGENTS.md +1 -1
- package/src/agent/memory.ts +143 -39
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,21 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|
|
6
6
|
|
|
7
7
|
The README mirrors the latest 5 entries — regenerate with `bun run changelog:sync`.
|
|
8
8
|
|
|
9
|
+
## [0.7.20] - 2026-06-26
|
|
10
|
+
_OKF concept-memory retrieval gains a hybrid reranker ported from memsearch. Injection priority no longer rides one raw keyword score — it fuses two complementary ranked channels by Reciprocal Rank Fusion (RRF): IDF-weighted lexical relevance (the sparse/BM25 channel, so rare discriminating terms steer recall) and concept-graph proximity (the local dense/semantic-neighbour channel, so a hub linked from multiple query hits surfaces even with no keyword of its own). All embedding-free and deterministic, layered atop the existing failure-first tier and pinned-invariant reserved budget._
|
|
11
|
+
|
|
12
|
+
### Added
|
|
13
|
+
- **IDF-weighted lexical scoring (`buildCorpusStats` / `tokenIdf`).** `src/agent/memory.ts` adds `CorpusStats` (corpus size + per-token document frequency) built once per retrieval pass, and BM25-style non-negative IDF `ln(1 + (N − df + 0.5)/(df + 0.5))`. `scoreConcept` now takes optional `stats` and scales each token's field weight (title ≫ tags ≫ type/description ≫ body) by its IDF, so a rare specific term outranks a concept hit only by common filler. Always > 0 for a present token, so the score-presence semantics the failure-gate and filters depend on are preserved; omitting `stats` keeps the raw field-weighted count.
|
|
14
|
+
- **Reciprocal Rank Fusion (`reciprocalRankFusion`, k=60).** Fuses several ranked id-lists into one score map — each list contributes `1/(k + rank)` summed across lists, rank-based (not score-based) so a strong lexical hit and a strong graph-proximity hit combine on equal, scale-free footing, and a concept ranked in BOTH channels rises above one strong in only one. Tunable `k`; an empty channel is a no-op.
|
|
15
|
+
- **Graph-proximity channel (`graphProximityOrder`).** Ranks concepts by how many distinct query-hit seeds reach them via 1-hop links (using the public `expandByGraph` API), the local stand-in for memsearch's dense-vector channel: a strongly-connected neighbour surfaces even when its own lexical score is weak. Input order breaks ties.
|
|
16
|
+
|
|
17
|
+
### Changed
|
|
18
|
+
- **`priorityOrder` fuses lexical ⊕ graph by RRF instead of raw score + a boolean `related` flag.** The failure-first and high-confidence "core" tiers are unchanged; below them, the prior `score`-then-`related` ordering is replaced by the fused RRF rank, with input order as the stable final tiebreak. `searchConcepts` now IDF-weights via `buildCorpusStats` too. The failure boost still only fires on an actual query hit (score > 0).
|
|
19
|
+
|
|
20
|
+
### Verified
|
|
21
|
+
- `bun run typecheck` clean; full suite green — 2022 pass / 0 fail across 234 files — including new `test/memory-search-okf.test.ts` cases (`buildCorpusStats` document-frequency counting, `scoreConcept` IDF scaling, a rare term outranking a higher raw-weight common term, `reciprocalRankFusion` cross-channel summation + tunable `k` + empty-channel no-op, graph proximity lifting a multiply-linked hub above an unlinked equal-score concept). `scripts/rrf-runtime-check.ts` drives the production `memoryPromptSection` against a real on-disk bundle.
|
|
22
|
+
|
|
23
|
+
|
|
9
24
|
## [0.7.19] - 2026-06-26
|
|
10
25
|
_The live model picker gains gajae-code's `/model` provider tabs, and skill invocation is consolidated onto a single `$` entrypoint. The picker now shows an `ALL` tab plus one tab per provider that `tab`/`shift+tab` cycles, and skills (including their declared aliases) are invoked only via `$` — the slash palette stays builtins-only._
|
|
11
26
|
|
package/README.ja.md
CHANGED
|
@@ -205,11 +205,11 @@ CI は `.github/workflows/npm-publish.yml` で公開します — GitHub リリ
|
|
|
205
205
|
## 変更履歴 (Changelog)
|
|
206
206
|
|
|
207
207
|
<!-- CHANGELOG:START (auto-generated from CHANGELOG.md — run `bun run changelog:sync`) -->
|
|
208
|
+
- **[0.7.20]** (2026-06-26) — OKF concept-memory retrieval gains a hybrid reranker ported from memsearch. Injection priority no longer rides one raw keyword score — it fuses two complementary ranked channels by Reciprocal Rank Fusion (RRF): IDF-weighted lexical relevance (the sparse/BM25 channel, so rare discriminating terms steer recall) and concept-graph proximity (the local dense/semantic-neighbour channel, so a hub linked from multiple query hits surfaces even with no keyword of its own). All embedding-free and deterministic, layered atop the existing failure-first tier and pinned-invariant reserved budget.
|
|
208
209
|
- **[0.7.19]** (2026-06-26) — The live model picker gains gajae-code's `/model` provider tabs, and skill invocation is consolidated onto a single `$` entrypoint. The picker now shows an `ALL` tab plus one tab per provider that `tab`/`shift+tab` cycles, and skills (including their declared aliases) are invoked only via `$` — the slash palette stays builtins-only.
|
|
209
210
|
- **[0.7.18]** (2026-06-26) — Slash-command discovery and the `/model` flow reach gajae-code parity. The slash palette/autocomplete now fuzzy-matches command names (with a description fallback for intent-style queries), resolved skills can contribute their own `/aliases` as real dispatchable commands, and `/model` runs gjc's two-menu target → reasoning flow so a picked model can be assigned to the default or any subagent role with its own thinking budget.
|
|
210
211
|
- **[0.7.17]** (2026-06-25) — Developer workflow parity (gjc `dev:link`/`dev:doctor`, adapted for jeo's zero-native-dep Bun runtime): the global `jeo` command can be linked to run this checkout's source hot to every edit, with a drift doctor that flags when `jeo` resolves to a compiled binary or an installed copy instead. README gains "Skill migration and bundled skill inspection" + "Development" sections. Also ships OKF concept-memory search/scoring with budget-aware injection and a round of workflow-prompt hardening (anti-punt, todo-first planning, verdict discipline) that keeps every loop escape hatch intact.
|
|
211
212
|
- **[0.7.16]** (2026-06-25) — Bundled skills get a safe installer: `jeo skills sync` materializes the built-in workflow skills into `~/.jeo/skills`, preserving local edits by default, with a CI-friendly `--check` drift report and a `--force` overwrite.
|
|
212
|
-
- **[0.7.15]** (2026-06-25) — Rollback release: the runtime source and test suite are restored to the **0.7.9** state. The intervening 0.7.10–0.7.14 runtime changes (quiet-mode/`-q`, `/resume` transcript rework, `mutatedFiles` plumbing, Ollama `num_ctx`, tmux REPL reaping, and related engine/provider/TUI edits) are reverted. Published as a new version because npm cannot re-issue an existing version number; `latest` now serves code identical to 0.7.9.
|
|
213
213
|
|
|
214
214
|
See [CHANGELOG.md](CHANGELOG.md) for the full history.
|
|
215
215
|
<!-- CHANGELOG:END -->
|
package/README.ko.md
CHANGED
|
@@ -205,11 +205,11 @@ CI는 `.github/workflows/npm-publish.yml`로 배포합니다 — GitHub 릴리
|
|
|
205
205
|
## 변경 이력 (Changelog)
|
|
206
206
|
|
|
207
207
|
<!-- CHANGELOG:START (auto-generated from CHANGELOG.md — run `bun run changelog:sync`) -->
|
|
208
|
+
- **[0.7.20]** (2026-06-26) — OKF concept-memory retrieval gains a hybrid reranker ported from memsearch. Injection priority no longer rides one raw keyword score — it fuses two complementary ranked channels by Reciprocal Rank Fusion (RRF): IDF-weighted lexical relevance (the sparse/BM25 channel, so rare discriminating terms steer recall) and concept-graph proximity (the local dense/semantic-neighbour channel, so a hub linked from multiple query hits surfaces even with no keyword of its own). All embedding-free and deterministic, layered atop the existing failure-first tier and pinned-invariant reserved budget.
|
|
208
209
|
- **[0.7.19]** (2026-06-26) — The live model picker gains gajae-code's `/model` provider tabs, and skill invocation is consolidated onto a single `$` entrypoint. The picker now shows an `ALL` tab plus one tab per provider that `tab`/`shift+tab` cycles, and skills (including their declared aliases) are invoked only via `$` — the slash palette stays builtins-only.
|
|
209
210
|
- **[0.7.18]** (2026-06-26) — Slash-command discovery and the `/model` flow reach gajae-code parity. The slash palette/autocomplete now fuzzy-matches command names (with a description fallback for intent-style queries), resolved skills can contribute their own `/aliases` as real dispatchable commands, and `/model` runs gjc's two-menu target → reasoning flow so a picked model can be assigned to the default or any subagent role with its own thinking budget.
|
|
210
211
|
- **[0.7.17]** (2026-06-25) — Developer workflow parity (gjc `dev:link`/`dev:doctor`, adapted for jeo's zero-native-dep Bun runtime): the global `jeo` command can be linked to run this checkout's source hot to every edit, with a drift doctor that flags when `jeo` resolves to a compiled binary or an installed copy instead. README gains "Skill migration and bundled skill inspection" + "Development" sections. Also ships OKF concept-memory search/scoring with budget-aware injection and a round of workflow-prompt hardening (anti-punt, todo-first planning, verdict discipline) that keeps every loop escape hatch intact.
|
|
211
212
|
- **[0.7.16]** (2026-06-25) — Bundled skills get a safe installer: `jeo skills sync` materializes the built-in workflow skills into `~/.jeo/skills`, preserving local edits by default, with a CI-friendly `--check` drift report and a `--force` overwrite.
|
|
212
|
-
- **[0.7.15]** (2026-06-25) — Rollback release: the runtime source and test suite are restored to the **0.7.9** state. The intervening 0.7.10–0.7.14 runtime changes (quiet-mode/`-q`, `/resume` transcript rework, `mutatedFiles` plumbing, Ollama `num_ctx`, tmux REPL reaping, and related engine/provider/TUI edits) are reverted. Published as a new version because npm cannot re-issue an existing version number; `latest` now serves code identical to 0.7.9.
|
|
213
213
|
|
|
214
214
|
See [CHANGELOG.md](CHANGELOG.md) for the full history.
|
|
215
215
|
<!-- CHANGELOG:END -->
|
package/README.md
CHANGED
|
@@ -232,11 +232,11 @@ Huge thanks to [gajae-code](https://github.com/Yeachan-Heo/gajae-code) for the i
|
|
|
232
232
|
## Changelog
|
|
233
233
|
|
|
234
234
|
<!-- CHANGELOG:START (auto-generated from CHANGELOG.md — run `bun run changelog:sync`) -->
|
|
235
|
+
- **[0.7.20]** (2026-06-26) — OKF concept-memory retrieval gains a hybrid reranker ported from memsearch. Injection priority no longer rides one raw keyword score — it fuses two complementary ranked channels by Reciprocal Rank Fusion (RRF): IDF-weighted lexical relevance (the sparse/BM25 channel, so rare discriminating terms steer recall) and concept-graph proximity (the local dense/semantic-neighbour channel, so a hub linked from multiple query hits surfaces even with no keyword of its own). All embedding-free and deterministic, layered atop the existing failure-first tier and pinned-invariant reserved budget.
|
|
235
236
|
- **[0.7.19]** (2026-06-26) — The live model picker gains gajae-code's `/model` provider tabs, and skill invocation is consolidated onto a single `$` entrypoint. The picker now shows an `ALL` tab plus one tab per provider that `tab`/`shift+tab` cycles, and skills (including their declared aliases) are invoked only via `$` — the slash palette stays builtins-only.
|
|
236
237
|
- **[0.7.18]** (2026-06-26) — Slash-command discovery and the `/model` flow reach gajae-code parity. The slash palette/autocomplete now fuzzy-matches command names (with a description fallback for intent-style queries), resolved skills can contribute their own `/aliases` as real dispatchable commands, and `/model` runs gjc's two-menu target → reasoning flow so a picked model can be assigned to the default or any subagent role with its own thinking budget.
|
|
237
238
|
- **[0.7.17]** (2026-06-25) — Developer workflow parity (gjc `dev:link`/`dev:doctor`, adapted for jeo's zero-native-dep Bun runtime): the global `jeo` command can be linked to run this checkout's source hot to every edit, with a drift doctor that flags when `jeo` resolves to a compiled binary or an installed copy instead. README gains "Skill migration and bundled skill inspection" + "Development" sections. Also ships OKF concept-memory search/scoring with budget-aware injection and a round of workflow-prompt hardening (anti-punt, todo-first planning, verdict discipline) that keeps every loop escape hatch intact.
|
|
238
239
|
- **[0.7.16]** (2026-06-25) — Bundled skills get a safe installer: `jeo skills sync` materializes the built-in workflow skills into `~/.jeo/skills`, preserving local edits by default, with a CI-friendly `--check` drift report and a `--force` overwrite.
|
|
239
|
-
- **[0.7.15]** (2026-06-25) — Rollback release: the runtime source and test suite are restored to the **0.7.9** state. The intervening 0.7.10–0.7.14 runtime changes (quiet-mode/`-q`, `/resume` transcript rework, `mutatedFiles` plumbing, Ollama `num_ctx`, tmux REPL reaping, and related engine/provider/TUI edits) are reverted. Published as a new version because npm cannot re-issue an existing version number; `latest` now serves code identical to 0.7.9.
|
|
240
240
|
|
|
241
241
|
See [CHANGELOG.md](CHANGELOG.md) for the full history.
|
|
242
242
|
<!-- CHANGELOG:END -->
|
package/README.zh.md
CHANGED
|
@@ -205,11 +205,11 @@ CI 通过 `.github/workflows/npm-publish.yml` 发布 — GitHub 发布 release
|
|
|
205
205
|
## 更新日志 (Changelog)
|
|
206
206
|
|
|
207
207
|
<!-- CHANGELOG:START (auto-generated from CHANGELOG.md — run `bun run changelog:sync`) -->
|
|
208
|
+
- **[0.7.20]** (2026-06-26) — OKF concept-memory retrieval gains a hybrid reranker ported from memsearch. Injection priority no longer rides one raw keyword score — it fuses two complementary ranked channels by Reciprocal Rank Fusion (RRF): IDF-weighted lexical relevance (the sparse/BM25 channel, so rare discriminating terms steer recall) and concept-graph proximity (the local dense/semantic-neighbour channel, so a hub linked from multiple query hits surfaces even with no keyword of its own). All embedding-free and deterministic, layered atop the existing failure-first tier and pinned-invariant reserved budget.
|
|
208
209
|
- **[0.7.19]** (2026-06-26) — The live model picker gains gajae-code's `/model` provider tabs, and skill invocation is consolidated onto a single `$` entrypoint. The picker now shows an `ALL` tab plus one tab per provider that `tab`/`shift+tab` cycles, and skills (including their declared aliases) are invoked only via `$` — the slash palette stays builtins-only.
|
|
209
210
|
- **[0.7.18]** (2026-06-26) — Slash-command discovery and the `/model` flow reach gajae-code parity. The slash palette/autocomplete now fuzzy-matches command names (with a description fallback for intent-style queries), resolved skills can contribute their own `/aliases` as real dispatchable commands, and `/model` runs gjc's two-menu target → reasoning flow so a picked model can be assigned to the default or any subagent role with its own thinking budget.
|
|
210
211
|
- **[0.7.17]** (2026-06-25) — Developer workflow parity (gjc `dev:link`/`dev:doctor`, adapted for jeo's zero-native-dep Bun runtime): the global `jeo` command can be linked to run this checkout's source hot to every edit, with a drift doctor that flags when `jeo` resolves to a compiled binary or an installed copy instead. README gains "Skill migration and bundled skill inspection" + "Development" sections. Also ships OKF concept-memory search/scoring with budget-aware injection and a round of workflow-prompt hardening (anti-punt, todo-first planning, verdict discipline) that keeps every loop escape hatch intact.
|
|
211
212
|
- **[0.7.16]** (2026-06-25) — Bundled skills get a safe installer: `jeo skills sync` materializes the built-in workflow skills into `~/.jeo/skills`, preserving local edits by default, with a CI-friendly `--check` drift report and a `--force` overwrite.
|
|
212
|
-
- **[0.7.15]** (2026-06-25) — Rollback release: the runtime source and test suite are restored to the **0.7.9** state. The intervening 0.7.10–0.7.14 runtime changes (quiet-mode/`-q`, `/resume` transcript rework, `mutatedFiles` plumbing, Ollama `num_ctx`, tmux REPL reaping, and related engine/provider/TUI edits) are reverted. Published as a new version because npm cannot re-issue an existing version number; `latest` now serves code identical to 0.7.9.
|
|
213
213
|
|
|
214
214
|
See [CHANGELOG.md](CHANGELOG.md) for the full history.
|
|
215
215
|
<!-- CHANGELOG:END -->
|
package/package.json
CHANGED
package/src/agent/AGENTS.md
CHANGED
|
@@ -18,7 +18,7 @@ The core runtime loop, tool registry, session management, and state persistence
|
|
|
18
18
|
| `json.ts` | Brief description of purpose |
|
|
19
19
|
| `loop.ts` | The primary execution loop orchestrating model calls and tool execution |
|
|
20
20
|
| `loop-guards.ts` | Intermediate-judgment classification (gjc ultragoal-guard parity): named `GuardState` taxonomy, `GUARD_LIMITS` thresholds, and pure classifiers (`isVerificationSignal`, `repeatHint`, `classifyDoneGate` — incl. `done_stale_verification` for a passing test/build that predates the last edit) consumed by `engine.ts` |
|
|
21
|
-
| `memory.ts` | OKF concept-bundle memory: session distill, query-aware budget injection (failure-first
|
|
21
|
+
| `memory.ts` | OKF concept-bundle memory: session distill, query-aware budget injection with a **hybrid reranker ported from memsearch** — `priorityOrder` fuses two ranked channels via `reciprocalRankFusion` (RRF, k=60): lexical IDF relevance (`buildCorpusStats`/`scoreConcept`, the sparse/BM25 channel so rare discriminating terms steer retrieval) ⊕ graph proximity (`graphProximityOrder`, the local dense/semantic-neighbour channel — a hub linked from multiple query hits surfaces even with no keyword of its own); all embedding-free, deterministic, atop the failure-first tier + pinned-invariant reserved budget. Plus mid-loop `recordFailedAttempt` capture, shared `upsertConceptFile` writer, legacy MEMORY.md migration (`migrateLegacyMemory`) + `JEO_MEMORY_LEGACY` rollback toggle |
|
|
22
22
|
| `memory-okf.ts` | OKF v0.1 format layer: frontmatter parse/serialize, concept IDs, conformance validation |
|
|
23
23
|
| `memory-graph.ts` | Concept cross-link graph: build/expand (1-hop search), broken-link-tolerant lint, optional graphify detection |
|
|
24
24
|
| `model-recency.ts` | Brief description of purpose |
|
package/src/agent/memory.ts
CHANGED
|
@@ -17,7 +17,7 @@ import { callLlm, type Message } from "./loop";
|
|
|
17
17
|
import { jeoEnv } from "../util/env";
|
|
18
18
|
import { parseConcept, serializeConcept, slugify, isReservedFile, conceptId } from "./memory-okf";
|
|
19
19
|
import { tryExtractJsonObject } from "./json";
|
|
20
|
-
import { buildConceptGraph, expandByGraph, lintConceptGraph, type GraphLintReport } from "./memory-graph";
|
|
20
|
+
import { buildConceptGraph, expandByGraph, lintConceptGraph, type ConceptGraph, type GraphLintReport } from "./memory-graph";
|
|
21
21
|
|
|
22
22
|
/** On-disk document cap — the distill prompt instructs the model to stay under it. */
|
|
23
23
|
export const MEMORY_MAX_CHARS = 6_000;
|
|
@@ -314,9 +314,56 @@ function tokenize(query?: string): string[] {
|
|
|
314
314
|
return Array.from(new Set((query.toLowerCase().match(/[a-z0-9]+/g) ?? []).filter(t => t.length >= 3)));
|
|
315
315
|
}
|
|
316
316
|
|
|
317
|
+
/** Corpus statistics for IDF (inverse document frequency) weighting: total concept
|
|
318
|
+
* count `n` and per-token document frequency `df` (how many concepts' searchable
|
|
319
|
+
* text contains the token). Built once per retrieval pass over the loaded bundle. */
|
|
320
|
+
export interface CorpusStats {
|
|
321
|
+
n: number;
|
|
322
|
+
df: Map<string, number>;
|
|
323
|
+
}
|
|
324
|
+
|
|
325
|
+
/** Lowercased searchable text of a concept (title + tags + type + description +
|
|
326
|
+
* body) — the field corpus over which document frequency is measured. */
|
|
327
|
+
function searchableText(concept: Concept): string {
|
|
328
|
+
return [concept.title, concept.tags.join(" "), concept.type, concept.description, concept.body]
|
|
329
|
+
.join(" ")
|
|
330
|
+
.toLowerCase();
|
|
331
|
+
}
|
|
332
|
+
|
|
333
|
+
/** Build IDF corpus stats from the concept set: per distinct token, how many
|
|
334
|
+
* concepts contain it. Pure and deterministic — no embeddings, no network
|
|
335
|
+
* (memsearch's hybrid BM25 contribution adapted to jeo's local-first bundle). */
|
|
336
|
+
export function buildCorpusStats(concepts: Concept[]): CorpusStats {
|
|
337
|
+
const df = new Map<string, number>();
|
|
338
|
+
for (const c of concepts) {
|
|
339
|
+
const text = searchableText(c);
|
|
340
|
+
// Distinct tokens of THIS concept (length ≥ 3, mirroring tokenize) so each
|
|
341
|
+
// concept contributes at most 1 to a token's document frequency.
|
|
342
|
+
const seen = new Set((text.match(/[a-z0-9]+/g) ?? []).filter(t => t.length >= 3));
|
|
343
|
+
for (const t of seen) df.set(t, (df.get(t) ?? 0) + 1);
|
|
344
|
+
}
|
|
345
|
+
return { n: concepts.length, df };
|
|
346
|
+
}
|
|
347
|
+
|
|
348
|
+
/** BM25-style non-negative IDF for a token: ln(1 + (N − df + 0.5)/(df + 0.5)).
|
|
349
|
+
* A token that discriminates (appears in few concepts) outweighs a token that
|
|
350
|
+
* appears everywhere — so a query's rare, specific terms steer retrieval instead
|
|
351
|
+
* of common filler. Always > 0 for a present token (df ≥ 1), so the score > 0
|
|
352
|
+
* presence semantics the failure-gate and filters depend on are preserved. A
|
|
353
|
+
* token absent from the corpus only ever contributes through a field it does not
|
|
354
|
+
* match (weight 0), so its IDF is moot; we floor df at the token's own presence. */
|
|
355
|
+
function tokenIdf(token: string, stats: CorpusStats): number {
|
|
356
|
+
const df = stats.df.get(token) ?? 1;
|
|
357
|
+
return Math.log(1 + (stats.n - df + 0.5) / (df + 0.5));
|
|
358
|
+
}
|
|
359
|
+
|
|
317
360
|
/** Relevance score of a concept against query tokens. Field weights mirror
|
|
318
|
-
* llm-wiki's retrieval bias (title ≫ tags ≫ type/description ≫ body).
|
|
319
|
-
|
|
361
|
+
* llm-wiki's retrieval bias (title ≫ tags ≫ type/description ≫ body). When
|
|
362
|
+
* `stats` is supplied, each token's field weight is scaled by its corpus IDF
|
|
363
|
+
* (memsearch-style BM25 weighting) so a rare discriminating term ranks a concept
|
|
364
|
+
* above one hit only by common tokens. Without `stats` this is the raw
|
|
365
|
+
* field-weighted presence count (IDF = 1). 0 = no hit. */
|
|
366
|
+
export function scoreConcept(concept: Concept, tokens: string[], stats?: CorpusStats): number {
|
|
320
367
|
if (tokens.length === 0) return 0;
|
|
321
368
|
const title = concept.title.toLowerCase();
|
|
322
369
|
const desc = concept.description.toLowerCase();
|
|
@@ -325,70 +372,127 @@ export function scoreConcept(concept: Concept, tokens: string[]): number {
|
|
|
325
372
|
const tags = concept.tags.map(t => t.toLowerCase());
|
|
326
373
|
let score = 0;
|
|
327
374
|
for (const t of tokens) {
|
|
328
|
-
|
|
329
|
-
if (
|
|
330
|
-
if (
|
|
331
|
-
if (
|
|
332
|
-
if (
|
|
375
|
+
let weight = 0;
|
|
376
|
+
if (title.includes(t)) weight += 5;
|
|
377
|
+
if (tags.some(tag => tag.includes(t))) weight += 3;
|
|
378
|
+
if (type.includes(t)) weight += 2;
|
|
379
|
+
if (desc.includes(t)) weight += 2;
|
|
380
|
+
if (body.includes(t)) weight += 1;
|
|
381
|
+
score += stats ? weight * tokenIdf(t, stats) : weight;
|
|
333
382
|
}
|
|
334
383
|
return score;
|
|
335
384
|
}
|
|
336
385
|
|
|
337
386
|
/** Search the bundle's concepts for a query, returning the relevant ones (score > 0)
|
|
338
|
-
* highest-score first. A type/tags/title/body keyword match all contribute
|
|
387
|
+
* highest-score first. A type/tags/title/body keyword match all contribute, scaled
|
|
388
|
+
* by corpus IDF so the most discriminating hits rank first. */
|
|
339
389
|
export function searchConcepts(concepts: Concept[], query: string): { concept: Concept; score: number }[] {
|
|
340
390
|
const tokens = tokenize(query);
|
|
391
|
+
const stats = buildCorpusStats(concepts);
|
|
341
392
|
return concepts
|
|
342
|
-
.map(concept => ({ concept, score: scoreConcept(concept, tokens) }))
|
|
393
|
+
.map(concept => ({ concept, score: scoreConcept(concept, tokens, stats) }))
|
|
343
394
|
.filter(r => r.score > 0)
|
|
344
395
|
.sort((a, b) => b.score - a.score);
|
|
345
396
|
}
|
|
346
397
|
|
|
398
|
+
/** Reciprocal Rank Fusion constant (memsearch / TREC standard k=60): dampens any
|
|
399
|
+
* single list's contribution so no one channel can dominate the fused order. */
|
|
400
|
+
const RRF_K = 60;
|
|
401
|
+
|
|
402
|
+
/** Fuse several ranked id-lists into one score map via Reciprocal Rank Fusion:
|
|
403
|
+
* each list contributes 1/(k + rank) (rank 0-based) for the ids it ranks, summed
|
|
404
|
+
* across lists. Rank-based (not score-based), so a strong lexical hit and a strong
|
|
405
|
+
* graph-proximity hit combine on equal, scale-free footing — memsearch's hybrid
|
|
406
|
+
* reranker (BM25 ⊕ dense) adapted to jeo's lexical ⊕ concept-graph channels. An id
|
|
407
|
+
* ranked in no list simply gets 0. */
|
|
408
|
+
export function reciprocalRankFusion(lists: string[][], k = RRF_K): Map<string, number> {
|
|
409
|
+
const fused = new Map<string, number>();
|
|
410
|
+
for (const list of lists) {
|
|
411
|
+
list.forEach((id, rank) => {
|
|
412
|
+
fused.set(id, (fused.get(id) ?? 0) + 1 / (k + rank));
|
|
413
|
+
});
|
|
414
|
+
}
|
|
415
|
+
return fused;
|
|
416
|
+
}
|
|
417
|
+
|
|
418
|
+
/** Rank concepts by graph proximity to the query seeds: the more distinct seeds a
|
|
419
|
+
* concept is reachable from (itself, or a 1-hop link to/from a seed), the higher it
|
|
420
|
+
* ranks. This is the local stand-in for memsearch's dense-vector channel — a
|
|
421
|
+
* strongly-connected neighbour surfaces even when its OWN lexical score is weak,
|
|
422
|
+
* the way a semantic neighbour does under dense retrieval. Only concepts reachable
|
|
423
|
+
* from ≥1 seed are returned; input order breaks ties. Uses the public graph API
|
|
424
|
+
* (expandByGraph) so it stays correct if the graph internals change. */
|
|
425
|
+
function graphProximityOrder(concepts: Concept[], seedIds: string[], graph: ConceptGraph): string[] {
|
|
426
|
+
if (seedIds.length === 0) return [];
|
|
427
|
+
const reach = new Map<string, number>();
|
|
428
|
+
for (const seed of seedIds) {
|
|
429
|
+
for (const id of expandByGraph([seed], graph, 1)) reach.set(id, (reach.get(id) ?? 0) + 1);
|
|
430
|
+
}
|
|
431
|
+
const index = new Map(concepts.map((c, i) => [conceptId(c.relPath), i] as const));
|
|
432
|
+
return [...reach.entries()]
|
|
433
|
+
.sort((a, b) => b[1] - a[1] || (index.get(a[0]) ?? 0) - (index.get(b[0]) ?? 0))
|
|
434
|
+
.map(([id]) => id);
|
|
435
|
+
}
|
|
436
|
+
|
|
437
|
+
|
|
347
438
|
/** Priority order for injection. Failure-first: a query-relevant FailedAttempt is
|
|
348
439
|
* surfaced AHEAD of everything else — resurfacing a known dead end is higher-leverage
|
|
349
440
|
* than reinforcing what already works ("못하는 게 없도록" — close the gaps, don't just
|
|
350
441
|
* polish strengths), and it is the mechanism by which the loop gets more precise the
|
|
351
442
|
* more it repeats (llm-wiki: accumulated failure knowledge sharpens future runs). Then
|
|
352
|
-
* high-confidence "core" concepts, then
|
|
353
|
-
*
|
|
354
|
-
*
|
|
355
|
-
*
|
|
356
|
-
*
|
|
443
|
+
* high-confidence "core" concepts, then HYBRID relevance.
|
|
444
|
+
*
|
|
445
|
+
* Hybrid relevance (memsearch's reranker, ported): two complementary ranked channels
|
|
446
|
+
* are fused by Reciprocal Rank Fusion instead of one raw score with a boolean boost —
|
|
447
|
+
* (1) lexical: query-hit concepts ranked by IDF-weighted score (the "sparse"/BM25
|
|
448
|
+
* channel), and
|
|
449
|
+
* (2) graph proximity: concepts reachable from the query-hit seeds via 1-hop links,
|
|
450
|
+
* ranked by how many seeds reach them (the local "dense"/semantic-neighbour
|
|
451
|
+
* channel — Sprint 04 graph expansion, now a ranked signal, not just a flag).
|
|
452
|
+
* Fusing by rank lets a strongly-linked neighbour with a weak lexical score still
|
|
453
|
+
* outrank unrelated noise, and lets a concept that is BOTH a lexical hit and a graph
|
|
454
|
+
* hub rise above a concept strong in only one channel. Input order is the stable
|
|
455
|
+
* final tiebreak. The failure boost only fires when the query actually hits the
|
|
456
|
+
* concept (score > 0), so an unrelated FailedAttempt never crowds out relevant
|
|
457
|
+
* context. */
|
|
357
458
|
function priorityOrder(concepts: Concept[], query?: string): Concept[] {
|
|
358
459
|
const tokens = tokenize(query);
|
|
359
|
-
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
|
|
364
|
-
|
|
365
|
-
|
|
460
|
+
const stats = buildCorpusStats(concepts);
|
|
461
|
+
const scored = concepts.map(c => ({ c, id: conceptId(c.relPath), score: scoreConcept(c, tokens, stats) }));
|
|
462
|
+
|
|
463
|
+
// Channel 1 — lexical ("sparse"/BM25): query-hit concepts, IDF score descending.
|
|
464
|
+
// Stable sort keeps input order on ties.
|
|
465
|
+
const lexicalOrder = scored.filter(s => s.score > 0).slice().sort((a, b) => b.score - a.score).map(s => s.id);
|
|
466
|
+
|
|
467
|
+
// Channel 2 — graph proximity ("dense"): neighbours of the query-hit seeds.
|
|
468
|
+
let graphOrder: string[] = [];
|
|
469
|
+
if (tokens.length > 0 && lexicalOrder.length > 0) {
|
|
470
|
+
graphOrder = graphProximityOrder(concepts, lexicalOrder, buildConceptGraph(concepts));
|
|
366
471
|
}
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
|
|
377
|
-
|
|
378
|
-
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
})
|
|
472
|
+
|
|
473
|
+
const fused = reciprocalRankFusion([lexicalOrder, graphOrder]);
|
|
474
|
+
|
|
475
|
+
return scored
|
|
476
|
+
.map(({ c, id, score }, i) => ({
|
|
477
|
+
concept: c,
|
|
478
|
+
i,
|
|
479
|
+
// Query-relevant past failure → resurface first so the loop is reminded of
|
|
480
|
+
// what NOT to repeat for the current task. Gated on score > 0: a stale,
|
|
481
|
+
// unrelated dead end stays out of the way.
|
|
482
|
+
failure: c.type === "FailedAttempt" && score > 0,
|
|
483
|
+
core: c.confidence === "high",
|
|
484
|
+
rrf: fused.get(id) ?? 0,
|
|
485
|
+
}))
|
|
382
486
|
.sort((a, b) => {
|
|
383
487
|
if (a.failure !== b.failure) return a.failure ? -1 : 1;
|
|
384
488
|
if (a.core !== b.core) return a.core ? -1 : 1;
|
|
385
|
-
if (b.
|
|
386
|
-
if (a.related !== b.related) return a.related ? -1 : 1;
|
|
489
|
+
if (b.rrf !== a.rrf) return b.rrf - a.rrf;
|
|
387
490
|
return a.i - b.i;
|
|
388
491
|
})
|
|
389
492
|
.map(s => s.concept);
|
|
390
493
|
}
|
|
391
494
|
|
|
495
|
+
|
|
392
496
|
/** Group items by their `type` into ordered `{ header, list }` sections: TYPE_LAYOUT
|
|
393
497
|
* order first, then any unknown types under their raw type name (lenient). The one
|
|
394
498
|
* place that encodes the section ordering — shared by render and index. */
|