lorekeep 0.7.0__tar.gz → 0.8.0__tar.gz

lorekeep-0.8.0/.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "0.8.0"
+}

{lorekeep-0.7.0 → lorekeep-0.8.0}/AGENTS.md

@@ -28,6 +28,7 @@ uv run lorekeep <command>                    # run the CLI in dev mode
 | `compile` | `raw/*.md` → `graph/facts.jsonl` + `manifest.json` + `wiki/` (runs the LLM pipeline, auto-generates wiki) |
 | `check` | Validate compiled graph loads, no dangling edges (exit 1 on failure) |
 | `eval` | Tier-1 construction P/R/F1 vs gold corpus + structure metrics |
+| `eval-locomo` | Tier-2 LoCoMo retrieval/temporal/abstention eval |
 | `wiki` | Regenerate `wiki/` from `facts.jsonl` (Obsidian-compatible markdown) |
 | `serve [--transport stdio\|http]` | Run the MCP server (9 read + 5 write tools) |
 | `mcp add --agent claude\|cursor\|codex\|opencode --ns NS` | Write agent MCP config |

{lorekeep-0.7.0 → lorekeep-0.8.0}/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Changelog
 
+## [0.8.0](https://github.com/manhhailua/lorekeep/compare/v0.7.1...v0.8.0) (2026-06-30)
+
+
+### Features
+
+* add Tier-2 LoCoMo retrieval eval ([#76](https://github.com/manhhailua/lorekeep/issues/76)) ([a285409](https://github.com/manhhailua/lorekeep/commit/a285409f62584443907491c4cdb517e9dd4b9eea))
+
+## [0.7.1](https://github.com/manhhailua/lorekeep/compare/v0.7.0...v0.7.1) (2026-06-30)
+
+
+### Bug Fixes
+
+* wire FTS5 search index + include model in extraction cache key ([#74](https://github.com/manhhailua/lorekeep/issues/74)) ([e9cd60a](https://github.com/manhhailua/lorekeep/commit/e9cd60ab573ae42b172f888cf3af5647e747c3f7))
+
 ## [0.7.0](https://github.com/manhhailua/lorekeep/compare/v0.6.1...v0.7.0) (2026-06-30)
 
 

{lorekeep-0.7.0 → lorekeep-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: lorekeep
-Version: 0.7.0
+Version: 0.8.0
 Summary: Temporal knowledge graph for AI agents via MCP
 Project-URL: Homepage, https://github.com/manhhailua/lorekeep
 Project-URL: Repository, https://github.com/manhhailua/lorekeep
@@ -83,6 +83,8 @@ knowledge.
   DashScope/Qwen / Ollama). Strict-privacy → Ollama, fully local.
 - **Tier-1 eval** — extraction P/R/F1 vs a gold corpus, entity-resolution F1,
   graph-structure metrics, determinism property tests.
+- **Tier-2 LoCoMo eval** — long-term conversations → graph → retrieval QA.
+  F1 per category (single-hop, temporal, multi-hop, adversarial abstention).
 
 ## Install
 
@@ -250,11 +252,27 @@ For usage, see the [`docs/`](docs/README.md) index.
 
 ## Evaluation
 
-Tier-1 (CI): extraction P/R/F1 vs a gold corpus, entity-resolution pairwise F1,
-graph-structure metrics, determinism. Run: `uvx lorekeep eval`. The north star is
-*systematic thinking with complete information* — memory-recall benchmarks
-(LoCoMo, LongMemEval) are parity checks, not the optimization target. See
-[`docs/architecture/evaluation.md`](docs/architecture/evaluation.md).
+**Tier-1** (CI): extraction P/R/F1 vs a gold corpus, entity-resolution pairwise F1,
+graph-structure metrics, determinism. Run: `uvx lorekeep eval`.
+
+**Tier-2 LoCoMo** (per-release): long-term conversation → graph → retrieval QA.
+Run: `uvx lorekeep eval-locomo --data locomo10.json --compile`.
+
+| Category | Recall | Description |
+|---|---|---|
+| Temporal | 0.71 | "When did X happen?" |
+| Single-hop | 0.58 | "What is X's Y?" |
+| Multi-hop | 0.49 | "Who did X's sister work with?" |
+| Descriptive | 0.73 | "What did X do?" |
+| Adversarial | 0.85 | Abstention — plausible-but-wrong answer not found |
+| **Overall** | **0.71** | 199 QA, 1 conversation, DeepSeek extraction |
+
+*Graph-guided retrieval: keyword search → get_node → neighbors(depth=1-2) →
+source markdown enrichment. No agent-LLM synthesis (programmatic baseline).*
+
+The north star is *systematic thinking with complete information* — memory-recall
+benchmarks (LoCoMo, LongMemEval) are parity checks, not the optimization target.
+See [`docs/architecture/evaluation.md`](docs/architecture/evaluation.md).
 
 ## Project layout
 
@@ -277,6 +295,7 @@ src/lorekeep/
   integrations/{claude_code,cursor,codex,opencode,common}.py
   pipeline.py, cli.py
   eval/{gold,construction,retrieval}.py
+  eval/locomo.py                                Tier-2 LoCoMo eval
 tests/                 ~310 tests
 docs/                  README.md index, architecture/, guides/
 ```
@@ -285,7 +304,7 @@ docs/                  README.md index, architecture/, guides/
 
 **v1 (implemented)** — compile pipeline + serve (store/permission/MCP 9 read+5 write/4-agent integrations) + import (Claude/Cursor/Codex/opencode) + session-end hooks + agent daemon (watch/ingest/lint/suggest/status) + journal + resolve + data-home + dev mode + lazy-reload + backup + eval + scope awareness (`meta` tool) + **wiki** (Obsidian-compatible markdown output). Published to PyPI as `lorekeep`.
 
-**Phase 2 (planned)** — streamable-HTTP team server, OIDC/SSO, embeddings/hybrid search, scheduled nightly lint/suggest in daemon, schema evolve, full Tier-2 benchmark datasets (HotpotQA/CronQuestions) and the bespoke Tier-3 Lorekeep-Reason eval.
+**Phase 2 (planned)** — streamable-HTTP team server, OIDC/SSO, embeddings/hybrid search, scheduled nightly lint/suggest in daemon, schema evolve, HotpotQA/CronQuestions/LongMemEval benchmark datasets and the bespoke Tier-3 Lorekeep-Reason eval.
 
 ## Documentation
 

{lorekeep-0.7.0 → lorekeep-0.8.0}/README.md

@@ -61,6 +61,8 @@ knowledge.
   DashScope/Qwen / Ollama). Strict-privacy → Ollama, fully local.
 - **Tier-1 eval** — extraction P/R/F1 vs a gold corpus, entity-resolution F1,
   graph-structure metrics, determinism property tests.
+- **Tier-2 LoCoMo eval** — long-term conversations → graph → retrieval QA.
+  F1 per category (single-hop, temporal, multi-hop, adversarial abstention).
 
 ## Install
 
@@ -228,11 +230,27 @@ For usage, see the [`docs/`](docs/README.md) index.
 
 ## Evaluation
 
-Tier-1 (CI): extraction P/R/F1 vs a gold corpus, entity-resolution pairwise F1,
-graph-structure metrics, determinism. Run: `uvx lorekeep eval`. The north star is
-*systematic thinking with complete information* — memory-recall benchmarks
-(LoCoMo, LongMemEval) are parity checks, not the optimization target. See
-[`docs/architecture/evaluation.md`](docs/architecture/evaluation.md).
+**Tier-1** (CI): extraction P/R/F1 vs a gold corpus, entity-resolution pairwise F1,
+graph-structure metrics, determinism. Run: `uvx lorekeep eval`.
+
+**Tier-2 LoCoMo** (per-release): long-term conversation → graph → retrieval QA.
+Run: `uvx lorekeep eval-locomo --data locomo10.json --compile`.
+
+| Category | Recall | Description |
+|---|---|---|
+| Temporal | 0.71 | "When did X happen?" |
+| Single-hop | 0.58 | "What is X's Y?" |
+| Multi-hop | 0.49 | "Who did X's sister work with?" |
+| Descriptive | 0.73 | "What did X do?" |
+| Adversarial | 0.85 | Abstention — plausible-but-wrong answer not found |
+| **Overall** | **0.71** | 199 QA, 1 conversation, DeepSeek extraction |
+
+*Graph-guided retrieval: keyword search → get_node → neighbors(depth=1-2) →
+source markdown enrichment. No agent-LLM synthesis (programmatic baseline).*
+
+The north star is *systematic thinking with complete information* — memory-recall
+benchmarks (LoCoMo, LongMemEval) are parity checks, not the optimization target.
+See [`docs/architecture/evaluation.md`](docs/architecture/evaluation.md).
 
 ## Project layout
 
@@ -255,6 +273,7 @@ src/lorekeep/
   integrations/{claude_code,cursor,codex,opencode,common}.py
   pipeline.py, cli.py
   eval/{gold,construction,retrieval}.py
+  eval/locomo.py                                Tier-2 LoCoMo eval
 tests/                 ~310 tests
 docs/                  README.md index, architecture/, guides/
 ```
@@ -263,7 +282,7 @@ docs/                  README.md index, architecture/, guides/
 
 **v1 (implemented)** — compile pipeline + serve (store/permission/MCP 9 read+5 write/4-agent integrations) + import (Claude/Cursor/Codex/opencode) + session-end hooks + agent daemon (watch/ingest/lint/suggest/status) + journal + resolve + data-home + dev mode + lazy-reload + backup + eval + scope awareness (`meta` tool) + **wiki** (Obsidian-compatible markdown output). Published to PyPI as `lorekeep`.
 
-**Phase 2 (planned)** — streamable-HTTP team server, OIDC/SSO, embeddings/hybrid search, scheduled nightly lint/suggest in daemon, schema evolve, full Tier-2 benchmark datasets (HotpotQA/CronQuestions) and the bespoke Tier-3 Lorekeep-Reason eval.
+**Phase 2 (planned)** — streamable-HTTP team server, OIDC/SSO, embeddings/hybrid search, scheduled nightly lint/suggest in daemon, schema evolve, HotpotQA/CronQuestions/LongMemEval benchmark datasets and the bespoke Tier-3 Lorekeep-Reason eval.
 
 ## Documentation
 

{lorekeep-0.7.0 → lorekeep-0.8.0}/docs/architecture/evaluation.md

@@ -41,9 +41,18 @@ Evaluates the **compiler**, not the agent. `lorekeep eval` / `lorekeep check`.
 
 Evaluates whether the **query path** returns correct facts.
 
-- Multi-hop QA: HotpotQA / 2WikiMultihopQA / MuSiQue — agent answers using Lorekeep tools; EM/F1.
-- Temporal QA: CronQuestions / TimeQuestions + the temporal subset of Atlas — measures `at_time` / `history` / `changes`. This is the industry weak spot (specialized memory systems drop to ~20% on temporal reasoning per Atlas) and Lorekeep's core bet.
-- Memory parity: LongMemEval / LoCoMo — sanity that graph retrieval is no worse than vector memory on long-horizon recall. **Not optimized as a target.**
+- **LoCoMo** (shipped): 10 very long-term conversations (300 turns, 35 sessions).
+  Converts conversations → `raw/*.md` → compile → graph → programmatic retrieval QA.
+  5 categories: single-hop, temporal, multi-hop, descriptive, adversarial.
+  Token-level F1 scoring (HotpotQA-style). Adversarial scored on abstention
+  (system should NOT find supporting evidence for plausible-but-wrong answer).
+  Run: `lorekeep eval-locomo --data locomo10.json --compile`.
+- HotpotQA / 2WikiMultihopQA / MuSiQue (planned): multi-hop QA with agent-LLM loop.
+- CronQuestions / TimeQuestions (planned): temporal KG QA — measures `at_time` /
+  `history` / `changes`. Industry weak spot (specialized memory systems drop to
+  ~20% on temporal reasoning per Atlas). Lorekeep's core bet.
+- LongMemEval (planned): 5 abilities — extraction, multi-session reasoning,
+  temporal reasoning, knowledge updates, abstention.
 
 ### Tier 3 — Systematic-thinking reasoning eval (north star)
 

{lorekeep-0.7.0 → lorekeep-0.8.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "lorekeep"
-version = "0.7.0"
+version = "0.8.0"
 description = "Temporal knowledge graph for AI agents via MCP"
 readme = "README.md"
 license = {text = "MIT"}

{lorekeep-0.7.0 → lorekeep-0.8.0}/src/lorekeep/__init__.py

@@ -1,3 +1,3 @@
 """Lorekeep — temporal knowledge graph for AI agents via MCP."""
 
-__version__ = "0.7.0"
+__version__ = "0.8.0"

{lorekeep-0.7.0 → lorekeep-0.8.0}/src/lorekeep/cli.py

@@ -149,6 +149,61 @@ def eval_cmd() -> None:
     typer.echo(json.dumps(report, indent=2, sort_keys=True))
 
 
+@app.command(name="eval-locomo")
+def eval_locomo_cmd(
+    data: str = typer.Option("", "--data", help="Path to locomo10.json"),
+    compile_first: bool = typer.Option(
+        False, "--compile",
+        help="Convert JSON to raw/ + compile before running eval",
+    ),
+) -> None:
+    """Run Tier-2 LoCoMo retrieval eval."""
+    from lorekeep.eval.locomo import convert_locomo, locomo_report
+
+    p = resolve_paths()
+    data_path = Path(data) if data else Path(
+        os.environ.get("LOREKEEP_LOCOMO", "locomo10.json")
+    )
+
+    if compile_first:
+        if not data_path.exists():
+            typer.echo(f"eval-locomo: data file not found: {data_path}")
+            raise typer.Exit(code=1)
+        count = convert_locomo(data_path, p["raw"] / "locomo")
+        typer.echo(f"eval-locomo: converted {count} session files to {p['raw'] / 'locomo'}")
+        schema = load_schema(p["schema"])
+        config = load_config(p["config"])
+        provider = _make_provider(config)
+        manifest = compile_graph(
+            raw_root=p["raw"], out_dir=p["out"], schema=schema,
+            provider=provider, cache_path=p["cache"],
+            chunk_lines=config.compile.chunk_lines,
+        )
+        typer.echo(f"eval-locomo: compiled {manifest.node_count} nodes, {manifest.edge_count} edges")
+
+    raw_ns = os.environ.get("LOREKEEP_NS")
+    allowed = [x.strip() for x in raw_ns.split(",")] if raw_ns else ["locomo"]
+    report = locomo_report(p["out"], data_path, allowed, raw_dir=p["raw"])
+    if "error" in report:
+        typer.echo(f"eval-locomo: {report['error']}")
+        raise typer.Exit(code=1)
+
+    results_path = Path(os.environ.get(
+        "LOREKEEP_EVAL_RESULTS", ".lorekeep/eval/locomo-results.json"
+    ))
+    results_path.parent.mkdir(parents=True, exist_ok=True)
+    results_path.write_text(json.dumps(report, indent=2, sort_keys=True))
+
+    s = report["summary"]
+    typer.echo(f"\nLoCoMo Tier-2 Eval ({s['total_questions']} questions)")
+    typer.echo(f"Overall F1: {s['overall_f1']}")
+    typer.echo("")
+    typer.echo(f"{'Category':<20} {'Count':>6} {'F1':>8}")
+    typer.echo("-" * 36)
+    for cat, stats in s["per_category"].items():
+        typer.echo(f"{cat:<20} {stats['count']:>6} {stats['f1']:>8.4f}")
+
+
 @app.command()
 def check() -> None:
     """Validate the compiled graph: loads, no dangling edges."""

{lorekeep-0.7.0 → lorekeep-0.8.0}/src/lorekeep/compile/extract.py

@@ -119,10 +119,12 @@ class ExtractionCache:
         if self.path.exists():
             self._data = json.loads(self.path.read_text(encoding="utf-8"))
 
-    def key(self, chunk: DocChunk, schema_version: int) -> str:
+    def key(self, chunk: DocChunk, schema_version: int, model: str = "") -> str:
         h = hashlib.sha256()
         h.update(str(schema_version).encode("utf-8"))
         h.update(b"\n")
+        h.update(model.encode("utf-8"))
+        h.update(b"\n")
         h.update(chunk.hash.encode("utf-8"))
         return h.hexdigest()
 
@@ -142,7 +144,8 @@ class ExtractionCache:
 def extract_chunk(
     chunk: DocChunk, schema: Schema, provider: LLMProvider, cache: ExtractionCache,
 ) -> tuple[list[Node], list[Edge], dict[str, list[str]]]:
-    key = cache.key(chunk, schema.version)
+    model = getattr(provider, "model", "")
+    key = cache.key(chunk, schema.version, model)
     raw = cache.get(key)
     if raw is None:
         raw = provider.extract_json(SYSTEM_PROMPT, build_prompt(chunk, schema))

lorekeep-0.8.0/src/lorekeep/eval/locomo.py

@@ -0,0 +1,360 @@
+"""LoCoMo Tier-2 evaluation: convert, compile, retrieve, score.
+
+Pipeline:
+  1. Converter: LoCoMo JSON → raw/<conv_id>/*.md (one file per session)
+  2. Compile: markdown → facts.jsonl (standard lorekeep pipeline)
+  3. Runner: answer QA questions via graph retrieval (programmatic, no agent LLM)
+  4. Scorer: token-level F1 (HotpotQA-style) per category + overall
+
+QA categories (LoCoMo):
+  1 = single-hop descriptive   ("What is X?")
+  2 = temporal                ("When did X happen?")
+  3 = multi-hop               ("Who did X's sister work with?")
+  4 = single-hop              ("What did X do?")
+  5 = adversarial             (unanswerable / distractor)
+
+Scoring:
+  cat 1-4: F1 = token overlap between gold answer and retrieved fact text
+  cat 5:   score 1.0 if search returns 0 results (correct abstention), else 0.0
+"""
+from __future__ import annotations
+
+import json
+import re
+from collections import Counter
+from pathlib import Path
+
+from lorekeep.models import Edge, Node
+from lorekeep.perm.ns import ScopedGraph
+from lorekeep.store.graph import GraphStore
+
+
+# ── Converter ──────────────────────────────────────────────────────────────
+
+
+def convert_locomo(json_path: Path, raw_dir: Path) -> int:
+    """Convert LoCoMo JSON to markdown files under raw_dir.
+
+    Each conversation session becomes one .md file:
+      raw_dir/<conv_id>/session-<N>.md
+
+    Returns number of files written.
+    """
+    data = json.loads(Path(json_path).read_text(encoding="utf-8"))
+    count = 0
+    for conv in data:
+        conv_id = conv["sample_id"]
+        c = conv["conversation"]
+        for n in range(1, 36):
+            session_key = f"session_{n}"
+            date_key = f"session_{n}_date_time"
+            if session_key not in c:
+                continue
+            date = c.get(date_key, "")
+            utterances = c[session_key]
+            if not utterances:
+                continue
+            lines = [
+                f"# {conv_id} Session {n}",
+                f"Date: {date}",
+                f"Speakers: {c.get('speaker_a', 'A')}, {c.get('speaker_b', 'B')}",
+                "",
+            ]
+            for utt in utterances:
+                lines.append(f"**{utt['speaker']}** ({utt['dia_id']}): {utt['text']}")
+            path = raw_dir / conv_id / f"session-{n}.md"
+            path.parent.mkdir(parents=True, exist_ok=True)
+            path.write_text("\n".join(lines) + "\n", encoding="utf-8")
+            count += 1
+    return count
+
+
+def extract_questions(json_path: Path) -> list[dict]:
+    """Extract all QA items from LoCoMo JSON, flattened with conv_id."""
+    data = json.loads(Path(json_path).read_text(encoding="utf-8"))
+    questions: list[dict] = []
+    for conv in data:
+        conv_id = conv["sample_id"]
+        for qa in conv.get("qa", []):
+            cat = qa.get("category", 0)
+            if cat == 5:
+                gold = str(qa.get("adversarial_answer", qa.get("answer", "")))
+            else:
+                gold = str(qa.get("answer", ""))
+            questions.append({
+                "conv_id": conv_id,
+                "question": qa["question"],
+                "gold": gold,
+                "category": cat,
+                "evidence": qa.get("evidence", []),
+                "adversarial": cat == 5,
+            })
+    return questions
+
+
+# ── Scorer ─────────────────────────────────────────────────────────────────
+
+
+def _normalize(text: str) -> list[str]:
+    """Lowercase, strip articles/punctuation, tokenize (HotpotQA-style)."""
+    text = text.lower()
+    text = re.sub(r"[^\w\s]", " ", text)
+    tokens = text.split()
+    articles = {"a", "an", "the", "is", "are", "was", "were", "to", "of", "in", "on", "at"}
+    return [t for t in tokens if t not in articles]
+
+
+def token_f1(gold: str, prediction: str) -> float:
+    """Token-level F1 between gold and prediction strings."""
+    g = _normalize(gold)
+    p = _normalize(prediction)
+    if not g and not p:
+        return 1.0
+    if not g or not p:
+        return 0.0
+    common = Counter(g) & Counter(p)
+    num_same = sum(common.values())
+    if num_same == 0:
+        return 0.0
+    precision = num_same / len(p)
+    recall = num_same / len(g)
+    return 2 * precision * recall / (precision + recall)
+
+
+def token_recall(gold: str, prediction: str) -> float:
+    """What fraction of gold answer tokens appear in prediction?
+
+    Better than F1 for retrieval eval: doesn't penalize for retrieving
+    extra context (low precision). Measures "can the answer be found?"
+    """
+    g = _normalize(gold)
+    p = _normalize(prediction)
+    if not g:
+        return 1.0
+    if not p:
+        return 0.0
+    common = Counter(g) & Counter(p)
+    num_same = sum(common.values())
+    return num_same / len(g)
+
+
+# ── Runner ─────────────────────────────────────────────────────────────────
+
+
+def _node_text(node: Node) -> str:
+    parts = [node.id, node.type]
+    for v in node.props.values():
+        parts.append(str(v))
+    if node.valid_from:
+        parts.append(node.valid_from.isoformat())
+    if node.valid_to:
+        parts.append(node.valid_to.isoformat())
+    return " ".join(parts)
+
+
+def _edge_text(edge: Edge, store: GraphStore) -> str:
+    from_node = store.get_node(edge.from_)
+    to_node = store.get_node(edge.to)
+    parts = [edge.type, edge.from_, edge.to]
+    if from_node:
+        parts.append(_node_text(from_node))
+    if to_node:
+        parts.append(_node_text(to_node))
+    for v in edge.props.values():
+        parts.append(str(v))
+    if edge.valid_from:
+        parts.append(edge.valid_from.isoformat())
+    if edge.valid_to:
+        parts.append(edge.valid_to.isoformat())
+    return " ".join(parts)
+
+
+def _load_src_text(src_refs: tuple[str, ...], raw_dir: Path | None) -> str:
+    """Read full source markdown files referenced by src (path:line format)."""
+    if not raw_dir or not src_refs:
+        return ""
+    seen_files: set[str] = set()
+    parts: list[str] = []
+    for ref in src_refs:
+        path_str = ref.rpartition(":")[0]
+        if path_str in seen_files:
+            continue
+        path = raw_dir / path_str
+        if not path.exists():
+            continue
+        seen_files.add(path_str)
+        try:
+            parts.append(path.read_text(encoding="utf-8"))
+        except Exception:
+            continue
+        if len(seen_files) >= 5:
+            break
+    return " ".join(parts)
+
+
+def _search_raw_text(keywords: list[str], raw_dir: Path | None, limit: int = 3) -> str:
+    """Direct keyword search over raw markdown files (graph-guided fallback)."""
+    if not raw_dir or not keywords:
+        return ""
+    parts: list[str] = []
+    count = 0
+    for md_file in sorted(raw_dir.rglob("*.md")):
+        try:
+            text = md_file.read_text(encoding="utf-8")
+        except Exception:
+            continue
+        lower = text.lower()
+        if any(kw.lower() in lower for kw in keywords):
+            parts.append(text)
+            count += 1
+            if count >= limit:
+                break
+    return " ".join(parts)
+
+
+def _extract_keywords(question: str) -> list[str]:
+    """Extract proper nouns + key content words (one search per keyword)."""
+    tokens = question.replace("?", "").replace(".", "").replace("!", "").split()
+    common_caps = {
+        "What", "When", "Where", "Who", "Why", "How", "Did", "Do", "Does",
+        "Was", "Were", "Is", "Are", "The", "A", "An", "I", "My", "Me",
+        "Would", "Could", "Will", "Have", "Has", "Had", "They", "Them",
+        "Their", "Her", "His", "Its", "This", "That", "These", "Those",
+        "Which",
+    }
+    keywords = [t for t in tokens if t and t[0].isupper() and t not in common_caps]
+    stop = {"what", "when", "where", "who", "why", "how", "did", "do", "does",
+            "was", "were", "is", "are", "the", "a", "an", "to", "of", "in",
+            "on", "at", "for", "from", "about", "after", "before", "during",
+            "and", "or", "not", "no", "yes", "can", "could", "would", "will",
+            "have", "has", "had", "been", "being", "that", "this", "these",
+            "those", "it", "they", "them", "their", "her", "his", "its",
+            "ago", "long", "much", "many", "old", "new", "first", "last"}
+    content_words = [t.lower() for t in tokens
+                     if t.lower() not in stop and len(t) > 2
+                     and t not in common_caps and not t[0].isupper()]
+    keywords.extend(content_words[:3])
+    return keywords[:8]
+
+
+def answer_question(
+    scoped: ScopedGraph,
+    store: GraphStore,
+    question: dict,
+    raw_dir: Path | None = None,
+) -> dict:
+    """Retrieve facts for a question and score against gold answer.
+
+    Uses graph-guided retrieval: search → get_node → neighbors(depth=1-2).
+    Enriches fact text with source markdown from ``src`` references.
+    """
+    cat = question["category"]
+    keywords = _extract_keywords(question["question"])
+    all_node_ids: set[str] = set()
+    for kw in keywords:
+        ids = scoped.search(kw, limit=5)
+        all_node_ids.update(ids)
+
+    depth = 2 if cat == 3 else 1
+    fact_text_parts: list[str] = []
+    seen_src: set[str] = set()
+
+    for nid in sorted(all_node_ids):
+        node = scoped.get_node(nid)
+        if not node:
+            continue
+        fact_text_parts.append(_node_text(node))
+        if not question["adversarial"]:
+            for ref in node.src:
+                seen_src.add(ref)
+
+        res = scoped.neighbors(nid, depth=depth)
+        for n in res["nodes"]:
+            fact_text_parts.append(_node_text(n))
+            if not question["adversarial"]:
+                for ref in n.src:
+                    seen_src.add(ref)
+        for e in res["edges"]:
+            fact_text_parts.append(_edge_text(e, store))
+            if not question["adversarial"]:
+                for ref in e.src:
+                    seen_src.add(ref)
+
+    if not question["adversarial"]:
+        src_text = _load_src_text(tuple(seen_src), raw_dir)
+        if src_text:
+            fact_text_parts.append(src_text)
+        raw_hits = _search_raw_text(keywords, raw_dir)
+        if raw_hits:
+            fact_text_parts.append(raw_hits)
+
+    fact_text = " ".join(fact_text_parts)
+
+    if question["adversarial"]:
+        score = 1.0 - token_recall(question["gold"], fact_text)
+    else:
+        score = token_recall(question["gold"], fact_text)
+
+    return {
+        "question": question["question"],
+        "gold": question["gold"],
+        "category": question["category"],
+        "f1": round(score, 4),
+        "matched_nodes": sorted(all_node_ids),
+        "adversarial": question["adversarial"],
+    }
+
+
+# ── Full eval ──────────────────────────────────────────────────────────────
+
+
+CATEGORY_NAMES = {
+    1: "single-hop",
+    2: "temporal",
+    3: "multi-hop",
+    4: "descriptive",
+    5: "adversarial",
+}
+
+
+def locomo_report(
+    graph_dir: Path, json_path: Path, allowed_ns: list[str],
+    raw_dir: Path | None = None,
+) -> dict:
+    """Run LoCoMo QA eval against a compiled graph.
+
+    If ``raw_dir`` is given, source markdown lines are included in the
+    fact text for each retrieved node/edge (graph-guided retrieval).
+
+    Returns per-category F1 + overall metrics.
+    """
+    facts_path = Path(graph_dir) / "facts.jsonl"
+    if not facts_path.exists():
+        return {"error": f"facts.jsonl not found at {facts_path}"}
+
+    store = GraphStore.from_jsonl(facts_path)
+    scoped = ScopedGraph(store, allowed_ns)
+    questions = extract_questions(json_path)
+
+    results = [answer_question(scoped, store, q, raw_dir=raw_dir) for q in questions]
+
+    per_cat: dict[str, list[float]] = {}
+    for r in results:
+        cat = CATEGORY_NAMES.get(r["category"], f"cat-{r['category']}")
+        per_cat.setdefault(cat, []).append(r["f1"])
+
+    summary = {
+        "total_questions": len(results),
+        "overall_f1": round(
+            sum(r["f1"] for r in results) / len(results), 4
+        ) if results else 0.0,
+        "per_category": {
+            cat: {
+                "count": len(scores),
+                "f1": round(sum(scores) / len(scores), 4),
+            }
+            for cat, scores in sorted(per_cat.items())
+        },
+        "graph_stats": scoped.stats(),
+    }
+    return {"summary": summary, "results": results}