scriptoria 0.5.2__tar.gz → 0.6.1__tar.gz

{scriptoria-0.5.2 → scriptoria-0.6.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: scriptoria
-Version: 0.5.2
+Version: 0.6.1
 Summary: Deterministic scriptorium-keeper (the `scrip` CLI): staleness, provenance integrity, and fact queries for an agent-compiled knowledge base
 Project-URL: Homepage, https://github.com/coredipper/scriptorium
 Project-URL: Changelog, https://github.com/coredipper/scriptorium/blob/main/CHANGELOG.md
@@ -53,11 +53,25 @@ uv run --project scrip scrip --help
 |---|---|
 | `scrip status` | Report `STALE` / `OK` / `UNCOMPILED` artifacts from the dependency graph. `--no-cache` recomputes from files; `--rebuild-manifest` regenerates the cache. |
 | `scrip verify` | Check every provenance anchor still resolves to text in its source; check referenced sources exist and `claim_id`s are unique. Fails on `BROKEN` and `AMBIGUOUS` by default; `--allow-ambiguous` downgrades `AMBIGUOUS` to a warning. |
-| `scrip query [claims\|entities\|edges\|contradictions]` | Structured query over `vault/facts/*.ndjson` via DuckDB. `--sql "<duckdb>"`, `--where`, `--limit`. |
-| `scrip index` | v0 stub for the embeddings retrieval rung. |
+| `scrip query [claims\|entities\|edges\|contradictions\|reconciliations]` | Structured query over `vault/facts/*.ndjson` via DuckDB. `--sql "<duckdb>"`, `--where`, `--limit`. |
+| `scrip ingest <url\|file>` | Fetch/read a source and write canonical `vault/raw/<slug>.md` plus sidecar metadata. HTML/PDF need the optional `[ingest]` extra. |
+| `scrip new concept\|entity <slug> --from raw/...` | Scaffold a derived wiki page for the agent to fill. |
+| `scrip anchor "<quote>" --source raw/<slug>` | Mint a content-anchored footnote for a verbatim source quote. |
+| `scrip fact add --table claims\|entities\|edges\|reconciliations` | Validate and append facts under the write lock; claims mint verified anchors, reconciliations mint ids/timestamps. |
+| `scrip span --claim <id>` | Resolve a claim anchor and print the cited span. |
+| `scrip similar --title ... --from ...` | Score overlap with existing wiki pages before PROMOTE. |
+| `scrip search "<question>"` | Retrieve source blocks for a miss; uses embeddings if an index exists, otherwise lexical grep. Add `--long-docs pageindex` to try the PageIndex cache first. |
+| `scrip index` | Build the optional embeddings index over `vault/raw/` when `scriptoria[embeddings]` is installed; otherwise exits cleanly and `search` falls back to grep. |
+| `scrip pageindex build raw/<slug>` | Build the optional PageIndex cache for one raw source when a compatible backend is importable; otherwise exits cleanly. |
+| `scrip pageindex search "<question>" [--source raw/<slug>]` | Search cached PageIndex sections, returning only snippets mapped back to canonical `vault/raw/` text. |
+| `scrip watch` / `scrip unlock` | Watch vault health in a poll loop; clear a stale advisory write lock. |
 
 Every command accepts `--root DIR` and `--json`.
 
+The optional `scrip-harness` package adds the model-bearing commands, including
+`scrip-harness answer "<question>"`, which gathers evidence with these primitives
+and verifies every citation before printing or saving an answer.
+
 ## Exit codes
 
 | Code | Meaning |

scriptoria-0.6.1/README.md

@@ -0,0 +1,54 @@
+# scrip
+
+The deterministic keeper of a [scriptorium](../README.md) knowledge base.
+
+The agent owns judgment (synthesis, fact extraction, reconciliation, promotion).
+`scrip` owns only what LLMs are unreliable or expensive at: **content hashing**,
+**staleness** over a dependency graph, **provenance-anchor integrity**, and
+**structured queries** over the facts layer. Files are the source of truth;
+`.kb/manifest.json` is only a regenerable speed cache.
+
+## Install
+
+```sh
+uv tool install ./scrip      # installs the `scrip` command
+# or, without installing:
+uv run --project scrip scrip --help
+```
+
+## Commands
+
+| Command | What it does |
+|---|---|
+| `scrip status` | Report `STALE` / `OK` / `UNCOMPILED` artifacts from the dependency graph. `--no-cache` recomputes from files; `--rebuild-manifest` regenerates the cache. |
+| `scrip verify` | Check every provenance anchor still resolves to text in its source; check referenced sources exist and `claim_id`s are unique. Fails on `BROKEN` and `AMBIGUOUS` by default; `--allow-ambiguous` downgrades `AMBIGUOUS` to a warning. |
+| `scrip query [claims\|entities\|edges\|contradictions\|reconciliations]` | Structured query over `vault/facts/*.ndjson` via DuckDB. `--sql "<duckdb>"`, `--where`, `--limit`. |
+| `scrip ingest <url\|file>` | Fetch/read a source and write canonical `vault/raw/<slug>.md` plus sidecar metadata. HTML/PDF need the optional `[ingest]` extra. |
+| `scrip new concept\|entity <slug> --from raw/...` | Scaffold a derived wiki page for the agent to fill. |
+| `scrip anchor "<quote>" --source raw/<slug>` | Mint a content-anchored footnote for a verbatim source quote. |
+| `scrip fact add --table claims\|entities\|edges\|reconciliations` | Validate and append facts under the write lock; claims mint verified anchors, reconciliations mint ids/timestamps. |
+| `scrip span --claim <id>` | Resolve a claim anchor and print the cited span. |
+| `scrip similar --title ... --from ...` | Score overlap with existing wiki pages before PROMOTE. |
+| `scrip search "<question>"` | Retrieve source blocks for a miss; uses embeddings if an index exists, otherwise lexical grep. Add `--long-docs pageindex` to try the PageIndex cache first. |
+| `scrip index` | Build the optional embeddings index over `vault/raw/` when `scriptoria[embeddings]` is installed; otherwise exits cleanly and `search` falls back to grep. |
+| `scrip pageindex build raw/<slug>` | Build the optional PageIndex cache for one raw source when a compatible backend is importable; otherwise exits cleanly. |
+| `scrip pageindex search "<question>" [--source raw/<slug>]` | Search cached PageIndex sections, returning only snippets mapped back to canonical `vault/raw/` text. |
+| `scrip watch` / `scrip unlock` | Watch vault health in a poll loop; clear a stale advisory write lock. |
+
+Every command accepts `--root DIR` and `--json`.
+
+The optional `scrip-harness` package adds the model-bearing commands, including
+`scrip-harness answer "<question>"`, which gathers evidence with these primitives
+and verifies every citation before printing or saving an answer.
+
+## Exit codes
+
+| Code | Meaning |
+|---|---|
+| `0` | clean / success |
+| `1` | actionable finding (stale artifacts; broken citations) |
+| `2` | usage error |
+| `3` | data error (malformed frontmatter / NDJSON; missing source; duplicate id) |
+| `4` | internal error |
+
+Code `1` is an *expected* outcome the agent branches on, not a crash.

{scriptoria-0.5.2 → scriptoria-0.6.1}/pyproject.toml

@@ -2,7 +2,7 @@
 # Distribution name on PyPI is `scriptoria` (scrip/scriptorium were taken); the
 # CLI command and the import package both remain `scrip`.
 name = "scriptoria"
-version = "0.5.2"
+version = "0.6.1"
 description = "Deterministic scriptorium-keeper (the `scrip` CLI): staleness, provenance integrity, and fact queries for an agent-compiled knowledge base"
 readme = "README.md"
 requires-python = ">=3.10"

{scriptoria-0.5.2 → scriptoria-0.6.1}/src/scrip/__init__.py

@@ -13,7 +13,7 @@ from __future__ import annotations
 
 from pathlib import Path
 
-__version__ = "0.5.2"
+__version__ = "0.6.1"
 
 # --- canonical vault layout ------------------------------------------------
 # ``root`` is the repo/instance root: the directory containing ``vault/``.

{scriptoria-0.5.2 → scriptoria-0.6.1}/src/scrip/blocks.py

@@ -35,17 +35,13 @@ def _is_blank(line: str) -> bool:
 
 def split_blocks(text: str) -> list[dict]:
     """Return a list of ``{"block_id", "span": [start, end], "hash"}``."""
-    # Index every line with its [start, end) char offsets.
-    spans: list[tuple[int, int, str]] = []
+    # ⚡ Bolt: Single pass block segmentation to avoid intermediate O(N) allocation
+    ranges: list[list[int]] = []  # [start, end] per block
+    cur: list[int] | None = None
     start = 0
+
     for line in text.splitlines(keepends=True):
         end = start + len(line)
-        spans.append((start, end, line))
-        start = end
-
-    ranges: list[list[int]] = []  # [start, end] per block
-    cur: list[int] | None = None
-    for s, e, line in spans:
         if _is_blank(line):
             if cur is not None:
                 ranges.append(cur)
@@ -54,12 +50,14 @@ def split_blocks(text: str) -> list[dict]:
             if cur is not None:
                 ranges.append(cur)
                 cur = None
-            ranges.append([s, e])  # a heading is its own block
+            ranges.append([start, end])  # a heading is its own block
         else:
             if cur is None:
-                cur = [s, e]
+                cur = [start, end]
             else:
-                cur[1] = e
+                cur[1] = end
+        start = end
+
     if cur is not None:
         ranges.append(cur)
 

{scriptoria-0.5.2 → scriptoria-0.6.1}/src/scrip/cli.py

@@ -224,15 +224,72 @@ def cmd_search(args: argparse.Namespace) -> int:
     from . import retrieval
 
     root = resolve_root(args.root)
-    out = retrieval.search(root, args.query, k=args.k)
+    out = retrieval.search(root, args.query, k=args.k, long_docs=args.long_docs)
     if args.json:
         _emit(out)
     else:
-        tag = out["method"] + (" — INDEX STALE, rebuild with `scrip index`" if out["stale_index"] else "")
+        stale_hint = "INDEX STALE, rebuild with `scrip index`"
+        if out["method"] == "pageindex":
+            stale_hint = "INDEX STALE, rebuild with `scrip pageindex build raw/<slug>`"
+        tag = out["method"] + (f" - {stale_hint}" if out["stale_index"] else "")
         print(f"[{tag}] top {len(out['results'])} for: {args.query!r}")
         for r in out["results"]:
-            score = f"{r['score']:.3f}" if isinstance(r["score"], float) else str(r["score"])
-            print(f"  {r['source_id']}#{r['block_id']}  ({score})")
+            score_value = r.get("score", "")
+            score = f"{score_value:.3f}" if isinstance(score_value, float) else str(score_value)
+            locator = r.get("block_id") or r.get("section_id") or "?"
+            print(f"  {r['source_id']}#{locator}  ({score})")
+            print(f"    {r.get('snippet', '')}")
+        if not out["results"]:
+            print("  (no matches)")
+    return 0
+
+
+def cmd_pageindex_build(args: argparse.Namespace) -> int:
+    from . import pageindex_adapter
+
+    root = resolve_root(args.root)
+    out = pageindex_adapter.build_index(root, args.source)
+    if args.json:
+        _emit(out)
+    else:
+        if out["status"] == "built":
+            print(f"indexed {out['sections_indexed']} section(s) for {out['source_id']} into .kb/pageindex/")
+        else:
+            print(out["message"])
+    return 0
+
+
+def cmd_pageindex_search(args: argparse.Namespace) -> int:
+    from . import pageindex_adapter
+
+    root = resolve_root(args.root)
+    source_id = args.source
+    if source_id is not None and not source_id.startswith("raw/"):
+        source_id = f"raw/{source_id}"
+    out = pageindex_adapter.search(root, args.query, k=args.k, source_id=source_id)
+    if out is None:
+        payload = {
+            "status": "missing",
+            "message": "no usable PageIndex cache; run `scrip pageindex build raw/<slug>` first",
+            "method": "pageindex",
+            "stale_index": False,
+            "results": [],
+        }
+        if args.json:
+            _emit(payload)
+        else:
+            print(payload["message"])
+        return 0
+    if args.json:
+        _emit(out)
+    else:
+        stale_hint = " - INDEX STALE, rebuild with `scrip pageindex build raw/<slug>`"
+        tag = out["method"] + (stale_hint if out["stale_index"] else "")
+        print(f"[{tag}] top {len(out['results'])} for: {args.query!r}")
+        for r in out["results"]:
+            score_value = r.get("score", "")
+            score = f"{score_value:.3f}" if isinstance(score_value, float) else str(score_value)
+            print(f"  {r['source_id']}#{r.get('section_id', '?')}  ({score})")
             print(f"    {r['snippet']}")
         if not out["results"]:
             print("  (no matches)")
@@ -569,8 +626,35 @@ def build_parser() -> argparse.ArgumentParser:
     )
     psr.add_argument("query", help="the question / search text")
     psr.add_argument("-k", type=int, default=5, help="number of results (default 5)")
+    psr.add_argument(
+        "--long-docs",
+        choices=["pageindex"],
+        help="try a long-document adapter cache before embeddings/grep",
+    )
     psr.set_defaults(func=cmd_search)
 
+    ppidx = sub.add_parser(
+        "pageindex",
+        help="optional PageIndex long-document cache (regenerable, not source of truth)",
+    )
+    pidx_sub = ppidx.add_subparsers(dest="pageindex_command", required=True, metavar="<action>")
+    ppb = pidx_sub.add_parser(
+        "build",
+        parents=[common],
+        help="build a PageIndex cache for one raw source",
+    )
+    ppb.add_argument("source", metavar="raw/<slug>", help="raw source to index")
+    ppb.set_defaults(func=cmd_pageindex_build)
+    pps = pidx_sub.add_parser(
+        "search",
+        parents=[common],
+        help="search the PageIndex cache for verbatim raw-source snippets",
+    )
+    pps.add_argument("query", help="the question / search text")
+    pps.add_argument("--source", metavar="raw/<slug>", help="restrict to one cached source")
+    pps.add_argument("-k", type=int, default=5, help="number of results (default 5)")
+    pps.set_defaults(func=cmd_pageindex_search)
+
     pul = sub.add_parser(
         "unlock",
         parents=[common],

{scriptoria-0.5.2 → scriptoria-0.6.1}/src/scrip/facts.py

@@ -26,6 +26,11 @@ from pathlib import Path
 
 import yaml
 
+try:
+    from yaml import CSafeLoader as SafeLoader
+except ImportError:
+    from yaml import SafeLoader
+
 from . import anchors, facts_dir, lock
 from .errors import DataError, UsageError
 
@@ -337,7 +342,7 @@ def _load_meta(root: Path) -> dict:
             "confidence": 0.0,
         }
     try:
-        data = yaml.safe_load(p.read_text(encoding="utf-8")) or {}
+        data = yaml.load(p.read_text(encoding="utf-8"), Loader=SafeLoader) or {}
     except yaml.YAMLError as e:
         raise DataError(f"invalid facts/_meta.yaml: {e}") from e
     if not isinstance(data, dict):

{scriptoria-0.5.2 → scriptoria-0.6.1}/src/scrip/frontmatter.py

@@ -12,6 +12,11 @@ from pathlib import Path
 
 import yaml
 
+try:
+    from yaml import CSafeLoader as SafeLoader
+except ImportError:
+    from yaml import SafeLoader
+
 from .errors import DataError
 
 FENCE = "---"
@@ -28,7 +33,7 @@ def parse(text: str) -> tuple[dict, str]:
             fm_text = "".join(lines[1:i])
             body = "".join(lines[i + 1 :])
             try:
-                meta = yaml.safe_load(fm_text)
+                meta = yaml.load(fm_text, Loader=SafeLoader)
             except yaml.YAMLError as e:
                 raise DataError(f"invalid YAML frontmatter: {e}") from e
             if meta is None:
@@ -53,7 +58,7 @@ def _read_frontmatter(f) -> tuple[bool, dict]:
         raise DataError("unterminated frontmatter (missing closing '---')")
 
     try:
-        meta = yaml.safe_load("".join(fm_lines))
+        meta = yaml.load("".join(fm_lines), Loader=SafeLoader)
     except yaml.YAMLError as e:
         raise DataError(f"invalid YAML frontmatter: {e}") from e
     if meta is None:

{scriptoria-0.5.2 → scriptoria-0.6.1}/src/scrip/graph.py

@@ -13,6 +13,11 @@ from pathlib import Path
 
 import yaml
 
+try:
+    from yaml import CSafeLoader as SafeLoader
+except ImportError:
+    from yaml import SafeLoader
+
 from . import blocks as blocks_mod
 from . import facts_dir, frontmatter, hashing, raw_dir, wiki_dir
 from . import manifest as manifest_mod
@@ -92,7 +97,7 @@ def scan_derived(root: Path) -> dict:
     fmeta = facts_dir(root) / "_meta.yaml"
     if fmeta.exists():
         try:
-            data = yaml.safe_load(fmeta.read_text(encoding="utf-8"))
+            data = yaml.load(fmeta.read_text(encoding="utf-8"), Loader=SafeLoader)
         except yaml.YAMLError as e:
             raise DataError(f"invalid facts/_meta.yaml: {e}") from e
         if data is None:
@@ -246,7 +251,7 @@ def stamp_artifacts(root: Path, paths: list[str] | None = None) -> list[dict]:
         ih = hashing.input_hash(deps)
         path = root / d["path"]
         if path.name == "_meta.yaml":
-            data = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
+            data = yaml.load(path.read_text(encoding="utf-8"), Loader=SafeLoader) or {}
             data["input-hash"] = ih
             data["last-compiled"] = now
             path.write_text(

scriptoria-0.6.1/src/scrip/pageindex_adapter.py

@@ -0,0 +1,371 @@
+"""Optional PageIndex cache adapter for long-document retrieval.
+
+PageIndex is treated like embeddings: a regenerable cache under ``.kb/``, never
+the source of truth. The only durable text remains ``vault/raw/<slug>.md``; every
+search result snippet persisted here must be copied verbatim from that raw text
+so a later answer can still cite it with ``scrip anchor``.
+
+The real backend is deliberately optional. Tests monkeypatch ``_get_backend``;
+production can provide a module exposing ``build``/``build_index`` and optionally
+``search``. Without a backend, the CLI reports an unavailable adapter and normal
+``scrip search`` falls back to embeddings/grep.
+"""
+
+from __future__ import annotations
+
+import importlib
+import json
+from datetime import datetime, timezone
+from pathlib import Path
+
+from . import hashing, raw_dir
+from .errors import DataError
+
+INDEX_SCHEMA = 1
+
+_backend = None
+_backend_tried = False
+
+
+def _pageindex_dir(root: Path) -> Path:
+    return root / ".kb" / "pageindex"
+
+
+def _source_id(source: str) -> str:
+    sid = source if source.startswith("raw/") else f"raw/{source}"
+    slug = sid[len("raw/") :]
+    if "/" in slug or "\\" in slug or slug.startswith(".") or ".." in slug:
+        raise DataError(f"unsafe source id: {source}")
+    return sid
+
+
+def _source_path(root: Path, source_id: str) -> Path:
+    p = raw_dir(root) / f"{source_id[len('raw/'):]}.md"
+    if not p.exists():
+        raise DataError(f"source does not exist: {source_id}")
+    return p
+
+
+def _cache_dir(root: Path, source_id: str) -> Path:
+    return _pageindex_dir(root) / source_id[len("raw/") :]
+
+
+def _now() -> str:
+    return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
+
+
+def _get_backend():
+    """Return an optional PageIndex backend module/object.
+
+    The adapter accepts either ``pageindex`` or ``page_index`` as import names.
+    A usable backend exposes ``build`` or ``build_index``; ``search`` is optional
+    because cached records can still be ranked lexically.
+    """
+    global _backend, _backend_tried
+    if _backend_tried:
+        return _backend
+    _backend_tried = True
+    for name in ("pageindex", "page_index"):
+        try:
+            mod = importlib.import_module(name)
+        except Exception:
+            continue
+        if any(hasattr(mod, attr) for attr in ("build", "build_index")):
+            _backend = mod
+            break
+    return _backend
+
+
+def available() -> bool:
+    return _get_backend() is not None
+
+
+def _backend_name(backend) -> str:
+    return getattr(backend, "__name__", backend.__class__.__name__)
+
+
+def _backend_version(backend) -> str:
+    return str(getattr(backend, "__version__", "unknown"))
+
+
+def _call_build(backend, *, source_id: str, text: str):
+    if hasattr(backend, "build_index"):
+        return backend.build_index(source_id=source_id, text=text)
+    if hasattr(backend, "build"):
+        return backend.build(source_id=source_id, text=text)
+    if callable(backend):
+        return backend(source_id=source_id, text=text)
+    raise RuntimeError("PageIndex backend does not expose build/build_index")
+
+
+def _call_search(backend, *, query: str, items: list[dict], k: int):
+    if backend is not None and hasattr(backend, "search"):
+        return backend.search(query=query, items=items, k=k)
+    return _lexical_rank(query, items, k)
+
+
+def _candidate_records(raw):
+    if raw is None:
+        return []
+    if isinstance(raw, list):
+        return raw
+    if isinstance(raw, dict):
+        for key in ("items", "sections", "nodes", "results"):
+            val = raw.get(key)
+            if isinstance(val, list):
+                return val
+    return []
+
+
+def _coerce_items(raw, *, source_id: str, text: str) -> list[dict]:
+    """Normalize backend records and keep only snippets present in raw text."""
+    items: list[dict] = []
+    for i, rec in enumerate(_candidate_records(raw)):
+        if not isinstance(rec, dict):
+            continue
+        snippet = str(rec.get("snippet") or rec.get("text") or rec.get("content") or "")
+        span = rec.get("span_hint") or rec.get("span")
+        if (
+            isinstance(span, list)
+            and len(span) == 2
+            and all(isinstance(n, int) for n in span)
+            and 0 <= span[0] < span[1] <= len(text)
+        ):
+            # Prefer the canonical raw slice. Backend summaries are not citable.
+            snippet = text[span[0] : span[1]].strip()
+        if not snippet:
+            continue
+        start = text.find(snippet)
+        if start < 0:
+            # The contract requires verbatim evidence; discard summaries.
+            continue
+        span_hint = [start, start + len(snippet)]
+        items.append(
+            {
+                "source_id": source_id,
+                "section_id": str(rec.get("section_id") or rec.get("id") or i),
+                "snippet": snippet,
+                "span_hint": span_hint,
+                "score": float(rec.get("score") or 0.0),
+                "method": "pageindex",
+            }
+        )
+    return items
+
+
+def _cached_item_maps(items: list[dict]) -> tuple[dict[tuple[str, str], dict], dict[str, list[dict]]]:
+    by_section: dict[tuple[str, str], dict] = {}
+    by_snippet: dict[str, list[dict]] = {}
+    for item in items:
+        source_id = item.get("source_id")
+        section_id = item.get("section_id")
+        snippet = item.get("snippet")
+        if isinstance(source_id, str) and section_id is not None:
+            by_section[(source_id, str(section_id))] = item
+        if isinstance(snippet, str):
+            by_snippet.setdefault(snippet, []).append(item)
+    return by_section, by_snippet
+
+
+def _float_score(value, fallback: float) -> float:
+    try:
+        return float(value)
+    except (TypeError, ValueError):
+        return fallback
+
+
+def _match_cached_item(
+    rec: dict, by_section: dict[tuple[str, str], dict], by_snippet: dict[str, list[dict]]
+) -> dict | None:
+    source_id = rec.get("source_id")
+    if isinstance(source_id, str):
+        try:
+            source_id = _source_id(source_id)
+        except DataError:
+            source_id = None
+    else:
+        source_id = None
+    section_id = rec.get("section_id") or rec.get("id")
+    if source_id is not None and section_id is not None:
+        item = by_section.get((source_id, str(section_id)))
+        if item is not None:
+            return item
+    snippet = rec.get("snippet") or rec.get("text") or rec.get("content")
+    if isinstance(snippet, str):
+        for item in by_snippet.get(snippet, []):
+            if source_id is None or item.get("source_id") == source_id:
+                if section_id is None or str(item.get("section_id")) == str(section_id):
+                    return item
+    return None
+
+
+def _canonical_search_results(raw, items: list[dict], k: int) -> list[dict]:
+    by_section, by_snippet = _cached_item_maps(items)
+    results: list[dict] = []
+    seen: set[tuple[str, str, tuple[int, ...]]] = set()
+    for rec in _candidate_records(raw):
+        if not isinstance(rec, dict):
+            continue
+        item = _match_cached_item(rec, by_section, by_snippet)
+        if item is None:
+            continue
+        span = item.get("span_hint")
+        span_key = tuple(span) if isinstance(span, list) and all(isinstance(n, int) for n in span) else ()
+        key = (str(item.get("source_id")), str(item.get("section_id")), span_key)
+        if key in seen:
+            continue
+        seen.add(key)
+        out = dict(item)
+        out["score"] = _float_score(rec.get("score"), _float_score(item.get("score"), 0.0))
+        out["method"] = "pageindex"
+        results.append(out)
+        if len(results) >= k:
+            break
+    return results
+
+
+def build_index(root: Path, source: str) -> dict:
+    """Build a PageIndex cache for one raw source.
+
+    Returns ``{"status": "built"|"unavailable", ...}`` rather than raising for
+    a missing backend so the CLI can fail soft like ``scrip index``.
+    """
+    backend = _get_backend()
+    if backend is None:
+        return {
+            "status": "unavailable",
+            "message": (
+                "PageIndex backend not installed; see docs/pageindex-adapter.md. "
+                "Normal `scrip search` still falls back to embeddings/grep."
+            ),
+        }
+    source_id = _source_id(source)
+    path = _source_path(root, source_id)
+    text = path.read_text(encoding="utf-8")
+    raw = _call_build(backend, source_id=source_id, text=text)
+    items = _coerce_items(raw, source_id=source_id, text=text)
+    d = _cache_dir(root, source_id)
+    d.mkdir(parents=True, exist_ok=True)
+    raw_hash = hashing.content_hash_file(path)
+    (d / "tree.json").write_text(
+        json.dumps({"source_id": source_id, "items": items}, indent=2, ensure_ascii=False)
+        + "\n",
+        encoding="utf-8",
+    )
+    (d / "meta.json").write_text(
+        json.dumps(
+            {
+                "source_id": source_id,
+                "raw_content_hash": raw_hash,
+                "backend": _backend_name(backend),
+                "backend_version": _backend_version(backend),
+                "schema": INDEX_SCHEMA,
+                "created_at": _now(),
+            },
+            indent=2,
+            ensure_ascii=False,
+        )
+        + "\n",
+        encoding="utf-8",
+    )
+    return {"status": "built", "source_id": source_id, "sections_indexed": len(items)}
+
+
+def _load_cached(root: Path, source_id: str | None = None) -> tuple[list[dict], bool]:
+    base = _pageindex_dir(root)
+    if not base.is_dir():
+        return [], False
+    dirs = (
+        [_cache_dir(root, _source_id(source_id))]
+        if source_id
+        else sorted(p for p in base.iterdir() if p.is_dir())
+    )
+    items: list[dict] = []
+    stale = False
+    for d in dirs:
+        tree_p, meta_p = d / "tree.json", d / "meta.json"
+        if not tree_p.exists() or not meta_p.exists():
+            continue
+        try:
+            tree = json.loads(tree_p.read_text(encoding="utf-8"))
+            meta = json.loads(meta_p.read_text(encoding="utf-8"))
+        except (json.JSONDecodeError, OSError, UnicodeDecodeError):
+            continue
+        sid = meta.get("source_id") or tree.get("source_id")
+        if not isinstance(sid, str):
+            continue
+        try:
+            source_path = _source_path(root, sid)
+            current = hashing.content_hash_file(source_path)
+            text = source_path.read_text(encoding="utf-8")
+        except DataError:
+            stale = True
+            continue
+        if meta.get("schema") != INDEX_SCHEMA:
+            stale = True
+            continue
+        if meta.get("raw_content_hash") != current:
+            stale = True
+        vals = tree.get("items") if isinstance(tree, dict) else None
+        if isinstance(vals, list):
+            items.extend(_revalidate_cached_items(vals, source_id=sid, text=text))
+    return items, stale
+
+
+def _revalidate_cached_items(raw_items: list, *, source_id: str, text: str) -> list[dict]:
+    items: list[dict] = []
+    for item in raw_items:
+        if not isinstance(item, dict):
+            continue
+        cached_source = item.get("source_id")
+        if isinstance(cached_source, str):
+            try:
+                if _source_id(cached_source) != source_id:
+                    continue
+            except DataError:
+                continue
+        snippet = item.get("snippet")
+        if not isinstance(snippet, str) or not snippet:
+            continue
+        start = text.find(snippet)
+        if start < 0:
+            continue
+        out = dict(item)
+        out["source_id"] = source_id
+        out["snippet"] = snippet
+        out["span_hint"] = [start, start + len(snippet)]
+        out["score"] = _float_score(item.get("score"), 0.0)
+        out["method"] = "pageindex"
+        items.append(out)
+    return items
+
+
+def _terms(query: str) -> list[str]:
+    return [t for t in hashing.normalize(query).split() if t]
+
+
+def _lexical_rank(query: str, items: list[dict], k: int) -> list[dict]:
+    terms = _terms(query)
+    ranked: list[dict] = []
+    for item in items:
+        low = str(item.get("snippet") or "").lower()
+        score = sum(low.count(t) for t in terms)
+        if score <= 0:
+            continue
+        out = dict(item)
+        out["score"] = float(score)
+        out["method"] = "pageindex"
+        ranked.append(out)
+    ranked.sort(key=lambda r: (-float(r["score"]), str(r.get("source_id")), str(r.get("section_id"))))
+    return ranked[:k]
+
+
+def search(root: Path, query: str, k: int = 5, source_id: str | None = None) -> dict | None:
+    """Return ``{method, stale_index, results}``, or ``None`` with no cache."""
+    items, stale = _load_cached(root, source_id)
+    if not items:
+        return None
+    backend = _get_backend()
+    raw_results = _call_search(backend, query=query, items=items, k=k)
+    results = _canonical_search_results(raw_results, items, k)
+    return {"method": "pageindex", "stale_index": stale, "results": results}

{scriptoria-0.5.2 → scriptoria-0.6.1}/src/scrip/retrieval.py

@@ -11,7 +11,7 @@ import re
 from pathlib import Path
 
 from . import blocks as blocks_mod
-from . import embeddings, raw_dir
+from . import embeddings, pageindex_adapter, raw_dir
 
 _WORD = re.compile(r"\w+")
 
@@ -41,8 +41,12 @@ def grep_search(root: Path, query: str, k: int = 5) -> list[dict]:
     return hits[:k]
 
 
-def search(root: Path, query: str, k: int = 5) -> dict:
+def search(root: Path, query: str, k: int = 5, long_docs: str | None = None) -> dict:
     """Return ``{method, stale_index, results}``."""
+    if long_docs == "pageindex":
+        p = pageindex_adapter.search(root, query, k)
+        if p is not None:
+            return p
     v = embeddings.vector_search(root, query, k)
     if v is not None:
         results, stale = v

scriptoria-0.6.1/tests/test_pageindex_cmd.py

@@ -0,0 +1,186 @@
+"""PageIndex adapter tests. Hermetic: no real PageIndex dependency is installed;
+the backend is monkeypatched with a deterministic tiny object."""
+
+import json
+
+from scrip import cli, pageindex_adapter, retrieval
+
+
+class FakePageIndex:
+    __version__ = "test"
+
+    def build_index(self, *, source_id, text):
+        start = text.index("Alpha section explains caching.")
+        end = start + len("Alpha section explains caching.")
+        return {
+            "sections": [
+                {"section_id": "alpha", "span_hint": [start, end], "score": 0.9},
+                # Summaries that are not verbatim raw text must not enter cache.
+                {"section_id": "summary", "snippet": "a generated summary absent from raw"},
+            ]
+        }
+
+    def search(self, *, query, items, k):
+        return [it for it in items if query.lower() in it["snippet"].lower()][:k]
+
+
+def _fake_backend(monkeypatch):
+    monkeypatch.setattr(pageindex_adapter, "_get_backend", lambda: FakePageIndex())
+
+
+def test_pageindex_build_unavailable_exits_cleanly(kb, capsys, monkeypatch):
+    monkeypatch.setattr(pageindex_adapter, "_get_backend", lambda: None)
+    kb.add_raw("paper", "# Paper\n\nAlpha section explains caching.\n")
+    rc = cli.main(["pageindex", "build", "raw/paper", "--json", "--root", str(kb.root)])
+    assert rc == 0
+    data = json.loads(capsys.readouterr().out)
+    assert data["status"] == "unavailable"
+
+
+def test_pageindex_build_persists_only_verbatim_raw_snippets(kb, capsys, monkeypatch):
+    _fake_backend(monkeypatch)
+    kb.add_raw(
+        "paper",
+        "# Paper\n\nAlpha section explains caching.\n\nBeta section covers indexing.\n",
+    )
+    rc = cli.main(["pageindex", "build", "raw/paper", "--json", "--root", str(kb.root)])
+    assert rc == 0
+    data = json.loads(capsys.readouterr().out)
+    assert data["status"] == "built"
+    assert data["sections_indexed"] == 1
+
+    tree = json.loads(
+        (kb.root / ".kb" / "pageindex" / "paper" / "tree.json").read_text(encoding="utf-8")
+    )
+    [item] = tree["items"]
+    assert item["source_id"] == "raw/paper"
+    assert item["section_id"] == "alpha"
+    assert item["snippet"] == "Alpha section explains caching."
+    assert "summary absent" not in json.dumps(tree)
+
+
+def test_pageindex_search_cli_and_retrieval_integration(kb, capsys, monkeypatch):
+    _fake_backend(monkeypatch)
+    kb.add_raw(
+        "paper",
+        "# Paper\n\nAlpha section explains caching.\n\nBeta section covers indexing.\n",
+    )
+    assert cli.main(["pageindex", "build", "paper", "--root", str(kb.root)]) == 0
+    capsys.readouterr()
+
+    assert cli.main(["pageindex", "search", "Alpha", "--json", "--root", str(kb.root)]) == 0
+    out = json.loads(capsys.readouterr().out)
+    assert out["method"] == "pageindex"
+    assert out["results"][0]["snippet"] == "Alpha section explains caching."
+
+    routed = retrieval.search(kb.root, "Alpha", long_docs="pageindex")
+    assert routed["method"] == "pageindex"
+    assert routed["results"][0]["source_id"] == "raw/paper"
+
+
+def test_search_long_docs_pageindex_cli_prints_section_results(kb, capsys, monkeypatch):
+    _fake_backend(monkeypatch)
+    kb.add_raw("paper", "# Paper\n\nAlpha section explains caching.\n")
+    assert cli.main(["pageindex", "build", "paper", "--root", str(kb.root)]) == 0
+    capsys.readouterr()
+
+    rc = cli.main(["search", "Alpha", "--long-docs", "pageindex", "--root", str(kb.root)])
+    assert rc == 0
+    out = capsys.readouterr().out
+    assert "[pageindex]" in out
+    assert "raw/paper#alpha" in out
+    assert "Alpha section explains caching." in out
+
+
+def test_pageindex_search_reuses_cached_verbatim_snippet(kb, monkeypatch):
+    class SummarySearchBackend(FakePageIndex):
+        def search(self, *, query, items, k):
+            return [
+                {
+                    "source_id": "raw/paper",
+                    "section_id": "alpha",
+                    "snippet": "fabricated summary absent from raw",
+                    "score": 99,
+                }
+            ]
+
+    monkeypatch.setattr(pageindex_adapter, "_get_backend", lambda: SummarySearchBackend())
+    kb.add_raw("paper", "# Paper\n\nAlpha section explains caching.\n")
+    pageindex_adapter.build_index(kb.root, "paper")
+
+    out = pageindex_adapter.search(kb.root, "Alpha")
+    assert out is not None
+    assert out["results"][0]["snippet"] == "Alpha section explains caching."
+    assert "fabricated summary" not in json.dumps(out)
+
+
+def test_pageindex_cache_revalidates_snippets_after_raw_change(kb, monkeypatch):
+    _fake_backend(monkeypatch)
+    kb.add_raw("paper", "# Paper\n\nAlpha section explains caching.\n")
+    pageindex_adapter.build_index(kb.root, "paper")
+    kb.mutate_raw("paper", "# Paper\n\nNew intro.\n\nAlpha section explains caching.\n")
+
+    out = pageindex_adapter.search(kb.root, "Alpha")
+    assert out is not None
+    assert out["stale_index"] is True
+    assert out["results"][0]["snippet"] == "Alpha section explains caching."
+
+
+def test_pageindex_cache_drops_stale_snippets_absent_from_raw(kb, monkeypatch):
+    _fake_backend(monkeypatch)
+    kb.add_raw("paper", "# Paper\n\nAlpha section explains caching.\n")
+    pageindex_adapter.build_index(kb.root, "paper")
+    kb.mutate_raw("paper", "# Paper\n\nAlpha section changed completely.\n")
+
+    assert pageindex_adapter.search(kb.root, "Alpha") is None
+    out = retrieval.search(kb.root, "Alpha", long_docs="pageindex")
+    assert out["method"] == "grep"
+    assert "changed completely" in out["results"][0]["snippet"]
+
+
+def test_pageindex_lexical_fallback_uses_query_time_scores(kb, monkeypatch):
+    class RankedBuildBackend:
+        __version__ = "test"
+
+        def build_index(self, *, source_id, text):
+            alpha_start = text.index("Alpha Alpha section.")
+            beta_start = text.index("Beta section.")
+            return {
+                "sections": [
+                    {
+                        "section_id": "alpha",
+                        "span_hint": [alpha_start, alpha_start + len("Alpha Alpha section.")],
+                        "score": 1,
+                    },
+                    {
+                        "section_id": "beta",
+                        "span_hint": [beta_start, beta_start + len("Beta section.")],
+                        "score": 100,
+                    },
+                ]
+            }
+
+    monkeypatch.setattr(pageindex_adapter, "_get_backend", lambda: RankedBuildBackend())
+    kb.add_raw("paper", "# Paper\n\nAlpha Alpha section.\n\nBeta section.\n")
+    pageindex_adapter.build_index(kb.root, "paper")
+    monkeypatch.setattr(pageindex_adapter, "_get_backend", lambda: None)
+
+    out = pageindex_adapter.search(kb.root, "Alpha section")
+    assert out is not None
+    assert [r["section_id"] for r in out["results"]] == ["alpha", "beta"]
+    assert [r["score"] for r in out["results"]] == [3.0, 1.0]
+
+
+def test_search_long_docs_pageindex_falls_back_without_cache(kb):
+    kb.add_raw("paper", "# Paper\n\nAlpha section explains caching.\n")
+    out = retrieval.search(kb.root, "Alpha", long_docs="pageindex")
+    assert out["method"] == "grep"
+    assert out["results"][0]["source_id"] == "raw/paper"
+
+
+def test_pageindex_search_missing_cache_json(kb, capsys):
+    rc = cli.main(["pageindex", "search", "anything", "--json", "--root", str(kb.root)])
+    assert rc == 0
+    out = json.loads(capsys.readouterr().out)
+    assert out["status"] == "missing"
+    assert out["results"] == []

{scriptoria-0.5.2 → scriptoria-0.6.1}/uv.lock

@@ -1141,7 +1141,7 @@ wheels = [
 
 [[package]]
 name = "scriptoria"
-version = "0.5.2"
+version = "0.6.1"
 source = { editable = "." }
 dependencies = [
     { name = "duckdb" },

scriptoria-0.5.2/README.md

@@ -1,40 +0,0 @@
-# scrip
-
-The deterministic keeper of a [scriptorium](../README.md) knowledge base.
-
-The agent owns judgment (synthesis, fact extraction, reconciliation, promotion).
-`scrip` owns only what LLMs are unreliable or expensive at: **content hashing**,
-**staleness** over a dependency graph, **provenance-anchor integrity**, and
-**structured queries** over the facts layer. Files are the source of truth;
-`.kb/manifest.json` is only a regenerable speed cache.
-
-## Install
-
-```sh
-uv tool install ./scrip      # installs the `scrip` command
-# or, without installing:
-uv run --project scrip scrip --help
-```
-
-## Commands
-
-| Command | What it does |
-|---|---|
-| `scrip status` | Report `STALE` / `OK` / `UNCOMPILED` artifacts from the dependency graph. `--no-cache` recomputes from files; `--rebuild-manifest` regenerates the cache. |
-| `scrip verify` | Check every provenance anchor still resolves to text in its source; check referenced sources exist and `claim_id`s are unique. Fails on `BROKEN` and `AMBIGUOUS` by default; `--allow-ambiguous` downgrades `AMBIGUOUS` to a warning. |
-| `scrip query [claims\|entities\|edges\|contradictions]` | Structured query over `vault/facts/*.ndjson` via DuckDB. `--sql "<duckdb>"`, `--where`, `--limit`. |
-| `scrip index` | v0 stub for the embeddings retrieval rung. |
-
-Every command accepts `--root DIR` and `--json`.
-
-## Exit codes
-
-| Code | Meaning |
-|---|---|
-| `0` | clean / success |
-| `1` | actionable finding (stale artifacts; broken citations) |
-| `2` | usage error |
-| `3` | data error (malformed frontmatter / NDJSON; missing source; duplicate id) |
-| `4` | internal error |
-
-Code `1` is an *expected* outcome the agent branches on, not a crash.