gather-engine 2.1.0__tar.gz → 2.2.0__tar.gz

{gather_engine-2.1.0 → gather_engine-2.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gather-engine
-Version: 2.1.0
+Version: 2.2.0
 Summary: Accountable research intake for difficult sources, with provenance receipts and witnessed digests.
 Author: Zain Dana Harper
 License-Expression: LicenseRef-FSL-1.1-MIT
@@ -34,7 +34,7 @@ Dynamic: license-file
 [![downloads](https://img.shields.io/pypi/dm/gather-engine?label=downloads&style=flat-square&labelColor=14041b)](https://pypi.org/project/gather-engine/)
 ![python: 3.11+](https://img.shields.io/badge/python-3.11%2B-blue?style=flat-square&labelColor=14041b)
 ![deps: none (core)](https://img.shields.io/badge/core%20deps-none-success?style=flat-square&labelColor=14041b)
-![version: 2.1.0](https://img.shields.io/badge/version-2.1.0-26dfe8?style=flat-square&labelColor=14041b)
+![version: 2.2.0](https://img.shields.io/badge/version-2.2.0-26dfe8?style=flat-square&labelColor=14041b)
 
 gather pulls research out of the places most tools break on: arXiv papers, authenticated JSON APIs, JavaScript-rendered pages via a real headless browser, scanned images through OCR, and audio through transcription, alongside video, web, feeds, and local docs. The core runs with zero third-party runtime dependencies, and the same engine is reachable from the CLI, MCP tools, and plain Python. Every run writes a receipt you can re-check.
 
@@ -42,7 +42,7 @@ gather pulls research out of the places most tools break on: arXiv papers, authe
 
 ## Current status
 
-`gather-engine 2.1.0` is the current source version. Core intake,
+`gather-engine 2.2.0` is the current source version. Core intake,
 content-addressed corpus storage, exact UTF-8 source-byte receipts, readable
 context selection, descriptor handoff, pilot monitoring, CLI, Python API, and MCP
 surfaces are present in this checkout. Browser rendering, OCR, audio, and fast
@@ -82,7 +82,9 @@ fabricated field cannot reach the corpus by being plausible.
 - **Track elements across redesigns.** Fingerprint a scraped element once, then relocate it in any later version of the page and get a typed MATCH, RELOCATED, DRIFT, or GONE verdict (`gather.track`, Python API).
 - **Structured extraction with a hallucination check.** `gather.schema_extract` binds schema fields to source nodes, and `verify_record` rejects any LLM-proposed field value not grounded in the fetched content.
 - **Streaming extraction.** `gather.stream` parses an HTML stream chunk by chunk and emits partial-update commits as blocks complete, each folded into a hash chain, so a streamed extraction is replayable.
-- **Hard-source adapters behind one shape.** Video with captions and comments (`yt-dlp`), static web, RSS/Atom feeds, local docs, arXiv, PDFs (`pdftotext`), authenticated JSON APIs (token from env, never logged), JS-rendered pages (headless Chromium), scanned images (`tesseract`), and audio (`whisper`). Each external tool is optional and only needed for its own adapter.
+- **Hard-source adapters behind one shape.** Video with captions and comments (`yt-dlp` first, with the official YouTube Data API as a fallback when you supply your own key; each item records which path served it and how fast), Reddit through its official Data API with your own app credentials, static web, RSS/Atom feeds, local docs, arXiv, PDFs (`pdftotext`), authenticated JSON APIs (token from env, never logged), JS-rendered pages (headless Chromium), scanned images (`tesseract`), and audio (`whisper`). Each external tool is optional and only needed for its own adapter.
+- **Cited reports, checked in code.** `gather report` asks a model on your own machine to answer from fixed excerpts with exact quotes, then checks every quote against the excerpt it cites and marks any it cannot verify.
+- **Filter ledgers.** `--scope TERMS --ledger DIR` writes one row per dropped item with its reason code, plus the input, so `gather ledger verify` can recompute every count.
 - **Channel and playlist intake.** `gather channel <url> --store DIR` lists a channel's videos, shorts, and streams tabs (or one playlist) and gathers every entry with bounded concurrency, resumable from a per-pass ledger. Metadata and comments run as one pass and captions as another (`--no-captions`, `--captions-only`), with polite pacing, bounded backoff on HTTP 429 and session rate limits, a stop at the first bot check (never retried or answered), one caption track per video, and a run summary that counts every outcome by reason.
 - **Scholarly-graph federation.** `gather scholar` queries OpenAlex, Semantic Scholar, and Crossref in one call, dedupes results by normalized DOI (never a fuzzy title match), and can capture citation edges as first-class records with `--edges`.
 - **A durable local corpus.** Any fetch command takes `--store DIR`: bodies are content-addressed and deduped by hash, new writes preserve exact UTF-8 source bytes, and `gather corpus list|verify|digest|runs|search|stats|prune|availability|context` inspects, re-checks, and queries what you stored.
@@ -250,7 +252,7 @@ Keep the README, package metadata, and examples aligned with current behavior be
 
 ---
 
-**[Zentropy Labs](https://github.com/ZentropyLabs-ai)** · order out of entropy. An independent lab building evidence-first tools that leave a re-checkable artifact behind. Built by Zain Dana Harper in Seattle. The full workbench is at [Project Telos](https://harperz9.github.io).
+Built by **[Zain Dana Harper](https://harperz9.github.io)** in Seattle: evidence-first tools that leave a re-checkable artifact behind. The full workbench is at [Project Telos](https://harperz9.github.io).
 
 ## Local client packages
 

{gather_engine-2.1.0 → gather_engine-2.2.0}/README.md

@@ -8,7 +8,7 @@
 [![downloads](https://img.shields.io/pypi/dm/gather-engine?label=downloads&style=flat-square&labelColor=14041b)](https://pypi.org/project/gather-engine/)
 ![python: 3.11+](https://img.shields.io/badge/python-3.11%2B-blue?style=flat-square&labelColor=14041b)
 ![deps: none (core)](https://img.shields.io/badge/core%20deps-none-success?style=flat-square&labelColor=14041b)
-![version: 2.1.0](https://img.shields.io/badge/version-2.1.0-26dfe8?style=flat-square&labelColor=14041b)
+![version: 2.2.0](https://img.shields.io/badge/version-2.2.0-26dfe8?style=flat-square&labelColor=14041b)
 
 gather pulls research out of the places most tools break on: arXiv papers, authenticated JSON APIs, JavaScript-rendered pages via a real headless browser, scanned images through OCR, and audio through transcription, alongside video, web, feeds, and local docs. The core runs with zero third-party runtime dependencies, and the same engine is reachable from the CLI, MCP tools, and plain Python. Every run writes a receipt you can re-check.
 
@@ -16,7 +16,7 @@ gather pulls research out of the places most tools break on: arXiv papers, authe
 
 ## Current status
 
-`gather-engine 2.1.0` is the current source version. Core intake,
+`gather-engine 2.2.0` is the current source version. Core intake,
 content-addressed corpus storage, exact UTF-8 source-byte receipts, readable
 context selection, descriptor handoff, pilot monitoring, CLI, Python API, and MCP
 surfaces are present in this checkout. Browser rendering, OCR, audio, and fast
@@ -56,7 +56,9 @@ fabricated field cannot reach the corpus by being plausible.
 - **Track elements across redesigns.** Fingerprint a scraped element once, then relocate it in any later version of the page and get a typed MATCH, RELOCATED, DRIFT, or GONE verdict (`gather.track`, Python API).
 - **Structured extraction with a hallucination check.** `gather.schema_extract` binds schema fields to source nodes, and `verify_record` rejects any LLM-proposed field value not grounded in the fetched content.
 - **Streaming extraction.** `gather.stream` parses an HTML stream chunk by chunk and emits partial-update commits as blocks complete, each folded into a hash chain, so a streamed extraction is replayable.
-- **Hard-source adapters behind one shape.** Video with captions and comments (`yt-dlp`), static web, RSS/Atom feeds, local docs, arXiv, PDFs (`pdftotext`), authenticated JSON APIs (token from env, never logged), JS-rendered pages (headless Chromium), scanned images (`tesseract`), and audio (`whisper`). Each external tool is optional and only needed for its own adapter.
+- **Hard-source adapters behind one shape.** Video with captions and comments (`yt-dlp` first, with the official YouTube Data API as a fallback when you supply your own key; each item records which path served it and how fast), Reddit through its official Data API with your own app credentials, static web, RSS/Atom feeds, local docs, arXiv, PDFs (`pdftotext`), authenticated JSON APIs (token from env, never logged), JS-rendered pages (headless Chromium), scanned images (`tesseract`), and audio (`whisper`). Each external tool is optional and only needed for its own adapter.
+- **Cited reports, checked in code.** `gather report` asks a model on your own machine to answer from fixed excerpts with exact quotes, then checks every quote against the excerpt it cites and marks any it cannot verify.
+- **Filter ledgers.** `--scope TERMS --ledger DIR` writes one row per dropped item with its reason code, plus the input, so `gather ledger verify` can recompute every count.
 - **Channel and playlist intake.** `gather channel <url> --store DIR` lists a channel's videos, shorts, and streams tabs (or one playlist) and gathers every entry with bounded concurrency, resumable from a per-pass ledger. Metadata and comments run as one pass and captions as another (`--no-captions`, `--captions-only`), with polite pacing, bounded backoff on HTTP 429 and session rate limits, a stop at the first bot check (never retried or answered), one caption track per video, and a run summary that counts every outcome by reason.
 - **Scholarly-graph federation.** `gather scholar` queries OpenAlex, Semantic Scholar, and Crossref in one call, dedupes results by normalized DOI (never a fuzzy title match), and can capture citation edges as first-class records with `--edges`.
 - **A durable local corpus.** Any fetch command takes `--store DIR`: bodies are content-addressed and deduped by hash, new writes preserve exact UTF-8 source bytes, and `gather corpus list|verify|digest|runs|search|stats|prune|availability|context` inspects, re-checks, and queries what you stored.
@@ -224,7 +226,7 @@ Keep the README, package metadata, and examples aligned with current behavior be
 
 ---
 
-**[Zentropy Labs](https://github.com/ZentropyLabs-ai)** · order out of entropy. An independent lab building evidence-first tools that leave a re-checkable artifact behind. Built by Zain Dana Harper in Seattle. The full workbench is at [Project Telos](https://harperz9.github.io).
+Built by **[Zain Dana Harper](https://harperz9.github.io)** in Seattle: evidence-first tools that leave a re-checkable artifact behind. The full workbench is at [Project Telos](https://harperz9.github.io).
 
 ## Local client packages
 

{gather_engine-2.1.0 → gather_engine-2.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "gather-engine"
-version = "2.1.0"
+version = "2.2.0"
 description = "Accountable research intake for difficult sources, with provenance receipts and witnessed digests."
 readme = "README.md"
 requires-python = ">=3.11"

{gather_engine-2.1.0 → gather_engine-2.2.0}/src/gather/__init__.py

@@ -28,7 +28,7 @@ from gather.scope import filter_scope, in_scope
 from gather.source import Catalog, Source
 from gather.store import Corpus
 
-__version__ = "2.1.0"
+__version__ = "2.2.0"
 
 __all__ = [
     "Catalog", "Corpus", "CorpusRootDescriptor", "CorpusRootIdentity", "Digest", "Item", "NullProvenanceProvider", "NullSynthesizer",

gather_engine-2.2.0/src/gather/citecheck.py

@@ -0,0 +1,119 @@
+"""A code check on every citation in a report. Pure: no network, no model.
+
+A report cites an excerpt by quoting it and naming its number: ``"the exact words" [2]``. For
+each citation the check decides, by string comparison alone:
+
+- ``verified``: the quoted span appears in excerpt 2 (after Unicode, quote-mark, dash and
+  whitespace normalisation, and case folding);
+- ``not-in-source``: excerpt 2 exists and does not contain the span;
+- ``unknown-source``: there is no excerpt 2;
+- ``too-short``: the quote has fewer than ``MIN_QUOTE_WORDS`` words, so a match would prove
+  little; it is not counted as verified;
+- ``unchecked``: a bracketed number with no quote before it, so there is nothing to compare.
+
+Sentences with no citation at all are listed as uncited. Nothing is hidden: every citation and
+every uncited sentence is in the result. Precision is verified citations over all citations.
+
+What this does not check: whether the sentence around a verified quote says what the excerpt
+means. A quote can be exact and the claim around it can still stretch it.
+"""
+
+from __future__ import annotations
+
+import re
+import unicodedata
+from dataclasses import dataclass
+
+CHECK_SCHEMA = "gather.citation-check/1"
+MIN_QUOTE_WORDS = 4
+STATUSES = ("verified", "not-in-source", "unknown-source", "too-short", "unchecked")
+
+_MARKS = str.maketrans({"“": '"', "”": '"', "„": '"', "«": '"', "»": '"',
+                        "‘": "'", "’": "'", "–": "-", "—": "-", "−": "-",
+                        " ": " "})
+# a double-quoted span, optional space, then [n]
+_QUOTE_CITE = re.compile(r'"([^"]{1,2000})"\s*\[(\d{1,4})\]')
+_BARE_CITE = re.compile(r"\[(\d{1,4})\]")
+_LIST_MARK = re.compile(r"^(?:[-*+]|\d{1,3}[.)])\s+")
+_SENTENCE = re.compile(r'(?<=[.!?])["\')\]]*\s+(?=["(\[A-Z0-9])')
+
+
+def normalize(text: str) -> str:
+    """NFKC, straight quote marks and hyphens, collapsed whitespace, case folded."""
+    text = unicodedata.normalize("NFKC", text).translate(_MARKS)
+    return re.sub(r"\s+", " ", text).strip().casefold()
+
+
+def _strip_edges(quote: str) -> str:
+    return normalize(quote).strip(" .,;:!?'")
+
+
+@dataclass(frozen=True, slots=True)
+class Citation:
+    sentence: int
+    source: int
+    quote: str | None
+    status: str
+
+    def to_dict(self) -> dict:
+        return {"sentence": self.sentence, "source": self.source, "quote": self.quote, "status": self.status}
+
+
+def _split_line(line: str) -> list[str]:
+    """Split one line at sentence ends that fall outside a double-quoted span."""
+    parts, start = [], 0
+    for m in _SENTENCE.finditer(line):
+        if line[:m.start()].count('"') % 2 == 0:
+            parts.append(line[start:m.start() + len(m.group(0).rstrip())].strip())
+            start = m.end()
+    parts.append(line[start:].strip())
+    return [p for p in parts if p]
+
+
+def sentences(report: str) -> list[str]:
+    """The report's sentences: lines split at sentence ends outside quotes, list markers removed,
+    headings and blank lines dropped."""
+    out: list[str] = []
+    for raw in report.splitlines():
+        line = raw.strip()
+        if not line or line.startswith("#"):
+            continue
+        out.extend(_split_line(_LIST_MARK.sub("", line.translate(_MARKS))))
+    return out
+
+
+def _status(quote: str, source: int, excerpts: list[str]) -> str:
+    if not 1 <= source <= len(excerpts):
+        return "unknown-source"
+    span = _strip_edges(quote)
+    if len(span.split()) < MIN_QUOTE_WORDS:
+        return "too-short"
+    return "verified" if span in normalize(excerpts[source - 1]) else "not-in-source"
+
+
+def _sentence_citations(n: int, sentence: str, excerpts: list[str]) -> list[Citation]:
+    found = [Citation(n, int(m.group(2)), m.group(1), _status(m.group(1), int(m.group(2)), excerpts))
+             for m in _QUOTE_CITE.finditer(sentence.translate(_MARKS))]
+    bare_text = _QUOTE_CITE.sub("", sentence.translate(_MARKS))
+    found += [Citation(n, int(m.group(1)), None, "unchecked") for m in _BARE_CITE.finditer(bare_text)]
+    return found
+
+
+def check_report(report: str, excerpts: list[str]) -> dict:
+    """Check every citation in ``report`` against ``excerpts`` (excerpt 1 is ``excerpts[0]``)."""
+    cites: list[Citation] = []
+    uncited: list[str] = []
+    for n, sentence in enumerate(sentences(report)):
+        found = _sentence_citations(n, sentence, excerpts)
+        cites.extend(found)
+        if not found:
+            uncited.append(sentence)
+    counts = {s: sum(1 for c in cites if c.status == s) for s in STATUSES}
+    total = len(cites)
+    return {
+        "schema": CHECK_SCHEMA, "excerpts": len(excerpts), "citations": [c.to_dict() for c in cites],
+        "counts": counts, "total": total,
+        "precision": round(counts["verified"] / total, 4) if total else None,
+        "uncited_sentences": uncited,
+        "does_not_prove": "that the claim around a verified quote matches the excerpt's meaning",
+    }

{gather_engine-2.1.0 → gather_engine-2.2.0}/src/gather/cli.py

@@ -4,6 +4,7 @@ import argparse
 import sys
 
 from gather import __version__
+from gather.cli_ext import add_extension_parsers
 from gather.commands import (
     cmd_api,
     cmd_arxiv,
@@ -11,12 +12,10 @@ from gather.commands import (
     cmd_docs,
     cmd_feed,
     cmd_ocr,
-    cmd_parse,
     cmd_pdf,
     cmd_run,
     cmd_scholar,
     cmd_transcribe,
-    cmd_video,
     cmd_web,
 )
 from gather.corpus_cmd import cmd_corpus
@@ -29,7 +28,6 @@ from gather.pilot_commands import (
     cmd_pilot_run,
     cmd_pilot_verify,
 )
-from gather.video_cmd import add_channel_parser, add_ytdlp_options
 from gather.web_commands import (
     cmd_caps,
     cmd_crawl,
@@ -50,6 +48,8 @@ def _add_common(p: argparse.ArgumentParser) -> None:
     p.add_argument("--scope", default=None, help="comma-separated scope terms; keep items mentioning any")
     p.add_argument("--json", action="store_true", help="emit the catalog and digest as JSON")
     p.add_argument("--store", default=None, metavar="DIR", help="persist gathered items into a corpus at DIR")
+    p.add_argument("--ledger", default=None, metavar="DIR",
+                   help="write a filter ledger (ledger.json, input.jsonl) of what --scope dropped into DIR")
 
 
 def _add_flagship_commands(sub) -> None:
@@ -124,21 +124,7 @@ def build_parser() -> argparse.ArgumentParser:
     sub = parser.add_subparsers(dest="command")
     _add_flagship_commands(sub)
 
-    parse = sub.add_parser("parse", help="parse a saved yt-dlp info.json (+ optional .vtt), offline, no network")
-    parse.add_argument("info", help="path to a yt-dlp info.json")
-    parse.add_argument("--vtt", default=None, help="path to a .vtt captions file")
-    parse.add_argument("--auto-captions", action="store_true",
-                       help="captions are machine-generated: collapse rolling-window growth, stamp auto-caption")
-    _add_common(parse)
-    parse.set_defaults(func=cmd_parse)
-
-    video = sub.add_parser("video", help="fetch a video via yt-dlp (needs yt-dlp on PATH and network)")
-    video.add_argument("url")
-    video.add_argument("--comments", action="store_true", help="also gather comments")
-    add_ytdlp_options(video)
-    _add_common(video)
-    video.set_defaults(func=cmd_video)
-    add_channel_parser(sub)
+    add_extension_parsers(sub, _add_common)
 
     web = sub.add_parser("web", help="fetch a static web page via http(s) and extract readable text")
     web.add_argument("url")

gather_engine-2.2.0/src/gather/cli_ext.py

@@ -0,0 +1,19 @@
+"""The subcommands that live in their own modules, registered in one place for ``gather.cli``."""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+
+
+def add_extension_parsers(sub, add_common: Callable) -> None:
+    from gather.ledger_cmd import add_ledger_parser
+    from gather.reddit_cmd import add_reddit_parser
+    from gather.report_cmd import add_report_parsers
+    from gather.video_cmd import add_channel_parser
+    from gather.youtube_cmd import add_video_parsers
+
+    add_video_parsers(sub, add_common)
+    add_channel_parser(sub)
+    add_reddit_parser(sub, add_common)
+    add_report_parsers(sub, add_common)
+    add_ledger_parser(sub)

{gather_engine-2.1.0 → gather_engine-2.2.0}/src/gather/client_mcp.py

@@ -156,23 +156,22 @@ def main(argv=None):
     return 0
 
 def definitions(network=None):
-    from gather.mcp import _tool_defs
-    tools = [d for d in _tool_defs() if d["name"] in {"gather.docs"}]
+    from gather.docs_tool import annotate, docs_tool
+    tools = [annotate(docs_tool())]
     if network is not None and network.origins:
-        tools.append({"name": "gather.fetch", "description": "Read an allowed origin; return untrusted source text and a byte receipt. No redirects or credentials.",
-                      "inputSchema": {"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"], "additionalProperties": False}})
+        tools.append(annotate({"name": "gather.fetch", "description": "Read an allowed origin; return untrusted source text and a byte receipt. No redirects or credentials.",
+                      "inputSchema": {"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"], "additionalProperties": False}}))
     return tools
 
 
 def invoke(name, args, root):
-    from gather.grants import NONE
-    from gather.mcp import call_tool
+    from gather.docs_tool import call_docs
     # Single files avoid a directory walk following a file link outside the grant.
     path = confined(root, args["path"])
     if not path.is_file():
         raise ClientRefusal("local client intake requires one text file per call")
     args["path"] = str(path)
-    return call_tool(name, args, grants=NONE)
+    return call_docs(args)
 
 
 if __name__ == "__main__":

{gather_engine-2.1.0 → gather_engine-2.2.0}/src/gather/commands.py

@@ -13,12 +13,22 @@ def _scope(args) -> list[str]:
     return _split(args.scope)
 
 
-def _emit(items, scope, as_json, store=None) -> int:
+def _scoped(items, scope, ledger=None):
+    """The scope filter; with ``ledger``, every drop is also written to a ledger beside its input."""
+    if not ledger:
+        from gather.scope import filter_scope
+        return filter_scope(items, scope)
+    from gather.filter_ledger import scope_with_ledger, write_ledger
+    kept, record = scope_with_ledger(items, scope)
+    write_ledger(ledger, record, items)
+    return kept, record["dropped"]
+
+
+def _emit(items, scope, as_json, store=None, ledger=None) -> int:
     from gather.digest import digest, verify_digest
     from gather.payloads import catalog_digest_payload
-    from gather.scope import filter_scope
 
-    kept, dropped = filter_scope(items, scope)
+    kept, dropped = _scoped(items, scope, ledger)
     d = digest(kept)
     stored = None
     if store:
@@ -48,7 +58,7 @@ def _fetch_and_emit(fetch, args, fail: str = "fetch failed") -> int:
     except Exception as exc:
         print(f"{fail}: {exc}", file=sys.stderr)
         return 1
-    return _emit(items, _scope(args), args.json, store=args.store)
+    return _emit(items, _scope(args), args.json, store=args.store, ledger=getattr(args, "ledger", None))
 
 
 def cmd_parse(args) -> int:
@@ -61,13 +71,13 @@ def cmd_parse(args) -> int:
         with open(args.vtt, encoding="utf-8") as f:
             vtt = f.read()
     items = parse_video(info_json, vtt, fetched_at=time.time(), auto_captions=args.auto_captions)
-    return _emit(items, _scope(args), args.json, store=args.store)
+    return _emit(items, _scope(args), args.json, store=args.store, ledger=args.ledger)
 
 
 def cmd_video(args) -> int:
-    from gather.video_cmd import video_source_from_args
+    from gather.youtube_cmd import route_from_args
     try:
-        src = video_source_from_args(args, with_comments=args.comments)
+        src = route_from_args(args, with_comments=args.comments)
     except ValueError as exc:
         print(f"fetch failed: {exc}", file=sys.stderr)
         return 2
@@ -115,17 +125,16 @@ def cmd_scholar(args) -> int:
     except Exception as exc:
         print(f"scholar failed: {exc}", file=sys.stderr)
         return 1
-    return _emit_with_edges(items, edges, _scope(args), args.json, store=args.store)
+    return _emit_with_edges(items, edges, _scope(args), args.json, store=args.store, ledger=args.ledger)
 
 
-def _emit_with_edges(items, edges, scope, as_json, store=None) -> int:
+def _emit_with_edges(items, edges, scope, as_json, store=None, ledger=None) -> int:
     """Emit scholar items plus citation-edge receipts, folding the edges into the same witnessed
     digest as the papers, so the seal covers the graph (nodes and links), not just the nodes."""
     from gather.digest import digest_of_receipts, verify_digest
-    from gather.scope import filter_scope
     from gather.store import Corpus
 
-    kept, dropped = filter_scope(items, scope)
+    kept, dropped = _scoped(items, scope, ledger)
     item_receipts = [
         {"kind": i.kind, "id": i.id, "title": i.title, "source": i.provenance.source,
          "ref": i.provenance.ref, "method": i.provenance.method, "sha256": i.provenance.sha256,

gather_engine-2.2.0/src/gather/docs_tool.py

@@ -0,0 +1,93 @@
+"""The gather.docs tool and the MCP tool annotations.
+
+Shared by the full MCP server and the local client profile, so the client can
+serve gather.docs without importing the full tool surface.
+"""
+from __future__ import annotations
+
+import json
+
+from gather.localpath import require_local
+from gather.payloads import catalog_digest_payload
+from gather.scope import filter_scope
+
+
+def scope_terms(raw: object) -> list[str]:
+    if raw is None:
+        return []
+    if isinstance(raw, str):
+        return [term.strip() for term in raw.split(",") if term.strip()]
+    if isinstance(raw, list) and all(isinstance(term, str) for term in raw):
+        return [term.strip() for term in raw if term.strip()]
+    raise ValueError("scope must be a comma-separated string or a list of strings")
+
+
+def payload_from_items(items, scope: list[str]) -> dict:
+    kept, dropped = filter_scope(items, scope)
+    return catalog_digest_payload(kept, dropped=dropped)
+
+
+def _hints(title: str, *, read_only: bool = True, idempotent: bool = True,
+           open_world: bool = False) -> dict:
+    return {"title": title, "readOnlyHint": read_only, "destructiveHint": False,
+            "idempotentHint": idempotent, "openWorldHint": open_world}
+
+
+# MCP tool annotations. A hint describes the tool to the client and grants
+# nothing; launch grants still refuse network sources, commands and credentials.
+TOOL_ANNOTATIONS = {
+    "gather.status": _hints("Gather status"),
+    "gather.doctor": _hints("Gather readiness check"),
+    "gather.docs": _hints("Read local documents with receipts"),
+    "gather.arxiv": _hints("Fetch arXiv metadata", idempotent=False, open_world=True),
+    "gather.federation": _hints("Validate or plan a source registry"),
+    "gather.run": _hints("Run a gather config", read_only=False, idempotent=False,
+                         open_world=True),
+    "gather.context": _hints("Select verified corpus context"),
+    "gather.pilot": _hints("Run or verify a gather pilot", read_only=False, idempotent=False,
+                           open_world=True),
+    "gather.fetch": _hints("Read an allowed web origin", idempotent=False, open_world=True),
+}
+
+
+def annotate(tool: dict) -> dict:
+    notes = dict(TOOL_ANNOTATIONS[tool["name"]])
+    return {**tool, "title": notes["title"], "annotations": notes}
+
+
+def docs_tool() -> dict:
+    """The raw gather.docs definition, before annotation."""
+    return {
+        "name": "gather.docs",
+        "description": "Read a local text file or directory and return catalog rows plus digest receipts.",
+        "inputSchema": {
+            "type": "object",
+            "properties": {
+                "path": {"type": "string", "description": "local file or directory to read; a network "
+                                                          "or device path returns NON_LOCAL_PATH"},
+                "scope": {
+                    "description": (
+                        "optional post-fetch content filter: keep items whose title or body "
+                        "contains any term as a case-insensitive substring; omit for unfiltered capture"
+                    ),
+                    "oneOf": [
+                        {"type": "string"},
+                        {"type": "array", "items": {"type": "string"}},
+                    ],
+                },
+            },
+            "required": ["path"],
+        },
+    }
+
+
+def call_docs(args: dict) -> str:
+    """Run gather.docs: read one local path and return catalog rows plus digest receipts."""
+    from gather.docs import DocsSource
+
+    path = args.get("path")
+    if not isinstance(path, str) or not path:
+        raise ValueError("gather.docs requires a non-empty path")
+    require_local(path, label="path")
+    payload = payload_from_items(DocsSource().fetch(path), scope_terms(args.get("scope")))
+    return json.dumps(payload, indent=2, ensure_ascii=False, sort_keys=True)

gather_engine-2.2.0/src/gather/filter_ledger.py

@@ -0,0 +1,111 @@
+"""Filter ledgers: what a filter dropped, item by item, with one reason code each.
+
+A filter that keeps some items and drops others makes a judgment a reader cannot see in the kept
+set alone. ``apply_filters`` runs named filters in order; the first filter that rejects an item
+gives its reason code, so every dropped item has exactly one. The ledger records the input
+(count and a digest over the items' content hashes, in order), one row per dropped item (index,
+id, content hash, reason), counts by reason, and the parameters of each filter.
+
+``write_ledger`` saves the ledger and the input items beside it (``ledger.json`` and
+``input.jsonl``), so a second person can recompute every count. ``verify_ledger`` does that
+recomputation: it rebuilds the input digest, checks every row against the input, checks that
+kept plus dropped equals the input and that the counts by reason match the rows, and, for the
+built-in scope filter, runs the filter again and compares.
+
+A ledger makes the filtering inspectable. It does not show that the filtering was fair, and a
+reason code can still carry a judgment inside its label.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+from collections.abc import Callable, Sequence
+from dataclasses import dataclass, field
+from typing import Any
+
+from gather.item import Item, make_item
+from gather.scope import in_scope
+
+LEDGER_SCHEMA = "gather.filter-ledger/1"
+SCOPE_REASON = "out-of-scope"
+DOES_NOT_PROVE = "that the filtering was fair; it makes the filtering inspectable"
+
+
+@dataclass(frozen=True, slots=True)
+class Filter:
+    """One named filter: ``keep(item)`` returns True to keep; a drop is recorded as ``reason``."""
+
+    reason: str
+    keep: Callable[[Item], bool]
+    params: dict[str, Any] = field(default_factory=dict)
+
+
+def scope_filter(terms: Sequence[str]) -> Filter:
+    term_list = [t for t in terms if t]
+    return Filter(SCOPE_REASON, lambda it: in_scope(it, term_list), {"filter": "scope", "terms": term_list})
+
+
+def input_digest(hashes: Sequence[str]) -> str:
+    return hashlib.sha256("\n".join(hashes).encode("utf-8")).hexdigest()
+
+
+def apply_filters(items: Sequence[Item], filters: Sequence[Filter]) -> tuple[list[Item], dict]:
+    """The kept items, in order, and the ledger of the dropped."""
+    kept: list[Item] = []
+    rows: list[dict] = []
+    for index, it in enumerate(items):
+        reason = next((f.reason for f in filters if not f.keep(it)), None)
+        if reason is None:
+            kept.append(it)
+        else:
+            rows.append({"index": index, "id": it.id, "sha256": it.provenance.sha256, "reason": reason})
+    by_reason = {f.reason: sum(1 for r in rows if r["reason"] == f.reason) for f in filters}
+    ledger = {
+        "schema": LEDGER_SCHEMA,
+        "filters": [{"reason": f.reason, **f.params} for f in filters],
+        "input": {"count": len(items), "digest": input_digest([i.provenance.sha256 for i in items])},
+        "kept": len(kept), "dropped": len(rows), "by_reason": by_reason, "rows": rows,
+        "does_not_prove": DOES_NOT_PROVE,
+    }
+    return kept, ledger
+
+
+def scope_with_ledger(items: Sequence[Item], terms: Sequence[str]) -> tuple[list[Item], dict]:
+    """The scope filter with its ledger. No terms keeps everything and drops nothing."""
+    return apply_filters(items, [scope_filter(terms)] if [t for t in terms if t] else [])
+
+
+def _item_row(it: Item) -> dict:
+    p = it.provenance
+    return {"kind": it.kind, "id": it.id, "title": it.title, "text": it.text, "source": p.source,
+            "ref": p.ref, "method": p.method, "fetched_at": p.fetched_at, "sha256": p.sha256}
+
+
+def write_ledger(directory: str, ledger: dict, items: Sequence[Item]) -> dict[str, str]:
+    """Write ``ledger.json`` and ``input.jsonl`` into ``directory`` (created if absent)."""
+    os.makedirs(directory, exist_ok=True)
+    paths = {"ledger": os.path.join(directory, "ledger.json"), "input": os.path.join(directory, "input.jsonl")}
+    with open(paths["input"], "w", encoding="utf-8") as f:
+        for it in items:
+            f.write(json.dumps(_item_row(it), ensure_ascii=False, sort_keys=True) + "\n")
+    with open(paths["ledger"], "w", encoding="utf-8") as f:
+        json.dump(ledger, f, indent=2, ensure_ascii=False)
+    return paths
+
+
+def read_input(path: str) -> list[Item]:
+    """Items from an ``input.jsonl``. A row whose text no longer matches its hash is an error."""
+    items = []
+    with open(path, encoding="utf-8") as f:
+        for n, line in enumerate(f, 1):
+            if not line.strip():
+                continue
+            r = json.loads(line)
+            it = make_item(kind=r["kind"], id=r["id"], title=r["title"], text=r["text"], source=r["source"],
+                           ref=r["ref"], method=r["method"], fetched_at=r["fetched_at"])
+            if it.provenance.sha256 != r["sha256"]:
+                raise ValueError(f"input line {n}: text does not match its sha256")
+            items.append(it)
+    return items