gather-engine 2.1.0__tar.gz → 2.3.0__tar.gz

{gather_engine-2.1.0 → gather_engine-2.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gather-engine
-Version: 2.1.0
+Version: 2.3.0
 Summary: Accountable research intake for difficult sources, with provenance receipts and witnessed digests.
 Author: Zain Dana Harper
 License-Expression: LicenseRef-FSL-1.1-MIT
@@ -34,7 +34,7 @@ Dynamic: license-file
 [![downloads](https://img.shields.io/pypi/dm/gather-engine?label=downloads&style=flat-square&labelColor=14041b)](https://pypi.org/project/gather-engine/)
 ![python: 3.11+](https://img.shields.io/badge/python-3.11%2B-blue?style=flat-square&labelColor=14041b)
 ![deps: none (core)](https://img.shields.io/badge/core%20deps-none-success?style=flat-square&labelColor=14041b)
-![version: 2.1.0](https://img.shields.io/badge/version-2.1.0-26dfe8?style=flat-square&labelColor=14041b)
+![version: 2.3.0](https://img.shields.io/badge/version-2.3.0-26dfe8?style=flat-square&labelColor=14041b)
 
 gather pulls research out of the places most tools break on: arXiv papers, authenticated JSON APIs, JavaScript-rendered pages via a real headless browser, scanned images through OCR, and audio through transcription, alongside video, web, feeds, and local docs. The core runs with zero third-party runtime dependencies, and the same engine is reachable from the CLI, MCP tools, and plain Python. Every run writes a receipt you can re-check.
 
@@ -42,7 +42,7 @@ gather pulls research out of the places most tools break on: arXiv papers, authe
 
 ## Current status
 
-`gather-engine 2.1.0` is the current source version. Core intake,
+`gather-engine 2.3.0` is the current source version. Core intake,
 content-addressed corpus storage, exact UTF-8 source-byte receipts, readable
 context selection, descriptor handoff, pilot monitoring, CLI, Python API, and MCP
 surfaces are present in this checkout. Browser rendering, OCR, audio, and fast
@@ -82,7 +82,9 @@ fabricated field cannot reach the corpus by being plausible.
 - **Track elements across redesigns.** Fingerprint a scraped element once, then relocate it in any later version of the page and get a typed MATCH, RELOCATED, DRIFT, or GONE verdict (`gather.track`, Python API).
 - **Structured extraction with a hallucination check.** `gather.schema_extract` binds schema fields to source nodes, and `verify_record` rejects any LLM-proposed field value not grounded in the fetched content.
 - **Streaming extraction.** `gather.stream` parses an HTML stream chunk by chunk and emits partial-update commits as blocks complete, each folded into a hash chain, so a streamed extraction is replayable.
-- **Hard-source adapters behind one shape.** Video with captions and comments (`yt-dlp`), static web, RSS/Atom feeds, local docs, arXiv, PDFs (`pdftotext`), authenticated JSON APIs (token from env, never logged), JS-rendered pages (headless Chromium), scanned images (`tesseract`), and audio (`whisper`). Each external tool is optional and only needed for its own adapter.
+- **Hard-source adapters behind one shape.** Video with captions and comments (`yt-dlp` first, with the official YouTube Data API as a fallback when you supply your own key; each item records which path served it and how fast), Reddit through its official Data API with your own app credentials, static web, RSS/Atom feeds, local docs, arXiv, PDFs (`pdftotext`), authenticated JSON APIs (token from env, never logged), JS-rendered pages (headless Chromium), scanned images (`tesseract`), and audio (`whisper`). Each external tool is optional and only needed for its own adapter.
+- **Cited reports, checked in code.** `gather report` asks a model on your own machine to answer from fixed excerpts with exact quotes, then checks every quote against the excerpt it cites and marks any it cannot verify. An answer that cites an excerpt without quoting it is sent back for a rewrite, up to twice.
+- **Filter ledgers.** `--scope TERMS --ledger DIR` writes one row per dropped item with its reason code, plus the input, so `gather ledger verify` can recompute every count.
 - **Channel and playlist intake.** `gather channel <url> --store DIR` lists a channel's videos, shorts, and streams tabs (or one playlist) and gathers every entry with bounded concurrency, resumable from a per-pass ledger. Metadata and comments run as one pass and captions as another (`--no-captions`, `--captions-only`), with polite pacing, bounded backoff on HTTP 429 and session rate limits, a stop at the first bot check (never retried or answered), one caption track per video, and a run summary that counts every outcome by reason.
 - **Scholarly-graph federation.** `gather scholar` queries OpenAlex, Semantic Scholar, and Crossref in one call, dedupes results by normalized DOI (never a fuzzy title match), and can capture citation edges as first-class records with `--edges`.
 - **A durable local corpus.** Any fetch command takes `--store DIR`: bodies are content-addressed and deduped by hash, new writes preserve exact UTF-8 source bytes, and `gather corpus list|verify|digest|runs|search|stats|prune|availability|context` inspects, re-checks, and queries what you stored.
@@ -250,7 +252,7 @@ Keep the README, package metadata, and examples aligned with current behavior be
 
 ---
 
-**[Zentropy Labs](https://github.com/ZentropyLabs-ai)** · order out of entropy. An independent lab building evidence-first tools that leave a re-checkable artifact behind. Built by Zain Dana Harper in Seattle. The full workbench is at [Project Telos](https://harperz9.github.io).
+Built by **[Zain Dana Harper](https://harperz9.github.io)** in Seattle: evidence-first tools that leave a re-checkable artifact behind. The full workbench is at [Project Telos](https://harperz9.github.io).
 
 ## Local client packages
 

{gather_engine-2.1.0 → gather_engine-2.3.0}/README.md

@@ -8,7 +8,7 @@
 [![downloads](https://img.shields.io/pypi/dm/gather-engine?label=downloads&style=flat-square&labelColor=14041b)](https://pypi.org/project/gather-engine/)
 ![python: 3.11+](https://img.shields.io/badge/python-3.11%2B-blue?style=flat-square&labelColor=14041b)
 ![deps: none (core)](https://img.shields.io/badge/core%20deps-none-success?style=flat-square&labelColor=14041b)
-![version: 2.1.0](https://img.shields.io/badge/version-2.1.0-26dfe8?style=flat-square&labelColor=14041b)
+![version: 2.3.0](https://img.shields.io/badge/version-2.3.0-26dfe8?style=flat-square&labelColor=14041b)
 
 gather pulls research out of the places most tools break on: arXiv papers, authenticated JSON APIs, JavaScript-rendered pages via a real headless browser, scanned images through OCR, and audio through transcription, alongside video, web, feeds, and local docs. The core runs with zero third-party runtime dependencies, and the same engine is reachable from the CLI, MCP tools, and plain Python. Every run writes a receipt you can re-check.
 
@@ -16,7 +16,7 @@ gather pulls research out of the places most tools break on: arXiv papers, authe
 
 ## Current status
 
-`gather-engine 2.1.0` is the current source version. Core intake,
+`gather-engine 2.3.0` is the current source version. Core intake,
 content-addressed corpus storage, exact UTF-8 source-byte receipts, readable
 context selection, descriptor handoff, pilot monitoring, CLI, Python API, and MCP
 surfaces are present in this checkout. Browser rendering, OCR, audio, and fast
@@ -56,7 +56,9 @@ fabricated field cannot reach the corpus by being plausible.
 - **Track elements across redesigns.** Fingerprint a scraped element once, then relocate it in any later version of the page and get a typed MATCH, RELOCATED, DRIFT, or GONE verdict (`gather.track`, Python API).
 - **Structured extraction with a hallucination check.** `gather.schema_extract` binds schema fields to source nodes, and `verify_record` rejects any LLM-proposed field value not grounded in the fetched content.
 - **Streaming extraction.** `gather.stream` parses an HTML stream chunk by chunk and emits partial-update commits as blocks complete, each folded into a hash chain, so a streamed extraction is replayable.
-- **Hard-source adapters behind one shape.** Video with captions and comments (`yt-dlp`), static web, RSS/Atom feeds, local docs, arXiv, PDFs (`pdftotext`), authenticated JSON APIs (token from env, never logged), JS-rendered pages (headless Chromium), scanned images (`tesseract`), and audio (`whisper`). Each external tool is optional and only needed for its own adapter.
+- **Hard-source adapters behind one shape.** Video with captions and comments (`yt-dlp` first, with the official YouTube Data API as a fallback when you supply your own key; each item records which path served it and how fast), Reddit through its official Data API with your own app credentials, static web, RSS/Atom feeds, local docs, arXiv, PDFs (`pdftotext`), authenticated JSON APIs (token from env, never logged), JS-rendered pages (headless Chromium), scanned images (`tesseract`), and audio (`whisper`). Each external tool is optional and only needed for its own adapter.
+- **Cited reports, checked in code.** `gather report` asks a model on your own machine to answer from fixed excerpts with exact quotes, then checks every quote against the excerpt it cites and marks any it cannot verify. An answer that cites an excerpt without quoting it is sent back for a rewrite, up to twice.
+- **Filter ledgers.** `--scope TERMS --ledger DIR` writes one row per dropped item with its reason code, plus the input, so `gather ledger verify` can recompute every count.
 - **Channel and playlist intake.** `gather channel <url> --store DIR` lists a channel's videos, shorts, and streams tabs (or one playlist) and gathers every entry with bounded concurrency, resumable from a per-pass ledger. Metadata and comments run as one pass and captions as another (`--no-captions`, `--captions-only`), with polite pacing, bounded backoff on HTTP 429 and session rate limits, a stop at the first bot check (never retried or answered), one caption track per video, and a run summary that counts every outcome by reason.
 - **Scholarly-graph federation.** `gather scholar` queries OpenAlex, Semantic Scholar, and Crossref in one call, dedupes results by normalized DOI (never a fuzzy title match), and can capture citation edges as first-class records with `--edges`.
 - **A durable local corpus.** Any fetch command takes `--store DIR`: bodies are content-addressed and deduped by hash, new writes preserve exact UTF-8 source bytes, and `gather corpus list|verify|digest|runs|search|stats|prune|availability|context` inspects, re-checks, and queries what you stored.
@@ -224,7 +226,7 @@ Keep the README, package metadata, and examples aligned with current behavior be
 
 ---
 
-**[Zentropy Labs](https://github.com/ZentropyLabs-ai)** · order out of entropy. An independent lab building evidence-first tools that leave a re-checkable artifact behind. Built by Zain Dana Harper in Seattle. The full workbench is at [Project Telos](https://harperz9.github.io).
+Built by **[Zain Dana Harper](https://harperz9.github.io)** in Seattle: evidence-first tools that leave a re-checkable artifact behind. The full workbench is at [Project Telos](https://harperz9.github.io).
 
 ## Local client packages
 

{gather_engine-2.1.0 → gather_engine-2.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "gather-engine"
-version = "2.1.0"
+version = "2.3.0"
 description = "Accountable research intake for difficult sources, with provenance receipts and witnessed digests."
 readme = "README.md"
 requires-python = ">=3.11"

{gather_engine-2.1.0 → gather_engine-2.3.0}/src/gather/__init__.py

@@ -28,7 +28,7 @@ from gather.scope import filter_scope, in_scope
 from gather.source import Catalog, Source
 from gather.store import Corpus
 
-__version__ = "2.1.0"
+__version__ = "2.3.0"
 
 __all__ = [
     "Catalog", "Corpus", "CorpusRootDescriptor", "CorpusRootIdentity", "Digest", "Item", "NullProvenanceProvider", "NullSynthesizer",

gather_engine-2.3.0/src/gather/citecheck.py

@@ -0,0 +1,119 @@
+"""A code check on every citation in a report. Pure: no network, no model.
+
+A report cites an excerpt by quoting it and naming its number: ``"the exact words" [2]``. For
+each citation the check decides, by string comparison alone:
+
+- ``verified``: the quoted span appears in excerpt 2 (after Unicode, quote-mark, dash and
+  whitespace normalisation, and case folding);
+- ``not-in-source``: excerpt 2 exists and does not contain the span;
+- ``unknown-source``: there is no excerpt 2;
+- ``too-short``: the quote has fewer than ``MIN_QUOTE_WORDS`` words, so a match would prove
+  little; it is not counted as verified;
+- ``unchecked``: a bracketed number with no quote before it, so there is nothing to compare.
+
+Sentences with no citation at all are listed as uncited. Nothing is hidden: every citation and
+every uncited sentence is in the result. Precision is verified citations over all citations.
+
+What this does not check: whether the sentence around a verified quote says what the excerpt
+means. A quote can be exact and the claim around it can still stretch it.
+"""
+
+from __future__ import annotations
+
+import re
+import unicodedata
+from dataclasses import dataclass
+
+CHECK_SCHEMA = "gather.citation-check/1"
+MIN_QUOTE_WORDS = 4
+STATUSES = ("verified", "not-in-source", "unknown-source", "too-short", "unchecked")
+
+_MARKS = str.maketrans({"“": '"', "”": '"', "„": '"', "«": '"', "»": '"',
+                        "‘": "'", "’": "'", "–": "-", "—": "-", "−": "-",
+                        " ": " "})
+# a double-quoted span, optional space, then [n]
+_QUOTE_CITE = re.compile(r'"([^"]{1,2000})"\s*\[(\d{1,4})\]')
+_BARE_CITE = re.compile(r"\[(\d{1,4})\]")
+_LIST_MARK = re.compile(r"^(?:[-*+]|\d{1,3}[.)])\s+")
+_SENTENCE = re.compile(r'(?<=[.!?])["\')\]]*\s+(?=["(\[A-Z0-9])')
+
+
+def normalize(text: str) -> str:
+    """NFKC, straight quote marks and hyphens, collapsed whitespace, case folded."""
+    text = unicodedata.normalize("NFKC", text).translate(_MARKS)
+    return re.sub(r"\s+", " ", text).strip().casefold()
+
+
+def _strip_edges(quote: str) -> str:
+    return normalize(quote).strip(" .,;:!?'")
+
+
+@dataclass(frozen=True, slots=True)
+class Citation:
+    sentence: int
+    source: int
+    quote: str | None
+    status: str
+
+    def to_dict(self) -> dict:
+        return {"sentence": self.sentence, "source": self.source, "quote": self.quote, "status": self.status}
+
+
+def _split_line(line: str) -> list[str]:
+    """Split one line at sentence ends that fall outside a double-quoted span."""
+    parts, start = [], 0
+    for m in _SENTENCE.finditer(line):
+        if line[:m.start()].count('"') % 2 == 0:
+            parts.append(line[start:m.start() + len(m.group(0).rstrip())].strip())
+            start = m.end()
+    parts.append(line[start:].strip())
+    return [p for p in parts if p]
+
+
+def sentences(report: str) -> list[str]:
+    """The report's sentences: lines split at sentence ends outside quotes, list markers removed,
+    headings and blank lines dropped."""
+    out: list[str] = []
+    for raw in report.splitlines():
+        line = raw.strip()
+        if not line or line.startswith("#"):
+            continue
+        out.extend(_split_line(_LIST_MARK.sub("", line.translate(_MARKS))))
+    return out
+
+
+def _status(quote: str, source: int, excerpts: list[str]) -> str:
+    if not 1 <= source <= len(excerpts):
+        return "unknown-source"
+    span = _strip_edges(quote)
+    if len(span.split()) < MIN_QUOTE_WORDS:
+        return "too-short"
+    return "verified" if span in normalize(excerpts[source - 1]) else "not-in-source"
+
+
+def _sentence_citations(n: int, sentence: str, excerpts: list[str]) -> list[Citation]:
+    found = [Citation(n, int(m.group(2)), m.group(1), _status(m.group(1), int(m.group(2)), excerpts))
+             for m in _QUOTE_CITE.finditer(sentence.translate(_MARKS))]
+    bare_text = _QUOTE_CITE.sub("", sentence.translate(_MARKS))
+    found += [Citation(n, int(m.group(1)), None, "unchecked") for m in _BARE_CITE.finditer(bare_text)]
+    return found
+
+
+def check_report(report: str, excerpts: list[str]) -> dict:
+    """Check every citation in ``report`` against ``excerpts`` (excerpt 1 is ``excerpts[0]``)."""
+    cites: list[Citation] = []
+    uncited: list[str] = []
+    for n, sentence in enumerate(sentences(report)):
+        found = _sentence_citations(n, sentence, excerpts)
+        cites.extend(found)
+        if not found:
+            uncited.append(sentence)
+    counts = {s: sum(1 for c in cites if c.status == s) for s in STATUSES}
+    total = len(cites)
+    return {
+        "schema": CHECK_SCHEMA, "excerpts": len(excerpts), "citations": [c.to_dict() for c in cites],
+        "counts": counts, "total": total,
+        "precision": round(counts["verified"] / total, 4) if total else None,
+        "uncited_sentences": uncited,
+        "does_not_prove": "that the claim around a verified quote matches the excerpt's meaning",
+    }

{gather_engine-2.1.0 → gather_engine-2.3.0}/src/gather/cli.py

@@ -4,6 +4,7 @@ import argparse
 import sys
 
 from gather import __version__
+from gather.cli_ext import add_extension_parsers
 from gather.commands import (
     cmd_api,
     cmd_arxiv,
@@ -11,12 +12,10 @@ from gather.commands import (
     cmd_docs,
     cmd_feed,
     cmd_ocr,
-    cmd_parse,
     cmd_pdf,
     cmd_run,
     cmd_scholar,
     cmd_transcribe,
-    cmd_video,
     cmd_web,
 )
 from gather.corpus_cmd import cmd_corpus
@@ -29,7 +28,6 @@ from gather.pilot_commands import (
     cmd_pilot_run,
     cmd_pilot_verify,
 )
-from gather.video_cmd import add_channel_parser, add_ytdlp_options
 from gather.web_commands import (
     cmd_caps,
     cmd_crawl,
@@ -50,6 +48,8 @@ def _add_common(p: argparse.ArgumentParser) -> None:
     p.add_argument("--scope", default=None, help="comma-separated scope terms; keep items mentioning any")
     p.add_argument("--json", action="store_true", help="emit the catalog and digest as JSON")
     p.add_argument("--store", default=None, metavar="DIR", help="persist gathered items into a corpus at DIR")
+    p.add_argument("--ledger", default=None, metavar="DIR",
+                   help="write a filter ledger (ledger.json, input.jsonl) of what --scope dropped into DIR")
 
 
 def _add_flagship_commands(sub) -> None:
@@ -124,21 +124,7 @@ def build_parser() -> argparse.ArgumentParser:
     sub = parser.add_subparsers(dest="command")
     _add_flagship_commands(sub)
 
-    parse = sub.add_parser("parse", help="parse a saved yt-dlp info.json (+ optional .vtt), offline, no network")
-    parse.add_argument("info", help="path to a yt-dlp info.json")
-    parse.add_argument("--vtt", default=None, help="path to a .vtt captions file")
-    parse.add_argument("--auto-captions", action="store_true",
-                       help="captions are machine-generated: collapse rolling-window growth, stamp auto-caption")
-    _add_common(parse)
-    parse.set_defaults(func=cmd_parse)
-
-    video = sub.add_parser("video", help="fetch a video via yt-dlp (needs yt-dlp on PATH and network)")
-    video.add_argument("url")
-    video.add_argument("--comments", action="store_true", help="also gather comments")
-    add_ytdlp_options(video)
-    _add_common(video)
-    video.set_defaults(func=cmd_video)
-    add_channel_parser(sub)
+    add_extension_parsers(sub, _add_common)
 
     web = sub.add_parser("web", help="fetch a static web page via http(s) and extract readable text")
     web.add_argument("url")

gather_engine-2.3.0/src/gather/cli_ext.py

@@ -0,0 +1,19 @@
+"""The subcommands that live in their own modules, registered in one place for ``gather.cli``."""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+
+
+def add_extension_parsers(sub, add_common: Callable) -> None:
+    from gather.ledger_cmd import add_ledger_parser
+    from gather.reddit_cmd import add_reddit_parser
+    from gather.report_cmd import add_report_parsers
+    from gather.video_cmd import add_channel_parser
+    from gather.youtube_cmd import add_video_parsers
+
+    add_video_parsers(sub, add_common)
+    add_channel_parser(sub)
+    add_reddit_parser(sub, add_common)
+    add_report_parsers(sub, add_common)
+    add_ledger_parser(sub)

{gather_engine-2.1.0 → gather_engine-2.3.0}/src/gather/client_mcp.py

@@ -156,23 +156,22 @@ def main(argv=None):
     return 0
 
 def definitions(network=None):
-    from gather.mcp import _tool_defs
-    tools = [d for d in _tool_defs() if d["name"] in {"gather.docs"}]
+    from gather.docs_tool import annotate, docs_tool
+    tools = [annotate(docs_tool())]
     if network is not None and network.origins:
-        tools.append({"name": "gather.fetch", "description": "Read an allowed origin; return untrusted source text and a byte receipt. No redirects or credentials.",
-                      "inputSchema": {"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"], "additionalProperties": False}})
+        tools.append(annotate({"name": "gather.fetch", "description": "Read an allowed origin; return untrusted source text and a byte receipt. No redirects or credentials.",
+                      "inputSchema": {"type": "object", "properties": {"url": {"type": "string"}}, "required": ["url"], "additionalProperties": False}}))
     return tools
 
 
 def invoke(name, args, root):
-    from gather.grants import NONE
-    from gather.mcp import call_tool
+    from gather.docs_tool import call_docs
     # Single files avoid a directory walk following a file link outside the grant.
     path = confined(root, args["path"])
     if not path.is_file():
         raise ClientRefusal("local client intake requires one text file per call")
     args["path"] = str(path)
-    return call_tool(name, args, grants=NONE)
+    return call_docs(args)
 
 
 if __name__ == "__main__":

{gather_engine-2.1.0 → gather_engine-2.3.0}/src/gather/commands.py

@@ -13,12 +13,22 @@ def _scope(args) -> list[str]:
     return _split(args.scope)
 
 
-def _emit(items, scope, as_json, store=None) -> int:
+def _scoped(items, scope, ledger=None):
+    """The scope filter; with ``ledger``, every drop is also written to a ledger beside its input."""
+    if not ledger:
+        from gather.scope import filter_scope
+        return filter_scope(items, scope)
+    from gather.filter_ledger import scope_with_ledger, write_ledger
+    kept, record = scope_with_ledger(items, scope)
+    write_ledger(ledger, record, items)
+    return kept, record["dropped"]
+
+
+def _emit(items, scope, as_json, store=None, ledger=None) -> int:
     from gather.digest import digest, verify_digest
     from gather.payloads import catalog_digest_payload
-    from gather.scope import filter_scope
 
-    kept, dropped = filter_scope(items, scope)
+    kept, dropped = _scoped(items, scope, ledger)
     d = digest(kept)
     stored = None
     if store:
@@ -48,7 +58,7 @@ def _fetch_and_emit(fetch, args, fail: str = "fetch failed") -> int:
     except Exception as exc:
         print(f"{fail}: {exc}", file=sys.stderr)
         return 1
-    return _emit(items, _scope(args), args.json, store=args.store)
+    return _emit(items, _scope(args), args.json, store=args.store, ledger=getattr(args, "ledger", None))
 
 
 def cmd_parse(args) -> int:
@@ -61,13 +71,13 @@ def cmd_parse(args) -> int:
         with open(args.vtt, encoding="utf-8") as f:
             vtt = f.read()
     items = parse_video(info_json, vtt, fetched_at=time.time(), auto_captions=args.auto_captions)
-    return _emit(items, _scope(args), args.json, store=args.store)
+    return _emit(items, _scope(args), args.json, store=args.store, ledger=args.ledger)
 
 
 def cmd_video(args) -> int:
-    from gather.video_cmd import video_source_from_args
+    from gather.youtube_cmd import route_from_args
     try:
-        src = video_source_from_args(args, with_comments=args.comments)
+        src = route_from_args(args, with_comments=args.comments)
     except ValueError as exc:
         print(f"fetch failed: {exc}", file=sys.stderr)
         return 2
@@ -115,17 +125,16 @@ def cmd_scholar(args) -> int:
     except Exception as exc:
         print(f"scholar failed: {exc}", file=sys.stderr)
         return 1
-    return _emit_with_edges(items, edges, _scope(args), args.json, store=args.store)
+    return _emit_with_edges(items, edges, _scope(args), args.json, store=args.store, ledger=args.ledger)
 
 
-def _emit_with_edges(items, edges, scope, as_json, store=None) -> int:
+def _emit_with_edges(items, edges, scope, as_json, store=None, ledger=None) -> int:
     """Emit scholar items plus citation-edge receipts, folding the edges into the same witnessed
     digest as the papers, so the seal covers the graph (nodes and links), not just the nodes."""
     from gather.digest import digest_of_receipts, verify_digest
-    from gather.scope import filter_scope
     from gather.store import Corpus
 
-    kept, dropped = filter_scope(items, scope)
+    kept, dropped = _scoped(items, scope, ledger)
     item_receipts = [
         {"kind": i.kind, "id": i.id, "title": i.title, "source": i.provenance.source,
          "ref": i.provenance.ref, "method": i.provenance.method, "sha256": i.provenance.sha256,

gather_engine-2.3.0/src/gather/docs_tool.py

@@ -0,0 +1,93 @@
+"""The gather.docs tool and the MCP tool annotations.
+
+Shared by the full MCP server and the local client profile, so the client can
+serve gather.docs without importing the full tool surface.
+"""
+from __future__ import annotations
+
+import json
+
+from gather.localpath import require_local
+from gather.payloads import catalog_digest_payload
+from gather.scope import filter_scope
+
+
+def scope_terms(raw: object) -> list[str]:
+    if raw is None:
+        return []
+    if isinstance(raw, str):
+        return [term.strip() for term in raw.split(",") if term.strip()]
+    if isinstance(raw, list) and all(isinstance(term, str) for term in raw):
+        return [term.strip() for term in raw if term.strip()]
+    raise ValueError("scope must be a comma-separated string or a list of strings")
+
+
+def payload_from_items(items, scope: list[str]) -> dict:
+    kept, dropped = filter_scope(items, scope)
+    return catalog_digest_payload(kept, dropped=dropped)
+
+
+def _hints(title: str, *, read_only: bool = True, idempotent: bool = True,
+           open_world: bool = False) -> dict:
+    return {"title": title, "readOnlyHint": read_only, "destructiveHint": False,
+            "idempotentHint": idempotent, "openWorldHint": open_world}
+
+
+# MCP tool annotations. A hint describes the tool to the client and grants
+# nothing; launch grants still refuse network sources, commands and credentials.
+TOOL_ANNOTATIONS = {
+    "gather.status": _hints("Gather status"),
+    "gather.doctor": _hints("Gather readiness check"),
+    "gather.docs": _hints("Read local documents with receipts"),
+    "gather.arxiv": _hints("Fetch arXiv metadata", idempotent=False, open_world=True),
+    "gather.federation": _hints("Validate or plan a source registry"),
+    "gather.run": _hints("Run a gather config", read_only=False, idempotent=False,
+                         open_world=True),
+    "gather.context": _hints("Select verified corpus context"),
+    "gather.pilot": _hints("Run or verify a gather pilot", read_only=False, idempotent=False,
+                           open_world=True),
+    "gather.fetch": _hints("Read an allowed web origin", idempotent=False, open_world=True),
+}
+
+
+def annotate(tool: dict) -> dict:
+    notes = dict(TOOL_ANNOTATIONS[tool["name"]])
+    return {**tool, "title": notes["title"], "annotations": notes}
+
+
+def docs_tool() -> dict:
+    """The raw gather.docs definition, before annotation."""
+    return {
+        "name": "gather.docs",
+        "description": "Read a local text file or directory and return catalog rows plus digest receipts.",
+        "inputSchema": {
+            "type": "object",
+            "properties": {
+                "path": {"type": "string", "description": "local file or directory to read; a network "
+                                                          "or device path returns NON_LOCAL_PATH"},
+                "scope": {
+                    "description": (
+                        "optional post-fetch content filter: keep items whose title or body "
+                        "contains any term as a case-insensitive substring; omit for unfiltered capture"
+                    ),
+                    "oneOf": [
+                        {"type": "string"},
+                        {"type": "array", "items": {"type": "string"}},
+                    ],
+                },
+            },
+            "required": ["path"],
+        },
+    }
+
+
+def call_docs(args: dict) -> str:
+    """Run gather.docs: read one local path and return catalog rows plus digest receipts."""
+    from gather.docs import DocsSource
+
+    path = args.get("path")
+    if not isinstance(path, str) or not path:
+        raise ValueError("gather.docs requires a non-empty path")
+    require_local(path, label="path")
+    payload = payload_from_items(DocsSource().fetch(path), scope_terms(args.get("scope")))
+    return json.dumps(payload, indent=2, ensure_ascii=False, sort_keys=True)

gather_engine-2.3.0/src/gather/filter_ledger.py

@@ -0,0 +1,111 @@
+"""Filter ledgers: what a filter dropped, item by item, with one reason code each.
+
+A filter that keeps some items and drops others makes a judgment a reader cannot see in the kept
+set alone. ``apply_filters`` runs named filters in order; the first filter that rejects an item
+gives its reason code, so every dropped item has exactly one. The ledger records the input
+(count and a digest over the items' content hashes, in order), one row per dropped item (index,
+id, content hash, reason), counts by reason, and the parameters of each filter.
+
+``write_ledger`` saves the ledger and the input items beside it (``ledger.json`` and
+``input.jsonl``), so a second person can recompute every count. ``verify_ledger`` does that
+recomputation: it rebuilds the input digest, checks every row against the input, checks that
+kept plus dropped equals the input and that the counts by reason match the rows, and, for the
+built-in scope filter, runs the filter again and compares.
+
+A ledger makes the filtering inspectable. It does not show that the filtering was fair, and a
+reason code can still carry a judgment inside its label.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import os
+from collections.abc import Callable, Sequence
+from dataclasses import dataclass, field
+from typing import Any
+
+from gather.item import Item, make_item
+from gather.scope import in_scope
+
+LEDGER_SCHEMA = "gather.filter-ledger/1"
+SCOPE_REASON = "out-of-scope"
+DOES_NOT_PROVE = "that the filtering was fair; it makes the filtering inspectable"
+
+
+@dataclass(frozen=True, slots=True)
+class Filter:
+    """One named filter: ``keep(item)`` returns True to keep; a drop is recorded as ``reason``."""
+
+    reason: str
+    keep: Callable[[Item], bool]
+    params: dict[str, Any] = field(default_factory=dict)
+
+
+def scope_filter(terms: Sequence[str]) -> Filter:
+    term_list = [t for t in terms if t]
+    return Filter(SCOPE_REASON, lambda it: in_scope(it, term_list), {"filter": "scope", "terms": term_list})
+
+
+def input_digest(hashes: Sequence[str]) -> str:
+    return hashlib.sha256("\n".join(hashes).encode("utf-8")).hexdigest()
+
+
+def apply_filters(items: Sequence[Item], filters: Sequence[Filter]) -> tuple[list[Item], dict]:
+    """The kept items, in order, and the ledger of the dropped."""
+    kept: list[Item] = []
+    rows: list[dict] = []
+    for index, it in enumerate(items):
+        reason = next((f.reason for f in filters if not f.keep(it)), None)
+        if reason is None:
+            kept.append(it)
+        else:
+            rows.append({"index": index, "id": it.id, "sha256": it.provenance.sha256, "reason": reason})
+    by_reason = {f.reason: sum(1 for r in rows if r["reason"] == f.reason) for f in filters}
+    ledger = {
+        "schema": LEDGER_SCHEMA,
+        "filters": [{"reason": f.reason, **f.params} for f in filters],
+        "input": {"count": len(items), "digest": input_digest([i.provenance.sha256 for i in items])},
+        "kept": len(kept), "dropped": len(rows), "by_reason": by_reason, "rows": rows,
+        "does_not_prove": DOES_NOT_PROVE,
+    }
+    return kept, ledger
+
+
+def scope_with_ledger(items: Sequence[Item], terms: Sequence[str]) -> tuple[list[Item], dict]:
+    """The scope filter with its ledger. No terms keeps everything and drops nothing."""
+    return apply_filters(items, [scope_filter(terms)] if [t for t in terms if t] else [])
+
+
+def _item_row(it: Item) -> dict:
+    p = it.provenance
+    return {"kind": it.kind, "id": it.id, "title": it.title, "text": it.text, "source": p.source,
+            "ref": p.ref, "method": p.method, "fetched_at": p.fetched_at, "sha256": p.sha256}
+
+
+def write_ledger(directory: str, ledger: dict, items: Sequence[Item]) -> dict[str, str]:
+    """Write ``ledger.json`` and ``input.jsonl`` into ``directory`` (created if absent)."""
+    os.makedirs(directory, exist_ok=True)
+    paths = {"ledger": os.path.join(directory, "ledger.json"), "input": os.path.join(directory, "input.jsonl")}
+    with open(paths["input"], "w", encoding="utf-8") as f:
+        for it in items:
+            f.write(json.dumps(_item_row(it), ensure_ascii=False, sort_keys=True) + "\n")
+    with open(paths["ledger"], "w", encoding="utf-8") as f:
+        json.dump(ledger, f, indent=2, ensure_ascii=False)
+    return paths
+
+
+def read_input(path: str) -> list[Item]:
+    """Items from an ``input.jsonl``. A row whose text no longer matches its hash is an error."""
+    items = []
+    with open(path, encoding="utf-8") as f:
+        for n, line in enumerate(f, 1):
+            if not line.strip():
+                continue
+            r = json.loads(line)
+            it = make_item(kind=r["kind"], id=r["id"], title=r["title"], text=r["text"], source=r["source"],
+                           ref=r["ref"], method=r["method"], fetched_at=r["fetched_at"])
+            if it.provenance.sha256 != r["sha256"]:
+                raise ValueError(f"input line {n}: text does not match its sha256")
+            items.append(it)
+    return items