gather-engine 2.5.0__tar.gz → 2.6.0__tar.gz

{gather_engine-2.5.0 → gather_engine-2.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gather-engine
-Version: 2.5.0
+Version: 2.6.0
 Summary: Collect web pages, papers, APIs, PDFs, scans, audio and video into a local corpus where every item records how it arrived.
 Author-email: Zain Dana Harper <zaindharper@gmail.com>
 License-Expression: AGPL-3.0-only
@@ -24,13 +24,17 @@ Requires-Dist: pytest>=8; extra == "dev"
 Requires-Dist: pytest-cov>=5; extra == "dev"
 Requires-Dist: ruff>=0.6; extra == "dev"
 Requires-Dist: mypy>=1.10; extra == "dev"
+Requires-Dist: Pillow>=10.1; extra == "dev"
 Provides-Extra: fast
 Requires-Dist: lxml>=5; extra == "fast"
 Provides-Extra: browser
 Requires-Dist: playwright>=1.4; extra == "browser"
+Provides-Extra: frames
+Requires-Dist: Pillow>=10.1; extra == "frames"
 Provides-Extra: all
 Requires-Dist: lxml>=5; extra == "all"
 Requires-Dist: playwright>=1.4; extra == "all"
+Requires-Dist: Pillow>=10.1; extra == "all"
 Dynamic: license-file
 
 <picture>
@@ -46,11 +50,11 @@ Gather is for researchers, journalists and developers who feed sources to a pers
 
 ## Install and first run
 
-Python 3.11 or newer. Version 2.5.0 is on PyPI as `gather-engine`.
+Python 3.11 or newer. Version 2.6.0 is on PyPI as `gather-engine`.
 
 ```bash
 pip install gather-engine
-gather --version                      # gather 2.5.0
+gather --version                      # gather 2.6.0
 ```
 
 Extract a saved page into hashed blocks, each tied to its place in the page:
@@ -124,6 +128,7 @@ fabricated field cannot reach the corpus by being plausible.
 - **Hard-source adapters behind one shape.** Video with captions and comments (`yt-dlp` first, with the official YouTube Data API as a fallback when you supply your own key; each item records which path served it and how fast), Reddit through its official Data API with your own app credentials, static web, RSS/Atom feeds, local docs, arXiv, PDFs (`pdftotext`), authenticated JSON APIs (token from env, never logged), JS-rendered pages (headless Chromium), scanned images (`tesseract`), and audio (`whisper`). Each external tool is optional and only needed for its own adapter.
 - **Cited reports, checked in code.** `gather report` asks a model on your own machine to answer from fixed excerpts with exact quotes, then checks every quote against the excerpt it cites and marks any it cannot verify. An answer that cites an excerpt without quoting it is sent back for a rewrite, up to twice.
 - **Filter ledgers.** `--scope TERMS --ledger DIR` writes one row per dropped item with its reason code, plus the input, so `gather ledger verify` can recompute every count.
+- **A video as one checkable bundle, with frames.** `gather video-bundle <url> --out DIR` saves the metadata and chapters, the caption track and transcripts, the comment tree, a balanced comment synthesis, and contact sheets of 24 to 96 frames across the whole video with the time printed on each. The frames come from the site's preview storyboard, so no video file is downloaded; they are small (320 by 180 pixels at best on YouTube) and the bundle records their real size. `--fast` reads captions and frames while the comments are still being read and keeps requests 1 second apart: a 234-second video that took 162 seconds with 3-second sleeps took 37 seconds. Frame sheets need the optional `frames` extra (Pillow). See [USAGE.md](https://github.com/HarperZ9/gather/blob/main/USAGE.md).
 - **Channel and playlist intake.** `gather channel <url> --store DIR` lists a channel's videos, shorts, and streams tabs (or one playlist) and gathers every entry with bounded concurrency, resumable from a per-pass ledger. Metadata and comments run as one pass and captions as another (`--no-captions`, `--captions-only`), with polite pacing, bounded backoff on HTTP 429 and session rate limits, a stop at the first bot check (never retried or answered), one caption track per video, and a run summary that counts every outcome by reason.
 - **Scholarly-graph federation.** `gather scholar` queries OpenAlex, Semantic Scholar, and Crossref in one call, dedupes results by normalized DOI (never a fuzzy title match), and can capture citation edges as first-class records with `--edges`.
 - **A durable local corpus.** Any fetch command takes `--store DIR`: bodies are content-addressed and deduped by hash, new writes preserve exact UTF-8 source bytes, and `gather corpus list|verify|digest|runs|search|stats|prune|availability|context` inspects, re-checks, and queries what you stored.
@@ -220,6 +225,24 @@ gather docs ./research-notes --scope "rubik,group theory"
 gather corpus verify ./corpus                  # re-hash every stored body; non-zero exit on corruption
 ```
 
+Turn a folder of posts saved by the CMMNS browser extension into tables, offline and by fixed rules:
+
+```bash
+gather saves ./saves --out ./saves-tables      # items, entities and media as CSV and JSON Lines
+gather saves --probe ./clip.mp4                # container, size, duration and audio fields of one file
+```
+
+The same folder gives byte-identical tables on every run. The language column is a rule and the entities are regular expressions; [USAGE.md](https://github.com/HarperZ9/gather/blob/main/USAGE.md#saved-posts-as-tables) lists the columns and the limits.
+
+Take any local file apart into a tree of its parts, then rebuild it byte for byte:
+
+```bash
+gather decompose ./clip.wav --out ./clip-tree     # tree.json, parts/, and measurement tables
+gather rebuild ./clip-tree --out ./clip-again.wav # same SHA-256, every node checked
+```
+
+ZIP, PNG, WAV, JSON, text and CMMNS saves get structural handlers; every other file is kept as one byte range and still rebuilds exactly. PCM WAV files also get loudness (BS.1770), tempo, key, spectrum and decay estimates. [USAGE.md](https://github.com/HarperZ9/gather/blob/main/USAGE.md#take-a-file-apart-and-put-it-back) lists the record, the handlers and the limits.
+
 Offline demo, no install of extra tools, nothing downloaded:
 
 ```bash
@@ -302,7 +325,7 @@ The `web` adapter reads static HTML and does not run JavaScript; a client-render
 - [docs/WEB-ENGINE-UPLIFT.md](https://github.com/HarperZ9/gather/blob/main/docs/WEB-ENGINE-UPLIFT.md): the web-data engine roadmap and benchmarks.
 - [docs/ENTERPRISE-READINESS.md](https://github.com/HarperZ9/gather/blob/main/docs/ENTERPRISE-READINESS.md): context envelopes, action receipts, and host-neutral operation for unattended agents.
 - [docs/PILOT.md](https://github.com/HarperZ9/gather/blob/main/docs/PILOT.md): the accountable pilot evidence engine, its manifest boundary, and the private/shared evidence split.
-- [CHANGELOG.md](https://github.com/HarperZ9/gather/blob/main/CHANGELOG.md): version history. Current release: 2.5.0.
+- [CHANGELOG.md](https://github.com/HarperZ9/gather/blob/main/CHANGELOG.md): version history. Current release: 2.6.0.
 
 Peer projects: [crucible](https://github.com/HarperZ9/crucible) (judgment), [index](https://github.com/HarperZ9/index) (code maps), [forum](https://github.com/HarperZ9/forum) (orchestration), [telos](https://github.com/HarperZ9/telos) (the engine).
 

gather_engine-2.5.0/src/gather_engine.egg-info/PKG-INFO → gather_engine-2.6.0/README.md

@@ -1,38 +1,3 @@
-Metadata-Version: 2.4
-Name: gather-engine
-Version: 2.5.0
-Summary: Collect web pages, papers, APIs, PDFs, scans, audio and video into a local corpus where every item records how it arrived.
-Author-email: Zain Dana Harper <zaindharper@gmail.com>
-License-Expression: AGPL-3.0-only
-Project-URL: Homepage, https://harperz9.github.io/gather.html
-Project-URL: Repository, https://github.com/HarperZ9/gather
-Project-URL: Issues, https://github.com/HarperZ9/gather/issues
-Project-URL: Changelog, https://github.com/HarperZ9/gather/blob/main/CHANGELOG.md
-Keywords: research,intake,ingestion,provenance,transcripts,arxiv,accountability,ocr,transcription,headless-browser,synthesis,web-scraping,ssrf-safe
-Classifier: Development Status :: 4 - Beta
-Classifier: Environment :: Console
-Classifier: Intended Audience :: Developers
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3 :: Only
-Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
-Requires-Python: >=3.11
-Description-Content-Type: text/markdown
-License-File: LICENSE
-License-File: NOTICE
-Provides-Extra: dev
-Requires-Dist: pytest>=8; extra == "dev"
-Requires-Dist: pytest-cov>=5; extra == "dev"
-Requires-Dist: ruff>=0.6; extra == "dev"
-Requires-Dist: mypy>=1.10; extra == "dev"
-Provides-Extra: fast
-Requires-Dist: lxml>=5; extra == "fast"
-Provides-Extra: browser
-Requires-Dist: playwright>=1.4; extra == "browser"
-Provides-Extra: all
-Requires-Dist: lxml>=5; extra == "all"
-Requires-Dist: playwright>=1.4; extra == "all"
-Dynamic: license-file
-
 <picture>
   <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/HarperZ9/gather/main/docs/art/hero-dark.svg">
   <img src="https://raw.githubusercontent.com/HarperZ9/gather/main/docs/art/hero-light.svg" alt="gather: Research intake for web pages, papers, scanned PDFs, audio and video. Five bundles of lines, solid, dashed, dotted, wavy and stepped, sweep in from the right and pass through a ring into one bright core. One line stops at the ring with a short cross mark." width="100%">
@@ -46,11 +11,11 @@ Gather is for researchers, journalists and developers who feed sources to a pers
 
 ## Install and first run
 
-Python 3.11 or newer. Version 2.5.0 is on PyPI as `gather-engine`.
+Python 3.11 or newer. Version 2.6.0 is on PyPI as `gather-engine`.
 
 ```bash
 pip install gather-engine
-gather --version                      # gather 2.5.0
+gather --version                      # gather 2.6.0
 ```
 
 Extract a saved page into hashed blocks, each tied to its place in the page:
@@ -124,6 +89,7 @@ fabricated field cannot reach the corpus by being plausible.
 - **Hard-source adapters behind one shape.** Video with captions and comments (`yt-dlp` first, with the official YouTube Data API as a fallback when you supply your own key; each item records which path served it and how fast), Reddit through its official Data API with your own app credentials, static web, RSS/Atom feeds, local docs, arXiv, PDFs (`pdftotext`), authenticated JSON APIs (token from env, never logged), JS-rendered pages (headless Chromium), scanned images (`tesseract`), and audio (`whisper`). Each external tool is optional and only needed for its own adapter.
 - **Cited reports, checked in code.** `gather report` asks a model on your own machine to answer from fixed excerpts with exact quotes, then checks every quote against the excerpt it cites and marks any it cannot verify. An answer that cites an excerpt without quoting it is sent back for a rewrite, up to twice.
 - **Filter ledgers.** `--scope TERMS --ledger DIR` writes one row per dropped item with its reason code, plus the input, so `gather ledger verify` can recompute every count.
+- **A video as one checkable bundle, with frames.** `gather video-bundle <url> --out DIR` saves the metadata and chapters, the caption track and transcripts, the comment tree, a balanced comment synthesis, and contact sheets of 24 to 96 frames across the whole video with the time printed on each. The frames come from the site's preview storyboard, so no video file is downloaded; they are small (320 by 180 pixels at best on YouTube) and the bundle records their real size. `--fast` reads captions and frames while the comments are still being read and keeps requests 1 second apart: a 234-second video that took 162 seconds with 3-second sleeps took 37 seconds. Frame sheets need the optional `frames` extra (Pillow). See [USAGE.md](https://github.com/HarperZ9/gather/blob/main/USAGE.md).
 - **Channel and playlist intake.** `gather channel <url> --store DIR` lists a channel's videos, shorts, and streams tabs (or one playlist) and gathers every entry with bounded concurrency, resumable from a per-pass ledger. Metadata and comments run as one pass and captions as another (`--no-captions`, `--captions-only`), with polite pacing, bounded backoff on HTTP 429 and session rate limits, a stop at the first bot check (never retried or answered), one caption track per video, and a run summary that counts every outcome by reason.
 - **Scholarly-graph federation.** `gather scholar` queries OpenAlex, Semantic Scholar, and Crossref in one call, dedupes results by normalized DOI (never a fuzzy title match), and can capture citation edges as first-class records with `--edges`.
 - **A durable local corpus.** Any fetch command takes `--store DIR`: bodies are content-addressed and deduped by hash, new writes preserve exact UTF-8 source bytes, and `gather corpus list|verify|digest|runs|search|stats|prune|availability|context` inspects, re-checks, and queries what you stored.
@@ -220,6 +186,24 @@ gather docs ./research-notes --scope "rubik,group theory"
 gather corpus verify ./corpus                  # re-hash every stored body; non-zero exit on corruption
 ```
 
+Turn a folder of posts saved by the CMMNS browser extension into tables, offline and by fixed rules:
+
+```bash
+gather saves ./saves --out ./saves-tables      # items, entities and media as CSV and JSON Lines
+gather saves --probe ./clip.mp4                # container, size, duration and audio fields of one file
+```
+
+The same folder gives byte-identical tables on every run. The language column is a rule and the entities are regular expressions; [USAGE.md](https://github.com/HarperZ9/gather/blob/main/USAGE.md#saved-posts-as-tables) lists the columns and the limits.
+
+Take any local file apart into a tree of its parts, then rebuild it byte for byte:
+
+```bash
+gather decompose ./clip.wav --out ./clip-tree     # tree.json, parts/, and measurement tables
+gather rebuild ./clip-tree --out ./clip-again.wav # same SHA-256, every node checked
+```
+
+ZIP, PNG, WAV, JSON, text and CMMNS saves get structural handlers; every other file is kept as one byte range and still rebuilds exactly. PCM WAV files also get loudness (BS.1770), tempo, key, spectrum and decay estimates. [USAGE.md](https://github.com/HarperZ9/gather/blob/main/USAGE.md#take-a-file-apart-and-put-it-back) lists the record, the handlers and the limits.
+
 Offline demo, no install of extra tools, nothing downloaded:
 
 ```bash
@@ -302,7 +286,7 @@ The `web` adapter reads static HTML and does not run JavaScript; a client-render
 - [docs/WEB-ENGINE-UPLIFT.md](https://github.com/HarperZ9/gather/blob/main/docs/WEB-ENGINE-UPLIFT.md): the web-data engine roadmap and benchmarks.
 - [docs/ENTERPRISE-READINESS.md](https://github.com/HarperZ9/gather/blob/main/docs/ENTERPRISE-READINESS.md): context envelopes, action receipts, and host-neutral operation for unattended agents.
 - [docs/PILOT.md](https://github.com/HarperZ9/gather/blob/main/docs/PILOT.md): the accountable pilot evidence engine, its manifest boundary, and the private/shared evidence split.
-- [CHANGELOG.md](https://github.com/HarperZ9/gather/blob/main/CHANGELOG.md): version history. Current release: 2.5.0.
+- [CHANGELOG.md](https://github.com/HarperZ9/gather/blob/main/CHANGELOG.md): version history. Current release: 2.6.0.
 
 Peer projects: [crucible](https://github.com/HarperZ9/crucible) (judgment), [index](https://github.com/HarperZ9/index) (code maps), [forum](https://github.com/HarperZ9/forum) (orchestration), [telos](https://github.com/HarperZ9/telos) (the engine).
 

{gather_engine-2.5.0 → gather_engine-2.6.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "gather-engine"
-version = "2.5.0"
+version = "2.6.0"
 description = "Collect web pages, papers, APIs, PDFs, scans, audio and video into a local corpus where every item records how it arrived."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -26,12 +26,13 @@ classifiers = [
 dependencies = []
 
 [project.optional-dependencies]
-dev = ["pytest>=8", "pytest-cov>=5", "ruff>=0.6", "mypy>=1.10"]
+dev = ["pytest>=8", "pytest-cov>=5", "ruff>=0.6", "mypy>=1.10", "Pillow>=10.1"]
 # Capability backends (wedge 5 parity layer). The core stays zero-dep; these are
 # opt-in edges. Missing edge => honest UNVERIFIABLE or stdlib fallback, never a fake.
 fast = ["lxml>=5"]          # native-speed parsing (2x on large docs); stdlib is the fallback
 browser = ["playwright>=1.4"]  # JS render (needs `playwright install chromium`)
-all = ["lxml>=5", "playwright>=1.4"]
+frames = ["Pillow>=10.1"]     # contact sheets of video frames (gather video-bundle); without it the sheets are not cut
+all = ["lxml>=5", "playwright>=1.4", "Pillow>=10.1"]
 
 [project.scripts]
 gather = "gather.cli:main"

{gather_engine-2.5.0 → gather_engine-2.6.0}/src/gather/__init__.py

@@ -28,7 +28,7 @@ from gather.scope import filter_scope, in_scope
 from gather.source import Catalog, Source
 from gather.store import Corpus
 
-__version__ = "2.5.0"
+__version__ = "2.6.0"
 
 __all__ = [
     "Catalog", "Corpus", "CorpusRootDescriptor", "CorpusRootIdentity", "Digest", "Item", "NullProvenanceProvider", "NullSynthesizer",

{gather_engine-2.5.0 → gather_engine-2.6.0}/src/gather/cli_ext.py

@@ -6,9 +6,11 @@ from collections.abc import Callable
 
 
 def add_extension_parsers(sub, add_common: Callable) -> None:
+    from gather.decompose_cmd import add_decompose_parsers
     from gather.ledger_cmd import add_ledger_parser
     from gather.reddit_cmd import add_reddit_parser
     from gather.report_cmd import add_report_parsers
+    from gather.saves_cmd import add_saves_parser
     from gather.video_bundle_cmd import add_bundle_parsers
     from gather.video_cmd import add_channel_parser
     from gather.youtube_cmd import add_video_parsers
@@ -19,3 +21,5 @@ def add_extension_parsers(sub, add_common: Callable) -> None:
     add_reddit_parser(sub, add_common)
     add_report_parsers(sub, add_common)
     add_ledger_parser(sub)
+    add_saves_parser(sub)
+    add_decompose_parsers(sub)

gather_engine-2.6.0/src/gather/decompose_binary.py

@@ -0,0 +1,127 @@
+"""Structural handlers for PNG and WAV, and the raw fallback.
+
+Each ``build_*`` takes a ``Maker`` and returns the root node for the whole source. A handler
+raises ``ValueError`` or ``struct.error`` when the bytes are not what it expects, and the front
+door then tries the next handler. Bytes after the last understood chunk stay in the tree as
+``raw`` nodes.
+"""
+
+from __future__ import annotations
+
+import struct
+import zlib
+
+from gather.decompose_tree import Maker
+
+PNG_SIGNATURE = b"\x89PNG\r\n\x1a\n"
+_MAX_CHUNKS = 200000
+
+
+def build_raw(m: Maker, ctx, depth: int) -> dict:
+    """The whole source as one byte range."""
+    return m.leaf("raw", 0, m.src.size)
+
+
+def _crc(m: Maker, start: int, length: int) -> int:
+    value = 0
+    for data in m.src.chunks(start, length):
+        value = zlib.crc32(data, value)
+    return value
+
+
+def _ihdr(data: bytes) -> dict:
+    width, height, depth, colour, compression, filtering, interlace = struct.unpack(">IIBBBBB", data[:13])
+    return {"width": width, "height": height, "bit_depth": depth, "colour_type": colour, "interlace": interlace}
+
+
+def _png_chunk(m: Maker, pos: int) -> dict | None:
+    """One chunk at ``pos`` as length, type, data and CRC fields, or None when it does not fit."""
+    head = m.src.read(pos, 8)
+    if len(head) < 8:
+        return None
+    length, kind = struct.unpack(">I4s", head)
+    if pos + 12 + length > m.src.size:
+        return None
+    stored = struct.unpack(">I", m.src.read(pos + 8 + length, 4))[0]
+    name = kind.decode("latin-1")
+    fields = [m.leaf("length", pos, 4, value=length), m.leaf("type", pos + 4, 4, value=name)]
+    if length:
+        fields.append(m.leaf("data", pos + 8, length))
+    ok = _crc(m, pos + 4, 4 + length) == stored
+    fields.append(m.leaf("crc", pos + 8 + length, 4, value=f"{stored:08x}", matches=ok))
+    params = {"type": name, "crc_ok": ok}
+    if kind == b"IHDR" and length >= 13:
+        params.update(_ihdr(m.src.read(pos + 8, 13)))
+    return m.group("png-chunk", fields, **params)
+
+
+def build_png(m: Maker, ctx, depth: int) -> dict:
+    if m.src.read(0, 8) != PNG_SIGNATURE:
+        raise ValueError("not a PNG signature")
+    parts = [m.leaf("png-signature", 0, 8)]
+    pos = 8
+    for _ in range(_MAX_CHUNKS):
+        chunk = _png_chunk(m, pos)
+        if chunk is None:
+            break
+        parts.append(chunk)
+        pos += chunk["length"]
+    return m.cover("png", 0, m.src.size, parts, chunks=len(parts) - 1)
+
+
+def _fmt(data: bytes) -> dict:
+    tag, channels, rate, byte_rate, align, bits = struct.unpack("<HHIIHH", data[:16])
+    return {"format_tag": tag, "channels": channels, "sample_rate": rate, "byte_rate": byte_rate,
+            "block_align": align, "bits_per_sample": bits}
+
+
+def _wav_data(m: Maker, start: int, length: int, fmt: dict) -> list[dict]:
+    """The data chunk's payload as sample frames, plus any bytes short of a whole frame."""
+    align = fmt.get("block_align", 0)
+    if not align or not length:
+        return [m.leaf("data", start, length)] if length else []
+    whole = length - length % align
+    parts = [m.leaf("pcm-frames", start, whole, unit="sample frame", unit_size=align, unit_count=whole // align,
+                    channels=fmt["channels"], bits_per_sample=fmt["bits_per_sample"])] if whole else []
+    if length - whole:
+        parts.append(m.leaf("raw", start + whole, length - whole))
+    return parts
+
+
+def _wav_chunk(m: Maker, pos: int, fmt: dict) -> dict | None:
+    head = m.src.read(pos, 8)
+    if len(head) < 8:
+        return None
+    ident, length = struct.unpack("<4sI", head)
+    if pos + 8 + length > m.src.size:
+        return None
+    name = ident.decode("latin-1")
+    fields = [m.leaf("id", pos, 4, value=name), m.leaf("size", pos + 4, 4, value=length)]
+    params: dict = {"id": name}
+    if ident == b"fmt " and length >= 16:
+        fmt.update(_fmt(m.src.read(pos + 8, 16)))
+        params.update(fmt)
+    if ident == b"data":
+        fields += _wav_data(m, pos + 8, length, fmt)
+    elif length:
+        fields.append(m.leaf("data", pos + 8, length))
+    if length & 1 and pos + 8 + length < m.src.size:
+        fields.append(m.leaf("pad", pos + 8 + length, 1))
+    return m.group("riff-chunk", fields, **params)
+
+
+def build_wav(m: Maker, ctx, depth: int) -> dict:
+    head = m.src.read(0, 12)
+    if len(head) < 12 or head[:4] != b"RIFF" or head[8:12] != b"WAVE":
+        raise ValueError("not a RIFF WAVE header")
+    parts = [m.leaf("riff-id", 0, 4, value="RIFF"), m.leaf("riff-size", 4, 4, value=struct.unpack("<I", head[4:8])[0]),
+             m.leaf("form", 8, 4, value="WAVE")]
+    fmt: dict = {}
+    pos = 12
+    for _ in range(_MAX_CHUNKS):
+        chunk = _wav_chunk(m, pos, fmt)
+        if chunk is None:
+            break
+        parts.append(chunk)
+        pos += chunk["length"]
+    return m.cover("wav", 0, m.src.size, parts, **fmt)

gather_engine-2.6.0/src/gather/decompose_cmd.py

@@ -0,0 +1,80 @@
+"""CLI glue for ``gather decompose`` and ``gather rebuild``.
+
+``gather decompose FILE --out DIR`` takes one local file apart into a tree (``tree.json`` plus
+``parts/``) by fixed rules: the handler is chosen from the file's bytes, every byte is kept, and
+the same file gives the same tree. ``gather rebuild DIR --out FILE`` puts the file back together
+byte for byte and checks every node; ``--verify`` checks without writing.
+
+Exit status: 0 on success; 1 when a rebuild or a verify finds a node that fails its check; 2 on
+bad options, an input that is not a local file, or a tree that cannot be read.
+"""
+
+from __future__ import annotations
+
+import json
+import sys
+
+
+def add_decompose_parsers(sub) -> None:
+    d = sub.add_parser("decompose", help="take one local file apart into a tree of its parts, offline, by fixed rules")
+    d.add_argument("file", help="a local file; it is read as data and never executed")
+    d.add_argument("--out", required=True, metavar="DIR", help="an empty or new folder for tree.json and parts/")
+    d.add_argument("--no-content", action="store_true", help="write the structural tree only, no measurement tables")
+    d.add_argument("--json", action="store_true", help="print the summary as JSON")
+    d.set_defaults(func=cmd_decompose)
+
+    r = sub.add_parser("rebuild", help="rebuild a file byte for byte from a decompose tree, checking every node")
+    r.add_argument("tree", help="a folder written by gather decompose")
+    r.add_argument("--out", metavar="FILE", help="the file to write")
+    r.add_argument("--verify", action="store_true", help="check the tree without writing a file")
+    r.add_argument("--companion", metavar="NAME", help="rebuild a companion file recorded in the tree instead")
+    r.add_argument("--json", action="store_true", help="print the report as JSON")
+    r.set_defaults(func=cmd_rebuild)
+
+
+def summary(doc: dict) -> dict:
+    """The facts about a tree worth printing: no node list, no local path."""
+    return {"format": doc["format"], "source": doc["source"], "handler": doc["handler"], "declined": doc["declined"],
+            "rebuild": doc["rebuild"], "unparsed_bytes": sum(r["length"] for r in doc["unparsed"]),
+            "companions": [c["name"] for c in doc.get("companions", [])],
+            "content": {row["component"]: row["status"] for row in doc.get("content", [])}}
+
+
+def cmd_decompose(args) -> int:
+    from gather.decompose_run import decompose_file
+    from gather.localpath import require_local
+
+    try:
+        doc = decompose_file(require_local(args.file, label="file"), args.out, content=not args.no_content)
+    except (OSError, ValueError) as exc:
+        print(f"decompose: {exc}", file=sys.stderr)
+        return 2
+    facts = summary(doc)
+    if args.json:
+        print(json.dumps(facts, indent=2))
+    else:
+        print(f"decompose: {facts['source']['name']} ({facts['source']['bytes']} bytes) by {facts['handler']}; "
+              f"rebuild {facts['rebuild']}; {facts['unparsed_bytes']} bytes kept as raw ranges; "
+              f"{len(facts['content'])} content components; tree in {args.out}")
+    return 0
+
+
+def cmd_rebuild(args) -> int:
+    from gather.decompose_rebuild import rebuild
+
+    if bool(args.out) == bool(args.verify):
+        print("rebuild: give --out FILE to rebuild, or --verify to check only", file=sys.stderr)
+        return 2
+    try:
+        report = rebuild(args.tree, args.out, args.companion)
+    except (OSError, ValueError) as exc:
+        print(f"rebuild: {exc}", file=sys.stderr)
+        return 2
+    if args.json:
+        print(json.dumps(report, indent=2))
+    else:
+        print(f"rebuild: {report['bytes']} bytes, sha256 {report['sha256']}, "
+              f"{'matches the original' if report['ok'] else 'DOES NOT match'}")
+        for problem in report["problems"]:
+            print(f"  {problem['node']} ({problem['kind']}): {problem['problem']}", file=sys.stderr)
+    return 0 if report["ok"] else 1

gather_engine-2.6.0/src/gather/decompose_content.py

@@ -0,0 +1,164 @@
+"""The content level of ``gather decompose``: measurements and estimates, kept apart from the tree.
+
+``content_stage(path, out)`` writes ``components.csv`` and ``components.jsonl`` (one row per
+component: what it is, the table that holds it, how it was produced, whether it is deterministic,
+and its status) plus one table per kind. A component that needs a decoder or a tool that is not
+here gets a row that says so; nothing is guessed and nothing is fetched. No child process is
+started: an installed transcriber or stem separator is only detected, and its row reads
+``planned``.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import importlib.util
+import os
+
+from gather import __version__
+from gather.decompose_document import document_tables, is_pdf
+from gather.decompose_dsp import decay_time, key, spectrum, tempo
+from gather.decompose_image import PALETTE_RULE, png_palette
+from gather.decompose_pcm import WINDOW_RULE, integrated_loudness, levels, read_pcm, sig
+from gather.media_probe import probe
+from gather.saves_write import csv_bytes, jsonl_bytes
+from gather.spawn import find_tool
+
+COLUMNS = ("kind", "component", "path", "produced_by", "tool", "tool_version", "deterministic", "status", "sha256")
+NO_DECODER = "not run (needs a local decoder; none in the standard library)"
+NO_TOOL = "not run (tool not installed)"
+MEASURED = "yes (fixed rules, rounded to 6 significant digits)"
+MODEL = "no (model output; repeatable only with pinned weights, a fixed seed and one device)"
+_AUDIO = (("sample peak", "largest absolute sample"), ("RMS level", "root mean square of all samples"),
+          ("crest factor", "sample peak minus RMS level"), ("DC offset", "mean of all samples"),
+          ("integrated loudness", "ITU-R BS.1770 K-weighting with absolute and relative gates"),
+          ("tempo estimate", "onset-envelope autocorrelation, 60 to 200 BPM"),
+          ("key estimate", "Goertzel chroma matched to Krumhansl and Schmuckler profiles"),
+          ("spectrum summary", "averaged 4096-point FFT: octave-band energy and centroid"),
+          ("decay time estimate", "Schroeder integration after the loudest frame, -5 to -25 dB fit; "
+                                  "an estimate from programme material, not a room measurement"))
+_VIDEO = ("shot boundaries", "keyframes", "frame rate", "colour summary per shot")
+
+
+def _row(kind: str, component: str, path: str = "", how: str = "", deterministic: str = "yes",
+         status: str = "ok", tool: str = "gather") -> dict:
+    return {"kind": kind, "component": component, "path": path, "produced_by": how, "tool": tool,
+            "tool_version": __version__ if tool == "gather" else "", "deterministic": deterministic,
+            "status": status, "sha256": ""}
+
+
+def _measure(name: str, value, unit: str, method: str) -> dict:
+    shown = sig(value) if isinstance(value, float) else value
+    return {"measure": name, "value": shown, "unit": unit, "method": method, "window": WINDOW_RULE}
+
+
+def _audio_tables(path: str) -> tuple[dict, dict]:
+    """(tables, status per component) for a PCM WAV. Raises ValueError when the PCM cannot be read."""
+    pcm = read_pcm(path)
+    mono = pcm.mono()
+    level, loud = levels(pcm), integrated_loudness(pcm)
+    bpm, tonic, spec = tempo(mono, pcm.rate), key(mono, pcm.rate), spectrum(mono, pcm.rate)
+    decay = decay_time(mono, pcm.rate)
+    how = dict(_AUDIO)
+    measures = [
+        _measure("frames_analysed", pcm.frames, "frames", "count"), _measure("frames_total", pcm.frames_total, "frames", "count"),
+        _measure("sample_peak", level["sample_peak_dbfs"], "dBFS", how["sample peak"]),
+        _measure("rms_level", level["rms_dbfs"], "dBFS", how["RMS level"]),
+        _measure("crest_factor", level["crest_factor_db"], "dB", how["crest factor"]),
+        _measure("dc_offset", level["dc_offset"], "full scale", how["DC offset"]),
+        _measure("integrated_loudness", loud, "LUFS", how["integrated loudness"]),
+        _measure("tempo_estimate", bpm, "BPM", how["tempo estimate"]),
+        _measure("key_estimate", tonic[0] if tonic else None, "key", how["key estimate"]),
+        _measure("key_profile_correlation", tonic[1] if tonic else None, "r", how["key estimate"]),
+        _measure("spectral_centroid", spec["centroid_hz"], "Hz", how["spectrum summary"]),
+        _measure("decay_time_60db_estimate", decay, "s", how["decay time estimate"])]
+    bands = [{"band_centre_hz": centre, "energy_db_rel_total": None if value is None else sig(value)}
+             for centre, value in spec["bands"]]
+    missing = {"integrated loudness": (loud, "no 400 ms block passes the gates"),
+               "tempo estimate": (bpm, "under 4 s of audio, or no onsets"), "key estimate": (tonic, "too short or silent"),
+               "decay time estimate": (decay, "the tail does not fall 25 dB"),
+               "sample peak": (level["sample_peak_dbfs"], "silent"), "RMS level": (level["rms_dbfs"], "silent"),
+               "crest factor": (level["crest_factor_db"], "silent"), "spectrum summary": (spec["centroid_hz"], "silent")}
+    status = {name: f"not run ({why})" for name, (value, why) in missing.items() if value is None}
+    return {"audio_measurements.csv": (("measure", "value", "unit", "method", "window"), measures),
+            "audio_spectrum.csv": (("band_centre_hz", "energy_db_rel_total"), bands)}, status
+
+
+def _audio_rows(path: str, container: str, tables: dict) -> list[dict]:
+    if container != "wav":
+        return [_row("audio", name, how=how, deterministic=MEASURED, status=NO_DECODER) for name, how in _AUDIO]
+    try:
+        made, status = _audio_tables(path)
+    except ValueError as exc:
+        return [_row("audio", name, how=how, deterministic=MEASURED, status=f"not run ({exc})") for name, how in _AUDIO]
+    tables.update(made)
+    return [_row("audio", name, "audio_spectrum.csv" if name == "spectrum summary" else "audio_measurements.csv", how,
+                 MEASURED, status.get(name, "ok")) for name, how in _AUDIO]
+
+
+def _tool_rows(kinds: set[str]) -> list[dict]:
+    """Rows for components that need a local tool: detected only, never started here."""
+    rows = []
+    if kinds & {"audio", "video"}:
+        stems = importlib.util.find_spec("remastering") is not None and bool(os.environ.get("REMASTER_MODELS"))
+        rows.append(_row("audio", "stems", how="stem separation by the remastering package (htdemucs_ft)",
+                         deterministic=MODEL, status="planned" if stems else NO_TOOL, tool="remastering"))
+        rows.append(_row("speech", "transcript with timestamps", how="local speech-to-text through gather.transcribe",
+                         deterministic=MODEL, status="planned" if find_tool("whisper") else NO_TOOL, tool="whisper"))
+    if "image" in kinds:
+        rows.append(_row("image", "embedded text", how="local text recognition through gather.ocr",
+                         deterministic="no (recognition engine output; version dependent)",
+                         status="planned" if find_tool("tesseract") else NO_TOOL, tool="tesseract"))
+    return rows
+
+
+def _image_rows(path: str, container: str, tables: dict) -> list[dict]:
+    how = f"pixel count per colour bin: {PALETTE_RULE}"
+    if container != "png":
+        status = "planned" if container == "gif" else NO_DECODER
+        return [_row("image", "palette", how=how, status=status)]
+    try:
+        tables["image_palette.csv"] = (("rank", "colour", "pixels", "share"), png_palette(path))
+    except ValueError as exc:
+        return [_row("image", "palette", how=how, status=f"not run ({exc})")]
+    return [_row("image", "palette", "image_palette.csv", how)]
+
+
+def _rows(path: str, tables: dict) -> list[dict]:
+    facts, _ = probe(path, with_hash=False)
+    container, kind = facts["container"], facts["kind"]
+    if container == "unknown":
+        if is_pdf(path):
+            return [_row("document", "text", how="local text extraction through gather.pdf", tool="pdftotext",
+                         deterministic="no (tool output; version dependent)",
+                         status="planned" if find_tool("pdftotext") else NO_TOOL)]
+        made = document_tables(path)
+        tables.update(made)
+        return [_row("document", name[len("document_"):-len(".csv")].replace("_", " "), name,
+                     "fixed rules over the decoded text") for name in made]
+    tables["container.csv"] = (("field", "value"), [{"field": k, "value": v} for k, v in facts.items() if k != "sha256"])
+    rows = [_row(kind, "container facts", "container.csv", "magic bytes and the container's own header fields")]
+    if kind == "image":
+        rows += _image_rows(path, container, tables)
+    if kind == "video":
+        rows += [_row("video", name, how="needs decoded frames", status=NO_DECODER) for name in _VIDEO]
+    if kind in ("audio", "video"):
+        rows += _audio_rows(path, container, tables)
+    return rows + _tool_rows({kind})
+
+
+def content_stage(path: str, out: str) -> list[dict]:
+    """Write the content tables for ``path`` into ``out`` and return the component rows."""
+    tables: dict = {}
+    rows = _rows(path, tables)
+    digests = {}
+    for name, (columns, table_rows) in sorted(tables.items()):
+        data = csv_bytes(columns, table_rows)
+        with open(os.path.join(out, name), "wb") as f:
+            f.write(data)
+        digests[name] = hashlib.sha256(data).hexdigest()
+    for row in rows:
+        row["sha256"] = digests.get(row["path"], "")
+    for name, encode in (("components.csv", csv_bytes), ("components.jsonl", jsonl_bytes)):
+        with open(os.path.join(out, name), "wb") as f:
+            f.write(encode(COLUMNS, rows))
+    return rows