gather-engine 2.4.0__tar.gz → 2.6.0__tar.gz

{gather_engine-2.4.0 → gather_engine-2.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: gather-engine
-Version: 2.4.0
+Version: 2.6.0
 Summary: Collect web pages, papers, APIs, PDFs, scans, audio and video into a local corpus where every item records how it arrived.
 Author-email: Zain Dana Harper <zaindharper@gmail.com>
 License-Expression: AGPL-3.0-only
@@ -24,13 +24,17 @@ Requires-Dist: pytest>=8; extra == "dev"
 Requires-Dist: pytest-cov>=5; extra == "dev"
 Requires-Dist: ruff>=0.6; extra == "dev"
 Requires-Dist: mypy>=1.10; extra == "dev"
+Requires-Dist: Pillow>=10.1; extra == "dev"
 Provides-Extra: fast
 Requires-Dist: lxml>=5; extra == "fast"
 Provides-Extra: browser
 Requires-Dist: playwright>=1.4; extra == "browser"
+Provides-Extra: frames
+Requires-Dist: Pillow>=10.1; extra == "frames"
 Provides-Extra: all
 Requires-Dist: lxml>=5; extra == "all"
 Requires-Dist: playwright>=1.4; extra == "all"
+Requires-Dist: Pillow>=10.1; extra == "all"
 Dynamic: license-file
 
 <picture>
@@ -46,11 +50,11 @@ Gather is for researchers, journalists and developers who feed sources to a pers
 
 ## Install and first run
 
-Python 3.11 or newer. Version 2.4.0 is on PyPI as `gather-engine`.
+Python 3.11 or newer. Version 2.6.0 is on PyPI as `gather-engine`.
 
 ```bash
 pip install gather-engine
-gather --version                      # gather 2.4.0
+gather --version                      # gather 2.6.0
 ```
 
 Extract a saved page into hashed blocks, each tied to its place in the page:
@@ -124,6 +128,7 @@ fabricated field cannot reach the corpus by being plausible.
 - **Hard-source adapters behind one shape.** Video with captions and comments (`yt-dlp` first, with the official YouTube Data API as a fallback when you supply your own key; each item records which path served it and how fast), Reddit through its official Data API with your own app credentials, static web, RSS/Atom feeds, local docs, arXiv, PDFs (`pdftotext`), authenticated JSON APIs (token from env, never logged), JS-rendered pages (headless Chromium), scanned images (`tesseract`), and audio (`whisper`). Each external tool is optional and only needed for its own adapter.
 - **Cited reports, checked in code.** `gather report` asks a model on your own machine to answer from fixed excerpts with exact quotes, then checks every quote against the excerpt it cites and marks any it cannot verify. An answer that cites an excerpt without quoting it is sent back for a rewrite, up to twice.
 - **Filter ledgers.** `--scope TERMS --ledger DIR` writes one row per dropped item with its reason code, plus the input, so `gather ledger verify` can recompute every count.
+- **A video as one checkable bundle, with frames.** `gather video-bundle <url> --out DIR` saves the metadata and chapters, the caption track and transcripts, the comment tree, a balanced comment synthesis, and contact sheets of 24 to 96 frames across the whole video with the time printed on each. The frames come from the site's preview storyboard, so no video file is downloaded; they are small (320 by 180 pixels at best on YouTube) and the bundle records their real size. `--fast` reads captions and frames while the comments are still being read and keeps requests 1 second apart: a 234-second video that took 162 seconds with 3-second sleeps took 37 seconds. Frame sheets need the optional `frames` extra (Pillow). See [USAGE.md](https://github.com/HarperZ9/gather/blob/main/USAGE.md).
 - **Channel and playlist intake.** `gather channel <url> --store DIR` lists a channel's videos, shorts, and streams tabs (or one playlist) and gathers every entry with bounded concurrency, resumable from a per-pass ledger. Metadata and comments run as one pass and captions as another (`--no-captions`, `--captions-only`), with polite pacing, bounded backoff on HTTP 429 and session rate limits, a stop at the first bot check (never retried or answered), one caption track per video, and a run summary that counts every outcome by reason.
 - **Scholarly-graph federation.** `gather scholar` queries OpenAlex, Semantic Scholar, and Crossref in one call, dedupes results by normalized DOI (never a fuzzy title match), and can capture citation edges as first-class records with `--edges`.
 - **A durable local corpus.** Any fetch command takes `--store DIR`: bodies are content-addressed and deduped by hash, new writes preserve exact UTF-8 source bytes, and `gather corpus list|verify|digest|runs|search|stats|prune|availability|context` inspects, re-checks, and queries what you stored.
@@ -220,6 +225,24 @@ gather docs ./research-notes --scope "rubik,group theory"
 gather corpus verify ./corpus                  # re-hash every stored body; non-zero exit on corruption
 ```
 
+Turn a folder of posts saved by the CMMNS browser extension into tables, offline and by fixed rules:
+
+```bash
+gather saves ./saves --out ./saves-tables      # items, entities and media as CSV and JSON Lines
+gather saves --probe ./clip.mp4                # container, size, duration and audio fields of one file
+```
+
+The same folder gives byte-identical tables on every run. The language column is a rule and the entities are regular expressions; [USAGE.md](https://github.com/HarperZ9/gather/blob/main/USAGE.md#saved-posts-as-tables) lists the columns and the limits.
+
+Take any local file apart into a tree of its parts, then rebuild it byte for byte:
+
+```bash
+gather decompose ./clip.wav --out ./clip-tree     # tree.json, parts/, and measurement tables
+gather rebuild ./clip-tree --out ./clip-again.wav # same SHA-256, every node checked
+```
+
+ZIP, PNG, WAV, JSON, text and CMMNS saves get structural handlers; every other file is kept as one byte range and still rebuilds exactly. PCM WAV files also get loudness (BS.1770), tempo, key, spectrum and decay estimates. [USAGE.md](https://github.com/HarperZ9/gather/blob/main/USAGE.md#take-a-file-apart-and-put-it-back) lists the record, the handlers and the limits.
+
 Offline demo, no install of extra tools, nothing downloaded:
 
 ```bash
@@ -302,7 +325,7 @@ The `web` adapter reads static HTML and does not run JavaScript; a client-render
 - [docs/WEB-ENGINE-UPLIFT.md](https://github.com/HarperZ9/gather/blob/main/docs/WEB-ENGINE-UPLIFT.md): the web-data engine roadmap and benchmarks.
 - [docs/ENTERPRISE-READINESS.md](https://github.com/HarperZ9/gather/blob/main/docs/ENTERPRISE-READINESS.md): context envelopes, action receipts, and host-neutral operation for unattended agents.
 - [docs/PILOT.md](https://github.com/HarperZ9/gather/blob/main/docs/PILOT.md): the accountable pilot evidence engine, its manifest boundary, and the private/shared evidence split.
-- [CHANGELOG.md](https://github.com/HarperZ9/gather/blob/main/CHANGELOG.md): version history. Current release: 2.4.0.
+- [CHANGELOG.md](https://github.com/HarperZ9/gather/blob/main/CHANGELOG.md): version history. Current release: 2.6.0.
 
 Peer projects: [crucible](https://github.com/HarperZ9/crucible) (judgment), [index](https://github.com/HarperZ9/index) (code maps), [forum](https://github.com/HarperZ9/forum) (orchestration), [telos](https://github.com/HarperZ9/telos) (the engine).
 

gather_engine-2.4.0/src/gather_engine.egg-info/PKG-INFO → gather_engine-2.6.0/README.md

@@ -1,38 +1,3 @@
-Metadata-Version: 2.4
-Name: gather-engine
-Version: 2.4.0
-Summary: Collect web pages, papers, APIs, PDFs, scans, audio and video into a local corpus where every item records how it arrived.
-Author-email: Zain Dana Harper <zaindharper@gmail.com>
-License-Expression: AGPL-3.0-only
-Project-URL: Homepage, https://harperz9.github.io/gather.html
-Project-URL: Repository, https://github.com/HarperZ9/gather
-Project-URL: Issues, https://github.com/HarperZ9/gather/issues
-Project-URL: Changelog, https://github.com/HarperZ9/gather/blob/main/CHANGELOG.md
-Keywords: research,intake,ingestion,provenance,transcripts,arxiv,accountability,ocr,transcription,headless-browser,synthesis,web-scraping,ssrf-safe
-Classifier: Development Status :: 4 - Beta
-Classifier: Environment :: Console
-Classifier: Intended Audience :: Developers
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3 :: Only
-Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
-Requires-Python: >=3.11
-Description-Content-Type: text/markdown
-License-File: LICENSE
-License-File: NOTICE
-Provides-Extra: dev
-Requires-Dist: pytest>=8; extra == "dev"
-Requires-Dist: pytest-cov>=5; extra == "dev"
-Requires-Dist: ruff>=0.6; extra == "dev"
-Requires-Dist: mypy>=1.10; extra == "dev"
-Provides-Extra: fast
-Requires-Dist: lxml>=5; extra == "fast"
-Provides-Extra: browser
-Requires-Dist: playwright>=1.4; extra == "browser"
-Provides-Extra: all
-Requires-Dist: lxml>=5; extra == "all"
-Requires-Dist: playwright>=1.4; extra == "all"
-Dynamic: license-file
-
 <picture>
   <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/HarperZ9/gather/main/docs/art/hero-dark.svg">
   <img src="https://raw.githubusercontent.com/HarperZ9/gather/main/docs/art/hero-light.svg" alt="gather: Research intake for web pages, papers, scanned PDFs, audio and video. Five bundles of lines, solid, dashed, dotted, wavy and stepped, sweep in from the right and pass through a ring into one bright core. One line stops at the ring with a short cross mark." width="100%">
@@ -46,11 +11,11 @@ Gather is for researchers, journalists and developers who feed sources to a pers
 
 ## Install and first run
 
-Python 3.11 or newer. Version 2.4.0 is on PyPI as `gather-engine`.
+Python 3.11 or newer. Version 2.6.0 is on PyPI as `gather-engine`.
 
 ```bash
 pip install gather-engine
-gather --version                      # gather 2.4.0
+gather --version                      # gather 2.6.0
 ```
 
 Extract a saved page into hashed blocks, each tied to its place in the page:
@@ -124,6 +89,7 @@ fabricated field cannot reach the corpus by being plausible.
 - **Hard-source adapters behind one shape.** Video with captions and comments (`yt-dlp` first, with the official YouTube Data API as a fallback when you supply your own key; each item records which path served it and how fast), Reddit through its official Data API with your own app credentials, static web, RSS/Atom feeds, local docs, arXiv, PDFs (`pdftotext`), authenticated JSON APIs (token from env, never logged), JS-rendered pages (headless Chromium), scanned images (`tesseract`), and audio (`whisper`). Each external tool is optional and only needed for its own adapter.
 - **Cited reports, checked in code.** `gather report` asks a model on your own machine to answer from fixed excerpts with exact quotes, then checks every quote against the excerpt it cites and marks any it cannot verify. An answer that cites an excerpt without quoting it is sent back for a rewrite, up to twice.
 - **Filter ledgers.** `--scope TERMS --ledger DIR` writes one row per dropped item with its reason code, plus the input, so `gather ledger verify` can recompute every count.
+- **A video as one checkable bundle, with frames.** `gather video-bundle <url> --out DIR` saves the metadata and chapters, the caption track and transcripts, the comment tree, a balanced comment synthesis, and contact sheets of 24 to 96 frames across the whole video with the time printed on each. The frames come from the site's preview storyboard, so no video file is downloaded; they are small (320 by 180 pixels at best on YouTube) and the bundle records their real size. `--fast` reads captions and frames while the comments are still being read and keeps requests 1 second apart: a 234-second video that took 162 seconds with 3-second sleeps took 37 seconds. Frame sheets need the optional `frames` extra (Pillow). See [USAGE.md](https://github.com/HarperZ9/gather/blob/main/USAGE.md).
 - **Channel and playlist intake.** `gather channel <url> --store DIR` lists a channel's videos, shorts, and streams tabs (or one playlist) and gathers every entry with bounded concurrency, resumable from a per-pass ledger. Metadata and comments run as one pass and captions as another (`--no-captions`, `--captions-only`), with polite pacing, bounded backoff on HTTP 429 and session rate limits, a stop at the first bot check (never retried or answered), one caption track per video, and a run summary that counts every outcome by reason.
 - **Scholarly-graph federation.** `gather scholar` queries OpenAlex, Semantic Scholar, and Crossref in one call, dedupes results by normalized DOI (never a fuzzy title match), and can capture citation edges as first-class records with `--edges`.
 - **A durable local corpus.** Any fetch command takes `--store DIR`: bodies are content-addressed and deduped by hash, new writes preserve exact UTF-8 source bytes, and `gather corpus list|verify|digest|runs|search|stats|prune|availability|context` inspects, re-checks, and queries what you stored.
@@ -220,6 +186,24 @@ gather docs ./research-notes --scope "rubik,group theory"
 gather corpus verify ./corpus                  # re-hash every stored body; non-zero exit on corruption
 ```
 
+Turn a folder of posts saved by the CMMNS browser extension into tables, offline and by fixed rules:
+
+```bash
+gather saves ./saves --out ./saves-tables      # items, entities and media as CSV and JSON Lines
+gather saves --probe ./clip.mp4                # container, size, duration and audio fields of one file
+```
+
+The same folder gives byte-identical tables on every run. The language column is a rule and the entities are regular expressions; [USAGE.md](https://github.com/HarperZ9/gather/blob/main/USAGE.md#saved-posts-as-tables) lists the columns and the limits.
+
+Take any local file apart into a tree of its parts, then rebuild it byte for byte:
+
+```bash
+gather decompose ./clip.wav --out ./clip-tree     # tree.json, parts/, and measurement tables
+gather rebuild ./clip-tree --out ./clip-again.wav # same SHA-256, every node checked
+```
+
+ZIP, PNG, WAV, JSON, text and CMMNS saves get structural handlers; every other file is kept as one byte range and still rebuilds exactly. PCM WAV files also get loudness (BS.1770), tempo, key, spectrum and decay estimates. [USAGE.md](https://github.com/HarperZ9/gather/blob/main/USAGE.md#take-a-file-apart-and-put-it-back) lists the record, the handlers and the limits.
+
 Offline demo, no install of extra tools, nothing downloaded:
 
 ```bash
@@ -302,7 +286,7 @@ The `web` adapter reads static HTML and does not run JavaScript; a client-render
 - [docs/WEB-ENGINE-UPLIFT.md](https://github.com/HarperZ9/gather/blob/main/docs/WEB-ENGINE-UPLIFT.md): the web-data engine roadmap and benchmarks.
 - [docs/ENTERPRISE-READINESS.md](https://github.com/HarperZ9/gather/blob/main/docs/ENTERPRISE-READINESS.md): context envelopes, action receipts, and host-neutral operation for unattended agents.
 - [docs/PILOT.md](https://github.com/HarperZ9/gather/blob/main/docs/PILOT.md): the accountable pilot evidence engine, its manifest boundary, and the private/shared evidence split.
-- [CHANGELOG.md](https://github.com/HarperZ9/gather/blob/main/CHANGELOG.md): version history. Current release: 2.4.0.
+- [CHANGELOG.md](https://github.com/HarperZ9/gather/blob/main/CHANGELOG.md): version history. Current release: 2.6.0.
 
 Peer projects: [crucible](https://github.com/HarperZ9/crucible) (judgment), [index](https://github.com/HarperZ9/index) (code maps), [forum](https://github.com/HarperZ9/forum) (orchestration), [telos](https://github.com/HarperZ9/telos) (the engine).
 

{gather_engine-2.4.0 → gather_engine-2.6.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "gather-engine"
-version = "2.4.0"
+version = "2.6.0"
 description = "Collect web pages, papers, APIs, PDFs, scans, audio and video into a local corpus where every item records how it arrived."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -26,12 +26,13 @@ classifiers = [
 dependencies = []
 
 [project.optional-dependencies]
-dev = ["pytest>=8", "pytest-cov>=5", "ruff>=0.6", "mypy>=1.10"]
+dev = ["pytest>=8", "pytest-cov>=5", "ruff>=0.6", "mypy>=1.10", "Pillow>=10.1"]
 # Capability backends (wedge 5 parity layer). The core stays zero-dep; these are
 # opt-in edges. Missing edge => honest UNVERIFIABLE or stdlib fallback, never a fake.
 fast = ["lxml>=5"]          # native-speed parsing (2x on large docs); stdlib is the fallback
 browser = ["playwright>=1.4"]  # JS render (needs `playwright install chromium`)
-all = ["lxml>=5", "playwright>=1.4"]
+frames = ["Pillow>=10.1"]     # contact sheets of video frames (gather video-bundle); without it the sheets are not cut
+all = ["lxml>=5", "playwright>=1.4", "Pillow>=10.1"]
 
 [project.scripts]
 gather = "gather.cli:main"

{gather_engine-2.4.0 → gather_engine-2.6.0}/src/gather/__init__.py

@@ -28,7 +28,7 @@ from gather.scope import filter_scope, in_scope
 from gather.source import Catalog, Source
 from gather.store import Corpus
 
-__version__ = "2.4.0"
+__version__ = "2.6.0"
 
 __all__ = [
     "Catalog", "Corpus", "CorpusRootDescriptor", "CorpusRootIdentity", "Digest", "Item", "NullProvenanceProvider", "NullSynthesizer",

{gather_engine-2.4.0 → gather_engine-2.6.0}/src/gather/cli_ext.py

@@ -6,14 +6,20 @@ from collections.abc import Callable
 
 
 def add_extension_parsers(sub, add_common: Callable) -> None:
+    from gather.decompose_cmd import add_decompose_parsers
     from gather.ledger_cmd import add_ledger_parser
     from gather.reddit_cmd import add_reddit_parser
     from gather.report_cmd import add_report_parsers
+    from gather.saves_cmd import add_saves_parser
+    from gather.video_bundle_cmd import add_bundle_parsers
     from gather.video_cmd import add_channel_parser
     from gather.youtube_cmd import add_video_parsers
 
     add_video_parsers(sub, add_common)
     add_channel_parser(sub)
+    add_bundle_parsers(sub)
     add_reddit_parser(sub, add_common)
     add_report_parsers(sub, add_common)
     add_ledger_parser(sub)
+    add_saves_parser(sub)
+    add_decompose_parsers(sub)

gather_engine-2.6.0/src/gather/comment_balance.py

@@ -0,0 +1,234 @@
+"""Balanced comment synthesis: what a comment section adds, with critique and support side by side.
+
+``balance(comments)`` reads a video's comments (yt-dlp's records, or a bundle's thread file) and
+returns a ``gather.comment-balance/1`` record:
+
+- every comment in exactly one category (``gather.comment_signals``), with its evidence, so
+  nothing is dropped without a reason a reader can check;
+- substantive comments clustered into claims by shared vocabulary, ranked by comments and likes,
+  each with its critique, support and neutral counts and the comment ids behind it;
+- groups of likely coordinated promotion (two or more promotional comments that share signals or
+  wording) and near-duplicate text from several comments;
+- the balance: counts per category and stance, the critique-to-support ratio, and the share of
+  likes that went to promotion.
+
+A reply keeps its parent and thread ids. A reply that agrees with its parent takes the parent's
+stance toward the video, so agreement with a critique counts as critique.
+
+The record names no commenter: author fields are never read, @mentions are removed from excerpts,
+and each excerpt is under 15 words. It is deterministic and uses no model, so the item it becomes
+is ``compiled`` (``gather.derive``), never ``synthesized``. Pure.
+"""
+
+from __future__ import annotations
+
+import math
+import re
+from collections import Counter
+from typing import Any
+
+from gather.comment_signals import AGREE_OPEN, INSULT, MENTION, URL, classify, tokens
+
+FORMAT = "gather.comment-balance/1"
+EXCERPT_WORDS = 14
+CLAIM_SIM = 0.2
+NEAR_DUP = 0.6
+
+
+def _likes(c: dict) -> int:
+    v = c.get("like_count", c.get("score"))
+    return v if isinstance(v, int) and v >= 0 else 0
+
+
+def excerpt(text: str, limit: int = EXCERPT_WORDS) -> str:
+    """The first sentence's opening words, at most ``limit``, with links and @mentions removed."""
+    plain = re.sub(r"\s+", " ", MENTION.sub("", URL.sub("", text or ""))).strip()
+    first = re.split(r"(?<=[.!?])\s", plain, maxsplit=1)[0]
+    words = first.split()
+    return " ".join(words[:limit]) + (" ..." if len(words) > limit else "")
+
+
+def flatten(tree: list[dict]) -> list[dict]:
+    """A thread file's nested comments as a flat list, each reply with its parent id."""
+    out: list[dict] = []
+    stack = [(c, None) for c in reversed(tree)]
+    while stack:
+        c, parent = stack.pop()
+        out.append({**{k: v for k, v in c.items() if k != "replies"}, "parent": c.get("parent") or parent})
+        stack.extend((r, c.get("id")) for r in reversed(c.get("replies") or []))
+    return out
+
+
+def _records(comments: list[dict]) -> list[dict]:
+    ids = {str(c.get("id")) for c in comments}
+    recs: list[dict[str, Any]] = []
+    for c in comments:
+        parent = c.get("parent")
+        parent = None if parent in (None, "", "root") else str(parent)
+        s = classify(str(c.get("text") or ""), from_channel=bool(c.get("author_is_uploader")))
+        recs.append({"id": str(c.get("id")), "parent": parent, "parent_missing": bool(parent and parent not in ids),
+                     "category": s.category, "stance": s.stance, "evidence": s.evidence, "promo": s.promo,
+                     "likes": _likes(c), "words": s.words, "question": s.question, "pointer": s.pointer,
+                     "_text": str(c.get("text") or ""), "_tokens": tokens(str(c.get("text") or ""))})
+    by = {r["id"]: r for r in recs}
+    for r in recs:
+        root, seen = r, set()
+        while root["parent"] and root["parent"] in by and root["id"] not in seen:
+            seen.add(root["id"])
+            root = by[root["parent"]]
+        r["thread"] = root["id"]
+        p = by.get(r["parent"] or "")
+        opens = r["stance"] == "support" or bool(AGREE_OPEN.match(r["_text"]))
+        agrees = p is not None and opens and p["category"] == "substantive" and p["stance"] in ("critique", "support")
+        r["toward_video"] = p["stance"] if agrees and p else r["stance"]
+        r["agrees_with_parent"] = agrees
+    return recs
+
+
+def _shingles(toks: list[str]) -> set[tuple[str, ...]]:
+    return {tuple(toks[i:i + 3]) for i in range(max(0, len(toks) - 2))}
+
+
+def _jaccard(a: set, b: set) -> float:
+    return len(a & b) / len(a | b) if a and b else 0.0
+
+
+def _groups(recs: list[dict], linked) -> list[list[dict]]:
+    parent = list(range(len(recs)))
+
+    def find(i: int) -> int:
+        while parent[i] != i:
+            parent[i] = parent[parent[i]]
+            i = parent[i]
+        return i
+
+    for i in range(len(recs)):
+        for j in range(i + 1, len(recs)):
+            if linked(recs[i], recs[j]):
+                parent[find(i)] = find(j)
+    out: dict[int, list[dict]] = {}
+    for i, r in enumerate(recs):
+        out.setdefault(find(i), []).append(r)
+    return [g for g in out.values() if len(g) > 1]
+
+
+def mark_coordination(recs: list[dict]) -> list[dict]:
+    """Groups of likely coordinated comments, marked ``coordinated_spam`` in place: promotional
+    comments that share two signals or a third of their wording, and three or more comments of
+    eight words or more with near-identical text."""
+    promo = [r for r in recs if r["category"] == "promotional"]
+    groups: list[dict[str, Any]] = [{"kind": "promotion with shared signals", "members": g} for g in _groups(
+        promo, lambda a, b: len(set(a["promo"]) & set(b["promo"])) >= 2
+        or _jaccard(_shingles(a["_tokens"]), _shingles(b["_tokens"])) >= 0.33)]
+    long = [r for r in recs if r["words"] >= 8 and r["category"] not in ("channel", "promotional")]
+    for dup in _groups(long, lambda a, b: _jaccard(_shingles(a["_tokens"]), _shingles(b["_tokens"])) >= NEAR_DUP):
+        if len(dup) >= 3:
+            groups.append({"kind": "near-identical text", "members": dup})
+    out = []
+    for n, g in enumerate(groups, 1):
+        shared = sorted(set.intersection(*(set(m["promo"]) for m in g["members"]))) if g["kind"].startswith("promotion") else []
+        for m in g["members"]:
+            m["category"], m["group"] = "coordinated_spam", n
+        out.append({"group": n, "kind": g["kind"], "comments": [m["id"] for m in g["members"]], "shared_signals": shared,
+                    "likes": sum(m["likes"] for m in g["members"])})
+    return out
+
+
+def _vectors(recs: list[dict]) -> list[dict[str, float]]:
+    """tf-idf vectors of unit length. A word in more than a third of the comments (the video's own
+    topic word) and a word only one comment uses carry no grouping weight."""
+    df = Counter(t for r in recs for t in set(r["_tokens"]))
+    n = len(recs)
+    vecs = []
+    for r in recs:
+        tf = Counter(t for t in r["_tokens"] if 1 < df[t] <= max(2, n / 3))
+        v = {t: (1 + math.log(c)) * math.log(1 + n / df[t]) for t, c in tf.items()}
+        norm = math.sqrt(sum(x * x for x in v.values())) or 1.0
+        vecs.append({t: x / norm for t, x in v.items()})
+    return vecs
+
+
+def _cos(a: dict[str, float], b: dict[str, float]) -> float:
+    if len(a) > len(b):
+        a, b = b, a
+    return sum(x * b.get(t, 0.0) for t, x in a.items())
+
+
+def cluster_claims(recs: list[dict]) -> list[dict]:
+    """Substantive comments grouped into claims. Most-liked first, each comment joins the closest
+    claim at or above ``CLAIM_SIM``; a reply below that joins its parent's claim, so a conversation
+    stays together. Claims are ranked by comments, then likes."""
+    sub = sorted((r for r in recs if r["category"] == "substantive"), key=lambda r: (-r["likes"], r["id"]))
+    vecs = dict(zip((r["id"] for r in sub), _vectors(sub)))
+    claims: list[dict] = []
+    home: dict[str, dict] = {}
+    for r in sub:
+        v = vecs[r["id"]]
+        best, score = None, 0.0
+        for c in claims:
+            s = _cos(v, c["centroid"]) / (math.sqrt(sum(x * x for x in c["centroid"].values())) or 1.0)
+            if s > score:
+                best, score = c, s
+        if score < CLAIM_SIM:
+            best = home.get(r["parent"] or "") or None
+        if best is None:
+            best = {"centroid": {}, "members": []}
+            claims.append(best)
+        best["members"].append(r)
+        for t, x in v.items():
+            best["centroid"][t] = best["centroid"].get(t, 0.0) + x
+        home[r["id"]] = best
+    return [_claim(c) for c in sorted(claims, key=lambda c: (-len(c["members"]), -sum(m["likes"] for m in c["members"])))]
+
+
+def _claim(c: dict) -> dict:
+    members = c["members"]
+    # the excerpt comes from the most-liked member that attacks no one, when there is one
+    civil = [m for m in members if not INSULT.search(m["_text"])] or members
+    top = max(civil, key=lambda m: (m["likes"], m["words"]))
+    stances = Counter(m["toward_video"] for m in members)
+    terms = [t for t, _ in sorted(c["centroid"].items(), key=lambda kv: -kv[1])[:4]]
+    return {"terms": terms, "comments": [m["id"] for m in members], "top_level": sum(1 for m in members if not m["parent"]),
+            "replies": sum(1 for m in members if m["parent"]), "critique": stances["critique"], "support": stances["support"],
+            "mixed": stances["mixed"], "neutral": stances["neutral"], "likes": sum(m["likes"] for m in members),
+            "excerpt": {"comment": top["id"], "text": excerpt(top["_text"])}, "threads": sorted({m["thread"] for m in members})}
+
+
+CATEGORIES = ("substantive", "reaction", "ad_hominem", "dismissal", "promotional", "coordinated_spam", "channel")
+
+
+def _balance_line(crit: int, supp: int) -> str:
+    if not crit and not supp:
+        return "No substantive comment takes a side, so there is no balance to report."
+    ratio = min(crit, supp) / max(crit, supp)
+    if ratio >= 0.5:
+        return f"Critique and support are both well represented: {crit} and {supp} substantive comments."
+    side, other = ("critique", "support") if crit > supp else ("support", "critique")
+    return f"Weighted toward {side}: {max(crit, supp)} {side} against {min(crit, supp)} {other} among substantive comments."
+
+
+def balance(comments: list[dict], *, title: str = "", url: str = "", max_claims: int = 20) -> dict:
+    """The balanced synthesis of one video's comments. ``comments`` are flat records with ``id``,
+    ``parent`` (None, ``"root"`` or a comment id), ``text``, ``like_count`` and optionally
+    ``author_is_uploader``; any author field is ignored. Pure and deterministic."""
+    recs = _records(comments)
+    groups = mark_coordination(recs)
+    claims = cluster_claims(recs)
+    cats = Counter(r["category"] for r in recs)
+    sub = [r for r in recs if r["category"] == "substantive"]
+    toward = Counter(r["toward_video"] for r in sub)
+    likes = sum(r["likes"] for r in recs)
+    spam_likes = sum(r["likes"] for r in recs if r["category"] in ("promotional", "coordinated_spam"))
+    top10 = sorted(recs, key=lambda r: -r["likes"])[:10]
+    public = [{k: v for k, v in r.items() if not k.startswith("_") and k != "promo"} for r in recs]
+    return {
+        "format": FORMAT, "title": title, "url": url, "method": "compiled",
+        "counts": {"comments": len(recs), "top_level": sum(1 for r in recs if not r["parent"]), "replies": sum(1 for r in recs if r["parent"]),
+                   "replies_parent_missing": sum(1 for r in recs if r["parent_missing"]), **{c: cats[c] for c in CATEGORIES}},
+        "balance": {"critique": toward["critique"], "support": toward["support"], "mixed": toward["mixed"], "neutral": toward["neutral"],
+                    "questions": sum(1 for r in sub if r["question"]), "pointers": sum(1 for r in sub if r["pointer"]),
+                    "sentence": _balance_line(toward["critique"], toward["support"]),
+                    "likes_total": likes, "likes_to_promotion": spam_likes,
+                    "top10_by_likes": dict(Counter(r["category"] if r["category"] != "substantive" else f"substantive:{r['toward_video']}" for r in top10))},
+        "claims": claims[:max_claims], "claims_total": len(claims), "coordinated": groups, "comments": public,
+    }

gather_engine-2.6.0/src/gather/comment_report.py

@@ -0,0 +1,78 @@
+"""The balanced comment synthesis as a page a person reads, and as a receipted Gather item.
+
+``report_markdown`` renders a ``gather.comment-balance/1`` record: the balance first, then the
+substantive claims ranked with their counts and one short excerpt each, then what was set aside and
+why, every group with the comment ids behind it. It names no commenter. ``balance_item`` wraps the
+page as a ``compiled`` derived item over the comment items it read (``gather.derive``), so the
+receipt says it was assembled by rules and points back to the exact comments.
+"""
+
+from __future__ import annotations
+
+from gather.derive import COMPILED, derive
+from gather.item import Item
+
+SET_ASIDE = (
+    ("coordinated_spam", "Likely coordinated spam"),
+    ("promotional", "Promotion from a single comment"),
+    ("ad_hominem", "About the person, not the argument"),
+    ("dismissal", "Dismissal with no reason given"),
+    ("reaction", "Short reactions"),
+    ("channel", "From the channel itself"),
+)
+
+
+def _ids(ids: list[str], limit: int = 12) -> str:
+    shown = ", ".join(ids[:limit])
+    return shown + (f" and {len(ids) - limit} more" if len(ids) > limit else "")
+
+
+def _claim_lines(n: int, c: dict) -> list[str]:
+    sides = ", ".join(f"{c[k]} {k}" for k in ("critique", "support", "mixed", "neutral") if c[k])
+    return [f"{n}. **{', '.join(c['terms']) or 'untitled'}**: {len(c['comments'])} comments ({c['top_level']} top-level, "
+            f"{c['replies']} replies; {sides}), {c['likes']} likes.",
+            f"   Excerpt: \"{c['excerpt']['text']}\" (comment {c['excerpt']['comment']})",
+            f"   Comments: {_ids(c['comments'])}", ""]
+
+
+def report_markdown(rec: dict) -> str:
+    """The record as Markdown: balance, ranked claims, what was set aside, and how it was made."""
+    k, b = rec["counts"], rec["balance"]
+    out = [f"# Comment balance{': ' + rec['title'] if rec.get('title') else ''}", ""]
+    if rec.get("url"):
+        out += [f"Source: {rec['url']}", ""]
+    missing = k["replies_parent_missing"]
+    orphans = f", {missing} of them with a parent that was not loaded" if missing else ""
+    out += [f"{k['comments']} comments read: {k['top_level']} top-level and {k['replies']} replies{orphans}.",
+            "Commenters are not named. Each excerpt is under 15 words.", "", "## Balance", "", f"- {b['sentence']}",
+            f"- Substantive comments: {k['substantive']} ({b['critique']} critique, {b['support']} support, {b['mixed']} mixed, "
+            f"{b['neutral']} neutral; {b['questions']} ask a question, {b['pointers']} point to a source).",
+            f"- Set aside: {k['coordinated_spam']} likely coordinated spam, {k['promotional']} other promotion, {k['ad_hominem']} about the "
+            f"person, {k['dismissal']} dismissals, {k['reaction']} short reactions, {k['channel']} from the channel.",
+            f"- Likes: {b['likes_total']} in all, {b['likes_to_promotion']} of them on promotion.",
+            f"- The 10 most-liked comments: {', '.join(f'{v} {n}' for n, v in sorted(b['top10_by_likes'].items(), key=lambda kv: -kv[1]))}.", "",
+            "## Substantive claims, ranked", ""]
+    for n, c in enumerate(rec["claims"], 1):
+        out += _claim_lines(n, c)
+    if rec["claims_total"] > len(rec["claims"]):
+        out += [f"{rec['claims_total'] - len(rec['claims'])} smaller claims are in the JSON record.", ""]
+    out += ["## Set aside, with the reason", ""]
+    for g in rec["coordinated"]:
+        why = f"shared signals: {', '.join(g['shared_signals'])}" if g["shared_signals"] else g["kind"]
+        out.append(f"- Group {g['group']}, {g['kind']}: {len(g['comments'])} comments, {g['likes']} likes ({why}). Comments: {_ids(g['comments'])}")
+    for cat, label in SET_ASIDE[1:]:
+        ids = [r["id"] for r in rec["comments"] if r["category"] == cat]
+        if ids:
+            out.append(f"- {label}: {len(ids)}. Comments: {_ids(ids, 8)}")
+    out += ["", "## How this was made", "",
+            "Deterministic rules, no model: the method is `compiled`. Categories come from word patterns, and an argument "
+            "marker outweighs an insult, so a harsh critique that gives a reason stays substantive. Claims group comments by "
+            "shared vocabulary. Sarcasm and irony can be misread. A reply that agrees with its parent counts toward the "
+            "parent's side.", ""]
+    return "\n".join(out)
+
+
+def balance_item(rec: dict, inputs: list[Item], *, fetched_at: float, ref: str) -> Item:
+    """The page as a ``compiled`` item derived from the comment items it read."""
+    return derive(inputs, report_markdown(rec), fetched_at=fetched_at, ref=ref, method=COMPILED,
+                  title=f"comment balance: {rec.get('title') or ref}", kind="synthesis", source="synthesis")