aprf-reader 0.1.1__tar.gz → 0.3.0__tar.gz

{aprf_reader-0.1.1/src/aprf_reader.egg-info → aprf_reader-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aprf-reader
-Version: 0.1.1
+Version: 0.3.0
 Summary: Summarise APRF (Aggregate Performance Reporting) inbox placement reports.
 License: MIT
 Requires-Python: >=3.11
@@ -15,9 +15,16 @@ Dynamic: license-file
 
 Reads APRF (Aggregate Performance Reporting) reports and tells you where the
 mail you signed landed: how much reached the inbox, how much was treated as
-unwanted, and how recipients reacted, day by day and per DKIM selector.
+unwanted, and how recipients reacted, day by day and per DKIM selector. It
+points out the days worth a look, such as a jump in unwanted mail.
 
-**Status: 0.1, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
+Two ways to use it, with the same results:
+
+- **In your browser:** [aprf.dns-audit.com](https://aprf.dns-audit.com). Drop
+  your reports on the page. They are read on your computer and never uploaded.
+- **On the command line:** `pipx install aprf-reader`.
+
+**Status: 0.3, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
 
 ## What APRF is
 
@@ -39,22 +46,39 @@ environment, or with pip (`pip3` on some systems) inside a virtual environment.
 
 ```bash
 pipx install aprf-reader
-aprf-reader reports/                      # Markdown summary to the terminal
+aprf-reader reports/                      # overview in the terminal
+aprf-reader APRF.mbox                     # a mail folder exported as mbox
+aprf-reader reports/ --markdown           # Markdown with daily tables
 aprf-reader reports/ --html summary.html  # and a page you can open or share
+aprf-reader reports/ --json               # the summary as data, for scripts
 ```
 
+In a terminal you get an overview: for each stream, each count's latest value,
+its daily range, and a trend line with a gap for each day with no report, then
+the changes worth a look. Piped or written to a file with `--md`, the output
+is Markdown with the full daily tables.
+
 Point it at report files (`.json`, `.json.gz`), saved report emails (`.eml`),
-or folders of them. Folders are searched recursively. Files it skips are listed
+mailbox exports (`.mbox`), or folders of them. The easiest way to collect reports
+is a mail filter that moves them into one folder, then an export of that folder:
+Gmail through [Google Takeout](https://takeout.google.com/) (pick Mail and the
+label), Thunderbird with the ImportExportTools NG add-on, or Apple Mail's
+Mailbox > Export Mailbox. Comcast's reports come from
+`aprf-noreply@alerts.xfinity.com` with subjects starting `APRF:`. Folders are searched recursively. Files it skips are listed
 on stderr and at the end of the summary, each with a reason.
 
 Each stream (provider, signing domain, selector) gets a table with one row per
 day and segment, a trend line per count, and a row for each day with no report.
+A change worth a look is a day when unwanted mail or negative reactions at
+least doubled and grew by 200 or more, or inbox mail fell by half and by 200
+or more, against the report before. Comcast counts in steps of 100, so smaller
+moves are left out as rounding.
 A change in the report's `version` field is called out, since the draft uses it
 to mark a change in what a category counts.
 
 ## What it does
 
-- Read report files (`.json`, `.json.gz` and `.eml`) or a folder of them, and skip
+- Read report files (`.json`, `.json.gz`, `.eml` and `.mbox`) or a folder of them, and skip
   duplicates and files it cannot read, with a reason for each.
 - Summarise placement per signing domain and selector: inbox, unwanted and
   recipient reactions, by day and as a trend.
@@ -67,6 +91,17 @@ to mark a change in what a category counts.
 - When the same day arrives twice, keep the first copy and say so. The draft
   has not yet decided whether a resend replaces the original.
 
+## How the web page is built
+
+`web/src/aprf.js` is a JavaScript port of the Python reader.
+`tests/test_web_parity.py` runs both on the same files (JSON, gzip, saved
+emails, forwarded emails, mailbox exports, duplicates and hostile text) and
+requires the same JSON, Markdown and HTML output, so the page and the command
+line cannot drift apart. `node web/build.js` inlines everything into one
+`web/dist/index.html` whose Content Security Policy blocks every network
+request, so it also works saved to disk. A merge to main that passes the
+tests deploys it to Cloudflare.
+
 ## Related
 
 - [dns-audit.com](https://dns-audit.com) checks a domain for an APRF record and

{aprf_reader-0.1.1 → aprf_reader-0.3.0}/README.md

@@ -2,9 +2,16 @@
 
 Reads APRF (Aggregate Performance Reporting) reports and tells you where the
 mail you signed landed: how much reached the inbox, how much was treated as
-unwanted, and how recipients reacted, day by day and per DKIM selector.
+unwanted, and how recipients reacted, day by day and per DKIM selector. It
+points out the days worth a look, such as a jump in unwanted mail.
 
-**Status: 0.1, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
+Two ways to use it, with the same results:
+
+- **In your browser:** [aprf.dns-audit.com](https://aprf.dns-audit.com). Drop
+  your reports on the page. They are read on your computer and never uploaded.
+- **On the command line:** `pipx install aprf-reader`.
+
+**Status: 0.3, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
 
 ## What APRF is
 
@@ -26,22 +33,39 @@ environment, or with pip (`pip3` on some systems) inside a virtual environment.
 
 ```bash
 pipx install aprf-reader
-aprf-reader reports/                      # Markdown summary to the terminal
+aprf-reader reports/                      # overview in the terminal
+aprf-reader APRF.mbox                     # a mail folder exported as mbox
+aprf-reader reports/ --markdown           # Markdown with daily tables
 aprf-reader reports/ --html summary.html  # and a page you can open or share
+aprf-reader reports/ --json               # the summary as data, for scripts
 ```
 
+In a terminal you get an overview: for each stream, each count's latest value,
+its daily range, and a trend line with a gap for each day with no report, then
+the changes worth a look. Piped or written to a file with `--md`, the output
+is Markdown with the full daily tables.
+
 Point it at report files (`.json`, `.json.gz`), saved report emails (`.eml`),
-or folders of them. Folders are searched recursively. Files it skips are listed
+mailbox exports (`.mbox`), or folders of them. The easiest way to collect reports
+is a mail filter that moves them into one folder, then an export of that folder:
+Gmail through [Google Takeout](https://takeout.google.com/) (pick Mail and the
+label), Thunderbird with the ImportExportTools NG add-on, or Apple Mail's
+Mailbox > Export Mailbox. Comcast's reports come from
+`aprf-noreply@alerts.xfinity.com` with subjects starting `APRF:`. Folders are searched recursively. Files it skips are listed
 on stderr and at the end of the summary, each with a reason.
 
 Each stream (provider, signing domain, selector) gets a table with one row per
 day and segment, a trend line per count, and a row for each day with no report.
+A change worth a look is a day when unwanted mail or negative reactions at
+least doubled and grew by 200 or more, or inbox mail fell by half and by 200
+or more, against the report before. Comcast counts in steps of 100, so smaller
+moves are left out as rounding.
 A change in the report's `version` field is called out, since the draft uses it
 to mark a change in what a category counts.
 
 ## What it does
 
-- Read report files (`.json`, `.json.gz` and `.eml`) or a folder of them, and skip
+- Read report files (`.json`, `.json.gz`, `.eml` and `.mbox`) or a folder of them, and skip
   duplicates and files it cannot read, with a reason for each.
 - Summarise placement per signing domain and selector: inbox, unwanted and
   recipient reactions, by day and as a trend.
@@ -54,6 +78,17 @@ to mark a change in what a category counts.
 - When the same day arrives twice, keep the first copy and say so. The draft
   has not yet decided whether a resend replaces the original.
 
+## How the web page is built
+
+`web/src/aprf.js` is a JavaScript port of the Python reader.
+`tests/test_web_parity.py` runs both on the same files (JSON, gzip, saved
+emails, forwarded emails, mailbox exports, duplicates and hostile text) and
+requires the same JSON, Markdown and HTML output, so the page and the command
+line cannot drift apart. `node web/build.js` inlines everything into one
+`web/dist/index.html` whose Content Security Policy blocks every network
+request, so it also works saved to disk. A merge to main that passes the
+tests deploys it to Cloudflare.
+
 ## Related
 
 - [dns-audit.com](https://dns-audit.com) checks a domain for an APRF record and

{aprf_reader-0.1.1 → aprf_reader-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "aprf-reader"
-version = "0.1.1"
+version = "0.3.0"
 description = "Summarise APRF (Aggregate Performance Reporting) inbox placement reports."
 readme = "README.md"
 license = { text = "MIT" }

{aprf_reader-0.1.1 → aprf_reader-0.3.0}/src/aprf_reader/__init__.py

@@ -1,3 +1,3 @@
 """Read APRF aggregate performance reports."""
 
-__version__ = "0.1.1"
+__version__ = "0.3.0"

{aprf_reader-0.1.1 → aprf_reader-0.3.0}/src/aprf_reader/cli.py

@@ -3,19 +3,25 @@
 from __future__ import annotations
 
 import argparse
+import json
+import os
+import shutil
 import sys
 from pathlib import Path
 
 from . import __version__
 from .load import load
-from .render import to_html, to_markdown
+from .render import to_html, to_markdown, to_terminal
+from .summary import to_dict
 
 
 def main(argv: list[str] | None = None) -> int:
     p = argparse.ArgumentParser(prog="aprf-reader", description="Summarise APRF placement reports.")
-    p.add_argument("paths", nargs="+", type=Path, help="report files (.json, .json.gz, .eml) or folders")
-    p.add_argument("--md", type=Path, help="write the Markdown summary here instead of printing it")
+    p.add_argument("paths", nargs="+", type=Path, help="report files (.json, .json.gz, .eml, .mbox) or folders")
+    p.add_argument("--md", type=Path, help="write the summary here instead of printing it")
     p.add_argument("--html", type=Path, help="also write a self-contained HTML page here")
+    p.add_argument("--markdown", action="store_true", help="print Markdown with daily tables, even in a terminal")
+    p.add_argument("--json", action="store_true", help="print the summary as JSON")
     p.add_argument("--version", action="version", version=f"aprf-reader {__version__}")
     args = p.parse_args(argv)
 
@@ -29,11 +35,17 @@ def main(argv: list[str] | None = None) -> int:
         print("no APRF reports found", file=sys.stderr)
         return 1
 
-    md = to_markdown(loaded)
+    if args.json:
+        text = json.dumps(to_dict(loaded.reports, loaded.skipped), indent=2) + "\n"
+    elif args.md or args.markdown or not sys.stdout.isatty():
+        text = to_markdown(loaded)  # files and pipes get Markdown, which other tools read
+    else:
+        color = "NO_COLOR" not in os.environ and os.environ.get("TERM") != "dumb"
+        text = to_terminal(loaded, color=color, width=shutil.get_terminal_size().columns)
     if args.md:
-        args.md.write_text(md, encoding="utf-8")
+        args.md.write_text(text, encoding="utf-8")
     else:
-        sys.stdout.write(md)
+        sys.stdout.write(text)
     if args.html:
         args.html.write_text(to_html(loaded), encoding="utf-8")
     return 0

aprf_reader-0.3.0/src/aprf_reader/load.py

@@ -0,0 +1,139 @@
+"""Find report files, open them, and drop duplicates."""
+
+from __future__ import annotations
+
+import gzip
+import hashlib
+import io
+import mailbox
+import zlib
+from collections.abc import Iterable, Iterator
+from dataclasses import dataclass, field
+from email import policy
+from email.message import Message
+from email.parser import BytesParser
+from pathlib import Path
+
+from .report import Report, ReportError, parse
+
+SUFFIXES = (".json", ".json.gz", ".eml", ".mbox")
+# A daily report is a few KB; this stops a stray huge file or gzip bomb
+MAX_BYTES = 10 * 1024 * 1024
+
+
+@dataclass
+class Loaded:
+    reports: list[Report] = field(default_factory=list)
+    skipped: list[tuple[str, str]] = field(default_factory=list)  # (where, reason)
+
+
+def _files(paths: Iterable[Path]) -> Iterator[Path]:
+    for p in paths:
+        if p.is_dir():
+            yield from sorted(f for f in p.rglob("*") if f.is_file() and (f.name.lower().endswith(SUFFIXES) or _is_mbox(f)))
+        else:
+            yield p
+
+
+def _is_mbox(path: Path) -> bool:
+    # Apple Mail exports a folder "Name.mbox" holding a file named just "mbox"
+    return path.name.lower() == "mbox" or path.name.lower().endswith(".mbox")
+
+
+def _gunzip(data: bytes) -> bytes:
+    with gzip.GzipFile(fileobj=io.BytesIO(data)) as g:
+        out = g.read(MAX_BYTES + 1)
+    if len(out) > MAX_BYTES:
+        raise ReportError("larger than 10 MB unpacked")
+    return out
+
+
+def _attachments(msg: Message, label: str) -> Iterator[tuple[str, bytes]]:
+    """Yield (label, report JSON bytes) for each report attached to one email."""
+    for part in msg.walk():
+        if part.is_multipart():
+            continue
+        fname = (part.get_filename() or "").lower()
+        ctype = part.get_content_type()
+        data = part.get_payload(decode=True) or b""
+        where = f"{label} ({fname or ctype})"
+        if len(data) > MAX_BYTES:
+            raise ReportError(f"{where} is larger than 10 MB")
+        if fname.endswith(".gz") or ctype in ("application/gzip", "application/x-gzip") or data[:2] == b"\x1f\x8b":
+            yield where, _gunzip(data)
+        # By name or type, or an attachment of unhelpful type that is a JSON array or object
+        elif fname.endswith(".json") or ctype == "application/json" or (
+                part.get_content_disposition() == "attachment" and data.lstrip()[:1] in (b"[", b"{")):
+            yield where, data
+
+
+def _payloads(path: Path) -> Iterator[tuple[str, bytes]]:
+    """Yield (label, report JSON bytes) for each report in a file."""
+    data = path.read_bytes()
+    if len(data) > MAX_BYTES:
+        raise ReportError("larger than 10 MB")
+    if path.name.lower().endswith(".eml"):
+        found = list(_attachments(BytesParser(policy=policy.default).parsebytes(data), str(path)))
+        if not found:
+            raise ReportError("no report attachment")
+        yield from found
+    elif data[:2] == b"\x1f\x8b":
+        yield str(path), _gunzip(data)
+    else:
+        yield str(path), data
+
+
+def _mbox(path: Path, out: Loaded) -> Iterator[tuple[str, bytes]]:
+    """Yield reports from a mailbox export; one skip line for all mail with none attached."""
+    box = mailbox.mbox(path, factory=lambda f: BytesParser(policy=policy.default).parse(f), create=False)
+    without = 0
+    try:
+        for i, msg in enumerate(box, 1):
+            label = f"{path} message {i}"
+            try:
+                found = list(_attachments(msg, label))
+            # OSError covers gzip.BadGzipFile; one bad message must not end the mailbox
+            except (OSError, EOFError, zlib.error, ReportError) as e:
+                out.skipped.append((label, str(e)))
+                continue
+            without += not found
+            yield from found
+    finally:
+        box.close()
+    if without:
+        out.skipped.append((str(path), f"{without} message{'s' if without != 1 else ''} with no report attached"))
+
+
+def load(paths: Iterable[Path]) -> Loaded:
+    out = Loaded()
+    seen_bytes: set[str] = set()
+    seen_keys: dict[tuple, str] = {}
+
+    def add(where: str, data: bytes) -> None:
+        digest = hashlib.sha256(data).hexdigest()
+        if digest in seen_bytes:
+            out.skipped.append((where, "duplicate of a file already read"))
+            return
+        seen_bytes.add(digest)
+        try:
+            reports = parse(data)
+        except ReportError as e:
+            out.skipped.append((where, str(e)))
+            return
+        for r in reports:
+            # The draft does not yet say whether a resend replaces the first; keep the first
+            if r.key in seen_keys:
+                out.skipped.append((where, f"same day already reported in {seen_keys[r.key]}; kept that one"))
+                continue
+            seen_keys[r.key] = where
+            out.reports.append(r)
+
+    for path in _files(paths):
+        try:
+            # An mbox can be large, so it is read one message at a time
+            payloads = _mbox(path, out) if _is_mbox(path) else list(_payloads(path))
+            for where, data in payloads:
+                add(where, data)
+        except (OSError, EOFError, zlib.error, ReportError) as e:
+            out.skipped.append((str(path), str(e)))
+    return out

{aprf_reader-0.1.1 → aprf_reader-0.3.0}/src/aprf_reader/render.py

@@ -3,6 +3,8 @@
 from __future__ import annotations
 
 import html
+import re
+from datetime import date
 
 from . import __version__
 from .load import Loaded
@@ -56,6 +58,8 @@ def to_markdown(loaded: Loaded) -> str:
             else:
                 lines.append(f"| {row.day} | {_md(row.segment)} | " + " | ".join(_n(v) for v in row.values) + " |")
         lines.append("")
+        if s.changes:
+            lines += ["Changes worth a look:", ""] + [f"- {_md(c)}" for c in s.changes] + [""]
         if s.notes:
             lines += [f"- {_md(n)}" for n in s.notes] + [""]
     if loaded.skipped:
@@ -118,6 +122,8 @@ def to_html(loaded: Loaded) -> str:
             else:
                 parts.append(f"<tr><td>{row.day}<td>{e(row.segment)}" + "".join(f"<td>{_n(v)}" for v in row.values))
         parts.append("</tbody></table></div>")
+        if s.changes:
+            parts.append("<p>Changes worth a look:</p><ul>" + "".join(f"<li>{e(c)}" for c in s.changes) + "</ul>")
         if s.notes:
             parts.append("<ul>" + "".join(f"<li>{e(n)}" for n in s.notes) + "</ul>")
     if loaded.skipped:
@@ -125,3 +131,92 @@ def to_html(loaded: Loaded) -> str:
     parts.append(f"<p class=muted>Read by aprf-reader {e(__version__)} against {DRAFT}. Nothing left this computer.</p>")
     parts.append("</body></html>")
     return "\n".join(parts) + "\n"
+
+
+# ---- terminal view ----
+
+BARS = "▁▂▃▄▅▆▇█"
+
+
+class _Style:
+    def __init__(self, color: bool):
+        self.on = color
+
+    def __call__(self, code: str, text: str) -> str:
+        return f"\033[{code}m{text}\033[0m" if self.on else text
+
+    def bold(self, t: str) -> str: return self("1", t)
+    def dim(self, t: str) -> str: return self("2", t)
+    def accent(self, t: str) -> str: return self("1;34", t)
+    def warn(self, t: str) -> str: return self("33", t)
+    def good(self, t: str) -> str: return self("32", t)
+
+
+def _bars(points: list[tuple[int, int]], width: int) -> str:
+    """One character per day, a space for a day with no report; days share a cell when they do not fit."""
+    span = points[-1][0] + 1
+    cells: list[int | None] = [None] * min(span, width)
+    for x, v in points:
+        i = x * len(cells) // span
+        cells[i] = v if cells[i] is None else max(cells[i], v)
+    top = max(v for _, v in points) or 1
+    if min(v for _, v in points) == top:
+        # A flat line is the same every day; a full bar would read as a peak
+        return "".join(" " if v is None else BARS[3] for v in cells)
+    return "".join(" " if v is None else BARS[min(7, v * 7 // top)] if v else BARS[0] for v in cells)
+
+
+def _date(d) -> str:
+    return f"{d:%b} {d.day}"
+
+
+def _range(a, b) -> str:
+    if a == b:
+        return f"{_date(a)}, {a.year}"
+    return (_date(a) if a.year == b.year else f"{_date(a)}, {a.year}") + f" to {_date(b)}, {b.year}"
+
+
+def _wrap(text: str, width: int, indent: str = "") -> list[str]:
+    import textwrap
+    return textwrap.wrap(text, width=max(40, width), initial_indent=indent, subsequent_indent=indent) or [indent]
+
+
+def to_terminal(loaded: Loaded, color: bool = True, width: int = 100) -> str:
+    """A readable overview for a terminal: per stream, each count's latest value, range and trend."""
+    st = _Style(color)
+    ss = streams(loaded.reports)
+    out = [st.bold("APRF placement summary") + st.dim(f"  {_counts(loaded, ss)}"), ""]
+    out += [st.dim(line) for line in _wrap(CAVEAT, min(width, 100))] + [""]
+    for s in ss:
+        out.append(st.accent(s.domain) + "  " + st.bold(f"selector {s.selector}"))
+        meta = [s.source, _range(s.first, s.last), _plural(len(s.by_day), "report")]
+        line = st.dim(" · ".join(meta))
+        if s.missing:
+            line += st.dim(" · ") + st.warn(f"{_plural(s.missing, 'day')} with no report")
+        out += [line, ""]
+        names = [k for _, k in s.columns]
+        nw = max([len(n) for n in names] + [5])
+        trends = [s.trend(i) for i in range(len(names))]
+        latest = [f"{t[-1][1]:,}" for t in trends]
+        ranges = [f"{min(v for _, v in t):,} to {max(v for _, v in t):,}" for t in trends]
+        lw, rw = max(len(x) for x in latest + ["latest"]), max(len(x) for x in ranges + ["range"])
+        bw = max(10, min(60, width - (2 + nw + 2 + lw + 2 + rw + 2)))
+        out.append(st.dim(f"  {'':<{nw}}  {'latest':>{lw}}  {'range':<{rw}}  trend"))
+        for name, t, l, r in zip(names, trends, latest, ranges):
+            out.append(f"  {name:<{nw}}  {st.bold(f'{l:>{lw}}')}  {r:<{rw}}  {st.accent(_bars(t, bw)) if len(t) > 1 else st.dim('one day')}")
+        out.append("")
+        if s.changes:
+            out.append(st.bold("  Changes worth a look"))
+            out += [f"  {st.warn('!')} " + re.sub(r"(\d{4})-(\d\d)-(\d\d)", lambda m: _date(date.fromisoformat(m[0])), c)
+                    for c in s.changes]
+        else:
+            out.append(f"  {st.good('✓')} " + st.dim("No big day-to-day moves in inbox, unwanted, or negative counts."))
+        for n in s.notes:
+            out += _wrap(n, width - 4, "  · ")
+        out.append("")
+    if loaded.skipped:
+        out.append(st.warn(f"Skipped {len(loaded.skipped)}"))
+        out += [st.dim(f"  {w}: {r}") for w, r in loaded.skipped]
+        out.append("")
+    out.append(st.dim(f"aprf-reader {__version__}, {DRAFT}. Daily tables: --markdown, or --html for a page."))
+    return "\n".join(out) + "\n"

{aprf_reader-0.1.1 → aprf_reader-0.3.0}/src/aprf_reader/summary.py

@@ -10,6 +10,11 @@ from functools import cached_property
 from .report import ENGAGEMENT_NAMES, NO_SDI, PLACEMENT_NAMES, Report
 
 
+# Counts where a jump up is bad news, or a drop is
+CHANGES = {"unwanted": "rise", "negative": "rise", "inbox": "fall"}
+MIN_MOVE = 200
+
+
 @dataclass(frozen=True)
 class Row:
     day: date
@@ -69,6 +74,27 @@ class Stream:
         return [((d - self.first).days, sum(getattr(s, a).get(k, 0) for s in r.segments))
                 for d, r in sorted(self.by_day.items())]
 
+    @cached_property
+    def changes(self) -> list[str]:
+        """Day-to-day moves big enough to look at, between one report day and the next."""
+        out = []
+        for col, (_, k) in enumerate(self.columns):
+            rule = CHANGES.get(k)
+            if rule is None:
+                continue
+            pts = self.trend(col)
+            for (x0, old), (x1, new) in zip(pts, pts[1:]):
+                # Comcast's buckets are 100 wide, so a move under two buckets is noise
+                if rule == "rise" and new >= 2 * old and new - old >= MIN_MOVE:
+                    verb = "rose"
+                elif rule == "fall" and 2 * new <= old and old - new >= MIN_MOVE:
+                    verb = "fell"
+                else:
+                    continue
+                d0, d1 = self.first + timedelta(x0), self.first + timedelta(x1)
+                out.append(f"{k} {verb} from {old:,} on {d0} to {new:,} on {d1}.")
+        return sorted(out, key=lambda t: t.rsplit(" on ", 1)[1])
+
     @cached_property
     def notes(self) -> list[str]:
         out = []
@@ -92,3 +118,20 @@ def streams(reports: list[Report]) -> list[Stream]:
     for r in reports:
         groups[(r.source, r.dkim_domain.lower(), r.dkim_selector)][r.day] = r
     return [Stream(src, dom, sel, days) for (src, dom, sel), days in sorted(groups.items())]
+
+
+def to_dict(reports: list[Report], skipped: list[tuple[str, str]]) -> dict:
+    """The summary as plain data, for --json and for checking the web page against."""
+    return {
+        "reports": len(reports),
+        "streams": [{
+            "source": s.source, "domain": s.domain, "selector": s.selector,
+            "first": str(s.first), "last": str(s.last), "missing_days": s.missing,
+            "columns": [k for _, k in s.columns],
+            "days": [{"day": str(d), "version": r.version, "segments": [
+                {"segment": list(seg.path), "classification": seg.classification, "engagement": seg.engagement}
+                for seg in r.segments]} for d, r in sorted(s.by_day.items())],
+            "changes": s.changes, "notes": s.notes,
+        } for s in streams(reports)],
+        "skipped": [{"where": w, "reason": r} for w, r in skipped],
+    }

{aprf_reader-0.1.1 → aprf_reader-0.3.0/src/aprf_reader.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aprf-reader
-Version: 0.1.1
+Version: 0.3.0
 Summary: Summarise APRF (Aggregate Performance Reporting) inbox placement reports.
 License: MIT
 Requires-Python: >=3.11
@@ -15,9 +15,16 @@ Dynamic: license-file
 
 Reads APRF (Aggregate Performance Reporting) reports and tells you where the
 mail you signed landed: how much reached the inbox, how much was treated as
-unwanted, and how recipients reacted, day by day and per DKIM selector.
+unwanted, and how recipients reacted, day by day and per DKIM selector. It
+points out the days worth a look, such as a jump in unwanted mail.
 
-**Status: 0.1, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
+Two ways to use it, with the same results:
+
+- **In your browser:** [aprf.dns-audit.com](https://aprf.dns-audit.com). Drop
+  your reports on the page. They are read on your computer and never uploaded.
+- **On the command line:** `pipx install aprf-reader`.
+
+**Status: 0.3, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
 
 ## What APRF is
 
@@ -39,22 +46,39 @@ environment, or with pip (`pip3` on some systems) inside a virtual environment.
 
 ```bash
 pipx install aprf-reader
-aprf-reader reports/                      # Markdown summary to the terminal
+aprf-reader reports/                      # overview in the terminal
+aprf-reader APRF.mbox                     # a mail folder exported as mbox
+aprf-reader reports/ --markdown           # Markdown with daily tables
 aprf-reader reports/ --html summary.html  # and a page you can open or share
+aprf-reader reports/ --json               # the summary as data, for scripts
 ```
 
+In a terminal you get an overview: for each stream, each count's latest value,
+its daily range, and a trend line with a gap for each day with no report, then
+the changes worth a look. Piped or written to a file with `--md`, the output
+is Markdown with the full daily tables.
+
 Point it at report files (`.json`, `.json.gz`), saved report emails (`.eml`),
-or folders of them. Folders are searched recursively. Files it skips are listed
+mailbox exports (`.mbox`), or folders of them. The easiest way to collect reports
+is a mail filter that moves them into one folder, then an export of that folder:
+Gmail through [Google Takeout](https://takeout.google.com/) (pick Mail and the
+label), Thunderbird with the ImportExportTools NG add-on, or Apple Mail's
+Mailbox > Export Mailbox. Comcast's reports come from
+`aprf-noreply@alerts.xfinity.com` with subjects starting `APRF:`. Folders are searched recursively. Files it skips are listed
 on stderr and at the end of the summary, each with a reason.
 
 Each stream (provider, signing domain, selector) gets a table with one row per
 day and segment, a trend line per count, and a row for each day with no report.
+A change worth a look is a day when unwanted mail or negative reactions at
+least doubled and grew by 200 or more, or inbox mail fell by half and by 200
+or more, against the report before. Comcast counts in steps of 100, so smaller
+moves are left out as rounding.
 A change in the report's `version` field is called out, since the draft uses it
 to mark a change in what a category counts.
 
 ## What it does
 
-- Read report files (`.json`, `.json.gz` and `.eml`) or a folder of them, and skip
+- Read report files (`.json`, `.json.gz`, `.eml` and `.mbox`) or a folder of them, and skip
   duplicates and files it cannot read, with a reason for each.
 - Summarise placement per signing domain and selector: inbox, unwanted and
   recipient reactions, by day and as a trend.
@@ -67,6 +91,17 @@ to mark a change in what a category counts.
 - When the same day arrives twice, keep the first copy and say so. The draft
   has not yet decided whether a resend replaces the original.
 
+## How the web page is built
+
+`web/src/aprf.js` is a JavaScript port of the Python reader.
+`tests/test_web_parity.py` runs both on the same files (JSON, gzip, saved
+emails, forwarded emails, mailbox exports, duplicates and hostile text) and
+requires the same JSON, Markdown and HTML output, so the page and the command
+line cannot drift apart. `node web/build.js` inlines everything into one
+`web/dist/index.html` whose Content Security Policy blocks every network
+request, so it also works saved to disk. A merge to main that passes the
+tests deploys it to Cloudflare.
+
 ## Related
 
 - [dns-audit.com](https://dns-audit.com) checks a domain for an APRF record and

{aprf_reader-0.1.1 → aprf_reader-0.3.0}/src/aprf_reader.egg-info/SOURCES.txt

@@ -13,4 +13,5 @@ src/aprf_reader.egg-info/dependency_links.txt
 src/aprf_reader.egg-info/entry_points.txt
 src/aprf_reader.egg-info/requires.txt
 src/aprf_reader.egg-info/top_level.txt
-tests/test_reader.py
+tests/test_reader.py
+tests/test_web_parity.py

{aprf_reader-0.1.1 → aprf_reader-0.3.0}/tests/test_reader.py

@@ -1,5 +1,7 @@
 import gzip
 import json
+import mailbox
+import re
 from datetime import date
 from email.message import EmailMessage
 from pathlib import Path
@@ -91,6 +93,45 @@ def test_load_gz_eml_folder_and_skips(tmp_path):
     assert "duplicate" in reasons["copy.json"] and "same day" in reasons["resend.json"]
 
 
+def report_email(day: int, payload: bytes, subtype: str, filename: str | None) -> EmailMessage:
+    msg = EmailMessage()
+    msg["From"] = "aprf-noreply@alerts.xfinity.com"
+    msg["Subject"] = f"APRF: day {day}"
+    msg.set_content("report attached")
+    msg.add_attachment(payload, maintype="application", subtype=subtype, filename=filename)
+    return msg
+
+
+def test_attachment_found_by_content(tmp_path):
+    # A provider may label the file octet-stream or give it no useful name
+    raw = json.dumps(report(0)).encode()
+    (tmp_path / "plain.eml").write_bytes(report_email(0, raw, "octet-stream", "20261001_example.com_s1_Comcast").as_bytes())
+    gz = gzip.compress(json.dumps(report(1)).encode())
+    (tmp_path / "gz.eml").write_bytes(report_email(1, gz, "octet-stream", None).as_bytes())
+    loaded = load([tmp_path])
+    assert sorted(r.day.day for r in loaded.reports) == [1, 2] and not loaded.skipped
+
+
+def test_mbox_export(tmp_path):
+    (tmp_path / "Reports.mbox").mkdir()  # Apple Mail's export layout
+    box = mailbox.mbox(tmp_path / "Reports.mbox" / "mbox")
+    box.add(report_email(0, gzip.compress(json.dumps(report(0)).encode()), "gzip", "a.json.gz"))
+    box.add(report_email(1, json.dumps(report(1)).encode(), "json", "b.json"))
+    note = EmailMessage()
+    note["Subject"] = "not a report"
+    note.set_content("hello")
+    box.add(note)
+    box.add(note)
+    box.add(report_email(2, b"\x1f\x8b broken", "gzip", "c.json.gz"))
+    box.add(report_email(3, b"not gzip at all", "gzip", "d.json.gz"))
+    box.add(report_email(4, json.dumps(report(4)).encode(), "json", "e.json"))
+    box.close()
+    loaded = load([tmp_path])
+    assert sorted(r.day.day for r in loaded.reports) == [1, 2, 5]  # read on past bad messages
+    reasons = [r for _, r in loaded.skipped]
+    assert len(reasons) == 3 and "2 messages with no report attached" in reasons
+
+
 def test_hostile_files_skipped_not_crashed(tmp_path):
     (tmp_path / "deep.json").write_text("[" * 100_000 + "]" * 100_000)
     write(tmp_path / "far.json", report(0, report_end=10**20))
@@ -147,3 +188,49 @@ def test_cli(tmp_path, capsys):
     assert out.read_text().startswith("<!doctype html>")
     (tmp_path / "empty").mkdir()
     assert main([str(tmp_path / "empty")]) == 1
+
+
+def test_changes_flag_big_moves_only():
+    (s1, s2) = streams(parse((FIX / "sample_six_weeks.json").read_bytes()))
+    assert s1.selector == "news" and s1.changes == [
+        "unwanted rose from 300 on 2026-09-24 to 2,500 on 2026-09-25.",
+        "negative rose from 100 on 2026-09-24 to 400 on 2026-09-25.",
+        "inbox fell from 2,200 on 2026-09-25 to 600 on 2026-09-26.",
+    ]
+    assert s2.changes == []  # steady stream, nothing flagged
+    # 10 to 200 is many times higher but under two buckets of movement
+    (s,) = streams([parse(json.dumps(report(0, inbox=100)).encode())[0],
+                    parse(json.dumps(report(1, inbox=100) | {"body": [{"classification": {"unwanted": 200}}]}).encode())[0]])
+    assert not any("unwanted" in c for c in s.changes)
+
+
+def test_json_output(capsys):
+    assert main([str(FIX / "sample_six_weeks.json"), "--json"]) == 0
+    doc = json.loads(capsys.readouterr().out)
+    assert doc["reports"] == 80 and [s["selector"] for s in doc["streams"]] == ["news", "receipts"]
+    news = doc["streams"][0]
+    assert news["missing_days"] == 2 and news["columns"] == ["inbox", "unwanted", "positive", "negative"]
+    assert news["days"][0]["segments"][0]["segment"] == [] and len(news["changes"]) == 3
+
+
+def test_terminal_view():
+    from aprf_reader.render import to_terminal
+    loaded = load([FIX / "sample_six_weeks.json", FIX / "comcast_beta.json"])
+    out = to_terminal(loaded, color=False, width=100)
+    assert "example.com  selector news" in out and "2 days with no report" in out
+    assert "! unwanted rose from 300 on Sep 24 to 2,500 on Sep 25." in out
+    assert "✓ No big day-to-day moves" in out and "one day" in out
+    assert "\033[" not in out and "\033[" in to_terminal(loaded, color=True)
+    assert max(len(line) for line in out.splitlines()) <= 100
+    inbox = next(line for line in out.splitlines() if line.startswith("  inbox"))
+    assert re.search("[▁-█] +[▁-█]", inbox)  # the two days with no report show as a break
+
+
+def test_cli_terminal_when_interactive(monkeypatch, capsys):
+    monkeypatch.setattr("sys.stdout.isatty", lambda: True)
+    monkeypatch.setenv("NO_COLOR", "1")
+    assert main([str(FIX / "sample_six_weeks.json")]) == 0
+    out = capsys.readouterr().out
+    assert out.startswith("APRF placement summary") and "\033[" not in out
+    assert main([str(FIX / "sample_six_weeks.json"), "--markdown"]) == 0
+    assert capsys.readouterr().out.startswith("# APRF placement summary")

aprf_reader-0.3.0/tests/test_web_parity.py

@@ -0,0 +1,105 @@
+"""The web page (web/src/aprf.js) must read files exactly as the command line does."""
+
+import gzip
+import json
+import mailbox
+import shutil
+import subprocess
+from email.message import EmailMessage
+from pathlib import Path
+
+import pytest
+
+from aprf_reader import __version__
+from aprf_reader.load import load
+from aprf_reader.render import to_html, to_markdown
+from aprf_reader.summary import to_dict
+
+ROOT = Path(__file__).parent.parent
+FIX = Path(__file__).parent / "fixtures"
+NODE = shutil.which("node")
+
+pytestmark = pytest.mark.skipif(NODE is None, reason="node is not installed")
+
+
+def web(mode: str, paths: list[Path]) -> str:
+    return subprocess.run([NODE, str(ROOT / "web/test/run.js"), mode, *map(str, paths)], check=True,
+                          capture_output=True, text=True, env={"APRF_VERSION": __version__}).stdout
+
+
+def same(paths: list[Path]) -> None:
+    loaded = load(paths)
+    assert json.loads(web("--json", paths)) == to_dict(loaded.reports, loaded.skipped)
+    assert web("--md", paths) == to_markdown(loaded)
+    assert web("--html", paths) == to_html(loaded)
+
+
+def report(day: int, selector: str = "s1", inbox: int = 100) -> dict:
+    start = 1790812800 + day * 86400
+    return {"header": {"version": 1, "source": "Comcast/Xfinity", "dkim_domain": "Example.com",
+                       "dkim_selector": selector, "report_start": start, "report_end": start + 86399,
+                       "sdi_used": " - N/A - Beta - Not in use"},
+            "body": [{"classification": {"inbox": inbox, "unwanted": 0}, "engagement": {"positive": 0}}]}
+
+
+def email(payload: bytes, subtype: str, filename: str | None, cte: str | None = None) -> EmailMessage:
+    msg = EmailMessage()
+    msg["From"] = "aprf-noreply@example.net"
+    msg["Subject"] = "APRF: test"
+    msg.set_content("Report attached.")
+    # cte=None would mean raw binary, not the base64 default
+    msg.add_attachment(payload, maintype="application", subtype=subtype, filename=filename, **({"cte": cte} if cte else {}))
+    return msg
+
+
+@pytest.mark.parametrize("name", ["draft_sample_1.json", "draft_sample_2.json", "comcast_beta.json", "sample_six_weeks.json"])
+def test_fixtures(name):
+    same([FIX / name])
+
+
+def test_all_fixtures_together():
+    same(sorted(FIX.glob("*.json")))
+
+
+def test_files_emails_and_mailboxes(tmp_path):
+    def put(name: str, data: bytes) -> Path:
+        (tmp_path / name).write_bytes(data)
+        return tmp_path / name
+
+    raw = lambda day, **kw: json.dumps(report(day, **kw)).encode()
+    paths = [
+        put("a.json", raw(0)),
+        put("b.json.gz", gzip.compress(raw(1))),
+        put("c.eml", email(gzip.compress(raw(2)), "gzip", "20261003_example.com_s1_Comcast.json.gz").as_bytes()),
+        put("d.eml", email(raw(3), "octet-stream", None).as_bytes()),
+        put("e.eml", email(raw(4), "json", "r.json", cte="quoted-printable").as_bytes()),
+        put("copy.json", raw(0)),
+        put("resend.json", raw(0, inbox=900)),
+        put("note.eml", email(b"hello", "pdf", "invoice.pdf").as_bytes()),
+    ]
+    # A report email forwarded as an attachment
+    outer = EmailMessage()
+    outer["Subject"] = "Fwd: APRF"
+    outer.set_content("See attached.")
+    outer.add_attachment(email(raw(5, inbox=3000), "json", "f.json"))
+    paths.append(put("fwd.eml", outer.as_bytes()))
+    box = mailbox.mbox(tmp_path / "Reports.mbox")
+    for day in (6, 7, 9):
+        box.add(email(gzip.compress(raw(day, selector="s2")), "gzip", f"{day}.json.gz"))
+    box.add(email(raw(8, selector="s2"), "json", "same.json"))
+    plain = EmailMessage()
+    plain.set_content("From the team: not a report")
+    box.add(plain)
+    box.close()
+    paths.append(tmp_path / "Reports.mbox")
+    same(paths)
+
+
+def test_hostile_text_escaped_the_same(tmp_path):
+    obj = report(0)
+    hostile = "<script>alert('x')</script>|a\\b \"q\" & é\nnew"
+    obj["header"].update(source=hostile, extra_info=hostile, sdi_used=hostile)
+    obj["body"][0]["segment"] = [hostile, "two"]
+    obj["body"][0]["classification"][hostile] = 7
+    (tmp_path / "h.json").write_text(json.dumps(obj))
+    same([tmp_path / "h.json"])

aprf_reader-0.1.1/src/aprf_reader/load.py

@@ -1,98 +0,0 @@
-"""Find report files, open them, and drop duplicates."""
-
-from __future__ import annotations
-
-import gzip
-import hashlib
-import io
-import zlib
-from collections.abc import Iterable, Iterator
-from dataclasses import dataclass, field
-from email import policy
-from email.parser import BytesParser
-from pathlib import Path
-
-from .report import Report, ReportError, parse
-
-SUFFIXES = (".json", ".json.gz", ".eml")
-# A daily report is a few KB; this stops a stray huge file or gzip bomb
-MAX_BYTES = 10 * 1024 * 1024
-
-
-@dataclass
-class Loaded:
-    reports: list[Report] = field(default_factory=list)
-    skipped: list[tuple[str, str]] = field(default_factory=list)  # (where, reason)
-
-
-def _files(paths: Iterable[Path]) -> Iterator[Path]:
-    for p in paths:
-        if p.is_dir():
-            yield from sorted(f for f in p.rglob("*") if f.is_file() and f.name.lower().endswith(SUFFIXES))
-        else:
-            yield p
-
-
-def _gunzip(data: bytes) -> bytes:
-    with gzip.GzipFile(fileobj=io.BytesIO(data)) as g:
-        out = g.read(MAX_BYTES + 1)
-    if len(out) > MAX_BYTES:
-        raise ReportError("larger than 10 MB unpacked")
-    return out
-
-
-def _payloads(path: Path) -> Iterator[tuple[str, bytes]]:
-    """Yield (label, report JSON bytes) for each report in a file."""
-    data = path.read_bytes()
-    if len(data) > MAX_BYTES:
-        raise ReportError("larger than 10 MB")
-    name = path.name.lower()
-    if name.endswith(".eml"):
-        msg = BytesParser(policy=policy.default).parsebytes(data)
-        found = False
-        for part in msg.walk():
-            fname = (part.get_filename() or "").lower()
-            ctype = part.get_content_type()
-            if fname.endswith(".json.gz") or ctype == "application/gzip":
-                found = True
-                yield f"{path} ({fname or ctype})", _gunzip(part.get_payload(decode=True) or b"")
-            elif fname.endswith(".json") or ctype == "application/json":
-                found = True
-                yield f"{path} ({fname or ctype})", part.get_payload(decode=True) or b""
-        if not found:
-            raise ReportError("no .json or .json.gz attachment")
-    elif data[:2] == b"\x1f\x8b":
-        yield str(path), _gunzip(data)
-    else:
-        yield str(path), data
-
-
-def load(paths: Iterable[Path]) -> Loaded:
-    out = Loaded()
-    seen_bytes: set[str] = set()
-    seen_keys: dict[tuple, str] = {}
-    for path in _files(paths):
-        try:
-            payloads = list(_payloads(path))
-        except (OSError, EOFError, zlib.error, ReportError) as e:
-            out.skipped.append((str(path), str(e)))
-            continue
-        for where, data in payloads:
-            digest = hashlib.sha256(data).hexdigest()
-            if digest in seen_bytes:
-                out.skipped.append((where, "duplicate of a file already read"))
-                continue
-            seen_bytes.add(digest)
-            try:
-                reports = parse(data)
-            except ReportError as e:
-                out.skipped.append((where, str(e)))
-                continue
-            for r in reports:
-                # The draft does not yet say whether a resend replaces the first; keep the first
-                if r.key in seen_keys:
-                    out.skipped.append((where, f"same day already reported in {seen_keys[r.key]}; kept that one"))
-                    continue
-                seen_keys[r.key] = where
-                out.reports.append(r)
-    return out