aprf-reader 0.2.0__tar.gz → 0.3.1__tar.gz

{aprf_reader-0.2.0/src/aprf_reader.egg-info → aprf_reader-0.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aprf-reader
-Version: 0.2.0
+Version: 0.3.1
 Summary: Summarise APRF (Aggregate Performance Reporting) inbox placement reports.
 License: MIT
 Requires-Python: >=3.11
@@ -15,9 +15,16 @@ Dynamic: license-file
 
 Reads APRF (Aggregate Performance Reporting) reports and tells you where the
 mail you signed landed: how much reached the inbox, how much was treated as
-unwanted, and how recipients reacted, day by day and per DKIM selector.
+unwanted, and how recipients reacted, day by day and per DKIM selector. It
+points out the days worth a look, such as a jump in unwanted mail.
 
-**Status: 0.2, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
+Two ways to use it, with the same results:
+
+- **In your browser:** [aprf.dns-audit.com](https://aprf.dns-audit.com). Drop
+  your reports on the page. They are read on your computer and never uploaded.
+- **On the command line:** `pipx install aprf-reader`.
+
+**Status: 0.3, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
 
 ## What APRF is
 
@@ -39,11 +46,18 @@ environment, or with pip (`pip3` on some systems) inside a virtual environment.
 
 ```bash
 pipx install aprf-reader
-aprf-reader reports/                      # Markdown summary to the terminal
-aprf-reader reports/ --html summary.html  # and a page you can open or share
+aprf-reader reports/                      # overview in the terminal
 aprf-reader APRF.mbox                     # a mail folder exported as mbox
+aprf-reader reports/ --markdown           # Markdown with daily tables
+aprf-reader reports/ --html summary.html  # and a page you can open or share
+aprf-reader reports/ --json               # the summary as data, for scripts
 ```
 
+In a terminal you get an overview: for each stream, each count's latest value,
+its daily range, and a trend line with a gap for each day with no report, then
+the changes worth a look. Piped or written to a file with `--md`, the output
+is Markdown with the full daily tables.
+
 Point it at report files (`.json`, `.json.gz`), saved report emails (`.eml`),
 mailbox exports (`.mbox`), or folders of them. The easiest way to collect reports
 is a mail filter that moves them into one folder, then an export of that folder:
@@ -55,6 +69,10 @@ on stderr and at the end of the summary, each with a reason.
 
 Each stream (provider, signing domain, selector) gets a table with one row per
 day and segment, a trend line per count, and a row for each day with no report.
+A change worth a look is a day when unwanted mail or negative reactions at
+least doubled and grew by 200 or more, or inbox mail fell by half and by 200
+or more, against the report before. Comcast counts in steps of 100, so smaller
+moves are left out as rounding.
 A change in the report's `version` field is called out, since the draft uses it
 to mark a change in what a category counts.
 
@@ -73,6 +91,17 @@ to mark a change in what a category counts.
 - When the same day arrives twice, keep the first copy and say so. The draft
   has not yet decided whether a resend replaces the original.
 
+## How the web page is built
+
+`web/src/aprf.js` is a JavaScript port of the Python reader.
+`tests/test_web_parity.py` runs both on the same files (JSON, gzip, saved
+emails, forwarded emails, mailbox exports, duplicates and hostile text) and
+requires the same JSON, Markdown and HTML output, so the page and the command
+line cannot drift apart. `node web/build.js` inlines everything into one
+`web/dist/index.html` whose Content Security Policy blocks every network
+request, so it also works saved to disk. A merge to main that passes the
+tests deploys it to Cloudflare.
+
 ## Related
 
 - [dns-audit.com](https://dns-audit.com) checks a domain for an APRF record and

{aprf_reader-0.2.0 → aprf_reader-0.3.1}/README.md

@@ -2,9 +2,16 @@
 
 Reads APRF (Aggregate Performance Reporting) reports and tells you where the
 mail you signed landed: how much reached the inbox, how much was treated as
-unwanted, and how recipients reacted, day by day and per DKIM selector.
+unwanted, and how recipients reacted, day by day and per DKIM selector. It
+points out the days worth a look, such as a jump in unwanted mail.
 
-**Status: 0.2, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
+Two ways to use it, with the same results:
+
+- **In your browser:** [aprf.dns-audit.com](https://aprf.dns-audit.com). Drop
+  your reports on the page. They are read on your computer and never uploaded.
+- **On the command line:** `pipx install aprf-reader`.
+
+**Status: 0.3, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
 
 ## What APRF is
 
@@ -26,11 +33,18 @@ environment, or with pip (`pip3` on some systems) inside a virtual environment.
 
 ```bash
 pipx install aprf-reader
-aprf-reader reports/                      # Markdown summary to the terminal
-aprf-reader reports/ --html summary.html  # and a page you can open or share
+aprf-reader reports/                      # overview in the terminal
 aprf-reader APRF.mbox                     # a mail folder exported as mbox
+aprf-reader reports/ --markdown           # Markdown with daily tables
+aprf-reader reports/ --html summary.html  # and a page you can open or share
+aprf-reader reports/ --json               # the summary as data, for scripts
 ```
 
+In a terminal you get an overview: for each stream, each count's latest value,
+its daily range, and a trend line with a gap for each day with no report, then
+the changes worth a look. Piped or written to a file with `--md`, the output
+is Markdown with the full daily tables.
+
 Point it at report files (`.json`, `.json.gz`), saved report emails (`.eml`),
 mailbox exports (`.mbox`), or folders of them. The easiest way to collect reports
 is a mail filter that moves them into one folder, then an export of that folder:
@@ -42,6 +56,10 @@ on stderr and at the end of the summary, each with a reason.
 
 Each stream (provider, signing domain, selector) gets a table with one row per
 day and segment, a trend line per count, and a row for each day with no report.
+A change worth a look is a day when unwanted mail or negative reactions at
+least doubled and grew by 200 or more, or inbox mail fell by half and by 200
+or more, against the report before. Comcast counts in steps of 100, so smaller
+moves are left out as rounding.
 A change in the report's `version` field is called out, since the draft uses it
 to mark a change in what a category counts.
 
@@ -60,6 +78,17 @@ to mark a change in what a category counts.
 - When the same day arrives twice, keep the first copy and say so. The draft
   has not yet decided whether a resend replaces the original.
 
+## How the web page is built
+
+`web/src/aprf.js` is a JavaScript port of the Python reader.
+`tests/test_web_parity.py` runs both on the same files (JSON, gzip, saved
+emails, forwarded emails, mailbox exports, duplicates and hostile text) and
+requires the same JSON, Markdown and HTML output, so the page and the command
+line cannot drift apart. `node web/build.js` inlines everything into one
+`web/dist/index.html` whose Content Security Policy blocks every network
+request, so it also works saved to disk. A merge to main that passes the
+tests deploys it to Cloudflare.
+
 ## Related
 
 - [dns-audit.com](https://dns-audit.com) checks a domain for an APRF record and

{aprf_reader-0.2.0 → aprf_reader-0.3.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "aprf-reader"
-version = "0.2.0"
+version = "0.3.1"
 description = "Summarise APRF (Aggregate Performance Reporting) inbox placement reports."
 readme = "README.md"
 license = { text = "MIT" }

{aprf_reader-0.2.0 → aprf_reader-0.3.1}/src/aprf_reader/__init__.py

@@ -1,3 +1,3 @@
 """Read APRF aggregate performance reports."""
 
-__version__ = "0.2.0"
+__version__ = "0.3.1"

{aprf_reader-0.2.0 → aprf_reader-0.3.1}/src/aprf_reader/cli.py

@@ -3,19 +3,25 @@
 from __future__ import annotations
 
 import argparse
+import json
+import os
+import shutil
 import sys
 from pathlib import Path
 
 from . import __version__
 from .load import load
-from .render import to_html, to_markdown
+from .render import to_html, to_markdown, to_terminal
+from .summary import to_dict
 
 
 def main(argv: list[str] | None = None) -> int:
     p = argparse.ArgumentParser(prog="aprf-reader", description="Summarise APRF placement reports.")
     p.add_argument("paths", nargs="+", type=Path, help="report files (.json, .json.gz, .eml, .mbox) or folders")
-    p.add_argument("--md", type=Path, help="write the Markdown summary here instead of printing it")
+    p.add_argument("--md", type=Path, help="write the summary here instead of printing it")
     p.add_argument("--html", type=Path, help="also write a self-contained HTML page here")
+    p.add_argument("--markdown", action="store_true", help="print Markdown with daily tables, even in a terminal")
+    p.add_argument("--json", action="store_true", help="print the summary as JSON")
     p.add_argument("--version", action="version", version=f"aprf-reader {__version__}")
     args = p.parse_args(argv)
 
@@ -29,11 +35,17 @@ def main(argv: list[str] | None = None) -> int:
         print("no APRF reports found", file=sys.stderr)
         return 1
 
-    md = to_markdown(loaded)
+    if args.json:
+        text = json.dumps(to_dict(loaded.reports, loaded.skipped), indent=2) + "\n"
+    elif args.md or args.markdown or not sys.stdout.isatty():
+        text = to_markdown(loaded)  # files and pipes get Markdown, which other tools read
+    else:
+        color = "NO_COLOR" not in os.environ and os.environ.get("TERM") != "dumb"
+        text = to_terminal(loaded, color=color, width=shutil.get_terminal_size().columns)
     if args.md:
-        args.md.write_text(md, encoding="utf-8")
+        args.md.write_text(text, encoding="utf-8")
     else:
-        sys.stdout.write(md)
+        sys.stdout.write(text)
     if args.html:
         args.html.write_text(to_html(loaded), encoding="utf-8")
     return 0

{aprf_reader-0.2.0 → aprf_reader-0.3.1}/src/aprf_reader/load.py

@@ -14,7 +14,7 @@ from email.message import Message
 from email.parser import BytesParser
 from pathlib import Path
 
-from .report import Report, ReportError, parse
+from .report import Report, ReportError, parse_items
 
 SUFFIXES = (".json", ".json.gz", ".eml", ".mbox")
 # A daily report is a few KB; this stops a stray huge file or gzip bomb
@@ -116,16 +116,20 @@ def load(paths: Iterable[Path]) -> Loaded:
             return
         seen_bytes.add(digest)
         try:
-            reports = parse(data)
+            items = parse_items(data)
         except ReportError as e:
             out.skipped.append((where, str(e)))
             return
-        for r in reports:
+        for i, r in enumerate(items, 1):
+            label = f"{where} report {i}" if len(items) > 1 else where
+            if isinstance(r, ReportError):
+                out.skipped.append((label, str(r)))
+                continue
             # The draft does not yet say whether a resend replaces the first; keep the first
             if r.key in seen_keys:
-                out.skipped.append((where, f"same day already reported in {seen_keys[r.key]}; kept that one"))
+                out.skipped.append((label, f"same day already reported in {seen_keys[r.key]}; kept that one"))
                 continue
-            seen_keys[r.key] = where
+            seen_keys[r.key] = label
             out.reports.append(r)
 
     for path in _files(paths):

{aprf_reader-0.2.0 → aprf_reader-0.3.1}/src/aprf_reader/render.py

@@ -3,12 +3,16 @@
 from __future__ import annotations
 
 import html
+import re
+from datetime import date
 
 from . import __version__
 from .load import Loaded
 from .report import DRAFT
 from .summary import Row, Stream, streams
 
+NO_COUNTS = "These reports carry no counts."
+
 CAVEAT = (
     "Counts are bucket upper bounds chosen by each provider: a ceiling, not an exact figure. "
     "Placement counts mail received that day; engagement counts actions taken that day on mail "
@@ -48,6 +52,9 @@ def to_markdown(loaded: Loaded) -> str:
     for s in ss:
         lines += [f"## {_md(s.source)}: {_md(s.domain)}, selector {_md(s.selector)}", "", _span(s), ""]
         cols = [k for _, k in s.columns]
+        if not cols:
+            lines += [NO_COUNTS, ""]
+            continue
         lines.append("| Day | Segment | " + " | ".join(_md(c) for c in cols) + " |")
         lines.append("|---|---|" + "---:|" * len(cols))
         for row in s.rows:
@@ -56,6 +63,8 @@ def to_markdown(loaded: Loaded) -> str:
             else:
                 lines.append(f"| {row.day} | {_md(row.segment)} | " + " | ".join(_n(v) for v in row.values) + " |")
         lines.append("")
+        if s.changes:
+            lines += ["Changes worth a look:", ""] + [f"- {_md(c)}" for c in s.changes] + [""]
         if s.notes:
             lines += [f"- {_md(n)}" for n in s.notes] + [""]
     if loaded.skipped:
@@ -108,6 +117,9 @@ def to_html(loaded: Loaded) -> str:
     for s in ss:
         cols = [k for _, k in s.columns]
         parts.append(f"<h2>{e(s.source)}: {e(s.domain)}, selector {e(s.selector)}</h2>")
+        if not cols:
+            parts.append(f"<p class=muted>{e(_span(s))}</p><p>{NO_COUNTS}</p>")
+            continue
         parts.append(f"<p class=muted>{e(_span(s))}</p><div class=wrap><table><thead><tr><th>Day<th>Segment")
         parts.append("".join(f"<th>{e(c)}" for c in cols) + "</thead><tbody>")
         parts.append("<tr class=trend><td>Trend<td class=muted>sum per day")
@@ -118,6 +130,8 @@ def to_html(loaded: Loaded) -> str:
             else:
                 parts.append(f"<tr><td>{row.day}<td>{e(row.segment)}" + "".join(f"<td>{_n(v)}" for v in row.values))
         parts.append("</tbody></table></div>")
+        if s.changes:
+            parts.append("<p>Changes worth a look:</p><ul>" + "".join(f"<li>{e(c)}" for c in s.changes) + "</ul>")
         if s.notes:
             parts.append("<ul>" + "".join(f"<li>{e(n)}" for n in s.notes) + "</ul>")
     if loaded.skipped:
@@ -125,3 +139,103 @@ def to_html(loaded: Loaded) -> str:
     parts.append(f"<p class=muted>Read by aprf-reader {e(__version__)} against {DRAFT}. Nothing left this computer.</p>")
     parts.append("</body></html>")
     return "\n".join(parts) + "\n"
+
+
+# ---- terminal view ----
+
+BARS = "▁▂▃▄▅▆▇█"
+
+
+class _Style:
+    def __init__(self, color: bool):
+        self.on = color
+
+    def __call__(self, code: str, text: str) -> str:
+        return f"\033[{code}m{text}\033[0m" if self.on else text
+
+    def bold(self, t: str) -> str: return self("1", t)
+    def dim(self, t: str) -> str: return self("2", t)
+    def accent(self, t: str) -> str: return self("1;34", t)
+    def warn(self, t: str) -> str: return self("33", t)
+    def good(self, t: str) -> str: return self("32", t)
+
+
+def _bars(points: list[tuple[int, int]], width: int) -> str:
+    """One character per day, a space for a day with no report; days share a cell when they do not fit."""
+    span = points[-1][0] + 1
+    cells: list[int | None] = [None] * min(span, width)
+    for x, v in points:
+        i = x * len(cells) // span
+        cells[i] = v if cells[i] is None else max(cells[i], v)
+    top = max(v for _, v in points) or 1
+    if min(v for _, v in points) == top:
+        # A flat line is the same every day; a full bar would read as a peak
+        return "".join(" " if v is None else BARS[3] for v in cells)
+    return "".join(" " if v is None else BARS[min(7, v * 7 // top)] if v else BARS[0] for v in cells)
+
+
+def _spread(points: list[tuple[int, int]]) -> str:
+    lo, hi = min(v for _, v in points), max(v for _, v in points)
+    if lo == hi:
+        return f"{lo:,}" if len(points) == 1 else f"{lo:,} every day"
+    return f"{lo:,} to {hi:,}"
+
+
+def _date(d) -> str:
+    return f"{d:%b} {d.day}"
+
+
+def _range(a, b) -> str:
+    if a == b:
+        return f"{_date(a)}, {a.year}"
+    return (_date(a) if a.year == b.year else f"{_date(a)}, {a.year}") + f" to {_date(b)}, {b.year}"
+
+
+def _wrap(text: str, width: int, indent: str = "") -> list[str]:
+    import textwrap
+    return textwrap.wrap(text, width=max(40, width), initial_indent=indent, subsequent_indent=indent) or [indent]
+
+
+def to_terminal(loaded: Loaded, color: bool = True, width: int = 100) -> str:
+    """A readable overview for a terminal: per stream, each count's latest value, range and trend."""
+    st = _Style(color)
+    ss = streams(loaded.reports)
+    out = [st.bold("APRF placement summary") + st.dim(f"  {_counts(loaded, ss)}"), ""]
+    out += [st.dim(line) for line in _wrap(CAVEAT, min(width, 100))] + [""]
+    for s in ss:
+        out.append(st.accent(s.domain) + "  " + st.bold(f"selector {s.selector}"))
+        meta = [s.source, _range(s.first, s.last), _plural(len(s.by_day), "report")]
+        line = st.dim(" · ".join(meta))
+        if s.missing:
+            line += st.dim(" · ") + st.warn(f"{_plural(s.missing, 'day')} with no report")
+        out += [line, ""]
+        names = [k for _, k in s.columns]
+        if not names:
+            out += ["  " + st.dim(NO_COUNTS), ""]
+            continue
+        nw = max([len(n) for n in names] + [5])
+        trends = [s.trend(i) for i in range(len(names))]
+        latest = [f"{t[-1][1]:,}" for t in trends]
+        ranges = [_spread(t) for t in trends]
+        lw, rw = max(len(x) for x in latest + ["latest"]), max(len(x) for x in ranges + ["range"])
+        bw = max(10, min(60, width - (2 + nw + 2 + lw + 2 + rw + 2)))
+        out.append(st.dim(f"  {'':<{nw}}  {'latest':>{lw}}  {'range':<{rw}}  trend"))
+        for name, t, l, r in zip(names, trends, latest, ranges):
+            out.append(f"  {name:<{nw}}  {st.bold(f'{l:>{lw}}')}  {r:<{rw}}  {st.accent(_bars(t, bw)) if len(t) > 1 else st.dim('one day')}")
+        out.append("")
+        if s.changes:
+            out.append(st.bold("  Changes worth a look"))
+            out += [f"  {st.warn('!')} " + re.sub(r"(\d{4})-(\d\d)-(\d\d)", lambda m: _date(date.fromisoformat(m[0])), c)
+                    for c in s.changes]
+        else:
+            out.append(f"  {st.good('✓')} " + st.dim("No big day-to-day moves in inbox, unwanted, or negative counts."))
+        for n in s.notes:
+            out += _wrap(n, width - 4, "  · ")
+        out.append("")
+    if loaded.skipped:
+        out.append(st.warn(f"Skipped {len(loaded.skipped)}"))
+        for w, r in loaded.skipped:
+            out += [st.dim(line) for line in _wrap(f"{w}: {r}", width - 2, "  ")]
+        out.append("")
+    out.append(st.dim(f"aprf-reader {__version__}, {DRAFT}. Daily tables: --markdown, or --html for a page."))
+    return "\n".join(out) + "\n"

{aprf_reader-0.2.0 → aprf_reader-0.3.1}/src/aprf_reader/report.py

@@ -15,10 +15,16 @@ DRAFT = "draft-ietf-mailmaint-aprf-00"
 # Names the draft defines, in display order; providers may add others
 PLACEMENT_NAMES = ("inbox", "unwanted")
 ENGAGEMENT_NAMES = ("positive", "neutral", "negative")
-# sdi_used values (first word) that mean no SDI was applied (section 5.2.1)
+# sdi_used values that mean no SDI was applied (section 5.2.1), compared on
+# the first word without case or trailing punctuation
 NO_SDI = ("", "N/A", "N/F")
 
 
+def uses_sdi(value: str) -> bool:
+    first = value.split(" ", 1)[0].rstrip(";,.").upper()
+    return first not in NO_SDI
+
+
 class ReportError(ValueError):
     """The data is not an APRF report this module can read."""
 
@@ -50,15 +56,45 @@ class Report:
         # Section 8: the report's date is its ending date, in UTC
         return _day(self.report_end)
 
+    @property
+    def stream_key(self) -> tuple[str, str, str]:
+        return (self.source, name_key(self.dkim_domain), self.dkim_selector.lower())
+
     @property
     def key(self) -> tuple[str, str, str, date]:
-        return (self.source, self.dkim_domain.lower(), self.dkim_selector, self.day)
+        return (*self.stream_key, self.day)
+
+
+def name_key(domain: str) -> str:
+    """DNS names compare without case, and bücher.example is xn--bcher-kva.example."""
+    d = domain.lower()
+    # Only plain letters, digits, dots and hyphens; the web page converts the same set
+    # shortcut: Python uses IDNA 2003 and browsers UTS 46, which differ on a few letters such as ß
+    if d.isascii() or not all(c.isalnum() or c in ".-" for c in d):
+        return d
+    try:
+        return d.encode("idna").decode("ascii")
+    except UnicodeError:
+        return d
 
 
 def _day(epoch: int) -> date:
     return datetime.fromtimestamp(epoch, UTC).date()
 
 
+# JavaScript numbers are exact only up to here; the web page reads the same files
+MAX_COUNT = 2**53 - 1
+
+
+def _int(v: object) -> int | None:
+    """An integer, or a float with nothing after the point (100.0); otherwise None."""
+    if isinstance(v, bool):
+        return None
+    if isinstance(v, float) and v.is_integer():
+        v = int(v)
+    return v if isinstance(v, int) else None
+
+
 def _counts(value: object, where: str) -> dict[str, int]:
     if value is None:
         return {}
@@ -66,9 +102,10 @@ def _counts(value: object, where: str) -> dict[str, int]:
         raise ReportError(f"{where} is not an object")
     out = {}
     for name, n in value.items():
-        if isinstance(n, bool) or not isinstance(n, int) or n < 0:
-            raise ReportError(f"{where}.{name} is not a count: {n!r}")
-        out[str(name)] = n
+        k = _int(n)
+        if k is None or k < 0 or k > MAX_COUNT:
+            raise ReportError(f"{where}.{name} is not a count: {json.dumps(n, ensure_ascii=False)}")
+        out[str(name)] = k
     return out
 
 
@@ -103,13 +140,19 @@ def _report(obj: object) -> Report:
     for name in ("dkim_domain", "dkim_selector"):
         if not isinstance(h.get(name), str) or not h[name]:
             raise ReportError(f"header.{name} is missing")
-    for name in ("report_start", "report_end"):
-        if isinstance(h.get(name), bool) or not isinstance(h.get(name), int):
+    start, end = _int(h.get("report_start")), _int(h.get("report_end"))
+    for name, v in (("report_start", start), ("report_end", end)):
+        if v is None:
             raise ReportError(f"header.{name} is not an epoch time")
+        # Seconds past 1e11 are beyond the year 5138; up to 1e14 that is milliseconds
+        if 10**11 <= abs(v) < 10**14:
+            raise ReportError(f"header.{name} looks like milliseconds, not seconds: {v}")
+    if end < start:
+        raise ReportError("header.report_end is before report_start")
     try:
-        _day(h["report_end"])
+        _day(end)
     except (OverflowError, OSError, ValueError):
-        raise ReportError(f"header.report_end is out of range: {h['report_end']}") from None
+        raise ReportError(f"header.report_end is out of range: {end}") from None
     body = obj.get("body", [])
     if not isinstance(body, list):
         raise ReportError("body is not a list")
@@ -117,8 +160,8 @@ def _report(obj: object) -> Report:
         source=_text(h.get("source")) or "unknown",
         dkim_domain=h["dkim_domain"],
         dkim_selector=h["dkim_selector"],
-        report_start=h["report_start"],
-        report_end=h["report_end"],
+        report_start=start,
+        report_end=end,
         version=_text(h.get("version")),
         # Comcast's beta sends " - N/A - Beta - Not in use"
         sdi_used=_text(h.get("sdi_used")).strip(" -"),
@@ -128,8 +171,9 @@ def _report(obj: object) -> Report:
     )
 
 
-def parse(data: bytes) -> list[Report]:
-    """Parse one report document. The draft's samples wrap reports in a list."""
+def parse_items(data: bytes) -> list[Report | ReportError]:
+    """Each report in a document, or the reason it cannot be read, so one bad
+    report does not cost the rest of the file."""
     try:
         doc = json.loads(data)
     except (UnicodeDecodeError, json.JSONDecodeError) as e:
@@ -139,4 +183,19 @@ def parse(data: bytes) -> list[Report]:
     items = doc if isinstance(doc, list) else [doc]
     if not items:
         raise ReportError("empty list")
-    return [_report(o) for o in items]
+    out: list[Report | ReportError] = []
+    for o in items:
+        try:
+            out.append(_report(o))
+        except ReportError as e:
+            out.append(e)
+    return out
+
+
+def parse(data: bytes) -> list[Report]:
+    """Parse one report document. The draft's samples wrap reports in a list."""
+    items = parse_items(data)
+    for r in items:
+        if isinstance(r, ReportError):
+            raise r
+    return items  # type: ignore[return-value]

{aprf_reader-0.2.0 → aprf_reader-0.3.1}/src/aprf_reader/summary.py

@@ -7,7 +7,12 @@ from dataclasses import dataclass
 from datetime import date, timedelta
 from functools import cached_property
 
-from .report import ENGAGEMENT_NAMES, NO_SDI, PLACEMENT_NAMES, Report
+from .report import ENGAGEMENT_NAMES, PLACEMENT_NAMES, Report, uses_sdi
+
+
+# Counts where a jump up is bad news, or a drop is
+CHANGES = {"unwanted": "rise", "negative": "rise", "inbox": "fall"}
+MIN_MOVE = 200
 
 
 @dataclass(frozen=True)
@@ -59,7 +64,7 @@ class Stream:
                 out.append(Row(prev + timedelta(1), d - timedelta(1), None))
             for s in self.by_day[d].segments:
                 vals = tuple(getattr(s, a).get(k) for a, k in self.columns)
-                out.append(Row(d, d, " / ".join(s.path) or "all mail", vals))
+                out.append(Row(d, d, " / ".join(p or "(blank)" for p in s.path) or "all mail", vals))
             prev = d
         return out
 
@@ -69,6 +74,27 @@ class Stream:
         return [((d - self.first).days, sum(getattr(s, a).get(k, 0) for s in r.segments))
                 for d, r in sorted(self.by_day.items())]
 
+    @cached_property
+    def changes(self) -> list[str]:
+        """Day-to-day moves big enough to look at, between one report day and the next."""
+        out = []
+        for col, (_, k) in enumerate(self.columns):
+            rule = CHANGES.get(k)
+            if rule is None:
+                continue
+            pts = self.trend(col)
+            for (x0, old), (x1, new) in zip(pts, pts[1:]):
+                # Comcast's buckets are 100 wide, so a move under two buckets is noise
+                if rule == "rise" and new >= 2 * old and new - old >= MIN_MOVE:
+                    verb = "rose"
+                elif rule == "fall" and 2 * new <= old and old - new >= MIN_MOVE:
+                    verb = "fell"
+                else:
+                    continue
+                d0, d1 = self.first + timedelta(x0), self.first + timedelta(x1)
+                out.append(f"{k} {verb} from {old:,} on {d0} to {new:,} on {d1}.")
+        return sorted(out, key=lambda t: t.rsplit(" on ", 1)[1])
+
     @cached_property
     def notes(self) -> list[str]:
         out = []
@@ -80,8 +106,7 @@ class Stream:
             prev = r.version
         # "TBD" is the draft samples' placeholder
         out += [f"Provider note: {t}" for t in sorted({r.extra_info for r in self.by_day.values()} - {"", "TBD"})]
-        sdi = sorted({r.sdi_used for r in self.by_day.values()
-                      if r.sdi_used.split(" ", 1)[0] not in NO_SDI})
+        sdi = sorted({r.sdi_used for r in self.by_day.values() if uses_sdi(r.sdi_used)})
         if sdi:
             out.append("SDI used: " + ", ".join(sdi))
         return out
@@ -90,5 +115,22 @@ class Stream:
 def streams(reports: list[Report]) -> list[Stream]:
     groups: dict[tuple[str, str, str], dict[date, Report]] = defaultdict(dict)
     for r in reports:
-        groups[(r.source, r.dkim_domain.lower(), r.dkim_selector)][r.day] = r
+        groups[r.stream_key][r.day] = r
     return [Stream(src, dom, sel, days) for (src, dom, sel), days in sorted(groups.items())]
+
+
+def to_dict(reports: list[Report], skipped: list[tuple[str, str]]) -> dict:
+    """The summary as plain data, for --json and for checking the web page against."""
+    return {
+        "reports": len(reports),
+        "streams": [{
+            "source": s.source, "domain": s.domain, "selector": s.selector,
+            "first": str(s.first), "last": str(s.last), "missing_days": s.missing,
+            "columns": [k for _, k in s.columns],
+            "days": [{"day": str(d), "version": r.version, "segments": [
+                {"segment": list(seg.path), "classification": seg.classification, "engagement": seg.engagement}
+                for seg in r.segments]} for d, r in sorted(s.by_day.items())],
+            "changes": s.changes, "notes": s.notes,
+        } for s in streams(reports)],
+        "skipped": [{"where": w, "reason": r} for w, r in skipped],
+    }

{aprf_reader-0.2.0 → aprf_reader-0.3.1/src/aprf_reader.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aprf-reader
-Version: 0.2.0
+Version: 0.3.1
 Summary: Summarise APRF (Aggregate Performance Reporting) inbox placement reports.
 License: MIT
 Requires-Python: >=3.11
@@ -15,9 +15,16 @@ Dynamic: license-file
 
 Reads APRF (Aggregate Performance Reporting) reports and tells you where the
 mail you signed landed: how much reached the inbox, how much was treated as
-unwanted, and how recipients reacted, day by day and per DKIM selector.
+unwanted, and how recipients reacted, day by day and per DKIM selector. It
+points out the days worth a look, such as a jump in unwanted mail.
 
-**Status: 0.2, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
+Two ways to use it, with the same results:
+
+- **In your browser:** [aprf.dns-audit.com](https://aprf.dns-audit.com). Drop
+  your reports on the page. They are read on your computer and never uploaded.
+- **On the command line:** `pipx install aprf-reader`.
+
+**Status: 0.3, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
 
 ## What APRF is
 
@@ -39,11 +46,18 @@ environment, or with pip (`pip3` on some systems) inside a virtual environment.
 
 ```bash
 pipx install aprf-reader
-aprf-reader reports/                      # Markdown summary to the terminal
-aprf-reader reports/ --html summary.html  # and a page you can open or share
+aprf-reader reports/                      # overview in the terminal
 aprf-reader APRF.mbox                     # a mail folder exported as mbox
+aprf-reader reports/ --markdown           # Markdown with daily tables
+aprf-reader reports/ --html summary.html  # and a page you can open or share
+aprf-reader reports/ --json               # the summary as data, for scripts
 ```
 
+In a terminal you get an overview: for each stream, each count's latest value,
+its daily range, and a trend line with a gap for each day with no report, then
+the changes worth a look. Piped or written to a file with `--md`, the output
+is Markdown with the full daily tables.
+
 Point it at report files (`.json`, `.json.gz`), saved report emails (`.eml`),
 mailbox exports (`.mbox`), or folders of them. The easiest way to collect reports
 is a mail filter that moves them into one folder, then an export of that folder:
@@ -55,6 +69,10 @@ on stderr and at the end of the summary, each with a reason.
 
 Each stream (provider, signing domain, selector) gets a table with one row per
 day and segment, a trend line per count, and a row for each day with no report.
+A change worth a look is a day when unwanted mail or negative reactions at
+least doubled and grew by 200 or more, or inbox mail fell by half and by 200
+or more, against the report before. Comcast counts in steps of 100, so smaller
+moves are left out as rounding.
 A change in the report's `version` field is called out, since the draft uses it
 to mark a change in what a category counts.
 
@@ -73,6 +91,17 @@ to mark a change in what a category counts.
 - When the same day arrives twice, keep the first copy and say so. The draft
   has not yet decided whether a resend replaces the original.
 
+## How the web page is built
+
+`web/src/aprf.js` is a JavaScript port of the Python reader.
+`tests/test_web_parity.py` runs both on the same files (JSON, gzip, saved
+emails, forwarded emails, mailbox exports, duplicates and hostile text) and
+requires the same JSON, Markdown and HTML output, so the page and the command
+line cannot drift apart. `node web/build.js` inlines everything into one
+`web/dist/index.html` whose Content Security Policy blocks every network
+request, so it also works saved to disk. A merge to main that passes the
+tests deploys it to Cloudflare.
+
 ## Related
 
 - [dns-audit.com](https://dns-audit.com) checks a domain for an APRF record and

{aprf_reader-0.2.0 → aprf_reader-0.3.1}/src/aprf_reader.egg-info/SOURCES.txt

@@ -13,4 +13,5 @@ src/aprf_reader.egg-info/dependency_links.txt
 src/aprf_reader.egg-info/entry_points.txt
 src/aprf_reader.egg-info/requires.txt
 src/aprf_reader.egg-info/top_level.txt
-tests/test_reader.py
+tests/test_reader.py
+tests/test_web_parity.py

{aprf_reader-0.2.0 → aprf_reader-0.3.1}/tests/test_reader.py

@@ -1,6 +1,7 @@
 import gzip
 import json
 import mailbox
+import re
 from datetime import date
 from email.message import EmailMessage
 from pathlib import Path
@@ -187,3 +188,70 @@ def test_cli(tmp_path, capsys):
     assert out.read_text().startswith("<!doctype html>")
     (tmp_path / "empty").mkdir()
     assert main([str(tmp_path / "empty")]) == 1
+
+
+def test_changes_flag_big_moves_only():
+    (s1, s2) = streams(parse((FIX / "sample_six_weeks.json").read_bytes()))
+    assert s1.selector == "news" and s1.changes == [
+        "unwanted rose from 300 on 2026-09-24 to 2,500 on 2026-09-25.",
+        "negative rose from 100 on 2026-09-24 to 400 on 2026-09-25.",
+        "inbox fell from 2,200 on 2026-09-25 to 600 on 2026-09-26.",
+    ]
+    assert s2.changes == []  # steady stream, nothing flagged
+    # 10 to 200 is many times higher but under two buckets of movement
+    (s,) = streams([parse(json.dumps(report(0, inbox=100)).encode())[0],
+                    parse(json.dumps(report(1, inbox=100) | {"body": [{"classification": {"unwanted": 200}}]}).encode())[0]])
+    assert not any("unwanted" in c for c in s.changes)
+
+
+def test_json_output(capsys):
+    assert main([str(FIX / "sample_six_weeks.json"), "--json"]) == 0
+    doc = json.loads(capsys.readouterr().out)
+    assert doc["reports"] == 80 and [s["selector"] for s in doc["streams"]] == ["news", "receipts"]
+    news = doc["streams"][0]
+    assert news["missing_days"] == 2 and news["columns"] == ["inbox", "unwanted", "positive", "negative"]
+    assert news["days"][0]["segments"][0]["segment"] == [] and len(news["changes"]) == 3
+
+
+def test_terminal_view():
+    from aprf_reader.render import to_terminal
+    loaded = load([FIX / "sample_six_weeks.json", FIX / "comcast_beta.json"])
+    out = to_terminal(loaded, color=False, width=100)
+    assert "example.com  selector news" in out and "2 days with no report" in out
+    assert "! unwanted rose from 300 on Sep 24 to 2,500 on Sep 25." in out
+    assert "✓ No big day-to-day moves" in out and "one day" in out
+    assert "\033[" not in out and "\033[" in to_terminal(loaded, color=True)
+    assert max(len(line) for line in out.splitlines()) <= 100
+    inbox = next(line for line in out.splitlines() if line.startswith("  inbox"))
+    assert re.search("[▁-█] +[▁-█]", inbox)  # the two days with no report show as a break
+
+
+def test_cli_terminal_when_interactive(monkeypatch, capsys):
+    monkeypatch.setattr("sys.stdout.isatty", lambda: True)
+    monkeypatch.setenv("NO_COLOR", "1")
+    assert main([str(FIX / "sample_six_weeks.json")]) == 0
+    out = capsys.readouterr().out
+    assert out.startswith("APRF placement summary") and "\033[" not in out
+    assert main([str(FIX / "sample_six_weeks.json"), "--markdown"]) == 0
+    assert capsys.readouterr().out.startswith("# APRF placement summary")
+
+
+def test_stress_set_one_bad_report_does_not_cost_the_file():
+    loaded = load([FIX / "stress_100.json"])
+    assert len(loaded.reports) == 88
+    reasons = {w.rsplit(" ", 1)[-1]: r for w, r in loaded.skipped}
+    assert reasons["73"] == 'body[0].engagement.positive is not a count: "100"'
+    assert reasons["79"] == "header.report_end is before report_start"
+    assert reasons["81"].startswith("header.report_start looks like milliseconds")
+    assert "report 63" in reasons["64"]  # bücher.example is xn--bcher-kva.example
+    assert "report 3" in reasons["70"]  # DNS-AUDIT.COM / SELECTOR1 is dns-audit.com / selector1
+    ss = {(s.source, s.domain, s.selector): s for s in streams(loaded.reports)}
+    obj = report(0)
+    obj["body"][0]["classification"]["unwanted"] = 100.0
+    assert parse(json.dumps(obj).encode())[0].segments[0].classification["unwanted"] == 100
+    for sel in ("sdi0", "sdi1", "sdi2", "sdi3", "sdi4"):  # n/a, "N/A ", "", null, N/F;
+        assert not ss[("Comcast/Xfinity", "dns-audit.com", sel)].notes, sel
+    blank = ss[("Comcast/Xfinity", "dns-audit.com", "s13")]
+    assert "(blank)" in [r.segment for r in blank.rows]
+    assert not ss[("Comcast/Xfinity", "dns-audit.com", "s8")].columns
+    assert "These reports carry no counts." in to_markdown(loaded)

aprf_reader-0.3.1/tests/test_web_parity.py

@@ -0,0 +1,106 @@
+"""The web page (web/src/aprf.js) must read files exactly as the command line does."""
+
+import gzip
+import json
+import mailbox
+import shutil
+import subprocess
+from email.message import EmailMessage
+from pathlib import Path
+
+import pytest
+
+from aprf_reader import __version__
+from aprf_reader.load import load
+from aprf_reader.render import to_html, to_markdown
+from aprf_reader.summary import to_dict
+
+ROOT = Path(__file__).parent.parent
+FIX = Path(__file__).parent / "fixtures"
+NODE = shutil.which("node")
+
+pytestmark = pytest.mark.skipif(NODE is None, reason="node is not installed")
+
+
+def web(mode: str, paths: list[Path]) -> str:
+    return subprocess.run([NODE, str(ROOT / "web/test/run.js"), mode, *map(str, paths)], check=True,
+                          capture_output=True, text=True, env={"APRF_VERSION": __version__}).stdout
+
+
+def same(paths: list[Path]) -> None:
+    loaded = load(paths)
+    assert json.loads(web("--json", paths)) == to_dict(loaded.reports, loaded.skipped)
+    assert web("--md", paths) == to_markdown(loaded)
+    assert web("--html", paths) == to_html(loaded)
+
+
+def report(day: int, selector: str = "s1", inbox: int = 100) -> dict:
+    start = 1790812800 + day * 86400
+    return {"header": {"version": 1, "source": "Comcast/Xfinity", "dkim_domain": "Example.com",
+                       "dkim_selector": selector, "report_start": start, "report_end": start + 86399,
+                       "sdi_used": " - N/A - Beta - Not in use"},
+            "body": [{"classification": {"inbox": inbox, "unwanted": 0}, "engagement": {"positive": 0}}]}
+
+
+def email(payload: bytes, subtype: str, filename: str | None, cte: str | None = None) -> EmailMessage:
+    msg = EmailMessage()
+    msg["From"] = "aprf-noreply@example.net"
+    msg["Subject"] = "APRF: test"
+    msg.set_content("Report attached.")
+    # cte=None would mean raw binary, not the base64 default
+    msg.add_attachment(payload, maintype="application", subtype=subtype, filename=filename, **({"cte": cte} if cte else {}))
+    return msg
+
+
+@pytest.mark.parametrize("name", ["draft_sample_1.json", "draft_sample_2.json", "comcast_beta.json", "sample_six_weeks.json",
+                                  "stress_100.json"])
+def test_fixtures(name):
+    same([FIX / name])
+
+
+def test_all_fixtures_together():
+    same(sorted(FIX.glob("*.json")))
+
+
+def test_files_emails_and_mailboxes(tmp_path):
+    def put(name: str, data: bytes) -> Path:
+        (tmp_path / name).write_bytes(data)
+        return tmp_path / name
+
+    raw = lambda day, **kw: json.dumps(report(day, **kw)).encode()
+    paths = [
+        put("a.json", raw(0)),
+        put("b.json.gz", gzip.compress(raw(1))),
+        put("c.eml", email(gzip.compress(raw(2)), "gzip", "20261003_example.com_s1_Comcast.json.gz").as_bytes()),
+        put("d.eml", email(raw(3), "octet-stream", None).as_bytes()),
+        put("e.eml", email(raw(4), "json", "r.json", cte="quoted-printable").as_bytes()),
+        put("copy.json", raw(0)),
+        put("resend.json", raw(0, inbox=900)),
+        put("note.eml", email(b"hello", "pdf", "invoice.pdf").as_bytes()),
+    ]
+    # A report email forwarded as an attachment
+    outer = EmailMessage()
+    outer["Subject"] = "Fwd: APRF"
+    outer.set_content("See attached.")
+    outer.add_attachment(email(raw(5, inbox=3000), "json", "f.json"))
+    paths.append(put("fwd.eml", outer.as_bytes()))
+    box = mailbox.mbox(tmp_path / "Reports.mbox")
+    for day in (6, 7, 9):
+        box.add(email(gzip.compress(raw(day, selector="s2")), "gzip", f"{day}.json.gz"))
+    box.add(email(raw(8, selector="s2"), "json", "same.json"))
+    plain = EmailMessage()
+    plain.set_content("From the team: not a report")
+    box.add(plain)
+    box.close()
+    paths.append(tmp_path / "Reports.mbox")
+    same(paths)
+
+
+def test_hostile_text_escaped_the_same(tmp_path):
+    obj = report(0)
+    hostile = "<script>alert('x')</script>|a\\b \"q\" & é\nnew"
+    obj["header"].update(source=hostile, extra_info=hostile, sdi_used=hostile)
+    obj["body"][0]["segment"] = [hostile, "two"]
+    obj["body"][0]["classification"][hostile] = 7
+    (tmp_path / "h.json").write_text(json.dumps(obj))
+    same([tmp_path / "h.json"])