aprf-reader 0.4.1__tar.gz → 0.6.0__tar.gz

{aprf_reader-0.4.1/src/aprf_reader.egg-info → aprf_reader-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aprf-reader
-Version: 0.4.1
+Version: 0.6.0
 Summary: Summarise APRF (Aggregate Performance Reporting) inbox placement reports.
 License: MIT
 Requires-Python: >=3.11
@@ -20,11 +20,11 @@ points out the days worth a look, such as a jump in unwanted mail.
 
 Two ways to use it, with the same results:
 
-- **In your browser:** [aprf.dns-audit.com](https://aprf.dns-audit.com). Drop
-  your reports on the page. They are read on your computer and never uploaded.
+- **On the web:** [aprf.dns-audit.com](https://aprf.dns-audit.com). Drop or
+  paste your reports on the page.
 - **On the command line:** `pipx install aprf-reader`.
 
-**Status: 0.4, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
+**Status: 0.6, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
 
 ## What APRF is
 
@@ -51,6 +51,7 @@ aprf-reader APRF.mbox                     # a mail folder exported as mbox
 aprf-reader reports/ --markdown           # Markdown with daily tables
 aprf-reader reports/ --html summary.html  # and a page you can open or share
 aprf-reader reports/ --json               # the summary as data, for scripts
+aprf-reader --imap imap.gmail.com --user you@gmail.com --folder APRF
 ```
 
 In a terminal you get an overview: for each stream, each count's latest value,
@@ -59,12 +60,26 @@ the changes worth a look. Piped or written to a file with `--md`, the output
 is Markdown with the full daily tables.
 
 Point it at report files (`.json`, `.json.gz`), saved report emails (`.eml`),
-mailbox exports (`.mbox`), or folders of them. The easiest way to collect reports
+mailbox exports (`.mbox`), `.zip` files of any of those, or folders of them. The easiest way to collect reports
 is a mail filter that moves them into one folder, then an export of that folder:
 Gmail through [Google Takeout](https://takeout.google.com/) (pick Mail and the
 label), Thunderbird with the ImportExportTools NG add-on, or Apple Mail's
 Mailbox > Export Mailbox. Comcast's reports come from
-`aprf-noreply@alerts.xfinity.com` with subjects starting `APRF:`. Folders are searched recursively. Files it skips are listed
+`aprf-noreply@alerts.xfinity.com` with subjects starting `APRF:`. Folders are searched recursively.
+
+Or skip the export and read the folder straight from your mailbox over IMAP:
+
+```bash
+aprf-reader --imap imap.gmail.com --user you@gmail.com --folder APRF --since 2026-10-01
+```
+
+It opens the folder read-only and fetches with `BODY.PEEK`, so nothing is
+marked read, moved or deleted. It reads messages whose subject contains
+`APRF:`; `--all-mail` reads every message in the folder. The password comes
+from the `APRF_IMAP_PASSWORD` environment variable, or it asks. Gmail and
+Outlook.com need an app password, not your account password, and a Gmail
+label is read as a folder of the same name. The connection uses TLS on port
+993; give another port as `--imap host:port`. Files it skips are listed
 on stderr and at the end of the summary, each with a reason.
 
 Each stream (provider, signing domain, selector) gets a table with one row per
@@ -78,14 +93,16 @@ to mark a change in what a category counts.
 
 ## What it does
 
-- Read report files (`.json`, `.json.gz`, `.eml` and `.mbox`) or a folder of them, and skip
+- Read report files (`.json`, `.json.gz`, `.eml`, `.mbox` and `.zip`), a folder of them, or a
+  mailbox folder over IMAP, and skip
   duplicates and files it cannot read, with a reason for each.
 - Summarise placement per signing domain and selector: inbox, unwanted and
   recipient reactions, by day and as a trend.
 - Treat the counts as what the draft says they are: rounded into buckets, so
   they are a signal, not exact rates.
 - Write a Markdown summary and a single self-contained HTML page you can open
-  or share. Nothing is uploaded anywhere.
+  or share. The command line tool sends nothing anywhere; with `--imap` it
+  only talks to the mail server you name.
 - Follow the draft as it changes. The report format lives in one module
   (`src/aprf_reader/report.py`), so a new draft revision is a small update.
 - Check each report against the draft and skip, with the reason, any it
@@ -152,8 +169,8 @@ repeats a report already read, such as the same report as `.json` and as
 emails, forwarded emails, mailbox exports, duplicates and hostile text) and
 requires the same JSON, Markdown and HTML output, so the page and the command
 line cannot drift apart. `node web/build.js` inlines everything into one
-`web/dist/index.html` whose Content Security Policy blocks every network
-request, so it also works saved to disk. A merge to main that passes the
+`web/dist/index.html` whose Content Security Policy allows one connection, the
+usage note to dns-audit.com, and it also works saved to disk. A merge to main that passes the
 tests deploys it to Cloudflare.
 
 ## Related

{aprf_reader-0.4.1 → aprf_reader-0.6.0}/README.md

@@ -7,11 +7,11 @@ points out the days worth a look, such as a jump in unwanted mail.
 
 Two ways to use it, with the same results:
 
-- **In your browser:** [aprf.dns-audit.com](https://aprf.dns-audit.com). Drop
-  your reports on the page. They are read on your computer and never uploaded.
+- **On the web:** [aprf.dns-audit.com](https://aprf.dns-audit.com). Drop or
+  paste your reports on the page.
 - **On the command line:** `pipx install aprf-reader`.
 
-**Status: 0.4, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
+**Status: 0.6, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
 
 ## What APRF is
 
@@ -38,6 +38,7 @@ aprf-reader APRF.mbox                     # a mail folder exported as mbox
 aprf-reader reports/ --markdown           # Markdown with daily tables
 aprf-reader reports/ --html summary.html  # and a page you can open or share
 aprf-reader reports/ --json               # the summary as data, for scripts
+aprf-reader --imap imap.gmail.com --user you@gmail.com --folder APRF
 ```
 
 In a terminal you get an overview: for each stream, each count's latest value,
@@ -46,12 +47,26 @@ the changes worth a look. Piped or written to a file with `--md`, the output
 is Markdown with the full daily tables.
 
 Point it at report files (`.json`, `.json.gz`), saved report emails (`.eml`),
-mailbox exports (`.mbox`), or folders of them. The easiest way to collect reports
+mailbox exports (`.mbox`), `.zip` files of any of those, or folders of them. The easiest way to collect reports
 is a mail filter that moves them into one folder, then an export of that folder:
 Gmail through [Google Takeout](https://takeout.google.com/) (pick Mail and the
 label), Thunderbird with the ImportExportTools NG add-on, or Apple Mail's
 Mailbox > Export Mailbox. Comcast's reports come from
-`aprf-noreply@alerts.xfinity.com` with subjects starting `APRF:`. Folders are searched recursively. Files it skips are listed
+`aprf-noreply@alerts.xfinity.com` with subjects starting `APRF:`. Folders are searched recursively.
+
+Or skip the export and read the folder straight from your mailbox over IMAP:
+
+```bash
+aprf-reader --imap imap.gmail.com --user you@gmail.com --folder APRF --since 2026-10-01
+```
+
+It opens the folder read-only and fetches with `BODY.PEEK`, so nothing is
+marked read, moved or deleted. It reads messages whose subject contains
+`APRF:`; `--all-mail` reads every message in the folder. The password comes
+from the `APRF_IMAP_PASSWORD` environment variable, or it asks. Gmail and
+Outlook.com need an app password, not your account password, and a Gmail
+label is read as a folder of the same name. The connection uses TLS on port
+993; give another port as `--imap host:port`. Files it skips are listed
 on stderr and at the end of the summary, each with a reason.
 
 Each stream (provider, signing domain, selector) gets a table with one row per
@@ -65,14 +80,16 @@ to mark a change in what a category counts.
 
 ## What it does
 
-- Read report files (`.json`, `.json.gz`, `.eml` and `.mbox`) or a folder of them, and skip
+- Read report files (`.json`, `.json.gz`, `.eml`, `.mbox` and `.zip`), a folder of them, or a
+  mailbox folder over IMAP, and skip
   duplicates and files it cannot read, with a reason for each.
 - Summarise placement per signing domain and selector: inbox, unwanted and
   recipient reactions, by day and as a trend.
 - Treat the counts as what the draft says they are: rounded into buckets, so
   they are a signal, not exact rates.
 - Write a Markdown summary and a single self-contained HTML page you can open
-  or share. Nothing is uploaded anywhere.
+  or share. The command line tool sends nothing anywhere; with `--imap` it
+  only talks to the mail server you name.
 - Follow the draft as it changes. The report format lives in one module
   (`src/aprf_reader/report.py`), so a new draft revision is a small update.
 - Check each report against the draft and skip, with the reason, any it
@@ -139,8 +156,8 @@ repeats a report already read, such as the same report as `.json` and as
 emails, forwarded emails, mailbox exports, duplicates and hostile text) and
 requires the same JSON, Markdown and HTML output, so the page and the command
 line cannot drift apart. `node web/build.js` inlines everything into one
-`web/dist/index.html` whose Content Security Policy blocks every network
-request, so it also works saved to disk. A merge to main that passes the
+`web/dist/index.html` whose Content Security Policy allows one connection, the
+usage note to dns-audit.com, and it also works saved to disk. A merge to main that passes the
 tests deploys it to Cloudflare.
 
 ## Related

{aprf_reader-0.4.1 → aprf_reader-0.6.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "aprf-reader"
-version = "0.4.1"
+version = "0.6.0"
 description = "Summarise APRF (Aggregate Performance Reporting) inbox placement reports."
 readme = "README.md"
 license = { text = "MIT" }

{aprf_reader-0.4.1 → aprf_reader-0.6.0}/src/aprf_reader/__init__.py

@@ -1,3 +1,3 @@
 """Read APRF aggregate performance reports."""
 
-__version__ = "0.4.1"
+__version__ = "0.6.0"

aprf_reader-0.6.0/src/aprf_reader/cli.py

@@ -0,0 +1,84 @@
+"""Command line entry point: aprf-reader <paths...>"""
+
+from __future__ import annotations
+
+import argparse
+import getpass
+import json
+import os
+import shutil
+import sys
+from datetime import date
+from pathlib import Path
+
+from . import __version__
+from .imap import ImapError, fetch
+from .load import load
+from .render import to_html, to_markdown, to_terminal
+from .summary import to_dict
+
+
+def main(argv: list[str] | None = None) -> int:
+    p = argparse.ArgumentParser(prog="aprf-reader", description="Summarise APRF placement reports.")
+    p.add_argument("paths", nargs="*", type=Path,
+                   help="report files (.json, .json.gz, .eml, .mbox, .zip) or folders; - reads a pasted report or email from stdin")
+    p.add_argument("--md", type=Path, help="write the summary here instead of printing it")
+    p.add_argument("--html", type=Path, help="also write a self-contained HTML page here")
+    p.add_argument("--markdown", action="store_true", help="print Markdown with daily tables, even in a terminal")
+    p.add_argument("--json", action="store_true", help="print the summary as JSON")
+    imap = p.add_argument_group("reading a mailbox over IMAP (read-only: nothing is marked read, moved or deleted)")
+    imap.add_argument("--imap", metavar="HOST[:PORT]", help="IMAP server, such as imap.gmail.com; TLS on port 993 unless given")
+    imap.add_argument("--user", help="IMAP login name; the password is read from APRF_IMAP_PASSWORD or asked for")
+    imap.add_argument("--folder", default="INBOX", help="folder to read (default INBOX)")
+    imap.add_argument("--since", type=date.fromisoformat, metavar="YYYY-MM-DD", help="only mail received on or after this day")
+    imap.add_argument("--all-mail", action="store_true", help='read every message in the folder, not only subjects with "APRF:"')
+    p.add_argument("--version", action="version", version=f"aprf-reader {__version__}")
+    args = p.parse_args(argv)
+    if not args.paths and not args.imap:
+        p.error("give report files or folders, or --imap")
+    if args.imap and not args.user:
+        p.error("--imap needs --user")
+
+    paths = [x for x in args.paths if str(x) != "-"]
+    for path in paths:
+        if not path.exists():
+            p.error(f"no such file or folder: {path}")
+    pasted = sys.stdin.buffer.read() if len(paths) < len(args.paths) else None
+    messages = []
+    if args.imap:
+        host, _, port = args.imap.partition(":")
+        if port and not port.isdigit():
+            p.error(f"--imap port is not a number: {port}")
+        password = os.environ.get("APRF_IMAP_PASSWORD") or getpass.getpass(f"IMAP password for {args.user}: ")
+        try:
+            # Fetched in full before reading, so a dropped connection leaves no half summary
+            messages = list(fetch(host, args.user, password, args.folder, int(port or 993), args.since, args.all_mail))
+        except (ImapError, OSError) as e:
+            print(f"aprf-reader: {args.imap}: {e}", file=sys.stderr)
+            return 2
+        print(f"read {len(messages)} message{'s' if len(messages) != 1 else ''} from {args.folder} on {host}", file=sys.stderr)
+    loaded = load(paths, pasted, messages, f"imap://{args.imap}/{args.folder}" if args.imap else "mailbox")
+    for where, reason in loaded.skipped:
+        print(f"skipped {where}: {reason}", file=sys.stderr)
+    if not loaded.reports:
+        print("no APRF reports found", file=sys.stderr)
+        return 1
+
+    if args.json:
+        text = json.dumps(to_dict(loaded), indent=2) + "\n"
+    elif args.md or args.markdown or not sys.stdout.isatty():
+        text = to_markdown(loaded)  # files and pipes get Markdown, which other tools read
+    else:
+        color = "NO_COLOR" not in os.environ and os.environ.get("TERM") != "dumb"
+        text = to_terminal(loaded, color=color, width=shutil.get_terminal_size().columns)
+    if args.md:
+        args.md.write_text(text, encoding="utf-8")
+    else:
+        sys.stdout.write(text)
+    if args.html:
+        args.html.write_text(to_html(loaded), encoding="utf-8")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

aprf_reader-0.6.0/src/aprf_reader/imap.py

@@ -0,0 +1,77 @@
+"""Read report emails straight from a mailbox over IMAP, without changing anything there."""
+
+from __future__ import annotations
+
+import imaplib
+import re
+from collections.abc import Iterator
+from datetime import date
+
+# Comcast's subjects start "APRF:"; IMAP SUBJECT matches a substring, without case
+SUBJECT = "APRF:"
+# Messages fetched per round trip
+BATCH = 50
+# A few years of daily reports for a few selectors; more is a wrong folder
+MAX_MESSAGES = 20_000
+
+
+class ImapError(Exception):
+    pass
+
+
+def _check(typ: str, data: list, what: str) -> list:
+    if typ != "OK":
+        detail = b" ".join(d for d in data if isinstance(d, bytes)).decode(errors="replace")
+        raise ImapError(f"{what} failed: {detail or typ}")
+    return data
+
+
+def _quote(s: str) -> str:
+    return '"' + s.replace("\\", "\\\\").replace('"', '\\"') + '"'
+
+
+def criteria(since: date | None = None, every: bool = False) -> list[str]:
+    out = [] if every else ["SUBJECT", _quote(SUBJECT)]
+    if since:
+        out += ["SINCE", since.strftime("%d-%b-%Y")]
+    return out or ["ALL"]
+
+
+def fetch(host: str, user: str, password: str, folder: str = "INBOX", port: int = 993,
+          since: date | None = None, every: bool = False, conn=None) -> Iterator[tuple[str, bytes]]:
+    """Yield (label, raw message bytes) for each matching message in one folder.
+
+    The folder is opened read-only and bodies are fetched with BODY.PEEK, so no
+    message is marked read, moved or deleted."""
+    m = conn or imaplib.IMAP4_SSL(host, port, timeout=60)
+    try:
+        try:
+            _check(*m.login(user, password), "login")
+        except imaplib.IMAP4.error as e:
+            raise ImapError(f"login failed: {e}") from None
+        try:
+            _check(*m.select(_quote(folder), readonly=True), f"opening folder {folder}")
+        except imaplib.IMAP4.error as e:
+            raise ImapError(f"opening folder {folder} failed: {e}") from None
+        (ids,) = _check(*m.uid("SEARCH", *criteria(since, every)), "search")
+        uids = (ids or b"").split()
+        if len(uids) > MAX_MESSAGES:
+            raise ImapError(f"{len(uids):,} messages match, more than {MAX_MESSAGES:,}; "
+                            "pick the folder the reports are filed in, or use --since")
+        for i in range(0, len(uids), BATCH):
+            chunk = b",".join(uids[i:i + BATCH]).decode()
+            data = _check(*m.uid("FETCH", chunk, "(UID RFC822.SIZE BODY.PEEK[])"), "fetch")
+            for item in data:
+                if not isinstance(item, tuple):
+                    continue  # the closing ")" of each response
+                head, body = item
+                # Most servers put UID before the body; one that puts it after still has a sequence number
+                uid = re.search(rb"UID (\d+)", head)
+                which = f"uid {uid.group(1).decode()}" if uid else f"message {head.split(b' ', 1)[0].decode(errors='replace')}"
+                label = f"imap://{host}/{folder} {which}"
+                yield label, body
+    finally:
+        try:
+            m.logout()
+        except (imaplib.IMAP4.error, OSError):
+            pass

{aprf_reader-0.4.1 → aprf_reader-0.6.0}/src/aprf_reader/load.py

@@ -7,6 +7,7 @@ import hashlib
 import io
 import mailbox
 import re
+import zipfile
 import zlib
 from collections.abc import Iterable, Iterator
 from dataclasses import dataclass, field
@@ -17,7 +18,9 @@ from pathlib import Path
 
 from .report import Report, ReportError, parse_items
 
-SUFFIXES = (".json", ".json.gz", ".eml", ".mbox")
+SUFFIXES = (".json", ".json.gz", ".eml", ".mbox", ".zip")
+# A zip of a year of daily reports for a few selectors is a few thousand files
+MAX_ZIP_ENTRIES = 20_000
 # A daily report is a few KB; this stops a stray huge file or gzip bomb
 MAX_BYTES = 10 * 1024 * 1024
 
@@ -97,20 +100,50 @@ def _attachments(msg: Message, label: str) -> Iterator[tuple[str, bytes, Mail]]:
             yield where, data, mail(False)
 
 
-def _payloads(path: Path) -> Iterator[tuple[str, bytes, Mail | None]]:
-    """Yield (label, report JSON bytes, how it arrived) for each report in a file."""
-    data = path.read_bytes()
+def _payloads(label: str, data: bytes) -> Iterator[tuple[str, bytes, Mail | None]]:
+    """Yield (label, report JSON bytes, how it arrived) for each report in one file's bytes."""
     if len(data) > MAX_BYTES:
         raise ReportError("larger than 10 MB")
-    if path.name.lower().endswith(".eml"):
-        found = list(_attachments(BytesParser(policy=policy.default).parsebytes(data), str(path)))
+    if label.lower().endswith(".eml"):
+        found = list(_attachments(BytesParser(policy=policy.default).parsebytes(data), label))
         if not found:
             raise ReportError("no report attachment, which section 8 calls for")
         yield from found
     elif data[:2] == b"\x1f\x8b":
-        yield str(path), _gunzip(data), None
+        yield label, _gunzip(data), None
     else:
-        yield str(path), data, None
+        yield label, data, None
+
+
+def _zip(path: Path, out: Loaded) -> Iterator[tuple[str, bytes]]:
+    """Yield (label, bytes) for each report file in a zip, in name order; a bad member is skipped, not fatal."""
+    try:
+        z = zipfile.ZipFile(path)
+    except zipfile.BadZipFile:
+        raise ReportError("not a readable zip file") from None
+    with z:
+        # __MACOSX holds Finder metadata copies named like the real files
+        members = sorted((i for i in z.infolist() if not i.is_dir() and not i.filename.startswith("__MACOSX/")
+                          and i.filename.lower().endswith(SUFFIXES)), key=lambda i: i.filename)
+        if len(members) > MAX_ZIP_ENTRIES:
+            raise ReportError(f"more than {MAX_ZIP_ENTRIES:,} report files in one zip")
+        for i in members:
+            label = f"{path}/{i.filename}"
+            name = i.filename.lower()
+            if name.endswith((".zip", ".mbox")):
+                out.skipped.append((label, "a zip or mailbox inside a zip is not read; unzip it first"))
+            elif i.flag_bits & 1:
+                out.skipped.append((label, "encrypted in the zip"))
+            elif i.file_size > MAX_BYTES:
+                out.skipped.append((label, "larger than 10 MB"))
+            else:
+                try:
+                    with z.open(i) as f:
+                        data = f.read(MAX_BYTES + 1)
+                except (zipfile.BadZipFile, NotImplementedError, zlib.error, EOFError, OSError):
+                    out.skipped.append((label, "could not be unpacked from the zip"))
+                    continue
+                yield label, data
 
 
 def _mbox(path: Path, out: Loaded) -> Iterator[tuple[str, bytes, Mail | None]]:
@@ -190,9 +223,25 @@ def _mail_warnings(mail: Mail, items: list) -> list[tuple[str, str]]:
     return out
 
 
-def load(paths: Iterable[Path]) -> Loaded:
+def pasted_name(data: bytes) -> str:
+    """A name for pasted text, so it is read like a file: a report's JSON, or a whole
+    email such as Gmail's Show original, attachment and all."""
+    head = data.lstrip()[:1]
+    if head in (b"[", b"{"):
+        return "pasted.json"
+    return "pasted.json.gz" if data[:2] == b"\x1f\x8b" else "pasted.eml"
+
+
+def _message(label: str, data: bytes) -> list[tuple[str, bytes, Mail]]:
+    """Reports in one raw email fetched from a mailbox; mail with none is counted, not skipped."""
+    if len(data) > MAX_BYTES:
+        raise ReportError("larger than 10 MB")
+    return list(_attachments(BytesParser(policy=policy.default).parsebytes(data), label))
+
+
+def load(paths: Iterable[Path], pasted: bytes | None = None,
+         messages: Iterable[tuple[str, bytes]] = (), source: str = "mailbox") -> Loaded:
     out = Loaded()
-    seen_bytes: set[str] = set()
     seen_keys: dict[tuple, tuple[str, Report]] = {}
 
     parsed: dict[str, list | ReportError] = {}
@@ -232,10 +281,38 @@ def load(paths: Iterable[Path]) -> Loaded:
 
     for path in _files(paths):
         try:
+            if path.name.lower().endswith(".zip"):
+                for label, data in _zip(path, out):
+                    try:
+                        for where, d, mail in list(_payloads(label, data)):
+                            add(where, d, mail)
+                    except (OSError, EOFError, zlib.error, ReportError) as e:
+                        out.skipped.append((label, str(e)))
+                continue
             # An mbox can be large, so it is read one message at a time
-            payloads = _mbox(path, out) if _is_mbox(path) else list(_payloads(path))
+            payloads = _mbox(path, out) if _is_mbox(path) else list(_payloads(str(path), path.read_bytes()))
             for where, data, mail in payloads:
                 add(where, data, mail)
         except (OSError, EOFError, zlib.error, ReportError) as e:
             out.skipped.append((str(path), str(e)))
+    # Raw emails from IMAP, read like an mbox: one line for all mail with no report
+    without = 0
+    for label, data in messages:
+        try:
+            found = _message(label, data)
+        except (OSError, EOFError, zlib.error, ReportError) as e:
+            out.skipped.append((label, str(e)))
+            continue
+        without += not found
+        for where, d, mail in found:
+            add(where, d, mail)
+    if without:
+        out.skipped.append((source, f"{without} message{'s' if without != 1 else ''} with no report attached"))
+    if pasted is not None:
+        name = pasted_name(pasted)
+        try:
+            for where, data, mail in list(_payloads(name, pasted)):
+                add(where, data, mail)
+        except (OSError, EOFError, zlib.error, ReportError) as e:
+            out.skipped.append((name, str(e)))
     return out

{aprf_reader-0.4.1 → aprf_reader-0.6.0/src/aprf_reader.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aprf-reader
-Version: 0.4.1
+Version: 0.6.0
 Summary: Summarise APRF (Aggregate Performance Reporting) inbox placement reports.
 License: MIT
 Requires-Python: >=3.11
@@ -20,11 +20,11 @@ points out the days worth a look, such as a jump in unwanted mail.
 
 Two ways to use it, with the same results:
 
-- **In your browser:** [aprf.dns-audit.com](https://aprf.dns-audit.com). Drop
-  your reports on the page. They are read on your computer and never uploaded.
+- **On the web:** [aprf.dns-audit.com](https://aprf.dns-audit.com). Drop or
+  paste your reports on the page.
 - **On the command line:** `pipx install aprf-reader`.
 
-**Status: 0.4, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
+**Status: 0.6, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
 
 ## What APRF is
 
@@ -51,6 +51,7 @@ aprf-reader APRF.mbox                     # a mail folder exported as mbox
 aprf-reader reports/ --markdown           # Markdown with daily tables
 aprf-reader reports/ --html summary.html  # and a page you can open or share
 aprf-reader reports/ --json               # the summary as data, for scripts
+aprf-reader --imap imap.gmail.com --user you@gmail.com --folder APRF
 ```
 
 In a terminal you get an overview: for each stream, each count's latest value,
@@ -59,12 +60,26 @@ the changes worth a look. Piped or written to a file with `--md`, the output
 is Markdown with the full daily tables.
 
 Point it at report files (`.json`, `.json.gz`), saved report emails (`.eml`),
-mailbox exports (`.mbox`), or folders of them. The easiest way to collect reports
+mailbox exports (`.mbox`), `.zip` files of any of those, or folders of them. The easiest way to collect reports
 is a mail filter that moves them into one folder, then an export of that folder:
 Gmail through [Google Takeout](https://takeout.google.com/) (pick Mail and the
 label), Thunderbird with the ImportExportTools NG add-on, or Apple Mail's
 Mailbox > Export Mailbox. Comcast's reports come from
-`aprf-noreply@alerts.xfinity.com` with subjects starting `APRF:`. Folders are searched recursively. Files it skips are listed
+`aprf-noreply@alerts.xfinity.com` with subjects starting `APRF:`. Folders are searched recursively.
+
+Or skip the export and read the folder straight from your mailbox over IMAP:
+
+```bash
+aprf-reader --imap imap.gmail.com --user you@gmail.com --folder APRF --since 2026-10-01
+```
+
+It opens the folder read-only and fetches with `BODY.PEEK`, so nothing is
+marked read, moved or deleted. It reads messages whose subject contains
+`APRF:`; `--all-mail` reads every message in the folder. The password comes
+from the `APRF_IMAP_PASSWORD` environment variable, or it asks. Gmail and
+Outlook.com need an app password, not your account password, and a Gmail
+label is read as a folder of the same name. The connection uses TLS on port
+993; give another port as `--imap host:port`. Files it skips are listed
 on stderr and at the end of the summary, each with a reason.
 
 Each stream (provider, signing domain, selector) gets a table with one row per
@@ -78,14 +93,16 @@ to mark a change in what a category counts.
 
 ## What it does
 
-- Read report files (`.json`, `.json.gz`, `.eml` and `.mbox`) or a folder of them, and skip
+- Read report files (`.json`, `.json.gz`, `.eml`, `.mbox` and `.zip`), a folder of them, or a
+  mailbox folder over IMAP, and skip
   duplicates and files it cannot read, with a reason for each.
 - Summarise placement per signing domain and selector: inbox, unwanted and
   recipient reactions, by day and as a trend.
 - Treat the counts as what the draft says they are: rounded into buckets, so
   they are a signal, not exact rates.
 - Write a Markdown summary and a single self-contained HTML page you can open
-  or share. Nothing is uploaded anywhere.
+  or share. The command line tool sends nothing anywhere; with `--imap` it
+  only talks to the mail server you name.
 - Follow the draft as it changes. The report format lives in one module
   (`src/aprf_reader/report.py`), so a new draft revision is a small update.
 - Check each report against the draft and skip, with the reason, any it
@@ -152,8 +169,8 @@ repeats a report already read, such as the same report as `.json` and as
 emails, forwarded emails, mailbox exports, duplicates and hostile text) and
 requires the same JSON, Markdown and HTML output, so the page and the command
 line cannot drift apart. `node web/build.js` inlines everything into one
-`web/dist/index.html` whose Content Security Policy blocks every network
-request, so it also works saved to disk. A merge to main that passes the
+`web/dist/index.html` whose Content Security Policy allows one connection, the
+usage note to dns-audit.com, and it also works saved to disk. A merge to main that passes the
 tests deploys it to Cloudflare.
 
 ## Related

{aprf_reader-0.4.1 → aprf_reader-0.6.0}/src/aprf_reader.egg-info/SOURCES.txt

@@ -3,6 +3,7 @@ README.md
 pyproject.toml
 src/aprf_reader/__init__.py
 src/aprf_reader/cli.py
+src/aprf_reader/imap.py
 src/aprf_reader/load.py
 src/aprf_reader/render.py
 src/aprf_reader/report.py
@@ -13,5 +14,6 @@ src/aprf_reader.egg-info/dependency_links.txt
 src/aprf_reader.egg-info/entry_points.txt
 src/aprf_reader.egg-info/requires.txt
 src/aprf_reader.egg-info/top_level.txt
+tests/test_imap.py
 tests/test_reader.py
 tests/test_web_parity.py

aprf_reader-0.6.0/tests/test_imap.py

@@ -0,0 +1,105 @@
+import gzip
+import imaplib
+import json
+from datetime import date
+
+import pytest
+
+from aprf_reader import cli, imap
+from aprf_reader.imap import ImapError, criteria, fetch
+from aprf_reader.load import load
+
+from test_reader import report, report_email
+
+
+class FakeImap:
+    """Answers the calls fetch() makes the way imaplib returns a server's replies."""
+
+    def __init__(self, messages, password="right", folders=("INBOX", "APRF")):
+        self.messages = messages  # {uid: raw bytes}
+        self.password, self.folders = password, folders
+        self.calls = []
+
+    def login(self, user, password):
+        if password != self.password:
+            raise imaplib.IMAP4.error("[AUTHENTICATIONFAILED] Invalid credentials")
+        return "OK", [b"logged in"]
+
+    def select(self, folder, readonly=False):
+        self.calls.append(("select", folder, readonly))
+        if folder.strip('"') not in self.folders:
+            return "NO", [b"[NONEXISTENT] Unknown Mailbox"]
+        return "OK", [str(len(self.messages)).encode()]
+
+    def uid(self, command, *args):
+        self.calls.append((command, *args))
+        if command == "SEARCH":
+            return "OK", [b" ".join(str(u).encode() for u in self.messages)]
+        out = []
+        for u in args[0].split(","):
+            body = self.messages[int(u)]
+            out += [(f"{u} (UID {u} RFC822.SIZE {len(body)} BODY[] {{{len(body)}}}".encode(), body), b")"]
+        return "OK", out
+
+    def logout(self):
+        self.calls.append(("logout",))
+        return "BYE", [b""]
+
+
+def mails(n=3):
+    out = {}
+    for d in range(n):
+        out[100 + d] = report_email(d, gzip.compress(json.dumps(report(d)).encode()), "gzip", f"{d}.json.gz").as_bytes()
+    return out
+
+
+def test_fetch_is_read_only_and_batched(monkeypatch):
+    monkeypatch.setattr(imap, "BATCH", 2)
+    fake = FakeImap(mails(5))
+    got = list(fetch("imap.example.com", "u", "right", "APRF", conn=fake))
+    assert [label for label, _ in got] == [f"imap://imap.example.com/APRF uid {u}" for u in range(100, 105)]
+    assert ("select", '"APRF"', True) in fake.calls  # EXAMINE, not SELECT
+    fetches = [c for c in fake.calls if c[0] == "FETCH"]
+    assert len(fetches) == 3 and all("BODY.PEEK[]" in c[2] for c in fetches)  # PEEK leaves \\Seen alone
+    assert fake.calls[-1] == ("logout",)
+
+
+def test_criteria():
+    assert criteria() == ["SUBJECT", '"APRF:"']
+    assert criteria(date(2026, 10, 1)) == ["SUBJECT", '"APRF:"', "SINCE", "01-Oct-2026"]
+    assert criteria(every=True) == ["ALL"]
+
+
+def test_login_and_folder_errors_are_plain():
+    with pytest.raises(ImapError, match="login failed: .*Invalid credentials"):
+        list(fetch("h", "u", "wrong", conn=FakeImap({})))
+    with pytest.raises(ImapError, match="opening folder Nope failed: .*Unknown Mailbox"):
+        list(fetch("h", "u", "right", "Nope", conn=FakeImap({})))
+
+
+def test_fetched_mail_read_like_an_mbox():
+    msgs = list(mails(2).items())
+    note = b"Subject: APRF: hello\r\n\r\nno attachment\r\n"
+    loaded = load([], None, [(f"m{u}", b) for u, b in msgs] + [("m9", note), ("m10", note)], "imap://h/APRF")
+    assert sorted(r.day.day for r in loaded.reports) == [1, 2]
+    assert loaded.skipped == [("imap://h/APRF", "2 messages with no report attached")]
+
+
+def test_cli_imap(monkeypatch, capsys):
+    fake = FakeImap(mails(2))
+    monkeypatch.setattr(imaplib, "IMAP4_SSL", lambda host, port, timeout: fake)
+    monkeypatch.setenv("APRF_IMAP_PASSWORD", "right")
+    assert cli.main(["--imap", "imap.example.com", "--user", "u", "--folder", "APRF", "--json"]) == 0
+    out, err = capsys.readouterr()
+    assert json.loads(out)["reports"] == 2
+    assert "read 2 messages from APRF on imap.example.com" in err
+
+
+def test_cli_imap_failure_and_usage(monkeypatch, capsys):
+    monkeypatch.setattr(imaplib, "IMAP4_SSL", lambda host, port, timeout: FakeImap({}))
+    monkeypatch.setenv("APRF_IMAP_PASSWORD", "wrong")
+    assert cli.main(["--imap", "h", "--user", "u"]) == 2
+    assert "login failed" in capsys.readouterr().err
+    for argv in ([], ["--imap", "h"], ["--imap", "h:x", "--user", "u"]):
+        with pytest.raises(SystemExit):
+            cli.main(argv)

{aprf_reader-0.4.1 → aprf_reader-0.6.0}/tests/test_reader.py

@@ -249,7 +249,7 @@ def test_stress_set_one_bad_report_does_not_cost_the_file():
     dup = {where.rsplit(" ", 1)[-1]: d for w, d, where in loaded.warnings if w.startswith("the same day came twice")}
     assert dup["64"] == f"{FIX}/stress_100.json report 63 kept, version 1 kept, version 2 dropped"
     assert dup["69"].endswith("report 68 kept, with different counts")  # bücher.example
-    assert "report 3 kept" in dup["70"]  # DNS-AUDIT.COM / SELECTOR1 is dns-audit.com / selector1
+    assert "report 3 kept" in dup["70"]  # SENDER.EXAMPLE / SELECTOR1 is sender.example / selector1
     assert not {"64", "69", "70"} & set(reasons)
     warned = {w for w, _, _ in loaded.warnings}
     assert "sdi_used is not N/A, N/F, or the sdi value as published in DNS (section 5.2.1)" in warned
@@ -259,13 +259,13 @@ def test_stress_set_one_bad_report_does_not_cost_the_file():
     assert reasons["79"] == "header.report_end is before report_start (section 5.1)"
     assert reasons["81"].startswith("header.report_start looks like milliseconds")
     ss = {(s.source, s.domain, s.selector): s for s in streams(loaded.reports)}
-    assert ("Big MBP Co", "dns-audit.com", "s11") in ss  # another provider, same day: its own stream
+    assert ("Big MBP Co", "sender.example", "s11") in ss  # another provider, same day: its own stream
     obj = report(0)
     obj["body"][0]["classification"]["unwanted"] = 100.0
     assert parse(json.dumps(obj).encode())[0].segments[0].classification["unwanted"] == 100
     for sel in ("sdi0", "sdi1", "sdi2", "sdi3", "sdi4"):  # n/a, "N/A ", "", null, N/F;
-        assert not ss[("Comcast/Xfinity", "dns-audit.com", sel)].notes, sel
-    blank = ss[("Comcast/Xfinity", "dns-audit.com", "s13")]
+        assert not ss[("Comcast/Xfinity", "sender.example", sel)].notes, sel
+    blank = ss[("Comcast/Xfinity", "sender.example", "s13")]
     assert "(blank)" in [r.segment for r in blank.rows]
-    assert not ss[("Comcast/Xfinity", "dns-audit.com", "s8")].columns
+    assert not ss[("Comcast/Xfinity", "sender.example", "s8")].columns
     assert "These reports carry no counts." in to_markdown(loaded)

{aprf_reader-0.4.1 → aprf_reader-0.6.0}/tests/test_web_parity.py

@@ -3,6 +3,7 @@
 import gzip
 import json
 import mailbox
+import re
 import shutil
 import subprocess
 from email.message import EmailMessage
@@ -22,16 +23,24 @@ NODE = shutil.which("node")
 pytestmark = pytest.mark.skipif(NODE is None, reason="node is not installed")
 
 
-def web(mode: str, paths: list[Path]) -> str:
-    return subprocess.run([NODE, str(ROOT / "web/test/run.js"), mode, *map(str, paths)], check=True,
-                          capture_output=True, text=True, env={"APRF_VERSION": __version__}).stdout
+def web(mode: str, paths: list, paste: bytes | None = None) -> str:
+    args = [*map(str, paths)] + (["-"] if paste is not None else [])
+    return subprocess.run([NODE, str(ROOT / "web/test/run.js"), mode, *args], check=True, input=paste or b"",
+                          capture_output=True, env={"APRF_VERSION": __version__}).stdout.decode()
+
+
+def _engine(doc: dict) -> dict:
+    """Python and browsers word JSON parse errors differently; compare only our part."""
+    cut = lambda r: r.split(": ", 1)[0] if r.startswith("not valid JSON") else r
+    return {**doc, "skipped": [{**k, "reason": cut(k["reason"])} for k in doc["skipped"]]}
 
 
 def same(paths: list[Path]) -> None:
     loaded = load(paths)
-    assert json.loads(web("--json", paths)) == to_dict(loaded)
-    assert web("--md", paths) == to_markdown(loaded)
-    assert web("--html", paths) == to_html(loaded)
+    assert _engine(json.loads(web("--json", paths))) == _engine(to_dict(loaded))
+    text = lambda t: re.sub(r"(not valid JSON, which section 5 requires): [^\n<]*", r"\1", t)
+    assert text(web("--md", paths)) == text(to_markdown(loaded))
+    assert text(web("--html", paths)) == text(to_html(loaded))
 
 
 def report(day: int, selector: str = "s1", inbox: int = 100) -> dict:
@@ -128,3 +137,46 @@ def test_section_8_checks_the_same(tmp_path):
     assert "attachment name stands in for a character a file name cannot hold, such as / or *" in by_file["safe.eml"]
     assert {"attachment name does not follow section 8", "subject does not follow section 8",
             "attachment Content-Type is not application/json, or application/gzip when compressed"} <= by_file["wrong.eml"]
+
+
+def test_zips_read_the_same(tmp_path):
+    import zipfile
+    raw = lambda day, **kw: json.dumps(report(day, **kw)).encode()
+    z = tmp_path / "reports.zip"
+    with zipfile.ZipFile(z, "w") as f:
+        f.writestr("b/20261002.json", raw(1), compress_type=zipfile.ZIP_DEFLATED)
+        f.writestr("a/20261001.json", raw(0), compress_type=zipfile.ZIP_STORED)
+        f.writestr("c/20261003.json.gz", gzip.compress(raw(2)))
+        f.writestr("d/20261004.eml", email(raw(3), "json", "x.json").as_bytes(), compress_type=zipfile.ZIP_DEFLATED)
+        f.writestr("__MACOSX/a/._20261001.json", b"\x00\x05\x16\x07 Finder junk")
+        f.writestr("e/inner.zip", b"PK nested")
+        f.writestr("e/box.mbox", b"From x\n\nhi\n")
+        f.writestr("f/huge.json", b" " * (11 * 1024 * 1024), compress_type=zipfile.ZIP_DEFLATED)
+        f.writestr("g/notes.txt", b"not a report suffix")
+        f.writestr("h/broken.json", b"{oops", compress_type=zipfile.ZIP_DEFLATED)
+        f.writestr("i/crc.json", raw(5), compress_type=zipfile.ZIP_STORED)
+    # Flip one byte of the stored crc.json so its CRC no longer matches
+    data = bytearray(z.read_bytes())
+    at = data.index(b'"dkim_selector"', data.index(b"i/crc.json") + 10)
+    data[at + 2] ^= 0x01
+    z.write_bytes(bytes(data))
+    (tmp_path / "fake.zip").write_bytes(b"not a zip at all")
+    same([z, tmp_path / "fake.zip"])
+    loaded = load([z, tmp_path / "fake.zip"])
+    assert sorted(r.day.day for r in loaded.reports) == [1, 2, 3, 4]
+    reasons = {w.rsplit("/", 1)[-1]: r for w, r in loaded.skipped}
+    assert reasons["crc.json"] == "could not be unpacked from the zip"
+    assert reasons["inner.zip"] == reasons["box.mbox"] == "a zip or mailbox inside a zip is not read; unzip it first"
+    assert reasons["huge.json"] == "larger than 10 MB" and reasons["fake.zip"] == "not a readable zip file"
+    assert "broken.json" in reasons and not any("MACOSX" in w for w, _ in loaded.skipped)
+
+
+@pytest.mark.parametrize("kind", ["json", "email", "gzip", "junk"])
+def test_pastes_read_the_same(kind):
+    raw = json.dumps(report(0)).encode()
+    paste = {"json": b"  \n" + raw, "email": email(gzip.compress(raw), "gzip", "r.json.gz").as_bytes(),
+             "gzip": gzip.compress(raw), "junk": b"hello, not a report"}[kind]
+    loaded = load([], paste)
+    assert _engine(json.loads(web("--json", [], paste))) == _engine(to_dict(loaded))
+    assert (len(loaded.reports), loaded.skipped[0][0] if loaded.skipped else None) == (
+        {"json": (1, None), "email": (1, None), "gzip": (1, None), "junk": (0, "pasted.eml")}[kind])

aprf_reader-0.4.1/src/aprf_reader/cli.py

@@ -1,55 +0,0 @@
-"""Command line entry point: aprf-reader <paths...>"""
-
-from __future__ import annotations
-
-import argparse
-import json
-import os
-import shutil
-import sys
-from pathlib import Path
-
-from . import __version__
-from .load import load
-from .render import to_html, to_markdown, to_terminal
-from .summary import to_dict
-
-
-def main(argv: list[str] | None = None) -> int:
-    p = argparse.ArgumentParser(prog="aprf-reader", description="Summarise APRF placement reports.")
-    p.add_argument("paths", nargs="+", type=Path, help="report files (.json, .json.gz, .eml, .mbox) or folders")
-    p.add_argument("--md", type=Path, help="write the summary here instead of printing it")
-    p.add_argument("--html", type=Path, help="also write a self-contained HTML page here")
-    p.add_argument("--markdown", action="store_true", help="print Markdown with daily tables, even in a terminal")
-    p.add_argument("--json", action="store_true", help="print the summary as JSON")
-    p.add_argument("--version", action="version", version=f"aprf-reader {__version__}")
-    args = p.parse_args(argv)
-
-    for path in args.paths:
-        if not path.exists():
-            p.error(f"no such file or folder: {path}")
-    loaded = load(args.paths)
-    for where, reason in loaded.skipped:
-        print(f"skipped {where}: {reason}", file=sys.stderr)
-    if not loaded.reports:
-        print("no APRF reports found", file=sys.stderr)
-        return 1
-
-    if args.json:
-        text = json.dumps(to_dict(loaded), indent=2) + "\n"
-    elif args.md or args.markdown or not sys.stdout.isatty():
-        text = to_markdown(loaded)  # files and pipes get Markdown, which other tools read
-    else:
-        color = "NO_COLOR" not in os.environ and os.environ.get("TERM") != "dumb"
-        text = to_terminal(loaded, color=color, width=shutil.get_terminal_size().columns)
-    if args.md:
-        args.md.write_text(text, encoding="utf-8")
-    else:
-        sys.stdout.write(text)
-    if args.html:
-        args.html.write_text(to_html(loaded), encoding="utf-8")
-    return 0
-
-
-if __name__ == "__main__":
-    sys.exit(main())