aprf-reader 0.3.1__tar.gz → 0.4.1__tar.gz

{aprf_reader-0.3.1/src/aprf_reader.egg-info → aprf_reader-0.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aprf-reader
-Version: 0.3.1
+Version: 0.4.1
 Summary: Summarise APRF (Aggregate Performance Reporting) inbox placement reports.
 License: MIT
 Requires-Python: >=3.11
@@ -24,7 +24,7 @@ Two ways to use it, with the same results:
   your reports on the page. They are read on your computer and never uploaded.
 - **On the command line:** `pipx install aprf-reader`.
 
-**Status: 0.3, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
+**Status: 0.4, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
 
 ## What APRF is
 
@@ -88,8 +88,62 @@ to mark a change in what a category counts.
   or share. Nothing is uploaded anywhere.
 - Follow the draft as it changes. The report format lives in one module
   (`src/aprf_reader/report.py`), so a new draft revision is a small update.
-- When the same day arrives twice, keep the first copy and say so. The draft
-  has not yet decided whether a resend replaces the original.
+- Check each report against the draft and skip, with the reason, any it
+  cannot trust. One bad report in a file does not cost the rest of the file.
+
+## Checking against the draft
+
+Every report is checked against draft-ietf-mailmaint-aprf-00, and every
+message names the section, so a list of them can go straight into a review of
+the draft. On the web page, Copy for a draft review copies them as text.
+
+**Skipped**, because the report cannot be trusted:
+
+| Problem | Section |
+|---|---|
+| Not valid JSON | 5 |
+| A required header field missing or of the wrong type; times not in epoch seconds | 5.2.1 |
+| Not one UTC day from 00:00:00 to 23:59:59, or ending before it starts | 5.1 |
+| Body not a list of segments; a segment not an object | 5.2.2 |
+| A placement or engagement count that is not a whole number | 6, 7 |
+| A saved email with no report attached | 8 |
+
+**Warnings**, where the report is still read:
+
+| Warning | Section |
+|---|---|
+| `sdi_used` is not N/A, N/F, or the `sdi` value as published in DNS, or is missing | 5.2.1 |
+| Attachment name or subject not `<yyyymmdd>_<DKIM domain>_<DKIM selector>_<source>` | 8 |
+| Message not `multipart/report`; attachment not `application/json` (or `application/gzip` for `.json.gz`) | 8, 8.1 |
+| One attachment holding several reports | 8 |
+| A placement key other than the four section 6 names, or an engagement key other than positive, neutral and negative | 6, 5.2.2 |
+| A name or subject that writes `/` or `*` some other way, as a file name must | 8 is silent |
+| The same day twice from one provider | left open, note in 8.1 |
+
+Comcast's beta sends `sdi_used` as ` - N/A - Beta - Not in use` and a source
+of `Comcast/Xfinity`, so its reports carry the `sdi_used` warning, and a
+correct file name for them cannot follow section 8 exactly. A file that
+repeats a report already read, such as the same report as `.json` and as
+`.eml`, is counted once and summed up in one line.
+
+## How it decides
+
+- **The same day twice, from the same provider:** it keeps the first copy it
+  reads and skips the other, saying whether the skipped copy had a different
+  `version` or different counts. The draft leaves resends open (a note in
+  section 8.1 asks whether a later report should overwrite or be discarded),
+  and `version` is a free string with no order, so there is no defined way
+  to pick the newer one. Files are read in name order, so the first copy is
+  the first by file name, then by position in the file.
+- **The same day from two providers:** both are kept, as separate streams.
+- **Names:** signing domains and selectors compare without case, and an
+  international domain matches its `xn--` form, so `bücher.example` and
+  `xn--bcher-kva.example` are one stream.
+- **The reporting window:** section 5.1 says a report MUST cover one UTC day,
+  from 00:00:00 to 23:59:59. A report that covers more, or is shifted off the
+  day boundary, is skipped with a reason that names section 5.1.
+- **Counts:** whole numbers only. `100.0` reads as 100; `"100"` and negative
+  numbers are refused.
 
 ## How the web page is built
 

{aprf_reader-0.3.1 → aprf_reader-0.4.1}/README.md

@@ -11,7 +11,7 @@ Two ways to use it, with the same results:
   your reports on the page. They are read on your computer and never uploaded.
 - **On the command line:** `pipx install aprf-reader`.
 
-**Status: 0.3, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
+**Status: 0.4, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
 
 ## What APRF is
 
@@ -75,8 +75,62 @@ to mark a change in what a category counts.
   or share. Nothing is uploaded anywhere.
 - Follow the draft as it changes. The report format lives in one module
   (`src/aprf_reader/report.py`), so a new draft revision is a small update.
-- When the same day arrives twice, keep the first copy and say so. The draft
-  has not yet decided whether a resend replaces the original.
+- Check each report against the draft and skip, with the reason, any it
+  cannot trust. One bad report in a file does not cost the rest of the file.
+
+## Checking against the draft
+
+Every report is checked against draft-ietf-mailmaint-aprf-00, and every
+message names the section, so a list of them can go straight into a review of
+the draft. On the web page, Copy for a draft review copies them as text.
+
+**Skipped**, because the report cannot be trusted:
+
+| Problem | Section |
+|---|---|
+| Not valid JSON | 5 |
+| A required header field missing or of the wrong type; times not in epoch seconds | 5.2.1 |
+| Not one UTC day from 00:00:00 to 23:59:59, or ending before it starts | 5.1 |
+| Body not a list of segments; a segment not an object | 5.2.2 |
+| A placement or engagement count that is not a whole number | 6, 7 |
+| A saved email with no report attached | 8 |
+
+**Warnings**, where the report is still read:
+
+| Warning | Section |
+|---|---|
+| `sdi_used` is not N/A, N/F, or the `sdi` value as published in DNS, or is missing | 5.2.1 |
+| Attachment name or subject not `<yyyymmdd>_<DKIM domain>_<DKIM selector>_<source>` | 8 |
+| Message not `multipart/report`; attachment not `application/json` (or `application/gzip` for `.json.gz`) | 8, 8.1 |
+| One attachment holding several reports | 8 |
+| A placement key other than the four section 6 names, or an engagement key other than positive, neutral and negative | 6, 5.2.2 |
+| A name or subject that writes `/` or `*` some other way, as a file name must | 8 is silent |
+| The same day twice from one provider | left open, note in 8.1 |
+
+Comcast's beta sends `sdi_used` as ` - N/A - Beta - Not in use` and a source
+of `Comcast/Xfinity`, so its reports carry the `sdi_used` warning, and a
+correct file name for them cannot follow section 8 exactly. A file that
+repeats a report already read, such as the same report as `.json` and as
+`.eml`, is counted once and summed up in one line.
+
+## How it decides
+
+- **The same day twice, from the same provider:** it keeps the first copy it
+  reads and skips the other, saying whether the skipped copy had a different
+  `version` or different counts. The draft leaves resends open (a note in
+  section 8.1 asks whether a later report should overwrite or be discarded),
+  and `version` is a free string with no order, so there is no defined way
+  to pick the newer one. Files are read in name order, so the first copy is
+  the first by file name, then by position in the file.
+- **The same day from two providers:** both are kept, as separate streams.
+- **Names:** signing domains and selectors compare without case, and an
+  international domain matches its `xn--` form, so `bücher.example` and
+  `xn--bcher-kva.example` are one stream.
+- **The reporting window:** section 5.1 says a report MUST cover one UTC day,
+  from 00:00:00 to 23:59:59. A report that covers more, or is shifted off the
+  day boundary, is skipped with a reason that names section 5.1.
+- **Counts:** whole numbers only. `100.0` reads as 100; `"100"` and negative
+  numbers are refused.
 
 ## How the web page is built
 

{aprf_reader-0.3.1 → aprf_reader-0.4.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "aprf-reader"
-version = "0.3.1"
+version = "0.4.1"
 description = "Summarise APRF (Aggregate Performance Reporting) inbox placement reports."
 readme = "README.md"
 license = { text = "MIT" }

{aprf_reader-0.3.1 → aprf_reader-0.4.1}/src/aprf_reader/__init__.py

@@ -1,3 +1,3 @@
 """Read APRF aggregate performance reports."""
 
-__version__ = "0.3.1"
+__version__ = "0.4.1"

{aprf_reader-0.3.1 → aprf_reader-0.4.1}/src/aprf_reader/cli.py

@@ -36,7 +36,7 @@ def main(argv: list[str] | None = None) -> int:
         return 1
 
     if args.json:
-        text = json.dumps(to_dict(loaded.reports, loaded.skipped), indent=2) + "\n"
+        text = json.dumps(to_dict(loaded), indent=2) + "\n"
     elif args.md or args.markdown or not sys.stdout.isatty():
         text = to_markdown(loaded)  # files and pipes get Markdown, which other tools read
     else:

aprf_reader-0.4.1/src/aprf_reader/load.py

@@ -0,0 +1,241 @@
+"""Find report files, open them, and drop duplicates."""
+
+from __future__ import annotations
+
+import gzip
+import hashlib
+import io
+import mailbox
+import re
+import zlib
+from collections.abc import Iterable, Iterator
+from dataclasses import dataclass, field
+from email import policy
+from email.message import Message
+from email.parser import BytesParser
+from pathlib import Path
+
+from .report import Report, ReportError, parse_items
+
+SUFFIXES = (".json", ".json.gz", ".eml", ".mbox")
+# A daily report is a few KB; this stops a stray huge file or gzip bomb
+MAX_BYTES = 10 * 1024 * 1024
+
+
+@dataclass
+class Loaded:
+    reports: list[Report] = field(default_factory=list)
+    skipped: list[tuple[str, str]] = field(default_factory=list)  # (where, reason)
+    # (warning, detail, where): read, but departs from the draft or meets a case it leaves open
+    warnings: list[tuple[str, str, str]] = field(default_factory=list)
+    # Files whose report bytes were already read, often the same report as .json and .eml
+    copies: int = 0
+
+
+@dataclass(frozen=True)
+class Mail:
+    """How a report arrived by email, for the section 8 checks."""
+    name: str  # attachment file name as sent
+    ctype: str  # attachment Content-Type
+    gz: bool
+    subject: str
+    msg_ctype: str  # Content-Type of the message that carries it
+
+DUPLICATE_DAY = ("the same day came twice from one provider; the first copy read was kept. "
+                 "The draft leaves this open (a note in section 8.1)")
+
+
+def _files(paths: Iterable[Path]) -> Iterator[Path]:
+    for p in paths:
+        if p.is_dir():
+            yield from sorted(f for f in p.rglob("*") if f.is_file() and (f.name.lower().endswith(SUFFIXES) or _is_mbox(f)))
+        else:
+            yield p
+
+
+def _is_mbox(path: Path) -> bool:
+    # Apple Mail exports a folder "Name.mbox" holding a file named just "mbox"
+    return path.name.lower() == "mbox" or path.name.lower().endswith(".mbox")
+
+
+def _gunzip(data: bytes) -> bytes:
+    with gzip.GzipFile(fileobj=io.BytesIO(data)) as g:
+        out = g.read(MAX_BYTES + 1)
+    if len(out) > MAX_BYTES:
+        raise ReportError("larger than 10 MB unpacked")
+    return out
+
+
+def _leaves(m: Message, owner: Message) -> Iterator[tuple[Message, Message]]:
+    """Each non-multipart part with the email it belongs to; an attached email owns its own parts."""
+    if m.get_content_type() == "message/rfc822":
+        for inner in m.get_payload():
+            yield from _leaves(inner, inner)
+    elif m.is_multipart():
+        for p in m.get_payload():
+            yield from _leaves(p, owner)
+    else:
+        yield m, owner
+
+
+def _attachments(msg: Message, label: str) -> Iterator[tuple[str, bytes, Mail]]:
+    """Yield (label, report JSON bytes, how it arrived) for each report attached to one email."""
+    for part, owner in _leaves(msg, msg):
+        name = part.get_filename() or ""
+        fname = name.lower()
+        ctype = part.get_content_type()
+        data = part.get_payload(decode=True) or b""
+        where = f"{label} ({fname or ctype})"
+        if len(data) > MAX_BYTES:
+            raise ReportError(f"{where} is larger than 10 MB")
+        mail = lambda gz: Mail(name, ctype, gz, " ".join(str(owner.get("subject", "")).split()), owner.get_content_type())
+        if fname.endswith(".gz") or ctype in ("application/gzip", "application/x-gzip") or data[:2] == b"\x1f\x8b":
+            yield where, _gunzip(data), mail(True)
+        # By name or type, or an attachment of unhelpful type that is a JSON array or object
+        elif fname.endswith(".json") or ctype == "application/json" or (
+                part.get_content_disposition() == "attachment" and data.lstrip()[:1] in (b"[", b"{")):
+            yield where, data, mail(False)
+
+
+def _payloads(path: Path) -> Iterator[tuple[str, bytes, Mail | None]]:
+    """Yield (label, report JSON bytes, how it arrived) for each report in a file."""
+    data = path.read_bytes()
+    if len(data) > MAX_BYTES:
+        raise ReportError("larger than 10 MB")
+    if path.name.lower().endswith(".eml"):
+        found = list(_attachments(BytesParser(policy=policy.default).parsebytes(data), str(path)))
+        if not found:
+            raise ReportError("no report attachment, which section 8 calls for")
+        yield from found
+    elif data[:2] == b"\x1f\x8b":
+        yield str(path), _gunzip(data), None
+    else:
+        yield str(path), data, None
+
+
+def _mbox(path: Path, out: Loaded) -> Iterator[tuple[str, bytes, Mail | None]]:
+    """Yield reports from a mailbox export; one skip line for all mail with none attached."""
+    box = mailbox.mbox(path, factory=lambda f: BytesParser(policy=policy.default).parse(f), create=False)
+    without = 0
+    try:
+        for i, msg in enumerate(box, 1):
+            label = f"{path} message {i}"
+            try:
+                found = list(_attachments(msg, label))
+            # OSError covers gzip.BadGzipFile; one bad message must not end the mailbox
+            except (OSError, EOFError, zlib.error, ReportError) as e:
+                out.skipped.append((label, str(e)))
+                continue
+            without += not found
+            yield from found
+    finally:
+        box.close()
+    if without:
+        out.skipped.append((str(path), f"{without} message{'s' if without != 1 else ''} with no report attached"))
+
+
+def _differs(kept: Report, dropped: Report) -> str:
+    if kept.version != dropped.version:
+        return f", version {kept.version or 'none'} kept, version {dropped.version or 'none'} dropped"
+    return ", with different counts" if kept.segments != dropped.segments else ", same counts"
+
+
+def _name(r: Report) -> str:
+    # Section 8: the ending date, then domain, selector and source with spaces removed
+    return f"{r.day:%Y%m%d}_{r.dkim_domain}_{r.dkim_selector}_{r.source.replace(' ', '')}"
+
+
+# Characters a file name cannot hold on common systems; "Comcast/Xfinity" and a "*" roll-up
+# selector both carry one, and section 8 only says to remove spaces
+UNSAFE = '/\\:*?"<>|'
+NAME_UNSAFE = ("attachment name stands in for a character a file name cannot hold, such as / or *; "
+               "section 8 does not say how to write those")
+SUBJECT_UNSAFE = ("subject stands in for / or * as a file name must, though section 8 gives the source "
+                  "and selector as they are")
+
+
+def _loose(expected: str, actual: str) -> bool:
+    """Equal, ignoring case, if each file-name-unsafe character in expected may be any character."""
+    pattern = "".join("." if c in UNSAFE else re.escape(c) for c in expected.lower())
+    return re.fullmatch(pattern, actual.lower()) is not None
+
+
+def _mail_warnings(mail: Mail, items: list) -> list[tuple[str, str]]:
+    out = []
+    if mail.msg_ctype != "multipart/report":
+        out.append(("message Content-Type is not multipart/report (section 8)", mail.msg_ctype))
+    want = "application/gzip" if mail.gz else "application/json"
+    if mail.ctype != want:
+        out.append(("attachment Content-Type is not application/json, or application/gzip when "
+                    "compressed (sections 8 and 8.1)", mail.ctype))
+    if len(items) > 1:
+        out.append(("one attachment holds several reports; the section 8 name and subject fit one", f"{len(items)} reports"))
+    elif isinstance(items[0], Report):
+        base = _name(items[0])
+        file = base + (".json.gz" if mail.gz else ".json")
+        if mail.name.lower() == file.lower():
+            pass
+        elif _loose(file, mail.name):
+            out.append((NAME_UNSAFE, f"{mail.name}; section 8 gives {file}"))
+        else:
+            out.append(("attachment name does not follow section 8: <yyyymmdd>_<DKIM domain>_<DKIM selector>_<source>.json",
+                        f"{mail.name or '(none)'}; expected {file}"))
+        if mail.subject.lower() == f"APRF: {base}".lower():
+            pass
+        elif _loose(f"APRF: {base}", mail.subject):
+            out.append((SUBJECT_UNSAFE, f"{mail.subject}; section 8 gives APRF: {base}"))
+        else:
+            out.append(("subject does not follow section 8: APRF: <yyyymmdd>_<DKIM domain>_<DKIM selector>_<source>",
+                        f"{mail.subject or '(none)'}; expected APRF: {base}"))
+    return out
+
+
+def load(paths: Iterable[Path]) -> Loaded:
+    out = Loaded()
+    seen_bytes: set[str] = set()
+    seen_keys: dict[tuple, tuple[str, Report]] = {}
+
+    parsed: dict[str, list | ReportError] = {}
+
+    def add(where: str, data: bytes, mail: Mail | None) -> None:
+        digest = hashlib.sha256(data).hexdigest()
+        copy = digest in parsed
+        if not copy:
+            try:
+                parsed[digest] = parse_items(data)
+            except ReportError as e:
+                parsed[digest] = e
+        items = parsed[digest]
+        # An email is checked even when its report was already read from another file
+        if mail is not None and not isinstance(items, ReportError):
+            out.warnings += [(w, d, where) for w, d in _mail_warnings(mail, items)]
+        if copy:
+            out.copies += 1
+            return
+        if isinstance(items, ReportError):
+            out.skipped.append((where, str(items)))
+            return
+        for i, r in enumerate(items, 1):
+            label = f"{where} report {i}" if len(items) > 1 else where
+            if isinstance(r, ReportError):
+                out.skipped.append((label, str(r)))
+                continue
+            out.warnings += [(w, d, label) for w, d in r.warnings]
+            # The draft leaves open whether a resend replaces the first (an open note in
+            # section 8.1), and its version is a free string with no order; keep the first
+            if r.key in seen_keys:
+                first_label, first = seen_keys[r.key]
+                out.warnings.append((DUPLICATE_DAY, f"{first_label} kept{_differs(first, r)}", label))
+                continue
+            seen_keys[r.key] = (label, r)
+            out.reports.append(r)
+
+    for path in _files(paths):
+        try:
+            # An mbox can be large, so it is read one message at a time
+            payloads = _mbox(path, out) if _is_mbox(path) else list(_payloads(path))
+            for where, data, mail in payloads:
+                add(where, data, mail)
+        except (OSError, EOFError, zlib.error, ReportError) as e:
+            out.skipped.append((str(path), str(e)))
+    return out

{aprf_reader-0.3.1 → aprf_reader-0.4.1}/src/aprf_reader/render.py

@@ -9,7 +9,14 @@ from datetime import date
 from . import __version__
 from .load import Loaded
 from .report import DRAFT
-from .summary import Row, Stream, streams
+from .summary import Row, Stream, copies_line, grouped, streams
+
+WARNINGS_NOTE = "Read anyway. Each names the draft section it departs from, or the case the draft leaves open."
+
+
+def _warning(w: str, times: int, where: str, detail: str) -> str:
+    return f"{w}. {_plural(times, 'time')}, first in {where}" + (f": {detail}" if detail else "")
+
 
 NO_COUNTS = "These reports carry no counts."
 
@@ -67,8 +74,12 @@ def to_markdown(loaded: Loaded) -> str:
             lines += ["Changes worth a look:", ""] + [f"- {_md(c)}" for c in s.changes] + [""]
         if s.notes:
             lines += [f"- {_md(n)}" for n in s.notes] + [""]
+    if loaded.warnings:
+        lines += ["## Warnings", "", WARNINGS_NOTE, ""] + [f"- {_md(_warning(*g))}" for g in grouped(loaded.warnings)] + [""]
     if loaded.skipped:
         lines += ["## Skipped", ""] + [f"- {_md(w)}: {_md(r)}" for w, r in loaded.skipped] + [""]
+    if loaded.copies:
+        lines += [copies_line(loaded.copies), ""]
     lines.append(f"Read by aprf-reader {__version__} against {DRAFT}.")
     return "\n".join(lines) + "\n"
 
@@ -134,8 +145,13 @@ def to_html(loaded: Loaded) -> str:
             parts.append("<p>Changes worth a look:</p><ul>" + "".join(f"<li>{e(c)}" for c in s.changes) + "</ul>")
         if s.notes:
             parts.append("<ul>" + "".join(f"<li>{e(n)}" for n in s.notes) + "</ul>")
+    if loaded.warnings:
+        parts.append(f"<h2>Warnings</h2><p class=muted>{WARNINGS_NOTE}</p><ul>"
+                     + "".join(f"<li>{e(_warning(*g))}" for g in grouped(loaded.warnings)) + "</ul>")
     if loaded.skipped:
         parts.append("<h2>Skipped</h2><ul>" + "".join(f"<li>{e(w)}: {e(r)}" for w, r in loaded.skipped) + "</ul>")
+    if loaded.copies:
+        parts.append(f"<p class=muted>{e(copies_line(loaded.copies))}</p>")
     parts.append(f"<p class=muted>Read by aprf-reader {e(__version__)} against {DRAFT}. Nothing left this computer.</p>")
     parts.append("</body></html>")
     return "\n".join(parts) + "\n"
@@ -232,6 +248,14 @@ def to_terminal(loaded: Loaded, color: bool = True, width: int = 100) -> str:
         for n in s.notes:
             out += _wrap(n, width - 4, "  · ")
         out.append("")
+    if loaded.warnings:
+        out.append(st.warn(f"Warnings {len(grouped(loaded.warnings))}"))
+        out += [st.dim(line) for line in _wrap(WARNINGS_NOTE, width - 2, "  ")]
+        for g in grouped(loaded.warnings):
+            out += _wrap(_warning(*g), width - 2, "  ")
+        out.append("")
+    if loaded.copies:
+        out += [st.dim(line) for line in _wrap(copies_line(loaded.copies), width)] + [""]
     if loaded.skipped:
         out.append(st.warn(f"Skipped {len(loaded.skipped)}"))
         for w, r in loaded.skipped:

{aprf_reader-0.3.1 → aprf_reader-0.4.1}/src/aprf_reader/report.py

@@ -7,6 +7,7 @@ edit to this file.
 from __future__ import annotations
 
 import json
+import re
 from dataclasses import dataclass, field
 from datetime import UTC, date, datetime
 
@@ -25,6 +26,14 @@ def uses_sdi(value: str) -> bool:
     return first not in NO_SDI
 
 
+DAY = 86400
+ONE_DAY = "not one UTC day from 00:00:00 to 23:59:59, which draft section 5.1 requires"
+# Placement names section 6 gives as examples; engagement names section 5.2.2 lists
+NAMED_CLASSIFICATION = ("inbox", "unwanted", "forwarded", "promotional")
+# sdi as published in DNS (section 4): a header field name, a comma, one separator character
+SDI_FORMAT = re.compile(r"[!-9;-~]+,\S")
+
+
 class ReportError(ValueError):
     """The data is not an APRF report this module can read."""
 
@@ -50,6 +59,8 @@ class Report:
     contact_info: str
     extra_info: str
     segments: tuple[Segment, ...]
+    # (warning, detail) where the report departs from the draft but can still be read
+    warnings: tuple[tuple[str, str], ...] = ()
 
     @property
     def day(self) -> date:
@@ -95,23 +106,23 @@ def _int(v: object) -> int | None:
     return v if isinstance(v, int) else None
 
 
-def _counts(value: object, where: str) -> dict[str, int]:
+def _counts(value: object, where: str, section: str) -> dict[str, int]:
     if value is None:
         return {}
     if not isinstance(value, dict):
-        raise ReportError(f"{where} is not an object")
+        raise ReportError(f"{where} is not an object ({section})")
     out = {}
     for name, n in value.items():
         k = _int(n)
         if k is None or k < 0 or k > MAX_COUNT:
-            raise ReportError(f"{where}.{name} is not a count: {json.dumps(n, ensure_ascii=False)}")
+            raise ReportError(f"{where}.{name} is not a count ({section}): {json.dumps(n, ensure_ascii=False, separators=(',', ':'))}")
         out[str(name)] = k
     return out
 
 
 def _segment(item: object, i: int) -> Segment:
     if not isinstance(item, dict):
-        raise ReportError(f"body[{i}] is not an object")
+        raise ReportError(f"body[{i}] is not an object (section 5.2.2)")
     seg = item.get("segment")
     # Section 5.2.2 shows a string, the samples a list
     if seg is None:
@@ -121,11 +132,11 @@ def _segment(item: object, i: int) -> Segment:
     elif isinstance(seg, list) and all(isinstance(s, str) for s in seg):
         path = tuple(seg)
     else:
-        raise ReportError(f"body[{i}].segment is not a string or list of strings")
+        raise ReportError(f"body[{i}].segment is not a string or list of strings (section 5.2.2)")
     return Segment(
         path,
-        _counts(item.get("classification"), f"body[{i}].classification"),
-        _counts(item.get("engagement"), f"body[{i}].engagement"),
+        _counts(item.get("classification"), f"body[{i}].classification", "section 6"),
+        _counts(item.get("engagement"), f"body[{i}].engagement", "section 7"),
     )
 
 
@@ -135,27 +146,30 @@ def _text(value: object) -> str:
 
 def _report(obj: object) -> Report:
     if not isinstance(obj, dict) or not isinstance(obj.get("header"), dict):
-        raise ReportError("no header object")
+        raise ReportError("no header object (section 5.2.1)")
     h = obj["header"]
     for name in ("dkim_domain", "dkim_selector"):
         if not isinstance(h.get(name), str) or not h[name]:
-            raise ReportError(f"header.{name} is missing")
+            raise ReportError(f"header.{name} is missing (section 5.2.1)")
     start, end = _int(h.get("report_start")), _int(h.get("report_end"))
     for name, v in (("report_start", start), ("report_end", end)):
         if v is None:
-            raise ReportError(f"header.{name} is not an epoch time")
+            raise ReportError(f"header.{name} is not an epoch time (section 5.2.1)")
         # Seconds past 1e11 are beyond the year 5138; up to 1e14 that is milliseconds
         if 10**11 <= abs(v) < 10**14:
-            raise ReportError(f"header.{name} looks like milliseconds, not seconds: {v}")
+            raise ReportError(f"header.{name} looks like milliseconds, not epoch seconds (section 5.2.1): {v}")
     if end < start:
-        raise ReportError("header.report_end is before report_start")
+        raise ReportError("header.report_end is before report_start (section 5.1)")
     try:
         _day(end)
     except (OverflowError, OSError, ValueError):
-        raise ReportError(f"header.report_end is out of range: {end}") from None
+        raise ReportError(f"header.report_end is out of range (section 5.2.1): {end}") from None
+    if start % DAY or end != start + DAY - 1:
+        raise ReportError(ONE_DAY)
     body = obj.get("body", [])
     if not isinstance(body, list):
-        raise ReportError("body is not a list")
+        raise ReportError("body is not a list of segments (section 5.2.2)")
+    segments = tuple(_segment(s, i) for i, s in enumerate(body))
     return Report(
         source=_text(h.get("source")) or "unknown",
         dkim_domain=h["dkim_domain"],
@@ -167,17 +181,44 @@ def _report(obj: object) -> Report:
         sdi_used=_text(h.get("sdi_used")).strip(" -"),
         contact_info=_text(h.get("contact_info")),
         extra_info=_text(h.get("extra_info")),
-        segments=tuple(_segment(s, i) for i, s in enumerate(body)),
+        segments=segments,
+        warnings=_warnings(h, segments),
     )
 
 
+def _warnings(h: dict, segments: tuple[Segment, ...]) -> tuple[tuple[str, str], ...]:
+    out = []
+    if "sdi_used" not in h:
+        out.append(("no sdi_used field; section 5.2.1 requires N/A when no SDI is used", ""))
+    else:
+        raw = h["sdi_used"]
+        if not (raw in ("N/A", "N/F") or (isinstance(raw, str) and SDI_FORMAT.fullmatch(raw))):
+            out.append(("sdi_used is not N/A, N/F, or the sdi value as published in DNS (section 5.2.1)",
+                        json.dumps(raw, ensure_ascii=False, separators=(',', ':'))))
+    keys = lambda attr: sorted({k for s in segments for k in getattr(s, attr)})
+    for k in keys("classification"):
+        if k not in NAMED_CLASSIFICATION:
+            out.append((f"classification key {json.dumps(k, ensure_ascii=False)} is not one section 6 names "
+                        "(inbox, unwanted, forwarded, promotional); read as given", ""))
+    for k in keys("engagement"):
+        if k not in ENGAGEMENT_NAMES:
+            out.append((f"engagement key {json.dumps(k, ensure_ascii=False)} is not positive, neutral or "
+                        "negative (section 5.2.2); read as given", ""))
+    return tuple(out)
+
+
+def _no_constant(name: str) -> None:
+    # Python's json takes NaN and Infinity; RFC 8259 and browsers do not
+    raise json.JSONDecodeError(f"{name} is not valid JSON", name, 0)
+
+
 def parse_items(data: bytes) -> list[Report | ReportError]:
     """Each report in a document, or the reason it cannot be read, so one bad
     report does not cost the rest of the file."""
     try:
-        doc = json.loads(data)
+        doc = json.loads(data, parse_constant=_no_constant)
     except (UnicodeDecodeError, json.JSONDecodeError) as e:
-        raise ReportError(f"not JSON: {e}") from None
+        raise ReportError(f"not valid JSON, which section 5 requires: {e}") from None
     except RecursionError:
         raise ReportError("JSON nested too deeply") from None
     items = doc if isinstance(doc, list) else [doc]

{aprf_reader-0.3.1 → aprf_reader-0.4.1}/src/aprf_reader/summary.py

@@ -119,8 +119,25 @@ def streams(reports: list[Report]) -> list[Stream]:
     return [Stream(src, dom, sel, days) for (src, dom, sel), days in sorted(groups.items())]
 
 
-def to_dict(reports: list[Report], skipped: list[tuple[str, str]]) -> dict:
+def grouped(warnings: list[tuple[str, str, str]]) -> list[tuple[str, int, str, str]]:
+    """(warning, times, first where, first detail), in the order first seen."""
+    out: dict[str, list] = {}
+    for w, d, where in warnings:
+        if w in out:
+            out[w][1] += 1
+        else:
+            out[w] = [w, 1, where, d]
+    return [tuple(v) for v in out.values()]
+
+
+def copies_line(n: int) -> str:
+    return (f"{n} file{'s' if n != 1 else ''} repeated a report already read, such as the same report "
+            "saved as .json and as .eml, and counted once.")
+
+
+def to_dict(loaded) -> dict:
     """The summary as plain data, for --json and for checking the web page against."""
+    reports, skipped = loaded.reports, loaded.skipped
     return {
         "reports": len(reports),
         "streams": [{
@@ -133,4 +150,6 @@ def to_dict(reports: list[Report], skipped: list[tuple[str, str]]) -> dict:
             "changes": s.changes, "notes": s.notes,
         } for s in streams(reports)],
         "skipped": [{"where": w, "reason": r} for w, r in skipped],
+        "warnings": [{"warning": w, "times": n, "first": where, "detail": d} for w, n, where, d in grouped(loaded.warnings)],
+        "copies": loaded.copies,
     }

{aprf_reader-0.3.1 → aprf_reader-0.4.1/src/aprf_reader.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aprf-reader
-Version: 0.3.1
+Version: 0.4.1
 Summary: Summarise APRF (Aggregate Performance Reporting) inbox placement reports.
 License: MIT
 Requires-Python: >=3.11
@@ -24,7 +24,7 @@ Two ways to use it, with the same results:
   your reports on the page. They are read on your computer and never uploaded.
 - **On the command line:** `pipx install aprf-reader`.
 
-**Status: 0.3, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
+**Status: 0.4, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
 
 ## What APRF is
 
@@ -88,8 +88,62 @@ to mark a change in what a category counts.
   or share. Nothing is uploaded anywhere.
 - Follow the draft as it changes. The report format lives in one module
   (`src/aprf_reader/report.py`), so a new draft revision is a small update.
-- When the same day arrives twice, keep the first copy and say so. The draft
-  has not yet decided whether a resend replaces the original.
+- Check each report against the draft and skip, with the reason, any it
+  cannot trust. One bad report in a file does not cost the rest of the file.
+
+## Checking against the draft
+
+Every report is checked against draft-ietf-mailmaint-aprf-00, and every
+message names the section, so a list of them can go straight into a review of
+the draft. On the web page, Copy for a draft review copies them as text.
+
+**Skipped**, because the report cannot be trusted:
+
+| Problem | Section |
+|---|---|
+| Not valid JSON | 5 |
+| A required header field missing or of the wrong type; times not in epoch seconds | 5.2.1 |
+| Not one UTC day from 00:00:00 to 23:59:59, or ending before it starts | 5.1 |
+| Body not a list of segments; a segment not an object | 5.2.2 |
+| A placement or engagement count that is not a whole number | 6, 7 |
+| A saved email with no report attached | 8 |
+
+**Warnings**, where the report is still read:
+
+| Warning | Section |
+|---|---|
+| `sdi_used` is not N/A, N/F, or the `sdi` value as published in DNS, or is missing | 5.2.1 |
+| Attachment name or subject not `<yyyymmdd>_<DKIM domain>_<DKIM selector>_<source>` | 8 |
+| Message not `multipart/report`; attachment not `application/json` (or `application/gzip` for `.json.gz`) | 8, 8.1 |
+| One attachment holding several reports | 8 |
+| A placement key other than the four section 6 names, or an engagement key other than positive, neutral and negative | 6, 5.2.2 |
+| A name or subject that writes `/` or `*` some other way, as a file name must | 8 is silent |
+| The same day twice from one provider | left open, note in 8.1 |
+
+Comcast's beta sends `sdi_used` as ` - N/A - Beta - Not in use` and a source
+of `Comcast/Xfinity`, so its reports carry the `sdi_used` warning, and a
+correct file name for them cannot follow section 8 exactly. A file that
+repeats a report already read, such as the same report as `.json` and as
+`.eml`, is counted once and summed up in one line.
+
+## How it decides
+
+- **The same day twice, from the same provider:** it keeps the first copy it
+  reads and skips the other, saying whether the skipped copy had a different
+  `version` or different counts. The draft leaves resends open (a note in
+  section 8.1 asks whether a later report should overwrite or be discarded),
+  and `version` is a free string with no order, so there is no defined way
+  to pick the newer one. Files are read in name order, so the first copy is
+  the first by file name, then by position in the file.
+- **The same day from two providers:** both are kept, as separate streams.
+- **Names:** signing domains and selectors compare without case, and an
+  international domain matches its `xn--` form, so `bücher.example` and
+  `xn--bcher-kva.example` are one stream.
+- **The reporting window:** section 5.1 says a report MUST cover one UTC day,
+  from 00:00:00 to 23:59:59. A report that covers more, or is shifted off the
+  day boundary, is skipped with a reason that names section 5.1.
+- **Counts:** whole numbers only. `100.0` reads as 100; `"100"` and negative
+  numbers are refused.
 
 ## How the web page is built
 

{aprf_reader-0.3.1 → aprf_reader-0.4.1}/tests/test_reader.py

@@ -88,9 +88,12 @@ def test_load_gz_eml_folder_and_skips(tmp_path):
 
     loaded = load([tmp_path])
     assert sorted(r.day.day for r in loaded.reports) == [1, 2, 3]
-    reasons = dict((Path(w.split(" ")[0]).name, r) for w, r in loaded.skipped)
-    assert set(reasons) == {"copy.json", "resend.json", "broken.json"}
-    assert "duplicate" in reasons["copy.json"] and "same day" in reasons["resend.json"]
+    assert [Path(w).name for w, _ in loaded.skipped] == ["broken.json"]
+    assert loaded.copies == 1  # copy.json has a.json's bytes
+    warned = {(w.split(";")[0].split(".")[0], Path(where.split(" ")[0]).name) for w, _, where in loaded.warnings}
+    assert ("the same day came twice from one provider", "resend.json") in warned
+    # The test email is multipart/mixed; its name and subject follow section 8
+    assert {w for w, _, where in loaded.warnings if "c.eml" in where} == {"message Content-Type is not multipart/report (section 8)"}
 
 
 def report_email(day: int, payload: bytes, subtype: str, filename: str | None) -> EmailMessage:
@@ -238,14 +241,25 @@ def test_cli_terminal_when_interactive(monkeypatch, capsys):
 
 def test_stress_set_one_bad_report_does_not_cost_the_file():
     loaded = load([FIX / "stress_100.json"])
-    assert len(loaded.reports) == 88
+    assert len(loaded.reports) == 86
     reasons = {w.rsplit(" ", 1)[-1]: r for w, r in loaded.skipped}
-    assert reasons["73"] == 'body[0].engagement.positive is not a count: "100"'
-    assert reasons["79"] == "header.report_end is before report_start"
+    # s18 spans two days and s19 is shifted an hour: draft section 5.1 requires one UTC day
+    assert "section 5.1" in reasons["77"] and "section 5.1" in reasons["78"]
+    # Same day from the same provider: keep the first, warn, and say how the copy differs
+    dup = {where.rsplit(" ", 1)[-1]: d for w, d, where in loaded.warnings if w.startswith("the same day came twice")}
+    assert dup["64"] == f"{FIX}/stress_100.json report 63 kept, version 1 kept, version 2 dropped"
+    assert dup["69"].endswith("report 68 kept, with different counts")  # bücher.example
+    assert "report 3 kept" in dup["70"]  # DNS-AUDIT.COM / SELECTOR1 is dns-audit.com / selector1
+    assert not {"64", "69", "70"} & set(reasons)
+    warned = {w for w, _, _ in loaded.warnings}
+    assert "sdi_used is not N/A, N/F, or the sdi value as published in DNS (section 5.2.1)" in warned
+    assert any(w.startswith('classification key "spam"') for w in warned)
+    assert not any('"promotional"' in w or '"forwarded"' in w for w in warned)  # section 6 names them
+    assert reasons["73"] == 'body[0].engagement.positive is not a count (section 7): "100"'
+    assert reasons["79"] == "header.report_end is before report_start (section 5.1)"
     assert reasons["81"].startswith("header.report_start looks like milliseconds")
-    assert "report 63" in reasons["64"]  # bücher.example is xn--bcher-kva.example
-    assert "report 3" in reasons["70"]  # DNS-AUDIT.COM / SELECTOR1 is dns-audit.com / selector1
     ss = {(s.source, s.domain, s.selector): s for s in streams(loaded.reports)}
+    assert ("Big MBP Co", "dns-audit.com", "s11") in ss  # another provider, same day: its own stream
     obj = report(0)
     obj["body"][0]["classification"]["unwanted"] = 100.0
     assert parse(json.dumps(obj).encode())[0].segments[0].classification["unwanted"] == 100

{aprf_reader-0.3.1 → aprf_reader-0.4.1}/tests/test_web_parity.py

@@ -29,7 +29,7 @@ def web(mode: str, paths: list[Path]) -> str:
 
 def same(paths: list[Path]) -> None:
     loaded = load(paths)
-    assert json.loads(web("--json", paths)) == to_dict(loaded.reports, loaded.skipped)
+    assert json.loads(web("--json", paths)) == to_dict(loaded)
     assert web("--md", paths) == to_markdown(loaded)
     assert web("--html", paths) == to_html(loaded)
 
@@ -104,3 +104,27 @@ def test_hostile_text_escaped_the_same(tmp_path):
     obj["body"][0]["classification"][hostile] = 7
     (tmp_path / "h.json").write_text(json.dumps(obj))
     same([tmp_path / "h.json"])
+
+
+def test_section_8_checks_the_same(tmp_path):
+    raw = json.dumps(report(9)).encode()  # 2026-10-10, Example.com, s1, Comcast/Xfinity
+    good = email(raw, "json", "20261010_Example.com_s1_Comcast/Xfinity.json")
+    good.replace_header("Subject", "APRF: 20261010_Example.com_s1_Comcast/Xfinity")
+    good.set_type("multipart/report")
+    safe = email(gzip.compress(json.dumps(report(10)).encode()), "gzip", "20261011_example.com_s1_Comcast-Xfinity.json.gz")
+    safe.replace_header("Subject", "APRF: 20261011_example.com_s1_Comcast-Xfinity")
+    wrong = email(json.dumps(report(11)).encode(), "octet-stream", "report.json")
+    paths = []
+    for name, msg in (("good.eml", good), ("safe.eml", safe), ("wrong.eml", wrong)):
+        (tmp_path / name).write_bytes(msg.as_bytes())
+        paths.append(tmp_path / name)
+    same(paths)
+    loaded = load(paths)
+    by_file = {}
+    for w, _, where in loaded.warnings:
+        by_file.setdefault(Path(where.split(" ")[0]).name, set()).add(w.split(";")[0].split(":")[0].split(" (")[0])
+    # Name, subject and both Content-Types follow section 8; only Comcast's beta sdi_used remains
+    assert by_file["good.eml"] == {"sdi_used is not N/A, N/F, or the sdi value as published in DNS"}
+    assert "attachment name stands in for a character a file name cannot hold, such as / or *" in by_file["safe.eml"]
+    assert {"attachment name does not follow section 8", "subject does not follow section 8",
+            "attachment Content-Type is not application/json, or application/gzip when compressed"} <= by_file["wrong.eml"]

aprf_reader-0.3.1/src/aprf_reader/load.py

@@ -1,143 +0,0 @@
-"""Find report files, open them, and drop duplicates."""
-
-from __future__ import annotations
-
-import gzip
-import hashlib
-import io
-import mailbox
-import zlib
-from collections.abc import Iterable, Iterator
-from dataclasses import dataclass, field
-from email import policy
-from email.message import Message
-from email.parser import BytesParser
-from pathlib import Path
-
-from .report import Report, ReportError, parse_items
-
-SUFFIXES = (".json", ".json.gz", ".eml", ".mbox")
-# A daily report is a few KB; this stops a stray huge file or gzip bomb
-MAX_BYTES = 10 * 1024 * 1024
-
-
-@dataclass
-class Loaded:
-    reports: list[Report] = field(default_factory=list)
-    skipped: list[tuple[str, str]] = field(default_factory=list)  # (where, reason)
-
-
-def _files(paths: Iterable[Path]) -> Iterator[Path]:
-    for p in paths:
-        if p.is_dir():
-            yield from sorted(f for f in p.rglob("*") if f.is_file() and (f.name.lower().endswith(SUFFIXES) or _is_mbox(f)))
-        else:
-            yield p
-
-
-def _is_mbox(path: Path) -> bool:
-    # Apple Mail exports a folder "Name.mbox" holding a file named just "mbox"
-    return path.name.lower() == "mbox" or path.name.lower().endswith(".mbox")
-
-
-def _gunzip(data: bytes) -> bytes:
-    with gzip.GzipFile(fileobj=io.BytesIO(data)) as g:
-        out = g.read(MAX_BYTES + 1)
-    if len(out) > MAX_BYTES:
-        raise ReportError("larger than 10 MB unpacked")
-    return out
-
-
-def _attachments(msg: Message, label: str) -> Iterator[tuple[str, bytes]]:
-    """Yield (label, report JSON bytes) for each report attached to one email."""
-    for part in msg.walk():
-        if part.is_multipart():
-            continue
-        fname = (part.get_filename() or "").lower()
-        ctype = part.get_content_type()
-        data = part.get_payload(decode=True) or b""
-        where = f"{label} ({fname or ctype})"
-        if len(data) > MAX_BYTES:
-            raise ReportError(f"{where} is larger than 10 MB")
-        if fname.endswith(".gz") or ctype in ("application/gzip", "application/x-gzip") or data[:2] == b"\x1f\x8b":
-            yield where, _gunzip(data)
-        # By name or type, or an attachment of unhelpful type that is a JSON array or object
-        elif fname.endswith(".json") or ctype == "application/json" or (
-                part.get_content_disposition() == "attachment" and data.lstrip()[:1] in (b"[", b"{")):
-            yield where, data
-
-
-def _payloads(path: Path) -> Iterator[tuple[str, bytes]]:
-    """Yield (label, report JSON bytes) for each report in a file."""
-    data = path.read_bytes()
-    if len(data) > MAX_BYTES:
-        raise ReportError("larger than 10 MB")
-    if path.name.lower().endswith(".eml"):
-        found = list(_attachments(BytesParser(policy=policy.default).parsebytes(data), str(path)))
-        if not found:
-            raise ReportError("no report attachment")
-        yield from found
-    elif data[:2] == b"\x1f\x8b":
-        yield str(path), _gunzip(data)
-    else:
-        yield str(path), data
-
-
-def _mbox(path: Path, out: Loaded) -> Iterator[tuple[str, bytes]]:
-    """Yield reports from a mailbox export; one skip line for all mail with none attached."""
-    box = mailbox.mbox(path, factory=lambda f: BytesParser(policy=policy.default).parse(f), create=False)
-    without = 0
-    try:
-        for i, msg in enumerate(box, 1):
-            label = f"{path} message {i}"
-            try:
-                found = list(_attachments(msg, label))
-            # OSError covers gzip.BadGzipFile; one bad message must not end the mailbox
-            except (OSError, EOFError, zlib.error, ReportError) as e:
-                out.skipped.append((label, str(e)))
-                continue
-            without += not found
-            yield from found
-    finally:
-        box.close()
-    if without:
-        out.skipped.append((str(path), f"{without} message{'s' if without != 1 else ''} with no report attached"))
-
-
-def load(paths: Iterable[Path]) -> Loaded:
-    out = Loaded()
-    seen_bytes: set[str] = set()
-    seen_keys: dict[tuple, str] = {}
-
-    def add(where: str, data: bytes) -> None:
-        digest = hashlib.sha256(data).hexdigest()
-        if digest in seen_bytes:
-            out.skipped.append((where, "duplicate of a file already read"))
-            return
-        seen_bytes.add(digest)
-        try:
-            items = parse_items(data)
-        except ReportError as e:
-            out.skipped.append((where, str(e)))
-            return
-        for i, r in enumerate(items, 1):
-            label = f"{where} report {i}" if len(items) > 1 else where
-            if isinstance(r, ReportError):
-                out.skipped.append((label, str(r)))
-                continue
-            # The draft does not yet say whether a resend replaces the first; keep the first
-            if r.key in seen_keys:
-                out.skipped.append((label, f"same day already reported in {seen_keys[r.key]}; kept that one"))
-                continue
-            seen_keys[r.key] = label
-            out.reports.append(r)
-
-    for path in _files(paths):
-        try:
-            # An mbox can be large, so it is read one message at a time
-            payloads = _mbox(path, out) if _is_mbox(path) else list(_payloads(path))
-            for where, data in payloads:
-                add(where, data)
-        except (OSError, EOFError, zlib.error, ReportError) as e:
-            out.skipped.append((str(path), str(e)))
-    return out