aprf-reader 0.3.0__tar.gz → 0.4.0__tar.gz

{aprf_reader-0.3.0/src/aprf_reader.egg-info → aprf_reader-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: aprf-reader
-Version: 0.3.0
+Version: 0.4.0
 Summary: Summarise APRF (Aggregate Performance Reporting) inbox placement reports.
 License: MIT
 Requires-Python: >=3.11
@@ -24,7 +24,7 @@ Two ways to use it, with the same results:
   your reports on the page. They are read on your computer and never uploaded.
 - **On the command line:** `pipx install aprf-reader`.
 
-**Status: 0.3, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
+**Status: 0.4, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
 
 ## What APRF is
 
@@ -88,8 +88,62 @@ to mark a change in what a category counts.
   or share. Nothing is uploaded anywhere.
 - Follow the draft as it changes. The report format lives in one module
   (`src/aprf_reader/report.py`), so a new draft revision is a small update.
-- When the same day arrives twice, keep the first copy and say so. The draft
-  has not yet decided whether a resend replaces the original.
+- Check each report against the draft and skip, with the reason, any it
+  cannot trust. One bad report in a file does not cost the rest of the file.
+
+## Checking against the draft
+
+Every report is checked against draft-ietf-mailmaint-aprf-00, and every
+message names the section, so a list of them can go straight into a review of
+the draft. On the web page, Copy for a draft review copies them as text.
+
+**Skipped**, because the report cannot be trusted:
+
+| Problem | Section |
+|---|---|
+| Not valid JSON | 5 |
+| A required header field missing or of the wrong type; times not in epoch seconds | 5.2.1 |
+| Not one UTC day from 00:00:00 to 23:59:59, or ending before it starts | 5.1 |
+| Body not a list of segments; a segment not an object | 5.2.2 |
+| A placement or engagement count that is not a whole number | 6, 7 |
+| A saved email with no report attached | 8 |
+
+**Warnings**, where the report is still read:
+
+| Warning | Section |
+|---|---|
+| `sdi_used` is not N/A, N/F, or the `sdi` value as published in DNS, or is missing | 5.2.1 |
+| Attachment name or subject not `<yyyymmdd>_<DKIM domain>_<DKIM selector>_<source>` | 8 |
+| Message not `multipart/report`; attachment not `application/json` (or `application/gzip` for `.json.gz`) | 8, 8.1 |
+| One attachment holding several reports | 8 |
+| A placement key other than the four section 6 names, or an engagement key other than positive, neutral and negative | 6, 5.2.2 |
+| A name or subject that writes `/` or `*` some other way, as a file name must | 8 is silent |
+| The same day twice from one provider | left open, note in 8.1 |
+
+Comcast's beta sends `sdi_used` as ` - N/A - Beta - Not in use` and a source
+of `Comcast/Xfinity`, so its reports carry the `sdi_used` warning, and a
+correct file name for them cannot follow section 8 exactly. A file that
+repeats a report already read, such as the same report as `.json` and as
+`.eml`, is counted once and summed up in one line.
+
+## How it decides
+
+- **The same day twice, from the same provider:** it keeps the first copy it
+  reads and skips the other, saying whether the skipped copy had a different
+  `version` or different counts. The draft leaves resends open (a note in
+  section 8.1 asks whether a later report should overwrite or be discarded),
+  and `version` is a free string with no order, so there is no defined way
+  to pick the newer one. Files are read in name order, so the first copy is
+  the first by file name, then by position in the file.
+- **The same day from two providers:** both are kept, as separate streams.
+- **Names:** signing domains and selectors compare without case, and an
+  international domain matches its `xn--` form, so `bücher.example` and
+  `xn--bcher-kva.example` are one stream.
+- **The reporting window:** section 5.1 says a report MUST cover one UTC day,
+  from 00:00:00 to 23:59:59. A report that covers more, or is shifted off the
+  day boundary, is skipped with a reason that names section 5.1.
+- **Counts:** whole numbers only. `100.0` reads as 100; `"100"` and negative
+  numbers are refused.
 
 ## How the web page is built
 

{aprf_reader-0.3.0 → aprf_reader-0.4.0}/README.md

@@ -11,7 +11,7 @@ Two ways to use it, with the same results:
   your reports on the page. They are read on your computer and never uploaded.
 - **On the command line:** `pipx install aprf-reader`.
 
-**Status: 0.3, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
+**Status: 0.4, beta.** Reads reports in the format of draft-ietf-mailmaint-aprf-00.
 
 ## What APRF is
 
@@ -75,8 +75,62 @@ to mark a change in what a category counts.
   or share. Nothing is uploaded anywhere.
 - Follow the draft as it changes. The report format lives in one module
   (`src/aprf_reader/report.py`), so a new draft revision is a small update.
-- When the same day arrives twice, keep the first copy and say so. The draft
-  has not yet decided whether a resend replaces the original.
+- Check each report against the draft and skip, with the reason, any it
+  cannot trust. One bad report in a file does not cost the rest of the file.
+
+## Checking against the draft
+
+Every report is checked against draft-ietf-mailmaint-aprf-00, and every
+message names the section, so a list of them can go straight into a review of
+the draft. On the web page, Copy for a draft review copies them as text.
+
+**Skipped**, because the report cannot be trusted:
+
+| Problem | Section |
+|---|---|
+| Not valid JSON | 5 |
+| A required header field missing or of the wrong type; times not in epoch seconds | 5.2.1 |
+| Not one UTC day from 00:00:00 to 23:59:59, or ending before it starts | 5.1 |
+| Body not a list of segments; a segment not an object | 5.2.2 |
+| A placement or engagement count that is not a whole number | 6, 7 |
+| A saved email with no report attached | 8 |
+
+**Warnings**, where the report is still read:
+
+| Warning | Section |
+|---|---|
+| `sdi_used` is not N/A, N/F, or the `sdi` value as published in DNS, or is missing | 5.2.1 |
+| Attachment name or subject not `<yyyymmdd>_<DKIM domain>_<DKIM selector>_<source>` | 8 |
+| Message not `multipart/report`; attachment not `application/json` (or `application/gzip` for `.json.gz`) | 8, 8.1 |
+| One attachment holding several reports | 8 |
+| A placement key other than the four section 6 names, or an engagement key other than positive, neutral and negative | 6, 5.2.2 |
+| A name or subject that writes `/` or `*` some other way, as a file name must | 8 is silent |
+| The same day twice from one provider | left open, note in 8.1 |
+
+Comcast's beta sends `sdi_used` as ` - N/A - Beta - Not in use` and a source
+of `Comcast/Xfinity`, so its reports carry the `sdi_used` warning, and a
+correct file name for them cannot follow section 8 exactly. A file that
+repeats a report already read, such as the same report as `.json` and as
+`.eml`, is counted once and summed up in one line.
+
+## How it decides
+
+- **The same day twice, from the same provider:** it keeps the first copy it
+  reads and skips the other, saying whether the skipped copy had a different
+  `version` or different counts. The draft leaves resends open (a note in
+  section 8.1 asks whether a later report should overwrite or be discarded),
+  and `version` is a free string with no order, so there is no defined way
+  to pick the newer one. Files are read in name order, so the first copy is
+  the first by file name, then by position in the file.
+- **The same day from two providers:** both are kept, as separate streams.
+- **Names:** signing domains and selectors compare without case, and an
+  international domain matches its `xn--` form, so `bücher.example` and
+  `xn--bcher-kva.example` are one stream.
+- **The reporting window:** section 5.1 says a report MUST cover one UTC day,
+  from 00:00:00 to 23:59:59. A report that covers more, or is shifted off the
+  day boundary, is skipped with a reason that names section 5.1.
+- **Counts:** whole numbers only. `100.0` reads as 100; `"100"` and negative
+  numbers are refused.
 
 ## How the web page is built
 

{aprf_reader-0.3.0 → aprf_reader-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "aprf-reader"
-version = "0.3.0"
+version = "0.4.0"
 description = "Summarise APRF (Aggregate Performance Reporting) inbox placement reports."
 readme = "README.md"
 license = { text = "MIT" }

{aprf_reader-0.3.0 → aprf_reader-0.4.0}/src/aprf_reader/__init__.py

@@ -1,3 +1,3 @@
 """Read APRF aggregate performance reports."""
 
-__version__ = "0.3.0"
+__version__ = "0.4.0"

{aprf_reader-0.3.0 → aprf_reader-0.4.0}/src/aprf_reader/cli.py

@@ -36,7 +36,7 @@ def main(argv: list[str] | None = None) -> int:
         return 1
 
     if args.json:
-        text = json.dumps(to_dict(loaded.reports, loaded.skipped), indent=2) + "\n"
+        text = json.dumps(to_dict(loaded), indent=2) + "\n"
     elif args.md or args.markdown or not sys.stdout.isatty():
         text = to_markdown(loaded)  # files and pipes get Markdown, which other tools read
     else:

aprf_reader-0.4.0/src/aprf_reader/load.py

@@ -0,0 +1,241 @@
+"""Find report files, open them, and drop duplicates."""
+
+from __future__ import annotations
+
+import gzip
+import hashlib
+import io
+import mailbox
+import re
+import zlib
+from collections.abc import Iterable, Iterator
+from dataclasses import dataclass, field
+from email import policy
+from email.message import Message
+from email.parser import BytesParser
+from pathlib import Path
+
+from .report import Report, ReportError, parse_items
+
+SUFFIXES = (".json", ".json.gz", ".eml", ".mbox")
+# A daily report is a few KB; this stops a stray huge file or gzip bomb
+MAX_BYTES = 10 * 1024 * 1024
+
+
+@dataclass
+class Loaded:
+    reports: list[Report] = field(default_factory=list)
+    skipped: list[tuple[str, str]] = field(default_factory=list)  # (where, reason)
+    # (warning, detail, where): read, but departs from the draft or meets a case it leaves open
+    warnings: list[tuple[str, str, str]] = field(default_factory=list)
+    # Files whose report bytes were already read, often the same report as .json and .eml
+    copies: int = 0
+
+
+@dataclass(frozen=True)
+class Mail:
+    """How a report arrived by email, for the section 8 checks."""
+    name: str  # attachment file name as sent
+    ctype: str  # attachment Content-Type
+    gz: bool
+    subject: str
+    msg_ctype: str  # Content-Type of the message that carries it
+
+DUPLICATE_DAY = ("the same day came twice from one provider; the first copy read was kept. "
+                 "The draft leaves this open (a note in section 8.1)")
+
+
+def _files(paths: Iterable[Path]) -> Iterator[Path]:
+    for p in paths:
+        if p.is_dir():
+            yield from sorted(f for f in p.rglob("*") if f.is_file() and (f.name.lower().endswith(SUFFIXES) or _is_mbox(f)))
+        else:
+            yield p
+
+
+def _is_mbox(path: Path) -> bool:
+    # Apple Mail exports a folder "Name.mbox" holding a file named just "mbox"
+    return path.name.lower() == "mbox" or path.name.lower().endswith(".mbox")
+
+
+def _gunzip(data: bytes) -> bytes:
+    with gzip.GzipFile(fileobj=io.BytesIO(data)) as g:
+        out = g.read(MAX_BYTES + 1)
+    if len(out) > MAX_BYTES:
+        raise ReportError("larger than 10 MB unpacked")
+    return out
+
+
+def _leaves(m: Message, owner: Message) -> Iterator[tuple[Message, Message]]:
+    """Each non-multipart part with the email it belongs to; an attached email owns its own parts."""
+    if m.get_content_type() == "message/rfc822":
+        for inner in m.get_payload():
+            yield from _leaves(inner, inner)
+    elif m.is_multipart():
+        for p in m.get_payload():
+            yield from _leaves(p, owner)
+    else:
+        yield m, owner
+
+
+def _attachments(msg: Message, label: str) -> Iterator[tuple[str, bytes, Mail]]:
+    """Yield (label, report JSON bytes, how it arrived) for each report attached to one email."""
+    for part, owner in _leaves(msg, msg):
+        name = part.get_filename() or ""
+        fname = name.lower()
+        ctype = part.get_content_type()
+        data = part.get_payload(decode=True) or b""
+        where = f"{label} ({fname or ctype})"
+        if len(data) > MAX_BYTES:
+            raise ReportError(f"{where} is larger than 10 MB")
+        mail = lambda gz: Mail(name, ctype, gz, " ".join(str(owner.get("subject", "")).split()), owner.get_content_type())
+        if fname.endswith(".gz") or ctype in ("application/gzip", "application/x-gzip") or data[:2] == b"\x1f\x8b":
+            yield where, _gunzip(data), mail(True)
+        # By name or type, or an attachment of unhelpful type that is a JSON array or object
+        elif fname.endswith(".json") or ctype == "application/json" or (
+                part.get_content_disposition() == "attachment" and data.lstrip()[:1] in (b"[", b"{")):
+            yield where, data, mail(False)
+
+
+def _payloads(path: Path) -> Iterator[tuple[str, bytes, Mail | None]]:
+    """Yield (label, report JSON bytes, how it arrived) for each report in a file."""
+    data = path.read_bytes()
+    if len(data) > MAX_BYTES:
+        raise ReportError("larger than 10 MB")
+    if path.name.lower().endswith(".eml"):
+        found = list(_attachments(BytesParser(policy=policy.default).parsebytes(data), str(path)))
+        if not found:
+            raise ReportError("no report attachment, which section 8 calls for")
+        yield from found
+    elif data[:2] == b"\x1f\x8b":
+        yield str(path), _gunzip(data), None
+    else:
+        yield str(path), data, None
+
+
+def _mbox(path: Path, out: Loaded) -> Iterator[tuple[str, bytes, Mail | None]]:
+    """Yield reports from a mailbox export; one skip line for all mail with none attached."""
+    box = mailbox.mbox(path, factory=lambda f: BytesParser(policy=policy.default).parse(f), create=False)
+    without = 0
+    try:
+        for i, msg in enumerate(box, 1):
+            label = f"{path} message {i}"
+            try:
+                found = list(_attachments(msg, label))
+            # OSError covers gzip.BadGzipFile; one bad message must not end the mailbox
+            except (OSError, EOFError, zlib.error, ReportError) as e:
+                out.skipped.append((label, str(e)))
+                continue
+            without += not found
+            yield from found
+    finally:
+        box.close()
+    if without:
+        out.skipped.append((str(path), f"{without} message{'s' if without != 1 else ''} with no report attached"))
+
+
+def _differs(kept: Report, dropped: Report) -> str:
+    if kept.version != dropped.version:
+        return f", version {kept.version or 'none'} kept, version {dropped.version or 'none'} dropped"
+    return ", with different counts" if kept.segments != dropped.segments else ", same counts"
+
+
+def _name(r: Report) -> str:
+    # Section 8: the ending date, then domain, selector and source with spaces removed
+    return f"{r.day:%Y%m%d}_{r.dkim_domain}_{r.dkim_selector}_{r.source.replace(' ', '')}"
+
+
+# Characters a file name cannot hold on common systems; "Comcast/Xfinity" and a "*" roll-up
+# selector both carry one, and section 8 only says to remove spaces
+UNSAFE = '/\\:*?"<>|'
+NAME_UNSAFE = ("attachment name stands in for a character a file name cannot hold, such as / or *; "
+               "section 8 does not say how to write those")
+SUBJECT_UNSAFE = ("subject stands in for / or * as a file name must, though section 8 gives the source "
+                  "and selector as they are")
+
+
+def _loose(expected: str, actual: str) -> bool:
+    """Equal, ignoring case, if each file-name-unsafe character in expected may be any character."""
+    pattern = "".join("." if c in UNSAFE else re.escape(c) for c in expected.lower())
+    return re.fullmatch(pattern, actual.lower()) is not None
+
+
+def _mail_warnings(mail: Mail, items: list) -> list[tuple[str, str]]:
+    out = []
+    if mail.msg_ctype != "multipart/report":
+        out.append(("message Content-Type is not multipart/report (section 8)", mail.msg_ctype))
+    want = "application/gzip" if mail.gz else "application/json"
+    if mail.ctype != want:
+        out.append(("attachment Content-Type is not application/json, or application/gzip when "
+                    "compressed (sections 8 and 8.1)", mail.ctype))
+    if len(items) > 1:
+        out.append(("one attachment holds several reports; the section 8 name and subject fit one", f"{len(items)} reports"))
+    elif isinstance(items[0], Report):
+        base = _name(items[0])
+        file = base + (".json.gz" if mail.gz else ".json")
+        if mail.name.lower() == file.lower():
+            pass
+        elif _loose(file, mail.name):
+            out.append((NAME_UNSAFE, f"{mail.name}; section 8 gives {file}"))
+        else:
+            out.append(("attachment name does not follow section 8: <yyyymmdd>_<DKIM domain>_<DKIM selector>_<source>.json",
+                        f"{mail.name or '(none)'}; expected {file}"))
+        if mail.subject.lower() == f"APRF: {base}".lower():
+            pass
+        elif _loose(f"APRF: {base}", mail.subject):
+            out.append((SUBJECT_UNSAFE, f"{mail.subject}; section 8 gives APRF: {base}"))
+        else:
+            out.append(("subject does not follow section 8: APRF: <yyyymmdd>_<DKIM domain>_<DKIM selector>_<source>",
+                        f"{mail.subject or '(none)'}; expected APRF: {base}"))
+    return out
+
+
+def load(paths: Iterable[Path]) -> Loaded:
+    out = Loaded()
+    seen_bytes: set[str] = set()
+    seen_keys: dict[tuple, tuple[str, Report]] = {}
+
+    parsed: dict[str, list | ReportError] = {}
+
+    def add(where: str, data: bytes, mail: Mail | None) -> None:
+        digest = hashlib.sha256(data).hexdigest()
+        copy = digest in parsed
+        if not copy:
+            try:
+                parsed[digest] = parse_items(data)
+            except ReportError as e:
+                parsed[digest] = e
+        items = parsed[digest]
+        # An email is checked even when its report was already read from another file
+        if mail is not None and not isinstance(items, ReportError):
+            out.warnings += [(w, d, where) for w, d in _mail_warnings(mail, items)]
+        if copy:
+            out.copies += 1
+            return
+        if isinstance(items, ReportError):
+            out.skipped.append((where, str(items)))
+            return
+        for i, r in enumerate(items, 1):
+            label = f"{where} report {i}" if len(items) > 1 else where
+            if isinstance(r, ReportError):
+                out.skipped.append((label, str(r)))
+                continue
+            out.warnings += [(w, d, label) for w, d in r.warnings]
+            # The draft leaves open whether a resend replaces the first (an open note in
+            # section 8.1), and its version is a free string with no order; keep the first
+            if r.key in seen_keys:
+                first_label, first = seen_keys[r.key]
+                out.warnings.append((DUPLICATE_DAY, f"{first_label} kept{_differs(first, r)}", label))
+                continue
+            seen_keys[r.key] = (label, r)
+            out.reports.append(r)
+
+    for path in _files(paths):
+        try:
+            # An mbox can be large, so it is read one message at a time
+            payloads = _mbox(path, out) if _is_mbox(path) else list(_payloads(path))
+            for where, data, mail in payloads:
+                add(where, data, mail)
+        except (OSError, EOFError, zlib.error, ReportError) as e:
+            out.skipped.append((str(path), str(e)))
+    return out

{aprf_reader-0.3.0 → aprf_reader-0.4.0}/src/aprf_reader/render.py

@@ -9,7 +9,16 @@ from datetime import date
 from . import __version__
 from .load import Loaded
 from .report import DRAFT
-from .summary import Row, Stream, streams
+from .summary import Row, Stream, copies_line, grouped, streams
+
+WARNINGS_NOTE = "Read anyway. Each names the draft section it departs from, or the case the draft leaves open."
+
+
+def _warning(w: str, times: int, where: str, detail: str) -> str:
+    return f"{w}. {_plural(times, 'time')}, first in {where}" + (f": {detail}" if detail else "")
+
+
+NO_COUNTS = "These reports carry no counts."
 
 CAVEAT = (
     "Counts are bucket upper bounds chosen by each provider: a ceiling, not an exact figure. "
@@ -50,6 +59,9 @@ def to_markdown(loaded: Loaded) -> str:
     for s in ss:
         lines += [f"## {_md(s.source)}: {_md(s.domain)}, selector {_md(s.selector)}", "", _span(s), ""]
         cols = [k for _, k in s.columns]
+        if not cols:
+            lines += [NO_COUNTS, ""]
+            continue
         lines.append("| Day | Segment | " + " | ".join(_md(c) for c in cols) + " |")
         lines.append("|---|---|" + "---:|" * len(cols))
         for row in s.rows:
@@ -62,8 +74,12 @@ def to_markdown(loaded: Loaded) -> str:
             lines += ["Changes worth a look:", ""] + [f"- {_md(c)}" for c in s.changes] + [""]
         if s.notes:
             lines += [f"- {_md(n)}" for n in s.notes] + [""]
+    if loaded.warnings:
+        lines += ["## Warnings", "", WARNINGS_NOTE, ""] + [f"- {_md(_warning(*g))}" for g in grouped(loaded.warnings)] + [""]
     if loaded.skipped:
         lines += ["## Skipped", ""] + [f"- {_md(w)}: {_md(r)}" for w, r in loaded.skipped] + [""]
+    if loaded.copies:
+        lines += [copies_line(loaded.copies), ""]
     lines.append(f"Read by aprf-reader {__version__} against {DRAFT}.")
     return "\n".join(lines) + "\n"
 
@@ -112,6 +128,9 @@ def to_html(loaded: Loaded) -> str:
     for s in ss:
         cols = [k for _, k in s.columns]
         parts.append(f"<h2>{e(s.source)}: {e(s.domain)}, selector {e(s.selector)}</h2>")
+        if not cols:
+            parts.append(f"<p class=muted>{e(_span(s))}</p><p>{NO_COUNTS}</p>")
+            continue
         parts.append(f"<p class=muted>{e(_span(s))}</p><div class=wrap><table><thead><tr><th>Day<th>Segment")
         parts.append("".join(f"<th>{e(c)}" for c in cols) + "</thead><tbody>")
         parts.append("<tr class=trend><td>Trend<td class=muted>sum per day")
@@ -126,8 +145,13 @@ def to_html(loaded: Loaded) -> str:
             parts.append("<p>Changes worth a look:</p><ul>" + "".join(f"<li>{e(c)}" for c in s.changes) + "</ul>")
         if s.notes:
             parts.append("<ul>" + "".join(f"<li>{e(n)}" for n in s.notes) + "</ul>")
+    if loaded.warnings:
+        parts.append(f"<h2>Warnings</h2><p class=muted>{WARNINGS_NOTE}</p><ul>"
+                     + "".join(f"<li>{e(_warning(*g))}" for g in grouped(loaded.warnings)) + "</ul>")
     if loaded.skipped:
         parts.append("<h2>Skipped</h2><ul>" + "".join(f"<li>{e(w)}: {e(r)}" for w, r in loaded.skipped) + "</ul>")
+    if loaded.copies:
+        parts.append(f"<p class=muted>{e(copies_line(loaded.copies))}</p>")
     parts.append(f"<p class=muted>Read by aprf-reader {e(__version__)} against {DRAFT}. Nothing left this computer.</p>")
     parts.append("</body></html>")
     return "\n".join(parts) + "\n"
@@ -166,6 +190,13 @@ def _bars(points: list[tuple[int, int]], width: int) -> str:
     return "".join(" " if v is None else BARS[min(7, v * 7 // top)] if v else BARS[0] for v in cells)
 
 
+def _spread(points: list[tuple[int, int]]) -> str:
+    lo, hi = min(v for _, v in points), max(v for _, v in points)
+    if lo == hi:
+        return f"{lo:,}" if len(points) == 1 else f"{lo:,} every day"
+    return f"{lo:,} to {hi:,}"
+
+
 def _date(d) -> str:
     return f"{d:%b} {d.day}"
 
@@ -195,10 +226,13 @@ def to_terminal(loaded: Loaded, color: bool = True, width: int = 100) -> str:
             line += st.dim(" · ") + st.warn(f"{_plural(s.missing, 'day')} with no report")
         out += [line, ""]
         names = [k for _, k in s.columns]
+        if not names:
+            out += ["  " + st.dim(NO_COUNTS), ""]
+            continue
         nw = max([len(n) for n in names] + [5])
         trends = [s.trend(i) for i in range(len(names))]
         latest = [f"{t[-1][1]:,}" for t in trends]
-        ranges = [f"{min(v for _, v in t):,} to {max(v for _, v in t):,}" for t in trends]
+        ranges = [_spread(t) for t in trends]
         lw, rw = max(len(x) for x in latest + ["latest"]), max(len(x) for x in ranges + ["range"])
         bw = max(10, min(60, width - (2 + nw + 2 + lw + 2 + rw + 2)))
         out.append(st.dim(f"  {'':<{nw}}  {'latest':>{lw}}  {'range':<{rw}}  trend"))
@@ -214,9 +248,18 @@ def to_terminal(loaded: Loaded, color: bool = True, width: int = 100) -> str:
         for n in s.notes:
             out += _wrap(n, width - 4, "  · ")
         out.append("")
+    if loaded.warnings:
+        out.append(st.warn(f"Warnings {len(grouped(loaded.warnings))}"))
+        out += [st.dim(line) for line in _wrap(WARNINGS_NOTE, width - 2, "  ")]
+        for g in grouped(loaded.warnings):
+            out += _wrap(_warning(*g), width - 2, "  ")
+        out.append("")
+    if loaded.copies:
+        out += [st.dim(line) for line in _wrap(copies_line(loaded.copies), width)] + [""]
     if loaded.skipped:
         out.append(st.warn(f"Skipped {len(loaded.skipped)}"))
-        out += [st.dim(f"  {w}: {r}") for w, r in loaded.skipped]
+        for w, r in loaded.skipped:
+            out += [st.dim(line) for line in _wrap(f"{w}: {r}", width - 2, "  ")]
         out.append("")
     out.append(st.dim(f"aprf-reader {__version__}, {DRAFT}. Daily tables: --markdown, or --html for a page."))
     return "\n".join(out) + "\n"