@bobfrankston/rmfmail 1.2.288 → 1.2.290

package/.commitmsg

@@ -1,29 +1,39 @@
-Trust: a message that never crossed a trust boundary proves itself
+Spam banner: an unknown rule must be able to condemn on its own
 
-Bob 2026-08-27, on his own reply to John Levine about spam handling: "why is
-my own mail spam". It carried no Authentication-Results at all — his own
-authenticated submission never left his systems, so nothing added one — and
-`proved` therefore came back false, dropping it to a caution banner that told
-him not to open the links in prose he had just written. What flagged it was
-`URIBL_DBL_SPAM` 8.0 on blueorchestra.org: the domain he was QUOTING, in a
-discussion about spam. Guilt by association at its purest.
+Bob 2026-08-28: "we have a problem since, of course, things like conference
+announcements use bulk mailers." An IEEE/ICECER call for papers, DMARC-proved,
+BAYES_00, scored exactly 4.5 of 4.5 — DCC_CHECK 3.0 + KAM_MARKETINGBL 1.0 +
+**1.5 GB_AWSTRACK_REDIR, Amazon SES's click-tracker** — and the one rule this
+file had never heard of pushed a proved announcement into "Check this message
+before acting on it / Links and attachments here should not be opened".
 
-SpamAssassin states two proofs in its own rule list that such mail carries
-instead of an AR header:
+The "unknown means caution" policy was right; testing it by PRESENCE was not.
+An unknown rule now keeps a proved sender at caution only when it could have
+flagged the message BY ITSELF — its own scores summed against the server's own
+threshold. That adds no constant to tune: "enough to condemn" already has a
+number and it is the one the server used. KAM_BENEFICIARY, the case the policy
+exists for, is worth 10.0 alone and still cautions; a 1.5 click-tracker cannot.
+Unscored rules (no X-Spam-Report) leave the sum unknown, and unknown stays
+cautious.
 
-  ALL_TRUSTED    Passed through trusted hosts only via SMTP
-  DKIM_VALID_AU  Message has a valid DKIM or DK signature from author's domain
+Two classification fixes found by sampling what still cautioned a PROVED
+sender across the store:
 
-DKIM_VALID_AU is the same evidence DMARC alignment rests on — the author
-domain signed it and the signature verified — so it proves the From line on
-its own. ALL_TRUSTED says nothing untrusted handled the message in transit.
-`proved` now comes from any of the three, and `provedBy` names which, so the
-banner says WHICH proof rather than asserting DMARC that isn't there. His
-message is now a grey note: "…but bob.ma proved it sent this. Valid signature
-from bob.ma, the sending domain itself (DKIM_VALID_AU)."
+- Branded ESP click-trackers (GB_AWSTRACK*, SendGrid, Mailchimp) are bulk-mail
+  machinery. Open redirectors (URI_PHP_REDIR and friends) stay unclassified —
+  a tracking domain the sending platform owns is not an abused redirect.
+- KAM_*URIBL_* ("Body contains URI listed in PCCC WILD RBL", 9.0) and
+  SH_HBL_EMAILS ("Email address listed in email blocklist", 8.0) are the same
+  evidence as URIBL_SBL — somebody else's list about a host the message
+  mentions — and were the bulk of the remaining noise: a Business Insider
+  newsletter, an MIT cancer-center invitation, a campaign mailing.
+  KAM_INFOUSMEBIZ judges the TLD a domain lives in, which is guilt by
+  neighbourhood.
 
-Forgery still outranks proof — across the store the danger count is unchanged
-at 294 of 1,131 flagged; caution 599, quiet 238.
-
-Also: the Insert menu gains Blockquote and Quoted text (paste) — rmf-tiny
-0.1.53, see its .commitmsg.
+Over the same 20,308 messages, of 1,131 flagged: 294 danger (unchanged — proof
+never overrides forgery), 278 caution, 559 quiet. What still cautions a proved
+sender is now almost entirely content-scam rules: ADVANCE_FEE, DEAR_BENEFICIARY,
+KAM_CRIM, UNDISC_MONEY, BITCOIN_DIRECT. The two false positives left in that
+set — a Substack news digest about displacement tripping KAM_CRIM, a Qatar
+Airways statement tripping BITCOIN_DIRECT — are exactly what sender trust is
+for, and are deliberately NOT being fixed by inventing more rule exceptions.

package/TODO.md

@@ -230,7 +230,7 @@ Items blocked on a design decision. Short-form question here; full context in th
 | **C129** | OPEN | **Android prod/dev split via filename/URL channel** — User 2026-05-09 (revised): no separate MAUI project. Same source tree builds; output filenames distinguish channels: Phase 1 (start now) writes `rmfmaildev.apk` alongside today's `mailx-maui.apk`. Phase 2 (when a release is declared) switches to `rmfmail-<version>.apk` versioned + `rmfmaildev.apk` for latest, with promotion = re-pointing `rmfmail.apk` at a chosen version. Single APK contents per build; channel is runtime setting in AppUpdater (Settings toggle for which URL to poll). Side-by-side install on one device NOT supported by this design (single Android package id) — that's a separate need. See `docs/prod-android.md` for full plan. Phase 1 ~30 min; full flow ~2.5 hours when activated. |
 | **C130** | OPEN | **Reset prefetch error budget after fresh sync** — Android: `errors`/`failedThisSession` accumulate within a prefetch session. Already resets on next syncAll cycle (since each `prefetchBodies(account)` call is fresh). Verify this works in practice via logit; if a long sync session burns through budget and stays stuck, add an explicit reset on `bodyAvailable` events that fire while prefetch is still running. |
 | **C164** | RESEARCHED | **Thundermail / JMAP provider** — full protocol survey at [`docs/thundermail-jmap.md`](thundermail-jmap.md), probed live 2026-08-17. Thundermail is Stalwart, so there is no proprietary API: IMAP 993 / SMTP 465 work **today with zero code** (an `accounts.jsonc` entry and nothing else), CardDAV `/dav/card` + CalDAV `/dav/cal` are http-basic, and JMAP is live at `https://mail.thundermail.com/jmap/` advertising core / mail / submission / calendars / contacts / filenode / sieve / quota plus **WebSocket push** (`supportsPush: true`). JMAP is the attractive bet: it kills the IDLE lane, the 5-lane connection semaphore and the Dovecot 20-conn ceiling, needs no transport injection (one path for Android and desktop, cf. C125), and `Email/changes` is native set-difference sync. **From address: resolved — good.** The 15-alias cap is `@thundermail.com`/`@tb.pro` only; custom domains get unlimited aliases plus a catch-all (empty-name alias), and with catch-all on you can send from any prefix unregistered — their own example is the per-vendor pattern already used on `bob.ma`. mailx needs ~nothing here (per-message From identity shipped v1.2.201); only Stalwart-side `MAIL FROM`/DKIM validation for an unregistered prefix is untested. **Still blocking/staged:** JMAP-calendars and `filenode` are unratified drafts; OAuth `scopes_supported` covers mail only (no calendars/contacts scope); invite-only beta so there is nothing to test against; and **no Drive equivalent** — Thunderbird Send is one-shot file *sharing*, not a listable revisable store, so mailx's shared-config dependency (`accounts.jsonc` / `contacts.jsonc` / preferences / allowlist on GDrive) has no home. **C161 is therefore the prerequisite, not JMAP** — the `_rmfmail` hidden-IMAP-folder state channel removes the Drive dependency for every account type at once and keeps working unchanged if JMAP lands later. Until C161 ships, adopting Thundermail is a half-migration that trades a Google dependency for a Thunderbird one. |
-| **C165** | PARTIAL | **Spam-banner false positives — trust the sender, do not tune the threshold** — full diagnosis and a "What shipped" section at [`docs/spam-false-positives.md`](spam-false-positives.md). Bob 2026-08-27: the red "This message is not what it says it is" banner fired on Constant Contact / Adobe MAX / Mokin newsletters he subscribed to, because `serverSpamVerdict()` consumed SpamAssassin’s **bottom line** (a delivery decision) to answer a different question (is this forged) and never parsed the `tests=` list in the same header. **Shipped v1.2.286 (mailx-types 0.1.64, mailx-store 0.1.96):** `tests=` is classified into forgery / association / bulk / neutral / other; `X-Spam-Report` supplies each rule’s score and description so findings name their evidence; the **topmost** `Authentication-Results` is read for a DMARC-aligned pass with `header.from=` checked against the real From. Severity follows the rules — danger only on a forgery rule or a server-recorded SPF/DKIM/DMARC failure (phishing/malware URL listings count); a new neutral `info` severity (no headline, no "do not open" footer, grey chip) when the sender is proved and every scoring rule is classifiable, or the score is all bulk; caution otherwise, **including a proved sender whose rules are unclassified** — unknown is never a reason to go quiet (326 messages in the store). `spamScoreOf()` and the banner now come from one analysis so they cannot disagree. Measured over 20,291 messages: of 1,131 flagged, 294 stay red, 672 caution, 165 quiet; the 2026-08-25 phishes stay red. Tests: all four samples verbatim + both ways the DMARC proof could be gamed. **Still owed:** sender trust — `allowlist.jsonc` already is the approved-sender store (`getAllowlist()` at `mailx-service/index.ts:1980`) and the viewer already has “Trust all senders at ‹domain›”, but nothing consults it when computing `trust`/`spamScore` (`store.ts:586`) and the banner has no inline trust action. Open Qs unchanged: trust key granularity (shared ESP domain vs Reply-To org vs Delivered-To tag), whether trust suppresses the whole banner or only the bulk/association half (leaning: never suppress forgery evidence), and auto-trust on reply/move-out-of-Junk. |
+| **C165** | PARTIAL | **Spam-banner false positives — trust the sender, do not tune the threshold** — full diagnosis and a "What shipped" section at [`docs/spam-false-positives.md`](spam-false-positives.md). Bob 2026-08-27: the red "This message is not what it says it is" banner fired on Constant Contact / Adobe MAX / Mokin newsletters he subscribed to, because `serverSpamVerdict()` consumed SpamAssassin’s **bottom line** (a delivery decision) to answer a different question (is this forged) and never parsed the `tests=` list in the same header. **Shipped v1.2.286–289:** `tests=` is classified into forgery / association / bulk / neutral / other; `X-Spam-Report` supplies each rule’s score and description so findings name their evidence; the **topmost** `Authentication-Results` is read for a DMARC-aligned pass with `header.from=` checked against the real From. Severity follows the rules — danger only on a forgery rule or a server-recorded SPF/DKIM/DMARC failure (phishing/malware URL listings count); a new neutral `info` severity (no headline, no "do not open" footer, grey chip) when the sender is proved and every scoring rule is classifiable, or the score is all bulk; caution otherwise, **including a proved sender whose rules are unclassified** — unknown is never a reason to go quiet (326 messages in the store). `spamScoreOf()` and the banner now come from one analysis so they cannot disagree. Measured over 20,291 messages: of 1,131 flagged, 294 stay red, 672 caution, 165 quiet; the 2026-08-25 phishes stay red. Tests: all four samples verbatim + both ways the DMARC proof could be gamed. Follow-up v1.2.288: proof of the From line accepts `DKIM_VALID_AU` and `ALL_TRUSTED` as well as DMARC (`provedBy` names which, in banner and chip tooltip) — Bob's OWN mail carries no `Authentication-Results` at all and was getting a caution banner over a URL it quoted. **John Levine 2026-08-27 confirms the direction:** DCC means bulk, "you have to whitelist the senders … I haven't tried to do a global one because everyone's different" — bulk is a fact about the mail, wanted is a fact about the reader. Server-side whitelisting is out for Bob (he whitelists `@bob.ma` and "that gets abused"), so the client list must key on evidence (DKIM-verified domain, Delivered-To tag), not the From line. **Still owed:** sender trust — `allowlist.jsonc` already is the approved-sender store (`getAllowlist()` at `mailx-service/index.ts:1980`) and the viewer already has “Trust all senders at ‹domain›”, but nothing consults it when computing `trust`/`spamScore` (`store.ts:586`) and the banner has no inline trust action. Open Qs unchanged: trust key granularity (shared ESP domain vs Reply-To org vs Delivered-To tag), whether trust suppresses the whole banner or only the bulk/association half (leaning: never suppress forgery evidence), and auto-trust on reply/move-out-of-Junk. |
 
 <a id="concerns"></a>
 

package/client/android-bootstrap.bundle.js

@@ -2308,7 +2308,10 @@ function analyzeServerSpam(input) {
   });
   const scored = (r) => !(r.score <= 0);
   const present = (cls) => rules.some((r) => r.cls === cls && scored(r));
-  const kind = dmarc.authFailures.length || present("forgery") ? "forgery" : present("association") ? "association" : present("other") ? "unclassified" : rules.length ? "bulk" : "unclassified";
+  const unknown = rules.filter((r) => r.cls === "other" && scored(r));
+  const unknownWeight = unknown.reduce((sum, r) => sum + r.score, 0);
+  const unknownCouldFlagAlone = !unknown.length ? false : !Number.isFinite(unknownWeight) || !Number.isFinite(threshold) ? true : unknownWeight >= threshold;
+  const kind = dmarc.authFailures.length || present("forgery") ? "forgery" : present("association") ? "association" : unknownCouldFlagAlone ? "unclassified" : rules.length ? "bulk" : "unclassified";
   const has = (name) => rules.some((r) => r.name === name);
   const provedBy = dmarc.pass ? "dmarc" : has("DKIM_VALID_AU") ? "dkim" : has("ALL_TRUSTED") ? "trusted" : "";
   const fromDomain = (bare(input.fromAddress).split("@")[1] || "").toLowerCase();
@@ -2530,6 +2533,7 @@ function spamScoreOf(input) {
     flagged: a.flagged,
     kind: a.kind,
     proved: a.proved,
+    provedBy: a.provedBy,
     reasons: a.reasons
   };
 }
@@ -2553,7 +2557,18 @@ var init_trust = __esm({
       /^SH_DBL/,
       /^SEM_URIBL/,
       /^SPAMHAUS_/,
-      /^RCVD_IN_(SBL|XBL|PBL|BL_SPAMCOP|SORBS|BRBL|VALIDITY|MSPIKE_[LZ])/
+      /^RCVD_IN_(SBL|XBL|PBL|BL_SPAMCOP|SORBS|BRBL|VALIDITY|MSPIKE_[LZ])/,
+      // Blocklist rules that happen to live under other prefixes. KAM_*URIBL_*
+      // ("Body contains URI listed in PCCC WILD RBL", 9.0) and SH_HBL_EMAILS
+      // ("Email address listed in email blocklist", 8.0) are the same kind of
+      // evidence as URIBL_SBL — somebody else's list, about a host or address
+      // the message mentions — and they were the bulk of what still cautioned a
+      // proved sender: a Business Insider newsletter, an MIT cancer-center
+      // invitation, a campaign mailing. KAM_INFOUSMEBIZ judges the TLD a domain
+      // lives in, which is guilt by neighbourhood.
+      /^KAM_(BODY_|FROM_)?URIBL/,
+      /^SH_HBL/,
+      /^KAM_INFOUSMEBIZ$/
     ];
     BULK_RULES = [
       /^DCC_/,
@@ -2581,7 +2596,14 @@ var init_trust = __esm({
       /^MPART_ALT_DIFF/,
       /^TVD_/,
       /^T_REMOTE_IMAGE$/,
-      /^UNPARSEABLE_RELAY$/
+      /^UNPARSEABLE_RELAY$/,
+      // Branded ESP click-trackers. Not open redirectors (URI_PHP_REDIR and
+      // friends stay unclassified): these are the sending platform's own
+      // tracking domain, which is mass-mail machinery by definition.
+      /^GB_AWSTRACK/,
+      /AWSTRACK/,
+      /^GB_SENDGRID/,
+      /^GB_MAILCHIMP/
     ];
     NEUTRAL_RULES = [
       /^BAYES_/,