@bobfrankston/rmfmail 1.2.285 → 1.2.287

package/.commitmsg

@@ -1,37 +1,33 @@
-Close the window where three daemons could start at once
+Grammar squiggles: stop inventing words, stop underlining white space
 
-"This does not look like a single instance ... but not one of the windows came
-to the front." It wasn't one. Three daemons were running:
+Three complaints from one compose session (Bob 2026-08-27, forwarding a
+message), all of them the checker arguing with prose that was fine:
 
-  PID 1452    23:52:15.9
-  PID 34012   23:52:19.9   <- the only one instance.json knew about
-  PID 94964   23:52:25.0
+1. "keypoint". Harper's CompoundNouns rule fires on any noun+noun pair whose
+   joined form is in its dictionary, so "One key point in my recent efforts"
+   came back as an error suggesting a word he did not write. Disabled in the
+   daemon next to AvoidCurses — same objection as that one: the tool
+   proofreads, it does not get a vote on word choice. Its siblings (MergeWords,
+   MakeupCompoundNoun, OvertimeCompoundNoun, ThereAfterCompound,
+   EasyGoingCompoundAdjective) are named in a comment there, one line each to
+   turn off if they start nagging.
 
-Launched 4 and 5 seconds apart, each with its own window, and replace-on-launch
-could never clean them up because it only ever knew about the last one to
-register. Win+4 could not raise "the" window because there were three of them
-and nothing owned the taskbar identity.
+2. A blue dotted line across two blank lines. blockText concatenated a block's
+   text nodes with no separator, so the last word of a paragraph was glued to
+   the first word of the next: "…of the network" + "To: x@y.com" reached Harper
+   as "networkTo", and it reported — correctly, for the string it was given —
+   that the word should be split. The span then covered a junction that on
+   screen is two empty lines, and getClientRects painted a full-width underline
+   over the whitespace. Verified both halves against the real engine: gluing
+   produces `networkTo` and `FrankstonSubject` lints that vanish once the
+   breaks are there. blockText now emits "\n" at every line-breaking element,
+   and spanToRange refuses any span whose ends sit in different line boxes, so
+   no squiggle can ever be drawn across a paragraph boundary again.
 
-The hole: the launcher spawns a DETACHED daemon and exits immediately, while
-the daemon does not register itself until it has built the store, started the
-popout server and opened its window — several seconds later. Every launch
-arriving inside that gap reads instance.json, sees nothing alive, and starts
-another daemon. Three launches within ten seconds, three daemons.
+3. Lints on the sentence being typed. The unfinished sentence under the caret
+   is now exempt — a thought that is not out yet is not an error.
 
-The launcher now claims the slot for its child the moment spawn returns, before
-it exits. The child is alive from that instant and pidIsMailx is a liveness
-check, so the claim is valid straight away; when the daemon later reaches its
-own registration it writes the same PID again. A launch landing mid-boot now
-takes the activate path added in the previous release and drops
-pending-activate.json, which the daemon consumes as soon as it starts watching
-— which is exactly why that path checks for a request that arrived while it was
-booting.
-
-Also: startedAt was recomputed on EVERY write, and every popup and popout
-window rewrites this file, so the field reported the last window event rather
-than when the daemon started. It now survives childPids updates. That one cost
-a wrong turn while diagnosing this — the file claimed the daemon had started
-seconds ago when it had been up for minutes.
-
-Ships separately from the activate fix in 1.2.284 — verified that release did
-not already contain it.
+The pure helpers (collectBlocks, blockText, lineBox, caretOffset,
+sentenceStart, spanToRange) moved to module scope and are exported, because
+they hold the decisions worth testing; tests/dom/harper-lint.test.ts covers all
+three rules in jsdom, using the forward markup client/app.ts actually builds.

package/TODO.md

@@ -144,6 +144,7 @@ None of these are truly showstoppers — mailx is usable — but they're the nex
 
 | # | Status | Item |
 |---|---|---|
+| **C165** | PARTIAL | Spam banner fires on subscribed newsletters (Constant Contact / Adobe / Mokin). **Rule classification + DMARC veto shipped v1.2.286** — severity now comes from what the `tests=` rules test, not from the score; red is reserved for forgery. **Still owed: sender trust** via `allowlist.jsonc` + an inline trust action on the banner. **Do not tune the `threshold/2` chip floor.** [`docs/spam-false-positives.md`](spam-false-positives.md). [S] |
 | [**S56**](#ext56) | OPEN | Row-objects own the preview pane — the last open slice of the old "S1 local-first refactor" umbrella. All S1 sub-shipments (tombstones, opaque UUIDs, stable row UUID, Message-ID move-detection, pink rows, viewer-reacts-to-list-replace, unified-inbox pink) are in Done under their version tags; this is what remains. ✓ (can start the `focus()`/`unfocus()` seam; full migration is more than one session) |
 | [**S9**](#ext9) | PARTIAL | Predownloading gaps — 60s prefetch landed v1.0.321; per-account prefetch guard already in mailx-imap (`prefetchingAccounts` Set). Remaining: batch body prefetch (C24) for 10–50× speedup. |
 | [**S51**](#ext51) ❓ | PARTIAL | Calendar Thunderbird-style sidebar. Sidebar UI + visible-by-default + 4-column grid reflow shipped v1.0.375/376. **Still owed**: Google Calendar live fetch via service-side proxy. Open Q: keep the full-screen calendar modal, or retire in favor of sidebar-only? |
@@ -229,6 +230,7 @@ Items blocked on a design decision. Short-form question here; full context in th
 | **C129** | OPEN | **Android prod/dev split via filename/URL channel** — User 2026-05-09 (revised): no separate MAUI project. Same source tree builds; output filenames distinguish channels: Phase 1 (start now) writes `rmfmaildev.apk` alongside today's `mailx-maui.apk`. Phase 2 (when a release is declared) switches to `rmfmail-<version>.apk` versioned + `rmfmaildev.apk` for latest, with promotion = re-pointing `rmfmail.apk` at a chosen version. Single APK contents per build; channel is runtime setting in AppUpdater (Settings toggle for which URL to poll). Side-by-side install on one device NOT supported by this design (single Android package id) — that's a separate need. See `docs/prod-android.md` for full plan. Phase 1 ~30 min; full flow ~2.5 hours when activated. |
 | **C130** | OPEN | **Reset prefetch error budget after fresh sync** — Android: `errors`/`failedThisSession` accumulate within a prefetch session. Already resets on next syncAll cycle (since each `prefetchBodies(account)` call is fresh). Verify this works in practice via logit; if a long sync session burns through budget and stays stuck, add an explicit reset on `bodyAvailable` events that fire while prefetch is still running. |
 | **C164** | RESEARCHED | **Thundermail / JMAP provider** — full protocol survey at [`docs/thundermail-jmap.md`](thundermail-jmap.md), probed live 2026-08-17. Thundermail is Stalwart, so there is no proprietary API: IMAP 993 / SMTP 465 work **today with zero code** (an `accounts.jsonc` entry and nothing else), CardDAV `/dav/card` + CalDAV `/dav/cal` are http-basic, and JMAP is live at `https://mail.thundermail.com/jmap/` advertising core / mail / submission / calendars / contacts / filenode / sieve / quota plus **WebSocket push** (`supportsPush: true`). JMAP is the attractive bet: it kills the IDLE lane, the 5-lane connection semaphore and the Dovecot 20-conn ceiling, needs no transport injection (one path for Android and desktop, cf. C125), and `Email/changes` is native set-difference sync. **From address: resolved — good.** The 15-alias cap is `@thundermail.com`/`@tb.pro` only; custom domains get unlimited aliases plus a catch-all (empty-name alias), and with catch-all on you can send from any prefix unregistered — their own example is the per-vendor pattern already used on `bob.ma`. mailx needs ~nothing here (per-message From identity shipped v1.2.201); only Stalwart-side `MAIL FROM`/DKIM validation for an unregistered prefix is untested. **Still blocking/staged:** JMAP-calendars and `filenode` are unratified drafts; OAuth `scopes_supported` covers mail only (no calendars/contacts scope); invite-only beta so there is nothing to test against; and **no Drive equivalent** — Thunderbird Send is one-shot file *sharing*, not a listable revisable store, so mailx's shared-config dependency (`accounts.jsonc` / `contacts.jsonc` / preferences / allowlist on GDrive) has no home. **C161 is therefore the prerequisite, not JMAP** — the `_rmfmail` hidden-IMAP-folder state channel removes the Drive dependency for every account type at once and keeps working unchanged if JMAP lands later. Until C161 ships, adopting Thundermail is a half-migration that trades a Google dependency for a Thunderbird one. |
+| **C165** | PARTIAL | **Spam-banner false positives — trust the sender, do not tune the threshold** — full diagnosis and a "What shipped" section at [`docs/spam-false-positives.md`](spam-false-positives.md). Bob 2026-08-27: the red "This message is not what it says it is" banner fired on Constant Contact / Adobe MAX / Mokin newsletters he subscribed to, because `serverSpamVerdict()` consumed SpamAssassin’s **bottom line** (a delivery decision) to answer a different question (is this forged) and never parsed the `tests=` list in the same header. **Shipped v1.2.286 (mailx-types 0.1.64, mailx-store 0.1.96):** `tests=` is classified into forgery / association / bulk / neutral / other; `X-Spam-Report` supplies each rule’s score and description so findings name their evidence; the **topmost** `Authentication-Results` is read for a DMARC-aligned pass with `header.from=` checked against the real From. Severity follows the rules — danger only on a forgery rule or a server-recorded SPF/DKIM/DMARC failure (phishing/malware URL listings count); a new neutral `info` severity (no headline, no "do not open" footer, grey chip) when the sender is proved and every scoring rule is classifiable, or the score is all bulk; caution otherwise, **including a proved sender whose rules are unclassified** — unknown is never a reason to go quiet (326 messages in the store). `spamScoreOf()` and the banner now come from one analysis so they cannot disagree. Measured over 20,291 messages: of 1,131 flagged, 294 stay red, 672 caution, 165 quiet; the 2026-08-25 phishes stay red. Tests: all four samples verbatim + both ways the DMARC proof could be gamed. **Still owed:** sender trust — `allowlist.jsonc` already is the approved-sender store (`getAllowlist()` at `mailx-service/index.ts:1980`) and the viewer already has “Trust all senders at ‹domain›”, but nothing consults it when computing `trust`/`spamScore` (`store.ts:586`) and the banner has no inline trust action. Open Qs unchanged: trust key granularity (shared ESP domain vs Reply-To org vs Delivered-To tag), whether trust suppresses the whole banner or only the bulk/association half (leaning: never suppress forgery evidence), and auto-trust on reply/move-out-of-Junk. |
 
 <a id="concerns"></a>
 

package/client/android-bootstrap.bundle.js

@@ -2226,18 +2226,141 @@ function bare(addr) {
   const m = (addr || "").match(/[^\s<>,;]+@[^\s<>,;]+/);
   return (m ? m[0] : addr || "").toLowerCase().replace(/[.,;]+$/, "");
 }
-function serverSpamVerdict(lines) {
-  const flag = header(lines, "x-spam-flag");
-  const status = header(lines, "x-spam-status");
-  if (!/^yes/i.test(flag) && !/^yes/i.test(status))
+function classifyRule(name, dmarcFailed) {
+  if (dmarcFailed && name === "DKIM_INVALID")
+    return "forgery";
+  for (const re of FORGERY_RULES)
+    if (re.test(name))
+      return "forgery";
+  for (const re of ASSOCIATION_RULES)
+    if (re.test(name))
+      return "association";
+  for (const re of NEUTRAL_RULES)
+    if (re.test(name))
+      return "neutral";
+  for (const re of BULK_RULES)
+    if (re.test(name))
+      return "bulk";
+  return "other";
+}
+function dmarcProof(input) {
+  const auth = headerAll(input.headerLines, "authentication-results")[0] || "";
+  if (!auth)
+    return { pass: false, failed: false, authFailures: [], domain: "", policy: "" };
+  const authFailures = [
+    /\bspf=(fail|softfail)\b/i.test(auth) ? "SPF" : "",
+    /\bdkim=(fail|permerror)\b/i.test(auth) ? "DKIM" : "",
+    /\bdmarc=(fail|permerror)\b/i.test(auth) ? "DMARC" : ""
+  ].filter(Boolean);
+  const failed = authFailures.includes("DMARC");
+  if (!/\bdmarc=pass\b/i.test(auth))
+    return { pass: false, failed, authFailures, domain: "", policy: "" };
+  const stated = (auth.match(/header\.from=([^\s;,]+)/i)?.[1] || "").toLowerCase();
+  const policy = (auth.match(/policy\.dmarc=([^\s;,]+)/i)?.[1] || "").toLowerCase();
+  const fromDomain = (bare(input.fromAddress).split("@")[1] || "").toLowerCase();
+  if (stated && fromDomain && stated !== fromDomain && !stated.endsWith("." + fromDomain) && !fromDomain.endsWith("." + stated)) {
+    return { pass: false, failed, authFailures, domain: "", policy };
+  }
+  return { pass: true, failed: false, authFailures, domain: stated || fromDomain, policy };
+}
+function unfold(value) {
+  return value.replace(/\r?\n/g, "");
+}
+function parseSpamReport(report) {
+  const out = /* @__PURE__ */ new Map();
+  let current = null;
+  for (const raw of report.split(/\r?\n/)) {
+    const line = raw.replace(/^[ \t]*\*?[ \t]*/, "").trimEnd();
+    if (!line)
+      continue;
+    const start = line.match(/^(-?\d+(?:\.\d+)?)\s+([A-Z0-9_]{3,})\s*(.*)$/);
+    if (start) {
+      current = { score: Number(start[1]), description: start[3].trim() };
+      out.set(start[2], current);
+      continue;
+    }
+    if (current && !line.startsWith("["))
+      current.description += " " + line;
+  }
+  for (const rule of out.values())
+    rule.description = rule.description.replace(/\s+/g, " ").replace(/^(BODY|RAW|HEADER|URI|RBL):\s*/i, "").trim();
+  return out;
+}
+function analyzeServerSpam(input) {
+  const status = unfold(header(input.headerLines, "x-spam-status"));
+  const flag = header(input.headerLines, "x-spam-flag");
+  if (!status && !flag)
+    return null;
+  const score = Number(status.match(/score=(-?[\d.]+)/)?.[1]);
+  const threshold = Number(status.match(/required=(-?[\d.]+)/)?.[1]);
+  const flagged = /^yes/i.test(status) || /^yes/i.test(flag);
+  const dmarc = dmarcProof(input);
+  const report = parseSpamReport(header(input.headerLines, "x-spam-report"));
+  const names = (status.match(/tests=([\s\S]*?)(?=\s+[a-z_]+=|$)/)?.[1] || "").replace(/\s+/g, "").split(",").map((n) => n.toUpperCase()).filter(Boolean);
+  const rules = names.map((name) => {
+    const found = report.get(name);
+    return {
+      name,
+      score: found ? found.score : NaN,
+      description: found ? found.description : "",
+      cls: classifyRule(name, dmarc.failed)
+    };
+  });
+  const scored = (r) => !(r.score <= 0);
+  const present = (cls) => rules.some((r) => r.cls === cls && scored(r));
+  const kind = dmarc.authFailures.length || present("forgery") ? "forgery" : present("association") ? "association" : present("other") ? "unclassified" : rules.length ? "bulk" : "unclassified";
+  const reasons = rules.filter((r) => r.cls !== "neutral" && scored(r)).sort((a, b) => (b.score || 0) - (a.score || 0)).slice(0, 3).map((r) => {
+    const num = Number.isFinite(r.score) ? `${r.score.toFixed(1)} ` : "";
+    return r.description ? `${num}${r.name} (${r.description})` : `${num}${r.name}`;
+  });
+  return {
+    score,
+    threshold,
+    flagged,
+    rules,
+    kind,
+    proved: dmarc.pass,
+    provedDomain: dmarc.domain,
+    provedPolicy: dmarc.policy,
+    bayesHam: rules.some((r) => r.name === "BAYES_00" || r.name === "BAYES_01"),
+    authFailures: dmarc.authFailures,
+    reasons
+  };
+}
+function serverSpamVerdict(input) {
+  const a = analyzeServerSpam(input);
+  if (!a || !a.flagged)
     return null;
-  const score = status.match(/score=([-\d.]+)/)?.[1];
-  const required = status.match(/required=([-\d.]+)/)?.[1];
+  const numbers = Number.isFinite(a.score) && Number.isFinite(a.threshold) ? `SpamAssassin score ${a.score} of ${a.threshold}` : "flagged by SpamAssassin";
+  const why = a.reasons.length ? ` \u2014 ${a.reasons.join("; ")}` : "";
+  const bayes = a.bayesHam ? "; your own trained filter puts it at 0-1% spam (BAYES_00)" : "";
+  if (a.kind === "forgery")
+    return {
+      id: "server-spam-verdict",
+      severity: "danger",
+      text: "Your mail server classified this as spam, and the rules that fired test who sent it.",
+      detail: a.authFailures.length ? `${a.authFailures.join(" and ")} authentication failed at your server. ${numbers}${why}` : `${numbers}${why}`
+    };
+  if (a.proved && a.kind !== "unclassified")
+    return {
+      id: "server-spam-verdict",
+      severity: "info",
+      text: `Your mail server scored this as spam, but ${a.provedDomain} proved it sent this.`,
+      detail: `DMARC pass, aligned${a.provedPolicy ? `, policy ${a.provedPolicy}` : ""}. ${numbers}${why}${bayes}`
+    };
+  if (a.kind === "bulk")
+    return {
+      id: "server-spam-verdict",
+      severity: "info",
+      text: "Your mail server rates this bulk mail \u2014 nothing that scored tests who sent it.",
+      detail: `${numbers}${why}${bayes}`
+    };
+  const proof = a.proved ? `${a.provedDomain} passed DMARC, so the From line is genuine. ` : "";
   return {
     id: "server-spam-verdict",
-    severity: "danger",
+    severity: "caution",
     text: "Your mail server classified this as spam before delivering it.",
-    detail: score && required ? `SpamAssassin score ${score}, threshold ${required}` : status || flag || "X-Spam-Flag: YES"
+    detail: `${proof}${numbers}${why}${bayes}`
   };
 }
 function selfSpoof(input) {
@@ -2336,14 +2459,14 @@ function redirectorLink(bodyHtml) {
 }
 function assessMessageTrust(input) {
   const findings = [
-    serverSpamVerdict(input.headerLines),
+    serverSpamVerdict(input),
     selfSpoof(input),
     relayAuthMismatch(input),
     zeroWidthObfuscation(input.bodyText || ""),
     hiddenLinkOverlay(input.bodyHtml || ""),
     redirectorLink(input.bodyHtml || "")
   ].filter(Boolean);
-  const rank = { danger: 0, caution: 1 };
+  const rank = { danger: 0, caution: 1, info: 2 };
   return findings.sort((a, b) => rank[a.severity] - rank[b.severity]);
 }
 function relayAuthMismatch(input) {
@@ -2381,20 +2504,80 @@ function zeroWidthObfuscation(bodyText) {
     detail: `${count} zero-width characters hidden inside words \u2014 legitimate mail has no reason to do this`
   };
 }
-function spamScoreOf(headerLines) {
-  const status = header(headerLines, "x-spam-status");
-  if (!status)
-    return null;
-  const score = Number(status.match(/score=([-\d.]+)/)?.[1]);
-  const threshold = Number(status.match(/required=([-\d.]+)/)?.[1]);
-  if (!Number.isFinite(score) || !Number.isFinite(threshold))
+function spamScoreOf(input) {
+  const a = analyzeServerSpam(input);
+  if (!a || !Number.isFinite(a.score) || !Number.isFinite(a.threshold))
     return null;
-  return { score, threshold, flagged: /^yes/i.test(status) };
+  return {
+    score: a.score,
+    threshold: a.threshold,
+    flagged: a.flagged,
+    kind: a.kind,
+    proved: a.proved,
+    reasons: a.reasons
+  };
 }
-var OVERLAY_MIN_EDGE_PX, RELAY_IDENTITY_HEADERS, ZERO_WIDTH_MIN_OCCURRENCES;
+var FORGERY_RULES, ASSOCIATION_RULES, BULK_RULES, NEUTRAL_RULES, OVERLAY_MIN_EDGE_PX, RELAY_IDENTITY_HEADERS, ZERO_WIDTH_MIN_OCCURRENCES;
 var init_trust = __esm({
   "packages/mailx-types/trust.js"() {
     "use strict";
+    FORGERY_RULES = [
+      /^SPF_(HELO_)?(FAIL|SOFTFAIL)$/,
+      /^(KAM_)?DMARC_(FAIL|REJECT|QUAR)/,
+      /^DKIM_ADSP_(ALL|DISCARD|NXDOMAIN)$/,
+      /^FORGED_/,
+      /SPOOF/,
+      /_PHISH/,
+      /PHISHING/,
+      /_MALW/,
+      /MALWARE/
+    ];
+    ASSOCIATION_RULES = [
+      /^URIBL_(SBL|BLACK|RED|DBL|ABUSE)/,
+      /^SH_DBL/,
+      /^SEM_URIBL/,
+      /^SPAMHAUS_/,
+      /^RCVD_IN_(SBL|XBL|PBL|BL_SPAMCOP|SORBS|BRBL|VALIDITY|MSPIKE_[LZ])/
+    ];
+    BULK_RULES = [
+      /^DCC_/,
+      /^PYZOR_/,
+      /^RAZOR2_/,
+      /^MAILING_LIST/,
+      /^LIST_/,
+      // KAM_* by name only, never the whole family: it holds marketing and list
+      // rules alongside scam-CONTENT rules (KAM_BENEFICIARY is worth 10.0,
+      // KAM_FAKE_NORTONLOW 6.5), and calling those bulk would quiet the loudest
+      // true positives in the store.
+      /^KAM_(BODY_)?MARKETINGBL/,
+      /^KAM_UNSUB/,
+      /^KAM_TRACKIMAGE$/,
+      /^KAM_REALLYHUGEIMGSRC$/,
+      /^KAM_DMARC_STATUS$/,
+      /^KAM_COUPON/,
+      /^HTML_/,
+      /^MIME_HTML/,
+      /^WORD_INVIS$/,
+      /^URIBL_(GREY|CSS|BLOCKED|CT_SURBL)/,
+      /^SH_BODYURI/,
+      /^HEADER_FROM_DIFFERENT_DOMAINS$/,
+      // every ESP on earth trips this
+      /^MPART_ALT_DIFF/,
+      /^TVD_/,
+      /^T_REMOTE_IMAGE$/,
+      /^UNPARSEABLE_RELAY$/
+    ];
+    NEUTRAL_RULES = [
+      /^BAYES_/,
+      /^SPF_(PASS|NONE|HELO_(PASS|NONE|NEUTRAL))$/,
+      /^DKIM_(SIGNED|VALID|VALID_AU|VALID_EF|INVALID)$/,
+      /^DKIMWL_/,
+      /^RCVD_IN_(DNSWL|MSPIKE_H|IADB)/,
+      /^SHORTCIRCUIT$/,
+      /^AWL$/,
+      /^NO_RELAYS$/,
+      /^ALL_TRUSTED$/
+    ];
     OVERLAY_MIN_EDGE_PX = 2e3;
     RELAY_IDENTITY_HEADERS = [
       "x-source-auth",