nondisposable 0.4.0 → 0.6.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9d2c949362fe7f9506247f3ae809ced6dca5a842086fa540346c8e89ea5dc5fd
-  data.tar.gz: a13fa7bec1531089fe68895529ecab5d9f05a2653292e2e39a1f4dfdeb818cf7
+  metadata.gz: 96ee85d55e05d52d472cb9a4b9897e318499282c35818579621daec1bcd7bf9e
+  data.tar.gz: f633e949328cb2972dfdff7c44f33072b29dae4b5bd95ab116af24ec504f676c
 SHA512:
-  metadata.gz: 696ce476987cdd1a90008b8065a9f8af9d6c19035e42cbd3cb0b03cf9b36ec0bf3fd9158d474552a8d2729e4d454ec19ac7ee31e24af844f17a42e27346532bc
-  data.tar.gz: 1a220333975bbd3ba6229a29172e42b4e9c91ddc6cc4ee97aca417ab743c3756c5ff024f70d57963f0c7643378d4d1a4153135632d7f65cd4a96068bb3bb3dc0
+  metadata.gz: 71c6fbbfcac0534c320decc1cee69147497984a9dc12f19569a2231eacc192a5a2cc328e88decdc5e8f692384bf8ae39705a9c77ff134fb5cccc02fc455c133d
+  data.tar.gz: 8abdd8d29e8bfeb5f51da7cc98a6b213a8af428b561a8be6a9d2e84bce470f96a5e2c5bbeeff2f9a4e0dda0d719f5e696562c97b9643278a5d92a5c90ae8ff06

data/CHANGELOG.md

@@ -1,3 +1,62 @@
+## [0.6.1] - 2026-10-17
+
+### Fixed
+
+- **A populated table is no longer replaced with the bundled list by an update** (an explicit `rollback` still can; see below). In 0.6.0, when a PRIMARY source with a bundled list (the default `disposable-email-domains`) had never been fetched successfully, `DomainListUpdater` put the list bundled with the gem in its place and published it, even over a populated table. **Who is affected:** any host with the `nondisposable_source_snapshots` table whose domain table was populated before the default source had an accepted snapshot of its own. Typically that means an upgrade from 0.4/0.5, with `Sources::DEFAULT` or a multi-source configuration such as `Sources::RECOMMENDED`. If the default source failed on such a host's first update, the table was replaced by the bundled snapshot (plus any other primary that succeeded), and every domain added upstream since the gem's release stopped being blocked until the next successful run. `update` returned false, but the list had already shrunk. Now the bundled stand-in is used **only on an empty table** (a fresh install). With a populated table, publishing is skipped with an error log and the table is kept as it is, until every primary has an accepted snapshot of its own. **The deliberate exception is `DomainListUpdater.rollback`:** withdrawing a poisoned list must always succeed, so there the bundled list still stands in for a primary without a snapshot, whatever the table holds. The emptiness check also bypasses the query cache, so a stale cached answer can never let the bundled list replace rows it never saw. Reported by a codex review in CarHey.
+
+## [0.6.0] - 2026-10-17
+
+Ships everything listed under 0.5.0 too: 0.5.0 was never released on its own.
+
+More than one blocklist, without trusting any of them more than it has earned. Triggered by a real incident: signups from `mail-burn.com`, a live 10-minute inbox the default list did not carry. The measurement behind every default is in `docs/evidence/2026-10-17-multi-source-measurement.md`. With the default configuration you still get the one list you had before.
+
+### Added
+
+- **Multiple blocklist sources** (`config.sources`, default `Nondisposable::Sources::DEFAULT`, which is today's single list). Each `Nondisposable::Source` has a `trust` (`:primary` blocks everything it lists; `:corroborating` needs `config.corroboration_threshold`, default 2, independent origins) and declares its `derived_from` lineage, so a list that copies another never counts as a second vote. Measured sources ship as constants (`DISPOSABLE_EMAIL_DOMAINS`, `FAKEFILTER`, `DISPOSABLE`, `MAILCHECKER`, `GROUNDCAT`, `WESBOS`) with licences and lineage.
+- **`Nondisposable::Sources::RECOMMENDED`**: the default list plus fakefilter, both primary. +3,275 domains on 2026-10-17, none of them a known real provider. It does **not** include disposable/disposable as a primary; see the README for the 84,341 domains, real mail providers among them, that would add.
+- **Source snapshots** (`nondisposable_source_snapshots`): every fetch of every source is recorded (status, time, sha256, count, list), and the stored list is rebuilt from each source's last good snapshot in one transaction, under a database advisory lock on PostgreSQL/MySQL (process lock on SQLite). Seeding takes the same lock.
+- **`rails generate nondisposable:upgrade`**: an additive migration for existing installs. Until it is migrated (or after rolling it back), the updater keeps the single-source behaviour, so deploying before migrating is safe. Fresh installs get the table from `nondisposable:install`.
+- **Update guards**: streamed size cap (`max_bytes`, 8 MB) and wall-clock `deadline` (30 s) per source; no redirects followed; https only; a snapshot is refused when over 1% of its lines are not domains, when it is below `minimum_count`, when it is under half the largest snapshot still retained or over three times the last good one, or when it equals a rolled-back snapshot. Counts are of unique canonical domains. Additions and removals are logged per source. Sizes count storable domains only (no junk, public suffixes or real providers). The list is never published empty; a never-fetched default source stands in its bundled list, and any other never-fetched primary blocks publishing. Rolled-back payloads stay refused permanently; failures never prune the last good snapshots.
+- **Concurrency limits (documented):** on SQLite the publish lock is per process, so run the refresh job from a single runner. During a rolling deploy from 0.5, pause the refresh job until every process runs 0.6 (old processes replace the table without the lock).
+- **`DomainListUpdater.reset_baseline(source_name)`**: accept a genuine upstream shrink below half of the source's high-water mark.
+- **`DomainListUpdater.publish`** (rebuild from snapshots without downloading, e.g. after a config change) and **`DomainListUpdater.rollback(source_name)`** (withdraw a source's newest snapshot and republish).
+- **Public Suffix List guard**: a bundled snapshot of publicsuffix.org's list (MPL 2.0, unmodified) and `Nondisposable::PublicSuffix`. Public suffixes (`com.br`, `co.uk`, `it.com`, `ddns.net`) are never stored, never accepted from `additional_domains`, and never used as a parent-match candidate, so one bad upstream line cannot block a whole namespace. Refresh with `rake nondisposable:psl:update`.
+- **`data/never_disposable.txt`**: real providers and ISPs that upstream lists were found to carry, plus paid mailbox providers; unioned at runtime with `data/email_providers.txt`. Each entry protects its subdomains (`bak.rr.com` under `rr.com`). An upstream source listing one is logged as possible poisoning.
+- **Privacy relays as a policy** (`config.block_privacy_relays`, default `false`, list in `data/privacy_relays.txt`): Hide My Email, DuckDuckGo, Firefox Relay, SimpleLogin/Proton Pass and addy.io are allowed even when a source lists them, or refused even when none does.
+- **`Nondisposable::Domain.normalize`**: one canonical form (stripped, downcased, no trailing dot, IDN to punycode) used by the checks and by list ingestion. `Bücher.de.` and `xn--bcher-kva.de` are the same domain.
+
+### Changed
+
+- ⚠️ **`DisposableDomain.disposable?` / `Nondisposable.disposable?` answer `true` for a name longer than DNS allows** (`Domain.too_long?`: over 253 characters or a label over 63 once canonical, or over 1,024 code points of raw input after stripping invisible characters). Such a name can never be deliverable, and a check that keeps junk out must not call it fine. Names exactly at the limits are judged normally. The validator rejects them earlier with `invalid_tld_error_message`.
+- ⚠️ **Precedence**: one table, applied both when checking and when building the list: `excluded_domains` > `never_disposable` > `additional_domains` > privacy-relay policy > sources. Before, `additional_domains` was checked first, so a domain in both `additional_domains` and `excluded_domains` was **blocked**; it is now **allowed**. `additional_domains` can no longer block a real provider listed in `never_disposable` (ignored with a warning).
+- ⚠️ Two domains the default upstream list carries are **no longer blocked**: `sify.com` (Indian ISP) and `21cn.com` (China Telecom webmail).
+- Entries that cannot be DNS names (labels over 63 characters, names over 253, IP addresses, HTML) are dropped from downloaded lists. The single-source path also refuses a payload with more than 1% such lines instead of storing it.
+- ⚠️ **The parent-domain walk has no depth cap any more**: it checks every parent down to the registrable domain, so `a.b.c.d.tempmail.com` is blocked when `tempmail.com` is listed (or in `additional_domains`). It used to stop after 3 parent labels, which let whoever controls a listed domain escape with a fourth. Still a single `WHERE name IN (...)` query; DNS caps a name at 253 characters. It never climbs above the registrable domain, so a public suffix (`co.uk`), or a name above a wildcard suffix (`sch.uk`), can never match. `DisposableDomain::PARENT_MATCH_DEPTH` is kept but unused.
+- The table is updated by writing only the difference (still in one transaction), instead of deleting and reinserting every row.
+
+## [0.5.0] - Not released separately (shipped in 0.6.0)
+
+Lookalike detection you can actually switch on. Replaying the 0.4.0 rule ("one edit from a provider") against the Tranco top 1M flagged 707 domains, and many are large real organisations whose staff would have been refused at signup: `ge.com`, `ml.com`, `mi.com`, `se.com` → `me.com`; `ea.com`, `aa.com` → `ya.com`; `bol.com`, `aon.com`, `uol.com` → `aol.com`; `one.com` → `ono.com`; `hive.com`, `life.com` → `live.com`; `grail.com`, `pmail.com` → `gmail.com`; `web.dev` → `web.de`; `yahoo.com.au` → `yahoo.com.ar`. Offline, nothing can tell those apart from `gmial.com` with certainty, so a guess no longer refuses anybody on its own.
+
+### Changed
+
+- **Behavior change for `reject_lookalike_domains = true`: it is now softer, never stricter.** Precedence, first match wins: an exact provider is accepted → an impossible TLD with a nameable correction (`gmail.con`, `gmailmcom`) is **refused**, whether or not `check_tld` is on → a reviewed typo from the new bundled `data/known_typo_domains.txt` (41 entries, e.g. `gmial.com`, `hotmial.com`, `yaho.com`) is **refused** → any other lookalike gets the same "Did you mean …?" error, but **confirmable**: it is skipped when the record says the user confirmed the address as typed. Disposable providers are still refused first, blocked TLDs are still refused after a confirmation, and a host's `additional_email_providers` outranks the known-typo list.
+- **`Nondisposable.suggestion_for` suggests less, and better.** Provider names of four characters or fewer (`me`, `ya`, `aol`, `mail`, `live`…) only take edits after the name, and every edit must look like a finger slip on QWERTY (swap, dropped letter other than the first, doubled letter, extra vowel, neighbouring key, accented same key). On a retrospective 30% holdout of the labelled corpus (split chosen before tuning but committed with the tuned filters, so not pre-registered): precision 8% → 45%, recall 81% of reviewed typos. Reviewed typos are always corrected, even where the ranker alone would stay silent (`jmail.com`).
+- `lookalike_distance = 0` now switches off the whole lookalike layer, known-typo list included.
+- The confirmable error is added with type `:lookalike_domain` (and `suggestion:` in its details) so a form can tell it apart: `errors.of_kind?(:email, :lookalike_domain)`. The message is unchanged and still names the correction; hard lookalike errors keep their plain-message form.
+- Domains are normalised the same way on every path (`Nondisposable.disposable?`, `additional_domains` / `excluded_domains`, the validator, the TLD check, suggestions, the provider and known-typo lists): stripped, lowercased, trailing dot removed, fullwidth and ideographic dots read as dots, NFKC plus the invisible characters UTS #46 drops (soft hyphen, zero-width space), unicode labels converted to punycode. `x@ｍａｉｌｉｎａｔｏｒ.com` and `x@foo．tk` no longer slip past their list entries. `user@tempmail.com.` and `user@bücher.de` (listed as `xn--bcher-kva.de`) are now caught. The local part is never touched.
+- **Stored rows and 0.4.0 compatibility.** List rows are stored in the canonical (punycode) form, and lookups also match a unicode row written by 0.4.0, under the same exclusion rules (a canonical `excluded_domains` entry still wins over a legacy `bücher.de` row and its subdomains). So that a mixed-version deploy, or a rollback to 0.4.0, never silently unblocks anything, a domain added in unicode through `additional_domains` is stored **twice**: `xn--bcher-kva.de` and `bücher.de`. The row count includes these twins. ⚠️ **Rollback to 0.4.0: unicode entries from downloaded lists are stored only in punycode, so a 0.4.0 reader would not match them; no current source ships raw unicode** (all six measured lists send IDNs as `xn--`, 2026-10-17). (A 0.4.0 reader also matches `additional_domains` in memory, so the twin row matters mostly for a rollback in which the config changed too.)
+- Normalisation is bounded on hostile input. NFKC runs only on names of up to 1,024 code points, and not on a run of more than 30 combining marks (not stream-safe, UAX #15). The DNS limits (253 per name, 63 per label) apply to the **normalised** result, so a valid label typed in decomposed form (`e` + combining acute) still matches.
+- **Behavior change: a domain longer than DNS allows is refused** by the validator, with `invalid_tld_error_message`, whatever is switched on: over 253 characters, a label over 63 (judged on the punycode form), or over 1,024 code points before normalisation. No mail can reach such an address and no list can hold it, so it can never be "not disposable". Invisible padding (soft hyphens and the like) is stripped first, so it can neither trigger this nor hide a listed domain. Values without an `@` are still left to `format:`. `Nondisposable::Domain.too_long?(domain)` exposes the check.
+
+### Added
+
+- **`config.reject_lookalike_domains = :confirmable`**, the explicit name for the mode above. Strings (`"confirmable"`, `"true"`, `"false"`) are accepted for YAML/ENV config; any other value raises `ArgumentError` at boot instead of silently meaning "on".
+- **`Nondisposable::ConfirmableEmail`**: `include` it and call `confirmable_email :email` to get a virtual, boolean-cast `email_lookalike_confirmed` attribute for a "Yes, my email is right" checkbox. It is reset after every save, so a later lookalike is asked about again. Without the concern, any `<attribute>_lookalike_confirmed` method returning exactly `true` works.
+- **`Nondisposable::KnownTypos`** and `data/known_typo_domains.txt`: the only lookalikes refused outright, each with direct evidence that nobody has a mailbox there (redirect to the provider or the provider's own DNS, parking/for-sale network, typosquat redirect or catch-all, or registered with no MX, A or AAAA record, all lookups answered NOERROR). The evidence is in `test/fixtures/lookalike_corpus.csv` (713 rows: 44 typo, 559 real, 110 uncertain), built by `script/collect_lookalike_evidence.rb` and `script/label_lookalike_corpus.rb`; the suite fails if an entry loses its evidence row or if any non-typo row can be refused.
+- **`Nondisposable::Domain.normalize` / `.from_email`**: the shared canonical form, including a zero-dependency RFC 3492 punycode encoder.
+- `Nondisposable::Suggestion.guess(domain)`: the ranker alone, without the reviewed list.
+
 ## [0.4.0] - 2026-08-25
 
 Catching the other kind of bad address. A disposable address is someone hiding from you; a typo is someone who wanted to reach you and can't. `user@gmail.con` isn't a throwaway — it's a real person whose account nobody will ever be able to reach, because `.con` has never existed. Both new checks are **opt-in**: upgrading this gem will not start rejecting addresses that were fine yesterday.

data/README.md

@@ -115,6 +115,11 @@ Nondisposable.configure do |config|
   # tempmail.com is on the list. Set to false for exact matches only.
   config.check_parent_domains = true
 
+  # --- Where the blocklist comes from (see "Blocklist sources") ---
+  config.sources = Nondisposable::Sources::DEFAULT  # or ::RECOMMENDED
+  config.corroboration_threshold = 2                # independent lists a :corroborating entry needs
+  config.block_privacy_relays = false               # Hide My Email, DuckDuckGo, SimpleLogin, addy.io…
+
   # --- Catching typos (both OFF by default) ---
 
   # Reject addresses whose TLD isn't in the IANA root zone: gmail.con, outlook.ed
@@ -136,9 +141,28 @@ end
 
 #### Parent domain matching
 
-With `check_parent_domains` enabled (the default since 0.3.0), `user@x.tempmail.com` is blocked when `tempmail.com` is on the list. The check walks up to 3 parent labels (`a.b.c.tempmail.com` → `b.c.tempmail.com` → `c.tempmail.com` → `tempmail.com`) and never matches against bare TLDs like `com`. All candidates are checked in a single indexed query.
+With `check_parent_domains` enabled (the default since 0.3.0), `user@x.tempmail.com` is blocked when `tempmail.com` is on the list. The check walks every parent down to the registrable domain (`a.b.c.d.tempmail.com` → … → `d.tempmail.com` → `tempmail.com`), however deep, and never matches against a public suffix like `com` or `co.uk`. All candidates are checked in a single indexed query. (Until 0.6.0 the walk stopped after 3 parent labels, so whoever controlled a listed domain could escape it with a fourth.)
+
+Since 0.6.0 the walk also never matches a **public suffix** — `co.uk`, `com.br`, `github.io` — using a snapshot of the [Public Suffix List](https://publicsuffix.org) bundled with the gem (no runtime dependency, no network). Upstream lists really do carry entries like `it.com` and `ddns.net`, under which anybody can register a name; one such line would otherwise refuse every unrelated company below it. Public suffixes are also never stored and never accepted from `additional_domains`.
+
+If you need to allowlist a specific subdomain under a listed parent, add the exact subdomain to `excluded_domains`; an exact exclusion always wins. Excluding a parent domain also unblocks its subdomains (unless a more specific subdomain is itself listed).
 
-This is a deliberately minimal, dependency-free approximation of "registrable domain" matching: the gem doesn't ship a full [public suffix list](https://publicsuffix.org), so it can't tell that `co.uk` is a public suffix — which is harmless in practice, because public suffixes don't appear on the blocklist. If you need to allowlist a specific subdomain under a listed parent, add the exact subdomain to `excluded_domains`; an exact exclusion always wins. Excluding a parent domain also unblocks its subdomains.
+#### Who wins when lists disagree
+
+One precedence table, applied both when an address is checked and when the stored list is built. Top wins:
+
+| # | List | Effect |
+|---|---|---|
+| 1 | `config.excluded_domains` (yours) | allowed |
+| 2 | `data/never_disposable.txt` (bundled: real providers and ISPs, plus `data/email_providers.txt`) | allowed, including subdomains |
+| 3 | `config.additional_domains` (yours) | blocked |
+| 4 | `data/privacy_relays.txt` (bundled) | `config.block_privacy_relays` decides |
+| 5 | the upstream sources | blocked |
+
+> [!IMPORTANT]
+> **Changed in 0.6.0:** before, `additional_domains` was checked first, so a domain in both `additional_domains` and `excluded_domains` was blocked. Now your exclusion wins. And because row 2 outranks row 3, `additional_domains` can no longer block a real provider such as `gmail.com` or `mail.ru` (it is ignored with a warning in the log): that list is for throwaway domains the upstream lists missed, and refusing a whole real provider is a business rule that belongs in your own validation, where it can tell the user why.
+
+Row 2 is why upgrading also **unblocks** two domains the default upstream list carried on 2026-10-17: `sify.com` (an Indian ISP) and `21cn.com` (China Telecom's webmail). A false positive is a real person who cannot sign up; if you find one, it belongs in `never_disposable.txt` (open a PR) or in your `excluded_domains`.
 
 #### Failure mode
 
@@ -202,10 +226,64 @@ Nondisposable.suggestion_for("someone@mail.com")  # => nil  (a real provider)
 <% end %>
 ```
 
-Setting `reject_lookalike_domains = true` turns the same guess into a validation error naming the correction. Do that deliberately: a suggestion is a guess about intent, and a wrong guess stops a real person signing up with their real address. Two things keep that rare — matching stops at **one** edit by default (`lookalike_distance`), and an exact match against the bundled provider list always wins, which is why that list includes awkward pairs like `mail.com` (one insertion from `gmail.com`). Add anything we've missed with `config.additional_email_providers`; doing so both makes it a suggestion candidate and stops its users being told they mistyped.
-
 The matching uses optimal string alignment distance rather than plain Levenshtein, so an adjacent swap counts as the one mistake a human actually made: `gmial.com` is distance 1, not 2.
 
+One edit is not enough to tell a typo from a real domain, though. Replayed against the [Tranco](https://tranco-list.eu) top 1M, "one edit from a provider" flags **707** domains, and many of them are large real organisations:
+
+- `ge.com` (General Electric), `ml.com` (Merrill Lynch), `mi.com` (Xiaomi), `se.com` (Schneider Electric) → `me.com`
+- `ea.com` (Electronic Arts), `aa.com` (American Airlines) → `ya.com`
+- `bol.com`, `aon.com`, `uol.com` (a major Brazilian mail provider) → `aol.com`
+- `one.com` → `ono.com`; `hive.com`, `life.com` → `live.com`; `grail.com`, `pmail.com` → `gmail.com`
+- `web.dev` → `web.de`; `yahoo.com.au` → `yahoo.com.ar`
+
+So since 0.5.0 suggestions are filtered before anyone sees them: a provider name of four characters or fewer (`me`, `ya`, `aol`, `mail`, `live`…) only takes edits after the name (`me.cmo` yes, `ge.com` no), and every edit has to look like a finger — a swap, a dropped letter (not the first), a doubled letter, an extra vowel, a neighbouring key on QWERTY, or the same key with an accent. A letter from across the keyboard (`grail`, `pmail`, `.au` for `.ar`) is a different word, not a slip. On a 30% holdout of the labelled corpus that keeps 13 of 16 reviewed typos and raises precision from 8% to 45% — measured retrospectively: the split was chosen before tuning but only committed with the tuned filters, so treat it as an estimate, not a pre-registered result. Better, still a guess — which is why a guess never refuses anyone on its own.
+
+#### `reject_lookalike_domains`: hard for known typos, confirmable for everything else
+
+```ruby
+config.reject_lookalike_domains = :confirmable # `true` means the same thing
+```
+
+| The domain is… | Example | Result |
+|---|---|---|
+| an exact provider (bundled or `additional_email_providers`) | `gmail.com`, `mail.com` | accepted |
+| an impossible TLD with a nameable correction (`check_tld` on or not) | `gmail.con`, `gmailmcom` | **refused**: "Did you mean …gmail.com?" |
+| a reviewed typo in [`data/known_typo_domains.txt`](data/known_typo_domains.txt) | `gmial.com`, `hotmial.com`, `yaho.com` | **refused**: "Did you mean …?" |
+| any other lookalike the ranker suggests | `gnail.com` | **confirmable**: same message, unless the user confirmed the address |
+| anything else | `ge.com`, `yahoo.com.au` | accepted |
+
+A disposable provider is still refused first, and a blocked TLD is still refused after a confirmation. A host's `additional_email_providers` outranks the known-typo list.
+
+The known-typo list is short (41 entries) on purpose. An entry needs direct evidence that nobody has a mailbox there — a redirect to the provider or the provider's own DNS (`yaho.com`, `homail.com`), a parking or for-sale network (`gamail.com`, `hotmial.com`), a typosquat redirect or catch-all (`gmial.com`, `gmali.com`), or a registered domain with no MX, A or AAAA record (`gmal.com`). "No MX" alone is not enough: SMTP falls back to the A record. The evidence for every entry is in `test/fixtures/lookalike_corpus.csv`, and the suite fails if an entry loses it.
+
+**Confirmable means the user can say "yes, this address is right".** Give the model somewhere to put that answer:
+
+```ruby
+class User < ApplicationRecord
+  include Nondisposable::ConfirmableEmail
+  confirmable_email :email # adds a virtual email_lookalike_confirmed attribute
+
+  validates :email, nondisposable: true
+end
+```
+
+```erb
+<%# Only when the confirmable error is showing: %>
+<% if @user.errors.of_kind?(:email, :lookalike_domain) %>
+  <%= f.check_box :email_lookalike_confirmed %>
+  <%= f.label :email_lookalike_confirmed, "Yes, my email address is right" %>
+<% end %>
+```
+
+Permit `:email_lookalike_confirmed` in your strong parameters. It's virtual — never persisted, and reset after every save — so it answers for one save; the error also carries the suggestion in `errors.details[:email]` (`suggestion:`). No concern needed if you'd rather wire it yourself: the validator only looks for a method named `<attribute>_lookalike_confirmed` that returns exactly `true`.
+
+> [!WARNING]
+> Without the checkbox, a confirmable error has no way past it — a real `gnail.com` user would be stuck. Wire the confirmation before you switch this on, or keep using `suggestion_for` as a hint only.
+
+Before 0.5.0, `true` refused every lookalike outright. If you set it, upgrading makes it **softer**, never stricter: reviewed typos and impossible TLDs are still refused, every other lookalike becomes confirmable, and many of the old matches (`ge.com`, `bol.com`) are no longer suggested at all.
+
+Matching stops at **one** edit by default (`lookalike_distance`; 2 is coarser and documented as risky, 0 switches the whole layer off, list included). Add anything we've missed with `config.additional_email_providers`; doing so both makes it a suggestion candidate and stops its users being told they mistyped.
+
 #### How the two combine
 
 At most one error is added, however many checks fire — and whenever the gem can name a correction, that phrasing wins:
@@ -214,7 +292,9 @@ At most one error is added, however many checks fire — and whenever the gem ca
 |---|---|---|
 | `user@gmail.con` | "looks like a typo. Did you mean user@gmail.com?" | same |
 | `user@zzz.con` | "doesn't look like a real email address" | same |
-| `user@gmail.co` | accepted | "looks like a typo. Did you mean user@gmail.com?" |
+| `user@gmail.co` | accepted | "looks like a typo. Did you mean user@gmail.com?" (a reviewed typo: refused) |
+| `user@gnail.com` | accepted | same message, confirmable |
+| `user@ge.com` | accepted | accepted |
 | `user@tempmail.com` | "provider is not allowed" | same |
 
 "Doesn't look like a real email address" is true but useless when we know what they meant, so a nameable correction always outranks the generic message.
@@ -230,6 +310,8 @@ Nondisposable.valid_tld?('user@example.con')  # => false
 Nondisposable.suggestion_for('user@gmial.com') # => "user@gmail.com"
 ```
 
+Every check compares domains in one canonical form — lowercased, without a trailing dot, unicode labels converted to punycode, so `user@Bücher.DE.` and `user@xn--bcher-kva.de` get the same verdict. The local part is never touched (case and `+tags` are the user's business). The same normalisation is available as `Nondisposable::Domain.normalize(domain)` and `Nondisposable::Domain.from_email(email)`.
+
 ## Updating disposable domains
 
 To manually update the list of disposable domains, run:
@@ -238,7 +320,7 @@ To manually update the list of disposable domains, run:
 Nondisposable::DomainListUpdater.update
 ```
 
-The fetch uses 10-second open/read timeouts and replaces the whole table atomically. If the remote fetch fails and your table is empty (e.g. a fresh install without network access), the updater automatically seeds from the blocklist snapshot bundled with the gem so you're never left unprotected; you can also trigger that explicitly with `Nondisposable::DomainListUpdater.seed` (it only seeds an empty table, and never overwrites existing data).
+The fetch uses 10-second open/read timeouts and replaces the whole table atomically (see [Blocklist sources](#blocklist-sources) for how several lists are merged and guarded). If the remote fetch fails and your table is empty (e.g. a fresh install without network access), the updater automatically seeds from the blocklist snapshot bundled with the gem so you're never left unprotected; you can also trigger that explicitly with `Nondisposable::DomainListUpdater.seed` (it only seeds an empty table, and never overwrites existing data).
 
 It's important you keep your disposable domain list up to date. `nondisposable` will read from the latest version of the [`disposable-email-domains`](https://github.com/disposable-email-domains/disposable-email-domains) list, which is typically updated every few days.
 
@@ -253,6 +335,95 @@ production:
     schedule: every day at 3am US/Pacific
 ```
 
+## Blocklist sources
+
+Out of the box the gem uses the one list it always has, [`disposable-email-domains`](https://github.com/disposable-email-domains/disposable-email-domains). Upgrading changes nothing. You can add more:
+
+```ruby
+config.sources = Nondisposable::Sources::RECOMMENDED
+```
+
+`RECOMMENDED` adds [fakefilter](https://github.com/7c/fakefilter), which watches temp-mail generators directly. On 2026-10-17 it added **3,275 domains** (13,168 in total, against 9,893), and none of them was a known real provider.
+
+> [!NOTE]
+> **More lists do not catch fresh burners.** `mail-burn.com`, the domain that prompted this feature, was live and handing out inboxes on 2026-10-17. Of the measured lists only disposable/disposable carried it, from its own crawl. groundcat had it too, but only because it copies disposable/disposable. That is **one origin**, and corroboration rightly refuses to count copies, so no precision-first combination of public lists blocks it. A fresh burner shows up in one crawler's list first. Until a list exists that publishes *evidence* for each fresh domain, block a domain you have verified yourself with `config.additional_domains`, and send it upstream.
+
+### Trust and lineage
+
+Every source is a `Nondisposable::Source` with a `trust`:
+
+- `:primary`: every domain it lists is blocked (subject to the precedence table).
+- `:corroborating`: a domain it lists is blocked only once `config.corroboration_threshold` (default 2) **independent origins** list it.
+
+"Independent" is the key word. Public blocklists copy each other: [`disposable/disposable`](https://github.com/disposable/disposable) declares that it imports mailchecker, fakefilter, wesbos and unkn0w, and groundcat imports disposable/disposable. Two copies of one mistake are one observation, so each source declares what it is `derived_from`, and a vote only counts when none of its upstreams that you also load carries the same domain.
+
+Measured on 2026-10-17, that rule has teeth. With lineage declared, adding disposable/disposable, mailchecker and groundcat as corroborating sources to `RECOMMENDED` adds **zero** domains: everything they agree on is already explained by an upstream you load. Ignoring lineage instead would have "corroborated" 68,692 more, 64 of them in the [Tranco](https://tranco-list.eu) top 100k (`compaq.com`, `cafe24.com`, `24hourfitness.com`, `asana.biz`…). That is why `RECOMMENDED` is two primaries and nothing else.
+
+The other measured sources ship as constants, so you can opt in knowingly:
+
+| Constant | List | Licence | Trust | Notes |
+|---|---|---|---|---|
+| `DISPOSABLE_EMAIL_DOMAINS` | disposable-email-domains/disposable-email-domains | CC0 | primary | the default; also the bundled seed |
+| `FAKEFILTER` | 7c/fakefilter | BSD-3 | primary | in `RECOMMENDED` |
+| `DISPOSABLE` | disposable/disposable-email-domains | MIT | corroborating | ~97k domains, aggregates most others |
+| `MAILCHECKER` | FGRibreau/mailchecker | MIT | corroborating | lists privacy relays |
+| `GROUNDCAT` | groundcat/disposable-email-domain-list | MIT | corroborating | aggregator |
+| `WESBOS` | wesbos/burner-email-providers | MIT | corroborating | no declared lineage, but 57% of it is also in mailchecker |
+
+> [!WARNING]
+> **About promoting `DISPOSABLE` to primary.** It is the broadest list, and the only one that carried `mail-burn.com` on 2026-10-17. On top of `RECOMMENDED` it adds 84,341 domains (70 of them in the Tranco top 100k, listed in `docs/evidence/2026-10-17-disposable-as-primary-tranco-top100k.csv`), and among them are real mail domains: China Unicom (`wo.com.cn`), the Japanese ISP ODN (`odn.ad.jp`), UQ mobile (`uqmobile.jp`), the Moscow ISP MTU (`mtu-net.ru`), plus `cafe24.com`, `24hourfitness.com`, `foxwoods.com` and more. The ones we verified are now in `never_disposable.txt`, but the rest of the list is unknown. If you still want it, `Nondisposable::Sources::DISPOSABLE.with(trust: :primary)` does it. For one specific burner domain, `config.additional_domains` is the precise tool.
+
+Your own source is a Hash or a `Source`:
+
+```ruby
+config.sources = Nondisposable::Sources::RECOMMENDED + [
+  { name: "ours", url: "https://example.com/burners.txt", trust: :corroborating, derived_from: %w[mailchecker] }
+]
+```
+
+Sources must be `https`, plain text, one domain per line (`#` comments allowed). Names and URLs must be unique, and at least one source must be `:primary`.
+
+### Privacy relays
+
+Apple Hide My Email, DuckDuckGo, Firefox Relay, SimpleLogin/Proton Pass and addy.io forward to a real, long-lived inbox. They are not throwaways, but they do let one person hold many addresses. That is a policy decision, so it is a setting: `config.block_privacy_relays = false` (the default) **allows** them even when an upstream list carries them (mailchecker does), and `true` **refuses** them even when none does. The list is `data/privacy_relays.txt`; each domain covers its subdomains.
+
+### Upgrading an existing install
+
+Multi-source lists need one new table. The migration is additive, so you can deploy first and migrate later:
+
+```bash
+rails generate nondisposable:upgrade
+rails db:migrate
+```
+
+Until the table exists (and again if you roll the migration back), the updater uses the default list exactly as before and logs a warning if you have configured other sources. During a rolling deploy an old process may still publish the single-source list once; the next update from a new process rebuilds the full one.
+
+### How an update is published
+
+Each source is fetched into its own row in `nondisposable_source_snapshots` (status, timestamp, sha256, count, the list itself). The stored list is then **rebuilt** from each source's last good snapshot, in one transaction, under a database advisory lock on PostgreSQL and MySQL (a process lock on SQLite, which already allows only one writer). Seeding takes the same lock. A new snapshot is refused, and the source's last good one kept, when:
+
+- the download fails, takes longer than the source's `deadline` (30 s), or is larger than its `max_bytes` (8 MB, checked while streaming);
+- more than 1% of its lines are not domain names (an HTML error page, a format change);
+- it has fewer than `minimum_count` domains (1,000 for the presets);
+- it is less than half the source's high-water mark, the largest size ever accepted for it (kept on its newest good snapshot, so neither pruning nor a series of 49% cuts can wear it down; `reset_baseline` is the human override), or more than three times its last good one;
+- it is identical to a snapshot a human rolled back.
+
+Every size is a count of *storable* domains (not junk, not a public suffix, not a known real provider, not a privacy relay), counted the same way whatever your settings, so a payload padded with such lines cannot pass for a full list and changing `block_privacy_relays` or `excluded_domains` never trips a guard. The list is never published empty. A primary that has never been fetched successfully blocks publishing, and the current table is kept as it is. Two exceptions let the default source's bundled list stand in for it: an EMPTY table (a fresh install), and an explicit `rollback`, which must always be able to withdraw a poisoned list. So an update can never replace a fresh seed with less than it, nor a table upgraded from 0.4/0.5 (with any source configuration) with an older bundled snapshot. Additions and removals are logged per source. A source that lists a known real provider, or a public suffix, is logged as a possible poisoning, and the entry is dropped. Accepting a snapshot, publishing and rolling back all happen under the same lock; downloads happen outside it.
+
+> [!WARNING]
+> **Two concurrency limits, by design:**
+>
+> - **SQLite: run the refresh from one process.** PostgreSQL and MySQL get a database-wide advisory lock; SQLite has none, so the lock is per process. Two *processes* refreshing at once can interleave (one can republish a merge it read before the other rolled a source back). Schedule `DisposableEmailDomainListUpdateJob` on a single job runner, and run `rollback` from that same process or when no update is running.
+> - **Rolling deploys: quiesce the old updater.** Processes still running 0.5 or earlier replace the table without taking the lock, so their single-source list can overwrite a multi-source one, or leave back a row a new process just withdrew. When upgrading: pause the scheduled refresh job, deploy, run `rails g nondisposable:upgrade && rails db:migrate`, then resume it and run `Nondisposable::DomainListUpdater.update` once.
+
+Two more tools:
+
+```ruby
+Nondisposable::DomainListUpdater.publish             # rebuild from snapshots, no download (after a config change)
+Nondisposable::DomainListUpdater.rollback("fakefilter") # withdraw that source's newest snapshot (its payload stays refused) and republish
+Nondisposable::DomainListUpdater.reset_baseline("fakefilter") # accept a genuine shrink: the guard now measures from today's size
+```
+
 ## Updating the TLD list
 
 The TLD snapshot is the deliberate opposite of the disposable list: it ships **inside the gem**, not in your database, and there is nothing for your app to schedule. Disposable domains number in the thousands and change every few days; TLDs are ~1,438 strings that change a handful of times a year, so a frozen `Set` costs one file read at boot, answers in O(1) with no query per signup, needs no migration, and can't be empty on a fresh install.
@@ -271,6 +442,14 @@ It refuses to overwrite a good list with an implausibly short one (a captive por
 Nondisposable::Tld.version # => "2026082301, Last Updated Mon Aug 24 07:07:01 2026 UTC"
 ```
 
+The bundled Public Suffix List (`data/public_suffix_list.dat`, redistributed unmodified under the MPL 2.0) is refreshed the same way, with the same safeguards:
+
+```bash
+rake nondisposable:psl:update    # rewrites data/public_suffix_list.dat from publicsuffix.org
+```
+
+A stale PSL can only be less protective, never lock anyone out: a suffix created after the snapshot simply isn't known to be one.
+
 ## Troubleshooting
 
 ### SSL certificate verify failed (unable to get certificate CRL)

data/app/models/nondisposable/disposable_domain.rb

@@ -4,52 +4,71 @@ module Nondisposable
   class DisposableDomain < ApplicationRecord
     validates :name, presence: true, uniqueness: { case_sensitive: false }
 
-    # How many leading labels the parent-domain walk strips when
-    # check_parent_domains is enabled: user@a.b.c.tempmail.com is checked as
-    # a.b.c.tempmail.com, b.c.tempmail.com, c.tempmail.com and tempmail.com.
-    #
-    # Trade-off (documented instead of pulling in a full public-suffix-list
-    # dependency): the walk never queries single-label suffixes (bare TLDs like
-    # "com"), but without a PSL it cannot tell that multi-label suffixes like
-    # "co.uk" are public — those candidates are harmless unless a public suffix
-    # is deliberately added to the list. Conversely, a throwaway address nested
-    # more than 3 labels below a listed domain escapes the walk; blocklisted
-    # domains are registrable domains, so 3 levels covers practical abuse.
+    # Kept so code that referenced it still loads; no longer used. Until 0.6.0
+    # the parent walk stopped after this many labels, which let anybody who
+    # controls a listed domain escape it with `a.b.c.d.tempmail.com`. The walk
+    # now covers every parent down to the registrable domain: DNS itself caps
+    # a name at 253 characters, so the candidate list is bounded anyway.
     PARENT_MATCH_DEPTH = 3
 
+    # Kept (and equal to Domain::MAX_RAW_LENGTH) for code that referenced it.
+    # The raw-length bound itself now lives in Domain.normalize, which applies
+    # it AFTER stripping invisible characters: judging raw length here let
+    # padding hide a listed domain.
+    MAX_INPUT_LENGTH = Nondisposable::Domain::MAX_RAW_LENGTH
+
     class << self
+      # Whether `domain` is disposable, according to the precedence table in
+      # Nondisposable::Precedence: host exclusions, then the bundled
+      # never-disposable list, then host additions, then the privacy-relay
+      # policy, and only then this table.
       def disposable?(domain)
-        return false if domain.blank?
+        # A name longer than DNS allows (over 253 characters or a label over
+        # 63 once canonical, or over 1,024 code points of raw input after
+        # invisible characters are stripped) is TRUE: it can never be a
+        # deliverable domain, and a check whose job is keeping junk out must
+        # not call junk fine. Judged by Domain.too_long?, never on the raw
+        # length: a raw cap let invisible padding (U+00AD x 1013 +
+        # "tempmail.com") hide a listed name. Domain.normalize bounds its own
+        # work on oversized input, so the parent walk below never runs on one.
+        raw = domain
+        return false if raw.to_s.strip.empty?
+        return true if Nondisposable::Domain.too_long?(raw)
 
-        domain = domain.to_s.downcase
-        config = Nondisposable.configuration
-        candidates = match_candidates(domain)
+        domain = Nondisposable::Domain.normalize(raw)
+        return false if domain.empty?
+        # Not UTF-8: no list holds it and no mail server can deliver to it.
+        return false if domain.encoding == Encoding::BINARY && !domain.ascii_only?
 
-        additional = config.additional_domains.map { |d| d.to_s.downcase }
-        return true if (candidates & additional).any? # rubocop compat: Array#intersect? needs Ruby >= 3.1, gem supports 3.0
+        verdict, candidates = Nondisposable::Precedence.decide(domain, match_candidates(domain))
+        return verdict == :block unless verdict == :lookup
 
-        excluded = config.excluded_domains.map { |d| d.to_s.downcase }
-        # An exact exclusion always wins, so a specific subdomain can be
-        # allowlisted even when a parent domain is on the list.
-        return false if excluded.include?(domain)
+        # Rows written before 0.5.0 stored a unicode domain as it came
+        # (lowercased, not punycode). Match that spelling of every candidate
+        # still eligible, so upgrading never unblocks them; the next publish
+        # rewrites them canonically.
+        legacy = Nondisposable::Domain.unicode(raw)
+        unless legacy == domain || legacy.ascii_only?
+          candidates |= match_candidates(legacy).select { |name| candidates.include?(Nondisposable::Domain.normalize(name)) }
+        end
 
-        where(name: candidates - excluded).exists?
+        candidates.any? && where(name: candidates).exists? # one WHERE name IN (...)
       end
 
-      # The exact domain plus up to PARENT_MATCH_DEPTH parent domains,
-      # never descending below two labels (bare TLDs are never candidates).
+      # With check_parent_domains (the default): the domain and EVERY parent
+      # down to, and including, its registrable domain:
+      #
+      #   a.b.c.d.tempmail.com => a.b.c.d.tempmail.com … d.tempmail.com, tempmail.com
+      #
+      # Never above the registrable domain: past it lie public suffixes (`com`,
+      # `co.uk`) and names like `sch.uk` that sit above a wildcard suffix
+      # (`*.sch.uk`), all belonging to unrelated registrants. Without
+      # check_parent_domains: the domain alone. Expects a normalized domain.
       def match_candidates(domain)
-        return [domain] unless Nondisposable.configuration.check_parent_domains
+        namespace = Nondisposable::Precedence.namespace(domain)
+        return namespace if Nondisposable.configuration.check_parent_domains
 
-        labels = domain.split('.')
-        candidates = [domain]
-        1.upto(PARENT_MATCH_DEPTH) do |depth|
-          parent = labels.drop(depth)
-          break if parent.size < 2
-
-          candidates << parent.join('.')
-        end
-        candidates
+        namespace.first(1)
       end
     end
 

data/app/models/nondisposable/source_snapshot.rb

@@ -0,0 +1,66 @@
+# frozen_string_literal: true
+
+module Nondisposable
+  # What one source said on one fetch — or why we refused to listen.
+  #
+  # The merged nondisposable_disposable_domains table cannot answer "which
+  # list put this domain here?", and it cannot be rebuilt when one source has a
+  # bad day: a single domain may come from several sources, and a domain one
+  # corroborating source lists must survive until a second one agrees. So each
+  # source keeps its own recent history here, and the merged table is always
+  # REBUILT from each source's last good snapshot (DomainListUpdater.publish).
+  #
+  # Statuses:
+  #   ok          accepted; `domains` holds the canonical list, one per line
+  #   rejected    downloaded, but failed a guard (too small, shrank or grew
+  #               implausibly, mostly junk); `error` says which
+  #   failed      the download itself failed; `error` says how
+  #   rolled_back an ok snapshot a human withdrew (DomainListUpdater.rollback)
+  #
+  # Only `ok` snapshots ever feed the published list. The table is optional:
+  # without it (hosts that have not run `rails g nondisposable:upgrade`), the
+  # updater keeps the single-source behaviour of 0.5 and earlier.
+  class SourceSnapshot < ApplicationRecord
+    self.table_name = 'nondisposable_source_snapshots'
+
+    STATUSES = %w[ok rejected failed rolled_back].freeze
+
+    # Rows kept per source: the newest RETAINED_PER_SOURCE of any status, plus
+    # the newest RETAINED_OK_PER_SOURCE good ones however old (failures must
+    # never evict the fallback, nor the shrink guard's baseline), plus EVERY
+    # rolled-back row (its sha256 is what refuses the same poison next time).
+    RETAINED_PER_SOURCE = 5
+    RETAINED_OK_PER_SOURCE = 3
+
+    validates :source, :status, :fetched_at, presence: true
+    validates :status, inclusion: { in: STATUSES }
+
+    scope :ok, -> { where(status: 'ok') }
+    scope :newest_first, -> { order(fetched_at: :desc, id: :desc) }
+
+    class << self
+      # Is the table there? Checked on every update rather than memoized, so a
+      # long-running process notices the upgrade migration without a restart.
+      def available?
+        connection_pool.with_connection { |connection| connection.data_source_exists?(table_name) }
+      rescue StandardError
+        false
+      end
+
+      def last_good(source_name)
+        ok.where(source: source_name).newest_first.first
+      end
+
+      def prune(source_name)
+        rows = where(source: source_name)
+        keep = rows.newest_first.limit(RETAINED_PER_SOURCE).pluck(:id) +
+               rows.ok.newest_first.limit(RETAINED_OK_PER_SOURCE).pluck(:id)
+        rows.where.not(id: keep).where.not(status: 'rolled_back').delete_all
+      end
+    end
+
+    def domain_list
+      domains.to_s.split("\n")
+    end
+  end
+end

data/data/known_typo_domains.txt

@@ -0,0 +1,99 @@
+# Reviewed typo domains, used by Nondisposable::KnownTypos.
+#
+#   <typo domain> <the provider it is a misspelling of>
+#
+# These are the ONLY lookalike domains the validator refuses outright (when
+# `reject_lookalike_domains` is on). Everything else one edit from a provider
+# is a guess, and a guess only ever produces a confirmable error. So the bar
+# for an entry is high, and it is the same bar for every entry:
+#
+#   1. The domain is a misspelling of a provider in data/email_providers.txt.
+#   2. There is direct evidence that nobody has a real mailbox there, of one of
+#      these kinds:
+#        - it redirects to the provider, or is held on the provider's own DNS
+#          (a defensive registration: yaho.com, homail.com);
+#        - it is parked or for sale through a parking/marketplace network
+#          (nameservers or lander of parkingcrew, sedo, bodis, afternic,
+#          above.com…), or publishes a Null MX (RFC 7505: "accepts no mail");
+#        - it is a typosquat: it redirects visitors to an unrelated monetised
+#          page, or its mail goes to a catch-all run for misspelled domains;
+#        - it is registered but has no MX, A or AAAA record, so mail to it
+#          cannot be delivered at all (RFC 5321 §5.1 has nothing to fall
+#          back to). Every lookup must have answered NOERROR — recorded in the
+#          evidence's dns_status column, and the labelling script refuses the
+#          verdict otherwise — never inferred from a lookup that timed out.
+#   3. The evidence is recorded, row by row, in test/fixtures/lookalike_corpus.csv
+#      (collected by script/collect_lookalike_evidence.rb on 2026-10-17), and
+#      the suite fails if an entry here loses its `typo` row there.
+#
+# What is NOT enough on its own: "no MX record" (SMTP falls back to the A
+# record), a blank homepage, or a short name. When the evidence is ambiguous
+# the domain stays OFF this list and remains confirmable — a wrong entry here
+# is a person who cannot sign up and cannot say "yes, this is my address".
+#
+# Never generated, never bulk-imported. Add an entry with its evidence row.
+
+# --- frontier.com ---
+ffrontier.com frontier.com
+
+# --- globo.com ---
+globh.com     globo.com
+
+# --- gmail.com ---
+gamail.com    gmail.com
+gimail.com    gmail.com
+gm4il.com     gmail.com
+gmai.com      gmail.com
+gmaii.com     gmail.com
+gmail.cm      gmail.com
+gmail.co      gmail.com
+gmaio.com     gmail.com
+gmal.com      gmail.com
+gmali.com     gmail.com
+gmeil.com     gmail.com
+gmial.com     gmail.com
+gmmail.com    gmail.com
+gmuil.com     gmail.com
+hmail.com     gmail.com
+jmail.com     gmail.com
+
+# --- hotmail.com ---
+hitmail.com   hotmail.com
+homail.com    hotmail.com
+hotamil.com   hotmail.com
+hotmai.com    hotmail.com
+hotmal.com    hotmail.com
+hotmasil.com  hotmail.com
+hotmial.com   hotmail.com
+otmail.com    hotmail.com
+
+# --- icloud.com ---
+icloude.com   icloud.com
+
+# --- outlook.com ---
+autlook.com   outlook.com
+outlok.com    outlook.com
+outloo.com    outlook.com
+
+# --- runbox.com ---
+rfunbox.com   runbox.com
+
+# --- yahoo.com ---
+yahho.com     yahoo.com
+yaho.com      yahoo.com
+yaoo.com      yahoo.com
+yhaoo.com     yahoo.com
+yhoo.com      yahoo.com
+
+# --- yahoo.de ---
+yaho.de       yahoo.de
+
+# --- yahoo.it ---
+yhoo.it       yahoo.it
+
+# --- yandex.ru ---
+ayndex.ru     yandex.ru
+yandez.ru     yandex.ru
+
+# --- zoho.com ---
+zoho.co       zoho.com