nondisposable 0.3.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +73 -0
- data/README.md +305 -3
- data/Rakefile +5 -0
- data/app/models/nondisposable/disposable_domain.rb +53 -34
- data/app/models/nondisposable/source_snapshot.rb +66 -0
- data/data/email_providers.txt +276 -0
- data/data/iana_tlds.txt +1439 -0
- data/data/known_typo_domains.txt +99 -0
- data/data/never_disposable.txt +480 -0
- data/data/privacy_relays.txt +42 -0
- data/data/public_suffix_list.dat +16571 -0
- data/gemfiles/rails_7.2.gemfile +1 -0
- data/gemfiles/rails_8.0.gemfile +1 -0
- data/gemfiles/rails_8.1.gemfile +1 -0
- data/lib/generators/nondisposable/install_generator.rb +1 -0
- data/lib/generators/nondisposable/templates/create_nondisposable_source_snapshots.rb.erb +42 -0
- data/lib/generators/nondisposable/templates/nondisposable.rb +67 -1
- data/lib/generators/nondisposable/upgrade_generator.rb +43 -0
- data/lib/nondisposable/confirmable_email.rb +75 -0
- data/lib/nondisposable/domain.rb +238 -0
- data/lib/nondisposable/domain_list_updater.rb +510 -88
- data/lib/nondisposable/email_validator.rb +101 -3
- data/lib/nondisposable/known_typos.rb +64 -0
- data/lib/nondisposable/precedence.rb +209 -0
- data/lib/nondisposable/public_suffix.rb +147 -0
- data/lib/nondisposable/public_suffix_list_updater.rb +88 -0
- data/lib/nondisposable/source.rb +233 -0
- data/lib/nondisposable/source_fetcher.rb +69 -0
- data/lib/nondisposable/source_merge.rb +127 -0
- data/lib/nondisposable/suggestion.rb +336 -0
- data/lib/nondisposable/tld.rb +190 -0
- data/lib/nondisposable/tld_list_updater.rb +101 -0
- data/lib/nondisposable/version.rb +1 -1
- data/lib/nondisposable.rb +153 -6
- data/lib/tasks/nondisposable.rake +66 -0
- metadata +24 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 9d0c0a8ad776c00c83c1e8b1e3001c652c5e5b414bb8f79216d3544192843e42
|
|
4
|
+
data.tar.gz: f2ab66119b84f495ce62d2104ab1c05701441796a969ae793e0fe033c69e6447
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 5d2c0d50d9296c92465713fcdf8c97ddfcb3f9b4536ff299b80687f4d0ea12c877a718511bd340d3f1bba897c3e1cc38bdd65ee10c9a3c6f37272cffd3a376cb
|
|
7
|
+
data.tar.gz: bd6983052fe4a8980a041c99f8aa7a4c41b35c947b5de246d39b10cdbf20c2ad36b2114dd1dad6b481388bcff02a4712977d73bf90c7d132bd0cbd594a5dc245
|
data/CHANGELOG.md
CHANGED
|
@@ -1,3 +1,76 @@
|
|
|
1
|
+
## [0.6.0] - 2026-10-17
|
|
2
|
+
|
|
3
|
+
Ships everything listed under 0.5.0 too: 0.5.0 was never released on its own.
|
|
4
|
+
|
|
5
|
+
More than one blocklist, without trusting any of them more than it has earned. Triggered by a real incident: signups from `mail-burn.com`, a live 10-minute inbox the default list did not carry. The measurement behind every default is in `docs/evidence/2026-10-17-multi-source-measurement.md`. With the default configuration you still get the one list you had before.
|
|
6
|
+
|
|
7
|
+
### Added
|
|
8
|
+
|
|
9
|
+
- **Multiple blocklist sources** (`config.sources`, default `Nondisposable::Sources::DEFAULT`, which is today's single list). Each `Nondisposable::Source` has a `trust` (`:primary` blocks everything it lists; `:corroborating` needs `config.corroboration_threshold`, default 2, independent origins) and declares its `derived_from` lineage, so a list that copies another never counts as a second vote. Measured sources ship as constants (`DISPOSABLE_EMAIL_DOMAINS`, `FAKEFILTER`, `DISPOSABLE`, `MAILCHECKER`, `GROUNDCAT`, `WESBOS`) with licences and lineage.
|
|
10
|
+
- **`Nondisposable::Sources::RECOMMENDED`**: the default list plus fakefilter, both primary. +3,275 domains on 2026-10-17, none of them a known real provider. It does **not** include disposable/disposable as a primary; see the README for the 84,341 domains, real mail providers among them, that would add.
|
|
11
|
+
- **Source snapshots** (`nondisposable_source_snapshots`): every fetch of every source is recorded (status, time, sha256, count, list), and the stored list is rebuilt from each source's last good snapshot in one transaction, under a database advisory lock on PostgreSQL/MySQL (process lock on SQLite). Seeding takes the same lock.
|
|
12
|
+
- **`rails generate nondisposable:upgrade`**: an additive migration for existing installs. Until it is migrated (or after rolling it back), the updater keeps the single-source behaviour, so deploying before migrating is safe. Fresh installs get the table from `nondisposable:install`.
|
|
13
|
+
- **Update guards**: streamed size cap (`max_bytes`, 8 MB) and wall-clock `deadline` (30 s) per source; no redirects followed; https only; a snapshot is refused when over 1% of its lines are not domains, when it is below `minimum_count`, when it is under half the largest snapshot still retained or over three times the last good one, or when it equals a rolled-back snapshot. Counts are of unique canonical domains. Additions and removals are logged per source. Sizes count storable domains only (no junk, public suffixes or real providers). The list is never published empty; a never-fetched default source stands in its bundled list, and any other never-fetched primary blocks publishing. Rolled-back payloads stay refused permanently; failures never prune the last good snapshots.
|
|
14
|
+
- **Concurrency limits (documented):** on SQLite the publish lock is per process, so run the refresh job from a single runner. During a rolling deploy from 0.5, pause the refresh job until every process runs 0.6 (old processes replace the table without the lock).
|
|
15
|
+
- **`DomainListUpdater.reset_baseline(source_name)`**: accept a genuine upstream shrink below half of the source's high-water mark.
|
|
16
|
+
- **`DomainListUpdater.publish`** (rebuild from snapshots without downloading, e.g. after a config change) and **`DomainListUpdater.rollback(source_name)`** (withdraw a source's newest snapshot and republish).
|
|
17
|
+
- **Public Suffix List guard**: a bundled snapshot of publicsuffix.org's list (MPL 2.0, unmodified) and `Nondisposable::PublicSuffix`. Public suffixes (`com.br`, `co.uk`, `it.com`, `ddns.net`) are never stored, never accepted from `additional_domains`, and never used as a parent-match candidate, so one bad upstream line cannot block a whole namespace. Refresh with `rake nondisposable:psl:update`.
|
|
18
|
+
- **`data/never_disposable.txt`**: real providers and ISPs that upstream lists were found to carry, plus paid mailbox providers; unioned at runtime with `data/email_providers.txt`. Each entry protects its subdomains (`bak.rr.com` under `rr.com`). An upstream source listing one is logged as possible poisoning.
|
|
19
|
+
- **Privacy relays as a policy** (`config.block_privacy_relays`, default `false`, list in `data/privacy_relays.txt`): Hide My Email, DuckDuckGo, Firefox Relay, SimpleLogin/Proton Pass and addy.io are allowed even when a source lists them, or refused even when none does.
|
|
20
|
+
- **`Nondisposable::Domain.normalize`**: one canonical form (stripped, downcased, no trailing dot, IDN to punycode) used by the checks and by list ingestion. `Bücher.de.` and `xn--bcher-kva.de` are the same domain.
|
|
21
|
+
|
|
22
|
+
### Changed
|
|
23
|
+
|
|
24
|
+
- ⚠️ **`DisposableDomain.disposable?` / `Nondisposable.disposable?` answer `true` for a name longer than DNS allows** (`Domain.too_long?`: over 253 characters or a label over 63 once canonical, or over 1,024 code points of raw input after stripping invisible characters). Such a name can never be deliverable, and a check that keeps junk out must not call it fine. Names exactly at the limits are judged normally. The validator rejects them earlier with `invalid_tld_error_message`.
|
|
25
|
+
- ⚠️ **Precedence**: one table, applied both when checking and when building the list: `excluded_domains` > `never_disposable` > `additional_domains` > privacy-relay policy > sources. Before, `additional_domains` was checked first, so a domain in both `additional_domains` and `excluded_domains` was **blocked**; it is now **allowed**. `additional_domains` can no longer block a real provider listed in `never_disposable` (ignored with a warning).
|
|
26
|
+
- ⚠️ Two domains the default upstream list carries are **no longer blocked**: `sify.com` (Indian ISP) and `21cn.com` (China Telecom webmail).
|
|
27
|
+
- Entries that cannot be DNS names (labels over 63 characters, names over 253, IP addresses, HTML) are dropped from downloaded lists. The single-source path also refuses a payload with more than 1% such lines instead of storing it.
|
|
28
|
+
- ⚠️ **The parent-domain walk has no depth cap any more**: it checks every parent down to the registrable domain, so `a.b.c.d.tempmail.com` is blocked when `tempmail.com` is listed (or in `additional_domains`). It used to stop after 3 parent labels, which let whoever controls a listed domain escape with a fourth. Still a single `WHERE name IN (...)` query; DNS caps a name at 253 characters. It never climbs above the registrable domain, so a public suffix (`co.uk`), or a name above a wildcard suffix (`sch.uk`), can never match. `DisposableDomain::PARENT_MATCH_DEPTH` is kept but unused.
|
|
29
|
+
- The table is updated by writing only the difference (still in one transaction), instead of deleting and reinserting every row.
|
|
30
|
+
|
|
31
|
+
## [0.5.0] - Not released separately (shipped in 0.6.0)
|
|
32
|
+
|
|
33
|
+
Lookalike detection you can actually switch on. Replaying the 0.4.0 rule ("one edit from a provider") against the Tranco top 1M flagged 707 domains, and many are large real organisations whose staff would have been refused at signup: `ge.com`, `ml.com`, `mi.com`, `se.com` → `me.com`; `ea.com`, `aa.com` → `ya.com`; `bol.com`, `aon.com`, `uol.com` → `aol.com`; `one.com` → `ono.com`; `hive.com`, `life.com` → `live.com`; `grail.com`, `pmail.com` → `gmail.com`; `web.dev` → `web.de`; `yahoo.com.au` → `yahoo.com.ar`. Offline, nothing can tell those apart from `gmial.com` with certainty, so a guess no longer refuses anybody on its own.
|
|
34
|
+
|
|
35
|
+
### Changed
|
|
36
|
+
|
|
37
|
+
- **Behavior change for `reject_lookalike_domains = true`: it is now softer, never stricter.** Precedence, first match wins: an exact provider is accepted → an impossible TLD with a nameable correction (`gmail.con`, `gmailmcom`) is **refused**, whether or not `check_tld` is on → a reviewed typo from the new bundled `data/known_typo_domains.txt` (41 entries, e.g. `gmial.com`, `hotmial.com`, `yaho.com`) is **refused** → any other lookalike gets the same "Did you mean …?" error, but **confirmable**: it is skipped when the record says the user confirmed the address as typed. Disposable providers are still refused first, blocked TLDs are still refused after a confirmation, and a host's `additional_email_providers` outranks the known-typo list.
|
|
38
|
+
- **`Nondisposable.suggestion_for` suggests less, and better.** Provider names of four characters or fewer (`me`, `ya`, `aol`, `mail`, `live`…) only take edits after the name, and every edit must look like a finger slip on QWERTY (swap, dropped letter other than the first, doubled letter, extra vowel, neighbouring key, accented same key). On a retrospective 30% holdout of the labelled corpus (split chosen before tuning but committed with the tuned filters, so not pre-registered): precision 8% → 45%, recall 81% of reviewed typos. Reviewed typos are always corrected, even where the ranker alone would stay silent (`jmail.com`).
|
|
39
|
+
- `lookalike_distance = 0` now switches off the whole lookalike layer, known-typo list included.
|
|
40
|
+
- The confirmable error is added with type `:lookalike_domain` (and `suggestion:` in its details) so a form can tell it apart: `errors.of_kind?(:email, :lookalike_domain)`. The message is unchanged and still names the correction; hard lookalike errors keep their plain-message form.
|
|
41
|
+
- Domains are normalised the same way on every path (`Nondisposable.disposable?`, `additional_domains` / `excluded_domains`, the validator, the TLD check, suggestions, the provider and known-typo lists): stripped, lowercased, trailing dot removed, fullwidth and ideographic dots read as dots, NFKC plus the invisible characters UTS #46 drops (soft hyphen, zero-width space), unicode labels converted to punycode. `x@mailinator.com` and `x@foo.tk` no longer slip past their list entries. `user@tempmail.com.` and `user@bücher.de` (listed as `xn--bcher-kva.de`) are now caught. The local part is never touched.
|
|
42
|
+
- **Stored rows and 0.4.0 compatibility.** List rows are stored in the canonical (punycode) form, and lookups also match a unicode row written by 0.4.0, under the same exclusion rules (a canonical `excluded_domains` entry still wins over a legacy `bücher.de` row and its subdomains). So that a mixed-version deploy, or a rollback to 0.4.0, never silently unblocks anything, a domain added in unicode through `additional_domains` is stored **twice**: `xn--bcher-kva.de` and `bücher.de`. The row count includes these twins. ⚠️ **Rollback to 0.4.0: unicode entries from downloaded lists are stored only in punycode, so a 0.4.0 reader would not match them; no current source ships raw unicode** (all six measured lists send IDNs as `xn--`, 2026-10-17). (A 0.4.0 reader also matches `additional_domains` in memory, so the twin row matters mostly for a rollback in which the config changed too.)
|
|
43
|
+
- Normalisation is bounded on hostile input. NFKC runs only on names of up to 1,024 code points, and not on a run of more than 30 combining marks (not stream-safe, UAX #15). The DNS limits (253 per name, 63 per label) apply to the **normalised** result, so a valid label typed in decomposed form (`e` + combining acute) still matches.
|
|
44
|
+
- **Behavior change: a domain longer than DNS allows is refused** by the validator, with `invalid_tld_error_message`, whatever is switched on: over 253 characters, a label over 63 (judged on the punycode form), or over 1,024 code points before normalisation. No mail can reach such an address and no list can hold it, so it can never be "not disposable". Invisible padding (soft hyphens and the like) is stripped first, so it can neither trigger this nor hide a listed domain. Values without an `@` are still left to `format:`. `Nondisposable::Domain.too_long?(domain)` exposes the check.
|
|
45
|
+
|
|
46
|
+
### Added
|
|
47
|
+
|
|
48
|
+
- **`config.reject_lookalike_domains = :confirmable`**, the explicit name for the mode above. Strings (`"confirmable"`, `"true"`, `"false"`) are accepted for YAML/ENV config; any other value raises `ArgumentError` at boot instead of silently meaning "on".
|
|
49
|
+
- **`Nondisposable::ConfirmableEmail`**: `include` it and call `confirmable_email :email` to get a virtual, boolean-cast `email_lookalike_confirmed` attribute for a "Yes, my email is right" checkbox. It is reset after every save, so a later lookalike is asked about again. Without the concern, any `<attribute>_lookalike_confirmed` method returning exactly `true` works.
|
|
50
|
+
- **`Nondisposable::KnownTypos`** and `data/known_typo_domains.txt`: the only lookalikes refused outright, each with direct evidence that nobody has a mailbox there (redirect to the provider or the provider's own DNS, parking/for-sale network, typosquat redirect or catch-all, or registered with no MX, A or AAAA record, all lookups answered NOERROR). The evidence is in `test/fixtures/lookalike_corpus.csv` (713 rows: 44 typo, 559 real, 110 uncertain), built by `script/collect_lookalike_evidence.rb` and `script/label_lookalike_corpus.rb`; the suite fails if an entry loses its evidence row or if any non-typo row can be refused.
|
|
51
|
+
- **`Nondisposable::Domain.normalize` / `.from_email`**: the shared canonical form, including a zero-dependency RFC 3492 punycode encoder.
|
|
52
|
+
- `Nondisposable::Suggestion.guess(domain)`: the ranker alone, without the reviewed list.
|
|
53
|
+
|
|
54
|
+
## [0.4.0] - 2026-08-25
|
|
55
|
+
|
|
56
|
+
Catching the other kind of bad address. A disposable address is someone hiding from you; a typo is someone who wanted to reach you and can't. `user@gmail.con` isn't a throwaway — it's a real person whose account nobody will ever be able to reach, because `.con` has never existed. Both new checks are **opt-in**: upgrading this gem will not start rejecting addresses that were fine yesterday.
|
|
57
|
+
|
|
58
|
+
### Added
|
|
59
|
+
|
|
60
|
+
- **TLD validation** (`config.check_tld`, default `false`): rejects addresses whose TLD isn't in the IANA root zone. The gem now bundles a snapshot of [data.iana.org/TLD/tlds-alpha-by-domain.txt](https://data.iana.org/TLD/tlds-alpha-by-domain.txt) (~1,438 entries, ~9 KB) loaded once into a frozen `Set` — no migration, no table, no query per signup, and it can't be empty on a fresh install. A domain with no dot at all (`example@gmailmcom`) has no TLD and is rejected too; IP literals and unicode TLDs are deliberately left alone.
|
|
61
|
+
- **`config.additional_tlds`**: accept a TLD delegated after your installed version shipped, without waiting for a release. This is the escape hatch that keeps a stale snapshot from ever locking anyone out.
|
|
62
|
+
- **`Nondisposable::Tld::SPECIAL_USE`**: the RFC 6761 / RFC 2606 reserved names (`test`, `example`, `invalid`, `localhost`, `local`, `onion`). ⚠️ **Heads up when you switch `check_tld` on**: these are reserved so they can never be delegated, so they aren't in the root zone and are rejected — which is correct for a signup form, and will also turn your fixtures at `user@example.test` red. One line, scoped to where it belongs: `config.additional_tlds = Nondisposable::Tld::SPECIAL_USE if Rails.env.local?`
|
|
63
|
+
- **`config.blocked_tlds`** and **`config.allowed_tlds`**: refuse real TLDs you don't want (the free Freenom set `tk ml ga cf gq` is the usual reason), or invert it into an allowlist.
|
|
64
|
+
- **Lookalike detection**: `Nondisposable.suggestion_for("someone@gmial.com") # => "someone@gmail.com"`, matching against a bundled list of ~240 well-known providers using optimal string alignment distance, so an adjacent swap counts as the one mistake a human actually made. This catches what a TLD check structurally cannot — `.co`, `.cm` and `.om` are Colombia, Cameroon and Oman, all real. Always available, never blocks anything.
|
|
65
|
+
- **`config.reject_lookalike_domains`** (default `false`): turns that suggestion into a validation error naming the correction. Off by default on purpose — a suggestion is a guess about intent, and a wrong guess stops a real person signing up with their real address. `config.lookalike_distance` (default `1`) and `config.additional_email_providers` tune it.
|
|
66
|
+
- **`Nondisposable.valid_tld?(email)`** and **`Nondisposable.suggestion_for(email)`** as direct checks, alongside the existing `disposable?`.
|
|
67
|
+
- **`rake nondisposable:tlds:update`**: refreshes the bundled snapshot from IANA. A maintenance task for a checkout of this gem, not something host apps run — it refuses to overwrite a good list with an implausibly short one, writes atomically, and preserves IANA's version header (readable via `Nondisposable::Tld.version`).
|
|
68
|
+
- New error messages: `invalid_tld_error_message`, `blocked_tld_error_message`, `lookalike_error_message` (which interpolates `%{suggestion}`).
|
|
69
|
+
|
|
70
|
+
### Changed
|
|
71
|
+
|
|
72
|
+
- The validator now adds **at most one error** however many checks fire, and prefers the most useful message available: whenever the gem can name a correction, that phrasing wins over the generic "doesn't look like a real email address". The disposable check still takes precedence over both — a correct spelling wouldn't help a throwaway provider.
|
|
73
|
+
|
|
1
74
|
## [0.3.0] - 2026-08-09
|
|
2
75
|
|
|
3
76
|
### Added
|
data/README.md
CHANGED
|
@@ -17,6 +17,17 @@ That's it! You're done.
|
|
|
17
17
|
|
|
18
18
|
The gem also provides a job you can run daily to keep your disposable domain list up to date.
|
|
19
19
|
|
|
20
|
+
It can also catch the other kind of bad address — the typo. `user@gmail.con` isn't a throwaway, it's a real person whose account nobody will ever be able to reach, because `.con` doesn't exist. Two opt-in checks:
|
|
21
|
+
|
|
22
|
+
```ruby
|
|
23
|
+
Nondisposable.configure do |config|
|
|
24
|
+
config.check_tld = true # reject TLDs that aren't in the IANA root zone
|
|
25
|
+
config.reject_lookalike_domains = true # and addresses one keystroke from a real provider
|
|
26
|
+
end
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
See [Catching typos](#catching-typos).
|
|
30
|
+
|
|
20
31
|
## Installation
|
|
21
32
|
|
|
22
33
|
Add this line to your application's Gemfile:
|
|
@@ -103,19 +114,191 @@ Nondisposable.configure do |config|
|
|
|
103
114
|
# Also match parent domains: an email at x.tempmail.com is blocked when
|
|
104
115
|
# tempmail.com is on the list. Set to false for exact matches only.
|
|
105
116
|
config.check_parent_domains = true
|
|
117
|
+
|
|
118
|
+
# --- Where the blocklist comes from (see "Blocklist sources") ---
|
|
119
|
+
config.sources = Nondisposable::Sources::DEFAULT # or ::RECOMMENDED
|
|
120
|
+
config.corroboration_threshold = 2 # independent lists a :corroborating entry needs
|
|
121
|
+
config.block_privacy_relays = false # Hide My Email, DuckDuckGo, SimpleLogin, addy.io…
|
|
122
|
+
|
|
123
|
+
# --- Catching typos (both OFF by default) ---
|
|
124
|
+
|
|
125
|
+
# Reject addresses whose TLD isn't in the IANA root zone: gmail.con, outlook.ed
|
|
126
|
+
config.check_tld = true
|
|
127
|
+
config.additional_tlds = [] # accept a TLD newer than this gem
|
|
128
|
+
config.blocked_tlds = %w[tk ml ga cf gq] # refuse real TLDs you don't want
|
|
129
|
+
config.allowed_tlds = nil # or allowlist: %w[es com] rejects everything else
|
|
130
|
+
|
|
131
|
+
# Reject addresses one keystroke from a well-known provider: gmail.co, gmial.com
|
|
132
|
+
config.reject_lookalike_domains = false
|
|
133
|
+
config.lookalike_distance = 1 # edits that still count as a typo
|
|
134
|
+
config.additional_email_providers = [] # your own domains / regional providers
|
|
135
|
+
|
|
136
|
+
config.invalid_tld_error_message = "doesn't look like a real email address"
|
|
137
|
+
config.blocked_tld_error_message = "domain ending is not allowed"
|
|
138
|
+
config.lookalike_error_message = "looks like a typo. Did you mean %{suggestion}?"
|
|
106
139
|
end
|
|
107
140
|
```
|
|
108
141
|
|
|
109
142
|
#### Parent domain matching
|
|
110
143
|
|
|
111
|
-
With `check_parent_domains` enabled (the default since 0.3.0), `user@x.tempmail.com` is blocked when `tempmail.com` is on the list. The check walks
|
|
144
|
+
With `check_parent_domains` enabled (the default since 0.3.0), `user@x.tempmail.com` is blocked when `tempmail.com` is on the list. The check walks every parent down to the registrable domain (`a.b.c.d.tempmail.com` → … → `d.tempmail.com` → `tempmail.com`), however deep, and never matches against a public suffix like `com` or `co.uk`. All candidates are checked in a single indexed query. (Until 0.6.0 the walk stopped after 3 parent labels, so whoever controlled a listed domain could escape it with a fourth.)
|
|
145
|
+
|
|
146
|
+
Since 0.6.0 the walk also never matches a **public suffix** — `co.uk`, `com.br`, `github.io` — using a snapshot of the [Public Suffix List](https://publicsuffix.org) bundled with the gem (no runtime dependency, no network). Upstream lists really do carry entries like `it.com` and `ddns.net`, under which anybody can register a name; one such line would otherwise refuse every unrelated company below it. Public suffixes are also never stored and never accepted from `additional_domains`.
|
|
147
|
+
|
|
148
|
+
If you need to allowlist a specific subdomain under a listed parent, add the exact subdomain to `excluded_domains`; an exact exclusion always wins. Excluding a parent domain also unblocks its subdomains (unless a more specific subdomain is itself listed).
|
|
149
|
+
|
|
150
|
+
#### Who wins when lists disagree
|
|
151
|
+
|
|
152
|
+
One precedence table, applied both when an address is checked and when the stored list is built. Top wins:
|
|
112
153
|
|
|
113
|
-
|
|
154
|
+
| # | List | Effect |
|
|
155
|
+
|---|---|---|
|
|
156
|
+
| 1 | `config.excluded_domains` (yours) | allowed |
|
|
157
|
+
| 2 | `data/never_disposable.txt` (bundled: real providers and ISPs, plus `data/email_providers.txt`) | allowed, including subdomains |
|
|
158
|
+
| 3 | `config.additional_domains` (yours) | blocked |
|
|
159
|
+
| 4 | `data/privacy_relays.txt` (bundled) | `config.block_privacy_relays` decides |
|
|
160
|
+
| 5 | the upstream sources | blocked |
|
|
161
|
+
|
|
162
|
+
> [!IMPORTANT]
|
|
163
|
+
> **Changed in 0.6.0:** before, `additional_domains` was checked first, so a domain in both `additional_domains` and `excluded_domains` was blocked. Now your exclusion wins. And because row 2 outranks row 3, `additional_domains` can no longer block a real provider such as `gmail.com` or `mail.ru` (it is ignored with a warning in the log): that list is for throwaway domains the upstream lists missed, and refusing a whole real provider is a business rule that belongs in your own validation, where it can tell the user why.
|
|
164
|
+
|
|
165
|
+
Row 2 is why upgrading also **unblocks** two domains the default upstream list carried on 2026-10-17: `sify.com` (an Indian ISP) and `21cn.com` (China Telecom's webmail). A false positive is a real person who cannot sign up; if you find one, it belongs in `never_disposable.txt` (open a PR) or in your `excluded_domains`.
|
|
114
166
|
|
|
115
167
|
#### Failure mode
|
|
116
168
|
|
|
117
169
|
By default (`on_check_failure = :allow`), if the disposable check raises — say, the database is briefly unavailable — the signup goes through and an error is logged. This is availability-first: a broken check should not lock everyone out of signup. If you'd rather fail closed (reject signups whenever the check cannot run, as versions before 0.3.0 did), set `config.on_check_failure = :reject`.
|
|
118
170
|
|
|
171
|
+
### Catching typos
|
|
172
|
+
|
|
173
|
+
A disposable address is someone hiding from you. A typo is someone who wanted to reach you and won't be able to. Both leave you with a useless row in the users table, so `nondisposable` can catch both — but the typo checks are **opt-in**, because upgrading a gem should never start rejecting addresses that were fine yesterday.
|
|
174
|
+
|
|
175
|
+
#### `config.check_tld` — is that a real domain ending?
|
|
176
|
+
|
|
177
|
+
Every TLD that exists is in the [IANA root zone database](https://data.iana.org/TLD/tlds-alpha-by-domain.txt), and the gem ships a snapshot of it (~1,438 entries, about 9 KB, loaded once into a frozen `Set`). `.con` has never been in it, and never will be.
|
|
178
|
+
|
|
179
|
+
```ruby
|
|
180
|
+
config.check_tld = true
|
|
181
|
+
|
|
182
|
+
Nondisposable.valid_tld?("user@gmail.com") # => true
|
|
183
|
+
Nondisposable.valid_tld?("user@gmail.con") # => false
|
|
184
|
+
Nondisposable.valid_tld?("example@gmailmcom") # => false (no TLD at all)
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
Details worth knowing:
|
|
188
|
+
|
|
189
|
+
- **No dot, no TLD.** `example@gmailmcom` models a typo observed in production: the dot was missed entirely. A domain with no TLD can't end in a real one, so it's rejected. If your app accepts single-label intranet addresses like `you@localhost`, leave this off for that model.
|
|
190
|
+
- **IP literals are left alone.** `user@192.168.0.1` and `user@[10.0.0.1]` are a format question, and a TLD check has no business answering it. Pair with `format:` if you care.
|
|
191
|
+
- **Unicode TLDs are never rejected.** IANA lists internationalised TLDs in punycode (`XN--FIQS8S`), and converting `例え.テスト` to that form needs an IDN library this gem deliberately doesn't depend on. Rather than reject every unicode address, it declines to judge them. Punycode-form addresses — what mail clients actually send — are checked normally.
|
|
192
|
+
- **A stale snapshot can't trap you.** If ICANN delegates a TLD after your gem version shipped, `config.additional_tlds = %w[newtld]` accepts it immediately, with no release to wait for.
|
|
193
|
+
|
|
194
|
+
> [!IMPORTANT]
|
|
195
|
+
> **`.test` and `.example` are rejected, and that will turn your test suite red.**
|
|
196
|
+
>
|
|
197
|
+
> RFC 6761 and RFC 2606 reserve `test`, `example`, `invalid` and `localhost` precisely so they can never be delegated — which is why they aren't in the root zone, and why your fixtures live at `user@example.test` in the first place. Rejecting them is right for a signup form (no human types `me@home.test`, and mail could never be delivered there), but it's a configuration question, not a bug:
|
|
198
|
+
>
|
|
199
|
+
> ```ruby
|
|
200
|
+
> # config/initializers/nondisposable.rb
|
|
201
|
+
> config.additional_tlds = Nondisposable::Tld::SPECIAL_USE if Rails.env.local?
|
|
202
|
+
> ```
|
|
203
|
+
>
|
|
204
|
+
> Scope it to your non-production environments, so production still refuses an address nobody could ever answer.
|
|
205
|
+
|
|
206
|
+
`blocked_tlds` refuses TLDs that are perfectly real but that you'd rather not see — the historically free Freenom set (`tk`, `ml`, `ga`, `cf`, `gq`) is the usual suspect. `allowed_tlds` inverts it into an allowlist; be careful, `%w[es]` turns away every `.com` customer you have.
|
|
207
|
+
|
|
208
|
+
#### `config.reject_lookalike_domains` — did they mean gmail.com?
|
|
209
|
+
|
|
210
|
+
A TLD check structurally **cannot** catch the most common typos, because `.co` (Colombia), `.cm` (Cameroon) and `.om` (Oman) are all real, delegated TLDs. `user@gmail.co` is a well-formed address at a real TLD and still almost always a finger that slipped off the `m`.
|
|
211
|
+
|
|
212
|
+
So the second check asks a different question: is this domain one edit away from a well-known provider, while not being one itself?
|
|
213
|
+
|
|
214
|
+
```ruby
|
|
215
|
+
Nondisposable.suggestion_for("someone@gmial.com") # => "someone@gmail.com"
|
|
216
|
+
Nondisposable.suggestion_for("someone@gmail.co") # => "someone@gmail.com"
|
|
217
|
+
Nondisposable.suggestion_for("someone@gmail.com") # => nil
|
|
218
|
+
Nondisposable.suggestion_for("someone@mail.com") # => nil (a real provider)
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
`suggestion_for` is always available and **never blocks anything** — show it as a hint and let the human decide. That's the gentler way to use this, and the one to reach for first:
|
|
222
|
+
|
|
223
|
+
```erb
|
|
224
|
+
<% if (did_you_mean = Nondisposable.suggestion_for(@user.email)) %>
|
|
225
|
+
<p>Did you mean <%= did_you_mean %>?</p>
|
|
226
|
+
<% end %>
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
The matching uses optimal string alignment distance rather than plain Levenshtein, so an adjacent swap counts as the one mistake a human actually made: `gmial.com` is distance 1, not 2.
|
|
230
|
+
|
|
231
|
+
One edit is not enough to tell a typo from a real domain, though. Replayed against the [Tranco](https://tranco-list.eu) top 1M, "one edit from a provider" flags **707** domains, and many of them are large real organisations:
|
|
232
|
+
|
|
233
|
+
- `ge.com` (General Electric), `ml.com` (Merrill Lynch), `mi.com` (Xiaomi), `se.com` (Schneider Electric) → `me.com`
|
|
234
|
+
- `ea.com` (Electronic Arts), `aa.com` (American Airlines) → `ya.com`
|
|
235
|
+
- `bol.com`, `aon.com`, `uol.com` (a major Brazilian mail provider) → `aol.com`
|
|
236
|
+
- `one.com` → `ono.com`; `hive.com`, `life.com` → `live.com`; `grail.com`, `pmail.com` → `gmail.com`
|
|
237
|
+
- `web.dev` → `web.de`; `yahoo.com.au` → `yahoo.com.ar`
|
|
238
|
+
|
|
239
|
+
So since 0.5.0 suggestions are filtered before anyone sees them: a provider name of four characters or fewer (`me`, `ya`, `aol`, `mail`, `live`…) only takes edits after the name (`me.cmo` yes, `ge.com` no), and every edit has to look like a finger — a swap, a dropped letter (not the first), a doubled letter, an extra vowel, a neighbouring key on QWERTY, or the same key with an accent. A letter from across the keyboard (`grail`, `pmail`, `.au` for `.ar`) is a different word, not a slip. On a 30% holdout of the labelled corpus that keeps 13 of 16 reviewed typos and raises precision from 8% to 45% — measured retrospectively: the split was chosen before tuning but only committed with the tuned filters, so treat it as an estimate, not a pre-registered result. Better, still a guess — which is why a guess never refuses anyone on its own.
|
|
240
|
+
|
|
241
|
+
#### `reject_lookalike_domains`: hard for known typos, confirmable for everything else
|
|
242
|
+
|
|
243
|
+
```ruby
|
|
244
|
+
config.reject_lookalike_domains = :confirmable # `true` means the same thing
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
| The domain is… | Example | Result |
|
|
248
|
+
|---|---|---|
|
|
249
|
+
| an exact provider (bundled or `additional_email_providers`) | `gmail.com`, `mail.com` | accepted |
|
|
250
|
+
| an impossible TLD with a nameable correction (`check_tld` on or not) | `gmail.con`, `gmailmcom` | **refused**: "Did you mean …gmail.com?" |
|
|
251
|
+
| a reviewed typo in [`data/known_typo_domains.txt`](data/known_typo_domains.txt) | `gmial.com`, `hotmial.com`, `yaho.com` | **refused**: "Did you mean …?" |
|
|
252
|
+
| any other lookalike the ranker suggests | `gnail.com` | **confirmable**: same message, unless the user confirmed the address |
|
|
253
|
+
| anything else | `ge.com`, `yahoo.com.au` | accepted |
|
|
254
|
+
|
|
255
|
+
A disposable provider is still refused first, and a blocked TLD is still refused after a confirmation. A host's `additional_email_providers` outranks the known-typo list.
|
|
256
|
+
|
|
257
|
+
The known-typo list is short (41 entries) on purpose. An entry needs direct evidence that nobody has a mailbox there — a redirect to the provider or the provider's own DNS (`yaho.com`, `homail.com`), a parking or for-sale network (`gamail.com`, `hotmial.com`), a typosquat redirect or catch-all (`gmial.com`, `gmali.com`), or a registered domain with no MX, A or AAAA record (`gmal.com`). "No MX" alone is not enough: SMTP falls back to the A record. The evidence for every entry is in `test/fixtures/lookalike_corpus.csv`, and the suite fails if an entry loses it.
|
|
258
|
+
|
|
259
|
+
**Confirmable means the user can say "yes, this address is right".** Give the model somewhere to put that answer:
|
|
260
|
+
|
|
261
|
+
```ruby
|
|
262
|
+
class User < ApplicationRecord
|
|
263
|
+
include Nondisposable::ConfirmableEmail
|
|
264
|
+
confirmable_email :email # adds a virtual email_lookalike_confirmed attribute
|
|
265
|
+
|
|
266
|
+
validates :email, nondisposable: true
|
|
267
|
+
end
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
```erb
|
|
271
|
+
<%# Only when the confirmable error is showing: %>
|
|
272
|
+
<% if @user.errors.of_kind?(:email, :lookalike_domain) %>
|
|
273
|
+
<%= f.check_box :email_lookalike_confirmed %>
|
|
274
|
+
<%= f.label :email_lookalike_confirmed, "Yes, my email address is right" %>
|
|
275
|
+
<% end %>
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
Permit `:email_lookalike_confirmed` in your strong parameters. It's virtual — never persisted, and reset after every save — so it answers for one save; the error also carries the suggestion in `errors.details[:email]` (`suggestion:`). No concern needed if you'd rather wire it yourself: the validator only looks for a method named `<attribute>_lookalike_confirmed` that returns exactly `true`.
|
|
279
|
+
|
|
280
|
+
> [!WARNING]
|
|
281
|
+
> Without the checkbox, a confirmable error has no way past it — a real `gnail.com` user would be stuck. Wire the confirmation before you switch this on, or keep using `suggestion_for` as a hint only.
|
|
282
|
+
|
|
283
|
+
Before 0.5.0, `true` refused every lookalike outright. If you set it, upgrading makes it **softer**, never stricter: reviewed typos and impossible TLDs are still refused, every other lookalike becomes confirmable, and many of the old matches (`ge.com`, `bol.com`) are no longer suggested at all.
|
|
284
|
+
|
|
285
|
+
Matching stops at **one** edit by default (`lookalike_distance`; 2 is coarser and documented as risky, 0 switches the whole layer off, list included). Add anything we've missed with `config.additional_email_providers`; doing so both makes it a suggestion candidate and stops its users being told they mistyped.
|
|
286
|
+
|
|
287
|
+
#### How the two combine
|
|
288
|
+
|
|
289
|
+
At most one error is added, however many checks fire — and whenever the gem can name a correction, that phrasing wins:
|
|
290
|
+
|
|
291
|
+
| Address | `check_tld` only | with `reject_lookalike_domains` |
|
|
292
|
+
|---|---|---|
|
|
293
|
+
| `user@gmail.con` | "looks like a typo. Did you mean user@gmail.com?" | same |
|
|
294
|
+
| `user@zzz.con` | "doesn't look like a real email address" | same |
|
|
295
|
+
| `user@gmail.co` | accepted | "looks like a typo. Did you mean user@gmail.com?" (a reviewed typo: refused) |
|
|
296
|
+
| `user@gnail.com` | accepted | same message, confirmable |
|
|
297
|
+
| `user@ge.com` | accepted | accepted |
|
|
298
|
+
| `user@tempmail.com` | "provider is not allowed" | same |
|
|
299
|
+
|
|
300
|
+
"Doesn't look like a real email address" is true but useless when we know what they meant, so a nameable correction always outranks the generic message.
|
|
301
|
+
|
|
119
302
|
### Direct Check
|
|
120
303
|
|
|
121
304
|
You can also check if an email is disposable directly:
|
|
@@ -123,8 +306,12 @@ You can also check if an email is disposable directly:
|
|
|
123
306
|
```ruby
|
|
124
307
|
Nondisposable.disposable?('user@example.com') # => false
|
|
125
308
|
Nondisposable.disposable?('user@disposable-email.com') # => true
|
|
309
|
+
Nondisposable.valid_tld?('user@example.con') # => false
|
|
310
|
+
Nondisposable.suggestion_for('user@gmial.com') # => "user@gmail.com"
|
|
126
311
|
```
|
|
127
312
|
|
|
313
|
+
Every check compares domains in one canonical form — lowercased, without a trailing dot, unicode labels converted to punycode, so `user@Bücher.DE.` and `user@xn--bcher-kva.de` get the same verdict. The local part is never touched (case and `+tags` are the user's business). The same normalisation is available as `Nondisposable::Domain.normalize(domain)` and `Nondisposable::Domain.from_email(email)`.
|
|
314
|
+
|
|
128
315
|
## Updating disposable domains
|
|
129
316
|
|
|
130
317
|
To manually update the list of disposable domains, run:
|
|
@@ -133,7 +320,7 @@ To manually update the list of disposable domains, run:
|
|
|
133
320
|
Nondisposable::DomainListUpdater.update
|
|
134
321
|
```
|
|
135
322
|
|
|
136
|
-
The fetch uses 10-second open/read timeouts and replaces the whole table atomically. If the remote fetch fails and your table is empty (e.g. a fresh install without network access), the updater automatically seeds from the blocklist snapshot bundled with the gem so you're never left unprotected; you can also trigger that explicitly with `Nondisposable::DomainListUpdater.seed` (it only seeds an empty table, and never overwrites existing data).
|
|
323
|
+
The fetch uses 10-second open/read timeouts and replaces the whole table atomically (see [Blocklist sources](#blocklist-sources) for how several lists are merged and guarded). If the remote fetch fails and your table is empty (e.g. a fresh install without network access), the updater automatically seeds from the blocklist snapshot bundled with the gem so you're never left unprotected; you can also trigger that explicitly with `Nondisposable::DomainListUpdater.seed` (it only seeds an empty table, and never overwrites existing data).
|
|
137
324
|
|
|
138
325
|
It's important you keep your disposable domain list up to date. `nondisposable` will read from the latest version of the [`disposable-email-domains`](https://github.com/disposable-email-domains/disposable-email-domains) list, which is typically updated every few days.
|
|
139
326
|
|
|
@@ -148,6 +335,121 @@ production:
|
|
|
148
335
|
schedule: every day at 3am US/Pacific
|
|
149
336
|
```
|
|
150
337
|
|
|
338
|
+
## Blocklist sources
|
|
339
|
+
|
|
340
|
+
Out of the box the gem uses the one list it always has, [`disposable-email-domains`](https://github.com/disposable-email-domains/disposable-email-domains). Upgrading changes nothing. You can add more:
|
|
341
|
+
|
|
342
|
+
```ruby
|
|
343
|
+
config.sources = Nondisposable::Sources::RECOMMENDED
|
|
344
|
+
```
|
|
345
|
+
|
|
346
|
+
`RECOMMENDED` adds [fakefilter](https://github.com/7c/fakefilter), which watches temp-mail generators directly. On 2026-10-17 it added **3,275 domains** (13,168 in total, against 9,893), and none of them was a known real provider.
|
|
347
|
+
|
|
348
|
+
> [!NOTE]
|
|
349
|
+
> **More lists do not catch fresh burners.** `mail-burn.com`, the domain that prompted this feature, was live and handing out inboxes on 2026-10-17. Of the measured lists only disposable/disposable carried it, from its own crawl. groundcat had it too, but only because it copies disposable/disposable. That is **one origin**, and corroboration rightly refuses to count copies, so no precision-first combination of public lists blocks it. A fresh burner shows up in one crawler's list first. Until a list exists that publishes *evidence* for each fresh domain, block a domain you have verified yourself with `config.additional_domains`, and send it upstream.
|
|
350
|
+
|
|
351
|
+
### Trust and lineage
|
|
352
|
+
|
|
353
|
+
Every source is a `Nondisposable::Source` with a `trust`:
|
|
354
|
+
|
|
355
|
+
- `:primary`: every domain it lists is blocked (subject to the precedence table).
|
|
356
|
+
- `:corroborating`: a domain it lists is blocked only once `config.corroboration_threshold` (default 2) **independent origins** list it.
|
|
357
|
+
|
|
358
|
+
"Independent" is the key word. Public blocklists copy each other: [`disposable/disposable`](https://github.com/disposable/disposable) declares that it imports mailchecker, fakefilter, wesbos and unkn0w, and groundcat imports disposable/disposable. Two copies of one mistake are one observation, so each source declares what it is `derived_from`, and a vote only counts when none of its upstreams that you also load carries the same domain.
|
|
359
|
+
|
|
360
|
+
Measured on 2026-10-17, that rule has teeth. With lineage declared, adding disposable/disposable, mailchecker and groundcat as corroborating sources to `RECOMMENDED` adds **zero** domains: everything they agree on is already explained by an upstream you load. Ignoring lineage instead would have "corroborated" 68,692 more, 64 of them in the [Tranco](https://tranco-list.eu) top 100k (`compaq.com`, `cafe24.com`, `24hourfitness.com`, `asana.biz`…). That is why `RECOMMENDED` is two primaries and nothing else.
|
|
361
|
+
|
|
362
|
+
The other measured sources ship as constants, so you can opt in knowingly:
|
|
363
|
+
|
|
364
|
+
| Constant | List | Licence | Trust | Notes |
|
|
365
|
+
|---|---|---|---|---|
|
|
366
|
+
| `DISPOSABLE_EMAIL_DOMAINS` | disposable-email-domains/disposable-email-domains | CC0 | primary | the default; also the bundled seed |
|
|
367
|
+
| `FAKEFILTER` | 7c/fakefilter | BSD-3 | primary | in `RECOMMENDED` |
|
|
368
|
+
| `DISPOSABLE` | disposable/disposable-email-domains | MIT | corroborating | ~97k domains, aggregates most others |
|
|
369
|
+
| `MAILCHECKER` | FGRibreau/mailchecker | MIT | corroborating | lists privacy relays |
|
|
370
|
+
| `GROUNDCAT` | groundcat/disposable-email-domain-list | MIT | corroborating | aggregator |
|
|
371
|
+
| `WESBOS` | wesbos/burner-email-providers | MIT | corroborating | no declared lineage, but 57% of it is also in mailchecker |
|
|
372
|
+
|
|
373
|
+
> [!WARNING]
|
|
374
|
+
> **About promoting `DISPOSABLE` to primary.** It is the broadest list, and the only one that carried `mail-burn.com` on 2026-10-17. On top of `RECOMMENDED` it adds 84,341 domains (70 of them in the Tranco top 100k, listed in `docs/evidence/2026-10-17-disposable-as-primary-tranco-top100k.csv`), and among them are real mail domains: China Unicom (`wo.com.cn`), the Japanese ISP ODN (`odn.ad.jp`), UQ mobile (`uqmobile.jp`), the Moscow ISP MTU (`mtu-net.ru`), plus `cafe24.com`, `24hourfitness.com`, `foxwoods.com` and more. The ones we verified are now in `never_disposable.txt`, but the rest of the list is unknown. If you still want it, `Nondisposable::Sources::DISPOSABLE.with(trust: :primary)` does it. For one specific burner domain, `config.additional_domains` is the precise tool.
|
|
375
|
+
|
|
376
|
+
Your own source is a Hash or a `Source`:
|
|
377
|
+
|
|
378
|
+
```ruby
|
|
379
|
+
config.sources = Nondisposable::Sources::RECOMMENDED + [
|
|
380
|
+
{ name: "ours", url: "https://example.com/burners.txt", trust: :corroborating, derived_from: %w[mailchecker] }
|
|
381
|
+
]
|
|
382
|
+
```
|
|
383
|
+
|
|
384
|
+
Sources must be `https`, plain text, one domain per line (`#` comments allowed). Names and URLs must be unique, and at least one source must be `:primary`.
|
|
385
|
+
|
|
386
|
+
### Privacy relays
|
|
387
|
+
|
|
388
|
+
Apple Hide My Email, DuckDuckGo, Firefox Relay, SimpleLogin/Proton Pass and addy.io forward to a real, long-lived inbox. They are not throwaways, but they do let one person hold many addresses. That is a policy decision, so it is a setting: `config.block_privacy_relays = false` (the default) **allows** them even when an upstream list carries them (mailchecker does), and `true` **refuses** them even when none does. The list is `data/privacy_relays.txt`; each domain covers its subdomains.
|
|
389
|
+
|
|
390
|
+
### Upgrading an existing install
|
|
391
|
+
|
|
392
|
+
Multi-source lists need one new table. The migration is additive, so you can deploy first and migrate later:
|
|
393
|
+
|
|
394
|
+
```bash
|
|
395
|
+
rails generate nondisposable:upgrade
|
|
396
|
+
rails db:migrate
|
|
397
|
+
```
|
|
398
|
+
|
|
399
|
+
Until the table exists (and again if you roll the migration back), the updater uses the default list exactly as before and logs a warning if you have configured other sources. During a rolling deploy an old process may still publish the single-source list once; the next update from a new process rebuilds the full one.
|
|
400
|
+
|
|
401
|
+
### How an update is published
|
|
402
|
+
|
|
403
|
+
Each source is fetched into its own row in `nondisposable_source_snapshots` (status, timestamp, sha256, count, the list itself). The stored list is then **rebuilt** from each source's last good snapshot, in one transaction, under a database advisory lock on PostgreSQL and MySQL (a process lock on SQLite, which already allows only one writer). Seeding takes the same lock. A new snapshot is refused, and the source's last good one kept, when:
|
|
404
|
+
|
|
405
|
+
- the download fails, takes longer than the source's `deadline` (30 s), or is larger than its `max_bytes` (8 MB, checked while streaming);
|
|
406
|
+
- more than 1% of its lines are not domain names (an HTML error page, a format change);
|
|
407
|
+
- it has fewer than `minimum_count` domains (1,000 for the presets);
|
|
408
|
+
- it is less than half the source's high-water mark, the largest size ever accepted for it (kept on its newest good snapshot, so neither pruning nor a series of 49% cuts can wear it down; `reset_baseline` is the human override), or more than three times its last good one;
|
|
409
|
+
- it is identical to a snapshot a human rolled back.
|
|
410
|
+
|
|
411
|
+
Every size is a count of *storable* domains (not junk, not a public suffix, not a known real provider, not a privacy relay), counted the same way whatever your settings, so a payload padded with such lines cannot pass for a full list and changing `block_privacy_relays` or `excluded_domains` never trips a guard. The list is never published empty. A primary that has never been fetched successfully either stands in its bundled list (the default source) or blocks publishing (any other), so a first run can never replace the seed with less than it. Additions and removals are logged per source. A source that lists a known real provider, or a public suffix, is logged as a possible poisoning, and the entry is dropped. Accepting a snapshot, publishing and rolling back all happen under the same lock; downloads happen outside it.
|
|
412
|
+
|
|
413
|
+
> [!WARNING]
|
|
414
|
+
> **Two concurrency limits, by design:**
|
|
415
|
+
>
|
|
416
|
+
> - **SQLite: run the refresh from one process.** PostgreSQL and MySQL get a database-wide advisory lock; SQLite has none, so the lock is per process. Two *processes* refreshing at once can interleave (one can republish a merge it read before the other rolled a source back). Schedule `DisposableEmailDomainListUpdateJob` on a single job runner, and run `rollback` from that same process or when no update is running.
|
|
417
|
+
> - **Rolling deploys: quiesce the old updater.** Processes still running 0.5 or earlier replace the table without taking the lock, so their single-source list can overwrite a multi-source one, or leave back a row a new process just withdrew. When upgrading: pause the scheduled refresh job, deploy, run `rails g nondisposable:upgrade && rails db:migrate`, then resume it and run `Nondisposable::DomainListUpdater.update` once.
|
|
418
|
+
|
|
419
|
+
Two more tools:
|
|
420
|
+
|
|
421
|
+
```ruby
|
|
422
|
+
Nondisposable::DomainListUpdater.publish # rebuild from snapshots, no download (after a config change)
|
|
423
|
+
Nondisposable::DomainListUpdater.rollback("fakefilter") # withdraw that source's newest snapshot (its payload stays refused) and republish
|
|
424
|
+
Nondisposable::DomainListUpdater.reset_baseline("fakefilter") # accept a genuine shrink: the guard now measures from today's size
|
|
425
|
+
```
|
|
426
|
+
|
|
427
|
+
## Updating the TLD list
|
|
428
|
+
|
|
429
|
+
The TLD snapshot is the deliberate opposite of the disposable list: it ships **inside the gem**, not in your database, and there is nothing for your app to schedule. Disposable domains number in the thousands and change every few days; TLDs are ~1,438 strings that change a handful of times a year, so a frozen `Set` costs one file read at boot, answers in O(1) with no query per signup, needs no migration, and can't be empty on a fresh install.
|
|
430
|
+
|
|
431
|
+
If ICANN delegates a TLD your installed version doesn't know about, don't wait for a release — that's what `config.additional_tlds` is for.
|
|
432
|
+
|
|
433
|
+
Maintainers refresh the snapshot from a checkout of this gem:
|
|
434
|
+
|
|
435
|
+
```bash
|
|
436
|
+
rake nondisposable:tlds:update # rewrites data/iana_tlds.txt from data.iana.org
|
|
437
|
+
```
|
|
438
|
+
|
|
439
|
+
It refuses to overwrite a good list with an implausibly short one (a captive portal or a truncated body served with a `200`), writes to a temp file and moves it into place so an interrupted run can't leave half a root zone behind, and keeps IANA's own version header so you can always see which root zone you're shipping:
|
|
440
|
+
|
|
441
|
+
```ruby
|
|
442
|
+
Nondisposable::Tld.version # => "2026082301, Last Updated Mon Aug 24 07:07:01 2026 UTC"
|
|
443
|
+
```
|
|
444
|
+
|
|
445
|
+
The bundled Public Suffix List (`data/public_suffix_list.dat`, redistributed unmodified under the MPL 2.0) is refreshed the same way, with the same safeguards:
|
|
446
|
+
|
|
447
|
+
```bash
|
|
448
|
+
rake nondisposable:psl:update # rewrites data/public_suffix_list.dat from publicsuffix.org
|
|
449
|
+
```
|
|
450
|
+
|
|
451
|
+
A stale PSL can only be less protective, never lock anyone out: a suffix created after the snapshot simply isn't known to be one.
|
|
452
|
+
|
|
151
453
|
## Troubleshooting
|
|
152
454
|
|
|
153
455
|
### SSL certificate verify failed (unable to get certificate CRL)
|
data/Rakefile
CHANGED
|
@@ -3,6 +3,11 @@
|
|
|
3
3
|
require "bundler/gem_tasks"
|
|
4
4
|
require "rake/testtask"
|
|
5
5
|
|
|
6
|
+
# nondisposable:tlds:update — refreshes data/iana_tlds.txt from the IANA root
|
|
7
|
+
# zone. A maintenance task for this checkout, not something host apps run; see
|
|
8
|
+
# the header of lib/nondisposable/tld_list_updater.rb.
|
|
9
|
+
Dir.glob("lib/tasks/*.rake").each { |task| load task }
|
|
10
|
+
|
|
6
11
|
Rake::TestTask.new(:test) do |t|
|
|
7
12
|
t.libs << "lib"
|
|
8
13
|
t.libs << "test"
|
|
@@ -4,52 +4,71 @@ module Nondisposable
|
|
|
4
4
|
class DisposableDomain < ApplicationRecord
|
|
5
5
|
validates :name, presence: true, uniqueness: { case_sensitive: false }
|
|
6
6
|
|
|
7
|
-
#
|
|
8
|
-
#
|
|
9
|
-
# a.b.c.
|
|
10
|
-
#
|
|
11
|
-
#
|
|
12
|
-
# dependency): the walk never queries single-label suffixes (bare TLDs like
|
|
13
|
-
# "com"), but without a PSL it cannot tell that multi-label suffixes like
|
|
14
|
-
# "co.uk" are public — those candidates are harmless unless a public suffix
|
|
15
|
-
# is deliberately added to the list. Conversely, a throwaway address nested
|
|
16
|
-
# more than 3 labels below a listed domain escapes the walk; blocklisted
|
|
17
|
-
# domains are registrable domains, so 3 levels covers practical abuse.
|
|
7
|
+
# Kept so code that referenced it still loads; no longer used. Until 0.6.0
|
|
8
|
+
# the parent walk stopped after this many labels, which let anybody who
|
|
9
|
+
# controls a listed domain escape it with `a.b.c.d.tempmail.com`. The walk
|
|
10
|
+
# now covers every parent down to the registrable domain: DNS itself caps
|
|
11
|
+
# a name at 253 characters, so the candidate list is bounded anyway.
|
|
18
12
|
PARENT_MATCH_DEPTH = 3
|
|
19
13
|
|
|
14
|
+
# Kept (and equal to Domain::MAX_RAW_LENGTH) for code that referenced it.
|
|
15
|
+
# The raw-length bound itself now lives in Domain.normalize, which applies
|
|
16
|
+
# it AFTER stripping invisible characters: judging raw length here let
|
|
17
|
+
# padding hide a listed domain.
|
|
18
|
+
MAX_INPUT_LENGTH = Nondisposable::Domain::MAX_RAW_LENGTH
|
|
19
|
+
|
|
20
20
|
class << self
|
|
21
|
+
# Whether `domain` is disposable, according to the precedence table in
|
|
22
|
+
# Nondisposable::Precedence: host exclusions, then the bundled
|
|
23
|
+
# never-disposable list, then host additions, then the privacy-relay
|
|
24
|
+
# policy, and only then this table.
|
|
21
25
|
def disposable?(domain)
|
|
22
|
-
|
|
26
|
+
# A name longer than DNS allows (over 253 characters or a label over
|
|
27
|
+
# 63 once canonical, or over 1,024 code points of raw input after
|
|
28
|
+
# invisible characters are stripped) is TRUE: it can never be a
|
|
29
|
+
# deliverable domain, and a check whose job is keeping junk out must
|
|
30
|
+
# not call junk fine. Judged by Domain.too_long?, never on the raw
|
|
31
|
+
# length: a raw cap let invisible padding (U+00AD x 1013 +
|
|
32
|
+
# "tempmail.com") hide a listed name. Domain.normalize bounds its own
|
|
33
|
+
# work on oversized input, so the parent walk below never runs on one.
|
|
34
|
+
raw = domain
|
|
35
|
+
return false if raw.to_s.strip.empty?
|
|
36
|
+
return true if Nondisposable::Domain.too_long?(raw)
|
|
23
37
|
|
|
24
|
-
domain =
|
|
25
|
-
|
|
26
|
-
|
|
38
|
+
domain = Nondisposable::Domain.normalize(raw)
|
|
39
|
+
return false if domain.empty?
|
|
40
|
+
# Not UTF-8: no list holds it and no mail server can deliver to it.
|
|
41
|
+
return false if domain.encoding == Encoding::BINARY && !domain.ascii_only?
|
|
27
42
|
|
|
28
|
-
|
|
29
|
-
return
|
|
43
|
+
verdict, candidates = Nondisposable::Precedence.decide(domain, match_candidates(domain))
|
|
44
|
+
return verdict == :block unless verdict == :lookup
|
|
30
45
|
|
|
31
|
-
|
|
32
|
-
#
|
|
33
|
-
#
|
|
34
|
-
|
|
46
|
+
# Rows written before 0.5.0 stored a unicode domain as it came
|
|
47
|
+
# (lowercased, not punycode). Match that spelling of every candidate
|
|
48
|
+
# still eligible, so upgrading never unblocks them; the next publish
|
|
49
|
+
# rewrites them canonically.
|
|
50
|
+
legacy = Nondisposable::Domain.unicode(raw)
|
|
51
|
+
unless legacy == domain || legacy.ascii_only?
|
|
52
|
+
candidates |= match_candidates(legacy).select { |name| candidates.include?(Nondisposable::Domain.normalize(name)) }
|
|
53
|
+
end
|
|
35
54
|
|
|
36
|
-
where(name: candidates
|
|
55
|
+
candidates.any? && where(name: candidates).exists? # one WHERE name IN (...)
|
|
37
56
|
end
|
|
38
57
|
|
|
39
|
-
#
|
|
40
|
-
#
|
|
58
|
+
# With check_parent_domains (the default): the domain and EVERY parent
|
|
59
|
+
# down to, and including, its registrable domain:
|
|
60
|
+
#
|
|
61
|
+
# a.b.c.d.tempmail.com => a.b.c.d.tempmail.com … d.tempmail.com, tempmail.com
|
|
62
|
+
#
|
|
63
|
+
# Never above the registrable domain: past it lie public suffixes (`com`,
|
|
64
|
+
# `co.uk`) and names like `sch.uk` that sit above a wildcard suffix
|
|
65
|
+
# (`*.sch.uk`), all belonging to unrelated registrants. Without
|
|
66
|
+
# check_parent_domains: the domain alone. Expects a normalized domain.
|
|
41
67
|
def match_candidates(domain)
|
|
42
|
-
|
|
68
|
+
namespace = Nondisposable::Precedence.namespace(domain)
|
|
69
|
+
return namespace if Nondisposable.configuration.check_parent_domains
|
|
43
70
|
|
|
44
|
-
|
|
45
|
-
candidates = [domain]
|
|
46
|
-
1.upto(PARENT_MATCH_DEPTH) do |depth|
|
|
47
|
-
parent = labels.drop(depth)
|
|
48
|
-
break if parent.size < 2
|
|
49
|
-
|
|
50
|
-
candidates << parent.join('.')
|
|
51
|
-
end
|
|
52
|
-
candidates
|
|
71
|
+
namespace.first(1)
|
|
53
72
|
end
|
|
54
73
|
end
|
|
55
74
|
|