valid_email_checker 0.1.0

data/NOTICE.md

@@ -0,0 +1,56 @@
+# Notices and licensing
+
+## What this gem embeds
+
+`valid_email_checker` is a wrapper. All of the actual email verification is
+done by the Rust crate **`check-if-email-exists`**, by Reacher, a verbatim copy
+of which is included in this repository at:
+
+    ext/valid_email_checker/vendor/check-if-email-exists/
+
+That copy is the upstream `core` package. Only two lines of its `Cargo.toml`
+differ from upstream — `publish` and `readme` — and the reason is recorded in a
+comment there. The exact upstream version and commit are recorded in
+`ext/valid_email_checker/vendor/UPSTREAM.json` and are also readable at runtime:
+
+```ruby
+ValidEmailChecker.upstream
+# => {name: "check-if-email-exists", version: "0.11.7", commit: "81da93e...", ...}
+```
+
+* Upstream project: https://github.com/reacherhq/check-if-email-exists
+* Upstream copyright: Copyright (C) 2018-2023 Reacher
+
+## License — please read this before deploying
+
+Upstream `check-if-email-exists` is offered under a **dual license**:
+
+1. **AGPL-3.0**, or
+2. a **commercial license** sold by Reacher at https://reacher.email/pricing
+
+Because this gem embeds and links that code, the combined work is a derivative
+of it. This gem is therefore distributed under the **AGPL-3.0** (see
+`LICENSE.txt`), and the AGPL's terms apply to your use of it.
+
+The consequence most people need to know about is **AGPL section 13**. If you
+run a modified version of this software and let users interact with it over a
+network, you must offer those users the corresponding source code of your
+modified version. Unlike the GPL, for AGPL-covered code this obligation is
+triggered by providing network access, not only by distributing binaries — so
+using this gem inside a public-facing web application is exactly the case the
+clause is written for.
+
+Nothing here is legal advice. If the AGPL does not suit how you intend to
+deploy, buy the commercial license from Reacher; that is precisely the reason
+the dual-license option exists. Note that a commercial license covers the
+upstream crate, and you would also need to arrange terms for this wrapper.
+
+## Third-party Rust dependencies
+
+Building the extension compiles the upstream crate's dependency tree
+(`tokio`, `reqwest`, `rustls`, `hickory-resolver`, `mailchecker` and others),
+each under its own license, predominantly MIT and Apache-2.0. To produce a
+full report:
+
+    cargo install cargo-license
+    cargo license --manifest-path ext/valid_email_checker/Cargo.toml

data/README.md

@@ -0,0 +1,316 @@
+# valid_email_checker
+
+Check whether an email address really exists — without sending any mail.
+
+This gem embeds the Rust crate
+[`check-if-email-exists`](https://github.com/reacherhq/check-if-email-exists)
+and calls it in-process through a native extension. There is no HTTP service to
+run and no API key to buy. A full check will:
+
+- validate the syntax, and suggest a correction for a near-miss domain;
+- resolve the domain's MX records;
+- open an SMTP conversation with the mail exchanger and ask about the recipient,
+  without ever completing a message;
+- report whether the address is disposable, role-based, catch-all, disabled, or
+  over quota.
+
+> **Licensing:** the embedded crate is AGPL-3.0, so this gem is too. If you plan
+> to use it in a network service, read [NOTICE.md](NOTICE.md) first — AGPL
+> section 13 will apply to you. Upstream sells a commercial license for exactly
+> this situation.
+
+## Install
+
+The gem compiles the embedded Rust crate at install time, so you need a Rust
+toolchain (see [rustup.rs](https://rustup.rs)):
+
+```console
+$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+$ gem install valid_email_checker
+```
+
+Or in a `Gemfile`:
+
+```ruby
+gem "valid_email_checker"
+```
+
+A cold build takes a few minutes — the dependency tree includes `tokio`,
+`rustls` and `reqwest`. Requires Ruby >= 3.1 and Rust >= 1.75.
+
+## Usage
+
+```ruby
+require "valid_email_checker"
+
+result = ValidEmailChecker.check("someone@gmail.com")
+
+result.reachable      # => :safe
+result.safe?          # => true
+result.deliverable?   # => true
+result.catch_all?     # => false
+result.disposable?    # => false
+result.mx_records     # => ["gmail-smtp-in.l.google.com.", ...]
+result.duration       # => 1.83
+```
+
+### The four verdicts
+
+`#reachable` is the headline answer. Everything else is supporting detail.
+
+| Verdict    | Meaning | Predicate |
+| ---------- | ------- | --------- |
+| `:safe`    | The address exists and has no quality problems. | `safe?` |
+| `:risky`   | It exists, but it is catch-all, disposable, role-based, or the inbox is full. Mail may bounce. | `risky?` |
+| `:invalid` | Malformed, or the domain has no MX records, or the server rejected the recipient. Do not send. | `invalid?` |
+| `:unknown` | No usable answer. Says nothing about the address — see below. | `unknown?` |
+
+`#valid?` is an alias for `safe?`, deliberately strict. Whether `:risky` or
+`:unknown` is good enough is a policy decision, so make it explicitly:
+
+```ruby
+case result.reachable
+when :safe          then accept
+when :risky         then accept_with_warning unless result.disposable?
+when :invalid       then reject
+when :unknown       then queue_for_retry
+end
+```
+
+### Please read this: why you will see `:unknown`
+
+SMTP verification is a best-effort signal, not a guarantee, and `:unknown` is
+common for reasons that have nothing to do with the address:
+
+- **Outbound port 25 is blocked** on most residential networks and by most
+  cloud providers (AWS, GCP, Azure, Heroku, Fly). Without a SOCKS5 proxy or a
+  host that permits port 25, essentially every check returns `:unknown`.
+- **Mail exchangers are defensive.** They greylist, rate-limit, and blacklist.
+  A server that has never seen your IP may simply refuse to answer.
+- **Reputation matters.** Set `from_email` and `hello_name` to a domain you
+  actually control, with matching forward and reverse DNS. This makes a large
+  difference in practice.
+
+`#error` and `#error_description` say what went wrong:
+
+```ruby
+result.error              # => {type: "AsyncSmtpError", message: "transient: ..."}
+result.error_description  # => "IpBlacklisted"  (or "NeedsRDNS", ...)
+result.timed_out?         # => false
+```
+
+Also note that **catch-all domains cannot be verified per-address** — the
+server accepts every recipient, so those come back `:risky`. That is a property
+of the domain, not a shortcoming of the check.
+
+### Syntax only, no network
+
+Instant and side-effect free, so it is safe inline in a request cycle or a
+model validation:
+
+```ruby
+ValidEmailChecker.syntax_valid?("someone@example.com")   # => true
+
+syntax = ValidEmailChecker.syntax("someone@gmail.co")
+syntax.valid?            # => true
+syntax.suggestion        # => "someone@gmail.com"
+syntax.domain            # => "gmail.co"
+
+# Provider quirks removed — Gmail ignores dots and "+" tags:
+ValidEmailChecker.syntax("Some.One+news@gmail.com").normalized_email
+# => "someone@gmail.com"
+```
+
+A suggestion is only offered for an address that parses. Notorious typo domains
+such as `gmial.com` are on upstream's blocklist, so they come back plainly
+invalid with no domain and no suggestion.
+
+### Checking many addresses
+
+`check_many` runs the whole batch on one async runtime with a single GVL
+release, so `concurrency` verifications are in flight at once. Results come
+back in the order given.
+
+```ruby
+results = ValidEmailChecker.check_many(addresses, concurrency: 20)
+results.select(&:safe?).map(&:email)
+```
+
+This is the main reason to embed the crate rather than shell out to it: on this
+machine, four addresses at `concurrency: 4` took 4.0s versus 13.5s serially.
+
+Be careful raising `concurrency` when the addresses share a domain — you are
+opening that many simultaneous SMTP connections to one mail exchanger, which is
+a good way to get rate-limited or blacklisted.
+
+### Threading
+
+`check` and `check_many` release the GVL for the duration of the verification,
+so other Ruby threads keep running. Calling them from a thread pool is fine.
+The async runtime is created per call rather than cached, so there is no shared
+runtime to contend on.
+
+### Forking servers
+
+**Do not verify an address in a process that will later fork.** Loading the gem
+is fine; calling `check` or `check_many` is not.
+
+On macOS, Apple's `Network.framework` is not fork-safe. The first verification
+in a process initializes XPC and dispatch state for the system resolver, and a
+child that inherits that state deadlocks inside the framework on its own first
+connection — permanently, ignoring the configured timeout. Rather than let that
+look like a slow mail server, the gem tracks which process verified and raises
+`ValidEmailChecker::ForkAfterVerificationError` instead:
+
+```ruby
+ValidEmailChecker.check("someone@example.com")   # in the parent
+fork { ValidEmailChecker.check("other@example.com") }
+# => ValidEmailChecker::ForkAfterVerificationError
+```
+
+In practice this costs nothing, because the normal arrangement is already the
+safe one: let Puma or Unicorn preload your app, and verify only inside workers.
+Only a verification during the preload phase itself — a boot-time health check,
+say — will trip it. Syntax checks are unaffected, since they touch no sockets.
+
+The guard applies on macOS only, and can be lifted with
+`VALID_EMAIL_CHECKER_ALLOW_FORK=1` if you have reason to believe your platform
+is unaffected.
+
+## Configuration
+
+Set process-wide defaults:
+
+```ruby
+ValidEmailChecker.configure do |config|
+  config.from_email = "verify@example.com"   # MAIL FROM: — use a domain you own
+  config.hello_name = "example.com"          # EHLO — ditto, with matching rDNS
+  config.timeout    = 15                     # seconds, per address
+  config.smtp_port  = 587                    # if 25 is blocked
+end
+```
+
+Any setting can be overridden per call:
+
+```ruby
+ValidEmailChecker.check("someone@example.com", timeout: 5, retries: 2)
+```
+
+| Option | Default | Notes |
+| ------ | ------- | ----- |
+| `from_email` | `"reacher@gmail.com"` | Address for `MAIL FROM:`. Some servers check it. |
+| `hello_name` | `"gmail.com"` | Name for `EHLO`. Should be a domain you own. |
+| `smtp_port` | `25` | Try `587` or `2525` where 25 is blocked. |
+| `smtp_timeout` | `nil` | Seconds, per SMTP connection. |
+| `retries` | `1` | Total SMTP connections. `2` can get past greylisting. |
+| `timeout` | `30` | Seconds, wall-clock ceiling per address. `nil` to disable. |
+| `concurrency` | `10` | For `check_many` only. |
+| `proxy` | `nil` | `{host:, port:, username:, password:, timeout:}`, SOCKS5. |
+| `check_gravatar` | `false` | Adds an HTTP round trip. |
+| `haveibeenpwned_api_key` | `nil` | Enables the breach lookup. |
+| `yahoo` | `:api` | `:api`, `:smtp` or `:headless`. |
+| `hotmail_b2c` | `:smtp` | `:smtp` or `:headless`. |
+| `webdriver_addr` | `nil` | Required by `:headless`, e.g. `"http://localhost:9515"`. |
+| `backend_name` | `"valid_email_checker"` | Recorded in the result's debug section. |
+
+Bad options raise `ValidEmailChecker::ConfigurationError` before any network
+access happens.
+
+### Timeouts
+
+`timeout` is a wall-clock ceiling per address, defaulting to 30 seconds.
+Upstream has no such ceiling; this gem adds one so a single unresponsive mail
+server cannot block a caller indefinitely.
+
+A timed-out result comes back as `:unknown` with `timed_out?` true. It carries
+no partial detail — no MX records, no SMTP findings — because the verification
+is abandoned wholesale. If you want the MX result independently of the SMTP
+stage, that is a good reason to keep the ceiling generous.
+
+### Proxies
+
+Where port 25 is blocked, route through a SOCKS5 proxy:
+
+```ruby
+ValidEmailChecker.configure do |config|
+  config.proxy = { host: "proxy.example.com", port: 1080, username: "u", password: "p" }
+end
+```
+
+### Yahoo and Hotmail
+
+Two providers need special handling, and this gem's defaults differ from
+upstream's so that a fresh install works with nothing else running: Yahoo
+defaults to `:api` and consumer Hotmail/Outlook to `:smtp`. Upstream defaults
+both to `:headless`, which drives the provider's password-recovery page through
+a WebDriver. To opt into that, run `chromedriver` and point the gem at it:
+
+```ruby
+ValidEmailChecker.configure do |config|
+  config.yahoo          = :headless
+  config.hotmail_b2c    = :headless
+  config.webdriver_addr = "http://localhost:9515"
+end
+```
+
+## Command line
+
+```console
+$ valid_email_checker someone@gmail.com info@example.com
+safe     someone@gmail.com
+risky    info@example.com  (role account, catch-all)
+
+$ valid_email_checker --syntax-only someone@gmail.co
+valid    someone@gmail.co  (suggestion: someone@gmail.com)
+
+$ valid_email_checker --json --timeout 10 someone@gmail.com
+[{"input": "someone@gmail.com", "is_reachable": "safe", ...}]
+
+$ valid_email_checker --stdin --concurrency 20 < addresses.txt
+```
+
+Exit status is `0` when every address came back `:safe`, `1` when any was not,
+and `2` for a usage or configuration error. Run `--help` for all options.
+
+## The raw upstream result
+
+Every field the crate produces is available verbatim, under `:syntax`, `:mx`,
+`:smtp`, `:misc` and `:debug`:
+
+```ruby
+result.to_h[:smtp]   # => {can_connect_smtp: true, is_catch_all: false, ...}
+result.to_h[:debug]  # => {backend_name: "...", start_time: "...", ...}
+```
+
+Note that when a stage fails, the crate replaces that section with
+`{error: {...}}` instead of the fields it normally holds. The predicate methods
+on `Result` already tolerate this; code reading `to_h` directly should too.
+
+## Development
+
+```console
+$ bundle install
+$ bundle exec rake compile     # build the extension
+$ bundle exec rake test        # offline tests only
+$ bundle exec rake test_network  # also the tests needing DNS and port 25
+```
+
+Tests that reach the network are skipped by default, since most networks block
+outbound port 25 — including, most likely, yours.
+
+To update the embedded crate to upstream's latest commit (or a tag, e.g.
+`rake "vendor:update[v0.11.8]"`):
+
+```console
+$ bundle exec rake vendor:update
+```
+
+This replaces `ext/valid_email_checker/vendor/check-if-email-exists/` with a
+fresh copy of upstream's `core` package, re-applies the two `Cargo.toml` changes
+noted in [NOTICE.md](NOTICE.md), updates
+`ext/valid_email_checker/vendor/UPSTREAM.json` and refreshes `Cargo.lock`.
+
+## License
+
+AGPL-3.0-only. See [LICENSE.txt](LICENSE.txt) and, importantly,
+[NOTICE.md](NOTICE.md).

data/exe/valid_email_checker

@@ -0,0 +1,148 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "optparse"
+require "valid_email_checker"
+
+options = {}
+format = :text
+syntax_only = false
+
+parser = OptionParser.new do |opts|
+  opts.banner = <<~BANNER
+    Check whether email addresses exist, without sending mail.
+
+    Usage: valid_email_checker [options] EMAIL [EMAIL...]
+           valid_email_checker [options] --stdin < addresses.txt
+
+  BANNER
+
+  opts.on("--stdin", "Read addresses from stdin, one per line") do
+    options[:stdin] = true
+  end
+
+  opts.on("--syntax-only", "Only parse the addresses; no network access") do
+    syntax_only = true
+  end
+
+  opts.on("--json", "Emit the full upstream result as JSON") { format = :json }
+
+  opts.on("--from-email EMAIL", "Address for the MAIL FROM: command") do |value|
+    options[:from_email] = value
+  end
+
+  opts.on("--hello-name NAME", "Name for the EHLO command; use a domain you own") do |value|
+    options[:hello_name] = value
+  end
+
+  opts.on("--smtp-port PORT", Integer, "SMTP port (default 25)") do |value|
+    options[:smtp_port] = value
+  end
+
+  opts.on("--timeout SECONDS", Float, "Per-address wall-clock budget (default 30)") do |value|
+    options[:timeout] = value
+  end
+
+  opts.on("--retries N", Integer, "Total SMTP connections per address (default 1)") do |value|
+    options[:retries] = value
+  end
+
+  opts.on("--concurrency N", Integer, "Addresses to verify at once (default 10)") do |value|
+    options[:concurrency] = value
+  end
+
+  opts.on("--proxy HOST:PORT", "SOCKS5 proxy, needed where port 25 is blocked") do |value|
+    host, port = value.rpartition(":").values_at(0, 2)
+    if host.empty? || port.to_i.zero?
+      warn "error: --proxy expects HOST:PORT, got #{value.inspect}"
+      exit 2
+    end
+    options[:proxy] = { host: host, port: port.to_i }
+  end
+
+  opts.on("--gravatar", "Also look up a Gravatar for each address") do
+    options[:check_gravatar] = true
+  end
+
+  opts.on("-v", "--version", "Print versions and exit") do
+    upstream = ValidEmailChecker.upstream
+    puts "valid_email_checker #{ValidEmailChecker::VERSION}"
+    puts "#{upstream[:name]} #{upstream[:version]} (#{upstream[:commit][0, 12]})"
+    exit 0
+  end
+
+  opts.on("-h", "--help", "Print this message and exit") do
+    puts opts
+    exit 0
+  end
+end
+
+parser.parse!
+
+emails = options.delete(:stdin) ? $stdin.read.split("\n") : ARGV
+emails = emails.map(&:strip).reject(&:empty?)
+
+if emails.empty?
+  warn parser.help
+  exit 2
+end
+
+# Exit status is meant for scripting: 0 only when every address came back
+# clearly good, 1 when any was invalid or could not be determined.
+def report_text(result)
+  marker = case result.reachable
+           when :safe then "safe   "
+           when :risky then "risky  "
+           when :invalid then "invalid"
+           else "unknown"
+           end
+
+  notes = []
+  notes << "catch-all" if result.catch_all?
+  notes << "disposable" if result.disposable?
+  notes << "role account" if result.role_account?
+  notes << "full inbox" if result.full_inbox?
+  notes << "disabled" if result.disabled?
+  notes << "timed out" if result.timed_out?
+  notes << "suggestion: #{result.syntax.suggestion}" if result.syntax.suggestion?
+  if result.error? && !result.timed_out?
+    notes << "#{result.error[:type]}: #{result.error[:message]}".lines.first.to_s.strip
+  end
+
+  line = "#{marker}  #{result.email}"
+  line += "  (#{notes.join(", ")})" unless notes.empty?
+  puts line
+end
+
+begin
+  if syntax_only
+    results = emails.map { |email| [email, ValidEmailChecker.syntax(email)] }
+
+    if format == :json
+      puts JSON.pretty_generate(results.map { |email, s| { input: email }.merge(s.to_h) })
+    else
+      results.each do |email, s|
+        suffix = s.suggestion? ? "  (suggestion: #{s.suggestion})" : ""
+        puts "#{s.valid? ? "valid  " : "invalid"}  #{email}#{suffix}"
+      end
+    end
+
+    exit results.all? { |_, s| s.valid? } ? 0 : 1
+  end
+
+  results = ValidEmailChecker.check_many(emails, **options)
+
+  if format == :json
+    puts JSON.pretty_generate(results.map(&:to_h))
+  else
+    results.each { |result| report_text(result) }
+  end
+
+  exit results.all?(&:safe?) ? 0 : 1
+rescue ValidEmailChecker::ConfigurationError, ArgumentError => e
+  warn "error: #{e.message}"
+  exit 2
+rescue Interrupt
+  warn "interrupted"
+  exit 130
+end

data/ext/valid_email_checker/Cargo.toml

@@ -0,0 +1,27 @@
+[package]
+name = "valid_email_checker"
+version = "0.1.0"
+edition = "2021"
+rust-version = "1.75"
+publish = false
+license = "AGPL-3.0-only"
+description = "Ruby native extension wrapping the embedded check-if-email-exists crate"
+
+[lib]
+name = "valid_email_checker"
+crate-type = ["cdylib"]
+
+[dependencies]
+magnus = "0.8"
+rb-sys = "0.9"
+serde = { version = "1.0", features = ["derive"] }
+serde_json = "1.0"
+futures = "0.3"
+tokio = { version = "1", features = ["rt-multi-thread", "time"] }
+
+# The upstream project, embedded in this gem under vendor/. Pinned by the
+# commit recorded in vendor/UPSTREAM.json; refresh with `rake vendor:update`.
+check-if-email-exists = { path = "vendor/check-if-email-exists" }
+
+# Note: build profiles live in the workspace root Cargo.toml, not here --
+# cargo ignores `[profile.*]` in a workspace member.

data/ext/valid_email_checker/extconf.rb

@@ -0,0 +1,26 @@
+# frozen_string_literal: true
+
+require "mkmf"
+require "rb_sys/mkmf"
+
+# Compiling the embedded crate pulls in tokio, rustls and reqwest, so a cold
+# build takes a few minutes. Fail with a clear message rather than a cryptic
+# one if the toolchain is missing.
+unless find_executable("cargo")
+  abort <<~MSG
+
+    valid_email_checker needs a Rust toolchain to build.
+
+    It embeds the `check-if-email-exists` crate and compiles it during
+    installation. Install Rust with:
+
+        curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
+
+    or see https://rustup.rs, then reinstall this gem.
+
+  MSG
+end
+
+create_rust_makefile("valid_email_checker/valid_email_checker") do |r|
+  r.profile = ENV.fetch("VALID_EMAIL_CHECKER_PROFILE", "release").to_sym
+end