gman 7.0.6 → 8.1.0

data/docs/README.md

@@ -4,7 +4,7 @@
 
 A ruby gem to check if the owner of a given email address or website is working for THE MAN (a.k.a verifies government domains). It will also provide you with metadata about the domain, such as the country, state, city, or agency, where applicable. It does this by leveraging the power of [Naughty or Nice](https://github.com/benbalter/naughty_or_nice), the [Public Suffix List](http://publicsuffix.org/), and the associated [Ruby Gem](https://github.com/weppos/publicsuffix-ruby).
 
-You could theoretically [use regex](https://gist.github.com/benbalter/6147066), but either you'll get a bunch of false positives, or your regex will be insanely complicated. `gov.uk`, may be valid, for example, but `gov.fr` is not (it's `gouv.fr`, for what it's worth). The solution? Use Public Suffix to verify that it's a valid public domain, then maintain [a crowd-sourced sub-list of known global government and military domains](https://github.com/benbalter/gman/blob/master/config/domains.txt). It should cover all US and international, government and military domains for both email and website verification.
+You could theoretically [use regex](https://gist.github.com/benbalter/6147066), but either you'll get a bunch of false positives, or your regex will be insanely complicated. `gov.uk`, may be valid, for example, but `gov.fr` is not (it's `gouv.fr`, for what it's worth). The solution? Use Public Suffix to verify that it's a valid public domain, then maintain [a crowd-sourced sub-list of known global government and military domains](https://github.com/benbalter/gman/blob/main/config/domains.txt). It should cover all US and international, government and military domains for both email and website verification.
 
 See a domains that's missing or one that shouldn't be there? [We'd love you to contribute](CONTRIBUTING.md).
 
@@ -18,6 +18,8 @@ Or add this to your `Gemfile` before doing a `bundle install`:
 
 `gem 'gman'`
 
+Gman requires Ruby 3.2 or newer.
+
 ## Usage
 
 ### In general

data/lib/gman/domain_list.rb

@@ -2,9 +2,9 @@
 
 class Gman
   class DomainList
-    COMMENT_REGEX = %r{//[/\s]*(.*)$}i.freeze
+    COMMENT_REGEX = %r{//[/\s]*(.*)$}i
 
-    attr_writer :data, :path, :contents
+    attr_writer :path, :contents
 
     class << self
       # The current, government domain list
@@ -41,6 +41,12 @@ class Gman
                     end
     end
 
+    # Replaces the list data, a hash of group => domains
+    def data=(data)
+      @data = data
+      clear_cache
+    end
+
     # Returns the parsed contents of the domain list as a hash
     # in the form for group => domains
     def data
@@ -73,7 +79,7 @@ class Gman
 
     # Return an array of strings representing all domains on the list
     def domains
-      data.values.flatten.compact.sort.uniq
+      @domains ||= data.values.flatten.compact.sort.uniq.freeze
     end
 
     # Return the total number of domains in the list
@@ -84,11 +90,12 @@ class Gman
     # Alphabetize groups and domains within each group
     # We need to ensure exceptions appear after their coresponding rules
     def alphabetize
-      @data = data.sort_by { |k, _v| k.downcase }.to_h
-      @data.map do |_group, domains|
+      self.data = data.sort_by { |k, _v| k.downcase }.to_h
+      @data.each_value do |domains|
         domains.sort! { |a, b| sort_with_exceptions(a, b) }
         domains.uniq!
       end
+      clear_cache
     end
 
     # Write the domain list to disk
@@ -112,14 +119,60 @@ class Gman
     end
     alias to_public_suffix to_s
 
-    # Given a domain, find any domain on the list that includes that domain
-    # E.g., `fcc.gov` would be the parent of `data.fcc.gov`
+    # Returns the group a list entry belongs to, e.g., "US Federal" for "fcc.gov"
+    # The entry must match the list verbatim. If an entry appears in more than
+    # one group, the first group wins.
+    def group_for(entry)
+      group_index[entry]
+    end
+
+    # Given a domain, find the list entry that covers it, using public suffix
+    # rules. E.g., `fcc.gov` would be the parent of `data.fcc.gov`, and
+    # `*.foo.gov` the parent of `bar.foo.gov`.
+    #
+    # If more than one entry matches, the longest wins. A domain under an
+    # exception rule (e.g., `!mail.gov.ua`) has no parent. A domain that is
+    # itself on the list returns the entry that covers its parent, if any.
+    #
+    # Returns the list entry as written (e.g., `*.foo.gov`), or nil
     def parent_domain(domain)
-      domains.find { |c| domain =~ /\.#{Regexp.escape(c)}$/ }
+      rule = parent_list.find(domain, default: nil)
+      if rule && !rule.is_a?(PublicSuffix::Rule::Exception) && rule.value == domain
+        ancestor = domain.split('.', 2)[1]
+        rule = ancestor && parent_list.find(ancestor, default: nil)
+      end
+      return if rule.nil? || rule.is_a?(PublicSuffix::Rule::Exception)
+
+      rule.rule
+    end
+
+    # Returns the exception rule (e.g., `!mail.gov.ua`) that excludes the given
+    # domain from the list, or nil if none does
+    def exception_for(domain)
+      rule = parent_list.find(domain, default: nil)
+      rule.rule if rule.is_a?(PublicSuffix::Rule::Exception)
     end
 
     private
 
+    # Clear values derived from data. Call after mutating data in place.
+    def clear_cache
+      @domains = @parent_list = @group_index = nil
+    end
+
+    # PublicSuffix::List of the current data, for parent lookups. Unlike
+    # #public_suffix_list, it is rebuilt when the data changes.
+    def parent_list
+      @parent_list ||= PublicSuffix::List.parse(domains.join("\n"))
+    end
+
+    # Hash of list entry => group, for constant-time group lookups
+    def group_index
+      @group_index ||= data.each_with_object({}) do |(group, domains), index|
+        domains.each { |domain| index[domain] ||= group }
+      end
+    end
+
     # Parse a public-suffix formatted string into a hash of groups => [domains]
     def string_to_hash(string)
       return unless string

data/lib/gman/identifier.rb

@@ -45,7 +45,7 @@ class Gman
   def state
     if matches
       matches[4].upcase
-    elsif dotgov_listing['State']
+    elsif dotgov_listing && dotgov_listing['State']
       dotgov_listing['State']
     elsif list_category
       matches = list_category.match(/usagov([A-Z]{2})/)
@@ -60,7 +60,7 @@ class Gman
   def federal?
     return false unless dotgov_listing
 
-    domain_type =~ /^Federal/i
+    domain_type.to_s.match?(/^Federal/i)
   end
 
   def city?
@@ -87,7 +87,7 @@ class Gman
     if matches
       matches[1] == 'state'
     elsif dotgov_listing
-      domain_type == 'State/Local Govt' || domain_type == 'State'
+      ['State/Local Govt', 'State'].include?(domain_type)
     else
       false
     end
@@ -110,12 +110,8 @@ class Gman
   def list_category
     return @list_category if defined?(@list_category)
 
-    match = Gman.list.public_suffix_list.find(domain.to_s)
-    return @list_category = nil unless match
-
-    regex = %r{// ([^\n]+)\n?[^/]*\n#{Regexp.escape(match.value)}\n}im
-    matches = Gman.list.contents.match(regex)
-    @list_category = matches ? matches[1] : nil
+    match = Gman.list.public_suffix_list.find(domain.to_s, default: nil)
+    @list_category = match && Gman.list.group_for(match.value)
   end
 
   def matches
@@ -128,9 +124,7 @@ class Gman
     return @dotgov_listing if defined? @dotgov_listing
     return unless dotgov?
 
-    @dotgov_listing = Gman.dotgov_list.find do |listing|
-      listing['Domain Name'].casecmp("#{domain.sld}.gov").zero?
-    end
+    @dotgov_listing = Gman.dotgov_index["#{domain.sld}.gov".downcase]
   end
 
   class << self
@@ -138,6 +132,11 @@ class Gman
       @dotgov_list ||= CSV.read(dotgov_list_path, headers: true)
     end
 
+    # Hash of lowercase domain name => dotgov listing, for constant-time lookup
+    def dotgov_index
+      @dotgov_index ||= dotgov_list.to_h { |listing| [listing['Domain Name'].downcase, listing] }
+    end
+
     private
 
     def dotgov_list_path

data/lib/gman/importer.rb

@@ -6,9 +6,10 @@ require 'yaml'
 require 'open-uri'
 require 'resolv'
 require 'logger'
+require 'addressable/uri'
 require 'swot'
 require_relative '../gman'
-require_relative './domain_list'
+require_relative 'domain_list'
 
 class Gman
   class Importer
@@ -71,7 +72,7 @@ class Gman
 
     def normalize_domain(domain)
       domain = Gman.new(domain).to_s
-      domain.to_s.downcase.strip.gsub(/^www./, '').gsub(%r{/$}, '')
+      domain.to_s.downcase.strip.delete_prefix('www.').delete_suffix('/')
     end
 
     def valid_domain?(domain, options = {})
@@ -152,34 +153,33 @@ class Gman
 
       if current.domains.include?(domain)
         reject(domain, 'duplicate')
-      else
-        parent = current.parent_domain(domain)
+      elsif (parent = current.parent_domain(domain))
         reject(domain, "subdomain of #{parent}")
+      else
+        reject(domain, "excluded by #{current.exception_for(domain)}")
       end
     end
 
+    # A domain is a dupe if it's on the list, covered by a list entry, or
+    # excluded by an exception rule
     def dupe?(domain)
-      current.domains.include?(domain) || current.parent_domain(domain)
+      current.domains.include?(domain) || current.parent_domain(domain) || current.exception_for(domain)
     end
 
     def normalize_domains!
-      domain_list.to_h.each_value do |domains|
-        domains.map! { |domain| normalize_domain(domain) }
-        domains.uniq!
+      domain_list.data = domain_list.data.transform_values do |domains|
+        domains.map { |domain| normalize_domain(domain) }.uniq
       end
     end
 
     def ensure_validity!(options = {})
-      domain_list.data.each_value do |domains|
-        domains.select! { |domain| valid_domain?(domain, options) }
+      domain_list.data = domain_list.data.transform_values do |domains|
+        domains.select { |domain| valid_domain?(domain, options) }
       end
     end
 
     def add_to_current
-      domain_list.data.each do |group, domains|
-        current.data[group] ||= []
-        current.data[group].concat domains
-      end
+      current.data = current.data.merge(domain_list.data) { |_group, existing, added| existing + added }
       current.write
     end
 

data/lib/gman/locality.rb

@@ -33,7 +33,7 @@ class Gman
       |
         (#{Regexp.union(LOCALITY_DOMAINS)})\.[a-z-]+
       )\.(#{Regexp.union(STATES)})\.us
-    /x.freeze
+    /x
 
     def self.valid?(domain)
       !domain.to_s.match(Locality::REGEX).nil?

data/lib/gman/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 class Gman
-  VERSION = '7.0.6'
+  VERSION = '8.1.0'
 end

data/lib/gman.rb

@@ -45,9 +45,7 @@ class Gman
   def valid?
     return @valid if defined?(@valid)
 
-    @valid = false unless valid_domain?
-    @valid = false if academic?
-    @valid ||= locality? || public_suffix_valid?
+    @valid = valid_domain? && (locality? || public_suffix_valid?)
   end
 
   def locality?
@@ -57,14 +55,18 @@ class Gman
   private
 
   def valid_domain?
-    @valid_domain ||= !domain.nil? && !academic?
+    !domain.nil? && !academic?
   end
 
   def academic?
-    @academic ||= domain && Gman.academic_list.valid?(to_s)
+    return @academic if defined?(@academic)
+
+    @academic = !domain.nil? && Gman.academic_list.valid?(to_s)
   end
 
   def public_suffix_valid?
-    @public_suffix_valid ||= Gman.list.valid?(to_s)
+    return @public_suffix_valid if defined?(@public_suffix_valid)
+
+    @public_suffix_valid = Gman.list.valid?(to_s)
   end
 end

metadata

@@ -1,29 +1,28 @@
 --- !ruby/object:Gem::Specification
 name: gman
 version: !ruby/object:Gem::Version
-  version: 7.0.6
+  version: 8.1.0
 platform: ruby
 authors:
 - Ben Balter
-autorequire:
 bindir: bin
 cert_chain: []
-date: 2022-12-12 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: colored
+  name: csv
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.2'
+        version: '3.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.2'
+        version: '3.0'
 - !ruby/object:Gem::Dependency
   name: iso_country_codes
   requirement: !ruby/object:Gem::Requirement
@@ -66,160 +65,6 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '3.0'
-- !ruby/object:Gem::Dependency
-  name: addressable
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '2.3'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '2.3'
-- !ruby/object:Gem::Dependency
-  name: mechanize
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '2.7'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '2.7'
-- !ruby/object:Gem::Dependency
-  name: parallel
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.6'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.6'
-- !ruby/object:Gem::Dependency
-  name: pry
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '0.10'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '0.10'
-- !ruby/object:Gem::Dependency
-  name: rspec
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '3.5'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '3.5'
-- !ruby/object:Gem::Dependency
-  name: rubocop
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.0'
-- !ruby/object:Gem::Dependency
-  name: rubocop-performance
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.5'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.5'
-- !ruby/object:Gem::Dependency
-  name: rubocop-rspec
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '2.0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '2.0'
-- !ruby/object:Gem::Dependency
-  name: ruby-prof
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.4'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.4'
-- !ruby/object:Gem::Dependency
-  name: ruby-progressbar
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.10'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.10'
-- !ruby/object:Gem::Dependency
-  name: swot
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '1.0'
 description: |2
       A ruby gem to check if the owner of a given email address is working for
       THE MAN.
@@ -230,38 +75,13 @@ executables:
 extensions: []
 extra_rdoc_files: []
 files:
-- ".github/CODEOWNERS"
-- ".github/ISSUE_TEMPLATE/bug_report.md"
-- ".github/ISSUE_TEMPLATE/feature_request.md"
-- ".github/config.yml"
-- ".github/dependabot.yml"
-- ".github/funding.yml"
-- ".github/no-response.yml"
-- ".github/release-drafter.yml"
-- ".github/settings.yml"
-- ".github/stale.yml"
-- ".github/workflows/ci.yml"
-- ".github/workflows/clean.yml"
-- ".github/workflows/codeql-analysis.yml"
-- ".github/workflows/validate.yml"
-- ".github/workflows/vendor.yml"
-- ".gitignore"
-- ".rspec"
-- ".rubocop.yml"
-- ".rubocop_todo.yml"
-- Gemfile
 - LICENSE
 - bin/gman
 - bin/gman_filter
 - config/domains.txt
 - config/vendor/academic.txt
 - config/vendor/dotgovs.csv
-- docs/CODE_OF_CONDUCT.md
-- docs/CONTRIBUTING.md
 - docs/README.md
-- docs/SECURITY.md
-- docs/_config.yml
-- gman.gemspec
 - lib/gman.rb
 - lib/gman/country_codes.rb
 - lib/gman/domain_list.rb
@@ -269,40 +89,11 @@ files:
 - lib/gman/importer.rb
 - lib/gman/locality.rb
 - lib/gman/version.rb
-- script/add
-- script/alphabetize
-- script/bootstrap
-- script/cibuild
-- script/console
-- script/dedupe
-- script/profile
-- script/prune
-- script/reconcile-us
-- script/release
-- script/validate-domains
-- script/vendor
-- script/vendor-federal-de
-- script/vendor-gov-list
-- script/vendor-public-suffix
-- script/vendor-swot
-- script/vendor-us
-- spec/fixtures/domains.txt
-- spec/fixtures/obama.txt
-- spec/gman/bin_spec.rb
-- spec/gman/country_code_spec.rb
-- spec/gman/domain_list_spec.rb
-- spec/gman/domains_spec.rb
-- spec/gman/identifier_spec.rb
-- spec/gman/importer_spec.rb
-- spec/gman/locality_spec.rb
-- spec/gman_spec.rb
-- spec/spec_helper.rb
 homepage: https://github.com/benbalter/gman
 licenses:
 - MIT
 metadata:
   rubygems_mfa_required: 'true'
-post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -310,18 +101,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '2.5'
-  - - "<"
-    - !ruby/object:Gem::Version
-      version: '4.0'
+      version: '3.2'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.2.33
-signing_key:
+rubygems_version: 3.6.9
 specification_version: 4
-summary: Check if a given domain or email address belong to a governemnt entity
+summary: Check if a given domain or email address belong to a government entity
 test_files: []

data/.github/CODEOWNERS

@@ -1,3 +0,0 @@
-# Require @benbalter's :+1: for changes to the .github repo-config files
-# mainly due to https://github.com/probot/settings privilege escalation
-.github/* @benbalter

data/.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,28 +0,0 @@
----
-name: Bug report
-about: Create a report to help us improve
-
----
-
-### Describe the bug
-
-A clear and concise description of what the bug is.
-
-### Steps to reproduce the behavior
-
-1. Go to '...'
-2. Click on '....'
-3. Scroll down to '....'
-4. See error
-
-### Expected behavior
-
-A clear and concise description of what you expected to happen.
-
-### Screenshots
-
-If applicable, add screenshots to help explain your problem.
-
-### Additional context
-
-Add any other context about the problem here.

data/.github/ISSUE_TEMPLATE/feature_request.md

@@ -1,21 +0,0 @@
----
-name: Feature request
-about: Suggest an idea for this project
-
----
-
-### Is your feature request related to a problem? Please describe the problem you're trying to solve.
-
-A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
-
-### Describe the solution you'd like
-
-A clear and concise description of what you want to happen.
-
-### Describe alternatives you've considered
-
-A clear and concise description of any alternative solutions or features you've considered.
-
-### Additional context
-
-Add any other context or screenshots about the feature request here.

data/.github/config.yml

@@ -1,23 +0,0 @@
-# Behaviorbot config. See https://github.com/behaviorbot/ for more information.
-# Note: Please Don't edit this file directly.
-# Edit https://github.com/benbalter/shared-community-files instead.
-
-# Configuration for update-docs - https://github.com/behaviorbot/update-docs
-updateDocsComment: "Thanks for the pull request! If you are making any changes to the user-facing functionality, please be sure to update the documentation in the `README` or `docs/` folder alongside your change. :heart:"
-
-# Configuration for request-info - https://github.com/behaviorbot/request-info
-requestInfoReplyComment: Thanks for this. Do you mind providing a bit more information about what problem you're trying to solve?
-requestInfoLabelToAdd: more-information-needed
-
-# Configuration for new-issue-welcome - https://github.com/behaviorbot/new-issue-welcome
-#newIssueWelcomeComment: >
-#  Welcome!
-
-# Configuration for new-pr-welcome - https://github.com/behaviorbot/new-pr-welcome
-newPRWelcomeComment: Welcome! Congrats on your first pull request to Gman. If you haven't already, please be sure to check out [the contributing guidelines](https://github.com/benbalter/gman/blob/master/docs/CONTRIBUTING.md).
-
-# Configuration for first-pr-merge - https://github.com/behaviorbot/first-pr-merge
-firstPRMergeComment: "Congrats on getting your first pull request to Gman merged! Without amazing humans like you submitting pull requests, we couldn’t run this project. You rock! :tada:<br /><br />If you're interested in tackling another bug or feature, take a look at [the open issues](https://github.com/benbalter/gman/issues), especially those [labeled `help wanted`](https://github.com/benbalter/gman/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22)."
-
-# Bug workaround
-contact_links: []