gman 6.0.1 → 7.0.4

data/docs/CODE_OF_CONDUCT.md

@@ -0,0 +1,46 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at ben@balter.com. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/

data/docs/CONTRIBUTING.md

@@ -0,0 +1,92 @@
+# Contributing to Gman
+
+Hi there! We're thrilled that you'd like to contribute to Gman. Your help is essential for keeping it great.
+
+Gman is an open source project supported by the efforts of an entire community and built one contribution at a time by users like you. We'd love for you to get involved. Whatever your level of skill or however much time you can give, your contribution is greatly appreciated. There are many ways to contribute, from writing tutorials or blog posts, improving the documentation, submitting bug reports and feature requests, helping other users by commenting on issues, or writing code which can be incorporated into Gman itself.
+
+Following these guidelines helps to communicate that you respect the time of the developers managing and developing this open source project. In return, they should reciprocate that respect in addressing your issue, assessing changes, and helping you finalize your pull requests.
+
+
+
+## How to report a bug
+
+Think you found a bug? Please check [the list of open issues](https://github.com/benbalter/gman/issues) to see if your bug has already been reported. If it hasn't please [submit a new issue](https://github.com/benbalter/gman/issues/new).
+
+Here are a few tips for writing *great* bug reports:
+
+* Describe the specific problem (e.g., "widget doesn't turn clockwise" versus "getting an error")
+* Include the steps to reproduce the bug, what you expected to happen, and what happened instead
+* Check that you are using the latest version of the project and its dependencies
+* Include what version of the project your using, as well as any relevant dependencies
+* Only include one bug per issue. If you have discovered two bugs, please file two issues
+* Include screenshots or screencasts whenever possible
+* Even if you don't know how to fix the bug, including a failing test may help others track it down
+
+**If you find a security vulnerability, do not open an issue. Please email ben@balter.com instead.**
+
+## How to suggest a feature or enhancement
+
+If you find yourself wishing for a feature that doesn't exist in Gman, you are probably not alone. There are bound to be others out there with similar needs. Many of the features that Gman has today have been added because our users saw the need.
+
+Feature requests are welcome. But take a moment to find out whether your idea fits with the scope and goals of the project. It's up to you to make a strong case to convince the project's developers of the merits of this feature. Please provide as much detail and context as possible, including describing the problem you're trying to solve.
+
+[Open an issue](https://github.com/benbalter/gman/issues/new) which describes the feature you would like to see, why you want it, how it should work, etc.
+
+## Domains
+
+Domains live in [`config/domains.txt`](../config/domains.txt) as a list of TLDs and SLD+TLDs.
+
+Right now, the only valid government top level domains (TLDs), represent the US government and are `.gov`, and `.mil`. Secondary domains (e.g., `gov.uk`, or `mil.au`) represent non-US government entities.
+
+To add or remove a domain from the list of known government domains, simply edit the `domains.txt` file.
+
+
+## Your first contribution
+
+We'd love for you to contribute to the project. Unsure where to begin contributing to Gman? You can start by looking through these "good first issue" and "help wanted" issues:
+
+* [Good first issues](https://github.com/benbalter/gman/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) - issues which should only require a few lines of code and a test or two
+* [Help wanted issues](https://github.com/benbalter/gman/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22) - issues which may be a bit more involved, but are specifically seeking community contributions
+
+*p.s. Feel free to ask for help; everyone is a beginner at first* :smiley_cat:
+
+## How to propose changes
+
+Here's a few general guidelines for proposing changes:
+
+* If you are changing any user-facing functionality, please be sure to update the documentation
+* If you are adding a new behavior or changing an existing behavior, please be sure to update the corresponding test(s)
+* Each pull request should implement **one** feature or bug fix. If you want to add or fix more than one thing, submit more than one pull request
+* Do not commit changes to files that are irrelevant to your feature or bug fix
+* Don't bump the version number in your pull request (it will be bumped prior to release)
+* Write [a good commit message](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html)
+
+At a high level, [the process for proposing changes](https://guides.github.com/introduction/flow/) is:
+
+1. [Fork](https://github.com/benbalter/gman/fork) and clone the project
+2. Configure and install the dependencies: `script/bootstrap`
+3. Make sure the tests pass on your machine: `script/cibuild`
+4. Create a descriptively named branch: `git checkout -b my-branch-name`
+5. Make your change, add tests and documentation, and make sure the tests still pass
+6. Push to your fork and [submit a pull request](https://github.com/benbalter/gman/compare) describing your change
+7. Pat your self on the back and wait for your pull request to be reviewed and merged
+
+**Interesting in submitting your first Pull Request?** It's easy! You can learn how from this *free* series [How to Contribute to an Open Source Project on GitHub](https://egghead.io/series/how-to-contribute-to-an-open-source-project-on-github)
+
+## Bootstrapping your local development environment
+
+`script/bootstrap`
+
+## Running tests
+
+`script/cibuild`
+
+## Code of conduct
+
+This project is governed by [the Contributor Covenant Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code.
+
+## Additional Resources
+
+* [Contributing to Open Source on GitHub](https://guides.github.com/activities/contributing-to-open-source/)
+* [Using Pull Requests](https://help.github.com/articles/using-pull-requests/)
+* [GitHub Help](https://help.github.com)

data/{README.md → docs/README.md}

@@ -1,6 +1,6 @@
-# Gman Gem
+# Gman
 
-[![Build Status](https://travis-ci.org/benbalter/gman.png)](https://travis-ci.org/benbalter/gman) [![Gem Version](https://badge.fury.io/rb/gman.png)](http://badge.fury.io/rb/gman)
+[![Build Status](https://travis-ci.org/benbalter/gman.png)](https://travis-ci.org/benbalter/gman) [![Gem Version](https://badge.fury.io/rb/gman.png)](http://badge.fury.io/rb/gman) [![PRs Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat-square)](http://makeapullrequest.com)
 
 A ruby gem to check if the owner of a given email address or website is working for THE MAN (a.k.a verifies government domains). It will also provide you with metadata about the domain, such as the country, state, city, or agency, where applicable. It does this by leveraging the power of [Naughty or Nice](https://github.com/benbalter/naughty_or_nice), the [Public Suffix List](http://publicsuffix.org/), and the associated [Ruby Gem](https://github.com/weppos/publicsuffix-ruby).
 
@@ -72,7 +72,7 @@ domain.country.name                #=> "United States"
 domain.country.alpha2              #=> "US"
 domain.country.alpha3              #=> "USA"
 domain.country.currency            #=> "USD"
-domain.conutry.calling_code        #=> "+1"
+domain.country.calling_code        #=> "+1"
 ```
 
 ### Check if a country is on the US Sanctions list

data/docs/SECURITY.md

@@ -0,0 +1,3 @@
+# Security Policy
+
+To report a security vulnerability, please email [ben@balter.com](mailto:ben@balter.com).

data/docs/_config.yml

@@ -0,0 +1,2 @@
+title: Gman
+description: A ruby gem to verify if the owner of a given email address, website, or domain is a government agency.

data/gman.gemspec

@@ -1,14 +1,16 @@
+# frozen_string_literal: true
+
 require File.expand_path './lib/gman/version', File.dirname(__FILE__)
 
 Gem::Specification.new do |s|
   s.name = 'gman'
-  s.summary = <<-EOF
+  s.summary = <<-SUMMARY
     Check if a given domain or email address belong to a governemnt entity
-  EOF
-  s.description = <<-EOF
+  SUMMARY
+  s.description = <<-DESC
     A ruby gem to check if the owner of a given email address is working for
     THE MAN.
-  EOF
+  DESC
   s.version = Gman::VERSION
   s.authors = ['Ben Balter']
   s.email = 'ben.balter@github.com'
@@ -20,24 +22,23 @@ Gem::Specification.new do |s|
   s.executables   = `git ls-files -- bin/*`.split("\n").map do |f|
     File.basename(f)
   end
-  s.require_paths = ['lib']
 
   s.require_paths = ['lib']
-  s.required_ruby_version = '~> 2.0'
+  s.required_ruby_version = '~> 2.5'
 
-  s.add_dependency('swot', '~> 1.0')
-  s.add_dependency('iso_country_codes', '~> 0.6')
-  s.add_dependency('naughty_or_nice', '~> 2.0')
   s.add_dependency('colored', '~> 1.2')
+  s.add_dependency('iso_country_codes', '~> 0.6')
+  s.add_dependency('naughty_or_nice', '= 2.1.1')
+  s.add_dependency('public_suffix', '>= 3.0')
 
-  s.add_development_dependency('rake', '~> 10.4')
-  s.add_development_dependency('shoulda', '~> 3.5')
-  s.add_development_dependency('rdoc', '~> 4.2')
-  s.add_development_dependency('bundler', '~> 1.10')
-  s.add_development_dependency('pry', '~> 0.10')
-  s.add_development_dependency('parallel', '~> 1.6')
-  s.add_development_dependency('mechanize', '~> 2.7')
   s.add_development_dependency('addressable', '~> 2.3')
+  s.add_development_dependency('mechanize', '~> 2.7')
+  s.add_development_dependency('parallel', '~> 1.6')
+  s.add_development_dependency('pry', '~> 0.10')
+  s.add_development_dependency('rspec', '~> 3.5')
+  s.add_development_dependency('rubocop', '~> 1.0')
+  s.add_development_dependency('rubocop-performance', '~> 1.5')
+  s.add_development_dependency('rubocop-rspec', '~> 2.0')
   s.add_development_dependency('ruby-prof', '~> 0.15')
-  s.add_development_dependency('rubocop', '~> 0.37')
+  s.add_development_dependency('swot', '~> 1.0')
 end

data/lib/gman.rb

@@ -1,38 +1,41 @@
+# frozen_string_literal: true
+
+$LOAD_PATH.unshift(File.dirname(__FILE__))
+
 require 'naughty_or_nice'
-require 'swot'
 require 'iso_country_codes'
 require 'csv'
 require_relative 'gman/version'
 require_relative 'gman/country_codes'
-require_relative 'gman/locality'
 require_relative 'gman/identifier'
 
 class Gman
   include NaughtyOrNice
 
+  autoload :DomainList, 'gman/domain_list'
+  autoload :Importer,   'gman/importer'
+  autoload :Locality,   'gman/locality'
+
   class << self
-    # returns an instance of our custom public suffix list
-    # list behaves like PublicSuffix::List
-    # but is limited to our whitelisted domains
     def list
-      @list ||= PublicSuffix::List.parse(list_contents)
+      @list ||= DomainList.new(path: list_path)
+    end
+
+    def academic_list
+      @academic_list ||= DomainList.new(path: academic_list_path)
     end
 
     def config_path
-      File.expand_path '../config', File.dirname(__FILE__)
+      @config_path ||= File.expand_path '../config', File.dirname(__FILE__)
     end
 
     # Returns the absolute path to the domain list
     def list_path
-      if ENV['GMAN_STUB_DOMAINS']
-        File.expand_path '../test/fixtures/domains.txt', File.dirname(__FILE__)
-      else
-        File.expand_path 'domains.txt', config_path
-      end
+      File.expand_path 'domains.txt', config_path
     end
 
-    def list_contents
-      @list_contents ||= File.new(list_path, 'r:utf-8').read
+    def academic_list_path
+      File.expand_path 'vendor/academic.txt', config_path
     end
   end
 
@@ -43,25 +46,26 @@ class Gman
     @valid ||= begin
       return false unless valid_domain?
       return false if academic?
+
       locality? || public_suffix_valid?
     end
   end
 
+  def locality?
+    Locality.valid?(domain)
+  end
+
   private
 
   def valid_domain?
-    domain && domain.valid? && !academic?
+    @valid_domain ||= !domain.nil? && !academic?
   end
 
   def academic?
-    domain && Swot.is_academic?(domain)
+    @academic ||= domain && Gman.academic_list.valid?(to_s)
   end
 
-  # domain is on the domain list and
-  # domain is not explicitly blacklisted and
-  # domain matches a standard public suffix list rule
   def public_suffix_valid?
-    rule = Gman.list.find(to_s)
-    !rule.nil? && rule.type != :exception && rule.allow?(".#{domain}")
+    @public_suffix_valid ||= Gman.list.valid?(to_s)
   end
 end

data/lib/gman/country_codes.rb

@@ -1,19 +1,21 @@
+# frozen_string_literal: true
+
 class Gman
   # Map last part of TLD to alpha2 country code
   ALPHA2_MAP = {
-    ac:     'sh',
-    uk:     'gb',
-    su:     'ru',
-    tp:     'tl',
-    yu:     'rs',
-    gov:    'us',
-    mil:    'us',
-    org:    'us',
-    com:    'us',
-    net:    'us',
-    edu:    'us',
+    ac: 'sh',
+    uk: 'gb',
+    su: 'ru',
+    tp: 'tl',
+    yu: 'rs',
+    gov: 'us',
+    mil: 'us',
+    org: 'us',
+    com: 'us',
+    net: 'us',
+    edu: 'us',
     travel: 'us',
-    info:   'us'
+    info: 'us'
   }.freeze
 
   # Returns the two character alpha county code represented by the domain
@@ -21,13 +23,10 @@ class Gman
   # e.g., United States = US, United Kingdom = GB
   def alpha2
     return unless domain
+
     @alpha2 ||= begin
       alpha2 = domain.tld.split('.').last
-      if ALPHA2_MAP[alpha2.to_sym]
-        ALPHA2_MAP[alpha2.to_sym]
-      else
-        alpha2
-      end
+      ALPHA2_MAP[alpha2.to_sym] || alpha2
     end
   end
 
@@ -38,6 +37,7 @@ class Gman
   # Gman.new("foo.gov").country.currency => "USD"
   def country
     return @country if defined? @country
+
     @country ||= begin
       IsoCountryCodes.find(alpha2) if alpha2
     rescue IsoCountryCodes::UnknownCodeError

data/lib/gman/domain_list.rb

@@ -1,39 +1,106 @@
+# frozen_string_literal: true
+
 class Gman
   class DomainList
-    attr_accessor :list
-    alias to_h list
+    COMMENT_REGEX = %r{//[/\s]*(.*)$}i.freeze
+
+    attr_writer :data, :path, :contents
+
+    class << self
+      # The current, government domain list
+      def current
+        DomainList.new(path: Gman.list_path)
+      end
+
+      def from_file(path)
+        DomainList.new(path: path)
+      end
+
+      def from_hash(hash)
+        DomainList.new(data: hash)
+      end
 
-    COMMENT_REGEX = %r{//[/\s]*(.*)$}i
+      def from_public_suffix(string)
+        DomainList.new(contents: string)
+      end
+      alias from_string from_public_suffix
+    end
+
+    def initialize(path: nil, contents: nil, data: nil)
+      @path     = path
+      @contents = contents
+      @data     = data.reject { |_, domains| domains.compact.empty? } if data
+    end
+
+    # Returns the raw content of the domain list as a string
+    def contents
+      @contents ||= if path
+                      File.new(path, 'r:utf-8').read
+                    else
+                      to_s
+                    end
+    end
+
+    # Returns the parsed contents of the domain list as a hash
+    # in the form for group => domains
+    def data
+      @data ||= string_to_hash(contents)
+    end
+    alias to_h data
 
-    def initialize(list)
-      @list = list.reject { |_group, domains| domains.compact.empty? }
+    # Returns the path to the domain list on disk
+    def path
+      @path ||= Gman.list_path
     end
 
+    # returns an instance of our custom public suffix list
+    # list behaves like PublicSuffix::List
+    # but is limited to our whitelisted domains
+    def public_suffix_list
+      @public_suffix_list ||= PublicSuffix::List.parse(contents)
+    end
+
+    # domain is on the domain list
+    def valid?(domain)
+      rule = public_suffix_list.find(domain, default: nil)
+      !(rule.nil? || rule.is_a?(PublicSuffix::Rule::Exception))
+    end
+
+    # Returns an array of strings representing the list groups
     def groups
-      list.keys
+      data.keys
     end
 
+    # Return an array of strings representing all domains on the list
     def domains
-      list.values.flatten.compact.sort.uniq
+      data.values.flatten.compact.sort.uniq
     end
 
+    # Return the total number of domains in the list
     def count
       domains.count
     end
 
+    # Alphabetize groups and domains within each group
+    # We need to ensure exceptions appear after their coresponding rules
     def alphabetize
-      @list = @list.sort_by { |k, _v| k.downcase }.to_h
-      @list.each { |_group, domains| domains.sort!.uniq! }
+      @data = data.sort_by { |k, _v| k.downcase }.to_h
+      @data.map do |_group, domains|
+        domains.sort! { |a, b| sort_with_exceptions(a, b) }
+        domains.uniq!
+      end
     end
 
+    # Write the domain list to disk
     def write
       alphabetize
-      File.write(Gman.list_path, to_public_suffix)
+      File.write(path, to_public_suffix)
     end
 
-    def to_public_suffix
-      current_group = output = ''
-      list.sort_by { |group, _domains| group.downcase }.each do |group, domains|
+    # The string representation of the domain list, in public suffix format
+    def to_s
+      current_group = output = +''
+      data.sort_by { |group, _| group.downcase }.each do |group, domains|
         if group != current_group
           output << "\n\n" unless current_group.empty? # first entry
           output << "// #{group}\n"
@@ -43,44 +110,59 @@ class Gman
       end
       output
     end
+    alias to_public_suffix to_s
 
-    def self.current
-      current = File.open(Gman.list_path).read
-      DomainList.from_public_suffix(current)
+    # Given a domain, find any domain on the list that includes that domain
+    # E.g., `fcc.gov` would be the parent of `data.fcc.gov`
+    def parent_domain(domain)
+      domains.find { |c| domain =~ /\.#{Regexp.escape(c)}$/ }
     end
 
-    def self.from_public_suffix(string)
-      string = string.gsub(/\r\n?/, "\n").split("\n")
-      hash = array_to_hash(string)
-      DomainList.new(hash)
+    private
+
+    # Parse a public-suffix formatted string into a hash of groups => [domains]
+    def string_to_hash(string)
+      return unless string
+
+      lines = string_to_array(string)
+      array_to_hash(lines)
     end
 
-    def parent_domain(domain)
-      domains.find { |c| domain =~ /\.#{Regexp.escape(c)}$/ }
+    def string_to_array(string)
+      string.gsub(/\r\n?/, "\n").split("\n")
     end
 
-    class << self
-      private
-
-      # Given an array of comments/domains in public suffix format
-      # Converts to a hash in the form of :group => [domain1, domain2...]
-      def array_to_hash(domains)
-        domain_hash = {}
-        group = ''
-        domains.each do |line|
-          if line =~ COMMENT_REGEX
-            group = COMMENT_REGEX.match(line)[1]
-          else
-            safe_push(domain_hash, group, line.downcase)
-          end
+    def array_to_hash(lines)
+      domain_hash = {}
+      group = ''
+      lines.each do |line|
+        if COMMENT_REGEX.match?(line)
+          group = COMMENT_REGEX.match(line)[1]
+        else
+          safe_push(domain_hash, group, line.downcase)
         end
-        domain_hash
       end
+      domain_hash
+    end
+
+    # Add a value to an array in a hash, creating the array if necessary
+    # hash  - the hash
+    # key   - the key within that hash to add the value to
+    # value - the single value to push into the array at hash[key]
+    def safe_push(hash, key, value)
+      return if value.empty?
+
+      hash[key] ||= []
+      hash[key].push value
+    end
 
-      def safe_push(hash, key, value)
-        return if value.empty?
-        hash[key] ||= []
-        hash[key].push value
+    def sort_with_exceptions(left, right)
+      if left.start_with?('!') && !right.start_with?('!')
+        1
+      elsif right.start_with?('!') && !left.start_with?('!')
+        -1
+      else
+        left <=> right
       end
     end
   end