gman 4.3.1 → 4.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 11648796c6d730940bc52ae469f1e8d3ae6d0e0d
-  data.tar.gz: 29a866b7263245d010583e8ac2578d17f9ae7a73
+  metadata.gz: 94f77accd5bd6449f6f58a0cc0e2e3566555003d
+  data.tar.gz: 570a30e2c58019f9ca18bed1c797f6d0d2260cce
 SHA512:
-  metadata.gz: 554d33a96022f7db57269e42ba0e819e9af5eb83e40565f94ed79a78446522f7d9aa14b5965da5dd2da4cef78c96a6a3994877943461e556d847082999c33262
-  data.tar.gz: 486cf049f003be3409df9db65336adcbf177e57fb9561b5818d06dac36396293e8d801011db577872cc1868f0c0e57936c8fc7af3c774408e9f067dd357022fc
+  metadata.gz: ffd38055c145a9a48b85c0c83de8b2e0a60636750f89978859e52aa1f908551d90d12b6f117c058fa02c9aa742fe660ba56e0d0fdee8e0166e4d8784bfce762b
+  data.tar.gz: e937086398523d2030af950c07ded2b644d0a2d491eb7e5f4b23fa8f4f33a21e9af285f0bd561a392359ff5ff5085e084cecc45e7e4d9138e1940c5d1d9ae87e

data/.gitignore

@@ -0,0 +1,4 @@
+*.gem
+.bundle
+tmp
+Gemfile.lock

data/.travis.yml

@@ -0,0 +1,4 @@
+langauage: ruby
+script: "script/cibuild"
+sudo: false
+cache: bundler

data/CONTRIBUTING.md

@@ -0,0 +1,22 @@
+# Contributing to Gman
+
+## How to contribute
+
+1. Fork the project
+2. Create a descriptive branch
+3. Make your change
+4. Submit a pull request
+
+## Code
+
+Open an issue, or submit a pull request
+
+## Domains
+
+Domains live in `./config/domains.txt` as a list of TLDs and SLD+TLDs.
+
+Right now, the only valid government top level domains (TLDs), represent the US government and are `.gov`, and `.mil`.
+
+Secondary domains (e.g., `gov.uk`, or `mil.au`) detect non-US government entities.
+
+To add or remove a domain from the list of known government domains, simply edit the `domains.txt` file.

data/Gemfile

@@ -0,0 +1,6 @@
+source 'https://rubygems.org'
+gemspec
+
+group :development, :test do
+  gem 'net-dns'
+end

data/README.md

@@ -0,0 +1,119 @@
+# Gman Gem
+
+[![Build Status](https://travis-ci.org/benbalter/gman.png)](https://travis-ci.org/benbalter/gman) [![Gem Version](https://badge.fury.io/rb/gman.png)](http://badge.fury.io/rb/gman)
+
+A ruby gem to check if the owner of a given email address or website is working for THE MAN (a.k.a verifies government domains). It will also provide you with metadata about the domain, such as the country, state, city, or agency, where applicable. It does this by leveraging the power of [Naughty or Nice](https://github.com/benbalter/naughty_or_nice), the [Public Suffix List](http://publicsuffix.org/), and the associated [Ruby Gem](https://github.com/weppos/publicsuffix-ruby).
+
+You could theoretically [use regex](https://gist.github.com/benbalter/6147066), but either you'll get a bunch of false positives, or your regex will be insanely complicated. `gov.uk`, may be valid, for example, but `gov.fr` is not (it's `gouv.fr`, for what it's worth). The solution? Use Public Suffix to verify that it's a valid public domain, then maintain [a crowd-sourced sub-list of known global government and military domains](https://github.com/benbalter/gman/blob/master/config/domains.txt). It should cover all US and international, government and military domains for both email and website verification.
+
+See a domains that's missing or one that shouldn't be there? [We'd love you to contribute](CONTRIBUTING.md).
+
+## Installation
+
+Gman is a Ruby gem, so you'll need a little Ruby-fu to get it working. Simply
+
+`gem install gman`
+
+Or add this to your `Gemfile` before doing a `bundle install`:
+
+`gem 'gman'`
+
+## Usage
+
+### In general
+
+### Verify email addresses
+
+```ruby
+Gman.valid? "foo@bar.gov" #=> true
+Gman.valid? "foo@bar.com" #=> false
+```
+
+### Verify domain
+
+```ruby
+Gman.valid? "http://foo.bar.gov" #=> true
+Gman.valid? "foo.bar.gov"        #=> true
+Gman.valid? "foo.gov"            #=> true
+Gman.valid? "foo.biz"            #=> false
+```
+
+### Determine the type of domain
+
+```ruby
+domain = Gman.new "whitehouse.gov"
+domain.type     #=> :federal
+domain.federal? #=> true
+domain.state?   #=> false
+domain.city?    #=> false
+domain.county?  #=> false
+```
+
+### Get information about the domain's geographic location (.gov and .us only)
+
+```ruby
+domain = Gman.new "illinois.gov"
+domain.state #=> "IL"
+domain.city  #=> "springfield"
+```
+
+### Get information about a .gov domain's owner
+
+```ruby
+domain = Gman.new "whitehouse.gov"
+domain.agency   #=> "Executive Office of the President"
+```
+
+### Get the ISO Country Code information represented by a government domain
+
+```ruby
+domain = Gman.new "whitehouse.gov" #=> #<Gman domain="whitehouse.gov" valid=true>
+domain.country.name                #=> "United States"
+domain.country.alpha2              #=> "US"
+domain.country.alpha3              #=> "USA"
+domain.country.currency            #=> "USD"
+domain.conutry.calling_code        #=> "+1"
+```
+
+### Command line
+
+#### Getting information about a given domain
+
+```
+$ gman whitehouse.gov
+Domain  : whitehouse.gov
+Valid government domain
+Type    : federal
+Country : United States
+State   : DC
+City    : Washington
+Agency  : Executive Office of the President
+```
+
+The command line tool will accept any domain-like string (email, url, etc.)
+
+```
+$ gman foo@illinois.gov
+Domain  : illinois.gov
+Valid government domain
+Type    : state
+Country : United States
+State   : IL
+City    : Springfield
+```
+
+#### Filter
+
+Filters newline-separated email addresses from stdin. Example usage:
+
+```
+$ gman_filter < path/to/list/of/addresses.txt
+```
+
+## Contributing
+
+Contributions welcome! Please see [the contribution guidelines](CONTRIBUTING.md) for code contributions or for details on how to add, update, or delete government domains.
+
+## Credits
+
+Heavily inspired by [swot](https://github.com/leereilly/swot). Thanks [@leereilly](https://github.com/leereilly)!

data/Rakefile

@@ -0,0 +1,29 @@
+require 'rubygems'
+require 'bundler'
+begin
+  Bundler.setup(:default, :development)
+rescue Bundler::BundlerError => e
+  $stderr.puts e.message
+  $stderr.puts "Run `bundle install` to install missing gems"
+  exit e.status_code
+end
+require 'rake'
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/test_gman*.rb'
+  test.verbose = true
+end
+
+desc "Open console with gman loaded"
+task :console do
+  exec "irb -r ./lib/gman.rb"
+end
+
+desc "Validate the domain list"
+Rake::TestTask.new(:domains) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/test_domains*.rb'
+  test.verbose = true
+end

data/bin/gman

@@ -0,0 +1,42 @@
+#! /usr/bin/env ruby
+# Given a domain-link string, returns information about the domain
+
+require 'colorize'
+require_relative "../lib/gman"
+
+# Convenience method to simplify the command-line logic
+class IsoCountryCodes
+  class Code
+    def to_s
+      name
+    end
+  end
+end
+
+domain = ARGV[0]
+
+if domain.nil?
+  puts "USAGE: gman <domain or email address>".red
+  exit 1
+end
+
+gman = Gman.new(domain)
+
+puts "Domain  : #{gman.domain}"
+
+if gman.domain_parts.nil?
+  puts "Status  : " + "Invalid domain".red
+  exit 1
+end
+
+if !gman.valid?
+  puts "Status  : " + "Not a government domain".red
+  exit 1
+end
+
+puts "Status  : " + "Valid government domain".green
+
+["type", "country", "state", "city", "agency"].each do |key|
+  value = gman.send(key)
+  puts "#{key.capitalize.ljust(8)}: #{value}" if value
+end

data/gman.gemspec

@@ -0,0 +1,29 @@
+Gem::Specification.new do |s|
+  s.name = "gman"
+  s.summary = "Check if a given domain or email address belong to a governemnt entity"
+  s.description = "A ruby gem to check if the owner of a given email address is working for THE MAN."
+  s.version = '4.4.0'
+  s.authors = ["Ben Balter"]
+  s.email = "ben.balter@github.com"
+  s.homepage = "https://github.com/benbalter/gman"
+  s.licenses = ["MIT"]
+
+  s.files                 = `git ls-files`.split("\n")
+  s.test_files            = `git ls-files -- {test,spec,features}/*`.split("\n")
+  s.executables           = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
+  s.require_paths         = ["lib"]
+
+  s.require_paths = ["lib"]
+  s.add_dependency( "swot", '~> 0.4.2' )
+  s.add_dependency( "iso_country_codes", "~> 0.6" )
+  s.add_dependency( "naughty_or_nice", "~> 0.0.2" )
+  s.add_dependency( "colorize", "~> 0.7" )
+
+  s.add_development_dependency( "rake" )
+  s.add_development_dependency( "shoulda" )
+  s.add_development_dependency( "rdoc" )
+  s.add_development_dependency( "bundler" )
+  s.add_development_dependency( "pry" )
+  s.add_development_dependency( "parallel" )
+
+end

data/lib/gman/identifier.rb

@@ -13,6 +13,8 @@ class Gman < NaughtyOrNice
       :federal
     elsif county?
       :county
+    elsif list_category.nil?
+      nil
     elsif list_category.include?("usagov")
       :unknown
     else

data/lib/gman/parser.rb

@@ -0,0 +1,59 @@
+# Utility functions for parsing and manipulating public-suffix formatted domain lists
+require 'net/dns'
+require 'net/dns/resolver'
+
+class Gman < NaughtyOrNice
+  class Parser
+
+    COMMENT_REGEX = /\/\/[\/\s]*(.*)$/i
+
+    class << self
+
+      # Given a public-suffix list formatted file
+      # Converts to a hash in the form of :group => [domain1, domain2...]
+      def file_to_hash(file)
+        array_to_hash(file_to_array(file))
+      end
+
+      # Given a public-suffix list formatted file
+      # Convert it into an array of comments and domains representing each line
+      def file_to_array(file)
+        domains = File.open(file).read
+        domains.gsub! /\r\n?/, "\n" # Normalize line endings
+        domains = domains.split("\n")
+      end
+
+      # Given an array of comments/domains in public suffix format
+      # Converts to a hash in the form of :group => [domain1, domain2...]
+      def array_to_hash(domains)
+        group = ""
+        domain_hash = {}
+        domains.each do |line|
+          next if line.empty?
+          if match = COMMENT_REGEX.match(line)
+            group = match[1]
+          else
+            domain_hash[group] = [] if domain_hash[group].nil?
+            domain_hash[group].push line.downcase
+          end
+        end
+        domain_hash
+      end
+
+      def resolver
+        @resolver ||= begin
+          resolver = Net::DNS::Resolver.new
+          resolver.nameservers = ["8.8.8.8","8.8.4.4", "208.67.222.222", "208.67.220.220"]
+          resolver
+        end
+      end
+
+      # Verifies that the given domain has an MX record, and thus is valid
+      def domain_resolves?(domain)
+        resolver.search(domain).header.anCount > 0 ||
+        resolver.search(domain, Net::DNS::NS).header.anCount > 0 ||
+        resolver.search(domain, Net::DNS::MX).header.anCount > 0
+      end
+    end
+  end
+end

data/script/alphabetize

@@ -0,0 +1,42 @@
+#! /usr/bin/env ruby
+#
+# Alphabetizes entries in the domains.txt file
+#
+# usage: script/alphabetize
+
+require_relative "../lib/gman"
+
+# Read in existing list
+domains = File.open(Gman.list_path).read
+domains = domains.gsub /\r\n?/, "\n" #normalize line endings
+domains = domains.split("\n")
+
+# Split list into grouped hash
+group = ""
+domain_hash = {}
+domains.each do |line|
+  next if line.empty?
+  if match = /\/\/[\/\s]*(.*)$/i.match(line)
+    group = match[1]
+  else
+    domain_hash[group] = [] if domain_hash[group].nil?
+    domain_hash[group].push line.downcase
+  end
+end
+
+# Sort by groups
+domain_hash = domain_hash.sort_by { |k,v| k.downcase }.to_h
+
+# Sort within groups
+domain_hash.each do |group, domains|
+  domain_hash[group].sort!
+end
+
+output = ""
+domain_hash.each do |group, domains|
+  output << "// #{group}\n"
+  output << domains.join("\n")
+  output << "\n\n"
+end
+
+File.write Gman.list_path, output.strip

data/script/build

@@ -0,0 +1,27 @@
+#!/usr/bin/env ruby
+# Propagates an initial list of best-guess government domains
+
+require "public_suffix"
+require "yaml"
+
+# https://gist.github.com/benbalter/6147066
+REGEX = /(\.g[ou]{1,2}(v|b|vt)|\.mil|\.gc|\.fed)(\.[a-z]{2})?$/i
+
+YAML_FILE = File.dirname(__FILE__) + "/../lib/domains.yml"
+domains = YAML.load_file YAML_FILE
+domains = [] unless domains
+
+PublicSuffix::List.default.each do |rule|
+  domain = nil
+
+  if rule.parts.length == 1
+    domain = rule.parts.first if ".#{rule.value}" =~ REGEX
+  else
+    domain = rule.parts.pop(2).join(".") if ".#{rule.value}" =~ REGEX
+  end
+
+  domains.push domain unless domain.nil? or domains.include? domain
+end
+
+domains = domains.sort
+File.open(YAML_FILE, 'w+') {|f| f.write(domains.to_yaml)}

data/script/cibuild

@@ -0,0 +1,7 @@
+#!/bin/sh
+
+set -e
+
+bundle exec rake test
+bundle exec script/dedupe
+bundle exec script/state-domains

data/script/console

@@ -0,0 +1,3 @@
+#! /bin/sh
+
+bundle exec pry -r ./lib/gman.rb

data/script/dedupe

@@ -0,0 +1,38 @@
+#! /usr/bin/env ruby
+
+require 'yaml'
+require 'open-uri'
+require './lib/gman'
+require './lib/gman/parser'
+
+
+current = Gman::Parser.file_to_array( Gman::list_path )
+domain_hash = Gman::Parser.array_to_hash(current)
+domain_list = domain_hash.flat_map { |k,v| v }
+
+puts "Checking for duplicate domains in the domain list..."
+puts "Current list contains #{domain_list.count} domains..."
+
+SOURCE = "https://raw.githubusercontent.com/GSA/govt-urls/master/government-urls.yaml"
+source_hash = YAML.load(open(SOURCE).read)
+source_list = source_hash.flat_map { |k,v| v }
+
+dupes = []
+domain_hash.each do |group,domains|
+  domains.each do |domain|
+    if domain_list.count(domain) > 1 && source_list.count(domain) <= 1
+      dupes.push(domain)
+    end
+  end
+end
+
+dupes.uniq!
+
+puts "Found #{dupes.count} dupes!"
+
+if dupes.count > 0
+  puts dupes.inspect
+  exit 1
+else
+  exit 0
+end

data/script/prune

@@ -0,0 +1,17 @@
+#! /usr/bin/env ruby
+# Given an array of domains, removes them from the list
+# Example usage: script/prune foo.invalid, bar.invalid, foo.bar.invalid
+
+domains = ARGV
+domains = domains.clone.map { |d| d.gsub ",", "" }
+
+list = File.open("./config/domains.txt").read
+puts "Starting list: #{list.size} lines"
+
+domains.each do |domain|
+  list.gsub! /^#{domain}$\n/, ""
+end
+
+puts "Ending list: #{list.size} lines"
+
+File.write "./config/domains.txt", list

data/script/release

@@ -0,0 +1,38 @@
+#!/bin/sh
+# Tag and push a release.
+
+set -e
+
+# Make sure we're in the project root.
+
+cd $(dirname "$0")/..
+
+# Build a new gem archive.
+
+rm -rf gman-*.gem
+gem build -q gman.gemspec
+
+# Make sure we're on the master branch.
+
+(git branch | grep -q '* master') || {
+  echo "Only release from the master branch."
+  exit 1
+}
+
+# Figure out what version we're releasing.
+
+tag=v`ls gman-*.gem | sed 's/^gman-\(.*\)\.gem$/\1/'`
+
+# Make sure we haven't released this version before.
+
+git fetch -t origin
+
+(git tag -l | grep -q "$tag") && {
+  echo "Whoops, there's already a '${tag}' tag."
+  exit 1
+}
+
+# Tag it and bag it.
+
+gem push gman-*.gem && git tag "$tag" &&
+  git push origin master && git push origin "$tag"

data/script/state-domains

@@ -0,0 +1,38 @@
+#! /usr/bin/env ruby
+# Strips domains in the form of e.g., city.<locality>.<state>.us from the domain list
+
+require './lib/gman'
+require './lib/gman/parser'
+
+current = Gman::Parser.file_to_array( Gman::list_path )
+domain_hash = Gman::Parser.array_to_hash(current)
+
+puts "Checking for state gov regex'd domains in the list..."
+puts "Starting with #{current.size} domains..."
+
+domain_hash.each do |group, domains|
+  next unless group =~ /usagov[A-Z]{2}/
+  state = group[-2,2].downcase
+  domain_hash[group].reject! { |d| d =~ Gman::LOCALITY_REGEX }
+  domain_hash[group].uniq!
+  domain_hash[group].sort!
+end
+
+# PublicSuffix Formatted Output
+current_group = ""
+output = ""
+domain_hash.each do |group, domains|
+  if group != current_group
+    output << "\n\n" unless current_group.empty? # first entry
+    output << "// #{group}\n"
+    current_group = group
+  end
+  output << domains.join("\n")
+end
+
+File.open(Gman.list_path, "w") { |file| file.write output }
+
+result = Gman::Parser.file_to_array( Gman::list_path )
+puts "New list contains #{result.size} domains. Fin."
+
+exit 1 if current.size != result.size

data/script/vendor-de

@@ -0,0 +1,44 @@
+#! /usr/bin/env ruby
+
+require 'csv'
+require 'open-uri'
+require './lib/gman'
+require './lib/gman/parser'
+
+source = "http://www.mik.nrw.de/nc/themen-aufgaben/kommunales/kommunale-adressen.html?tx_szkommunaldb_pi1%5Bexport%5D=csv"
+
+csv = open(source).read.force_encoding("iso-8859-1").encode("UTF-8")
+
+# For some reason, the header row is actually the last row
+# Pop the last line off the file and prepend it at the begining
+# So that when we pass it to CSV it detects the headers properly
+lines = csv.split("\n")
+lines.unshift lines.pop
+csv = lines.join("\n")
+
+data = CSV.parse(csv, :headers => true, :col_sep => ";")
+domains = data.map { |row| row["Internet"].to_s.downcase.strip.gsub /^www./, "" }
+
+domains.reject! { |domain| domain.empty? }
+domains.select! { |domain| PublicSuffix.valid?(".#{domain}") } # Validate domain
+domains.reject! { |domain| Swot::is_academic?(domain) }  # Reject academic domains
+
+current = Gman::Parser.file_to_array( Gman::list_path )
+current_hash = Gman::Parser.array_to_hash(current)
+
+current_hash["German Municipalities"] = domains
+current_hash = current_hash.sort_by { |group, domains| group.downcase }
+
+# PublicSuffix Formatted Output
+current_group = ""
+output = ""
+current_hash.each do |group, domains|
+  if group != current_group
+    output << "\n\n" unless current_group.empty? # first entry
+    output << "// #{group}\n"
+    current_group = group
+  end
+  output << domains.join("\n")
+end
+
+File.open(Gman.list_path, "w") { |file| file.write output }

data/script/vendor-gov-list

@@ -0,0 +1,5 @@
+#!/bin/sh
+
+DATE=2014-12-01
+
+wget "https://raw.githubusercontent.com/GSA/data/gh-pages/dotgov-domains/$DATE-full.csv" -O config/vendor/dotgovs.csv

data/script/vendor-us

@@ -0,0 +1,82 @@
+#! /usr/bin/env ruby
+#
+# Vendors the USA.gov-maintained list of US domains into domains.txt
+# Source: https://github.com/GSA-OCSIT/govt-urls
+#
+# Normalizes and cleans inputs, validates domains, rejects academic domains, and
+# sorts, ensures uniqueness, and merges into the existing lib/domains.txt list
+#
+# Usage: script/vendor-us
+#
+# Will automatically fetch latest version of the list and merge
+# You can check for changes and commit via `git status`
+#
+# It's also probably a good idea to run `script/ci-build` for good measure
+
+require 'rubygems'
+require 'public_suffix'
+require 'swot'
+require 'yaml'
+require 'open-uri'
+require './lib/gman'
+require './lib/gman/parser'
+
+SOURCE = "https://raw.githubusercontent.com/GSA/govt-urls/master/government-urls.yaml"
+BLACKLIST = ["usagovQUASI", "usagovFED", "usagovPW"]
+domain_hash = {}
+
+domain_hash = YAML.load(open(SOURCE).read)
+puts "found #{domain_hash.map { |group,domains| domains.count }.inject(:+)} domains..."
+
+# Normalize ALL THE THINGS
+domain_hash.each do |group, domains|
+  domains.map! { |domain| domain.strip } # Strip trailing slashes
+  domains.map! { |domain| domain.gsub /\/$/, "" } # Strip trailing slashes
+  domains.map! { |domain| domain.downcase } # make lower case
+  domains.reject! { |domain| domain.empty? }      # Reject empty strings
+end
+
+# filter
+domain_hash.reject! { |group,domain| BLACKLIST.include?(group) } # Group blacklist
+domain_hash.each do |group, domains|
+  puts "Filtering #{group}..."
+  domains.reject! { |domain| domain.match /\// }           # Reject URLs
+  domains.select! { |domain| PublicSuffix.valid?(domain) } # Validate domain
+  domains.reject! { |domain| Swot::is_academic?(domain) }  # Reject academic domains
+end
+puts "Filtered down to #{domain_hash.map { |group,domains| domains.count }.inject(:+)} domains"
+
+# Grab existing list
+current = Gman::Parser.file_to_array( Gman::list_path )
+current_hash = Gman::Parser.array_to_hash(current)
+puts "Current list contains #{current.size} domains... merging"
+
+# Lazy deep merge
+domain_hash.each do |group,domains|
+  current_hash[group] = [] if current_hash[group].nil?
+  current_hash[group].concat domains
+  current_hash[group].sort! # Alphabetize
+  current_hash[group].uniq! # Ensure uniqueness
+end
+
+# Sort by group
+current_hash = current_hash.sort_by { |group, domains| group.downcase }
+
+# PublicSuffix Formatted Output
+current_group = ""
+output = ""
+current_hash.each do |group, domains|
+  if group != current_group
+    output << "\n\n" unless current_group.empty? # first entry
+    output << "// #{group}\n"
+    current_group = group
+  end
+  output << domains.join("\n")
+end
+
+puts "merged. Writing..."
+
+File.open(Gman.list_path, "w") { |file| file.write output }
+
+result = Gman::Parser.file_to_array( Gman::list_path )
+puts "New list contains #{result.size} domains. Fin."

data/test/helper.rb

@@ -0,0 +1,26 @@
+require 'rubygems'
+require 'bundler'
+require 'minitest/autorun'
+require 'parallel'
+require 'open3'
+
+begin
+  Bundler.setup(:default, :development)
+rescue Bundler::BundlerError => e
+  $stderr.puts e.message
+  $stderr.puts "Run `bundle install` to install missing gems"
+  exit e.status_code
+end
+
+require 'shoulda'
+
+$LOAD_PATH.unshift(File.join(File.dirname(__FILE__), '..', 'lib'))
+$LOAD_PATH.unshift(File.dirname(__FILE__))
+require 'gman'
+require 'net/dns'
+require 'net/dns/resolver'
+require './lib/gman/parser'
+
+def test_bin(*args)
+  output, status = Open3.capture2e("bundle", "exec", "gman", *args)
+end

data/test/obama.txt

@@ -0,0 +1,5 @@
+barry@dcpchicago.org
+prof.obama@uchicago.edu
+mr.senator@obama.senate.gov
+president@whitehouse.gov
+commander.in.chief@us.army.mil

data/test/test_domains.rb

@@ -0,0 +1,60 @@
+require File.join(File.dirname(__FILE__), 'helper')
+
+class TestDomains < Minitest::Test
+
+  WHITELIST = [ "non-us gov", "non-us mil", "US Federal"]
+  DOMAINS = Gman::Parser.file_to_hash(Gman.list_path)
+
+  def whitelisted?(domain)
+    WHITELIST.each do |group|
+      return true if DOMAINS[group].include? domain
+    end
+    false
+  end
+
+  should "only contain resolvable domains" do
+    unresolvables = []
+    Parallel.each(Gman.list, :in_threads => 2) do |entry|
+      next if whitelisted? entry.name
+      resolves = Gman::Parser.domain_resolves?(entry.name)
+      unresolvables.push entry.name unless resolves
+    end
+    assert_equal [], unresolvables
+  end
+
+  should "not contain any educational domains" do
+    Parallel.each(Gman.list, :in_threads => 2) do |entry|
+      assert_equal false, Swot::is_academic?(entry.name), "#{entry.name} is an academic domain"
+    end
+  end
+
+  should "not contain any invalid domains" do
+    Parallel.each(Gman.list, :in_threads => 2) do |entry|
+      assert_equal true, PublicSuffix.valid?("foo.#{entry.name}"), "#{entry.name} is not a valid domain"
+    end
+  end
+
+  should "pass any url on the list" do
+    Parallel.each(Gman.list, :in_threads => 2) do |entry|
+      assert_equal true, Gman.valid?("http://foo.#{entry.name}/bar"), "http://foo.#{entry.name}/bar is not a valid"
+    end
+  end
+
+  should "pass any email on the list" do
+    Parallel.each(Gman.list, :in_threads => 2) do |entry|
+      assert_equal true, Gman.valid?("foo@bar.#{entry.name}"), "foo@bar.#{entry.name} is not a valid"
+    end
+  end
+
+  should "pass any domain on the list" do
+    Parallel.each(Gman.list, :in_threads => 2) do |entry|
+      assert_equal true, Gman.valid?("foo.#{entry.name}"), "foo.#{entry.name} is not a valid domain"
+    end
+  end
+
+  should "identify the coutnry for any domain on the list" do
+    Parallel.each(Gman.list, :in_threads => 2) do |entry|
+      Gman.new("foo.#{entry.name}").country.name
+    end
+  end
+end

data/test/test_gman.rb

@@ -0,0 +1,57 @@
+require File.join(File.dirname(__FILE__), 'helper')
+
+VALID = [  "foo.gov",
+            "http://foo.mil",
+            "foo@bar.gc.ca",
+            "foo.gov.au",
+            "https://www.foo.gouv.fr",
+            "foo@ci.champaign.il.us",
+            "foo.bar.baz.gov.au",
+            "foo@bar.gov.uk",
+            ".gov",
+            "foo.fed.us",
+            "foo.state.il.us",
+            "state.il.us",
+            "foo@af.mil",
+            "gov.in"
+        ]
+
+INVALID = [ "foo.bar.com",
+            "bar@foo.biz",
+            "http://www.foo.biz",
+            "foo.uk",
+            "gov",
+            "foo@k12.champaign.il.us",
+            "foo@kii.gov.by",
+            "foo",
+            "",
+            nil,
+            " ",
+            "foo.city.il.us",
+            "foo.ci.il.us",
+            "foo.zx.us",
+            "foo@mail.gov.ua"
+          ]
+
+class TestGman < Minitest::Test
+
+  VALID.each do |domain|
+    should "recognize #{domain} as a government domain" do
+      assert Gman::valid?(domain)
+    end
+  end
+
+  INVALID.each do |domain|
+    should "recognize #{domain} as a non-government domain" do
+      refute Gman::valid?(domain)
+    end
+  end
+
+  should "not allow educational domains" do
+    assert_equal false, Gman::valid?("foo@gwu.edu")
+  end
+
+  should "returns the path to domains.txt" do
+    assert_equal true, File.exists?(Gman.list_path)
+  end
+end

data/test/test_gman_bin.rb

@@ -0,0 +1,56 @@
+require_relative "helper"
+
+class TestGmanBin < Minitest::Test
+
+  def setup
+    @output, @status = test_bin("whitehouse.gov")
+  end
+
+  should "parse the domain" do
+    output, status = test_bin("bar.gov")
+    assert_match /Domain  : bar.gov/, output
+
+    output, status = test_bin("foo@bar.gov")
+    assert_match /Domain  : bar.gov/, output
+
+    output, status = test_bin("http://bar.gov/foo")
+    assert_match /Domain  : bar.gov/, output
+  end
+
+  should "err on invalid domains" do
+    output, status = test_bin("foo.invalid")
+    assert_equal 1, status.exitstatus
+    assert_match /Invalid domain/, output
+  end
+
+  should "err on non-government domains" do
+    output, status = test_bin("github.com")
+    assert_equal 1, status.exitstatus
+    assert_match /Not a government domain/, output
+  end
+
+  should "know the type" do
+    assert_match /federal/, @output
+    assert_equal 0, @status.exitstatus
+  end
+
+  should "know the agency" do
+    assert_match /Executive Office of the President/, @output
+    assert_equal 0, @status.exitstatus
+  end
+
+  should "know the country" do
+    assert_match /United States/, @output
+    assert_equal 0, @status.exitstatus
+  end
+
+  should "know the city" do
+    assert_match /Washington/, @output
+    assert_equal 0, @status.exitstatus
+  end
+
+  should "know the state" do
+    assert_match /DC/, @output
+    assert_equal 0, @status.exitstatus
+  end
+end

data/test/test_gman_country_codes.rb

@@ -0,0 +1,10 @@
+require File.join(File.dirname(__FILE__), 'helper')
+
+class TestGmanCountryCodes < Minitest::Test
+  should "determine a domain's country" do
+    assert_equal "United States", Gman.new("whitehouse.gov").country.name
+    assert_equal "United States", Gman.new("army.mil").country.name
+    assert_equal "United Kingdom", Gman.new("foo.gov.uk").country.name
+    assert_equal "Canada", Gman.new("foo.gc.ca").country.name
+  end
+end

data/test/test_gman_filter.rb

@@ -0,0 +1,19 @@
+HERE = File.dirname(__FILE__)
+require File.join(HERE, 'helper')
+
+class TestGmanFilter < Minitest::Test
+
+  txt_path = File.join(HERE, "obama.txt")
+  exec_path = File.join(HERE, "..", "bin", "gman_filter")
+
+  should "remove non-gov/mil addresses" do
+    filtered = `#{exec_path} < #{txt_path}`
+    expected = %w(
+      mr.senator@obama.senate.gov
+      president@whitehouse.gov
+      commander.in.chief@us.army.mil
+    ).join("\n") + "\n"
+    assert_equal filtered, expected
+  end
+
+end

data/test/test_gman_identifier.rb

@@ -0,0 +1,106 @@
+require File.join(File.dirname(__FILE__), 'helper')
+
+class TestGmanIdentifier < Minitest::Test
+  should "Parse the dotgov list" do
+    assert Gman.dotgov_list
+    assert_equal CSV::Table, Gman.dotgov_list.class
+    assert_equal CSV::Row, Gman.dotgov_list.first.class
+    assert Gman.dotgov_list.first["Domain Name"]
+  end
+
+  context "locality domains" do
+    should "detect state domains" do
+      domain = Gman.new("state.ak.us")
+      assert domain.state?
+
+      refute domain.dotgov?
+      refute domain.city?
+      refute domain.federal?
+      refute domain.county?
+
+      assert_equal :state, domain.type
+      assert_equal "AK", domain.state
+    end
+
+    should "detect city domains" do
+      domain = Gman.new("ci.champaign.il.us")
+      assert domain.city?
+
+      refute domain.dotgov?
+      refute domain.state?
+      refute domain.federal?
+      refute domain.county?
+
+      assert_equal :city, domain.type
+      assert_equal "IL", domain.state
+    end
+  end
+
+  context "dotgovs" do
+    should "detect federal dotgovs" do
+      domain = Gman.new "whitehouse.gov"
+      assert domain.federal?
+      assert domain.dotgov?
+
+      refute domain.city?
+      refute domain.state?
+      refute domain.county?
+
+      assert_equal :federal, domain.type
+      assert_equal "DC", domain.state
+      assert_equal "Washington", domain.city
+      assert_equal "Executive Office of the President", domain.agency
+    end
+
+    should "detect state dotgovs" do
+      domain = Gman.new "illinois.gov"
+      assert domain.state?
+      assert domain.dotgov?
+
+      refute domain.city?
+      refute domain.federal?
+      refute domain.county?
+
+      assert_equal :state, domain.type
+      assert_equal "IL", domain.state
+      assert_equal "Springfield", domain.city
+    end
+
+    should "detect county dotgovs" do
+      domain = Gman.new "ALLEGHENYCOUNTYPA.GOV"
+      assert domain.county?
+      assert domain.dotgov?
+
+      refute domain.city?
+      refute domain.federal?
+      refute domain.state?
+
+      assert_equal :county, domain.type
+      assert_equal "PA", domain.state
+      assert_equal "Pittsburgh", domain.city
+    end
+
+    should "detect the list category" do
+      assert_equal "US Federal", Gman.new("whitehouse.gov").send("list_category")
+    end
+  end
+
+  context "non-dotgov domains" do
+    should "determine a domain's group" do
+      assert_equal "usagovIN", Gman.new("cityofperu.org").send("list_category")
+      assert_equal :unknown, Gman.new("cityofperu.org").type
+
+      assert_equal "Canada municipal", Gman.new("acme.ca").send("list_category")
+      assert_equal :"Canada municipal", Gman.new("acme.ca").type
+
+      assert_equal "Canada federal", Gman.new("canada.ca").send("list_category")
+      assert_equal :"Canada federal", Gman.new("canada.ca").type
+    end
+
+    should "detect the state" do
+      assert_equal "PR", Gman.new("sanjuan.pr").state
+      assert_equal "OR", Gman.new("ashland.or.us").state
+      refute Gman.new("canada.ca").state
+    end
+  end
+end

data/test/test_gman_locality.rb

@@ -0,0 +1,10 @@
+require File.join(File.dirname(__FILE__), 'helper')
+
+class TestGmanLocality < Minitest::Test
+  should "parse the alpha2" do
+    assert_equal "us", Gman.new("whitehouse.gov").alpha2
+    assert_equal "us", Gman.new("army.mil").alpha2
+    assert_equal "gb", Gman.new("foo.gov.uk").alpha2
+    assert_equal "ca", Gman.new("gov.ca").alpha2
+  end
+end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: gman
 version: !ruby/object:Gem::Version
-  version: 4.3.1
+  version: 4.4.0
 platform: ruby
 authors:
 - Ben Balter
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-01-22 00:00:00.000000000 Z
+date: 2015-01-28 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: swot
@@ -30,14 +30,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.4'
+        version: '0.6'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '0.4'
+        version: '0.6'
 - !ruby/object:Gem::Dependency
   name: naughty_or_nice
   requirement: !ruby/object:Gem::Requirement
@@ -52,6 +52,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: 0.0.2
+- !ruby/object:Gem::Dependency
+  name: colorize
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.7'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.7'
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
@@ -140,18 +154,48 @@ description: A ruby gem to check if the owner of a given email address is workin
   for THE MAN.
 email: ben.balter@github.com
 executables:
+- gman
 - gman_filter
 extensions: []
 extra_rdoc_files: []
 files:
+- ".gitignore"
+- ".travis.yml"
+- CONTRIBUTING.md
+- Gemfile
 - LICENSE
+- README.md
+- Rakefile
+- bin/gman
 - bin/gman_filter
 - config/domains.txt
 - config/vendor/dotgovs.csv
+- gman.gemspec
 - lib/gman.rb
 - lib/gman/country_codes.rb
 - lib/gman/identifier.rb
 - lib/gman/locality.rb
+- lib/gman/parser.rb
+- script/alphabetize
+- script/build
+- script/cibuild
+- script/console
+- script/dedupe
+- script/prune
+- script/release
+- script/state-domains
+- script/vendor-de
+- script/vendor-gov-list
+- script/vendor-us
+- test/helper.rb
+- test/obama.txt
+- test/test_domains.rb
+- test/test_gman.rb
+- test/test_gman_bin.rb
+- test/test_gman_country_codes.rb
+- test/test_gman_filter.rb
+- test/test_gman_identifier.rb
+- test/test_gman_locality.rb
 homepage: https://github.com/benbalter/gman
 licenses:
 - MIT
@@ -176,4 +220,13 @@ rubygems_version: 2.2.0
 signing_key: 
 specification_version: 4
 summary: Check if a given domain or email address belong to a governemnt entity
-test_files: []
+test_files:
+- test/helper.rb
+- test/obama.txt
+- test/test_domains.rb
+- test/test_gman.rb
+- test/test_gman_bin.rb
+- test/test_gman_country_codes.rb
+- test/test_gman_filter.rb
+- test/test_gman_identifier.rb
+- test/test_gman_locality.rb