my_representatives 0.0.2

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: b0f5e144e58bda1fd4a0723829000e1f962b4ab2
+  data.tar.gz: c7a2f8c5caafe1cd8d473c17ce5ce0c6854cab7f
+SHA512:
+  metadata.gz: b4c48b64e2d08ad906eb9cf48c4375d7e25ab7bcd0d2b711f1267ab11018bc2d3d7d6bc6a0a20df8421ed2975eca15a83a5f9e8698c615b093648d94e3bc8796
+  data.tar.gz: 3511bdfef1dcd5216f601f0d009c131c88c2c4b865f2e7864a7d5909559f4399b21ac1f52fe0437f271f798bab502f7be35578cd60578a0c1c51de70a03ed623

data/.gitignore

@@ -0,0 +1,4 @@
+*.gem
+.DS_Store
+Gemfile.lock
+tmp

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+
+gemspec

data/LICENSE.md

@@ -0,0 +1,8 @@
+The MIT License (MIT)
+Copyright (c) 2017 Jeremy Tennant
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,60 @@
+# MyRepresentatives
+Aggregates contact information for Australian State and Commonwealth Members, and Senators. Data is output as ```MyRepresentatives::Person``` and ```MyRepresentatives::Electorate``` ruby objects, or in CSV files.
+
+## Quick Start
+**Install the gem**
+```ruby
+$ gem install MyRepresentatives
+```
+**Start IRB**
+```ruby
+$ irb
+irb(main):001:0> require 'my_representatives'
+=> true
+irb(main):002:0> MyRepresentatives.new
+...
+```
+*A CSV file will be output to 'tmp/csv_all_representatives.csv' (relative to the working directory).*
+
+**Or perhaps something like this:**
+```ruby
+$ irb
+require 'my_representatives'
+# State of Victoria Lower House (Legislative Assembly) - from CSV
+representatives = MyRepresentatives::VIC::CSVLower.new
+representatives.people
+# Or Victoria's Upper House (Legislative Council) - scraped from the web
+representatives = MyRepresentatives::VIC::WebUpper.new
+representatives.people
+...
+# Commonwealth Upper House (Senate) - merge CSV and Web Scraping to form a 'complete' data source
+representatives = MyRepresentatives::Commonwealth::MergeUpper.new
+representatives.people
+...
+# Northern Territory Lower House (Legislative Assembly) - Scrape from the web
+representatives = MyRepresentatives::NT::WebLower.new
+representatives.people
+...
+```
+
+**Looking for more?**
+
+Browse the source!
+
+## Edge cases
+Tasmania and Northern Territory governments do not, at this time, supply CSV files (although they do offer Microsoft Word documents).
+
+#### Tasmania
+This gem presently uses manually created CSV files for Tasmania's [House of Assembly](https://gist.github.com/tennantje/c94bd62340bd078b9b1ae12168f48388) and [Legislative Council](https://gist.github.com/tennantje/c662bd7a2ac44d2e5839ad73d6d0a80e) based on these official sources - [House of Assembly](http://www.parliament.tas.gov.au/HA/HAMem.doc) and  [Legislative Council](http://www.parliament.tas.gov.au/LC/LCMem.doc). Neither document are regularly checked for accuracy, therefore if you identify a change in government - please notify its author. Unfortunately there is no support for web scraping, as there is no scrapable data on the Tasmanian Parliament website at this time.
+
+#### Northern Territory
+Unfortunately there is no support for Northern Territory CSV at this time. There is, however, the  ```MyRepresentatives:::NT::WebLower``` class if you're happy for the data to be scraped from the NT parliament website (which is a good source of data).
+
+## Dependencies
+Some State governments (Queensland and Western Australia) supply data in Microsoft Excel's legacy XLS format (rather than the modern XLSX format - or even CSV (our preference)). MyRepresentatives uses the **excel2csv** gem to convert XLS files to CSV. At this time, excel2csv requires [JDK](http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html) to be installed on your system.
+
+## Accuracy of Data
+Per the License, "THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND" please be aware that scraping data from websites is problematic - structural changes to Government websites are outside the control of the authors of this gem, and have the ability to break some or all compatibility without notice. In developing this gem, I have also identified several errors in the CSV files published by official sources - so this is also a significant roadblock in ensuring the data this gem produces is accurate.
+
+## License
+MIT - For terms refer to LICENSE.md

data/lib/my_representatives/abbreviatable.rb

@@ -0,0 +1,29 @@
+module MyRepresentatives
+  module Abbreviatable
+
+    def state_abbreviation(str)
+      if Static::STATES.include?(str.upcase)
+        str.upcase
+      elsif str.upcase.strip == "AUSTRALIAN CAPITAL TERRITORY"
+        "ACT"
+      elsif str.upcase.strip == "NEW SOUTH WALES"
+        "NSW"
+      elsif str.upcase.strip == "NORTHERN TERRITORY"
+        "NT"
+      elsif str.upcase.strip == "QUEENSLAND"
+        "QLD"
+      elsif str.upcase.strip == "SOUTH AUSTRALIA"
+        "SA"
+      elsif str.upcase.strip == "TASMANIA"
+        "TAS"
+      elsif str.upcase.strip == "VICTORIA"
+        "VIC"
+      elsif str.upcase.strip == "WESTERN AUSTRALIA"
+        "WA"
+      else
+        "UNKNOWN" # Will likely trigger an error elsewhere in the app, but that's okay
+      end
+    end
+
+  end # Abbreviatable
+end # MyRepresentatives

data/lib/my_representatives/act/csv_lower.rb

@@ -0,0 +1,118 @@
+module MyRepresentatives
+  module ACT
+    class CSVLower
+      include MyRepresentatives::Fileable
+      include MyRepresentatives::Guessable
+
+      attr_accessor :csv_url, :csv_filename, :people, :logger
+
+      def initialize
+        @logger = Logger.new(STDOUT)
+        @csv_url = "http://www.parliament.act.gov.au/__data/assets/file/0011/874235/Members-mail-merge-list-2016-1108.csv"
+        @csv_filename = "tmp/csv_act_lower.csv"
+        @people = []
+
+        csv_from_url
+        people_from_csv
+      end
+
+
+      private
+
+        def csv_from_url
+          create_tmp
+
+          begin
+            open(@csv_filename, "wb") do |file|
+              open(@csv_url) do |uri|
+                file.write(uri.read)
+              end
+            end
+          rescue SocketError => err
+            @logger.debug("Unable to connect to #{@csv_url}")
+            raise err
+          end
+        end
+
+        def people_from_csv
+          CSV.foreach(@csv_filename, headers: true) do |row|
+
+            # Setup the electorate
+            electorate_name = find_electorate(row)
+            electorate      = Electorate.new(electorate_name)
+            electorate.state_act_lower!
+
+            # Setup the Representative (Person)
+            person = Person.new(electorate)
+
+            person.title                      = find_title(row)
+            person.first_name                 = find_first_name(row)
+            person.last_name                  = find_last_name(row)
+            person.email                      = find_email(row)
+            person.phone                      = find_phone(row)
+            person.party_name                 = find_party(row)
+            person.honorifics                 = "MLA"
+            person.formal_name                = format_formal_name(person)
+            person.physical_address           = format_physical_address(row)
+            person.postal_address             = format_postal_address(row)
+            person.gender                     = guess_gender(person.title)
+            person.preferred_name             = nil
+            person.salutation                 = nil
+            person.image_url                  = nil
+            person.homepage_url               = nil
+
+            @people << person
+          end
+        end
+
+        def find_first_name(arr)
+          arr["F_Name"]
+        end
+
+        def find_last_name(arr)
+          if arr && arr["L_Name"]
+            arr["L_Name"].gsub(" MLA","")
+          else
+            nil
+          end
+        end
+
+        def find_electorate(arr)
+          if arr && arr["JobTitle"]
+            arr["JobTitle"].gsub("Member for ","")
+          else
+            nil
+          end
+        end
+
+        def find_email(arr)
+          arr["Email"]
+        end
+
+        def find_party(arr)
+          arr["Party"]
+        end
+
+        def find_phone(arr)
+          arr["Phone"]
+        end
+
+        def find_title(arr)
+          arr["Title"]
+        end
+
+        def format_formal_name(person)
+          "#{person.title} #{person.first_name} #{person.last_name} #{person.honorifics}"
+        end
+
+        def format_physical_address(arr)
+          "#{arr["Address1"]}\n#{arr["City"]} #{arr["State"]} #{arr["PostCode"]}"
+        end
+
+        def format_postal_address(arr)
+          "#{arr["Address1"]}\n#{arr["City"]} #{arr["State"]} #{arr["PostCode"]}"
+        end
+
+    end # CSVLower
+  end # ACT
+end # MyRepresentatives

data/lib/my_representatives/act/merge_lower.rb

@@ -0,0 +1,38 @@
+module MyRepresentatives
+  module ACT
+    class MergeLower
+      attr_accessor :csv_people, :web_people, :people
+
+      def initialize
+        @csv_people = MyRepresentatives::ACT::CSVLower.new.people
+        @web_people = MyRepresentatives::ACT::WebLower.new.people
+        @people = []
+        check_and_update_person
+      end
+
+      private
+
+        def check_and_update_person
+          @csv_people.each do |person|
+            if @web_people.any? { |wp| wp.phone == person.phone} || @web_people.any? { |wp| wp.email == person.email}
+              index = @web_people.find_index {|wp| wp.phone == person.phone}
+
+              if !index
+                index = @web_people.find_index { |wp| wp.email == person.email }
+              end
+
+              if index
+                person.homepage_url     = @web_people[index].homepage_url
+                person.image_url        = @web_people[index].image_url
+                person.successful_merge = true
+              end
+
+            end
+            @people << person
+          end
+        end
+
+
+    end # Merge
+  end # ACT
+end # MyRepresentatives

data/lib/my_representatives/act/web_index.rb

@@ -0,0 +1,37 @@
+module MyRepresentatives
+  module ACT
+    class WebIndex
+      attr_accessor :index_url, :document, :representative_documents
+
+      def initialize
+        @logger = Logger.new(STDOUT)
+        @index_url                = "http://www.parliament.act.gov.au/members/current"
+        @document                 = find_representatives
+        @representative_documents = []
+        representative_documents_from_document
+      end
+
+
+      private
+
+        def find_representatives
+          begin
+            request = open(@index_url)
+            Nokogiri::HTML(request)
+          rescue SocketError => err
+            @logger.debug("Unable to connect to #{@index_url}")
+            raise err
+          end
+        end
+
+        def representative_documents_from_document
+          table = @document.css("table.tablesorter")
+
+          table.css("tbody").css("tr").each do |row|
+            @representative_documents << row unless row.text.strip.empty?
+          end
+        end
+
+    end # WebIndex
+  end # ACT
+end # MyRep

data/lib/my_representatives/act/web_index_row.rb

@@ -0,0 +1,86 @@
+module MyRepresentatives
+  module ACT
+    class WebIndexRow
+      attr_accessor :document
+
+      def initialize(document)
+        raise MyRepresentatives::NokogiriDocumentExpectedError unless document && document.is_a?(Nokogiri::XML::Element)
+        @document = document
+        raise MyRepresentatives::NokogiriDocumentPropertiesError unless test_document_for_validity
+      end
+
+      def first_name
+        begin
+          @document.css("a")[0].text.split(",")[1]
+        rescue
+          nil
+        end
+      end
+
+      def last_name
+        begin
+          @document.css("a")[0].text.split(",")[0]
+        rescue
+        end
+      end
+
+      def electorate_name
+        begin
+          @document.css("td")[2].text
+        rescue
+        end
+      end
+
+      def party_name
+        begin
+          @document.css("td")[3].text
+        rescue
+        end
+      end
+
+      def phone
+        begin
+          text = @document.css("td")[4].text
+          regex = /Ph:(.+?)$/
+          regex.match(text)[1].strip
+        rescue
+          nil
+        end
+      end
+
+      def email
+        begin
+          text = @document.css("td")[4].text
+          regex = /^(.+?@.+?.gov.au)/
+          regex.match(text)[1].strip
+        rescue
+          nil
+        end
+      end
+
+      def address
+        nil
+      end
+
+      def homepage_url
+        begin
+          @document.css("a")[0].attr("href")
+        rescue
+          nil
+        end
+      end
+
+      private
+
+        def test_document_for_validity
+          if @document && @document.css("a") && @document.css("td")
+            true
+          else
+            @logger.debug("Failed to find a valid document to scrape")
+            false
+          end
+        end
+
+    end # WebIndexRow
+  end # ACT
+end # MyRepresentatives

data/lib/my_representatives/act/web_lower.rb

@@ -0,0 +1,65 @@
+module MyRepresentatives
+  module ACT
+    class WebLower
+
+      attr_accessor :documents, :people
+
+      def initialize
+        @logger = Logger.new(STDOUT)
+        @documents = WebIndex.new.representative_documents
+        @people = []
+
+        @documents.each do |document|
+          row = WebIndexRow.new(document)
+          create_person_from_row(row)
+        end
+
+        @people.each do |person|
+          update_person(person)
+        end
+      end
+
+      private
+
+        def create_person_from_row(row)
+          @logger.info("Attempting to create #{row.first_name} #{row.last_name}")
+
+          # Setup the electorate
+          electorate_name = row.electorate_name
+          electorate      = MyRepresentatives::Electorate.new(electorate_name)
+          electorate.state_act_lower!
+
+          # Setup the Representative (Person)
+          person = MyRepresentatives::Person.new(electorate)
+          person.first_name                 = row.first_name
+          person.last_name                  = row.last_name
+          person.email                      = row.email
+          person.phone                      = row.phone
+          person.party_name                 = row.party_name
+          person.physical_address           = row.address
+          person.postal_address             = row.address
+          person.homepage_url               = row.homepage_url
+          person.honorifics                 = "MLA"
+          person.formal_name                = format_formal_name(person)
+          person.title                      = nil
+          person.gender                     = nil
+          person.preferred_name             = nil
+          person.salutation                 = nil
+          person.image_url                  = nil
+
+          @people << person
+
+        end
+
+        def update_person(person)
+          details_page = WebShow.new(person.homepage_url)
+          person.image_url = details_page.image_url
+        end
+
+        def format_formal_name(person)
+          "#{person.first_name} #{person.last_name} #{person.honorifics}"
+        end
+
+    end # WebLower
+  end # ACT
+end # MyRepresentatives

data/lib/my_representatives/act/web_show.rb

@@ -0,0 +1,46 @@
+module MyRepresentatives
+  module ACT
+    class WebShow
+
+      attr_accessor :url, :document, :logger
+
+      def initialize(url)
+        @logger = Logger.new(STDOUT)
+        raise MyRepresentatives::InvalidURLError unless url && url.is_a?(String)
+        @url = url
+        fetch_document
+        raise MyRepresentatives::NokogiriDocumentPropertiesError unless test_document_for_validity
+      end
+
+      def image_url
+        begin
+          @document.css("div.section").css("img")[0].attr('src')
+        rescue NoMethodError
+          nil
+        end
+      end
+
+      private
+
+        def fetch_document
+          begin
+            request   = open(@url)
+            @document = Nokogiri::HTML(request)
+          rescue => err
+            @logger.debug("Failed to connect to the url: #{@url}")
+            raise err
+          end
+        end
+
+        def test_document_for_validity
+          if @document && @document.css("div.section") && @document.css("div.section").css("img")
+            true
+          else
+            @logger.debug("Failed to find a valid document to scrape")
+            false
+          end
+        end
+
+    end # WebShow
+  end # ACT
+end # MyRepresentatives

data/lib/my_representatives/commonwealth/commonwealth.rb

@@ -0,0 +1,5 @@
+module MyRepresentatives
+  module Commonwealth
+    
+  end
+end

data/lib/my_representatives/commonwealth/csv_email.rb

@@ -0,0 +1,93 @@
+module MyRepresentatives
+  module Commonwealth
+    class CSVEmail
+      include MyRepresentatives::Abbreviatable
+      include MyRepresentatives::Fileable
+      include MyRepresentatives::Guessable
+
+      attr_accessor :csv_url, :csv_filename, :people, :logger
+
+      def initialize
+        @logger       = Logger.new(STDOUT)
+        # Email addresses for Commonwealth Members and Senators tend to be a little harder to find
+        # At this time, this gem uses a csv manually created by https://github.com/tennantje/.
+        # https://gist.github.com/tennantje/8ee584d9c534ce6a57910668b277394d
+        @csv_url      = "https://gist.githubusercontent.com/tennantje/8ee584d9c534ce6a57910668b277394d/raw"
+        @csv_filename = "tmp/csv_commonwealth_emails.csv"
+        @people       = []
+
+        csv_from_url
+        people_from_csv
+      end
+
+      private
+
+        def csv_from_url
+          create_tmp
+          begin
+            open(@csv_filename, "wb") do |file|
+              open(@csv_url) do |uri|
+                 file.write(uri.read)
+              end
+            end
+          rescue SocketError => err
+            @logger.debug("Unable to connect to #{@csv_url}")
+            raise err
+          end
+        end
+
+        def people_from_csv
+          CSV.foreach(@csv_filename, headers: true) do |row|
+
+            # Setup the electorate
+            electorate_name  = "Not necessary"                # These values don't matter, and are used to force-validate the object
+            electorate_state = "ACT"                          # These values don't matter, and are used to force-validate the object
+            electorate       = Electorate.new(electorate_name)# These values don't matter, and are used to force-validate the object
+            electorate.commonwealth_lower!(electorate_state)  # These values don't matter, and are used to force-validate the object
+
+            # Setup the Representative (Person)
+            person = Person.new(electorate)
+
+            person.first_name                 = find_first_name(row)
+            person.last_name                  = find_last_name(row)
+            person.formal_name                = find_formal_name(row)
+            person.phone                      = find_phone(row)
+            person.email                      = find_email(row)
+            person.title                      = nil
+            person.honorifics                 = nil
+            person.party_name                 = nil
+            person.salutation                 = nil
+            person.gender                     = nil
+            person.preferred_name             = nil
+            person.physical_address           = nil
+            person.postal_address             = nil
+            person.image_url                  = nil
+            person.homepage_url               = nil
+
+            @people << person
+          end
+        end
+
+        def find_first_name(arr)
+          arr["first"]
+        end
+
+        def find_last_name(arr)
+          arr["last"]
+        end
+
+        def find_formal_name(arr)
+          arr["formal_name"]
+        end
+
+        def find_email(arr)
+          arr["email"]
+        end
+
+        def find_phone(arr)
+          arr["phone"]
+        end
+
+    end #CSVLower
+  end # Commonwealth
+end # MyRepresentatives