hulse 0.5 → 0.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 904848309e824d9e0c4ded85597ba3f0e7409d46
-  data.tar.gz: 9dc85ec003bf1b9c1c5cd3b8602150413109b283
+  metadata.gz: 24805bee84bd7c6a90544f7d70c98c0eb32661c2
+  data.tar.gz: f8e91bcffaca717a42d71e1f5bd1903fa52da684
 SHA512:
-  metadata.gz: 8277fdc3417493c205504d907a17b68aa4d5d9dcb5cd8a7898e4f21db786912450814749218468ee997e1dffde0d3f97eed145ffba2edf89dff804dd22862f0c
-  data.tar.gz: 97c18eccfd0850e1b116223e132ee5b066881b1b472e250489baa59d48b80ca30df3d6ebef679685a19adff93babfebdb9b4ec973941a8427adf5f791827137d
+  metadata.gz: 3171a1108d9677464ff0ddd67f3780ae37128dd0118f68efba14215f1033647ebd92fd19d6da0fbf7d7e14a14dce7649ddd17a48f702638f6770a393e06465eb
+  data.tar.gz: c7739d66a725f05131c0d4e696a6647bd0c2b535eb453a9ef06b52a53e26459516673e1a420b097936aed8da074bbdfaa5acdcad1187151b8074255d7c6e9e34

data/README.md

@@ -1,6 +1,8 @@
 # Hulse
 
-Hulse is a Ruby gem for accessing House and Senate roll call votes from the official sources on [house.gov](http://clerk.house.gov/evs/2013/index.asp) and [senate.gov](http://www.senate.gov/pagelayout/legislative/a_three_sections_with_teasers/votes.htm). It works using Ruby 1.9.3 and 2.x.
+Hulse is a Ruby gem for accessing raw data and information published by and about the U.S. Congress. It works using Ruby 1.9.3, 2.0, 2.1 and 2.2. It is not a wrapper for existing non-governmental APIs of congressional information. Instead, it loads, parses and scrapes official sources.
+
+Hulse can be used to get House and Senate roll call votes from the official sources on [house.gov](http://clerk.house.gov/evs/2013/index.asp) and [senate.gov](http://www.senate.gov/pagelayout/legislative/a_three_sections_with_teasers/votes.htm).
 
 Hulse has two vote classes, `HouseVote` and `SenateVote`, which create Ruby objects using the XML attributes available from roll call vote data (voice votes are not covered by Hulse or available as data from official sources). Hulse makes a few changes, renaming some attributes for clarity and consistency, and collapsing each House vote's date and time into a single datetime attribute. Otherwise it does not alter the original data.
 
@@ -8,6 +10,8 @@ Hulse has two vote classes, `HouseVote` and `SenateVote`, which create Ruby obje
 
 Hulse also has two member classes, `HouseMember` and `SenateMember`, which create Ruby objects using the XML made available by the Clerk of the House and the Secretary of the Senate. House members have some basic information, including the unique `bioguide_id`, along with office details and committee and subcommittee assignment data. For vacant seats, information on the seat's previous occupant is available. Senate members have less information, but their data includes the Senate class and the URLs of their websites and email forms.
 
+Hulse has two other classes, `HouseFloor` and `Record`. The former provides a wrapper to [XML data on floor activity](http://clerk.house.gov/floorsummary/floor.aspx?day=20150729) published by the Clerk of the House, including timestamps and descriptions. The `Record` class provides a basic wrapper to the [Congressional Record](https://www.congress.gov/congressional-record), the daily listing of activities by the House and Senate, as well as some methods for accessing specific portions of it, particularly the titles and permalinks of articles.
+
 Hulse is named for [Carl Hulse](https://www.nytimes.com/learning/students/ask_reporters/Carl_Hulse.html), a longtime congressional correspondent for The New York Times.
 
 ## Installation
@@ -47,6 +51,18 @@ For Senate votes, you can grab a year's SenateVote objects (with a limited set o
 2.0.0p353 :007 > senate_2013.first
 => <Hulse::SenateVote:0x000001017f0d58 @congress=113, @session=1, @year=2013, @vote_number="00291", @vote_date=<Date: 2013-12-20 ((2456647j,0s,0n),+0s,2299161j)>, @issue="PN921", @question="On the Cloture Motion", @vote_result="Agreed to", @vote_count={:yeas=>"59", :nays=>"34"}, @vote_title="Motion to Invoke Cloture on the Nomination of Janet L. Yellen to be Chairman of the Board of Governors of the Federal Reserve System">
 ```
+House and Senate members have a `current` class method that retrieves the latest XML data from the House and Senate websites and creates Ruby objects. The House file has more data, including vacancies, than the Senate file does.
+
+```ruby
+irb(main):003:> require 'hulse'
+irb(main):003:0> members = Hulse::HouseMember.current
+irb(main):004:0> members.first
+=> <Hulse::HouseMember:0x007fc6cb37f020 @bioguide_id="Y000033", @sort_name="YOUNG,DON", @last_name="Young", @first_name="Don", @middle_name=nil, @suffix=nil, @courtesy="Mr.", @official_name="Don Young", @formal_name="Mr. Young of Alaska", @party="R", @caucus_party="R", @state_postal="AK", @state_name="Alaska", @district="At Large", @district_code="AK00", @hometown="Fort Yukon", @office_building="RHOB", @office_room="2314", @office_zip="20515-0200", @phone="(202) 225-5765", @last_elected_date=#<Date: 2014-11-04 ((2456966j,0s,0n),+0s,2299161j)>, @sworn_date=#<Date: 2015-01-12 ((2457035j,0s,0n),+0s,2299161j)>, @committees=[{"comcode"=>"II00", "rank"=>"2"}, {"comcode"=>"PW00", "rank"=>"2"}], @subcommittees=[{"subcomcode"=>"II10", "rank"=>"2"}, {"subcomcode"=>"II13", "rank"=>"2"}, {"subcomcode"=>"II24", "rank"=>"1", "leadership"=>"Chairman"}, {"subcomcode"=>"PW05", "rank"=>"2"}, {"subcomcode"=>"PW07", "rank"=>"2"}, {"subcomcode"=>"PW12", "rank"=>"2"}], @is_vacant=false, @footnote=nil, @predecessor=nil, @vacancy_date=nil>
+```
+
+## Tests
+
+Hulse uses `MiniTest` for development. To run the tests, do `rake test`.
 
 ## Authors
 

data/hulse.gemspec

@@ -22,6 +22,7 @@ Gem::Specification.new do |spec|
   spec.add_development_dependency "rake"
   spec.add_development_dependency "minitest"
   spec.add_dependency "httparty"
+  spec.add_dependency "nokogiri"
   spec.add_dependency "oj"
   spec.add_dependency "american_date"
 end

data/lib/hulse/house_floor.rb

@@ -0,0 +1,40 @@
+module Hulse
+  class HouseFloor
+
+    attr_reader :legislative_day, :finished, :actions, :congress, :session, :url, :next_session
+
+    def initialize(params={})
+      params.each_pair do |k,v|
+        instance_variable_set("@#{k}", v)
+      end
+    end
+
+    def self.latest_dates
+      url = "http://clerk.house.gov/floorsummary/floor-rss.ashx"
+      response = HTTParty.get(url)
+      xml = response.parsed_response
+      xml['rss']['channel']['item'].map{|i| Date.parse(i['pubDate']).to_s}.uniq
+    end
+
+    def self.date(date)
+      url = "http://clerk.house.gov/floorsummary/Download.aspx?file=#{date.to_s.gsub('-','')}.xml"
+      response = HTTParty.get(url)
+      xml = HTTParty::Parser.call(response.parsed_response, :xml)
+      self.create_from_xml(xml)
+    end
+
+    def self.create_from_xml(xml)
+      actions = []
+      xml['legislative_activity']['floor_actions']['floor_action'].each do |action|
+        actions << action.inject({}){|memo,(k,v)| memo[k.to_sym] = v; memo}
+      end
+      results = self.new(congress: xml['legislative_activity']['legislative_congress']['congress'].to_i,
+      session: xml['legislative_activity']['legislative_congress']['session'].to_i,
+      legislative_day: xml['legislative_activity']['legislative_day']['__content__'].strip,
+      finished: xml['legislative_activity']['floor_actions']['legislative_day_finished']['__content__'],
+      url: "http://clerk.house.gov/floorsummary/floor.aspx?day=#{xml['legislative_activity']['legislative_day']['date']}",
+      next_session: DateTime.parse(xml['legislative_activity']['floor_actions']['legislative_day_finished']['next_legislative_day_convenes']),
+      actions: actions)
+    end
+  end
+end

data/lib/hulse/record.rb

@@ -0,0 +1,104 @@
+module Hulse
+  class Record
+
+    attr_reader :date, :section, :topics, :html
+
+    def initialize(params={})
+      params.each_pair do |k,v|
+        instance_variable_set("@#{k}", v)
+      end
+    end
+
+    def to_s
+      section
+    end
+
+    def self.base_url(date=nil)
+      if date
+        year, month, day = date.year, date.month, date.day
+      else
+        date = Date.today-1
+        year, month, day = date.year, date.month, date.day
+      end
+      "https://www.congress.gov/congressional-record/#{year}/#{month}/#{day}/"
+    end
+
+    def self.daily_digest(date=nil)
+      doc = HTTParty.get(base_url(date)+'daily-digest')
+      html = Nokogiri::HTML(doc.parsed_response)
+      (html/:pre).text
+    end
+
+    def self.senate(date=nil)
+      doc = HTTParty.get(base_url(date)+'senate-section')
+      create_from_html(Nokogiri::HTML(doc.parsed_response), date, 'senate')
+    end
+
+    def self.house(date=nil)
+      doc = HTTParty.get(base_url(date)+'house-section')
+      create_from_html(Nokogiri::HTML(doc.parsed_response), date, 'house')
+    end
+
+    def self.extension_of_remarks(date=nil)
+      doc = HTTParty.get(base_url(date)+'extensions-of-remarks-section')
+      create_from_html(Nokogiri::HTML(doc.parsed_response), date, 'extension')
+    end
+
+    def self.create_from_html(html, date, section)
+      section_topics = topics(html)
+      self.new(date: date,
+        section: section,
+        topics: section_topics,
+        html: html
+      )
+    end
+
+    def self.topics(html)
+      (html/:td).map{|d| d.children[1]}.compact.map{|l| {url: l['href'], title: l.text.strip}}
+    end
+
+    def has_senate_explanations?
+      topics.select{|l| l[:title].include?("VOTE EXPLANATION")}.empty? ? false : true
+    end
+
+    def senate_explanations
+      topics.select{|l| l[:title].include?("VOTE EXPLANATION")}
+    end
+
+    def has_personal_explanations?
+      topics.select{|l| l[:title].include?("PERSONAL EXPLANATION")}.empty? ? false : true
+    end
+
+    def personal_explanations
+      topics.select{|l| l[:title].include?("PERSONAL EXPLANATION")}
+    end
+
+    def has_committee_elections?
+      topics.select{|l| l[:title].include?("COMMITTEE ELECTION")}.empty? ? false : true
+    end
+
+    def committee_elections
+      topics.select{|l| l[:title].include?("COMMITTEE ELECTION")}
+    end
+
+    def has_committee_resignations?
+      topics.select{|l| l[:title].include?("COMMITTEE RESIGNATION")}.empty? ? false : true
+    end
+
+    def chairman_designations
+      topics.select{|l| l[:title].include?("DESIGNATING THE CHAIRMAN")}
+    end
+
+    def ranking_designations
+      topics.select{|l| l[:title].include?("DESIGNATING THE RANKING")}
+    end
+
+    def leaves_of_absence
+      topics.select{|l| l[:title].include?("LEAVE OF ABSENCE")}
+    end
+
+    def committee_leaves_of_absence
+      topics.select{|l| l[:title].include?("COMMITTEE LEAVE OF ABSENCE")}
+    end
+  end
+end

data/lib/hulse/version.rb

@@ -1,3 +1,3 @@
 module Hulse
-  VERSION = "0.5"
+  VERSION = "0.6"
 end

data/lib/hulse.rb

@@ -1,8 +1,11 @@
 require 'httparty'
 require 'oj'
+require 'nokogiri'
 require "hulse/version"
+require "hulse/record"
 require "hulse/house_vote"
 require "hulse/house_member"
+require "hulse/house_floor"
 require "hulse/senate_vote"
 require "hulse/senate_member"
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: hulse
 version: !ruby/object:Gem::Version
-  version: '0.5'
+  version: '0.6'
 platform: ruby
 authors:
 - Derek Willis
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-04-23 00:00:00.000000000 Z
+date: 2015-08-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -66,6 +66,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: nokogiri
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: oj
   requirement: !ruby/object:Gem::Requirement
@@ -109,8 +123,10 @@ files:
 - Rakefile
 - hulse.gemspec
 - lib/hulse.rb
+- lib/hulse/house_floor.rb
 - lib/hulse/house_member.rb
 - lib/hulse/house_vote.rb
+- lib/hulse/record.rb
 - lib/hulse/senate_member.rb
 - lib/hulse/senate_vote.rb
 - lib/hulse/version.rb
@@ -136,7 +152,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.2.2
+rubygems_version: 2.4.5
 signing_key: 
 specification_version: 4
 summary: Hulse is a Ruby gem for accessing House and Senate roll call votes and member