statement 1.0.1 → 1.0.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (4) hide show
  1. data/README.md +2 -2
  2. data/lib/statement/feed.rb +8 -3
  3. data/lib/statement/version.rb +1 -1
  4. metadata +21 -21
data/README.md CHANGED
@@ -44,11 +44,11 @@ puts results.first
44
44
  {:source=>"http://blumenauer.house.gov/index.php?option=com_bca-rss-syndicator&feed_id=1", :url=>"http://blumenauer.house.gov/index.php?option=com_content&amp;view=article&amp;id=2203:blumenauer-qwe-need-a-national-system-that-speaks-to-the-transportation-challenges-of-todayq&amp;catid=66:2013-press-releases", :title=>"Blumenauer: &quot;We need a national system that speaks to the transportation challenges of ...", :date=>#<Date: 2013-04-24 ((2456407j,0s,0n),+0s,2299161j)>, :domain=>"blumenauer.house.gov"}
45
45
  ```
46
46
 
47
- If you have a batch of RSS URLs, you can pass them to Feed's `batch` class method, which will use Typhoeus to fetch them in parallel:
47
+ If you have a batch of RSS URLs, you can pass them to Feed's `batch` class method, which will use Typhoeus to fetch them in parallel and returns a two-element array of results and failed urls:
48
48
 
49
49
  ```ruby
50
50
  urls = ['http://aderholt.house.gov/common/rss//index.cfm?rss=20', 'http://andrews.house.gov/rss.xml', "http://alexander.house.gov/common/rss/?rss=24", "http://amash.house.gov/rss.xml"]
51
- results = Statement::Feed.batch(urls)
51
+ results, failures = Statement::Feed.batch(urls)
52
52
  ```
53
53
 
54
54
  The sites that require HTML scraping are detailed in individual methods, and can be called individually or in bulk:
data/lib/statement/feed.rb CHANGED
@@ -10,17 +10,22 @@ module Statement
10
10
 
11
11
  def self.batch(urls)
12
12
  results = []
13
+ failures = []
13
14
  hydra = Typhoeus::Hydra.new
14
15
  urls.each do |url|
15
16
  req = Typhoeus::Request.new(url)
16
17
  req.on_complete do |response|
17
- doc = Nokogiri::XML(response.body)
18
- results << parse_rss(doc, url)
18
+ if response.success?
19
+ doc = Nokogiri::XML(response.body)
20
+ results << parse_rss(doc, url)
21
+ else
22
+ failures << url
23
+ end
19
24
  end
20
25
  hydra.queue(req)
21
26
  end
22
27
  hydra.run
23
- results.flatten
28
+ [results.flatten, failures]
24
29
  end
25
30
 
26
31
  def self.open_rss(url)
data/lib/statement/version.rb CHANGED
@@ -1,3 +1,3 @@
1
1
  module Statement
2
- VERSION = "1.0.1"
2
+ VERSION = "1.0.2"
3
3
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: statement
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.1
4
+ version: 1.0.2
5
5
  prerelease:
6
6
  platform: ruby
7
7
  authors:
@@ -13,7 +13,7 @@ date: 2013-06-12 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: bundler
16
- requirement: &2156333660 !ruby/object:Gem::Requirement
16
+ requirement: &2160424300 !ruby/object:Gem::Requirement
17
17
  none: false
18
18
  requirements:
19
19
  - - ~>
@@ -21,10 +21,10 @@ dependencies:
21
21
  version: '1.3'
22
22
  type: :development
23
23
  prerelease: false
24
- version_requirements: *2156333660
24
+ version_requirements: *2160424300
25
25
  - !ruby/object:Gem::Dependency
26
26
  name: rake
27
- requirement: &2156333060 !ruby/object:Gem::Requirement
27
+ requirement: &2160422860 !ruby/object:Gem::Requirement
28
28
  none: false
29
29
  requirements:
30
30
  - - ! '>='
@@ -32,10 +32,10 @@ dependencies:
32
32
  version: '0'
33
33
  type: :development
34
34
  prerelease: false
35
- version_requirements: *2156333060
35
+ version_requirements: *2160422860
36
36
  - !ruby/object:Gem::Dependency
37
37
  name: webmock
38
- requirement: &2156332240 !ruby/object:Gem::Requirement
38
+ requirement: &2160421900 !ruby/object:Gem::Requirement
39
39
  none: false
40
40
  requirements:
41
41
  - - ! '>='
@@ -43,10 +43,10 @@ dependencies:
43
43
  version: '0'
44
44
  type: :development
45
45
  prerelease: false
46
- version_requirements: *2156332240
46
+ version_requirements: *2160421900
47
47
  - !ruby/object:Gem::Dependency
48
48
  name: minitest
49
- requirement: &2156331080 !ruby/object:Gem::Requirement
49
+ requirement: &2160421160 !ruby/object:Gem::Requirement
50
50
  none: false
51
51
  requirements:
52
52
  - - ! '>='
@@ -54,10 +54,10 @@ dependencies:
54
54
  version: '0'
55
55
  type: :development
56
56
  prerelease: false
57
- version_requirements: *2156331080
57
+ version_requirements: *2160421160
58
58
  - !ruby/object:Gem::Dependency
59
59
  name: american_date
60
- requirement: &2156330340 !ruby/object:Gem::Requirement
60
+ requirement: &2160420500 !ruby/object:Gem::Requirement
61
61
  none: false
62
62
  requirements:
63
63
  - - ! '>='
@@ -65,10 +65,10 @@ dependencies:
65
65
  version: '0'
66
66
  type: :runtime
67
67
  prerelease: false
68
- version_requirements: *2156330340
68
+ version_requirements: *2160420500
69
69
  - !ruby/object:Gem::Dependency
70
70
  name: nokogiri
71
- requirement: &2156329800 !ruby/object:Gem::Requirement
71
+ requirement: &2160419820 !ruby/object:Gem::Requirement
72
72
  none: false
73
73
  requirements:
74
74
  - - ! '>='
@@ -76,10 +76,10 @@ dependencies:
76
76
  version: '0'
77
77
  type: :runtime
78
78
  prerelease: false
79
- version_requirements: *2156329800
79
+ version_requirements: *2160419820
80
80
  - !ruby/object:Gem::Dependency
81
81
  name: koala
82
- requirement: &2156329180 !ruby/object:Gem::Requirement
82
+ requirement: &2160434240 !ruby/object:Gem::Requirement
83
83
  none: false
84
84
  requirements:
85
85
  - - ! '>='
@@ -87,10 +87,10 @@ dependencies:
87
87
  version: '0'
88
88
  type: :runtime
89
89
  prerelease: false
90
- version_requirements: *2156329180
90
+ version_requirements: *2160434240
91
91
  - !ruby/object:Gem::Dependency
92
92
  name: oj
93
- requirement: &2156328380 !ruby/object:Gem::Requirement
93
+ requirement: &2160433240 !ruby/object:Gem::Requirement
94
94
  none: false
95
95
  requirements:
96
96
  - - ! '>='
@@ -98,10 +98,10 @@ dependencies:
98
98
  version: '0'
99
99
  type: :runtime
100
100
  prerelease: false
101
- version_requirements: *2156328380
101
+ version_requirements: *2160433240
102
102
  - !ruby/object:Gem::Dependency
103
103
  name: twitter
104
- requirement: &2156344140 !ruby/object:Gem::Requirement
104
+ requirement: &2160432780 !ruby/object:Gem::Requirement
105
105
  none: false
106
106
  requirements:
107
107
  - - ! '>='
@@ -109,10 +109,10 @@ dependencies:
109
109
  version: '0'
110
110
  type: :runtime
111
111
  prerelease: false
112
- version_requirements: *2156344140
112
+ version_requirements: *2160432780
113
113
  - !ruby/object:Gem::Dependency
114
114
  name: typhoeus
115
- requirement: &2156343620 !ruby/object:Gem::Requirement
115
+ requirement: &2160432280 !ruby/object:Gem::Requirement
116
116
  none: false
117
117
  requirements:
118
118
  - - ! '>='
@@ -120,7 +120,7 @@ dependencies:
120
120
  version: '0'
121
121
  type: :runtime
122
122
  prerelease: false
123
- version_requirements: *2156343620
123
+ version_requirements: *2160432280
124
124
  description: Crawls congressional websites for press releases.
125
125
  email:
126
126
  - dwillis@gmail.com