statement 1.0.1 → 1.0.2

Sign up to get free protection for your applications and to get access to all the features.
Files changed (4) hide show
  1. data/README.md +2 -2
  2. data/lib/statement/feed.rb +8 -3
  3. data/lib/statement/version.rb +1 -1
  4. metadata +21 -21
data/README.md CHANGED
@@ -44,11 +44,11 @@ puts results.first
44
44
  {:source=>"http://blumenauer.house.gov/index.php?option=com_bca-rss-syndicator&feed_id=1", :url=>"http://blumenauer.house.gov/index.php?option=com_content&amp;view=article&amp;id=2203:blumenauer-qwe-need-a-national-system-that-speaks-to-the-transportation-challenges-of-todayq&amp;catid=66:2013-press-releases", :title=>"Blumenauer: &quot;We need a national system that speaks to the transportation challenges of ...", :date=>#<Date: 2013-04-24 ((2456407j,0s,0n),+0s,2299161j)>, :domain=>"blumenauer.house.gov"}
45
45
  ```
46
46
 
47
- If you have a batch of RSS URLs, you can pass them to Feed's `batch` class method, which will use Typhoeus to fetch them in parallel:
47
+ If you have a batch of RSS URLs, you can pass them to Feed's `batch` class method, which will use Typhoeus to fetch them in parallel and returns a two-element array of results and failed urls:
48
48
 
49
49
  ```ruby
50
50
  urls = ['http://aderholt.house.gov/common/rss//index.cfm?rss=20', 'http://andrews.house.gov/rss.xml', "http://alexander.house.gov/common/rss/?rss=24", "http://amash.house.gov/rss.xml"]
51
- results = Statement::Feed.batch(urls)
51
+ results, failures = Statement::Feed.batch(urls)
52
52
  ```
53
53
 
54
54
  The sites that require HTML scraping are detailed in individual methods, and can be called individually or in bulk:
data/lib/statement/feed.rb CHANGED
@@ -10,17 +10,22 @@ module Statement
10
10
 
11
11
  def self.batch(urls)
12
12
  results = []
13
+ failures = []
13
14
  hydra = Typhoeus::Hydra.new
14
15
  urls.each do |url|
15
16
  req = Typhoeus::Request.new(url)
16
17
  req.on_complete do |response|
17
- doc = Nokogiri::XML(response.body)
18
- results << parse_rss(doc, url)
18
+ if response.success?
19
+ doc = Nokogiri::XML(response.body)
20
+ results << parse_rss(doc, url)
21
+ else
22
+ failures << url
23
+ end
19
24
  end
20
25
  hydra.queue(req)
21
26
  end
22
27
  hydra.run
23
- results.flatten
28
+ [results.flatten, failures]
24
29
  end
25
30
 
26
31
  def self.open_rss(url)
data/lib/statement/version.rb CHANGED
@@ -1,3 +1,3 @@
1
1
  module Statement
2
- VERSION = "1.0.1"
2
+ VERSION = "1.0.2"
3
3
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: statement
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.1
4
+ version: 1.0.2
5
5
  prerelease:
6
6
  platform: ruby
7
7
  authors:
@@ -13,7 +13,7 @@ date: 2013-06-12 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: bundler
16
- requirement: &2156333660 !ruby/object:Gem::Requirement
16
+ requirement: &2160424300 !ruby/object:Gem::Requirement
17
17
  none: false
18
18
  requirements:
19
19
  - - ~>
@@ -21,10 +21,10 @@ dependencies:
21
21
  version: '1.3'
22
22
  type: :development
23
23
  prerelease: false
24
- version_requirements: *2156333660
24
+ version_requirements: *2160424300
25
25
  - !ruby/object:Gem::Dependency
26
26
  name: rake
27
- requirement: &2156333060 !ruby/object:Gem::Requirement
27
+ requirement: &2160422860 !ruby/object:Gem::Requirement
28
28
  none: false
29
29
  requirements:
30
30
  - - ! '>='
@@ -32,10 +32,10 @@ dependencies:
32
32
  version: '0'
33
33
  type: :development
34
34
  prerelease: false
35
- version_requirements: *2156333060
35
+ version_requirements: *2160422860
36
36
  - !ruby/object:Gem::Dependency
37
37
  name: webmock
38
- requirement: &2156332240 !ruby/object:Gem::Requirement
38
+ requirement: &2160421900 !ruby/object:Gem::Requirement
39
39
  none: false
40
40
  requirements:
41
41
  - - ! '>='
@@ -43,10 +43,10 @@ dependencies:
43
43
  version: '0'
44
44
  type: :development
45
45
  prerelease: false
46
- version_requirements: *2156332240
46
+ version_requirements: *2160421900
47
47
  - !ruby/object:Gem::Dependency
48
48
  name: minitest
49
- requirement: &2156331080 !ruby/object:Gem::Requirement
49
+ requirement: &2160421160 !ruby/object:Gem::Requirement
50
50
  none: false
51
51
  requirements:
52
52
  - - ! '>='
@@ -54,10 +54,10 @@ dependencies:
54
54
  version: '0'
55
55
  type: :development
56
56
  prerelease: false
57
- version_requirements: *2156331080
57
+ version_requirements: *2160421160
58
58
  - !ruby/object:Gem::Dependency
59
59
  name: american_date
60
- requirement: &2156330340 !ruby/object:Gem::Requirement
60
+ requirement: &2160420500 !ruby/object:Gem::Requirement
61
61
  none: false
62
62
  requirements:
63
63
  - - ! '>='
@@ -65,10 +65,10 @@ dependencies:
65
65
  version: '0'
66
66
  type: :runtime
67
67
  prerelease: false
68
- version_requirements: *2156330340
68
+ version_requirements: *2160420500
69
69
  - !ruby/object:Gem::Dependency
70
70
  name: nokogiri
71
- requirement: &2156329800 !ruby/object:Gem::Requirement
71
+ requirement: &2160419820 !ruby/object:Gem::Requirement
72
72
  none: false
73
73
  requirements:
74
74
  - - ! '>='
@@ -76,10 +76,10 @@ dependencies:
76
76
  version: '0'
77
77
  type: :runtime
78
78
  prerelease: false
79
- version_requirements: *2156329800
79
+ version_requirements: *2160419820
80
80
  - !ruby/object:Gem::Dependency
81
81
  name: koala
82
- requirement: &2156329180 !ruby/object:Gem::Requirement
82
+ requirement: &2160434240 !ruby/object:Gem::Requirement
83
83
  none: false
84
84
  requirements:
85
85
  - - ! '>='
@@ -87,10 +87,10 @@ dependencies:
87
87
  version: '0'
88
88
  type: :runtime
89
89
  prerelease: false
90
- version_requirements: *2156329180
90
+ version_requirements: *2160434240
91
91
  - !ruby/object:Gem::Dependency
92
92
  name: oj
93
- requirement: &2156328380 !ruby/object:Gem::Requirement
93
+ requirement: &2160433240 !ruby/object:Gem::Requirement
94
94
  none: false
95
95
  requirements:
96
96
  - - ! '>='
@@ -98,10 +98,10 @@ dependencies:
98
98
  version: '0'
99
99
  type: :runtime
100
100
  prerelease: false
101
- version_requirements: *2156328380
101
+ version_requirements: *2160433240
102
102
  - !ruby/object:Gem::Dependency
103
103
  name: twitter
104
- requirement: &2156344140 !ruby/object:Gem::Requirement
104
+ requirement: &2160432780 !ruby/object:Gem::Requirement
105
105
  none: false
106
106
  requirements:
107
107
  - - ! '>='
@@ -109,10 +109,10 @@ dependencies:
109
109
  version: '0'
110
110
  type: :runtime
111
111
  prerelease: false
112
- version_requirements: *2156344140
112
+ version_requirements: *2160432780
113
113
  - !ruby/object:Gem::Dependency
114
114
  name: typhoeus
115
- requirement: &2156343620 !ruby/object:Gem::Requirement
115
+ requirement: &2160432280 !ruby/object:Gem::Requirement
116
116
  none: false
117
117
  requirements:
118
118
  - - ! '>='
@@ -120,7 +120,7 @@ dependencies:
120
120
  version: '0'
121
121
  type: :runtime
122
122
  prerelease: false
123
- version_requirements: *2156343620
123
+ version_requirements: *2160432280
124
124
  description: Crawls congressional websites for press releases.
125
125
  email:
126
126
  - dwillis@gmail.com