em_aws 0.2.9 → 0.3.0.beta
Sign up to get free protection for your applications and to get access to all the features.
- data/Gemfile +1 -1
- data/Gemfile.lock +6 -11
- data/HISTORY.md +11 -0
- data/README.md +23 -18
- data/em_aws.gemspec +3 -2
- data/lib/aws/core/http/em_http_handler.rb +53 -53
- data/lib/em_aws.rb +0 -1
- data/lib/em_aws/version.rb +1 -1
- data/spec/em_http_handler_spec.rb +19 -45
- metadata +10 -9
data/Gemfile
CHANGED
data/Gemfile.lock
CHANGED
@@ -1,24 +1,24 @@
|
|
1
1
|
PATH
|
2
2
|
remote: .
|
3
3
|
specs:
|
4
|
-
em_aws (0.
|
5
|
-
aws-sdk (~> 1.
|
4
|
+
em_aws (0.3.0.beta)
|
5
|
+
aws-sdk (~> 1.9.3)
|
6
6
|
em-http-request
|
7
7
|
em-synchrony
|
8
8
|
hot_tub (~> 0.2.4)
|
9
9
|
|
10
10
|
GEM
|
11
|
-
remote:
|
11
|
+
remote: https://rubygems.org/
|
12
12
|
specs:
|
13
13
|
ZenTest (4.8.4)
|
14
14
|
addressable (2.3.4)
|
15
|
-
aws-sdk (1.
|
15
|
+
aws-sdk (1.9.3)
|
16
16
|
json (~> 1.4)
|
17
17
|
nokogiri (>= 1.4.4)
|
18
18
|
uuidtools (~> 2.1)
|
19
19
|
builder (3.2.0)
|
20
20
|
cookiejar (0.3.0)
|
21
|
-
diff-lcs (1.2.
|
21
|
+
diff-lcs (1.2.4)
|
22
22
|
em-http-request (1.0.3)
|
23
23
|
addressable (>= 2.2.3)
|
24
24
|
cookiejar
|
@@ -30,15 +30,11 @@ GEM
|
|
30
30
|
em-synchrony (1.0.3)
|
31
31
|
eventmachine (>= 1.0.0.beta.1)
|
32
32
|
eventmachine (1.0.3)
|
33
|
-
eventmachine (1.0.3-java)
|
34
33
|
eventmachine_httpserver (0.2.1)
|
35
34
|
hot_tub (0.2.5)
|
36
35
|
http_parser.rb (0.5.3)
|
37
|
-
http_parser.rb (0.5.3-java)
|
38
36
|
json (1.7.7)
|
39
|
-
json (1.7.7-java)
|
40
37
|
nokogiri (1.5.9)
|
41
|
-
nokogiri (1.5.9-java)
|
42
38
|
rspec (2.13.0)
|
43
39
|
rspec-core (~> 2.13.0)
|
44
40
|
rspec-expectations (~> 2.13.0)
|
@@ -47,10 +43,9 @@ GEM
|
|
47
43
|
rspec-expectations (2.13.0)
|
48
44
|
diff-lcs (>= 1.1.3, < 2.0)
|
49
45
|
rspec-mocks (2.13.1)
|
50
|
-
uuidtools (2.1.
|
46
|
+
uuidtools (2.1.4)
|
51
47
|
|
52
48
|
PLATFORMS
|
53
|
-
java
|
54
49
|
ruby
|
55
50
|
|
56
51
|
DEPENDENCIES
|
data/HISTORY.md
CHANGED
@@ -4,6 +4,17 @@ EmAws Changelog
|
|
4
4
|
HEAD
|
5
5
|
=======
|
6
6
|
|
7
|
+
0.3.0.beta
|
8
|
+
=======
|
9
|
+
|
10
|
+
- requires AWS-SDK 1.9.3 for thread saftey issues in 1.9.0-1.9.2
|
11
|
+
- refactors client API to expose EM-Http-Request client options directly
|
12
|
+
|
13
|
+
0.2.9
|
14
|
+
=======
|
15
|
+
|
16
|
+
- AWS-SDK 1.9+ breaks EmAws 0.2, require AWS-SDK <= 1.8.5
|
17
|
+
|
7
18
|
0.2.8
|
8
19
|
=======
|
9
20
|
|
data/README.md
CHANGED
@@ -1,13 +1,19 @@
|
|
1
1
|
# EmAws
|
2
2
|
An EM-Synchrony handler for Ruby [AWS-SDK-Ruby](https://github.com/aws/aws-sdk-ruby)
|
3
|
-
|
4
3
|
## Installation
|
5
4
|
|
6
5
|
em_aws is available through [Rubygems](https://rubygems.org/gems/em_aws) and can be installed via:
|
7
6
|
|
8
7
|
$ gem install em_aws
|
9
8
|
|
10
|
-
|
9
|
+
### Requirements
|
10
|
+
|
11
|
+
* EmAws 0.3.x requires AWS-SDK-Ruby ~> 1.9.3
|
12
|
+
* EmAws 0.2.x is available for those using AWS-SDK-Ruby <= 1.8.5
|
13
|
+
* AWS-SDK-Ruby 1.9.0 to 1.9.2 are not compatible with any version of EmAws; see [here](https://github.com/aws/aws-sdk-ruby/issues/237)
|
14
|
+
|
15
|
+
## Rails 3 setup
|
16
|
+
|
11
17
|
Setup [AWS-SDK-Ruby](https://github.com/aws/aws-sdk-ruby/blob/master/README.rdoc) as you would normally.
|
12
18
|
|
13
19
|
Assuming you've already setup async-rails, add em_aws to your gemfile:
|
@@ -18,24 +24,26 @@ Then run:
|
|
18
24
|
|
19
25
|
bundle install
|
20
26
|
|
21
|
-
|
27
|
+
Add the following to your aws.rb initializer:
|
22
28
|
|
23
29
|
require 'aws-sdk'
|
24
30
|
require 'aws/core/http/em_http_handler'
|
25
|
-
AWS.eager_autoload! # AWS lazyloading is not threadsafe
|
26
31
|
AWS.config(
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
32
|
+
:http_handler => AWS::Http::EMHttpHandler.new(
|
33
|
+
:proxy => { :host => '127.0.0.1', # proxy address
|
34
|
+
:port => 9000, # proxy port
|
35
|
+
:type => :socks5 },
|
36
|
+
:pool_size => 20, # Default is 0, set to > 0 to enable pooling
|
37
|
+
:async => false)) # If set to true all requests are handle asynchronously
|
38
|
+
# and initially return nil
|
31
39
|
|
32
40
|
Your done.
|
33
41
|
|
34
|
-
All requests to AWS will use EM-Synchrony's implementation of em-http-request for non-block HTTP requests and fiber management.
|
42
|
+
All requests to AWS will use EM-Synchrony's implementation of em-http-request for non-block HTTP requests and fiber management. See [EM-HTTP-Request](https://github.com/igrigorik/em-http-request/wiki/Issuing-Requests#available-connection--request-parameters) for all client options
|
35
43
|
|
36
44
|
## Connection Pooling (keep-alive)
|
37
|
-
|
38
|
-
to 0 which will leave the connection open for as long as the client allows. Connects
|
45
|
+
|
46
|
+
We use [HotTub](https://github.com/JoshMcKin/hot_tub) to manage connection pooling. To enable connection pooling set the :pool_size to anything greater than 0. By default :inactivity_timeout is set to 0 which will leave the connection open for as long as the client allows. Connects
|
39
47
|
are created lazy, so pools grow until they meet the set pool size.
|
40
48
|
|
41
49
|
require 'aws-sdk'
|
@@ -44,16 +52,13 @@ are created lazy, so pools grow until they meet the set pool size.
|
|
44
52
|
:http_handler => AWS::Http::EMHttpHandler.new({
|
45
53
|
:pool_size => 20,
|
46
54
|
:inactivity_timeout => 0, # number of seconds to timeout stale connections in the pool,
|
47
|
-
:never_block => true
|
48
|
-
:proxy => {:host => "http://myproxy.com",:port => 80})
|
55
|
+
:never_block => true) # if we run out of connections, create a new one
|
49
56
|
)
|
50
57
|
|
51
58
|
## Streaming
|
52
|
-
|
53
|
-
|
54
|
-
|
55
|
-
# You can pass an IO object in the :data option instead but the object must
|
56
|
-
# respond to :path. We cannot stream from memory at this time.
|
59
|
+
Streaming from disk,You can pass an IO object in the :data option instead but the object must
|
60
|
+
respond to :path. We cannot stream from memory at this time.
|
61
|
+
|
57
62
|
EM.synchrony do
|
58
63
|
s3 = AWS::S3.new
|
59
64
|
s3.buckets['bucket_name'].objects["foo.txt"].write(:file => "path_to_file")
|
data/em_aws.gemspec
CHANGED
@@ -8,13 +8,14 @@ Gem::Specification.new do |s|
|
|
8
8
|
s.platform = Gem::Platform::RUBY
|
9
9
|
s.authors = ["Joshua Mckinney"]
|
10
10
|
s.email = ["joshmckin@gmail.com"]
|
11
|
-
s.homepage = ""
|
11
|
+
s.homepage = "https://github.com/JoshMcKin/em_aws"
|
12
|
+
s.license = "MIT"
|
12
13
|
s.summary = %q{Adds EM-Synchrony support to AWS-SDK gem}
|
13
14
|
s.description = %q{Adds EM-Synchrony support to AWS-SDK gem}
|
14
15
|
|
15
16
|
s.rubyforge_project = "em_aws"
|
16
17
|
|
17
|
-
s.add_runtime_dependency "aws-sdk", "~> 1.
|
18
|
+
s.add_runtime_dependency "aws-sdk", "~> 1.9.3"
|
18
19
|
s.add_runtime_dependency "em-synchrony"
|
19
20
|
s.add_runtime_dependency "em-http-request"
|
20
21
|
s.add_runtime_dependency "hot_tub", "~> 0.2.4"
|
@@ -12,17 +12,23 @@ module AWS
|
|
12
12
|
# http://www.mikeperham.com/2010/04/03/introducing-phat-an-asynchronous-rails-app/
|
13
13
|
# for examples of Aync-Rails application
|
14
14
|
#
|
15
|
-
# In Rails add the following to
|
15
|
+
# In Rails add the following to your aws.rb initializer
|
16
16
|
#
|
17
17
|
# require 'aws-sdk'
|
18
18
|
# require 'aws/core/http/em_http_handler'
|
19
19
|
# AWS.config(
|
20
20
|
# :http_handler => AWS::Http::EMHttpHandler.new(
|
21
|
-
#
|
22
|
-
#
|
23
|
-
#
|
24
|
-
# :
|
25
|
-
#
|
21
|
+
# :proxy => {:host => '127.0.0.1', # proxy address
|
22
|
+
# :port => 9000, # proxy port
|
23
|
+
# :type => :socks5},
|
24
|
+
# :pool_size => 20, # Default is 0, set to > 0 to enable pooling
|
25
|
+
# :async => false)) # If set to true all requests are handle asynchronously
|
26
|
+
# # and initially return nil
|
27
|
+
#
|
28
|
+
# EM-AWS exposes all connections options for EM-Http-Request at initialization
|
29
|
+
# For more information on available options see https://github.com/igrigorik/em-http-request/wiki/Issuing-Requests#available-connection--request-parameters
|
30
|
+
# If Options from the request section of the above link are present, they
|
31
|
+
# set on every request but may be over written by the request object
|
26
32
|
class EMHttpHandler
|
27
33
|
|
28
34
|
EM_PASS_THROUGH_ERRORS = [
|
@@ -32,32 +38,22 @@ module AWS
|
|
32
38
|
LocalJumpError, SignalException, ScriptError,
|
33
39
|
SystemStackError, RegexpError, IndexError,
|
34
40
|
]
|
35
|
-
|
36
|
-
attr_reader :
|
41
|
+
|
42
|
+
attr_reader :default_options, :client_options, :pool_options, :pool
|
37
43
|
|
38
44
|
# Constructs a new HTTP handler using EM-Synchrony.
|
39
45
|
# @param [Hash] options Default options to send to EM-Synchrony on
|
40
46
|
# each request. These options will be sent to +get+, +post+,
|
41
47
|
# +head+, +put+, or +delete+ when a request is made. Note
|
42
48
|
# that +:body+, +:head+, +:parser+, and +:ssl_ca_file+ are
|
43
|
-
# ignored. If you need to set the CA file
|
44
|
-
#
|
45
|
-
# {AWS::Configuration} instead.
|
49
|
+
# ignored. If you need to set the CA file see:
|
50
|
+
# https://github.com/igrigorik/em-http-request/wiki/Issuing-Requests#available-connection--request-parameters
|
46
51
|
def initialize options = {}
|
47
|
-
@
|
48
|
-
@client_options =
|
49
|
-
|
50
|
-
|
51
|
-
|
52
|
-
@pool_options = {
|
53
|
-
:with_pool => true,
|
54
|
-
:size => ((options[:pool_size].to_i || 5)),
|
55
|
-
:never_block => (options[:never_block].nil? ? true : options[:never_block]),
|
56
|
-
:blocking_timeout => (options[:pool_timeout] || 10)
|
57
|
-
}
|
58
|
-
if @pool_options[:size] > 0
|
59
|
-
@pool = HotTub::Session.new(@pool_options) { |url| EM::HttpRequest.new(url,@client_options)}
|
60
|
-
end
|
52
|
+
@default_options = options
|
53
|
+
@client_options = fetch_client_options
|
54
|
+
@pool_options = fetch_pool_options
|
55
|
+
@pool = HotTub::Session.new(pool_options) { |url|
|
56
|
+
EM::HttpRequest.new(url,client_options)} if with_pool?
|
61
57
|
end
|
62
58
|
|
63
59
|
def handle(request,response,&read_block)
|
@@ -66,7 +62,7 @@ module AWS
|
|
66
62
|
else
|
67
63
|
EM.synchrony do
|
68
64
|
process_request(request,response,&read_block)
|
69
|
-
|
65
|
+
pool.close_all if pool
|
70
66
|
EM.stop
|
71
67
|
end
|
72
68
|
end
|
@@ -90,14 +86,38 @@ module AWS
|
|
90
86
|
else
|
91
87
|
EM.synchrony do
|
92
88
|
process_request(request,response,true,&read_block)
|
93
|
-
|
89
|
+
pool.close_all if @pool
|
94
90
|
EM.stop
|
95
91
|
end
|
96
92
|
end
|
97
93
|
end
|
98
94
|
|
95
|
+
def with_pool?
|
96
|
+
(default_options[:pool_size].to_i > 0)
|
97
|
+
end
|
98
|
+
|
99
99
|
private
|
100
100
|
|
101
|
+
def fetch_client_options
|
102
|
+
co = ({} || default_options.dup)
|
103
|
+
co.delete(:pool_size)
|
104
|
+
co.delete(:never_block)
|
105
|
+
co.delete(:blocking_timeout)
|
106
|
+
co[:inactivity_timeout] ||= 0
|
107
|
+
co[:connect_timeout] ||= 10
|
108
|
+
co[:keepalive] = true if with_pool?
|
109
|
+
co
|
110
|
+
end
|
111
|
+
|
112
|
+
def fetch_pool_options
|
113
|
+
{
|
114
|
+
:with_pool => true,
|
115
|
+
:size => ((default_options[:pool_size].to_i || 5)),
|
116
|
+
:never_block => (default_options[:never_block].nil? ? true : default_options[:never_block]),
|
117
|
+
:blocking_timeout => (default_options[:blocking_timeout] || 10)
|
118
|
+
}
|
119
|
+
end
|
120
|
+
|
101
121
|
def fetch_url(request)
|
102
122
|
"#{(request.use_ssl? ? "https" : "http")}://#{request.host}:#{request.port}"
|
103
123
|
end
|
@@ -110,29 +130,9 @@ module AWS
|
|
110
130
|
{:head => headers}
|
111
131
|
end
|
112
132
|
|
113
|
-
def fetch_proxy(request)
|
114
|
-
opts={}
|
115
|
-
if request.proxy_uri
|
116
|
-
opts[:proxy] = {:host => request.proxy_uri.host,:port => request.proxy_uri.port}
|
117
|
-
end
|
118
|
-
opts
|
119
|
-
end
|
120
|
-
|
121
|
-
def fetch_ssl(request)
|
122
|
-
opts = {}
|
123
|
-
if request.use_ssl? && request.ssl_verify_peer?
|
124
|
-
opts[:private_key_file] = request.ssl_ca_file
|
125
|
-
opts[:cert_chain_file]= request.ssl_ca_file
|
126
|
-
end
|
127
|
-
opts
|
128
|
-
end
|
129
|
-
|
130
133
|
def fetch_request_options(request)
|
131
|
-
opts =
|
132
|
-
|
133
|
-
merge(fetch_proxy(request)).
|
134
|
-
merge(fetch_ssl(request)))
|
135
|
-
opts[:query] = request.querystring
|
134
|
+
opts = client_options.merge(fetch_headers(request))
|
135
|
+
opts[:query] = request.querystring
|
136
136
|
if request.body_stream.respond_to?(:path)
|
137
137
|
opts[:file] = request.body_stream.path
|
138
138
|
else
|
@@ -145,14 +145,14 @@ module AWS
|
|
145
145
|
def fetch_response(request,opts={},&read_block)
|
146
146
|
method = "a#{request.http_method}".downcase.to_sym # aget, apost, aput, adelete, ahead
|
147
147
|
url = fetch_url(request)
|
148
|
-
if
|
149
|
-
|
150
|
-
req = connection.send(method,
|
148
|
+
if pool
|
149
|
+
pool.run(url) do |connection|
|
150
|
+
req = connection.send(method, opts)
|
151
151
|
req.stream &read_block if block_given?
|
152
152
|
return EM::Synchrony.sync req unless opts[:async]
|
153
153
|
end
|
154
154
|
else
|
155
|
-
clnt_opts =
|
155
|
+
clnt_opts = client_options.merge(:inactivity_timeout => request.read_timeout)
|
156
156
|
req = EM::HttpRequest.new(url,clnt_opts).send(method,opts)
|
157
157
|
req.stream &read_block if block_given?
|
158
158
|
return EM::Synchrony.sync req unless opts[:async]
|
data/lib/em_aws.rb
CHANGED
data/lib/em_aws/version.rb
CHANGED
@@ -137,54 +137,28 @@ module AWS::Core
|
|
137
137
|
opts[:file].should eql(my_io.path)
|
138
138
|
end
|
139
139
|
end
|
140
|
+
|
140
141
|
end
|
141
|
-
describe '#fetch_proxy' do
|
142
|
-
context ':proxy_uri' do
|
143
|
-
it 'passes proxy address and port from the request' do
|
144
|
-
req.proxy_uri = URI.parse('https://user:pass@proxy.com:443/path?query')
|
145
|
-
handler.send(:fetch_proxy,(req))[:proxy][:host].should == 'proxy.com'
|
146
|
-
handler.send(:fetch_proxy,(req))[:proxy][:port].should == 443
|
147
|
-
end
|
148
|
-
end
|
149
142
|
|
150
|
-
|
151
|
-
|
152
|
-
|
153
|
-
|
154
|
-
|
155
|
-
|
156
|
-
|
143
|
+
describe '#fetch_client_options' do
|
144
|
+
it "should remove pool related options" do
|
145
|
+
opts = handler.send(:fetch_client_options)
|
146
|
+
opts.has_key?(:size).should be_false
|
147
|
+
opts.has_key?(:never_block).should be_false
|
148
|
+
opts.has_key?(:blocking_timeout).should be_false
|
149
|
+
end
|
150
|
+
context "with_pool is true" do
|
151
|
+
it "should set keepalive as true" do
|
152
|
+
handler.stub(:with_pool?).and_return(true)
|
153
|
+
opts = handler.send(:fetch_client_options)
|
154
|
+
opts[:keepalive].should be_true
|
157
155
|
end
|
158
|
-
|
159
|
-
|
160
|
-
|
161
|
-
|
162
|
-
|
163
|
-
|
164
|
-
context 'ssl_verify_peer? is true' do
|
165
|
-
|
166
|
-
before(:each) do
|
167
|
-
req.ssl_verify_peer = true
|
168
|
-
req.ssl_ca_file = "foobar.txt"
|
169
|
-
end
|
170
|
-
|
171
|
-
it 'should use the ssl_ca_file attribute of the request' do
|
172
|
-
handler.send(:fetch_ssl,(req))[:private_key_file].should == "foobar.txt"
|
173
|
-
end
|
174
|
-
|
175
|
-
it 'should use the ssl_ca_file attribute of the request' do
|
176
|
-
handler.send(:fetch_ssl,(req))[:cert_chain_file].should == "foobar.txt"
|
177
|
-
end
|
178
|
-
end
|
179
|
-
|
180
|
-
it 'should not set the ssl_ca_file option without ssl_verify_peer?' do
|
181
|
-
handler.send(:fetch_ssl,(req)).should_not include(:private_key_file)
|
182
|
-
end
|
183
|
-
end
|
184
|
-
|
185
|
-
it 'should not set the ssl_ca_file option without use_ssl?' do
|
186
|
-
handler.send(:fetch_ssl,(req)).should_not include(:private_key_file)
|
187
|
-
end
|
156
|
+
end
|
157
|
+
context "with_pool is false" do
|
158
|
+
it "should keepalive be false" do
|
159
|
+
handler.stub(:with_pool?).and_return(false)
|
160
|
+
opts = handler.send(:fetch_client_options)
|
161
|
+
opts[:keepalive].should_not be_true
|
188
162
|
end
|
189
163
|
end
|
190
164
|
end
|
metadata
CHANGED
@@ -1,15 +1,15 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: em_aws
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
5
|
-
prerelease:
|
4
|
+
version: 0.3.0.beta
|
5
|
+
prerelease: 6
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
8
8
|
- Joshua Mckinney
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2013-04-
|
12
|
+
date: 2013-04-30 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: aws-sdk
|
@@ -18,7 +18,7 @@ dependencies:
|
|
18
18
|
requirements:
|
19
19
|
- - ~>
|
20
20
|
- !ruby/object:Gem::Version
|
21
|
-
version: 1.
|
21
|
+
version: 1.9.3
|
22
22
|
type: :runtime
|
23
23
|
prerelease: false
|
24
24
|
version_requirements: !ruby/object:Gem::Requirement
|
@@ -26,7 +26,7 @@ dependencies:
|
|
26
26
|
requirements:
|
27
27
|
- - ~>
|
28
28
|
- !ruby/object:Gem::Version
|
29
|
-
version: 1.
|
29
|
+
version: 1.9.3
|
30
30
|
- !ruby/object:Gem::Dependency
|
31
31
|
name: em-synchrony
|
32
32
|
requirement: !ruby/object:Gem::Requirement
|
@@ -130,8 +130,9 @@ files:
|
|
130
130
|
- spec/em_http_handler_spec.rb
|
131
131
|
- spec/patches_spec.rb
|
132
132
|
- spec/spec_helper.rb
|
133
|
-
homepage:
|
134
|
-
licenses:
|
133
|
+
homepage: https://github.com/JoshMcKin/em_aws
|
134
|
+
licenses:
|
135
|
+
- MIT
|
135
136
|
post_install_message:
|
136
137
|
rdoc_options: []
|
137
138
|
require_paths:
|
@@ -145,9 +146,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
145
146
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
146
147
|
none: false
|
147
148
|
requirements:
|
148
|
-
- - ! '
|
149
|
+
- - ! '>'
|
149
150
|
- !ruby/object:Gem::Version
|
150
|
-
version:
|
151
|
+
version: 1.3.1
|
151
152
|
requirements: []
|
152
153
|
rubyforge_project: em_aws
|
153
154
|
rubygems_version: 1.8.25
|