nsq-ruby-maglev- 1.2.1.0

Sign up to get free protection for your applications and to get access to all the features.
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 404388ace1303e56c07f6b18284a81abe2cd03e8
4
+ data.tar.gz: 77946db3d3f9d10a2d515ff75ac3ca48888fbc55
5
+ SHA512:
6
+ metadata.gz: 106aa12520e1d473c57ff760eff08267c5bd1294a0586dc06ee607d513722a0e9651435eea6a2ef4403d72f70bfd50e412909e4bfc93740e58e756d89c490132
7
+ data.tar.gz: 24d46489acd7a824001d0a6720e0b9bedc1b259e27a974fbe7e5c8b33b83259fc3555aea1ae1749785e31ac6def0d7eb543b3a9cd8e018055c73cbc9d816a98e
@@ -0,0 +1,22 @@
1
+ Copyright (c) 2014 Wistia, Inc.
2
+
3
+ MIT License
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining
6
+ a copy of this software and associated documentation files (the
7
+ "Software"), to deal in the Software without restriction, including
8
+ without limitation the rights to use, copy, modify, merge, publish,
9
+ distribute, sublicense, and/or sell copies of the Software, and to
10
+ permit persons to whom the Software is furnished to do so, subject to
11
+ the following conditions:
12
+
13
+ The above copyright notice and this permission notice shall be
14
+ included in all copies or substantial portions of the Software.
15
+
16
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
17
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
18
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
19
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
20
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
21
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
22
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,311 @@
1
+ # nsq-ruby
2
+
3
+ nsq-ruby is a simple NSQ client library written in Ruby.
4
+
5
+ - The code is straightforward.
6
+ - It has no dependencies.
7
+ - It's well tested.
8
+ - It's being used in production and has processed billions of messages.
9
+
10
+
11
+ ## Quick start
12
+
13
+ ### Publish messages
14
+
15
+ ```Ruby
16
+ require 'nsq'
17
+ producer = Nsq::Producer.new(
18
+ nsqd: '127.0.0.1:4150',
19
+ topic: 'some-topic'
20
+ )
21
+
22
+ # Write a message to NSQ
23
+ producer.write('some-message')
24
+
25
+ # Write a bunch of messages to NSQ (uses mpub)
26
+ producer.write('one', 'two', 'three', 'four', 'five')
27
+
28
+ # Close the connection
29
+ producer.terminate
30
+ ```
31
+
32
+ ### Consume messages
33
+
34
+ ```Ruby
35
+ require 'nsq'
36
+ consumer = Nsq::Consumer.new(
37
+ nsqlookupd: '127.0.0.1:4161',
38
+ topic: 'some-topic',
39
+ channel: 'some-channel'
40
+ )
41
+
42
+ # Pop a message off the queue
43
+ msg = consumer.pop
44
+ puts msg.body
45
+ msg.finish
46
+
47
+ # Close the connections
48
+ consumer.terminate
49
+ ```
50
+
51
+
52
+ ## Producer
53
+
54
+ ### Initialization
55
+
56
+ The Nsq::Producer constructor takes the following options:
57
+
58
+ | Option | Description | Default |
59
+ |---------------|----------------------------------------|--------------------|
60
+ | `topic` | Topic to which to publish messages | |
61
+ | `nsqd` | Host and port of the nsqd instance | '127.0.0.1:4150' |
62
+ | `nsqlookupd` | Use lookupd to automatically discover nsqds | |
63
+
64
+ For example, if you'd like to publish messages to a single nsqd.
65
+
66
+ ```Ruby
67
+ producer = Nsq::Producer.new(
68
+ nsqd: '6.7.8.9:4150',
69
+ topic: 'topic-of-great-esteem'
70
+ )
71
+ ```
72
+
73
+ Alternatively, you can use nsqlookupd to find all nsqd nodes in the cluster.
74
+ When you instantiate Nsq::Producer in this way, it will automatically maintain
75
+ connections to all nsqd instances. When you publish a message, it will be sent
76
+ to a random nsqd instance.
77
+
78
+ ```Ruby
79
+ producer = Nsq::Producer.new(
80
+ nsqlookupd: ['1.2.3.4:4161', '6.7.8.9:4161'],
81
+ topic: 'topic-of-great-esteem'
82
+ )
83
+ ```
84
+
85
+
86
+ ### `#write`
87
+
88
+ Publishes one or more message to nsqd. If you give it a single argument, it will
89
+ send it to nsqd via `PUB`. If you give it multiple arguments, it will send all
90
+ those messages to nsqd via `MPUB`. It will automatically call `to_s` on any
91
+ arguments you give it.
92
+
93
+ ```Ruby
94
+ # Send a single message via PUB
95
+ producer.write(123)
96
+
97
+ # Send three messages via MPUB
98
+ producer.write(456, 'another-message', { key: 'value' }.to_json)
99
+ ```
100
+
101
+ If it's connection to nsqd fails, it will automatically try to reconnect with
102
+ exponential backoff. Any messages that were sent to `#write` will be queued
103
+ and transmitted after reconnecting.
104
+
105
+ **Note** we don't wait for nsqd to acknowledge our writes. As a result, if the
106
+ connection to nsqd fails, you can lose messages. This is acceptable for our use
107
+ cases, mostly because we are sending messages to a local nsqd instance and
108
+ failure is very rare.
109
+
110
+ ### `#connected?`
111
+
112
+ Returns true if it's currently connected to nsqd and false if not.
113
+
114
+ ### `#terminate`
115
+
116
+ Closes the connection to nsqd and stops it from trying to automatically
117
+ reconnect.
118
+
119
+ This is automatically called `at_exit`, but it's good practice to close your
120
+ producers when you're done with them.
121
+
122
+
123
+ ## Consumer
124
+
125
+ ### Initialization
126
+
127
+ | Option | Description | Default |
128
+ |----------------------|-----------------------------------------------|--------------------|
129
+ | `topic` | Topic to consume messages from | |
130
+ | `channel` | Channel name for this consumer | |
131
+ | `nsqlookupd` | Use lookupd to automatically discover nsqds | |
132
+ | `nsqd` | Connect directly to a single nsqd instance | '127.0.0.1:4150' |
133
+ | `max_in_flight` | Max number of messages for this consumer to have in flight at a time | 1 |
134
+ | `discovery_interval` | Seconds between queue discovery via nsqlookupd | 60.0 |
135
+ | `msg_timeout` | Milliseconds before nsqd will timeout a message | 60000 |
136
+
137
+
138
+ For example:
139
+
140
+ ```Ruby
141
+ consumer = Nsq::Consumer.new(
142
+ topic: 'the-topic',
143
+ channel: 'my-channel',
144
+ nsqlookupd: ['127.0.0.1:4161', '4.5.6.7:4161'],
145
+ max_in_flight: 100,
146
+ discovery_interval: 30,
147
+ msq_timeout: 120_000
148
+ )
149
+ ```
150
+
151
+ Notes:
152
+
153
+ - `nsqlookupd` can be a string or array of strings for each nsqlookupd service
154
+ you'd like to use. The format is `"<host>:<http-port>"`. If you specify
155
+ `nsqlookupd`, it ignores the `nsqd` option.
156
+ - `max_in_flight` is for the total max in flight across all the connections,
157
+ but to make the implementation of `nsq-ruby` as simple as possible, the minimum
158
+ `max_in_flight` _per_ connection is 1. So if you set `max_in_flight` to 1 and
159
+ are connected to 3 nsqds, you may have up to 3 messages in flight at a time.
160
+
161
+
162
+ ### `#pop`
163
+
164
+ `nsq-ruby` works by maintaining a local queue of in flight messages from NSQ.
165
+ To get at these messages, just call pop.
166
+
167
+ ```Ruby
168
+ message = consumer.pop
169
+ ```
170
+
171
+ If there are messages on the queue, `pop` will return one immediately. If there
172
+ are no messages on the queue, `pop` will block execution until one arrives.
173
+
174
+
175
+ ### `#size`
176
+
177
+ `size` returns the size of the local message queue.
178
+
179
+
180
+ ### `#terminate`
181
+
182
+ Gracefully closes all connections and stops the consumer. You should call this
183
+ when you're finished with a consumer object.
184
+
185
+
186
+ ## Message
187
+
188
+ The `Message` object is what you get when you call `pop` on a consumer.
189
+ Once you have a message, you'll likely want to get its contents using the `#body`
190
+ method, and then call `#finish` once you're done with it.
191
+
192
+ ### `body`
193
+
194
+ Returns the body of the message as a UTF-8 encoded string.
195
+
196
+ ### `attempts`
197
+
198
+ Returns the number of times this message was attempted to be processed. For
199
+ most messages this should be 1 (since it will be your first attempt processing
200
+ them). If it's more than 1, that means that you requeued the message or it
201
+ timed out in flight.
202
+
203
+ ### `timestamp`
204
+
205
+ Returns the time this message was originally sent to NSQ as a `Time` object.
206
+
207
+ ### `#finish`
208
+
209
+ Notify NSQ that you've completed processing of this message.
210
+
211
+ ### `#touch`
212
+
213
+ Tells NSQ to reset the message timeout for this message so you have more time
214
+ to process it.
215
+
216
+ ### `#requeue(timeout = 0)`
217
+
218
+ Tells NSQ to requeue this message. Called with no arguments, this will requeue
219
+ the message and it will be available to be received immediately.
220
+
221
+ Optionally you can pass a number of milliseconds as an argument. This tells
222
+ NSQ to delay its requeueing by that number of milliseconds.
223
+
224
+
225
+ ## Logging
226
+
227
+ By default, `nsq-ruby` doesn't log anything. To enable logging, use
228
+ `Nsq.logger=` and point it at a Ruby Logger instance. Like this:
229
+
230
+ ```Ruby
231
+ Nsq.logger = Logger.new(STDOUT)
232
+ ```
233
+
234
+
235
+ ## Requirements
236
+
237
+ NSQ v0.2.29 or later for IDENTIFY metadata specification (0.2.28) and per-
238
+ connection timeout support (0.2.29).
239
+
240
+
241
+ ### Supports
242
+
243
+ - Discovery via nsqlookupd
244
+ - Automatic reconnection to nsqd
245
+ - Producing to all nsqd instances automatically via nsqlookupd
246
+
247
+
248
+ ### Does not support
249
+
250
+ - TLS
251
+ - Compression
252
+ - Backoff
253
+ - Authentication
254
+
255
+ If you need more advanced features, like these, you should check out
256
+ [Krakow](https://github.com/chrisroberts/krakow), a more fully featured NSQ
257
+ client for Ruby.
258
+
259
+
260
+ ## Testing
261
+
262
+ Run the tests like this:
263
+
264
+ ```
265
+ rake spec
266
+ ```
267
+
268
+ Want a deluge of logging while running the specs to help determine what is
269
+ going on?
270
+
271
+ ```
272
+ VERBOSE=true rake spec
273
+ ```
274
+
275
+
276
+ ## Is this production ready?
277
+
278
+ Yes! It's used in several critical parts of our infrastructure at
279
+ [Wistia](http://wistia.com) and currently produces and consumes tens of
280
+ millions of messages a day.
281
+
282
+
283
+ ## Authors
284
+
285
+ - Robby Grossman (@freerobby)
286
+ - Brendan Schwartz (@bschwartz)
287
+ - Marshall Moutenot (@mmoutenot)
288
+
289
+
290
+ ## MIT License
291
+
292
+ Copyright (C) 2014 Wistia, Inc.
293
+
294
+ Permission is hereby granted, free of charge, to any person obtaining a copy of
295
+ this software and associated documentation files (the "Software"), to deal in
296
+ the Software without restriction, including without limitation the rights to
297
+ use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
298
+ of the Software, and to permit persons to whom the Software is furnished to do
299
+ so, subject to the following conditions:
300
+
301
+ The above copyright notice and this permission notice shall be included in all
302
+ copies or substantial portions of the Software.
303
+
304
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
305
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
306
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
307
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
308
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
309
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
310
+ SOFTWARE.
311
+
@@ -0,0 +1,13 @@
1
+ require_relative 'version'
2
+
3
+ require_relative 'nsq/logger'
4
+
5
+ require_relative 'nsq/exceptions'
6
+
7
+ require_relative 'nsq/frames/frame'
8
+ require_relative 'nsq/frames/error'
9
+ require_relative 'nsq/frames/response'
10
+ require_relative 'nsq/frames/message'
11
+
12
+ require_relative 'nsq/consumer'
13
+ require_relative 'nsq/producer'
@@ -0,0 +1,117 @@
1
+ require_relative 'discovery'
2
+ require_relative 'connection'
3
+ require_relative 'logger'
4
+
5
+ module Nsq
6
+ class ClientBase
7
+ include Nsq::AttributeLogger
8
+ @@log_attributes = [:topic]
9
+
10
+ attr_reader :topic
11
+ attr_reader :connections
12
+
13
+ def connected?
14
+ @connections.values.any?(&:connected?)
15
+ end
16
+
17
+
18
+ def terminate
19
+ @discovery_thread.kill if @discovery_thread
20
+ drop_all_connections
21
+ end
22
+
23
+
24
+ private
25
+
26
+ # discovers nsqds from an nsqlookupd repeatedly
27
+ #
28
+ # opts:
29
+ # nsqlookups: ['127.0.0.1:4161'],
30
+ # topic: 'topic-to-find-nsqds-for',
31
+ # interval: 60
32
+ #
33
+ def discover_repeatedly(opts = {})
34
+ @discovery_thread = Thread.new do
35
+
36
+ @discovery = Discovery.new(opts[:nsqlookupds])
37
+
38
+ loop do
39
+ begin
40
+ nsqds = nsqds_from_lookupd(opts[:topic])
41
+ drop_and_add_connections(nsqds)
42
+ rescue DiscoveryException
43
+ # We can't connect to any nsqlookupds. That's okay, we'll just
44
+ # leave our current nsqd connections alone and try again later.
45
+ warn 'Could not connect to any nsqlookupd instances in discovery loop'
46
+ end
47
+ sleep opts[:interval]
48
+ end
49
+
50
+ end
51
+
52
+ @discovery_thread.abort_on_exception = true
53
+ end
54
+
55
+
56
+ def nsqds_from_lookupd(topic = nil)
57
+ if topic
58
+ @discovery.nsqds_for_topic(topic)
59
+ else
60
+ @discovery.nsqds
61
+ end
62
+ end
63
+
64
+
65
+ def drop_and_add_connections(nsqds)
66
+ # drop nsqd connections that are no longer in lookupd
67
+ missing_nsqds = @connections.keys - nsqds
68
+ missing_nsqds.each do |nsqd|
69
+ drop_connection(nsqd)
70
+ end
71
+
72
+ # add new ones
73
+ new_nsqds = nsqds - @connections.keys
74
+ new_nsqds.each do |nsqd|
75
+ begin
76
+ add_connection(nsqd)
77
+ rescue Exception => ex
78
+ error "Failed to connect to nsqd @ #{nsqd}: #{ex}"
79
+ end
80
+ end
81
+
82
+ # balance RDY state amongst the connections
83
+ connections_changed
84
+ end
85
+
86
+
87
+ def add_connection(nsqd, options = {})
88
+ info "+ Adding connection #{nsqd}"
89
+ host, port = nsqd.split(':')
90
+ connection = Connection.new({
91
+ host: host,
92
+ port: port
93
+ }.merge(options))
94
+ @connections[nsqd] = connection
95
+ end
96
+
97
+
98
+ def drop_connection(nsqd)
99
+ info "- Dropping connection #{nsqd}"
100
+ connection = @connections.delete(nsqd)
101
+ connection.close if connection
102
+ connections_changed
103
+ end
104
+
105
+
106
+ def drop_all_connections
107
+ @connections.keys.each do |nsqd|
108
+ drop_connection(nsqd)
109
+ end
110
+ end
111
+
112
+
113
+ # optional subclass hook
114
+ def connections_changed
115
+ end
116
+ end
117
+ end
@@ -0,0 +1,402 @@
1
+ require 'json'
2
+ require 'socket'
3
+ require 'timeout'
4
+
5
+ require_relative 'frames/error'
6
+ require_relative 'frames/message'
7
+ require_relative 'frames/response'
8
+ require_relative 'logger'
9
+
10
+ module Nsq
11
+ class Connection
12
+ include Nsq::AttributeLogger
13
+ @@log_attributes = [:host, :port]
14
+
15
+ attr_reader :host
16
+ attr_reader :port
17
+ attr_accessor :max_in_flight
18
+ attr_reader :presumed_in_flight
19
+
20
+ USER_AGENT = "nsq-ruby/#{Nsq::Version::STRING}"
21
+ RESPONSE_HEARTBEAT = '_heartbeat_'
22
+ RESPONSE_OK = 'OK'
23
+
24
+
25
+ def initialize(opts = {})
26
+ @host = opts[:host] || (raise ArgumentError, 'host is required')
27
+ @port = opts[:port] || (raise ArgumentError, 'host is required')
28
+ @queue = opts[:queue]
29
+ @topic = opts[:topic]
30
+ @channel = opts[:channel]
31
+ @msg_timeout = opts[:msg_timeout] || 60_000 # 60s
32
+ @max_in_flight = opts[:max_in_flight] || 1
33
+
34
+ if @msg_timeout < 1000
35
+ raise ArgumentError, 'msg_timeout cannot be less than 1000. it\'s in milliseconds.'
36
+ end
37
+
38
+ # for outgoing communication
39
+ @write_queue = Queue.new
40
+
41
+ # For indicating that the connection has died.
42
+ # We use a Queue so we don't have to poll. Used to communicate across
43
+ # threads (from write_loop and read_loop to connect_and_monitor).
44
+ @death_queue = Queue.new
45
+
46
+ @connected = false
47
+ @presumed_in_flight = 0
48
+
49
+ open_connection
50
+ start_monitoring_connection
51
+ end
52
+
53
+
54
+ def connected?
55
+ @connected
56
+ end
57
+
58
+
59
+ # close the connection and don't try to re-open it
60
+ def close
61
+ stop_monitoring_connection
62
+ close_connection
63
+ end
64
+
65
+
66
+ def sub(topic, channel)
67
+ write "SUB #{topic} #{channel}\n"
68
+ end
69
+
70
+
71
+ def rdy(count)
72
+ write "RDY #{count}\n"
73
+ end
74
+
75
+
76
+ def fin(message_id)
77
+ write "FIN #{message_id}\n"
78
+ decrement_in_flight
79
+ end
80
+
81
+
82
+ def req(message_id, timeout)
83
+ write "REQ #{message_id} #{timeout}\n"
84
+ decrement_in_flight
85
+ end
86
+
87
+
88
+ def touch(message_id)
89
+ write "TOUCH #{message_id}\n"
90
+ end
91
+
92
+
93
+ def pub(topic, message)
94
+ write ["PUB #{topic}\n", message.bytesize, message].pack('a*Na*')
95
+ end
96
+
97
+
98
+ def mpub(topic, messages)
99
+ body = messages.map do |message|
100
+ [message.bytesize, message].pack('Na*')
101
+ end.join
102
+
103
+ write ["MPUB #{topic}\n", body.bytesize, messages.size, body].pack('a*NNa*')
104
+ end
105
+
106
+
107
+ # Tell the server we are ready for more messages!
108
+ def re_up_ready
109
+ rdy(@max_in_flight)
110
+ # assume these messages are coming our way. yes, this might not be the
111
+ # case, but it's much easier to manage our RDY state with the server if
112
+ # we treat things this way.
113
+ @presumed_in_flight = @max_in_flight
114
+ end
115
+
116
+
117
+ private
118
+
119
+ def cls
120
+ write "CLS\n"
121
+ end
122
+
123
+
124
+ def nop
125
+ write "NOP\n"
126
+ end
127
+
128
+
129
+ def write(raw)
130
+ @write_queue.push(raw)
131
+ end
132
+
133
+
134
+ def write_to_socket(raw)
135
+ debug ">>> #{raw.inspect}"
136
+ @socket.write(raw)
137
+ end
138
+
139
+
140
+ def identify
141
+ hostname = Socket.gethostname
142
+ metadata = {
143
+ client_id: Socket.gethostbyname(hostname).flatten.compact.first,
144
+ hostname: hostname,
145
+ feature_negotiation: true,
146
+ heartbeat_interval: 30_000, # 30 seconds
147
+ output_buffer: 16_000, # 16kb
148
+ output_buffer_timeout: 250, # 250ms
149
+ tls_v1: false,
150
+ snappy: false,
151
+ deflate: false,
152
+ sample_rate: 0, # disable sampling
153
+ user_agent: USER_AGENT,
154
+ msg_timeout: @msg_timeout
155
+ }.to_json
156
+ write_to_socket ["IDENTIFY\n", metadata.length, metadata].pack('a*Na*')
157
+
158
+ # Now wait for the response!
159
+ frame = receive_frame
160
+ server = JSON.parse(frame.data)
161
+
162
+ if @max_in_flight > server['max_rdy_count']
163
+ raise "max_in_flight is set to #{@max_in_flight}, server only supports #{server['max_rdy_count']}"
164
+ end
165
+
166
+ @server_version = server['version']
167
+ end
168
+
169
+
170
+ def handle_response(frame)
171
+ if frame.data == RESPONSE_HEARTBEAT
172
+ debug 'Received heartbeat'
173
+ nop
174
+ elsif frame.data == RESPONSE_OK
175
+ debug 'Received OK'
176
+ else
177
+ die "Received response we don't know how to handle: #{frame.data}"
178
+ end
179
+ end
180
+
181
+
182
+ def receive_frame
183
+ if buffer = @socket.read(8)
184
+ size, type = buffer.unpack('NN')
185
+ size -= 4 # we want the size of the data part and type already took up 4 bytes
186
+ data = @socket.read(size)
187
+ frame_class = frame_class_for_type(type)
188
+ return frame_class.new(data, self)
189
+ end
190
+ end
191
+
192
+
193
+ FRAME_CLASSES = [Response, Error, Message]
194
+ def frame_class_for_type(type)
195
+ raise "Bad frame type specified: #{type}" if type > FRAME_CLASSES.length - 1
196
+ [Response, Error, Message][type]
197
+ end
198
+
199
+
200
+ def decrement_in_flight
201
+ @presumed_in_flight -= 1
202
+
203
+ if server_needs_rdy_re_ups?
204
+ # now that we're less than @max_in_flight we might need to re-up our RDY state
205
+ threshold = (@max_in_flight * 0.2).ceil
206
+ re_up_ready if @presumed_in_flight <= threshold
207
+ end
208
+ end
209
+
210
+
211
+ def start_read_loop
212
+ @read_loop_thread ||= Thread.new{read_loop}
213
+ end
214
+
215
+
216
+ def stop_read_loop
217
+ @read_loop_thread.kill if @read_loop_thread
218
+ @read_loop_thread = nil
219
+ end
220
+
221
+
222
+ def read_loop
223
+ loop do
224
+ frame = receive_frame
225
+ if frame.is_a?(Response)
226
+ handle_response(frame)
227
+ elsif frame.is_a?(Error)
228
+ error "Error received: #{frame.data}"
229
+ elsif frame.is_a?(Message)
230
+ debug "<<< #{frame.body}"
231
+ @queue.push(frame) if @queue
232
+ else
233
+ raise 'No data from socket'
234
+ end
235
+ end
236
+ rescue Exception => ex
237
+ die(ex)
238
+ end
239
+
240
+
241
+ def start_write_loop
242
+ @write_loop_thread ||= Thread.new{write_loop}
243
+ end
244
+
245
+
246
+ def stop_write_loop
247
+ @stop_write_loop = true
248
+ @write_loop_thread.join(1) if @write_loop_thread
249
+ @write_loop_thread = nil
250
+ end
251
+
252
+
253
+ def write_loop
254
+ @stop_write_loop = false
255
+ data = nil
256
+ loop do
257
+ data = @write_queue.pop
258
+ write_to_socket(data)
259
+ break if @stop_write_loop && @write_queue.size == 0
260
+ end
261
+ rescue Exception => ex
262
+ # requeue PUB and MPUB commands
263
+ if data =~ /^M?PUB/
264
+ debug "Requeueing to write_queue: #{data.inspect}"
265
+ @write_queue.push(data)
266
+ end
267
+ die(ex)
268
+ end
269
+
270
+
271
+ # Waits for death of connection
272
+ def start_monitoring_connection
273
+ @connection_monitor_thread ||= Thread.new{monitor_connection}
274
+ @connection_monitor_thread.abort_on_exception = true
275
+ end
276
+
277
+
278
+ def stop_monitoring_connection
279
+ @connection_monitor_thread.kill if @connection_monitor_thread
280
+ @connection_monitor = nil
281
+ end
282
+
283
+
284
+ def monitor_connection
285
+ loop do
286
+ # wait for death, hopefully it never comes
287
+ cause_of_death = @death_queue.pop
288
+ warn "Died from: #{cause_of_death}"
289
+
290
+ debug 'Reconnecting...'
291
+ reconnect
292
+ debug 'Reconnected!'
293
+
294
+ # clear all death messages, since we're now reconnected.
295
+ # we don't want to complete this loop and immediately reconnect again.
296
+ @death_queue.clear
297
+ end
298
+ end
299
+
300
+
301
+ # close the connection if it's not already closed and try to reconnect
302
+ # over and over until we succeed!
303
+ def reconnect
304
+ close_connection
305
+ with_retries do
306
+ open_connection
307
+ end
308
+ end
309
+
310
+
311
+ def open_connection
312
+ @socket = TCPSocket.new(@host, @port)
313
+ # write the version and IDENTIFY directly to the socket to make sure
314
+ # it gets to nsqd ahead of anything in the `@write_queue`
315
+ write_to_socket ' V2'
316
+ identify
317
+
318
+ start_read_loop
319
+ start_write_loop
320
+ @connected = true
321
+
322
+ # we need to re-subscribe if there's a topic specified
323
+ if @topic
324
+ debug "Subscribing to #{@topic}"
325
+ sub(@topic, @channel)
326
+ re_up_ready
327
+ end
328
+ end
329
+
330
+
331
+ # closes the connection and stops listening for messages
332
+ def close_connection
333
+ cls if connected?
334
+ stop_read_loop
335
+ stop_write_loop
336
+ @socket = nil
337
+ @connected = false
338
+ end
339
+
340
+
341
+ # this is called when there's a connection error in the read or write loop
342
+ # it triggers `connect_and_monitor` to try to reconnect
343
+ def die(reason)
344
+ @connected = false
345
+ @death_queue.push(reason)
346
+ end
347
+
348
+
349
+ # Retry the supplied block with exponential backoff.
350
+ #
351
+ # Borrowed liberally from:
352
+ # https://github.com/ooyala/retries/blob/master/lib/retries.rb
353
+ def with_retries(&block)
354
+ base_sleep_seconds = 0.5
355
+ max_sleep_seconds = 300 # 5 minutes
356
+
357
+ # Let's do this thing
358
+ attempts = 0
359
+ start_time = Time.now
360
+
361
+ begin
362
+ attempts += 1
363
+ return block.call(attempts)
364
+
365
+ rescue Errno::ECONNREFUSED, Errno::ECONNRESET, Errno::EHOSTUNREACH,
366
+ Errno::ENETDOWN, Errno::ENETUNREACH, Errno::ETIMEDOUT, Timeout::Error => ex
367
+
368
+ raise ex if attempts >= 100
369
+
370
+ # The sleep time is an exponentially-increasing function of base_sleep_seconds.
371
+ # But, it never exceeds max_sleep_seconds.
372
+ sleep_seconds = [base_sleep_seconds * (2 ** (attempts - 1)), max_sleep_seconds].min
373
+ # Randomize to a random value in the range sleep_seconds/2 .. sleep_seconds
374
+ sleep_seconds = sleep_seconds * (0.5 * (1 + rand()))
375
+ # But never sleep less than base_sleep_seconds
376
+ sleep_seconds = [base_sleep_seconds, sleep_seconds].max
377
+
378
+ warn "Failed to connect: #{ex}. Retrying in #{sleep_seconds.round(1)} seconds."
379
+
380
+ snooze(sleep_seconds)
381
+
382
+ retry
383
+ end
384
+ end
385
+
386
+
387
+ # Se we can stub for testing and reconnect in a tight loop
388
+ def snooze(t)
389
+ sleep(t)
390
+ end
391
+
392
+
393
+ def server_needs_rdy_re_ups?
394
+ # versions less than 0.3.0 need RDY re-ups
395
+ # see: https://github.com/bitly/nsq/blob/master/ChangeLog.md#030---2014-11-18
396
+ major, minor, patch = @server_version.split('.').map(&:to_i)
397
+ major == 0 && minor <= 2
398
+ end
399
+
400
+
401
+ end
402
+ end
@@ -0,0 +1,84 @@
1
+ require_relative 'client_base'
2
+
3
+ module Nsq
4
+ class Consumer < ClientBase
5
+
6
+ attr_reader :max_in_flight
7
+
8
+ def initialize(opts = {})
9
+ if opts[:nsqlookupd]
10
+ @nsqlookupds = [opts[:nsqlookupd]].flatten
11
+ else
12
+ @nsqlookupds = []
13
+ end
14
+
15
+ @topic = opts[:topic] || raise(ArgumentError, 'topic is required')
16
+ @channel = opts[:channel] || raise(ArgumentError, 'channel is required')
17
+ @max_in_flight = opts[:max_in_flight] || 1
18
+ @discovery_interval = opts[:discovery_interval] || 60
19
+ @msg_timeout = opts[:msg_timeout]
20
+
21
+ # This is where we queue up the messages we receive from each connection
22
+ @messages = opts[:queue] || Queue.new
23
+
24
+ # This is where we keep a record of our active nsqd connections
25
+ # The key is a string with the host and port of the instance (e.g.
26
+ # '127.0.0.1:4150') and the key is the Connection instance.
27
+ @connections = {}
28
+
29
+ if !@nsqlookupds.empty?
30
+ discover_repeatedly(
31
+ nsqlookupds: @nsqlookupds,
32
+ topic: @topic,
33
+ interval: @discovery_interval
34
+ )
35
+ else
36
+ # normally, we find nsqd instances to connect to via nsqlookupd(s)
37
+ # in this case let's connect to an nsqd instance directly
38
+ add_connection(opts[:nsqd] || '127.0.0.1:4150', max_in_flight: @max_in_flight)
39
+ end
40
+
41
+ at_exit{terminate}
42
+ end
43
+
44
+
45
+ # pop the next message off the queue
46
+ def pop
47
+ @messages.pop
48
+ end
49
+
50
+
51
+ # returns the number of messages we have locally in the queue
52
+ def size
53
+ @messages.size
54
+ end
55
+
56
+
57
+ private
58
+ def add_connection(nsqd, options = {})
59
+ super(nsqd, {
60
+ topic: @topic,
61
+ channel: @channel,
62
+ queue: @messages,
63
+ msg_timeout: @msg_timeout,
64
+ max_in_flight: 1
65
+ }.merge(options))
66
+ end
67
+
68
+ # Be conservative, but don't set a connection's max_in_flight below 1
69
+ def max_in_flight_per_connection(number_of_connections = @connections.length)
70
+ [@max_in_flight / number_of_connections, 1].max
71
+ end
72
+
73
+ def connections_changed
74
+ redistribute_ready
75
+ end
76
+
77
+ def redistribute_ready
78
+ @connections.values.each do |connection|
79
+ connection.max_in_flight = max_in_flight_per_connection
80
+ connection.re_up_ready
81
+ end
82
+ end
83
+ end
84
+ end
@@ -0,0 +1,96 @@
1
+ require 'json'
2
+ require 'net/http'
3
+ require 'uri'
4
+
5
+ require_relative 'logger'
6
+
7
+ # Connects to nsqlookup's to find the nsqd instances for a given topic
8
+ module Nsq
9
+ class Discovery
10
+ include Nsq::AttributeLogger
11
+
12
+ # lookupd addresses must be formatted like so: '<host>:<http-port>'
13
+ def initialize(lookupds)
14
+ @lookupds = lookupds
15
+ end
16
+
17
+ # Returns an array of nsqds instances
18
+ #
19
+ # nsqd instances returned are strings in this format: '<host>:<tcp-port>'
20
+ #
21
+ # discovery.nsqds
22
+ # #=> ['127.0.0.1:4150', '127.0.0.1:4152']
23
+ #
24
+ # If all nsqlookupd's are unreachable, raises Nsq::DiscoveryException
25
+ #
26
+ def nsqds
27
+ gather_nsqds_from_all_lookupds do |lookupd|
28
+ get_nsqds(lookupd)
29
+ end
30
+ end
31
+
32
+ # Returns an array of nsqds instances that have messages for
33
+ # that topic.
34
+ #
35
+ # nsqd instances returned are strings in this format: '<host>:<tcp-port>'
36
+ #
37
+ # discovery.nsqds_for_topic('a-topic')
38
+ # #=> ['127.0.0.1:4150', '127.0.0.1:4152']
39
+ #
40
+ # If all nsqlookupd's are unreachable, raises Nsq::DiscoveryException
41
+ #
42
+ def nsqds_for_topic(topic)
43
+ gather_nsqds_from_all_lookupds do |lookupd|
44
+ get_nsqds(lookupd, topic)
45
+ end
46
+ end
47
+
48
+
49
+ private
50
+
51
+ def gather_nsqds_from_all_lookupds
52
+ nsqd_list = @lookupds.map do |lookupd|
53
+ yield(lookupd)
54
+ end.flatten
55
+
56
+ # All nsqlookupds were unreachable, raise an error!
57
+ if nsqd_list.length > 0 && nsqd_list.all? { |nsqd| nsqd.nil? }
58
+ raise DiscoveryException
59
+ end
60
+
61
+ nsqd_list.compact.uniq
62
+ end
63
+
64
+ # Returns an array of nsqd addresses
65
+ # If there's an error, return nil
66
+ def get_nsqds(lookupd, topic = nil)
67
+ uri_scheme = 'http://' unless lookupd.match(%r(https?://))
68
+ uri = URI.parse("#{uri_scheme}#{lookupd}")
69
+
70
+ uri.query = "ts=#{Time.now.to_i}"
71
+ if topic
72
+ uri.path = '/lookup'
73
+ uri.query += "&topic=#{topic}"
74
+ else
75
+ uri.path = '/nodes'
76
+ end
77
+
78
+ begin
79
+ body = Net::HTTP.get(uri)
80
+ data = JSON.parse(body)
81
+
82
+ if data['data'] && data['data']['producers']
83
+ data['data']['producers'].map do |producer|
84
+ "#{producer['broadcast_address']}:#{producer['tcp_port']}"
85
+ end
86
+ else
87
+ []
88
+ end
89
+ rescue Exception => e
90
+ error "Error during discovery for #{lookupd}: #{e}"
91
+ nil
92
+ end
93
+ end
94
+
95
+ end
96
+ end
@@ -0,0 +1,5 @@
1
+ module Nsq
2
+ # Raised when nsqlookupd discovery fails
3
+ class DiscoveryException < Exception; end
4
+ end
5
+
@@ -0,0 +1,6 @@
1
+ require_relative 'frame'
2
+
3
+ module Nsq
4
+ class Error < Frame
5
+ end
6
+ end
@@ -0,0 +1,16 @@
1
+ require_relative '../logger'
2
+
3
+ module Nsq
4
+ class Frame
5
+ include Nsq::AttributeLogger
6
+ @@log_attributes = [:connection]
7
+
8
+ attr_reader :data
9
+ attr_reader :connection
10
+
11
+ def initialize(data, connection)
12
+ @data = data
13
+ @connection = connection
14
+ end
15
+ end
16
+ end
@@ -0,0 +1,34 @@
1
+ require_relative 'frame'
2
+
3
+ module Nsq
4
+ class Message < Frame
5
+
6
+ attr_reader :attempts
7
+ attr_reader :id
8
+ attr_reader :body
9
+
10
+ def initialize(data, connection)
11
+ super
12
+ ts_1, ts_2, @attempts, @id, @body = @data.unpack('NNs>a16a*')
13
+ @timestamp_in_nanoseconds = (ts_1 * (2**32)) + ts_2
14
+ @body.force_encoding('UTF-8')
15
+ end
16
+
17
+ def finish
18
+ connection.fin(id)
19
+ end
20
+
21
+ def requeue(timeout = 0)
22
+ connection.req(id, timeout)
23
+ end
24
+
25
+ def touch
26
+ connection.touch(id)
27
+ end
28
+
29
+ def timestamp
30
+ Time.at(@timestamp_in_nanoseconds / 1_000_000_000.0)
31
+ end
32
+
33
+ end
34
+ end
@@ -0,0 +1,6 @@
1
+ require_relative 'frame'
2
+
3
+ module Nsq
4
+ class Response < Frame
5
+ end
6
+ end
@@ -0,0 +1,38 @@
1
+ require 'logger'
2
+ module Nsq
3
+ @@logger = Logger.new(nil)
4
+
5
+
6
+ def self.logger
7
+ @@logger
8
+ end
9
+
10
+
11
+ def self.logger=(new_logger)
12
+ @@logger = new_logger
13
+ end
14
+
15
+
16
+ module AttributeLogger
17
+ def self.included(klass)
18
+ klass.send :class_variable_set, :@@log_attributes, []
19
+ end
20
+
21
+ %w(fatal error warn info debug).map{|m| m.to_sym}.each do |level|
22
+ define_method level do |msg|
23
+ Nsq.logger.send(level, "#{prefix} #{msg}")
24
+ end
25
+ end
26
+
27
+
28
+ private
29
+ def prefix
30
+ attrs = self.class.send(:class_variable_get, :@@log_attributes)
31
+ if attrs.count > 0
32
+ "[#{attrs.map{|a| "#{a.to_s}: #{self.send(a)}"}.join(' ')}] "
33
+ else
34
+ ''
35
+ end
36
+ end
37
+ end
38
+ end
@@ -0,0 +1,63 @@
1
+ require_relative 'client_base'
2
+
3
+ module Nsq
4
+ class Producer < ClientBase
5
+ attr_reader :topic
6
+
7
+ def initialize(opts = {})
8
+ @connections = {}
9
+ @topic = opts[:topic] || raise(ArgumentError, 'topic is required')
10
+ @discovery_interval = opts[:discovery_interval] || 60
11
+
12
+ nsqlookupds = []
13
+ if opts[:nsqlookupd]
14
+ nsqlookupds = [opts[:nsqlookupd]].flatten
15
+ discover_repeatedly(
16
+ nsqlookupds: nsqlookupds,
17
+ interval: @discovery_interval
18
+ )
19
+
20
+ elsif opts[:nsqd]
21
+ nsqds = [opts[:nsqd]].flatten
22
+ nsqds.each{|d| add_connection(d)}
23
+
24
+ else
25
+ add_connection('127.0.0.1:4150')
26
+ end
27
+
28
+ at_exit{terminate}
29
+ end
30
+
31
+
32
+ def write(*raw_messages)
33
+ # stringify the messages
34
+ messages = raw_messages.map(&:to_s)
35
+
36
+ # get a suitable connection to write to
37
+ connection = connection_for_write
38
+
39
+ if messages.length > 1
40
+ connection.mpub(@topic, messages)
41
+ else
42
+ connection.pub(@topic, messages.first)
43
+ end
44
+ end
45
+
46
+
47
+ private
48
+ def connection_for_write
49
+ # Choose a random Connection that's currently connected
50
+ # Or, if there's nothing connected, just take any random one
51
+ connections_currently_connected = connections.select{|_,c| c.connected?}
52
+ connection = connections_currently_connected.values.sample || connections.values.sample
53
+
54
+ # Raise an exception if there's no connection available
55
+ unless connection
56
+ raise 'No connections available'
57
+ end
58
+
59
+ connection
60
+ end
61
+
62
+ end
63
+ end
@@ -0,0 +1,9 @@
1
+ module Nsq
2
+ module Version
3
+ MAJOR = 1
4
+ MINOR = 2
5
+ PATCH = 1
6
+ BUILD = 0
7
+ STRING = [MAJOR, MINOR, PATCH, BUILD].compact.join('.')
8
+ end
9
+ end
metadata ADDED
@@ -0,0 +1,116 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: nsq-ruby-maglev-
3
+ version: !ruby/object:Gem::Version
4
+ version: 1.2.1.0
5
+ platform: ruby
6
+ authors:
7
+ - Wistia
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2015-08-14 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: bundler
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - "~>"
18
+ - !ruby/object:Gem::Version
19
+ version: '1.0'
20
+ type: :development
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - "~>"
25
+ - !ruby/object:Gem::Version
26
+ version: '1.0'
27
+ - !ruby/object:Gem::Dependency
28
+ name: jeweler
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - "~>"
32
+ - !ruby/object:Gem::Version
33
+ version: '2.0'
34
+ type: :development
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - "~>"
39
+ - !ruby/object:Gem::Version
40
+ version: '2.0'
41
+ - !ruby/object:Gem::Dependency
42
+ name: nsq-cluster
43
+ requirement: !ruby/object:Gem::Requirement
44
+ requirements:
45
+ - - "~>"
46
+ - !ruby/object:Gem::Version
47
+ version: '1.1'
48
+ type: :development
49
+ prerelease: false
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - "~>"
53
+ - !ruby/object:Gem::Version
54
+ version: '1.1'
55
+ - !ruby/object:Gem::Dependency
56
+ name: rspec
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - "~>"
60
+ - !ruby/object:Gem::Version
61
+ version: '3.0'
62
+ type: :development
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - "~>"
67
+ - !ruby/object:Gem::Version
68
+ version: '3.0'
69
+ description: ''
70
+ email: dev@wistia.com
71
+ executables: []
72
+ extensions: []
73
+ extra_rdoc_files:
74
+ - LICENSE.txt
75
+ - README.md
76
+ files:
77
+ - LICENSE.txt
78
+ - README.md
79
+ - lib/nsq.rb
80
+ - lib/nsq/client_base.rb
81
+ - lib/nsq/connection.rb
82
+ - lib/nsq/consumer.rb
83
+ - lib/nsq/discovery.rb
84
+ - lib/nsq/exceptions.rb
85
+ - lib/nsq/frames/error.rb
86
+ - lib/nsq/frames/frame.rb
87
+ - lib/nsq/frames/message.rb
88
+ - lib/nsq/frames/response.rb
89
+ - lib/nsq/logger.rb
90
+ - lib/nsq/producer.rb
91
+ - lib/version.rb
92
+ homepage: http://github.com/johnnyt/nsq-ruby
93
+ licenses:
94
+ - MIT
95
+ metadata: {}
96
+ post_install_message:
97
+ rdoc_options: []
98
+ require_paths:
99
+ - lib
100
+ required_ruby_version: !ruby/object:Gem::Requirement
101
+ requirements:
102
+ - - ">="
103
+ - !ruby/object:Gem::Version
104
+ version: '0'
105
+ required_rubygems_version: !ruby/object:Gem::Requirement
106
+ requirements:
107
+ - - ">="
108
+ - !ruby/object:Gem::Version
109
+ version: '0'
110
+ requirements: []
111
+ rubyforge_project:
112
+ rubygems_version: 2.2.2
113
+ signing_key:
114
+ specification_version: 4
115
+ summary: Ruby client library for NSQ
116
+ test_files: []