message_bus 1.0.9 → 1.0.10

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of message_bus might be problematic. Click here for more details.

checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: fcb3053a5e28fd2c050064c43f0065c3b8ee7bf1
4
- data.tar.gz: b766e21019d5f45e4a8083e9aab9b8e8fa3af947
3
+ metadata.gz: e399202ac9751b63a940cf2442247905bc3394c3
4
+ data.tar.gz: cca646b9e0cead6959171bccc487015983caae22
5
5
  SHA512:
6
- metadata.gz: f4637d0abd63d7c955e1155923bf1e74c6abc8c2aeda6f8f2c8d480c7f76bbadbcaf5f602667b37a1c5618f06d2fe9c1295e270ef20bdbc9fec25f078301eb2c
7
- data.tar.gz: 0f41d3a209691da9d2866530b20d73de6d99fc3aa87456fa6d79d6a5ebec363f1fc727963d77d63f393c7359efd75dde80fd544049a624ecae8ef1ab4ccb7388
6
+ metadata.gz: 0e8caed956222e13faabee6bb0d0b33552fd9d030d4dd481d39baf16a208f9073e1be6c43a9a38e193ba07e8d31ccc65073a97685ea5dc0f72c6dd4833ddd813
7
+ data.tar.gz: bd0253ae0cead17c8508e91fe9ea0bd6bf94f8cdd16ad8822aa914e8853e3cd141380aada6b6ac62e9c13f96622ebd062f2b5916845f23672a757cc279820b98
data/CHANGELOG CHANGED
@@ -1,3 +1,9 @@
1
+ 01-05-2015
2
+ - Version: 1.0.10
3
+ - Feature: no longer depends on EventMachine (only used for Thin backend)
4
+ - Feature: realiable pub sub will queue messages in memory if redis is readonly, configurable
5
+ - Fix: if redis is flushed we will continue to deliver messages
6
+
1
7
  23-03-2015
2
8
  - Version 1.0.9
3
9
  - Fix: inherit off StandardError not Exception for all exceptions raised
@@ -67,6 +73,5 @@
67
73
  - Improve robustness under failure conditions
68
74
 
69
75
  30-09-2013
70
-
71
76
  - Fix failures in Ruby 1.9
72
77
  - Set up rack hijack by default in light of passengers new setting
data/Guardfile CHANGED
@@ -1,7 +1,51 @@
1
- guard 'rspec' do
2
- watch(%r{^spec/.+_spec\.rb$})
3
- watch(%r{^lib/(.+)\.rb$}) { |m| "spec/lib/#{m[1]}_spec.rb" }
4
- watch('spec/spec_helper.rb') { "spec" }
5
- end
1
+ # A sample Guardfile
2
+ # More info at https://github.com/guard/guard#readme
3
+
4
+ ## Uncomment and set this to only include directories you want to watch
5
+ directories %w(lib spec)
6
+
7
+ ## Uncomment to clear the screen before every task
8
+ # clearing :on
9
+
10
+ ## Guard internally checks for changes in the Guardfile and exits.
11
+ ## If you want Guard to automatically start up again, run guard in a
12
+ ## shell loop, e.g.:
13
+ ##
14
+ ## $ while bundle exec guard; do echo "Restarting Guard..."; done
15
+ ##
16
+ ## Note: if you are using the `directories` clause above and you are not
17
+ ## watching the project directory ('.'), then you will want to move
18
+ ## the Guardfile to a watched dir and symlink it back, e.g.
19
+ #
20
+ # $ mkdir config
21
+ # $ mv Guardfile config/
22
+ # $ ln -s config/Guardfile .
23
+ #
24
+ # and, you'll have to watch "config/Guardfile" instead of "Guardfile"
25
+
26
+ # Note: The cmd option is now required due to the increasing number of ways
27
+ # rspec may be run, below are examples of the most common uses.
28
+ # * bundler: 'bundle exec rspec'
29
+ # * bundler binstubs: 'bin/rspec'
30
+ # * spring: 'bin/rspec' (This will use spring if running and you have
31
+ # installed the spring binstubs per the docs)
32
+ # * zeus: 'zeus rspec' (requires the server to be started separately)
33
+ # * 'just' rspec: 'rspec'
6
34
 
35
+ guard :rspec, cmd: "bundle exec rspec" do
36
+ require "guard/rspec/dsl"
37
+ dsl = Guard::RSpec::Dsl.new(self)
7
38
 
39
+ # Feel free to open issues for suggestions and improvements
40
+
41
+ # RSpec files
42
+ rspec = dsl.rspec
43
+ watch(rspec.spec_helper) { rspec.spec_dir }
44
+ watch(rspec.spec_support) { rspec.spec_dir }
45
+ watch(rspec.spec_files)
46
+
47
+ # Ruby files
48
+ ruby = dsl.ruby
49
+ dsl.watch_spec_files_for(ruby.lib_files)
50
+
51
+ end
data/README.md CHANGED
@@ -54,7 +54,7 @@ end
54
54
  MessageBus.client_filter("/channel") do |user_id, message|
55
55
  # return message if client is allowed to see this message
56
56
  # allows you to inject server side filtering of messages based on arbitrary rules
57
- # also allows you to override the message a client will see
57
+ # also allows you to override the message a clients will see
58
58
  # be sure to .dup the message if you wish to change it
59
59
  end
60
60
 
@@ -96,7 +96,6 @@ MessageBus.subscribe("/channel", function(data){
96
96
 
97
97
  ```
98
98
 
99
-
100
99
  ## Similar projects
101
100
 
102
101
  Faye - http://faye.jcoglan.com/
@@ -1,4 +1,5 @@
1
1
  require 'message_bus'
2
+ require 'stackprof'
2
3
 
3
4
  if defined?(PhusionPassenger)
4
5
  PhusionPassenger.on_event(:starting_worker_process) do |forked|
@@ -17,8 +18,14 @@ end
17
18
  # Rack::MiniProfiler.config.storage = Rack::MiniProfiler::MemoryStore
18
19
 
19
20
  # use Rack::MiniProfiler
21
+ # StackProf.start(mode: :cpu)
22
+ # Thread.new {
23
+ # sleep 10
24
+ # StackProf.stop
25
+ # File.write("test.prof",Marshal.dump(StackProf.results))
26
+ # }
27
+
20
28
  MessageBus.long_polling_interval = 1000 * 2
21
- MessageBus.rack_hijack_enabled = true
22
- MessageBus.max_active_clients = 3
29
+ MessageBus.max_active_clients = 10000
23
30
  use MessageBus::Rack::Middleware
24
31
  run lambda { |env| [200, {"Content-Type" => "text/html"}, ["Howdy"]] }
@@ -24,13 +24,19 @@ class Chat < Sinatra::Base
24
24
  <head>
25
25
  <script src="/jquery-1.8.2.js"></script>
26
26
  <script src="/message-bus.js"></script>
27
+ <style>
28
+ #panel { position: fixed; bottom: 0; background-color: #FFFFFF; }
29
+ #messages { padding-bottom: 40px; }
30
+ .hidden { display: none; }
31
+ </style>
27
32
  </head>
28
33
  <body>
29
- <p>Chat Demo</p>
30
- <div id='messages'></div>
31
- <div id='panel'>
34
+ <p>This is a trivial chat demo... It is implemented as a <a href="https://github.com/SamSaffron/message_bus/blob/master/examples/chat/chat.rb">Sinatra app</a>. The <a href="https://github.com/SamSaffron/message_bus">message_bus</a> can easily be added to any Rails/Rack app. <small>This app can be deployed with <a href="https://github.com/discourse/discourse_docker">Discourse Docker</a> using <a href="https://github.com/SamSaffron/message_bus/blob/master/examples/chat/docker_container/chat.yml">this template</a>.</small></p>
35
+ <div id='messages' class="hidden"></div>
36
+ <div id='panel' class="hidden">
32
37
  <form>
33
38
  <textarea cols=80 rows=2></textarea>
39
+ <button id="send">send</button>
34
40
  </form>
35
41
  </div>
36
42
  <div id='your-name'>Enter your name: <input type='text'/>
@@ -39,26 +45,40 @@ class Chat < Sinatra::Base
39
45
  $(function() {
40
46
  var name;
41
47
 
42
- $('#messages, #panel').hide();
48
+ $('#messages, #panel').addClass('hidden');
43
49
 
44
50
  $('#your-name input').keyup(function(e){
45
51
  if(e.keyCode == 13) {
46
52
  name = $(this).val();
47
- $('#your-name').hide();
48
- $('#messages, #panel').show();
53
+ $('#your-name').addClass('hidden');
54
+ $('#messages, #panel').removeClass('hidden');
55
+ $('#send').text("send (" + name + ")");
49
56
  }
50
57
  });
51
58
 
59
+ var safe = function(html){
60
+ return String(html).replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;').replace(/"/g, '&quot;');
61
+ };
52
62
 
53
63
  MessageBus.subscribe("/message", function(msg){
54
- $('#messages').append("<p>"+ msg.name + " said: " + msg.data + "</p>");
64
+ $('#messages').append("<p>"+ safe(msg.name) + " said: " + safe(msg.data) + "</p>");
65
+ $(document.body).scrollTop(document.body.scrollHeight);
55
66
  }, 0); // last id is zero, so getting backlog
56
67
 
68
+ var submit = function(){
69
+ var val = $('form textarea').val().trim();
70
+ if (val.length === 0) {
71
+ return;
72
+ }
73
+ $.post("/message", { data: val, name: name} );
74
+ $('textarea').val("");
75
+ };
76
+
77
+ $('#send').click(function(){submit(); return false;});
57
78
 
58
79
  $('textarea').keyup(function(e){
59
80
  if(e.keyCode == 13) {
60
- $.post("/message", { data: $('form textarea').val(), name: name} );
61
- $('textarea').val("");
81
+ submit();
62
82
  }
63
83
  });
64
84
 
@@ -0,0 +1,80 @@
1
+ base_image: "samsaffron/discourse_base:1.0.11"
2
+
3
+ update_pups: false
4
+
5
+ params:
6
+ home: /var/www/message_bus
7
+
8
+ templates:
9
+ - "templates/redis.template.yml"
10
+
11
+ expose:
12
+ - "80:80"
13
+
14
+ volumes:
15
+ - volume:
16
+ host: /var/docker/shared/chat
17
+ guest: /shared
18
+
19
+ hooks:
20
+ after_redis:
21
+ - exec:
22
+ cmd:
23
+ - useradd chat -s /bin/bash -m -U
24
+ - exec:
25
+ background: true
26
+ cmd: "sudo -u redis /usr/bin/redis-server /etc/redis/redis.conf --dbfilename test.rdb"
27
+ - exec: mkdir -p /var/www
28
+ - exec: cd /var/www && git clone --depth 1 https://github.com/SamSaffron/message_bus.git
29
+ - exec:
30
+ cmd:
31
+ - gem install unicorn
32
+ - gem install redis
33
+ - gem install sinatra
34
+ - gem install eventmachine
35
+ - file:
36
+ path: /etc/service/unicorn/run
37
+ chmod: "+x"
38
+ contents: |
39
+ #!/bin/bash
40
+ exec 2>&1
41
+ # redis
42
+ cd $home/examples/chat
43
+ exec sudo -E -u chat LD_PRELOAD=/usr/lib/libjemalloc.so.1 unicorn -p 8080 -E production
44
+ - exec: rm /etc/nginx/sites-enabled/default
45
+ - replace:
46
+ filename: /etc/nginx/nginx.conf
47
+ from: pid /run/nginx.pid;
48
+ to: daemon off;
49
+ - file:
50
+ path: /etc/nginx/conf.d/chat.conf
51
+ contents: |
52
+ upstream chat {
53
+ server localhost:8080;
54
+ }
55
+ server {
56
+ listen 80;
57
+ gzip on;
58
+ gzip_types application/json text/css application/x-javascript;
59
+ gzip_min_length 1000;
60
+ server_name chat.samsaffron.com;
61
+ keepalive_timeout 65;
62
+ location / {
63
+ try_files $uri @chat;
64
+ }
65
+ location @chat{
66
+ proxy_set_header Host $http_host;
67
+ proxy_set_header X-Real-IP $remote_addr;
68
+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
69
+ proxy_set_header X-Forwarded-Proto http;
70
+ proxy_pass http://chat;
71
+ }
72
+ }
73
+ - file:
74
+ path: /etc/service/nginx/run
75
+ chmod: "+x"
76
+ contents: |
77
+ #!/bin/sh
78
+ exec 2>&1
79
+ exec /usr/sbin/nginx
80
+
@@ -0,0 +1,9 @@
1
+ #!/bin/bash
2
+
3
+ result=`cd /root/logster && git pull | grep "Already up-to-date"`
4
+
5
+ if [ -z "$result" ]; then
6
+ echo "updating..."
7
+ cd /var/docker && ./launcher bootstrap logster && ./launcher destroy logster && ./launcher start logster
8
+ fi
9
+
data/lib/message_bus.rb CHANGED
@@ -1,8 +1,5 @@
1
- # require 'thin'
2
- # require 'eventmachine'
3
- # require 'rack'
4
- # require 'redis'
5
-
1
+ require "monitor"
2
+ require "set"
6
3
  require "message_bus/version"
7
4
  require "message_bus/message"
8
5
  require "message_bus/reliable_pub_sub"
@@ -12,7 +9,6 @@ require "message_bus/message_handler"
12
9
  require "message_bus/diagnostics"
13
10
  require "message_bus/rack/middleware"
14
11
  require "message_bus/rack/diagnostics"
15
- require "monitor.rb"
16
12
 
17
13
  # we still need to take care of the logger
18
14
  if defined?(::Rails)
@@ -367,7 +363,11 @@ module MessageBus::Implementation
367
363
 
368
364
  def new_subscriber_thread
369
365
  Thread.new do
370
- global_subscribe_thread unless @destroyed
366
+ begin
367
+ global_subscribe_thread unless @destroyed
368
+ rescue => e
369
+ MessageBus.logger.warn "Unexpected error in subscriber thread #{e}"
370
+ end
371
371
  end
372
372
  end
373
373
 
@@ -99,15 +99,28 @@ class MessageBus::Client
99
99
 
100
100
  protected
101
101
 
102
+
103
+ # heavily optimised to avoid all uneeded allocations
104
+ NEWLINE="\r\n".freeze
105
+ COLON_SPACE = ": ".freeze
106
+ HTTP_11 = "HTTP/1.1 200 OK\r\n".freeze
107
+ CONTENT_LENGTH = "Content-Length: ".freeze
108
+ CONNECTION_CLOSE = "Connection: close\r\n".freeze
109
+
102
110
  def write_and_close(data)
103
111
  if @io
104
- @io.write("HTTP/1.1 200 OK\r\n")
112
+ @io.write(HTTP_11)
105
113
  @headers.each do |k,v|
106
- @io.write("#{k}: #{v}\r\n")
114
+ @io.write(k)
115
+ @io.write(COLON_SPACE)
116
+ @io.write(v)
117
+ @io.write(NEWLINE)
107
118
  end
108
- @io.write("Content-Length: #{data.bytes.to_a.length}\r\n")
109
- @io.write("Connection: close\r\n")
110
- @io.write("\r\n")
119
+ @io.write(CONTENT_LENGTH)
120
+ @io.write(data.bytes.to_a.length)
121
+ @io.write(NEWLINE)
122
+ @io.write(CONNECTION_CLOSE)
123
+ @io.write(NEWLINE)
111
124
  @io.write(data)
112
125
  @io.close
113
126
  @io = nil
@@ -5,22 +5,33 @@ class MessageBus::Rack::Middleware
5
5
 
6
6
  def start_listener
7
7
  unless @started_listener
8
+ require 'message_bus/timer_thread'
8
9
 
9
- require 'eventmachine'
10
- require 'message_bus/em_ext'
10
+ @timer_thread = MessageBus::TimerThread.new
11
+ @timer_thread.on_error do |e|
12
+ @bus.logger.warn "Failed to process job: #{e} #{e.backtrace}"
13
+ end
14
+
15
+ thin = defined?(Thin::Server) && ObjectSpace.each_object(Thin::Server).to_a.first
16
+ thin_running = thin && thin.running?
11
17
 
12
18
  @subscription = @bus.subscribe do |msg|
13
- if EM.reactor_running?
14
- EM.next_tick do
15
- begin
16
- @connection_manager.notify_clients(msg) if @connection_manager
17
- rescue
18
- @bus.logger.warn "Failed to notify clients: #{$!} #{$!.backtrace}"
19
- end
19
+ run = proc do
20
+ begin
21
+ @connection_manager.notify_clients(msg) if @connection_manager
22
+ rescue
23
+ @bus.logger.warn "Failed to notify clients: #{$!} #{$!.backtrace}"
20
24
  end
21
25
  end
26
+
27
+ if thin_running
28
+ EM.next_tick(&run)
29
+ else
30
+ @timer_thread.queue(&run)
31
+ end
32
+
33
+ @started_listener = true
22
34
  end
23
- @started_listener = true
24
35
  end
25
36
  end
26
37
 
@@ -34,6 +45,7 @@ class MessageBus::Rack::Middleware
34
45
  def stop_listener
35
46
  if @subscription
36
47
  @bus.unsubscribe(&@subscription)
48
+ @timer_thread.stop if @timer_thread
37
49
  @started_listener = false
38
50
  end
39
51
  end
@@ -101,11 +113,8 @@ class MessageBus::Rack::Middleware
101
113
  return [200, headers, ["OK"]]
102
114
  end
103
115
 
104
- ensure_reactor
105
-
106
116
  long_polling = @bus.long_polling_enabled? &&
107
117
  env['QUERY_STRING'] !~ /dlp=t/.freeze &&
108
- EM.reactor_running? &&
109
118
  @connection_manager.client_count < @bus.max_active_clients
110
119
 
111
120
  if backlog.length > 0
@@ -152,22 +161,10 @@ class MessageBus::Rack::Middleware
152
161
  end
153
162
  end
154
163
 
155
- def ensure_reactor
156
- # ensure reactor is running
157
- if EM.reactor_pid != Process.pid
158
- Thread.new { EM.run }
159
- i = 100
160
- while !EM.reactor_running? && i > 0
161
- sleep 0.001
162
- i -= 1
163
- end
164
- end
165
- end
166
-
167
164
  def add_client_with_timeout(client)
168
165
  @connection_manager.add_client(client)
169
166
 
170
- client.cleanup_timer = ::EM::Timer.new( @bus.long_polling_interval.to_f / 1000) {
167
+ client.cleanup_timer = @timer_thread.queue( @bus.long_polling_interval.to_f / 1000) {
171
168
  begin
172
169
  client.cleanup_timer = nil
173
170
  client.ensure_closed!
@@ -10,6 +10,8 @@ require 'redis'
10
10
 
11
11
  class MessageBus::ReliablePubSub
12
12
  attr_reader :subscribed
13
+ attr_accessor :max_publish_retries, :max_publish_wait, :max_backlog_size,
14
+ :max_global_backlog_size, :max_in_memory_publish_backlog
13
15
 
14
16
  UNSUB_MESSAGE = "$$UNSUBSCRIBE"
15
17
 
@@ -22,38 +24,18 @@ class MessageBus::ReliablePubSub
22
24
  end
23
25
  end
24
26
 
25
- def max_publish_retries=(val)
26
- @max_publish_retries = val
27
- end
28
-
29
- def max_publish_retries
30
- @max_publish_retries ||= 10
31
- end
32
-
33
- def max_publish_wait=(ms)
34
- @max_publish_wait = ms
35
- end
36
-
37
- def max_publish_wait
38
- @max_publish_wait ||= 500
39
- end
40
-
41
27
  # max_backlog_size is per multiplexed channel
42
28
  def initialize(redis_config = {}, max_backlog_size = 1000)
43
29
  @redis_config = redis_config
44
30
  @max_backlog_size = 1000
45
- # we can store a ton here ...
31
+ # we can store a lot of messages, since only one queue
46
32
  @max_global_backlog_size = 100000
47
- end
48
-
49
- # amount of global backlog we can spin through
50
- def max_global_backlog_size=(val)
51
- @max_global_backlog_size = val
52
- end
53
-
54
- # per channel backlog size
55
- def max_backlog_size=(val)
56
- @max_backlog_size = val
33
+ @max_publish_retries = 10
34
+ @max_publish_wait = 500 #ms
35
+ @max_in_memory_publish_backlog = 1000
36
+ @in_memory_backlog = []
37
+ @lock = Mutex.new
38
+ @flush_backlog_thread = nil
57
39
  end
58
40
 
59
41
  def new_redis_connection
@@ -97,7 +79,7 @@ class MessageBus::ReliablePubSub
97
79
  end
98
80
  end
99
81
 
100
- def publish(channel, data)
82
+ def publish(channel, data, queue_in_memory=true)
101
83
  redis = pub_redis
102
84
  backlog_id_key = backlog_id_key(channel)
103
85
  backlog_key = backlog_key(channel)
@@ -130,6 +112,65 @@ class MessageBus::ReliablePubSub
130
112
  end
131
113
 
132
114
  backlog_id
115
+
116
+ rescue Redis::CommandError => e
117
+ if queue_in_memory &&
118
+ e.message =~ /^READONLY/
119
+
120
+ @lock.synchronize do
121
+ @in_memory_backlog << [channel,data]
122
+ if @in_memory_backlog.length > @max_in_memory_publish_backlog
123
+ @in_memory_backlog.delete_at(0)
124
+ MessageBus.logger.warn("Dropping old message cause max_in_memory_publish_backlog is full")
125
+ end
126
+ end
127
+
128
+ if @flush_backlog_thread == nil
129
+ @lock.synchronize do
130
+ if @flush_backlog_thread == nil
131
+ @flush_backlog_thread = Thread.new{ensure_backlog_flushed}
132
+ end
133
+ end
134
+ end
135
+ nil
136
+ else
137
+ raise
138
+ end
139
+ end
140
+
141
+ def ensure_backlog_flushed
142
+ while true
143
+ try_again = false
144
+
145
+ @lock.synchronize do
146
+ break if @in_memory_backlog.length == 0
147
+
148
+ begin
149
+ publish(*@in_memory_backlog[0],false)
150
+ rescue Redis::CommandError => e
151
+ if e.message =~ /^READONLY/
152
+ try_again = true
153
+ else
154
+ MessageBus.logger.warn("Dropping undeliverable message #{e}")
155
+ end
156
+ rescue => e
157
+ MessageBus.logger.warn("Dropping undeliverable message #{e}")
158
+ end
159
+
160
+ @in_memory_backlog.delete_at(0) unless try_again
161
+ end
162
+
163
+ if try_again
164
+ sleep 0.005
165
+ # in case we are not connected to the correct server
166
+ # which can happen when sharing ips
167
+ pub_redis.client.reconnect
168
+ end
169
+ end
170
+ ensure
171
+ @lock.synchronize do
172
+ @flush_backlog_thread = nil
173
+ end
133
174
  end
134
175
 
135
176
  def last_id(channel)
@@ -197,6 +238,10 @@ class MessageBus::ReliablePubSub
197
238
  end
198
239
 
199
240
  def process_global_backlog(highest_id, raise_error, &blk)
241
+ if highest_id > pub_redis.get(global_id_key).to_i
242
+ highest_id = 0
243
+ end
244
+
200
245
  global_backlog(highest_id).each do |old|
201
246
  if highest_id + 1 == old.global_id
202
247
  yield old
@@ -209,6 +254,7 @@ class MessageBus::ReliablePubSub
209
254
  end
210
255
  end
211
256
  end
257
+
212
258
  highest_id
213
259
  end
214
260
 
@@ -263,10 +309,11 @@ class MessageBus::ReliablePubSub
263
309
  end
264
310
  m = MessageBus::Message.decode m
265
311
 
266
- # we have 2 options
312
+ # we have 3 options
267
313
  #
268
314
  # 1. message came in the correct order GREAT, just deal with it
269
315
  # 2. message came in the incorrect order COMPLICATED, wait a tiny bit and clear backlog
316
+ # 3. message came in the incorrect order and is lowest than current highest id, reset
270
317
 
271
318
  if highest_id.nil? || m.global_id == highest_id + 1
272
319
  highest_id = m.global_id
@@ -0,0 +1,87 @@
1
+ class MessageBus::TimerThread
2
+
3
+ attr_reader :jobs
4
+
5
+ class Cancelable
6
+ NOOP = proc{}
7
+
8
+ def initialize(job)
9
+ @job = job
10
+ end
11
+ def cancel
12
+ @job[1] = NOOP
13
+ end
14
+ end
15
+
16
+ def initialize
17
+ @stopped = false
18
+ @jobs = []
19
+ @mutex = Mutex.new
20
+ @next = nil
21
+ @thread = Thread.new{do_work}
22
+ @on_error = lambda{|e| STDERR.puts "Exception while processing Timer:\n #{e.backtrace.join("\n")}"}
23
+ end
24
+
25
+ def stop
26
+ @stopped = true
27
+ end
28
+
29
+ # queue a block to run after a certain delay (in seconds)
30
+ def queue(delay=0, &block)
31
+ queue_time = Time.new.to_f + delay
32
+ job = [queue_time, block]
33
+
34
+ @mutex.synchronize do
35
+ i = @jobs.length
36
+ while i > 0
37
+ i -= 1
38
+ current,_ = @jobs[i]
39
+ i+=1 and break if current < queue_time
40
+ end
41
+ @jobs.insert(i, job)
42
+ @next = queue_time if i==0
43
+ end
44
+
45
+ unless @thread.alive?
46
+ @mutex.synchronize do
47
+ @thread = Thread.new{do_work} unless @thread.alive?
48
+ end
49
+ end
50
+
51
+ if @thread.status == "sleep".freeze
52
+ @thread.wakeup
53
+ end
54
+
55
+ Cancelable.new(job)
56
+ end
57
+
58
+ def on_error(&block)
59
+ @on_error = block
60
+ end
61
+
62
+ protected
63
+
64
+ def do_work
65
+ while !@stopped
66
+ if @next && @next <= Time.new.to_f
67
+ _,blk = @jobs.shift
68
+ begin
69
+ blk.call
70
+ rescue => e
71
+ @on_error.call(e) if @on_error
72
+ end
73
+ @mutex.synchronize do
74
+ @next,_ = @jobs[0]
75
+ end
76
+ end
77
+ unless @next && @next <= Time.new.to_f
78
+ sleep_time = 1000
79
+ @mutex.synchronize do
80
+ sleep_time = @next-Time.new.to_f if @next
81
+ end
82
+ sleep [0,sleep_time].max
83
+ end
84
+ end
85
+ end
86
+
87
+ end
@@ -1,3 +1,3 @@
1
1
  module MessageBus
2
- VERSION = "1.0.9"
2
+ VERSION = "1.0.10"
3
3
  end
data/message_bus.gemspec CHANGED
@@ -16,6 +16,5 @@ Gem::Specification.new do |gem|
16
16
  gem.require_paths = ["lib"]
17
17
  gem.version = MessageBus::VERSION
18
18
  gem.add_runtime_dependency 'rack', '>= 1.1.3'
19
- gem.add_runtime_dependency 'eventmachine'
20
19
  gem.add_runtime_dependency 'redis'
21
20
  end
@@ -6,7 +6,13 @@ describe MessageBus::Client do
6
6
  describe "subscriptions" do
7
7
 
8
8
  before do
9
- @client = MessageBus::Client.new :client_id => 'abc'
9
+ @bus = MessageBus::Instance.new
10
+ @client = MessageBus::Client.new client_id: 'abc', message_bus: @bus
11
+ end
12
+
13
+ after do
14
+ @bus.reset!
15
+ @bus.destroy
10
16
  end
11
17
 
12
18
  it "should provide a list of subscriptions" do
@@ -16,7 +22,7 @@ describe MessageBus::Client do
16
22
 
17
23
  it "should provide backlog for subscribed channel" do
18
24
  @client.subscribe('/hello', nil)
19
- MessageBus.publish '/hello', 'world'
25
+ @bus.publish '/hello', 'world'
20
26
  log = @client.backlog
21
27
  log.length.should == 1
22
28
  log[0].channel.should == '/hello'
@@ -18,6 +18,26 @@ describe MessageBus do
18
18
  @bus.destroy
19
19
  end
20
20
 
21
+ it "should recover from a redis flush" do
22
+
23
+ data = nil
24
+ @bus.subscribe("/chuck") do |msg|
25
+ data = msg.data
26
+ end
27
+ @bus.publish("/chuck", {:norris => true})
28
+ @bus.publish("/chuck", {:norris => true})
29
+ @bus.publish("/chuck", {:norris => true})
30
+
31
+ @bus.reliable_pub_sub.pub_redis.flushall
32
+
33
+ @bus.publish("/chuck", {:yeager => true})
34
+
35
+ wait_for(2000){ data["yeager"]}
36
+
37
+ data["yeager"].should == true
38
+
39
+ end
40
+
21
41
  it "should automatically decode hashed messages" do
22
42
  data = nil
23
43
  @bus.subscribe("/chuck") do |msg|
@@ -147,7 +167,9 @@ describe MessageBus do
147
167
  end
148
168
 
149
169
  @bus.publish("/hello", "world")
170
+
150
171
  wait_for(2000){ data }
172
+
151
173
  if child = Process.fork
152
174
  wait_for(2000) { data == "ready" }
153
175
  @bus.publish("/hello", "world1")
@@ -164,7 +186,10 @@ describe MessageBus do
164
186
 
165
187
  $stdout.reopen("/dev/null", "w")
166
188
  $stderr.reopen("/dev/null", "w")
167
- exit
189
+
190
+ # having some issues with exit here
191
+ # TODO find and fix
192
+ Process.kill "KILL", Process.pid
168
193
  end
169
194
 
170
195
  end
@@ -8,8 +8,6 @@ describe MessageBus::ReliablePubSub do
8
8
  end
9
9
 
10
10
  def work_it
11
- Signal.trap("HUP") { exit }
12
-
13
11
  bus = new_bus
14
12
  $stdout.reopen("/dev/null", "w")
15
13
  $stderr.reopen("/dev/null", "w")
@@ -29,7 +27,6 @@ describe MessageBus::ReliablePubSub do
29
27
  end
30
28
 
31
29
  it 'gets every response from child processes' do
32
- pid = nil
33
30
  Redis.new(:db => 10).flushdb
34
31
  begin
35
32
  pids = (1..10).map{spawn_child}
@@ -51,7 +48,7 @@ describe MessageBus::ReliablePubSub do
51
48
  ensure
52
49
  if pids
53
50
  pids.each do |pid|
54
- Process.kill("HUP", pid)
51
+ Process.kill("KILL", pid)
55
52
  Process.wait(pid)
56
53
  end
57
54
  end
@@ -12,6 +12,31 @@ describe MessageBus::ReliablePubSub do
12
12
  @bus.reset!
13
13
  end
14
14
 
15
+ context "readonly" do
16
+ before do
17
+ @bus.pub_redis.slaveof "127.0.0.80", "666"
18
+ end
19
+
20
+ after do
21
+ @bus.pub_redis.slaveof "no", "one"
22
+ end
23
+
24
+ it "should be able to store messages in memory for a period while in read only" do
25
+ @bus.max_in_memory_publish_backlog = 2
26
+
27
+ 3.times do
28
+ result = @bus.publish "/foo", "bar"
29
+ result.should == nil
30
+ end
31
+
32
+ @bus.pub_redis.slaveof "no", "one"
33
+ sleep 0.01
34
+
35
+ @bus.backlog("/foo", 0).map(&:data).should == ["bar","bar"]
36
+
37
+ end
38
+ end
39
+
15
40
  it "should be able to access the backlog" do
16
41
  @bus.publish "/foo", "bar"
17
42
  @bus.publish "/foo", "baz"
@@ -0,0 +1,65 @@
1
+ require 'spec_helper'
2
+ require 'message_bus/timer_thread'
3
+
4
+ describe MessageBus::TimerThread do
5
+ before do
6
+ @timer = MessageBus::TimerThread.new
7
+ end
8
+
9
+ after do
10
+ @timer.stop
11
+ end
12
+
13
+ it "allows you to cancel timers" do
14
+ success = true
15
+ @timer.queue(0.005){success=false}.cancel
16
+ sleep(0.006)
17
+ success.should == true
18
+ end
19
+
20
+ it "queues jobs in the correct order" do
21
+ counter = 0
22
+ failed = nil
23
+
24
+ items = (0..5).to_a.shuffle
25
+ items.map do |i|
26
+ # threading introduces a delay meaning we need to wait a long time
27
+ Thread.new do
28
+ @timer.queue(i/5.0) do
29
+ failed = true if counter != i
30
+ counter += 1
31
+ end
32
+ end
33
+ end
34
+
35
+ wait_for(1500) {
36
+ counter == items.length
37
+ }
38
+
39
+ counter.should == items.length
40
+ failed.should == nil
41
+ end
42
+
43
+ it "should call the error callback if something goes wrong" do
44
+ error = nil
45
+
46
+ @timer.queue do
47
+ boom
48
+ end
49
+
50
+ @timer.on_error do |e|
51
+ error = e
52
+ end
53
+
54
+ @timer.queue do
55
+ boom
56
+ end
57
+
58
+ wait_for(10) do
59
+ error
60
+ end
61
+
62
+ error.class.should == NameError
63
+ end
64
+
65
+ end
data/spec/spec_helper.rb CHANGED
@@ -8,6 +8,11 @@ RSpec.configure do |config|
8
8
  config.mock_with :rspec do |mocks|
9
9
  mocks.syntax = :should
10
10
  end
11
+
12
+ # to debug hanging tests
13
+ # config.before :each do |x|
14
+ # p x.metadata[:location]
15
+ # end
11
16
  end
12
17
 
13
18
  def wait_for(timeout_milliseconds)
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: message_bus
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.9
4
+ version: 1.0.10
5
5
  platform: ruby
6
6
  authors:
7
7
  - Sam Saffron
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2015-03-23 00:00:00.000000000 Z
11
+ date: 2015-05-03 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: rack
@@ -24,20 +24,6 @@ dependencies:
24
24
  - - ">="
25
25
  - !ruby/object:Gem::Version
26
26
  version: 1.1.3
27
- - !ruby/object:Gem::Dependency
28
- name: eventmachine
29
- requirement: !ruby/object:Gem::Requirement
30
- requirements:
31
- - - ">="
32
- - !ruby/object:Gem::Version
33
- version: '0'
34
- type: :runtime
35
- prerelease: false
36
- version_requirements: !ruby/object:Gem::Requirement
37
- requirements:
38
- - - ">="
39
- - !ruby/object:Gem::Version
40
- version: '0'
41
27
  - !ruby/object:Gem::Dependency
42
28
  name: redis
43
29
  requirement: !ruby/object:Gem::Requirement
@@ -80,6 +66,8 @@ files:
80
66
  - examples/bench/unicorn.conf.rb
81
67
  - examples/chat/chat.rb
82
68
  - examples/chat/config.ru
69
+ - examples/chat/docker_container/chat.yml
70
+ - examples/chat/docker_container/update_chat
83
71
  - examples/minimal/Gemfile
84
72
  - examples/minimal/config.ru
85
73
  - lib/message_bus.rb
@@ -94,6 +82,7 @@ files:
94
82
  - lib/message_bus/rack/thin_ext.rb
95
83
  - lib/message_bus/rails/railtie.rb
96
84
  - lib/message_bus/reliable_pub_sub.rb
85
+ - lib/message_bus/timer_thread.rb
97
86
  - lib/message_bus/version.rb
98
87
  - message_bus.gemspec
99
88
  - spec/lib/assets/asset_encoding_spec.rb
@@ -106,6 +95,7 @@ files:
106
95
  - spec/lib/middleware_spec.rb
107
96
  - spec/lib/multi_process_spec.rb
108
97
  - spec/lib/reliable_pub_sub_spec.rb
98
+ - spec/lib/timer_thread_spec.rb
109
99
  - spec/spec_helper.rb
110
100
  - vendor/assets/javascripts/message-bus.js
111
101
  homepage: https://github.com/SamSaffron/message_bus
@@ -128,7 +118,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
128
118
  version: '0'
129
119
  requirements: []
130
120
  rubyforge_project:
131
- rubygems_version: 2.2.2
121
+ rubygems_version: 2.4.5
132
122
  signing_key:
133
123
  specification_version: 4
134
124
  summary: ''
@@ -143,4 +133,5 @@ test_files:
143
133
  - spec/lib/middleware_spec.rb
144
134
  - spec/lib/multi_process_spec.rb
145
135
  - spec/lib/reliable_pub_sub_spec.rb
136
+ - spec/lib/timer_thread_spec.rb
146
137
  - spec/spec_helper.rb