gilmour 0.3.4 → 0.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: c95ad64d9f1f755d37f31ebfc42d86f480f609b3
-  data.tar.gz: 4a6e00ddbf3eacc9eb31beee7319bd54f4152f53
+  metadata.gz: e019128d85d1f18f07ee9c90a8444950e17e77e4
+  data.tar.gz: 424ecd2839a45bd7f89ed8e5abf9c256315200bc
 SHA512:
-  metadata.gz: 0c94da1ec0e755bd6106d62bf938dd40b88eb6d845f88ea754c8505f750f6433b27d4bc2842eb260f6faf92999bd140f7b3ed3f22628f33c0ace29982353b5c6
-  data.tar.gz: 0e029a1923918f2374d0a3476693f267dd3f724e44e032ce4cf194313c709c0f0db2790a5c97b4420d23c32aa9288227523555442a30bfb0a823646aa8f1afe2
+  metadata.gz: 4bd6a702576f5cc27f791a6b25f6520dd08ae3212fa1ab1adf056976e8086f44de057e37aef55bbac3ae9ad278efe690b23adadf3a285d1600c8c39f870fd012
+  data.tar.gz: 2540aabffac9e90f958d28170ce143fd5194efb01ed77ad5dfbdc2d0e17e6337c12191491f7700f04b8024c1e178d4950b1268f5e3e91949c3a18302d5d948b8

data/.travis.yml

@@ -14,3 +14,4 @@ script:
   - bundle exec rspec spec/test_service_base.rb -b --format documentation
   - bundle exec rspec spec/test_subscriber_redis.rb -b --format documentation
   - bundle exec rspec spec/test_subscriber_redis_forked.rb -b --format documentation
+  - bundle exec rspec spec/test_waiter.rb -b --format documentation

data/README.md

@@ -3,7 +3,35 @@
 Gilmour is a framework for writing micro-services that exchange data over
 non-http transports. Currently the supported backend is Redis PubSub.
 Redis pubsub channels are used like "routes".
-The DSL provided is similar to Sinatra.
+Gilmour started off simply as a non-http alternative to Sinatra, but has grown into a feature rich microservices communication library and framework.
+
+## Patterns
+
+Gilmour enables two patterns of communication:
+
+### The request-response pattern
+This is the most common pattern found in microservice architectures. An entity
+sends a `request`, and a subscriber processes it and sends a `reply`. See
+`examples/fibonacci.rb` as an example.  Gilmour also allows horizontal scaling
+by using __exclusion groups__. When there are multiple instances of a subscriber running, only one subscriber from an exclusion group processes that request.
+Eg, if you run `container.rb` from the examples multiple times simultaneously, running `echoclient.rb` or `forkechoclient.rb` will produce the same result, irrespective of how many instances of the container are running. (More of containers later).
+
+### The signal-slot pattern
+A less common, but very powerful pattern is event driven service design, or the signal-slot pattern (a term borrowed from the Qt library). One service emits or signals events, which are picked up by one or more subscribers and processed. None of the subscribers send a reply, because a reply really has no meaning in this context. See `examples/signal_slot.rb`.
+
+Which pattern to chose, is primarily dependent on the ownership of failure. If the subscribers own the failures they encounter, then you should consider the signal-slot pattern.
+
+It is also possible to have slots as well as reply subscribers for a given message. Gilmour provides a utility `broadcast` which internally does a signal and a request.
+
+## Composition
+
+Microservices increase the granularity of our services oriented architectures. Borrowing from unix philosohy, they should do one thing and do it well. However, for this to be really useful, there should be a facility such as is provided by unix shells. Unix shells allow the composition of small commands using the following methods
+
+* Composition: `cmd1 | cmd2 | cmd2`
+* AndAnd: `cmd1 && cmd2 && cmd3`
+* Batch: `cmd1; cmd2; cmd3 > out` or `(cmd1; cmd2; cmd3) > out`
+
+Also, you should be able to use these methods of composition in any combination and also in a nexted manner - `(cmd1 | cmd2) && cmd3`. Gilmour enables you to do just that. See `examples/composition.rb`.
 
 ## Protocol
 
@@ -13,17 +41,44 @@ The structure of the payload is a simple JSON as shown below:
     {
 	  data: The actual payload,
 	  sender: The origin of the request (unique for each request),
-	  code: The respoonse code if this is a response
+	  code: The response code if this is a response. This uses HTTP error codes
     }
 
+Gilmour has builtin support for error publishing. The error channel is separate from the response channels and everything that is put on the error channel is prefixed by the per-request unique sender uuid, which makes debugging easier. The error packet protocol is
+
+    {
+		code: response code (non-200)
+		sender: the "sender" from the request
+		topic: the topic to which the request was sent
+		request_data: the request payload
+		userdata: implementation dependent debug infomation
+		backtrace: the backtrace of the error
+		timestamp: the timestamp of the error
+    }
+
+## Redis backend conventions
+
 The `sender` field actually represents a unique sender key. Any reponse that
 is to be sent to the request is sent on the "reservered" topic
 `response.<sender>` on the same exchange.
 
-## Usage Examples
+For the request-reply pattern, the topic is prefixed with `gilmour.request`.
+For the signal-slot pettern, the topic is prefixed with `gilmour.slot`.
+
+The topic on which errors are published is `gilmour.error`.
+
+## Auto-loading
+
+Gilmour code can be structured such that the subscribers are auto-loaded from the filesystem. This enables creation of containers which house multiple microservices, with a choice of which ones to enable. See `examples/container.rb`.
+
+## Health monitoring
+
+Gilmour supports health monitoring heartbeats. Every gilmour process, which has health monitoring enabled, listens for hearbeat pings and responds accordingly. External monitors can use this to monitor the health. [The health bulletin](https://github.com/gilmour-libs/health-bulletin) and [the health-tools](	https://github.com/gilmour-libs/health-tools) are examples of such tools.
 
-See the `examples` directory for examples of usage
+## More Usage Examples
 
+See the `examples` directory 
+  
 ## Specs
 
 To run the specs, set `REDIS_HOST` (default is localhost) and `REDIS_PORT` (default is 6379)

data/examples/composition.rb

@@ -0,0 +1,98 @@
+require 'gilmour'
+
+class Server
+  include Gilmour::Base
+end
+server = Server.new
+gilmour = server.enable_backend('redis')
+
+gilmour.reply_to 'one' do
+  respond request.body.merge({'one' => 'foo'})
+end
+
+gilmour.reply_to 'two' do
+  respond request.body.merge({'two' => 'foo'})
+end
+
+gilmour.reply_to 'badtwo' do
+  respond request.body.merge({'badtwo' => 'foo'}), 500
+end
+
+gilmour.reply_to 'three' do
+  respond request.body.merge({'three' => 'foo'})
+end
+
+composed = gilmour.compose([{topic: 'one'}, {topic: 'two'},
+                            {topic: 'three', message: {'anotherthree' => 'anotherthree'}}])
+compose_waiter = Gilmour::Waiter.new
+composed.execute({zero: 'foo'}) do |data, code|
+  puts "composition:"
+  puts data # data will have the keys 'one', 'two', 'three' and 'anotherthree'
+  compose_waiter.signal
+end
+compose_waiter.wait
+
+andand = gilmour.andand([{topic: 'one'}, {topic: 'two'},
+                         {topic: 'three', message: {'anotherthree' => 'anotherthree'}}])
+andand_waiter = Gilmour::Waiter.new
+andand.execute do |data, code|
+  puts "\nandand:"
+  puts data # data will have the keys 'three' and 'anotherthree'
+  andand_waiter.signal
+end
+andand_waiter.wait
+
+
+error_andand = gilmour.andand([{topic: 'one'}, {topic: 'badtwo'},
+                               {topic: 'three', message: {'anotherthree' => 'anotherthree'}}])
+error_andand_waiter = Gilmour::Waiter.new
+continuation = nil
+error_andand.execute do |data, code, c|
+  continuation = c # the part of the pipeline that was not executed
+  puts "\nerror_andand:\n code - #{code}" # should be 500 fron badtwo
+  puts "error_andand:\n data - #{data}" # this will have keys one and badtwo
+  error_andand_waiter.signal
+end
+error_andand_waiter.wait
+
+continuation_waiter = Gilmour::Waiter.new
+continuation.execute do |data, code|
+  puts "\ncontinuation:"
+  puts data # This will have keys three and anotherthree
+  continuation_waiter.signal
+end
+continuation_waiter.wait
+
+
+batch = gilmour.batch([{topic: 'one'}, {topic: 'badtwo'},
+                       {topic: 'three', message: {'anotherthree' => 'anotherthree'}}])
+batch_waiter = Gilmour::Waiter.new
+batch.execute do |data, code|
+  puts "\nbatch:"
+  puts data # this will have keys three and anotherthree (wont abort on badtwo)
+  batch_waiter.signal
+end
+batch_waiter.wait
+
+batch_record =  gilmour.batch([{topic: 'one'}, {topic: 'badtwo'},
+                               {topic: 'three', message: {'anotherthree' => 'anotherthree'}}],
+true) # true will record all responses in an array
+batch_record_waiter = Gilmour::Waiter.new
+batch_record.execute do |data, code|
+  puts "\nbatch with record:"
+  puts data # This is an array
+  batch_record_waiter.signal
+end
+batch_record_waiter.wait
+
+t_andand = gilmour.andand([{topic: 'one'}, {topic: 'two'}])
+# Use the above composition inside another
+compose = gilmour.compose([t_andand, {topic: 'three'}])
+combination_waiter = Gilmour::Waiter.new
+compose.execute do |data, code|
+  puts "\nnested composition"
+  puts data # will have keys two and three
+  combination_waiter.signal
+end
+combination_waiter.wait
+

data/examples/container.rb

@@ -0,0 +1,19 @@
+# encoding: utf-8
+# Example of running as a container with subscribers
+# Run ruby server.rb first, followed by echoclient.rb
+require 'gilmour'
+
+class EventServer
+  include Gilmour::Base
+
+  def initialize
+    enable_backend('redis')
+    registered_subscribers.each do |sub|
+      sub.backend = 'redis'
+    end
+    start(true)
+  end
+  load_all './subscribers'
+end
+
+EventServer.new

data/examples/echoclient.rb

@@ -1,19 +1,23 @@
 require 'securerandom'
-require 'gilmour/backends/redis'
+require 'gilmour'
+
+class Server
+  include Gilmour::Base
+end
+
 
 def redis_send_and_recv(message, key)
-  redis = Gilmour::RedisBackend.new({})
-  redis.setup_subscribers({})
+  server = Server.new
+  gilmour = server.enable_backend('redis')
   count = 0
   loop do
-    waiter = Thread.new { loop { sleep 1 } }
-    newkey = "#{key}.#{SecureRandom.hex(2)}"
-    redis.publish(count, newkey) do |data, code|
+    waiter = Gilmour::Waiter.new
+    gilmour.request!(count, key) do |data, code|
       puts "Client got response: #{code}: #{data}"
-      waiter.kill
+      waiter.signal
     end
     count = count + 1
-    waiter.join
+    waiter.wait
     sleep 1
   end
 end

data/examples/fibonacci.rb

@@ -0,0 +1,32 @@
+require 'gilmour'
+
+class Server
+  include Gilmour::Base
+end
+server = Server.new
+gilmour = server.enable_backend('redis')
+
+gilmour.reply_to "fib", excl_group: 'fib', timeout: 1 do
+  first = request.body['first']
+  second = request.body['second']
+  respond('next' => first + second)
+end
+
+handler = proc do |first, second|
+  first ||= 1
+  second ||= 1
+  gilmour.request!({ 'first' => first, 'second' => second }, 'fib',
+                   timeout: 5) do |resp, code|
+    if code != 200
+      $stderr.puts "Something went wrong in the response! Aborting"
+      exit
+    end
+    next_val = resp['next']
+    puts next_val
+    EM.add_timer(1) { handler.call(second, resp['next']) }
+  end
+end
+
+EM.next_tick { handler.call }
+server.start true
+

data/examples/forkechoclient.rb

@@ -0,0 +1,23 @@
+require 'securerandom'
+require 'gilmour'
+
+class Server
+  include Gilmour::Base
+end
+
+def redis_send_and_recv(message, key)
+  server = Server.new
+  gilmour = server.enable_backend('redis')
+  loop do
+    waiter = Gilmour::Waiter.new
+    puts "Process: Sending: #{key}"
+    gilmour.request!(message, key) do |data, code|
+      puts "Process: Received: #{data}"
+      waiter.signal
+    end
+    waiter.wait
+    sleep 1
+  end
+end
+
+redis_send_and_recv('Ping', 'forkecho')

data/examples/signal_slot.rb

@@ -0,0 +1,35 @@
+# Run this as LOG_LEVEL=info ruby relay.rb
+require 'gilmour'
+
+class Server
+  include Gilmour::Base
+end
+
+server = Server.new
+gilmour = server.enable_backend('redis')
+
+waiter = Gilmour::Waiter.new
+
+waiter.add
+gilmour.slot "user.added" do
+  # This one will send out email notifications
+  logger.info "Sent email notification for -"
+  logger.info request.body
+  waiter.done
+end
+
+waiter.add
+gilmour.slot "user.added" do
+  # This one will send out push notifications
+  logger.info "Sent push notification for -"
+  logger.info request.body
+  waiter.done
+end
+
+EM.next_tick {
+  gilmour.signal!({ username: 'foo', email: 'foo@bar.com' }, 'user.added')
+}
+
+waiter.wait
+
+

data/examples/subscribers/echo.rb

@@ -0,0 +1,22 @@
+class EchoSubscriber < EventServer
+  reply_to 'echo', excl_group: 'echo' do
+    if request.body == 'quit'
+      respond nil
+    else
+      $stderr.puts request.body
+      respond "#{request.topic}"
+    end
+  end
+
+  reply_to 'forkecho', fork:true, excl_group: 'forkecho' do
+    if request.body == 'quit'
+      respond nil
+    else
+      $stderr.puts "Forked response. pid: #{Process.pid}"
+      respond "#{request.topic}"
+    end
+  end
+
+end
+
+

data/lib/gilmour/backends/backend.rb

@@ -3,6 +3,7 @@ require 'socket'
 require 'securerandom'
 
 require_relative '../protocol'
+require_relative '../composers'
 
 module Gilmour
   ErrorChannel = "gilmour.error"
@@ -12,55 +13,59 @@ module Gilmour
     SUPPORTED_BACKENDS = %w(redis)
     @@registry = {}
 
-    def report_errors?
+    include Gilmour::Composers
+    attr_accessor :broadcast_errors
+
+    def report_errors?  #:nodoc:
       #Override this method to adjust if you want errors to be reported.
-      return true
+      return false
     end
 
-    def register_health_check
+    def register_health_check #:nodoc:
       raise NotImplementedError.new
     end
 
-    def unregister_health_check
+    def unregister_health_check #:nodoc:
       raise NotImplementedError.new
     end
 
-    def self.implements(backend_name)
+    def self.implements(backend_name) #:nodoc:
       @@registry[backend_name] = self
     end
 
-    def self.get(backend_name)
+    def self.get(backend_name) #:nodoc:
       @@registry[backend_name]
     end
 
-    # :nodoc:
     # This should be implemented by the derived class
     # subscriptions is a hash in the format -
-    # { topic => [handler1, handler2, ...],
-    #   topic2 => [handler3, handler4, ...],
-    #   ...
-    # }
+    #   { topic => [handler1, handler2, ...],
+    #     topic2 => [handler3, handler4, ...],
+    #     ...
+    #   }
     # where handler is a hash
-    # { :handler => handler_proc,
-    #   :subscriber => subscriber_derived_class
-    # }
-    def setup_subscribers(subscriptions)
+    #   { :handler => handler_proc,
+    #     :subscriber => subscriber_derived_class
+    #   }
+    def setup_subscribers(subscriptions) #:nodoc:
     end
 
+    # This is the underlying method for all publishes. Use only if you know
+    # what you are doing. Use the request! and signal! methods instead.
     # Sends a message
     # If optional block is given, it will be executed when a response is received
     # or if timeout occurs
     # +message+:: The body of the message (any object that is serialisable)
     # +destination+:: The channel to post to
-    # +opts+::
-    #   ++timeout+:: Sender side timeout
+    # +opts+:: Options
+    #          timeout:: Sender side timeout
     #
     def publish(message, destination, opts = {}, code = 0, &blk)
       payload, sender = Gilmour::Protocol.create_request(message, code)
 
       EM.defer do # Because publish can be called from outside the event loop
         begin
-          send(sender, destination, payload, opts, &blk)
+          send_message(sender, destination, payload, opts, &blk)
         rescue Exception => e
           GLogger.debug e.message
           GLogger.debug e.backtrace
@@ -69,32 +74,149 @@ module Gilmour
       sender
     end
 
+    def request_destination(dest) #:nodoc:
+      'gilmour.request.' + dest
+    end
+
+    def slot_destination(dest) #:nodoc:
+      'gilmour.slot.' + dest
+    end
+
+    # Sends a request
+    # If optional block is given, it will be executed when a response is received
+    # or if timeout occurs. You usually want to provide this (otherwise consider
+    # using slots)
+    # Params:
+    # +message+:: The body of the message (any object that is serialisable)
+    # +destination+:: The channel to post to
+    # +opts+:: Options
+    #          timeout:: Sender side timeout
+    #          confirm_subscriber:: Confirms that active subscriber exists,
+    #          else calls blk with 404 error code.
+    def request!(message, dest, opts = {}, &blk)
+      opts[:confirm_subscriber] = true
+      publish(message, request_destination(dest), opts, 0, &blk)
+    end
+
+    # Emits a signal
+    # Params:
+    # +message+:: The body of the message (any object that is serialisable)
+    # +destination+:: The channel to post to
+    def signal!(message, dest, opts = {})
+      GLogger.error('Signal cannot have a callback. Ignoring!') if block_given?
+      publish(message, slot_destination(dest), opts)
+    end
+
+    # Emits a signal and sends a request
+    def broadcast(message, destination, opts = {}, code = 0, &blk)
+      request(message, destination, opts, code, &blk)
+      signal(message, destination, opts)
+    end
+
     def emit_error(message)
       raise NotImplementedError.new
     end
 
     # Adds a new handler for the given _topic_
-    def add_listener(topic, &handler)
+    def add_listener(topic, opts={}, &blk)
       raise "Not implemented by child class"
     end
 
+    def listeners(topic) #:nodoc:
+      raise NotImplementedError.new
+    end
+
+    def excl_dups?(topic, opts) #:nodoc:
+      group = exclusive_group(opts)
+      existing = listeners(topic).select { |l| exclusive_group(l) == group }
+      !existing.empty?
+    end
+
+    # Sets up a reply listener
+    # Params:
+    # +topic+:: The topic to listen on
+    # +options+:: Options Hash
+    #             timeout:: a 504 error code is sent after this time and the
+    #             execution for the request is terminated
+    #             fork:: The request will be processed in a forked process.
+    #             useful for long running executions
+    #             excl_group:: The exclustion group for this handler (see README for explanation)
+    # +blk+:: The block to the executed for the request
+    def reply_to(topic, options={}, &blk)
+      opts = options.dup
+      group = exclusive_group(opts)
+      if group.empty?
+        group = "_default"
+        opts[:excl_group] = group
+        GLogger.warn("Using default exclusion group for #{topic}")
+      end
+      req_topic = request_destination(topic)
+      if excl_dups?(req_topic, opts)
+        raise RuntimeError.new("Duplicate reply handler for #{topic}:#{group}")
+      end
+      opts[:type] = :reply
+      opts[:excl] = true
+      add_listener(req_topic, opts, &blk)
+    end
+    
+    # Sets up a slot listener
+    # Params:
+    # +topic+:: The topic to listen on
+    # +options+:: Options hash
+    #             timeout:: a 504 error code is sent after this time and the
+    #             execution for the request is terminated
+    #             fork:: The request will be processed in a forked process.
+    #             useful for long running executions
+    # +blk+:: The block to the executed for the request
+    def slot(topic, options={}, &blk)
+      opts = options.dup
+      stopic = slot_destination(topic)
+      if opts[:excl] && excl_dups?(stopic, opts)
+        raise RuntimeError.new("Duplicate reply handler for #{topic}:#{group}")
+      end
+      opts[:type] = :slot
+      #TODO: Check whether topic has a registered subscriber class?
+      # or leave it to a linter??
+      add_listener(stopic, opts, &blk)
+    end
+
     # Removes existing _handler_ for the _topic_
-    def remove_listener(topic, &handler)
+    # Use only if you have registered the handler with add_listener
+    # or listen_to
+    def remove_listener(topic, handler)
       raise "Not implemented by child class"
     end
 
-    def acquire_ex_lock(sender)
+    # Removes existing slot handler for the _topic_
+    def remove_slot(topic, handler)
+      remove_listener(slot_destination(topic), handler)
+    end
+
+    # Removes existing reply handler for the _topic_
+    def remove_reply(topic, handler)
+      remove_listener(request_destination(topic), handler)
+    end
+
+    def acquire_ex_lock(sender) #:nodoc:
       raise "Not implemented by child class"
     end
 
-    def send_response(sender, body, code)
+    def send_response(sender, body, code) #:nodoc:
       raise "Not implemented by child class"
     end
 
-    def execute_handler(topic, payload, sub)
+    def exclusive_group(sub) #:nodoc:
+      (sub[:excl_group] || sub[:subscriber]).to_s
+    end
+
+    def execute_handler(topic, payload, sub) #:nodoc:
       data, sender = Gilmour::Protocol.parse_request(payload)
       if sub[:exclusive]
-        lock_key = sender + sub[:subscriber].to_s
+        group = exclusive_group(sub)
+        if group.empty?
+          raise RuntimeError.new("Exclusive flag without group encountered!")
+        end
+        lock_key = sender + group
         acquire_ex_lock(lock_key) { _execute_handler(topic, data, sender, sub) }
       else
         _execute_handler(topic, data, sender, sub)
@@ -104,29 +226,32 @@ module Gilmour
       GLogger.debug e.backtrace
     end
 
-    def _execute_handler(topic, data, sender, sub)
+    def _execute_handler(topic, data, sender, sub) #:nodoc:
+      respond = (sub[:type] != :slot)
       Gilmour::Responder.new(
-        sender, topic, data, self, sub[:timeout], sub[:fork]
+        sender, topic, data, self, timeout: sub[:timeout],
+        fork: sub[:fork], respond: respond
       ).execute(sub[:handler])
     rescue Exception => e
       GLogger.debug e.message
       GLogger.debug e.backtrace
     end
 
-    def send
+    def send_message #:nodoc:
       raise "Not implemented by child class"
     end
 
-    def self.load_backend(name)
+    def self.load_backend(name) #:nodoc:
       require_relative name
     end
 
-    def self.load_all_backends
+    def self.load_all_backends #:nodoc:
       SUPPORTED_BACKENDS.each do |f|
         load_backend f
       end
     end
 
+    # stop the backend
     def stop(sender, body, code)
       raise "Not implemented by child class"
     end