liebre 0.1.21 → 0.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 1e5a7ee5a915ac1eef958be46fcaa5dc4f84c6d5
-  data.tar.gz: 162226313a78c9c3373b56edc4a66844ff69568c
+  metadata.gz: d8d00979bc470fd55b7dce291b96658266ea74ac
+  data.tar.gz: 48dff8d83158aa0a84f74a5aaac36e1f86de64b1
 SHA512:
-  metadata.gz: f75296a952c2a8f02de319343df92d8f9caf839ec819a57a909877fffff5dfd99046d7cf0d65ed38fbffe5c76c3bdc6b2739d8f8f556901ce13dd0799ebf9da5
-  data.tar.gz: 18b09cb54e1430ef62e2c7a807c3561cef1b8a4650987ad3f3075196fc8fc7f4eb4431dd41a788627ed6e4bc08ac34eb33f914cfed50105b72675f1fbc88dbc3
+  metadata.gz: b4d5fe0344c0cecf9ff3313ff58521406d20b9f6df88aae242bcfc450860f8dabd1180a3f0ec11371297fa2a1792508d279fd7d1b4bbc03dfcafc303dc9631bf
+  data.tar.gz: 20f95bcd2593a5bf0ef664f242fb3d53a7d429d785b83106643eafec38afaee2e021a512f4deb4b442a3f03ecae3f3fae0c9e623e4df0bf6bd1306fdad3e3b2c

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--require spec_helper

data/Gemfile.lock

@@ -1,28 +1,29 @@
 PATH
   remote: .
   specs:
-    liebre (0.1.15)
-      bunny (~> 2.5, >= 2.5.1)
+    liebre (0.2.0)
+      concurrent-ruby
 
 GEM
   remote: https://rubygems.org/
   specs:
     amq-protocol (2.0.1)
-    bunny (2.5.1)
+    bunny (2.2.2)
       amq-protocol (>= 2.0.1)
     coderay (1.1.1)
-    diff-lcs (1.2.5)
+    concurrent-ruby (1.0.5)
+    diff-lcs (1.3)
     method_source (0.8.2)
     pry (0.10.4)
       coderay (~> 1.1.0)
       method_source (~> 0.8.1)
       slop (~> 3.4)
-    rake (11.2.2)
+    rake (12.0.0)
     rspec (3.5.0)
       rspec-core (~> 3.5.0)
       rspec-expectations (~> 3.5.0)
       rspec-mocks (~> 3.5.0)
-    rspec-core (3.5.2)
+    rspec-core (3.5.4)
       rspec-support (~> 3.5.0)
     rspec-expectations (3.5.0)
       diff-lcs (>= 1.2.0, < 2.0)
@@ -38,10 +39,11 @@ PLATFORMS
 
 DEPENDENCIES
   bundler (~> 1.6)
+  bunny
   liebre!
   pry
   rake
   rspec
 
 BUNDLED WITH
-   1.13.6
+   1.14.6

data/{LICENSE → LICENSE.txt}

@@ -1,4 +1,4 @@
-Copyright (c) 2016 jcabotc
+Copyright (c) 2017 jcabotc
 
 MIT License
 

data/README.md

@@ -1,268 +1,565 @@
 # Liebre
 
-## Intro
+A library to interact with AMQP servers.
 
-**Liebre stands for hare in spanish.**
+Liebre stands for hare in spanish.
 
-This is a gem that handles RabbitMQ consumers, publishers and RPCs, based on [bunny](https://github.com/ruby-amqp/bunny).
+## Installation
 
-* It allows to create classes that will be invoked everytime a message it's received in its subscribed queue. 
-* It handles RPCs as a special Consumer where a callback message is returned to the exchange.
-* You can use its Publisher to send message to an exchange.
-* There is also a Publisher RPC method to send a message and wait for its response.
+Add this line to your application's Gemfile:
+
+    gem 'liebre'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install liebre
+
+## Introduction
+
+The Liebre library provides 4 abstractions, or **actors**, to interact with the server:
+
+  * **Publisher**: Publishes messages to an exchange
+  * **Consumer**: Binds a queue to an exchange, and consumes messages
+  * **RPC Client**: Publishes messages to an exchange and blocks until a response is received through an exclusive queue
+  * **RPC Server**: Binds a queue to an exchange, consumes messages, and replies the caller by putting a message at the specified queue
+
+Each actor has its own thread and its own channel.  Some actors (Consumer and RPC Server)
+also have their own thread pool in order to be able to handle messages concurrently.
+
+It leverages [concurrent-ruby](https://github.com/ruby-concurrency/concurrent-ruby) `Concurrent::Async` mixin to
+implement the actors.
 
 ## Configuration
-It is based on 2 config files:
 
-* rabbitmq.yml: it contains RabbitMQ connection configurations and can be enviroment dependant (default path `config/rabbitmq.yml`), it must contain, at least the `default` connection
-  * without environment set:
+Liebre accepts the following configuration options:
+
+```ruby
+Liebre.configure do |config|
+  config.logger      = $logger
+  config.adapter     = Liebre::Adapter::Bunny
+  config.connections = connections_config
+  config.actors      = actors_config
+end
 ```
-default:
-  :host: localhost
-  :port: 5672
-  :user: guest
-  :pass: guest
-  :vhost: /
-  :threaded: true
-  :heartbeat: 2
+
+### Logger
+
+The logger configuration option is optional and defaults to `Logger.new(nil)`.
+Liebre will log in the following events:
+
+  * An actor is started
+  * An actor is stopped
+  * Some error happened on the actor's thread
+
+Any other logging should be done from the application.
+
+### Adapter
+
+It specifies the adapter to use to interact with the server. The only adapter that ships with
+the library is the `Liebre::Adapter::Bunny` adapter that uses the `bunny` gem.
+
+**IMPORTANT**: Note that you should have the `bunny` gem available to use the
+`Liebre::Adapter::Bunny` adapter.
+
+### Connections
+
+On startup Liebre will establish a set of connections with one or more AMQP servers. Actors can be
+started on any of this connections.
+
+```ruby
+{one: {host: "foo.com", port: 123},
+ other: {}}
 ```
-  * with environment set:
+
+The example above will establish two connections, the configuration for each connection
+will be given with no modification to the adapter.
+
+### Actors
+
+The configuration of all actors:
+
+```ruby
+{
+  publishers: {
+    my_publisher: {connection: :one,
+                   resources: {
+                     exchange: {name: "foo", type: "fanout"}}},
+    my_other_pub: {connection: :other,
+                   resources: {
+                     exchange: {name: "bar", type: "direct", opts: {durable: true}}}}
+  },
+  consumers: {
+    my_consumer: {connection: :one,
+                  handler: MyApp::Consumer,
+                  prefetch_count: 10,
+                  pool_size: 10,
+                  resources: {
+                    exchange: {name: "baz", type: "fanout"},
+                    queue: {name: "baz_queue"},
+                    bind: [{routing_key: "a_key"}, {routing_key: "another"}]}},
+  },
+  rpc_clients: {
+    my_rpc_client: {connection: :other,
+                    resources: {
+                      exchange: {name: "qux", type: "fanout"}}},
+  },
+  rpc_servers: {
+    my_rpc_server: {connection: :one,
+                    handler: MyApp::RPCServer,
+                    prefetch_count: 5,
+                    pool_size: 5,
+                    resources: {
+                      exchange: {name: "quux", type: "fanout"},
+                      queue: {name: "quux_queue"}}}
+  }
+}
 ```
-development:
-  default:
-    :host: localhost
-    :port: 5672
-    :user: guest
-    :pass: guest
-    :vhost: /
-    :threaded: true
-    :heartbeat: 2
-  rpc:
-    :host: localhost
-    :port: 5672
-    :user: guest
-    :pass: guest
-    :vhost: rpc
-    :threaded: true
-    :heartbeat: 2
-test:
-  default:
-    :host: localhost
-    :port: 5672
-    :user: guest
-    :pass: guest
-    :vhost: /
-    :threaded: true
-    :heartbeat: 2
-  rpc:
-    :host: localhost
-    :port: 5672
-    :user: guest
-    :pass: guest
-    :vhost: rpc
-    :threaded: true
-    :heartbeat: 2
-
-production:
-  ...
+
+The example above will start 5 actors:
+
+  * A publisher to the `"foo"` exchange over the connection `:one`
+  * A publisher to the `"bar"` exchange over the connection `:other`
+  * A consumer of the "baz" queue that will consume messages over the connection `:one` and process them by running `MyApp::Consumer` handler on a pool of 10 dedicated threads
+  * A rpc client to the `"qux"` exchange over the connection `:other`
+  * A rpc server of the "quux" queue that will consume messages over the connection `:one` and process them by running `MyApp::RPCServer` handler on a pool of 5 dedicated threads
+
+Common options:
+
+  * `:connection` - The name of the connection to open the channel over
+  * `:resources` - Configuration about queues and exchanges. Depends on the actor
+
+Options for message handlers (Consumer and RPC Server):
+
+  * `:handler` - The class to use to handle messages. Its interface depends on the actor
+  * `:prefetch_count` - The prefetch count of the actor's channel
+  * `:pool_size` - The number of dedicated threads that will be used to handle messages concurrently
+
+## Start the engine
+
+### On an existing service
+
+A `Liebre::Engine` is an object that is able to start and stop a given configuration.
+
+```ruby
+require 'yaml'
+require 'liebre'
+
+config = Liebre::Config.new
+
+config.adapter     = Liebre::Adapter::Bunny
+config.connections = YAML.load_file("config/rabbitmq.yml")
+config.actors      = YAML.load_file("config/liebre.yml")
+
+engine = Liebre::Engine.new(config)
+engine.start
 ```
 
-* liebre.yml (default path `config/liebre.yml`)
+The example above will establish the specified connections and start all actors with
+their own threads and thread pools.
+
+Usually only one engine is required per application. To simplify the process, a default
+engine that uses the default config is provided.
+
+```ruby
+require 'liebre'
+
+Liebre.configure do |config|
+  config.adapter     = Liebre::Adapter::Bunny
+  config.connections = YAML.load_file("config/rabbitmq.yml")
+  config.actors      = YAML.load_file("config/liebre.yml")
+end
+
+Liebre.engine.start
+```
+
+This kind of startup is commonly used on rack applications. The previous code
+can be called from a Rails initializer.
+
+### As a standalone application
+
+If you wish to start Liebre as a standalone application `Liebre::Runner` is provided.
+
+```ruby
+require 'liebre'
+
+Liebre.configure do |config|
+  # some config
+end
+
+runner = Liebre::Runner.new(engine: Liebre.engine)
+runner.run
 ```
-rpc_request_timeout: 5
-
-consumers:
-  some_consumer:
-    class_name: MyConsumer
-    rpc: false
-    pool_size: 1
-    num_threads: 4
-    exchange:
-      name: "consumer_exchange"
-      type: "fanout"
-      opts:
-        durable: false
-    queue:
-      name: "consumer_queue"
-      opts:
-        durable: false
-    bind:
-      routing_key: #key a string or an array of strings
-        - key_1
-        - key_2
-        
-  some_rpc:
-    class_name: MyRPC
-    rpc: true
-    connection_name: rpc
-    pool_size: 1
-    exchange:
-      name: "rpc_exchange"
-      type: "fanout"
-      opts:
-        durable: false
-    queue:
-      name: "rpc_queue"
-      opts:
-        durable: false
-  
-publishers:
-  some_publisher:
-    exchange:
-      name: "consumer_exchange"
-      type: "fanout"
-      opts:
-        durable: false
-  rpc_publisher:
-    connection_name: rpc
-    exchange:
-      name: "rpc_exchange"
-      type: "fanout"
-      opts:
-        durable: false
+
+The example above starts the runner with the default engine
+
+It sets up some system signals in order to respond gracefully to unix `kill`
+and sleeps the main thread forever.
+
+The previous pattern is so common that a shortcut is provided:
+
+```ruby
+require 'liebre'
+
+Liebre.configure do |config|
+  # some config
+end
+
+Liebre.start
 ```
 
-### Consumers
+### Start partially
 
-An entry for each consumer in your app, consumer options:
-* `class_name` (mandatory): The class that will be invoked everytime a message is received
-* `rpc` is a flag to specify the consumer behaviour (default false)
-* `connection_name`: the name of the connection to use (default `default`)
-* `pool_size` of the connection channel (default 1)
-* `num_threads`: number of consumers of the queue (default 1)
-* `exchange` a hash of options:
-  * `name`: the exchange name
-  * `type`: the exchange type (fanout, direct or topic)
-  * `opts`: options hash to pass to bunny exchange function
-* `queue` a hash of options:
-  * `name`: the queue name
-  * `opts`: options hash to pass to bunny queue function
-* `bind` a hash of options (optional):
-  * `routing_key`: the binding routing key, it can be a single string or an array of strings
+Imagine the following setup: You have an application that may be started as a rack
+application and also as a worker with no rack interface to consume from rabbitmq as
+a background tasks system.
 
-### Publishers
+When you start the application with rack you may use publishers or rpc clients, but
+you don't want to start the consumers and rpc servers.
 
-An entry for each exchange you want to publish to, options:
-* `connection_name`: the name of the connection to use (default `default`)
-* `exchange` a hash of options:
-  * `name`: the exchange name
-  * `type`: the exchange type (fanout, direct or topic)
-  * `opts`: options hash to pass to bunny exchange function
+When you start the application with `Liebre::Runner` you want all actors to start.
 
-### Other configurations
+To handle this kind of scenarios `Liebre` keeps track of the already started actors
+and prevents them to start twice on a given engine.
 
-`rpc_request_timeout`: set the timeout for an RPC request
+A solution to this case may be the following:
 
-### Change default paths and set env and Logger
+Start all publishers and consumers on an initialized that will be shared between
+both ways to start the application:
 
-You can change these defaults in an initializer with something like:
+```ruby
+require 'liebre'
 
+Liebre.configure do |config|
+  # some config
+end
+
+Liebre.engine.start(only: [:publishers, :rpc_clients])
 ```
-Liebre::Config.config_path = "your/custom/path"
-Liebre::Config.connection_path = "your/custom/path"
 
-Liebre::Config.env = "production"
-Liebre::Config.logger = Logger.new(...)
+This code will configure `Liebre` and starts all publishers and rpc clients.
+
+When you start your application with `Liebre::Runner` you can do the following:
+
+```ruby
+# run all your initializers including the one above
 
+require 'liebre'
+Liebre.start
 ```
 
-## Usage
+This code will start all actors, and publishers and rpc clients will only be started
+once.
 
-There are 2 different consumer usages: `Consumer` and `RPC`.
+## Actors
 
-### Consumer
+Once a liebre engine has been started a `Liebre::Repository` has been populated with all actors
+that can be fetched by name.
 
-You only need to create a class with this simple interface:
+```ruby
+# ... start the engine
+repo = engine.repo
 
+publisher_1 = repo.publisher(:my_publisher)
+publisher_2 = repo.publisher(:my_other_pub)
+consumer    = repo.consumer(:my_consumer)
+rpc_client  = repo.rpc_server(:my_rpc_server)
+rpc_server  = repo.rpc_server(:my_rpc_server)
 ```
-class MyConsumer
-    
-  def initialize payload, meta
-    @payload = payload #the content of the message
-    @meta = meta #the meta information (also called properties)
-  end
 
-  def call
-    #do your stuff here
-    #return :ack to anknowledge the message, 
-    #return :reject to requeue it 
-    #or return :error to send it to the dead-letter-exchange
-    :ack
-  end
+### Publisher
+
+A publisher is an actor that declares an exchange on startup and provides a method to publish
+messages to that exchange.
+
+The `:resources` section of the actor's configuration requires the exchange specification:
+
+```ruby
+require 'liebre'
 
+Liebre.configure do |config|
+  config.adapter = Liebre::Adapter::Bunny
+  config.connections = {default: {}}
+
+  config.actors = {
+    publishers: {
+      my_pub: {
+        connection: :default,
+        resources: {
+          exchange: {name: "foo",
+                     type: "direct",
+                     opts: {durable: true}}
+        }
+      }
+    }
+  }
 end
+
+Liebre.engine.start
+
+publisher = Liebre.repo.publisher(:my_pub)
+publisher.publish("some_data")
+publisher.publish("more_data", :routing_key => "my_key")
 ```
 
-Every time a message is received, a new instance of this class will be created.
+The exchange specification:
 
-### RPC
+  * `:name` - The name of the exchange
+  * `:type` - The exchange type (`"direct"`, `"fanout"`, etc)
+  * `:opts` - (defaults to `{}`) The exchange options
 
-You only need to create a class with this simple interface:
+Once started the `#publish` method can be used to send messages to the configured
+exchange.
+
+### Consumer
+
+A consumer is an actor that declares an exchange, a queue and binds them on startup.
+It owns a thread pool to handle consumed messages.
+
+After declaration it subscribes to the queue and starts consuming messages. Once a message
+is consumed a handler for that message is started in a thread of the pool.
+
+The handler class must implement the following interface:
+
+  * `#initialize` receives 3 arguments: `payload`, `meta`, and `callback`.
+  * `#call`
+
+The handler class is initialized with that 3 arguments:
+
+  * `payload` - The actual content of the message
+  * `meta` - Message metadata that includes the headers and other information (depends on the adapter)
+  * `callback` - An object that responds to `#ack`, `#nack`, and `#reject`
+
+```ruby
+class MyHandler
 
-```
-class MyRPC
-    
   def initialize payload, meta, callback
-    @payload = payload #the content of the message
-    @meta = meta #the meta information (also called properties)
-    @callback = callback #a Proc that will publish an answer
+    @payload  = payload
+    @callback = callback
   end
 
   def call
-    #do your stuff here
-    @callback.call("your response")
+    case payload
+      when "0" then zero()
+      when "1" then one()
+      else raise "unknown!"
+    end
   end
 
+private
+
+  def zero
+    puts "yay!"
+    callback.ack()
+  end
+
+  def one
+    puts "wtf!"
+    callback.reject(requeue: false)
+  end
+
+  attr_reader :payload, :callback
+
 end
 ```
 
-Every time a message is received, a new instance of this class will be created.
+The handler above will print `"yay!"` and ack the message when the payload is `"0"`,
+print `"wtf!"` and reject the message when the payload is `"1"`, and will raise (and therefore
+will be rejected by liebre) when the handler raises an error.
 
-There are 2 ways to publish a message: `enqueue` and `enqueue_and_wait`.
+The `:resources` section of the actor's configuration requires the exchange and the queue,
+and may include binding options.
 
-### `enqueue`
+```ruby
+require 'liebre'
+
+Liebre.configure do |config|
+  config.adapter = Liebre::Adapter::Bunny
+  config.connections = {default: {}}
+
+  config.actors = {
+    consumers: {
+      my_con: {
+        connection: :default,
+        handler: MyHandler,
+        prefetch_count: 5,
+        pool_size: 5,
+        resources: {
+          exchange: {name: "foo",
+                     type: "direct",
+                     opts: {durable: true}},
+          queue: {name: "bar",
+                  opts: {durable: true}},
+          bind: [{routing_key: "one"},
+                 {routing_key: "other"}]
+        }
+      }
+    }
+  }
+end
 
+Liebre.engine.start
 ```
-publisher = Liebre::Publisher.new("some_publisher_name")
 
-publisher.enqueue "hello", :routing_key => "consumer_queue"
+The exchange specification is the same as for the publisher.
 
-publisher.enqueue "bye", :routing_key => "consumer_queue"
+The queue specification:
 
-```
+  * `:name` - The name of the queue
+  * `:opts` - (defaults to `{}`) The queue options
 
-### `enqueue_and_wait` (alias `rpc`)
+The bind specification is optional and defaults to `{}`. `:bind` value may be:
 
-```
-rpc_publisher = Liebre::Publisher.new("rpc_publisher")
+  * not present - the queue is bound to the exchange once with `{}` as options
+  * a hash - the queues is bound once to the exchange with that options
+  * a list of hashes - the queues is bound to the exchange once per hash of options
 
-response = publisher.enqueue_and_wait "hello", :routing_key => "consumer_queue"
+The example above declares the exchange `"foo"`, declares the queue `"bar"`,
+binds the queue to the exchange twice: one with the `"one"` routing key and
+another with the `"other"` routing key).
 
-another_response = publisher.rpc "bye", :routing_key => "consumer_queue"
+It starts a thread pool of 5 threads and starts consuming messages.
 
-```
+For each message the consumer receives a handler is instantiated and `#call` is called on
+it in one of the threads of its pool.
 
-### Installation and Execution
+### RPC Client
 
-Add the following to your Gemfile:
-```
-gem "liebre", ">~ 0.1"
-```
+A RPC client is an actor that declares an exchange, and declares a temporary queue with
+the options `exclusive`, and `auto_delete`.
 
-In your Raketask add:
+After declaration it subscribes to the queue.
 
-```
-require "liebre/tasks"
-```
+The `:resources` section of the rpc client configuration must specify the exchange.
 
-Then you just need to run:
-```
-rake liebre:run
+```ruby
+require 'liebre'
+
+Liebre.configure do |config|
+  config.adapter = Liebre::Adapter::Bunny
+  config.connections = {default: {}}
+
+  config.actors = {
+    rpc_client: {
+      my_client: {
+        connection: :default,
+        resources: {
+          exchange: {name: "foo",
+                     type: "direct",
+                     opts: {durable: true}},
+          queue: {prefix: "client_responses"}
+        }
+      }
+    }
+  }
+end
+
+Liebre.engine.start
+
+client = Liebre.repo.rpc_client(:my_pub)
+client.request("data")                     # => rpc server response (or nil on timeout, 5 seconds by default)
+client.request("data", routing_key: "bar") # => rpc server response (or nil on timeout, 5 seconds by default)
+client.request("data", {}, 15)             # => rpc server response (or nil on timeout after 15 seconds)
 ```
 
-Alternative way:
+The exchange specification is the same as for the publisher.
+
+The queue specification is optional and includes a prefix for the queue's name. The name of the
+queue will be that prefix followed by a random token, for example: `"client_responses_q23jrefdzXw"`.
+
+When a request is performed the client will block until the response is received or the timeout is reached.
+
+### RPC Server
+
+A rpc server is an actor that declares an exchange, a queue and binds them on startup.
+It owns a thread pool to handle requests.
+
+After declaration it subscribes to the queue and starts consuming messages. Once a message
+is consumed a handler for that message is started in a thread of the pool.
+
+The handler class must implement the following interface:
+
+  * `#initialize` receives 3 arguments: `payload`, `meta`, and `callback`.
+  * `#call`
+
+The handler class is initialized with that 3 arguments:
+
+  * `payload` - The actual content of the message
+  * `meta` - Message metadata that includes the headers and other information (depends on the adapter)
+  * `callback` - An object that responds to `#reply`
+
+```ruby
+class MyHandler
+
+  def initialize payload, meta, callback
+    @payload  = payload
+    @callback = callback
+  end
+
+  def call
+    case payload
+      when "0" then callback.reply("zero")
+      when "1" then callback.reply("one")
+      else raise "unknown!"
+    end
+  end
+
+private
+
+  attr_reader :payload, :callback
+
+end
 ```
+
+The handler above will reply to the client with "zero" when payload is "0",
+reply with "one" when the payload is "1", and will raise (and therefore not reply)
+when the handler raises error.
+
+The `:resources` section of the actor's configuration requires the exchange and the queue,
+and may include binding options.
+
+```ruby
 require 'liebre'
 
-Liebre::Runner.new.start
-```
+Liebre.configure do |config|
+  config.adapter = Liebre::Adapter::Bunny
+  config.connections = {default: {}}
+
+  config.actors = {
+    rpc_servers: {
+      my_rpc_server: {
+        connection: :default,
+        handler: MyHandler,
+        prefetch_count: 5,
+        pool_size: 5,
+        resources: {
+          exchange: {name: "foo",
+                     type: "direct",
+                     opts: {durable: true}},
+          queue: {name: "bar",
+                  opts: {durable: true}},
+          bind: [{routing_key: "one"},
+                 {routing_key: "other"}]
+        }
+      }
+    }
+  }
+end
+
+Liebre.engine.start
+```
+
+The exchange specification is the same as for the publisher.
+The queue and bind specifications are the same as for the consumer.
+
+The example above declares the exchange `"foo"`, declares the queue `"bar"`,
+binds the queue to the exchange twice: one with the `"one"` routing key and
+another with the `"other"` routing key).
+
+It starts a thread pool of 5 threads and starts consuming requests.
+
+For each message the consumer receives it instantiates and runs `#call` on
+the new handler in one of the threads of its pool.