rabbit_feed 0.3.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: b89c207774f809ccb829ba29d9b11299dc2872b0
+  data.tar.gz: 9ff5943d611beed002969fcad6b62e9d75b4f1c2
+SHA512:
+  metadata.gz: 2e612b191caae6071e6e5df07e0fc45f081353a68eae365be1ed5548a41d7b23ac82395436ca7e494d3f53d94724e4941041edd9986ed706d48b0e3e26e949ca
+  data.tar.gz: d76e833bf491449b3b641dd2346aa1f7b76de28acf7e1ff02d3f63416af311ec1d464a7a8f84f8fff3509251b0edc1c4150118b1318256a45e642a71b598c0f5

data/.gitignore

@@ -0,0 +1,11 @@
+# Bundler directory
+.bundle
+# Gem package directory
+pkg
+# OSX dotfile
+.DS_Store
+# Log files
+*.log
+*.iml
+.idea/
+*.gem

data/.rspec

@@ -0,0 +1,3 @@
+-r rutabaga
+--order rand
+--color

data/Brewfile

@@ -0,0 +1,4 @@
+# MQ server
+rabbitmq
+# For load testing
+siege

data/DEVELOPING.md

@@ -0,0 +1,140 @@
+# Developing RabbitFeed
+
+## Prerequisites
+
+### Ruby
+
+RabbitFeed has been tested on Ruby v2.0.
+
+Install the required gems with
+
+        gem install bundler
+        bundle install
+
+### RabbitMQ running locally
+
+        brew project
+        rabbitmq-server
+
+The management interface can be found [here](http://localhost:15672/). The default login is `guest/guest`. You can view exchanges [here](http://localhost:15672/#/exchanges) and queues [here](http://localhost:15672/#/queues).
+
+
+## Running Tests
+
+This will run the specs and the features:
+
+    bundle exec rake
+
+## Running the example project
+
+After doing any dev work, it is good practice to verify you haven't broken the examples. Run the examples like this:
+
+    ./run_example
+
+You should see something similar to the following output:
+
+    Starting non rails application consumer...
+    /opt/boxen/rbenv/versions/2.0.0-p451/bin/ruby -S rspec ./spec/lib/non_rails_app/event_handler_spec.rb
+    NonRailsApp::EventHandler - Consumed event: user_updates_beaver with payload: {"beaver_name"=>"beaver"}
+    .
+
+    Finished in 0.00349 seconds
+    1 example, 0 failures
+
+    Randomized with seed 59391
+
+    Non rails application consumer started
+    Starting rails application consumer...
+    /opt/boxen/rbenv/versions/2.0.0-p451/bin/ruby -S rspec ./spec/controllers/beavers_controller_spec.rb
+    ...
+
+    Finished in 0.04397 seconds
+    3 examples, 0 failures
+
+    Randomized with seed 18883
+
+    Rails application consumer started
+    Starting rails application...
+    Rails application started
+    Triggering event...
+    NonRailsApp::EventHandler - Consumed event: user_creates_beaver with payload: {"application"=>"rails_app", "host"=>"macjfleck.home", "environment"=>"development", "version"=>"1.0.0", "name"=>"user_creates_beaver", "created_at_utc"=>"2014-05-05T14:16:44.045395Z", "beaver_name"=>"05/05/14 15:16:43"}
+    Event triggered
+    RailsApp::EventHandler - Consumed event: application_acknowledges_event with payload: {"application"=>"non_rails_app", "host"=>"macjfleck.home", "environment"=>"development", "version"=>"1.0.0", "name"=>"application_acknowledges_event", "created_at_utc"=>"2014-05-05T14:16:44.057279Z", "beaver_name"=>"05/05/14 15:16:43", "event_name"=>"user_creates_beaver"}
+    Stopping non rails application consumer...
+    Non rails application consumer stopped
+    Stopping rails application consumer...
+    Rails application consumer stopped
+    Stopping rails application...
+    Rails application stopped
+
+## Performance Benchmarking
+
+There is a script that can be run to benchmark the tool. It benchmarks three areas:
+
+1. Producing events during HTTP requests within a rails application (using [siege](http://www.joedog.org/siege-home/))
+2. Producing events directly
+3. Consuming events
+
+To run the benchmarking script
+
+    brew project
+    ./run_benchmark
+
+As of 20140925, running the benchmark on this hardware:
+
+    MacBook Pro Retina, Mid 2012
+    Processor  2.6 GHz Intel Core i7
+    Memory  8 GB 1600 MHz DDR3
+    Software  OS X 10.8.5 (12F45)
+
+Results in this output:
+
+    Starting test of rails application...
+    Starting rails application...
+    -- create_table("beavers", {:force=>true})
+       -> 0.0140s
+    -- initialize_schema_migrations_table()
+       -> 0.0196s
+    Rails application started
+          done.
+
+    Transactions:            200 hits
+    Availability:         100.00 %
+    Elapsed time:           1.33 secs
+    Data transferred:         0.16 MB
+    Response time:            0.06 secs
+    Transaction rate:       150.38 trans/sec
+    Throughput:           0.12 MB/sec
+    Concurrency:            8.76
+    Successful transactions:         100
+    Failed transactions:             0
+    Longest transaction:          0.36
+    Shortest transaction:         0.00
+
+    Stopping rails application...
+    Rails application stopped
+    Rails application test complete
+
+
+    Starting standalone publishing and consuming benchmark...
+    Publishing 5000 events...
+           user     system      total        real
+       2.460000   0.310000   2.770000 (  2.779719)
+    Consuming 5000 events...
+           user     system      total        real
+       2.380000   0.590000   2.970000 (  3.500515)
+    Benchmark complete
+
+## Connection Recovery Testing
+
+A critical piece of RabbitFeed is the ability to recover from network connectivity problems. This means...
+
+* When publishing, all events that are published make it to the queue
+* When consuming, the consumer re-establishes its connection to the queue automatically
+
+To simulate network connectivity problems, there is a recovery test script that can be run like this:
+
+    brew project
+    ./run_recovery_test
+
+The script will publish and then consume 5000 mesages with the network dropping out every half-second.

data/Gemfile

@@ -0,0 +1,15 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in rabbit_feed.gemspec
+gemspec
+
+group :development, :test do
+  gem 'pry-plus'
+  gem 'rake'
+end
+
+group :test do
+  gem 'codeclimate-test-reporter'
+  gem 'rutabaga'
+  gem 'timecop'
+end

data/Gemfile.lock

@@ -0,0 +1,121 @@
+PATH
+  remote: .
+  specs:
+    rabbit_feed (0.3.1)
+      activemodel
+      activesupport
+      avro
+      bunny
+      connection_pool
+      pidfile
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    activemodel (4.1.6)
+      activesupport (= 4.1.6)
+      builder (~> 3.1)
+    activesupport (4.1.6)
+      i18n (~> 0.6, >= 0.6.9)
+      json (~> 1.7, >= 1.7.7)
+      minitest (~> 5.1)
+      thread_safe (~> 0.1)
+      tzinfo (~> 1.1)
+    amq-protocol (1.9.2)
+    avro (1.7.7)
+      multi_json
+    binding_of_caller (0.7.2)
+      debug_inspector (>= 0.0.1)
+    bond (0.4.3)
+    builder (3.2.2)
+    bunny (1.6.3)
+      amq-protocol (>= 1.9.2)
+    codeclimate-test-reporter (0.2.0)
+      simplecov (>= 0.7.1, < 1.0.0)
+    coderay (1.1.0)
+    columnize (0.3.6)
+    connection_pool (2.0.0)
+    debug_inspector (0.0.2)
+    debugger (1.6.6)
+      columnize (>= 0.3.1)
+      debugger-linecache (~> 1.2.0)
+      debugger-ruby_core_source (~> 1.3.2)
+    debugger-linecache (1.2.0)
+    debugger-ruby_core_source (1.3.2)
+    diff-lcs (1.2.5)
+    docile (1.1.2)
+    gherkin (2.12.2)
+      multi_json (~> 1.3)
+    i18n (0.6.11)
+    interception (0.3)
+    jist (1.5.1)
+      json
+    json (1.8.1)
+    method_source (0.8.2)
+    minitest (5.4.2)
+    multi_json (1.9.2)
+    pidfile (0.3.0)
+    pry (0.9.12.4)
+      coderay (~> 1.0)
+      method_source (~> 0.8)
+      slop (~> 3.4)
+    pry-debugger (0.2.2)
+      debugger (~> 1.3)
+      pry (~> 0.9.10)
+    pry-doc (0.4.6)
+      pry (>= 0.9)
+      yard (>= 0.8)
+    pry-docmore (0.1.1)
+      pry
+      pry-doc
+    pry-plus (1.0.0)
+      bond
+      jist
+      pry-debugger
+      pry-doc
+      pry-docmore
+      pry-rescue
+      pry-stack_explorer
+    pry-rescue (1.2.0)
+      interception (>= 0.3)
+      pry
+    pry-stack_explorer (0.4.9.1)
+      binding_of_caller (>= 0.7)
+      pry (>= 0.9.11)
+    rake (10.2.2)
+    rspec (2.14.1)
+      rspec-core (~> 2.14.0)
+      rspec-expectations (~> 2.14.0)
+      rspec-mocks (~> 2.14.0)
+    rspec-core (2.14.8)
+    rspec-expectations (2.14.5)
+      diff-lcs (>= 1.1.3, < 2.0)
+    rspec-mocks (2.14.6)
+    rutabaga (0.0.4)
+      turnip (>= 1.1.0)
+    simplecov (0.8.2)
+      docile (~> 1.1.0)
+      multi_json
+      simplecov-html (~> 0.8.0)
+    simplecov-html (0.8.0)
+    slop (3.4.7)
+    thread_safe (0.3.4)
+    timecop (0.4.4)
+    turnip (1.2.1)
+      gherkin (>= 2.5)
+      rspec (>= 2.0, < 4.0)
+    tzinfo (1.2.2)
+      thread_safe (~> 0.1)
+    yard (0.8.7.3)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  codeclimate-test-reporter
+  pry-plus
+  rabbit_feed!
+  rake
+  rspec
+  rutabaga
+  timecop

data/LICENSE.txt

@@ -0,0 +1,9 @@
+Copyright (c) 2013-2014 Simply Business
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,304 @@
+# RabbitFeed
+
+![RabbitFeed logo](https://cloud.githubusercontent.com/assets/768254/3286432/b4e65e7e-f548-11e3-9c91-f7d04f489cf3.png)
+
+A gem providing asynchronous event publish and subscribe capabilities with RabbitMQ.
+
+## Core concepts
+
+* Fire and forget: Application can publish an event and it has no knowledge/care of how that event is consumed.
+* Persistent events: Once an event has been published, it will persist until it has been processed successfully.
+* Self-describing events: The event not only contains a payload, but also a schema that describes that payload.
+* Multiple subscribers: Multiple applications can subscribe to the same events.
+* Event versioning: Consumers can customize event handling based on the event version.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'rabbit_feed', git: 'git@github.com:simplybusiness/rabbit_feed.git'
+
+### Configuration
+
+Create a `config/rabbit_feed.yml` file. The following options can be specified:
+
+    environment:
+      host: RabbitMQ host
+      user: RabbitMQ user name
+      password: RabbitMQ password
+      application: Application name
+
+Sample:
+
+    development:
+      host: localhost
+      user: guest
+      password: guest
+      application: beavers
+
+### Initialisation
+
+If installing in a rails application, the following should be defined in `config/initializers/rabbit_feed.rb`:
+
+```ruby
+RabbitFeed.instance_eval do
+  self.log                     = Logger.new (Rails.root.join 'log', 'rabbit_feed.log')
+  self.environment             = Rails.env
+  self.configuration_file_path = Rails.root.join 'config', 'rabbit_feed.yml'
+end
+# Define the events (if producing)
+EventDefinitions do
+  define_event('user_creates_beaver', version: '1.0.0') do
+    defined_as do
+      'A beaver has been created'
+    end
+    payload_contains do
+      field('beaver_name', type: 'string', definition: 'The name of the beaver')
+    end
+  end
+end
+# Define the event routing (if consuming)
+EventRouting do
+  accept_from('beavers') do
+    event('foo') do |event|
+      # Do something...
+    end
+  end
+end
+```
+
+## Producing events
+
+The producer defines the events and their payloads using the [Event Definitions DSL](https://github.com/simplybusiness/rabbit_feed#event-definitions-dsl). In a rails app, this can be defined in the [initialiser](https://github.com/simplybusiness/rabbit_feed#initialisation).
+
+To produce an event:
+
+```ruby
+require 'rabbit_feed'
+RabbitFeed::Producer.publish_event 'Event name', { 'payload_field' => 'payload field value' }
+```
+
+**Event name:** This tells you what the event is.
+
+**Event payload:** This is the data about the event. This should be a hash.
+
+The event will be published to the configured exchange on RabbitMQ (`amq.topic` by default) with a routing key having the pattern of:  `[environment].[producer application name].[event name]`.
+
+### Returned Events
+
+In the case that there are no consumers configured to subscribe to an event, the event will be returned to the producer. The returned event will be logged, and if your project uses [Airbrake](https://airbrake.io), an error will be reported there.
+
+### Testing the Producer
+
+To prevent RabbitFeed from firing events during tests, add the following to `spec_helper.rb`:
+
+```ruby
+config.before :each do
+  RabbitFeed::Producer.stub!
+end
+```
+
+#### RSpec
+
+To verify that your application publishes an event, use the custom RSpec matcher provided with this application.
+
+Add the following RSpec configuration to `spec_helper.rb`:
+
+```ruby
+RSpec.configure do |config|
+  config.include(RabbitFeed::TestingSupport::RSpecMatchers)
+end
+```
+
+The expectation looks like this:
+
+```ruby
+require 'spec_helper'
+
+describe BeaversController do
+
+  describe 'POST create' do
+    it 'publishes a create event' do
+      expect{
+        post :create, beaver: { name: 'beaver' }
+      }.to publish_event('user_creates_beaver', { 'beaver_name' => 'beaver' })
+    end
+  end
+end
+```
+
+## Consuming events
+
+The consumer defines to which events it will subscribe as well as how it handles events using the [Event Routing DSL](https://github.com/simplybusiness/rabbit_feed#event-routing-dsl). In a rails app, this can be defined in the [initialiser](https://github.com/simplybusiness/rabbit_feed#initialisation).
+
+An `Event` contains the following information:
+
+    `environment` The environment in which the event was created (e.g. development, test, production)
+    `application` The name of the application that generated the event (as specified in rabbit_feed.yml)
+    `version` The version of the event
+    `name` The name of the event
+    `host` The hostname of the server on which the event was generated
+    `created_at_utc` The time (in UTC) that the event was created
+    `payload` The payload of the event
+
+### Running the consumer
+
+    bundle exec rabbit_feed consume --environment development
+
+More information about the consumer command line options can be found [here](https://github.com/simplybusiness/rabbit_feed#consumer).
+
+### Event Processing Errors
+
+In the case that your consumer raises an error whilst processing an event, the error will be logged. If your project uses [Airbrake](https://airbrake.io), the error will also be reported there. The event that was being processed will remain on the RabbitMQ queue, and will be redelivered to the consumer until it is processed without error.
+
+### Testing the Consumer
+
+If you want to test that your routes are behaving as expected without actually using RabbitMQ, you can include the module `TestHelpers` in your tests and then invoke `rabbit_feed_consumer.consume_event(event)`. Following is an example:
+
+```ruby
+describe 'consuming events' do
+
+  include RabbitFeed::TestingSupport::TestingHelpers
+
+  accumulator = []
+
+  let(:define_route) do
+    EventRouting do
+      accept_from('app') do
+        event('ev') do |event|
+          accumulator << event
+        end
+      end
+    end
+  end
+
+  let(:event) { {'application' => 'app', 'name' => 'ev', 'stuff' => 'some_stuff'} }
+
+  before { define_route }
+
+  it 'route to the correct service' do
+    rabbit_feed_consumer.consume_event(event)
+    expect(accumulator.size).to eq(1)
+  end
+end
+```
+
+## Command Line Tools
+
+### Event Publish
+
+    bundle exec bin/rabbit_feed produce --payload 'Event payload' --name 'Event name' --environment test --config spec/fixtures/configuration.yml --logfile test.log --require rabbit_feed.rb --verbose
+
+Publishes an event. Note: until you've specified the [event definitions](https://github.com/simplybusiness/rabbit_feed#event-definitions-dsl), this will not publish any events. Options are as follows:
+
+    --payload The payload of the event
+    --name The name of the event
+    --environment The environment to run in
+    --config The location of the rabbit_feed configuration file
+    --logfile The location of the log file
+    --require The project file containing the dependancies
+    --verbose Turns on DEBUG logging
+    --help Print the available options
+
+### Consumer
+
+    bundle exec bin/rabbit_feed consume --environment test --config spec/fixtures/configuration.yml --logfile test.log --require rabbit_feed.rb --pidfile rabbit_feed.pid --verbose --daemon
+
+Starts a consumer. Note: until you've specified the [event routing](https://github.com/simplybusiness/rabbit_feed#event-routing-dsl), this will not receive any events. Options are as follows:
+
+    --environment The environment to run in
+    --config The location of the rabbit_feed configuration file
+    --logfile The location of the log file
+    --require The project file containing the dependancies (only necessary if running with non-rails application)
+    --pidfile The location at which to write a pid file
+    --verbose Turns on DEBUG logging
+    --daemon Run the consumer as a daemon
+    --help Print the available options
+
+## Event Definitions DSL
+
+Provides a means to define all events that are published by an application. Defines the event names and the payload associated with each event. The DSL is converted into a schema that is serialized along with the event payload, meaning the events are self-describing. This is accomplished using Apache [Avro](http://avro.apache.org/docs/current/). This also validates the event payload against its schema before it is published.
+
+Event definitions are cumulative, meaning you can load multiple `EventDefinitions` blocks.
+
+Here is an example DSL:
+
+```ruby
+EventDefinitions do
+  define_event('user_creates_beaver', version: '1.0.0') do
+    defined_as do
+      'A beaver has been created'
+    end
+    payload_contains do
+      field('beaver_name', type: 'string', definition: 'The name of the beaver')
+    end
+  end
+
+  define_event('user_updates_beaver', version: '1.0.0') do
+    defined_as do
+      'A beaver has been updated'
+    end
+    payload_contains do
+      field('beaver_name', type: 'string', definition: 'The name of the beaver')
+    end
+  end
+end
+```
+
+This defines two events:
+
+1. `user_creates_beaver`
+2. `user_updates_beaver`
+
+Each event has a mandatory string field in its payload, called `beaver_name`.
+
+The available field types are described [here](http://avro.apache.org/docs/current/spec.html#schema_primitive).
+
+Publishing a `user_creates_beaver` event would look like this:
+
+```ruby
+RabbitFeed::Producer.publish_event 'user_creates_beaver', { 'beaver_name' => @beaver.name }
+```
+
+## Event Routing DSL
+
+Provides a means for consumers to specify to which events it will subscribe as well as how it handles events. This is accomplished using a custom DSL backed by a RabbitMQ [topic](http://www.rabbitmq.com/tutorials/tutorial-five-ruby.html) exchange.
+
+Event routing definitions are cumulative, meaning you can load multiple `EventRouting` blocks.
+
+Here is an example DSL:
+
+```ruby
+EventRouting do
+  accept_from('beavers') do
+    event('user_created_beaver') do |event|
+      puts event.payload
+    end
+    event('user_updated_beaver') do |event|
+      puts event.payload
+    end
+  end
+end
+```
+
+This will subscribe to specified events originating from the `beavers` application. We have specified that we would like to subcribe to `user_created_beaver` and `user_updated_beaver` events. If either event type is received, we have specified that its payload will be printed to the screen.
+
+When the consumer is started, it will create its queue named using this pattern: `[environment].[consumer application name]`. It will bind the queue to the `amq.topic` exchange on the routing keys as defined in the event routing. In this example, it will bind on:
+
+    environment.beavers.user_created_beaver
+    environment.beavers.user_updated_beaver
+
+_Note: The consumer queues will automatically expire (delete) after 7 days without any consumer connections. This is to prevent unused queues from hanging around once their associated consumer has been terminated._
+
+## Delivery Semantics
+
+RabbitFeed provides 'at least once' delivery semantics. There are two use-cases where an event may be delivered more than once:
+
+1. If the subscriber raises an exception whilst processing an event, RabbitFeed will re-deliver the event to the subscriber until the event is processed without error.
+1. If an event is pushed to the subscriber, and the subscriber loses connectivity with RabbitMQ before it can send an acknowledgement back to RabbitMQ, RabbitMQ will push the event again once connectivity has been restored.
+
+It is advisable to run RabbitFeed in a [clustered](https://www.rabbitmq.com/clustering.html) RabbitMQ environment to prevent the loss of messages in the case that a RabbitMQ node is lost. By default, RabbitFeed will declare queues to be mirrored across all nodes of the cluster.
+
+## Developing
+
+_See [./DEVELOPING.md](./DEVELOPING.md) for instructions on how to develop RabbitFeed_