promiscuous 0.9.3.1 → 0.10

data/README.md

@@ -3,119 +3,22 @@ Promiscuous
 
 [![Build Status](https://secure.travis-ci.org/crowdtap/promiscuous.png?branch=master)](https://secure.travis-ci.org/crowdtap/promiscuous)
 
-Promiscuous offers an automatic way of propagating your model data across one or
-more applications. It supports Mongoid2, Mongoid3 and ActiveRecord.
-It uses [RabbitMQ](http://www.rabbitmq.com/).
-
-Usage
-------
-
-From a publisher side (app that owns the data), create a Publisher per model.
-as shown below.
-
-From your subscribers side (apps that receive updates from the publisher),
-create a Subscriber per model, as shown below.
-
-Example
---------
-
-### In your publisher app
-
-```ruby
-# initializer
-Promiscuous::AMQP.configure(:app => 'crowdtap',
-                            :server_uri => 'amqp://user:password@host:port/vhost')
-
-# publisher
-class ModelPublisher < Promiscuous::Publisher::Mongoid
-  publish :to => 'crowdtap/model',
-          :class => :Model,
-          :attributes => [:field_1, :field_2, :field_3]
-end
-```
-
-### In your subscriber app
-
-```ruby
-# initializer
-Promiscuous::AMQP.configure(:app => 'sniper',
-                            :server_uri => 'amqp://user:password@host:port/vhost',
-                            :error_handler => some_proc)
-
-# subscriber
-class ModelSubscriber < Promiscuous::Subscriber::Mongoid
-  subscribe :from => 'crowdtap/model',
-            :attributes => [:field_1, :field_2, :field_3],
-            :class => Model, # optional
-            :foreign_key => :publisher_id # optional
-end
-```
-
-### Starting the subscriber worker
-
-    rake promiscuous:replicate
-
-How does it work ?
-------------------
-
-1. On the publisher side, Promiscuous hooks into the after_create/update/destroy callbacks.
-2. When a model changes, Promiscuous sends a message to RabbitMQ, to the
-   'promiscuous' [topic exchange](http://www.rabbitmq.com/tutorials/tutorial-five-python.html).
-3. RabbitMQ routes the messages to each application through queues.
-   We use one queue per application (TODO explain why we need one queue).
-4. Subscribers apps are running the promiscuous worker, listening on their own queues,
-   executing the create/update/destroy on their databases.
-
-Note that we use a single exchange to preserve the ordering of data updates
-across application so that subscribers always see a consistant state of the
-system.
-
-Synching databases
--------------------
-
-Documents are created if not present when receiving an update on a non existing
-document.
-
-TODO: Explain how to sync databases.
-
-WARNING/TODO
-------------
-
-Promiscuous does **not** handle:
-- ActiveRecord polymorphism.
-- Any of the Mongoid atomic operatiors, such as inc, or add_to_set.
-- Association magic. Example:
-  ```ruby
-  # This will NOT replicate particiation_ids:
-  m = Member.first
-  m.particiations = [Participation.first]
-  m.save
-  
-  # On the other hand, this will:
-  m = Member.first
-  m.particiation_ids = [Participation.first.ids]
-  m.save
-  ```
-
-Furthermore, it can be racy. Consider this scenario with two interleaving
-requests A and B:
-
-1. (A) Update mongo doc X.value = 1
-2. (B) Update mongo doc X.value = 2
-3. (B) Publish 'X.value = 2' to Rabbit
-4. (A) Publish 'X.value = 1' to Rabbit
-
-At the end of the scenario, on the publisher side, the document X has value
-equal to 2, while on the subscriber side, the document has a value of 1.  This
-will likely not occur in most scenarios BUT BEWARE.  We have plans to fix this
-issue by using version numbers and mongo's amazing findandmodify.
-
-Backend: bunny / ruby-amqp
---------------------------
-
-Your publisher app may not run an eventmachine loop, which is required for
-ruby-amqp. Bunny on the other hand allows a non-eventmachine based application
-to publish messages to rabbitmq.
+Promiscuous is designed to facilitate designing a
+[service-oriented architecture](http://en.wikipedia.org/wiki/Service-oriented_architecture)
+in Ruby.
+
+Promiscuous offers an automatic way of propagating your data across one or more
+applications. It supports Mongoid2, Mongoid3 and ActiveRecord.
+It relies on [RabbitMQ](http://www.rabbitmq.com/) to push data around.
+
+Philosophy
+----------
+
+In order for a service-oriented system to be successful, services *must* be
+loosely coupled.  The traditional Ruby way of tackling this problem is to
+provide RESTful APIs.
+Sadly, this come to a cost since one must write controllers, integration tests, etc.
+Promiscuous to the rescue
 
 Compatibility
 -------------

data/lib/promiscuous/amqp/rubyamqp.rb

@@ -11,12 +11,12 @@ module Promiscuous
           raise "Please use amqp://user:password@host:port/vhost" if uri.scheme != 'amqp'
 
           {
-            :host => uri.host,
-            :port => uri.port,
+            :host   => uri.host,
+            :port   => uri.port,
             :scheme => uri.scheme,
-            :user => uri.user,
-            :pass => uri.password,
-            :vhost => uri.path.empty? ? "/" : uri.path,
+            :user   => uri.user,
+            :pass   => uri.password,
+            :vhost  => uri.path.empty? ? "/" : uri.path,
          }
         end
 
@@ -26,7 +26,8 @@ module Promiscuous
       end
 
       def self.disconnect
-        channel.close
+        self.channel.close if self.channel
+        self.channel = nil
       end
 
       def self.subscribe(options={}, &block)

data/lib/promiscuous/common/worker.rb

@@ -0,0 +1,17 @@
+module Promiscuous::Common::Worker
+  extend ActiveSupport::Concern
+
+  def initialize
+    self.stop = false
+  end
+
+  def unit_of_work
+    if defined?(Mongoid)
+      Mongoid.unit_of_work { yield }
+    else
+      yield
+    end
+  end
+
+  included { attr_accessor :stop }
+end

data/lib/promiscuous/common.rb

@@ -2,4 +2,5 @@ module Promiscuous::Common
   autoload :Options,      'promiscuous/common/options'
   autoload :Lint,         'promiscuous/common/lint'
   autoload :ClassHelpers, 'promiscuous/common/class_helpers'
+  autoload :Worker,       'promiscuous/common/worker'
 end

data/lib/promiscuous/publisher/amqp.rb

@@ -2,7 +2,7 @@ module Promiscuous::Publisher::AMQP
   extend ActiveSupport::Concern
   include Promiscuous::Publisher::Envelope
 
-  def amqp_publish
+  def publish
     Promiscuous::AMQP.publish(:key => to, :payload => payload.to_json)
   end
 

data/lib/promiscuous/publisher/error.rb

@@ -0,0 +1,18 @@
+class Promiscuous::Publisher::Error < RuntimeError
+  attr_accessor :inner, :instance
+
+  def initialize(inner, instance)
+    super(inner)
+    set_backtrace(inner.backtrace)
+    self.inner = inner
+    self.instance = instance
+  end
+
+  def message
+    "#{inner.message} while processing #{instance}"
+  end
+
+  def to_s
+    message
+  end
+end

data/lib/promiscuous/publisher/model.rb

@@ -33,7 +33,7 @@ module Promiscuous::Publisher::Model
         [:create, :update, :destroy].each do |operation|
           __send__("after_#{operation}", "promiscuous_publish_#{operation}".to_sym)
           define_method "promiscuous_publish_#{operation}" do
-            self.class.promiscuous_publisher.new(:instance => self, :operation => operation).amqp_publish
+            self.class.promiscuous_publisher.new(:instance => self, :operation => operation).publish
           end
         end
         alias :promiscuous_sync :promiscuous_publish_update

data/lib/promiscuous/publisher/mongoid/defer.rb

@@ -0,0 +1,52 @@
+module Promiscuous::Publisher::Mongoid::Defer
+  extend ActiveSupport::Concern
+
+  mattr_accessor :klasses
+  self.klasses = {}
+
+  def publish
+    super unless should_defer?
+  end
+
+  def should_defer?
+    if options.has_key?(:defer)
+      options[:defer]
+    else
+      operation == :update
+    end
+  end
+
+  def self.hook_mongoid
+    return if @mongoid_hooked
+    @mongoid_hooked = true
+
+    Moped::Query.class_eval do
+      alias_method :update_orig, :update
+      def update(change, flags = nil)
+        if klass = Promiscuous::Publisher::Mongoid::Defer.klasses[@collection.name]
+          psp_field = klass.aliased_fields["promiscous_sync_pending"]
+          change = change.dup
+          change['$set'] ||= {}
+          change['$set'].merge!(psp_field => true)
+        end
+        update_orig(change, flags)
+      end
+    end
+  end
+
+  included do
+    klass.class_eval do
+      cattr_accessor :publisher_defer_hooked
+      return if self.publisher_defer_hooked
+      self.publisher_defer_hooked = true
+
+      # TODO Make sure we are not overriding a field, although VERY unlikly
+      psp_field = :_psp
+      field psp_field, :as => :promiscous_sync_pending, :type => Boolean
+      index({psp_field => 1}, :background => true, :sparse => true)
+
+      Promiscuous::Publisher::Mongoid::Defer.hook_mongoid
+      Promiscuous::Publisher::Mongoid::Defer.klasses[collection.name] = self
+    end
+  end
+end

data/lib/promiscuous/publisher/mongoid.rb

@@ -1,5 +1,6 @@
 class Promiscuous::Publisher::Mongoid < Promiscuous::Publisher::Base
   autoload :Embedded, 'promiscuous/publisher/mongoid/embedded'
+  autoload :Defer,    'promiscuous/publisher/mongoid/defer'
 
   include Promiscuous::Publisher::Class
   include Promiscuous::Publisher::Attributes
@@ -13,6 +14,11 @@ class Promiscuous::Publisher::Mongoid < Promiscuous::Publisher::Base
       include Promiscuous::Publisher::Mongoid::Embedded
     else
       include Promiscuous::Publisher::Model
+      include Promiscuous::Publisher::Mongoid::Defer if mongoid3?
     end
   end
+
+  def self.mongoid3?
+    Gem.loaded_specs['mongoid'].version >= Gem::Version.new('3.0.0')
+  end
 end

data/lib/promiscuous/publisher/worker.rb

@@ -0,0 +1,49 @@
+class Promiscuous::Publisher::Worker
+  include Promiscuous::Common::Worker
+
+  def self.poll_delay
+    # TODO Configurable globally
+    # TODO Configurable per publisher
+    1.second
+  end
+
+  def replicate
+    EM.defer proc { self.replicate_once },
+             proc { EM::Timer.new(self.class.poll_delay) { replicate } }
+  end
+
+  def replicate_once
+    return if self.stop
+    begin
+      self.unit_of_work do
+        Promiscuous::Publisher::Mongoid::Defer.klasses.values.each do |klass|
+          replicate_collection(klass)
+        end
+      end
+    rescue Exception => e
+      self.stop = true
+      unless e.is_a?(Promiscuous::Publisher::Error)
+        e = Promiscuous::Publisher::Error.new(e, nil)
+      end
+      Promiscuous.error "[publish] FATAL #{e}"
+      Promiscuous::Config.error_handler.try(:call, e)
+    end
+  end
+
+  def replicate_collection(klass)
+    return if self.stop
+    # TODO Check for indexes and if not there, bail out
+    psp_field = klass.aliased_fields["promiscous_sync_pending"]
+    while instance = klass.where(psp_field => true).find_and_modify({'$unset' => {psp_field => 1}})
+      replicate_instance(instance)
+    end
+  end
+
+  def replicate_instance(instance)
+    return if self.stop
+    instance.class.promiscuous_publisher.new(:instance => instance, :operation => :update, :defer => false).publish
+  rescue Exception => e
+    # TODO set back the psp field
+    raise Promiscuous::Publisher::Error.new(e, instance)
+  end
+end

data/lib/promiscuous/publisher.rb

@@ -10,6 +10,8 @@ module Promiscuous::Publisher
   autoload :Model,        'promiscuous/publisher/model'
   autoload :Mongoid,      'promiscuous/publisher/mongoid'
   autoload :Polymorphic,  'promiscuous/publisher/polymorphic'
+  autoload :Worker,       'promiscuous/publisher/worker'
+  autoload :Error,        'promiscuous/publisher/error'
 
   def self.lint(*args)
     Lint.lint(*args)

data/lib/promiscuous/railtie/replicate.rake

@@ -1,18 +1,38 @@
 namespace :promiscuous do
-  desc 'Run the subscribers worker'
+  # Note This rake task can be loaded without Rails
+  desc 'Run the workers'
   task :replicate => :environment do |t|
-    require 'promiscuous/worker'
     require 'eventmachine'
     require 'em-synchrony'
 
     EM.synchrony do
-      Promiscuous::Loader.load_descriptors :subscribers if defined?(Rails)
-      Promiscuous::AMQP.disconnect
-      Promiscuous::Config.backend = :rubyamqp
-      Promiscuous::AMQP.connect
+      trap_signals
+      force_backend :rubyamqp
+
+      Promiscuous::Loader.load_descriptors if defined?(Rails)
 
       Promiscuous::Worker.replicate
-      $stderr.puts "Replicating with #{Promiscuous::Subscriber::AMQP.subscribers.count} subscribers"
+
+      msg = "Replicating with #{Promiscuous::Subscriber::AMQP.subscribers.count} subscribers" +
+            " and #{Promiscuous::Publisher::Mongoid::Defer.klasses.count} publishers"
+      Promiscuous.info msg
+      $stderr.puts msg
     end
   end
+
+  def trap_signals
+    %w(SIGTERM SIGINT).each do |signal|
+      Signal.trap(signal) do
+        Promiscuous.info "Exiting..."
+        Promiscuous::Worker.stop
+        EM.stop
+      end
+    end
+  end
+
+  def force_backend(backend)
+    Promiscuous::AMQP.disconnect
+    Promiscuous::Config.backend = backend
+    Promiscuous::AMQP.connect
+  end
 end

data/lib/promiscuous/subscriber/amqp.rb

@@ -16,7 +16,7 @@ module Promiscuous::Subscriber::AMQP
   included { use_option :from }
 
   module ClassMethods
-    def from=(value)
+    def from=(_)
       super
       old_sub = Promiscuous::Subscriber::AMQP.subscribers[from]
       raise "The subscriber '#{old_sub}' already listen on '#{from}'" if old_sub

data/lib/promiscuous/subscriber/mongoid/embedded.rb

@@ -2,7 +2,7 @@ module Promiscuous::Subscriber::Mongoid::Embedded
   extend ActiveSupport::Concern
 
   def fetch
-    old_value.nil? ? klass.new.tap { |m| m.id = id } : old_value
+    (old_value || klass.new).tap { |m| m.id = id }
   end
 
   def old_value

data/lib/promiscuous/subscriber/polymorphic.rb

@@ -30,7 +30,6 @@ module Promiscuous::Subscriber::Polymorphic
 
     def polymorphic_subscriber_from(payload)
       type = payload.is_a?(Hash) ? payload['type'] : nil
-      raise "The payload is missing the type information'" if type.nil?
       polymorphic_map[type] || self
     end
   end

data/lib/promiscuous/subscriber/worker.rb

@@ -0,0 +1,31 @@
+class Promiscuous::Subscriber::Worker
+  include Promiscuous::Common::Worker
+
+  def replicate
+    Promiscuous::AMQP.subscribe(subscribe_options) do |metadata, payload|
+      # Note: This code always runs on the root Fiber,
+      # so ordering is always preserved
+      begin
+        unless self.stop
+          Promiscuous.info "[receive] #{payload}"
+          self.unit_of_work { Promiscuous::Subscriber.process(JSON.parse(payload)) }
+          metadata.ack
+        end
+      rescue Exception => e
+        e = Promiscuous::Subscriber::Error.new(e, payload)
+
+        # TODO Discuss with Arjun about having an error queue.
+        self.stop = true
+        Promiscuous::AMQP.disconnect
+        Promiscuous.error "[receive] FATAL #{e}"
+        Promiscuous::Config.error_handler.try(:call, e)
+      end
+    end
+  end
+
+  def subscribe_options
+    queue_name = "#{Promiscuous::Config.app}.promiscuous"
+    bindings = Promiscuous::Subscriber::AMQP.subscribers.keys
+    {:queue_name => queue_name, :bindings => bindings}
+  end
+end

data/lib/promiscuous/subscriber.rb

@@ -12,6 +12,7 @@ module Promiscuous::Subscriber
   autoload :Polymorphic,  'promiscuous/subscriber/polymorphic'
   autoload :Upsert,       'promiscuous/subscriber/upsert'
   autoload :Observer,     'promiscuous/subscriber/observer'
+  autoload :Worker,       'promiscuous/subscriber/worker'
 
   def self.lint(*args)
     Lint.lint(*args)

data/lib/promiscuous/version.rb

@@ -1,3 +1,3 @@
 module Promiscuous
-  VERSION = '0.9.3.1'
+  VERSION = '0.10'
 end

data/lib/promiscuous/worker.rb

@@ -1,51 +1,15 @@
-module Promiscuous
-  module Worker
-    mattr_accessor :stop
-
-    def self.replicate
-      self.stop = false
-      self.trap_signals unless ENV['TEST_ENV']
-
-      Promiscuous::AMQP.subscribe(subscribe_options) do |metadata, payload|
-        begin
-          unless self.stop
-            Promiscuous.info "[receive] #{payload}"
-            self.mongoid_wrapper { Promiscuous::Subscriber.process(JSON.parse(payload)) }
-            metadata.ack
-          end
-        rescue Exception => e
-          e = Promiscuous::Subscriber::Error.new(e, payload)
-
-          self.stop = true
-          Promiscuous::AMQP.disconnect
-          Promiscuous.error "[receive] FATAL #{e}"
-          Promiscuous::Config.error_handler.call(e) if Promiscuous::Config.error_handler
-        end
-      end
-    end
-
-    def self.mongoid_wrapper
-      if defined?(Mongoid)
-        Mongoid.unit_of_work { yield }
-      else
-        yield
-      end
-    end
-
-    def self.trap_signals
-      %w(SIGTERM SIGINT).each do |signal|
-        Signal.trap(signal) do
-          self.stop = true
-          EM.stop
-          Promiscuous.info "exiting gracefully"
-        end
-      end
-    end
+module Promiscuous::Worker
+  mattr_accessor :workers
+  self.workers = []
+
+  def self.replicate
+    self.workers << Promiscuous::Publisher::Worker.new
+    self.workers << Promiscuous::Subscriber::Worker.new
+    self.workers.each { |w| w.replicate }
+  end
 
-    def self.subscribe_options
-      queue_name = "#{Promiscuous::Config.app}.promiscuous"
-      bindings = Promiscuous::Subscriber::AMQP.subscribers.keys
-      {:queue_name => queue_name, :bindings => bindings}
-    end
+  def self.stop
+    self.workers.each { |w| w.stop = true }
+    self.workers.clear
   end
 end

data/lib/promiscuous.rb

@@ -9,6 +9,7 @@ module Promiscuous
   autoload :Publisher,  'promiscuous/publisher'
   autoload :Subscriber, 'promiscuous/subscriber'
   autoload :Observer,   'promiscuous/observer'
+  autoload :Worker,     'promiscuous/worker'
 
   class << self
     def configure(&block)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: promiscuous
 version: !ruby/object:Gem::Version
-  version: 0.9.3.1
+  version: '0.10'
   prerelease: 
 platform: ruby
 authors:
@@ -10,7 +10,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-10-03 00:00:00.000000000 Z
+date: 2012-10-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport
@@ -105,8 +105,8 @@ files:
 - lib/promiscuous/amqp/rubyamqp.rb
 - lib/promiscuous/publisher/envelope.rb
 - lib/promiscuous/publisher/mongoid/embedded.rb
+- lib/promiscuous/publisher/mongoid/defer.rb
 - lib/promiscuous/publisher/active_record.rb
-- lib/promiscuous/publisher/amqp.rb
 - lib/promiscuous/publisher/attributes.rb
 - lib/promiscuous/publisher/base.rb
 - lib/promiscuous/publisher/lint.rb
@@ -115,17 +115,19 @@ files:
 - lib/promiscuous/publisher/lint/base.rb
 - lib/promiscuous/publisher/lint/class.rb
 - lib/promiscuous/publisher/lint/polymorphic.rb
-- lib/promiscuous/publisher/mongoid.rb
 - lib/promiscuous/publisher/polymorphic.rb
-- lib/promiscuous/publisher/model.rb
 - lib/promiscuous/publisher/class.rb
 - lib/promiscuous/publisher/mock.rb
+- lib/promiscuous/publisher/amqp.rb
+- lib/promiscuous/publisher/error.rb
+- lib/promiscuous/publisher/model.rb
+- lib/promiscuous/publisher/mongoid.rb
+- lib/promiscuous/publisher/worker.rb
 - lib/promiscuous/railtie/replicate.rake
 - lib/promiscuous/subscriber/envelope.rb
 - lib/promiscuous/subscriber/error.rb
 - lib/promiscuous/subscriber/mongoid/embedded.rb
 - lib/promiscuous/subscriber/active_record.rb
-- lib/promiscuous/subscriber/amqp.rb
 - lib/promiscuous/subscriber/lint.rb
 - lib/promiscuous/subscriber/lint/amqp.rb
 - lib/promiscuous/subscriber/lint/base.rb
@@ -137,23 +139,26 @@ files:
 - lib/promiscuous/subscriber/upsert.rb
 - lib/promiscuous/subscriber/model.rb
 - lib/promiscuous/subscriber/class.rb
-- lib/promiscuous/subscriber/polymorphic.rb
-- lib/promiscuous/subscriber/attributes.rb
 - lib/promiscuous/subscriber/observer.rb
+- lib/promiscuous/subscriber/attributes.rb
+- lib/promiscuous/subscriber/amqp.rb
+- lib/promiscuous/subscriber/polymorphic.rb
+- lib/promiscuous/subscriber/worker.rb
 - lib/promiscuous/config.rb
 - lib/promiscuous/amqp.rb
 - lib/promiscuous/common/lint.rb
 - lib/promiscuous/common/lint/base.rb
 - lib/promiscuous/common/options.rb
 - lib/promiscuous/common/class_helpers.rb
+- lib/promiscuous/common/worker.rb
 - lib/promiscuous/loader.rb
-- lib/promiscuous/publisher.rb
 - lib/promiscuous/railtie.rb
-- lib/promiscuous/worker.rb
-- lib/promiscuous/common.rb
 - lib/promiscuous/observer.rb
+- lib/promiscuous/common.rb
+- lib/promiscuous/publisher.rb
 - lib/promiscuous/subscriber.rb
 - lib/promiscuous/version.rb
+- lib/promiscuous/worker.rb
 - lib/promiscuous.rb
 - README.md
 homepage: http://github.com/crowdtap/promiscuous