chasqui 0.9.3 → 1.0.0.pre.rc1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: c7b83716114be3ede50a3559cb742d776cdecd1f
-  data.tar.gz: df453dfaa501a592ddc6b3c650fcae09a9343123
+  metadata.gz: e2a3b1a24a648748e026c9938d71f77aeda97a7f
+  data.tar.gz: cf1b2e7b5d261c31980b46edc42fcff4048374a2
 SHA512:
-  metadata.gz: 91b367f3cd65f4bceffd73ebbdd6ef9d9af344741d797ebb8909d7b372c8246c99cee930b3b0a3cf09f316cbd48e84f11f82cfcd16f2251acdaa63c089fba7be
-  data.tar.gz: 420990d6feaa68f81d4db1ec6076a0c2b8fa2e5bd137dcf41a222d89734fa8e560e5d2e8cbb07a455d22a827e020c43c744f886e78a6f17e3f60c9e255f73f55
+  metadata.gz: bedd779b43a73f7df0021bf31c19aa686093432e9488f7b1d32be64a066defc788b040596d19a662ddca6715245c36f0d2eb5a810b3ad26eaaf2d05ceceefdf1
+  data.tar.gz: a06828b09503066ea658447bcc0af0099339be201538421e94923bf138c8d25926eae97264c99291aca5259be3dfcf53fccf72ca6acf8f580ab092322f1ae48f

data/.gitignore

@@ -12,3 +12,4 @@
 *.o
 *.a
 mkmf.log
+.vagrant/

data/README.md

@@ -3,11 +3,13 @@
 
 # Chasqui
 
-Chasqui is a simple, lightweight, persistent implementation of the publish-subscribe (pub/sub)
-messaging pattern for service oriented architectures.
+Chasqui is a simple, lightweight, persistent implementation of the
+publish-subscribe (pub-sub) messaging pattern for service oriented
+architectures.
 
-Chasqui delivers messages to subscribers in a Resque-compatible format. If you are already
-using Resque and/or Sidekiq, Chasqui will make a wonderful companion to your architecture.
+Chasqui delivers messages to subscribers in a Resque-compatible format. If you
+are already using Resque and/or Sidekiq, Chasqui will make a wonderful
+companion to your architecture.
 
 ## Installation
 
@@ -25,22 +27,26 @@ Or install it yourself as:
 
 ## Dependencies
 
-Chasqui uses Redis to queue events and manage subscriptions. You can install
-redis with your favorite package manager, such as homebrew, yum, or apt, or if
-you prefer, you can run `vagrant up` to run Redis in a virtual machine.
+Chasqui uses [Redis](http://redis.io/) to queue events and manage
+subscriptions. You can install Redis with your favorite package manager, such
+as homebrew, yum, or apt, or if you prefer, you can run `vagrant up` to run
+Redis in a virtual machine.
 
 ## Quick Start
 
-Chasqui consistents of two components - a client and a broker. The broker's
-responsibility is to forward published events to registered subscribers. The
-client can both publish events and register subscribers.
+Chasqui consists of two components - a client and a broker. Clients can both
+publish to a channel and subscribe to events on one or more channels. The
+broker transforms incoming events into Resque (or Sidekiq) jobs and places them
+on one or more queues according to the currently registered subscribers. Under
+the hood, a subscriber is simply a Sidekiq/Resque worker that processes jobs
+placed on a queue.
 
 ### Start the broker
 
     chasqui -r redis://localhost:6379/0 -q my-app
 
-Your broker must use the same redis connection as your sidekiq (or resque)
-workers. For a list of available broker options, run `chasqui --help`.
+Your broker must use the same Redis connection as your Sidekiq/Resque workers.
+For a list of available broker options, see `chasqui --help`.
 
 ### Publish events
 
@@ -48,8 +54,7 @@ Publishing events is simple.
 
     # file: publisher.rb
     require 'chasqui'
-    Chasqui.publish 'user.sign-up', 'Luke Skywalker'
-    Chasqui.publish 'user.cancel', 'Dart Vader', 'invalid use of the force'
+    Chasqui.publish 'user.sign-up', name: 'Darth Vader'
 
 Be sure to run the publisher, broker, and subscribers in separate terminal
 windows.
@@ -58,34 +63,35 @@ windows.
 
 ### Subscribe to events
 
-Subscribing to events is also simple. The following example tells chasqui to
-forward events to the subscriber's 'my-app' queue, for which chasqui will
-generate the appropriate worker class. Within the subscriber block, you define
-one or more `on` blocks in which you place your application logic for handling
-an event.
+Subscribing to events is also simple. In the following example, we create a
+subscriber to handle events published to the `user.sign-up` channel.
 
     # file: subscriber1.rb
     require 'chasqui'
 
-    Chasqui.subscribe queue: 'my-app' do
+    class UserSignUpSubscriber
+      include Chasqui::Subscriber
+      subscribe channel: 'user.sign-up'
 
-      on 'user.sign-up' do |user_id|
-        # do something when the user signs up
+      def perform(payload)
+        # Do something when the user signs up.
+        #
+        # User.create(name: payload[:user])
+        #  => #<User:0X00fe346 @name="Darth Vader">
       end
-
-      on 'user.cancel' do |user_id, reason|
-        # do something else when user cancels
-      end
-
     end
 
-You can have as many subscribers as you like, but __each subscriber must have
-its own unique queue name__.
+### Running Sidekiq subscribers
 
 Here is how you can run the subscriber as a sidekiq worker:
 
     sidekiq -r subscriber.rb
 
+For more information on running Sidekiq, please refer to the
+[Sidekiq documentation](https://github.com/mperham/sidekiq).
+
+### Running Resque subscribers
+
 To run the resque worker, you first need to create a Rakefile.
 
     # Rakefile
@@ -104,25 +110,40 @@ Then you can run the resque worker to start processing events.
 
     rake resque:work
 
+For more information on running Resque workers, please refer to
+the [resque documentation](https://github.com/resque/resque).
+
 ## Why Chasqui?
 
-* Reduces coupling between applications
-* Integrates with the popular sidekiq and resque background worker libraries
-* Queues events for registered subscribers even if a subscriber is unavailable
+* Reduces coupling between applications.
+* Simple to learn and use.
+* Integrates with the popular Sidekiq and Resque background worker libraries.
+* Queues events for registered subscribers even if a subscriber is unavailable.
+* Subscribers can benefit from Sidekiq's built-in retry functionality for
+  failed jobs.
 
 ## Limitations
 
-In order for chasqui to work properly, the publisher, broker, and all
-subscribers must connect to the same Redis database.
+In order for Chasqui to work properly, the publisher, broker, and all
+subscribers must connect to the same Redis database. Chasqui is intentionally
+simple and may not have all of the features you would expect from a messaging
+system. If Chasqui is missing a feature that you think it should have, please
+consider [opening a GitHub issue](https://github.com/jbgo/chasqui/issues/new)
+to discuss your feature proposal. 
 
 ## Contributing
 
-* For new functionality, please open an issue for discussion before creating a pull request.
-* For bug fixes, you are welcome to create a pull request without first opening an issue.
+* For new functionality, please open an issue for discussion before creating a
+  pull request.
+* For bug fixes, you are welcome to create a pull request without first opening
+  an issue.
 * Except for documentation changes, tests are required with all pull requests.
-* Please be polite and respectful when discussing the project with maintainers and your fellow contributors.
+* Please be polite and respectful when discussing the project with maintainers
+  and your fellow contributors.
 
 ## Code of Conduct
 
-If you are unsure whether or not your communication may be inappropriate, please consult the [Chasqui Code of Conduct](code-of-conduct.md).
-If you even suspect harassment or abuse, please report it to the email address listed in the Code of Conduct.
+If you are unsure whether or not your communication may be inappropriate,
+please consult the [Chasqui Code of Conduct](code-of-conduct.md).  If you even
+suspect harassment or abuse, please report it to the email address listed in
+the Code of Conduct.

data/Vagrantfile

@@ -0,0 +1,15 @@
+# -*- mode: ruby -*-
+# vi: set ft=ruby :
+
+Vagrant.configure(2) do |config|
+  config.vm.box = 'hashicorp/precise64'
+
+  config.vm.network 'forwarded_port', guest: 6379, host: 6379
+
+  config.vm.provision "shell", inline: <<-SHELL
+    sudo apt-get update
+    sudo apt-get install -y redis-server
+    sudo sed -ie 's/bind 127.0.0.1/bind 0.0.0.0/' /etc/redis/redis.conf
+    sudo service redis-server restart
+  SHELL
+end

data/lib/chasqui.rb

@@ -7,88 +7,61 @@ require 'redis-namespace'
 require "chasqui/version"
 require "chasqui/config"
 require "chasqui/broker"
-require "chasqui/multi_broker"
+require "chasqui/brokers/redis_broker"
+require "chasqui/queue_adapter"
+require "chasqui/queue_adapters/redis_queue_adapter"
 require "chasqui/subscriber"
-require "chasqui/subscription"
-require "chasqui/workers/worker"
-require "chasqui/workers/resque_worker"
-require "chasqui/workers/sidekiq_worker"
+require "chasqui/subscriptions"
+require "chasqui/worker"
 
 module Chasqui
+
+  class ConfigurationError < StandardError; end
+
   class << self
     extend Forwardable
-    def_delegators :config, :redis, :channel, :inbox, :inbox_queue, :logger
+    def_delegators :config, *CONFIG_SETTINGS
+    def_delegators :subscriptions, :register, :unregister
 
     def configure(&block)
-      @config ||= Config.new
-      yield @config
+      yield config
     end
 
     def config
       @config ||= Config.new
     end
 
-    def publish(event, *args)
-      redis.lpush inbox_queue, build_payload(event, *args).to_json
-    end
-
-    def subscribe(options={}, &block)
-      queue = options.fetch :queue
-      channel = options.fetch :channel, config.channel
-
-      create_subscription(queue, channel).tap do |subscription|
-        subscription.subscriber.evaluate(&block) if block_given?
-        redis.sadd subscription_key(channel), subscription.subscription_id
-      end
-    end
-
-    def unsubscribe(channel, options={}, &block)
-      queue = options.fetch :queue
-      subscription = subscriptions[queue.to_s]
-
-      if subscription
-        redis.srem subscription_key(channel), subscription.subscription_id
-        subscription.subscription_id
-      end
-    end
-
-    def subscription(queue)
-      subscriptions[queue.to_s]
+    def publish(channel, *args)
+      redis.lpush inbox_queue, build_event(channel, *args).to_json
     end
 
-    def subscriber_class_name(queue)
-      queue_name_constant = queue.split(':').last.gsub(/[^\w]/, '_')
-      "Subscriber__#{queue_name_constant}".to_sym
-    end
-
-    def subscription_key(channel)
-      "subscriptions:#{channel}"
+    def subscriptions
+      @subscriptions ||= Subscriptions.new build_queue_adapter
     end
 
     private
 
-    def create_subscription(queue, channel)
-      subscriptions[queue.to_s] ||= Subscription.new queue, channel
-    end
-
-    def subscriptions
-      @subscriptions ||= {}
-    end
-
-    def build_payload(event, *args)
+    def build_event(channel, *args)
       opts = extract_job_options!(*args)
 
-      payload = { event: event, channel: channel, data: args }
-      payload[:retry] = opts[:retry] || opts['retry'] if opts
+      payload = { channel: channel, payload: args }
+      payload[:retry] = fetch_option(opts, :retry, true) || false
       payload[:created_at] = Time.now.to_f.to_s
 
       payload
     end
 
     def extract_job_options!(*args)
-      if args.last.kind_of?(Hash)
-        args.last.delete(:job_options)
-      end
+      opts = args.last.delete(:job_options) if args.last.kind_of?(Hash)
+      opts ||= {}
+    end
+
+    def fetch_option(opts, key, default=nil)
+      opts.fetch key.to_sym, opts.fetch(key.to_s, default)
+    end
+
+    def build_queue_adapter
+      queue_adapter.new
     end
   end
 end

data/lib/chasqui/broker.rb

@@ -4,7 +4,7 @@ class Chasqui::Broker
   attr_reader :config, :redis, :redis_namespace
 
   extend Forwardable
-  def_delegators :@config, :inbox, :logger
+  def_delegators :@config, :inbox_queue, :logger
 
   ShutdownSignals = %w(INT QUIT ABRT TERM).freeze
 
@@ -24,7 +24,7 @@ class Chasqui::Broker
     install_signal_handlers
 
     logger.info "broker started with pid #{Process.pid}"
-    logger.info "configured to fetch events from #{inbox} on #{redis.inspect}"
+    logger.info "configured to fetch events from #{inbox_queue} on #{redis.inspect}"
 
     until_shutdown_requested { forward_event }
   end
@@ -35,7 +35,7 @@ class Chasqui::Broker
 
   class << self
     def start
-      Chasqui::MultiBroker.new.start
+      Chasqui::RedisBroker.new.start
     end
   end
 

data/lib/chasqui/{multi_broker.rb → brokers/redis_broker.rb}

@@ -1,6 +1,6 @@
 require 'securerandom'
 
-class Chasqui::MultiBroker < Chasqui::Broker
+class Chasqui::RedisBroker < Chasqui::Broker
 
   def forward_event
     event = receive or return
@@ -17,22 +17,22 @@ class Chasqui::MultiBroker < Chasqui::Broker
   end
 
   def in_progress_queue
-    with_namespace inbox, 'in_progress'
+    with_namespace inbox_queue, 'in_progress'
   end
 
-  def inbox_queue
-    with_namespace inbox
+  def namespaced_inbox_queue
+    with_namespace inbox_queue
   end
 
-  def build_job(queue, event)
+  def build_job(queue, job_class, event)
     {
-      class: "Chasqui::#{Chasqui.subscriber_class_name(queue)}",
+      class: job_class,
       args: [event],
       queue: 'my-queue',
       jid: SecureRandom.hex(12),
       created_at: (event['created_at'] || Time.now).to_f,
       enqueued_at: Time.now.to_f,
-      retry: !!event['retry']
+      retry: event['retry']
     }.to_json
   end
 
@@ -57,7 +57,7 @@ class Chasqui::MultiBroker < Chasqui::Broker
   end
 
   def dequeue
-    redis.brpoplpush(inbox_queue, in_progress_queue, timeout: config.broker_poll_interval).tap do |event|
+    redis.brpoplpush(namespaced_inbox_queue, in_progress_queue, timeout: config.broker_poll_interval).tap do |event|
       if event.nil?
         logger.debug "reached timeout for broker poll interval: #{config.broker_poll_interval} seconds"
       end
@@ -65,8 +65,8 @@ class Chasqui::MultiBroker < Chasqui::Broker
   end
 
   def dispatch(event, subscription_id)
-    backend, queue = subscription_id.split('/', 2)
-    job = build_job queue, event
+    backend, job_class, queue = subscription_id.split('/', 3)
+    job = build_job queue, job_class, event
 
     logger.debug "dispatching event queue=#{queue} backend=#{backend} job=#{job}"
 
@@ -79,7 +79,7 @@ class Chasqui::MultiBroker < Chasqui::Broker
   end
 
   def subscriptions_for(event)
-    subscription_key = Chasqui.subscription_key event['channel']
+    subscription_key = "subscriptions:#{event['channel']}"
     redis.smembers with_namespace(subscription_key)
   end
 

data/lib/chasqui/config.rb

@@ -1,10 +1,11 @@
 module Chasqui
 
   Defaults = {
+    default_queue: 'chasqui-subscribers',
     inbox_queue: 'inbox',
     redis_namespace: 'chasqui',
-    publish_channel: '__default',
-    broker_poll_interval: 3
+    broker_poll_interval: 3,
+    queue_adapter: -> { QueueAdapters::RedisQueueAdapter }
   }.freeze
 
   class ConfigurationError < StandardError
@@ -12,22 +13,31 @@ module Chasqui
 
   CONFIG_SETTINGS = [
     :broker_poll_interval,
-    :channel,
+    :channel_prefix,
+    :default_queue,
     :inbox_queue,
     :logger,
+    :queue_adapter,
     :redis,
     :worker_backend
   ]
 
   class Config < Struct.new(*CONFIG_SETTINGS)
-    def channel
-      self[:channel] ||= Defaults.fetch(:publish_channel)
+    def default_queue
+      self[:default_queue] ||= Defaults.fetch(:default_queue)
     end
 
     def inbox_queue
       self[:inbox_queue] ||= Defaults.fetch(:inbox_queue)
     end
-    alias inbox inbox_queue
+
+    def queue_adapter
+      self[:queue_adapter] ||= Defaults.fetch(:queue_adapter).call
+    end
+
+    def worker_backend
+      self[:worker_backend] ||= choose_worker_backend
+    end
 
     def redis
       unless self[:redis]
@@ -74,5 +84,15 @@ module Chasqui
     def broker_poll_interval
       self[:broker_poll_interval] ||= Defaults.fetch(:broker_poll_interval)
     end
+
+    private
+
+    def choose_worker_backend
+      if Object.const_defined? :Sidekiq
+        :sidekiq
+      elsif Object.const_defined? :Resque
+        :resque
+      end
+    end
   end
 end