elecnix-workling 0.4.2 → 0.4.9

data/CHANGES.markdown

@@ -1,3 +1,13 @@
+Version 0.4.8, 07.05.2009
+- implemented XMPP listener (experimental)
+- implemented method for explicitly naming queues, this is useful in conjunction with the XMPP listener
+- rewrote the workling_client script, now allows specifying configuration options on the command line. Makes it easier to manage workling instances in production, and allows running worklings for multiple apps on one machine
+- integrated support for custom exception notifications
+
+Version 0.4.2.3, 31.01.2009
+- introduced Workling.raises_exceptions. by default, this is true in test and development to help with bug tracking. 
+- added :threaded as the default spawn runner for test and development. helps problem tracing. 
+
 Version 0.4.2.2, 29.11.08
 - turned Workling.load_path into an Array.
 

data/README.markdown

@@ -1,3 +1,7 @@
+# *digitalhobbit/workling Fork Notes*
+
+*This particular Workling fork provides an SQS Client. See instructions below.*
+
 # Workling
 
 Workling gives your Rails App a simple API that you can use to make code run in the background, outside of the your request. 
@@ -56,7 +60,11 @@ Calling `async_my_method` on the worker class will trigger background work. This
 
 If an exception is raised in your Worker, it will not be propagated to the calling code by workling. This is because the code is called asynchronously, meaning that exceptions may be raised after the calling code has already returned. If you need your calling code to handle exceptional situations, you have to pass the error into the return store. 
 
-Workling does log all exceptions that propagate out of the worker methods. 
+Workling does log all exceptions that propagate out of the worker methods.
+
+Furthermore you can provide custom exception handling by defining the 
+  notify_exception(exception, method, options)
+in your worker class. If it is present it will be called for every exception
 
 ## Logging with Workling
 
@@ -64,6 +72,52 @@ Workling does log all exceptions that propagate out of the worker methods.
 
     logger.info("about to moo.")
 
+## Running Workling (in production)
+
+The workling daemon can be invoked using the command
+
+    script/workling_client run
+
+This will make it run in the development environment, with the default settings unless overridden in your development.rb (see below)
+
+For production use the script takes a couple of options
+
+    script/workling_client <daemon_options> -- <app_options>
+
+The daemon_options configure the runtime environment of the daemon and can be one of the following options:
+
+    --app-name APP_NAME
+    --dir DIR
+    --monitor
+    --ontop
+
+The app-name option is useful if you want to run worklings for multiple Rails apps on the same machine. Each app will need to have its unique name.
+The monitor option should not be specified if you are using Monit or god.
+The dir options specifies where the logs and the pid files are saved.
+
+The app_options allow you to specify the configuration of the workling interfaces via the command line:
+
+    --client CLIENT
+    --invoker INVOKER
+    --routing ROUTING
+    --load-path LOADPATH
+    --environment ENVIRONMENT
+
+The client, invoker and routing params take the full class names of the relevant plugin classes. load-path allows overriding where workling looks for workers and environment should be self explanatory.
+
+The following is a sample of how workling_client can be used with god and AMQP:
+
+    God.watch do |w|
+      script = "cd #{RAILS_ROOT} && workling_client"
+      w.name = "myapp-workling"
+      w.start = "#{script} start -a myapp-workling -- -e production -i Workling::Remote::Invokers::EventmachineSubscriber -c Workling::Clients::AmqpClient"
+      w.restart = "#{script} restart -a myapp-workling -- -e production -i Workling::Remote::Invokers::EventmachineSubscriber -c Workling::Clients::AmqpClient"
+      w.stop = "#{script} stop -a myapp-workling"
+
+      w.pid_file = "#{RAILS_ROOT}/log/myapp-workling.pid"
+    end
+
+
 ## What should I know about the Spawn Runner?
 
 Workling automatically detects and uses Spawn, if installed. Spawn basically forks Rails every time you invoke a workling. To see what sort of characteristics this has, go into script/console, and run this: 
@@ -75,9 +129,20 @@ You'll see that this executes pretty much instantly. Run 'top' in another termin
 
 You cannot run your workers on a remote machine or cluster them with spawn. You also have no persistence: if you've fired of a lot of work and everything dies, there's no way of picking up where you left off. 
 
-# Using the Starling runner
 
-If you want cross machine jobs with low latency and a low memory overhead, you might want to look into using the Starling Runner. 
+## Gohanlon addition
+
+To use Workling with Spawn, you can safely delete the config/workling.yml file generated by 'script/plugin install'.
+
+Also in your environment.rb file the following are useful 
+
+	# Run all jobs to be executed in the foreground
+	# Workling::Remote.dispatcher = Workling::Remote::Runners::NotRemoteRunner.new
+
+	# Execute jobs in a forked process using Spawn
+	Workling::Remote::Runners::SpawnRunner.options = { :method => :spawn }
+	Workling::Remote.dispatcher = Workling::Remote::Runners::SpawnRunner.new
+
 
 ## Installing Starling
 
@@ -219,6 +284,96 @@ Workling will now automatically detect and use Bj, unless you have also installe
 
     Workling::Remote.dispatcher = Workling::Remote::Runners::BackgroundjobRunner.new
 
+
+# Using XMPP listener
+
+NOTE: this code is highly experimental. It was implemented in order to facilitate the async communication between multiple Rails app running in the same domain/datacentre. However it is no longer in production use, since the setup proved to be both too complex to maintain and quite unreliable. The AMQP support turned out to be much more appropriate.
+I'm putting this here as a starting base for someone interested in using XMPP in such a way. Contact me at derfred on github if you want pointers.
+
+this client requires the xmpp4r gem
+
+in the config/environments/development.rb file (or production.rb etc)
+
+    Workling::Remote::Runners::ClientRunner.client = Workling::Clients::XmppClient.new
+    Workling::Remote.dispatcher = Workling::Remote::Runners::ClientRunner.new        # dont use the standard runner
+    Workling::Remote.invoker = Workling::Remote::Invokers::LoopedSubscriber          # does not work with the EventmachineSubscriber Invoker
+
+furthermore in the workling.yml file you need to set up the server details for your XMPP server
+
+    development:
+      listens_on: "localhost:22122"
+      jabber_id: "sub@localhost/laptop"
+      jabber_server: "localhost"
+      jabber_password: "sub"
+      jabber_service: "pubsub.derfredtop.local"
+
+for details on how to configure your XMPP server (ejabberd) check out the following howto:
+
+  http://keoko.wordpress.com/2008/12/17/xmpp-pubsub-with-ejabberd-and-xmpp4r/
+
+
+finally you need to expose your worker methods to XMPP nodes like so:
+
+    class NotificationWorker < Workling::Base
+
+      expose :receive_notification, :as => "/home/localhost/pub/sub"
+
+      def receive_notification(input)
+        # something here
+      end
+
+    end
+
+# Using SQS
+
+If you're running on Amazon EC2, you may want to leverage SQS (Simple Queue Service) to benefit from this highly scalable queue implementation without having to install any software.
+
+The SQS Client namespaces queues with an optional prefix as well as with the Rails environment, allowing us to distinguish between production and staging queues, for example. Queues are automatically created the first time they are accessed.
+
+Configuring Workling to use SQS is very straightforward and requires no additional software, with the exception of the RightAws gem.
+
+## Installing the SQS Client
+
+Install the RightAws gem:
+
+    1. sudo gem install right_aws
+
+Configure Workling to use the SqsClient. Add this to your environment:
+
+    Workling::Remote.dispatcher = Workling::Remote::Runners::ClientRunner.new
+    Workling::Remote.dispatcher.client = Workling::Clients::SqsClient.new
+
+Add your AWS key id and secret key to workling.yml:
+
+    production:
+      sqs_options:
+        aws_access_key_id: <your AWS access key id>
+        aws_secret_access_key: <your AWS secret access key>
+
+You can optionally override the following settings, although the defaults
+will likely be sufficient:
+
+        # Queue names consist of an optional prefix, followed by the environment
+        # and the name of the key.
+        prefix: foo_
+
+        # The number of SQS messages to retrieve at once. The maximum and default
+        # value is 10.
+        messages_per_req: 10
+
+        # The SQS visibility timeout for retrieved messages. Defaults to 30 seconds.
+        visibility_timeout: 30
+
+Now start the Workling Client:
+
+    1 ./script/workling_client start
+    
+You're good.
+
+## Limitations
+
+SQS messages need to be explicitly deleted from the queue. Otherwise, they will reappear after the visibility timeout. The SQS client currently deletes a message immediately before handing it to a worker, assuming that it will be processed successfully. A more robust implementation (which would require additional hooks in the Workling framework) would be to defer the deletion until after the message was successfully processed, allowing us to retry the message processing in case of an error.
+
 # Progress indicators and return stores
 
 Your worklings can write back to a return store. This allows you to write progress indicators, or access results from your workling. As above, this is fairly slim. Again, you can swap in any return store implementation you like without changing your code. They all behave like memcached. For tests, there is a memory return store, for production use there is currently a starling return store. You can easily add a new return store (over the database for instance) by subclassing `Workling::Return::Store::Base`. Configure it like this in your test environment:
@@ -365,11 +520,13 @@ Next, inside of `listen`, we need to iterate through all defined routes. There i
 
 That's it! We now have a more effective Invoker.
 
+
+
 # Contributors
 
 The following people contributed code to workling so far. Many thanks :) If I forgot anybody, I aplogise. Just drop me a note and I'll add you to the project so that you can amend this!
 
-Anybody who contributes fixes (with tests), or new functionality (whith tests) which is pulled into the main project, will also be be added to the project.
+Anybody who contributes fixes (with tests), or new functionality (with tests) which is pulled into the main project, will also be be added to the project.
 
 * Andrew Carter (ascarter)
 * Chris Gaffney (gaffneyc)

data/bin/workling_client

@@ -0,0 +1,28 @@
+#!/usr/bin/env ruby
+require 'rubygems'
+require 'daemons'
+require 'workling_server'
+
+daemon_options = {
+  :app_name   => "workling",
+  :dir_mode   => :normal,
+  :dir        => File.join(Dir.pwd, 'log'),
+  :log_output => true,
+  :multiple   => false,
+  :backtrace  => true,
+  :monitor    => false
+}.merge(WorklingServer.parse_daemon_options(ARGV))
+
+workling_options = {
+  :client_class => "Workling::Clients::MemcacheQueueClient",
+  :invoker_class => "Workling::Remote::Invokers::ThreadedPoller",
+  :routing_class => "Workling::Routing::ClassAndMethodRouting",
+  :rails_root => Dir.pwd,
+  :load_path => [ 'app/workers/**/*.rb' ],
+  :rails_env => (ENV['RAILS_ENV'] || "development").dup
+}.merge(WorklingServer.parse_workling_options(ARGV))
+
+Daemons.run_proc(daemon_options[:app_name], daemon_options) do
+  Dir.chdir(workling_options[:rails_root])
+  WorklingServer.run(workling_options)
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: elecnix-workling
 version: !ruby/object:Gem::Version 
-  version: 0.4.2
+  version: 0.4.9
 platform: ruby
 authors: 
 - Rany Keddo
@@ -15,8 +15,8 @@ dependencies: []
 
 description: easily do background work in rails, without commiting to a particular runner. comes with starling, bj and spawn runners.
 email: nicolas@marchildon.net
-executables: []
-
+executables: 
+- workling_client
 extensions: []
 
 extra_rdoc_files: []
@@ -26,78 +26,6 @@ files:
 - VERSION.yml
 - README.markdown
 - TODO.markdown
-- lib/workling.rb
-- lib/rude_q
-- lib/rude_q/client.rb
-- lib/tasks
-- lib/workling
-- lib/workling/remote.rb
-- lib/workling/rudeq.rb
-- lib/workling/remote
-- lib/workling/remote/runners
-- lib/workling/remote/runners/rudeq_runner.rb
-- lib/workling/remote/runners/not_remote_runner.rb
-- lib/workling/remote/runners/backgroundjob_runner.rb
-- lib/workling/remote/runners/base.rb
-- lib/workling/remote/runners/starling_runner.rb
-- lib/workling/remote/runners/client_runner.rb
-- lib/workling/remote/runners/spawn_runner.rb
-- lib/workling/remote/invokers
-- lib/workling/remote/invokers/base.rb
-- lib/workling/remote/invokers/basic_poller.rb
-- lib/workling/remote/invokers/eventmachine_subscriber.rb
-- lib/workling/remote/invokers/threaded_poller.rb
-- lib/workling/rudeq
-- lib/workling/rudeq/client.rb
-- lib/workling/rudeq/poller.rb
-- lib/workling/routing
-- lib/workling/routing/base.rb
-- lib/workling/routing/class_and_method_routing.rb
-- lib/workling/base.rb
-- lib/workling/discovery.rb
-- lib/workling/clients
-- lib/workling/clients/memcache_queue_client.rb
-- lib/workling/clients/base.rb
-- lib/workling/clients/amqp_client.rb
-- lib/workling/return
-- lib/workling/return/store
-- lib/workling/return/store/iterator.rb
-- lib/workling/return/store/starling_return_store.rb
-- lib/workling/return/store/base.rb
-- lib/workling/return/store/memory_return_store.rb
-- lib/workling/return/store/rudeq_return_store.rb
-- test/invoker_eventmachine_subscription_test.rb
-- test/discovery_test.rb
-- test/spawn_runner_test.rb
-- test/starling_runner_test.rb
-- test/not_remote_runner_test.rb
-- test/starling_return_store_test.rb
-- test/rudeq_client_test.rb
-- test/mocks
-- test/mocks/logger.rb
-- test/mocks/spawn.rb
-- test/mocks/client.rb
-- test/mocks/rude_queue.rb
-- test/runners
-- test/runners/thread_runner.rb
-- test/clients
-- test/clients/memory_queue_client.rb
-- test/invoker_basic_poller_test.rb
-- test/rudeq_poller_test.rb
-- test/remote_runner_test.rb
-- test/test_helper.rb
-- test/workers
-- test/workers/analytics
-- test/workers/analytics/invites.rb
-- test/workers/util.rb
-- test/memcachequeue_client_test.rb
-- test/rescue_test.rb
-- test/class_and_method_routing_test.rb
-- test/rudeq_return_store_test.rb
-- test/invoker_threaded_poller_test.rb
-- test/return_store_test.rb
-- test/memory_return_store_test.rb
-- test/rudeq_runner_test.rb
 has_rdoc: true
 homepage: http://github.com/elecnix/workling
 post_install_message: 

data/lib/rude_q/client.rb

@@ -1,11 +0,0 @@
-#
-#  A RudeQ client that behvaes somewhat like memcache-client
-#
-module RudeQ
-  class Client    
-    def initialize(*args); super(); end   
-    def set(key, value); RudeQueue.set(key, value); end;
-    def get(key); RudeQueue.get(key); end;
-    def stats; ActiveRecord::Base.connection; end
-  end
-end

data/lib/workling.rb

@@ -1,150 +0,0 @@
-#
-#  I can haz am in your Workling are belong to us! 
-#
-module Workling
-  class WorklingError < StandardError; end
-  class WorklingNotFoundError < WorklingError; end
-  class WorklingConnectionError < WorklingError; end
-  class QueueserverNotFoundError < WorklingError
-    def initialize
-      super "config/workling.yml configured to connect to queue server on #{ Workling.config[:listens_on] } for this environment. could not connect to queue server on this host:port. for starling users: pass starling the port with -p flag when starting it. If you don't want to use Starling, then explicitly set Workling::Remote.dispatcher (see README for an example)"
-    end
-  end
-
-  class ConfigurationError < WorklingError
-    def initialize
-      super File.exist?(File.join(RAILS_ROOT, 'config', 'starling.yml')) ? 
-        "config/starling.yml has been depracated. rename your config file to config/workling.yml then try again!" :
-        "config/workling.yml could not be loaded. check out README.markdown to see what this file should contain. "
-    end
-  end
-  
-  mattr_accessor :load_path
-  @@load_path = [ File.expand_path(File.join(File.dirname(__FILE__), '../../../../app/workers')) ]
-  VERSION = "0.4.2.2"
-  
-  #
-  # determine the runner to use if nothing is specifically set. workling will try to detect
-  # starling, spawn, or bj, in that order. if none of these are found, notremoterunner will
-  # be used. 
-  #
-  # this can be overridden by setting Workling::Remote.dispatcher, eg:
-  #   Workling::Remote.dispatcher = Workling::Remote::Runners::StarlingRunner.new
-  #
-  def self.default_runner
-    if RAILS_ENV == "test"
-      Workling::Remote::Runners::NotRemoteRunner.new
-    elsif starling_installed?
-      Workling::Remote::Runners::StarlingRunner.new
-    elsif spawn_installed?
-      Workling::Remote::Runners::SpawnRunner.new
-    elsif bj_installed?
-      Workling::Remote::Runners::BackgroundjobRunner.new
-    else
-      Workling::Remote::Runners::NotRemoteRunner.new
-    end
-  end
-  
-  #
-  # gets the worker instance, given a class. the optional method argument will cause an 
-  # exception to be raised if the worker instance does not respoind to said method. 
-  #
-  def self.find(clazz, method = nil)
-    begin
-      inst = clazz.to_s.camelize.constantize.new 
-    rescue NameError
-      raise_not_found(clazz, method)
-    end
-    raise_not_found(clazz, method) if method && !inst.respond_to?(method)
-    inst
-  end
-  
-  # returns Workling::Return::Store.instance. 
-  def self.return
-    Workling::Return::Store.instance
-  end
-
-  # is spawn installed?
-  def self.spawn_installed?
-    begin
-      require 'spawn'
-    rescue LoadError
-    end
-
-    Object.const_defined? "Spawn"
-  end
-
-  # is starling installed?  
-  def self.starling_installed?
-    begin
-      require 'starling' 
-    rescue LoadError
-    end
-      
-    Object.const_defined? "Starling"
-  end
-
-  # is bj installed?
-  def self.bj_installed?
-    Object.const_defined? "Bj"
-  end
-  
-  # tries to load fiveruns-memcache-client. if this isn't found, 
-  # memcache-client is searched for. if that isn't found, don't do anything. 
-  def self.try_load_a_memcache_client
-    begin
-      gem 'fiveruns-memcache-client'
-      require 'memcache'
-    rescue Gem::LoadError
-      begin
-        gem 'memcache-client'
-        require 'memcache'
-      rescue Gem::LoadError
-        Workling::Base.logger.info "WORKLING: couldn't find a memcache client - you need one for the starling runner. "
-      end
-    end
-  end
-  
-  # attempts to load amqp and writes out descriptive error message if not present
-  def self.try_load_an_amqp_client
-    begin
-      require 'mq'
-    rescue Exception => e
-      raise WorklingError.new(
-        "WORKLING: couldn't find the ruby amqp client - you need it for the amqp runner. " \
-        "Install from github: gem sources -a http://gems.github.com/ && sudo gem install tmm1-amqp "
-      )
-    end
-  end
-  
-  #
-  #  returns a config hash. reads RAILS_ROOT/config/workling.yml
-  #
-  def self.config
-    begin
-      config_path = File.join(RAILS_ROOT, 'config', 'workling.yml')
-      @@config ||=  YAML.load_file(config_path)[RAILS_ENV || 'development'].symbolize_keys
-      @@config[:memcache_options].symbolize_keys! if @@config[:memcache_options]
-      @@config 
-    rescue
-       # config files could not be read correctly
-      raise ConfigurationError.new
-    end
-  end
-  
-  #
-  #  Raises exceptions thrown inside of the worker. normally, these are logged to 
-  #  logger.error. it's easy to miss these log calls while developing, though. 
-  #
-  mattr_accessor :raise_exceptions
-  @@raise_exceptions = (RAILS_ENV == "test" || RAILS_ENV == "development")
-  
-  def self.raise_exceptions?
-    @@raise_exceptions
-  end
-  
-  private
-    def self.raise_not_found(clazz, method)
-      raise Workling::WorklingNotFoundError.new("could not find #{ clazz }:#{ method } workling. ") 
-    end
-end