xing-gearman-ruby 1.0.0 → 1.1.0

data/HOWTO

@@ -0,0 +1,146 @@
+= GEARMAN
+
+"Gearman provides a generic application framework to farm out work to other 
+machines or processes that are better suited to do the work. It allows you to 
+do work in parallel, to load balance processing, and to call functions between 
+languages. It can be used in a variety of applications, from high-availability 
+web sites to the transport of database replication events. In other words, it 
+is the nervous system for how distributed processing communicates."
+
+ - http://www.gearman.org/
+
+
+== Setting up a basic environment
+
+A very basic German environment will look like this:
+
+                       ----------  
+                       | Client |  
+                       ----------  
+                           |
+                     --------------     
+                     | Job Server |     
+                     --------------     
+                           |
+    ----------------------------------------------
+    |              |              |              |
+----------     ----------     ----------     ----------
+| Worker |     | Worker |     | Worker |     | Worker |
+----------     ----------     ----------     ----------
+
+And the behavior will be the following:
+
+ * JobServer: Acts as a message passing point.
+ * Client: Sends tasks to the JobServer. Will be connected to only one JobServer
+           in case more than one exits for failover purposes.
+ * Worker: Anounce his 'abilities' to the JobServer and waits for tasks.
+
+For the JobServer we recommend to use the offical Perl version, there's also a
+more performant C implementation of the server with support for persistent
+queues, bells and whistles but is not stable enough for production use at the 
+time of this document was wrote.
+
+The Client and the Worker can be implemented in any language. This way you can
+send tasks from a Ruby client server, to a Perl or C worker in order to get 
+better performance.
+
+== Installing the required software
+
+For the JobServer we recommend to use the offical Perl version, to install it:
+
+ * Mac OS X: sudo port install p5-gearman-server
+ * Debian/Ubuntu: sudo apt-get install gearman-server
+
+To get the Ruby libraries by Xing:
+
+ git clone git://github.com/xing/gearman-ruby.git
+
+== Gearman demo
+
+Now you're ready for you first experience with Gearman. In the cloned repository
+you'll find an 'examples' directory.
+
+Run the 'gearman_environment.sh' to build an environment like the one showed in
+the diagram above.
+
+  * Client: Will ask you for an arithmetic operation, like: 2+3
+            The code of the client is in: 'examples/calculus_client.rb'
+
+  * JobServer: The Perl server.
+
+  * Workers: You'll have 4 worker, one for each of the basic arithmetic 
+             operations.
+             The code of the worker is in: 'examples/calculus_worker.rb'
+
+There are other demos in the examples folder you can give a look at. Each demo usually
+consist in the client and server scripts.
+
+=== Creating clients and tasks
+
+In order to get a job scheduled by a Gearman server using the gearman ruby library, there
+are three main objects you must interact with: Gearman::Client, Gearman::Task and Gearman::TaskSet.
+Let's review all of them briefly:
+
+  - Gearman::Client -> the portion of the library storing the data about the connection to 
+                       the Gearman server.
+  - Gearman::Task -> a job execution request that will be dispatched by the Gearman server to 
+                     worker and whose result data will be returned to the client.
+  - Gearman::TaskSet -> a collection of tasks to be executed. The Taskset object will track the
+                        execution of the tasks with the info returned from the Gearman server
+                        and notify the client with the results or errors when all the tasks
+                        have completed their execution.
+
+To send a new task to the Gearman server, the client must build a new Gearman::Task object, add it to
+a Gearman::TaskSet that must hold a reference to a Gearman::Client and send the wait message to
+the TaskSet.
+The following code taken from examples/client.rb shows the process:
+
+----------------------------------------------------
+servers = ['localhost:4730', 'localhost:4731']
+  
+client = Gearman::Client.new(servers)
+taskset = Gearman::TaskSet.new(client)
+
+task = Gearman::Task.new('sleep', 20)
+task.on_complete {|d| puts d }
+
+taskset.add_task(task)
+taskset.wait(100)
+----------------------------------------------------
+
+The name of the function to be executed is the first parameter to the constructor of the Task object.
+Take into account that the string you pass as a parameter will be used 'as it' by the Gearman server
+to locate a suitable worker for that function.
+The second parameter is the argument to be sent to the worker that will execute, if the arguments for
+the task are complex, a serialization format like YAML or XML must be agreeded with the workers.
+The last and optional parameter is a hash of options. The following options are currently available:
+
+  - :priority -> (:high | :low) the priority of the job, a high priority job is executed before a low on
+  - :background -> (true | false) a background task will return no further information to the client.
+
+The execution of a task in a Gearman remote worker can fail, the worker can throw an exception, etc.
+All these events can be handled by the client of the ruby library registering callback blocks.
+The following events are currently available:
+
+  - on_complete -> the task was executed succesfully
+  - on_fail -> the task fail for some unknown reason
+  - on_retry -> after failure, the task is gonna be retried, the number of retries is passed
+  - on_exception -> the remote worker send an exception notification, the exception text is passed
+  - on_stauts -> a status update is sent by the remote worker
+
+In order to receive exception notifications in the client, this option must be sent in the server, the 
+method option_request can be used this task. The following example, extracted from the examples/client_exception.rb 
+demo script shows the process:
+
+----------------------------------------------------
+client = Gearman::Client.new(servers)
+#try this out
+client.option_request("exceptions")
+----------------------------------------------------
+
+This feature will only works if the server and workers have implemented support for the OPT_REQ and WORK_EXCEPTION
+messages of the Gearman protocol.
+
+
+Enjoy.
+

data/TODO

@@ -0,0 +1,8 @@
+- Failover strategies
+  - Client:
+     * If connected for the first time, try to connect to at least one server from the server array
+     * If already connected to a server, and it goes down, the client should go down as well
+  - Worker:
+     * If connected for the first time, try to connect to as many servers as it can. 
+       Loop trough the bad servers, trying to reconnect to them as well.
+     * If a already connected to a server, and it goes down, wait and try to reconnect again.      

data/VERSION.yml

@@ -1,4 +1,4 @@
 --- 
-:major: 1
-:minor: 0
 :patch: 0
+:major: 1
+:minor: 1

data/examples/calculus_client.rb

@@ -0,0 +1,39 @@
+#!/usr/bin/env ruby
+require 'rubygems'
+require '../lib/gearman'
+#Gearman::Util.debug = true
+
+# Connect to the local server (at the default port 7003) 
+client = Gearman::Client.new('localhost')
+taskset = Gearman::TaskSet.new(client)
+
+# Get something to echo
+puts '[client] Write a basic arithmetic operation:'
+input = gets
+
+operations = input.chomp.scan(/\d+[\+\-\*\/]\d+/).compact
+puts "[client] The following operations were found: #{operations.inspect}"
+
+# Setup a task for operation
+operations.each do |op|
+  # Determining the operation
+  case op
+    when /\+/
+      type, data = 'addition', op.split('+') 
+    when /\-/
+      type, data = 'subtraction', op.split('-') 
+    when /\*/
+      type, data = 'multiplication', op.split('*') 
+    when /\//
+      type, data = 'division', op.split('/') 
+  end
+
+  task = Gearman::Task.new(type, Marshal.dump(data.map {|v| v.to_i}))
+  task.on_complete {|r| puts "[client] #{type} result is: #{r}" }
+
+  # Sending the task to the server
+  puts "[client] Sending values: #{data.inspect}, to the '#{type}' worker"
+  taskset.add_task(task)
+  taskset.wait(100)
+end
+

data/examples/calculus_worker.rb

@@ -0,0 +1,45 @@
+#!/usr/bin/env ruby
+require 'rubygems'
+require '../lib/gearman'
+
+#Gearman::Util.debug = true
+
+worker = Gearman::Worker.new('localhost')
+worker.reconnect_sec = 2
+
+# Additon ability
+worker.add_ability('addition') do |data,job|
+  values = Marshal.load(data)
+  puts "[addition_worker] Calculating #{values.inspect}..."
+  # sleep 5
+  values.first + values.last
+end
+
+# Subtraction ability
+worker.add_ability('subtraction') do |data,job|
+  values = Marshal.load(data)
+  puts "[subtraction_worker] Calculating #{values.inspect}..."
+  # sleep 5
+  values.first - values.last
+end
+
+# Multiplication worker
+worker.add_ability('multiplication') do |data,job|
+  values = Marshal.load(data)
+  puts "[multiplication_worker] Calculating #{values.inspect}..."
+  # sleep 5
+  values.first * values.last
+end
+
+# Division worker
+worker.add_ability('division') do |data,job|
+  values = Marshal.load(data)
+  puts "[division_worker] Calculating #{data.inspect}..."
+  # sleep 5
+  values.first / values.last
+end
+
+# Running the workers
+loop do
+  worker.work
+end

data/examples/client_background.rb

@@ -0,0 +1,15 @@
+require 'rubygems'
+#require 'gearman'
+require '../lib/gearman'
+Gearman::Util.debug = true
+
+servers = ['localhost:4730', 'localhost:4731']
+  
+client = Gearman::Client.new(servers)
+taskset = Gearman::TaskSet.new(client)
+
+task = Gearman::Task.new('sleep', 20, { :background => true })
+task.on_complete {|d| puts d }
+
+taskset.add_task(task)
+taskset.wait(100)

data/examples/client_exception.rb

@@ -0,0 +1,18 @@
+require 'rubygems'
+require '../lib/gearman'
+Gearman::Util.debug = true
+
+servers = ['localhost:4730']
+
+client = Gearman::Client.new(servers)
+#try this out
+client.option_request("exceptions")
+
+taskset = Gearman::TaskSet.new(client)
+
+task = Gearman::Task.new('fail_with_exception', 20)
+task.on_complete {|d| puts d }
+task.on_exception {|message| puts message; false}
+
+taskset.add_task(task)
+taskset.wait(100)

data/examples/client_prefix.rb

@@ -0,0 +1,17 @@
+require 'rubygems'
+#require 'gearman'
+require '../lib/gearman'
+Gearman::Util.debug = true
+
+servers = ['localhost:4730', 'localhost:4731']
+  
+ability_name_with_prefix = Gearman::Util.ability_name_with_prefix("test","sleep")
+
+client = Gearman::Client.new(servers)
+taskset = Gearman::TaskSet.new(client)
+
+task = Gearman::Task.new(ability_name_with_prefix, 20)
+task.on_complete {|d| puts d }
+
+taskset.add_task(task)
+taskset.wait(100)

data/examples/gearman_environment.sh

@@ -0,0 +1,25 @@
+#!/bin/bash  
+
+# Start Gearmand
+echo ' + Starting Gearmand'
+gearmand --daemon --pidfile=/tmp/gearmand.pid
+
+# Start the client and the worker(s)
+echo ' + Starting calculus_worker.rb'
+ruby calculus_worker.rb & 
+
+sleep 3
+
+echo ' + Starting calculus_client.rb'
+ruby calculus_client.rb 
+
+echo ' +++ Example finished +++ '
+
+# Stop Gearmand
+echo ' - Stopping Gearmand'
+kill -9 `cat /tmp/gearmand.pid`
+
+# Stop the workers
+echo ' - Stopping calculus_worker.rb'
+kill -9 `ps ax|grep calculus_worker|grep ruby|awk -F' ' '{print $1}'`
+

data/examples/worker_exception.rb

@@ -0,0 +1,14 @@
+require 'rubygems'
+require '../lib/gearman'
+
+Gearman::Util.debug = true
+
+servers = ['localhost:4730']
+w = Gearman::Worker.new(servers)
+
+# Add a handler for a "sleep" function that takes a single argument, the
+# number of seconds to sleep before reporting success.
+w.add_ability('fail_with_exception') do |data,job|
+  raise Exception.new("fooexception")
+end
+loop { w.work }

data/examples/worker_prefix.rb

@@ -0,0 +1,25 @@
+require 'rubygems'
+#require 'gearman'
+require '../lib/gearman'
+
+Gearman::Util.debug = true
+
+servers = ['localhost:4730', 'localhost:4731']
+w = Gearman::Worker.new(servers)
+
+ability_name_with_prefix = Gearman::Util.ability_name_with_prefix("test","sleep")
+
+# Add a handler for a "sleep" function that takes a single argument, the
+# number of seconds to sleep before reporting success.
+w.add_ability(ability_name_with_prefix) do |data,job|
+ seconds = data
+ (1..seconds.to_i).each do |i|
+   sleep 1
+   print i
+   # Report our progress to the job server every second.
+   job.report_status(i, seconds)
+ end
+ # Report success.
+ true
+end
+loop { w.work }

data/gearman-ruby.gemspec

@@ -2,11 +2,11 @@
 
 Gem::Specification.new do |s|
   s.name = %q{gearman-ruby}
-  s.version = "1.0.0"
+  s.version = "1.1.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Daniel Erat", "Ladislav Martincik"]
-  s.date = %q{2009-06-25}
+  s.date = %q{2009-07-16}
   s.description = %q{Library for the Gearman distributed job system}
   s.email = %q{ladislav.martincik@xing.com}
   s.extra_rdoc_files = [
@@ -15,9 +15,11 @@ Gem::Specification.new do |s|
   ]
   s.files = [
     ".gitignore",
+     "HOWTO",
      "LICENSE",
      "README",
      "Rakefile",
+     "TODO",
      "VERSION.yml",
      "examples/Net/Gearman/Client.php",
      "examples/Net/Gearman/Connection.php",
@@ -30,13 +32,21 @@ Gem::Specification.new do |s|
      "examples/Net/Gearman/Set.php",
      "examples/Net/Gearman/Task.php",
      "examples/Net/Gearman/Worker.php",
+     "examples/calculus_client.rb",
+     "examples/calculus_worker.rb",
      "examples/client.php",
      "examples/client.rb",
+     "examples/client_background.rb",
+     "examples/client_exception.rb",
+     "examples/client_prefix.rb",
+     "examples/gearman_environment.sh",
      "examples/scale_image.rb",
      "examples/scale_image_worker.rb",
      "examples/server.rb",
      "examples/worker.php",
      "examples/worker.rb",
+     "examples/worker_exception.rb",
+     "examples/worker_prefix.rb",
      "gearman-ruby.gemspec",
      "lib/gearman.rb",
      "lib/gearman/client.rb",
@@ -49,29 +59,37 @@ Gem::Specification.new do |s|
      "test/client_test.rb",
      "test/mock_client_test.rb",
      "test/mock_worker_test.rb",
+     "test/util_test.rb",
      "test/worker_test.rb"
   ]
-  s.has_rdoc = true
   s.homepage = %q{http://github.com/xing/gearman-ruby}
   s.rdoc_options = ["--charset=UTF-8"]
   s.require_paths = ["lib"]
-  s.rubygems_version = %q{1.3.1}
+  s.rubygems_version = %q{1.3.4}
   s.summary = %q{Library for the Gearman distributed job system}
   s.test_files = [
     "test/client_test.rb",
      "test/mock_client_test.rb",
      "test/mock_worker_test.rb",
+     "test/util_test.rb",
      "test/worker_test.rb",
+     "examples/calculus_client.rb",
+     "examples/calculus_worker.rb",
      "examples/client.rb",
+     "examples/client_background.rb",
+     "examples/client_exception.rb",
+     "examples/client_prefix.rb",
      "examples/scale_image.rb",
      "examples/scale_image_worker.rb",
      "examples/server.rb",
-     "examples/worker.rb"
+     "examples/worker.rb",
+     "examples/worker_exception.rb",
+     "examples/worker_prefix.rb"
   ]
 
   if s.respond_to? :specification_version then
     current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
-    s.specification_version = 2
+    s.specification_version = 3
 
     if Gem::Version.new(Gem::RubyGemsVersion) >= Gem::Version.new('1.2.0') then
     else

data/lib/gearman/client.rb

@@ -23,8 +23,21 @@ class Client
     @server_counter = -1
     @bad_servers = []
   end
-  attr_reader :job_servers, :bad_servers  
-  attr_accessor :test_hostport, :task_create_timeout_sec  
+  attr_reader :job_servers, :bad_servers
+  attr_accessor :test_hostport, :task_create_timeout_sec
+
+  ##
+  # Set the options
+  #
+  # @options options to pass to the servers  "exeptions"
+  def option_request(opts)
+    Util.log "Send options request with #{opts}"
+    request = Util.pack_request("option_req", opts)
+    sock= self.get_socket(self.get_job_server)
+    Util.send_request(sock, request)
+    response = Util.read_response(sock, 20)
+    raise ProtocolError, response[1] if response[0]==:error
+  end
 
   ##
   # Set the job servers to be used by this client.

data/lib/gearman/task.rb

@@ -7,6 +7,7 @@ module Gearman
 # == Description
 # A task submitted to a Gearman job server.
 class Task
+
   ##
   # Create a new Task object.
   #
@@ -16,8 +17,8 @@ class Task
   def initialize(func, arg='', opts={})
     @func = func.to_s
     @arg = arg or ''  # TODO: use something more ref-like?
-    %w{uniq on_complete on_fail on_retry on_status retry_count
-       high_priority}.map {|s| s.to_sym }.each do |k|
+    %w{uniq on_complete on_fail on_retry on_exception on_status retry_count
+       priority background}.map {|s| s.to_sym }.each do |k|
       instance_variable_set "@#{k}", opts[k]
       opts.delete k
     end
@@ -29,7 +30,7 @@ class Task
     @retries_done = 0
     @hash = nil
   end
-  attr_accessor :uniq, :retry_count, :high_priority
+  attr_accessor :uniq, :retry_count, :priority, :background
   attr_reader :successful, :func, :arg
 
   ##
@@ -62,6 +63,14 @@ class Task
     @on_retry = f
   end
 
+  ##
+  # Set a block of code to be executed when a remote exception is sent by a worker.
+  # The block will receive the message of the exception passed from the worker.
+  # The user can return true for retrying or false to mark it as finished
+  def on_exception(&f)
+    @on_exception = f
+  end
+
   ##
   # Set a block of code to be executed when we receive a status update for
   # this task.  The block will receive two arguments, a numerator and
@@ -94,6 +103,20 @@ class Task
     true
   end
 
+  ##
+  # Record an exception and check whether we should be retried.
+  #
+  # @return  true if we should be resubmitted; false otherwise
+  def handle_exception(exception)
+    if @on_exception
+      should_retry = @on_exception.call(exception)
+      @retries_done += 1 if should_retry
+      should_retry
+    else
+      false
+    end
+  end
+
   ##
   # Handle a status update for the task.
   def handle_status(numerator, denominator)
@@ -116,11 +139,21 @@ class Task
   ##
   # Construct a packet to submit this task to a job server.
   #
-  # @param background  ??
   # @return            String representation of packet
-  def get_submit_packet(background=false)
-    mode = 'submit_job' +
-      (background ? '_bg' : @high_priority ? '_high' : '')
+  def get_submit_packet()
+    mode = 'submit_job'
+    if(@priority)
+      if(@priority == :high)
+        mode += "_high"
+      elsif(@priority == :low)
+        mode += "_low"
+      end
+    end
+
+    if(@background)
+      mode += "_bg"
+    end
+
     Util::pack_request(mode, [func, get_uniq_hash, arg].join("\0"))
   end
 end

data/lib/gearman/taskset.rb

@@ -92,13 +92,16 @@ class TaskSet
   def handle_job_created(hostport, data)
     Util.log "Got job_created with handle #{data} from #{hostport}"
     if not @task_waiting_for_handle
-      raise ProtocolError, "Got unexpected job_created notification " +
-        "with handle #{data} from #{hostport}"
+      raise ProtocolError, "Got unexpected job_created notification " + "with handle #{data} from #{hostport}"
     end
     js_handle = Util.handle_to_str(hostport, data)
     task = @task_waiting_for_handle
     @task_waiting_for_handle = nil
-    (@tasks_in_progress[js_handle] ||= []) << task
+    if(task.background)
+      @finished_tasks << task
+    else
+      (@tasks_in_progress[js_handle] ||= []) << task
+    end
     nil
   end
   private :handle_job_created
@@ -124,9 +127,32 @@ class TaskSet
     end
     nil
   end
-
   private :handle_work_complete
 
+  ##
+  # Handle a 'work_exception' response from a job server.
+  #
+  # @param hostport  "host:port" of job server
+  # @param data      data returned in packet from server
+  def handle_work_exception(hostport, data)
+    handle, exception = data.split("\0", 3)
+    Util.log "Got work_exception with handle #{handle} from #{hostport}: '#{exception}'"
+    js_handle = Util.handle_to_str(hostport, handle)
+    tasks = @tasks_in_progress.delete(js_handle)
+    if not tasks
+      raise ProtocolError, "Got unexpected work_status with handle " +
+        "#{handle} from #{hostport} (no task by that name)"
+    end
+    tasks.each do |t|
+      if t.handle_exception(exception)
+        add_task_internal(t, false)
+      else
+        @finished_tasks << t
+      end
+    end
+  end
+  private :handle_work_exception
+
   ##
   # Handle a 'work_fail' response from a job server.
   #
@@ -189,6 +215,8 @@ class TaskSet
       handle_work_fail(hostport, data)
     when :work_status
       handle_work_status(hostport, data)
+    when :work_exception
+      handle_work_exception(hostport, data)
     else
       Util.log "Got #{type.to_s} from #{hostport}"
     end
@@ -229,7 +257,7 @@ class TaskSet
     @sockets.values.each {|s| @client.return_socket(s) }
     @sockets = {}
     @finished_tasks.each do |t|
-      if not t.successful
+      if ( (t.background.nil? || t.background == false) && !t.successful)
         Util.log "Taskset failed"
         return false
       end

data/lib/gearman/testlib.rb

@@ -33,10 +33,10 @@ class FakeJobServer
     @tester.assert_true(sock.closed?)
   end
 
-  def expect_request(sock, exp_type, exp_data='')
-    head = sock.recv(12)
+  def expect_request(sock, exp_type, exp_data='', size=12)
+    head = sock.recv(size)
     magic, type, len = head.unpack('a4NN')
-    @tester.assert_equal("\0REQ", magic)
+    @tester.assert("\0REQ" == magic || "\000REQ" == magic)
     @tester.assert_equal(Gearman::Util::NUMS[exp_type.to_sym], type)
     data = len > 0 ? sock.recv(len) : ''
     @tester.assert_equal(exp_data, data)
@@ -52,8 +52,7 @@ class FakeJobServer
   end
 
   def send_response(sock, type, data='', bogus_size=nil)
-    type_num = Gearman::Util::NUMS[type.to_sym]
-    raise RuntimeError, "Invalid type #{type}" if not type_num
+    type_num = Gearman::Util::NUMS[type.to_sym] || 0
     response = "\0RES" + [type_num, (bogus_size or data.size)].pack('NN') + data
     sock.write(response)
   end

data/lib/gearman/util.rb

@@ -24,8 +24,11 @@ class Util
 
     6  => :noop,            # J->W: --
     7  => :submit_job,      # C->J: FUNC[0]UNIQ[0]ARGS
-    21 => :submit_job_high, # C->J: FUNC[0]UNIQ[0]ARGS
-    18 => :submit_job_bg,   # C->J: FUNC[0]UNIQ[0]ARGS
+    33 => :submit_job_low,     # C->J: FUNC[0]UNIQ[0]ARGS
+    34 => :submit_job_low_bg,  # C->J: FUNC[0]UNIQ[0]ARGS
+    21 => :submit_job_high,    # C->J: FUNC[0]UNIQ[0]ARGS
+    32 => :submit_job_high_bg, # C->J: FUNC[0]UNIQ[0]ARGS
+    18 => :submit_job_bg,      # C->J: FUNC[0]UNIQ[0]ARGS
 
     8  => :job_created,     # J->C: HANDLE
     9  => :grab_job,        # W->J: --
@@ -36,9 +39,14 @@ class Util
     13 => :work_complete,   # W->J/C: HANDLE[0]RES
     14 => :work_fail,       # W->J/C: HANDLE
 
+    25 => :work_exception,  # W->J: HANDLE[0]ARG
+    26 => :option_req,      # C->J: TEXT
+    27 => :option_res,      # J->C: TEXT
+
     15 => :get_status,      # C->J: HANDLE
     20 => :status_res,      # C->J: HANDLE[0]KNOWN[0]RUNNING[0]NUM[0]DENOM
 
+    25 => :work_exception,  # W->J/C: HANDLE[0]ARG
     16 => :echo_req,        # ?->J: TEXT
     17 => :echo_res,        # J->?: TEXT
 
@@ -198,15 +206,15 @@ class Util
       raise ServerDownException.new(ex.message)
     end
   end
-  
+
   def Util.ability_name_with_prefix(prefix,name)
     "#{prefix}\t#{name}"
   end
-  
+
   class << self
     alias :ability_name_for_perl :ability_name_with_prefix
   end
-  
+
 end
 
 end

data/lib/gearman/worker.rb

@@ -259,17 +259,25 @@ class Worker
       return false
     end
 
-    ret = ability.run(data, Job.new(sock, handle))
+    exception = nil
+    begin
+      ret = ability.run(data, Job.new(sock, handle))
+    rescue Exception => e
+      exception = e
+    end
+
 
-    cmd = nil
-    if ret
+    cmd = if ret && exception.nil?
       ret = ret.to_s
       Util.log "Sending work_complete for #{handle} with #{ret.size} byte(s) " +
         "to #{hostport}"
-      cmd = Util.pack_request(:work_complete, "#{handle}\0#{ret}")
-    else
+      Util.pack_request(:work_complete, "#{handle}\0#{ret}")
+    elsif exception.nil?
       Util.log "Sending work_fail for #{handle} to #{hostport}"
-      cmd = Util.pack_request(:work_fail, handle)
+      Util.pack_request(:work_fail, handle)
+    elsif exception
+      Util.log "Sending work_exception for #{handle} to #{hostport}"
+      Util.pack_request(:work_exception, "#{handle}\0#{exception.message}")
     end
 
     Util.send_request(sock, cmd)

data/test/client_test.rb

@@ -108,4 +108,38 @@ class TestClient < Test::Unit::TestCase
     end
     assert(should_be_true)
   end
+
+  def test_option_request_exceptions
+    this_server = FakeJobServer.new(self)
+    Thread.new do
+      server_socket = this_server.expect_connection
+      this_server.expect_request(server_socket, "option_req", "exceptions")
+      this_server.send_response(server_socket, :job_created, 'a')
+    end
+    client = Gearman::Client.new
+    hostport = "localhost:#{this_server.port}"
+    client.job_servers = [hostport]
+    client.option_request("exceptions")
+  end
+
+  def test_option_request_bad
+    this_server = FakeJobServer.new(self)
+    Thread.new do
+      server_socket = this_server.expect_connection
+      this_server.expect_request(server_socket, "option_req", "cccceptionsccc")
+      this_server.send_response(server_socket, :exception, 'a')
+    end
+
+    client = Gearman::Client.new
+    hostport = "localhost:#{this_server.port}"
+    client.job_servers = [hostport]
+    begin
+      client.option_request("cccceptionsccc")
+      assert(false)
+    rescue Gearman::ProtocolError
+      assert(true)
+    end
+  end
+
+
 end

data/test/mock_client_test.rb

@@ -56,6 +56,127 @@ class TestClient < Test::Unit::TestCase
     assert_equal(15, res2)
   end
 
+  ##
+  # Tests that the high priority option can be set in a job request
+  def test_client_submit_priority_high
+    server = FakeJobServer.new(self)
+    client, task1, task2, taskset, sock, res1, res2 = nil
+
+    s = TestScript.new
+    c = TestScript.new
+
+    server_thread = Thread.new { s.loop_forever }.run
+    client_thread = Thread.new { c.loop_forever }.run
+
+    c.exec { client = Gearman::Client.new("localhost:#{server.port}") }
+
+    c.exec { task1 = Gearman::Task.new('add', '5 2', { :priority => :high }) }
+    c.exec { task1.on_complete {|d| res1 = d.to_i } }
+    c.exec { taskset = Gearman::TaskSet.new(client) }
+    c.exec { taskset.add_task(task1) }
+    s.exec { sock = server.expect_connection }
+    s.wait
+
+    s.exec { server.expect_request(sock, :submit_job_high, "add\000\0005 2") }
+  end
+
+  ##
+  # Tests that the low priority option can be set in a job request
+  def test_client_submit_priority_low
+    server = FakeJobServer.new(self)
+    client, task1, task2, taskset, sock, res1, res2 = nil
+
+    s = TestScript.new
+    c = TestScript.new
+
+    server_thread = Thread.new { s.loop_forever }.run
+    client_thread = Thread.new { c.loop_forever }.run
+
+    c.exec { client = Gearman::Client.new("localhost:#{server.port}") }
+
+    c.exec { task1 = Gearman::Task.new('add', '5 2', { :priority => :low }) }
+    c.exec { task1.on_complete {|d| res1 = d.to_i } }
+    c.exec { taskset = Gearman::TaskSet.new(client) }
+    c.exec { taskset.add_task(task1) }
+    s.exec { sock = server.expect_connection }
+    s.wait
+
+    s.exec { server.expect_request(sock, :submit_job_low, "add\000\0005 2") }
+  end
+
+
+  ##
+  # Check that the client sends a correct background job request
+  def test_client_submit_background
+    server = FakeJobServer.new(self)
+    client, task1, task2, taskset, sock, res1, res2 = nil
+
+    s = TestScript.new
+    c = TestScript.new
+
+    server_thread = Thread.new { s.loop_forever }.run
+    client_thread = Thread.new { c.loop_forever }.run
+
+    c.exec { client = Gearman::Client.new("localhost:#{server.port}") }
+
+    c.exec { task1 = Gearman::Task.new('add', '5 2', { :background => :true }) }
+    c.exec { task1.on_complete {|d| res1 = d.to_i } }
+    c.exec { taskset = Gearman::TaskSet.new(client) }
+    c.exec { taskset.add_task(task1) }
+    s.exec { sock = server.expect_connection }
+    s.wait
+
+    s.exec { server.expect_request(sock, :submit_job_bg, "add\000\0005 2") }
+  end
+
+  ##
+  # Check that the client sends a correct background job with high priority request
+  def test_client_submit_background
+    server = FakeJobServer.new(self)
+    client, task1, task2, taskset, sock, res1, res2 = nil
+
+    s = TestScript.new
+    c = TestScript.new
+
+    server_thread = Thread.new { s.loop_forever }.run
+    client_thread = Thread.new { c.loop_forever }.run
+
+    c.exec { client = Gearman::Client.new("localhost:#{server.port}") }
+
+    c.exec { task1 = Gearman::Task.new('add', '5 2', { :background => :true, :priority => :high }) }
+    c.exec { task1.on_complete {|d| res1 = d.to_i } }
+    c.exec { taskset = Gearman::TaskSet.new(client) }
+    c.exec { taskset.add_task(task1) }
+    s.exec { sock = server.expect_connection }
+    s.wait
+
+    s.exec { server.expect_request(sock, :submit_job_high_bg, "add\000\0005 2") }
+  end
+
+  ##
+  # Check that the client sends a correct background job with low priority request
+  def test_client_submit_background
+    server = FakeJobServer.new(self)
+    client, task1, task2, taskset, sock, res1, res2 = nil
+
+    s = TestScript.new
+    c = TestScript.new
+
+    server_thread = Thread.new { s.loop_forever }.run
+    client_thread = Thread.new { c.loop_forever }.run
+
+    c.exec { client = Gearman::Client.new("localhost:#{server.port}") }
+
+    c.exec { task1 = Gearman::Task.new('add', '5 2', { :background => :true, :priority => :low }) }
+    c.exec { task1.on_complete {|d| res1 = d.to_i } }
+    c.exec { taskset = Gearman::TaskSet.new(client) }
+    c.exec { taskset.add_task(task1) }
+    s.exec { sock = server.expect_connection }
+    s.wait
+
+    s.exec { server.expect_request(sock, :submit_job_low_bg, "add\000\0005 2") }
+  end
+  
   ##
   # Test Client#do_task.
   def test_do_task
@@ -185,6 +306,39 @@ class TestClient < Test::Unit::TestCase
     assert_equal(false, setres)
   end
 
+  def test_exception
+    server = FakeJobServer.new(self)
+    client, task, taskset, sock = nil
+    res,exception, setres = nil
+
+    s = TestScript.new
+    c = TestScript.new
+
+    server_thread = Thread.new { s.loop_forever }.run
+    client_thread = Thread.new { c.loop_forever }.run
+
+    c.exec { client = Gearman::Client.new("localhost:#{server.port}") }
+
+    c.exec { task = Gearman::Task.new('func2', 'b') }
+    c.exec { task.on_complete {|d| res = d } }
+    c.exec { task.on_exception {|message| exception=message; false } }
+    c.exec { taskset = Gearman::TaskSet.new(client) }
+    c.exec { taskset.add_task(task) }
+    s.exec { sock = server.expect_connection }
+
+    s.exec { server.expect_request(sock, :submit_job, "func2\000\000b") }
+    s.exec { server.send_response(sock, :job_created, 'b') }
+
+    s.exec { server.send_response(sock, :work_exception, "b\0exceptionmsg") }
+
+    c.exec { setres = taskset.wait }
+    c.wait
+    s.wait
+
+    assert_equal(nil, res)
+    assert_equal('exceptionmsg',exception)
+  end
+
   ##
   # Test that user-supplied uniq values are handled correctly.
   def test_uniq

data/test/mock_worker_test.rb

@@ -210,4 +210,48 @@ class TestWorker < Test::Unit::TestCase
     s.wait
     w.wait
   end
+
+  def test_exception
+    @server = FakeJobServer.new(self)
+    worker = nil
+    sock = nil
+
+    s = TestScript.new
+    w = TestScript.new
+
+    server_thread = Thread.new { s.loop_forever }.run
+    worker_thread = Thread.new { w.loop_forever }.run
+
+    # Create a worker and wait for it to connect to us.
+    w.exec {
+      worker = Gearman::Worker.new(
+        "localhost:#{@server.port}", { :client_id => 'test' })
+    }
+    s.exec { sock = @server.expect_connection }
+    s.wait
+
+    # After it connects, it should send its ID, and it should tell us its
+    # abilities when we report them.
+    s.exec { @server.expect_request(sock, :set_client_id, 'test') }
+    w.exec { worker.add_ability('echo') {|d,j| raise Exception.new("fooexception") } }
+    s.exec { @server.expect_request(sock, :can_do, 'echo') }
+
+    # It should try to grab a job when we tell it to work.
+    w.exec { worker.work }
+    s.exec { @server.expect_request(sock, :grab_job) }
+
+    # If we tell it there aren't any jobs, it should go to sleep.
+    s.exec { @server.send_response(sock, :no_job) }
+    s.exec { @server.expect_request(sock, :pre_sleep) }
+
+    # When we send it a noop, it should wake up and ask for a job again.
+    s.exec { @server.send_response(sock, :noop) }
+    s.exec { @server.expect_request(sock, :grab_job) }
+
+    # When we give it a job, it should raise an excpetion and notify the server
+    s.exec { @server.send_response(sock, :job_assign, "a\0echo\0foo") }
+    s.exec { @server.expect_request(sock, :work_exception, "a\0fooexception") }
+
+    s.wait
+  end
 end

data/test/util_test.rb

@@ -0,0 +1,17 @@
+#!/usr/bin/env ruby
+
+$:.unshift('../lib')
+require 'gearman'
+require 'gearman/testlib'
+require 'test/unit'
+
+class TestUtil < Test::Unit::TestCase
+
+  def test_ability_prefix_name_builder
+    assert_equal(Gearman::Util.ability_name_with_prefix("test","a"),"test\ta")
+  end
+
+  def test_ability_name_for_perl
+    assert_equal(Gearman::Util.ability_name_for_perl("test","a"),"test\ta")
+  end  
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: xing-gearman-ruby
 version: !ruby/object:Gem::Version 
-  version: 1.0.0
+  version: 1.1.0
 platform: ruby
 authors: 
 - Daniel Erat
@@ -10,7 +10,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2009-06-25 00:00:00 -07:00
+date: 2009-07-16 00:00:00 -07:00
 default_executable: 
 dependencies: []
 
@@ -25,9 +25,11 @@ extra_rdoc_files:
 - README
 files: 
 - .gitignore
+- HOWTO
 - LICENSE
 - README
 - Rakefile
+- TODO
 - VERSION.yml
 - examples/Net/Gearman/Client.php
 - examples/Net/Gearman/Connection.php
@@ -40,13 +42,21 @@ files:
 - examples/Net/Gearman/Set.php
 - examples/Net/Gearman/Task.php
 - examples/Net/Gearman/Worker.php
+- examples/calculus_client.rb
+- examples/calculus_worker.rb
 - examples/client.php
 - examples/client.rb
+- examples/client_background.rb
+- examples/client_exception.rb
+- examples/client_prefix.rb
+- examples/gearman_environment.sh
 - examples/scale_image.rb
 - examples/scale_image_worker.rb
 - examples/server.rb
 - examples/worker.php
 - examples/worker.rb
+- examples/worker_exception.rb
+- examples/worker_prefix.rb
 - gearman-ruby.gemspec
 - lib/gearman.rb
 - lib/gearman/client.rb
@@ -59,8 +69,9 @@ files:
 - test/client_test.rb
 - test/mock_client_test.rb
 - test/mock_worker_test.rb
+- test/util_test.rb
 - test/worker_test.rb
-has_rdoc: true
+has_rdoc: false
 homepage: http://github.com/xing/gearman-ruby
 post_install_message: 
 rdoc_options: 
@@ -84,15 +95,23 @@ requirements: []
 rubyforge_project: 
 rubygems_version: 1.2.0
 signing_key: 
-specification_version: 2
+specification_version: 3
 summary: Library for the Gearman distributed job system
 test_files: 
 - test/client_test.rb
 - test/mock_client_test.rb
 - test/mock_worker_test.rb
+- test/util_test.rb
 - test/worker_test.rb
+- examples/calculus_client.rb
+- examples/calculus_worker.rb
 - examples/client.rb
+- examples/client_background.rb
+- examples/client_exception.rb
+- examples/client_prefix.rb
 - examples/scale_image.rb
 - examples/scale_image_worker.rb
 - examples/server.rb
 - examples/worker.rb
+- examples/worker_exception.rb
+- examples/worker_prefix.rb