gearman-ruby 2.0.0

data/.gitignore

@@ -0,0 +1 @@
+pkg/

data/HOWTO

@@ -0,0 +1,146 @@
+= GEARMAN
+
+"Gearman provides a generic application framework to farm out work to other 
+machines or processes that are better suited to do the work. It allows you to 
+do work in parallel, to load balance processing, and to call functions between 
+languages. It can be used in a variety of applications, from high-availability 
+web sites to the transport of database replication events. In other words, it 
+is the nervous system for how distributed processing communicates."
+
+ - http://www.gearman.org/
+
+
+== Setting up a basic environment
+
+A very basic German environment will look like this:
+
+                       ----------  
+                       | Client |  
+                       ----------  
+                           |
+                     --------------     
+                     | Job Server |     
+                     --------------     
+                           |
+    ----------------------------------------------
+    |              |              |              |
+----------     ----------     ----------     ----------
+| Worker |     | Worker |     | Worker |     | Worker |
+----------     ----------     ----------     ----------
+
+And the behavior will be the following:
+
+ * JobServer: Acts as a message passing point.
+ * Client: Sends tasks to the JobServer. Will be connected to only one JobServer
+           in case more than one exits for failover purposes.
+ * Worker: Anounce his 'abilities' to the JobServer and waits for tasks.
+
+For the JobServer we recommend to use the offical Perl version, there's also a
+more performant C implementation of the server with support for persistent
+queues, bells and whistles but is not stable enough for production use at the 
+time of this document was wrote.
+
+The Client and the Worker can be implemented in any language. This way you can
+send tasks from a Ruby client server, to a Perl or C worker in order to get 
+better performance.
+
+== Installing the required software
+
+For the JobServer we recommend to use the offical Perl version, to install it:
+
+ * Mac OS X: sudo port install p5-gearman-server
+ * Debian/Ubuntu: sudo apt-get install gearman-server
+
+To get the Ruby libraries by Xing:
+
+ git clone git://github.com/xing/gearman-ruby.git
+
+== Gearman demo
+
+Now you're ready for you first experience with Gearman. In the cloned repository
+you'll find an 'examples' directory.
+
+Run the 'gearman_environment.sh' to build an environment like the one showed in
+the diagram above.
+
+  * Client: Will ask you for an arithmetic operation, like: 2+3
+            The code of the client is in: 'examples/calculus_client.rb'
+
+  * JobServer: The Perl server.
+
+  * Workers: You'll have 4 worker, one for each of the basic arithmetic 
+             operations.
+             The code of the worker is in: 'examples/calculus_worker.rb'
+
+There are other demos in the examples folder you can give a look at. Each demo usually
+consist in the client and server scripts.
+
+=== Creating clients and tasks
+
+In order to get a job scheduled by a Gearman server using the gearman ruby library, there
+are three main objects you must interact with: Gearman::Client, Gearman::Task and Gearman::TaskSet.
+Let's review all of them briefly:
+
+  - Gearman::Client -> the portion of the library storing the data about the connection to 
+                       the Gearman server.
+  - Gearman::Task -> a job execution request that will be dispatched by the Gearman server to 
+                     worker and whose result data will be returned to the client.
+  - Gearman::TaskSet -> a collection of tasks to be executed. The Taskset object will track the
+                        execution of the tasks with the info returned from the Gearman server
+                        and notify the client with the results or errors when all the tasks
+                        have completed their execution.
+
+To send a new task to the Gearman server, the client must build a new Gearman::Task object, add it to
+a Gearman::TaskSet that must hold a reference to a Gearman::Client and send the wait message to
+the TaskSet.
+The following code taken from examples/client.rb shows the process:
+
+----------------------------------------------------
+servers = ['localhost:4730', 'localhost:4731']
+  
+client = Gearman::Client.new(servers)
+taskset = Gearman::TaskSet.new(client)
+
+task = Gearman::Task.new('sleep', 20)
+task.on_complete {|d| puts d }
+
+taskset.add_task(task)
+taskset.wait(100)
+----------------------------------------------------
+
+The name of the function to be executed is the first parameter to the constructor of the Task object.
+Take into account that the string you pass as a parameter will be used 'as it' by the Gearman server
+to locate a suitable worker for that function.
+The second parameter is the argument to be sent to the worker that will execute, if the arguments for
+the task are complex, a serialization format like YAML or XML must be agreeded with the workers.
+The last and optional parameter is a hash of options. The following options are currently available:
+
+  - :priority -> (:high | :low) the priority of the job, a high priority job is executed before a low on
+  - :background -> (true | false) a background task will return no further information to the client.
+
+The execution of a task in a Gearman remote worker can fail, the worker can throw an exception, etc.
+All these events can be handled by the client of the ruby library registering callback blocks.
+The following events are currently available:
+
+  - on_complete -> the task was executed succesfully
+  - on_fail -> the task fail for some unknown reason
+  - on_retry -> after failure, the task is gonna be retried, the number of retries is passed
+  - on_exception -> the remote worker send an exception notification, the exception text is passed
+  - on_stauts -> a status update is sent by the remote worker
+
+In order to receive exception notifications in the client, this option must be sent in the server, the 
+method option_request can be used this task. The following example, extracted from the examples/client_exception.rb 
+demo script shows the process:
+
+----------------------------------------------------
+client = Gearman::Client.new(servers)
+#try this out
+client.option_request("exceptions")
+----------------------------------------------------
+
+This feature will only works if the server and workers have implemented support for the OPT_REQ and WORK_EXCEPTION
+messages of the Gearman protocol.
+
+
+Enjoy.
+

data/LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2009 XING AG
+ 
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+ 
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+ 
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README

@@ -0,0 +1,9 @@
+gearman-ruby
+===============
+
+Library for the Gearman distributed job system
+
+COPYRIGHT
+=========
+
+Copyright (c) 2009 XING AG. See LICENSE for details.

data/Rakefile

@@ -0,0 +1,41 @@
+require 'rake'
+require 'rake/testtask'
+require 'rake/rdoctask'
+require 'rcov/rcovtask'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |s|
+    s.name = "gearman-ruby"
+    s.summary = "Library for the Gearman distributed job system"
+    s.email = "kim.altintop@gmail.com"
+    s.homepage = "http://github.com/kim/gearman-ruby"
+    s.description = "Library for the Gearman distributed job system"
+    s.authors = ["Kim Altintop"]
+    s.add_dependency 'eventmachine', '>= 0.12.8'
+  end
+rescue LoadError
+  puts "Jeweler not available. Install it with: sudo gem install technicalpickles-jeweler -s http://gems.github.com"
+end
+
+Rake::TestTask.new do |t|
+  t.libs << 'lib'
+  t.pattern = 'test/**/*_test.rb'
+  t.verbose = false
+end
+
+Rake::RDocTask.new do |rdoc|
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title = 'Gearman Ruby'
+  rdoc.options << '--line-numbers' << '--inline-source'
+  rdoc.rdoc_files.include('README*')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+Rcov::RcovTask.new do |t|
+  t.libs << "test"
+  t.test_files = FileList['test/*_test.rb']
+  t.verbose = true
+end
+
+task :default => :test

data/TODO

@@ -0,0 +1,8 @@
+- Failover strategies
+  - Client:
+     * If connected for the first time, try to connect to at least one server from the server array
+     * If already connected to a server, and it goes down, the client should go down as well
+  - Worker:
+     * If connected for the first time, try to connect to as many servers as it can. 
+       Loop trough the bad servers, trying to reconnect to them as well.
+     * If a already connected to a server, and it goes down, wait and try to reconnect again.      

data/VERSION.yml

@@ -0,0 +1,4 @@
+--- 
+:patch: 0
+:major: 2
+:minor: 0

data/examples/calculus_client.rb

@@ -0,0 +1,41 @@
+#!/usr/bin/env ruby
+require 'rubygems'
+require '../lib/gearman'
+#Gearman::Util.debug = true
+
+# Connect to the local server (at the default port 4730)
+client = Gearman::Client.new('localhost')
+taskset = Gearman::Taskset.new
+
+# Get something to echo
+puts '[client] Write a basic arithmetic operation:'
+input = gets
+
+operations = input.chomp.scan(/\d+[\+\-\*\/]\d+/).compact
+puts "[client] The following operations were found: #{operations.inspect}"
+
+# Setup a task for operation
+operations.each do |op|
+  # Determining the operation
+  case op
+    when /\+/
+      type, data = 'addition', op.split('+')
+    when /\-/
+      type, data = 'subtraction', op.split('-')
+    when /\*/
+      type, data = 'multiplication', op.split('*')
+    when /\//
+      type, data = 'division', op.split('/')
+  end
+
+  task = Gearman::Task.new(type, Marshal.dump(data.map {|v| v.to_i}))
+  task.on_complete {|r| puts "[client] #{type} result is: #{r}" }
+  task.on_warning {|w| puts "[client] warn: #{w}" }
+  task.on_fail { puts "[client] calculation failed" } 
+
+  # Sending the task to the server
+  puts "[client] Sending values: #{data.inspect}, to the '#{type}' worker"
+  taskset << task
+end
+
+client.run(taskset)

data/examples/calculus_worker.rb

@@ -0,0 +1,42 @@
+#!/usr/bin/env ruby
+require 'rubygems'
+require '../lib/gearman'
+
+#Gearman::Util.debug = true
+
+worker = Gearman::Worker.new('localhost')
+
+# Additon ability
+worker.add_ability('addition') do |data,job|
+  values = Marshal.load(data)
+  puts "[addition_worker] Calculating #{values.inspect}..."
+  # sleep 5
+  values.first + values.last
+end
+
+# Subtraction ability
+worker.add_ability('subtraction') do |data,job|
+  values = Marshal.load(data)
+  puts "[subtraction_worker] Calculating #{values.inspect}..."
+  # sleep 5
+  values.first - values.last
+end
+
+# Multiplication worker
+worker.add_ability('multiplication') do |data,job|
+  values = Marshal.load(data)
+  puts "[multiplication_worker] Calculating #{values.inspect}..."
+  # sleep 5
+  values.first * values.last
+end
+
+# Division worker
+worker.add_ability('division') do |data,job|
+  values = Marshal.load(data)
+  puts "[division_worker] Calculating #{data.inspect}..."
+  # sleep 5
+  values.first / values.last
+end
+
+# Running the workers
+worker.work

data/examples/client.rb

@@ -0,0 +1,19 @@
+require 'rubygems'
+#require 'gearman'
+require '../lib/gearman'
+Gearman::Util.debug = true
+
+servers = ['localhost:4730', 'localhost:4731']
+
+client = Gearman::Client.new(servers)
+taskset = Gearman::Taskset.new
+
+task = Gearman::Task.new('sleep', 2)
+task.on_complete {|d| puts "TASK 1: #{d}" }
+taskset << task
+
+task = Gearman::Task.new('sleep', 2)
+task.on_complete {|d| puts "TASK 2: #{d}" }
+taskset << task
+
+client.run(taskset)

data/examples/client_background.rb

@@ -0,0 +1,14 @@
+require 'rubygems'
+#require 'gearman'
+require '../lib/gearman'
+Gearman::Util.debug = true
+
+servers = ['localhost:4730', 'localhost:4731']
+
+client = Gearman::Client.new(servers)
+
+task = Gearman::Task.new('sleep', 20, :background => true, :poll_status_interval => 1)
+task.on_complete {|d| puts d } #never called
+task.on_status {|d| puts "Status: #{d}"}
+
+client.run(task)

data/examples/client_data.rb

@@ -0,0 +1,16 @@
+require 'rubygems'
+#require 'gearman'
+require '../lib/gearman'
+Gearman::Util.debug = true
+
+servers = ['localhost:4730']
+
+client = Gearman::Client.new(servers)
+taskset = Gearman::Taskset.new
+
+task = Gearman::Task.new('chunked_transfer')
+task.on_data {|d| puts d }
+task.on_complete {|d| puts d }
+
+taskset << task
+client.run taskset

data/examples/client_echo.rb

@@ -0,0 +1,16 @@
+require 'rubygems'
+require '../lib/gearman'
+
+servers = ['localhost:4730']
+client = Gearman::Client.new(servers)
+taskset = Gearman::TaskSet.new(client)
+
+100000.times do |i|
+  task = Gearman::Task.new('echo', "hello #{i}")
+  task.on_complete {|d| puts d }
+  taskset.add_task(task)
+end
+taskset.wait(100)
+
+taskset.add_task(Gearman::Task.new('quit'))
+taskset.wait(100)

data/examples/client_exception.rb

@@ -0,0 +1,17 @@
+require 'rubygems'
+require '../lib/gearman'
+Gearman::Util.debug = true
+
+servers = ['localhost:4730']
+
+client = Gearman::Client.new(servers)
+
+task = Gearman::Task.new('fail_with_exception', "void")
+task.retries = 2
+task.on_complete {|d| puts d }
+task.on_exception {|ex| puts "This should never be called" }
+task.on_warning {|warning| puts "WARNING: #{warning}" }
+task.on_retry { puts "PRE-RETRY HOOK: retry no. #{task.retries_done}" }
+task.on_fail { puts "TASK FAILED, GIVING UP" }
+
+client.run task

data/examples/client_prefix.rb

@@ -0,0 +1,15 @@
+require 'rubygems'
+#require 'gearman'
+require '../lib/gearman'
+Gearman::Util.debug = true
+
+servers = ['localhost:4730', 'localhost:4731']
+
+ability_name_with_prefix = Gearman::Util.ability_name_with_prefix("test","sleep")
+
+client = Gearman::Client.new(servers)
+
+task = Gearman::Task.new(ability_name_with_prefix, 20)
+task.on_complete {|d| puts d }
+
+client.run task

data/examples/evented_client.rb

@@ -0,0 +1,23 @@
+require 'rubygems'
+require "ruby-debug"
+require '../lib/gearman'
+
+Gearman::Util.debug = true
+
+servers = ['localhost:4730', 'localhost:4731']
+client = Gearman::Client.new(servers)
+
+taskset = Gearman::Taskset.new
+
+task = Gearman::Task.new('sleep', 2)
+task.on_status {|numerator, denominator| puts "TASK 1: Completed #{numerator} of #{denominator}"}
+task.on_complete {|d| puts "TASK 1: #{d}" }
+taskset << task
+
+task = Gearman::Task.new('sleep', 15, :poll_status_interval => 2, :uuid => nil)
+task.on_status {|numerator, denominator| puts "TASK 2: Completed #{numerator} of #{denominator}"}
+task.on_data {|data| puts "TASK 2 DATA: #{data}" }
+task.on_complete {|d| puts "TASK 2: #{d}" }
+taskset << task
+
+client.run(taskset)