job_reactor 0.5.1 → 0.5.2

data/README.markdown

@@ -5,8 +5,10 @@ JobReactor is a library for creating, scheduling and processing background jobs.
 It is asynchronous client-server distributed system based on [EventMachine][0].
 Inspired by [Resque][1], [Beanstalkd][2] ([Stalker][3]), [DelayedJob][4], and etc.
 
-JobReactor has not 'rails' integration for the time being.
-But it is very close. We need to test the system with different servers (clusters) and automate the initialization and re-start processes.
+To use JobReactor with [Ruby on Rails][9] you should start distributor in initializer using `JR.run` method (it launches EventMachine in separate thread).
+Then add rake task(s) which will run the node(s). If you use [Thin][10] server the solution is more complicated because 'Thin' use EventMachine too.
+So, 'rails' integration is not complete for the time being.
+We need to test the system with different servers (clusters) and automate the initialization and re-start processes.
 Collaborators, you are welcome!
 
 So, read the 'features' section and try JobReactor. You can do a lot with it.
@@ -34,9 +36,11 @@ In your main application:
 `application.rb`
 ``` ruby
 require 'job_reactor'
+
 JR.run do
-  JR.start_distributor('localhost', 5000)  #see documentation
+  JR.start_distributor('localhost', 5000)  #see lib/job_reactor/job_reactor.rb
 end
+
 sleep(1) until(JR.ready?)
 
 # The application
@@ -49,6 +53,7 @@ Define the 'my_job' in separate directory (files with job's definitions **must**
 `reactor_jobs/my_jobs.rb`
 ``` ruby
 include JobReactor
+
 job 'my_job' do |args|
   puts args[:arg1]
 end
@@ -57,14 +62,16 @@ And the last file - 'the worker code':
 `worker.rb`
 ``` ruby
 require 'job_reactor'
+
 JR.config[:job_directory] = 'reactor_jobs' #this default config, so you can omit this line
+
 JR.run! do
   JR.start_node({
   :storage => 'memory_storage',
   :name => 'worker_1',
   :server => ['localhost', 5001],
   :distributors => [['localhost', 5000]]
-  })                                         #see documentation
+  })                                         #see lib/job_reactor/job_reactor.rb
 end
 ```
 Run 'application.rb' in one terminal window and 'worker.rb' in another.
@@ -183,7 +190,9 @@ JR.enqueue('my_job', {arg1: 1}, {period: 100, node: 'my_favourite_node', not_nod
 
 The rule to use specified node is not strict if `JR.config[:always_use_specified_node]` is false (default).
 This means that distributor will try to send the job to the given node at first. But if the node is `locked` (maybe you have just sent another job to it and it is very busy) distributor will look for other node.
-The last two arguments are optional too. The first is 'success feedback' and the last is 'error feedback'. We use term 'feedback' to distinguish from 'callbacks' and 'errbacks'. 'feedback' is executed on the main application side while 'callbacks' on the node side. 'feedbacks' are the procs which will be called when node sent message that job is completed (successfully or not). The argunments for the 'feedback' are the arguments of the initial job plus all added on the node side.
+
+The last two arguments are optional. The first is 'success feedback' and the last is 'error feedback'. We use term 'feedback' to distinguish from 'callbacks' and 'errbacks'. 'feedback' is executed on the main application side while 'callbacks' on the node side. 'feedbacks' are the procs which will be called when node sent message that job is completed (successfully or not). The argunments for the 'feedback' are the arguments of the initial job plus all added on the node side.
+
 Example:
 
 ```ruby
@@ -199,7 +208,7 @@ JR.enqueue('my_job', {arg1: 1}, {}, success)
 ```
 
 The 'success' proc args will be {arg1: 1, result: 'Yay!'}.
-The same story is with 'error feedback'. Note that error feedback will be launched after all attempts failed on the node side.
+The same story is with 'error feedback'. __Note__, that error feedback will be launched after all attempts failed on the node side.
 See config: `JR.config[:max_attempt] = 10` and `JR.config[:retry_multiplier]`
 
 4. You disconnect node (stop it manually or node fails itself)
@@ -212,6 +221,76 @@ See config: `JR.config[:max_attempt] = 10` and `JR.config[:retry_multiplier]`
 ---------------------------------
 * Nodes will continue to work, but you won't be able to receive the results from node when you start the application again because all feedbacks are stored in memory.
 
+Callbacks and feedbacks
+============================
+'callbacks', 'errbacks', 'success feedback', and 'error feedback' helps you divide the __job__ into small relatively independent parts.
+
+To define `'job'` you use `JobReactor.job` method (see 'Quick start' section). The only arguments are 'job_name' and the block which is the job itself.
+
+You can define any number of callbacks and errbacks for the given job. Just use `JobReactor.job_callback` and `JobRector.job_errback` methods. The are three arguments for calbacks and errbacks. The name of the job, the name of callback/errback (optional) and the block.
+
+```ruby
+include JobReactor
+
+job 'test_job' do |args|
+  puts "job with args #{args}" 
+end
+
+job_callback 'test_job', 'first_callback' do |args|
+  puts "first callback with args #{args}"
+end
+
+job_callback 'test_job', 'second_callback' do |args|
+  puts "second callback with args #{args}"
+end
+
+job_errback 'test_job', 'first_errback' do |args|
+  puts "first errback with error #{args[:error]}"
+end
+
+job_errback 'test_job', 'second_errback' do |args|
+  puts 'another errback'
+end
+```
+
+Callbacks and errbacks acts as ordinary EventMachine::Deferrable callbacks and errbacks. The `'job'` is the first callack, first `'job_callback'` becomes second callback and so on. See `lib/job_reactor/job_reactor/job_parser.rb` for more information. When Node start job it calls `succeed` method on the 'job object' with given argument (args). This runs all callbacks sequentially. If error occurs in any callback Node calls `fail` method on the 'deferrable' object with the same args (plus merged `:error => 'Error message`).
+
+__Note__, you define jobs, callbacks and errbacks in top-level scope, so the `self` is `main` object.
+
+You can `merge!` additional key-value pairs to 'args' in the job to exchange information between job and it's callbacks.
+
+```ruby
+include JobReactor
+
+job 'test_job' do |args|
+  args.merge!(result: 'Hello')
+end
+
+job_callback 'test_job', 'first_callback' do |args|
+  puts args[:result]
+  args.merge!(another_result: 'world')
+end
+
+job_callback 'test_job', 'second_callback' do |args|
+  puts "#{args[:result]} #{args[:another_result]}"
+end
+```
+__Note__, if error occurs you can't see additional arguments in job errbacks.
+
+Another trick is `JR.config[:merge_job_itself_to_args]` option which is `false` by default. If you set this option to `true` you can see `:job_itself` key in `args`. The value contains many usefull information about job ('name', 'attempt', 'status', 'make_after', 'node', etc).
+
+Feedbacks are defined as a Proc object and attached to the 'job' when it is enqueued on the application side.
+
+```ruby
+success = Proc.new { |args| puts 'Success' }
+error = Proc.new { |args| puts 'Error' }
+JR.enqueue('my_job', {arg1: 1, arg2: 2}, {after: 100}, success, error)
+```
+
+This procs will be called when Node informs about success or error. The 'args' for the corresponding proc will be the same 'args' which is in the job (and it's callbacks) on the node side. So you can, for example, return any result by merging it to 'args' in the job (or it's callbacks).
+
+__Note__, feedbacks are kept in memory in your application, so they disappear when you restart the application.
+
 Job Storage
 ==========
 Now you can store your job in [Redis][5] storage (`'redis_storage`') or in memory (`'memory_storage'`).
@@ -242,7 +321,7 @@ The informaion about jobs is saved several times during processing. This informa
 * on_success - the unique id of success feedback on the distributor side;
 * on_error - the unique id of error feedback on the distributor side;
 
-By default JobReactor delete all completed and cancelled jobs, but you can configure it:
+By default JobReactor deletes all completed and cancelled jobs, but you can configure it:
 The default options are:
 
 ```ruby
@@ -276,3 +355,5 @@ The MIT License - Copyright (c) 2012 Anton Mishchuk
 [6]: https://github.com/igrigorik/em-http-request
 [7]: https://github.com/igrigorik/em-websocket
 [8]: https://github.com/madsimian/em-redis
+[9]: http://rubyonrails.org/
+[10]: http://code.macournoyer.com/thin/

data/lib/job_reactor/job_reactor/storages.rb

@@ -9,7 +9,7 @@
 
 # Defines storages for lazy loading
 
-# TODO 'NEXT RELEASE'
+# TODO
 # require 'active_record'
 # class JobReactor::ActiveRecordStorage < ::ActiveRecord::Base; end
 

data/lib/job_reactor/job_reactor.rb

@@ -110,7 +110,7 @@ module JobReactor
     # JR.enqueue 'job', { arg1: 'arg1'}, {}, success, error
     #
     def enqueue(name, args = { }, opts = { }, success_proc = nil, error_proc = nil)
-      hash = { 'name' => name, 'args' => args, 'attempt' => 0, 'status' => 'new' }
+      hash = { 'name' => name, 'args' => args, 'attempt' => 0, 'status' => 'new', 'defer' => 'false' }
 
       hash.merge!('period' => opts[:period]) if opts[:period]
       opts[:after] = (opts[:run_at] - Time.now) if opts[:run_at]
@@ -121,6 +121,8 @@ module JobReactor
 
       hash.merge!('distributor' => JR::Distributor.server)
 
+      hash.merge!('defer' => 'true') if opts[:defer]
+
       add_succ_feedbacks!(hash, success_proc) if success_proc.is_a? Proc
       add_err_feedbacks!(hash, error_proc) if error_proc.is_a? Proc
 

data/lib/job_reactor/node.rb

@@ -56,14 +56,22 @@ module JobReactor
     # It makes a job and run do_job.
     #
     def schedule(hash)
-      if hash['make_after'].to_i > 0
-        EM::Timer.new(hash['make_after']) do
-          self.storage.load(hash) { |hash| do_job(JR.make(hash)) }
+      run_job = Proc.new do
+        if hash['make_after'].to_i > 0
+          EM::Timer.new(hash['make_after']) do
+            self.storage.load(hash) { |hash| do_job(JR.make(hash)) }
+          end
+        else
+          EM.next_tick do
+            self.storage.load(hash) { |hash| do_job(JR.make(hash)) }
+          end
         end
+      end
+
+      if hash['defer'] == 'true'
+        EM.defer { run_job.call }
       else
-        EM.next_tick do
-          self.storage.load(hash) { |hash| do_job(JR.make(hash)) }
-        end
+        run_job.call
       end
     end
 

data/lib/job_reactor/storages/memory_storage.rb

@@ -10,7 +10,7 @@ module JobReactor
       end
 
       def load(hash, &block)
-        hash      = storage[hash['id']]
+        hash = storage[hash['id']]
         if hash
           hash_copy = { }
           hash.each { |k, v| hash_copy.merge!(k => v) }

data/lib/job_reactor/storages/redis_storage.rb

@@ -4,7 +4,7 @@ require 'em-redis'
 module JobReactor
   module RedisStorage
     @@storage = EM::Protocols::Redis.connect(host: JobReactor.config[:redis_host], port: JobReactor.config[:redis_port])
-    ATTRS = %w(id name args last_error run_at failed_at attempt period make_after status distributor on_success on_error)
+    ATTRS = %w(id name args last_error run_at failed_at attempt period make_after status distributor on_success on_error defer)
 
     class << self
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: job_reactor
 version: !ruby/object:Gem::Version
-  version: 0.5.1
+  version: 0.5.2
   prerelease: 
 platform: ruby
 authors:
@@ -10,7 +10,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-06-18 00:00:00.000000000 Z
+date: 2012-09-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: eventmachine
@@ -68,22 +68,22 @@ executables: []
 extensions: []
 extra_rdoc_files: []
 files:
+- lib/job_reactor.rb
 - lib/job_reactor/node/server.rb
 - lib/job_reactor/node/client.rb
+- lib/job_reactor/node.rb
+- lib/job_reactor/job_reactor.rb
 - lib/job_reactor/distributor/server.rb
 - lib/job_reactor/distributor/client.rb
-- lib/job_reactor/distributor.rb
-- lib/job_reactor/logger.rb
-- lib/job_reactor/storages/memory_storage.rb
 - lib/job_reactor/storages/redis_monitor.rb
+- lib/job_reactor/storages/memory_storage.rb
 - lib/job_reactor/storages/redis_storage.rb
+- lib/job_reactor/distributor.rb
+- lib/job_reactor/logger.rb
 - lib/job_reactor/job_reactor/storages.rb
-- lib/job_reactor/job_reactor/config.rb
-- lib/job_reactor/job_reactor/job_parser.rb
 - lib/job_reactor/job_reactor/exceptions.rb
-- lib/job_reactor/job_reactor.rb
-- lib/job_reactor/node.rb
-- lib/job_reactor.rb
+- lib/job_reactor/job_reactor/job_parser.rb
+- lib/job_reactor/job_reactor/config.rb
 - README.markdown
 homepage: http://github.com/antonmi/job_reactor
 licenses: []