job_reactor 0.5.0.beta3 → 0.5.0.beta4

data/README.markdown

@@ -5,9 +5,9 @@ Now we are in beta (need to complete documentation and fix some bugs)
 
 JobReactor is a library for creating, scheduling and processing background jobs.
 It is asynchronous client-server distributed system based on [EventMachine][0].
-Inspired by Resque, Stalker, DelayedJob, and etc.
+Inspired by [Resque][1], [Stalker][2], [DelayedJob][3], and etc.
 
-JobReactor hasn't 'rails' integration for the time being.
+JobReactor has not 'rails' integration for the time being.
 But it is very close. We need test the system with different servers (clusters) and automatize initialization and restart processes.
 Collaborators, you are welcome!
 
@@ -17,6 +17,9 @@ Quick start
 ===========
 Use `gem install job_reactor --pre` to try it.
 
+You need to install [Redis][6] if you want to persist your jobs.
+``$ sudo apt-get install redis-server ``
+
 In you main application:
 `application.rb`
 ``` ruby
@@ -51,7 +54,7 @@ end
 ```
 Run 'application.rb' in one terminal window and 'worker.rb' in another.
 Node connects to distributor, receives the job and works.
-Cool! But it was the simplest example. See 'examples' directory and read the wiki (coming soon).
+Cool! But it was the simplest example. See 'examples' directory and read 'advanced usage'(coming soon).
 
 Features
 =============
@@ -63,9 +66,9 @@ If you don't have many jobs you can leave only one node which will be connected
 2. High scalability
 -------------------
 Nodes and distributors are connected via TCP. So, you can run them on any machine you can connect to.
-Nodes may use different storage or the same one. So, you can store vitally important jobs in relational database and
+Nodes may use different storage or the same one. You can store vitally important jobs in database and
 simple insignificant jobs in memory.
-And more: your nodes may create jobs for others nodes and communicate with each other. See page [advance usage].
+And more: your nodes may create jobs for others nodes and communicate with each other. See page [advanced usage].
 3. Full job control
 -------------------
 You can add callback and errbacks to the job which will be called on the node.
@@ -83,7 +86,7 @@ If node is stopped or crashed it will retry stored jobs after start.
 5. EventMachine available
 -------------------------
 Remember, your jobs will be run inside EventMachine reactor! You can easily use the power of async nature of EventMachine.
-Use asynchronous [http requests], [websockets], [etc.], [etc.], and [etc]. See page [advance usage].
+Use asynchronous [em-http-request][4], [em-websocket][5], [etc.], [etc.], and [etc]. See page [advance usage].
 6. Deferred and periodic jobs
 -----------------------------
 You can use deferred jobs which will run 'after' some time or 'run_at' given time.
@@ -102,7 +105,7 @@ If no nodes are specified distributor will try to send the job to the first free
 10. Node based priorities
 -----------------------
 There are no priorities like in Delayed::Job or Stalker. Bud there are flexible node-based priorities.
-You can specify the node which should execute the job. You can reserve several nodes for high priority jobs.
+You can specify the node which should execute the job and the node is forbidden for given job. You can reserve several nodes for high priority jobs.
 
 
 
@@ -125,9 +128,17 @@ How it works
 #TODO
 
 
+License
+---------
+The MIT License - Copyright (c) 2012 Anton Mishchuk
+
 
 
 
-Links:
-------
-[0]: http://rubyeventmachine.com/
+[0]: http://rubyeventmachine.com
+[1]: https://github.com/defunkt/resque
+[2]: https://github.com/han/stalker
+[3]: https://github.com/tobi/delayed_job
+[4]: https://github.com/igrigorik/em-http-request
+[5]: https://github.com/igrigorik/em-websocket
+[6]: http://redis.io

data/lib/job_reactor/job_reactor/config.rb

@@ -17,6 +17,7 @@ JR.config[:log_job_processing] = true
 JR.config[:always_use_specified_node] = false #will send job to another node if specified node is not available
 JR.config[:remove_done_jobs] = true
 JR.config[:remove_cancelled_jobs] = true
+JR.config[:remove_failed_jobs] = false
 
 JR.config[:redis_host] = 'localhost'
 JR.config[:redis_port] = 6379

data/lib/job_reactor/job_reactor.rb

@@ -5,6 +5,7 @@ require 'job_reactor/job_reactor/config'
 require 'job_reactor/job_reactor/job_parser'
 require 'job_reactor/job_reactor/exceptions'
 require 'job_reactor/job_reactor/storages'
+require 'job_reactor/storages/redis_monitor'
 
 module JobReactor
 

data/lib/job_reactor/node.rb

@@ -54,11 +54,7 @@ module JobReactor
     def schedule(hash)
       EM::Timer.new(hash['make_after']) do  #Of course, we can start job immediately (unless it is 'after' job), but we let EM take care about it. Maybe there is another job is ready to start
         self.storage.load(hash) do |hash|
-          if job = JR.make(hash)  #If we decide fail silently. See JR.make
-            do_job(job)
-          else
-            #TODO Do nothing or raise exception ????
-          end
+          do_job(JR.make(hash))
         end
       end
     end
@@ -75,6 +71,7 @@ module JobReactor
     def do_job(job)
       job['run_at'] = Time.now
       job['status'] = 'in progress'
+      job['attempt'] += 1
       storage.save(job) do |job|
         begin
           args = job['args'].merge(JR.config[:merge_job_itself_to_args] ? {:job_itself => job.dup} : {})
@@ -136,10 +133,16 @@ module JobReactor
     #Tryes again or report error
     #
     def complete_rescue(job)
-      if job['attempt'].to_i < JobReactor.config[:max_attempt] - 1
+      if job['attempt'].to_i < JobReactor.config[:max_attempt]
         try_again(job)
       else
+        job['status'] = 'failed'
         report_error(job) if job['on_error']
+        if JR.config[:remove_failed_jobs]
+          storage.destroy(job)
+        else
+          storage.save(job)
+        end
       end
     end
 
@@ -159,7 +162,6 @@ module JobReactor
     # They will be rescheduled after period time.
     #
     def try_again(job)
-      job['attempt'] += 1
       if job['period'] && job['period'] > 0
         job['make_after'] = job['period']
       else

data/lib/job_reactor/storages/redis_monitor.rb

@@ -0,0 +1,58 @@
+require 'redis'
+module JobReactor
+  module RedisMonitor
+
+    ATTRS = %w(id name args last_error run_at failed_at attempt period make_after status distributor on_success on_error)
+
+    extend self
+
+    def storage
+      @@storage ||= Redis.new(host: JR.config[:redis_host], port: JR.config[:redis_port])
+    end
+
+    # Returns all job for given node.
+    #
+    def jobs_for(name, to_be_retried = false)
+      pattern = "*#{name}_*"
+      keys = storage.keys(pattern)
+      result = {}
+      keys.each do |key|
+        hash = self.load(key)
+        if to_be_retried
+          result.merge!(key => hash)  if hash['status'] != 'complete' && hash['status'] != 'cancelled' && hash['status'] != 'failed'
+        else
+          result.merge!(key => hash)
+        end
+      end
+      result
+    end
+
+    # Load job from storage.
+    #
+    def load(key)
+      hash = {}
+      record = storage.hmget(key, *ATTRS)
+      ATTRS.each_with_index do |attr, i|
+        hash[attr] = record[i]
+      end
+      ['attempt', 'period', 'make_after'].each do |attr|
+        hash[attr] = hash[attr].to_i
+      end
+      hash['args'] = Marshal.load(hash['args'])
+
+      hash
+    end
+
+    def destroy(key)
+      storage.del(key)
+    end
+
+    # Destroys all job for given node.
+    #
+    def destroy_all_jobs_for(name)
+      pattern = "*#{name}_*"
+      storage.del(*storage.keys(pattern))
+    end
+
+  end
+end

data/lib/job_reactor/storages/redis_storage.rb

@@ -60,7 +60,8 @@ module JobReactor
               hash['id'] = id
               hash['node'] = name
               self.load(hash) do |hash|
-                if hash['status'] != 'complete' && hash['status'] != 'cancelled' && hash['attempt'].to_i < JobReactor.config[:max_attempt]
+                if hash['status'] != 'complete' && hash['status'] != 'cancelled' && hash['status'] != 'failed'
+                else
                   block.call(hash)
                 end
               end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: job_reactor
 version: !ruby/object:Gem::Version
-  version: 0.5.0.beta3
+  version: 0.5.0.beta4
   prerelease: 6
 platform: ruby
 authors:
@@ -10,7 +10,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-06-02 00:00:00.000000000 Z
+date: 2012-06-05 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: eventmachine
@@ -28,6 +28,22 @@ dependencies:
     - - ! '>='
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: redis
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: em-redis
   requirement: !ruby/object:Gem::Requirement
@@ -58,6 +74,7 @@ files:
 - lib/job_reactor/distributor.rb
 - lib/job_reactor/logger.rb
 - lib/job_reactor/storages/memory_storage.rb
+- lib/job_reactor/storages/redis_monitor.rb
 - lib/job_reactor/storages/redis_storage.rb
 - lib/job_reactor/job_reactor/storages.rb
 - lib/job_reactor/job_reactor/config.rb