jobba 1.2.0 → 1.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: dcfba40866333af0570df573302aaac92ef2c985
-  data.tar.gz: 031ee23b4e6ef49a80c46f539e68b7772a621edc
+  metadata.gz: 28b15024f2860e2b7cc60d8a70d0f39f1bd9a112
+  data.tar.gz: e2498a809fd6583a2fa56077c78f2ed04b094a12
 SHA512:
-  metadata.gz: ddd79547cd5913d6b7c10962150ebec677055ab979ad0890685d2362fd0123cd312595efeb27ad8a39618ef4cc256cc99498bccfa889f95a6f8452db42a62fc0
-  data.tar.gz: e9442849d2a665b3b0253d5f1ada4ac74f61441e7976b3a40d058e758f72bef9c80c6fc467caea97c3fa590f1f3cf6d7f05f9c6f3188418bb8410dae2f0cbce5
+  metadata.gz: 2f49f2e17aa3b5cc43422aabe408a908afcd17d80973d9859ad6144e3949465fbb5a4ef2afca680ae2f0b4f8eb57478aeec296278810e8da5575764dfddf1ef8
+  data.tar.gz: 04c416209f2864a203e5517da500c8fa1258535757a56d1a17bac38debc4504e54f2888a36c06778741134e10d54d9eba0780ff15018e975235deb290d726888

data/README.md

@@ -123,6 +123,14 @@ There is also a special timestamp for when a kill is requested, `kill_requested_
 
 The order of states is not enforced, and you do not have to use all states.  However, note that you'll only be able to query for states you use (Jobba doesn't automatically travel through states you skip) and if you're using an unusual order your time-based queries will have to reflect that order.
 
+### Restarts
+
+Generally-speaking, you should only enter any state once.  Jobba only records the timestamp the first time you enter a state.
+
+The expection to this rule is that if call `started!` a second time, Jobba will note this as a restart.  The current values in the status will be archived and your status will look like a `started` status, with the exception that the `attempt` count will be incremented.  A restarted status can then enter `succeeded`, `failed`, or `killed` states and those timestamps will be stored.  `job_name` and `job_args` survive the restart.
+
+The `attempt` field is zero-indexed, so the first attempt is attempt `0`.
+
 ## Mark Progress
 
 If you want to have a way to track the progress of a job, you can call:
@@ -137,7 +145,18 @@ This is useful if you need to show a progress bar on your client, for example.
 
 ## Recording Job Errors
 
-...
+The status can keep track of a list of errors.  Errors can be anything, as long as they are JSON-friendly.
+
+```ruby
+my_status.add_error("oh nooo!!")
+my_status.add_error(msg: "oh nooo!!", data: 42)
+```
+
+Errors are available from an `errors` attribute
+
+```ruby
+my_status.errors # => ["oh nooo!!", {"msg" => "oh nooo!!", "data" => 42}]
+```
 
 ## Saving Job-specific Data
 
@@ -148,7 +167,9 @@ my_status.save({a: 'blah', b: [1,2,3]})
 my_status.save("some string")
 ```
 
-Note that if you `save` a hash with symbol keys it will be converted to string keys.  In fact, any argument passed in to `save` will be converted to JSON and parsed back so that the `data` attribute returns the same thing regardless of if `data` is retrieved immediately after being set or after being loaded from Redis.
+## Normalization of Saved Data and Errors
+
+Note that if you `save` or `add_error` contains a hash with symbol keys, those keys will be converted to strings.  In fact, any argument passed in to these methods will be converted to JSON and parsed back again so that the `data` and `errors` attributes returns the same thing regardless of if they are retrieved immediately after being set or after being loaded from Redis.
 
 ## Setting Job Name and Arguments
 
@@ -158,16 +179,24 @@ If you want to be able to query for all statuses for a certain kind of job, you
 my_status.set_job_name("MySpecialJob")
 ```
 
-If you want to be able to query for all statuses that take a certain argument as input, you can add job arguments to a status:
+If you want to be able to query for all statuses that take a certain argument as input, you can set job arguments on a status:
 
 ```ruby
-my_status.add_job_arg(arg_name, arg)
+my_status.set_job_args(arg_1_name: arg_2, arg_2_name: arg_2)
 ```
 
-where `arg_name` is what the argument is called in your job (e.g. `"input_1"`) and `arg` is a way to identify the argument (e.g. `"gid://app/Person/72"`).
+where the keys are what the argument is called in your job (e.g. `"input_1"`) and the values are a way to identify the argument (e.g. `"gid://app/Person/72"`).  The values must currently be strings.
 
 You probably will only want to track complex arguments, e.g. models in your application.  E.g. you could have a `Book` model and a `PublishBook` background job and you may want to see all of the `PublishBook` jobs that have status for the `Book` with ID `53`.
 
+Note that you can set job args with names that are either symbols or strings, but you can only read the args back by the string form of their name, e.g.
+
+```ruby
+my_status.set_job_args(foo: "bar")
+my_status.job_args['foo']          # => "bar"
+my_status.job_args[:foo]           # => nil
+```
+
 ## Killing Jobs
 
 While Jobba can't really kill jobs (it doesn't control your job-running library), it has a facility for marking that you'd like a job to be killed.
@@ -201,7 +230,7 @@ When you get hold of a `Status`, via `create!`, `find`, `find!`, or as the resul
 | `id` | A Jobba-created UUID |
 | `state` | one of the states above |
 | `progress` | a float between 0.0 and 1.0 |
-| `errors` | TBD |
+| `errors` | an array of errors |
 | `data` | job-specific data |
 | `job_name` | The name of the job |
 | `job_args` | An hash of job arguments, {arg_name: arg, ...} |
@@ -414,11 +443,25 @@ Note that, in operations having to do with time, this gem ignores anything beyon
 
 Jobba strives to do all of its operations as efficiently as possible using built-in Redis operations.  If you find a place where the efficiency can be improved, please submit an issue or a pull request.
 
+### Write from one; Read from many
+
+Jobba assumes that any job is being run at one time by only one worker.  Jobba makes no accomodations for multiple processes updating a Status at the same time; multiple processes reading of a Status are fine of course.
+
+## Development
+
+By default, this gem uses `fakeredis` instead of real Redis.  This is great most of the time, but occassionally `fakeredis` doesn't work exactly like real Redis.  If you want to use real Redis, just set the `USE_REAL_REDIS` environment variable to `true`, e.g.
+
+```
+$> USE_REAL_REDIS=true rspec
+```
+
+Travis runs the specs with both `fakeredis` and real Redis.
+
 ## TODO
 
 1. Provide job min, max, and average durations.
-2. Implement `add_error`.
 8. Specs that test scale.
+9. Move redis code in `set_job_args`, `set_job_name`, and `save` into `set` to match rest of code.
 
 
 

data/lib/jobba/clause_factory.rb

@@ -24,8 +24,6 @@ class Jobba::ClauseFactory
     end
   end
 
-  protected
-
   def self.timestamp_clause(timestamp_name, options)
     validate_timestamp_name!(timestamp_name)
 

data/lib/jobba/query.rb

@@ -95,7 +95,7 @@ class Jobba::Query
       load_default_clause if clauses.empty?
       working_set = nil
 
-      clauses.each_with_index do |clause, ii|
+      clauses.each do |clause|
         clause_set = clause.to_new_set
 
         if working_set.nil?

data/lib/jobba/state.rb

@@ -5,6 +5,11 @@ class Jobba::State
   def initialize(name, timestamp_name)
     @name = name
     @timestamp_name = timestamp_name
+    @timestamp_name_key = timestamp_name.to_sym
+  end
+
+  def timestamp_name_key
+    @timestamp_name_key
   end
 
   def self.from_name(state_name)
@@ -51,6 +56,11 @@ class Jobba::State
     UNKNOWN
   ].freeze
 
+  ALL_TIMESTAMP_SYMBOLS = ALL.collect{|state| state.timestamp_name_key}.freeze
+
+  def to_json
+    name.to_json
+  end
 end
 
 

data/lib/jobba/status.rb

@@ -20,21 +20,21 @@ module Jobba
     # Finds the job with the specified ID and returns it.  If no such ID
     # exists in the store, returns nil.
     def self.find(id)
-      if (hash = raw_redis_hash(id))
-        new(raw: hash)
-      else
+      if (hash = raw_redis_hash(id)).empty?
         nil
+      else
+        new(raw: hash)
       end
     end
 
     def self.local_attrs
-      %w(id state progress errors data kill_requested_at job_name job_args) +
+      %w(id state progress errors data kill_requested_at job_name job_args attempt) +
       State::ALL.collect(&:timestamp_name)
     end
 
     def reload!
       @json_encoded_attrs = self.class.raw_redis_hash(id)
-      self.class.local_attrs.each{|aa| send("#{aa}=",nil)}
+      clear_attrs
       self
     end
 
@@ -56,11 +56,12 @@ module Jobba
 
     State::ENTERABLE.each do |state|
       define_method("#{state.name}!") do
-        return self if state == self.state
-
         redis.multi do
-          leave_current_state!
-          enter_state!(state)
+          if state == State::STARTED && did_start?
+            restart!
+          elsif state != self.state
+            move_to_state!(state)
+          end
         end
 
         self
@@ -81,6 +82,10 @@ module Jobba
       !completed?
     end
 
+    def did_start?
+      !self.started_at.nil?
+    end
+
     def request_kill!
       time, usec_int = now
       if redis.hsetnx(job_key, :kill_requested_at, usec_int)
@@ -99,37 +104,40 @@ module Jobba
 
     def set_job_name(job_name)
       raise ArgumentError, "`job_name` must not be blank" if job_name.nil? || job_name.empty?
-      raise StandardError, "`job_name` can only be set once" if !self.job_name.nil?
 
       redis.multi do
+        redis.srem(job_name_key, id)
         set(job_name: job_name)
         redis.sadd(job_name_key, id)
       end
     end
 
-    def add_job_arg(arg_name, arg)
-      raise ArgumentError, "`arg_name` must not be blank" if arg_name.nil? || arg_name.empty?
-      raise ArgumentError, "`arg` must not be blank" if arg.nil? || arg.empty?
+    def set_job_args(args_hash={})
+      raise ArgumentError, "All values in the hash passed to `set_job_args` must be strings" \
+        if args_hash.values.any?{|val| !val.is_a?(String)}
+
+      args_hash = normalize_for_json(args_hash)
 
       redis.multi do
-        self.job_args[arg_name.to_sym] = arg
-        redis.hset(job_args_key, arg_name, arg)
-        redis.sadd(job_arg_key(arg), id)
+        delete_self_from_job_args_set!
+        set(job_args: args_hash)
+        add_self_to_job_args_set!
       end
     end
 
-    # def add_error(error, options = { })
-    #   options = { is_fatal: false }.merge(options)
-    #   @errors << { is_fatal: options[:is_fatal],
-    #                code: error.code,
-    #                message: error.message,
-    #                data: error.data }
-    #   set(errors: @errors)
-    # end
+    def add_error(error)
+      raise ArgumentError, "The argument to `add_error` cannot be nil" if error.nil?
+
+      errors.push(normalize_for_json(error))
+      set(errors: errors)
+    end
 
     def save(data)
-      normalized_data = JSON.parse(data.to_json, quirks_mode: true)
-      set(data: normalized_data)
+      set(data: normalize_for_json(data))
+    end
+
+    def normalize_for_json(input)
+      JSON.parse(input.to_json, quirks_mode: true)
     end
 
     def delete
@@ -141,21 +149,12 @@ module Jobba
     end
 
     def delete!
-      redis.multi do
-        redis.del(job_key)
-
-        State::ALL.each do |state|
-          redis.srem(state.name, id)
-          redis.zrem(state.timestamp_name, id)
-        end
-
-        redis.srem(job_name_key, id)
+      delete_in_redis!
+      delete_locally!
+    end
 
-        redis.del(job_args_key)
-        job_args.marshal_dump.values.each do |arg|
-          redis.srem(job_arg_key(arg), id)
-        end
-      end
+    def prior_attempts
+      [*0..attempt-1].collect{|ii| self.class.find!("#{id}:#{ii}")}
     end
 
     protected
@@ -165,28 +164,40 @@ module Jobba
     end
 
     def self.raw_redis_hash(id)
-      main_hash, job_args_hash = redis.multi do
-        redis.hgetall(job_key(id))
-        redis.hgetall(job_args_key(id))
-      end
+      redis.hgetall(job_key(id))
+    end
+
+    def restart!
+      # Identify the values we want the restarted status to have, archive
+      # the attempt (clears out redis and local for this status), then
+      # set the restarted values
+
+      restarted_values = {
+        id: id,
+        attempt: attempt+1,
+        recorded_at: recorded_at,
+        queued_at: queued_at,
+        progress: 0,
+        errors: [],
+        job_name: job_name,
+        job_args: job_args
+      }
 
-      return nil if main_hash.empty?
+      archive_attempt!
 
-      main_hash['job_args'] = job_args_hash.to_json if !job_args_hash.nil?
-      main_hash
+      set(restarted_values)
+      move_to_state!(State::STARTED)
     end
 
-    def leave_current_state!
-      redis.srem(state.name, id)
+    def archive_attempt!
+      archived_job_key = job_key(attempt)
+      redis.rename(job_key, archived_job_key)
+      redis.hset(archived_job_key, :id, "#{id}:#{attempt}".to_json)
+      delete_locally!
     end
 
-    def enter_state!(state)
-      time, usec_int = now
-      set(state: state.name, state.timestamp_name => usec_int)
-      self.state = state
-      self.send("#{state.timestamp_name}=",time)
-      redis.zadd(state.timestamp_name, usec_int, id)
-      redis.sadd(state.name, id)
+    def move_to_state!(state)
+      set(state: state, state.timestamp_name_key => Jobba::Time.now)
     end
 
     def initialize(attrs = {})
@@ -195,20 +206,25 @@ module Jobba
       @json_encoded_attrs = attrs[:raw]
 
       if @json_encoded_attrs.nil? || @json_encoded_attrs.empty?
-        @id       = attrs[:id]       || attrs['id']       || SecureRandom.uuid
-        @state    = attrs[:state]    || attrs['state']    || State::UNKNOWN
-        @progress = attrs[:progress] || attrs['progress'] || 0
-        @errors   = attrs[:errors]   || attrs['errors']   || []
-        @data     = attrs[:data]     || attrs['data']     || {}
+
+        @id       = attrs[:id]       || SecureRandom.uuid
+        @state    = attrs[:state]    || State::UNKNOWN
+        @progress = attrs[:progress] || 0
+        @errors   = attrs[:errors]   || []
+        @data     = attrs[:data]     || {}
+        @attempt  = attrs[:attempt]  || 0
+        @job_args = attrs[:job_args] || {}
+        @job_name = attrs[:job_name]
 
         if attrs[:persist]
           redis.multi do
             set({
               id: id,
               progress: progress,
-              errors: errors
+              errors: errors,
+              attempt: attempt
             })
-            enter_state!(state)
+            move_to_state!(state)
           end
         end
       end
@@ -224,59 +240,122 @@ module Jobba
       when /.*_at/
         attribute.nil? ? nil : Jobba::Utils.time_from_usec_int(attribute.to_i)
       when 'job_args'
-        OpenStruct.new(attribute)
+        attribute || {}
       else
         attribute
       end
     end
 
     def set(incoming_hash)
+      # in case the ID isn't set but is in the hash, set locally so other
+      # commands can use it
+      self.id = incoming_hash[:id] || id
+
       apply_consistency_rules!(incoming_hash)
-      set_hash_locally(incoming_hash)
+
       set_hash_in_redis(incoming_hash)
+      set_state_in_redis(incoming_hash)
+      set_state_timestamps_in_redis(incoming_hash)
+
+      set_hash_locally(incoming_hash)
     end
 
     def apply_consistency_rules!(hash)
       hash[:progress] = 1.0 if hash[:state] == State::SUCCEEDED
     end
 
+    def set_hash_in_redis(hash)
+      redis_key_value_array =
+        hash.to_a.flat_map do |kv_array|
+          key = kv_array[0]
+          value = kv_array[1]
+          value = Jobba::Utils.time_to_usec_int(value) if value.is_a?(::Time)
+
+          [key, value.to_json]
+        end
+
+      Jobba.redis.hmset(job_key, *redis_key_value_array)
+    end
+
+    def set_state_in_redis(hash)
+      return unless hash[:state]
+      redis.srem(state.name, id) unless state.nil? # leave old state if set
+      redis.sadd(hash[:state].name, id)            # enter new state
+    end
+
+    def set_state_timestamps_in_redis(hash)
+      timestamp_names = hash.keys & State::ALL_TIMESTAMP_SYMBOLS
+
+      timestamp_names.each do |timestamp_name|
+        usec_int = Utils.time_to_usec_int(hash[timestamp_name])
+        redis.zadd(timestamp_name, usec_int, id)
+      end
+    end
+
     def set_hash_locally(hash)
       hash.each{ |key, value| self.send("#{key}=", value) }
     end
 
-    def set_hash_in_redis(hash)
-      redis_key_value_array =
-        hash.to_a.flat_map{|kv_array| [kv_array[0], kv_array[1].to_json]}
+    def delete_in_redis!
+      redis.multi do
+        redis.del(job_key)
 
-      Jobba.redis.hmset(job_key, *redis_key_value_array)
+        State::ALL.each do |state|
+          redis.srem(state.name, id)
+          redis.zrem(state.timestamp_name, id)
+        end
+
+        redis.srem(job_name_key, id)
+
+        delete_self_from_job_args_set!
+      end
+
+      prior_attempts.each(&:delete!)
     end
 
-    def job_name_key
-      "job_name:#{job_name}"
+    def delete_locally!
+      clear_attrs
+      @json_encoded_attrs = nil
     end
 
-    def job_key
-      self.class.job_key(id)
+    def delete_self_from_job_args_set!
+      self.job_args.values.each do |arg|
+        redis.srem(job_arg_key(arg), id)
+      end
     end
 
-    def self.job_key(id)
-      raise(ArgumentError, "`id` cannot be nil") if id.nil?
-      "id:#{id}"
+    def add_self_to_job_args_set!
+      self.job_args.values.each do |arg|
+        redis.sadd(job_arg_key(arg), id)
+      end
     end
 
-    def job_args_key
-      self.class.job_args_key(id)
+    def clear_attrs
+      self.class.local_attrs.each{|aa| send("#{aa}=",nil)}
+    end
+
+    def job_name_key
+      "job_name:#{job_name}"
     end
 
-    def self.job_args_key(id)
+    def job_key(attempt=nil)
+      self.class.job_key(id, attempt)
+    end
+
+    def self.job_key(id, attempt=nil)
       raise(ArgumentError, "`id` cannot be nil") if id.nil?
-      "job_args:#{id}"
+      attempt.nil? ? "id:#{id}" : "id:#{id}:#{attempt}"
     end
 
     def job_arg_key(arg)
       "job_arg:#{arg}"
     end
 
+    def job_errors_key(id)
+      raise(ArgumentError, "`id` cannot be nil") if id.nil?
+      "#{id}:errors"
+    end
+
     def compute_fractional_progress(at, out_of)
       if at.nil?
         raise ArgumentError, "Must specify at least `at` argument to `progress` call"

data/lib/jobba/version.rb

@@ -1,3 +1,3 @@
 module Jobba
-  VERSION = "1.2.0"
+  VERSION = "1.3.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: jobba
 version: !ruby/object:Gem::Version
-  version: 1.2.0
+  version: 1.3.0
 platform: ruby
 authors:
 - JP Slavinsky
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2016-01-03 00:00:00.000000000 Z
+date: 2016-01-14 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: redis