jobba 1.1.0 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: be04439547ffdf5ad57d5ac3a3279ca10ed0ae5b
-  data.tar.gz: 13de0a5da8c25827bdef5a7dbb99143d3e12427b
+  metadata.gz: dcfba40866333af0570df573302aaac92ef2c985
+  data.tar.gz: 031ee23b4e6ef49a80c46f539e68b7772a621edc
 SHA512:
-  metadata.gz: 1099b20ffe944620cf5d712eea9c64410da0fa877ac9695c27dfee3cf557f2c348eeeaf9ca44088fa4c42fe0f1e8b6f1d27541412e0211ad5f7d900a1625baaa
-  data.tar.gz: a041d7540f6518291bd659b881dd03fc74d942529786b7c70145c9c327b6261facb44d489c7fd33672fe23a29d446c799ce362494d4d018f1c9824c9d4c5c64a
+  metadata.gz: ddd79547cd5913d6b7c10962150ebec677055ab979ad0890685d2362fd0123cd312595efeb27ad8a39618ef4cc256cc99498bccfa889f95a6f8452db42a62fc0
+  data.tar.gz: e9442849d2a665b3b0253d5f1ada4ac74f61441e7976b3a40d058e758f72bef9c80c6fc467caea97c3fa590f1f3cf6d7f05f9c6f3188418bb8410dae2f0cbce5

data/README.md

@@ -19,7 +19,7 @@ or
 $> gem install jobba
 ```
 
-Semantic versioning will begin with version `2.0.0`.
+Version 1.x.x follows the scheme, 1.major_change.minor_change.  Normal semantic versioning (major/minor/patch) will begin with version `2.0.0`.
 
 ## Configuration
 
@@ -237,7 +237,7 @@ You can also call `reload!` on a `Status` to have it reset its state to what is
 Once jobs are completed or otherwise no longer interesting, it'd be nice to clear them out of Redis.  You can do this with:
 
 ```ruby
-my_status.delete    # freaks out if `my_status` isn't complete
+my_status.delete    # freaks out if `my_status` isn't completed
 my_status.delete!   # always deletes
 ```
 
@@ -247,6 +247,12 @@ Jobba has an activerecord-like query interface for finding Status objects.
 
 ### Basic Query Examples
 
+**Getting All Statuses**
+
+```ruby
+Jobba.all
+```
+
 **State**
 
 ```ruby
@@ -305,6 +311,15 @@ Jobba.where(job_arg: "gid://app/MyModel/42")
 Jobba.where(job_arg: "gid://app/Person/86")
 ```
 
+**Status IDs**
+
+```ruby
+Jobba.where(id: nil)
+Jobba.where(id: [])
+Jobba.where(id: "some_id")
+Jobba.where(id: ["an_id", "another_id"])
+```
+
 ### Query Chaining
 
 Queries can be chained! (intersects the results of each `where` clause)
@@ -314,47 +329,79 @@ Jobba.where(state: :queued).where(recorded_at: {after: some_time})
 Jobba.where(job_name: "MyTroublesomeJob").where(state: :failed)
 ```
 
-### Operations on Queries
+### Sort Order
+
+Currently, results from queries are not guaranteed to be in any order.  You can sort them yourself using normal Ruby calls.
+
+### Running a Query to get Statuses
+
+```ruby
+Jobba.where(...).run
+```
+
+When you call `run` on a query, you'll get back a `Statuses` object, which is simply a collection of `Status` objects with a few convenience methods and bulk operations.
+
+**Bulk Methods on Statuses**
+
+* `delete_all`
+* `delete_all!`
+* `request_kill_all!`
+
+These work like describe above for individual `Status` objects.
+
+There is also a not-very-tested `multi` operation that takes a block and executes the block inside a Redis `multi` call.  Do not use it unless you really know what you are doing.
 
-When you have a query you can run the following methods on it.  These act like what you'd expect for a Ruby array.
+```ruby
+my_statuses.multi do |status, redis|
+  # do stuff on `status` using the `redis` connection
+end
+```
+
+**Array-like Methods on Statuses**
 
-* `first`
 * `any?`
 * `none?`
 * `all?`
-* `each`
-* `each_with_index`
 * `map`
 * `collect`
-* `select`
 * `empty?`
 * `count`
+* `select!`
+* `reject!`
 
-`empty?` and `count` are performed in Redis without bringing back all query results to Ruby.
-
-You can also call two special methods directly on `Jobba`:
+If you want to get an array of `Status` objects from a `Statuses` object, just call
 
 ```ruby
-Jobba.all     # returns all statuses
-Jobba.count   # returns count of all statuses
+a_statuses_object.to_a
 ```
 
-## Statuses Object
+`select!` and `reject!`, as you would expect, operate in place and also return `self`.
 
-Calling `all` on a query returns a `Statuses` object, which is just a collection of `Status` objects.  It has a few methods for doing bulk operations on all `Status` objects in it.
+### Passthrough Methods on Queries
 
-* `delete`
-* `delete!`
-* `request_kill!`
+As a convenience, if you call a method on `Query` that isn't defined there but is defined on `Statuses`, a new `Statuses` object will be created for you and your method called on it.
 
-These work like describe above for individual `Status` objects.
+```ruby
+Jobba.where(state: :queued).collect(&:queued_at)
+```
 
-There is also a not-very-tested `multi` operation that takes a block and executes the block inside a Redis `multi` call.
+is the same as
 
 ```ruby
-my_statuses.multi do |status, redis|
-  # do stuff on `status` using the `redis` connection
-end
+Jobba.where(state: :queued).run.collect(&:queued_at)
+```
+
+### Query Counts
+
+Notably, both `Query` and `Statuses` define the `count` and `empty?` methods.  Which ones you use affects if the counting is done in Redis or in Ruby:
+
+```ruby
+Jobba.where(...).count       # These count in Redis
+Jobba.where(...).empty?
+Jobba.all.count
+
+Jobba.where(...).run.count   # These pull data back to Ruby and count in Ruby
+Jobba.where(...).run.empty?
 ```
 
 ## Notes
@@ -372,9 +419,6 @@ Jobba strives to do all of its operations as efficiently as possible using built
 1. Provide job min, max, and average durations.
 2. Implement `add_error`.
 8. Specs that test scale.
-9. Make sure we're calling `multi` or `pipelined` everywhere we can.
-11. Make sure we're consistent on completed/complete incompleted/incomplete.
-12. Should more `Statuses` operations return `Statuses`, e.g. `each`, so that they can be chained?
 
 
 

data/lib/jobba/clause.rb

@@ -25,7 +25,7 @@ class Jobba::Clause
   end
 
   def to_new_set
-    new_key = "temp:#{SecureRandom.hex(10)}"
+    new_key = Jobba::Utils.temp_key
 
     # Make a copy of the data into new_key then filter values if indicated
     # (always making a copy gets normal sets into a sorted set key OR if

data/lib/jobba/clause_factory.rb

@@ -15,6 +15,8 @@ class Jobba::ClauseFactory
       Jobba::Clause.new(prefix: "job_name", suffixes: value)
     when :job_arg
       Jobba::Clause.new(prefix: "job_arg", suffixes: value)
+    when :id
+      Jobba::IdClause.new(value)
     when /.*_at/
       timestamp_clause(key, value)
     else

data/lib/jobba/exceptions.rb

@@ -1,5 +1,6 @@
 module Jobba
 
   class NotCompletedError < StandardError; end
+  class NotImplemented < StandardError; end
 
 end

data/lib/jobba/id_clause.rb

@@ -0,0 +1,14 @@
+class Jobba::IdClause
+
+  include Jobba::Common
+
+  def initialize(ids)
+    @ids = [ids].flatten.compact
+  end
+
+  def to_new_set
+    new_key = Jobba::Utils.temp_key
+    redis.zadd(new_key, @ids.collect{|id| [0, id]}) if @ids.any?
+    new_key
+  end
+end

data/lib/jobba/query.rb

@@ -1,4 +1,5 @@
 require 'jobba/clause'
+require 'jobba/id_clause'
 require 'jobba/clause_factory'
 
 class Jobba::Query
@@ -14,7 +15,7 @@ class Jobba::Query
   end
 
   def count
-    run(&COUNT_STATUSES)
+    _run(COUNT_STATUSES)
   end
 
   def empty?
@@ -25,9 +26,9 @@ class Jobba::Query
   # to run on the result of the executed `where`s.  So if we don't know what
   # the method is, execute the `where`s and pass the method to its output.
 
-  def method_missing(method_name, *args)
+  def method_missing(method_name, *args, &block)
     if Jobba::Statuses.instance_methods.include?(method_name)
-      run(&GET_STATUSES).send(method_name, *args)
+      run.send(method_name, *args, &block)
     else
       super
     end
@@ -37,6 +38,10 @@ class Jobba::Query
     Jobba::Statuses.instance_methods.include?(method_name) || super
   end
 
+  def run
+    _run(GET_STATUSES)
+  end
+
   protected
 
   attr_accessor :clauses
@@ -45,36 +50,73 @@ class Jobba::Query
     @clauses = []
   end
 
-  GET_STATUSES = ->(working_set) {
-    ids = Jobba.redis.zrange(working_set, 0, -1)
-    Jobba::Statuses.new(ids)
-  }
-
-  COUNT_STATUSES = ->(working_set) {
-    Jobba.redis.zcard(working_set)
-  }
+  class RunBlocks
+    attr_reader :redis_block, :output_block
 
-  def run(&working_set_block)
-
-    # TODO PUT IN MULTI BLOCKS WHERE WE CAN!
-
-    load_default_clause if clauses.empty?
-    working_set = nil
+    def initialize(redis_block, output_block)
+      @redis_block = redis_block
+      @output_block = output_block
+    end
 
-    clauses.each_with_index do |clause, ii|
-      clause_set = clause.to_new_set
+    def output_block_result_is_redis_block_result?
+      output_block.nil?
+    end
+  end
 
-      if working_set.nil?
-        working_set = clause_set
-      else
-        redis.zinterstore(working_set, [working_set, clause_set], weights: [0, 0])
-        redis.del(clause_set)
+  GET_STATUSES = RunBlocks.new(
+    ->(working_set, redis) {
+      redis.zrange(working_set, 0, -1)
+    },
+    ->(ids) {
+      Jobba::Statuses.new(ids)
+    }
+  )
+
+  COUNT_STATUSES = RunBlocks.new(
+    ->(working_set, redis) {
+      redis.zcard(working_set)
+    },
+    nil
+  )
+
+  def _run(run_blocks)
+    # Each clause in a query is converted to a sorted set (which may be filtered,
+    # e.g. in the case of timestamp clauses) and then the sets are successively
+    # intersected.
+    #
+    # Different users of this method have different uses for the final "working"
+    # set.  Because we want to bundle all of the creations and intersections of
+    # clause sets into one call to Redis (via a `multi` block), we have users
+    # of this method provide a final block to run on the working set within
+    # Redis (and within the `multi` call) and then another block to run on
+    # the output of the first block.
+
+    multi_result = redis.multi do
+      load_default_clause if clauses.empty?
+      working_set = nil
+
+      clauses.each_with_index do |clause, ii|
+        clause_set = clause.to_new_set
+
+        if working_set.nil?
+          working_set = clause_set
+        else
+          redis.zinterstore(working_set, [working_set, clause_set], weights: [0, 0])
+          redis.del(clause_set)
+        end
       end
-    end
 
-    working_set_block.call(working_set).tap do
+      # This is later accessed as `multi_result[-2]` since it is the second to last output
+      run_blocks.redis_block.call(working_set, redis)
+
       redis.del(working_set)
     end
+
+    redis_block_output = multi_result[-2]
+
+    run_blocks.output_block_result_is_redis_block_result? ?
+      redis_block_output :
+      run_blocks.output_block.call(redis_block_output)
   end
 
   def load_default_clause

data/lib/jobba/statuses.rb

@@ -5,15 +5,23 @@ class Jobba::Statuses
 
   attr_reader :ids
 
-  def all
-    load
+  def to_a
+    load.dup
+  end
+
+  def_delegators :@ids, :empty?
+
+  def_delegators :load, :any?, :none?, :all?, :count, :map, :collect
+
+  def select!(&block)
+    modify!(:select!, &block)
   end
 
-  def_delegator :@ids, :empty?
-  def_delegators :all, :first, :any?, :none?, :all?, :each, :each_with_index,
-                       :map, :collect, :select, :count
+  def reject!(&block)
+    modify!(:reject!, &block)
+  end
 
-  def delete
+  def delete_all
     if any?(&:incomplete?)
       raise(Jobba::NotCompletedError,
             "This status cannot be deleted because it isn't complete.  Use " \
@@ -23,7 +31,7 @@ class Jobba::Statuses
     delete!
   end
 
-  def delete!
+  def delete_all!
     load
     redis.multi do
       @cache.each(&:delete!)
@@ -32,7 +40,7 @@ class Jobba::Statuses
     @ids = []
   end
 
-  def request_kill!
+  def request_kill_all!
     load
     redis.multi do
       @cache.each(&:request_kill!)
@@ -61,11 +69,24 @@ class Jobba::Statuses
       end
     end
 
+    raw_statuses.reject!(&:empty?)
+
     raw_statuses.collect do |raw_status|
       Jobba::Status.new(raw: raw_status)
     end
   end
 
+  def modify!(method, &block)
+    raise Jobba::NotImplemented unless block_given?
+    load
+    if @cache.send(method, &block).nil?
+      nil
+    else
+      @ids = @cache.collect(&:id)
+      self
+    end
+  end
+
   def initialize(*ids)
     @ids = [ids].flatten.compact
     @cache = nil

data/lib/jobba/utils.rb

@@ -22,4 +22,8 @@ module Jobba::Utils
     Jobba::Time.at(int / 1000000, int % 1000000)
   end
 
+  def self.temp_key
+    "temp:#{SecureRandom.hex(10)}"
+  end
+
 end

data/lib/jobba/version.rb

@@ -1,3 +1,3 @@
 module Jobba
-  VERSION = "1.1.0"
+  VERSION = "1.2.0"
 end

data/lib/jobba.rb

@@ -14,16 +14,12 @@ require "jobba/query"
 
 module Jobba
 
-  def self.where(*args)
-    Query.new.where(*args)
-  end
-
   def self.all
-    Query.new.all
+    Query.new
   end
 
-  def self.count
-    Query.new.count
+  def self.where(*args)
+    all.where(*args)
   end
 
   def self.configure

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: jobba
 version: !ruby/object:Gem::Version
-  version: 1.1.0
+  version: 1.2.0
 platform: ruby
 authors:
 - JP Slavinsky
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2016-01-02 00:00:00.000000000 Z
+date: 2016-01-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: redis
@@ -133,6 +133,7 @@ files:
 - lib/jobba/common.rb
 - lib/jobba/configuration.rb
 - lib/jobba/exceptions.rb
+- lib/jobba/id_clause.rb
 - lib/jobba/query.rb
 - lib/jobba/state.rb
 - lib/jobba/status.rb