disc 0.0.26 → 0.0.27

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 68a355c65ac7b54ac4da0771e2b17af8b6b54899
-  data.tar.gz: 994f4f7fae76906db3590c54f8be1e5705e7f721
+  metadata.gz: 176f77e62f7e6d5d3fb12160f104e6f9920829f4
+  data.tar.gz: 3f081fa1f1835c21d92c6b52d20650d09b697e5a
 SHA512:
-  metadata.gz: 2c5a300f2a8d2c6d46901f4e3cf6a6ac2778abab5638dfa07caeeab807d3a701590c3d528e2557043ce15ac4e5d803088a36d81664ebed699fca433a2ae5c400
-  data.tar.gz: 8ca93a43cafb8cb9420b260ae83124d27a09a9b45e371fa1af94e66ba2c8e49f53c5ca64c4c6d28acf6a50ae27b71d0c4dfe14be47e10faec05c705021e2b86f
+  metadata.gz: 285379b94730d7a5e1eec6a5d551dfd42403452f91289a1fbce88c49d7f26457b59d9e32be25eb16c8a137888756f6ed851666ff64fc07532e0494e8deee8ef2
+  data.tar.gz: ad8a5da1ea7ce655c093796186abd7eb292fd22b2f68e0e7f13c9473643e93c28345893283966196578058fa121bbce70a8971df7f57d2e20b0cb298eeff41f8

data/.gems

@@ -1,5 +1,4 @@
 cutest:1.2.2
 byebug:5.0.0
 disque:0.0.6
-msgpack:0.6.2
 celluloid:0.17.0

data/README.md

@@ -2,11 +2,11 @@
 
 Disc fills the gap between your Ruby service objects and [antirez](http://antirez.com/)'s wonderful [Disque](https://github.com/antirez/disque) backend.
 
-<a href=https://www.flickr.com/photos/noodlefish/5321412234" target="blank_">
+<a href="https://www.flickr.com/photos/noodlefish/5321412234" target="blank_">
 ![Disc Wars!](https://cloud.githubusercontent.com/assets/437/8634016/b63ee0f8-27e6-11e5-9a78-51921bd32c88.jpg)
 </a>
 
-## Usage
+## Basic Usage
 
 1.  Install the gem
 
@@ -35,36 +35,99 @@ Disc fills the gap between your Ruby service objects and [antirez](http://antire
   CreateGameGrid.enqueue('light_cycle')
   ```
 
+
+4. Create a file that requires anything needed for your jobs to run
+
+  ```ruby
+# disc_init.rb
+  require 'disc/worker'
+  Dir['./jobs/**/*.rb'].each { |job| require job }
+  ```
+
+5. Run as many Disc Worker processes as you wish, requiring your `disc_init.rb` file
+
+  ```bash
+  $ QUEUES=urgent,default disc -r ./disc_init.rb
+  ```
+
 4. Or enqueue them to be performed at some time in the future, or on a queue other than it's default.
 
   ```ruby
   CreateGameGrid.enqueue(
     'disc_arena',
-    at: DateTime.new(2015, 12, 31),
+    at: DateTime.new(2020, 12, 31),
     queue: 'not_so_important'
   )
   ```
 
-5. Create a file that requires anything needed for your jobs to run
+## Disc Jobs
 
-  ```ruby
-# disc_init.rb
-  require 'ohm'
-  Dir['./jobs/**/*.rb'].each { |job| require job }
-  ```
+`Disc::Job` is a module you can include in your Ruby classes, this allows a Disc worker process to execute the code in them by adding a class method (`#enqueue`) with the following signature:
 
-6. Run as many Disc Worker processes as you wish, requiring your `disc_init.rb` file
+```Ruby
+def enqueue(arguments, at: nil, queue: nil, **options)
+end
+```
 
-  ```bash
-  $ QUEUES=urgent,default disc -r ./disc_init.rb
-  ```
+Signature documentation follows:
 
-## Notes about Jobs
+```ruby
+## Disc's `#enqueue` is the main user-facing method of a Disc job, it
+#  enqueues a job with a given set of arguments in Disque, so it can be
+#  picked up by a Disc worker process.
+#
+## Parameters:
+#
+## `arguments`  - an optional array of arguments with which to execute
+#                 the job's #perform method.
+#
+# `at`          - an optional named parameter specifying a moment in the
+#                 future in which to run the job, must respond to
+#                 `#to_time`.
+#
+## `queue`      - an optional named parameter specifying the name of the
+#                 queue in which to store the job, defaults to the class
+#                 Disc queue or to 'default' if no Disc queue is specified
+#                 in the class.
+#
+##  `**options` - an optional hash of options to forward internally to
+#                 [disque-rb](https://github.com/soveran/disque-rb)'s
+#                 `#push` method, valid options are:
+#
+##  `replicate: <count>`  - specifies the number of nodes the job should
+#                           be replicated to.
+#
+### `delay: <sec>`        - specifies a delay time in seconds for the job
+#                           to be delivered to a Disc worker, it is ignored
+#                           if using the `at` parameter.
+#
+### `ttl: <sec>`          - specifies the job's time to live in seconds:
+#                           after this time, the job is deleted even if
+#                           it was not successfully delivered. If not
+#                           specified, the default TTL is one day.
+#
+### `maxlen: <count>`     - specifies that if there are already <count>
+#                           messages queued for the specified queue name,
+#                           the message is refused.
+#
+### `async: true`         - asks the server to let the command return ASAP
+#                           and replicate the job to other nodes in the background.
+#
+#
+### CAVEATS
+#
+## For convenience, any object can be passed as the `arguments` parameter,
+#  `Array.wrap` will be used internally to preserve the array structure.
+#
+## The `arguments` parameter is serialized for storage using `Disc.serialize`
+#  and Disc workers picking it up use `Disc.deserialize` on it, both methods
+#  use standard library json but can be overriden by the user
+#
+```
 
-Jobs are fairly straightforward Ruby classes, internally Disc serializes them to MessagePack so they can be stored in Disque, this has a few implications:
+You can see [Disque's ADDJOB documentation](https://github.com/antirez/disque#addjob-queue_name-job-ms-timeout-replicate-count-delay-sec-retry-sec-ttl-sec-maxlen-count-async) for more details
 
-* Don't enqueue complex objects! Instead of `user`, enqueue `user.id`!
-* If your job takes multiple arguments, you'll want to pass all those arguments in the first parameter of `#enqueue` as an array.
+When a Disc worker process is assigned a job, it will create a new intance of the job's class and execute the `#perform` method with whatever arguments were previously passed to `#enqueue`.
 
 Example:
 

data/disc.gemspec

@@ -14,6 +14,5 @@ Gem::Specification.new do |s|
   s.executables.push('disc')
 
   s.add_dependency('disque', '~> 0.0.6')
-  s.add_dependency('msgpack', '>= 0.5.6', '< 0.6.3')
   s.add_dependency('clap', '~> 1.0')
 end

data/lib/active_job/queue_adapters/disc_adapter.rb

@@ -1,5 +1,6 @@
 require 'date'
 require 'msgpack'
+require 'disc/worker'
 
 module ActiveJob
   module QueueAdapters
@@ -11,10 +12,10 @@ module ActiveJob
       def self.enqueue_at(job, timestamp)
         Disc.disque.push(
           job.queue_name,
-          {
+          Disc.serialize({
             class: job.class.name,
             arguments: job.arguments
-          }.to_msgpack,
+          }),
           Disc.disque_timeout,
           delay: timestamp.nil? ? nil : (timestamp.to_time.to_i - DateTime.now.to_time.to_i)
         )

data/lib/disc.rb

@@ -1,13 +1,8 @@
 require 'date'
 require 'disque'
-require 'msgpack'
-
-require_relative 'disc/version'
+require 'json'
 
 class Disc
-  attr_reader :disque,
-    :disque_timeout
-
   def self.disque
     @disque ||= Disque.new(
       ENV.fetch('DISQUE_NODES', 'localhost:7711'),
@@ -21,7 +16,7 @@ class Disc
   end
 
   def self.disque_timeout
-    @disque_timeout ||= 100
+    @disque_timeout ||= ENV.fetch('DISQUE_TIMEOUT', 100)
   end
 
   def self.disque_timeout=(timeout)
@@ -36,6 +31,10 @@ class Disc
     @default_queue = queue
   end
 
+  def self.qlen(queue)
+    disque.call('QLEN', queue)
+  end
+
   def self.flush
     Disc.disque.call('DEBUG', 'FLUSHALL')
   end
@@ -44,119 +43,15 @@ class Disc
     $stderr.puts exception
   end
 
-  class Worker
-    attr_reader :disque,
-                :queues,
-                :timeout,
-                :count
-
-    def self.current
-      @current ||= new
-    end
-
-    def self.run
-      current.run
-    end
-
-    def self.stop
-      current.stop
-    end
-
-    def initialize(options = {})
-      @disque = options.fetch(:disque, Disc.disque)
-      @queues = options.fetch(
-        :queues,
-        ENV.fetch('QUEUES', Disc.default_queue)
-      ).split(',')
-      @count = Integer(
-        options.fetch(
-          :count,
-          ENV.fetch('DISQUE_COUNT', '1')
-        )
-      )
-      @timeout = Integer(
-        options.fetch(
-          :timeout,
-          ENV.fetch('DISQUE_TIMEOUT', '2000')
-        )
-      )
-
-      self.run if options[:run]
-      self
-    end
-
-    def stop
-      @stop = true
-    end
-
-    def run
-      $stdout.puts("Disc::Worker listening in #{queues}")
-      loop do
-        jobs = disque.fetch(from: queues, timeout: timeout, count: count)
-        Array(jobs).each do |queue, msgid, serialized_job|
-          begin
-            job = MessagePack.unpack(serialized_job)
-            instance = Object.const_get(job['class']).new
-            instance.perform(*job['arguments'])
-            disque.call('ACKJOB', msgid)
-          rescue => err
-            Disc.on_error(err, job.update('id' => msgid, 'queue' => queue))
-          end
-        end
-
-        break if @stop
-      end
-    ensure
-      disque.quit
-    end
+  def self.serialize(args)
+    JSON.dump(args)
   end
 
-  module Job
-    attr_reader :arguments,
-                :disque,
-                :disc_options
-
-    def self.included(base)
-      base.extend(ClassMethods)
-    end
-
-    module ClassMethods
-      def disque
-        defined?(@disque) ? @disque : Disc.disque
-      end
-
-      def disque=(disque)
-        @disque = disque
-      end
-
-      def disc(queue: nil, **options)
-        @queue = queue
-        @disc_options = options
-      end
-
-      def disc_options
-        @disc_options ||= {}
-      end
-
-      def queue
-        @queue || Disc.default_queue
-      end
-
-      def enqueue(args = [], at: nil, queue: nil, **options)
-        options = disc_options.merge(options).tap do |opt|
-          opt[:delay] = at.to_time.to_i - DateTime.now.to_time.to_i unless at.nil?
-        end
-
-        disque.push(
-          queue || self.queue,
-          {
-            class: self.name,
-            arguments: Array(args)
-          }.to_msgpack,
-          Disc.disque_timeout,
-          options
-        )
-      end
-    end
+  def self.deserialize(data)
+    JSON.parse(data)
   end
 end
+
+require_relative 'disc/job'
+require_relative 'disc/version'
+

data/lib/disc/job.rb

@@ -0,0 +1,127 @@
+module Disc::Job
+  attr_accessor :disque_id,
+                :arguments
+
+  def self.included(base)
+    base.extend(ClassMethods)
+  end
+
+  def info
+    return nil if disque_id.nil?
+
+    Hash[*self.class.disque.call("SHOW", disque_id)]
+  end
+
+  def state
+    info.fetch('state')
+  end
+
+  module ClassMethods
+    def [](disque_id)
+      job_data = disque.call("SHOW", disque_id)
+      return nil if job_data.nil?
+
+      job = self.new
+      job_data = Hash[*job_data]
+
+      job.disque_id = disque_id
+      job.arguments = Disc.deserialize(job_data.fetch('body')).fetch('arguments')
+
+      return job
+    end
+
+    def disque
+      defined?(@disque) ? @disque : Disc.disque
+    end
+
+    def disque=(disque)
+      @disque = disque
+    end
+
+    def disc(queue: nil, **options)
+      @queue = queue
+      @disc_options = options
+    end
+
+    def disc_options
+      @disc_options ||= {}
+    end
+
+    def queue
+      @queue || Disc.default_queue
+    end
+
+    def perform(arguments)
+      self.new.perform(*arguments)
+    end
+
+    ## Disc's `#enqueue` is the main user-facing method of a Disc job, it
+    #  enqueues a job with a given set of arguments in Disque, so it can be
+    #  picked up by a Disc worker process.
+    #
+    ## Parameters:
+    #
+    ## `arguments`  - an optional array of arguments with which to execute
+    #                 the job's #perform method.
+    #
+    ## `at`         - an optional named parameter specifying a moment in the
+    #                 future in which to run the job, must respond to
+    #                 `#to_time`.
+    #
+    ## `queue`      - an optional named parameter specifying the name of the
+    #                 queue in which to store the job, defaults to the class
+    #                 Disc queue or to 'default' if no Disc queue is specified
+    #                 in the class.
+    #
+    ##  `**options` - an optional hash of options to forward internally to
+    #                 [disque-rb](https://github.com/soveran/disque-rb)'s
+    #                 `#push` method, valid options are:
+    #
+    ##  `replicate: <count>`  - specifies the number of nodes the job should
+    #                           be replicated to.
+    #
+    ### `delay: <sec>`        - specifies a delay time in seconds for the job
+    #                           to be delivered to a Disc worker, it is ignored
+    #                           if using the `at` parameter.
+    #
+    ### `ttl: <sec>`          - specifies the job's time to live in seconds:
+    #                           after this time, the job is deleted even if
+    #                           it was not successfully delivered. If not
+    #                           specified, the default TTL is one day.
+    #
+    ### `maxlen: <count>`     - specifies that if there are already <count>
+    #                           messages queued for the specified queue name,
+    #                           the message is refused.
+    #
+    ### `async: true`         - asks the server to let the command return ASAP
+    #                           and replicate the job to other nodes in the background.
+    #
+    #
+    ### CAVEATS
+    #
+    ## For convenience, any object can be passed as the `arguments` parameter,
+    #  `Array.wrap` will be used internally to preserve the array structure.
+    #
+    ## The `arguments` parameter is serialized for storage using `Disc.serialize`
+    #  and Disc workers picking it up use `Disc.deserialize` on it, both methods
+    #  use standard library json but can be overriden by the user
+    #
+    def enqueue(args = [], at: nil, queue: nil, **options)
+      options = disc_options.merge(options).tap do |opt|
+        opt[:delay] = at.to_time.to_i - DateTime.now.to_time.to_i unless at.nil?
+      end
+
+      disque_id = disque.push(
+        queue || self.queue,
+        Disc.serialize({
+          class: self.name,
+          arguments: Array(args)
+        }),
+        Disc.disque_timeout,
+        options
+      )
+
+      return self[disque_id]
+    end
+  end
+end

data/lib/disc/version.rb

@@ -1,3 +1,3 @@
 class Disc
-  VERSION = "0.0.26"
+  VERSION = "0.0.27"
 end

data/lib/disc/worker.rb

@@ -0,0 +1,70 @@
+require 'disc'
+
+
+class Disc::Worker
+  attr_reader :disque,
+              :queues,
+              :timeout,
+              :count
+
+  def self.current
+    @current ||= new
+  end
+
+  def self.run
+    current.run
+  end
+
+  def self.stop
+    current.stop
+  end
+
+  def initialize(options = {})
+    @disque = options.fetch(:disque, Disc.disque)
+    @queues = options.fetch(
+      :queues,
+      ENV.fetch('QUEUES', Disc.default_queue)
+    ).split(',')
+    @count = Integer(
+      options.fetch(
+        :count,
+        ENV.fetch('DISQUE_COUNT', '1')
+      )
+    )
+    @timeout = Integer(
+      options.fetch(
+        :timeout,
+        ENV.fetch('DISQUE_TIMEOUT', '2000')
+      )
+    )
+
+    self.run if options[:run]
+    self
+  end
+
+  def stop
+    @stop = true
+  end
+
+  def run
+    $stdout.puts("Disc::Worker listening in #{queues}")
+    loop do
+      jobs = disque.fetch(from: queues, timeout: timeout, count: count)
+      Array(jobs).each do |queue, msgid, serialized_job|
+        begin
+          job = Disc.deserialize(serialized_job)
+          instance = Object.const_get(job['class']).new
+          instance.perform(*job['arguments'])
+          disque.call('ACKJOB', msgid)
+          $stdout.puts("Completed #{job['class']} id #{msgid}")
+        rescue => err
+          Disc.on_error(err, job.update('id' => msgid, 'queue' => queue))
+        end
+      end
+
+      break if @stop
+    end
+  ensure
+    disque.quit
+  end
+end

data/test/disc_test.rb

@@ -1,6 +1,5 @@
 require 'cutest'
 require 'disc'
-require 'msgpack'
 require 'pty'
 require 'timeout'
 
@@ -55,20 +54,20 @@ scope do
   end
 
   test 'jobs are enqueued to the correct Disque queue with appropriate parameters and class' do
-    jobid = Echoer.enqueue(['one argument', { random: 'data' }, 3])
+    job_instance = Echoer.enqueue(['one argument', { random: 'data' }, 3])
 
     jobs = Array(Disc.disque.fetch(from: ['test'], timeout: Disc.disque_timeout, count: 1))
     assert jobs.any?
     assert_equal 1, jobs.count
 
     jobs.first.tap do |queue, id, serialized_job|
-      job = MessagePack.unpack(serialized_job)
+      job = Disc.deserialize(serialized_job)
 
       assert job.has_key?('class')
       assert job.has_key?('arguments')
 
       assert_equal 'Echoer', job['class']
-      assert_equal jobid, id
+      assert_equal job_instance.disque_id, id
 
       args = job['arguments']
       assert_equal 3, args.size
@@ -78,8 +77,29 @@ scope do
     end
   end
 
+  test 'we get easy access to the job via the job id' do
+    job_instance = Echoer.enqueue(['one argument', { random: 'data' }, 3])
+
+    assert job_instance.is_a?(Echoer)
+    assert !job_instance.disque_id.nil?
+    assert !job_instance.info.nil?
+
+    job = Echoer[job_instance.disque_id]
+
+    assert job.is_a?(Echoer)
+    assert_equal 'queued', job.state
+    assert_equal 3, job.arguments.count
+    assert_equal 'one argument', job.arguments.first
+  end
+
+  test 'we can query the lenght of a given queue' do
+    job_instance = Echoer.enqueue(['one argument', { random: 'data' }, 3])
+
+    assert_equal 1, Disc.qlen(Echoer.queue)
+  end
+
   test 'enqueue at timestamp behaves properly' do
-    jobid = Echoer.enqueue(['one argument', { random: 'data' }, 3], at: Time.now + 1)
+    job_instance = Echoer.enqueue(['one argument', { random: 'data' }, 3], at: Time.now + 1)
 
     jobs = Array(Disc.disque.fetch(from: ['test'], timeout: Disc.disque_timeout, count: 1))
     assert jobs.empty?
@@ -95,8 +115,8 @@ scope do
 
     jobs.first.tap do |queue, id, serialized_job|
       assert_equal 'test', queue
-      assert_equal jobid, id
-      job = MessagePack.unpack(serialized_job)
+      assert_equal job_instance.disque_id, id
+      job = Disc.deserialize(serialized_job)
       assert job.has_key?('class')
       assert job.has_key?('arguments')
       assert_equal 'Echoer', job['class']
@@ -107,7 +127,7 @@ scope do
   test 'jobs are executed' do
     Echoer.enqueue(['one argument', { random: 'data' }, 3])
 
-    run('QUEUES=test ruby -Ilib bin/disc -r ./examples/echoer') do |cout, pid|
+    run('QUEUES=test ruby -Ilib bin/disc -r disc/worker -r ./examples/echoer') do |cout, pid|
       output = Timeout.timeout(1) { cout.take(3) }
       jobs = Disc.disque.fetch(from: ['test'], timeout: Disc.disque_timeout, count: 1)
       assert jobs.nil?
@@ -116,9 +136,9 @@ scope do
   end
 
   test 'Disc.on_error will catch unhandled exceptions and keep disc alive' do
-    Failer.enqueue('this can only end positively')
+    failer = Failer.enqueue('this can only end positively')
 
-    run('QUEUES=test ruby -Ilib bin/disc -r ./examples/failer') do |cout, pid|
+    run('QUEUES=test ruby -Ilib bin/disc -r disc/worker -r ./examples/failer') do |cout, pid|
       output = Timeout.timeout(1) { cout.take(5) }
       jobs = Disc.disque.fetch(from: ['test'], timeout: Disc.disque_timeout, count: 1)
       assert jobs.nil?
@@ -139,7 +159,7 @@ scope do
   end
 
   test 'enqueue supports delay' do
-    jobid = Echoer.enqueue(['one argument', { random: 'data' }, 3], delay: 2)
+    job_instance = Echoer.enqueue(['one argument', { random: 'data' }, 3], delay: 2)
 
     jobs = Array(Disc.disque.fetch(from: ['test'], timeout: Disc.disque_timeout, count: 1))
     assert jobs.empty?
@@ -155,7 +175,7 @@ scope do
   end
 
   test 'enqueue supports retry' do
-    jobid = Echoer.enqueue(['one argument', { random: 'data' }, 3], retry: 1)
+    job_instance = Echoer.enqueue(['one argument', { random: 'data' }, 3], retry: 1)
 
     jobs = Array(Disc.disque.fetch(from: ['test'], timeout: Disc.disque_timeout, count: 1))
     assert jobs.any?
@@ -168,7 +188,7 @@ scope do
   end
 
   test 'enqueue supports ttl' do
-    jobid = Echoer.enqueue(['one argument', { random: 'data' }, 3], ttl: 1)
+    job_instance = Echoer.enqueue(['one argument', { random: 'data' }, 3], ttl: 1)
 
     sleep 1.5
     jobs = Array(Disc.disque.fetch(from: ['test'], timeout: Disc.disque_timeout, count: 1))
@@ -184,7 +204,7 @@ scope do
   end
 
   test 'enqueue supports async' do
-    jobid = Echoer.enqueue(['one argument', { random: 'data' }, 3], async: true)
+    job_instance = Echoer.enqueue(['one argument', { random: 'data' }, 3], async: true)
 
     sleep 1 # async is too fast to reliably assert an empty queue, let's wait instead
     jobs = Array(Disc.disque.fetch(from: ['test'], timeout: Disc.disque_timeout, count: 1))

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: disc
 version: !ruby/object:Gem::Version
-  version: 0.0.26
+  version: 0.0.27
 platform: ruby
 authors:
 - pote
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-09-21 00:00:00.000000000 Z
+date: 2016-05-01 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: disque
@@ -24,26 +24,6 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: 0.0.6
-- !ruby/object:Gem::Dependency
-  name: msgpack
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: 0.5.6
-    - - "<"
-      - !ruby/object:Gem::Version
-        version: 0.6.3
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: 0.5.6
-    - - "<"
-      - !ruby/object:Gem::Version
-        version: 0.6.3
 - !ruby/object:Gem::Dependency
   name: clap
   requirement: !ruby/object:Gem::Requirement
@@ -83,8 +63,10 @@ files:
 - examples/greeter.rb
 - lib/active_job/queue_adapters/disc_adapter.rb
 - lib/disc.rb
+- lib/disc/job.rb
 - lib/disc/testing.rb
 - lib/disc/version.rb
+- lib/disc/worker.rb
 - test/disc_test.rb
 homepage: https://github.com/pote/disc
 licenses:
@@ -106,7 +88,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.4.5
+rubygems_version: 2.5.1
 signing_key: 
 specification_version: 4
 summary: A simple and powerful Disque job implementation