resque-stages 0.0.1

data/README.md

@@ -0,0 +1,250 @@
+# Resque::Stages
+
+A Resque plugin for executing jobs in stages.  Groups are created each
+with multiple stages.  Each stage contains a number of Jobs.  All jobs
+within a stage must complete before the next stage is executed.  What
+makes this gem special is that it will wait for jobs to Retry before
+they are considered complete and the next stage will be executed.
+
+Once all jobs in all stages are complete, the stages and jobs are all
+deleted - cleaning up after itself.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'resque-stages'
+```
+
+And then execute:
+
+    $ bundle install
+
+Or install it yourself as:
+
+    $ gem install resque-stages
+
+## Usage
+
+###Basic Usage
+
+Include the stages plugin in all jobs that will be enqueued as a part of
+a Stage.  Do not worry, you will not have to enqueue the job only as
+part of a stage.  You will be able to use the job inside of or outside
+of a stage.
+
+This gem will enqueue a job with extra paramters when it is enqueued
+as a part of a stage.  To support this, the
+`perform_job` method is added to your class and it will return an
+object which will have the un-altered arguments for the job - whether or
+not the job is called with the extra parameters.
+
+```Ruby
+class MyJob
+  extend Resque::Plugins::Retry
+  include Resque::Plugins::Stages
+  
+  def perform(*args)
+    job = perform_job(*args)
+    
+    real_perform(*job.args)
+  end
+
+  # The "old" perform function can be called here as-is because the
+  # perform_job will always ensure that the original parameters are called
+  def real_perform(my_param_1, my_param_2)
+  end
+end
+```
+
+To create a grouping of stages and execute the jobs, you then create a
+group and the stages you need in it:
+
+```Ruby
+def enqueue_stages
+  Resque::Plugins::Stages::StagedGroup.within_a_grouping do |grouping|
+    stage = grouping.stage(1)
+    
+    12.times do |index|
+      stage.enqueue MyJob, index, "parameter"
+    end
+
+    stage = grouping.stage(2)
+    stage.enqueue MyJob, -1, "summary"
+  end
+end
+```
+
+`within_a_grouping` will `initiate` the first stage when it completes.
+This will cause all jobs in stage 1 to complete before stage 2 executes.
+
+NOTE:  All jobs and stages in a grouping are kept until the last job in
+the last stage completes so that you can query the jobs to find out which
+ones completed successfully and which ones failed.
+
+To aid in querying information about the jobs in a stage, each job
+includes a single string value `status_message` which can be set at any
+time and will be available until all jobs are completed.
+
+###Compatibility with other Resque Plugins
+
+In general, this plugin should be compatible with other gems.  Some special
+notations on specific compatibility should be made.
+
+[resque-compressible](https://github.com/MishaConway/resque-compressible) -
+Special code has been added to ensure compatibility with this gem and
+compressed jobs.  The only condition for this to work properly is that the
+`Resque::Plugins::Stages` gem must be included AFTER the
+`Resque::Plugins::Compressible` gem is extended so that the code knows
+that it needs to accomidate resque-compressible.
+
+[resque-retry](https://github.com/lantins/resque-retry) -
+Special code has been added to ensure compatibility with this gem and
+Retryable jobs.  When a job is retried, we know it and mark the status
+of the job with a special status indicating that the job is pending being
+retried.  When enqueued, if the job is pending a retry, the pending job
+is enqueued, not a new copy of the job.  The only condition for this to
+work properly is that the `Resque::Plugins::Stages` gem must be included
+AFTER the `Resque::Plugins::Retry` gem is extended so that the code knows
+that it needs to accomidate resque-retry.
+
+[resque-job_history](https://github.com/RealNobody/resque-job_history) -
+No special code has been added for this gem, but this gem and gems like it
+(resque-cleaner, resque-history, etc.) which record the parameters for a
+job and allow it to be re-enqueued all have the same/similar potential
+problem of enqueueing a job with outdated job IDs because we include
+the job ID in the enqueued job.  If this happens and you use the
+`perform_job` method as described, there will be no problem.  A temporary
+job with a `nil` `staged_group_stage` will be created and the params
+will be available as always.
+
+[resque](https://github.com/resque/resque) - Enqueuing jobs directly.
+If you enque a job that is staged through `Resque` itself, the job will
+still work as designed if you properly use the `perform_job` method as
+described.  It will recognize that there is no actual job and create a
+temporary one with the right parameters for you to use.
+
+###API
+
+**StagedJob** - A job that is a part of a Stage
+
+The Staged Job is the job which has been enqueued and using
+the `perform_job` will be available to you within your `perform` method.
+
+```Ruby
+job = perform_job(*args)
+
+job.args               # The original args to the job
+job.blank?             # true if there is no job or stage/grouping etc.
+job.staged_group_stage # The stage that the job is a part of.
+job.status             # The status of the job
+                       # Valid statuses:
+                       #    :pending
+                       #    :queued
+                       #    :running
+                       #    :pending_re_run
+                       #    :failed
+                       #    :successful
+job.status_message     # A message that you can set on the job
+job.class_name         # The name of the jobs class
+job.queue_time         # The time that the job was initially enqueued
+job.delete             # Delete the job.  It may remain in the Resque queue
+job.enqueue_job        # Enque the job (Will enqueue from the delay queue if retrying)
+job.completed?         # True if the job is :failed or :successful
+job.queued?            # True if the job has been queued to Resque
+job.pending?           # True if the job is :pending or :pending_re_run
+
+# To get the group so you can see other stages: 
+job.staged_group_stage&.staged_group
+```
+
+**StagedGroupStage** - A Stage that is a part of a Group (contains Jobs)
+
+```Ruby
+job = perform_job(*args)
+stage = job.staged_group_stage
+
+## OR
+stage = group.stage(1)
+
+stage.enqueue                 # Add a job to a stage
+stage.enqueue_to              #   If a stage is running, then
+stage.enqueue_at              #   the job is enqueued immediately
+stage.enqueue_at_with_queue   #   using the enqueue method specified
+stage.enqueue_in              #   otherwise it is simply enqueued
+stage.enqueue_in_with_queue   #   when the stage is initiated
+stage.status                  # The status of the stage
+                              #   Valid statuses:
+                              #     :pending
+                              #     :running
+                              #     :complete
+stage.number                  # The stages number
+stage.staged_group            # The group the stage belongs to
+stage.jobs                    # The jobs for the stage in the order they
+                              #   were enqueued
+stage.num_jobs                # The number of jobs in the stage
+stage.delete                  # Delete the stage
+stage.initiate                # Enqueue all jobs in the stage.
+                              #   Once all jobs compelte, the next stage
+                              #   will be initiated.
+stage.blank?                  # Returns true if the stage does not really exist.
+```
+
+
+**StagedGroup** - A group of stages
+
+```Ruby
+Resque::Plugins::Stages::StagedGroup.within_a_grouping("description") do |group|
+  group.initiate      # Find the next stage that is not complete and initiate it
+  group.description   # The description that was provided for the group
+  group.created_at    # The date/time that the group was created.
+  group.current_stage # The first non-complete stage.
+  group.stage(number) # The indicated stage.  Will create a stage if none found.
+  group.stages        # A hash of all of the stages.  The stage numbers
+                      #   will be the key values.
+  group.delete        # Delete the group and all stages and jobs.
+  group.blank?        # Returns true if the group is not saved
+end
+```
+
+**Cleaner** - A cleaner utility class for fixing up mixed up jobs
+
+```Ruby
+Resque::Plugins::Stages::Cleaner.purge_all    # delete all values from Redis
+Resque::Plugins::Stages::Cleaner.cleanup_jobs # Create any stages or groups
+                                              #  needed for orphaned jobs.
+```
+
+## Screenshots
+
+### Groups
+![Pending Job Details](https://raw.githubusercontent.com/RealNobody/resque-stages/master/read_me/groups_list.png)
+
+### Stages
+![Pending Job Details](https://raw.githubusercontent.com/RealNobody/resque-stages/master/read_me/stages.png)
+
+### Jobs
+![Pending Job Details](https://raw.githubusercontent.com/RealNobody/resque-stages/master/read_me/stage.png)
+
+### Job
+![Pending Job Details](https://raw.githubusercontent.com/RealNobody/resque-stages/master/read_me/job.png)
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/resque-stages. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/[USERNAME]/resque-stages/blob/master/CODE_OF_CONDUCT.md).
+
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+
+## Code of Conduct
+
+Everyone interacting in the Resque::Stages project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/resque-stages/blob/master/CODE_OF_CONDUCT.md).

data/Rakefile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec

data/bin/console

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+#!/usr/bin/env ruby
+
+require "bundler/setup"
+require "resque/resque-stages"
+
+# You can add fixtures and/or initialization code here to make experimenting
+# with your gem easier. You can also use a different console, if you like.
+
+# (If you use this, don't forget to add pry to your Gemfile!)
+# require "pry"
+# Pry.start
+
+require "irb"
+IRB.start(__FILE__)

data/bin/setup

@@ -0,0 +1,8 @@
+#!/usr/bin/env bash
+set -euo pipefail
+IFS=$'\n\t'
+set -vx
+
+bundle install
+
+# Do any other automated setup that you need to do here

data/lib/resque-stages.rb

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+require "resque"
+require File.expand_path(File.join("resque", "plugins", "stages", "redis_access"), File.dirname(__FILE__))
+require File.expand_path(File.join("resque", "plugins", "stages", "staged_group_list"), File.dirname(__FILE__))
+require File.expand_path(File.join("resque", "plugins", "stages", "staged_group"), File.dirname(__FILE__))
+require File.expand_path(File.join("resque", "plugins", "stages", "staged_group_stage"), File.dirname(__FILE__))
+require File.expand_path(File.join("resque", "plugins", "stages", "staged_job"), File.dirname(__FILE__))
+require File.expand_path(File.join("resque", "plugins", "stages", "cleaner"), File.dirname(__FILE__))
+
+require File.expand_path(File.join("resque", "plugins", "stages"), File.dirname(__FILE__))

data/lib/resque/plugins/stages.rb

@@ -0,0 +1,110 @@
+# frozen_string_literal: true
+
+module Resque
+  module Plugins
+    # This module is added to any job class which needs to work within a stage.
+    #
+    # If the job is going to be retryable, this module needs to be included after
+    # the retry module is extended so that we know that the class is retryable.
+    module Stages
+      extend ActiveSupport::Concern
+
+      class Error < StandardError; end
+
+      included do
+        if singleton_class.included_modules.map(&:name).include?("Resque::Plugins::Retry")
+          try_again_callback :stages_report_try_again
+          give_up_callback :stages_report_giving_up
+        else
+          add_record_failure
+        end
+      end
+
+      # rubocop:disable Metrics/BlockLength
+      class_methods do
+        def perform_job(*args)
+          job = perform_job_from_param(args)
+
+          if job.nil?
+            job            = Resque::Plugins::Stages::StagedJob.new(SecureRandom.uuid)
+            job.class_name = name
+            job.args       = args
+          end
+
+          job
+        end
+
+        # rubocop:disable Metrics/AbcSize
+        # rubocop:disable Metrics/CyclomaticComplexity
+        def perform_job_from_param(args)
+          return if args.blank? || !args.first.is_a?(Hash)
+
+          hash            = args.first.with_indifferent_access
+          job             = Resque::Plugins::Stages::StagedJob.new(hash[:staged_job_id]) if hash.key?(:staged_job_id)
+          job&.class_name = name
+          job.args        = (hash.key?(:resque_compressed) ? args : args[1..]) if !job.nil? && job.blank?
+
+          job
+        end
+
+        # rubocop:enable Metrics/CyclomaticComplexity
+        # rubocop:enable Metrics/AbcSize
+
+        def before_perform_stages_successful(*args)
+          job = perform_job(*args)
+
+          return if job.blank?
+
+          job.status = :running
+        end
+
+        def after_perform_stages_successful(*args)
+          job = perform_job(*args)
+
+          return if job.blank?
+          return if job.status == :failed && Resque.inline?
+
+          job.status = :successful
+        end
+
+        def around_perform_stages_inline_around(*args)
+          yield
+        rescue StandardError => e
+          raise e unless Resque.inline?
+
+          job = perform_job(*args)
+          return if job.blank?
+
+          job.status = :failed
+        end
+
+        def stages_report_try_again(_exception, *args)
+          job = perform_job(*args)
+
+          return if job.blank?
+
+          job.status = :pending_re_run
+        end
+
+        def stages_report_giving_up(_exception, *args)
+          job = perform_job(*args)
+
+          return if job.blank?
+
+          job.status = :failed
+        end
+
+        def add_record_failure
+          define_singleton_method(:on_failure_stages_failed) do |_error, *args|
+            job = perform_job(*args)
+
+            return if job.blank?
+
+            job.status = :failed
+          end
+        end
+      end
+      # rubocop:enable Metrics/BlockLength
+    end
+  end
+end

data/lib/resque/plugins/stages/cleaner.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+module Resque
+  module Plugins
+    module Stages
+      # A class for cleaning up stranded objects for the Stages plugin
+      class Cleaner
+        include Resque::Plugins::Stages::RedisAccess
+
+        class << self
+          def redis
+            @redis ||= Resque::Plugins::Stages::Cleaner.new.redis
+          end
+
+          def purge_all
+            keys = redis.keys("*")
+
+            return if keys.blank?
+
+            redis.del(*keys)
+          end
+
+          def cleanup_jobs
+            jobs = redis.keys("StagedJob::*")
+
+            jobs.each do |job_key|
+              job = Resque::Plugins::Stages::StagedJob.new(job_key[11..])
+
+              job.verify
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/resque/plugins/stages/redis_access.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+module Resque
+  module Plugins
+    module Stages
+      # A module to add a `redis` method for a class in this gem that needs redis to get a reids object that is namespaced.
+      module RedisAccess
+        NAME_SPACE = "Resque::Plugins::Stages::"
+
+        def redis
+          @redis ||= Redis::Namespace.new(Resque::Plugins::Stages::RedisAccess::NAME_SPACE, redis: Resque.redis)
+        end
+      end
+    end
+  end
+end