rocketjob 5.4.1 → 6.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 97910a2bbdcc67b808677e01f9ef9650935deac9f47ee415ad7665227e06e2a3
-  data.tar.gz: 9c560b2a1deea93eb4c8e9cdf6fb28f2308e8f9a85457660c292ad38a2b76137
+  metadata.gz: e313f192b854d066258a614ceac1131851c8df94c7e08f7cea6681fff6946d69
+  data.tar.gz: 10804682bee08715671696db4610ce4f93679398398bc385e93619f5a3aca715
 SHA512:
-  metadata.gz: 768538825ba34f58d915205fb1c61066ef2d86d9c3743f487db381074e30bd75c420f99f017adbb338c0dd4f202a69cf699b0b4e2a9ebc0581423fb102c85284
-  data.tar.gz: 17d494818bf341a6b3340f4c0c5d5b77ec68c05eeb26d10eccb43c81221c14acd64ffe73ea1f610b514f233c84d5cc7cf3c2a10f4d1410b9b427308d22f5ad53
+  metadata.gz: 158675e5ddec87a8b277708887b037746e3f1573569edd5a8959eabdf5668b144553cbe6164844f2cf69bc603798b3b5b052697506dad8a29e8477afc62cc45f
+  data.tar.gz: 680efe5603de3649b7e09340a545d2e1df1e02697d34af9451869310e9f2a87bbd05423686bb7d40d60f104553f7311721870e52ac2171bb1491b3a8decaf439

data/README.md

@@ -1,5 +1,5 @@
 # Rocket Job
-[![Gem Version](https://img.shields.io/gem/v/rocketjob.svg)](https://rubygems.org/gems/rocketjob) [![Build Status](https://travis-ci.org/rocketjob/rocketjob.svg?branch=master)](https://travis-ci.org/rocketjob/rocketjob) [![Downloads](https://img.shields.io/gem/dt/rocketjob.svg)](https://rubygems.org/gems/semantic_logger) [![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](http://opensource.org/licenses/Apache-2.0) ![](https://img.shields.io/badge/status-Production%20Ready-blue.svg) [![Support](https://img.shields.io/badge/IRC%20(gitter)-Support-brightgreen.svg)](https://gitter.im/rocketjob/support)
+[![Gem Version](https://img.shields.io/gem/v/rocketjob.svg)](https://rubygems.org/gems/rocketjob) [![Downloads](https://img.shields.io/gem/dt/rocketjob.svg)](https://rubygems.org/gems/rocketjob) [![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](http://opensource.org/licenses/Apache-2.0) ![](https://img.shields.io/badge/status-Production%20Ready-blue.svg) [![Support](https://img.shields.io/badge/IRC%20(gitter)-Support-brightgreen.svg)](https://gitter.im/rocketjob/support)
 
 Ruby's missing batch system
 
@@ -17,11 +17,179 @@ Checkout https://rocketjob.io/
 * Questions? Join the chat room on Gitter for [rocketjob support](https://gitter.im/rocketjob/support)
 * [Report bugs](https://github.com/rocketjob/rocketjob/issues)
 
-## Rocket Job 4
+## Rocket Job v6
 
-Rocket Job Pro is now open sourced and included within Rocket Job. 
+- Support for Ruby v3 and Rails 6.
+- Major enhancements in Batch job support:
+    - Direct built-in Tabular support for all input and output categories.
+    - Multiple output file support, each with its own settings for:
+        - Compression
+            - GZip, Zip, BZip2 (Chunked for much faster loading into Apache Spark).
+        - Encryption
+            - PGP, Symmetric Encryption.
+        - File format
+            - CSV, PSV, JSON, Fixed Format, xlsx.
+- Significant error handling improvements, especially around throttle failures
+  that used to result in "hanging" jobs.
+- Support AWS DocumentDB in addition to MongoDB as the data store.
+- Removed use of Symbols to meet Symbol deprecation in MongoDB and Mongoid.
 
-The `RocketJob::Batch` plugin now adds batch processing capabilites to break up a single task into many
+### Upgrading to Rocket Job v6
+
+The following plugins have been deprecated and are no longer loaded by default.
+- `RocketJob::Batch::Tabular::Input`
+- `RocketJob::Batch::Tabular::Output`
+
+If your code relies on these plugins and you still want to upgrade to Rocket Job v6,
+add the following require statement to any jobs that still use them:
+
+~~~ruby
+require "rocket_job/batch/tabular"
+~~~
+
+It is important to migrate away from these plugins, since they will be removed in a future release.
+
+#### Scheduled Jobs
+
+For any scheduled jobs that include the `RocketJob::Plugins::Cron` plugin, the default behavior has changed
+so that the scheduled job instance is created immediately after the currently scheduled instance starts.
+
+To maintain the old behavior of creating the job when it fails, aborts, or completes, add the following line
+to each of the applicable jobs:
+
+~~~ruby
+self.cron_after_start = false
+~~~
+
+Additionally, scheduled jobs will now prevent a new one from being created when another scheduled instance 
+of the same job is already queued, or running with the _same_ `cron_schedule`.
+
+To maintain the old behavior of allowing multiple instances with the same cron schedule, add the following
+line to each of the applicable jobs:
+
+~~~ruby
+self.cron_singleton = false
+~~~
+
+##### Singleton
+
+Since Scheduled jobs now implement their own singleton logic, remove the singleton plugin from any scheduled jobs.
+
+#### Upgrading Batch Jobs to Rocket Job v6
+
+Rocket Job v6 replaces the array of symbol type for `input_categories` and `output_categories`
+with an array of `RocketJob::Category::Input` and `RocketJob::Category::Output`.
+
+Jobs that added or modified the input or output categories need to be upgraded. For example:
+~~~ruby
+class MyJob < RocketJob::Job
+  include RocketJob::Batch
+  
+  self.output_categories = [:main, :errors, :ignored]
+end
+~~~
+
+Needs to be changed to:
+~~~ruby
+class MyJob < RocketJob::Job
+  include RocketJob::Batch
+  
+  output_category name: :main
+  output_category name: :errors
+  output_category name: :ignored
+end
+~~~
+
+##### slice_size, encrypt, compress
+
+These fields have been removed from the job itself:
+~~~ruby
+class MyJob < RocketJob::Job
+  include RocketJob::Batch
+  
+  self.slice_sice = 1_000
+  self.encrypt    = true
+  self.compress   = true
+end
+~~~
+
+They are now specified on the `input_category` as follows:
+- `slice_size` just moves under `input_category`. 
+- `encrypt` becomes an option to `serializer`.
+- `compress` is now the default for all batch jobs so is not needed.
+
+If the serializer is set to `encrypt` then it is automatically compressed.
+
+~~~ruby
+class MyJob < RocketJob::Job
+  include RocketJob::Batch
+  
+  input_category slice_sice: 1_000, serializer: :encrypt
+end
+~~~
+
+##### collect_output, collect_nil_output
+
+The following fields have been moved from the job itself:
+~~~ruby
+class MyJob < RocketJob::Job
+  include RocketJob::Batch
+  
+  self.collect_output     = true
+  self.collect_nil_output = true
+end
+~~~
+
+Into the corresponding `output_category`:
+- `collect_output` no longer has any meaning. Output is collected anytime an `output_category` is defined.
+- `collect_nil_output` is now the option `nils` on the `output_category. 
+  It defaults to `false` so that by default any `nil` output from the `perform` method is not collected. 
+~~~ruby
+class MyJob < RocketJob::Job
+  include RocketJob::Batch
+
+  output_category nils: true
+end
+~~~
+
+##### name
+
+For both `input_category` and `output_category`, when the `name` argument is not supplied
+it defaults to `:main`.
+
+For Example:
+~~~ruby
+class MyJob < RocketJob::Job
+  include RocketJob::Batch
+
+  input_category name: :main, serializer: :encrypt
+  output_category name: :main
+end
+~~~
+
+Is the same as:
+~~~ruby
+class MyJob < RocketJob::Job
+  include RocketJob::Batch
+
+  input_category serializer: :encrypt
+  output_category
+end
+~~~
+
+##### Existing and inflight jobs
+
+When migrating to Rocket Job 6, it is recommended to load every job and then save it back again as part of the 
+deployment. When the job loads it will automatically convert itself from the old schema to the new v6 schema.
+
+In flight jobs should not be affected, other than it is important to shutdown all running batch 
+servers _before_ running any new instances.
+
+## Rocket Job v4
+
+Rocket Job Pro is now fully open source and included in Rocket Job under the Apache License. 
+
+The `RocketJob::Batch` plugin now adds batch processing capabilities to break up a single task into many
 concurrent workers processing slices of the entire job at the same time. 
 
 
@@ -33,7 +201,9 @@ class MyJob < RocketJob::Job
   
   self.description         = "Reverse names"
   self.destroy_on_complete = false
-  self.collect_output      = true
+  
+  # Collect the output for this job in the default output category: `:main`
+  output_category
 
   # Method to call by all available workers at the same time.
   # Reverse the characters for each line: 

data/bin/rocketjob_batch_perf

@@ -2,7 +2,7 @@
 require "rocketjob_batch"
 
 # Log to console
-SemanticLogger.add_appender(io: STDOUT, formatter: :color)
+SemanticLogger.add_appender(io: $stdout, formatter: :color)
 
 perf = RocketJob::Batch::Performance.new
 perf.parse(ARGV)

data/bin/rocketjob_perf

@@ -2,7 +2,7 @@
 require "rocketjob"
 
 # Log to console
-SemanticLogger.add_appender(io: STDOUT, formatter: :color)
+SemanticLogger.add_appender(io: $stdout, formatter: :color)
 
 perf = RocketJob::Performance.new
 perf.parse(ARGV)

data/lib/rocket_job/batch/categories.rb

@@ -0,0 +1,345 @@
+require "active_support/concern"
+
+module RocketJob
+  module Batch
+    module Categories
+      extend ActiveSupport::Concern
+
+      included do
+        after_initialize :rocketjob_categories_assign, if: :new_record?
+        after_initialize :rocketjob_categories_migrate, unless: :new_record?
+        before_perform :rocketjob_categories_input_render
+        after_perform :rocketjob_categories_output_render
+
+        # List of categories that this job can load input data into
+        embeds_many :input_categories, class_name: "RocketJob::Category::Input"
+
+        # List of categories that this job can save output data into
+        embeds_many :output_categories, class_name: "RocketJob::Category::Output"
+
+        # Internal attributes
+        class_attribute :defined_input_categories, instance_accessor: false, instance_predicate: false
+        class_attribute :defined_output_categories, instance_accessor: false, instance_predicate: false
+
+        # For RJMC to be able to edit jobs
+        accepts_nested_attributes_for :input_categories, :output_categories
+      end
+
+      module ClassMethods
+        # Define a new input category
+        # @see RocketJob::Category::Input
+        def input_category(**args)
+          category = RocketJob::Category::Input.new(**args)
+          if defined_input_categories.nil?
+            self.defined_input_categories = [category]
+          else
+            rocketjob_categories_set(category, defined_input_categories)
+          end
+        end
+
+        # Define a new output category
+        # @see RocketJob::Category::Output
+        def output_category(**args)
+          category = RocketJob::Category::Output.new(**args)
+          if defined_output_categories.nil?
+            self.defined_output_categories = [category]
+          else
+            rocketjob_categories_set(category, defined_output_categories)
+          end
+        end
+
+        # Builds this job instance from the supplied properties hash that may contain input and output categories.
+        # Keeps the defaults and merges in settings without replacing existing categories.
+        def from_properties(properties)
+          return super(properties) unless properties.key?("input_categories") || properties.key?("output_categories")
+
+          properties        = properties.dup
+          input_categories  = properties.delete("input_categories")
+          output_categories = properties.delete("output_categories")
+          job               = super(properties)
+          job.merge_input_categories(input_categories)
+          job.merge_output_categories(output_categories)
+          job
+        end
+
+        private
+
+        def rocketjob_categories_set(category, categories)
+          index = categories.find_index { |cat| cat.name == category.name }
+          index ? categories[index] = category : categories << category
+          category
+        end
+      end
+
+      def input_category(category_name = :main)
+        return category_name if category_name.is_a?(Category::Input)
+        raise(ArgumentError, "Cannot supply Output Category to input category") if category_name.is_a?(Category::Output)
+
+        category_name = category_name.to_sym
+        # find does not work against this association
+        input_categories.each { |category| return category if category.name == category_name }
+
+        unless category_name == :main
+          raise(
+            ArgumentError,
+            "Unknown Input Category: #{category_name.inspect}. Registered categories: #{input_categories.collect(&:name).join(',')}"
+          )
+        end
+
+        # Auto-register main input category when not defined
+        category = Category::Input.new(job: self)
+        self.input_categories << category
+        category
+      end
+
+      def output_category(category_name = :main)
+        return category_name if category_name.is_a?(Category::Output)
+        raise(ArgumentError, "Cannot supply Input Category to output category") if category_name.is_a?(Category::Input)
+
+        category_name = category_name.to_sym
+        # .find does not work against this association
+        output_categories.each { |category| return category if category.name == category_name }
+
+        raise(
+          ArgumentError,
+          "Unknown Output Category: #{category_name.inspect}. Registered categories: #{output_categories.collect(&:name).join(',')}"
+        )
+      end
+
+      # Returns [true|false] whether the named category has already been defined
+      def input_category?(category_name)
+        category_name = category_name.to_sym
+        # .find does not work against this association
+        input_categories.each { |catg| return true if catg.name == category_name }
+        false
+      end
+
+      def output_category?(category_name)
+        category_name = category_name.to_sym
+        # .find does not work against this association
+        output_categories.each { |catg| return true if catg.name == category_name }
+        false
+      end
+
+      def merge_input_categories(categories)
+        return if categories.blank?
+
+        categories.each do |properties|
+          category_name = (properties["name"] || properties[:name] || :main).to_sym
+          category      = input_category(category_name)
+          properties.each { |key, value| category.public_send("#{key}=".to_sym, value) }
+        end
+      end
+
+      def merge_output_categories(categories)
+        return if categories.blank?
+
+        categories.each do |properties|
+          category_name = (properties["name"] || properties[:name] || :main).to_sym
+          category      = output_category(category_name)
+          properties.each { |key, value| category.public_send("#{key}=".to_sym, value) }
+        end
+      end
+
+      private
+
+      def rocketjob_categories_assign
+        # Input categories defaults to :main if none was set in the class
+        if input_categories.empty?
+          self.input_categories =
+            if self.class.defined_input_categories
+              self.class.defined_input_categories.deep_dup
+            else
+              [RocketJob::Category::Input.new]
+            end
+        end
+
+        return if !self.class.defined_output_categories || !output_categories.empty?
+
+        # Input categories defaults to nil if none was set in the class
+        self.output_categories = self.class.defined_output_categories.deep_dup
+      end
+
+      # Render the output from the perform.
+      def rocketjob_categories_output_render
+        return if @rocket_job_output.nil?
+
+        # TODO: ..
+        return unless output_categories
+        return if output_categories.empty?
+
+        @rocket_job_output = rocketjob_categories_output_render_row(@rocket_job_output)
+      end
+
+      # Parse the input data before passing to the perform method
+      def rocketjob_categories_input_render
+        return if @rocket_job_input.nil?
+
+        @rocket_job_input = rocketjob_categories_input_render_row(@rocket_job_input)
+      end
+
+      def rocketjob_categories_input_render_row(row)
+        return if row.nil?
+
+        category = input_category
+        return row if category.nil? || !category.tabular?
+        return nil if row.blank?
+
+        tabular = category.tabular
+
+        # Return the row as-is if the required header has not yet been set.
+        if tabular.header?
+          raise(ArgumentError,
+                "The tabular header columns _must_ be set before attempting to parse data that requires it.")
+        end
+
+        tabular.record_parse(row)
+      end
+
+      def rocketjob_categories_output_render_row(row)
+        return if row.nil?
+
+        if row.is_a?(Batch::Result)
+          category  = output_category(row.category)
+          row.value = category.tabular.render(row.value) if category.tabular?
+          return row
+        end
+
+        if row.is_a?(Batch::Results)
+          results = Batch::Results.new
+          row.each { |result| results << rocketjob_categories_output_render_row(result) }
+          return results
+        end
+
+        category = output_category
+        return row unless category.tabular?
+        return nil if row.blank?
+
+        category.tabular.render(row)
+      end
+
+      # Migrate existing v5 batch jobs to v6
+      def rocketjob_categories_migrate
+        return unless attribute_present?(:input_categories) && self[:input_categories]&.first.is_a?(Symbol)
+
+        serializer = :none
+        if attribute_present?(:compress)
+          serializer = :compress if self[:compress]
+          remove_attribute(:compress)
+        end
+
+        if attribute_present?(:encrypt)
+          serializer = :encrypt if self[:encrypt]
+          remove_attribute(:encrypt)
+        end
+
+        slice_size = 100
+        if attribute_present?(:slice_size)
+          slice_size = self[:slice_size].to_i
+          remove_attribute(:slice_size)
+        end
+
+        main_input_format  = nil
+        main_input_mode    = :line
+        main_input_columns = nil
+        # Only migrate tabular attributes if the job also removed the tabular plugin.
+        unless respond_to?(:tabular_input_render)
+          if attribute_present?(:tabular_input_format)
+            main_input_format = self[:tabular_input_format]
+            remove_attribute(:tabular_input_format)
+          end
+
+          if attribute_present?(:tabular_input_mode)
+            main_input_mode = self[:tabular_input_mode]
+            remove_attribute(:tabular_input_mode)
+          end
+
+          if attribute_present?(:tabular_input_header)
+            main_input_columns = self[:tabular_input_header]
+            remove_attribute(:tabular_input_header)
+          end
+        end
+
+        file_name = nil
+        if attribute_present?(:upload_file_name)
+          file_name = self[:upload_file_name]
+          remove_attribute(:upload_file_name)
+        end
+
+        existing                = self[:input_categories]
+        self[:input_categories] = []
+        self[:input_categories] = existing.collect do |category_name|
+          RocketJob::Category::Input.new(
+            name:       category_name,
+            file_name:  file_name,
+            serializer: serializer,
+            slice_size: slice_size,
+            format:     [:main, "main"].include?(category_name) ? main_input_format : nil,
+            columns:    [:main, "main"].include?(category_name) ? main_input_columns : nil,
+            mode:       [:main, "main"].include?(category_name) ? main_input_mode : nil
+          ).as_document
+        end
+
+        collect_output = false
+        if attribute_present?(:collect_output)
+          collect_output = self[:collect_output]
+          remove_attribute(:collect_output)
+        end
+
+        collect_nil_output = true
+        if attribute_present?(:collect_nil_output)
+          collect_nil_output = self[:collect_nil_output]
+          remove_attribute(:collect_nil_output)
+        end
+
+        main_output_format  = nil
+        main_output_columns = nil
+        main_output_options = nil
+
+        # Only migrate tabular attributes if the job also removed the tabular plugin.
+        unless respond_to?(:tabular_output_render)
+          if attribute_present?(:tabular_output_format)
+            main_output_format = self[:tabular_output_format]
+            remove_attribute(:tabular_output_format)
+          end
+
+          if attribute_present?(:tabular_output_header)
+            main_output_columns = self[:tabular_output_header]
+            remove_attribute(:tabular_output_header)
+          end
+
+          if attribute_present?(:tabular_output_options)
+            main_output_options = self[:tabular_output_options]
+            remove_attribute(:tabular_output_options)
+          end
+        end
+
+        existing                 = self[:output_categories]
+        self[:output_categories] = []
+        if collect_output
+          if existing.blank?
+            self[:output_categories] = [
+              RocketJob::Category::Output.new(
+                nils:           collect_nil_output,
+                format:         main_output_format,
+                columns:        main_output_columns,
+                format_options: main_output_options
+              ).as_document
+            ]
+          elsif existing.first.is_a?(Symbol)
+            self[:output_categories] = existing.collect do |category_name|
+              RocketJob::Category::Output.new(
+                name:           category_name,
+                serializer:     serializer,
+                nils:           collect_nil_output,
+                format:         [:main, "main"].include?(category_name) ? main_output_format : nil,
+                columns:        [:main, "main"].include?(category_name) ? main_output_columns : nil,
+                format_options: [:main, "main"].include?(category_name) ? main_output_options : nil
+              ).as_document
+            end
+          end
+        end
+      end
+    end
+  end
+end