rocketjob 6.3.2 → 7.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d48a3c90185bfb29f8d18060f54cd0f707eec8679d3ece753600034d93d7e16a
-  data.tar.gz: 1fb550f51f4a1c2790bd0728bf06baea2b6750f42e118b9fe498255341d9baaa
+  metadata.gz: 4a97fb3e6b6ed4130f52e5796bdb0737661c20e83f158c1f00667629438ea316
+  data.tar.gz: 298727e29cd1f8d37cd4cab34c23d7a8b3fee805f0f264c784e72af873309640
 SHA512:
-  metadata.gz: 7f89a635e43ee05ae8d980b8db70052c7783477e6451cc37d71be4952ef00711dbc15a550e75947d8197b12108ea8122a216bff0ca7bb3e141f4b9a1ee0a2598
-  data.tar.gz: b51fa5e89696defe6326c85c32e00661b68b6146ed02f2222656fddfef2317468d303f36914649389fff477597bfda8747f09d8a22881d870603c9acef8e3d90
+  metadata.gz: df159598ced406d32f34802c65361c6f429075862722afb078324e265f4c71e779f12646292b1fe03d3a9e53bcd94c32f9f9e18aea3853260e21368b0e46bb5a
+  data.tar.gz: f1c0e817b687fa470f66ee3cbe7b519d52835813ad9ec99c96f092d557e1438f09358af1391b6288832df47c45249011d7d151668e782cf0207b25a431a274df

data/README.md

@@ -1,344 +1,66 @@
 # Rocket Job
-[![Gem Version](https://img.shields.io/gem/v/rocketjob.svg)](https://rubygems.org/gems/rocketjob) [![Downloads](https://img.shields.io/gem/dt/rocketjob.svg)](https://rubygems.org/gems/rocketjob) [![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](http://opensource.org/licenses/Apache-2.0) ![](https://img.shields.io/badge/status-Production%20Ready-blue.svg) 
+[![Gem Version](https://img.shields.io/gem/v/rocketjob.svg)](https://rubygems.org/gems/rocketjob) [![Downloads](https://img.shields.io/gem/dt/rocketjob.svg)](https://rubygems.org/gems/rocketjob) [![Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/reidmorrison/rocketjob/badges/coverage.json)](https://github.com/reidmorrison/rocketjob/actions/workflows/ci.yml) [![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](http://opensource.org/licenses/Apache-2.0) ![](https://img.shields.io/badge/status-Production%20Ready-blue.svg)
 
-Ruby's missing batch system
+**Process millions of records across thousands of workers.**
 
-Checkout https://rocketjob.io/
+Rocket Job is a distributed, priority-based, MongoDB-backed batch processing system for Ruby. Run
+conventional background jobs, or split a single job's input into slices and process it concurrently
+across thousands of workers, spilling from memory to disk so very large files never fall over.
 
-![Rocket Job](https://rocketjob.io/images/rocket/rocket-icon-512x512.png)
+Full documentation is at **[rocketjob.reidmorrison.com](https://rocketjob.reidmorrison.com/)**.
+
+![Rocket Job](https://rocketjob.reidmorrison.com/images/rocket/rocket-icon-512x512.png)
 
 ## Documentation
 
-* [Guide](http://rocketjob.io/)
-* [API Reference](http://www.rubydoc.info/gems/rocketjob/)
+* [Introduction](https://rocketjob.reidmorrison.com/) — what Rocket Job is and why it exists
+* [Installation](https://rocketjob.reidmorrison.com/installation.html)
+* [Programmer's Guide](https://rocketjob.reidmorrison.com/guide.html) — simple jobs
+* [Batch Guide](https://rocketjob.reidmorrison.com/batch.html) — parallel batch jobs
+* [Included Jobs](https://rocketjob.reidmorrison.com/jobs.html) and [Directory Monitor](https://rocketjob.reidmorrison.com/dirmon.html)
+* [Events](https://rocketjob.reidmorrison.com/events.html)
+* [Web UI (Mission Control)](https://rocketjob.reidmorrison.com/mission_control.html)
+* [Deployment](https://rocketjob.reidmorrison.com/deployment.html)
+* [Architecture and Internals](https://rocketjob.reidmorrison.com/architecture.html)
+* [API Reference](https://www.rubydoc.info/gems/rocketjob/)
 
 ## Support
 
 * Ask questions in [Rocket Job Discussions](https://github.com/reidmorrison/rocketjob/discussions)
-* [Report bugs](https://github.com/rocketjob/rocketjob/issues)
-
-## Rocket Job v6
-
-- Support for Ruby v3 and Rails 6.
-- Major enhancements in Batch job support:
-    - Direct built-in Tabular support for all input and output categories.
-    - Multiple output file support, each with its own settings for:
-        - Compression
-            - GZip, Zip, BZip2 (Chunked for much faster loading into Apache Spark).
-        - Encryption
-            - PGP, Symmetric Encryption.
-        - File format
-            - CSV, PSV, JSON, Fixed Format, xlsx.
-- Significant error handling improvements, especially around throttle failures
-  that used to result in "hanging" jobs.
-- Support AWS DocumentDB in addition to MongoDB as the data store.
-- Removed use of Symbols to meet Symbol deprecation in MongoDB and Mongoid.
-
-### Upgrading to Rocket Job v6
-
-The following plugins have been deprecated and are no longer loaded by default.
-- `RocketJob::Batch::Tabular::Input`
-- `RocketJob::Batch::Tabular::Output`
-
-If your code relies on these plugins and you still want to upgrade to Rocket Job v6,
-add the following require statement to any jobs that still use them:
-
-~~~ruby
-require "rocket_job/batch/tabular"
-~~~
-
-It is important to migrate away from these plugins, since they will be removed in a future release.
-
-#### Scheduled Jobs
-
-For any scheduled jobs that include the `RocketJob::Plugins::Cron` plugin, the default behavior has changed
-so that the scheduled job instance is created immediately after the currently scheduled instance starts.
-
-To maintain the old behavior of creating the job when it fails, aborts, or completes, add the following line
-to each of the applicable jobs:
-
-~~~ruby
-self.cron_after_start = false
-~~~
-
-Additionally, scheduled jobs will now prevent a new one from being created when another scheduled instance 
-of the same job is already queued, or running with the _same_ `cron_schedule`.
-
-To maintain the old behavior of allowing multiple instances with the same cron schedule, add the following
-line to each of the applicable jobs:
-
-~~~ruby
-self.cron_singleton = false
-~~~
-
-##### Singleton
-
-Since Scheduled jobs now implement their own singleton logic, remove the singleton plugin from any scheduled jobs.
-
-#### Upgrading Batch Jobs to Rocket Job v6
-
-Rocket Job v6 replaces the array of symbol type for `input_categories` and `output_categories`
-with an array of `RocketJob::Category::Input` and `RocketJob::Category::Output`.
-
-Jobs that added or modified the input or output categories need to be upgraded. For example:
-~~~ruby
-class MyJob < RocketJob::Job
-  include RocketJob::Batch
-  
-  self.output_categories = [:main, :errors, :ignored]
-end
-~~~
+* [Report bugs](https://github.com/reidmorrison/rocketjob/issues)
 
-Needs to be changed to:
-~~~ruby
-class MyJob < RocketJob::Job
-  include RocketJob::Batch
-  
-  output_category name: :main
-  output_category name: :errors
-  output_category name: :ignored
-end
-~~~
-
-##### slice_size, encrypt, compress
-
-These fields have been removed from the job itself:
-~~~ruby
-class MyJob < RocketJob::Job
-  include RocketJob::Batch
-
-  self.slice_size = 1_000
-  self.encrypt    = true
-  self.compress   = true
-end
-~~~
-
-They are now specified on the `input_category` as follows:
-- `slice_size` just moves under `input_category`. 
-- `encrypt` becomes an option to `serializer`.
-- `compress` is now the default for all batch jobs so is not needed.
-
-If the serializer is set to `encrypt` then it is automatically compressed.
-
-~~~ruby
-class MyJob < RocketJob::Job
-  include RocketJob::Batch
-
-  input_category slice_size: 1_000, serializer: :encrypt
-end
-~~~
-
-##### collect_output, collect_nil_output
-
-The following fields have been moved from the job itself:
-~~~ruby
-class MyJob < RocketJob::Job
-  include RocketJob::Batch
-  
-  self.collect_output     = true
-  self.collect_nil_output = true
-end
-~~~
-
-Into the corresponding `output_category`:
-- `collect_output` no longer has any meaning. Output is collected anytime an `output_category` is defined.
-- `collect_nil_output` is now the option `nils` on the `output_category. 
-  It defaults to `false` so that by default any `nil` output from the `perform` method is not collected. 
-~~~ruby
-class MyJob < RocketJob::Job
-  include RocketJob::Batch
-
-  output_category nils: true
-end
-~~~
-
-##### name
+## Upgrading
 
-For both `input_category` and `output_category`, when the `name` argument is not supplied
-it defaults to `:main`.
+See the [Upgrading guide](https://rocketjob.reidmorrison.com/upgrading.html) for the code and data changes needed
+between major versions. Per-release notes are in the
+[GitHub Releases](https://github.com/reidmorrison/rocketjob/releases).
 
-For Example:
-~~~ruby
-class MyJob < RocketJob::Job
-  include RocketJob::Batch
+## Contributing
 
-  input_category name: :main, serializer: :encrypt
-  output_category name: :main
-end
-~~~
+Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the development setup and how
+to run the test suite, and the [Architecture and Internals](https://rocketjob.reidmorrison.com/architecture.html)
+page for how Rocket Job is put together.
 
-Is the same as:
-~~~ruby
-class MyJob < RocketJob::Job
-  include RocketJob::Batch
+The documentation site lives in [`docs/`](docs/) as Jekyll markdown. To preview changes locally:
 
-  input_category serializer: :encrypt
-  output_category
-end
+~~~bash
+cd docs
+bundle update
+jekyll serve
 ~~~
 
-##### Existing and inflight jobs
-
-When migrating to Rocket Job 6, it is recommended to load every job and then save it back again as part of the 
-deployment. When the job loads it will automatically convert itself from the old schema to the new v6 schema.
-
-In flight jobs should not be affected, other than it is important to shutdown all running batch 
-servers _before_ running any new instances.
-
-## Rocket Job v4
-
-Rocket Job Pro is now fully open source and included in Rocket Job under the Apache License. 
-
-The `RocketJob::Batch` plugin now adds batch processing capabilities to break up a single task into many
-concurrent workers processing slices of the entire job at the same time. 
-
+Then open [http://127.0.0.1:4000](http://127.0.0.1:4000) and edit the files under `docs/`.
 
-Example:
-
-```ruby
-class MyJob < RocketJob::Job
-  include RocketJob::Batch
-  
-  self.description         = "Reverse names"
-  self.destroy_on_complete = false
-  
-  # Collect the output for this job in the default output category: `:main`
-  output_category
-
-  # Method to call by all available workers at the same time.
-  # Reverse the characters for each line: 
-  def perform(line)
-    line.reverse
-  end
-end
-```
-
-Upload a file for processing, for example `names.csv` which could contain:
-
-```
-jack
-jane
-bill
-john
-blake
-chris
-dave
-marc
-```
-
-To queue the above job for processing:
-
-```ruby
-job = MyJob.new
-job.upload('names.csv')
-job.save!
-```
-
-Once the job has completed, download the results into a file:
-
-```ruby
-job.download('names_reversed.csv')
-```
-
-## Contributing to the documentation
-
-To contribute to the documentation it is as easy as forking the repository
-and then editing the markdown pages directly via the github web interface.
-
-For more complex documentation changes checkout the source code locally.
-
-#### Local checkout
-
-* Fork the repository in github.
-* Checkout your fork of the source code locally.
-* Install Jekyll
-~~~
-    cd docs
-    bundle update
-~~~
-* Run Jekyll web server:
-~~~
-    jekyll serve
-~~~
-* Open a web browser to view the local documentation:
-    [http://127.0.0.1:4000](http://127.0.0.1:4000)
-* Edit the files in the `/docs` folder.
-* Refresh the page to see the changes.
-
-Once the changes are complete, submit a github pull request.
-
-## Upgrading to V3
-
-V3 replaces MongoMapper with Mongoid which supports the latest MongoDB Ruby client driver.
-
-### Upgrading Mongo Config file
-Replace `mongo.yml` with `mongoid.yml`.
-
-Start with the sample [mongoid.yml](https://github.com/rocketjob/rocketjob/blob/feature/mongoid/test/config/mongoid.yml).
- 
-For more information on the new [Mongoid config file](https://docs.mongodb.com/ruby-driver/master/tutorials/5.1.0/mongoid-installation/).
-
-Note: The `rocketjob` and `rocketjob_slices` clients in the above `mongoid.yml` file are required.
-
-### Other changes
-
-* Arguments are no longer supported, use fields for defining all named arguments for a job.
-
-* Replace usages of `rocket_job do` to set default values:
-
-~~~ruby
-  rocket_job do |job|
-    job.priority = 25
-  end
-~~~
-
-With:
-
-~~~ruby
-  self.priority = 25
-~~~
-
-* Replace `key` with `field` when adding attributes to a job:
-
-~~~ruby
-  key :inquiry_defaults, Hash
-~~~
-
-With:
-
-~~~ruby
-  field :inquiry_defaults, type: Hash, default: {}
-~~~
-
-* Replace usage of `public_rocket_job_properties` with the `user_editable` option:
-
-~~~ruby
-field :priority, type: Integer, default: 50, user_editable: true
-~~~
-
-## Ruby Support
-
-Rocket Job is tested and supported on the following Ruby platforms:
-- Ruby 2.1, 2.2, 2.3, 2.4, and above
-- JRuby 9.0.5 and above
-
-## Dependencies
+## Versioning
 
-* [MongoDB](https://www.mongodb.org)
-    * Persists job information.
-    * Version 2.7 or greater.
-* [Semantic Logger](https://rocketjob.github.io/semantic_logger)
-    * Highly concurrent scalable logging.
+This project uses [Semantic Versioning](https://semver.org/).
 
-## Versioning
+## License
 
-This project uses [Semantic Versioning](http://semver.org/).
+Apache License v2.0. See [LICENSE.txt](LICENSE.txt).
 
 ## Author
 
 [Reid Morrison](https://github.com/reidmorrison)
 
-## Contributors
-
-[Contributors](https://github.com/rocketjob/rocketjob/graphs/contributors)
-
+[Contributors](https://github.com/reidmorrison/rocketjob/graphs/contributors)

data/bin/rocketjob_batch_perf

@@ -1,5 +1,5 @@
 #!/usr/bin/env ruby
-require "rocketjob_batch"
+require "rocketjob"
 
 # Log to console
 SemanticLogger.add_appender(io: $stdout, formatter: :color)

data/lib/rocket_job/batch/categories.rb

@@ -29,6 +29,7 @@ module RocketJob
         # Define a new input category
         # @see RocketJob::Category::Input
         def input_category(**args)
+          RocketJob::Category::Input.warn_deprecated_options(args)
           category = RocketJob::Category::Input.new(**args)
           if defined_input_categories.nil?
             self.defined_input_categories = [category]
@@ -40,6 +41,7 @@ module RocketJob
         # Define a new output category
         # @see RocketJob::Category::Output
         def output_category(**args)
+          RocketJob::Category::Output.warn_deprecated_options(args)
           category = RocketJob::Category::Output.new(**args)
           if defined_output_categories.nil?
             self.defined_output_categories = [category]
@@ -51,12 +53,12 @@ module RocketJob
         # Builds this job instance from the supplied properties hash that may contain input and output categories.
         # Keeps the defaults and merges in settings without replacing existing categories.
         def from_properties(properties)
-          return super(properties) unless properties.key?("input_categories") || properties.key?("output_categories")
+          return super unless properties.key?("input_categories") || properties.key?("output_categories")
 
           properties        = properties.dup
           input_categories  = properties.delete("input_categories")
           output_categories = properties.delete("output_categories")
-          job               = super(properties)
+          job               = super
           job.merge_input_categories(input_categories)
           job.merge_output_categories(output_categories)
           job
@@ -120,28 +122,39 @@ module RocketJob
         false
       end
 
+      # Sets the supplied properties of each input category on it, such as those that a Dirmon entry or an
+      # UploadFileJob holds, and reports a deprecation warning for a deprecated option among them, see
+      # Category::Base.warn_deprecated_options.
       def merge_input_categories(categories)
         return if categories.blank?
 
         categories.each do |properties|
           category_name = (properties["name"] || properties[:name] || :main).to_sym
           category      = input_category(category_name)
-          properties.each { |key, value| category.public_send("#{key}=".to_sym, value) }
+          rocketjob_categories_merge(category, properties)
         end
       end
 
+      # Sets the supplied properties of each output category on it, see #merge_input_categories.
       def merge_output_categories(categories)
         return if categories.blank?
 
         categories.each do |properties|
           category_name = (properties["name"] || properties[:name] || :main).to_sym
           category      = output_category(category_name)
-          properties.each { |key, value| category.public_send("#{key}=".to_sym, value) }
+          rocketjob_categories_merge(category, properties)
         end
       end
 
       private
 
+      # Sets the supplied properties on the category, after reporting a deprecation warning for a deprecated option
+      # among them, which includes the category's schema, since an input category's columns are not deprecated with one.
+      def rocketjob_categories_merge(category, properties)
+        category.class.warn_deprecated_options(category.attributes.slice("schema").merge(properties.to_h.stringify_keys))
+        properties.each { |key, value| category.public_send(:"#{key}=", value) }
+      end
+
       def rocketjob_categories_assign
         # Input categories defaults to :main if none was set in the class
         if input_categories.empty?
@@ -218,7 +231,7 @@ module RocketJob
 
       # Migrate existing v5 batch jobs to v6
       def rocketjob_categories_migrate
-        return unless attribute_present?(:input_categories) && self[:input_categories]&.first.is_a?(Symbol)
+        return unless attributes["input_categories"]&.first.is_a?(Symbol)
 
         serializer = :none
         if attribute_present?(:compress)
@@ -314,27 +327,27 @@ module RocketJob
 
         existing                 = self[:output_categories]
         self[:output_categories] = []
-        if collect_output
-          if existing.blank?
-            self[:output_categories] = [
-              RocketJob::Category::Output.new(
-                nils:           collect_nil_output,
-                format:         main_output_format,
-                columns:        main_output_columns,
-                format_options: main_output_options
-              ).as_document
-            ]
-          elsif existing.first.is_a?(Symbol)
-            self[:output_categories] = existing.collect do |category_name|
-              RocketJob::Category::Output.new(
-                name:           category_name,
-                serializer:     serializer,
-                nils:           collect_nil_output,
-                format:         [:main, "main"].include?(category_name) ? main_output_format : nil,
-                columns:        [:main, "main"].include?(category_name) ? main_output_columns : nil,
-                format_options: [:main, "main"].include?(category_name) ? main_output_options : nil
-              ).as_document
-            end
+        return unless collect_output
+
+        if existing.blank?
+          self[:output_categories] = [
+            RocketJob::Category::Output.new(
+              nils:           collect_nil_output,
+              format:         main_output_format,
+              columns:        main_output_columns,
+              format_options: main_output_options
+            ).as_document
+          ]
+        elsif existing.first.is_a?(Symbol)
+          self[:output_categories] = existing.collect do |category_name|
+            RocketJob::Category::Output.new(
+              name:           category_name,
+              serializer:     serializer,
+              nils:           collect_nil_output,
+              format:         [:main, "main"].include?(category_name) ? main_output_format : nil,
+              columns:        [:main, "main"].include?(category_name) ? main_output_columns : nil,
+              format_options: [:main, "main"].include?(category_name) ? main_output_options : nil
+            ).as_document
           end
         end
       end