rocketjob 6.3.2 → 7.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: d48a3c90185bfb29f8d18060f54cd0f707eec8679d3ece753600034d93d7e16a
-  data.tar.gz: 1fb550f51f4a1c2790bd0728bf06baea2b6750f42e118b9fe498255341d9baaa
+  metadata.gz: 7cf0e18a83d97aadac0d2741a399bcb55fa33884ccf2660836a5a3cfd946f772
+  data.tar.gz: d95563d0c77495dcdb3498dccf7cfc22fefee78bd22df244b823fc3939ef5373
 SHA512:
-  metadata.gz: 7f89a635e43ee05ae8d980b8db70052c7783477e6451cc37d71be4952ef00711dbc15a550e75947d8197b12108ea8122a216bff0ca7bb3e141f4b9a1ee0a2598
-  data.tar.gz: b51fa5e89696defe6326c85c32e00661b68b6146ed02f2222656fddfef2317468d303f36914649389fff477597bfda8747f09d8a22881d870603c9acef8e3d90
+  metadata.gz: 8f379b7890ddce715719616c311b501a916bc081bfd8eaba5055fa124badd58084985d9f38da0c081a8b5bcff3c67108ba3ea5ca52d8f2a8d7a92aa7297c94e8
+  data.tar.gz: db4e047d9c07faed020db569f7268fc0dea981ede5792ec40b7636718b118514fc6edc8d08b961d3dfe1d4ecadada40f548dde490ed4ede5326bbcb12ef5bedd

data/README.md

@@ -1,344 +1,66 @@
 # Rocket Job
-[![Gem Version](https://img.shields.io/gem/v/rocketjob.svg)](https://rubygems.org/gems/rocketjob) [![Downloads](https://img.shields.io/gem/dt/rocketjob.svg)](https://rubygems.org/gems/rocketjob) [![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](http://opensource.org/licenses/Apache-2.0) ![](https://img.shields.io/badge/status-Production%20Ready-blue.svg) 
+[![Gem Version](https://img.shields.io/gem/v/rocketjob.svg)](https://rubygems.org/gems/rocketjob) [![Downloads](https://img.shields.io/gem/dt/rocketjob.svg)](https://rubygems.org/gems/rocketjob) [![License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](http://opensource.org/licenses/Apache-2.0) ![](https://img.shields.io/badge/status-Production%20Ready-blue.svg)
 
-Ruby's missing batch system
+**Process millions of records across thousands of workers.**
 
-Checkout https://rocketjob.io/
+Rocket Job is a distributed, priority-based, MongoDB-backed batch processing system for Ruby. Run
+conventional background jobs, or split a single job's input into slices and process it concurrently
+across thousands of workers, spilling from memory to disk so very large files never fall over.
 
-![Rocket Job](https://rocketjob.io/images/rocket/rocket-icon-512x512.png)
+Full documentation is at **[rocketjob.reidmorrison.com](https://rocketjob.reidmorrison.com/)**.
+
+![Rocket Job](https://rocketjob.reidmorrison.com/images/rocket/rocket-icon-512x512.png)
 
 ## Documentation
 
-* [Guide](http://rocketjob.io/)
-* [API Reference](http://www.rubydoc.info/gems/rocketjob/)
+* [Introduction](https://rocketjob.reidmorrison.com/) — what Rocket Job is and why it exists
+* [Installation](https://rocketjob.reidmorrison.com/installation.html)
+* [Programmer's Guide](https://rocketjob.reidmorrison.com/guide.html) — simple jobs
+* [Batch Guide](https://rocketjob.reidmorrison.com/batch.html) — parallel batch jobs
+* [Included Jobs](https://rocketjob.reidmorrison.com/jobs.html) and [Directory Monitor](https://rocketjob.reidmorrison.com/dirmon.html)
+* [Events](https://rocketjob.reidmorrison.com/events.html)
+* [Web UI (Mission Control)](https://rocketjob.reidmorrison.com/mission_control.html)
+* [Deployment](https://rocketjob.reidmorrison.com/deployment.html)
+* [Architecture and Internals](https://rocketjob.reidmorrison.com/architecture.html)
+* [API Reference](https://www.rubydoc.info/gems/rocketjob/)
 
 ## Support
 
 * Ask questions in [Rocket Job Discussions](https://github.com/reidmorrison/rocketjob/discussions)
-* [Report bugs](https://github.com/rocketjob/rocketjob/issues)
-
-## Rocket Job v6
-
-- Support for Ruby v3 and Rails 6.
-- Major enhancements in Batch job support:
-    - Direct built-in Tabular support for all input and output categories.
-    - Multiple output file support, each with its own settings for:
-        - Compression
-            - GZip, Zip, BZip2 (Chunked for much faster loading into Apache Spark).
-        - Encryption
-            - PGP, Symmetric Encryption.
-        - File format
-            - CSV, PSV, JSON, Fixed Format, xlsx.
-- Significant error handling improvements, especially around throttle failures
-  that used to result in "hanging" jobs.
-- Support AWS DocumentDB in addition to MongoDB as the data store.
-- Removed use of Symbols to meet Symbol deprecation in MongoDB and Mongoid.
-
-### Upgrading to Rocket Job v6
-
-The following plugins have been deprecated and are no longer loaded by default.
-- `RocketJob::Batch::Tabular::Input`
-- `RocketJob::Batch::Tabular::Output`
-
-If your code relies on these plugins and you still want to upgrade to Rocket Job v6,
-add the following require statement to any jobs that still use them:
-
-~~~ruby
-require "rocket_job/batch/tabular"
-~~~
-
-It is important to migrate away from these plugins, since they will be removed in a future release.
-
-#### Scheduled Jobs
-
-For any scheduled jobs that include the `RocketJob::Plugins::Cron` plugin, the default behavior has changed
-so that the scheduled job instance is created immediately after the currently scheduled instance starts.
-
-To maintain the old behavior of creating the job when it fails, aborts, or completes, add the following line
-to each of the applicable jobs:
-
-~~~ruby
-self.cron_after_start = false
-~~~
-
-Additionally, scheduled jobs will now prevent a new one from being created when another scheduled instance 
-of the same job is already queued, or running with the _same_ `cron_schedule`.
-
-To maintain the old behavior of allowing multiple instances with the same cron schedule, add the following
-line to each of the applicable jobs:
-
-~~~ruby
-self.cron_singleton = false
-~~~
-
-##### Singleton
-
-Since Scheduled jobs now implement their own singleton logic, remove the singleton plugin from any scheduled jobs.
-
-#### Upgrading Batch Jobs to Rocket Job v6
-
-Rocket Job v6 replaces the array of symbol type for `input_categories` and `output_categories`
-with an array of `RocketJob::Category::Input` and `RocketJob::Category::Output`.
-
-Jobs that added or modified the input or output categories need to be upgraded. For example:
-~~~ruby
-class MyJob < RocketJob::Job
-  include RocketJob::Batch
-  
-  self.output_categories = [:main, :errors, :ignored]
-end
-~~~
+* [Report bugs](https://github.com/reidmorrison/rocketjob/issues)
 
-Needs to be changed to:
-~~~ruby
-class MyJob < RocketJob::Job
-  include RocketJob::Batch
-  
-  output_category name: :main
-  output_category name: :errors
-  output_category name: :ignored
-end
-~~~
-
-##### slice_size, encrypt, compress
-
-These fields have been removed from the job itself:
-~~~ruby
-class MyJob < RocketJob::Job
-  include RocketJob::Batch
-
-  self.slice_size = 1_000
-  self.encrypt    = true
-  self.compress   = true
-end
-~~~
-
-They are now specified on the `input_category` as follows:
-- `slice_size` just moves under `input_category`. 
-- `encrypt` becomes an option to `serializer`.
-- `compress` is now the default for all batch jobs so is not needed.
-
-If the serializer is set to `encrypt` then it is automatically compressed.
-
-~~~ruby
-class MyJob < RocketJob::Job
-  include RocketJob::Batch
-
-  input_category slice_size: 1_000, serializer: :encrypt
-end
-~~~
-
-##### collect_output, collect_nil_output
-
-The following fields have been moved from the job itself:
-~~~ruby
-class MyJob < RocketJob::Job
-  include RocketJob::Batch
-  
-  self.collect_output     = true
-  self.collect_nil_output = true
-end
-~~~
-
-Into the corresponding `output_category`:
-- `collect_output` no longer has any meaning. Output is collected anytime an `output_category` is defined.
-- `collect_nil_output` is now the option `nils` on the `output_category. 
-  It defaults to `false` so that by default any `nil` output from the `perform` method is not collected. 
-~~~ruby
-class MyJob < RocketJob::Job
-  include RocketJob::Batch
-
-  output_category nils: true
-end
-~~~
-
-##### name
+## Upgrading
 
-For both `input_category` and `output_category`, when the `name` argument is not supplied
-it defaults to `:main`.
+See the [Upgrading guide](https://rocketjob.reidmorrison.com/upgrading.html) for the code and data changes needed
+between major versions. Per-release notes are in the
+[GitHub Releases](https://github.com/reidmorrison/rocketjob/releases).
 
-For Example:
-~~~ruby
-class MyJob < RocketJob::Job
-  include RocketJob::Batch
+## Contributing
 
-  input_category name: :main, serializer: :encrypt
-  output_category name: :main
-end
-~~~
+Contributions are welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the development setup and how
+to run the test suite, and the [Architecture and Internals](https://rocketjob.reidmorrison.com/architecture.html)
+page for how Rocket Job is put together.
 
-Is the same as:
-~~~ruby
-class MyJob < RocketJob::Job
-  include RocketJob::Batch
+The documentation site lives in [`docs/`](docs/) as Jekyll markdown. To preview changes locally:
 
-  input_category serializer: :encrypt
-  output_category
-end
+~~~bash
+cd docs
+bundle update
+jekyll serve
 ~~~
 
-##### Existing and inflight jobs
-
-When migrating to Rocket Job 6, it is recommended to load every job and then save it back again as part of the 
-deployment. When the job loads it will automatically convert itself from the old schema to the new v6 schema.
-
-In flight jobs should not be affected, other than it is important to shutdown all running batch 
-servers _before_ running any new instances.
-
-## Rocket Job v4
-
-Rocket Job Pro is now fully open source and included in Rocket Job under the Apache License. 
-
-The `RocketJob::Batch` plugin now adds batch processing capabilities to break up a single task into many
-concurrent workers processing slices of the entire job at the same time. 
-
+Then open [http://127.0.0.1:4000](http://127.0.0.1:4000) and edit the files under `docs/`.
 
-Example:
-
-```ruby
-class MyJob < RocketJob::Job
-  include RocketJob::Batch
-  
-  self.description         = "Reverse names"
-  self.destroy_on_complete = false
-  
-  # Collect the output for this job in the default output category: `:main`
-  output_category
-
-  # Method to call by all available workers at the same time.
-  # Reverse the characters for each line: 
-  def perform(line)
-    line.reverse
-  end
-end
-```
-
-Upload a file for processing, for example `names.csv` which could contain:
-
-```
-jack
-jane
-bill
-john
-blake
-chris
-dave
-marc
-```
-
-To queue the above job for processing:
-
-```ruby
-job = MyJob.new
-job.upload('names.csv')
-job.save!
-```
-
-Once the job has completed, download the results into a file:
-
-```ruby
-job.download('names_reversed.csv')
-```
-
-## Contributing to the documentation
-
-To contribute to the documentation it is as easy as forking the repository
-and then editing the markdown pages directly via the github web interface.
-
-For more complex documentation changes checkout the source code locally.
-
-#### Local checkout
-
-* Fork the repository in github.
-* Checkout your fork of the source code locally.
-* Install Jekyll
-~~~
-    cd docs
-    bundle update
-~~~
-* Run Jekyll web server:
-~~~
-    jekyll serve
-~~~
-* Open a web browser to view the local documentation:
-    [http://127.0.0.1:4000](http://127.0.0.1:4000)
-* Edit the files in the `/docs` folder.
-* Refresh the page to see the changes.
-
-Once the changes are complete, submit a github pull request.
-
-## Upgrading to V3
-
-V3 replaces MongoMapper with Mongoid which supports the latest MongoDB Ruby client driver.
-
-### Upgrading Mongo Config file
-Replace `mongo.yml` with `mongoid.yml`.
-
-Start with the sample [mongoid.yml](https://github.com/rocketjob/rocketjob/blob/feature/mongoid/test/config/mongoid.yml).
- 
-For more information on the new [Mongoid config file](https://docs.mongodb.com/ruby-driver/master/tutorials/5.1.0/mongoid-installation/).
-
-Note: The `rocketjob` and `rocketjob_slices` clients in the above `mongoid.yml` file are required.
-
-### Other changes
-
-* Arguments are no longer supported, use fields for defining all named arguments for a job.
-
-* Replace usages of `rocket_job do` to set default values:
-
-~~~ruby
-  rocket_job do |job|
-    job.priority = 25
-  end
-~~~
-
-With:
-
-~~~ruby
-  self.priority = 25
-~~~
-
-* Replace `key` with `field` when adding attributes to a job:
-
-~~~ruby
-  key :inquiry_defaults, Hash
-~~~
-
-With:
-
-~~~ruby
-  field :inquiry_defaults, type: Hash, default: {}
-~~~
-
-* Replace usage of `public_rocket_job_properties` with the `user_editable` option:
-
-~~~ruby
-field :priority, type: Integer, default: 50, user_editable: true
-~~~
-
-## Ruby Support
-
-Rocket Job is tested and supported on the following Ruby platforms:
-- Ruby 2.1, 2.2, 2.3, 2.4, and above
-- JRuby 9.0.5 and above
-
-## Dependencies
+## Versioning
 
-* [MongoDB](https://www.mongodb.org)
-    * Persists job information.
-    * Version 2.7 or greater.
-* [Semantic Logger](https://rocketjob.github.io/semantic_logger)
-    * Highly concurrent scalable logging.
+This project uses [Semantic Versioning](https://semver.org/).
 
-## Versioning
+## License
 
-This project uses [Semantic Versioning](http://semver.org/).
+Apache License v2.0. See [LICENSE.txt](LICENSE.txt).
 
 ## Author
 
 [Reid Morrison](https://github.com/reidmorrison)
 
-## Contributors
-
-[Contributors](https://github.com/rocketjob/rocketjob/graphs/contributors)
-
+[Contributors](https://github.com/reidmorrison/rocketjob/graphs/contributors)

data/bin/rocketjob_batch_perf

@@ -1,5 +1,5 @@
 #!/usr/bin/env ruby
-require "rocketjob_batch"
+require "rocketjob"
 
 # Log to console
 SemanticLogger.add_appender(io: $stdout, formatter: :color)

data/lib/rocket_job/batch/categories.rb

@@ -51,12 +51,12 @@ module RocketJob
         # Builds this job instance from the supplied properties hash that may contain input and output categories.
         # Keeps the defaults and merges in settings without replacing existing categories.
         def from_properties(properties)
-          return super(properties) unless properties.key?("input_categories") || properties.key?("output_categories")
+          return super unless properties.key?("input_categories") || properties.key?("output_categories")
 
           properties        = properties.dup
           input_categories  = properties.delete("input_categories")
           output_categories = properties.delete("output_categories")
-          job               = super(properties)
+          job               = super
           job.merge_input_categories(input_categories)
           job.merge_output_categories(output_categories)
           job
@@ -126,7 +126,7 @@ module RocketJob
         categories.each do |properties|
           category_name = (properties["name"] || properties[:name] || :main).to_sym
           category      = input_category(category_name)
-          properties.each { |key, value| category.public_send("#{key}=".to_sym, value) }
+          properties.each { |key, value| category.public_send(:"#{key}=", value) }
         end
       end
 
@@ -136,7 +136,7 @@ module RocketJob
         categories.each do |properties|
           category_name = (properties["name"] || properties[:name] || :main).to_sym
           category      = output_category(category_name)
-          properties.each { |key, value| category.public_send("#{key}=".to_sym, value) }
+          properties.each { |key, value| category.public_send(:"#{key}=", value) }
         end
       end
 
@@ -314,27 +314,27 @@ module RocketJob
 
         existing                 = self[:output_categories]
         self[:output_categories] = []
-        if collect_output
-          if existing.blank?
-            self[:output_categories] = [
-              RocketJob::Category::Output.new(
-                nils:           collect_nil_output,
-                format:         main_output_format,
-                columns:        main_output_columns,
-                format_options: main_output_options
-              ).as_document
-            ]
-          elsif existing.first.is_a?(Symbol)
-            self[:output_categories] = existing.collect do |category_name|
-              RocketJob::Category::Output.new(
-                name:           category_name,
-                serializer:     serializer,
-                nils:           collect_nil_output,
-                format:         [:main, "main"].include?(category_name) ? main_output_format : nil,
-                columns:        [:main, "main"].include?(category_name) ? main_output_columns : nil,
-                format_options: [:main, "main"].include?(category_name) ? main_output_options : nil
-              ).as_document
-            end
+        return unless collect_output
+
+        if existing.blank?
+          self[:output_categories] = [
+            RocketJob::Category::Output.new(
+              nils:           collect_nil_output,
+              format:         main_output_format,
+              columns:        main_output_columns,
+              format_options: main_output_options
+            ).as_document
+          ]
+        elsif existing.first.is_a?(Symbol)
+          self[:output_categories] = existing.collect do |category_name|
+            RocketJob::Category::Output.new(
+              name:           category_name,
+              serializer:     serializer,
+              nils:           collect_nil_output,
+              format:         [:main, "main"].include?(category_name) ? main_output_format : nil,
+              columns:        [:main, "main"].include?(category_name) ? main_output_columns : nil,
+              format_options: [:main, "main"].include?(category_name) ? main_output_options : nil
+            ).as_document
           end
         end
       end

data/lib/rocket_job/batch/io.rb

@@ -269,13 +269,15 @@ module RocketJob
       # * If an exception is raised while uploading data, the input collection is cleared out
       #   so that if a job is retried during an upload failure, data is not duplicated.
 
-      def upload(object = nil, category: :main, file_name: nil, stream_mode: nil, on_first: nil, columns: nil, slice_batch_size: nil, **args, &block)
+      def upload(object = nil, category: :main, file_name: nil, stream_mode: nil, on_first: nil, columns: nil,
+                 slice_batch_size: nil, **args, &block)
         input_collection = input(category)
 
         if block
           raise(ArgumentError, "Cannot supply both an object to upload, and a block.") if object
-          if stream_mode || columns || slice_batch_size || args.size > 0
-            raise(ArgumentError, "Unknown keyword arguments when uploading a block. Only accepts :category, :file_name, or :on_first")
+          if stream_mode || columns || slice_batch_size || args.size.positive?
+            raise(ArgumentError,
+                  "Unknown keyword arguments when uploading a block. Only accepts :category, :file_name, or :on_first")
           end
 
           category           = input_category(category)
@@ -292,8 +294,9 @@ module RocketJob
         count =
           case object
           when Range
-            if file_name || stream_mode || on_first || args.size > 0
-              raise(ArgumentError, "Unknown keyword arguments when uploading a Range. Only accepts :category, :columns, or :slice_batch_size")
+            if file_name || stream_mode || on_first || args.size.positive?
+              raise(ArgumentError,
+                    "Unknown keyword arguments when uploading a Range. Only accepts :category, :columns, or :slice_batch_size")
             end
 
             first = object.first
@@ -304,14 +307,16 @@ module RocketJob
               input_collection.upload_integer_range_in_reverse_order(last, first, slice_batch_size: slice_batch_size || 1_000)
             end
           when Mongoid::Criteria
-            if file_name || stream_mode || on_first || args.size > 0
-              raise(ArgumentError, "Unknown keyword arguments when uploading a Mongoid::Criteria. Only accepts :category, :columns, or :slice_batch_size")
+            if file_name || stream_mode || on_first || args.size.positive?
+              raise(ArgumentError,
+                    "Unknown keyword arguments when uploading a Mongoid::Criteria. Only accepts :category, :columns, or :slice_batch_size")
             end
 
             input_collection.upload_mongo_query(object, columns: columns, slice_batch_size: slice_batch_size, &block)
           when defined?(ActiveRecord::Relation) ? ActiveRecord::Relation : false
-            if file_name || stream_mode || on_first || args.size > 0
-              raise(ArgumentError, "Unknown keyword arguments when uploading an ActiveRecord::Relation. Only accepts :category, :columns, or :slice_batch_size")
+            if file_name || stream_mode || on_first || args.size.positive?
+              raise(ArgumentError,
+                    "Unknown keyword arguments when uploading an ActiveRecord::Relation. Only accepts :category, :columns, or :slice_batch_size")
             end
 
             input_collection.upload_arel(object, columns: columns, slice_batch_size: slice_batch_size, &block)
@@ -336,15 +341,15 @@ module RocketJob
       end
 
       # @deprecated
-      def upload_arel(arel, *column_names, category: :main, &block)
-        count             = input(category).upload_arel(arel, columns: column_names, &block)
+      def upload_arel(arel, *column_names, category: :main, &)
+        count             = input(category).upload_arel(arel, columns: column_names, &)
         self.record_count = (record_count || 0) + count
         count
       end
 
       # @deprecated
-      def upload_mongo_query(criteria, *column_names, category: :main, &block)
-        count             = input(category).upload_mongo_query(criteria, columns: column_names, &block)
+      def upload_mongo_query(criteria, *column_names, category: :main, &)
+        count             = input(category).upload_mongo_query(criteria, columns: column_names, &)
         self.record_count = (record_count || 0) + count
         count
       end
@@ -358,7 +363,8 @@ module RocketJob
 
       # @deprecated
       def upload_integer_range_in_reverse_order(start_id, last_id, category: :main, slice_batch_size: 1_000)
-        count             = input(category).upload_integer_range_in_reverse_order(start_id, last_id, slice_batch_size: slice_batch_size)
+        count             = input(category).upload_integer_range_in_reverse_order(start_id, last_id,
+                                                                                  slice_batch_size: slice_batch_size)
         self.record_count = (record_count || 0) + count
         count
       end
@@ -459,7 +465,7 @@ module RocketJob
           binary_header_line = output_collection.slice_class.to_binary(header_line) if header_line
 
           # Don't overwrite supplied stream options if any
-          stream = stream&.is_a?(IOStreams::Stream) ? stream.dup : IOStreams.new(category.file_name)
+          stream = stream.is_a?(IOStreams::Stream) ? stream.dup : IOStreams.new(category.file_name)
           stream.remove_from_pipeline(output_collection.slice_class.binary_format)
           stream.writer(**args) do |io|
             # TODO: Binary formats should return the record count, instead of the slice count.

data/lib/rocket_job/batch/logger.rb

@@ -29,7 +29,7 @@ module RocketJob
       #   - log_exception logs entire exception if raised
       #   - on_exception_level changes log level from info to error on exception
       #   - silence noisy jobs by raising log level
-      def rocket_job_batch_slice_logger(&block)
+      def rocket_job_batch_slice_logger(&)
         logger.measure_info(
           "Completed slice",
           metric:             "#{self.class.name}/slice",
@@ -37,7 +37,7 @@ module RocketJob
           on_exception_level: :error,
           silence:            log_level,
           payload:            {records: rocket_job_slice&.size},
-          &block
+          &
         )
       end
 

data/lib/rocket_job/batch/model.rb

@@ -66,14 +66,14 @@ module RocketJob
             percent = percent_complete
             if percent >= 5
               secs                        = seconds.to_f
-              h["est_remaining_duration"] = RocketJob.seconds_as_duration((((secs / percent) * 100) - secs))
+              h["est_remaining_duration"] = RocketJob.seconds_as_duration(((secs / percent) * 100) - secs)
             end
           end
         elsif completed?
           secs                  = seconds.to_f
           h["records_per_hour"] = ((record_count.to_f / secs) * 60 * 60).round if record_count&.positive? && (secs > 0.0)
         end
-        h.merge!(super(time_zone))
+        h.merge!(super)
         h.delete("result")
         # Worker name should be retrieved from the slices when processing
         h.delete("worker_name") if sub_state == :processing

data/lib/rocket_job/batch/performance.rb

@@ -81,7 +81,7 @@ module RocketJob
           o.on("-E", "--encrypt", "Turn on encryption") do
             self.encrypt = true
           end
-          o.on("-s", "--slice_size COUNT", "Slice size") do
+          o.on("-s", "--slice_size COUNT", "Slice size") do |arg|
             self.slice_size = arg.to_i
           end
         end

data/lib/rocket_job/batch/statistics.rb

@@ -50,9 +50,16 @@ module RocketJob
           return unless last
 
           last_target = paths.inject(in_memory) do |target, sub_key|
-            target.key?(sub_key) ? target[sub_key] : target[sub_key] = Hash.new(0)
+            # Assigning a Hash into a BSON::Document (Mongoid 9) stores a
+            # converted copy, so read the value back rather than reusing the
+            # assigned object, otherwise nested increments are lost.
+            target[sub_key] = Hash.new(0) unless target.key?(sub_key)
+            target[sub_key]
           end
-          last_target[last] += increment
+          # Mongoid 9 demongoizes Hash fields to a BSON::Document without a
+          # default, so the leaf value may be nil even when the field default
+          # is `Hash.new(0)`. Default missing leaves to zero before adding.
+          last_target[last] = (last_target[last] || 0) + increment
         end
       end
 

data/lib/rocket_job/batch/throttle.rb

@@ -45,11 +45,16 @@ module RocketJob
         #     Or, a block that will return the filter.
         #     Default: :throttle_filter_class (Throttle all jobs of this class)
         #
+        #   description: [String|Proc]
+        #     Human readable reason why the job is throttled, persisted to `throttled_by`
+        #     and shown in Mission Control. A Proc is called with the job and slice and
+        #     must return a String. Default: a humanized version of the method name.
+        #
         # Note: Throttles are executed in the order they are defined.
-        def define_batch_throttle(method_name, filter: :throttle_filter_class)
+        def define_batch_throttle(method_name, filter: :throttle_filter_class, description: nil)
           # Duplicate to prevent modifying parent class throttles
           definitions = rocket_job_batch_throttles ? rocket_job_batch_throttles.dup : ThrottleDefinitions.new
-          definitions.add(method_name, filter)
+          definitions.add(method_name, filter, description)
           self.rocket_job_batch_throttles = definitions
         end