burner 1.7.0 → 1.11.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 59c9f86d7e38af602d55db6f13473ef3943e7bb860bc531a49025c1cc42a4686
-  data.tar.gz: 732a8127ec38ea3e4160331fa05046d9fec8e4057a48ecefb309c12821f67254
+  metadata.gz: f687d9818e7a090c0999f144096eef52c5ba7093a069a8d2ef784f5abdb859d7
+  data.tar.gz: cae3bb23c2671ab69263fe7359c93a47777c72fc387ab93a03006c38a2bc83cd
 SHA512:
-  metadata.gz: 96c17f4c33e6aa21d9f6b5b075cee2d0fd456bef4b7b02c9401b9439925e2ae8b91c44ebae1e8bc1185d22215516b1f6d27ed6362760e14d8ddc069cceb8e22a
-  data.tar.gz: 71ddb54a2f7c286adf6c457aeffbb897989fe2d0d3a1c95cfbb0e86bb69db4dcae5bc06190485e43a6655ba664d02d2fce90977d9d391597261ccf4944986aeb
+  metadata.gz: 3d1740b55ec51e3459092d99165f75197dcad75d2dcce28ae285ed9dc1e1fcbb2a85002f8abeb868e0bc388424e8e85056bab1b41a54046023d06fbd836f0598
+  data.tar.gz: bffbcf8567d03252bc04fb8e207674595b271635da2ef4cec963cd6ac329e4e58194167f0b474dc522785cbaef10a239b504e12825c56fb9d55042f4733b1840

data/.tool-versions

@@ -0,0 +1 @@
+ruby 2.6.6

data/CHANGELOG.md

@@ -1,3 +1,34 @@
+# 1.11.0 (May 17th, 2021)
+
+Added Jobs:
+
+* b/collection/flat_file_parse
+
+# 1.10.0 (May 17th, 2021)
+
+Added Jobs:
+
+* b/collection/only_keys
+# 1.9.0 (April 13th, 2021)
+
+Added Jobs:
+
+* b/collection/pivot
+# 1.8.0 (March 31st, 2021)
+
+Added Jobs:
+
+* b/param/from_register
+* b/param/to_register
+
+Other:
+
+* Payload#param was added to access a param key's value.
+* Payload#update_param was added to update a param key's value.
+
+Internal Notes:
+
+Payload#register and Payload#params data stores have been internally consolidated while still maintaining the same public API surface area.
 
 # 1.7.0 (January 22nd, 2021)
 

data/README.md

@@ -266,11 +266,15 @@ This library only ships with very basic, rudimentary jobs that are meant to just
 * **b/collection/arrays_to_objects** [mappings, register]: Convert an array of arrays to an array of objects.
 * **b/collection/coalesce** [grouped_register, insensitive, key_mappings, keys, register, separator]: Merge two datasets together based on the key values of one dataset (array) with a grouped dataset (hash).  If insensitive (false by default) is true then each key's value will be converted/coerced to a lowercase string.
 * **b/collection/concatenate** [from_registers, to_register]: Concatenate each from_register's value and place the newly concatenated array into the to_register.  Note: this does not do any deep copying and should be assumed it is shallow copying all objects.
+* **b/collection/flat_file_parse** [keys_register, register, separator]: Map an array of arrays to an array of hashes.  These keys can be realized at run-time as they are pulled from the first entry in the array.  The `keys_register` will also be set to the keys used for mapping.
 * **b/collection/graph** [config, key, register]: Use [Hashematics](https://github.com/bluemarblepayroll/hashematics) to turn a flat array of objects into a deeply nested object tree.
 * **b/collection/group** [insensitive, keys, register, separator]: Take a register's value (an array of objects) and group the objects by the specified keys.  If insensitive (false by default) is true then each key's value will be converted/coerced to a lowercase string.
 * **b/collection/nested_aggregate** [register, key_mappings, key, separator]: Traverse a set of objects, resolving key's value for each object, optionally copying down key_mappings to the child records, then merging all the inner records together.
 * **b/collection/number** [key, register, separator, start_at]: This job can iterate over a set of records and sequence them (set the specified key to a sequential index value.)
 * **b/collection/objects_to_arrays** [mappings, register]: Convert an array of objects to an array of arrays.
+* **b/collection/only_keys** [keys_register, register, separator]: Limit an array of objects' keys to a specified set of keys.  These keys can be realized at run-time as they are pulled from another register (`keys_register`) thus making it dynamic.
+* **b/collection/pivot** [unique_keys, insensitive, other_keys, pivot_key, pivot_value_key, register, separator]:
+Take an array of objects and pivot a key into multiple keys.  It essentially takes all the values for a key and creates N number of keys (one per value.) Under the hood it uses HashMath's [Record](https://github.com/bluemarblepayroll/hash_math#record-the-hash-prototype) and [Table](https://github.com/bluemarblepayroll/hash_math#table-the-double-hash-hash-of-hashes) classes.
 * **b/collection/shift** [amount, register]: Remove the first N number of elements from an array.
 * **b/collection/transform** [attributes, exclusive, separator, register]: Iterate over all objects and transform each key per the attribute transformers specifications.  If exclusive is set to false then the current object will be overridden/merged.  Separator can also be set for key path support.  This job uses [Realize](https://github.com/bluemarblepayroll/realize), which provides its own extendable value-transformation pipeline.  If an attribute is not set with `explicit: true` then it will automatically start from the key's value from the record.  If `explicit: true` is started, then it will start from the record itself.
 * **b/collection/unpivot** [pivot_set, register]: Take an array of objects and unpivot specific sets of keys into rows.  Under the hood it uses [HashMath's Unpivot class](https://github.com/bluemarblepayroll/hash_math#unpivot-hash-key-coalescence-and-row-extrapolation).
@@ -297,6 +301,11 @@ By default all jobs will use the `Burner::Disks::Local` disk for its persistence
 * **b/io/row_reader** [data_key, disk, ignore_blank_path, ignore_file_not_found, path_key, register, separator]: Iterates over an array of objects, extracts a filepath from a key in each object, and attempts to load the file's content for each record.  The file's content will be stored at the specified data_key. By default missing paths or files will be treated as hard errors.  If you wish to ignore these then pass in true for ignore_blank_path and/or ignore_file_not_found.
 * **b/io/write** [binary, disk, path, register, supress_side_effect]: Write to a local file.  The path parameter can be interpolated using `Payload#params`.  If the contents are binary, pass in `binary: true` to open it up in binary+write mode.  By default, written files are also logged as WrittenFile instances to the Payload#side_effects array.  You can pass in supress_side_effect: true to disable this behavior.
 
+#### Parameters
+
+* **b/param/from_register** [param_key, register]: Copy the value of a register to a param key.
+* **b/param/to_register** [param_key, register]: Copy the value of a param key to a register.
+
 #### Serialization
 
 * **b/serialize/csv** [byte_order_mark, register]: Take an array of arrays and create a CSV.  You can optionally pre-pend a byte order mark, see Burner::Modeling::ByteOrderMark for acceptable options.

data/lib/burner/data.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+module Burner
+  # Defines a key value pair data store per our library.  It is basically a composite
+  # object around a hash with indifferent key typing.
+  class Data
+    extend Forwardable
+
+    def_delegators :internal_hash, :transform_keys
+
+    def initialize(hash = {})
+      @internal_hash = {}
+
+      (hash || {}).each { |k, v| self[k] = v }
+    end
+
+    def []=(key, value)
+      internal_hash[key.to_s] = value
+    end
+
+    def [](key)
+      internal_hash[key.to_s]
+    end
+
+    def to_h
+      internal_hash
+    end
+
+    def ==(other)
+      other.instance_of?(self.class) &&
+        to_h == other.to_h
+    end
+    alias eql? ==
+
+    private
+
+    attr_reader :internal_hash
+  end
+end

data/lib/burner/job.rb

@@ -46,15 +46,9 @@ module Burner
     protected
 
     def job_string_template(expression, output, payload)
-      templatable_params = payload.params
-                                  .merge(__id: output.id)
-                                  .merge(templatable_register_values(payload))
+      templatable_params = payload.params_and_registers_hash.merge(__id: output.id)
 
       Util::StringTemplate.instance.evaluate(expression, templatable_params)
     end
-
-    def templatable_register_values(payload)
-      payload.registers.transform_keys { |key| "__#{key}_register" }
-    end
   end
 end

data/lib/burner/job_with_dynamic_keys.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+require_relative 'job_with_register'
+
+module Burner
+  # Add on a register attribute to the configuration for a job.  This indicates that a job
+  # either accesses and/or mutates the payload's registers.
+  class JobWithDynamicKeys < JobWithRegister
+    attr_reader :keys_register,
+                :resolver
+
+    def initialize(
+      keys_register:,
+      name: '',
+      register: DEFAULT_REGISTER,
+      separator: BLANK
+    )
+      super(name: name, register: register)
+
+      @keys_register = keys_register.to_s
+      @resolver      = Objectable.resolver(separator: separator)
+
+      freeze
+    end
+  end
+end

data/lib/burner/job_with_register.rb

@@ -13,6 +13,8 @@ module Burner
   # Add on a register attribute to the configuration for a job.  This indicates that a job
   # either accesses and/or mutates the payload's registers.
   class JobWithRegister < Job
+    BLANK = ''
+
     attr_reader :register
 
     def initialize(name: '', register: DEFAULT_REGISTER)

data/lib/burner/jobs.rb

@@ -25,11 +25,14 @@ module Burner
     register 'b/collection/arrays_to_objects', Library::Collection::ArraysToObjects
     register 'b/collection/coalesce',          Library::Collection::Coalesce
     register 'b/collection/concatenate',       Library::Collection::Concatenate
-    register 'b/collection/graph',             Library::Collection::Graph
     register 'b/collection/group',             Library::Collection::Group
+    register 'b/collection/flat_file_parse',   Library::Collection::FlatFileParse
+    register 'b/collection/graph',             Library::Collection::Graph
     register 'b/collection/nested_aggregate',  Library::Collection::NestedAggregate
     register 'b/collection/number',            Library::Collection::Number
     register 'b/collection/objects_to_arrays', Library::Collection::ObjectsToArrays
+    register 'b/collection/only_keys',         Library::Collection::OnlyKeys
+    register 'b/collection/pivot',             Library::Collection::Pivot
     register 'b/collection/shift',             Library::Collection::Shift
     register 'b/collection/transform',         Library::Collection::Transform
     register 'b/collection/unpivot',           Library::Collection::Unpivot
@@ -48,6 +51,9 @@ module Burner
     register 'b/io/row_reader',                Library::IO::RowReader
     register 'b/io/write',                     Library::IO::Write
 
+    register 'b/param/from_register',          Library::Param::FromRegister
+    register 'b/param/to_register',            Library::Param::ToRegister
+
     register 'b/serialize/csv',                Library::Serialize::Csv
     register 'b/serialize/json',               Library::Serialize::Json
     register 'b/serialize/yaml',               Library::Serialize::Yaml

data/lib/burner/library.rb

@@ -7,7 +7,7 @@
 # LICENSE file in the root directory of this source tree.
 #
 
-require_relative 'job_with_register'
+require_relative 'job_with_dynamic_keys'
 
 require_relative 'library/echo'
 require_relative 'library/nothing'
@@ -16,11 +16,14 @@ require_relative 'library/sleep'
 require_relative 'library/collection/arrays_to_objects'
 require_relative 'library/collection/coalesce'
 require_relative 'library/collection/concatenate'
+require_relative 'library/collection/flat_file_parse'
 require_relative 'library/collection/graph'
 require_relative 'library/collection/group'
 require_relative 'library/collection/nested_aggregate'
 require_relative 'library/collection/number'
 require_relative 'library/collection/objects_to_arrays'
+require_relative 'library/collection/only_keys'
+require_relative 'library/collection/pivot'
 require_relative 'library/collection/shift'
 require_relative 'library/collection/transform'
 require_relative 'library/collection/unpivot'
@@ -39,6 +42,9 @@ require_relative 'library/io/read'
 require_relative 'library/io/row_reader'
 require_relative 'library/io/write'
 
+require_relative 'library/param/from_register'
+require_relative 'library/param/to_register'
+
 require_relative 'library/serialize/csv'
 require_relative 'library/serialize/json'
 require_relative 'library/serialize/yaml'

data/lib/burner/library/collection/flat_file_parse.rb

@@ -0,0 +1,57 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+module Burner
+  module Library
+    module Collection
+      # Convert an array of arrays to an array of objects.  The difference between this
+      # job and ArraysToObjects is that this one does not take in mappings and instead
+      # will use the first entry as the array of keys to positionally map to.
+      #
+      # For example, if a register had this:
+      #
+      #   [['id', 'first', 'last'], [1, 'frank', 'rizzo']]
+      #
+      # Then executing this job would result in this:
+      #
+      #   [{ 'id' => 1, 'first' => frank, 'last' => rizzo }]
+      #
+      # As a side-effect, the keys_register would result in this:
+      #
+      #   ['id', 'first', 'last']
+      #
+      # Expected Payload[register] input: array of arrays.
+      # Payload[register] output: An array of hashes.
+      class FlatFileParse < JobWithDynamicKeys
+        def perform(output, payload)
+          objects = array(payload[register])
+          keys    = array(objects.shift)
+          count   = objects.length
+
+          output.detail("Mapping #{count} array(s) to key(s): #{keys.join(', ')}")
+
+          payload[register]      = objects.map { |object| transform(object, keys) }
+          payload[keys_register] = keys
+        end
+
+        private
+
+        def transform(object, keys)
+          object.each_with_object({}).with_index do |(value, memo), index|
+            next if index >= keys.length
+
+            key = keys[index]
+
+            resolver.set(memo, key, value)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/burner/library/collection/only_keys.rb

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+module Burner
+  module Library
+    module Collection
+      # This job knows how to take an array of objects and limit it to a specific set of keys.
+      # The keys are pulled from another register which helps make it dynamic (you can load
+      # up this other register with a dynamic list of keys at run-time.)
+      #
+      # Expected Payload[register] input: array of objects.
+      # Payload[register] output: An array of objects.
+      class OnlyKeys < JobWithDynamicKeys
+        def perform(output, payload)
+          objects = array(payload[register])
+          count   = objects.length
+          keys    = array(payload[keys_register])
+
+          output.detail("Dynamically limiting #{count} object(s) with key(s): #{keys.join(', ')}")
+
+          payload[register] = objects.map { |object| transform(object, keys) }
+        end
+
+        private
+
+        def transform(object, keys)
+          keys.each_with_object({}) do |key, memo|
+            value = resolver.get(object, key)
+
+            resolver.set(memo, key, value)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/burner/library/collection/pivot.rb

@@ -0,0 +1,150 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+module Burner
+  module Library
+    module Collection
+      # Take an array of objects and pivot a key into multiple keys.  It essentially takes all
+      # the values for a key and creates N number of keys (one per value.)
+      # Under the hood it uses HashMath's Record and Table classes:
+      # https://github.com/bluemarblepayroll/hash_math
+      #
+      # An example of a normalized dataset that could be pivoted:
+      #
+      # records = [
+      #   { patient_id: 1, key: :first_name, value: 'bozo' },
+      #   { patient_id: 1, key: :last_name,  value: 'clown' },
+      #   { patient_id: 2, key: :first_name, value: 'frank' },
+      #   { patient_id: 2, key: :last_name,  value: 'rizzo' },
+      # ]
+      #
+      # Using the following job configuration:
+      #
+      # config = {
+      #   unique_key: :patient_id
+      # }
+      #
+      # Once ran through this job, it would set the register to:
+      #
+      # records = [
+      #   { patient_id: 1, first_name: 'bozo', last_name: 'clown' },
+      #   { patient_id: 2, first_name: 'frank', last_name: 'rizzo' },
+      # ]
+      #
+      # Expected Payload[register] input: array of objects.
+      # Payload[register] output: An array of objects.
+      class Pivot < JobWithRegister
+        DEFAULT_PIVOT_KEY       = :key
+        DEFAULT_PIVOT_VALUE_KEY = :value
+
+        attr_reader :insensitive,
+                    :other_keys,
+                    :non_pivoted_keys,
+                    :pivot_key,
+                    :pivot_value_key,
+                    :resolver,
+                    :unique_keys
+
+        def initialize(
+          unique_keys:,
+          insensitive: false,
+          name: '',
+          other_keys: [],
+          pivot_key: DEFAULT_PIVOT_KEY,
+          pivot_value_key: DEFAULT_PIVOT_KEY_VALUE,
+          register: DEFAULT_REGISTER,
+          separator: ''
+        )
+          super(name: name, register: register)
+
+          @insensitive      = insensitive || false
+          @pivot_key        = pivot_key.to_s
+          @pivot_value_key  = pivot_value_key.to_s
+          @resolver         = Objectable.resolver(separator: separator)
+          @unique_keys      = Array(unique_keys)
+          @other_keys       = Array(other_keys)
+          @non_pivoted_keys = @unique_keys + @other_keys
+
+          freeze
+        end
+
+        def perform(output, payload)
+          objects = array(payload[register])
+          table   = make_table(objects)
+
+          output.detail("Pivoting #{objects.length} object(s)")
+          output.detail("By key: #{pivot_key} and value: #{pivot_value_key}")
+
+          objects.each { |object| object_to_table(object, table) }
+
+          pivoted_objects = table.to_a.map(&:fields)
+
+          output.detail("Resulting dataset has #{pivoted_objects.length} object(s)")
+
+          payload[register] = pivoted_objects
+        end
+
+        private
+
+        def resolve_key(object)
+          key_to_use = resolver.get(object, pivot_key)
+
+          make_key(key_to_use)
+        end
+
+        def make_key(value)
+          insensitive ? value.to_s.downcase : value
+        end
+
+        def make_row_id(object)
+          unique_keys.map { |k| make_key(resolver.get(object, k)) }
+        end
+
+        def make_key_map(objects)
+          objects.each_with_object({}) do |object, key_map|
+            key = resolver.get(object, pivot_key)
+            unique_key = make_key(key)
+
+            key_map[unique_key] ||= Set.new
+
+            key_map[unique_key] << key
+          end
+        end
+
+        def make_record(objects)
+          key_map = make_key_map(objects)
+          keys    = non_pivoted_keys + key_map.values.map(&:first)
+
+          HashMath::Record.new(keys)
+        end
+
+        def make_table(objects)
+          HashMath::Table.new(make_record(objects))
+        end
+
+        def object_to_table(object, table)
+          row_id = make_row_id(object)
+
+          non_pivoted_keys.each do |key|
+            value = resolver.get(object, key)
+
+            table.add(row_id, key, value)
+          end
+
+          key_to_use   = resolve_key(object)
+          value_to_use = resolver.get(object, pivot_value_key)
+
+          table.add(row_id, key_to_use, value_to_use)
+
+          self
+        end
+      end
+    end
+  end
+end

data/lib/burner/library/collection/transform.rb

@@ -21,8 +21,6 @@ module Burner
       # Expected Payload[register] input: array of objects.
       # Payload[register] output: An array of objects.
       class Transform < JobWithRegister
-        BLANK = ''
-
         attr_reader :attribute_renderers,
                     :exclusive,
                     :resolver

data/lib/burner/library/param/base.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+module Burner
+  module Library
+    module Param
+      # Common logic shared across Param job subclasses.
+      class Base < JobWithRegister
+        attr_reader :param_key
+
+        def initialize(name: BLANK, param_key: BLANK, register: DEFAULT_REGISTER)
+          super(name: name, register: register)
+
+          @param_key = param_key.to_s
+
+          freeze
+        end
+      end
+    end
+  end
+end

data/lib/burner/library/param/from_register.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+require_relative 'base'
+
+module Burner
+  module Library
+    module Param
+      # Copy a register's value into a param key.  Generally speaking you should only be
+      # mutating registers, that way the params stay true to the passed in params for the
+      # pipeline.  But this job is available in case a param needs to be updated.
+      #
+      # Expected Payload[register] input: anything.
+      # Payload.params(param_key) output: whatever value was specified in the register.
+      class FromRegister < Base
+        def perform(output, payload)
+          output.detail("Pushing value from register: #{register} to param: #{param_key}")
+
+          payload.update_param(param_key, payload[register])
+        end
+      end
+    end
+  end
+end

data/lib/burner/library/param/to_register.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+require_relative 'base'
+
+module Burner
+  module Library
+    module Param
+      # Copy a param key's value into a register.
+      #
+      # Expected Payload.param(param_key) input: anything.
+      # Payload[register] output: whatever value was specified as the param_key's value.
+      class ToRegister < Base
+        def perform(output, payload)
+          output.detail("Pushing value to register: #{register} from param: #{param_key}")
+
+          payload[register] = payload.param(param_key)
+        end
+      end
+    end
+  end
+end

data/lib/burner/payload.rb

@@ -7,6 +7,8 @@
 # LICENSE file in the root directory of this source tree.
 #
 
+require_relative 'data'
+
 module Burner
   # The input for all Job#perform methods.  The main notion of this object is its 'registers'
   # attribute.  This registers attribute is a key-indifferent hash, accessible on Payload using
@@ -24,10 +26,15 @@ module Burner
   # The 'time' attribute is important in that it should for the replaying of pipelines and jobs.
   # Instead of having job's utilizing Time.now, Date.today, etc... they should rather opt to
   # use this value instead.
+  #
+  # The notion of params are somewhat conflated with registers here.  Both are hashes and both
+  # serve as data stores. The difference is registers are really meant to be the shared-data
+  # repository across jobs, while params are, more or less, the input into the _pipeline_.
+  # It is debatable if mutation of the params should be allowed but the design decision was made
+  # early on to allow for params to be mutable albeit with registers being the preferred mutable
+  # store.
   class Payload
-    attr_reader :params,
-                :registers,
-                :side_effects,
+    attr_reader :side_effects,
                 :time
 
     def initialize(
@@ -36,12 +43,32 @@ module Burner
       side_effects: [],
       time: Time.now.utc
     )
-      @params       = params || {}
-      @registers    = {}
+      @params       = Data.new(params)
+      @registers    = Data.new(registers)
       @side_effects = side_effects || []
       @time         = time || Time.now.utc
+    end
 
-      add_registers(registers)
+    # Backwards compatibility.  This allows for control over the underlying data structure
+    # while still maintaining the same public API as before.
+    def params
+      @params.to_h
+    end
+
+    # Backwards compatibility.  This allows for control over the underlying data structure
+    # while still maintaining the same public API as before.
+    def registers
+      @registers.to_h
+    end
+
+    # Law of Demeter: While params is an accessible hash, it is preferred that the
+    # public class methods are used.
+    def param(key)
+      @params[key]
+    end
+
+    def update_param(key, value)
+      tap { @params[key] = value }
     end
 
     # Add a side effect of a job.  This helps to keep track of things jobs do outside of its
@@ -52,12 +79,12 @@ module Burner
 
     # Set a register's value.
     def []=(key, value)
-      set(key, value)
+      @registers[key] = value
     end
 
     # Retrieve a register's value.
     def [](key)
-      registers[key.to_s]
+      @registers[key]
     end
 
     # Set halt_pipeline to true.  This will indicate to the pipeline to stop all
@@ -71,14 +98,11 @@ module Burner
       @halt_pipeline || false
     end
 
-    private
-
-    def set(key, value)
-      registers[key.to_s] = value
-    end
+    def params_and_registers_hash
+      registers_hash = @registers.transform_keys { |key| "__#{key}_register" }
+      params_hash    = @params.to_h
 
-    def add_registers(registers)
-      (registers || {}).each { |k, v| set(k, v) }
+      params_hash.merge(registers_hash)
     end
   end
 end

data/lib/burner/pipeline.rb

@@ -55,7 +55,7 @@ module Burner
     private
 
     def output_params(params, output)
-      if params.keys.any?
+      if params.any?
         output.write('Parameters:')
       else
         output.write('No parameters passed in.')

data/lib/burner/version.rb

@@ -8,5 +8,5 @@
 #
 
 module Burner
-  VERSION = '1.7.0'
+  VERSION = '1.11.0'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: burner
 version: !ruby/object:Gem::Version
-  version: 1.7.0
+  version: 1.11.0
 platform: ruby
 authors:
 - Matthew Ruggio
-autorequire: 
+autorequire:
 bindir: exe
 cert_chain: []
-date: 2021-01-22 00:00:00.000000000 Z
+date: 2021-05-18 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: acts_as_hashable
@@ -220,6 +220,7 @@ files:
 - ".gitignore"
 - ".rubocop.yml"
 - ".ruby-version"
+- ".tool-versions"
 - ".travis.yml"
 - CHANGELOG.md
 - CODE_OF_CONDUCT.md
@@ -234,21 +235,26 @@ files:
 - exe/burner
 - lib/burner.rb
 - lib/burner/cli.rb
+- lib/burner/data.rb
 - lib/burner/disks.rb
 - lib/burner/disks/local.rb
 - lib/burner/job.rb
 - lib/burner/job_set.rb
+- lib/burner/job_with_dynamic_keys.rb
 - lib/burner/job_with_register.rb
 - lib/burner/jobs.rb
 - lib/burner/library.rb
 - lib/burner/library/collection/arrays_to_objects.rb
 - lib/burner/library/collection/coalesce.rb
 - lib/burner/library/collection/concatenate.rb
+- lib/burner/library/collection/flat_file_parse.rb
 - lib/burner/library/collection/graph.rb
 - lib/burner/library/collection/group.rb
 - lib/burner/library/collection/nested_aggregate.rb
 - lib/burner/library/collection/number.rb
 - lib/burner/library/collection/objects_to_arrays.rb
+- lib/burner/library/collection/only_keys.rb
+- lib/burner/library/collection/pivot.rb
 - lib/burner/library/collection/shift.rb
 - lib/burner/library/collection/transform.rb
 - lib/burner/library/collection/unpivot.rb
@@ -266,6 +272,9 @@ files:
 - lib/burner/library/io/row_reader.rb
 - lib/burner/library/io/write.rb
 - lib/burner/library/nothing.rb
+- lib/burner/library/param/base.rb
+- lib/burner/library/param/from_register.rb
+- lib/burner/library/param/to_register.rb
 - lib/burner/library/serialize/csv.rb
 - lib/burner/library/serialize/json.rb
 - lib/burner/library/serialize/yaml.rb
@@ -304,7 +313,7 @@ metadata:
   documentation_uri: https://www.rubydoc.info/gems/burner
   homepage_uri: https://github.com/bluemarblepayroll/burner
   source_code_uri: https://github.com/bluemarblepayroll/burner
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -320,7 +329,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubygems_version: 3.0.3
-signing_key: 
+signing_key:
 specification_version: 4
 summary: Declarative and extendable processing pipeline
 test_files: []