burner 1.0.0.pre.alpha.9 → 1.1.0.pre.alpha

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a2af01edaf5ab67512cb8d3db4dbf3bba75a6272af78d78d43f3ceaec0ead386
-  data.tar.gz: 812095ecf0b4240f61128f1968380f35e6748a07486f268da358ab56425de013
+  metadata.gz: 9dea13fe31b6e7bc97eafb25201508afc0a05d2d4e2eb86aeab00004695674f1
+  data.tar.gz: 6569debdcc9ad51a0e180ee6b0bac192029b95d59ad767ddae7c6711dba9830d
 SHA512:
-  metadata.gz: 68fcfbf7d2b600889e805691b4e8289c82eb6366391c533333d8873b1521e41d342af00bbf68b01caad954fd8eb136016906b484727020af9ff5716bcfed0822
-  data.tar.gz: 33dc2ca22f71d2e6f54858bfa90e9f5c6bbbff969409fb1d6a8c05b459c105c77ff7e354780a93daca8241edd31fbb03c424091c3557bfb4facf8b43b9fe6264
+  metadata.gz: 20cb03d16e9495edd35a9323f9507780b006f0be2ef61b0423e198ada1bdcc672e7a7fc471d606bc5cd47d3f727237922285bf71dd9648381b92bc5b90327174
+  data.tar.gz: 977cd155d3a717af7a4fc4876694f27b665844780fadf23556cdf1d7fb120a124741c341dbf1d4725fad8a0dbe12260f7475e36d9cd7e1415618929b73fb9794

data/CHANGELOG.md

@@ -1,3 +1,14 @@
+# 1.1.0 (TBD)
+
+Added Jobs:
+
+* b/collection/coalesce
+* b/collection/group
+
+# 1.0.0 (November 5th, 2020)
+
+Initial version publication.
+
 # 0.0.1
 
 Shell

data/README.md

@@ -234,7 +234,10 @@ This library only ships with very basic, rudimentary jobs that are meant to just
 #### Collection
 
 * **b/collection/arrays_to_objects** [mappings, register]: Convert an array of arrays to an array of objects.
+* **b/collection/coalesce** [register, grouped_register, key_mappings, keys, separator]: Merge two datasets together based on the key values of one dataset (array) with a grouped dataset (hash).
+* **b/collection/concatenate** [from_registers, to_register]: Concatenate each from_register's value and place the newly concatenated array into the to_register.  Note: this does not do any deep copying and should be assumed it is shallow copying all objects.
 * **b/collection/graph** [config, key, register]: Use [Hashematics](https://github.com/bluemarblepayroll/hashematics) to turn a flat array of objects into a deeply nested object tree.
+* **b/collection/group** [keys, register, separator]: Take a register's value (an array of objects) and group the objects by the specified keys.
 * **b/collection/objects_to_arrays** [mappings, register]: Convert an array of objects to an array of arrays.
 * **b/collection/shift** [amount, register]: Remove the first N number of elements from an array.
 * **b/collection/transform** [attributes, exclusive, separator, register]: Iterate over all objects and transform each key per the attribute transformers specifications.  If exclusive is set to false then the current object will be overridden/merged.  Separator can also be set for key path support.  This job uses [Realize](https://github.com/bluemarblepayroll/realize), which provides its own extendable value-transformation pipeline.
@@ -260,11 +263,15 @@ This library only ships with very basic, rudimentary jobs that are meant to just
 * **b/serialize/json** [register]: Convert value to JSON.
 * **b/serialize/yaml** [register]: Convert value to YAML.
 
+#### Value
+
+* **b/value/copy** [from_register, to_register]: Copy from_register's value into the to_register.  Note: this does not do any deep copying and should be assumed it is shallow copying all objects.
+* **b/value/static** [register, value]: Set the value to any arbitrary value.
+
 #### General
 
-* **b/dummy** []: Do nothing
 * **b/echo** [message]: Write a message to the output.  The message parameter can be interpolated using  `Payload#params`.
-* **b/set** [register, value]: Set the value to any arbitrary value.
+* **b/nothing** []: Do nothing.
 * **b/sleep** [seconds]: Sleep the thread for X number of seconds.
 
 Notes:
@@ -273,6 +280,8 @@ Notes:
 
 ### Adding & Registering Jobs
 
+Note: Jobs have to be registered with a type in the Burner::Jobs factory.  All jobs that ship with this library are prefixed with `b/` in their type in order to provide a namespace for 'burner-specific' jobs vs. externally provided jobs.
+
 Where this library shines is when additional jobs are plugged in.  Burner uses its `Burner::Jobs` class as its class-level registry built with [acts_as_hashable](https://github.com/bluemarblepayroll/acts_as_hashable)'s acts_as_hashable_factory directive.
 
 Let's say we would like to register a job to parse a CSV:

data/burner.gemspec

@@ -32,7 +32,7 @@ Gem::Specification.new do |s|
   s.add_dependency('hashematics', '~>1.1')
   s.add_dependency('hash_math', '~>1.2')
   s.add_dependency('objectable', '~>1.0')
-  s.add_dependency('realize', '~>1.2')
+  s.add_dependency('realize', '~>1.3')
   s.add_dependency('stringento', '~>2.1')
 
   s.add_development_dependency('guard-rspec', '~>4.7')

data/lib/burner.rb

@@ -29,3 +29,9 @@ require_relative 'burner/util'
 
 # Main Entrypoint(s)
 require_relative 'burner/cli'
+
+# Top-level namespace
+module Burner
+  # All jobs that need to reference the main register should use this constant.
+  DEFAULT_REGISTER = 'default'
+end

data/lib/burner/job.rb

@@ -31,10 +31,11 @@ module Burner
     # The #perform method takes in two arguments: output (an instance of Burner::Output)
     # and payload (an instance of Burner::Payload).  Jobs can leverage output to emit
     # information to the pipeline's log(s).  The payload is utilized to pass data from job to job,
-    # with its most important attribute being #value.  The value attribute is mutable
-    # per the individual job's context (meaning of it is unknown without understanding a job's
-    # input and output value of #value.).  Therefore #value can mean anything and it is up to the
-    # engineers to clearly document the assumptions of its use.
+    # with its most important attribute being #registers.  The registers attribute is a mutable
+    # and accessible hash per the individual job's context
+    # (meaning of it is unknown without understanding a job's input and output value
+    # of #registers.).  Therefore #register key values can mean anything
+    # and it is up to consumers to clearly document the assumptions of its use.
     #
     # Returning false will short-circuit the pipeline right after the job method exits.
     # Returning anything else besides false just means "continue".
@@ -47,9 +48,15 @@ module Burner
     protected
 
     def job_string_template(expression, output, payload)
-      templatable_params = payload.params.merge(__id: output.id, __value: payload[''])
+      templatable_params = payload.params
+                                  .merge(__id: output.id)
+                                  .merge(templatable_register_values(payload))
 
       Util::StringTemplate.instance.evaluate(expression, templatable_params)
     end
+
+    def templatable_register_values(payload)
+      payload.registers.transform_keys { |key| "__#{key}_register" }
+    end
   end
 end

data/lib/burner/job_with_register.rb

@@ -15,7 +15,7 @@ module Burner
   class JobWithRegister < Job
     attr_reader :register
 
-    def initialize(name:, register: '')
+    def initialize(name:, register: DEFAULT_REGISTER)
       super(name: name)
 
       @register = register.to_s

data/lib/burner/jobs.rb

@@ -16,21 +16,23 @@ module Burner
   class Jobs
     acts_as_hashable_factory
 
-    # Dummy is the default as noted by the ''.  This means if a type is omitted, nil, or blank
-    # string then the dummy job will be used.
-    register 'b/dummy', '',                    Library::Dummy
+    # Nothing is the default as noted by the ''.  This means if a type is omitted, nil, or blank
+    # string then the nothing job will be used.
     register 'b/echo',                         Library::Echo
-    register 'b/set_value',                    Library::SetValue
+    register 'b/nothing', '',                  Library::Nothing
     register 'b/sleep',                        Library::Sleep
 
     register 'b/collection/arrays_to_objects', Library::Collection::ArraysToObjects
+    register 'b/collection/coalesce',          Library::Collection::Coalesce
+    register 'b/collection/concatenate',       Library::Collection::Concatenate
     register 'b/collection/graph',             Library::Collection::Graph
+    register 'b/collection/group',             Library::Collection::Group
     register 'b/collection/objects_to_arrays', Library::Collection::ObjectsToArrays
     register 'b/collection/shift',             Library::Collection::Shift
     register 'b/collection/transform',         Library::Collection::Transform
     register 'b/collection/unpivot',           Library::Collection::Unpivot
     register 'b/collection/values',            Library::Collection::Values
-    register 'b/collection/validate',          Library::Collection::Values
+    register 'b/collection/validate',          Library::Collection::Validate
 
     register 'b/deserialize/csv',              Library::Deserialize::Csv
     register 'b/deserialize/json',             Library::Deserialize::Json
@@ -43,5 +45,8 @@ module Burner
     register 'b/serialize/csv',                Library::Serialize::Csv
     register 'b/serialize/json',               Library::Serialize::Json
     register 'b/serialize/yaml',               Library::Serialize::Yaml
+
+    register 'b/value/copy',                   Library::Value::Copy
+    register 'b/value/static',                 Library::Value::Static
   end
 end

data/lib/burner/library.rb

@@ -9,24 +9,33 @@
 
 require_relative 'job_with_register'
 
+require_relative 'library/echo'
+require_relative 'library/nothing'
+require_relative 'library/sleep'
+
 require_relative 'library/collection/arrays_to_objects'
+require_relative 'library/collection/coalesce'
+require_relative 'library/collection/concatenate'
 require_relative 'library/collection/graph'
+require_relative 'library/collection/group'
 require_relative 'library/collection/objects_to_arrays'
 require_relative 'library/collection/shift'
 require_relative 'library/collection/transform'
 require_relative 'library/collection/unpivot'
 require_relative 'library/collection/validate'
 require_relative 'library/collection/values'
+
 require_relative 'library/deserialize/csv'
 require_relative 'library/deserialize/json'
 require_relative 'library/deserialize/yaml'
-require_relative 'library/dummy'
-require_relative 'library/echo'
+
 require_relative 'library/io/exist'
 require_relative 'library/io/read'
 require_relative 'library/io/write'
+
 require_relative 'library/serialize/csv'
 require_relative 'library/serialize/json'
 require_relative 'library/serialize/yaml'
-require_relative 'library/set_value'
-require_relative 'library/sleep'
+
+require_relative 'library/value/copy'
+require_relative 'library/value/static'

data/lib/burner/library/collection/arrays_to_objects.rb

@@ -14,8 +14,8 @@ module Burner
       # Burner::Modeling::KeyIndexMapping instances or hashable configurations which specifies
       # the index-to-key mappings to use.
       #
-      # Expected Payload#value input: array of arrays.
-      # Payload#value output: An array of hashes.
+      # Expected Payload[register] input: array of arrays.
+      # Payload[register] output: An array of hashes.
       #
       # An example using a configuration-first pipeline:
       #
@@ -23,7 +23,7 @@ module Burner
       #     jobs: [
       #       {
       #         name: 'set',
-      #         type: 'b/set_value',
+      #         type: 'b/value/static',
       #         value: [
       #           [1, 'funky']
       #         ]
@@ -47,10 +47,15 @@ module Burner
       #   }
       #
       #   Burner::Pipeline.make(config).execute
+      #
+      # Given the above example, the expected output would be:
+      #  [
+      #    { 'id' => 1, 'name' => 'funky' }
+      #  ]
       class ArraysToObjects < JobWithRegister
         attr_reader :mappings
 
-        def initialize(name:, mappings: [], register: '')
+        def initialize(name:, mappings: [], register: DEFAULT_REGISTER)
           super(name: name, register: register)
 
           @mappings = Modeling::KeyIndexMapping.array(mappings)

data/lib/burner/library/collection/coalesce.rb

@@ -0,0 +1,73 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+module Burner
+  module Library
+    module Collection
+      # This is generally used right after the Group job has been executed on a separate
+      # dataset in a separate register.  This job can match up specified values in its dataset
+      # with lookup values in another.  If it finds a match then it will (shallow) copy over
+      # the values into the respective dataset.
+      #
+      # Expected Payload[register] input: array of objects.
+      # Payload[register] output: array of objects.
+      class Coalesce < JobWithRegister
+        attr_reader :grouped_register, :key_mappings, :keys, :resolver
+
+        def initialize(
+          name:,
+          grouped_register:,
+          key_mappings: [],
+          keys: [],
+          register: DEFAULT_REGISTER,
+          separator: ''
+        )
+          super(name: name, register: register)
+
+          @grouped_register = grouped_register.to_s
+          @key_mappings     = Modeling::KeyMapping.array(key_mappings)
+          @keys             = Array(keys)
+          @resolver         = Objectable.resolver(separator: separator.to_s)
+
+          raise ArgumentError, 'at least one key is required' if @keys.empty?
+
+          freeze
+        end
+
+        def perform(output, payload)
+          payload[register] = array(payload[register])
+          count             = payload[register].length
+
+          output.detail("Coalescing based on key(s): #{keys} for #{count} records(s)")
+
+          payload[register].each do |record|
+            key    = make_key(record)
+            lookup = find_lookup(payload, key)
+
+            key_mappings.each do |key_mapping|
+              value = resolver.get(lookup, key_mapping.from)
+
+              resolver.set(record, key_mapping.to, value)
+            end
+          end
+        end
+
+        private
+
+        def find_lookup(payload, key)
+          (payload[grouped_register] || {})[key] || {}
+        end
+
+        def make_key(record)
+          keys.map { |key| resolver.get(record, key) }
+        end
+      end
+    end
+  end
+end

data/lib/burner/library/collection/concatenate.rb

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+module Burner
+  module Library
+    module Collection
+      # Take the list of from_registers and concatenate each of their values together.
+      # Each from_value will be coerced into an array if it not an array.
+      #
+      # Expected Payload[from_register] input: array of objects.
+      # Payload[to_register] output: An array of objects.
+      class Concatenate < Job
+        attr_reader :from_registers, :to_register
+
+        def initialize(name:, from_registers: [], to_register: DEFAULT_REGISTER)
+          super(name: name)
+
+          @from_registers = Array(from_registers)
+          @to_register    = to_register.to_s
+
+          freeze
+        end
+
+        def perform(output, payload)
+          output.detail("Concatenating registers: '#{from_registers}' to: '#{to_register}'")
+
+          payload[to_register] = from_registers.each_with_object([]) do |from_register, memo|
+            from_register_value = array(payload[from_register])
+
+            memo.concat(from_register_value)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/burner/library/collection/graph.rb

@@ -13,12 +13,17 @@ module Burner
       # Take an array of (denormalized) objects and create an object hierarchy from them.
       # Under the hood it uses Hashematics: https://github.com/bluemarblepayroll/hashematics.
       #
-      # Expected Payload#value input: array of objects.
-      # Payload#value output: An array of objects.
+      # Expected Payload[register] input: array of objects.
+      # Payload[register] output: An array of objects.
       class Graph < JobWithRegister
         attr_reader :key, :groups
 
-        def initialize(name:, key:, config: Hashematics::Configuration.new, register: '')
+        def initialize(
+          name:,
+          key:,
+          config: Hashematics::Configuration.new,
+          register: DEFAULT_REGISTER
+        )
           super(name: name, register: register)
 
           raise ArgumentError, 'key is required' if key.to_s.empty?

data/lib/burner/library/collection/group.rb

@@ -0,0 +1,66 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+module Burner
+  module Library
+    module Collection
+      # Take a register's value (an array of objects) and group the objects by the specified keys.
+      # It essentially creates a hash from an array. This is useful for creating a O(1) lookup
+      # which can then be used in conjunction with the Coalesce Job for another array of data.
+      #
+      # An example of this specific job:
+      #
+      # input: [{ id: 1, code: 'a' }, { id: 2, code: 'b' }]
+      # keys: [:code]
+      # output: { 'a' => { id: 1, code: 'a' }, 'b' => { id: 2, code: 'b' } }
+      #
+      # Expected Payload[register] input: array of objects.
+      # Payload[register] output: hash.
+      class Group < JobWithRegister
+        attr_reader :keys, :resolver
+
+        def initialize(
+          name:,
+          keys: [],
+          register: DEFAULT_REGISTER,
+          separator: ''
+        )
+          super(name: name, register: register)
+
+          @keys     = Array(keys)
+          @resolver = Objectable.resolver(separator: separator.to_s)
+
+          raise ArgumentError, 'at least one key is required' if @keys.empty?
+
+          freeze
+        end
+
+        def perform(output, payload)
+          payload[register] = array(payload[register])
+          count             = payload[register].length
+
+          output.detail("Grouping based on key(s): #{keys} for #{count} records(s)")
+
+          grouped_records = payload[register].each_with_object({}) do |record, memo|
+            key       = make_key(record)
+            memo[key] = record
+          end
+
+          payload[register] = grouped_records
+        end
+
+        private
+
+        def make_key(record)
+          keys.map { |key| resolver.get(record, key) }
+        end
+      end
+    end
+  end
+end

data/lib/burner/library/collection/objects_to_arrays.rb

@@ -15,39 +15,41 @@ module Burner
       # Burner::Modeling::KeyIndexMapping instances or hashable configurations which specifies
       # the key-to-index mappings to use.
       #
-      # Expected Payload#value input: array of hashes.
-      # Payload#value output: An array of arrays.
+      # Expected Payload[register] input: array of hashes.
+      # Payload[register] output: An array of arrays.
       #
       # An example using a configuration-first pipeline:
       #
-      #   config = {
-      #     jobs: [
-      #       {
-      #         name: 'set',
-      #         type: 'b/set_value',
-      #         value: [
-      #           [1, 'funky']
-      #         ]
-      #       },
-      #       {
-      #         name: 'map',
-      #         type: 'b/collection/objects_to_arrays',
-      #         mappings: [
-      #           { index: 0, key: 'id' },
-      #           { index: 1, key: 'name' }
-      #         ]
-      #       },
-      #       {
-      #         name: 'output',
-      #         type: 'b/echo',
-      #         message: 'value is currently: {__value}'
-      #       },
+      #  config = {
+      #    jobs: [
+      #      {
+      #        name: 'set',
+      #        type: 'b/value/static',
+      #        value: [
+      #          { 'id' => 1, 'name' => 'funky' }
+      #        ],
+      #        register: register
+      #      },
+      #      {
+      #        name: 'map',
+      #        type: 'b/collection/objects_to_arrays',
+      #        mappings: [
+      #          { index: 0, key: 'id' },
+      #          { index: 1, key: 'name' }
+      #        ],
+      #        register: register
+      #      },
+      #      {
+      #        name: 'output',
+      #        type: 'b/echo',
+      #        message: 'value is currently: {__value}'
+      #      },
       #
-      #     ],
-      #     steps: %w[set map output]
-      #   }
+      #    ],
+      #    steps: %w[set map output]
+      #  }
       #
-      #   Burner::Pipeline.make(config).execute
+      #  Burner::Pipeline.make(config).execute
       class ObjectsToArrays < JobWithRegister
         attr_reader :mappings
 
@@ -56,7 +58,7 @@ module Burner
         # nested hashes then set separator to '.'.  For more information, see the underlying
         # library that supports this dot-notation concept:
         #   https://github.com/bluemarblepayroll/objectable
-        def initialize(name:, mappings: [], register: '', separator: '')
+        def initialize(name:, mappings: [], register: DEFAULT_REGISTER, separator: '')
           super(name: name, register: register)
 
           @mappings = Modeling::KeyIndexMapping.array(mappings)

data/lib/burner/library/collection/shift.rb

@@ -14,8 +14,8 @@ module Burner
       # attribute.  The initial use case for this was to remove "header" rows from arrays,
       # like you would expect when parsing CSV files.
       #
-      # Expected Payload#value input: nothing.
-      # Payload#value output: An array with N beginning elements removed.
+      # Expected Payload[register] input: nothing.
+      # Payload[register] output: An array with N beginning elements removed.
       class Shift < JobWithRegister
         DEFAULT_AMOUNT = 0
 
@@ -23,7 +23,7 @@ module Burner
 
         attr_reader :amount
 
-        def initialize(name:, amount: DEFAULT_AMOUNT, register: '')
+        def initialize(name:, amount: DEFAULT_AMOUNT, register: DEFAULT_REGISTER)
           super(name: name, register: register)
 
           @amount = amount.to_i

data/lib/burner/library/collection/transform.rb

@@ -18,8 +18,8 @@ module Burner
       # For more information on the specific contract for attributes, see the
       # Burner::Modeling::Attribute class.
       #
-      # Expected Payload#value input: array of objects.
-      # Payload#value output: An array of objects.
+      # Expected Payload[register] input: array of objects.
+      # Payload[register] output: An array of objects.
       class Transform < JobWithRegister
         BLANK = ''
 
@@ -27,7 +27,13 @@ module Burner
                     :exclusive,
                     :resolver
 
-        def initialize(name:, attributes: [], exclusive: false, register: '', separator: BLANK)
+        def initialize(
+          name:,
+          attributes: [],
+          exclusive: false,
+          register: DEFAULT_REGISTER,
+          separator: BLANK
+        )
           super(name: name, register: register)
 
           @resolver  = Objectable.resolver(separator: separator)

data/lib/burner/library/collection/unpivot.rb

@@ -14,12 +14,16 @@ module Burner
       # Under the hood it uses HashMath's Unpivot class:
       # https://github.com/bluemarblepayroll/hash_math
       #
-      # Expected Payload#value input: array of objects.
-      # Payload#value output: An array of objects.
+      # Expected Payload[register] input: array of objects.
+      # Payload[register] output: An array of objects.
       class Unpivot < JobWithRegister
         attr_reader :unpivot
 
-        def initialize(name:, pivot_set: HashMath::Unpivot::PivotSet.new, register: '')
+        def initialize(
+          name:,
+          pivot_set: HashMath::Unpivot::PivotSet.new,
+          register: DEFAULT_REGISTER
+        )
           super(name: name, register: register)
 
           @unpivot = HashMath::Unpivot.new(pivot_set)

data/lib/burner/library/collection/validate.rb

@@ -14,8 +14,9 @@ module Burner
       # of validations.  The main register will include the valid objects and the invalid_register
       # will contain the invalid objects.
       #
-      # Expected Payload#value input: array of objects.
-      # Payload#value output: An array of objects.
+      # Expected Payload[register] input: array of objects.
+      # Payload[register] output: An array of objects that are valid.
+      # Payload[invalid_register] output: An array of objects that are invalid.
       class Validate < JobWithRegister
         DEFAULT_INVALID_REGISTER = 'invalid'
         DEFAULT_JOIN_CHAR        = ', '
@@ -32,7 +33,7 @@ module Burner
           invalid_register: DEFAULT_INVALID_REGISTER,
           join_char: DEFAULT_JOIN_CHAR,
           message_key: DEFAULT_MESSAGE_KEY,
-          register: '',
+          register: DEFAULT_REGISTER,
           separator: '',
           validations: []
         )

data/lib/burner/library/collection/values.rb

@@ -14,12 +14,12 @@ module Burner
       # If include_keys is true (it is false by default), then call #keys on the first
       # object and inject that as a "header" object.
       #
-      # Expected Payload#value input: array of objects.
-      # Payload#value output: An array of arrays.
+      # Expected Payload[register] input: array of objects.
+      # Payload[register] output: An array of arrays.
       class Values < JobWithRegister
         attr_reader :include_keys
 
-        def initialize(name:, include_keys: false, register: '')
+        def initialize(name:, include_keys: false, register: DEFAULT_REGISTER)
           super(name: name, register: register)
 
           @include_keys = include_keys || false

data/lib/burner/library/deserialize/csv.rb

@@ -12,8 +12,8 @@ module Burner
     module Deserialize
       # Take a CSV string and de-serialize into object(s).
       #
-      # Expected Payload#value input: nothing.
-      # Payload#value output: an array of arrays.  Each inner array represents one data row.
+      # Expected Payload[register] input: nothing.
+      # Payload[register] output: an array of arrays.  Each inner array represents one data row.
       class Csv < JobWithRegister
         # This currently only supports returning an array of arrays, including the header row.
         # In the future this could be extended to offer more customizable options, such as

data/lib/burner/library/deserialize/json.rb

@@ -12,8 +12,8 @@ module Burner
     module Deserialize
       # Take a JSON string and deserialize into object(s).
       #
-      # Expected Payload#value input: string of JSON data.
-      # Payload#value output: anything, as specified by the JSON de-serializer.
+      # Expected Payload[register] input: string of JSON data.
+      # Payload[register] output: anything, as specified by the JSON de-serializer.
       class Json < JobWithRegister
         def perform(_output, payload)
           payload[register] = JSON.parse(payload[register])

data/lib/burner/library/deserialize/yaml.rb

@@ -15,12 +15,12 @@ module Burner
       # YAML.  If you wish to ease this restriction, for example if you have custom serialization
       # for custom classes, then you can pass in safe: false.
       #
-      # Expected Payload#value input: string of YAML data.
-      # Payload#value output: anything as specified by the YAML de-serializer.
+      # Expected Payload[register] input: string of YAML data.
+      # Payload[register]output: anything as specified by the YAML de-serializer.
       class Yaml < JobWithRegister
         attr_reader :safe
 
-        def initialize(name:, register: '', safe: true)
+        def initialize(name:, register: DEFAULT_REGISTER, safe: true)
           super(name: name, register: register)
 
           @safe = safe

data/lib/burner/library/echo.rb

@@ -11,7 +11,7 @@ module Burner
   module Library
     # Output a simple message to the output.
     #
-    # Note: this does not use Payload#value.
+    # Note: this does not use Payload#registers.
     class Echo < Job
       attr_reader :message
 

data/lib/burner/library/io/base.rb

@@ -14,7 +14,7 @@ module Burner
       class Base < JobWithRegister
         attr_reader :path
 
-        def initialize(name:, path:, register: '')
+        def initialize(name:, path:, register: DEFAULT_REGISTER)
           super(name: name, register: register)
 
           raise ArgumentError, 'path is required' if path.to_s.empty?

data/lib/burner/library/io/exist.rb

@@ -15,7 +15,7 @@ module Burner
       # Check to see if a file exists.  If short_circuit is set to true and the file
       # does not exist then the job will return false and short circuit the pipeline.
       #
-      # Note: this does not use Payload#value.
+      # Note: this does not use Payload#registers.
       class Exist < Job
         attr_reader :path, :short_circuit
 

data/lib/burner/library/io/read.rb

@@ -14,12 +14,12 @@ module Burner
     module IO
       # Read value from disk.
       #
-      # Expected Payload#value input: nothing.
-      # Payload#value output: contents of the specified file.
+      # Expected Payload[register] input: nothing.
+      # Payload[register] output: contents of the specified file.
       class Read < Base
         attr_reader :binary
 
-        def initialize(name:, path:, binary: false, register: '')
+        def initialize(name:, path:, binary: false, register: DEFAULT_REGISTER)
           super(name: name, path: path, register: register)
 
           @binary = binary || false

data/lib/burner/library/io/write.rb

@@ -14,12 +14,12 @@ module Burner
     module IO
       # Write value to disk.
       #
-      # Expected Payload#value input: anything.
-      # Payload#value output: whatever was passed in.
+      # Expected Payload[register] input: anything.
+      # Payload[register] output: whatever was passed in.
       class Write < Base
         attr_reader :binary
 
-        def initialize(name:, path:, binary: false, register: '')
+        def initialize(name:, path:, binary: false, register: DEFAULT_REGISTER)
           super(name: name, path: path, register: register)
 
           @binary = binary || false

data/lib/burner/library/{dummy.rb → nothing.rb}

@@ -11,8 +11,8 @@ module Burner
   module Library
     # Do nothing.
     #
-    # Note: this does not use Payload#value.
-    class Dummy < Job
+    # Note: this does not use Payload#registers.
+    class Nothing < Job
       def perform(_output, _payload); end
     end
   end

data/lib/burner/library/serialize/csv.rb

@@ -12,8 +12,8 @@ module Burner
     module Serialize
       # Take an array of arrays and create a CSV.
       #
-      # Expected Payload#value input: array of arrays.
-      # Payload#value output: a serialized CSV string.
+      # Expected Payload[register] input: array of arrays.
+      # Payload[register] output: a serialized CSV string.
       class Csv < JobWithRegister
         def perform(_output, payload)
           payload[register] = CSV.generate(options) do |csv|

data/lib/burner/library/serialize/json.rb

@@ -12,8 +12,8 @@ module Burner
     module Serialize
       # Treat value like a Ruby object and serialize it using JSON.
       #
-      # Expected Payload#value input: anything.
-      # Payload#value output: string representing the output of the JSON serializer.
+      # Expected Payload[register] input: anything.
+      # Payload[register] output: string representing the output of the JSON serializer.
       class Json < JobWithRegister
         def perform(_output, payload)
           payload[register] = payload[register].to_json

data/lib/burner/library/serialize/yaml.rb

@@ -12,8 +12,8 @@ module Burner
     module Serialize
       # Treat value like a Ruby object and serialize it using YAML.
       #
-      # Expected Payload#value input: anything.
-      # Payload#value output: string representing the output of the YAML serializer.
+      # Expected Payload[register] input: anything.
+      # Payload[register] output: string representing the output of the YAML serializer.
       class Yaml < JobWithRegister
         def perform(_output, payload)
           payload[register] = payload[register].to_yaml

data/lib/burner/library/sleep.rb

@@ -11,7 +11,7 @@ module Burner
   module Library
     # Arbitrarily put thread to sleep for X number of seconds
     #
-    # Payload#value output: whatever value was specified in this job.
+    # Note: this does not use Payload#registers.
     class Sleep < Job
       attr_reader :seconds
 

data/lib/burner/library/value/copy.rb

@@ -0,0 +1,39 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+module Burner
+  module Library
+    module Value
+      # Copy one value in a register to another.  Note that this does *not* perform any type of
+      # deep copy, it simply points one register's value to another.  If you decide to later mutate
+      # one register then you may mutate the other.
+      #
+      # Expected Payload[from_register] input: anything.
+      # Payload[to_register] output: whatever value was specified in the from_register.
+      class Copy < Job
+        attr_reader :from_register, :to_register
+
+        def initialize(name:, to_register: DEFAULT_REGISTER, from_register: DEFAULT_REGISTER)
+          super(name: name)
+
+          @from_register = from_register.to_s
+          @to_register   = to_register.to_s
+
+          freeze
+        end
+
+        def perform(output, payload)
+          output.detail("Copying register: '#{from_register}' to: '#{to_register}'")
+
+          payload[to_register] = payload[from_register]
+        end
+      end
+    end
+  end
+end

data/lib/burner/library/value/static.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+module Burner
+  module Library
+    module Value
+      # Arbitrarily set the value of a register.
+      #
+      # Expected Payload[register] input: anything.
+      # Payload[register] output: whatever value was specified in this job.
+      class Static < JobWithRegister
+        attr_reader :value
+
+        def initialize(name:, register: DEFAULT_REGISTER, value: nil)
+          super(name: name, register: register)
+
+          @value = value
+
+          freeze
+        end
+
+        def perform(_output, payload)
+          payload[register] = value
+        end
+      end
+    end
+  end
+end

data/lib/burner/modeling.rb

@@ -10,4 +10,5 @@
 require_relative 'modeling/attribute'
 require_relative 'modeling/attribute_renderer'
 require_relative 'modeling/key_index_mapping'
+require_relative 'modeling/key_mapping'
 require_relative 'modeling/validations'

data/lib/burner/modeling/attribute.rb

@@ -10,7 +10,9 @@
 module Burner
   module Modeling
     # Defines a top-level key and the associated transformers for deriving the final value
-    # to set the key to.
+    # to set the key to.  The transformers that can be passed in can be any Realize::Transformers
+    # subclasses.  For more information, see the Realize library at:
+    # https://github.com/bluemarblepayroll/realize
     class Attribute
       acts_as_hashable
 

data/lib/burner/modeling/key_mapping.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+module Burner
+  module Modeling
+    # Generic mapping from a key to another key.
+    class KeyMapping
+      acts_as_hashable
+
+      attr_reader :from, :to
+
+      def initialize(from:, to:)
+        raise ArgumentError, 'from is required' if from.to_s.empty?
+        raise ArgumentError, 'to is required'   if to.to_s.empty?
+
+        @from = from.to_s
+        @to   = to.to_s
+
+        freeze
+      end
+    end
+  end
+end

data/lib/burner/modeling/validations/base.rb

@@ -27,7 +27,7 @@ module Burner
         end
 
         def message
-          @message.to_s.empty? ? "#{key}#{default_message}" : @message.to_s
+          @message.to_s.empty? ? "#{key} #{default_message}" : @message.to_s
         end
       end
     end

data/lib/burner/modeling/validations/blank.rb

@@ -16,14 +16,18 @@ module Burner
       class Blank < Base
         acts_as_hashable
 
+        BLANK_RE = /\A[[:space:]]*\z/.freeze
+
         def valid?(object, resolver)
-          resolver.get(object, key).to_s.empty?
+          value = resolver.get(object, key).to_s
+
+          value.empty? || BLANK_RE.match?(value)
         end
 
         private
 
         def default_message
-          ' must be blank'
+          'must be blank'
         end
       end
     end

data/lib/burner/modeling/validations/present.rb

@@ -7,23 +7,23 @@
 # LICENSE file in the root directory of this source tree.
 #
 
-require_relative 'base'
+require_relative 'blank'
 
 module Burner
   module Modeling
     class Validations
       # Check if a value is present.  If it is blank (null or empty) then it is invalid.
-      class Present < Base
+      class Present < Blank
         acts_as_hashable
 
         def valid?(object_value, resolver)
-          !resolver.get(object_value, key).to_s.empty?
+          !super(object_value, resolver)
         end
 
         private
 
         def default_message
-          ' is required'
+          'is required'
         end
       end
     end

data/lib/burner/payload.rb

@@ -8,16 +8,22 @@
 #
 
 module Burner
-  # The input for all Job#perform methods.  The main notion of this object is its "value"
-  # attribute.  This is dynamic and weak on purpose and is subject to whatever the Job#perform
-  # methods decides it is.  This definitely adds an order-of-magnitude complexity to this whole
-  # library and lifecycle, but I am not sure there is any other way around it: trying to build
-  # a generic, open-ended object pipeline to serve almost any use case.
+  # The input for all Job#perform methods.  The main notion of this object is its 'registers'
+  # attribute.  This registers attribute is a key-indifferent hash, accessible on Payload using
+  # the brackets setter and getter methods.  This is dynamic and weak on purpose and is subject
+  # to whatever the Job#perform methods decides it is.  This definitely adds an order-of-magnitude
+  # complexity to this whole library and lifecycle, but I am not sure there is any other way
+  # around it: trying to build a generic, open-ended processing pipeline to serve almost
+  # any use case.
   #
   # The side_effects attribute can also be utilized as a way for jobs to emit any data in a more
   # structured/additive manner.  The initial use case for this was for Burner's core IO jobs to
   # report back the files it has written in a more structured data way (as opposed to simply
   # writing some information to the output.)
+  #
+  # The 'time' attribute is important in that it should for the replaying of pipelines and jobs.
+  # Instead of having job's utilizing Time.now, Date.today, etc... they should rather opt to
+  # use this value instead.
   class Payload
     attr_reader :params,
                 :registers,

data/lib/burner/version.rb

@@ -8,5 +8,5 @@
 #
 
 module Burner
-  VERSION = '1.0.0-alpha.9'
+  VERSION = '1.1.0-alpha'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: burner
 version: !ruby/object:Gem::Version
-  version: 1.0.0.pre.alpha.9
+  version: 1.1.0.pre.alpha
 platform: ruby
 authors:
 - Matthew Ruggio
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-10-27 00:00:00.000000000 Z
+date: 2020-11-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: acts_as_hashable
@@ -72,14 +72,14 @@ dependencies:
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.2'
+        version: '1.3'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.2'
+        version: '1.3'
 - !ruby/object:Gem::Dependency
   name: stringento
   requirement: !ruby/object:Gem::Requirement
@@ -225,7 +225,10 @@ files:
 - lib/burner/jobs.rb
 - lib/burner/library.rb
 - lib/burner/library/collection/arrays_to_objects.rb
+- lib/burner/library/collection/coalesce.rb
+- lib/burner/library/collection/concatenate.rb
 - lib/burner/library/collection/graph.rb
+- lib/burner/library/collection/group.rb
 - lib/burner/library/collection/objects_to_arrays.rb
 - lib/burner/library/collection/shift.rb
 - lib/burner/library/collection/transform.rb
@@ -235,21 +238,23 @@ files:
 - lib/burner/library/deserialize/csv.rb
 - lib/burner/library/deserialize/json.rb
 - lib/burner/library/deserialize/yaml.rb
-- lib/burner/library/dummy.rb
 - lib/burner/library/echo.rb
 - lib/burner/library/io/base.rb
 - lib/burner/library/io/exist.rb
 - lib/burner/library/io/read.rb
 - lib/burner/library/io/write.rb
+- lib/burner/library/nothing.rb
 - lib/burner/library/serialize/csv.rb
 - lib/burner/library/serialize/json.rb
 - lib/burner/library/serialize/yaml.rb
-- lib/burner/library/set_value.rb
 - lib/burner/library/sleep.rb
+- lib/burner/library/value/copy.rb
+- lib/burner/library/value/static.rb
 - lib/burner/modeling.rb
 - lib/burner/modeling/attribute.rb
 - lib/burner/modeling/attribute_renderer.rb
 - lib/burner/modeling/key_index_mapping.rb
+- lib/burner/modeling/key_mapping.rb
 - lib/burner/modeling/validations.rb
 - lib/burner/modeling/validations/base.rb
 - lib/burner/modeling/validations/blank.rb

data/lib/burner/library/set_value.rb

@@ -1,32 +0,0 @@
-# frozen_string_literal: true
-
-#
-# Copyright (c) 2020-present, Blue Marble Payroll, LLC
-#
-# This source code is licensed under the MIT license found in the
-# LICENSE file in the root directory of this source tree.
-#
-
-module Burner
-  module Library
-    # Arbitrarily set value
-    #
-    # Expected Payload#value input: anything.
-    # Payload#value output: whatever value was specified in this job.
-    class SetValue < JobWithRegister
-      attr_reader :value
-
-      def initialize(name:, register: '', value: nil)
-        super(name: name, register: register)
-
-        @value = value
-
-        freeze
-      end
-
-      def perform(_output, payload)
-        payload[register] = value
-      end
-    end
-  end
-end