burner 1.4.0 → 1.7.0.pre.alpha

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 708bae8cd70ee165a49a692ede631830ba30e378d002c6dd1b986262def154f9
-  data.tar.gz: bb11da5222c638d97cf33e39633c4f7fca7c6e0a6ccf2cf75c24844d3b3d6159
+  metadata.gz: 015d18494cd63b8440f061092912d8c70d2b9327d4983f2e2465f5ccc0fbc971
+  data.tar.gz: d1fc16e3a9aed39dc9a910e178fd8dd6adabfd0f69320c525b7ef6b6b092541b
 SHA512:
-  metadata.gz: 4b15fff4b6c137d9d2d004181da58d8f4a2864605b3b286685449f8b8dd63bde6b89335cd4a619dc15bc57b261772f90c0eacd7679e13a16ed5897da75d66594
-  data.tar.gz: 2fe52efc1b645028b97332b0d91f919408fccab68a1474e6406ee0d1ac324d1b12d3cd518ed1817203307677611bcb76e13c00423c417faa86b2c3e3754f68a7
+  metadata.gz: 3429b4a33464567987939d911a8265854f2f35788a9fadeb294d7209ab5e09c2f4b1bd0bba77c93ada6c315ad0fae57079a6843bb67fb2bc49cce284939146ab
+  data.tar.gz: 4f31f6728c709dac0af4168122cba9d63409e8e018d8fb8140a0a1d29baad893b0a022ada878f535ef08f36b28697996c6d14b1a253416230be284c2b4e60b7b

data/CHANGELOG.md

@@ -1,3 +1,26 @@
+
+# 1.7.0 (TBD)
+
+Added Jobs:
+
+* b/collection/number
+* b/value/nest
+* b/value/transform
+
+Enhanced Jobs:
+
+* b/collection/coalesce and b/collection/group now support the notion of case and type-insensitivity (insensitive option).
+
+# 1.6.0 (December 22nd, 2020)
+
+Additions:
+
+* b/io/write now provides an optional `supress_side_effect` option.
+# 1.5.0 (December 21st, 2020)
+
+Added Jobs:
+
+* b/collection/zip
 # 1.4.0 (December 17th, 2020)
 
 Additions:

data/README.md

@@ -93,6 +93,54 @@ Some notes:
 * Jobs can be re-used (just like the output_id and output_value jobs).
 * If steps is nil then all jobs will execute in their declared order.
 
+### Omitting Job Names and Steps
+
+Job names are optional, but steps can only correspond to named jobs.  This means if steps is declared then anonymous jobs will have no way to be executed.  Here is the same pipeline as above, but without job names and steps:
+
+````ruby
+pipeline = {
+  jobs: [
+    {
+      type: 'b/io/read',
+      path: '{input_file}'
+    },
+    {
+      type: 'b/echo',
+      message: 'The job id is: {__id}'
+    },
+    {
+      type: 'b/echo',
+      message: 'The current value is: {__default_register}'
+    },
+    {
+      type: 'b/deserialize/json'
+    },
+    {
+      type: 'b/serialize/yaml'
+    },
+    {
+      type: 'b/echo',
+      message: 'The current value is: {__default_register}'
+    },
+    {
+      type: 'b/io/write',
+      path: '{output_file}'
+    }
+  ]
+}
+
+params = {
+  input_file: 'input.json',
+  output_file: 'output.yaml'
+}
+
+payload = Burner::Payload.new(params: params)
+
+Burner::Pipeline.make(pipeline).execute(payload: payload)
+````
+
+Like everything in software, there are trade-offs to the above two equivalent pipelines.  The former (one with steps and job names) has less jobs but is more verbose.  The latter (without steps and job names) has more jobs but reads terser.  Names also can aid in self-documenting your code/configuration so it may be a good idea to enforce at least names are used.
+
 ### Capturing Feedback / Output
 
 By default, output will be emitted to `$stdout`.  You can add or change listeners by passing in optional values into Pipeline#execute.  For example, say we wanted to capture the output from our json-to-yaml example:
@@ -216,17 +264,19 @@ This library only ships with very basic, rudimentary jobs that are meant to just
 #### Collection
 
 * **b/collection/arrays_to_objects** [mappings, register]: Convert an array of arrays to an array of objects.
-* **b/collection/coalesce** [register, grouped_register, key_mappings, keys, separator]: Merge two datasets together based on the key values of one dataset (array) with a grouped dataset (hash).
+* **b/collection/coalesce** [grouped_register, insensitive, key_mappings, keys, register, separator]: Merge two datasets together based on the key values of one dataset (array) with a grouped dataset (hash).  If insensitive (false by default) is true then each key's value will be converted/coerced to a lowercase string.
 * **b/collection/concatenate** [from_registers, to_register]: Concatenate each from_register's value and place the newly concatenated array into the to_register.  Note: this does not do any deep copying and should be assumed it is shallow copying all objects.
 * **b/collection/graph** [config, key, register]: Use [Hashematics](https://github.com/bluemarblepayroll/hashematics) to turn a flat array of objects into a deeply nested object tree.
-* **b/collection/group** [keys, register, separator]: Take a register's value (an array of objects) and group the objects by the specified keys.
+* **b/collection/group** [insensitive, keys, register, separator]: Take a register's value (an array of objects) and group the objects by the specified keys.  If insensitive (false by default) is true then each key's value will be converted/coerced to a lowercase string.
 * **b/collection/nested_aggregate** [register, key_mappings, key, separator]: Traverse a set of objects, resolving key's value for each object, optionally copying down key_mappings to the child records, then merging all the inner records together.
+* **b/collection/number** [key, register, separator, start_at]: This job can iterate over a set of records and sequence them (set the specified key to a sequential index value.)
 * **b/collection/objects_to_arrays** [mappings, register]: Convert an array of objects to an array of arrays.
 * **b/collection/shift** [amount, register]: Remove the first N number of elements from an array.
 * **b/collection/transform** [attributes, exclusive, separator, register]: Iterate over all objects and transform each key per the attribute transformers specifications.  If exclusive is set to false then the current object will be overridden/merged.  Separator can also be set for key path support.  This job uses [Realize](https://github.com/bluemarblepayroll/realize), which provides its own extendable value-transformation pipeline.  If an attribute is not set with `explicit: true` then it will automatically start from the key's value from the record.  If `explicit: true` is started, then it will start from the record itself.
 * **b/collection/unpivot** [pivot_set, register]: Take an array of objects and unpivot specific sets of keys into rows.  Under the hood it uses [HashMath's Unpivot class](https://github.com/bluemarblepayroll/hash_math#unpivot-hash-key-coalescence-and-row-extrapolation).
 * **b/collection/validate** [invalid_register, join_char, message_key, register, separator, validations]: Take an array of objects, run it through each declared validator, and split the objects into two registers.  The valid objects will be split into the current register while the invalid ones will go into the invalid_register as declared.  Optional arguments, join_char and message_key, help determine the compiled error messages.  The separator option can be utilized to use dot-notation for validating keys.  See each validation's options by viewing their classes within the `lib/modeling/validations` directory.
 * **b/collection/values** [include_keys, register]: Take an array of objects and call `#values` on each object. If include_keys is true (it is false by default), then call `#keys` on the first object and inject that as a "header" object.
+* **b/collection/zip** [base_register, register, with_register]: Combines `base_register` and `with_register`s' data to form one single array in `register`.  It will combine each element, positionally in each array to form the final array.  For example:  ['hello', 'bugs'] + ['world', 'bunny'] => [['hello', 'world'], ['bugs', 'bunny']]
 
 #### Compression
 
@@ -245,7 +295,7 @@ By default all jobs will use the `Burner::Disks::Local` disk for its persistence
 * **b/io/exist** [disk, path, short_circuit]: Check to see if a file exists. The path parameter can be interpolated using `Payload#params`.  If short_circuit was set to true (defaults to false) and the file does not exist then the pipeline will be short-circuited.
 * **b/io/read** [binary, disk, path, register]: Read in a local file.  The path parameter can be interpolated using `Payload#params`.  If the contents are binary, pass in `binary: true` to open it up in binary+read mode.
 * **b/io/row_reader** [data_key, disk, ignore_blank_path, ignore_file_not_found, path_key, register, separator]: Iterates over an array of objects, extracts a filepath from a key in each object, and attempts to load the file's content for each record.  The file's content will be stored at the specified data_key. By default missing paths or files will be treated as hard errors.  If you wish to ignore these then pass in true for ignore_blank_path and/or ignore_file_not_found.
-* **b/io/write** [binary, disk, path, register]: Write to a local file.  The path parameter can be interpolated using `Payload#params`.  If the contents are binary, pass in `binary: true` to open it up in binary+write mode.
+* **b/io/write** [binary, disk, path, register, supress_side_effect]: Write to a local file.  The path parameter can be interpolated using `Payload#params`.  If the contents are binary, pass in `binary: true` to open it up in binary+write mode.  By default, written files are also logged as WrittenFile instances to the Payload#side_effects array.  You can pass in supress_side_effect: true to disable this behavior.
 
 #### Serialization
 
@@ -256,7 +306,9 @@ By default all jobs will use the `Burner::Disks::Local` disk for its persistence
 #### Value
 
 * **b/value/copy** [from_register, to_register]: Copy from_register's value into the to_register.  Note: this does not do any deep copying and should be assumed it is shallow copying all objects.
+* **b/value/nest** [key, register]: This job will nest the current value within a new outer hash.  The specified key passed in will be the corresponding new hash key entry for the existing value.
 * **b/value/static** [register, value]: Set the value to any arbitrary value.
+* **b/value/transform** [register, separator, transformers]: Transform the current value of the register through a Realize::Pipeline.  This will transform the entire value, as opposed to the b/collection/transform job, which will iterate over each row/record in a dataset and transform each row/record.
 
 #### General
 

data/lib/burner/job.rb

@@ -18,9 +18,7 @@ module Burner
 
     attr_reader :name
 
-    def initialize(name:)
-      raise ArgumentError, 'name is required' if name.to_s.empty?
-
+    def initialize(name: '')
       @name = name.to_s
     end
 

data/lib/burner/job_set.rb

@@ -0,0 +1,64 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+require_relative 'jobs'
+
+module Burner
+  # This class understands how jobs fit together as a cohesive unit.  It does not know how to
+  # use them, but it knows how to group them together in a logical manner following some simple
+  # rules, such as:
+  #   - Jobs in a set should have unique names (unless the name is blank)
+  #   - Subsets of jobs can be extracted, by name, in constant time.
+  class JobSet
+    class DuplicateJobNameError < StandardError; end
+    class JobNotFoundError < StandardError; end
+
+    def initialize(jobs = [])
+      @jobs = Jobs.array(jobs).freeze
+
+      assert_unique_job_names
+    end
+
+    def jobs(names = nil)
+      return @jobs unless names
+
+      Array(names).map do |name|
+        job = named_jobs_by_name[name.to_s]
+
+        raise JobNotFoundError, "#{name} was not declared as a job" unless job
+
+        job
+      end
+    end
+
+    private
+
+    def named_jobs_by_name
+      @named_jobs_by_name ||= named_jobs.each_with_object({}) { |job, memo| memo[job.name] = job }
+    end
+
+    def named_jobs
+      @named_jobs ||= @jobs.reject { |job| job.name == '' }
+    end
+
+    def assert_unique_job_names
+      unique_job_names = Set.new
+
+      named_jobs.each do |job|
+        if unique_job_names.include?(job.name)
+          raise DuplicateJobNameError, "job with name: #{job.name} already declared"
+        end
+
+        unique_job_names << job.name
+      end
+
+      nil
+    end
+  end
+end

data/lib/burner/job_with_register.rb

@@ -15,10 +15,17 @@ module Burner
   class JobWithRegister < Job
     attr_reader :register
 
-    def initialize(name:, register: DEFAULT_REGISTER)
+    def initialize(name: '', register: DEFAULT_REGISTER)
       super(name: name)
 
       @register = register.to_s
     end
+
+    protected
+
+    # Helper method that knows how to ensure the register is an array.
+    def ensure_array(payload)
+      payload[register] = array(payload[register])
+    end
   end
 end

data/lib/burner/jobs.rb

@@ -28,12 +28,14 @@ module Burner
     register 'b/collection/graph',             Library::Collection::Graph
     register 'b/collection/group',             Library::Collection::Group
     register 'b/collection/nested_aggregate',  Library::Collection::NestedAggregate
+    register 'b/collection/number',            Library::Collection::Number
     register 'b/collection/objects_to_arrays', Library::Collection::ObjectsToArrays
     register 'b/collection/shift',             Library::Collection::Shift
     register 'b/collection/transform',         Library::Collection::Transform
     register 'b/collection/unpivot',           Library::Collection::Unpivot
     register 'b/collection/values',            Library::Collection::Values
     register 'b/collection/validate',          Library::Collection::Validate
+    register 'b/collection/zip',               Library::Collection::Zip
 
     register 'b/compress/row_reader',          Library::Compress::RowReader
 
@@ -51,6 +53,8 @@ module Burner
     register 'b/serialize/yaml',               Library::Serialize::Yaml
 
     register 'b/value/copy',                   Library::Value::Copy
+    register 'b/value/nest',                   Library::Value::Nest
     register 'b/value/static',                 Library::Value::Static
+    register 'b/value/transform',              Library::Value::Transform
   end
 end

data/lib/burner/library.rb

@@ -19,12 +19,14 @@ require_relative 'library/collection/concatenate'
 require_relative 'library/collection/graph'
 require_relative 'library/collection/group'
 require_relative 'library/collection/nested_aggregate'
+require_relative 'library/collection/number'
 require_relative 'library/collection/objects_to_arrays'
 require_relative 'library/collection/shift'
 require_relative 'library/collection/transform'
 require_relative 'library/collection/unpivot'
 require_relative 'library/collection/validate'
 require_relative 'library/collection/values'
+require_relative 'library/collection/zip'
 
 require_relative 'library/compress/row_reader'
 
@@ -42,4 +44,6 @@ require_relative 'library/serialize/json'
 require_relative 'library/serialize/yaml'
 
 require_relative 'library/value/copy'
+require_relative 'library/value/nest'
 require_relative 'library/value/static'
+require_relative 'library/value/transform'

data/lib/burner/library/collection/arrays_to_objects.rb

@@ -55,7 +55,7 @@ module Burner
       class ArraysToObjects < JobWithRegister
         attr_reader :mappings
 
-        def initialize(name:, mappings: [], register: DEFAULT_REGISTER)
+        def initialize(mappings: [], name: '', register: DEFAULT_REGISTER)
           super(name: name, register: register)
 
           @mappings = Modeling::KeyIndexMapping.array(mappings)

data/lib/burner/library/collection/coalesce.rb

@@ -18,19 +18,27 @@ module Burner
       # Expected Payload[register] input: array of objects.
       # Payload[register] output: array of objects.
       class Coalesce < JobWithRegister
-        attr_reader :grouped_register, :key_mappings, :keys, :resolver
+        include Util::Keyable
+
+        attr_reader :grouped_register,
+                    :insensitive,
+                    :key_mappings,
+                    :keys,
+                    :resolver
 
         def initialize(
-          name:,
           grouped_register:,
+          insensitive: false,
           key_mappings: [],
           keys: [],
+          name: '',
           register: DEFAULT_REGISTER,
           separator: ''
         )
           super(name: name, register: register)
 
           @grouped_register = grouped_register.to_s
+          @insensitive      = insensitive || false
           @key_mappings     = Modeling::KeyMapping.array(key_mappings)
           @keys             = Array(keys)
           @resolver         = Objectable.resolver(separator: separator.to_s)
@@ -41,13 +49,14 @@ module Burner
         end
 
         def perform(output, payload)
-          payload[register] = array(payload[register])
-          count             = payload[register].length
+          ensure_array(payload)
+
+          count = payload[register].length
 
           output.detail("Coalescing based on key(s): #{keys} for #{count} records(s)")
 
           payload[register].each do |record|
-            key    = make_key(record)
+            key    = make_key(record, keys, resolver, insensitive)
             lookup = find_lookup(payload, key)
 
             key_mappings.each do |key_mapping|
@@ -63,10 +72,6 @@ module Burner
         def find_lookup(payload, key)
           (payload[grouped_register] || {})[key] || {}
         end
-
-        def make_key(record)
-          keys.map { |key| resolver.get(record, key) }
-        end
       end
     end
   end

data/lib/burner/library/collection/concatenate.rb

@@ -18,7 +18,7 @@ module Burner
       class Concatenate < Job
         attr_reader :from_registers, :to_register
 
-        def initialize(name:, from_registers: [], to_register: DEFAULT_REGISTER)
+        def initialize(from_registers: [], name: '', to_register: DEFAULT_REGISTER)
           super(name: name)
 
           @from_registers = Array(from_registers)

data/lib/burner/library/collection/graph.rb

@@ -19,9 +19,9 @@ module Burner
         attr_reader :key, :groups
 
         def initialize(
-          name:,
           key:,
           config: Hashematics::Configuration.new,
+          name: '',
           register: DEFAULT_REGISTER
         )
           super(name: name, register: register)

data/lib/burner/library/collection/group.rb

@@ -16,6 +16,12 @@ module Burner
       # It is worth noting that the resulting hashes values are singular objects and not an array
       # like Ruby's Enumerable#group_by method.
       #
+      # If the insensitive option is set as true then each key's value will be coerced as
+      # a lowercase string.  This can help provide two types of insensitivity: case and type
+      # insensitivity.  This may be appropriate in some places but not others.  If any other
+      # value coercion is needed then another option would be to first transform the records
+      # before grouping them.
+      #
       # An example of this specific job:
       #
       # input: [{ id: 1, code: 'a' }, { id: 2, code: 'b' }]
@@ -25,18 +31,22 @@ module Burner
       # Expected Payload[register] input: array of objects.
       # Payload[register] output: hash.
       class Group < JobWithRegister
-        attr_reader :keys, :resolver
+        include Util::Keyable
+
+        attr_reader :insensitive, :keys, :resolver
 
         def initialize(
-          name:,
+          insensitive: false,
           keys: [],
+          name: '',
           register: DEFAULT_REGISTER,
           separator: ''
         )
           super(name: name, register: register)
 
-          @keys     = Array(keys)
-          @resolver = Objectable.resolver(separator: separator.to_s)
+          @insensitive = insensitive || false
+          @keys        = Array(keys)
+          @resolver    = Objectable.resolver(separator: separator.to_s)
 
           raise ArgumentError, 'at least one key is required' if @keys.empty?
 
@@ -50,18 +60,12 @@ module Burner
           output.detail("Grouping based on key(s): #{keys} for #{count} records(s)")
 
           grouped_records = payload[register].each_with_object({}) do |record, memo|
-            key       = make_key(record)
+            key       = make_key(record, keys, resolver, insensitive)
             memo[key] = record
           end
 
           payload[register] = grouped_records
         end
-
-        private
-
-        def make_key(record)
-          keys.map { |key| resolver.get(record, key) }
-        end
       end
     end
   end

data/lib/burner/library/collection/nested_aggregate.rb

@@ -21,7 +21,7 @@ module Burner
       class NestedAggregate < JobWithRegister
         attr_reader :key, :key_mappings, :resolver
 
-        def initialize(name:, key:, key_mappings: [], register: DEFAULT_REGISTER, separator: '')
+        def initialize(key:, key_mappings: [], name: '', register: DEFAULT_REGISTER, separator: '')
           super(name: name, register: register)
 
           raise ArgumentError, 'key is required' if key.to_s.empty?

data/lib/burner/library/collection/number.rb

@@ -0,0 +1,51 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+module Burner
+  module Library
+    module Collection
+      # This job can iterate over a set of records and sequence them (set the specified key to
+      # a sequential index value.)
+      #
+      # Expected Payload[register] input: array of objects.
+      # Payload[register] output: array of objects.
+      class Number < JobWithRegister
+        BLANK            = ''
+        DEFAULT_KEY      = 'number'
+        DEFAULT_START_AT = 1
+
+        attr_reader :key, :resolver, :start_at
+
+        def initialize(
+          key: DEFAULT_KEY,
+          name: BLANK,
+          register: Burner::DEFAULT_REGISTER,
+          separator: BLANK,
+          start_at: DEFAULT_START_AT
+        )
+          super(name: name, register: register)
+
+          @key      = key.to_s
+          @resolver = Objectable.resolver(separator: separator)
+          @start_at = start_at.to_i
+
+          freeze
+        end
+
+        def perform(output, payload)
+          output.detail("Setting '#{key}' for each record with values starting at #{start_at}")
+
+          ensure_array(payload).each.with_index(start_at) do |record, index|
+            resolver.set(record, key, index)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/burner/library/collection/objects_to_arrays.rb

@@ -58,7 +58,7 @@ module Burner
         # nested hashes then set separator to '.'.  For more information, see the underlying
         # library that supports this dot-notation concept:
         #   https://github.com/bluemarblepayroll/objectable
-        def initialize(name:, mappings: [], register: DEFAULT_REGISTER, separator: '')
+        def initialize(mappings: [], name: '', register: DEFAULT_REGISTER, separator: '')
           super(name: name, register: register)
 
           @mappings = Modeling::KeyIndexMapping.array(mappings)

data/lib/burner/library/collection/shift.rb

@@ -23,7 +23,7 @@ module Burner
 
         attr_reader :amount
 
-        def initialize(name:, amount: DEFAULT_AMOUNT, register: DEFAULT_REGISTER)
+        def initialize(amount: DEFAULT_AMOUNT, name: '', register: DEFAULT_REGISTER)
           super(name: name, register: register)
 
           @amount = amount.to_i

data/lib/burner/library/collection/transform.rb

@@ -28,9 +28,9 @@ module Burner
                     :resolver
 
         def initialize(
-          name:,
           attributes: [],
           exclusive: false,
+          name: '',
           register: DEFAULT_REGISTER,
           separator: BLANK
         )

data/lib/burner/library/collection/unpivot.rb

@@ -20,7 +20,7 @@ module Burner
         attr_reader :unpivot
 
         def initialize(
-          name:,
+          name: '',
           pivot_set: HashMath::Unpivot::PivotSet.new,
           register: DEFAULT_REGISTER
         )

data/lib/burner/library/collection/validate.rb

@@ -29,10 +29,10 @@ module Burner
                     :validations
 
         def initialize(
-          name:,
           invalid_register: DEFAULT_INVALID_REGISTER,
           join_char: DEFAULT_JOIN_CHAR,
           message_key: DEFAULT_MESSAGE_KEY,
+          name: '',
           register: DEFAULT_REGISTER,
           separator: '',
           validations: []

data/lib/burner/library/collection/values.rb

@@ -19,7 +19,7 @@ module Burner
       class Values < JobWithRegister
         attr_reader :include_keys
 
-        def initialize(name:, include_keys: false, register: DEFAULT_REGISTER)
+        def initialize(include_keys: false, name: '', register: DEFAULT_REGISTER)
           super(name: name, register: register)
 
           @include_keys = include_keys || false

data/lib/burner/library/collection/zip.rb

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+module Burner
+  module Library
+    module Collection
+      # This job can take two arrays and coalesces them by index.  For example:
+      #
+      # input:
+      #   base_register: [ 'hello', 'bugs' ]
+      #   with_register: [ 'world', 'bunny' ]
+      # output:
+      #  register: [ ['hello', 'world'], ['bugs', 'bunny'] ]
+      #
+      # Expected Payload[base_register] input: array of objects.
+      # Expected Payload[with_register] input: array of objects.
+      # Payload[register] output: An array of two-dimensional arrays.
+      class Zip < JobWithRegister
+        attr_reader :base_register, :with_register
+
+        def initialize(
+          with_register:,
+          base_register: DEFAULT_REGISTER,
+          name: '',
+          register: DEFAULT_REGISTER
+        )
+          super(name: name, register: register)
+
+          @base_register = base_register.to_s
+          @with_register = with_register.to_s
+
+          freeze
+        end
+
+        def perform(output, payload)
+          base_data = array(payload[base_register])
+          with_data = array(payload[with_register])
+
+          output.detail("Combining register: #{base_register} (#{base_data.length} record(s))")
+          output.detail("With register: #{with_register} (#{with_data.length} record(s))")
+
+          payload[register] = base_data.zip(with_data)
+        end
+      end
+    end
+  end
+end

data/lib/burner/library/compress/row_reader.rb

@@ -33,10 +33,10 @@ module Burner
                     :resolver
 
         def initialize(
-          name:,
           data_key: DEFAULT_DATA_KEY,
           ignore_blank_data: false,
           ignore_blank_path: false,
+          name: '',
           path_key: DEFAULT_PATH_KEY,
           register: DEFAULT_REGISTER,
           separator: ''

data/lib/burner/library/deserialize/yaml.rb

@@ -20,7 +20,7 @@ module Burner
       class Yaml < JobWithRegister
         attr_reader :safe
 
-        def initialize(name:, register: DEFAULT_REGISTER, safe: true)
+        def initialize(name: '', register: DEFAULT_REGISTER, safe: true)
           super(name: name, register: register)
 
           @safe = safe

data/lib/burner/library/echo.rb

@@ -15,7 +15,7 @@ module Burner
     class Echo < Job
       attr_reader :message
 
-      def initialize(name:, message: '')
+      def initialize(message: '', name: '')
         super(name: name)
 
         @message = message.to_s

data/lib/burner/library/io/exist.rb

@@ -17,7 +17,7 @@ module Burner
       class Exist < Job
         attr_reader :disk, :path, :short_circuit
 
-        def initialize(name:, path:, disk: {}, short_circuit: false)
+        def initialize(path:, disk: {}, name: '', short_circuit: false)
           super(name: name)
 
           raise ArgumentError, 'path is required' if path.to_s.empty?

data/lib/burner/library/io/open_file_base.rb

@@ -14,7 +14,7 @@ module Burner
       class OpenFileBase < JobWithRegister
         attr_reader :binary, :disk, :path
 
-        def initialize(name:, path:, binary: false, disk: {}, register: DEFAULT_REGISTER)
+        def initialize(path:, binary: false, disk: {}, name: '', register: DEFAULT_REGISTER)
           super(name: name, register: register)
 
           raise ArgumentError, 'path is required' if path.to_s.empty?

data/lib/burner/library/io/row_reader.rb

@@ -35,12 +35,12 @@ module Burner
                     :resolver
 
         def initialize(
-          name:,
           binary: false,
           data_key: DEFAULT_DATA_KEY,
           disk: {},
           ignore_blank_path: false,
           ignore_file_not_found: false,
+          name: '',
           path_key: DEFAULT_PATH_KEY,
           register: DEFAULT_REGISTER,
           separator: ''

data/lib/burner/library/io/write.rb

@@ -12,11 +12,34 @@ require_relative 'open_file_base'
 module Burner
   module Library
     module IO
-      # Write value to disk.
+      # Write value to disk.  By default, written files are also logged as WrittenFile
+      # instances to the Payload#side_effects array.  You can pass in
+      # supress_side_effect: true to disable this behavior.
       #
       # Expected Payload[register] input: anything.
       # Payload[register] output: whatever was passed in.
       class Write < OpenFileBase
+        attr_reader :supress_side_effect
+
+        def initialize(
+          path:,
+          binary: false,
+          disk: {},
+          name: '',
+          register: DEFAULT_REGISTER,
+          supress_side_effect: false
+        )
+          @supress_side_effect = supress_side_effect || false
+
+          super(
+            binary: binary,
+            disk: disk,
+            name: name,
+            path: path,
+            register: register
+          )
+        end
+
         def perform(output, payload)
           logical_filename  = job_string_template(path, output, payload)
           physical_filename = nil
@@ -29,6 +52,10 @@ module Burner
 
           output.detail("Wrote to: #{physical_filename}")
 
+          return if supress_side_effect
+
+          output.detail("Saving to side effects: #{logical_filename}")
+
           side_effect = SideEffects::WrittenFile.new(
             logical_filename: logical_filename,
             physical_filename: physical_filename,

data/lib/burner/library/serialize/csv.rb

@@ -18,7 +18,7 @@ module Burner
       class Csv < JobWithRegister
         attr_reader :byte_order_mark
 
-        def initialize(name:, byte_order_mark: nil, register: DEFAULT_REGISTER)
+        def initialize(byte_order_mark: nil, name: '', register: DEFAULT_REGISTER)
           super(name: name, register: register)
 
           @byte_order_mark = Modeling::ByteOrderMark.resolve(byte_order_mark)

data/lib/burner/library/sleep.rb

@@ -15,7 +15,7 @@ module Burner
     class Sleep < Job
       attr_reader :seconds
 
-      def initialize(name:, seconds: 0)
+      def initialize(name: '', seconds: 0)
         super(name: name)
 
         @seconds = seconds.to_f

data/lib/burner/library/value/copy.rb

@@ -19,7 +19,7 @@ module Burner
       class Copy < Job
         attr_reader :from_register, :to_register
 
-        def initialize(name:, to_register: DEFAULT_REGISTER, from_register: DEFAULT_REGISTER)
+        def initialize(from_register: DEFAULT_REGISTER, name: '', to_register: DEFAULT_REGISTER)
           super(name: name)
 
           @from_register = from_register.to_s

data/lib/burner/library/value/nest.rb

@@ -0,0 +1,37 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+module Burner
+  module Library
+    module Value
+      # This job will nest the current value within a new outer hash.  The specified key
+      # passed in will be the corresponding new hash key entry for the existing value.
+      #
+      # Expected Payload[from_register] input: anything.
+      # Payload[to_register] output: hash.
+      class Nest < JobWithRegister
+        DEFAULT_KEY = 'key'
+
+        attr_reader :key
+
+        def initialize(key: DEFAULT_KEY, name: '', register: Burner::DEFAULT_REGISTER)
+          super(name: name, register: register)
+
+          @key = key.to_s
+
+          freeze
+        end
+
+        def perform(_output, payload)
+          payload[register] = { key => payload[register] }
+        end
+      end
+    end
+  end
+end

data/lib/burner/library/value/static.rb

@@ -17,7 +17,7 @@ module Burner
       class Static < JobWithRegister
         attr_reader :value
 
-        def initialize(name:, register: DEFAULT_REGISTER, value: nil)
+        def initialize(name: '', register: DEFAULT_REGISTER, value: nil)
           super(name: name, register: register)
 
           @value = value

data/lib/burner/library/value/transform.rb

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+module Burner
+  module Library
+    module Value
+      # Transform the current value of the register through a Realize::Pipeline.  This will
+      # transform the entire value, as opposed to the b/collection/transform job, which will
+      # iterate over each row/record in a dataset and transform each row/record.
+      #
+      # Expected Payload[register] input: anything.
+      # Payload[register] output: anything.
+      class Transform < JobWithRegister
+        attr_reader :pipeline
+
+        def initialize(name: '', register: DEFAULT_REGISTER, separator: '', transformers: [])
+          super(name: name, register: register)
+
+          resolver = Objectable.resolver(separator: separator)
+
+          @pipeline = Realize::Pipeline.new(transformers, resolver: resolver)
+
+          freeze
+        end
+
+        def perform(_output, payload)
+          payload[register] = pipeline.transform(payload[register], payload.time)
+        end
+      end
+    end
+  end
+end

data/lib/burner/pipeline.rb

@@ -7,7 +7,7 @@
 # LICENSE file in the root directory of this source tree.
 #
 
-require_relative 'jobs'
+require_relative 'job_set'
 require_relative 'output'
 require_relative 'payload'
 require_relative 'step'
@@ -19,27 +19,13 @@ module Burner
   class Pipeline
     acts_as_hashable
 
-    class JobNotFoundError < StandardError; end
-    class DuplicateJobNameError < StandardError; end
-
     attr_reader :steps
 
     def initialize(jobs: [], steps: nil)
-      jobs = Jobs.array(jobs)
-
-      assert_unique_job_names(jobs)
-
-      jobs_by_name = jobs.map { |job| [job.name, job] }.to_h
-
-      step_names = steps ? Array(steps) : jobs_by_name.keys
-
-      @steps = step_names.map do |step_name|
-        job = jobs_by_name[step_name.to_s]
-
-        raise JobNotFoundError, "#{step_name} was not declared as a job" unless job
-
-        Step.new(job)
-      end
+      @steps = JobSet
+               .new(jobs)
+               .jobs(steps)
+               .map { |job| Step.new(job) }
     end
 
     # The main entry-point for kicking off a pipeline.
@@ -68,20 +54,6 @@ module Burner
 
     private
 
-    def assert_unique_job_names(jobs)
-      unique_job_names = Set.new
-
-      jobs.each do |job|
-        if unique_job_names.include?(job.name)
-          raise DuplicateJobNameError, "job with name: #{job.name} already declared"
-        end
-
-        unique_job_names << job.name
-      end
-
-      nil
-    end
-
     def output_params(params, output)
       if params.keys.any?
         output.write('Parameters:')

data/lib/burner/util.rb

@@ -8,4 +8,5 @@
 #
 
 require_relative 'util/arrayable'
+require_relative 'util/keyable'
 require_relative 'util/string_template'

data/lib/burner/util/keyable.rb

@@ -0,0 +1,23 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+module Burner
+  module Util
+    # Provide helper methods for keys.
+    module Keyable
+      def make_key(record, keys, resolver, insensitive)
+        keys.map do |key|
+          value = resolver.get(record, key)
+
+          insensitive ? value.to_s.downcase : value
+        end
+      end
+    end
+  end
+end

data/lib/burner/version.rb

@@ -8,5 +8,5 @@
 #
 
 module Burner
-  VERSION = '1.4.0'
+  VERSION = '1.7.0-alpha'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: burner
 version: !ruby/object:Gem::Version
-  version: 1.4.0
+  version: 1.7.0.pre.alpha
 platform: ruby
 authors:
 - Matthew Ruggio
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-12-17 00:00:00.000000000 Z
+date: 2021-01-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: acts_as_hashable
@@ -237,6 +237,7 @@ files:
 - lib/burner/disks.rb
 - lib/burner/disks/local.rb
 - lib/burner/job.rb
+- lib/burner/job_set.rb
 - lib/burner/job_with_register.rb
 - lib/burner/jobs.rb
 - lib/burner/library.rb
@@ -246,12 +247,14 @@ files:
 - lib/burner/library/collection/graph.rb
 - lib/burner/library/collection/group.rb
 - lib/burner/library/collection/nested_aggregate.rb
+- lib/burner/library/collection/number.rb
 - lib/burner/library/collection/objects_to_arrays.rb
 - lib/burner/library/collection/shift.rb
 - lib/burner/library/collection/transform.rb
 - lib/burner/library/collection/unpivot.rb
 - lib/burner/library/collection/validate.rb
 - lib/burner/library/collection/values.rb
+- lib/burner/library/collection/zip.rb
 - lib/burner/library/compress/row_reader.rb
 - lib/burner/library/deserialize/csv.rb
 - lib/burner/library/deserialize/json.rb
@@ -268,7 +271,9 @@ files:
 - lib/burner/library/serialize/yaml.rb
 - lib/burner/library/sleep.rb
 - lib/burner/library/value/copy.rb
+- lib/burner/library/value/nest.rb
 - lib/burner/library/value/static.rb
+- lib/burner/library/value/transform.rb
 - lib/burner/modeling.rb
 - lib/burner/modeling/attribute.rb
 - lib/burner/modeling/attribute_renderer.rb
@@ -287,6 +292,7 @@ files:
 - lib/burner/step.rb
 - lib/burner/util.rb
 - lib/burner/util/arrayable.rb
+- lib/burner/util/keyable.rb
 - lib/burner/util/string_template.rb
 - lib/burner/version.rb
 homepage: https://github.com/bluemarblepayroll/burner
@@ -309,9 +315,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '2.5'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">="
+  - - ">"
     - !ruby/object:Gem::Version
-      version: '0'
+      version: 1.3.1
 requirements: []
 rubygems_version: 3.0.3
 signing_key: