burner 1.6.0.pre.alpha → 1.9.0.pre.alpha

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a5d81ad3d936aefde8cb52a299a1b28d8baf8ca40cfefee478a1c1e525578aec
-  data.tar.gz: f69f993fa1a51a4c5ed6e88a7225a1646c7ef47a38369b09680532854e961f84
+  metadata.gz: eca471b05b356ad1e96f71c0173673109622e14dc7d7d46bdab0beccc1e03396
+  data.tar.gz: 55f81d7080c1f0eb4a4cc1d51df7534376306362a1d97a8369f3fd85c0107a3b
 SHA512:
-  metadata.gz: 4c670ad08031fc43df3aa1b995828ce507aa213e294c3ac4cfdef6a8d187a53224ad31bb44a959bb72cc220f06e9094dea4af6301bee5fea3d375c241bd0b493
-  data.tar.gz: 11ef5e19a6160c0a9e943de848beb2b03ffa35bb5c8401eab9884cdddb5d4e43d2bd31ee4bc2f36479b28f7409afe32d7941f969a87f28743a50fceb7ec4342c
+  metadata.gz: bbd76fedb82b0499f7f8913e5530a1c35e2d4fc613ef11e95fb8ad6c759a28a7f4737d5ae29a71e0409dbd8f45b03de5b96dc22b84e3b85384cb256d1742eb9c
+  data.tar.gz: 311f81afb55fdbf5548c9394ec1ee30e020833f205e4e1c53fef87630c2e77d0b6cbd6639689efaa8956ae7bb188555f35ffb1dca7c08c2cedb6881f959cd508

data/.tool-versions

@@ -0,0 +1 @@
+ruby 2.6.6

data/CHANGELOG.md

@@ -1,5 +1,41 @@
+# 1.9.0 (TBD)
 
-# 1.6.0 (TBD)
+Added Jobs:
+
+* b/collection/pivot
+# 1.8.0 (March 31st, 2021)
+
+Added Jobs:
+
+* b/param/from_register
+* b/param/to_register
+
+Other:
+
+* Payload#param was added to access a param key's value.
+* Payload#update_param was added to update a param key's value.
+
+Internal Notes:
+
+Payload#register and Payload#params data stores have been internally consolidated while still maintaining the same public API surface area.
+
+# 1.7.0 (January 22nd, 2021)
+
+Added Jobs:
+
+* b/collection/number
+* b/value/nest
+* b/value/transform
+
+Enhanced Jobs:
+
+* b/collection/coalesce and b/collection/group now support the notion of case and type-insensitivity (insensitive option).
+
+Changes:
+
+* Job names derived from Burner::Job are now optional.  Pipelines themselves now can handle jobs without names.
+
+# 1.6.0 (December 22nd, 2020)
 
 Additions:
 

data/README.md

@@ -93,6 +93,54 @@ Some notes:
 * Jobs can be re-used (just like the output_id and output_value jobs).
 * If steps is nil then all jobs will execute in their declared order.
 
+### Omitting Job Names and Steps
+
+Job names are optional, but steps can only correspond to named jobs.  This means if steps is declared then anonymous jobs will have no way to be executed.  Here is the same pipeline as above, but without job names and steps:
+
+````ruby
+pipeline = {
+  jobs: [
+    {
+      type: 'b/io/read',
+      path: '{input_file}'
+    },
+    {
+      type: 'b/echo',
+      message: 'The job id is: {__id}'
+    },
+    {
+      type: 'b/echo',
+      message: 'The current value is: {__default_register}'
+    },
+    {
+      type: 'b/deserialize/json'
+    },
+    {
+      type: 'b/serialize/yaml'
+    },
+    {
+      type: 'b/echo',
+      message: 'The current value is: {__default_register}'
+    },
+    {
+      type: 'b/io/write',
+      path: '{output_file}'
+    }
+  ]
+}
+
+params = {
+  input_file: 'input.json',
+  output_file: 'output.yaml'
+}
+
+payload = Burner::Payload.new(params: params)
+
+Burner::Pipeline.make(pipeline).execute(payload: payload)
+````
+
+Like everything in software, there are trade-offs to the above two equivalent pipelines.  The former (one with steps and job names) has less jobs but is more verbose.  The latter (without steps and job names) has more jobs but reads terser.  Names also can aid in self-documenting your code/configuration so it may be a good idea to enforce at least names are used.
+
 ### Capturing Feedback / Output
 
 By default, output will be emitted to `$stdout`.  You can add or change listeners by passing in optional values into Pipeline#execute.  For example, say we wanted to capture the output from our json-to-yaml example:
@@ -216,12 +264,15 @@ This library only ships with very basic, rudimentary jobs that are meant to just
 #### Collection
 
 * **b/collection/arrays_to_objects** [mappings, register]: Convert an array of arrays to an array of objects.
-* **b/collection/coalesce** [register, grouped_register, key_mappings, keys, separator]: Merge two datasets together based on the key values of one dataset (array) with a grouped dataset (hash).
+* **b/collection/coalesce** [grouped_register, insensitive, key_mappings, keys, register, separator]: Merge two datasets together based on the key values of one dataset (array) with a grouped dataset (hash).  If insensitive (false by default) is true then each key's value will be converted/coerced to a lowercase string.
 * **b/collection/concatenate** [from_registers, to_register]: Concatenate each from_register's value and place the newly concatenated array into the to_register.  Note: this does not do any deep copying and should be assumed it is shallow copying all objects.
 * **b/collection/graph** [config, key, register]: Use [Hashematics](https://github.com/bluemarblepayroll/hashematics) to turn a flat array of objects into a deeply nested object tree.
-* **b/collection/group** [keys, register, separator]: Take a register's value (an array of objects) and group the objects by the specified keys.
+* **b/collection/group** [insensitive, keys, register, separator]: Take a register's value (an array of objects) and group the objects by the specified keys.  If insensitive (false by default) is true then each key's value will be converted/coerced to a lowercase string.
 * **b/collection/nested_aggregate** [register, key_mappings, key, separator]: Traverse a set of objects, resolving key's value for each object, optionally copying down key_mappings to the child records, then merging all the inner records together.
+* **b/collection/number** [key, register, separator, start_at]: This job can iterate over a set of records and sequence them (set the specified key to a sequential index value.)
 * **b/collection/objects_to_arrays** [mappings, register]: Convert an array of objects to an array of arrays.
+* **b/collection/pivot** [unique_keys, insensitive, other_keys, pivot_key, pivot_value_key, register, separator]:
+Take an array of objects and pivot a key into multiple keys.  It essentially takes all the values for a key and creates N number of keys (one per value.) Under the hood it uses HashMath's [Record](https://github.com/bluemarblepayroll/hash_math#record-the-hash-prototype) and [Table](https://github.com/bluemarblepayroll/hash_math#table-the-double-hash-hash-of-hashes) classes.
 * **b/collection/shift** [amount, register]: Remove the first N number of elements from an array.
 * **b/collection/transform** [attributes, exclusive, separator, register]: Iterate over all objects and transform each key per the attribute transformers specifications.  If exclusive is set to false then the current object will be overridden/merged.  Separator can also be set for key path support.  This job uses [Realize](https://github.com/bluemarblepayroll/realize), which provides its own extendable value-transformation pipeline.  If an attribute is not set with `explicit: true` then it will automatically start from the key's value from the record.  If `explicit: true` is started, then it will start from the record itself.
 * **b/collection/unpivot** [pivot_set, register]: Take an array of objects and unpivot specific sets of keys into rows.  Under the hood it uses [HashMath's Unpivot class](https://github.com/bluemarblepayroll/hash_math#unpivot-hash-key-coalescence-and-row-extrapolation).
@@ -248,6 +299,11 @@ By default all jobs will use the `Burner::Disks::Local` disk for its persistence
 * **b/io/row_reader** [data_key, disk, ignore_blank_path, ignore_file_not_found, path_key, register, separator]: Iterates over an array of objects, extracts a filepath from a key in each object, and attempts to load the file's content for each record.  The file's content will be stored at the specified data_key. By default missing paths or files will be treated as hard errors.  If you wish to ignore these then pass in true for ignore_blank_path and/or ignore_file_not_found.
 * **b/io/write** [binary, disk, path, register, supress_side_effect]: Write to a local file.  The path parameter can be interpolated using `Payload#params`.  If the contents are binary, pass in `binary: true` to open it up in binary+write mode.  By default, written files are also logged as WrittenFile instances to the Payload#side_effects array.  You can pass in supress_side_effect: true to disable this behavior.
 
+#### Parameters
+
+* **b/param/from_register** [param_key, register]: Copy the value of a register to a param key.
+* **b/param/to_register** [param_key, register]: Copy the value of a param key to a register.
+
 #### Serialization
 
 * **b/serialize/csv** [byte_order_mark, register]: Take an array of arrays and create a CSV.  You can optionally pre-pend a byte order mark, see Burner::Modeling::ByteOrderMark for acceptable options.
@@ -257,7 +313,9 @@ By default all jobs will use the `Burner::Disks::Local` disk for its persistence
 #### Value
 
 * **b/value/copy** [from_register, to_register]: Copy from_register's value into the to_register.  Note: this does not do any deep copying and should be assumed it is shallow copying all objects.
+* **b/value/nest** [key, register]: This job will nest the current value within a new outer hash.  The specified key passed in will be the corresponding new hash key entry for the existing value.
 * **b/value/static** [register, value]: Set the value to any arbitrary value.
+* **b/value/transform** [register, separator, transformers]: Transform the current value of the register through a Realize::Pipeline.  This will transform the entire value, as opposed to the b/collection/transform job, which will iterate over each row/record in a dataset and transform each row/record.
 
 #### General
 

data/lib/burner/data.rb

@@ -0,0 +1,46 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+module Burner
+  # Defines a key value pair data store per our library.  It is basically a composite
+  # object around a hash with indifferent key typing.
+  class Data
+    extend Forwardable
+
+    def_delegators :internal_hash, :transform_keys
+
+    def initialize(hash = {})
+      @internal_hash = {}
+
+      (hash || {}).each { |k, v| self[k] = v }
+    end
+
+    def []=(key, value)
+      internal_hash[key.to_s] = value
+    end
+
+    def [](key)
+      internal_hash[key.to_s]
+    end
+
+    def to_h
+      internal_hash
+    end
+
+    def ==(other)
+      other.instance_of?(self.class) &&
+        to_h == other.to_h
+    end
+    alias eql? ==
+
+    private
+
+    attr_reader :internal_hash
+  end
+end

data/lib/burner/job.rb

@@ -18,9 +18,7 @@ module Burner
 
     attr_reader :name
 
-    def initialize(name:)
-      raise ArgumentError, 'name is required' if name.to_s.empty?
-
+    def initialize(name: '')
       @name = name.to_s
     end
 
@@ -48,15 +46,9 @@ module Burner
     protected
 
     def job_string_template(expression, output, payload)
-      templatable_params = payload.params
-                                  .merge(__id: output.id)
-                                  .merge(templatable_register_values(payload))
+      templatable_params = payload.params_and_registers_hash.merge(__id: output.id)
 
       Util::StringTemplate.instance.evaluate(expression, templatable_params)
     end
-
-    def templatable_register_values(payload)
-      payload.registers.transform_keys { |key| "__#{key}_register" }
-    end
   end
 end

data/lib/burner/job_set.rb

@@ -0,0 +1,64 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+require_relative 'jobs'
+
+module Burner
+  # This class understands how jobs fit together as a cohesive unit.  It does not know how to
+  # use them, but it knows how to group them together in a logical manner following some simple
+  # rules, such as:
+  #   - Jobs in a set should have unique names (unless the name is blank)
+  #   - Subsets of jobs can be extracted, by name, in constant time.
+  class JobSet
+    class DuplicateJobNameError < StandardError; end
+    class JobNotFoundError < StandardError; end
+
+    def initialize(jobs = [])
+      @jobs = Jobs.array(jobs).freeze
+
+      assert_unique_job_names
+    end
+
+    def jobs(names = nil)
+      return @jobs unless names
+
+      Array(names).map do |name|
+        job = named_jobs_by_name[name.to_s]
+
+        raise JobNotFoundError, "#{name} was not declared as a job" unless job
+
+        job
+      end
+    end
+
+    private
+
+    def named_jobs_by_name
+      @named_jobs_by_name ||= named_jobs.each_with_object({}) { |job, memo| memo[job.name] = job }
+    end
+
+    def named_jobs
+      @named_jobs ||= @jobs.reject { |job| job.name == '' }
+    end
+
+    def assert_unique_job_names
+      unique_job_names = Set.new
+
+      named_jobs.each do |job|
+        if unique_job_names.include?(job.name)
+          raise DuplicateJobNameError, "job with name: #{job.name} already declared"
+        end
+
+        unique_job_names << job.name
+      end
+
+      nil
+    end
+  end
+end

data/lib/burner/job_with_register.rb

@@ -15,10 +15,17 @@ module Burner
   class JobWithRegister < Job
     attr_reader :register
 
-    def initialize(name:, register: DEFAULT_REGISTER)
+    def initialize(name: '', register: DEFAULT_REGISTER)
       super(name: name)
 
       @register = register.to_s
     end
+
+    protected
+
+    # Helper method that knows how to ensure the register is an array.
+    def ensure_array(payload)
+      payload[register] = array(payload[register])
+    end
   end
 end

data/lib/burner/jobs.rb

@@ -28,7 +28,9 @@ module Burner
     register 'b/collection/graph',             Library::Collection::Graph
     register 'b/collection/group',             Library::Collection::Group
     register 'b/collection/nested_aggregate',  Library::Collection::NestedAggregate
+    register 'b/collection/number',            Library::Collection::Number
     register 'b/collection/objects_to_arrays', Library::Collection::ObjectsToArrays
+    register 'b/collection/pivot',             Library::Collection::Pivot
     register 'b/collection/shift',             Library::Collection::Shift
     register 'b/collection/transform',         Library::Collection::Transform
     register 'b/collection/unpivot',           Library::Collection::Unpivot
@@ -47,11 +49,16 @@ module Burner
     register 'b/io/row_reader',                Library::IO::RowReader
     register 'b/io/write',                     Library::IO::Write
 
+    register 'b/param/from_register',          Library::Param::FromRegister
+    register 'b/param/to_register',            Library::Param::ToRegister
+
     register 'b/serialize/csv',                Library::Serialize::Csv
     register 'b/serialize/json',               Library::Serialize::Json
     register 'b/serialize/yaml',               Library::Serialize::Yaml
 
     register 'b/value/copy',                   Library::Value::Copy
+    register 'b/value/nest',                   Library::Value::Nest
     register 'b/value/static',                 Library::Value::Static
+    register 'b/value/transform',              Library::Value::Transform
   end
 end

data/lib/burner/library.rb

@@ -19,7 +19,9 @@ require_relative 'library/collection/concatenate'
 require_relative 'library/collection/graph'
 require_relative 'library/collection/group'
 require_relative 'library/collection/nested_aggregate'
+require_relative 'library/collection/number'
 require_relative 'library/collection/objects_to_arrays'
+require_relative 'library/collection/pivot'
 require_relative 'library/collection/shift'
 require_relative 'library/collection/transform'
 require_relative 'library/collection/unpivot'
@@ -38,9 +40,14 @@ require_relative 'library/io/read'
 require_relative 'library/io/row_reader'
 require_relative 'library/io/write'
 
+require_relative 'library/param/from_register'
+require_relative 'library/param/to_register'
+
 require_relative 'library/serialize/csv'
 require_relative 'library/serialize/json'
 require_relative 'library/serialize/yaml'
 
 require_relative 'library/value/copy'
+require_relative 'library/value/nest'
 require_relative 'library/value/static'
+require_relative 'library/value/transform'

data/lib/burner/library/collection/arrays_to_objects.rb

@@ -55,7 +55,7 @@ module Burner
       class ArraysToObjects < JobWithRegister
         attr_reader :mappings
 
-        def initialize(name:, mappings: [], register: DEFAULT_REGISTER)
+        def initialize(mappings: [], name: '', register: DEFAULT_REGISTER)
           super(name: name, register: register)
 
           @mappings = Modeling::KeyIndexMapping.array(mappings)

data/lib/burner/library/collection/coalesce.rb

@@ -18,19 +18,27 @@ module Burner
       # Expected Payload[register] input: array of objects.
       # Payload[register] output: array of objects.
       class Coalesce < JobWithRegister
-        attr_reader :grouped_register, :key_mappings, :keys, :resolver
+        include Util::Keyable
+
+        attr_reader :grouped_register,
+                    :insensitive,
+                    :key_mappings,
+                    :keys,
+                    :resolver
 
         def initialize(
-          name:,
           grouped_register:,
+          insensitive: false,
           key_mappings: [],
           keys: [],
+          name: '',
           register: DEFAULT_REGISTER,
           separator: ''
         )
           super(name: name, register: register)
 
           @grouped_register = grouped_register.to_s
+          @insensitive      = insensitive || false
           @key_mappings     = Modeling::KeyMapping.array(key_mappings)
           @keys             = Array(keys)
           @resolver         = Objectable.resolver(separator: separator.to_s)
@@ -41,13 +49,14 @@ module Burner
         end
 
         def perform(output, payload)
-          payload[register] = array(payload[register])
-          count             = payload[register].length
+          ensure_array(payload)
+
+          count = payload[register].length
 
           output.detail("Coalescing based on key(s): #{keys} for #{count} records(s)")
 
           payload[register].each do |record|
-            key    = make_key(record)
+            key    = make_key(record, keys, resolver, insensitive)
             lookup = find_lookup(payload, key)
 
             key_mappings.each do |key_mapping|
@@ -63,10 +72,6 @@ module Burner
         def find_lookup(payload, key)
           (payload[grouped_register] || {})[key] || {}
         end
-
-        def make_key(record)
-          keys.map { |key| resolver.get(record, key) }
-        end
       end
     end
   end

data/lib/burner/library/collection/concatenate.rb

@@ -18,7 +18,7 @@ module Burner
       class Concatenate < Job
         attr_reader :from_registers, :to_register
 
-        def initialize(name:, from_registers: [], to_register: DEFAULT_REGISTER)
+        def initialize(from_registers: [], name: '', to_register: DEFAULT_REGISTER)
           super(name: name)
 
           @from_registers = Array(from_registers)