burner 1.1.0 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f1dd679013fe833b143b12340e68d5598ce4a3d53a6d5fcc6dc8c9df02bc7348
-  data.tar.gz: e347add004c9846a9d7e083d4d7f35b6aa664f0eb340b3bc0eceffcd3c0875e5
+  metadata.gz: a2c3e9af2bbfbf80cd5c5884e05b1282f52871566103baf4ca1b599fc76af8ca
+  data.tar.gz: e7c2eb8a00086e1937a4d9e39383d38a60c5ebaa5efa1d67da4e6af9741261cd
 SHA512:
-  metadata.gz: a92860ad5611e298eb1f2d1acaca9271324402ee375a2d842ac32a41f89c23a12abd8913596ee2fbe7fd59e071fd1b7ab77bbfc5317c57d2ca342dc75f6091a2
-  data.tar.gz: cb529e0a81f7a3bef34e9966463ca8f71b2025101ca4f6a6bd0cef03396fcb2a6ceb6bad7681079e4edc0630f21e5d6c19f7b6489fb2d198b716a93dfd5c3938
+  metadata.gz: e885c7bf710f613323bbc3fc49162815632d7785aa6e753f3cd53bb04d2b53e74f92be549f75b6997d4dedce5c2dabe01fd032796a6d6e2fadfd6e84c633b168
+  data.tar.gz: e09d491b3fb7ef1b79932ac6094adedd74f806170a2a11aa1ff1eda838b6bf76cec1684dbfaba12d07a6e071062d432bcd5f3939f275830fc13fb71cabb39f8c

data/CHANGELOG.md

@@ -1,3 +1,13 @@
+# 1.2.0 (November 25th, 2020)
+
+#### Enhancements:
+
+* All for a pipeline to be configured with null steps.  When null, just execute all jobs in positional order.
+* Allow Collection::Transform job attributes to implicitly start from a resolve transformer.  `explicit: true` can be passed in as an option in case the desire is to begin from the record and not a specific value.
+
+#### Added Jobs:
+
+* b/collection/nested_aggregate
 # 1.1.0 (November 16, 2020)
 
 Added Jobs:

data/README.md

@@ -91,38 +91,20 @@ Some notes:
 * The job's ID can be accessed using the `__id` key.
 * The current job's payload value can be accessed using the `__value` key.
 * Jobs can be re-used (just like the output_id and output_value jobs).
+* If steps is nil then all jobs will execute in their declared order.
 
 ### Capturing Feedback / Output
 
 By default, output will be emitted to `$stdout`.  You can add or change listeners by passing in optional values into Pipeline#execute.  For example, say we wanted to capture the output from our json-to-yaml example:
 
 ````ruby
-class StringOut
-  def initialize
-    @io = StringIO.new
-  end
-
-  def puts(msg)
-    tap { io.write("#{msg}\n") }
-  end
-
-  def read
-    io.rewind
-    io.read
-  end
-
-  private
-
-  attr_reader :io
-end
-
-string_out = StringOut.new
-output     = Burner::Output.new(outs: string_out)
-payload    = Burner::Payload.new(params: params)
+io      = StringIO.new
+output  = Burner::Output.new(outs: io)
+payload = Burner::Payload.new(params: params)
 
 Burner::Pipeline.make(pipeline).execute(output: output, payload: payload)
 
-log = string_out.read
+log = io.string
 ````
 
 The value of `log` should now look similar to:
@@ -238,9 +220,10 @@ This library only ships with very basic, rudimentary jobs that are meant to just
 * **b/collection/concatenate** [from_registers, to_register]: Concatenate each from_register's value and place the newly concatenated array into the to_register.  Note: this does not do any deep copying and should be assumed it is shallow copying all objects.
 * **b/collection/graph** [config, key, register]: Use [Hashematics](https://github.com/bluemarblepayroll/hashematics) to turn a flat array of objects into a deeply nested object tree.
 * **b/collection/group** [keys, register, separator]: Take a register's value (an array of objects) and group the objects by the specified keys.
+* **b/collection/nested_aggregate** [register, key_mappings, key, separator]: Traverse a set of objects, resolving key's value for each object, optionally copying down key_mappings to the child records, then merging all the inner records together.
 * **b/collection/objects_to_arrays** [mappings, register]: Convert an array of objects to an array of arrays.
 * **b/collection/shift** [amount, register]: Remove the first N number of elements from an array.
-* **b/collection/transform** [attributes, exclusive, separator, register]: Iterate over all objects and transform each key per the attribute transformers specifications.  If exclusive is set to false then the current object will be overridden/merged.  Separator can also be set for key path support.  This job uses [Realize](https://github.com/bluemarblepayroll/realize), which provides its own extendable value-transformation pipeline.
+* **b/collection/transform** [attributes, exclusive, separator, register]: Iterate over all objects and transform each key per the attribute transformers specifications.  If exclusive is set to false then the current object will be overridden/merged.  Separator can also be set for key path support.  This job uses [Realize](https://github.com/bluemarblepayroll/realize), which provides its own extendable value-transformation pipeline.  If an attribute is not set with `explicit: true` then it will automatically start from the key's value from the record.  If `explicit: true` is started, then it will start from the record itself.
 * **b/collection/unpivot** [pivot_set, register]: Take an array of objects and unpivot specific sets of keys into rows.  Under the hood it uses [HashMath's Unpivot class](https://github.com/bluemarblepayroll/hash_math#unpivot-hash-key-coalescence-and-row-extrapolation).
 * **b/collection/validate** [invalid_register, join_char, message_key, register, separator, validations]: Take an array of objects, run it through each declared validator, and split the objects into two registers.  The valid objects will be split into the current register while the invalid ones will go into the invalid_register as declared.  Optional arguments, join_char and message_key, help determine the compiled error messages.  The separator option can be utilized to use dot-notation for validating keys.  See each validation's options by viewing their classes within the `lib/modeling/validations` directory.
 * **b/collection/values** [include_keys, register]: Take an array of objects and call `#values` on each object. If include_keys is true (it is false by default), then call `#keys` on the first object and inject that as a "header" object.

data/lib/burner/jobs.rb

@@ -27,6 +27,7 @@ module Burner
     register 'b/collection/concatenate',       Library::Collection::Concatenate
     register 'b/collection/graph',             Library::Collection::Graph
     register 'b/collection/group',             Library::Collection::Group
+    register 'b/collection/nested_aggregate',  Library::Collection::NestedAggregate
     register 'b/collection/objects_to_arrays', Library::Collection::ObjectsToArrays
     register 'b/collection/shift',             Library::Collection::Shift
     register 'b/collection/transform',         Library::Collection::Transform

data/lib/burner/library.rb

@@ -18,6 +18,7 @@ require_relative 'library/collection/coalesce'
 require_relative 'library/collection/concatenate'
 require_relative 'library/collection/graph'
 require_relative 'library/collection/group'
+require_relative 'library/collection/nested_aggregate'
 require_relative 'library/collection/objects_to_arrays'
 require_relative 'library/collection/shift'
 require_relative 'library/collection/transform'

data/lib/burner/library/collection/nested_aggregate.rb

@@ -0,0 +1,67 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+module Burner
+  module Library
+    module Collection
+      # Iterate over a collection of objects, calling key on each object, then aggregating the
+      # returns of key together into one array.  This new derived array will be set as the value
+      # for the payload's register.  Leverage the key_mappings option to optionally copy down
+      # keys and values from outer to inner records.  This job is particularly useful
+      # if you have nested arrays but wish to deal with each level/depth in the aggregate.
+      #
+      # Expected Payload[register] input: array of objects.
+      # Payload[register] output: array of objects.
+      class NestedAggregate < JobWithRegister
+        attr_reader :key, :key_mappings, :resolver
+
+        def initialize(name:, key:, key_mappings: [], register: DEFAULT_REGISTER, separator: '')
+          super(name: name, register: register)
+
+          raise ArgumentError, 'key is required' if key.to_s.empty?
+
+          @key          = key.to_s
+          @key_mappings = Modeling::KeyMapping.array(key_mappings)
+          @resolver     = Objectable.resolver(separator: separator.to_s)
+
+          freeze
+        end
+
+        def perform(output, payload)
+          records = array(payload[register])
+          count   = records.length
+
+          output.detail("Aggregating on key: #{key} for #{count} records(s)")
+
+          # Outer loop on parent records
+          payload[register] = records.each_with_object([]) do |record, memo|
+            inner_records = resolver.get(record, key)
+
+            # Inner loop on child records
+            array(inner_records).each do |inner_record|
+              memo << copy_key_mappings(record, inner_record)
+            end
+          end
+        end
+
+        private
+
+        def copy_key_mappings(source_record, destination_record)
+          key_mappings.each do |key_mapping|
+            value = resolver.get(source_record, key_mapping.from)
+
+            resolver.set(destination_record, key_mapping.to, value)
+          end
+
+          destination_record
+        end
+      end
+    end
+  end
+end

data/lib/burner/modeling/attribute.rb

@@ -13,19 +13,37 @@ module Burner
     # to set the key to.  The transformers that can be passed in can be any Realize::Transformers
     # subclasses.  For more information, see the Realize library at:
     # https://github.com/bluemarblepayroll/realize
+    #
+    # Note that if explicit: true is set then no transformers will be automatically injected.
+    # If explicit is not true (default) then it will have a resolve job automatically injected
+    # in the beginning of the pipeline.  This is the observed default behavior, with the
+    # exception having to be initially cross-mapped using a custom resolve transformation.
     class Attribute
       acts_as_hashable
 
+      RESOLVE_TYPE = 'r/value/resolve'
+
       attr_reader :key, :transformers
 
-      def initialize(key:, transformers: [])
+      def initialize(key:, explicit: false, transformers: [])
         raise ArgumentError, 'key is required' if key.to_s.empty?
 
         @key          = key.to_s
-        @transformers = Realize::Transformers.array(transformers)
+        @transformers = base_transformers(explicit) + Realize::Transformers.array(transformers)
 
         freeze
       end
+
+      private
+
+      # When explicit, this will return an empty array.
+      # When not explicit, this will return an array with a basic transformer that simply
+      # gets the key's value.  This establishes a good majority base case.
+      def base_transformers(explicit)
+        return [] if explicit
+
+        [Realize::Transformers.make(type: RESOLVE_TYPE, key: key)]
+      end
     end
   end
 end

data/lib/burner/modeling/key_mapping.rb

@@ -9,18 +9,18 @@
 
 module Burner
   module Modeling
-    # Generic mapping from a key to another key.
+    # Generic mapping from a key to another key.  The argument 'to' is optional
+    # and if it is blank then the 'from' value will be used for the 'to' as well.
     class KeyMapping
       acts_as_hashable
 
       attr_reader :from, :to
 
-      def initialize(from:, to:)
+      def initialize(from:, to: '')
         raise ArgumentError, 'from is required' if from.to_s.empty?
-        raise ArgumentError, 'to is required'   if to.to_s.empty?
 
         @from = from.to_s
-        @to   = to.to_s
+        @to   = to.to_s.empty? ? @from : to.to_s
 
         freeze
       end

data/lib/burner/pipeline.rb

@@ -14,7 +14,8 @@ require_relative 'step'
 
 module Burner
   # The root package.  A Pipeline contains the job configurations along with the steps.  The steps
-  # referens jobs and tell you the order of the jobs to run.
+  # reference jobs and tell you the order of the jobs to run.  If steps is nil then all jobs
+  # will execute in their declared order.
   class Pipeline
     acts_as_hashable
 
@@ -23,14 +24,16 @@ module Burner
 
     attr_reader :steps
 
-    def initialize(jobs: [], steps: [])
+    def initialize(jobs: [], steps: nil)
       jobs = Jobs.array(jobs)
 
       assert_unique_job_names(jobs)
 
       jobs_by_name = jobs.map { |job| [job.name, job] }.to_h
 
-      @steps = Array(steps).map do |step_name|
+      step_names = steps ? Array(steps) : jobs_by_name.keys
+
+      @steps = step_names.map do |step_name|
         job = jobs_by_name[step_name.to_s]
 
         raise JobNotFoundError, "#{step_name} was not declared as a job" unless job

data/lib/burner/version.rb

@@ -8,5 +8,5 @@
 #
 
 module Burner
-  VERSION = '1.1.0'
+  VERSION = '1.2.0'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: burner
 version: !ruby/object:Gem::Version
-  version: 1.1.0
+  version: 1.2.0
 platform: ruby
 authors:
 - Matthew Ruggio
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-11-16 00:00:00.000000000 Z
+date: 2020-11-25 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: acts_as_hashable
@@ -229,6 +229,7 @@ files:
 - lib/burner/library/collection/concatenate.rb
 - lib/burner/library/collection/graph.rb
 - lib/burner/library/collection/group.rb
+- lib/burner/library/collection/nested_aggregate.rb
 - lib/burner/library/collection/objects_to_arrays.rb
 - lib/burner/library/collection/shift.rb
 - lib/burner/library/collection/transform.rb