burner 1.0.0.pre.alpha → 1.0.0.pre.alpha.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 920f3b0f0027e21b30da75f86335d75d2d351f49c8b68662de7226ade43474c6
-  data.tar.gz: b7f4189e37023f1d627ea5222df0f397a24a69c7ce3721b13ce714d2e274ee94
+  metadata.gz: 57b5b4290b72962e10ce7ea7244a1dcfb43894cff1d72ef4e0684de4cdea4a35
+  data.tar.gz: 2936c408e7ffa3e1e9883510d890329870d5c1a4c129fef1fb283103568c9508
 SHA512:
-  metadata.gz: d488edf32a95e5da64190c512f365d76e2414d16712cee9b56a400bd62e11ed7824a316a95923af15f0ecac3390e4be359d2e31ffc33fb13129309c4efce9403
-  data.tar.gz: 8640cfdb2fa4241fb2a493c2ac2aed62e08e1826446907f4b16d043276fffee656c787d2079545fe03d52b8f45756240a523ad878fc89063884868c3f94bfecd
+  metadata.gz: f514870aa2b12cc4fc34f3952cabf8c738985dac1a6602c7f41aec3f28b83738991d5cf82d429b62d76b0f3a96b85907e51d0e06db0ce33579d1dc74dc55409e
+  data.tar.gz: 2e9ee10f91bb28eb4d79091ae7106f6da640daaf6b5ac2f77e828e81743dfd55d6baca12a0e376c3be709483fe7a94e0f840f47ed2ecb5031233f4ff9ae7db3a

data/README.md

@@ -2,7 +2,7 @@
 
 [![Gem Version](https://badge.fury.io/rb/burner.svg)](https://badge.fury.io/rb/burner) [![Build Status](https://travis-ci.org/bluemarblepayroll/burner.svg?branch=master)](https://travis-ci.org/bluemarblepayroll/burner) [![Maintainability](https://api.codeclimate.com/v1/badges/dbc3757929b67504f6ca/maintainability)](https://codeclimate.com/github/bluemarblepayroll/burner/maintainability) [![Test Coverage](https://api.codeclimate.com/v1/badges/dbc3757929b67504f6ca/test_coverage)](https://codeclimate.com/github/bluemarblepayroll/burner/test_coverage) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-TODO
+This library serves as the skeleton for a processing engine.  It allows you to organize your code into Jobs, then stitch those jobs together as steps.
 
 ## Installation
 
@@ -20,7 +20,325 @@ bundle add burner
 
 ## Examples
 
-TODO
+The purpose of this library is to provide a framework for creating highly de-coupled functions (known as jobs), and then allow for the stitching of them back together in any arbitrary order (know as steps.)  Although our example will be somewhat specific and contrived, the only limit to what the jobs and order of jobs are is up to your imagination.
+
+### JSON-to-YAML File Converter
+
+All the jobs for this example are shipped with this library.  In this example, we will write a pipeline that can read a JSON file and convert it to YAML.  Pipelines are data-first so we can represent a pipeline using a hash:
+
+````ruby
+pipeline = {
+  jobs: [
+    {
+      name: :read,
+      type: 'io/read',
+      path: '{input_file}'
+    },
+    {
+      name: :output_id,
+      type: :echo,
+      message: 'The job id is: {__id}'
+    },
+    {
+      name: :output_value,
+      type: :echo,
+      message: 'The current value is: {__value}'
+    },
+    {
+      name: :parse,
+      type: 'deserialize/json'
+    },
+    {
+      name: :convert,
+      type: 'serialize/yaml'
+    },
+    {
+      name: :write,
+      type: 'io/write',
+      path: '{output_file}'
+    }
+  ],
+  steps: %i[
+    read
+    output_id
+    output_value
+    parse
+    convert
+    output_value
+    write
+  ]
+}
+
+params = {
+  input_file: 'input.json',
+  output_file: 'output.yaml'
+}
+
+payload = Burner::Payload.new(params: params)
+````
+
+Assuming we are running this script from a directory where an `input.json` file exists, we can then programatically process the pipeline:
+
+````ruby
+Burner::Pipeline.make(pipeline).execute(payload: payload)
+````
+
+We should now see a output.yaml file created.
+
+Some notes:
+
+* Some values are able to be string-interpolated using the provided Payload#params.  This allows for the passing runtime configuration/data into pipelines/jobs.
+* The job's ID can be accessed using the `__id` key.
+* The current job's payload value can be accessed using the `__value` key.
+* Jobs can be re-used (just like the output_id and output_value jobs).
+
+### Capturing Feedback / Output
+
+By default, output will be emitted to `$stdout`.  You can add or change listeners by passing in optional values into Pipeline#execute.  For example, say we wanted to capture the output from our json-to-yaml example:
+
+````ruby
+class StringOut
+  def initialize
+    @io = StringIO.new
+  end
+
+  def puts(msg)
+    tap { io.write("#{msg}\n") }
+  end
+
+  def read
+    io.rewind
+    io.read
+  end
+
+  private
+
+  attr_reader :io
+end
+
+string_out = StringOut.new
+output     = Burner::Output.new(outs: string_out)
+payload    = Burner::Payload.new(params: params)
+
+Burner::Pipeline.make(pipeline).execute(output: output, payload: payload)
+
+log = string_out.read
+````
+
+The value of `log` should now look similar to:
+
+````bash
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] Pipeline started with 7 step(s)
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] Parameters:
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - input_file: input.json
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - output_file: output.yaml
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] --------------------------------------------------------------------------------
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] [1] Burner::Jobs::IO::Read::read
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - Reading: spec/fixtures/input.json
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - Completed in: 0.0 second(s)
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] [2] Burner::Jobs::Echo::output_id
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - The job id is:
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - Completed in: 0.0 second(s)
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] [3] Burner::Jobs::Echo::output_value
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - The current value is:
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - Completed in: 0.0 second(s)
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] [4] Burner::Jobs::Deserialize::Json::parse
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - Completed in: 0.0 second(s)
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] [5] Burner::Jobs::Serialize::Yaml::convert
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - Completed in: 0.0 second(s)
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] [6] Burner::Jobs::Echo::output_value
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - The current value is:
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - Completed in: 0.0 second(s)
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] [7] Burner::Jobs::IO::Write::write
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - Writing: output.yaml
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - Completed in: 0.0 second(s)
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] --------------------------------------------------------------------------------
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] Pipeline ended, took 0.001 second(s) to complete
+````
+
+Notes:
+
+* The Job ID is specified as the leading UUID in each line.
+* `outs` can be provided an array of listeners, as long as each listener responds to `puts(msg)`.
+
+### Command Line Pipeline Processing
+
+This library also ships with a built-in script `exe/burner` that illustrates using the `Burner::Cli` API.  This class can take in an array of arguments (similar to a command-line) and execute a pipeline.  The first argument is the path to a YAML file with the pipeline's configuration and each subsequent argument is a param in `key=value` form.  Here is how the json-to-yaml example can utilize this interface:
+
+#### Create YAML Pipeline Configuration File
+
+Write the following json_to_yaml_pipeline.yaml file to disk:
+
+````yaml
+jobs:
+  - name: read
+    type: io/read
+    path: '{input_file}'
+
+  - name: output_id
+    type: echo
+    message: 'The job id is: {__id}'
+
+  - name: output_value
+    type: echo
+    message: 'The current value is: {__value}'
+
+  - name: parse
+    type: deserialize/json
+
+  - name: convert
+    type: serialize/yaml
+
+  - name: write
+    type: io/write
+    path: '{output_file}'
+
+steps:
+  - read
+  - output_id
+  - output_value
+  - parse
+  - convert
+  - output_value
+  - write
+````
+
+#### Run Using Script
+
+From the command-line, run:
+
+````bash
+bundle exec burner json_to_yaml_pipeline.yaml input_file=input.json output_file=output.yaml
+````
+
+The pipeline should be processed and output.yaml should be created.
+
+#### Run Using Programmatic API
+
+Instead of the script, you can invoke it using code:
+
+````ruby
+args = %w[
+  json_to_yaml_pipeline.yaml
+  input_file=input.json
+  output_file=output.yaml
+]
+
+Burner::Cli.new(args).invoke
+````
+
+### Core Job Library
+
+This library only ships with very basic, rudimentary jobs that are meant to just serve as a baseline:
+
+#### Collection
+
+* **collection/arrays_to_objects** [mappings]: Convert an array of arrays to an array of objects.
+* **collection/graph** [config, key]: Use (Hashematics)[https://github.com/bluemarblepayroll/hashematics] to turn a flat array of objects into a deeply nested object tree.
+* **collection/objects_to_arrays** [mappings]: Convert an array of objects to an array of arrays.
+* **collection/shift** [amount]: Remove the first N number of elements from an array.
+* **collection/transform** [attributes, exclusive, separator]: Iterate over all objects and transform each key per the attribute transformers specifications.  If exclusive is set to false then the current object will be overridden/merged.  Separator can also be set for key path support.  This job uses (Realize)[https://github.com/bluemarblepayroll/realize], which provides its own extendable value-transformation pipeline.
+* **collection/unpivot** [pivot_set]: Take an array of objects and unpivot specific sets of keys into rows.  Under the hood it uses [HashMath's Unpivot class](https://github.com/bluemarblepayroll/hash_math#unpivot-hash-key-coalescence-and-row-extrapolation).
+* **collection/values** [include_keys]: Take an array of objects and call `#values` on each object. If include_keys is true (it is false by default), then call `#keys` on the first object and inject that as a "header" object.
+
+#### De-serialization
+
+* **deserialize/csv** []: Take a CSV string and de-serialize into object(s).  Currently it will return an array of arrays, with each nested array representing one row.
+* **deserialize/json** []: Treat input as a string and de-serialize it to JSON.
+* **deserialize/yaml** [safe]: Treat input as a string and de-serialize it to YAML.  By default it will try and (safely de-serialize)[https://ruby-doc.org/stdlib-2.6.1/libdoc/psych/rdoc/Psych.html#method-c-safe_load] it (only using core classes).  If you wish to de-serialize it to any class type, pass in `safe: false`
+
+#### IO
+
+* **io/exist** [path, short_circuit]: Check to see if a file exists. The path parameter can be interpolated using `Payload#params`.  If short_circuit was set to true (defaults to false) and the file does not exist then the pipeline will be short-circuited.
+* **io/read** [binary, path]: Read in a local file.  The path parameter can be interpolated using `Payload#params`.  If the contents are binary, pass in `binary: true` to open it up in binary+read mode.
+* **io/write** [binary, path]: Write to a local file.  The path parameter can be interpolated using `Payload#params`.  If the contents are binary, pass in `binary: true` to open it up in binary+write mode.
+
+#### Serialization
+
+* **serialize/csv** []: Take an array of arrays and create a CSV.
+* **serialize/json** []: Convert value to JSON.
+* **serialize/yaml** []: Convert value to YAML.
+
+#### General
+
+* **dummy** []: Do nothing
+* **echo** [message]: Write a message to the output.  The message parameter can be interpolated using  `Payload#params`.
+* **set** [value]: Set the value to any arbitrary value.
+* **sleep** [seconds]: Sleep the thread for X number of seconds.
+
+
+### Adding & Registering Jobs
+
+Where this library shines is when additional jobs are plugged in.  Burner uses its `Burner::Jobs` class as its class-level registry built with (acts_as_hashable)[https://github.com/bluemarblepayroll/acts_as_hashable]'s acts_as_hashable_factory directive.
+
+Let's say we would like to register a job to parse a CSV:
+
+````ruby
+class ParseCsv < Burner::Job
+  def perform(output, payload)
+    payload.value = CSV.parse(payload.value, headers: true).map(&:to_h)
+
+    nil
+  end
+end
+
+Burner::Jobs.register('parse_csv', ParseCsv)
+````
+
+`parse_csv` is now recognized as a valid job and we can use it:
+
+````ruby
+pipeline = {
+  jobs: [
+    {
+      name: :read,
+      type: 'io/read',
+      path: '{input_file}'
+    },
+    {
+      name: :output_id,
+      type: :echo,
+      message: 'The job id is: {__id}'
+    },
+    {
+      name: :output_value,
+      type: :echo,
+      message: 'The current value is: {__value}'
+    },
+    {
+      name: :parse,
+      type: :parse_csv
+    },
+    {
+      name: :convert,
+      type: 'serialize/yaml'
+    },
+    {
+      name: :write,
+      type: 'io/write',
+      path: '{output_file}'
+    }
+  ],
+  steps: %i[
+    read
+    output_id
+    output_value
+    parse
+    convert
+    output_value
+    write
+  ]
+}
+
+params = {
+  input_file: File.join('spec', 'fixtures', 'cars.csv'),
+  output_file: File.join(TEMP_DIR, "#{SecureRandom.uuid}.yaml")
+}
+
+payload = Burner::Payload.new(params: params)
+
+Burner::Pipeline.make(pipeline).execute(output: output, payload: payload)
+````
 
 ## Contributing
 

data/burner.gemspec

@@ -29,7 +29,10 @@ Gem::Specification.new do |s|
   s.required_ruby_version = '>= 2.5'
 
   s.add_dependency('acts_as_hashable', '~>1.2')
+  s.add_dependency('hashematics', '~>1.1')
+  s.add_dependency('hash_math', '~>1.2')
   s.add_dependency('objectable', '~>1.0')
+  s.add_dependency('realize', '~>1.2')
   s.add_dependency('stringento', '~>2.1')
 
   s.add_development_dependency('guard-rspec', '~>4.7')

data/lib/burner.rb

@@ -9,11 +9,21 @@
 
 require 'acts_as_hashable'
 require 'benchmark'
+require 'csv'
+require 'forwardable'
+require 'hash_math'
+require 'hashematics'
 require 'json'
 require 'objectable'
+require 'realize'
 require 'securerandom'
 require 'singleton'
 require 'stringento'
+require 'time'
 require 'yaml'
 
+# Common/Shared
+require_relative 'burner/modeling'
+
+# Main Entrypoint(s)
 require_relative 'burner/cli'

data/lib/burner/cli.rb

@@ -12,18 +12,18 @@ require_relative 'pipeline'
 module Burner
   # Process a single string as a Pipeline.  This is mainly to back the command-line interface.
   class Cli
-    attr_reader :params, :pipeline
+    attr_reader :payload, :pipeline
 
     def initialize(args)
-      path       = args.first
-      cli_params = extract_cli_params(args)
-      config     = read_yaml(path)
-      @pipeline  = Burner::Pipeline.make(jobs: config['jobs'], steps: config['steps'])
-      @params    = (config['params'] || {}).merge(cli_params)
+      path      = args.first
+      params    = extract_cli_params(args)
+      config    = read_yaml(path)
+      @pipeline = Burner::Pipeline.make(jobs: config['jobs'], steps: config['steps'])
+      @payload  = Payload.new(params: params)
     end
 
     def execute
-      pipeline.execute(params: params)
+      pipeline.execute(payload: payload)
     end
 
     private

data/lib/burner/job.rb

@@ -28,8 +28,10 @@ module Burner
 
     private
 
-    def eval_string_template(expression, input)
-      string_template.evaluate(expression, input)
+    def job_string_template(expression, output, payload)
+      templatable_params = payload.params.merge(__id: output.id, __value: payload.value)
+
+      string_template.evaluate(expression, templatable_params)
     end
   end
 end

data/lib/burner/jobs.rb

@@ -8,12 +8,22 @@
 #
 
 require_relative 'job'
+require_relative 'jobs/collection/arrays_to_objects'
+require_relative 'jobs/collection/graph'
+require_relative 'jobs/collection/objects_to_arrays'
+require_relative 'jobs/collection/shift'
+require_relative 'jobs/collection/transform'
+require_relative 'jobs/collection/unpivot'
+require_relative 'jobs/collection/values'
+require_relative 'jobs/deserialize/csv'
 require_relative 'jobs/deserialize/json'
 require_relative 'jobs/deserialize/yaml'
 require_relative 'jobs/dummy'
 require_relative 'jobs/echo'
+require_relative 'jobs/io/exist'
 require_relative 'jobs/io/read'
 require_relative 'jobs/io/write'
+require_relative 'jobs/serialize/csv'
 require_relative 'jobs/serialize/json'
 require_relative 'jobs/serialize/yaml'
 require_relative 'jobs/set'
@@ -26,15 +36,25 @@ module Burner
   class Jobs
     acts_as_hashable_factory
 
-    register 'deserialize/json', Deserialize::Json
-    register 'deserialize/yaml', Deserialize::Yaml
-    register 'dummy', '',        Dummy
-    register 'echo',             Echo
-    register 'io/read',          IO::Read
-    register 'io/write',         IO::Write
-    register 'serialize/json',   Serialize::Json
-    register 'serialize/yaml',   Serialize::Yaml
-    register 'set',              Set
-    register 'sleep',            Sleep
+    register 'collection/arrays_to_objects', Collection::ArraysToObjects
+    register 'collection/graph',             Collection::Graph
+    register 'collection/objects_to_arrays', Collection::ObjectsToArrays
+    register 'collection/shift',             Collection::Shift
+    register 'collection/transform',         Collection::Transform
+    register 'collection/unpivot',           Collection::Unpivot
+    register 'collection/values',            Collection::Values
+    register 'deserialize/csv',              Deserialize::Csv
+    register 'deserialize/json',             Deserialize::Json
+    register 'deserialize/yaml',             Deserialize::Yaml
+    register 'dummy', '',                    Dummy
+    register 'echo',                         Echo
+    register 'io/exist',                     IO::Exist
+    register 'io/read',                      IO::Read
+    register 'io/write',                     IO::Write
+    register 'serialize/csv',                Serialize::Csv
+    register 'serialize/json',               Serialize::Json
+    register 'serialize/yaml',               Serialize::Yaml
+    register 'set',                          Set
+    register 'sleep',                        Sleep
   end
 end