burner 1.0.0.pre.alpha → 1.0.0.pre.alpha.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 920f3b0f0027e21b30da75f86335d75d2d351f49c8b68662de7226ade43474c6
-  data.tar.gz: b7f4189e37023f1d627ea5222df0f397a24a69c7ce3721b13ce714d2e274ee94
+  metadata.gz: 76ba9eda7f18a68f4c24c11d4724e2021c6f1d92ca4878fafb118e6c897c018c
+  data.tar.gz: 95e35512ece9b5f8acb912305d0609c86354977d3c8bd6e125a18db0584778fc
 SHA512:
-  metadata.gz: d488edf32a95e5da64190c512f365d76e2414d16712cee9b56a400bd62e11ed7824a316a95923af15f0ecac3390e4be359d2e31ffc33fb13129309c4efce9403
-  data.tar.gz: 8640cfdb2fa4241fb2a493c2ac2aed62e08e1826446907f4b16d043276fffee656c787d2079545fe03d52b8f45756240a523ad878fc89063884868c3f94bfecd
+  metadata.gz: 47b53c96680edfd6ca08e637d2ab6fd738684e9c2e7e8d08ca6d52a6361b8cf3cfa55af072cfeeb41a44838e76a4474aa229fc23b86f54558603c9af718f5fa0
+  data.tar.gz: 804da88a604544877d7b4f34bafcf1079f3282ad0c34da69466b83bfbd37fb30589f1cb38b02aafa4ccfd17f9e24aabf28385d06a8b80c3d2f552d4e71d3fd7d

data/README.md

@@ -2,7 +2,7 @@
 
 [![Gem Version](https://badge.fury.io/rb/burner.svg)](https://badge.fury.io/rb/burner) [![Build Status](https://travis-ci.org/bluemarblepayroll/burner.svg?branch=master)](https://travis-ci.org/bluemarblepayroll/burner) [![Maintainability](https://api.codeclimate.com/v1/badges/dbc3757929b67504f6ca/maintainability)](https://codeclimate.com/github/bluemarblepayroll/burner/maintainability) [![Test Coverage](https://api.codeclimate.com/v1/badges/dbc3757929b67504f6ca/test_coverage)](https://codeclimate.com/github/bluemarblepayroll/burner/test_coverage) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-TODO
+This library serves as the skeleton for a processing engine.  It allows you to organize your code into Jobs, then stitch those jobs together as steps.
 
 ## Installation
 
@@ -20,7 +20,297 @@ bundle add burner
 
 ## Examples
 
-TODO
+The purpose of this library is to provide a framework for creating highly de-coupled functions (known as jobs), and then allow for the stitching of them back together in any arbitrary order (know as steps.)  Although our example will be somewhat specific and contrived, the only limit to what the jobs and order of jobs are is up to your imagination.
+
+### JSON-to-YAML File Converter
+
+All the jobs for this example are shipped with this library.  In this example, we will write a pipeline that can read a JSON file and convert it to YAML.  Pipelines are data-first so we can represent a pipeline using a hash:
+
+````ruby
+pipeline = {
+  jobs: [
+    {
+      name: :read,
+      type: 'io/read',
+      path: '{input_file}'
+    },
+    {
+      name: :output_id,
+      type: :echo,
+      message: 'The job id is: {__id}'
+    },
+    {
+      name: :output_value,
+      type: :echo,
+      message: 'The current value is: {__value}'
+    },
+    {
+      name: :parse,
+      type: 'deserialize/json'
+    },
+    {
+      name: :convert,
+      type: 'serialize/yaml'
+    },
+    {
+      name: :write,
+      type: 'io/write',
+      path: '{output_file}'
+    }
+  ],
+  steps: %i[
+    read
+    output_id
+    output_value
+    parse
+    convert
+    output_value
+    write
+  ]
+}
+
+params = {
+  input_file: 'input.json',
+  output_file: 'output.yaml'
+}
+````
+
+Assuming we are running this script from a directory where an `input.json` file exists, we can then programatically process the pipeline:
+
+````ruby
+Burner::Pipeline.make(pipeline).execute(params: params)
+````
+
+We should now see a output.yaml file created.
+
+Some notes:
+
+* Some values are able to be string-interpolated using the provided params.  This allows for the passing runtime configuration/data into pipelines/jobs.
+* The job's ID can be accessed using the `__id` key.
+* The current job's payload value can be accessed using the `__value` key.
+* Jobs can be re-used (just like the output_id and output_value jobs).
+
+### Capturing Feedback / Output
+
+By default, output will be emitted to `$stdout`.  You can add or change listeners by passing in optional values into Pipeline#execute.  For example, say we wanted to capture the output from our json-to-yaml example:
+
+````ruby
+class StringOut
+  def initialize
+    @io = StringIO.new
+  end
+
+  def puts(msg)
+    tap { io.write("#{msg}\n") }
+  end
+
+  def read
+    io.rewind
+    io.read
+  end
+
+  private
+
+  attr_reader :io
+end
+
+string_out = StringOut.new
+output     = Burner::Output.new(outs: string_out)
+
+Burner::Pipeline.make(pipeline).execute(output: output, params: params)
+
+log = string_out.read
+````
+
+The value of `log` should now look similar to:
+
+````bash
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] Pipeline started with 7 step(s)
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] Parameters:
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - input_file: input.json
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - output_file: output.yaml
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] --------------------------------------------------------------------------------
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] [1] Burner::Jobs::IO::Read::read
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - Reading: spec/fixtures/input.json
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - Completed in: 0.0 second(s)
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] [2] Burner::Jobs::Echo::output_id
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - The job id is:
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - Completed in: 0.0 second(s)
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] [3] Burner::Jobs::Echo::output_value
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - The current value is:
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - Completed in: 0.0 second(s)
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] [4] Burner::Jobs::Deserialize::Json::parse
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - Completed in: 0.0 second(s)
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] [5] Burner::Jobs::Serialize::Yaml::convert
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - Completed in: 0.0 second(s)
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] [6] Burner::Jobs::Echo::output_value
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - The current value is:
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - Completed in: 0.0 second(s)
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] [7] Burner::Jobs::IO::Write::write
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - Writing: output.yaml
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC]   - Completed in: 0.0 second(s)
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] --------------------------------------------------------------------------------
+[8bdc394e-7047-4a1a-87ed-6c54ed690ed5 | 2020-10-14 13:49:59 UTC] Pipeline ended, took 0.001 second(s) to complete
+````
+
+Notes:
+
+* The Job ID is specified as the leading UUID in each line.
+* `outs` can be provided an array of listeners, as long as each listener responds to `puts(msg)`.
+
+### Command Line Pipeline Processing
+
+This library also ships with a built-in script `exe/burner` that illustrates using the `Burner::Cli` API.  This class can take in an array of arguments (similar to a command-line) and execute a pipeline.  The first argument is the path to a YAML file with the pipeline's configuration and each subsequent argument is a param in `key=value` form.  Here is how the json-to-yaml example can utilize this interface:
+
+#### Create YAML Pipeline Configuration File
+
+Write the following json_to_yaml_pipeline.yaml file to disk:
+
+````yaml
+jobs:
+  - name: read
+    type: io/read
+    path: '{input_file}'
+
+  - name: output_id
+    type: echo
+    message: 'The job id is: {__id}'
+
+  - name: output_value
+    type: echo
+    message: 'The current value is: {__value}'
+
+  - name: parse
+    type: deserialize/json
+
+  - name: convert
+    type: serialize/yaml
+
+  - name: write
+    type: io/write
+    path: '{output_file}'
+
+steps:
+  - read
+  - output_id
+  - output_value
+  - parse
+  - convert
+  - output_value
+  - write
+````
+
+#### Run Using Script
+
+From the command-line, run:
+
+````bash
+bundle exec burner json_to_yaml_pipeline.yaml input_file=input.json output_file=output.yaml
+````
+
+The pipeline should be processed and output.yaml should be created.
+
+#### Run Using Programmatic API
+
+Instead of the script, you can invoke it using code:
+
+````ruby
+args = %w[
+  json_to_yaml_pipeline.yaml
+  input_file=input.json
+  output_file=output.yaml
+]
+
+Burner::Cli.new(args).invoke
+````
+
+### Core Job Library
+
+This library only ships with very basic, rudimentary jobs that are meant to just serve as a baseline:
+
+* **deserialize/json** []: Treat input as a string and de-serialize it to JSON.
+* **deserialize/yaml** [safe]: Treat input as a string and de-serialize it to YAML.  By default it will try and (safely de-serialize)[https://ruby-doc.org/stdlib-2.6.1/libdoc/psych/rdoc/Psych.html#method-c-safe_load] it (only using core classes).  If you wish to de-serialize it to any class type, pass in `safe: false`
+* **dummy** []: Do nothing
+* **echo** [message]: Write a message to the output.  The message parameter can be interpolated using params.
+* **io/exist** [path, short_circuit]: Check to see if a file exists. The path parameter can be interpolated using params.  If short_circuit was set to true (defaults to false) and the file does not exist then the pipeline will be short-circuited.
+* **io/read** [binary, path]: Read in a local file.  The path parameter can be interpolated using params.  If the contents are binary, pass in `binary: true` to open it up in binary+read mode.
+* **io/write** [binary, path]: Write to a local file.  The path parameter can be interpolated using params.  If the contents are binary, pass in `binary: true` to open it up in binary+write mode.
+* **serialize/json** []: Convert value to JSON.
+* **serialize/yaml** []: Convert value to YAML.
+* **set** [value]: Set the value to any arbitrary value.
+* **sleep** [seconds]: Sleep the thread for X number of seconds.
+
+
+### Adding & Registering Jobs
+
+Where this library shines is when additional jobs are plugged in.  Burner uses its `Burner::Jobs` class as its class-level registry built with (acts_as_hashable)[https://github.com/bluemarblepayroll/acts_as_hashable]'s acts_as_hashable_factory directive.
+
+Let's say we would like to register a job to parse a CSV:
+
+````ruby
+class ParseCsv < Burner::Job
+  def perform(output, payload, params)
+    payload.value = CSV.parse(payload.value, headers: true).map(&:to_h)
+
+    nil
+  end
+end
+
+Burner::Jobs.register('parse_csv', ParseCsv)
+````
+
+`parse_csv` is now recognized as a valid job and we can use it:
+
+````ruby
+pipeline = {
+  jobs: [
+    {
+      name: :read,
+      type: 'io/read',
+      path: '{input_file}'
+    },
+    {
+      name: :output_id,
+      type: :echo,
+      message: 'The job id is: {__id}'
+    },
+    {
+      name: :output_value,
+      type: :echo,
+      message: 'The current value is: {__value}'
+    },
+    {
+      name: :parse,
+      type: :parse_csv
+    },
+    {
+      name: :convert,
+      type: 'serialize/yaml'
+    },
+    {
+      name: :write,
+      type: 'io/write',
+      path: '{output_file}'
+    }
+  ],
+  steps: %i[
+    read
+    output_id
+    output_value
+    parse
+    convert
+    output_value
+    write
+  ]
+}
+
+params = {
+  input_file: File.join('spec', 'fixtures', 'cars.csv'),
+  output_file: File.join(TEMP_DIR, "#{SecureRandom.uuid}.yaml")
+}
+
+Burner::Pipeline.make(pipeline).execute(output: output, params: params)
+````
 
 ## Contributing
 

data/exe/burner

@@ -18,4 +18,4 @@ if ARGV.empty?
 end
 
 # This should return exit code of 1 if it raises any hard errors.
-Burner::Cli.new(ARGV).execute
+Burner::Cli.new(ARGV).invoke

data/lib/burner/cli.rb

@@ -22,7 +22,7 @@ module Burner
       @params    = (config['params'] || {}).merge(cli_params)
     end
 
-    def execute
+    def invoke
       pipeline.execute(params: params)
     end
 

data/lib/burner/jobs.rb

@@ -12,6 +12,7 @@ require_relative 'jobs/deserialize/json'
 require_relative 'jobs/deserialize/yaml'
 require_relative 'jobs/dummy'
 require_relative 'jobs/echo'
+require_relative 'jobs/io/exist'
 require_relative 'jobs/io/read'
 require_relative 'jobs/io/write'
 require_relative 'jobs/serialize/json'
@@ -30,6 +31,7 @@ module Burner
     register 'deserialize/yaml', Deserialize::Yaml
     register 'dummy', '',        Dummy
     register 'echo',             Echo
+    register 'io/exist',         IO::Exist
     register 'io/read',          IO::Read
     register 'io/write',         IO::Write
     register 'serialize/json',   Serialize::Json

data/lib/burner/jobs/io/base.rb

@@ -12,17 +12,14 @@ module Burner
     module IO
       # Common configuration/code for all IO Job subclasses.
       class Base < Job
-        attr_reader :binary, :path
+        attr_reader :path
 
-        def initialize(name:, path:, binary: false)
+        def initialize(name:, path:)
           super(name: name)
 
           raise ArgumentError, 'path is required' if path.to_s.empty?
 
-          @path   = path.to_s
-          @binary = binary || false
-
-          freeze
+          @path = path.to_s
         end
 
         private
@@ -30,10 +27,6 @@ module Burner
         def compile_path(params)
           eval_string_template(path, params)
         end
-
-        def mode
-          binary ? 'wb' : 'w'
-        end
       end
     end
   end

data/lib/burner/jobs/io/exist.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+require_relative 'base'
+
+module Burner
+  class Jobs
+    module IO
+      # Check to see if a file exists.  If short_circuit is set to true and the file
+      # does not exist then the job will return false and short circuit the pipeline.
+      class Exist < Base
+        attr_reader :short_circuit
+
+        def initialize(name:, path:, short_circuit: false)
+          super(name: name, path: path)
+
+          @short_circuit = short_circuit || false
+
+          freeze
+        end
+
+        def perform(output, _payload, params)
+          compiled_path = compile_path(params)
+
+          exists = File.exist?(compiled_path)
+          verb   = exists ? 'does' : 'does not'
+
+          output.detail("The path: #{compiled_path} #{verb} exist")
+
+          # if anything but false is returned then the pipeline will not short circuit.  So
+          # we need to make sure we explicitly return false.
+          short_circuit && !exists ? false : nil
+        end
+      end
+    end
+  end
+end

data/lib/burner/jobs/io/read.rb

@@ -14,6 +14,16 @@ module Burner
     module IO
       # Read value from disk.
       class Read < Base
+        attr_reader :binary
+
+        def initialize(name:, path:, binary: false)
+          super(name: name, path: path)
+
+          @binary = binary || false
+
+          freeze
+        end
+
         def perform(output, payload, params)
           compiled_path = compile_path(params)
 

data/lib/burner/jobs/io/write.rb

@@ -14,6 +14,16 @@ module Burner
     module IO
       # Write value to disk.
       class Write < Base
+        attr_reader :binary
+
+        def initialize(name:, path:, binary: false)
+          super(name: name, path: path)
+
+          @binary = binary || false
+
+          freeze
+        end
+
         def perform(output, payload, params)
           compiled_path = compile_path(params)
 
@@ -21,7 +31,15 @@ module Burner
 
           output.detail("Writing: #{compiled_path}")
 
-          File.open(compiled_path, mode) { |io| io.write(payload.value) }
+          time_in_seconds = Benchmark.measure do
+            File.open(compiled_path, mode) { |io| io.write(payload.value) }
+          end.real
+
+          payload.add_written_file(
+            logical_filename: compiled_path,
+            physical_filename: compiled_path,
+            time_in_seconds: time_in_seconds
+          )
 
           nil
         end
@@ -39,6 +57,10 @@ module Burner
 
           nil
         end
+
+        def mode
+          binary ? 'wb' : 'w'
+        end
       end
     end
   end

data/lib/burner/payload.rb

@@ -7,6 +7,8 @@
 # LICENSE file in the root directory of this source tree.
 #
 
+require_relative 'written_file'
+
 module Burner
   # The input for all Job#perform methods.  The main notion of this object is its "value"
   # attribute.  This is dynamic and weak on purpose and is subject to whatever the Job#perform
@@ -16,11 +18,16 @@ module Burner
   class Payload
     attr_accessor :value
 
-    attr_reader :context
+    attr_reader :context, :written_files
+
+    def initialize(context: {}, value: nil, written_files: [])
+      @context       = context || {}
+      @value         = value
+      @written_files = written_files || []
+    end
 
-    def initialize(context: {}, value: nil)
-      @context = context || {}
-      @value   = value
+    def add_written_file(written_file)
+      tap { written_files << WrittenFile.make(written_file) }
     end
   end
 end

data/lib/burner/pipeline.rb

@@ -42,7 +42,14 @@ module Burner
       output.ruler
 
       time_in_seconds = Benchmark.measure do
-        steps.each { |step| step.perform(output, payload, params) }
+        steps.each do |step|
+          return_value = step.perform(output, payload, params)
+
+          if return_value.is_a?(FalseClass)
+            output.detail('Job returned false, ending pipeline.')
+            break
+          end
+        end
       end.real.round(3)
 
       output.ruler

data/lib/burner/step.rb

@@ -29,15 +29,19 @@ module Burner
     end
 
     def perform(output, payload, params)
+      return_value = nil
+
       output.title("#{job.class.name}#{SEPARATOR}#{job.name}")
 
       time_in_seconds = Benchmark.measure do
-        job.perform(output, payload, params)
+        job_params = (params || {}).merge(__id: output.id, __value: payload.value)
+
+        return_value = job.perform(output, payload, job_params)
       end.real.round(3)
 
       output.complete(time_in_seconds)
 
-      nil
+      return_value
     end
   end
 end

data/lib/burner/version.rb

@@ -8,5 +8,5 @@
 #
 
 module Burner
-  VERSION = '1.0.0-alpha'
+  VERSION = '1.0.0-alpha.1'
 end

data/lib/burner/written_file.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+#
+# Copyright (c) 2020-present, Blue Marble Payroll, LLC
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+#
+
+module Burner
+  # Describes a file that was generated by a Job.  If a Job emits a file, it should also add the
+  # file details to the Payload#written_files array using the Payload#add_written_file method.
+  class WrittenFile
+    acts_as_hashable
+
+    attr_reader :logical_filename,
+                :physical_filename,
+                :time_in_seconds
+
+    def initialize(logical_filename:, physical_filename:, time_in_seconds:)
+      @logical_filename  = logical_filename.to_s
+      @physical_filename = physical_filename.to_s
+      @time_in_seconds   = time_in_seconds.to_f
+
+      freeze
+    end
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: burner
 version: !ruby/object:Gem::Version
-  version: 1.0.0.pre.alpha
+  version: 1.0.0.pre.alpha.1
 platform: ruby
 authors:
 - Matthew Ruggio
@@ -184,6 +184,7 @@ files:
 - lib/burner/jobs/dummy.rb
 - lib/burner/jobs/echo.rb
 - lib/burner/jobs/io/base.rb
+- lib/burner/jobs/io/exist.rb
 - lib/burner/jobs/io/read.rb
 - lib/burner/jobs/io/write.rb
 - lib/burner/jobs/serialize/json.rb
@@ -196,6 +197,7 @@ files:
 - lib/burner/step.rb
 - lib/burner/string_template.rb
 - lib/burner/version.rb
+- lib/burner/written_file.rb
 homepage: https://github.com/bluemarblepayroll/burner
 licenses:
 - MIT