realize 1.0.0.pre.alpha → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 77bcd7b4fedcf6a51318375889b4ab11c4222015ddc7da3ada67315d5e6817dc
-  data.tar.gz: eb9d90c4d49a0ab19da44543b99e355c557012c1155740efb9613ac5e095e5b5
+  metadata.gz: 0f1004b6b97c8ebd485bac4fb9859ffd28511240f4a9943c73875664be35722c
+  data.tar.gz: 979d68bb86ecc8ba32251c2346a0183158686bf04fb9949f04646e7e16a6a03b
 SHA512:
-  metadata.gz: 48875ff1d61c486f648955f2e62b9d1527dd40b2be7454a531dce21df3186017f06d2275d80587653867dc10a249666effb2e11f85ad10b5d31ca5e51a27d4d1
-  data.tar.gz: e3cb298a91c99011883d3763b01585548679388cc2b3c93269571d2c96e7eca002727035cd156a6cc8799d854930de2aa637cec6e005385ef810cd11aa976572
+  metadata.gz: f2398a44d581a5e8685ef9c29acbb140fbc2488cb7ffc6411317254325154e30b72b3828b96df5aa7e17f299a79c802f4ece1a50486c0ec62d20bdcb9bd5f073
+  data.tar.gz: 9b87926382b38afda00be7ab8055ab26eea572228489f0d0efd341dd0b709d11d1becafa3e925d79baa5c30bccd0fab30130c8792e646891c902fc6cb7a23aa9

data/CHANGELOG.md

@@ -1,3 +1,7 @@
-# 1.0.0 (June 8th, 2020)
+# 1.1.0 (June 24th, 2020)
+
+Addition of r/collection/at_index, r/collection/first, and r/collection/last
+
+# 1.0.0 (June 9th, 2020)
 
 Initial Release, includes 15 initial transformers.

data/README.md

@@ -2,7 +2,11 @@
 
 [![Gem Version](https://badge.fury.io/rb/realize.svg)](https://badge.fury.io/rb/realize) [![Build Status](https://travis-ci.org/bluemarblepayroll/realize.svg?branch=master)](https://travis-ci.org/bluemarblepayroll/realize) [![Maintainability](https://api.codeclimate.com/v1/badges/115f0c5a1d0a4cce7230/maintainability)](https://codeclimate.com/github/bluemarblepayroll/realize/maintainability) [![Test Coverage](https://api.codeclimate.com/v1/badges/115f0c5a1d0a4cce7230/test_coverage)](https://codeclimate.com/github/bluemarblepayroll/realize/test_coverage) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
 
-TODO
+This library provides a pluggable and configurable data transformation framework.  The general use-case is:
+
+> We need to be able to configure the data transformation pipeline, within an application, for a system-to-system integration.
+
+It is currently used in production at Blue Marble to power the transformation pipeline within a larger ETL framework.
 
 ## Installation
 
@@ -20,7 +24,90 @@ bundle add realize
 
 ## Examples
 
-TODO
+### Basic Transformer Example
+
+Here is a simple record we will use for data derivation and transformation:
+
+````ruby
+record = {
+  id: 1,
+  created_at: '2020-03-04T12:34:56Z',
+  first: 'Frank',
+  last: 'Rizzo'
+}
+````
+
+Let's say we wanted to retrieve the created_at formatted as: 'MM/DD/YY', we could write:
+
+````ruby
+transformers = [
+  {
+    type: 'r/value/resolve',
+    key: :created_at
+  },
+  {
+    type: 'r/format/date',
+    output_format: '%D'
+  }
+]
+
+value = Realize.pipeline(transformers).transform(record) # 03/04/20
+````
+
+Notice how all built-in transformers are prefixed with 'r'.  This should help isolate the built-in transformers from potential externally registered transformers.
+
+### Transformer Gallery
+
+Here is a list of each built-in transformer, their options, and what their function is:
+
+#### Collection-oriented Transformers
+
+* **r/collection/at_index** [index]:  Takes an array (or coerces value to an array) and returns the value at the given index position.
+* **r/collection/first** []:  Takes an array (or coerces value to an array) and returns the value at the first index position.
+* **r/collection/last** []:  Takes an array (or coerces value to an array) and returns the value at the last index position.
+* **r/collection/sort** [key, direction]:  Takes an array (or coerces value to an array) and sort it either ascending or descending by some defined key's value.
+
+#### Filtering Transformers
+
+* **r/filter/by_key_record_value** [key, value]: Takes an array (or coerces value to an array) and selects only the records that match the key's value.  In this case the value is derived off of the main record.
+* **r/filter/by_key_value_presence** [key]: Takes an array (or coerces value to an array) and selects only the records where the key's value is present (not nil and not empty).
+* **r/filter/by_key_record_value** [key, value]: Takes an array (or coerces value to an array) and selects only the records that match the key's value.  In this case, the value is statically defined.
+* **r/filter/inactive** [key, value]: Takes an array (or coerces value to an array) and selects only the records where the current time is between the start and end times as derived from the record.  Note that *current time* can be passed in but defaults to current UTC time.
+
+#### Format-oriented Transformers
+
+* **r/format/date** [input_format, output_format]: Parses the incoming value into a Time object using the configured input_format and outputs it as formatted by the configured output_format.
+* **r/format/remove_whitespace** []: Removes all whitespace from the incoming value.
+* **r/format/string_replace** [original, replacement]: Replaces all occurrences of the configured original value with the replacement value.
+
+#### Logical Transformers
+
+* **r/logical/switch** [cases, default_transformers, key]: Provides a value-based logic branching.  If a value matches a specific case, the specific cases transformers will be executed.  If it does not match any case then the default_transformers will be executed.
+
+#### Value-oriented Transformers
+
+* **r/value/blank** []: Always return a blank string.
+* **r/value/map** [values]: Do a lookup on the value using the `values` hash.  If a value is found, then its corresponding value is returned.  If it isn't then the input is returned.
+* **r/value/null** []: Always returns null.
+* **r/value/resolve** [key]: Dynamically resolves a value based on the record's key.  It uses the [Objectable](https://github.com/bluemarblepayroll/objectable) library by default, which provides some goodies like dot-notation for nested objects and type-insensitivity (works for Ruby objects, hashes, structs, etc.)
+* **r/value/static** [value]: Always returns a hard-coded value.
+* **r/value/verbatim** []: Default transformer, simply echos back the input.
+
+### Plugging in Transformers
+
+Custom transformers can be externally created and registered as long as it complies with the general transformer interface:
+
+* constructor accepts keyword arguments from which the configuration provides
+* responds to a method called transform with the signature: transform(resolver, value, time, record)
+
+After you have implemented the custom transformer, you can externally register is by:
+
+````ruby
+Realize::Transformers.register('some_custom_transformer', SomeCustomTransformer)
+````
+
+You should now be able to use the type: 'some_custom_transformer' within your transformation configuration.
+
 
 ## Contributing
 

data/lib/realize.rb

@@ -18,11 +18,8 @@ require_relative 'realize/pipeline'
 # Top-level syntactic sugar.
 module Realize
   class << self
-    def pipeline(resolver: Objectable.resolver, transformers: [])
-      Pipeline.new(
-        resolver: resolver,
-        transformers: transformers
-      )
+    def pipeline(transformers = [], resolver: Objectable.resolver)
+      Pipeline.new(transformers, resolver: resolver)
     end
   end
 end

data/lib/realize/collection/at_index.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module Realize
+  class Collection
+    # Transformer to get an item of a collection
+    class AtIndex
+      acts_as_hashable
+
+      attr_reader :index
+
+      def initialize(index:)
+        raise ArgumentError, 'index is required' if index.to_s.empty?
+
+        @index = index.to_i
+
+        freeze
+      end
+
+      def transform(_resolver, value, _time, _record)
+        value.is_a?(Array) ? value[index] : value
+      end
+    end
+  end
+end

data/lib/realize/collection/first.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+module Realize
+  class Collection
+    # Transformer to get the first item of a collection
+    class First
+      acts_as_hashable
+
+      def transform(resolver, value, time, record)
+        AtIndex.new(index: 0).transform(resolver, value, time, record)
+      end
+    end
+  end
+end

data/lib/realize/collection/last.rb

@@ -0,0 +1,14 @@
+# frozen_string_literal: true
+
+module Realize
+  class Collection
+    # Transformer to get the last item of a collection
+    class Last
+      acts_as_hashable
+
+      def transform(resolver, value, time, record)
+        AtIndex.new(index: -1).transform(resolver, value, time, record)
+      end
+    end
+  end
+end

data/lib/realize/collection/sort.rb

@@ -1,16 +1,12 @@
 # frozen_string_literal: true
 
+require_relative 'sort/direction'
+
 module Realize
   class Collection
-    # Transformer to take an array of oibjects and sort by the given key
+    # Transformer to take an array of objects and sort by the given key
     # and by the given sort direction. Defaulting to ascending.
     class Sort
-      module Direction
-        ASCENDING = 'ASC'
-        ASC = 'ASC'
-        DESCENDING = 'DESC'
-        DESC = 'DESC'
-      end
       include Arrays
       include Direction
       acts_as_hashable
@@ -28,10 +24,10 @@ module Realize
         freeze
       end
 
-      def transform(_resolver, value, _time, _record)
+      def transform(resolver, value, _time, _record)
         records = array(value)
 
-        sorted = records.sort_by { |hsh| hsh[key.to_sym] }
+        sorted = records.sort_by { |hsh| resolver.get(hsh, key) }
 
         order == DESCENDING ? sorted.reverse : sorted
       end

data/lib/realize/collection/sort/direction.rb

@@ -1,35 +1,13 @@
 # frozen_string_literal: true
 
-require_relative 'sort/direction'
-
 module Realize
   class Collection
-    # Transformer to take an array of oibjects and sort by the given key
-    # and by the given sort direction. Defaulting to ascending.
     class Sort
-      include Arrays
-      include Direction
-      acts_as_hashable
-
-      DEFAULT_ORDER = ASCENDING
-
-      attr_reader :key, :order
-
-      def initialize(key:, order: DEFAULT_ORDER)
-        raise ArgumentError, 'key is required' if key.to_s.empty?
-
-        @key   = key
-        @order = Direction.const_get(order.to_s.upcase.to_sym)
-
-        freeze
-      end
-
-      def transform(_resolver, value, _time, _record)
-        records = array(value)
-
-        sorted = records.sort_by { |hsh| hsh[key.to_sym] }
-
-        order == DESCENDING ? sorted.reverse : sorted
+      # Constants describing the possible values of 'direction' for a Sort instance.
+      module Direction
+        # providing a few different ways to specify.
+        ASCENDING  = ASC  = 'ASC'
+        DESCENDING = DESC = 'DESC'
       end
     end
   end

data/lib/realize/format/remove_whitespace.rb

@@ -7,8 +7,6 @@ module Realize
     class RemoveWhitespace
       acts_as_hashable
 
-      def initialize(_opts = {}); end
-
       def transform(_resolver, value, _time, _record)
         value.to_s.gsub(/\s+/, ' ')
       end

data/lib/realize/pipeline.rb

@@ -6,7 +6,7 @@ module Realize
   class Pipeline
     attr_reader :resolver, :transformers
 
-    def initialize(resolver: Objectable.resolver, transformers: [])
+    def initialize(transformers = [], resolver: Objectable.resolver)
       raise ArgumentError, 'resolver is required' unless resolver
 
       @resolver     = resolver
@@ -15,7 +15,7 @@ module Realize
       freeze
     end
 
-    def transform(record, time)
+    def transform(record, time = Time.now.utc)
       transformers.inject(record) do |memo, transformer|
         transformer.transform(resolver, memo, time, record)
       end

data/lib/realize/transformers.rb

@@ -1,5 +1,8 @@
 # frozen_string_literal: true
 
+require_relative 'collection/at_index'
+require_relative 'collection/first'
+require_relative 'collection/last'
 require_relative 'collection/sort'
 require_relative 'filter/by_key_record_value'
 require_relative 'filter/by_key_value'
@@ -25,6 +28,9 @@ module Realize
     acts_as_hashable_factory
 
     register '',                               Value::Verbatim
+    register 'r/collection/at_index',          Collection::AtIndex
+    register 'r/collection/first',             Collection::First
+    register 'r/collection/last',              Collection::Last
     register 'r/collection/sort',              Collection::Sort
     register 'r/filter/by_key_record_value',   Filter::ByKeyRecordValue
     register 'r/filter/by_key_value',          Filter::ByKeyValue

data/lib/realize/value/blank.rb

@@ -6,8 +6,6 @@ module Realize
     class Blank
       acts_as_hashable
 
-      def initialize(_opts = {}); end
-
       def transform(_resolver, _value, _time, _record)
         ''
       end

data/lib/realize/value/null.rb

@@ -6,8 +6,6 @@ module Realize
     class Null
       acts_as_hashable
 
-      def initialize(_opts = {}); end
-
       def transform(_resolver, _value, _time, _record)
         nil
       end

data/lib/realize/value/verbatim.rb

@@ -6,11 +6,6 @@ module Realize
     class Verbatim
       acts_as_hashable
 
-      # This is here to satisfy an underlying issue in acts_as_hashable.
-      # The #make calls in the factory and hashable module should be calling #new with no
-      # args if no keys are detected.
-      def initialize(_opts = {}); end
-
       def transform(_resolver, value, _time, _record)
         value
       end

data/lib/realize/version.rb

@@ -8,5 +8,5 @@
 #
 
 module Realize
-  VERSION = '1.0.0-alpha'
+  VERSION = '1.1.0'
 end

data/realize.gemspec

@@ -11,8 +11,8 @@ Gem::Specification.new do |s|
     Derive and transform a value using a configuration-first pipeline.
   DESCRIPTION
 
-  s.authors     = ['Matthew Ruggio']
-  s.email       = ['mruggio@bluemarblepayroll.com']
+  s.authors     = ['Matthew Ruggio', 'Dan Dewar']
+  s.email       = ['mruggio@bluemarblepayroll.com', 'ddewar@bluemarblepayroll.com']
   s.files       = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
   s.bindir      = 'exe'
   s.executables = []
@@ -28,7 +28,7 @@ Gem::Specification.new do |s|
 
   s.required_ruby_version = '>= 2.5'
 
-  s.add_dependency('acts_as_hashable', '~>1')
+  s.add_dependency('acts_as_hashable', '~>1', '>=1.2.0')
   s.add_dependency('objectable', '~>1')
 
   s.add_development_dependency('guard-rspec', '~>4.7')

metadata

@@ -1,14 +1,15 @@
 --- !ruby/object:Gem::Specification
 name: realize
 version: !ruby/object:Gem::Version
-  version: 1.0.0.pre.alpha
+  version: 1.1.0
 platform: ruby
 authors:
 - Matthew Ruggio
+- Dan Dewar
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2020-06-08 00:00:00.000000000 Z
+date: 2020-06-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: acts_as_hashable
@@ -17,6 +18,9 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '1'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.2.0
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -24,6 +28,9 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '1'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.2.0
 - !ruby/object:Gem::Dependency
   name: objectable
   requirement: !ruby/object:Gem::Requirement
@@ -139,6 +146,7 @@ dependencies:
 description: "    Derive and transform a value using a configuration-first pipeline.\n"
 email:
 - mruggio@bluemarblepayroll.com
+- ddewar@bluemarblepayroll.com
 executables: []
 extensions: []
 extra_rdoc_files: []
@@ -159,6 +167,9 @@ files:
 - exe/.gitkeep
 - lib/realize.rb
 - lib/realize/arrays.rb
+- lib/realize/collection/at_index.rb
+- lib/realize/collection/first.rb
+- lib/realize/collection/last.rb
 - lib/realize/collection/sort.rb
 - lib/realize/collection/sort/direction.rb
 - lib/realize/filter/by_key_record_value.rb
@@ -200,9 +211,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '2.5'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">"
+  - - ">="
     - !ruby/object:Gem::Version
-      version: 1.3.1
+      version: '0'
 requirements: []
 rubygems_version: 3.0.3
 signing_key: