realize 1.0.0.pre.alpha.1 → 1.1.1.pre.alpha
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +11 -1
- data/README.md +89 -2
- data/lib/realize.rb +2 -5
- data/lib/realize/collection/at_index.rb +24 -0
- data/lib/realize/collection/first.rb +14 -0
- data/lib/realize/collection/last.rb +14 -0
- data/lib/realize/collection/sort.rb +5 -9
- data/lib/realize/collection/sort/direction.rb +5 -27
- data/lib/realize/format/date.rb +2 -4
- data/lib/realize/format/remove_whitespace.rb +0 -2
- data/lib/realize/pipeline.rb +3 -6
- data/lib/realize/transformers.rb +6 -0
- data/lib/realize/value/blank.rb +0 -2
- data/lib/realize/value/null.rb +0 -2
- data/lib/realize/value/verbatim.rb +0 -5
- data/lib/realize/version.rb +1 -1
- data/realize.gemspec +3 -3
- metadata +13 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 0d584e2249098a82c2719e6feb8410a4e21dada80f9ca42978ef316843fb78a5
|
4
|
+
data.tar.gz: e5537b12e40b3a65a14aef391e864a13e565ef9040986a0a05b2b19f4ea7851a
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 35d3820ea8b76292d3fc52e80c9e6e20f7fc89909c8df4b3910838f0aa1b986391b295d2feb3e161eac4a1816838e6d20c3baef4f26e2dc5456c6434a57f2068
|
7
|
+
data.tar.gz: 14ac8614d30bbeef7f5adefb5eeffd773997576e640ac9f22c323e5813ebb052d292bff25da8dd2eef8ffabca1bc388af9bda996336ba65667672f03bfdf84e2
|
data/CHANGELOG.md
CHANGED
@@ -1,3 +1,13 @@
|
|
1
|
-
# 1.
|
1
|
+
# 1.1.1 (TBD)
|
2
|
+
|
3
|
+
Fixes:
|
4
|
+
|
5
|
+
* Do not leverage #to_datetime even if it is available for date parsing. This removes un-intentional coupling of other libraries into Realize and keeps it based on Ruby standard and core libraries.
|
6
|
+
|
7
|
+
# 1.1.0 (June 24th, 2020)
|
8
|
+
|
9
|
+
Addition of r/collection/at_index, r/collection/first, and r/collection/last
|
10
|
+
|
11
|
+
# 1.0.0 (June 9th, 2020)
|
2
12
|
|
3
13
|
Initial Release, includes 15 initial transformers.
|
data/README.md
CHANGED
@@ -2,7 +2,11 @@
|
|
2
2
|
|
3
3
|
[![Gem Version](https://badge.fury.io/rb/realize.svg)](https://badge.fury.io/rb/realize) [![Build Status](https://travis-ci.org/bluemarblepayroll/realize.svg?branch=master)](https://travis-ci.org/bluemarblepayroll/realize) [![Maintainability](https://api.codeclimate.com/v1/badges/115f0c5a1d0a4cce7230/maintainability)](https://codeclimate.com/github/bluemarblepayroll/realize/maintainability) [![Test Coverage](https://api.codeclimate.com/v1/badges/115f0c5a1d0a4cce7230/test_coverage)](https://codeclimate.com/github/bluemarblepayroll/realize/test_coverage) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
|
4
4
|
|
5
|
-
|
5
|
+
This library provides a pluggable and configurable data transformation framework. The general use-case is:
|
6
|
+
|
7
|
+
> We need to be able to configure the data transformation pipeline, within an application, for a system-to-system integration.
|
8
|
+
|
9
|
+
It is currently used in production at Blue Marble to power the transformation pipeline within a larger ETL framework.
|
6
10
|
|
7
11
|
## Installation
|
8
12
|
|
@@ -20,7 +24,90 @@ bundle add realize
|
|
20
24
|
|
21
25
|
## Examples
|
22
26
|
|
23
|
-
|
27
|
+
### Basic Transformer Example
|
28
|
+
|
29
|
+
Here is a simple record we will use for data derivation and transformation:
|
30
|
+
|
31
|
+
````ruby
|
32
|
+
record = {
|
33
|
+
id: 1,
|
34
|
+
created_at: '2020-03-04T12:34:56Z',
|
35
|
+
first: 'Frank',
|
36
|
+
last: 'Rizzo'
|
37
|
+
}
|
38
|
+
````
|
39
|
+
|
40
|
+
Let's say we wanted to retrieve the created_at formatted as: 'MM/DD/YY', we could write:
|
41
|
+
|
42
|
+
````ruby
|
43
|
+
transformers = [
|
44
|
+
{
|
45
|
+
type: 'r/value/resolve',
|
46
|
+
key: :created_at
|
47
|
+
},
|
48
|
+
{
|
49
|
+
type: 'r/format/date',
|
50
|
+
output_format: '%D'
|
51
|
+
}
|
52
|
+
]
|
53
|
+
|
54
|
+
value = Realize.pipeline(transformers).transform(record) # 03/04/20
|
55
|
+
````
|
56
|
+
|
57
|
+
Notice how all built-in transformers are prefixed with 'r'. This should help isolate the built-in transformers from potential externally registered transformers.
|
58
|
+
|
59
|
+
### Transformer Gallery
|
60
|
+
|
61
|
+
Here is a list of each built-in transformer, their options, and what their function is:
|
62
|
+
|
63
|
+
#### Collection-oriented Transformers
|
64
|
+
|
65
|
+
* **r/collection/at_index** [index]: Takes an array (or coerces value to an array) and returns the value at the given index position.
|
66
|
+
* **r/collection/first** []: Takes an array (or coerces value to an array) and returns the value at the first index position.
|
67
|
+
* **r/collection/last** []: Takes an array (or coerces value to an array) and returns the value at the last index position.
|
68
|
+
* **r/collection/sort** [key, direction]: Takes an array (or coerces value to an array) and sort it either ascending or descending by some defined key's value.
|
69
|
+
|
70
|
+
#### Filtering Transformers
|
71
|
+
|
72
|
+
* **r/filter/by_key_record_value** [key, value]: Takes an array (or coerces value to an array) and selects only the records that match the key's value. In this case the value is derived off of the main record.
|
73
|
+
* **r/filter/by_key_value_presence** [key]: Takes an array (or coerces value to an array) and selects only the records where the key's value is present (not nil and not empty).
|
74
|
+
* **r/filter/by_key_record_value** [key, value]: Takes an array (or coerces value to an array) and selects only the records that match the key's value. In this case, the value is statically defined.
|
75
|
+
* **r/filter/inactive** [key, value]: Takes an array (or coerces value to an array) and selects only the records where the current time is between the start and end times as derived from the record. Note that *current time* can be passed in but defaults to current UTC time.
|
76
|
+
|
77
|
+
#### Format-oriented Transformers
|
78
|
+
|
79
|
+
* **r/format/date** [input_format, output_format]: Parses the incoming value into a Time object using the configured input_format and outputs it as formatted by the configured output_format.
|
80
|
+
* **r/format/remove_whitespace** []: Removes all whitespace from the incoming value.
|
81
|
+
* **r/format/string_replace** [original, replacement]: Replaces all occurrences of the configured original value with the replacement value.
|
82
|
+
|
83
|
+
#### Logical Transformers
|
84
|
+
|
85
|
+
* **r/logical/switch** [cases, default_transformers, key]: Provides a value-based logic branching. If a value matches a specific case, the specific cases transformers will be executed. If it does not match any case then the default_transformers will be executed.
|
86
|
+
|
87
|
+
#### Value-oriented Transformers
|
88
|
+
|
89
|
+
* **r/value/blank** []: Always return a blank string.
|
90
|
+
* **r/value/map** [values]: Do a lookup on the value using the `values` hash. If a value is found, then its corresponding value is returned. If it isn't then the input is returned.
|
91
|
+
* **r/value/null** []: Always returns null.
|
92
|
+
* **r/value/resolve** [key]: Dynamically resolves a value based on the record's key. It uses the [Objectable](https://github.com/bluemarblepayroll/objectable) library by default, which provides some goodies like dot-notation for nested objects and type-insensitivity (works for Ruby objects, hashes, structs, etc.)
|
93
|
+
* **r/value/static** [value]: Always returns a hard-coded value.
|
94
|
+
* **r/value/verbatim** []: Default transformer, simply echos back the input.
|
95
|
+
|
96
|
+
### Plugging in Transformers
|
97
|
+
|
98
|
+
Custom transformers can be externally created and registered as long as it complies with the general transformer interface:
|
99
|
+
|
100
|
+
* constructor accepts keyword arguments from which the configuration provides
|
101
|
+
* responds to a method called transform with the signature: transform(resolver, value, time, record)
|
102
|
+
|
103
|
+
After you have implemented the custom transformer, you can externally register is by:
|
104
|
+
|
105
|
+
````ruby
|
106
|
+
Realize::Transformers.register('some_custom_transformer', SomeCustomTransformer)
|
107
|
+
````
|
108
|
+
|
109
|
+
You should now be able to use the type: 'some_custom_transformer' within your transformation configuration.
|
110
|
+
|
24
111
|
|
25
112
|
## Contributing
|
26
113
|
|
data/lib/realize.rb
CHANGED
@@ -18,11 +18,8 @@ require_relative 'realize/pipeline'
|
|
18
18
|
# Top-level syntactic sugar.
|
19
19
|
module Realize
|
20
20
|
class << self
|
21
|
-
def pipeline(resolver: Objectable.resolver
|
22
|
-
Pipeline.new(
|
23
|
-
resolver: resolver,
|
24
|
-
transformers: transformers
|
25
|
-
)
|
21
|
+
def pipeline(transformers = [], resolver: Objectable.resolver)
|
22
|
+
Pipeline.new(transformers, resolver: resolver)
|
26
23
|
end
|
27
24
|
end
|
28
25
|
end
|
@@ -0,0 +1,24 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Realize
|
4
|
+
class Collection
|
5
|
+
# Transformer to get an item of a collection
|
6
|
+
class AtIndex
|
7
|
+
acts_as_hashable
|
8
|
+
|
9
|
+
attr_reader :index
|
10
|
+
|
11
|
+
def initialize(index:)
|
12
|
+
raise ArgumentError, 'index is required' if index.to_s.empty?
|
13
|
+
|
14
|
+
@index = index.to_i
|
15
|
+
|
16
|
+
freeze
|
17
|
+
end
|
18
|
+
|
19
|
+
def transform(_resolver, value, _time, _record)
|
20
|
+
value.is_a?(Array) ? value[index] : value
|
21
|
+
end
|
22
|
+
end
|
23
|
+
end
|
24
|
+
end
|
@@ -0,0 +1,14 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Realize
|
4
|
+
class Collection
|
5
|
+
# Transformer to get the first item of a collection
|
6
|
+
class First
|
7
|
+
acts_as_hashable
|
8
|
+
|
9
|
+
def transform(resolver, value, time, record)
|
10
|
+
AtIndex.new(index: 0).transform(resolver, value, time, record)
|
11
|
+
end
|
12
|
+
end
|
13
|
+
end
|
14
|
+
end
|
@@ -0,0 +1,14 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module Realize
|
4
|
+
class Collection
|
5
|
+
# Transformer to get the last item of a collection
|
6
|
+
class Last
|
7
|
+
acts_as_hashable
|
8
|
+
|
9
|
+
def transform(resolver, value, time, record)
|
10
|
+
AtIndex.new(index: -1).transform(resolver, value, time, record)
|
11
|
+
end
|
12
|
+
end
|
13
|
+
end
|
14
|
+
end
|
@@ -1,16 +1,12 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
+
require_relative 'sort/direction'
|
4
|
+
|
3
5
|
module Realize
|
4
6
|
class Collection
|
5
|
-
# Transformer to take an array of
|
7
|
+
# Transformer to take an array of objects and sort by the given key
|
6
8
|
# and by the given sort direction. Defaulting to ascending.
|
7
9
|
class Sort
|
8
|
-
module Direction
|
9
|
-
ASCENDING = 'ASC'
|
10
|
-
ASC = 'ASC'
|
11
|
-
DESCENDING = 'DESC'
|
12
|
-
DESC = 'DESC'
|
13
|
-
end
|
14
10
|
include Arrays
|
15
11
|
include Direction
|
16
12
|
acts_as_hashable
|
@@ -28,10 +24,10 @@ module Realize
|
|
28
24
|
freeze
|
29
25
|
end
|
30
26
|
|
31
|
-
def transform(
|
27
|
+
def transform(resolver, value, _time, _record)
|
32
28
|
records = array(value)
|
33
29
|
|
34
|
-
sorted = records.sort_by { |hsh| hsh
|
30
|
+
sorted = records.sort_by { |hsh| resolver.get(hsh, key) }
|
35
31
|
|
36
32
|
order == DESCENDING ? sorted.reverse : sorted
|
37
33
|
end
|
@@ -1,35 +1,13 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
-
require_relative 'sort/direction'
|
4
|
-
|
5
3
|
module Realize
|
6
4
|
class Collection
|
7
|
-
# Transformer to take an array of oibjects and sort by the given key
|
8
|
-
# and by the given sort direction. Defaulting to ascending.
|
9
5
|
class Sort
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
attr_reader :key, :order
|
17
|
-
|
18
|
-
def initialize(key:, order: DEFAULT_ORDER)
|
19
|
-
raise ArgumentError, 'key is required' if key.to_s.empty?
|
20
|
-
|
21
|
-
@key = key
|
22
|
-
@order = Direction.const_get(order.to_s.upcase.to_sym)
|
23
|
-
|
24
|
-
freeze
|
25
|
-
end
|
26
|
-
|
27
|
-
def transform(_resolver, value, _time, _record)
|
28
|
-
records = array(value)
|
29
|
-
|
30
|
-
sorted = records.sort_by { |hsh| hsh[key.to_sym] }
|
31
|
-
|
32
|
-
order == DESCENDING ? sorted.reverse : sorted
|
6
|
+
# Constants describing the possible values of 'direction' for a Sort instance.
|
7
|
+
module Direction
|
8
|
+
# providing a few different ways to specify.
|
9
|
+
ASCENDING = ASC = 'ASC'
|
10
|
+
DESCENDING = DESC = 'DESC'
|
33
11
|
end
|
34
12
|
end
|
35
13
|
end
|
data/lib/realize/format/date.rb
CHANGED
@@ -30,12 +30,10 @@ module Realize
|
|
30
30
|
return nil if value.to_s.empty?
|
31
31
|
|
32
32
|
date_time =
|
33
|
-
if
|
34
|
-
value.to_datetime
|
35
|
-
elsif input_format?
|
33
|
+
if input_format?
|
36
34
|
DateTime.strptime(value, input_format)
|
37
35
|
else
|
38
|
-
DateTime.parse(value)
|
36
|
+
DateTime.parse(value.to_s)
|
39
37
|
end
|
40
38
|
|
41
39
|
date_time.strftime(output_format)
|
data/lib/realize/pipeline.rb
CHANGED
@@ -6,7 +6,7 @@ module Realize
|
|
6
6
|
class Pipeline
|
7
7
|
attr_reader :resolver, :transformers
|
8
8
|
|
9
|
-
def initialize(resolver: Objectable.resolver
|
9
|
+
def initialize(transformers = [], resolver: Objectable.resolver)
|
10
10
|
raise ArgumentError, 'resolver is required' unless resolver
|
11
11
|
|
12
12
|
@resolver = resolver
|
@@ -15,11 +15,8 @@ module Realize
|
|
15
15
|
freeze
|
16
16
|
end
|
17
17
|
|
18
|
-
def transform(record,
|
19
|
-
|
20
|
-
value = opts.fetch(:value, record)
|
21
|
-
|
22
|
-
transformers.inject(value) do |memo, transformer|
|
18
|
+
def transform(record, time = Time.now.utc)
|
19
|
+
transformers.inject(record) do |memo, transformer|
|
23
20
|
transformer.transform(resolver, memo, time, record)
|
24
21
|
end
|
25
22
|
end
|
data/lib/realize/transformers.rb
CHANGED
@@ -1,5 +1,8 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
+
require_relative 'collection/at_index'
|
4
|
+
require_relative 'collection/first'
|
5
|
+
require_relative 'collection/last'
|
3
6
|
require_relative 'collection/sort'
|
4
7
|
require_relative 'filter/by_key_record_value'
|
5
8
|
require_relative 'filter/by_key_value'
|
@@ -25,6 +28,9 @@ module Realize
|
|
25
28
|
acts_as_hashable_factory
|
26
29
|
|
27
30
|
register '', Value::Verbatim
|
31
|
+
register 'r/collection/at_index', Collection::AtIndex
|
32
|
+
register 'r/collection/first', Collection::First
|
33
|
+
register 'r/collection/last', Collection::Last
|
28
34
|
register 'r/collection/sort', Collection::Sort
|
29
35
|
register 'r/filter/by_key_record_value', Filter::ByKeyRecordValue
|
30
36
|
register 'r/filter/by_key_value', Filter::ByKeyValue
|
data/lib/realize/value/blank.rb
CHANGED
data/lib/realize/value/null.rb
CHANGED
@@ -6,11 +6,6 @@ module Realize
|
|
6
6
|
class Verbatim
|
7
7
|
acts_as_hashable
|
8
8
|
|
9
|
-
# This is here to satisfy an underlying issue in acts_as_hashable.
|
10
|
-
# The #make calls in the factory and hashable module should be calling #new with no
|
11
|
-
# args if no keys are detected.
|
12
|
-
def initialize(_opts = {}); end
|
13
|
-
|
14
9
|
def transform(_resolver, value, _time, _record)
|
15
10
|
value
|
16
11
|
end
|
data/lib/realize/version.rb
CHANGED
data/realize.gemspec
CHANGED
@@ -11,8 +11,8 @@ Gem::Specification.new do |s|
|
|
11
11
|
Derive and transform a value using a configuration-first pipeline.
|
12
12
|
DESCRIPTION
|
13
13
|
|
14
|
-
s.authors = ['Matthew Ruggio']
|
15
|
-
s.email = ['mruggio@bluemarblepayroll.com']
|
14
|
+
s.authors = ['Matthew Ruggio', 'Dan Dewar']
|
15
|
+
s.email = ['mruggio@bluemarblepayroll.com', 'ddewar@bluemarblepayroll.com']
|
16
16
|
s.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
|
17
17
|
s.bindir = 'exe'
|
18
18
|
s.executables = []
|
@@ -28,7 +28,7 @@ Gem::Specification.new do |s|
|
|
28
28
|
|
29
29
|
s.required_ruby_version = '>= 2.5'
|
30
30
|
|
31
|
-
s.add_dependency('acts_as_hashable', '~>1')
|
31
|
+
s.add_dependency('acts_as_hashable', '~>1', '>=1.2.0')
|
32
32
|
s.add_dependency('objectable', '~>1')
|
33
33
|
|
34
34
|
s.add_development_dependency('guard-rspec', '~>4.7')
|
metadata
CHANGED
@@ -1,14 +1,15 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: realize
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.1.1.pre.alpha
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Matthew Ruggio
|
8
|
+
- Dan Dewar
|
8
9
|
autorequire:
|
9
10
|
bindir: exe
|
10
11
|
cert_chain: []
|
11
|
-
date: 2020-
|
12
|
+
date: 2020-09-09 00:00:00.000000000 Z
|
12
13
|
dependencies:
|
13
14
|
- !ruby/object:Gem::Dependency
|
14
15
|
name: acts_as_hashable
|
@@ -17,6 +18,9 @@ dependencies:
|
|
17
18
|
- - "~>"
|
18
19
|
- !ruby/object:Gem::Version
|
19
20
|
version: '1'
|
21
|
+
- - ">="
|
22
|
+
- !ruby/object:Gem::Version
|
23
|
+
version: 1.2.0
|
20
24
|
type: :runtime
|
21
25
|
prerelease: false
|
22
26
|
version_requirements: !ruby/object:Gem::Requirement
|
@@ -24,6 +28,9 @@ dependencies:
|
|
24
28
|
- - "~>"
|
25
29
|
- !ruby/object:Gem::Version
|
26
30
|
version: '1'
|
31
|
+
- - ">="
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: 1.2.0
|
27
34
|
- !ruby/object:Gem::Dependency
|
28
35
|
name: objectable
|
29
36
|
requirement: !ruby/object:Gem::Requirement
|
@@ -139,6 +146,7 @@ dependencies:
|
|
139
146
|
description: " Derive and transform a value using a configuration-first pipeline.\n"
|
140
147
|
email:
|
141
148
|
- mruggio@bluemarblepayroll.com
|
149
|
+
- ddewar@bluemarblepayroll.com
|
142
150
|
executables: []
|
143
151
|
extensions: []
|
144
152
|
extra_rdoc_files: []
|
@@ -159,6 +167,9 @@ files:
|
|
159
167
|
- exe/.gitkeep
|
160
168
|
- lib/realize.rb
|
161
169
|
- lib/realize/arrays.rb
|
170
|
+
- lib/realize/collection/at_index.rb
|
171
|
+
- lib/realize/collection/first.rb
|
172
|
+
- lib/realize/collection/last.rb
|
162
173
|
- lib/realize/collection/sort.rb
|
163
174
|
- lib/realize/collection/sort/direction.rb
|
164
175
|
- lib/realize/filter/by_key_record_value.rb
|