burner 1.0.0.pre.alpha.3 → 1.0.0.pre.alpha.4
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +37 -10
- data/burner.gemspec +3 -0
- data/lib/burner.rb +9 -0
- data/lib/burner/cli.rb +8 -10
- data/lib/burner/job.rb +4 -2
- data/lib/burner/jobs.rb +27 -11
- data/lib/burner/jobs/collection/arrays_to_objects.rb +43 -0
- data/lib/burner/jobs/collection/graph.rb +43 -0
- data/lib/burner/jobs/collection/objects_to_arrays.rb +54 -0
- data/lib/burner/jobs/collection/shift.rb +43 -0
- data/lib/burner/jobs/collection/transform.rb +64 -0
- data/lib/burner/jobs/collection/transform/attribute.rb +33 -0
- data/lib/burner/jobs/collection/transform/attribute_renderer.rb +36 -0
- data/lib/burner/jobs/collection/unpivot.rb +45 -0
- data/lib/burner/jobs/deserialize/csv.rb +28 -0
- data/lib/burner/jobs/deserialize/json.rb +1 -1
- data/lib/burner/jobs/deserialize/yaml.rb +1 -1
- data/lib/burner/jobs/dummy.rb +1 -1
- data/lib/burner/jobs/echo.rb +2 -2
- data/lib/burner/jobs/io/base.rb +0 -6
- data/lib/burner/jobs/io/exist.rb +2 -2
- data/lib/burner/jobs/io/read.rb +2 -2
- data/lib/burner/jobs/io/write.rb +2 -2
- data/lib/burner/jobs/serialize/csv.rb +38 -0
- data/lib/burner/jobs/serialize/json.rb +1 -1
- data/lib/burner/jobs/serialize/yaml.rb +1 -1
- data/lib/burner/jobs/set.rb +1 -1
- data/lib/burner/jobs/sleep.rb +1 -1
- data/lib/burner/modeling.rb +10 -0
- data/lib/burner/modeling/key_index_mapping.rb +29 -0
- data/lib/burner/payload.rb +11 -3
- data/lib/burner/pipeline.rb +3 -3
- data/lib/burner/step.rb +2 -4
- data/lib/burner/version.rb +1 -1
- metadata +56 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 6e96a8d656466219851ff06ebced5aacac54fe0b6a78de2e1f0e5bf76480e1ca
|
4
|
+
data.tar.gz: a4ba7e8d001f7e25ee7c404ed539089e51f295098d0062d14a3fcf473db64d7d
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 1cc60f86ee2931902ab8e6870f5bda78f953e6740c2a5d76dc306c12457e93511646b9cbfe4db8fa583a3242bc1d361715011ea98ea5682045356a6c6f269808
|
7
|
+
data.tar.gz: 5b9cf06f4426d1629f9c6672ee5c469052863e07bf01079b021494f5b6e0bc4ae0ef57be64e1a37d1652803039a1e11061c5415ca4a83f39672d4c55085feeac
|
data/README.md
CHANGED
@@ -73,19 +73,21 @@ params = {
|
|
73
73
|
input_file: 'input.json',
|
74
74
|
output_file: 'output.yaml'
|
75
75
|
}
|
76
|
+
|
77
|
+
payload = Burner::Payload.new(params: params)
|
76
78
|
````
|
77
79
|
|
78
80
|
Assuming we are running this script from a directory where an `input.json` file exists, we can then programatically process the pipeline:
|
79
81
|
|
80
82
|
````ruby
|
81
|
-
Burner::Pipeline.make(pipeline).execute(
|
83
|
+
Burner::Pipeline.make(pipeline).execute(payload: payload)
|
82
84
|
````
|
83
85
|
|
84
86
|
We should now see a output.yaml file created.
|
85
87
|
|
86
88
|
Some notes:
|
87
89
|
|
88
|
-
* Some values are able to be string-interpolated using the provided params. This allows for the passing runtime configuration/data into pipelines/jobs.
|
90
|
+
* Some values are able to be string-interpolated using the provided Payload#params. This allows for the passing runtime configuration/data into pipelines/jobs.
|
89
91
|
* The job's ID can be accessed using the `__id` key.
|
90
92
|
* The current job's payload value can be accessed using the `__value` key.
|
91
93
|
* Jobs can be re-used (just like the output_id and output_value jobs).
|
@@ -116,8 +118,9 @@ end
|
|
116
118
|
|
117
119
|
string_out = StringOut.new
|
118
120
|
output = Burner::Output.new(outs: string_out)
|
121
|
+
payload = Burner::Payload.new(params: params)
|
119
122
|
|
120
|
-
Burner::Pipeline.make(pipeline).execute(output: output,
|
123
|
+
Burner::Pipeline.make(pipeline).execute(output: output, payload: payload)
|
121
124
|
|
122
125
|
log = string_out.read
|
123
126
|
````
|
@@ -228,15 +231,37 @@ Burner::Cli.new(args).invoke
|
|
228
231
|
|
229
232
|
This library only ships with very basic, rudimentary jobs that are meant to just serve as a baseline:
|
230
233
|
|
234
|
+
#### Collection
|
235
|
+
|
236
|
+
* **collection/arrays_to_objects** [mappings]: Convert an array of arrays to an array of objects.
|
237
|
+
* **collection/graph** [config, key]: Use (Hashematics)[https://github.com/bluemarblepayroll/hashematics] to turn a flat array of objects into a deeply nested object tree.
|
238
|
+
* **collection/objects_to_arrays** [mappings]: Convert an array of objects to an array of arrays.
|
239
|
+
* **collection/shift** [amount]: Remove the first N number of elements from an array.
|
240
|
+
* **collection/transform** [attributes, exclusive, separator]: Iterate over all objects and transform each key per the attribute transformers specifications. If exclusive is set to false then the current object will be overridden/merged. Separator can also be set for key path support. This job uses (Realize)[https://github.com/bluemarblepayroll/realize], which provides its own extendable value-transformation pipeline.
|
241
|
+
* **collection/unpivot** [pivot_set]: Take an array of objects and unpivot specific sets of keys into rows. Under the hood it uses [HashMath's Unpivot class](https://github.com/bluemarblepayroll/hash_math#unpivot-hash-key-coalescence-and-row-extrapolation).
|
242
|
+
|
243
|
+
#### De-serialization
|
244
|
+
|
245
|
+
* **deserialize/csv** []: Take a CSV string and de-serialize into object(s). Currently it will return an array of arrays, with each nested array representing one row.
|
231
246
|
* **deserialize/json** []: Treat input as a string and de-serialize it to JSON.
|
232
247
|
* **deserialize/yaml** [safe]: Treat input as a string and de-serialize it to YAML. By default it will try and (safely de-serialize)[https://ruby-doc.org/stdlib-2.6.1/libdoc/psych/rdoc/Psych.html#method-c-safe_load] it (only using core classes). If you wish to de-serialize it to any class type, pass in `safe: false`
|
233
|
-
|
234
|
-
|
235
|
-
|
236
|
-
* **io/
|
237
|
-
* **io/
|
248
|
+
|
249
|
+
#### IO
|
250
|
+
|
251
|
+
* **io/exist** [path, short_circuit]: Check to see if a file exists. The path parameter can be interpolated using `Payload#params`. If short_circuit was set to true (defaults to false) and the file does not exist then the pipeline will be short-circuited.
|
252
|
+
* **io/read** [binary, path]: Read in a local file. The path parameter can be interpolated using `Payload#params`. If the contents are binary, pass in `binary: true` to open it up in binary+read mode.
|
253
|
+
* **io/write** [binary, path]: Write to a local file. The path parameter can be interpolated using `Payload#params`. If the contents are binary, pass in `binary: true` to open it up in binary+write mode.
|
254
|
+
|
255
|
+
#### Serialization
|
256
|
+
|
257
|
+
* **serialize/csv** []: Take an array of arrays and create a CSV.
|
238
258
|
* **serialize/json** []: Convert value to JSON.
|
239
259
|
* **serialize/yaml** []: Convert value to YAML.
|
260
|
+
|
261
|
+
#### General
|
262
|
+
|
263
|
+
* **dummy** []: Do nothing
|
264
|
+
* **echo** [message]: Write a message to the output. The message parameter can be interpolated using `Payload#params`.
|
240
265
|
* **set** [value]: Set the value to any arbitrary value.
|
241
266
|
* **sleep** [seconds]: Sleep the thread for X number of seconds.
|
242
267
|
|
@@ -249,7 +274,7 @@ Let's say we would like to register a job to parse a CSV:
|
|
249
274
|
|
250
275
|
````ruby
|
251
276
|
class ParseCsv < Burner::Job
|
252
|
-
def perform(output, payload
|
277
|
+
def perform(output, payload)
|
253
278
|
payload.value = CSV.parse(payload.value, headers: true).map(&:to_h)
|
254
279
|
|
255
280
|
nil
|
@@ -309,7 +334,9 @@ params = {
|
|
309
334
|
output_file: File.join(TEMP_DIR, "#{SecureRandom.uuid}.yaml")
|
310
335
|
}
|
311
336
|
|
312
|
-
Burner::
|
337
|
+
payload = Burner::Payload.new(params: params)
|
338
|
+
|
339
|
+
Burner::Pipeline.make(pipeline).execute(output: output, payload: payload)
|
313
340
|
````
|
314
341
|
|
315
342
|
## Contributing
|
data/burner.gemspec
CHANGED
@@ -29,7 +29,10 @@ Gem::Specification.new do |s|
|
|
29
29
|
s.required_ruby_version = '>= 2.5'
|
30
30
|
|
31
31
|
s.add_dependency('acts_as_hashable', '~>1.2')
|
32
|
+
s.add_dependency('hashematics', '~>1.1')
|
33
|
+
s.add_dependency('hash_math', '~>1.2')
|
32
34
|
s.add_dependency('objectable', '~>1.0')
|
35
|
+
s.add_dependency('realize', '~>1.2')
|
33
36
|
s.add_dependency('stringento', '~>2.1')
|
34
37
|
|
35
38
|
s.add_development_dependency('guard-rspec', '~>4.7')
|
data/lib/burner.rb
CHANGED
@@ -9,12 +9,21 @@
|
|
9
9
|
|
10
10
|
require 'acts_as_hashable'
|
11
11
|
require 'benchmark'
|
12
|
+
require 'csv'
|
12
13
|
require 'forwardable'
|
14
|
+
require 'hash_math'
|
15
|
+
require 'hashematics'
|
13
16
|
require 'json'
|
14
17
|
require 'objectable'
|
18
|
+
require 'realize'
|
15
19
|
require 'securerandom'
|
16
20
|
require 'singleton'
|
17
21
|
require 'stringento'
|
22
|
+
require 'time'
|
18
23
|
require 'yaml'
|
19
24
|
|
25
|
+
# Common/Shared
|
26
|
+
require_relative 'burner/modeling'
|
27
|
+
|
28
|
+
# Main Entrypoint(s)
|
20
29
|
require_relative 'burner/cli'
|
data/lib/burner/cli.rb
CHANGED
@@ -12,20 +12,18 @@ require_relative 'pipeline'
|
|
12
12
|
module Burner
|
13
13
|
# Process a single string as a Pipeline. This is mainly to back the command-line interface.
|
14
14
|
class Cli
|
15
|
-
attr_reader :
|
15
|
+
attr_reader :payload, :pipeline
|
16
16
|
|
17
17
|
def initialize(args)
|
18
|
-
path
|
19
|
-
|
20
|
-
config
|
21
|
-
@pipeline
|
22
|
-
@
|
18
|
+
path = args.first
|
19
|
+
params = extract_cli_params(args)
|
20
|
+
config = read_yaml(path)
|
21
|
+
@pipeline = Burner::Pipeline.make(jobs: config['jobs'], steps: config['steps'])
|
22
|
+
@payload = Payload.new(params: params)
|
23
23
|
end
|
24
24
|
|
25
|
-
def execute
|
26
|
-
|
27
|
-
|
28
|
-
pipeline.execute(output: output, params: final_params, payload: payload)
|
25
|
+
def execute
|
26
|
+
pipeline.execute(payload: payload)
|
29
27
|
end
|
30
28
|
|
31
29
|
private
|
data/lib/burner/job.rb
CHANGED
@@ -28,8 +28,10 @@ module Burner
|
|
28
28
|
|
29
29
|
private
|
30
30
|
|
31
|
-
def
|
32
|
-
|
31
|
+
def job_string_template(expression, output, payload)
|
32
|
+
templatable_params = payload.params.merge(__id: output.id, __value: payload.value)
|
33
|
+
|
34
|
+
string_template.evaluate(expression, templatable_params)
|
33
35
|
end
|
34
36
|
end
|
35
37
|
end
|
data/lib/burner/jobs.rb
CHANGED
@@ -8,6 +8,13 @@
|
|
8
8
|
#
|
9
9
|
|
10
10
|
require_relative 'job'
|
11
|
+
require_relative 'jobs/collection/arrays_to_objects'
|
12
|
+
require_relative 'jobs/collection/graph'
|
13
|
+
require_relative 'jobs/collection/objects_to_arrays'
|
14
|
+
require_relative 'jobs/collection/shift'
|
15
|
+
require_relative 'jobs/collection/transform'
|
16
|
+
require_relative 'jobs/collection/unpivot'
|
17
|
+
require_relative 'jobs/deserialize/csv'
|
11
18
|
require_relative 'jobs/deserialize/json'
|
12
19
|
require_relative 'jobs/deserialize/yaml'
|
13
20
|
require_relative 'jobs/dummy'
|
@@ -15,6 +22,7 @@ require_relative 'jobs/echo'
|
|
15
22
|
require_relative 'jobs/io/exist'
|
16
23
|
require_relative 'jobs/io/read'
|
17
24
|
require_relative 'jobs/io/write'
|
25
|
+
require_relative 'jobs/serialize/csv'
|
18
26
|
require_relative 'jobs/serialize/json'
|
19
27
|
require_relative 'jobs/serialize/yaml'
|
20
28
|
require_relative 'jobs/set'
|
@@ -27,16 +35,24 @@ module Burner
|
|
27
35
|
class Jobs
|
28
36
|
acts_as_hashable_factory
|
29
37
|
|
30
|
-
register '
|
31
|
-
register '
|
32
|
-
register '
|
33
|
-
register '
|
34
|
-
register '
|
35
|
-
register '
|
36
|
-
register '
|
37
|
-
register '
|
38
|
-
register '
|
39
|
-
register '
|
40
|
-
register '
|
38
|
+
register 'collection/arrays_to_objects', Collection::ArraysToObjects
|
39
|
+
register 'collection/graph', Collection::Graph
|
40
|
+
register 'collection/objects_to_arrays', Collection::ObjectsToArrays
|
41
|
+
register 'collection/shift', Collection::Shift
|
42
|
+
register 'collection/transform', Collection::Transform
|
43
|
+
register 'collection/unpivot', Collection::Unpivot
|
44
|
+
register 'deserialize/csv', Deserialize::Csv
|
45
|
+
register 'deserialize/json', Deserialize::Json
|
46
|
+
register 'deserialize/yaml', Deserialize::Yaml
|
47
|
+
register 'dummy', '', Dummy
|
48
|
+
register 'echo', Echo
|
49
|
+
register 'io/exist', IO::Exist
|
50
|
+
register 'io/read', IO::Read
|
51
|
+
register 'io/write', IO::Write
|
52
|
+
register 'serialize/csv', Serialize::Csv
|
53
|
+
register 'serialize/json', Serialize::Json
|
54
|
+
register 'serialize/yaml', Serialize::Yaml
|
55
|
+
register 'set', Set
|
56
|
+
register 'sleep', Sleep
|
41
57
|
end
|
42
58
|
end
|
@@ -0,0 +1,43 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
#
|
4
|
+
# Copyright (c) 2020-present, Blue Marble Payroll, LLC
|
5
|
+
#
|
6
|
+
# This source code is licensed under the MIT license found in the
|
7
|
+
# LICENSE file in the root directory of this source tree.
|
8
|
+
#
|
9
|
+
|
10
|
+
module Burner
|
11
|
+
class Jobs
|
12
|
+
module Collection
|
13
|
+
# Convert an array of arrays to an array of objects.
|
14
|
+
# Expected Payload#value input: array of arrays.
|
15
|
+
# Payload#value output: An array of hashes.
|
16
|
+
class ArraysToObjects < Job
|
17
|
+
attr_reader :mappings
|
18
|
+
|
19
|
+
def initialize(name:, mappings: [])
|
20
|
+
super(name: name)
|
21
|
+
|
22
|
+
@mappings = Modeling::KeyIndexMapping.array(mappings)
|
23
|
+
|
24
|
+
freeze
|
25
|
+
end
|
26
|
+
|
27
|
+
def perform(_output, payload)
|
28
|
+
payload.value = (payload.value || []).map { |array| index_to_key_map(array) }
|
29
|
+
|
30
|
+
nil
|
31
|
+
end
|
32
|
+
|
33
|
+
private
|
34
|
+
|
35
|
+
def index_to_key_map(array)
|
36
|
+
mappings.each_with_object({}) do |mapping, memo|
|
37
|
+
memo[mapping.key] = array[mapping.index]
|
38
|
+
end
|
39
|
+
end
|
40
|
+
end
|
41
|
+
end
|
42
|
+
end
|
43
|
+
end
|
@@ -0,0 +1,43 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
#
|
4
|
+
# Copyright (c) 2020-present, Blue Marble Payroll, LLC
|
5
|
+
#
|
6
|
+
# This source code is licensed under the MIT license found in the
|
7
|
+
# LICENSE file in the root directory of this source tree.
|
8
|
+
#
|
9
|
+
|
10
|
+
module Burner
|
11
|
+
class Jobs
|
12
|
+
module Collection
|
13
|
+
# Take an array of (denormalized) objects and create an object hierarchy from them.
|
14
|
+
# Under the hood it uses Hashematics: https://github.com/bluemarblepayroll/hashematics.
|
15
|
+
# Expected Payload#value input: array of objects.
|
16
|
+
# Payload#value output: An array of objects.
|
17
|
+
class Graph < Job
|
18
|
+
attr_reader :key, :groups
|
19
|
+
|
20
|
+
def initialize(name:, key:, config: Hashematics::Configuration.new)
|
21
|
+
super(name: name)
|
22
|
+
|
23
|
+
raise ArgumentError, 'key is required' if key.to_s.empty?
|
24
|
+
|
25
|
+
@groups = Hashematics::Configuration.new(config).groups
|
26
|
+
@key = key.to_s
|
27
|
+
|
28
|
+
freeze
|
29
|
+
end
|
30
|
+
|
31
|
+
def perform(output, payload)
|
32
|
+
graph = Hashematics::Graph.new(groups).add(payload.value || [])
|
33
|
+
|
34
|
+
output.detail("Graphing: #{key}")
|
35
|
+
|
36
|
+
payload.value = graph.data(key)
|
37
|
+
|
38
|
+
nil
|
39
|
+
end
|
40
|
+
end
|
41
|
+
end
|
42
|
+
end
|
43
|
+
end
|
@@ -0,0 +1,54 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
#
|
4
|
+
# Copyright (c) 2020-present, Blue Marble Payroll, LLC
|
5
|
+
#
|
6
|
+
# This source code is licensed under the MIT license found in the
|
7
|
+
# LICENSE file in the root directory of this source tree.
|
8
|
+
#
|
9
|
+
|
10
|
+
module Burner
|
11
|
+
class Jobs
|
12
|
+
module Collection
|
13
|
+
# Convert an array of objects to an array of arrays. You can leverage the separator
|
14
|
+
# option to support key paths and nested objects.
|
15
|
+
# Expected Payload#value input: array of hashes.
|
16
|
+
# Payload#value output: An array of arrays.
|
17
|
+
class ObjectsToArrays < Job
|
18
|
+
attr_reader :mappings
|
19
|
+
|
20
|
+
# If you wish to support nested objects you can pass in a string to use as a
|
21
|
+
# key path separator. For example: if you would like to recognize dot-notation for
|
22
|
+
# nested hashes then set separator to '.'.
|
23
|
+
def initialize(name:, mappings: [], separator: '')
|
24
|
+
super(name: name)
|
25
|
+
|
26
|
+
@mappings = Modeling::KeyIndexMapping.array(mappings)
|
27
|
+
@resolver = Objectable.resolver(separator: separator.to_s)
|
28
|
+
|
29
|
+
freeze
|
30
|
+
end
|
31
|
+
|
32
|
+
def perform(_output, payload)
|
33
|
+
payload.value = (payload.value || []).map { |object| key_to_index_map(object) }
|
34
|
+
|
35
|
+
nil
|
36
|
+
end
|
37
|
+
|
38
|
+
private
|
39
|
+
|
40
|
+
attr_reader :resolver
|
41
|
+
|
42
|
+
def key_to_index_map(object)
|
43
|
+
mappings.each_with_object(prototype_array) do |mapping, memo|
|
44
|
+
memo[mapping.index] = resolver.get(object, mapping.key)
|
45
|
+
end
|
46
|
+
end
|
47
|
+
|
48
|
+
def prototype_array
|
49
|
+
Array.new(mappings.length)
|
50
|
+
end
|
51
|
+
end
|
52
|
+
end
|
53
|
+
end
|
54
|
+
end
|
@@ -0,0 +1,43 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
#
|
4
|
+
# Copyright (c) 2020-present, Blue Marble Payroll, LLC
|
5
|
+
#
|
6
|
+
# This source code is licensed under the MIT license found in the
|
7
|
+
# LICENSE file in the root directory of this source tree.
|
8
|
+
#
|
9
|
+
|
10
|
+
module Burner
|
11
|
+
class Jobs
|
12
|
+
module Collection
|
13
|
+
# Take an array and remove the first N elements, where N is specified by the amount
|
14
|
+
# attribute.
|
15
|
+
# Expected Payload#value input: nothing.
|
16
|
+
# Payload#value output: An array with N beginning elements removed.
|
17
|
+
class Shift < Job
|
18
|
+
DEFAULT_AMOUNT = 0
|
19
|
+
|
20
|
+
private_constant :DEFAULT_AMOUNT
|
21
|
+
|
22
|
+
attr_reader :amount
|
23
|
+
|
24
|
+
def initialize(name:, amount: DEFAULT_AMOUNT)
|
25
|
+
super(name: name)
|
26
|
+
|
27
|
+
@amount = amount.to_i
|
28
|
+
|
29
|
+
freeze
|
30
|
+
end
|
31
|
+
|
32
|
+
def perform(output, payload)
|
33
|
+
output.detail("Shifting #{amount} entries.")
|
34
|
+
|
35
|
+
payload.value ||= []
|
36
|
+
payload.value.shift(amount)
|
37
|
+
|
38
|
+
nil
|
39
|
+
end
|
40
|
+
end
|
41
|
+
end
|
42
|
+
end
|
43
|
+
end
|
@@ -0,0 +1,64 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
#
|
4
|
+
# Copyright (c) 2020-present, Blue Marble Payroll, LLC
|
5
|
+
#
|
6
|
+
# This source code is licensed under the MIT license found in the
|
7
|
+
# LICENSE file in the root directory of this source tree.
|
8
|
+
#
|
9
|
+
|
10
|
+
require_relative 'transform/attribute'
|
11
|
+
require_relative 'transform/attribute_renderer'
|
12
|
+
|
13
|
+
module Burner
|
14
|
+
class Jobs
|
15
|
+
module Collection
|
16
|
+
# Iterate over all objects and return a new set of transformed objects. Under the hood
|
17
|
+
# this uses the Realize library: https://github.com/bluemarblepayroll/realize
|
18
|
+
# Expected Payload#value input: array of objects.
|
19
|
+
# Payload#value output: An array of objects.
|
20
|
+
class Transform < Job
|
21
|
+
BLANK = ''
|
22
|
+
|
23
|
+
attr_reader :attribute_renderers,
|
24
|
+
:exclusive,
|
25
|
+
:resolver
|
26
|
+
|
27
|
+
def initialize(name:, attributes: [], exclusive: false, separator: BLANK)
|
28
|
+
super(name: name)
|
29
|
+
|
30
|
+
@resolver = Objectable.resolver(separator: separator)
|
31
|
+
@exclusive = exclusive || false
|
32
|
+
|
33
|
+
@attribute_renderers = Attribute.array(attributes)
|
34
|
+
.map { |a| AttributeRenderer.new(a, resolver) }
|
35
|
+
|
36
|
+
freeze
|
37
|
+
end
|
38
|
+
|
39
|
+
def perform(output, payload)
|
40
|
+
payload.value = (payload.value || []).map { |row| transform(row, payload.time) }
|
41
|
+
|
42
|
+
attr_count = attribute_renderers.length
|
43
|
+
row_count = payload.value.length
|
44
|
+
|
45
|
+
output.detail("Transformed #{attr_count} attributes(s) for #{row_count} row(s)")
|
46
|
+
|
47
|
+
nil
|
48
|
+
end
|
49
|
+
|
50
|
+
private
|
51
|
+
|
52
|
+
def transform(row, time)
|
53
|
+
outgoing_row = exclusive ? {} : row
|
54
|
+
|
55
|
+
attribute_renderers.each_with_object(outgoing_row) do |attribute_renderer, memo|
|
56
|
+
value = attribute_renderer.transform(row, time)
|
57
|
+
|
58
|
+
resolver.set(memo, attribute_renderer.key, value)
|
59
|
+
end
|
60
|
+
end
|
61
|
+
end
|
62
|
+
end
|
63
|
+
end
|
64
|
+
end
|
@@ -0,0 +1,33 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
#
|
4
|
+
# Copyright (c) 2020-present, Blue Marble Payroll, LLC
|
5
|
+
#
|
6
|
+
# This source code is licensed under the MIT license found in the
|
7
|
+
# LICENSE file in the root directory of this source tree.
|
8
|
+
#
|
9
|
+
|
10
|
+
module Burner
|
11
|
+
class Jobs
|
12
|
+
module Collection
|
13
|
+
class Transform < Job
|
14
|
+
# Defines a top-level key and the associated transformers for deriving the final value
|
15
|
+
# to set the key to.
|
16
|
+
class Attribute
|
17
|
+
acts_as_hashable
|
18
|
+
|
19
|
+
attr_reader :key, :transformers
|
20
|
+
|
21
|
+
def initialize(key:, transformers: [])
|
22
|
+
raise ArgumentError, 'key is required' if key.to_s.empty?
|
23
|
+
|
24
|
+
@key = key.to_s
|
25
|
+
@transformers = Realize::Transformers.array(transformers)
|
26
|
+
|
27
|
+
freeze
|
28
|
+
end
|
29
|
+
end
|
30
|
+
end
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
@@ -0,0 +1,36 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
#
|
4
|
+
# Copyright (c) 2020-present, Blue Marble Payroll, LLC
|
5
|
+
#
|
6
|
+
# This source code is licensed under the MIT license found in the
|
7
|
+
# LICENSE file in the root directory of this source tree.
|
8
|
+
#
|
9
|
+
|
10
|
+
module Burner
|
11
|
+
class Jobs
|
12
|
+
module Collection
|
13
|
+
class Transform < Job
|
14
|
+
# Composed of an Attribute instance and a Pipeline instance. It knows how to
|
15
|
+
# render/transform an Attribute. Since this library is data-first, these intermediary
|
16
|
+
# objects are necessary for non-data-centric modeling.
|
17
|
+
class AttributeRenderer
|
18
|
+
extend Forwardable
|
19
|
+
|
20
|
+
attr_reader :attribute, :pipeline
|
21
|
+
|
22
|
+
def_delegators :attribute, :key
|
23
|
+
|
24
|
+
def_delegators :pipeline, :transform
|
25
|
+
|
26
|
+
def initialize(attribute, resolver)
|
27
|
+
@attribute = attribute
|
28
|
+
@pipeline = Realize::Pipeline.new(attribute.transformers, resolver: resolver)
|
29
|
+
|
30
|
+
freeze
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
34
|
+
end
|
35
|
+
end
|
36
|
+
end
|
@@ -0,0 +1,45 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
#
|
4
|
+
# Copyright (c) 2020-present, Blue Marble Payroll, LLC
|
5
|
+
#
|
6
|
+
# This source code is licensed under the MIT license found in the
|
7
|
+
# LICENSE file in the root directory of this source tree.
|
8
|
+
#
|
9
|
+
|
10
|
+
module Burner
|
11
|
+
class Jobs
|
12
|
+
module Collection
|
13
|
+
# Take an array of objects and un-pivot groups of keys into rows.
|
14
|
+
# Under the hood it uses HashMath's Unpivot class:
|
15
|
+
# https://github.com/bluemarblepayroll/hash_math
|
16
|
+
# Expected Payload#value input: array of objects.
|
17
|
+
# Payload#value output: An array of objects.
|
18
|
+
class Unpivot < Job
|
19
|
+
attr_reader :unpivot
|
20
|
+
|
21
|
+
def initialize(name:, pivot_set: HashMath::Unpivot::PivotSet.new)
|
22
|
+
super(name: name)
|
23
|
+
|
24
|
+
@unpivot = HashMath::Unpivot.new(pivot_set)
|
25
|
+
|
26
|
+
freeze
|
27
|
+
end
|
28
|
+
|
29
|
+
def perform(output, payload)
|
30
|
+
pivot_count = unpivot.pivot_set.pivots.length
|
31
|
+
key_count = unpivot.pivot_set.pivots.map { |p| p.keys.length }.sum
|
32
|
+
object_count = payload.value&.length || 0
|
33
|
+
|
34
|
+
message = "#{pivot_count} Pivots, Key(s): #{key_count} key(s), #{object_count} objects(s)"
|
35
|
+
|
36
|
+
output.detail(message)
|
37
|
+
|
38
|
+
payload.value = (payload.value || []).flat_map { |object| unpivot.expand(object) }
|
39
|
+
|
40
|
+
nil
|
41
|
+
end
|
42
|
+
end
|
43
|
+
end
|
44
|
+
end
|
45
|
+
end
|
@@ -0,0 +1,28 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
#
|
4
|
+
# Copyright (c) 2020-present, Blue Marble Payroll, LLC
|
5
|
+
#
|
6
|
+
# This source code is licensed under the MIT license found in the
|
7
|
+
# LICENSE file in the root directory of this source tree.
|
8
|
+
#
|
9
|
+
|
10
|
+
module Burner
|
11
|
+
class Jobs
|
12
|
+
module Deserialize
|
13
|
+
# Take a CSV string and de-serialize into object(s).
|
14
|
+
# Expected Payload#value input: nothing.
|
15
|
+
# Payload#value output: an array of arrays. Each inner array represents one data row.
|
16
|
+
class Csv < Job
|
17
|
+
# This currently only supports returning an array of arrays, including the header row.
|
18
|
+
# In the future this could be extended to offer more customizable options, such as
|
19
|
+
# making it return an array of hashes with the columns mapped, etc.)
|
20
|
+
def perform(_output, payload)
|
21
|
+
payload.value = CSV.new(payload.value, headers: false).to_a
|
22
|
+
|
23
|
+
nil
|
24
|
+
end
|
25
|
+
end
|
26
|
+
end
|
27
|
+
end
|
28
|
+
end
|
@@ -27,7 +27,7 @@ module Burner
|
|
27
27
|
# in a sandbox. By default, though, we will try and drive them towards using it
|
28
28
|
# in the safer alternative.
|
29
29
|
# rubocop:disable Security/YAMLLoad
|
30
|
-
def perform(output, payload
|
30
|
+
def perform(output, payload)
|
31
31
|
output.detail('Warning: loading YAML not using safe_load.') unless safe
|
32
32
|
|
33
33
|
payload.value = safe ? YAML.safe_load(payload.value) : YAML.load(payload.value)
|
data/lib/burner/jobs/dummy.rb
CHANGED
data/lib/burner/jobs/echo.rb
CHANGED
@@ -21,8 +21,8 @@ module Burner
|
|
21
21
|
freeze
|
22
22
|
end
|
23
23
|
|
24
|
-
def perform(output,
|
25
|
-
compiled_message =
|
24
|
+
def perform(output, payload)
|
25
|
+
compiled_message = job_string_template(message, output, payload)
|
26
26
|
|
27
27
|
output.detail(compiled_message)
|
28
28
|
|
data/lib/burner/jobs/io/base.rb
CHANGED
data/lib/burner/jobs/io/exist.rb
CHANGED
@@ -25,8 +25,8 @@ module Burner
|
|
25
25
|
freeze
|
26
26
|
end
|
27
27
|
|
28
|
-
def perform(output,
|
29
|
-
compiled_path =
|
28
|
+
def perform(output, payload)
|
29
|
+
compiled_path = job_string_template(path, output, payload)
|
30
30
|
|
31
31
|
exists = File.exist?(compiled_path)
|
32
32
|
verb = exists ? 'does' : 'does not'
|
data/lib/burner/jobs/io/read.rb
CHANGED
@@ -24,8 +24,8 @@ module Burner
|
|
24
24
|
freeze
|
25
25
|
end
|
26
26
|
|
27
|
-
def perform(output, payload
|
28
|
-
compiled_path =
|
27
|
+
def perform(output, payload)
|
28
|
+
compiled_path = job_string_template(path, output, payload)
|
29
29
|
|
30
30
|
output.detail("Reading: #{compiled_path}")
|
31
31
|
|
data/lib/burner/jobs/io/write.rb
CHANGED
@@ -24,8 +24,8 @@ module Burner
|
|
24
24
|
freeze
|
25
25
|
end
|
26
26
|
|
27
|
-
def perform(output, payload
|
28
|
-
compiled_path =
|
27
|
+
def perform(output, payload)
|
28
|
+
compiled_path = job_string_template(path, output, payload)
|
29
29
|
|
30
30
|
ensure_directory_exists(output, compiled_path)
|
31
31
|
|
@@ -0,0 +1,38 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
#
|
4
|
+
# Copyright (c) 2020-present, Blue Marble Payroll, LLC
|
5
|
+
#
|
6
|
+
# This source code is licensed under the MIT license found in the
|
7
|
+
# LICENSE file in the root directory of this source tree.
|
8
|
+
#
|
9
|
+
|
10
|
+
module Burner
|
11
|
+
class Jobs
|
12
|
+
module Serialize
|
13
|
+
# Take an array of arrays and create a CSV.
|
14
|
+
# Expected Payload#value input: array of arrays.
|
15
|
+
# Payload#value output: a serialized CSV string.
|
16
|
+
class Csv < Job
|
17
|
+
def perform(_output, payload)
|
18
|
+
payload.value = CSV.generate(options) do |csv|
|
19
|
+
(payload.value || []).each do |row|
|
20
|
+
csv << row
|
21
|
+
end
|
22
|
+
end
|
23
|
+
|
24
|
+
nil
|
25
|
+
end
|
26
|
+
|
27
|
+
private
|
28
|
+
|
29
|
+
def options
|
30
|
+
{
|
31
|
+
headers: false,
|
32
|
+
write_headers: false
|
33
|
+
}
|
34
|
+
end
|
35
|
+
end
|
36
|
+
end
|
37
|
+
end
|
38
|
+
end
|
data/lib/burner/jobs/set.rb
CHANGED
data/lib/burner/jobs/sleep.rb
CHANGED
@@ -0,0 +1,10 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
#
|
4
|
+
# Copyright (c) 2020-present, Blue Marble Payroll, LLC
|
5
|
+
#
|
6
|
+
# This source code is licensed under the MIT license found in the
|
7
|
+
# LICENSE file in the root directory of this source tree.
|
8
|
+
#
|
9
|
+
|
10
|
+
require_relative 'modeling/key_index_mapping'
|
@@ -0,0 +1,29 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
#
|
4
|
+
# Copyright (c) 2020-present, Blue Marble Payroll, LLC
|
5
|
+
#
|
6
|
+
# This source code is licensed under the MIT license found in the
|
7
|
+
# LICENSE file in the root directory of this source tree.
|
8
|
+
#
|
9
|
+
|
10
|
+
module Burner
|
11
|
+
module Modeling
|
12
|
+
# Generic relationship between a numeric index and a key.
|
13
|
+
class KeyIndexMapping
|
14
|
+
acts_as_hashable
|
15
|
+
|
16
|
+
attr_reader :index, :key
|
17
|
+
|
18
|
+
def initialize(index:, key:)
|
19
|
+
raise ArgumentError, 'index is required' if index.to_s.empty?
|
20
|
+
raise ArgumentError, 'key is required' if key.to_s.empty?
|
21
|
+
|
22
|
+
@index = index.to_i
|
23
|
+
@key = key.to_s
|
24
|
+
|
25
|
+
freeze
|
26
|
+
end
|
27
|
+
end
|
28
|
+
end
|
29
|
+
end
|
data/lib/burner/payload.rb
CHANGED
@@ -18,10 +18,18 @@ module Burner
|
|
18
18
|
class Payload
|
19
19
|
attr_accessor :value
|
20
20
|
|
21
|
-
attr_reader :
|
21
|
+
attr_reader :params,
|
22
|
+
:time,
|
23
|
+
:written_files
|
22
24
|
|
23
|
-
def initialize(
|
24
|
-
|
25
|
+
def initialize(
|
26
|
+
params: {},
|
27
|
+
time: Time.now.utc,
|
28
|
+
value: nil,
|
29
|
+
written_files: []
|
30
|
+
)
|
31
|
+
@params = params || {}
|
32
|
+
@time = time || Time.now.utc
|
25
33
|
@value = value
|
26
34
|
@written_files = written_files || []
|
27
35
|
end
|
data/lib/burner/pipeline.rb
CHANGED
@@ -35,15 +35,15 @@ module Burner
|
|
35
35
|
end
|
36
36
|
|
37
37
|
# The main entry-point for kicking off a pipeline.
|
38
|
-
def execute(output: Output.new,
|
38
|
+
def execute(output: Output.new, payload: Payload.new)
|
39
39
|
output.write("Pipeline started with #{steps.length} step(s)")
|
40
40
|
|
41
|
-
output_params(params, output)
|
41
|
+
output_params(payload.params, output)
|
42
42
|
output.ruler
|
43
43
|
|
44
44
|
time_in_seconds = Benchmark.measure do
|
45
45
|
steps.each do |step|
|
46
|
-
return_value = step.perform(output, payload
|
46
|
+
return_value = step.perform(output, payload)
|
47
47
|
|
48
48
|
if return_value.is_a?(FalseClass)
|
49
49
|
output.detail('Job returned false, ending pipeline.')
|
data/lib/burner/step.rb
CHANGED
@@ -28,15 +28,13 @@ module Burner
|
|
28
28
|
freeze
|
29
29
|
end
|
30
30
|
|
31
|
-
def perform(output, payload
|
31
|
+
def perform(output, payload)
|
32
32
|
return_value = nil
|
33
33
|
|
34
34
|
output.title("#{job.class.name}#{SEPARATOR}#{job.name}")
|
35
35
|
|
36
36
|
time_in_seconds = Benchmark.measure do
|
37
|
-
|
38
|
-
|
39
|
-
return_value = job.perform(output, payload, job_params)
|
37
|
+
return_value = job.perform(output, payload)
|
40
38
|
end.real.round(3)
|
41
39
|
|
42
40
|
output.complete(time_in_seconds)
|
data/lib/burner/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: burner
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.0.0.pre.alpha.
|
4
|
+
version: 1.0.0.pre.alpha.4
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Matthew Ruggio
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2020-10-
|
11
|
+
date: 2020-10-20 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: acts_as_hashable
|
@@ -24,6 +24,34 @@ dependencies:
|
|
24
24
|
- - "~>"
|
25
25
|
- !ruby/object:Gem::Version
|
26
26
|
version: '1.2'
|
27
|
+
- !ruby/object:Gem::Dependency
|
28
|
+
name: hashematics
|
29
|
+
requirement: !ruby/object:Gem::Requirement
|
30
|
+
requirements:
|
31
|
+
- - "~>"
|
32
|
+
- !ruby/object:Gem::Version
|
33
|
+
version: '1.1'
|
34
|
+
type: :runtime
|
35
|
+
prerelease: false
|
36
|
+
version_requirements: !ruby/object:Gem::Requirement
|
37
|
+
requirements:
|
38
|
+
- - "~>"
|
39
|
+
- !ruby/object:Gem::Version
|
40
|
+
version: '1.1'
|
41
|
+
- !ruby/object:Gem::Dependency
|
42
|
+
name: hash_math
|
43
|
+
requirement: !ruby/object:Gem::Requirement
|
44
|
+
requirements:
|
45
|
+
- - "~>"
|
46
|
+
- !ruby/object:Gem::Version
|
47
|
+
version: '1.2'
|
48
|
+
type: :runtime
|
49
|
+
prerelease: false
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - "~>"
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: '1.2'
|
27
55
|
- !ruby/object:Gem::Dependency
|
28
56
|
name: objectable
|
29
57
|
requirement: !ruby/object:Gem::Requirement
|
@@ -38,6 +66,20 @@ dependencies:
|
|
38
66
|
- - "~>"
|
39
67
|
- !ruby/object:Gem::Version
|
40
68
|
version: '1.0'
|
69
|
+
- !ruby/object:Gem::Dependency
|
70
|
+
name: realize
|
71
|
+
requirement: !ruby/object:Gem::Requirement
|
72
|
+
requirements:
|
73
|
+
- - "~>"
|
74
|
+
- !ruby/object:Gem::Version
|
75
|
+
version: '1.2'
|
76
|
+
type: :runtime
|
77
|
+
prerelease: false
|
78
|
+
version_requirements: !ruby/object:Gem::Requirement
|
79
|
+
requirements:
|
80
|
+
- - "~>"
|
81
|
+
- !ruby/object:Gem::Version
|
82
|
+
version: '1.2'
|
41
83
|
- !ruby/object:Gem::Dependency
|
42
84
|
name: stringento
|
43
85
|
requirement: !ruby/object:Gem::Requirement
|
@@ -179,6 +221,15 @@ files:
|
|
179
221
|
- lib/burner/cli.rb
|
180
222
|
- lib/burner/job.rb
|
181
223
|
- lib/burner/jobs.rb
|
224
|
+
- lib/burner/jobs/collection/arrays_to_objects.rb
|
225
|
+
- lib/burner/jobs/collection/graph.rb
|
226
|
+
- lib/burner/jobs/collection/objects_to_arrays.rb
|
227
|
+
- lib/burner/jobs/collection/shift.rb
|
228
|
+
- lib/burner/jobs/collection/transform.rb
|
229
|
+
- lib/burner/jobs/collection/transform/attribute.rb
|
230
|
+
- lib/burner/jobs/collection/transform/attribute_renderer.rb
|
231
|
+
- lib/burner/jobs/collection/unpivot.rb
|
232
|
+
- lib/burner/jobs/deserialize/csv.rb
|
182
233
|
- lib/burner/jobs/deserialize/json.rb
|
183
234
|
- lib/burner/jobs/deserialize/yaml.rb
|
184
235
|
- lib/burner/jobs/dummy.rb
|
@@ -187,10 +238,13 @@ files:
|
|
187
238
|
- lib/burner/jobs/io/exist.rb
|
188
239
|
- lib/burner/jobs/io/read.rb
|
189
240
|
- lib/burner/jobs/io/write.rb
|
241
|
+
- lib/burner/jobs/serialize/csv.rb
|
190
242
|
- lib/burner/jobs/serialize/json.rb
|
191
243
|
- lib/burner/jobs/serialize/yaml.rb
|
192
244
|
- lib/burner/jobs/set.rb
|
193
245
|
- lib/burner/jobs/sleep.rb
|
246
|
+
- lib/burner/modeling.rb
|
247
|
+
- lib/burner/modeling/key_index_mapping.rb
|
194
248
|
- lib/burner/output.rb
|
195
249
|
- lib/burner/payload.rb
|
196
250
|
- lib/burner/pipeline.rb
|