db_fuel 1.1.0 → 1.2.1.pre.alpha.3
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/.rubocop.yml +2 -1
- data/.tool-versions +1 -0
- data/README.md +103 -7
- data/db_fuel.gemspec +4 -4
- data/lib/db_fuel/library.rb +4 -0
- data/lib/db_fuel/library/active_record/base.rb +8 -38
- data/lib/db_fuel/library/active_record/find_or_insert.rb +10 -33
- data/lib/db_fuel/library/active_record/insert.rb +10 -41
- data/lib/db_fuel/library/active_record/update.rb +25 -29
- data/lib/db_fuel/library/active_record/update_all.rb +96 -0
- data/lib/db_fuel/library/active_record/upsert.rb +210 -0
- data/lib/db_fuel/library/dbee/base.rb +12 -3
- data/lib/db_fuel/library/dbee/query.rb +35 -3
- data/lib/db_fuel/library/dbee/range.rb +18 -6
- data/lib/db_fuel/modeling.rb +1 -0
- data/lib/db_fuel/modeling/attribute_renderer_set.rb +83 -0
- data/lib/db_fuel/version.rb +1 -1
- metadata +17 -11
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 50352c41173cfcd379df4c836b7005750af724460f4533e112b7acc539f3adaf
|
4
|
+
data.tar.gz: 1536941988c987c6dd90d6e581b60ca9be1d3e25df9b927e3b2f393f54fa0478
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 4d3bdb76cf43715d1bef3f92b5a6eb63e497c1a69cc9bfa113d648ed72f09900d5cae82aac991e45aea20629a274cffc0c0e1c808a8258c0e55a71a4ff1b5151
|
7
|
+
data.tar.gz: 6cf1100fc98dd991a18bada43391b3aa9db4dea2f8dba66ab382b3ef58fc4db5fce30e60120f8e425328ecc29e5cdf28b11a8fc0e679ebc7d8e43a5358115b08
|
data/.rubocop.yml
CHANGED
@@ -1,6 +1,7 @@
|
|
1
1
|
AllCops:
|
2
2
|
TargetRubyVersion: 2.5
|
3
3
|
NewCops: enable
|
4
|
+
SuggestExtensions: false
|
4
5
|
|
5
6
|
Layout/LineLength:
|
6
7
|
Max: 100
|
@@ -8,7 +9,7 @@ Layout/LineLength:
|
|
8
9
|
- db_fuel.gemspec
|
9
10
|
|
10
11
|
Metrics/BlockLength:
|
11
|
-
|
12
|
+
IgnoredMethods:
|
12
13
|
- let
|
13
14
|
- it
|
14
15
|
- describe
|
data/.tool-versions
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
ruby 2.6.6
|
data/README.md
CHANGED
@@ -24,14 +24,16 @@ Refer to the [Burner](https://github.com/bluemarblepayroll/burner) library for m
|
|
24
24
|
|
25
25
|
### ActiveRecord Jobs
|
26
26
|
|
27
|
-
* **db_fuel/active_record/find_or_insert** [table_name, attributes, debug, primary_key, register, separator, timestamps, unique_attributes]: An extension of the `db_fuel/active_record/insert` job that adds an existence check before SQL insertion. The `unique_attributes` will be converted to WHERE clauses for performing the existence check.
|
28
|
-
* **db_fuel/active_record/insert** [table_name, attributes, debug, primary_key, register, separator, timestamps]: This job can take the objects in a register and insert them into a database table. If primary_key is specified then its key will be set to the primary key. Note that composite primary keys are not supported. Attributes defines which object properties to convert to SQL. Refer to the class and constructor specification for more detail.
|
29
|
-
* **db_fuel/active_record/
|
27
|
+
* **db_fuel/active_record/find_or_insert** [name, table_name, attributes, debug, primary_key, register, separator, timestamps, unique_attributes]: An extension of the `db_fuel/active_record/insert` job that adds an existence check before SQL insertion. The `unique_attributes` will be converted to WHERE clauses for performing the existence check.
|
28
|
+
* **db_fuel/active_record/insert** [name, table_name, attributes, debug, primary_key, register, separator, timestamps]: This job can take the objects in a register and insert them into a database table. If primary_key is specified then its key will be set to the primary key. Note that composite primary keys are not supported. Attributes defines which object properties to convert to SQL. Refer to the class and constructor specification for more detail.
|
29
|
+
* **db_fuel/active_record/update_all** [name, table_name, attributes, debug, register, separator, timestamps, unique_attributes]: This job can take the objects in a register and updates them within a database table. Attributes defines which object properties to convert to SQL SET clauses while unique_attributes translate to WHERE clauses. One or more records may be updated at a time. Refer to the class and constructor specification for more detail.
|
30
|
+
* **db_fuel/active_record/update** [name, table_name, attributes, debug, register, primary_key, separator, timestamps, unique_attributes]: This job can take the unique objects in a register and updates them within a database table. Attributes defines which object properties to convert to SQL SET clauses while unique_attributes translate to WHERE clauses to find the records to update. The primary_key is used to update the unique record. Only one record will be updated per statement. Refer to the class and constructor specification for more detail.
|
31
|
+
* **db_fuel/active_record/upsert** [name, table_name, attributes, debug, primary_key, register, separator, timestamps, unique_attributes]: This job can take the objects in a register and either inserts or updates them within a database table. Attributes defines which object properties to convert to SQL SET clauses while each key in unique_attributes become a WHERE clause in order to check for the existence of a specific record. The updated record will use the primary_key specified to perform the UPDATE operation. Note that composite primary keys are not supported. Refer to the class and constructor specification for more detail.
|
30
32
|
|
31
33
|
### Dbee Jobs
|
32
34
|
|
33
|
-
* **db_fuel/dbee/query** [model, query, register]: Pass in a [Dbee](https://github.com/bluemarblepayroll/dbee) model and query and store the results in the specified register. Refer to the [Dbee](https://github.com/bluemarblepayroll/dbee) library directly on how to craft a model or query.
|
34
|
-
* **db_fuel/dbee/range** [key, key_path, model, query, register, separator]: Similar to `db_fuel/dbee/query` with the addition of being able to grab a list of values from the register to use as a Dbee EQUALS/IN filter. This helps to dynamically limit the resulting record set. The key is used to specify where to grab the list of values, while the key_path will be used to craft the [Dbee equal's filter](https://github.com/bluemarblepayroll/dbee/blob/master/lib/dbee/query/filters/equals.rb). Separator is exposed in case nested object support is necessary.
|
35
|
+
* **db_fuel/dbee/query** [model, query, register, debug]: Pass in a [Dbee](https://github.com/bluemarblepayroll/dbee) model and query and store the results in the specified register. Refer to the [Dbee](https://github.com/bluemarblepayroll/dbee) library directly on how to craft a model or query.
|
36
|
+
* **db_fuel/dbee/range** [key, key_path, model, query, register, separator, debug]: Similar to `db_fuel/dbee/query` with the addition of being able to grab a list of values from the register to use as a Dbee EQUALS/IN filter. This helps to dynamically limit the resulting record set. The key is used to specify where to grab the list of values, while the key_path will be used to craft the [Dbee equal's filter](https://github.com/bluemarblepayroll/dbee/blob/master/lib/dbee/query/filters/equals.rb). Separator is exposed in case nested object support is necessary.
|
35
37
|
|
36
38
|
## Examples
|
37
39
|
|
@@ -96,6 +98,10 @@ If we were to inspect the contents of `payload` we should see the patient's resu
|
|
96
98
|
payload['patients'] # array in form of: [ { "id" => 1, "first_name" => "Something" }, ... ]
|
97
99
|
````
|
98
100
|
|
101
|
+
Notes
|
102
|
+
|
103
|
+
* Set `debug: true` to print out SQL statement in the output (not for production use.)
|
104
|
+
|
99
105
|
### Limiting Result Sets
|
100
106
|
|
101
107
|
The `db_fuel/dbee/query` does not provide a way to dynamically connect the query to existing data. You are free to put any Dbee query filters in the query declaration, but what if you would like to further limit this based on the knowledge of a range of values? The `db_fuel/dbee/range` job is meant to do exactly this. On the surface it is mainly an extension of the `db_fuel/dbee/query` job.
|
@@ -149,6 +155,10 @@ payload['patients'] # array in form of: [ { "id" => 1, "first_name" => "Somethin
|
|
149
155
|
|
150
156
|
The only difference between the query and range jobs should be the latter is limited based on the incoming first names.
|
151
157
|
|
158
|
+
Notes
|
159
|
+
|
160
|
+
* Set `debug: true` to print out SQL statement in the output (not for production use.)
|
161
|
+
|
152
162
|
### Updating the Database
|
153
163
|
|
154
164
|
#### Inserting Records
|
@@ -242,7 +252,7 @@ Now only records where the chart_number does not match an existing record will b
|
|
242
252
|
|
243
253
|
#### Updating Records
|
244
254
|
|
245
|
-
Let's say we now want to update
|
255
|
+
Let's say we now want to update these unique records' last names:
|
246
256
|
|
247
257
|
````ruby
|
248
258
|
pipeline = {
|
@@ -264,6 +274,91 @@ pipeline = {
|
|
264
274
|
{ key: :last_name }
|
265
275
|
],
|
266
276
|
table_name: 'patients',
|
277
|
+
primary_key: {
|
278
|
+
key: :id
|
279
|
+
},
|
280
|
+
unique_attributes: [
|
281
|
+
{ key: :chart_number }
|
282
|
+
]
|
283
|
+
}
|
284
|
+
]
|
285
|
+
}
|
286
|
+
|
287
|
+
payload = Burner::Payload.new
|
288
|
+
|
289
|
+
Burner::Pipeline.make(pipeline).execute(payload: payload)
|
290
|
+
````
|
291
|
+
|
292
|
+
Each database record should have been updated with their new respective last names based on the primary key specified.
|
293
|
+
|
294
|
+
#### Updating All Records
|
295
|
+
|
296
|
+
Let's say we want to update those records' midddle names:
|
297
|
+
|
298
|
+
````ruby
|
299
|
+
pipeline = {
|
300
|
+
jobs: [
|
301
|
+
{
|
302
|
+
name: :load_patients,
|
303
|
+
type: 'b/value/static',
|
304
|
+
register: :patients,
|
305
|
+
value: [
|
306
|
+
{ chart_number: 'B0001', middle_name: 'Rabbit' },
|
307
|
+
{ chart_number: 'C0001', middle_name: 'Elf' }
|
308
|
+
]
|
309
|
+
},
|
310
|
+
{
|
311
|
+
name: 'update_patients',
|
312
|
+
type: 'db_fuel/active_record/update_all',
|
313
|
+
register: :patients,
|
314
|
+
attributes: [
|
315
|
+
{ key: :last_name }
|
316
|
+
],
|
317
|
+
table_name: 'patients',
|
318
|
+
unique_attributes: [
|
319
|
+
{ key: :chart_number }
|
320
|
+
]
|
321
|
+
}
|
322
|
+
]
|
323
|
+
}
|
324
|
+
|
325
|
+
payload = Burner::Payload.new
|
326
|
+
|
327
|
+
Burner::Pipeline.make(pipeline).execute(payload: payload)
|
328
|
+
````
|
329
|
+
|
330
|
+
Each database record should have been updated with their new respective middle names based on chart_number.
|
331
|
+
|
332
|
+
#### Upserting Records
|
333
|
+
|
334
|
+
Let's say we don't know if these chart_number values already exist or not.
|
335
|
+
So we want db_fuel to either insert a record if the chart_number doesn't exist or update the record if the chart_number already exists.
|
336
|
+
|
337
|
+
````ruby
|
338
|
+
pipeline = {
|
339
|
+
jobs: [
|
340
|
+
{
|
341
|
+
name: :load_patients,
|
342
|
+
type: 'b/value/static',
|
343
|
+
register: :patients,
|
344
|
+
value: [
|
345
|
+
{ chart_number: 'B0002', first_name: 'Babs', last_name: 'Bunny' },
|
346
|
+
{ chart_number: 'B0003', first_name: 'Daffy', last_name: 'Duck' }
|
347
|
+
]
|
348
|
+
},
|
349
|
+
{
|
350
|
+
name: 'update_patients',
|
351
|
+
type: 'db_fuel/active_record/upsert',
|
352
|
+
register: :patients,
|
353
|
+
attributes: [
|
354
|
+
{ key: :chart_number },
|
355
|
+
{ key: :first_name },
|
356
|
+
{ key: :last_name }
|
357
|
+
],
|
358
|
+
table_name: 'patients',
|
359
|
+
primary_key: {
|
360
|
+
key: :id
|
361
|
+
},
|
267
362
|
unique_attributes: [
|
268
363
|
{ key: :chart_number }
|
269
364
|
]
|
@@ -276,7 +371,8 @@ payload = Burner::Payload.new
|
|
276
371
|
Burner::Pipeline.make(pipeline).execute(payload: payload)
|
277
372
|
````
|
278
373
|
|
279
|
-
Each database record should have been updated with their
|
374
|
+
Each database record should have been either inserted or updated with their corresponding values. In this case Babs' last name
|
375
|
+
was switched back to Bunny and a new record was created for Daffy Duck.
|
280
376
|
|
281
377
|
Notes:
|
282
378
|
|
data/db_fuel.gemspec
CHANGED
@@ -11,8 +11,8 @@ Gem::Specification.new do |s|
|
|
11
11
|
This library adds database-centric jobs to the Burner library. Burner does not ship with database jobs out of the box.
|
12
12
|
DESCRIPTION
|
13
13
|
|
14
|
-
s.authors = ['Matthew Ruggio']
|
15
|
-
s.email = ['mruggio@bluemarblepayroll.com']
|
14
|
+
s.authors = ['Matthew Ruggio', 'John Bosko']
|
15
|
+
s.email = ['mruggio@bluemarblepayroll.com', 'jbosko@bluemarblepayroll.com']
|
16
16
|
s.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
|
17
17
|
s.bindir = 'exe'
|
18
18
|
s.executables = %w[]
|
@@ -42,7 +42,7 @@ Gem::Specification.new do |s|
|
|
42
42
|
|
43
43
|
s.add_dependency('activerecord', activerecord_version)
|
44
44
|
s.add_dependency('acts_as_hashable', '~>1.2')
|
45
|
-
s.add_dependency('burner', '~>1.
|
45
|
+
s.add_dependency('burner', '~>1.7')
|
46
46
|
s.add_dependency('dbee', '~>2.1')
|
47
47
|
s.add_dependency('dbee-active_record', '~>2.1')
|
48
48
|
s.add_dependency('objectable', '~>1.0')
|
@@ -51,7 +51,7 @@ Gem::Specification.new do |s|
|
|
51
51
|
s.add_development_dependency('pry', '~>0')
|
52
52
|
s.add_development_dependency('rake', '~> 13')
|
53
53
|
s.add_development_dependency('rspec', '~> 3.8')
|
54
|
-
s.add_development_dependency('rubocop', '~>
|
54
|
+
s.add_development_dependency('rubocop', '~>1.7.0')
|
55
55
|
s.add_development_dependency('simplecov', '~>0.18.5')
|
56
56
|
s.add_development_dependency('simplecov-console', '~>0.7.0')
|
57
57
|
s.add_development_dependency('sqlite3', '~>1')
|
data/lib/db_fuel/library.rb
CHANGED
@@ -10,6 +10,8 @@
|
|
10
10
|
require_relative 'library/active_record/find_or_insert'
|
11
11
|
require_relative 'library/active_record/insert'
|
12
12
|
require_relative 'library/active_record/update'
|
13
|
+
require_relative 'library/active_record/update_all'
|
14
|
+
require_relative 'library/active_record/upsert'
|
13
15
|
|
14
16
|
require_relative 'library/dbee/query'
|
15
17
|
require_relative 'library/dbee/range'
|
@@ -20,6 +22,8 @@ module Burner
|
|
20
22
|
register 'db_fuel/active_record/find_or_insert', DbFuel::Library::ActiveRecord::FindOrInsert
|
21
23
|
register 'db_fuel/active_record/insert', DbFuel::Library::ActiveRecord::Insert
|
22
24
|
register 'db_fuel/active_record/update', DbFuel::Library::ActiveRecord::Update
|
25
|
+
register 'db_fuel/active_record/update_all', DbFuel::Library::ActiveRecord::UpdateAll
|
26
|
+
register 'db_fuel/active_record/upsert', DbFuel::Library::ActiveRecord::Upsert
|
23
27
|
|
24
28
|
register 'db_fuel/dbee/query', DbFuel::Library::Dbee::Query
|
25
29
|
register 'db_fuel/dbee/range', DbFuel::Library::Dbee::Range
|
@@ -22,11 +22,12 @@ module DbFuel
|
|
22
22
|
attr_reader :attribute_renderers,
|
23
23
|
:db_provider,
|
24
24
|
:debug,
|
25
|
-
:resolver
|
25
|
+
:resolver,
|
26
|
+
:attribute_renderers_set
|
26
27
|
|
27
28
|
def initialize(
|
28
|
-
name:,
|
29
29
|
table_name:,
|
30
|
+
name: '',
|
30
31
|
attributes: [],
|
31
32
|
debug: false,
|
32
33
|
register: Burner::DEFAULT_REGISTER,
|
@@ -34,45 +35,14 @@ module DbFuel
|
|
34
35
|
)
|
35
36
|
super(name: name, register: register)
|
36
37
|
|
37
|
-
|
38
|
-
@
|
39
|
-
|
40
|
-
@db_provider
|
38
|
+
@resolver = Objectable.resolver(separator: separator)
|
39
|
+
@attribute_renderers_set = Modeling::AttributeRendererSet.new(attributes: attributes,
|
40
|
+
resolver: resolver)
|
41
|
+
@db_provider = DbProvider.new(table_name)
|
41
42
|
@debug = debug || false
|
42
43
|
end
|
43
44
|
|
44
|
-
|
45
|
-
|
46
|
-
def make_attribute_renderers(attributes)
|
47
|
-
Burner::Modeling::Attribute
|
48
|
-
.array(attributes)
|
49
|
-
.map { |a| Burner::Modeling::AttributeRenderer.new(a, resolver) }
|
50
|
-
end
|
51
|
-
|
52
|
-
def transform(attribute_renderers, row, time)
|
53
|
-
attribute_renderers.each_with_object({}) do |attribute_renderer, memo|
|
54
|
-
value = attribute_renderer.transform(row, time)
|
55
|
-
|
56
|
-
resolver.set(memo, attribute_renderer.key, value)
|
57
|
-
end
|
58
|
-
end
|
59
|
-
|
60
|
-
def created_at_timestamp_attribute
|
61
|
-
timestamp_attribute(CREATED_AT)
|
62
|
-
end
|
63
|
-
|
64
|
-
def updated_at_timestamp_attribute
|
65
|
-
timestamp_attribute(UPDATED_AT)
|
66
|
-
end
|
67
|
-
|
68
|
-
def timestamp_attribute(key)
|
69
|
-
Burner::Modeling::Attribute.make(
|
70
|
-
key: key,
|
71
|
-
transformers: [
|
72
|
-
{ type: NOW_TYPE }
|
73
|
-
]
|
74
|
-
)
|
75
|
-
end
|
45
|
+
protected
|
76
46
|
|
77
47
|
def debug_detail(output, message)
|
78
48
|
return unless debug
|
@@ -7,7 +7,7 @@
|
|
7
7
|
# LICENSE file in the root directory of this source tree.
|
8
8
|
#
|
9
9
|
|
10
|
-
require_relative '
|
10
|
+
require_relative 'upsert'
|
11
11
|
|
12
12
|
module DbFuel
|
13
13
|
module Library
|
@@ -19,11 +19,11 @@ module DbFuel
|
|
19
19
|
#
|
20
20
|
# Expected Payload[register] input: array of objects
|
21
21
|
# Payload[register] output: array of objects.
|
22
|
-
class FindOrInsert <
|
23
|
-
attr_reader :unique_attribute_renderers
|
22
|
+
class FindOrInsert < Upsert
|
23
|
+
# attr_reader :unique_attribute_renderers
|
24
24
|
|
25
25
|
# Arguments:
|
26
|
-
# name
|
26
|
+
# name: name of the job within the Burner::Pipeline.
|
27
27
|
#
|
28
28
|
# table_name [required]: name of the table to use for the INSERT statements.
|
29
29
|
#
|
@@ -51,8 +51,8 @@ module DbFuel
|
|
51
51
|
# unique_attributes: Each key will become a WHERE clause in order check for record
|
52
52
|
# existence before insertion attempt.
|
53
53
|
def initialize(
|
54
|
-
name:,
|
55
54
|
table_name:,
|
55
|
+
name: '',
|
56
56
|
attributes: [],
|
57
57
|
debug: false,
|
58
58
|
primary_key: nil,
|
@@ -61,6 +61,7 @@ module DbFuel
|
|
61
61
|
timestamps: true,
|
62
62
|
unique_attributes: []
|
63
63
|
)
|
64
|
+
|
64
65
|
super(
|
65
66
|
name: name,
|
66
67
|
table_name: table_name,
|
@@ -69,10 +70,9 @@ module DbFuel
|
|
69
70
|
primary_key: primary_key,
|
70
71
|
register: register,
|
71
72
|
separator: separator,
|
72
|
-
timestamps: timestamps
|
73
|
+
timestamps: timestamps,
|
74
|
+
unique_attributes: unique_attributes
|
73
75
|
)
|
74
|
-
|
75
|
-
@unique_attribute_renderers = make_attribute_renderers(unique_attributes)
|
76
76
|
end
|
77
77
|
|
78
78
|
def perform(output, payload)
|
@@ -82,14 +82,14 @@ module DbFuel
|
|
82
82
|
payload[register] = array(payload[register])
|
83
83
|
|
84
84
|
payload[register].each do |row|
|
85
|
-
exists =
|
85
|
+
exists = find_record(output, row, payload.time)
|
86
86
|
|
87
87
|
if exists
|
88
88
|
total_existed += 1
|
89
89
|
next
|
90
90
|
end
|
91
91
|
|
92
|
-
|
92
|
+
insert_record(output, row, payload.time)
|
93
93
|
|
94
94
|
total_inserted += 1
|
95
95
|
end
|
@@ -97,29 +97,6 @@ module DbFuel
|
|
97
97
|
output.detail("Total Existed: #{total_existed}")
|
98
98
|
output.detail("Total Inserted: #{total_inserted}")
|
99
99
|
end
|
100
|
-
|
101
|
-
private
|
102
|
-
|
103
|
-
def existence_check_and_mutate(output, row, time)
|
104
|
-
unique_row = transform(unique_attribute_renderers, row, time)
|
105
|
-
|
106
|
-
first_sql = db_provider.first_sql(unique_row)
|
107
|
-
debug_detail(output, "Find Statement: #{first_sql}")
|
108
|
-
|
109
|
-
first_record = db_provider.first(unique_row)
|
110
|
-
|
111
|
-
return false unless first_record
|
112
|
-
|
113
|
-
if primary_key
|
114
|
-
id = resolver.get(first_record, primary_key.column)
|
115
|
-
|
116
|
-
resolver.set(row, primary_key.key, id)
|
117
|
-
end
|
118
|
-
|
119
|
-
debug_detail(output, "Record Exists: #{first_record}")
|
120
|
-
|
121
|
-
true
|
122
|
-
end
|
123
100
|
end
|
124
101
|
end
|
125
102
|
end
|
@@ -7,7 +7,7 @@
|
|
7
7
|
# LICENSE file in the root directory of this source tree.
|
8
8
|
#
|
9
9
|
|
10
|
-
require_relative '
|
10
|
+
require_relative 'upsert'
|
11
11
|
|
12
12
|
module DbFuel
|
13
13
|
module Library
|
@@ -16,11 +16,11 @@ module DbFuel
|
|
16
16
|
#
|
17
17
|
# Expected Payload[register] input: array of objects
|
18
18
|
# Payload[register] output: array of objects.
|
19
|
-
class Insert <
|
20
|
-
attr_reader :primary_key
|
19
|
+
class Insert < Upsert
|
20
|
+
# attr_reader :primary_key
|
21
21
|
|
22
22
|
# Arguments:
|
23
|
-
# name
|
23
|
+
# name: name of the job within the Burner::Pipeline.
|
24
24
|
#
|
25
25
|
# table_name [required]: name of the table to use for the INSERT statements.
|
26
26
|
#
|
@@ -45,8 +45,8 @@ module DbFuel
|
|
45
45
|
# and updated_at columns will automatically have their values set
|
46
46
|
# to the current UTC timestamp.
|
47
47
|
def initialize(
|
48
|
-
name:,
|
49
48
|
table_name:,
|
49
|
+
name: '',
|
50
50
|
attributes: [],
|
51
51
|
debug: false,
|
52
52
|
primary_key: nil,
|
@@ -54,56 +54,25 @@ module DbFuel
|
|
54
54
|
separator: '',
|
55
55
|
timestamps: true
|
56
56
|
)
|
57
|
-
explicit_attributes = Burner::Modeling::Attribute.array(attributes)
|
58
57
|
|
59
|
-
attributes =
|
58
|
+
attributes = Burner::Modeling::Attribute.array(attributes)
|
60
59
|
|
61
60
|
super(
|
62
61
|
name: name,
|
63
62
|
table_name: table_name,
|
64
63
|
attributes: attributes,
|
65
64
|
debug: debug,
|
65
|
+
primary_key: primary_key,
|
66
66
|
register: register,
|
67
|
-
separator: separator
|
67
|
+
separator: separator,
|
68
|
+
timestamps: timestamps
|
68
69
|
)
|
69
|
-
|
70
|
-
@primary_key = Modeling::KeyedColumn.make(primary_key, nullable: true)
|
71
70
|
end
|
72
71
|
|
73
72
|
def perform(output, payload)
|
74
73
|
payload[register] = array(payload[register])
|
75
74
|
|
76
|
-
payload[register].each { |row|
|
77
|
-
end
|
78
|
-
|
79
|
-
private
|
80
|
-
|
81
|
-
def insert(output, row, time)
|
82
|
-
transformed_row = transform(attribute_renderers, row, time)
|
83
|
-
|
84
|
-
output_sql(output, transformed_row)
|
85
|
-
insert_and_mutate(output, transformed_row, row)
|
86
|
-
end
|
87
|
-
|
88
|
-
def output_sql(output, row)
|
89
|
-
sql = db_provider.insert_sql(row)
|
90
|
-
|
91
|
-
debug_detail(output, "Insert Statement: #{sql}")
|
92
|
-
end
|
93
|
-
|
94
|
-
def insert_and_mutate(output, row_to_insert, row_to_return)
|
95
|
-
id = db_provider.insert(row_to_insert)
|
96
|
-
|
97
|
-
resolver.set(row_to_return, primary_key.key, id) if primary_key
|
98
|
-
|
99
|
-
debug_detail(output, "Insert Return: #{row_to_return}")
|
100
|
-
end
|
101
|
-
|
102
|
-
def timestamp_attributes
|
103
|
-
[
|
104
|
-
created_at_timestamp_attribute,
|
105
|
-
updated_at_timestamp_attribute
|
106
|
-
]
|
75
|
+
payload[register].each { |row| insert_record(output, row, payload.time) }
|
107
76
|
end
|
108
77
|
end
|
109
78
|
end
|
@@ -7,22 +7,22 @@
|
|
7
7
|
# LICENSE file in the root directory of this source tree.
|
8
8
|
#
|
9
9
|
|
10
|
-
require_relative '
|
10
|
+
require_relative 'upsert'
|
11
11
|
|
12
12
|
module DbFuel
|
13
13
|
module Library
|
14
14
|
module ActiveRecord
|
15
|
-
# This job can take the objects in a register and updates them within database table.
|
15
|
+
# This job can take the unique objects in a register and updates them within database table.
|
16
16
|
# The attributes translate to SQL SET clauses and the unique_keys translate to
|
17
|
-
# WHERE clauses.
|
17
|
+
# WHERE clauses to find the records to update.
|
18
|
+
# The primary_key is used to update the unique record.
|
19
|
+
# Only one record will be updated per statement.
|
18
20
|
#
|
19
21
|
# Expected Payload[register] input: array of objects
|
20
22
|
# Payload[register] output: array of objects.
|
21
|
-
class Update <
|
22
|
-
attr_reader :unique_attribute_renderers
|
23
|
-
|
23
|
+
class Update < Upsert
|
24
24
|
# Arguments:
|
25
|
-
# name
|
25
|
+
# name: name of the job within the Burner::Pipeline.
|
26
26
|
#
|
27
27
|
# table_name [required]: name of the table to use for the INSERT statements.
|
28
28
|
#
|
@@ -35,6 +35,11 @@ module DbFuel
|
|
35
35
|
# returned objects will be printed in the output. Only use this option while
|
36
36
|
# debugging issues as it will fill up the output with (potentially too much) data.
|
37
37
|
#
|
38
|
+
# primary_key [required]: Primary key column for the corresponding table.
|
39
|
+
# Used as the WHERE clause for the UPDATE statement.
|
40
|
+
# Only one record will be updated at a time
|
41
|
+
# using the primary key specified.
|
42
|
+
#
|
38
43
|
# separator: Just like other jobs with a 'separator' option, if the objects require
|
39
44
|
# key-path notation or nested object support, you can set the separator
|
40
45
|
# to something non-blank (like a period for notation in the
|
@@ -43,33 +48,35 @@ module DbFuel
|
|
43
48
|
# timestamps: If timestamps is true (default behavior) then the updated_at column will
|
44
49
|
# automatically have its value set to the current UTC timestamp.
|
45
50
|
#
|
46
|
-
# unique_attributes: Each key will become a WHERE clause in order to only
|
47
|
-
# records.
|
51
|
+
# unique_attributes: Each key will become a WHERE clause in order to only find specific
|
52
|
+
# records. The UPDATE statement's WHERE
|
53
|
+
# clause will use the primary key specified.
|
48
54
|
def initialize(
|
49
|
-
name:,
|
50
55
|
table_name:,
|
56
|
+
name: '',
|
51
57
|
attributes: [],
|
52
58
|
debug: false,
|
59
|
+
primary_key: nil,
|
53
60
|
register: Burner::DEFAULT_REGISTER,
|
54
61
|
separator: '',
|
55
62
|
timestamps: true,
|
56
63
|
unique_attributes: []
|
57
64
|
)
|
58
|
-
explicit_attributes = Burner::Modeling::Attribute.array(attributes)
|
59
65
|
|
60
|
-
attributes =
|
66
|
+
attributes = Burner::Modeling::Attribute.array(attributes)
|
61
67
|
|
62
68
|
super(
|
63
69
|
name: name,
|
64
70
|
table_name: table_name,
|
65
71
|
attributes: attributes,
|
66
72
|
debug: debug,
|
73
|
+
primary_key: primary_key,
|
67
74
|
register: register,
|
68
|
-
separator: separator
|
75
|
+
separator: separator,
|
76
|
+
timestamps: timestamps,
|
77
|
+
unique_attributes: unique_attributes
|
69
78
|
)
|
70
79
|
|
71
|
-
@unique_attribute_renderers = make_attribute_renderers(unique_attributes)
|
72
|
-
|
73
80
|
freeze
|
74
81
|
end
|
75
82
|
|
@@ -79,14 +86,11 @@ module DbFuel
|
|
79
86
|
payload[register] = array(payload[register])
|
80
87
|
|
81
88
|
payload[register].each do |row|
|
82
|
-
|
83
|
-
where_object = transform(unique_attribute_renderers, row, payload.time)
|
84
|
-
|
85
|
-
sql = db_provider.update_sql(set_object, where_object)
|
89
|
+
rows_affected = 0
|
86
90
|
|
87
|
-
|
91
|
+
first_record = update_record(output, row, payload.time)
|
88
92
|
|
89
|
-
rows_affected =
|
93
|
+
rows_affected = 1 if first_record
|
90
94
|
|
91
95
|
debug_detail(output, "Individual Rows Affected: #{rows_affected}")
|
92
96
|
|
@@ -95,14 +99,6 @@ module DbFuel
|
|
95
99
|
|
96
100
|
output.detail("Total Rows Affected: #{total_rows_affected}")
|
97
101
|
end
|
98
|
-
|
99
|
-
private
|
100
|
-
|
101
|
-
def timestamp_attributes
|
102
|
-
[
|
103
|
-
updated_at_timestamp_attribute
|
104
|
-
]
|
105
|
-
end
|
106
102
|
end
|
107
103
|
end
|
108
104
|
end
|
@@ -0,0 +1,96 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
#
|
4
|
+
# Copyright (c) 2020-present, Blue Marble Payroll, LLC
|
5
|
+
#
|
6
|
+
# This source code is licensed under the MIT license found in the
|
7
|
+
# LICENSE file in the root directory of this source tree.
|
8
|
+
#
|
9
|
+
|
10
|
+
require_relative 'upsert'
|
11
|
+
|
12
|
+
module DbFuel
|
13
|
+
module Library
|
14
|
+
module ActiveRecord
|
15
|
+
# This job can take the objects in a register and updates them within database table.
|
16
|
+
# The attributes translate to SQL SET clauses
|
17
|
+
# and the unique_keys translate to WHERE clauses.
|
18
|
+
# One or more records may be updated at a time.
|
19
|
+
#
|
20
|
+
# Expected Payload[register] input: array of objects
|
21
|
+
# Payload[register] output: array of objects.
|
22
|
+
class UpdateAll < Upsert
|
23
|
+
# Arguments:
|
24
|
+
# name: name of the job within the Burner::Pipeline.
|
25
|
+
#
|
26
|
+
# table_name [required]: name of the table to use for the INSERT statements.
|
27
|
+
#
|
28
|
+
# attributes: Used to specify which object properties to put into the
|
29
|
+
# SQL statement and also allows for one last custom transformation
|
30
|
+
# pipeline, in case the data calls for SQL-specific transformers
|
31
|
+
# before mutation.
|
32
|
+
#
|
33
|
+
# debug: If debug is set to true (defaults to false) then the SQL statements and
|
34
|
+
# returned objects will be printed in the output. Only use this option while
|
35
|
+
# debugging issues as it will fill up the output with (potentially too much) data.
|
36
|
+
#
|
37
|
+
# separator: Just like other jobs with a 'separator' option, if the objects require
|
38
|
+
# key-path notation or nested object support, you can set the separator
|
39
|
+
# to something non-blank (like a period for notation in the
|
40
|
+
# form of: name.first).
|
41
|
+
#
|
42
|
+
# timestamps: If timestamps is true (default behavior) then the updated_at column will
|
43
|
+
# automatically have its value set to the current UTC timestamp.
|
44
|
+
#
|
45
|
+
# unique_attributes: Each key will become a WHERE clause in order to only update specific
|
46
|
+
# records.
|
47
|
+
def initialize(
|
48
|
+
table_name:,
|
49
|
+
name: '',
|
50
|
+
attributes: [],
|
51
|
+
debug: false,
|
52
|
+
register: Burner::DEFAULT_REGISTER,
|
53
|
+
separator: '',
|
54
|
+
timestamps: true,
|
55
|
+
unique_attributes: []
|
56
|
+
)
|
57
|
+
|
58
|
+
attributes = Burner::Modeling::Attribute.array(attributes)
|
59
|
+
|
60
|
+
super(
|
61
|
+
name: name,
|
62
|
+
table_name: table_name,
|
63
|
+
attributes: attributes,
|
64
|
+
debug: debug,
|
65
|
+
primary_key: nil,
|
66
|
+
register: register,
|
67
|
+
separator: separator,
|
68
|
+
timestamps: timestamps,
|
69
|
+
unique_attributes: unique_attributes
|
70
|
+
)
|
71
|
+
|
72
|
+
freeze
|
73
|
+
end
|
74
|
+
|
75
|
+
def perform(output, payload)
|
76
|
+
total_rows_affected = 0
|
77
|
+
|
78
|
+
payload[register] = array(payload[register])
|
79
|
+
|
80
|
+
payload[register].each do |row|
|
81
|
+
where_object = attribute_renderers_set
|
82
|
+
.transform(unique_attribute_renderers, row, payload.time)
|
83
|
+
|
84
|
+
rows_affected = update(output, row, payload.time, where_object)
|
85
|
+
|
86
|
+
debug_detail(output, "Individual Rows Affected: #{rows_affected}")
|
87
|
+
|
88
|
+
total_rows_affected += rows_affected
|
89
|
+
end
|
90
|
+
|
91
|
+
output.detail("Total Rows Affected: #{total_rows_affected}")
|
92
|
+
end
|
93
|
+
end
|
94
|
+
end
|
95
|
+
end
|
96
|
+
end
|
@@ -0,0 +1,210 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
#
|
4
|
+
# Copyright (c) 2020-present, Blue Marble Payroll, LLC
|
5
|
+
#
|
6
|
+
# This source code is licensed under the MIT license found in the
|
7
|
+
# LICENSE file in the root directory of this source tree.
|
8
|
+
#
|
9
|
+
|
10
|
+
require_relative 'base'
|
11
|
+
|
12
|
+
module DbFuel
|
13
|
+
module Library
|
14
|
+
module ActiveRecord
|
15
|
+
# This job will insert or update records.
|
16
|
+
# It will use the unique_keys to first run a query to see if it exists.
|
17
|
+
# Each unique_key becomes a WHERE clause. If a record is found it will then
|
18
|
+
# update the found record using the primary key specified.
|
19
|
+
# If a record is updated or created the record's id will be set to the primary_key.
|
20
|
+
#
|
21
|
+
# Expected Payload[register] input: array of objects
|
22
|
+
# Payload[register] output: array of objects.
|
23
|
+
class Upsert < Base
|
24
|
+
attr_reader :primary_key, :timestamps, :unique_attribute_renderers
|
25
|
+
|
26
|
+
# Arguments:
|
27
|
+
# name: name of the job within the Burner::Pipeline.
|
28
|
+
#
|
29
|
+
# table_name [required]: name of the table to use for the INSERT OR UPDATE statements.
|
30
|
+
#
|
31
|
+
# attributes: Used to specify which object properties to put into the
|
32
|
+
# SQL statement and also allows for one last custom transformation
|
33
|
+
# pipeline, in case the data calls for SQL-specific transformers
|
34
|
+
# before mutation.
|
35
|
+
#
|
36
|
+
# debug: If debug is set to true (defaults to false) then the SQL statements and
|
37
|
+
# returned objects will be printed in the output. Only use this option while
|
38
|
+
# debugging issues as it will fill
|
39
|
+
# up the output with (potentially too much) data.
|
40
|
+
#
|
41
|
+
# primary_key [required]: Used to set the object's property to the returned primary key
|
42
|
+
# from the INSERT statement or used as the
|
43
|
+
# WHERE clause for the UPDATE statement.
|
44
|
+
#
|
45
|
+
# separator: Just like other jobs with a 'separator' option, if the objects require
|
46
|
+
# key-path notation or nested object support, you can set the separator
|
47
|
+
# to something non-blank (like a period for notation in the
|
48
|
+
# form of: name.first).
|
49
|
+
#
|
50
|
+
# timestamps: If timestamps is true (default behavior) then the updated_at column will
|
51
|
+
# automatically have its value set
|
52
|
+
# to the current UTC timestamp if a record was updated.
|
53
|
+
# If a record was created the
|
54
|
+
# created_at and updated_at columns will be set.
|
55
|
+
#
|
56
|
+
# unique_attributes: Each key will become a WHERE clause in
|
57
|
+
# order to check for the existence of a specific record.
|
58
|
+
def initialize(
|
59
|
+
table_name:,
|
60
|
+
primary_key:,
|
61
|
+
name: '',
|
62
|
+
attributes: [],
|
63
|
+
debug: false,
|
64
|
+
register: Burner::DEFAULT_REGISTER,
|
65
|
+
separator: '',
|
66
|
+
timestamps: true,
|
67
|
+
unique_attributes: []
|
68
|
+
)
|
69
|
+
super(
|
70
|
+
name: name,
|
71
|
+
table_name: table_name,
|
72
|
+
attributes: attributes,
|
73
|
+
debug: debug,
|
74
|
+
register: register,
|
75
|
+
separator: separator
|
76
|
+
)
|
77
|
+
|
78
|
+
@primary_key = Modeling::KeyedColumn.make(primary_key, nullable: true)
|
79
|
+
|
80
|
+
@unique_attribute_renderers = attribute_renderers_set
|
81
|
+
.make_attribute_renderers(unique_attributes)
|
82
|
+
|
83
|
+
@timestamps = timestamps
|
84
|
+
|
85
|
+
freeze
|
86
|
+
end
|
87
|
+
|
88
|
+
def perform(output, payload)
|
89
|
+
raise ArgumentError, 'primary_key is required' unless primary_key
|
90
|
+
|
91
|
+
total_inserted = 0
|
92
|
+
total_updated = 0
|
93
|
+
|
94
|
+
payload[register] = array(payload[register])
|
95
|
+
|
96
|
+
payload[register].each do |row|
|
97
|
+
record_updated = insert_or_update(output, row, payload.time)
|
98
|
+
|
99
|
+
if record_updated
|
100
|
+
total_updated += 1
|
101
|
+
else
|
102
|
+
total_inserted += 1
|
103
|
+
end
|
104
|
+
end
|
105
|
+
|
106
|
+
output.detail("Total Updated: #{total_updated}")
|
107
|
+
output.detail("Total Inserted: #{total_inserted}")
|
108
|
+
end
|
109
|
+
|
110
|
+
protected
|
111
|
+
|
112
|
+
def find_record(output, row, time)
|
113
|
+
unique_row = attribute_renderers_set.transform(unique_attribute_renderers, row, time)
|
114
|
+
|
115
|
+
first_sql = db_provider.first_sql(unique_row)
|
116
|
+
|
117
|
+
debug_detail(output, "Find Statement: #{first_sql}")
|
118
|
+
|
119
|
+
first_record = db_provider.first(unique_row)
|
120
|
+
|
121
|
+
id = resolver.get(first_record, primary_key.column)
|
122
|
+
|
123
|
+
resolver.set(row, primary_key.key, id)
|
124
|
+
|
125
|
+
debug_detail(output, "Record Exists: #{first_record}") if first_record
|
126
|
+
|
127
|
+
first_record
|
128
|
+
end
|
129
|
+
|
130
|
+
def insert_record(output, row, time)
|
131
|
+
dynamic_attrs = if timestamps
|
132
|
+
# doing an INSERT and timestamps should be set
|
133
|
+
# set the created_at and updated_at fields
|
134
|
+
attribute_renderers_set.timestamp_created_attribute_renderers
|
135
|
+
else
|
136
|
+
attribute_renderers_set.attribute_renderers
|
137
|
+
end
|
138
|
+
|
139
|
+
set_object = attribute_renderers_set.transform(dynamic_attrs, row, time)
|
140
|
+
|
141
|
+
insert_sql = db_provider.insert_sql(set_object)
|
142
|
+
|
143
|
+
debug_detail(output, "Insert Statement: #{insert_sql}")
|
144
|
+
|
145
|
+
id = db_provider.insert(set_object)
|
146
|
+
|
147
|
+
# add the primary key name and value to row if primary_key was specified
|
148
|
+
resolver.set(row, primary_key.key, id) if primary_key
|
149
|
+
|
150
|
+
debug_detail(output, "Insert Return: #{row}")
|
151
|
+
end
|
152
|
+
|
153
|
+
# Updates only a single record. Lookups primary key to update the record.
|
154
|
+
def update_record(output, row, time)
|
155
|
+
raise ArgumentError, 'primary_key is required' unless primary_key
|
156
|
+
|
157
|
+
first_record = find_record(output, row, time)
|
158
|
+
|
159
|
+
if first_record
|
160
|
+
debug_detail(output, "Record Exists: #{first_record}")
|
161
|
+
|
162
|
+
id = resolver.get(first_record, primary_key.column)
|
163
|
+
|
164
|
+
where_object = { primary_key.column => id }
|
165
|
+
|
166
|
+
# update record using the primary key as the WHERE clause
|
167
|
+
update(output, row, time, where_object)
|
168
|
+
end
|
169
|
+
|
170
|
+
first_record
|
171
|
+
end
|
172
|
+
|
173
|
+
# Updates one or many records depending on where_object passed
|
174
|
+
def update(output, row, time, where_object)
|
175
|
+
dynamic_attrs = if timestamps
|
176
|
+
# doing an UPDATE and timestamps should be set,
|
177
|
+
# modify the updated_at field, don't modify the created_at field
|
178
|
+
attribute_renderers_set.timestamp_updated_attribute_renderers
|
179
|
+
else
|
180
|
+
attribute_renderers_set.attribute_renderers
|
181
|
+
end
|
182
|
+
|
183
|
+
set_object = attribute_renderers_set.transform(dynamic_attrs, row, time)
|
184
|
+
|
185
|
+
update_sql = db_provider.update_sql(set_object, where_object)
|
186
|
+
|
187
|
+
debug_detail(output, "Update Statement: #{update_sql}")
|
188
|
+
|
189
|
+
debug_detail(output, "Update Return: #{row}")
|
190
|
+
|
191
|
+
db_provider.update(set_object, where_object)
|
192
|
+
end
|
193
|
+
|
194
|
+
private
|
195
|
+
|
196
|
+
def insert_or_update(output, row, time)
|
197
|
+
first_record = update_record(output, row, time)
|
198
|
+
|
199
|
+
if first_record
|
200
|
+
first_record
|
201
|
+
else
|
202
|
+
# create the record
|
203
|
+
insert_record(output, row, time)
|
204
|
+
nil
|
205
|
+
end
|
206
|
+
end
|
207
|
+
end
|
208
|
+
end
|
209
|
+
end
|
210
|
+
end
|
@@ -14,7 +14,8 @@ module DbFuel
|
|
14
14
|
class Base < Burner::JobWithRegister
|
15
15
|
attr_reader :model,
|
16
16
|
:provider,
|
17
|
-
:query
|
17
|
+
:query,
|
18
|
+
:debug
|
18
19
|
|
19
20
|
# Arguments:
|
20
21
|
# - model: Dbee Model configuration
|
@@ -22,16 +23,18 @@ module DbFuel
|
|
22
23
|
# - register: Name of the register to use for gathering the IN clause values and where
|
23
24
|
# to store the resulting recordset.
|
24
25
|
def initialize(
|
25
|
-
name
|
26
|
+
name: '',
|
26
27
|
model: {},
|
27
28
|
query: {},
|
28
|
-
register: Burner::DEFAULT_REGISTER
|
29
|
+
register: Burner::DEFAULT_REGISTER,
|
30
|
+
debug: false
|
29
31
|
)
|
30
32
|
super(name: name, register: register)
|
31
33
|
|
32
34
|
@model = ::Dbee::Model.make(model)
|
33
35
|
@provider = ::Dbee::Providers::ActiveRecordProvider.new
|
34
36
|
@query = ::Dbee::Query.make(query)
|
37
|
+
@debug = debug || false
|
35
38
|
|
36
39
|
freeze
|
37
40
|
end
|
@@ -47,6 +50,12 @@ module DbFuel
|
|
47
50
|
|
48
51
|
payload[register] = records
|
49
52
|
end
|
53
|
+
|
54
|
+
def debug_detail(output, message)
|
55
|
+
return unless debug
|
56
|
+
|
57
|
+
output.detail(message)
|
58
|
+
end
|
50
59
|
end
|
51
60
|
end
|
52
61
|
end
|
@@ -18,16 +18,48 @@ module DbFuel
|
|
18
18
|
# Expected Payload[register] input: nothing
|
19
19
|
# Payload[register] output: array of objects.
|
20
20
|
class Query < Base
|
21
|
+
# Arguments:
|
22
|
+
# - name: Name of job.
|
23
|
+
# - model: Dbee Model configuration
|
24
|
+
# - query: Dbee Query configuration
|
25
|
+
#
|
26
|
+
# - register: Name of the register to use for gathering the IN clause values and where
|
27
|
+
# to store the resulting recordset.
|
28
|
+
#
|
29
|
+
# - debug: If debug is set to true (defaults to false) then the SQL statements
|
30
|
+
# will be printed in the output. Only use this option while
|
31
|
+
# debugging issues as it will fill
|
32
|
+
# up the output with (potentially too much) data.
|
33
|
+
def initialize(
|
34
|
+
name: '',
|
35
|
+
model: {},
|
36
|
+
query: {},
|
37
|
+
register: Burner::DEFAULT_REGISTER,
|
38
|
+
debug: false
|
39
|
+
)
|
40
|
+
super(
|
41
|
+
model: model,
|
42
|
+
name: name,
|
43
|
+
query: query,
|
44
|
+
register: register,
|
45
|
+
debug: debug
|
46
|
+
)
|
47
|
+
end
|
48
|
+
|
21
49
|
def perform(output, payload)
|
22
|
-
records = execute(sql)
|
50
|
+
records = execute(sql(output))
|
23
51
|
|
24
52
|
load_register(records, output, payload)
|
25
53
|
end
|
26
54
|
|
27
55
|
private
|
28
56
|
|
29
|
-
def sql
|
30
|
-
::Dbee.sql(model, query, provider)
|
57
|
+
def sql(output)
|
58
|
+
sql_statement = ::Dbee.sql(model, query, provider)
|
59
|
+
|
60
|
+
debug_detail(output, "Query SQL: #{sql_statement}")
|
61
|
+
|
62
|
+
sql_statement
|
31
63
|
end
|
32
64
|
end
|
33
65
|
end
|
@@ -32,14 +32,20 @@ module DbFuel
|
|
32
32
|
# - register: Name of the register to use for gathering the IN clause values and where
|
33
33
|
# to store the resulting recordset.
|
34
34
|
# - separator: Character to use to split the key-path for nested object support.
|
35
|
+
#
|
36
|
+
# - debug: If debug is set to true (defaults to false) then the SQL statements
|
37
|
+
# will be printed in the output. Only use this option while
|
38
|
+
# debugging issues as it will fill
|
39
|
+
# up the output with (potentially too much) data.
|
35
40
|
def initialize(
|
36
|
-
name:,
|
37
41
|
key:,
|
42
|
+
name: '',
|
38
43
|
key_path: '',
|
39
44
|
model: {},
|
40
45
|
query: {},
|
41
46
|
register: Burner::DEFAULT_REGISTER,
|
42
|
-
separator: ''
|
47
|
+
separator: '',
|
48
|
+
debug: false
|
43
49
|
)
|
44
50
|
raise ArgumentError, 'key is required' if key.to_s.empty?
|
45
51
|
|
@@ -51,12 +57,13 @@ module DbFuel
|
|
51
57
|
model: model,
|
52
58
|
name: name,
|
53
59
|
query: query,
|
54
|
-
register: register
|
60
|
+
register: register,
|
61
|
+
debug: debug
|
55
62
|
)
|
56
63
|
end
|
57
64
|
|
58
65
|
def perform(output, payload)
|
59
|
-
records = execute(sql(payload))
|
66
|
+
records = execute(sql(output, payload))
|
60
67
|
|
61
68
|
load_register(records, output, payload)
|
62
69
|
end
|
@@ -90,8 +97,13 @@ module DbFuel
|
|
90
97
|
)
|
91
98
|
end
|
92
99
|
|
93
|
-
def sql(payload)
|
94
|
-
|
100
|
+
def sql(output, payload)
|
101
|
+
dbee_query = compile_dbee_query(payload)
|
102
|
+
sql_statement = ::Dbee.sql(model, dbee_query, provider)
|
103
|
+
|
104
|
+
debug_detail(output, "Range SQL: #{sql_statement}")
|
105
|
+
|
106
|
+
sql_statement
|
95
107
|
end
|
96
108
|
end
|
97
109
|
end
|
data/lib/db_fuel/modeling.rb
CHANGED
@@ -0,0 +1,83 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
#
|
4
|
+
# Copyright (c) 2020-present, Blue Marble Payroll, LLC
|
5
|
+
#
|
6
|
+
# This source code is licensed under the MIT license found in the
|
7
|
+
# LICENSE file in the root directory of this source tree.
|
8
|
+
#
|
9
|
+
|
10
|
+
module DbFuel
|
11
|
+
module Modeling
|
12
|
+
# Creates attribute renderers based on attributes passed.
|
13
|
+
# Also constains methods to transform attribute renderers
|
14
|
+
# and include timestamp attributes if needed.
|
15
|
+
class AttributeRendererSet
|
16
|
+
CREATED_AT = :created_at
|
17
|
+
NOW_TYPE = 'r/value/now'
|
18
|
+
UPDATED_AT = :updated_at
|
19
|
+
|
20
|
+
attr_reader :attribute_renderers, :resolver
|
21
|
+
|
22
|
+
def initialize(attributes: [], resolver: nil)
|
23
|
+
raise ArgumentError, 'resolver is required' unless resolver
|
24
|
+
|
25
|
+
@resolver = resolver
|
26
|
+
@attribute_renderers = make_attribute_renderers(attributes)
|
27
|
+
|
28
|
+
freeze
|
29
|
+
end
|
30
|
+
|
31
|
+
# Adds the attributes for created_at and updated_at to the currrent attribute renderers.
|
32
|
+
def timestamp_created_attribute_renderers
|
33
|
+
timestamp_attributes = [created_at_timestamp_attribute, updated_at_timestamp_attribute]
|
34
|
+
|
35
|
+
timestamp_attributes.map do |a|
|
36
|
+
Burner::Modeling::AttributeRenderer.new(a, resolver)
|
37
|
+
end + attribute_renderers
|
38
|
+
end
|
39
|
+
|
40
|
+
# Adds the attribute for updated_at to the currrent attribute renderers.
|
41
|
+
def timestamp_updated_attribute_renderers
|
42
|
+
timestamp_attributes = [updated_at_timestamp_attribute]
|
43
|
+
|
44
|
+
timestamp_attributes.map do |a|
|
45
|
+
Burner::Modeling::AttributeRenderer.new(a, resolver)
|
46
|
+
end + attribute_renderers
|
47
|
+
end
|
48
|
+
|
49
|
+
def make_attribute_renderers(attributes)
|
50
|
+
Burner::Modeling::Attribute
|
51
|
+
.array(attributes)
|
52
|
+
.map { |a| Burner::Modeling::AttributeRenderer.new(a, resolver) }
|
53
|
+
end
|
54
|
+
|
55
|
+
def transform(attribute_renderers, row, time)
|
56
|
+
attribute_renderers.each_with_object({}) do |attribute_renderer, memo|
|
57
|
+
value = attribute_renderer.transform(row, time)
|
58
|
+
|
59
|
+
resolver.set(memo, attribute_renderer.key, value)
|
60
|
+
end
|
61
|
+
end
|
62
|
+
|
63
|
+
private
|
64
|
+
|
65
|
+
def created_at_timestamp_attribute
|
66
|
+
timestamp_attribute(CREATED_AT)
|
67
|
+
end
|
68
|
+
|
69
|
+
def updated_at_timestamp_attribute
|
70
|
+
timestamp_attribute(UPDATED_AT)
|
71
|
+
end
|
72
|
+
|
73
|
+
def timestamp_attribute(key)
|
74
|
+
Burner::Modeling::Attribute.make(
|
75
|
+
key: key,
|
76
|
+
transformers: [
|
77
|
+
{ type: NOW_TYPE }
|
78
|
+
]
|
79
|
+
)
|
80
|
+
end
|
81
|
+
end
|
82
|
+
end
|
83
|
+
end
|
data/lib/db_fuel/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,15 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: db_fuel
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.1.
|
4
|
+
version: 1.2.1.pre.alpha.3
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Matthew Ruggio
|
8
|
-
|
8
|
+
- John Bosko
|
9
|
+
autorequire:
|
9
10
|
bindir: exe
|
10
11
|
cert_chain: []
|
11
|
-
date:
|
12
|
+
date: 2021-02-16 00:00:00.000000000 Z
|
12
13
|
dependencies:
|
13
14
|
- !ruby/object:Gem::Dependency
|
14
15
|
name: activerecord
|
@@ -50,14 +51,14 @@ dependencies:
|
|
50
51
|
requirements:
|
51
52
|
- - "~>"
|
52
53
|
- !ruby/object:Gem::Version
|
53
|
-
version: '1.
|
54
|
+
version: '1.7'
|
54
55
|
type: :runtime
|
55
56
|
prerelease: false
|
56
57
|
version_requirements: !ruby/object:Gem::Requirement
|
57
58
|
requirements:
|
58
59
|
- - "~>"
|
59
60
|
- !ruby/object:Gem::Version
|
60
|
-
version: '1.
|
61
|
+
version: '1.7'
|
61
62
|
- !ruby/object:Gem::Dependency
|
62
63
|
name: dbee
|
63
64
|
requirement: !ruby/object:Gem::Requirement
|
@@ -162,14 +163,14 @@ dependencies:
|
|
162
163
|
requirements:
|
163
164
|
- - "~>"
|
164
165
|
- !ruby/object:Gem::Version
|
165
|
-
version:
|
166
|
+
version: 1.7.0
|
166
167
|
type: :development
|
167
168
|
prerelease: false
|
168
169
|
version_requirements: !ruby/object:Gem::Requirement
|
169
170
|
requirements:
|
170
171
|
- - "~>"
|
171
172
|
- !ruby/object:Gem::Version
|
172
|
-
version:
|
173
|
+
version: 1.7.0
|
173
174
|
- !ruby/object:Gem::Dependency
|
174
175
|
name: simplecov
|
175
176
|
requirement: !ruby/object:Gem::Requirement
|
@@ -216,6 +217,7 @@ description: " This library adds database-centric jobs to the Burner library.
|
|
216
217
|
does not ship with database jobs out of the box.\n"
|
217
218
|
email:
|
218
219
|
- mruggio@bluemarblepayroll.com
|
220
|
+
- jbosko@bluemarblepayroll.com
|
219
221
|
executables: []
|
220
222
|
extensions: []
|
221
223
|
extra_rdoc_files: []
|
@@ -224,6 +226,7 @@ files:
|
|
224
226
|
- ".gitignore"
|
225
227
|
- ".rubocop.yml"
|
226
228
|
- ".ruby-version"
|
229
|
+
- ".tool-versions"
|
227
230
|
- ".travis.yml"
|
228
231
|
- CHANGELOG.md
|
229
232
|
- CODE_OF_CONDUCT.md
|
@@ -242,10 +245,13 @@ files:
|
|
242
245
|
- lib/db_fuel/library/active_record/find_or_insert.rb
|
243
246
|
- lib/db_fuel/library/active_record/insert.rb
|
244
247
|
- lib/db_fuel/library/active_record/update.rb
|
248
|
+
- lib/db_fuel/library/active_record/update_all.rb
|
249
|
+
- lib/db_fuel/library/active_record/upsert.rb
|
245
250
|
- lib/db_fuel/library/dbee/base.rb
|
246
251
|
- lib/db_fuel/library/dbee/query.rb
|
247
252
|
- lib/db_fuel/library/dbee/range.rb
|
248
253
|
- lib/db_fuel/modeling.rb
|
254
|
+
- lib/db_fuel/modeling/attribute_renderer_set.rb
|
249
255
|
- lib/db_fuel/modeling/keyed_column.rb
|
250
256
|
- lib/db_fuel/version.rb
|
251
257
|
homepage: https://github.com/bluemarblepayroll/db_fuel
|
@@ -257,7 +263,7 @@ metadata:
|
|
257
263
|
documentation_uri: https://www.rubydoc.info/gems/db_fuel
|
258
264
|
homepage_uri: https://github.com/bluemarblepayroll/db_fuel
|
259
265
|
source_code_uri: https://github.com/bluemarblepayroll/db_fuel
|
260
|
-
post_install_message:
|
266
|
+
post_install_message:
|
261
267
|
rdoc_options: []
|
262
268
|
require_paths:
|
263
269
|
- lib
|
@@ -268,12 +274,12 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
268
274
|
version: '2.5'
|
269
275
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
270
276
|
requirements:
|
271
|
-
- - "
|
277
|
+
- - ">"
|
272
278
|
- !ruby/object:Gem::Version
|
273
|
-
version:
|
279
|
+
version: 1.3.1
|
274
280
|
requirements: []
|
275
281
|
rubygems_version: 3.0.3
|
276
|
-
signing_key:
|
282
|
+
signing_key:
|
277
283
|
specification_version: 4
|
278
284
|
summary: Dbee and ActiveRecord jobs for Burner
|
279
285
|
test_files: []
|