occams-record 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/README.md +131 -0
- data/lib/occams-record.rb +5 -0
- data/lib/occams-record/batches.rb +72 -0
- data/lib/occams-record/eager_loaders.rb +29 -0
- data/lib/occams-record/eager_loaders/base.rb +61 -0
- data/lib/occams-record/eager_loaders/belongs_to.rb +36 -0
- data/lib/occams-record/eager_loaders/habtm.rb +66 -0
- data/lib/occams-record/eager_loaders/has_many.rb +20 -0
- data/lib/occams-record/eager_loaders/has_one.rb +39 -0
- data/lib/occams-record/eager_loaders/polymorphic_belongs_to.rb +74 -0
- data/lib/occams-record/query.rb +101 -0
- data/lib/occams-record/result_row.rb +91 -0
- data/lib/occams-record/version.rb +7 -0
- metadata +77 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: dd652d5b028ca9870438dd907b3d5fba86eec92f
|
4
|
+
data.tar.gz: bb9033890d9f488deeb6d077b35d76b17b78fc4e
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: ebf6b5a27af8b77beab59ef971b2b53f5909bd586f461fb05c240e384f13705016fe5a71a5e5a772d7511ea8c10042a874cf284a7074eb00b35f277c15fc509d
|
7
|
+
data.tar.gz: b822de2b3cc6c0402e7ffc3b35c2424851d2b4c44e1316f05924c8a7489b2fb7a4cd2b20c7f0a4ba4c0e89806a0204ae155c1ce04d66352114cd1e24b7d37f7e
|
data/README.md
ADDED
@@ -0,0 +1,131 @@
|
|
1
|
+
# Occam's Record
|
2
|
+
|
3
|
+
> Do not multiply entities beyond necessity. -- Occam's Razor
|
4
|
+
|
5
|
+
EXPERIMENTAL. Occam's Record is a high-efficiency API for querying large sets with ActiveRecord. When loading thousands of records, ActiveRecord wastes a lot of RAM and CPU cycles on *things you'll never use.* Additionally, eagerly-loaded associations are forced to load each and every column, even if you only need a few.
|
6
|
+
|
7
|
+
For those stuck with ActiveRecord, OccamsRecord seeks to solve these issues by making some very specific trade-offs:
|
8
|
+
|
9
|
+
* OccamsRecord results are **read-only**.
|
10
|
+
* OccamsRecord objects are **purely database rows** - they don't have any instance methods from your Rails models.
|
11
|
+
|
12
|
+
**What does this buy you?**
|
13
|
+
|
14
|
+
* OccamsRecord results are **one-thid the size** of ActiveRecord results.
|
15
|
+
* OccamsRecord queries run **three times faster** than ActiveRecord queries, or more.
|
16
|
+
* When you're eager loading associations you may specify which columns to `SELECT`. (This can be a significant performance boost to both your database and Rails app, on top of the above numbers.)
|
17
|
+
|
18
|
+
**What don't you give up?**
|
19
|
+
|
20
|
+
* You can still write your queries using ActiveRecord's query builder, as well as your existing models' associations & scopes.
|
21
|
+
* You can still use ActiveRecord for everything else - small queries, creating, updating, and deleting records.
|
22
|
+
* You can still inject some instance methods into your results, if you must. See below.
|
23
|
+
|
24
|
+
**Is there evidence to back any of this up?**
|
25
|
+
|
26
|
+
Glad you asked. [Look over the results yourself.](https://github.com/jhollinger/occams-record/wiki/Measurements)
|
27
|
+
|
28
|
+
## Examples
|
29
|
+
|
30
|
+
**Simple example**
|
31
|
+
|
32
|
+
widgets = OccamsRecord.
|
33
|
+
query(Widget.order("name")).
|
34
|
+
eager_load(:category).
|
35
|
+
run
|
36
|
+
|
37
|
+
widgets[0].id
|
38
|
+
=> 1000
|
39
|
+
|
40
|
+
widgets[0].name
|
41
|
+
=> "Widget 1000"
|
42
|
+
|
43
|
+
widgets[0].category.name
|
44
|
+
=> "Category 1"
|
45
|
+
|
46
|
+
**More complicated example**
|
47
|
+
|
48
|
+
Notice that we're eager loading splines, but *only the fields that we need*. If that's a wide table, your DBA will thank you.
|
49
|
+
|
50
|
+
widgets = OccamsRecord.
|
51
|
+
query(Widget.order("name")).
|
52
|
+
eager_load(:category).
|
53
|
+
eager_load(:splines, -> { select("widget_id, description") }).
|
54
|
+
run
|
55
|
+
|
56
|
+
widgets[0].splines.map { |s| s.description }
|
57
|
+
=> ["Spline 1", "Spline 2", "Spline 3"]
|
58
|
+
|
59
|
+
widgets[1].splines.map { |s| s.description }
|
60
|
+
=> ["Spline 4", "Spline 5"]
|
61
|
+
|
62
|
+
**An insane example, but only half as insane as the one that prompted the creation of this library**
|
63
|
+
|
64
|
+
In addition to custom eager loading queries, we're also adding nested eager loading (and customizing those queries!).
|
65
|
+
|
66
|
+
widgets = OccamsRecord.
|
67
|
+
query(Widget.order("name")).
|
68
|
+
eager_load(:category).
|
69
|
+
|
70
|
+
# load order_items, but only the fields needed to identify which orders go with which widgets
|
71
|
+
eager_load(:order_items, -> { select("widget_id, order_id") }) {
|
72
|
+
|
73
|
+
# load the orders
|
74
|
+
eager_load(:orders) {
|
75
|
+
|
76
|
+
# load the customers who made the orders, but only their names
|
77
|
+
eager_load(:customer, -> { select("id, name") })
|
78
|
+
}
|
79
|
+
}.
|
80
|
+
run
|
81
|
+
|
82
|
+
## Injecting instance methods
|
83
|
+
|
84
|
+
By default your results will only have getters for selected columns and eager-loaded associations. If you must, you *can* inject extra methods into your results by putting those methods into a Module. NOTE this is discouraged, as you should try to maintain a clear separation between your persistence layer and your domain.
|
85
|
+
|
86
|
+
module MyWidgetMethods
|
87
|
+
def to_s
|
88
|
+
name
|
89
|
+
end
|
90
|
+
|
91
|
+
def expensive?
|
92
|
+
price_per_unit > 100
|
93
|
+
end
|
94
|
+
end
|
95
|
+
|
96
|
+
module MyOrderMethods
|
97
|
+
def description
|
98
|
+
"#{order_number} - #{date}"
|
99
|
+
end
|
100
|
+
end
|
101
|
+
|
102
|
+
widgets = OccamsRecord.
|
103
|
+
query(Widget.order("name"), use: MyWidgetMethods).
|
104
|
+
eager_load(:orders, use: MyOrderMethods).
|
105
|
+
run
|
106
|
+
|
107
|
+
widgets[0].to_s
|
108
|
+
=> "Widget A"
|
109
|
+
|
110
|
+
widgets[0].price_per_unit
|
111
|
+
=> 57.23
|
112
|
+
|
113
|
+
widgets[0].expensive?
|
114
|
+
=> false
|
115
|
+
|
116
|
+
widgets[0].orders[0].description
|
117
|
+
=> "O839SJZ98B 1/8/2017"
|
118
|
+
|
119
|
+
## Testing
|
120
|
+
|
121
|
+
To run the tests, simply run:
|
122
|
+
|
123
|
+
bundle install
|
124
|
+
bundle exec rake test
|
125
|
+
|
126
|
+
By default, bundler will install the latest (supported) version of ActiveRecord. To specify a version to test against, run:
|
127
|
+
|
128
|
+
AR=4.2 bundle update activerecord
|
129
|
+
bundle exec rake test
|
130
|
+
|
131
|
+
Look inside `Gemfile` to see all testable versions.
|
@@ -0,0 +1,72 @@
|
|
1
|
+
module OccamsRecord
|
2
|
+
#
|
3
|
+
# Methods for building batch finding methods.
|
4
|
+
#
|
5
|
+
module Batches
|
6
|
+
#
|
7
|
+
# Load records in batches of N and yield each record to a block if given.
|
8
|
+
# If no block is given, returns an Enumerator.
|
9
|
+
#
|
10
|
+
# @param batch_size [Integer]
|
11
|
+
# @return [Enumerator] will yield each record
|
12
|
+
#
|
13
|
+
def find_each(batch_size: 1000)
|
14
|
+
enum = Enumerator.new { |y|
|
15
|
+
batches(of: batch_size).each { |batch|
|
16
|
+
batch.each { |record| y.yield record }
|
17
|
+
}
|
18
|
+
}
|
19
|
+
if block_given?
|
20
|
+
enum.each { |record| yield record }
|
21
|
+
else
|
22
|
+
enum
|
23
|
+
end
|
24
|
+
end
|
25
|
+
|
26
|
+
#
|
27
|
+
# Load records in batches of N and yield each batch to a block if given.
|
28
|
+
# If no block is given, returns an Enumerator.
|
29
|
+
#
|
30
|
+
# @param batch_size [Integer]
|
31
|
+
# @return [Enumerator] will yield each batch
|
32
|
+
#
|
33
|
+
def find_in_batches(batch_size: 1000)
|
34
|
+
enum = batches(of: batch_size)
|
35
|
+
if block_given?
|
36
|
+
enum.each { |batch| yield batch }
|
37
|
+
else
|
38
|
+
enum
|
39
|
+
end
|
40
|
+
end
|
41
|
+
|
42
|
+
private
|
43
|
+
|
44
|
+
#
|
45
|
+
# Returns an Enumerator that yields batches of records, of size "of".
|
46
|
+
# NOTE ActiveRecord 5+ provides the 'in_batches' method to do something
|
47
|
+
# similiar, but 4.2 doesn't have it, so...
|
48
|
+
#
|
49
|
+
# @param of [Integer] batch size
|
50
|
+
# @return [Enumerator] yields batches
|
51
|
+
#
|
52
|
+
def batches(of:)
|
53
|
+
limit = scope.limit_value
|
54
|
+
batch_size = limit && limit < of ? limit : of
|
55
|
+
Enumerator.new do |y|
|
56
|
+
offset = scope.offset_value || 0
|
57
|
+
out_of_records, count = false, 0
|
58
|
+
|
59
|
+
until out_of_records
|
60
|
+
l = limit && batch_size > limit - count ? limit - count : batch_size
|
61
|
+
q = scope.offset(offset).limit(l)
|
62
|
+
results = Query.new(q, use: @use, query_logger: @query_logger, eager_loaders: @eager_loaders).run
|
63
|
+
|
64
|
+
y.yield results if results.any?
|
65
|
+
count += results.size
|
66
|
+
offset += count
|
67
|
+
out_of_records = results.size < batch_size || (limit && count >= limit)
|
68
|
+
end
|
69
|
+
end
|
70
|
+
end
|
71
|
+
end
|
72
|
+
end
|
@@ -0,0 +1,29 @@
|
|
1
|
+
module OccamsRecord
|
2
|
+
#
|
3
|
+
# Contains eager loaders for various kinds of associations.
|
4
|
+
#
|
5
|
+
module EagerLoaders
|
6
|
+
autoload :Base, 'occams-record/eager_loaders/base'
|
7
|
+
autoload :BelongsTo, 'occams-record/eager_loaders/belongs_to'
|
8
|
+
autoload :PolymorphicBelongsTo, 'occams-record/eager_loaders/polymorphic_belongs_to'
|
9
|
+
autoload :HasOne, 'occams-record/eager_loaders/has_one'
|
10
|
+
autoload :HasMany, 'occams-record/eager_loaders/has_many'
|
11
|
+
autoload :Habtm, 'occams-record/eager_loaders/habtm'
|
12
|
+
|
13
|
+
# Fetch the appropriate eager loader for the given association type.
|
14
|
+
def self.fetch!(ref)
|
15
|
+
case ref.macro
|
16
|
+
when :belongs_to
|
17
|
+
ref.options[:polymorphic] ? PolymorphicBelongsTo : BelongsTo
|
18
|
+
when :has_one
|
19
|
+
HasOne
|
20
|
+
when :has_many
|
21
|
+
HasMany
|
22
|
+
when :has_and_belongs_to_many
|
23
|
+
EagerLoaders::Habtm
|
24
|
+
else
|
25
|
+
raise "Unsupported association type `#{macro}`"
|
26
|
+
end
|
27
|
+
end
|
28
|
+
end
|
29
|
+
end
|
@@ -0,0 +1,61 @@
|
|
1
|
+
module OccamsRecord
|
2
|
+
module EagerLoaders
|
3
|
+
#
|
4
|
+
# Base class for eagoer loading an association.
|
5
|
+
#
|
6
|
+
class Base
|
7
|
+
# @return [String] association name
|
8
|
+
attr_reader :name
|
9
|
+
# @return [Module] optional Module to include in the result class
|
10
|
+
attr_reader :use
|
11
|
+
# @return [Proc] optional Proc for eager loading things on this association
|
12
|
+
attr_reader :eval_block
|
13
|
+
|
14
|
+
#
|
15
|
+
# @param ref [ActiveRecord::Association] the ActiveRecord association
|
16
|
+
# @param scope [Proc] a scope to apply to the query (optional)
|
17
|
+
# @param use [Module] optional Module to include in the result class
|
18
|
+
# @param eval_block [Proc] a block where you may perform eager loading on *this* association (optional)
|
19
|
+
#
|
20
|
+
def initialize(ref, scope = nil, use = nil, &eval_block)
|
21
|
+
@ref, @scope, @use, @eval_block = ref, scope, use, eval_block
|
22
|
+
@name, @model = ref.name.to_s, ref.klass
|
23
|
+
@assign = "#{@name}="
|
24
|
+
end
|
25
|
+
|
26
|
+
#
|
27
|
+
# Yield one or more ActiveRecord::Relation objects to a given block.
|
28
|
+
#
|
29
|
+
# @param rows [Array<OccamsRecord::ResultRow>] Array of rows used to calculate the query.
|
30
|
+
#
|
31
|
+
def query(rows)
|
32
|
+
raise 'Not Implemented'
|
33
|
+
end
|
34
|
+
|
35
|
+
#
|
36
|
+
# Merges the associated rows into the parent rows.
|
37
|
+
#
|
38
|
+
# @param assoc_rows [Array<OccamsRecord::ResultRow>]
|
39
|
+
# @param rows [Array<OccamsRecord::ResultRow>]
|
40
|
+
#
|
41
|
+
def merge!(assoc_rows, rows)
|
42
|
+
raise 'Not Implemented'
|
43
|
+
end
|
44
|
+
|
45
|
+
private
|
46
|
+
|
47
|
+
#
|
48
|
+
# Returns the base scope for the relation, including any scope defined on the association itself,
|
49
|
+
# and any optional scope passed into the eager loader.
|
50
|
+
#
|
51
|
+
# @return [ActiveRecord::Relation]
|
52
|
+
#
|
53
|
+
def base_scope
|
54
|
+
q = @ref.klass.all
|
55
|
+
q = q.instance_exec(&@ref.scope) if @ref.scope
|
56
|
+
q = q.instance_exec(&@scope) if @scope
|
57
|
+
q
|
58
|
+
end
|
59
|
+
end
|
60
|
+
end
|
61
|
+
end
|
@@ -0,0 +1,36 @@
|
|
1
|
+
module OccamsRecord
|
2
|
+
module EagerLoaders
|
3
|
+
# Eager loader for belongs_to associations.
|
4
|
+
class BelongsTo < Base
|
5
|
+
#
|
6
|
+
# Yield one or more ActiveRecord::Relation objects to a given block.
|
7
|
+
#
|
8
|
+
# @param rows [Array<OccamsRecord::ResultRow>] Array of rows used to calculate the query.
|
9
|
+
#
|
10
|
+
def query(rows)
|
11
|
+
ids = rows.map { |r| r.send @ref.foreign_key }.compact.uniq
|
12
|
+
yield base_scope.where(@ref.active_record_primary_key => ids)
|
13
|
+
end
|
14
|
+
|
15
|
+
#
|
16
|
+
# Merge the association rows into the given rows.
|
17
|
+
#
|
18
|
+
# @param assoc_rows [Array<OccamsRecord::ResultRow>] rows loaded from the association
|
19
|
+
# @param rows [Array<OccamsRecord::ResultRow>] rows loaded from the main model
|
20
|
+
#
|
21
|
+
def merge!(assoc_rows, rows)
|
22
|
+
pkey_col = @model.primary_key.to_s
|
23
|
+
assoc_rows_by_id = assoc_rows.reduce({}) { |a, assoc_row|
|
24
|
+
id = assoc_row.send pkey_col
|
25
|
+
a[id] = assoc_row
|
26
|
+
a
|
27
|
+
}
|
28
|
+
|
29
|
+
rows.each do |row|
|
30
|
+
fkey = row.send @ref.foreign_key
|
31
|
+
row.send @assign, fkey ? assoc_rows_by_id[fkey] : nil
|
32
|
+
end
|
33
|
+
end
|
34
|
+
end
|
35
|
+
end
|
36
|
+
end
|
@@ -0,0 +1,66 @@
|
|
1
|
+
module OccamsRecord
|
2
|
+
module EagerLoaders
|
3
|
+
# Eager loader for has_and_belongs_to_many associations.
|
4
|
+
class Habtm < Base
|
5
|
+
#
|
6
|
+
# Yield one or more ActiveRecord::Relation objects to a given block.
|
7
|
+
#
|
8
|
+
# @param rows [Array<OccamsRecord::ResultRow>] Array of rows used to calculate the query.
|
9
|
+
#
|
10
|
+
def query(rows)
|
11
|
+
assoc_ids = join_rows(rows).map { |row| row[1] }.compact.uniq
|
12
|
+
yield base_scope.where(@ref.association_primary_key => assoc_ids)
|
13
|
+
end
|
14
|
+
|
15
|
+
#
|
16
|
+
# Merge the association rows into the given rows.
|
17
|
+
#
|
18
|
+
# @param assoc_rows [Array<OccamsRecord::ResultRow>] rows loaded from the association
|
19
|
+
# @param rows [Array<OccamsRecord::ResultRow>] rows loaded from the main model
|
20
|
+
#
|
21
|
+
def merge!(assoc_rows, rows)
|
22
|
+
joins_by_id = join_rows(rows).reduce({}) { |a, join|
|
23
|
+
id = join[0]
|
24
|
+
a[id] ||= []
|
25
|
+
a[id] << join[1]
|
26
|
+
a
|
27
|
+
}
|
28
|
+
|
29
|
+
assoc_rows_by_id = assoc_rows.reduce({}) { |a, row|
|
30
|
+
id = row.send @ref.association_primary_key
|
31
|
+
a[id] = row
|
32
|
+
a
|
33
|
+
}
|
34
|
+
|
35
|
+
rows.each do |row|
|
36
|
+
id = row.send @ref.active_record_primary_key
|
37
|
+
assoc_fkeys = (joins_by_id[id] || []).uniq
|
38
|
+
associations = assoc_rows_by_id.values_at(*assoc_fkeys).compact.uniq
|
39
|
+
row.send @assign, associations
|
40
|
+
end
|
41
|
+
end
|
42
|
+
|
43
|
+
private
|
44
|
+
|
45
|
+
#
|
46
|
+
# Fetches (and caches) an array of rows from the join table. The rows are [fkey, assoc_fkey].
|
47
|
+
#
|
48
|
+
# @param rows [Array<OccamsRecord::ResultRow>]
|
49
|
+
# @return [Array<Array<String>>]
|
50
|
+
#
|
51
|
+
def join_rows(rows)
|
52
|
+
return @join_rows if defined? @join_rows
|
53
|
+
|
54
|
+
conn = @model.connection
|
55
|
+
join_table = conn.quote_table_name @ref.join_table
|
56
|
+
assoc_fkey = conn.quote_column_name @ref.association_foreign_key
|
57
|
+
fkey = conn.quote_column_name @ref.foreign_key
|
58
|
+
quoted_ids = rows.map { |r| conn.quote r.send @ref.active_record_primary_key }
|
59
|
+
|
60
|
+
@join_rows = conn.
|
61
|
+
exec_query("SELECT #{fkey}, #{assoc_fkey} FROM #{join_table} WHERE #{fkey} IN (#{quoted_ids.join ','})").
|
62
|
+
rows
|
63
|
+
end
|
64
|
+
end
|
65
|
+
end
|
66
|
+
end
|
@@ -0,0 +1,20 @@
|
|
1
|
+
module OccamsRecord
|
2
|
+
module EagerLoaders
|
3
|
+
# Eager loader for has_many associations.
|
4
|
+
class HasMany < HasOne
|
5
|
+
#
|
6
|
+
# Merge the association rows into the given rows.
|
7
|
+
#
|
8
|
+
# @param assoc_rows [Array<OccamsRecord::ResultRow>] rows loaded from the association
|
9
|
+
# @param rows [Array<OccamsRecord::ResultRow>] rows loaded from the main model
|
10
|
+
#
|
11
|
+
def merge!(assoc_rows, rows)
|
12
|
+
assoc_rows_by_fkey = assoc_rows.group_by(&@ref.foreign_key.to_sym)
|
13
|
+
rows.each do |row|
|
14
|
+
pkey = row.send @ref.active_record_primary_key
|
15
|
+
row.send @assign, assoc_rows_by_fkey[pkey] || []
|
16
|
+
end
|
17
|
+
end
|
18
|
+
end
|
19
|
+
end
|
20
|
+
end
|
@@ -0,0 +1,39 @@
|
|
1
|
+
module OccamsRecord
|
2
|
+
module EagerLoaders
|
3
|
+
# Eager loader for has_one associations.
|
4
|
+
class HasOne < Base
|
5
|
+
#
|
6
|
+
# Yield one or more ActiveRecord::Relation objects to a given block.
|
7
|
+
#
|
8
|
+
# @param rows [Array<OccamsRecord::ResultRow>] Array of rows used to calculate the query.
|
9
|
+
#
|
10
|
+
def query(rows)
|
11
|
+
ids = rows.map { |r| r.send @ref.active_record_primary_key }.compact.uniq
|
12
|
+
q = base_scope.where(@ref.foreign_key => ids)
|
13
|
+
q.where!(@ref.type => rows[0].class.try!(:model_name)) if @ref.options[:as]
|
14
|
+
yield q
|
15
|
+
end
|
16
|
+
|
17
|
+
#
|
18
|
+
# Merge the association rows into the given rows.
|
19
|
+
#
|
20
|
+
# @param assoc_rows [Array<OccamsRecord::ResultRow>] rows loaded from the association
|
21
|
+
# @param rows [Array<OccamsRecord::ResultRow>] rows loaded from the main model
|
22
|
+
#
|
23
|
+
def merge!(assoc_rows, rows)
|
24
|
+
fkey_col = @ref.foreign_key.to_s
|
25
|
+
assoc_rows_by_fkey = assoc_rows.reduce({}) { |a, assoc_row|
|
26
|
+
fid = assoc_row.send fkey_col
|
27
|
+
a[fid] = assoc_row
|
28
|
+
a
|
29
|
+
}
|
30
|
+
|
31
|
+
pkey_col = @ref.active_record_primary_key.to_s
|
32
|
+
rows.each do |row|
|
33
|
+
pkey = row.send pkey_col
|
34
|
+
row.send @assign, pkey ? assoc_rows_by_fkey[pkey] : nil
|
35
|
+
end
|
36
|
+
end
|
37
|
+
end
|
38
|
+
end
|
39
|
+
end
|
@@ -0,0 +1,74 @@
|
|
1
|
+
module OccamsRecord
|
2
|
+
module EagerLoaders
|
3
|
+
# Eager loader for polymorphic belongs tos
|
4
|
+
class PolymorphicBelongsTo
|
5
|
+
# @return [String] association name
|
6
|
+
attr_reader :name
|
7
|
+
# @return [Module] optional Module to include in the result class
|
8
|
+
attr_reader :use
|
9
|
+
# @return [Proc] optional Proc for eager loading things on this association
|
10
|
+
attr_reader :eval_block
|
11
|
+
|
12
|
+
#
|
13
|
+
# @param ref [ActiveRecord::Association] the ActiveRecord association
|
14
|
+
# @param scope [Proc] a scope to apply to the query (optional)
|
15
|
+
# @param use [Module] optional Module to include in the result class
|
16
|
+
# @param eval_block [Proc] a block where you may perform eager loading on *this* association (optional)
|
17
|
+
#
|
18
|
+
def initialize(ref, scope = nil, use = nil, &eval_block)
|
19
|
+
@ref, @name, @scope, @eval_block = ref, ref.name.to_s, scope, eval_block
|
20
|
+
@foreign_type = @ref.foreign_type.to_sym
|
21
|
+
@foreign_key = @ref.foreign_key.to_sym
|
22
|
+
@use = use
|
23
|
+
@assign = "#{@name}="
|
24
|
+
end
|
25
|
+
|
26
|
+
#
|
27
|
+
# Yield ActiveRecord::Relations to the given block, one for every "type" represented in the given rows.
|
28
|
+
#
|
29
|
+
# @param rows [Array<OccamsRecord::ResultRow>] Array of rows used to calculate the query.
|
30
|
+
#
|
31
|
+
def query(rows)
|
32
|
+
rows_by_type = rows.group_by(&@foreign_type)
|
33
|
+
rows_by_type.each do |type, rows_of_type|
|
34
|
+
model = type.constantize
|
35
|
+
ids = rows_of_type.map(&@foreign_key).uniq
|
36
|
+
q = base_scope(model).where(model.primary_key => ids)
|
37
|
+
yield q
|
38
|
+
end
|
39
|
+
end
|
40
|
+
|
41
|
+
#
|
42
|
+
# Merge associations of type N into rows of model N.
|
43
|
+
#
|
44
|
+
def merge!(assoc_rows_of_type, rows)
|
45
|
+
type = assoc_rows_of_type[0].class.try!(:model_name) || return
|
46
|
+
rows_of_type = rows.select { |r| r.send(@foreign_type) == type }
|
47
|
+
merge_model!(assoc_rows_of_type, rows_of_type, type.constantize)
|
48
|
+
end
|
49
|
+
|
50
|
+
private
|
51
|
+
|
52
|
+
def base_scope(model)
|
53
|
+
q = model.all
|
54
|
+
q = q.instance_exec(&@ref.scope) if @ref.scope
|
55
|
+
q = q.instance_exec(&@scope) if @scope
|
56
|
+
q
|
57
|
+
end
|
58
|
+
|
59
|
+
def merge_model!(assoc_rows, rows, model)
|
60
|
+
pkey_col = model.primary_key.to_s
|
61
|
+
assoc_rows_by_id = assoc_rows.reduce({}) { |a, assoc_row|
|
62
|
+
id = assoc_row.send pkey_col
|
63
|
+
a[id] = assoc_row
|
64
|
+
a
|
65
|
+
}
|
66
|
+
|
67
|
+
rows.each do |row|
|
68
|
+
fkey = row.send @ref.foreign_key
|
69
|
+
row.send @assign, fkey ? assoc_rows_by_id[fkey] : nil
|
70
|
+
end
|
71
|
+
end
|
72
|
+
end
|
73
|
+
end
|
74
|
+
end
|
@@ -0,0 +1,101 @@
|
|
1
|
+
require 'occams-record/batches'
|
2
|
+
|
3
|
+
module OccamsRecord
|
4
|
+
#
|
5
|
+
# Starts building a OccamsRecord::Query. Pass it a scope from any of ActiveRecord's query builder
|
6
|
+
# methods or associations. If you want to eager loaded associations, do NOT us ActiveRecord for it.
|
7
|
+
# Instead, use OccamsRecord::Query#eager_load. Finally, call `run` to run the query and get back an
|
8
|
+
# array of objects.
|
9
|
+
#
|
10
|
+
# results = OccamsRecord.
|
11
|
+
# query(Widget.order("name")).
|
12
|
+
# eager_load(:category).
|
13
|
+
# eager_load(:order_items, ->(q) { q.select("widget_id, order_id") }) {
|
14
|
+
# eager_load(:orders) {
|
15
|
+
# eager_load(:customer, ->(q) { q.select("name") })
|
16
|
+
# }
|
17
|
+
# }.
|
18
|
+
# run
|
19
|
+
#
|
20
|
+
# @param query [ActiveRecord::Relation]
|
21
|
+
# @param use [Module] optional Module to include in the result class
|
22
|
+
# @param query_logger [Array] (optional) an array into which all queries will be inserted for logging/debug purposes
|
23
|
+
# @return [OccamsRecord::Query]
|
24
|
+
#
|
25
|
+
def self.query(scope, use: nil, query_logger: nil)
|
26
|
+
Query.new(scope, use: use, query_logger: query_logger)
|
27
|
+
end
|
28
|
+
|
29
|
+
#
|
30
|
+
# Represents a query to be run and eager associations to be loaded. Use OccamsRecord.query to create your queries
|
31
|
+
# instead of instantiating objects directly.
|
32
|
+
#
|
33
|
+
class Query
|
34
|
+
# @return [ActiveRecord::Base]
|
35
|
+
attr_reader :model
|
36
|
+
# @return [ActiveRecord::Relation] scope for building the main SQL query
|
37
|
+
attr_reader :scope
|
38
|
+
# @return [ActiveRecord::Connection]
|
39
|
+
attr_reader :conn
|
40
|
+
|
41
|
+
include Batches
|
42
|
+
|
43
|
+
#
|
44
|
+
# Initialize a new query.
|
45
|
+
#
|
46
|
+
# @param scope [ActiveRecord::Relation]
|
47
|
+
# @param use [Module] optional Module to include in the result class
|
48
|
+
# @param query_logger [Array] (optional) an array into which all queries will be inserted for logging/debug purposes
|
49
|
+
# @param eager_loaders [OccamsRecord::EagerLoaders::Base]
|
50
|
+
# @param eval_block [Proc] block that will be eval'd on this instance. Can be used for eager loading. (optional)
|
51
|
+
#
|
52
|
+
def initialize(scope, use: nil, query_logger: nil, eager_loaders: [], &eval_block)
|
53
|
+
@model = scope.klass
|
54
|
+
@scope = scope
|
55
|
+
@eager_loaders = eager_loaders
|
56
|
+
@conn = model.connection
|
57
|
+
@use = use
|
58
|
+
@query_logger = query_logger
|
59
|
+
instance_eval(&eval_block) if eval_block
|
60
|
+
end
|
61
|
+
|
62
|
+
#
|
63
|
+
# Specify an association to be eager-loaded. You may optionally pass a block that accepts a scope
|
64
|
+
# which you may modify to customize the query. For maximum memory savings, always `select` only
|
65
|
+
# the colums you actually need.
|
66
|
+
#
|
67
|
+
# @param assoc [Symbol] name of association
|
68
|
+
# @param scope [Proc] a scope to apply to the query (optional)
|
69
|
+
# @param use [Module] optional Module to include in the result class
|
70
|
+
# @param eval_block [Proc] a block where you may perform eager loading on *this* association (optional)
|
71
|
+
#
|
72
|
+
def eager_load(assoc, scope = nil, use: nil, &eval_block)
|
73
|
+
ref = model.reflections[assoc.to_s]
|
74
|
+
raise "OccamsRecord: No assocation `:#{assoc}` on `#{model.name}`" if ref.nil?
|
75
|
+
@eager_loaders << EagerLoaders.fetch!(ref).new(ref, scope, use, &eval_block)
|
76
|
+
self
|
77
|
+
end
|
78
|
+
|
79
|
+
#
|
80
|
+
# Run the query and return the results.
|
81
|
+
#
|
82
|
+
# @return [Array<OccamsRecord::ResultRow>]
|
83
|
+
#
|
84
|
+
def run
|
85
|
+
sql = scope.to_sql
|
86
|
+
@query_logger << sql if @query_logger
|
87
|
+
result = conn.exec_query sql
|
88
|
+
row_class = OccamsRecord.build_result_row_class(model, result.columns, @eager_loaders.map(&:name), @use)
|
89
|
+
rows = result.rows.map { |row| row_class.new row }
|
90
|
+
|
91
|
+
@eager_loaders.each { |loader|
|
92
|
+
loader.query(rows) { |scope|
|
93
|
+
assoc_rows = Query.new(scope, use: loader.use, query_logger: @query_logger, &loader.eval_block).run
|
94
|
+
loader.merge! assoc_rows, rows
|
95
|
+
}
|
96
|
+
}
|
97
|
+
|
98
|
+
rows
|
99
|
+
end
|
100
|
+
end
|
101
|
+
end
|
@@ -0,0 +1,91 @@
|
|
1
|
+
module OccamsRecord
|
2
|
+
# ActiveRecord's internal type casting API changes from version to version.
|
3
|
+
TYPE_CAST_METHOD = case ActiveRecord::VERSION::MAJOR
|
4
|
+
when 4 then :type_cast_from_database
|
5
|
+
when 5 then :deserialize
|
6
|
+
end
|
7
|
+
|
8
|
+
#
|
9
|
+
# Dynamically build a class for a specific set of result rows. It inherits from OccamsRecord::ResultRow, and optionall includes
|
10
|
+
# a user-defined module.
|
11
|
+
#
|
12
|
+
# @param model [ActiveRecord::Base] the AR model representing the table (it holds column & type info).
|
13
|
+
# @param column_names [Array<String>] the column names in the result set. The order MUST match the order returned by the query.
|
14
|
+
# @param association_names [Array<String>] names of associations that will be eager loaded into the results.
|
15
|
+
# @param included_module [Module] (optional)
|
16
|
+
# @return [OccamsRecord::ResultRow] a class customized for this result set
|
17
|
+
#
|
18
|
+
def self.build_result_row_class(model, column_names, association_names, included_module = nil)
|
19
|
+
Class.new(ResultRow) do
|
20
|
+
include included_module if included_module
|
21
|
+
|
22
|
+
self.columns = column_names.map(&:to_s)
|
23
|
+
self.associations = association_names.map(&:to_s)
|
24
|
+
self.model_name = model.name
|
25
|
+
|
26
|
+
# Build getters & setters for associations. (We need setters b/c they're set AFTER the row is initialized
|
27
|
+
attr_accessor(*association_names)
|
28
|
+
|
29
|
+
# Build a getter for each attribute returned by the query. The values will be type converted on demand.
|
30
|
+
column_names.each_with_index do |col, idx|
|
31
|
+
type = model.attributes_builder.types[col.to_s] || raise("OccamsRecord: Column `#{col}` does not exist on model `#{model.name}`")
|
32
|
+
define_method col do
|
33
|
+
@cast_values_cache[idx] ||= type.send(TYPE_CAST_METHOD, @raw_values[idx])
|
34
|
+
end
|
35
|
+
end
|
36
|
+
end
|
37
|
+
end
|
38
|
+
|
39
|
+
#
|
40
|
+
# Abstract class for result rows.
|
41
|
+
#
|
42
|
+
class ResultRow
|
43
|
+
class << self
|
44
|
+
# Array of column names
|
45
|
+
attr_accessor :columns
|
46
|
+
# Array of associations names
|
47
|
+
attr_accessor :associations
|
48
|
+
# Name of Rails model
|
49
|
+
attr_accessor :model_name
|
50
|
+
end
|
51
|
+
self.columns = []
|
52
|
+
self.associations = []
|
53
|
+
|
54
|
+
#
|
55
|
+
# Initialize a new result row.
|
56
|
+
#
|
57
|
+
# @param raw_values [Array] array of raw values from db
|
58
|
+
#
|
59
|
+
def initialize(raw_values)
|
60
|
+
@raw_values = raw_values
|
61
|
+
@cast_values_cache = {}
|
62
|
+
end
|
63
|
+
|
64
|
+
#
|
65
|
+
# Return row as a Hash (recursive).
|
66
|
+
#
|
67
|
+
# @param symbolize_names [Boolean] if true, make Hash keys Symbols instead of Strings
|
68
|
+
# @return [Hash] a Hash with String or Symbol keys
|
69
|
+
#
|
70
|
+
def to_h(symbolize_names: false)
|
71
|
+
hash = self.class.columns.reduce({}) { |a, col_name|
|
72
|
+
key = symbolize_names ? col_name.to_sym : col_name
|
73
|
+
a[key] = send col_name
|
74
|
+
a
|
75
|
+
}
|
76
|
+
|
77
|
+
self.class.associations.reduce(hash) { |a, assoc_name|
|
78
|
+
key = symbolize_names ? assoc_name.to_sym : assoc_name
|
79
|
+
assoc = send assoc_name
|
80
|
+
a[key] = if assoc.is_a? Array
|
81
|
+
assoc.map { |x| x.to_h(symbolize_names: symbolize_names) }
|
82
|
+
elsif assoc
|
83
|
+
assoc.to_h(symbolize_names: symbolize_names)
|
84
|
+
end
|
85
|
+
a
|
86
|
+
}
|
87
|
+
end
|
88
|
+
|
89
|
+
alias_method :to_hash, :to_h
|
90
|
+
end
|
91
|
+
end
|
metadata
ADDED
@@ -0,0 +1,77 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: occams-record
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: 0.1.0
|
5
|
+
platform: ruby
|
6
|
+
authors:
|
7
|
+
- Jordan Hollinger
|
8
|
+
autorequire:
|
9
|
+
bindir: bin
|
10
|
+
cert_chain: []
|
11
|
+
date: 2017-08-19 00:00:00.000000000 Z
|
12
|
+
dependencies:
|
13
|
+
- !ruby/object:Gem::Dependency
|
14
|
+
name: activerecord
|
15
|
+
requirement: !ruby/object:Gem::Requirement
|
16
|
+
requirements:
|
17
|
+
- - ">="
|
18
|
+
- !ruby/object:Gem::Version
|
19
|
+
version: '4.2'
|
20
|
+
- - "<"
|
21
|
+
- !ruby/object:Gem::Version
|
22
|
+
version: '5.2'
|
23
|
+
type: :runtime
|
24
|
+
prerelease: false
|
25
|
+
version_requirements: !ruby/object:Gem::Requirement
|
26
|
+
requirements:
|
27
|
+
- - ">="
|
28
|
+
- !ruby/object:Gem::Version
|
29
|
+
version: '4.2'
|
30
|
+
- - "<"
|
31
|
+
- !ruby/object:Gem::Version
|
32
|
+
version: '5.2'
|
33
|
+
description: A faster, lower-memory querying API for ActiveRecord that returns results
|
34
|
+
as unadorned, read-only objects.
|
35
|
+
email: jordan.hollinger@gmail.com
|
36
|
+
executables: []
|
37
|
+
extensions: []
|
38
|
+
extra_rdoc_files: []
|
39
|
+
files:
|
40
|
+
- README.md
|
41
|
+
- lib/occams-record.rb
|
42
|
+
- lib/occams-record/batches.rb
|
43
|
+
- lib/occams-record/eager_loaders.rb
|
44
|
+
- lib/occams-record/eager_loaders/base.rb
|
45
|
+
- lib/occams-record/eager_loaders/belongs_to.rb
|
46
|
+
- lib/occams-record/eager_loaders/habtm.rb
|
47
|
+
- lib/occams-record/eager_loaders/has_many.rb
|
48
|
+
- lib/occams-record/eager_loaders/has_one.rb
|
49
|
+
- lib/occams-record/eager_loaders/polymorphic_belongs_to.rb
|
50
|
+
- lib/occams-record/query.rb
|
51
|
+
- lib/occams-record/result_row.rb
|
52
|
+
- lib/occams-record/version.rb
|
53
|
+
homepage: https://github.com/jhollinger/occams-record
|
54
|
+
licenses:
|
55
|
+
- MIT
|
56
|
+
metadata: {}
|
57
|
+
post_install_message:
|
58
|
+
rdoc_options: []
|
59
|
+
require_paths:
|
60
|
+
- lib
|
61
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
62
|
+
requirements:
|
63
|
+
- - ">="
|
64
|
+
- !ruby/object:Gem::Version
|
65
|
+
version: 2.1.0
|
66
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
67
|
+
requirements:
|
68
|
+
- - ">="
|
69
|
+
- !ruby/object:Gem::Version
|
70
|
+
version: '0'
|
71
|
+
requirements: []
|
72
|
+
rubyforge_project:
|
73
|
+
rubygems_version: 2.5.2
|
74
|
+
signing_key:
|
75
|
+
specification_version: 4
|
76
|
+
summary: The missing high-efficiency query API for ActiveRecord
|
77
|
+
test_files: []
|