occams-record 0.5.0 → 0.6.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +10 -2
- data/lib/occams-record.rb +1 -0
- data/lib/occams-record/batches.rb +9 -3
- data/lib/occams-record/eager_loaders/base.rb +0 -1
- data/lib/occams-record/eager_loaders/belongs_to.rb +2 -11
- data/lib/occams-record/eager_loaders/habtm.rb +2 -1
- data/lib/occams-record/eager_loaders/has_many.rb +2 -5
- data/lib/occams-record/eager_loaders/has_one.rb +3 -12
- data/lib/occams-record/eager_loaders/polymorphic_belongs_to.rb +5 -18
- data/lib/occams-record/merge.rb +57 -0
- data/lib/occams-record/version.rb +1 -1
- metadata +3 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: f60c7e2adfe9a547efe1d4c9eccd25b8baa7c733
|
4
|
+
data.tar.gz: 7955381fac5f5792b7e9a537dd5962199dff7116
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 2e47c6911ac5648a01c9feeadcee5c1dd1e0a3d5dc9b475108bbbb86a6e06a87f31863521862ee6c807c33e85b34b793a9469188c1d72789d2cb389d088cdaf7
|
7
|
+
data.tar.gz: 0a35392419e4527552da87d433754d0a2998d016d5a27f332f78d6932605cdb856e256175d06cfbd9f8c214bd4908b5ed6d8455c38169f0568c3ad4ac5442ce0
|
data/README.md
CHANGED
@@ -2,18 +2,22 @@
|
|
2
2
|
|
3
3
|
> Do not multiply entities beyond necessity. -- Occam's Razor
|
4
4
|
|
5
|
-
|
5
|
+
Occam's Record is a high-efficiency query API for ActiveRecord. When loading thousands of records, ActiveRecord wastes a lot of RAM and CPU cycles on *things you'll never use.* Additionally, eagerly-loaded associations are forced to load each and every column, even if you only need a few.
|
6
6
|
|
7
7
|
For those stuck with ActiveRecord, OccamsRecord seeks to solve these issues by making some very specific trade-offs:
|
8
8
|
|
9
9
|
* OccamsRecord results are **read-only**.
|
10
10
|
* OccamsRecord objects are **purely database rows** - they don't have any instance methods from your Rails models.
|
11
|
+
* OccamsRecord queries must specify each association that will be used. Otherwise they simply won't be availble.
|
11
12
|
|
12
13
|
**What does this buy you?**
|
13
14
|
|
14
15
|
* OccamsRecord results are **one-third the size** of ActiveRecord results.
|
15
16
|
* OccamsRecord queries run **three to five times faster** than ActiveRecord queries.
|
16
|
-
* When
|
17
|
+
* When eager loading associations you may specify which columns to `SELECT`. (This can be a significant performance boost to both your database and Rails app, on top of the above numbers.)
|
18
|
+
* When eager loading associations you may completely customize the query (`WHERE`, `ORDER BY`, `LIMIT`, etc.)
|
19
|
+
* By forcing eager loading of associations, OccamsRecord bypasses the primary cause of performance problems in Rails: N+1 queries.
|
20
|
+
* The forced eager loading helps the developer visualize the "shape" of the query/result, which can make obvious certain insights that would otherwise require lots of digging. For example, if you're eager loading 15 associations, maybe you need to add some redundant foreign keys or denormalize some fields.
|
17
21
|
|
18
22
|
**What don't you give up?**
|
19
23
|
|
@@ -25,6 +29,10 @@ For those stuck with ActiveRecord, OccamsRecord seeks to solve these issues by m
|
|
25
29
|
|
26
30
|
Glad you asked. [Look over the results yourself.](https://github.com/jhollinger/occams-record/wiki/Measurements)
|
27
31
|
|
32
|
+
**Why not use a different ORM?**
|
33
|
+
|
34
|
+
That's a great idea; check out [sequel](https://rubygems.org/gems/sequel) or [rom](https://rubygems.org/gems/rom)! But for large, legacy codebases heavily invested in ActiveRecord, switching ORMs often isn't practical. OccamsRecord can help you get some of those wins without throwing everything out.
|
35
|
+
|
28
36
|
## Usage
|
29
37
|
|
30
38
|
**Add to your Gemfile**
|
data/lib/occams-record.rb
CHANGED
@@ -1,12 +1,15 @@
|
|
1
1
|
module OccamsRecord
|
2
2
|
#
|
3
|
-
# Methods for building batch finding methods.
|
3
|
+
# Methods for building batch finding methods. It expects "model" and "scope" methods to be present.
|
4
4
|
#
|
5
5
|
module Batches
|
6
6
|
#
|
7
7
|
# Load records in batches of N and yield each record to a block if given.
|
8
8
|
# If no block is given, returns an Enumerator.
|
9
9
|
#
|
10
|
+
# NOTE Unlike ActiveRecord's find_each, order IS respected. The primary key will be appended
|
11
|
+
# to the ORDER BY clause to help ensure consistent batches.
|
12
|
+
#
|
10
13
|
# @param batch_size [Integer]
|
11
14
|
# @return [Enumerator] will yield each record
|
12
15
|
#
|
@@ -27,6 +30,9 @@ module OccamsRecord
|
|
27
30
|
# Load records in batches of N and yield each batch to a block if given.
|
28
31
|
# If no block is given, returns an Enumerator.
|
29
32
|
#
|
33
|
+
# NOTE Unlike ActiveRecord's find_in_batches, order IS respected. The primary key will be appended
|
34
|
+
# to the ORDER BY clause to help ensure consistent batches.
|
35
|
+
#
|
30
36
|
# @param batch_size [Integer]
|
31
37
|
# @return [Enumerator] will yield each batch
|
32
38
|
#
|
@@ -44,7 +50,7 @@ module OccamsRecord
|
|
44
50
|
#
|
45
51
|
# Returns an Enumerator that yields batches of records, of size "of".
|
46
52
|
# NOTE ActiveRecord 5+ provides the 'in_batches' method to do something
|
47
|
-
# similiar,
|
53
|
+
# similiar, although 4.2 does not. Also it does not respect ORDER BY.
|
48
54
|
#
|
49
55
|
# @param of [Integer] batch size
|
50
56
|
# @return [Enumerator] yields batches
|
@@ -58,7 +64,7 @@ module OccamsRecord
|
|
58
64
|
|
59
65
|
until out_of_records
|
60
66
|
l = limit && batch_size > limit - count ? limit - count : batch_size
|
61
|
-
q = scope.offset(offset).limit(l)
|
67
|
+
q = scope.order(model.primary_key.to_sym).offset(offset).limit(l)
|
62
68
|
results = Query.new(q, use: @use, query_logger: @query_logger, eager_loaders: @eager_loaders).run
|
63
69
|
|
64
70
|
y.yield results if results.any?
|
@@ -19,17 +19,8 @@ module OccamsRecord
|
|
19
19
|
# @param rows [Array<OccamsRecord::ResultRow>] rows loaded from the main model
|
20
20
|
#
|
21
21
|
def merge!(assoc_rows, rows)
|
22
|
-
|
23
|
-
|
24
|
-
id = assoc_row.send pkey_col
|
25
|
-
a[id] = assoc_row
|
26
|
-
a
|
27
|
-
}
|
28
|
-
|
29
|
-
rows.each do |row|
|
30
|
-
fkey = row.send @ref.foreign_key
|
31
|
-
row.send @assign, fkey ? assoc_rows_by_id[fkey] : nil
|
32
|
-
end
|
22
|
+
Merge.new(rows, name).
|
23
|
+
single!(assoc_rows, @ref.foreign_key.to_s, @model.primary_key.to_s)
|
33
24
|
end
|
34
25
|
end
|
35
26
|
end
|
@@ -32,11 +32,12 @@ module OccamsRecord
|
|
32
32
|
a
|
33
33
|
}
|
34
34
|
|
35
|
+
assign = "#{name}="
|
35
36
|
rows.each do |row|
|
36
37
|
id = row.send(@ref.active_record_primary_key).to_s
|
37
38
|
assoc_fkeys = (joins_by_id[id] || []).uniq
|
38
39
|
associations = assoc_rows_by_id.values_at(*assoc_fkeys).compact.uniq
|
39
|
-
row.send
|
40
|
+
row.send assign, associations
|
40
41
|
end
|
41
42
|
end
|
42
43
|
|
@@ -9,11 +9,8 @@ module OccamsRecord
|
|
9
9
|
# @param rows [Array<OccamsRecord::ResultRow>] rows loaded from the main model
|
10
10
|
#
|
11
11
|
def merge!(assoc_rows, rows)
|
12
|
-
|
13
|
-
|
14
|
-
pkey = row.send @ref.active_record_primary_key
|
15
|
-
row.send @assign, assoc_rows_by_fkey[pkey] || []
|
16
|
-
end
|
12
|
+
Merge.new(rows, name).
|
13
|
+
many!(assoc_rows, @ref.active_record_primary_key, @ref.foreign_key)
|
17
14
|
end
|
18
15
|
end
|
19
16
|
end
|
@@ -8,6 +8,7 @@ module OccamsRecord
|
|
8
8
|
# @param rows [Array<OccamsRecord::ResultRow>] Array of rows used to calculate the query.
|
9
9
|
#
|
10
10
|
def query(rows)
|
11
|
+
return if rows.empty?
|
11
12
|
ids = rows.map { |r| r.send @ref.active_record_primary_key }.compact.uniq
|
12
13
|
q = base_scope.where(@ref.foreign_key => ids)
|
13
14
|
q.where!(@ref.type => rows[0].class.try!(:model_name)) if @ref.options[:as]
|
@@ -21,18 +22,8 @@ module OccamsRecord
|
|
21
22
|
# @param rows [Array<OccamsRecord::ResultRow>] rows loaded from the main model
|
22
23
|
#
|
23
24
|
def merge!(assoc_rows, rows)
|
24
|
-
|
25
|
-
|
26
|
-
fid = assoc_row.send fkey_col
|
27
|
-
a[fid] = assoc_row
|
28
|
-
a
|
29
|
-
}
|
30
|
-
|
31
|
-
pkey_col = @ref.active_record_primary_key.to_s
|
32
|
-
rows.each do |row|
|
33
|
-
pkey = row.send pkey_col
|
34
|
-
row.send @assign, pkey ? assoc_rows_by_fkey[pkey] : nil
|
35
|
-
end
|
25
|
+
Merge.new(rows, name).
|
26
|
+
single!(assoc_rows, @ref.active_record_primary_key.to_s, @ref.foreign_key.to_s)
|
36
27
|
end
|
37
28
|
end
|
38
29
|
end
|
@@ -16,11 +16,10 @@ module OccamsRecord
|
|
16
16
|
# @param eval_block [Proc] a block where you may perform eager loading on *this* association (optional)
|
17
17
|
#
|
18
18
|
def initialize(ref, scope = nil, use = nil, &eval_block)
|
19
|
-
@ref, @
|
19
|
+
@ref, @scope, @use, @eval_block = ref, scope, use, eval_block
|
20
|
+
@name = ref.name.to_s
|
20
21
|
@foreign_type = @ref.foreign_type.to_sym
|
21
22
|
@foreign_key = @ref.foreign_key.to_sym
|
22
|
-
@use = use
|
23
|
-
@assign = "#{@name}="
|
24
23
|
end
|
25
24
|
|
26
25
|
#
|
@@ -45,7 +44,9 @@ module OccamsRecord
|
|
45
44
|
return if assoc_rows_of_type.empty?
|
46
45
|
type = assoc_rows_of_type[0].class.model_name
|
47
46
|
rows_of_type = rows.select { |r| r.send(@foreign_type) == type }
|
48
|
-
|
47
|
+
model = type.constantize
|
48
|
+
Merge.new(rows_of_type, name).
|
49
|
+
single!(assoc_rows_of_type, @ref.foreign_key.to_s, model.primary_key.to_s)
|
49
50
|
end
|
50
51
|
|
51
52
|
private
|
@@ -56,20 +57,6 @@ module OccamsRecord
|
|
56
57
|
q = q.instance_exec(&@scope) if @scope
|
57
58
|
q
|
58
59
|
end
|
59
|
-
|
60
|
-
def merge_model!(assoc_rows, rows, model)
|
61
|
-
pkey_col = model.primary_key.to_s
|
62
|
-
assoc_rows_by_id = assoc_rows.reduce({}) { |a, assoc_row|
|
63
|
-
id = assoc_row.send pkey_col
|
64
|
-
a[id] = assoc_row
|
65
|
-
a
|
66
|
-
}
|
67
|
-
|
68
|
-
rows.each do |row|
|
69
|
-
fkey = row.send @ref.foreign_key
|
70
|
-
row.send @assign, fkey ? assoc_rows_by_id[fkey] : nil
|
71
|
-
end
|
72
|
-
end
|
73
60
|
end
|
74
61
|
end
|
75
62
|
end
|
@@ -0,0 +1,57 @@
|
|
1
|
+
module OccamsRecord
|
2
|
+
#
|
3
|
+
# Represents a merge operation to be performed. Merges are always "left" merges. You initialize the
|
4
|
+
# Merge with the "left" records, and the name of the attribute into which "right" records will be placed.
|
5
|
+
#
|
6
|
+
# After initializing, perform a specific type of merge by calling the appropriate *! method.
|
7
|
+
#
|
8
|
+
class Merge
|
9
|
+
# @return [Array<OccamsRecord::ResultRow>] the rows into which associated rows will be merged
|
10
|
+
attr_reader :target_rows
|
11
|
+
|
12
|
+
#
|
13
|
+
# Initialize a new Merge operation.
|
14
|
+
#
|
15
|
+
# @param target_rows [Array<OccamsRecord::ResultRow] the rows into which associated rows should be merged
|
16
|
+
# @param assoc_name [String|Symbol] name of the attribute where associated rows will be put
|
17
|
+
#
|
18
|
+
def initialize(target_rows, assoc_name)
|
19
|
+
@target_rows = target_rows
|
20
|
+
@assign = "#{assoc_name}="
|
21
|
+
end
|
22
|
+
|
23
|
+
#
|
24
|
+
# Merge a single assoc_row into each target_rows (or nil if one can't be found).
|
25
|
+
# target_attr and assoc_attr are the matching keys on target_rows and assoc_rows, respectively.
|
26
|
+
#
|
27
|
+
# @param assoc_rows [Array<OccamsRecord::ResultRow>] rows to merge into target_rows
|
28
|
+
# @param target_attr [String|Symbol] name of the matching key on the target records
|
29
|
+
# @param assoc_attr [String] name of the matching key on the associated records
|
30
|
+
#
|
31
|
+
def single!(assoc_rows, target_attr, assoc_attr)
|
32
|
+
assoc_rows_by_id = assoc_rows.reduce({}) { |a, assoc_row|
|
33
|
+
id = assoc_row.send assoc_attr
|
34
|
+
a[id] = assoc_row
|
35
|
+
a
|
36
|
+
}
|
37
|
+
|
38
|
+
target_rows.each do |row|
|
39
|
+
attr = row.send target_attr
|
40
|
+
row.send @assign, attr ? assoc_rows_by_id[attr] : nil
|
41
|
+
end
|
42
|
+
end
|
43
|
+
|
44
|
+
#
|
45
|
+
# Merge an array of assoc_rows into the target_rows. Some target_rows may end up with 0 matching
|
46
|
+
# associations, and they'll be assigned empty arrays.
|
47
|
+
# target_attr and assoc_attr are the matching keys on target_rows and assoc_rows, respectively.
|
48
|
+
#
|
49
|
+
def many!(assoc_rows, target_attr, assoc_attr)
|
50
|
+
assoc_rows_by_attr = assoc_rows.group_by(&assoc_attr.to_sym)
|
51
|
+
target_rows.each do |row|
|
52
|
+
pkey = row.send target_attr
|
53
|
+
row.send @assign, assoc_rows_by_attr[pkey] || []
|
54
|
+
end
|
55
|
+
end
|
56
|
+
end
|
57
|
+
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: occams-record
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.6.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Jordan Hollinger
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2017-09-
|
11
|
+
date: 2017-09-11 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activerecord
|
@@ -47,6 +47,7 @@ files:
|
|
47
47
|
- lib/occams-record/eager_loaders/has_many.rb
|
48
48
|
- lib/occams-record/eager_loaders/has_one.rb
|
49
49
|
- lib/occams-record/eager_loaders/polymorphic_belongs_to.rb
|
50
|
+
- lib/occams-record/merge.rb
|
50
51
|
- lib/occams-record/query.rb
|
51
52
|
- lib/occams-record/result_row.rb
|
52
53
|
- lib/occams-record/version.rb
|