occams-record 0.9.0 → 0.10.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +32 -24
- data/lib/occams-record/batches.rb +2 -0
- data/lib/occams-record/eager_loaders.rb +7 -6
- data/lib/occams-record/eager_loaders/base.rb +5 -4
- data/lib/occams-record/eager_loaders/belongs_to.rb +1 -0
- data/lib/occams-record/eager_loaders/habtm.rb +1 -0
- data/lib/occams-record/eager_loaders/has_one.rb +1 -0
- data/lib/occams-record/eager_loaders/polymorphic_belongs_to.rb +6 -4
- data/lib/occams-record/query.rb +3 -2
- data/lib/occams-record/raw_query.rb +4 -4
- data/lib/occams-record/results.rb +6 -0
- data/lib/occams-record/version.rb +1 -1
- metadata +2 -2
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 9c305e7ef50172b06abc65501fda3a2e323a4b59
|
4
|
+
data.tar.gz: 8b3148052f587c1987485527473e99764132130a
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 4907ef959db18d068d58007f6f1df7d2aac60ce69cf1b9af4c91e89c518b8b2eb103d5574b6d03a34964a1456c62f18b372e0c50595a1bc9c2cb3b9a967afbbe
|
7
|
+
data.tar.gz: 39cbe4f0a8bb3af17944aba9d6c7839a6751b9b19053adbceb8df924815718d9c9dfaa596733bb63f1b3fb873815059c79264127d99b31a9621ffa69bdb99d4e
|
data/README.md
CHANGED
@@ -2,7 +2,7 @@
|
|
2
2
|
|
3
3
|
> Do not multiply entities beyond necessity. -- Occam's Razor
|
4
4
|
|
5
|
-
Occam's Record is a high-efficiency query API for ActiveRecord. When loading thousands of records, ActiveRecord wastes a lot of RAM and CPU cycles on *things you'll never use.* Additionally, eagerly-loaded associations are forced to load each and every column,
|
5
|
+
Occam's Record is a high-efficiency query API for ActiveRecord. When loading thousands of records, ActiveRecord wastes a lot of RAM and CPU cycles on *things you'll never use.* Additionally, eagerly-loaded associations are forced to load each and every column, each and every record, and all in a certain order.
|
6
6
|
|
7
7
|
For those stuck with ActiveRecord, OccamsRecord seeks to solve these issues by making some very specific trade-offs:
|
8
8
|
|
@@ -10,28 +10,9 @@ For those stuck with ActiveRecord, OccamsRecord seeks to solve these issues by m
|
|
10
10
|
* OccamsRecord objects are **purely database rows** - they don't have any instance methods from your Rails models.
|
11
11
|
* OccamsRecord queries must specify each association that will be used. Otherwise they simply won't be availble.
|
12
12
|
|
13
|
-
|
13
|
+
For more on the rational behind OccamsRecord, see the Rational section at the end of the README. But in short, OccamsRecord is 3x-5x faster, uses 1/3 of the memory, and eliminates the N+1 query problem.
|
14
14
|
|
15
|
-
|
16
|
-
* OccamsRecord queries run **three to five times faster** than ActiveRecord queries.
|
17
|
-
* When eager loading associations you may specify which columns to `SELECT`. (This can be a significant performance boost to both your database and Rails app, on top of the above numbers.)
|
18
|
-
* When eager loading associations you may completely customize the query (`WHERE`, `ORDER BY`, `LIMIT`, etc.)
|
19
|
-
* By forcing eager loading of associations, OccamsRecord bypasses the primary cause of performance problems in Rails: N+1 queries.
|
20
|
-
* Forced eager loading also makes you consider the "shape" of your data, which can help you identify areas that need refactored (e.g. redundant foreign keys, more denormalization, etc.)
|
21
|
-
|
22
|
-
**What don't you give up?**
|
23
|
-
|
24
|
-
* You can still write your queries using ActiveRecord's query builder, as well as your existing models' associations & scopes.
|
25
|
-
* You can still use ActiveRecord for everything else - small queries, creating, updating, and deleting records.
|
26
|
-
* You can still inject some instance methods into your results, if you must. See below.
|
27
|
-
|
28
|
-
**Is there evidence to back any of this up?**
|
29
|
-
|
30
|
-
Glad you asked. [Look over the results yourself.](https://github.com/jhollinger/occams-record/wiki/Measurements)
|
31
|
-
|
32
|
-
**Why not use a different ORM?**
|
33
|
-
|
34
|
-
That's a great idea; check out [sequel](https://rubygems.org/gems/sequel) or [rom](https://rubygems.org/gems/rom)! But for large, legacy codebases heavily invested in ActiveRecord, switching ORMs often isn't practical. OccamsRecord can help you get some of those wins without a rewrite.
|
15
|
+
**BREAKING CHANGE** to `eager_load` in version **0.10.0**. See the examples below or [HISTORY.md](https://github.com/jhollinger/occams-record/blob/v0.10.0/HISTORY.md) for the new usage.
|
35
16
|
|
36
17
|
## Usage
|
37
18
|
|
@@ -41,6 +22,8 @@ That's a great idea; check out [sequel](https://rubygems.org/gems/sequel) or [ro
|
|
41
22
|
gem 'occams-record'
|
42
23
|
```
|
43
24
|
|
25
|
+
Full documentation is available at [rubydoc.info/gems/occams-record](http://www.rubydoc.info/gems/occams-record).
|
26
|
+
|
44
27
|
**Simple example**
|
45
28
|
|
46
29
|
```ruby
|
@@ -89,8 +72,8 @@ widgets = OccamsRecord.
|
|
89
72
|
# load order_items, but only the fields needed to identify which orders go with which widgets
|
90
73
|
eager_load(:order_items, select: "widget_id, order_id") {
|
91
74
|
|
92
|
-
# load the orders
|
93
|
-
eager_load(:orders, -> { select("id, customer_id").order("order_date DESC") }) {
|
75
|
+
# load the orders ("q" has all the normal query methods and any scopes defined on Order)
|
76
|
+
eager_load(:orders, ->(q) { q.select("id, customer_id").order("order_date DESC") }) {
|
94
77
|
|
95
78
|
# load the customers who made the orders, but only their names
|
96
79
|
eager_load(:customer, select: "id, name")
|
@@ -168,6 +151,31 @@ widgets = OccamsRecord.
|
|
168
151
|
run
|
169
152
|
```
|
170
153
|
|
154
|
+
## Rational
|
155
|
+
|
156
|
+
**What does OccamsRecord buy you?**
|
157
|
+
|
158
|
+
* OccamsRecord results are **one-third the size** of ActiveRecord results.
|
159
|
+
* OccamsRecord queries run **three to five times faster** than ActiveRecord queries.
|
160
|
+
* When eager loading associations you may specify which columns to `SELECT`. (This can be a significant performance boost to both your database and Rails app, on top of the above numbers.)
|
161
|
+
* When eager loading associations you may completely customize the query (`WHERE`, `ORDER BY`, `LIMIT`, etc.)
|
162
|
+
* By forcing eager loading of associations, OccamsRecord bypasses the primary cause of performance problems in Rails: N+1 queries.
|
163
|
+
* Forced eager loading also makes you consider the "shape" of your data, which can help you identify areas that need refactored (e.g. redundant foreign keys, more denormalization, etc.)
|
164
|
+
|
165
|
+
**What don't you give up?**
|
166
|
+
|
167
|
+
* You can still write your queries using ActiveRecord's query builder, as well as your existing models' associations & scopes.
|
168
|
+
* You can still use ActiveRecord for everything else - small queries, creating, updating, and deleting records.
|
169
|
+
* You can still inject some instance methods into your results, if you must. See below.
|
170
|
+
|
171
|
+
**Is there evidence to back any of this up?**
|
172
|
+
|
173
|
+
Glad you asked. [Look over the results yourself.](https://github.com/jhollinger/occams-record/wiki/Measurements)
|
174
|
+
|
175
|
+
**Why not use a different ORM?**
|
176
|
+
|
177
|
+
That's a great idea; check out [sequel](https://rubygems.org/gems/sequel) or [rom](https://rubygems.org/gems/rom)! But for large, legacy codebases heavily invested in ActiveRecord, switching ORMs often isn't practical. OccamsRecord can help you get some of those wins without a rewrite.
|
178
|
+
|
171
179
|
## Unsupported features
|
172
180
|
|
173
181
|
The following `ActiveRecord` are not supported, and I have no plans to do so. However, I'd be glad to accept pull requests.
|
@@ -11,6 +11,7 @@ module OccamsRecord
|
|
11
11
|
# to the ORDER BY clause to help ensure consistent batches.
|
12
12
|
#
|
13
13
|
# @param batch_size [Integer]
|
14
|
+
# @yield [OccamsRecord::Results::Row]
|
14
15
|
# @return [Enumerator] will yield each record
|
15
16
|
#
|
16
17
|
def find_each(batch_size: 1000)
|
@@ -34,6 +35,7 @@ module OccamsRecord
|
|
34
35
|
# to the ORDER BY clause to help ensure consistent batches.
|
35
36
|
#
|
36
37
|
# @param batch_size [Integer]
|
38
|
+
# @yield [OccamsRecord::Results::Row]
|
37
39
|
# @return [Enumerator] will yield each batch
|
38
40
|
#
|
39
41
|
def find_in_batches(batch_size: 1000)
|
@@ -13,21 +13,22 @@ module OccamsRecord
|
|
13
13
|
# Methods for adding eager loading to a query.
|
14
14
|
module Builder
|
15
15
|
#
|
16
|
-
# Specify an association to be eager-loaded.
|
17
|
-
#
|
18
|
-
# the colums you actually need.
|
16
|
+
# Specify an association to be eager-loaded. For maximum memory savings, only SELECT the
|
17
|
+
# colums you actually need.
|
19
18
|
#
|
20
19
|
# @param assoc [Symbol] name of association
|
21
|
-
# @param scope [Proc] a scope to apply to the query (optional)
|
20
|
+
# @param scope [Proc] a scope to apply to the query (optional). It will be passed an
|
21
|
+
# ActiveRecord::Relation on which you may call all the normal query hethods (select, where, etc) as well as any scopes you've defined on the model.
|
22
22
|
# @param select [String] a custom SELECT statement, minus the SELECT (optional)
|
23
23
|
# @param use [Array<Module>] optional Module to include in the result class (single or array)
|
24
|
-
# @param
|
24
|
+
# @param as [Symbol] Load the association usign a different attribute name
|
25
|
+
# @yield a block where you may perform eager loading on *this* association (optional)
|
25
26
|
# @return [OccamsRecord::Query] returns self
|
26
27
|
#
|
27
28
|
def eager_load(assoc, scope = nil, select: nil, use: nil, as: nil, &eval_block)
|
28
29
|
ref = @model ? @model.reflections[assoc.to_s] : nil
|
29
30
|
raise "OccamsRecord: No assocation `:#{assoc}` on `#{@model&.name || '<model missing>'}`" if ref.nil?
|
30
|
-
scope ||= -> {
|
31
|
+
scope ||= ->(q) { q.select select } if select
|
31
32
|
@eager_loaders << eager_loader_for_association(ref).new(ref, scope, use: use, as: as, &eval_block)
|
32
33
|
self
|
33
34
|
end
|
@@ -13,10 +13,11 @@ module OccamsRecord
|
|
13
13
|
|
14
14
|
#
|
15
15
|
# @param ref [ActiveRecord::Association] the ActiveRecord association
|
16
|
-
# @param scope [Proc] a scope to apply to the query (optional)
|
16
|
+
# @param scope [Proc] a scope to apply to the query (optional). It will be passed an
|
17
|
+
# ActiveRecord::Relation on which you may call all the normal query hethods (select, where, etc) as well as any scopes you've defined on the model.
|
17
18
|
# @param use [Array(Module)] optional Module to include in the result class (single or array)
|
18
|
-
# @param as [Symbol] Load the association
|
19
|
-
# @
|
19
|
+
# @param as [Symbol] Load the association usign a different attribute name
|
20
|
+
# @yield perform eager loading on *this* association (optional)
|
20
21
|
#
|
21
22
|
def initialize(ref, scope = nil, use: nil, as: nil, &eval_block)
|
22
23
|
@ref, @scope, @use, @as, @eval_block = ref, scope, use, as, eval_block
|
@@ -54,7 +55,7 @@ module OccamsRecord
|
|
54
55
|
def base_scope
|
55
56
|
q = @ref.klass.all
|
56
57
|
q = q.instance_exec(&@ref.scope) if @ref.scope
|
57
|
-
q =
|
58
|
+
q = @scope.(q) if @scope
|
58
59
|
q
|
59
60
|
end
|
60
61
|
end
|
@@ -6,6 +6,7 @@ module OccamsRecord
|
|
6
6
|
# Yield one or more ActiveRecord::Relation objects to a given block.
|
7
7
|
#
|
8
8
|
# @param rows [Array<OccamsRecord::Results::Row>] Array of rows used to calculate the query.
|
9
|
+
# @yield
|
9
10
|
#
|
10
11
|
def query(rows)
|
11
12
|
ids = rows.map { |r| r.send @ref.foreign_key }.compact.uniq
|
@@ -6,6 +6,7 @@ module OccamsRecord
|
|
6
6
|
# Yield one or more ActiveRecord::Relation objects to a given block.
|
7
7
|
#
|
8
8
|
# @param rows [Array<OccamsRecord::Results::Row>] Array of rows used to calculate the query.
|
9
|
+
# @yield
|
9
10
|
#
|
10
11
|
def query(rows)
|
11
12
|
assoc_ids = join_rows(rows).map { |row| row[1] }.compact.uniq
|
@@ -11,10 +11,11 @@ module OccamsRecord
|
|
11
11
|
|
12
12
|
#
|
13
13
|
# @param ref [ActiveRecord::Association] the ActiveRecord association
|
14
|
-
# @param scope [Proc] a scope to apply to the query (optional)
|
14
|
+
# @param scope [Proc] a scope to apply to the query (optional). It will be passed an
|
15
|
+
# ActiveRecord::Relation on which you may call all the normal query hethods (select, where, etc) as well as any scopes you've defined on the model.
|
15
16
|
# @param use [Array<Module>] optional Module to include in the result class (single or array)
|
16
|
-
# @param as [Symbol] Load the association
|
17
|
-
# @
|
17
|
+
# @param as [Symbol] Load the association usign a different attribute name
|
18
|
+
# @yield perform eager loading on *this* association (optional)
|
18
19
|
#
|
19
20
|
def initialize(ref, scope = nil, use: nil, as: nil, &eval_block)
|
20
21
|
@ref, @scope, @use, @eval_block = ref, scope, use, eval_block
|
@@ -27,6 +28,7 @@ module OccamsRecord
|
|
27
28
|
# Yield ActiveRecord::Relations to the given block, one for every "type" represented in the given rows.
|
28
29
|
#
|
29
30
|
# @param rows [Array<OccamsRecord::Results::Row>] Array of rows used to calculate the query.
|
31
|
+
# @yield
|
30
32
|
#
|
31
33
|
def query(rows)
|
32
34
|
rows_by_type = rows.group_by(&@foreign_type)
|
@@ -55,7 +57,7 @@ module OccamsRecord
|
|
55
57
|
def base_scope(model)
|
56
58
|
q = model.all
|
57
59
|
q = q.instance_exec(&@ref.scope) if @ref.scope
|
58
|
-
q =
|
60
|
+
q = @scope.(q) if @scope
|
59
61
|
q
|
60
62
|
end
|
61
63
|
end
|
data/lib/occams-record/query.rb
CHANGED
@@ -17,7 +17,7 @@ module OccamsRecord
|
|
17
17
|
# }.
|
18
18
|
# run
|
19
19
|
#
|
20
|
-
# @param
|
20
|
+
# @param scope [ActiveRecord::Relation]
|
21
21
|
# @param use [Module] optional Module to include in the result class
|
22
22
|
# @param query_logger [Array] (optional) an array into which all queries will be inserted for logging/debug purposes
|
23
23
|
# @return [OccamsRecord::Query]
|
@@ -46,7 +46,7 @@ module OccamsRecord
|
|
46
46
|
# @param use [Array<Module>] optional Module to include in the result class (single or array)
|
47
47
|
# @param query_logger [Array] (optional) an array into which all queries will be inserted for logging/debug purposes
|
48
48
|
# @param eager_loaders [OccamsRecord::EagerLoaders::Base]
|
49
|
-
# @
|
49
|
+
# @yield will be eval'd on this instance. Can be used for eager loading. (optional)
|
50
50
|
#
|
51
51
|
def initialize(scope, use: nil, query_logger: nil, eager_loaders: [], &eval_block)
|
52
52
|
@model = scope.klass
|
@@ -88,6 +88,7 @@ module OccamsRecord
|
|
88
88
|
# If you pass a block, each result row will be yielded to it. If you don't,
|
89
89
|
# an Enumerable will be returned.
|
90
90
|
#
|
91
|
+
# @yield [OccamsRecord::Results::Row]
|
91
92
|
# @return Enumerable
|
92
93
|
#
|
93
94
|
def each
|
@@ -25,7 +25,7 @@ module OccamsRecord
|
|
25
25
|
# eager_load(:category).
|
26
26
|
# run
|
27
27
|
#
|
28
|
-
# @param sql [String] The SELECT statement to run. Binds should use
|
28
|
+
# @param sql [String] The SELECT statement to run. Binds should use Ruby's named string substitution.
|
29
29
|
# @param binds [Hash] Bind values (Symbol keys)
|
30
30
|
# @param use [Array<Module>] optional Module to include in the result class (single or array)
|
31
31
|
# @param query_logger [Array] (optional) an array into which all queries will be inserted for logging/debug purposes
|
@@ -50,12 +50,11 @@ module OccamsRecord
|
|
50
50
|
#
|
51
51
|
# Initialize a new query.
|
52
52
|
#
|
53
|
-
# @param sql [String] The SELECT statement to run. Binds should use
|
53
|
+
# @param sql [String] The SELECT statement to run. Binds should use Ruby's named string substitution.
|
54
54
|
# @param binds [Hash] Bind values (Symbol keys)
|
55
55
|
# @param use [Array<Module>] optional Module to include in the result class (single or array)
|
56
56
|
# @param eager_loaders [OccamsRecord::EagerLoaders::Base]
|
57
57
|
# @param query_logger [Array] (optional) an array into which all queries will be inserted for logging/debug purposes
|
58
|
-
# @param eval_block [Proc] block that will be eval'd on this instance. Can be used for eager loading. (optional)
|
59
58
|
#
|
60
59
|
def initialize(sql, binds, use: nil, eager_loaders: [], query_logger: nil)
|
61
60
|
@sql = sql
|
@@ -110,7 +109,8 @@ module OccamsRecord
|
|
110
109
|
# If you pass a block, each result row will be yielded to it. If you don't,
|
111
110
|
# an Enumerable will be returned.
|
112
111
|
#
|
113
|
-
# @
|
112
|
+
# @yield [OccansR::Results::Row]
|
113
|
+
# @return [Enumerable]
|
114
114
|
#
|
115
115
|
def each
|
116
116
|
if block_given?
|
@@ -1,4 +1,5 @@
|
|
1
1
|
module OccamsRecord
|
2
|
+
# Classes and methods for handing query results.
|
2
3
|
module Results
|
3
4
|
# ActiveRecord's internal type casting API changes from version to version.
|
4
5
|
CASTER = case ActiveRecord::VERSION::MAJOR
|
@@ -98,6 +99,11 @@ module OccamsRecord
|
|
98
99
|
|
99
100
|
alias_method :to_hash, :to_h
|
100
101
|
|
102
|
+
#
|
103
|
+
# Returns a string with the "real" model name and raw result values.
|
104
|
+
#
|
105
|
+
# @return [String]
|
106
|
+
#
|
101
107
|
def inspect
|
102
108
|
"#<OccamsRecord::Results::Row @model_name=#{self.class.model_name} @raw_values=#{@raw_values}>"
|
103
109
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: occams-record
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.10.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Jordan Hollinger
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2018-01-
|
11
|
+
date: 2018-01-21 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activerecord
|