ar-ondemand 1.1.4 → 1.1.5
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +8 -8
- data/CHANGELOG.md +7 -0
- data/CONTRIBUTORS.md +1 -0
- data/Gemfile.lock +1 -1
- data/LICENSE.md +18 -0
- data/README.md +140 -0
- data/lib/ar-ondemand/result.rb +5 -1
- data/lib/ar-ondemand/version.rb +1 -1
- data/spec/app/models/widget.rb +3 -0
- data/spec/db/schema.rb +16 -8
- data/spec/factories/widget.rb +9 -0
- data/spec/lib/on_demand_spec.rb +47 -0
- metadata +7 -1
checksums.yaml
CHANGED
@@ -1,15 +1,15 @@
|
|
1
1
|
---
|
2
2
|
!binary "U0hBMQ==":
|
3
3
|
metadata.gz: !binary |-
|
4
|
-
|
4
|
+
NDhlMjQ2ZTkzNjRkMTc0MmJiMGM3Y2VkYjQ1NDcwZDVlYTA4ZGQzNQ==
|
5
5
|
data.tar.gz: !binary |-
|
6
|
-
|
6
|
+
Mzk5ZDY1OTdmZjg5YzEzZmI2YTNkNDNhMGI0MTliYzQ2Nzc0ZjU1Mw==
|
7
7
|
SHA512:
|
8
8
|
metadata.gz: !binary |-
|
9
|
-
|
10
|
-
|
11
|
-
|
9
|
+
OTRlYjk5NzhhZjU4MmVkOGViMGFjOTkyYWY2MjQ2ZGUxMmE3ZTFlMzE1NWZm
|
10
|
+
YTQ1ZGFhZWI0NDZlYmUwMDUzYjA0OWIwMGJjMzM1OWRkMTcwNDliODIwZWVl
|
11
|
+
NzZhN2QzMDc1MWMwY2UxMWI0ZTNhNThjMTFhODBkZjk4MTVjZWI=
|
12
12
|
data.tar.gz: !binary |-
|
13
|
-
|
14
|
-
|
15
|
-
|
13
|
+
ZDRmOTExOTU0Y2E0ZmI0YjQ2YmUxZGVkOGVhMWU5YTlmNDU4N2FkM2JlYTVj
|
14
|
+
N2Q0MWU1ZDI2NzViNjY0YTU2ZTU5ZTE2NzAyMmEyNGMzZjE5NTZiNjAyNDZh
|
15
|
+
NmI1YjFkZWJiZDQ2YTdkNDY2OGQyMDNmOGQwMjYyYjk1YmI0ZjY=
|
data/CHANGELOG.md
ADDED
data/CONTRIBUTORS.md
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
* Steve Frank <lardcanoe@gmail.com>
|
data/Gemfile.lock
CHANGED
data/LICENSE.md
ADDED
@@ -0,0 +1,18 @@
|
|
1
|
+
The MIT License (MIT)
|
2
|
+
|
3
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy of
|
4
|
+
this software and associated documentation files (the "Software"), to deal in
|
5
|
+
the Software without restriction, including without limitation the rights to
|
6
|
+
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
|
7
|
+
the Software, and to permit persons to whom the Software is furnished to do so,
|
8
|
+
subject to the following conditions:
|
9
|
+
|
10
|
+
The above copyright notice and this permission notice shall be included in all
|
11
|
+
copies or substantial portions of the Software.
|
12
|
+
|
13
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
14
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
15
|
+
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
|
16
|
+
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
|
17
|
+
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
|
18
|
+
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
CHANGED
@@ -1,4 +1,144 @@
|
|
1
1
|
# ar-ondemand
|
2
2
|
|
3
|
+
The `ar-ondemand` gem adds functionality to ActiveRecord to help deal with AR's bloat.
|
4
|
+
|
3
5
|
[![Gem Version](https://badge.fury.io/rb/ar-ondemand.svg)](http://badge.fury.io/rb/ar-ondemand)
|
4
6
|
[![](https://ci.solanolabs.com:443/cloudhealthtech/ar-ondemand/badges/170027.png?badge_token=bd73a19d5421a68f29e22ad15ad080cbabc56ba7)](https://ci.solanolabs.com:443/cloudhealthtech/ar-ondemand/suites/170027)
|
7
|
+
|
8
|
+
# Getting Started
|
9
|
+
|
10
|
+
```
|
11
|
+
require 'ar-ondemand'
|
12
|
+
```
|
13
|
+
|
14
|
+
Please note that this library has been written for our needs, and even though it has gotten significant usage in
|
15
|
+
production environments, it hasn't seen the myriad ways others use and abuse ActiveRecord, so please experiment
|
16
|
+
locally first.
|
17
|
+
|
18
|
+
It has been used with ActiveRecord 3.2, MRI 1.9.3, JRuby 1.7 and the MySQL adapter.
|
19
|
+
|
20
|
+
# Functionality
|
21
|
+
|
22
|
+
## on_demand
|
23
|
+
|
24
|
+
This was the original impetus for the gem. The issue was that we had to compare ~500k records between the source
|
25
|
+
dataset and our database, and we had no idea which records were new, changed or deleted. We'd preload everything from
|
26
|
+
the database, but due to ActiveRecord's massive bloat, we'd constantly run into OOM exceptions. To get around that,
|
27
|
+
the concept of a [lightweight ActiveRecord object](https://github.com/CloudHealth/ar-ondemand/blob/master/lib/ar-ondemand/record.rb)
|
28
|
+
was introduced that had the absolute bare minimum needed to handle comparing the source data with the database.
|
29
|
+
|
30
|
+
With this new type of object, we could easily interate over the 500k records extremely quickly and determine what has
|
31
|
+
changed. The on-demand aspect comes when `.save` is called. If changes were noticed, it
|
32
|
+
[secretly](https://github.com/CloudHealth/ar-ondemand/blob/master/lib/ar-ondemand/record.rb#L67) instantiates an actual
|
33
|
+
ActiveRecord model so that all the real functionality you'd expect, such as validation and callbacks, occurs.
|
34
|
+
|
35
|
+
### Usage
|
36
|
+
|
37
|
+
```
|
38
|
+
assets = Widget.on_demand :identifier, {customer_id: 1, account_id: 42}
|
39
|
+
source.each do |dso|
|
40
|
+
w = assets[dso[:identifier]]
|
41
|
+
w.name = dso[:name]
|
42
|
+
w.foo = dso[:foo]
|
43
|
+
w.bar = dso[:bar]
|
44
|
+
ar_obj = w.save
|
45
|
+
# If new or changed, ar_obj will be an ActiveRecord instance, and ar_obj.id will now be set
|
46
|
+
end
|
47
|
+
```
|
48
|
+
|
49
|
+
## for_reading
|
50
|
+
|
51
|
+
How often do you just need to load a bunch of objects and read some properties? Pretty often, right? Now, have you ever
|
52
|
+
looked at how much memory ActiveRecord itself is consuming, as well as how much extra time it takes to create all the
|
53
|
+
instances of the objects compared to how long it took to extract from the database? It's bonkers! The `for_reading`
|
54
|
+
method makes it easy to get access to ActiveRecord-like functionality at 100th the cost.
|
55
|
+
|
56
|
+
### Usage
|
57
|
+
|
58
|
+
Let's say you have some Widget's. Instead of:
|
59
|
+
|
60
|
+
```
|
61
|
+
Widget.where(customer_id: 1).each { |r| ... }
|
62
|
+
```
|
63
|
+
|
64
|
+
use `for_reading` and get a significant speed boost, and use far less memory:
|
65
|
+
|
66
|
+
```
|
67
|
+
Widget.where(customer_id: 1).for_reading.each { |r| ... }
|
68
|
+
```
|
69
|
+
|
70
|
+
The big limitation is that you can't use `.includes` so you have to be writing a query for just that class.
|
71
|
+
|
72
|
+
### Batch Results
|
73
|
+
|
74
|
+
```
|
75
|
+
Widget.where(customer_id: 1).for_reading(batch_size: 50000).each { |b| b.each { |r| } }
|
76
|
+
```
|
77
|
+
|
78
|
+
### raw_results
|
79
|
+
|
80
|
+
This is just a nice little helper to get the raw database results, which you'd get by calling `ActiveRecord::Base.connection.select_all`
|
81
|
+
but for some reason you can't call that on a model.
|
82
|
+
|
83
|
+
```
|
84
|
+
ActiveRecord::Base.connection.select_all "select * from widgets"
|
85
|
+
ActiveRecord::Base.connection.select_all "select * from widgets where customer_id = 1"
|
86
|
+
ActiveRecord::Base.connection.select_all "select * from widgets where customer_id = 1 limit 100000"
|
87
|
+
```
|
88
|
+
|
89
|
+
use:
|
90
|
+
|
91
|
+
```
|
92
|
+
Widget.raw_results
|
93
|
+
Widget.where(customer_id: 1).raw_results
|
94
|
+
Widget.where(customer_id: 1).limit(100_000).raw_results
|
95
|
+
```
|
96
|
+
|
97
|
+
## for_streaming
|
98
|
+
|
99
|
+
Another use case was needing to iterate through 30,000,000 records. The code that needed this data was not setup to work
|
100
|
+
with batching, so the concept of streaming results was introduced. It simply uses an Enumerator to hide the fact that
|
101
|
+
batching is actually happening behind the scenes. The downside is that Enumerator made it ~10% slower. If speed matters,
|
102
|
+
change your code to use batching.
|
103
|
+
|
104
|
+
### Usage
|
105
|
+
|
106
|
+
```
|
107
|
+
def run
|
108
|
+
get_objects.each do |r|
|
109
|
+
do_something r
|
110
|
+
end
|
111
|
+
end
|
112
|
+
|
113
|
+
def get_objects
|
114
|
+
Widget.where(customer_id: 1).for_streaming
|
115
|
+
end
|
116
|
+
```
|
117
|
+
|
118
|
+
Additional usage:
|
119
|
+
|
120
|
+
```
|
121
|
+
Widget.where(customer_id: 1).for_streaming(batch_size: 100_000).each { |r| }
|
122
|
+
Widget.where(customer_id: 1).for_streaming(for_reading: true).each { |r| }
|
123
|
+
Widget.where(customer_id: 1).for_streaming(for_reading: true, batch_size: 1_000_000).each { |r| }
|
124
|
+
```
|
125
|
+
|
126
|
+
## delete_all_by_pk
|
127
|
+
|
128
|
+
Deleting many records or even a few records in a massive table can be an expensive operation, and can even lock up
|
129
|
+
your table during the duration of the delete, as well as perform a complete table scan. A common pattern to deal with
|
130
|
+
this is querying the table first to find the primary keys that meet the criteria and then doing a delete specifying
|
131
|
+
the primary keys as the `where` condition. This function does that all for you.
|
132
|
+
|
133
|
+
### Usage
|
134
|
+
|
135
|
+
```
|
136
|
+
Widget.delete_all_by_pk
|
137
|
+
Widget.where(Widget[:customer_id].eq(1).and(Widget[:usage].gt(42))).delete_all_by_pk
|
138
|
+
```
|
139
|
+
|
140
|
+
If you know you could be deleting millions, then we recommend batching the deletes:
|
141
|
+
|
142
|
+
```
|
143
|
+
Widget.where(Widget[:customer_id].eq(1).and(Widget[:usage].gt(42))).delete_all_by_pk(batch_size: 250_000)
|
144
|
+
```
|
data/lib/ar-ondemand/result.rb
CHANGED
@@ -60,7 +60,11 @@ module ActiveRecord
|
|
60
60
|
# TODO: Is using HashWithIndifferentAccess[] more efficient?
|
61
61
|
h = {}
|
62
62
|
@col_indexes.each_pair do |k, v|
|
63
|
-
|
63
|
+
if @column_types[k]
|
64
|
+
h[k] = @column_types[k].type_cast rec[v]
|
65
|
+
else
|
66
|
+
h[k] = rec[v]
|
67
|
+
end
|
64
68
|
end
|
65
69
|
h.with_indifferent_access
|
66
70
|
end
|
data/lib/ar-ondemand/version.rb
CHANGED
data/spec/db/schema.rb
CHANGED
@@ -1,11 +1,19 @@
|
|
1
1
|
ActiveRecord::Schema.define do
|
2
|
-
create_table
|
3
|
-
t.integer
|
4
|
-
t.integer
|
5
|
-
t.integer
|
6
|
-
t.string
|
7
|
-
t.text
|
8
|
-
t.datetime
|
9
|
-
t.datetime
|
2
|
+
create_table 'audit_records', force: true do |t|
|
3
|
+
t.integer 'customer_id'
|
4
|
+
t.integer 'model_id'
|
5
|
+
t.integer 'model_type_id'
|
6
|
+
t.string 'action', limit: 6
|
7
|
+
t.text 'description'
|
8
|
+
t.datetime 'created_at', null: false
|
9
|
+
t.datetime 'updated_at', null: false
|
10
|
+
end
|
11
|
+
|
12
|
+
create_table 'widgets', force: true do |t|
|
13
|
+
t.integer 'customer_id'
|
14
|
+
t.text 'identifier'
|
15
|
+
t.text 'description'
|
16
|
+
t.datetime 'created_at', null: false
|
17
|
+
t.datetime 'updated_at', null: false
|
10
18
|
end
|
11
19
|
end
|
@@ -0,0 +1,47 @@
|
|
1
|
+
require 'spec_helper'
|
2
|
+
require 'ar-ondemand'
|
3
|
+
|
4
|
+
describe 'OnDemand' do
|
5
|
+
before(:each) do
|
6
|
+
(1..25).each do
|
7
|
+
create(:widget)
|
8
|
+
end
|
9
|
+
end
|
10
|
+
|
11
|
+
context 'Existing' do
|
12
|
+
it 'should get asset' do
|
13
|
+
assets = ::Widget.where(customer_id: 1).on_demand :identifier, customer_id: 1
|
14
|
+
ids = assets.ids
|
15
|
+
expect(ids.length).to eql(25)
|
16
|
+
|
17
|
+
x = assets['w-1']
|
18
|
+
expect(x.identifier).to eql('w-1')
|
19
|
+
expect(x.id).to eql(1)
|
20
|
+
expect(x).to be_an_instance_of(::ActiveRecord::OnDemand::Record)
|
21
|
+
end
|
22
|
+
end
|
23
|
+
|
24
|
+
context 'Update Existing' do
|
25
|
+
it 'saving should get instance of a Widget' do
|
26
|
+
assets = ::Widget.where(customer_id: 1).on_demand :identifier, customer_id: 1
|
27
|
+
x = assets['w-26']
|
28
|
+
expect(x.identifier).to eql('w-26')
|
29
|
+
expect(x).to be_an_instance_of(::ActiveRecord::OnDemand::Record)
|
30
|
+
|
31
|
+
x.customer_id = 2
|
32
|
+
obj = x.save
|
33
|
+
expect(obj).to be_an_instance_of(::Widget)
|
34
|
+
end
|
35
|
+
end
|
36
|
+
|
37
|
+
context 'Create' do
|
38
|
+
it 'should create asset' do
|
39
|
+
assets = ::Widget.where(customer_id: 1).on_demand :identifier, customer_id: 1
|
40
|
+
x = assets['w-999']
|
41
|
+
expect(x.identifier).to eql('w-999')
|
42
|
+
expect(x.id).to eql(nil)
|
43
|
+
expect(x.persisted?).to eql(false)
|
44
|
+
expect(x).to be_an_instance_of(::Widget)
|
45
|
+
end
|
46
|
+
end
|
47
|
+
end
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: ar-ondemand
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.1.
|
4
|
+
version: 1.1.5
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Steve Frank
|
@@ -48,8 +48,11 @@ extensions: []
|
|
48
48
|
extra_rdoc_files: []
|
49
49
|
files:
|
50
50
|
- .gitignore
|
51
|
+
- CHANGELOG.md
|
52
|
+
- CONTRIBUTORS.md
|
51
53
|
- Gemfile
|
52
54
|
- Gemfile.lock
|
55
|
+
- LICENSE.md
|
53
56
|
- README.md
|
54
57
|
- ar-ondemand.gemspec
|
55
58
|
- lib/ar-ondemand.rb
|
@@ -62,12 +65,15 @@ files:
|
|
62
65
|
- lib/ar-ondemand/version.rb
|
63
66
|
- solano.yml
|
64
67
|
- spec/app/models/audit_record.rb
|
68
|
+
- spec/app/models/widget.rb
|
65
69
|
- spec/db/schema.rb
|
66
70
|
- spec/db/seeds.rb
|
67
71
|
- spec/factories/audit_record.rb
|
72
|
+
- spec/factories/widget.rb
|
68
73
|
- spec/lib/delete_all_by_pk_spec.rb
|
69
74
|
- spec/lib/for_reading_spec.rb
|
70
75
|
- spec/lib/for_streaming_spec.rb
|
76
|
+
- spec/lib/on_demand_spec.rb
|
71
77
|
- spec/spec_helper.rb
|
72
78
|
- spec/support/active_record.rb
|
73
79
|
homepage: https://github.com/CloudHealth/ar-ondemand
|