yam-db-charmer 1.7.01
Sign up to get free protection for your applications and to get access to all the features.
- data/.gitignore +4 -0
- data/CHANGES +184 -0
- data/LICENSE +21 -0
- data/Makefile +2 -0
- data/README.rdoc +612 -0
- data/Rakefile +4 -0
- data/db-charmer.gemspec +29 -0
- data/init.rb +1 -0
- data/lib/db_charmer/action_controller/force_slave_reads.rb +69 -0
- data/lib/db_charmer/active_record/association_preload.rb +23 -0
- data/lib/db_charmer/active_record/class_attributes.rb +101 -0
- data/lib/db_charmer/active_record/connection_switching.rb +81 -0
- data/lib/db_charmer/active_record/db_magic.rb +85 -0
- data/lib/db_charmer/active_record/migration/multi_db_migrations.rb +71 -0
- data/lib/db_charmer/active_record/multi_db_proxy.rb +77 -0
- data/lib/db_charmer/active_record/sharding.rb +40 -0
- data/lib/db_charmer/connection_factory.rb +76 -0
- data/lib/db_charmer/connection_proxy.rb +27 -0
- data/lib/db_charmer/core_extensions.rb +23 -0
- data/lib/db_charmer/force_slave_reads.rb +36 -0
- data/lib/db_charmer/rails2/abstract_adapter/log_formatting.rb +24 -0
- data/lib/db_charmer/rails2/active_record/master_slave_routing.rb +49 -0
- data/lib/db_charmer/rails2/active_record/named_scope/scope_proxy.rb +26 -0
- data/lib/db_charmer/rails3/abstract_adapter/connection_name.rb +38 -0
- data/lib/db_charmer/rails3/active_record/log_subscriber.rb +23 -0
- data/lib/db_charmer/rails3/active_record/master_slave_routing.rb +46 -0
- data/lib/db_charmer/rails3/active_record/relation/connection_routing.rb +147 -0
- data/lib/db_charmer/rails3/active_record/relation_method.rb +28 -0
- data/lib/db_charmer/sharding/connection.rb +31 -0
- data/lib/db_charmer/sharding/method/db_block_group_map.rb +257 -0
- data/lib/db_charmer/sharding/method/db_block_map.rb +211 -0
- data/lib/db_charmer/sharding/method/hash_map.rb +23 -0
- data/lib/db_charmer/sharding/method/range.rb +33 -0
- data/lib/db_charmer/sharding/method.rb +10 -0
- data/lib/db_charmer/sharding/stub_connection.rb +60 -0
- data/lib/db_charmer/sharding.rb +18 -0
- data/lib/db_charmer/version.rb +10 -0
- data/lib/db_charmer.rb +192 -0
- data/lib/tasks/databases.rake +82 -0
- metadata +178 -0
data/.gitignore
ADDED
data/CHANGES
ADDED
@@ -0,0 +1,184 @@
|
|
1
|
+
1.7.0 (2011-08-29):
|
2
|
+
|
3
|
+
Beta feature: Rails 3 support
|
4
|
+
|
5
|
+
Beta feature: Added force_slave_reads functionality. Now we could have models with slaves
|
6
|
+
that are not used by default, but could be turned on globally (per-controller or per-action).
|
7
|
+
|
8
|
+
Heavily reorganized the source code to match Rails code structure (class names, etc). This should
|
9
|
+
make it much easier for other contributors to work with the code.
|
10
|
+
Added smarter environment detection (using Rails.env, RAILS_ENV or RACK_ENV).
|
11
|
+
|
12
|
+
Changed dependencies a bit: instead of depending on rails, we now require specific components
|
13
|
+
(ActiveRecord, ActiveSupport, etc) + we do not require blankslate gem anymore.
|
14
|
+
|
15
|
+
Bugfixes: Fix for N+1 queries when accessing shard_info for db_block_group_map sharding method.
|
16
|
+
|
17
|
+
----------------------------------------------------------------------------------------
|
18
|
+
1.6.17-19 (2011-04-25):
|
19
|
+
|
20
|
+
Bugfixes: Do not touch database for sharded models until we really need to. Before 1.6.17
|
21
|
+
if a database server was dead and there were any other connection issues, class loading
|
22
|
+
in Ruby would be broken and sharded model class would not be initialized.
|
23
|
+
|
24
|
+
----------------------------------------------------------------------------------------
|
25
|
+
1.6.14 (2011-01-09):
|
26
|
+
|
27
|
+
Bugfixes: We do not support Rails 3, and now we prohibit any versions but 2.2 and 2.3 from
|
28
|
+
being used with db-charmer gem.
|
29
|
+
|
30
|
+
----------------------------------------------------------------------------------------
|
31
|
+
1.6.13 (2010-08-17):
|
32
|
+
|
33
|
+
Starting with this version we use Rails.env instead of RAILS_ENV to auto-detect rails
|
34
|
+
environment. If you use DbCharmer in non-rails project, please set DbCharmer.env manually.
|
35
|
+
|
36
|
+
Bugfixes: Thanks to Eric Lindvall we now allow connection names that have symbols ruby
|
37
|
+
wouldn't like for class names.
|
38
|
+
|
39
|
+
----------------------------------------------------------------------------------------
|
40
|
+
1.6.12 (2010-05-09):
|
41
|
+
|
42
|
+
Starting with this version we use Rails.cache (memcache or whatever you use in your project)
|
43
|
+
to cache sharding blocks information.
|
44
|
+
|
45
|
+
Bugfixes: Thanks to Allen Madsen (github user blatyo) we've fixed a few minor issues in
|
46
|
+
database connections handling.
|
47
|
+
|
48
|
+
----------------------------------------------------------------------------------------
|
49
|
+
1.6.11 (2010-04-16):
|
50
|
+
|
51
|
+
Bugfix: Change the way we allocate sharding blocks in block map sharding method to
|
52
|
+
prevent race-conditions from happening on block to shard assignments.
|
53
|
+
Breaking change: We require connections to exist by default in all connection factory
|
54
|
+
methods. If you need old behavior, pass should_exist=false explicitly.
|
55
|
+
|
56
|
+
----------------------------------------------------------------------------------------
|
57
|
+
1.6.10 (2010-04-09):
|
58
|
+
|
59
|
+
Multi-Db migrations changed. Now it is possible to call ActiveRecord::Migration.db_magic
|
60
|
+
and specify default migration connection that would be used by all migrations without
|
61
|
+
excplicitly switched connections.
|
62
|
+
|
63
|
+
----------------------------------------------------------------------------------------
|
64
|
+
1.6.9 (2010-04-08):
|
65
|
+
|
66
|
+
Bugfix release: now DbCharmer works without Rails.
|
67
|
+
|
68
|
+
----------------------------------------------------------------------------------------
|
69
|
+
1.6.7 (2010-04-07):
|
70
|
+
|
71
|
+
Changed the way we handle associations in on_db(:foo).find(:include) calls. Now we
|
72
|
+
switch association's connection only if its default connection is the same as the
|
73
|
+
master model's connection (not more "table does not exist" problems I hope).
|
74
|
+
|
75
|
+
----------------------------------------------------------------------------------------
|
76
|
+
1.6.5 (2010-04-05):
|
77
|
+
|
78
|
+
Bugfix release: Fixed :connection vs :slave in db_magic behaviour. Model.on_master should
|
79
|
+
run queries on the master, not on AR's default connection.
|
80
|
+
|
81
|
+
----------------------------------------------------------------------------------------
|
82
|
+
1.6.4 (2010-04-05):
|
83
|
+
|
84
|
+
Default behaviour changed: DbCharmer.connections_should_exist is true in all environments
|
85
|
+
by default. Old default behaviour was too misleading for many developers.
|
86
|
+
|
87
|
+
----------------------------------------------------------------------------------------
|
88
|
+
1.6.3 (2010-04-03):
|
89
|
+
|
90
|
+
Bugfix release: Modified stub connection initialization code to set default connections
|
91
|
+
for sharded models using shards enumeration or default shard features of sharding methods.
|
92
|
+
|
93
|
+
----------------------------------------------------------------------------------------
|
94
|
+
1.6.2 (2010-04-03):
|
95
|
+
|
96
|
+
Bugfix release: Modified our stub connection used on sharded models to fail on db-calling
|
97
|
+
methods only. Proxy the rest to a real shard connection. Another bug fixed in db_block_map
|
98
|
+
sharding method: we didn't increment block counters when assigning blocks to shards.
|
99
|
+
|
100
|
+
----------------------------------------------------------------------------------------
|
101
|
+
1.6.1 (2010-03-31):
|
102
|
+
|
103
|
+
Breaking change from now on all connection-switching methods (both in migrations and in
|
104
|
+
models) are controlled by a single option DbCharmer.connections_should_exist. This
|
105
|
+
option is false by default in all non-production environments. Check out README for
|
106
|
+
more details.
|
107
|
+
|
108
|
+
----------------------------------------------------------------------------------------
|
109
|
+
1.6.0 (2010-03-31):
|
110
|
+
|
111
|
+
The major (and arguably the only noticeable) change in this version is our simple database
|
112
|
+
sharding support. The feature is still in alpha stage and should not be used in production
|
113
|
+
without complete understanding of the principles of its work.
|
114
|
+
|
115
|
+
----------------------------------------------------------------------------------------
|
116
|
+
1.5.5 (2010-03-15):
|
117
|
+
|
118
|
+
Thanks to ngmoco.com (http://github.com/ngmoco) now DbCharmer supports one more use-case
|
119
|
+
for multi-db migrations. Now you can run the same migration on many databases at once.
|
120
|
+
For example, the following migration would create test_table on all three shard databases:
|
121
|
+
|
122
|
+
class MultiDbTest < ActiveRecord::Migration
|
123
|
+
db_magic :connections => [ :shard01, :shard02, :shard03 ]
|
124
|
+
|
125
|
+
def self.up
|
126
|
+
create_table :test_table do |t|
|
127
|
+
t.string :test_string
|
128
|
+
t.timestamps
|
129
|
+
end
|
130
|
+
end
|
131
|
+
|
132
|
+
def self.down
|
133
|
+
drop_table :test_table
|
134
|
+
end
|
135
|
+
end
|
136
|
+
|
137
|
+
----------------------------------------------------------------------------------------
|
138
|
+
1.5.4 (2010-03-12):
|
139
|
+
|
140
|
+
Added DbCharmer.with_remapped_databases, so that you can change the connection for
|
141
|
+
many models simultaneously, and implicitly. Very useful for work where you want to use
|
142
|
+
a particular slave for a whole range of database access.
|
143
|
+
|
144
|
+
----------------------------------------------------------------------------------------
|
145
|
+
1.5.3 (2010-03-10):
|
146
|
+
|
147
|
+
Few changes:
|
148
|
+
* Colorized connection names in the logs for development mode
|
149
|
+
* We do not log connection names when connection does not exist
|
150
|
+
|
151
|
+
----------------------------------------------------------------------------------------
|
152
|
+
1.5.1 (2010-03-06):
|
153
|
+
|
154
|
+
In this version we've added support for connection names logging in Rails queries log.
|
155
|
+
New log records have [connection_name] prefix for all queries that are executed on
|
156
|
+
non-standard connections:
|
157
|
+
|
158
|
+
[logs] LogRecord Columns (1.1ms) SHOW FIELDS FROM `log_records`
|
159
|
+
[logs] User Delete all (0.1ms) DELETE FROM `users`
|
160
|
+
[slave01] User Load (0.2ms) SELECT * FROM `users` WHERE (`users`.`login` = 'foo')
|
161
|
+
|
162
|
+
----------------------------------------------------------------------------------------
|
163
|
+
1.4.6 -> 1.5.0 (2010-03-05):
|
164
|
+
|
165
|
+
Major change in this version of DbCharmer is association preload support. For example,
|
166
|
+
let's say we have a schema:
|
167
|
+
|
168
|
+
class Post < ActiveRecord::Base
|
169
|
+
belongs_to :user
|
170
|
+
end
|
171
|
+
|
172
|
+
class User < ActiveRecord::Base
|
173
|
+
has_many :posts
|
174
|
+
end
|
175
|
+
|
176
|
+
Now, if we have the following call in our code:
|
177
|
+
|
178
|
+
User.on_db(:foo).all(:include => :posts)
|
179
|
+
|
180
|
+
In 1.4.6 it would load the users from connection :foo and posts from the
|
181
|
+
default connection, which is not what we would expect from this line of code.
|
182
|
+
So, starting 1.5.0 all finder calls on models having :include parameter would
|
183
|
+
switch associated models' connections to the same connection as the main model
|
184
|
+
in the call.
|
data/LICENSE
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
The MIT License
|
2
|
+
|
3
|
+
Copyright (c) 2011, Oleksiy Kovyrin
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
7
|
+
in the Software without restriction, including without limitation the rights
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
10
|
+
furnished to do so, subject to the following conditions:
|
11
|
+
|
12
|
+
The above copyright notice and this permission notice shall be included in
|
13
|
+
all copies or substantial portions of the Software.
|
14
|
+
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
|
21
|
+
THE SOFTWARE.
|
data/Makefile
ADDED
data/README.rdoc
ADDED
@@ -0,0 +1,612 @@
|
|
1
|
+
= DB Charmer - ActiveRecord Connection Magic Plugin
|
2
|
+
|
3
|
+
+DbCharmer+ is a simple yet powerful plugin for ActiveRecord that significantly extends its ability to work with
|
4
|
+
multiple databases and/or database servers. The major features we add to ActiveRecord are:
|
5
|
+
|
6
|
+
1. Simple management for AR model connections (+switch_connection_to+ method)
|
7
|
+
2. Switching of default AR model connections to separate servers/databases
|
8
|
+
3. Ability to easily choose where your query should go (<tt>Model.on_*</tt> methods family)
|
9
|
+
4. Automated master/slave queries routing (selects go to a slave, updates handled by the master).
|
10
|
+
5. Multiple database migrations with very flexible query routing controls.
|
11
|
+
6. Simple database sharding functionality with multiple sharding methods (value, range, mapping table).
|
12
|
+
|
13
|
+
For more information on the project, you can check out our web site at http://kovyrin.github.com/db-charmer.
|
14
|
+
|
15
|
+
|
16
|
+
== Installation
|
17
|
+
|
18
|
+
There are two options when approaching +DbCharmer+ installation:
|
19
|
+
* using the gem (recommended and the only way of using it with Rails 3.0+)
|
20
|
+
* install as a Rails plugin (works in Rails 2.x only)
|
21
|
+
|
22
|
+
To install as a gem, add this to your Gemfile:
|
23
|
+
|
24
|
+
gem 'db-charmer', :require => 'db_charmer'
|
25
|
+
|
26
|
+
To install +DbCharmer+ as a Rails plugin use the following command:
|
27
|
+
|
28
|
+
./script/plugin install git://github.com/kovyrin/db-charmer.git
|
29
|
+
|
30
|
+
_Notice_: If you use +DbCharmer+ in a non-rails project, you may need to set <tt>DbCharmer.env</tt> to a correct value
|
31
|
+
before using any of its connection management methods. Correct value here is a valid <tt>database.yml</tt>
|
32
|
+
first-level section name.
|
33
|
+
|
34
|
+
|
35
|
+
== Easy ActiveRecord Connection Management
|
36
|
+
|
37
|
+
As a part of this plugin we've added +switch_connection_to+ method that accepts many different kinds
|
38
|
+
of db connections specifications and uses them on a model. We support:
|
39
|
+
|
40
|
+
1. Strings and symbols as the names of connection configuration blocks in database.yml.
|
41
|
+
2. ActiveRecord models (we'd use connection currently set up on a model).
|
42
|
+
3. Database connections (<tt>Model.connection</tt>)
|
43
|
+
4. Nil values to reset model to default connection.
|
44
|
+
|
45
|
+
Sample code:
|
46
|
+
|
47
|
+
class Foo < ActiveRecord::Model; end
|
48
|
+
|
49
|
+
Foo.switch_connection_to(:blah)
|
50
|
+
Foo.switch_connection_to('foo')
|
51
|
+
Foo.switch_connection_to(Bar)
|
52
|
+
Foo.switch_connection_to(Baz.connection)
|
53
|
+
Foo.switch_connection_to(nil)
|
54
|
+
|
55
|
+
Sample <tt>database.yml</tt> configuration:
|
56
|
+
|
57
|
+
production:
|
58
|
+
blah:
|
59
|
+
adapter: mysql
|
60
|
+
username: blah
|
61
|
+
host: blah.local
|
62
|
+
database: blah
|
63
|
+
|
64
|
+
foo:
|
65
|
+
adapter: mysql
|
66
|
+
username: foo
|
67
|
+
host: foo.local
|
68
|
+
database: foo
|
69
|
+
|
70
|
+
The +switch_connection_to+ method has an optional second parameter +should_exist+ which is true
|
71
|
+
by default. This parameter is used when the method is called with a string or a symbol connection
|
72
|
+
name and there is no such connection configuration in the database.yml file. If this parameter
|
73
|
+
is +true+, an exception would be raised, otherwise, the error would be ignored and no connection
|
74
|
+
change would happen.
|
75
|
+
|
76
|
+
This is really useful when in development mode or in a tests you do not want to create many different
|
77
|
+
databases on your local machine and just want to put all your tables in a single database.
|
78
|
+
|
79
|
+
*Warning*: All the connection switching calls would switch connection *only* for those classes the
|
80
|
+
method called on. You can't call the +switch_connection_to+ method and switch connection for a
|
81
|
+
base class in some hierarchy (for example, you can't switch AR::Base connection and see all your
|
82
|
+
models switched to the new connection, use the classic +establish_connection+ instead).
|
83
|
+
|
84
|
+
|
85
|
+
== Multiple DB Migrations
|
86
|
+
|
87
|
+
In every application that works with many databases, there is need in a convenient schema migrations mechanism.
|
88
|
+
|
89
|
+
All Rails users already have this mechanism - rails migrations. So in +DbCharmer+, we've made it possible
|
90
|
+
to seamlessly use multiple databases in Rails migrations.
|
91
|
+
|
92
|
+
There are two methods available in migrations to operate on more than one database:
|
93
|
+
|
94
|
+
1. Global connection change method - used to switch whole migration to a non-default database.
|
95
|
+
2. Block-level connection change method - could be used to do only a part of a migration on a non-default db.
|
96
|
+
|
97
|
+
Migration class example (global connection rewrite):
|
98
|
+
|
99
|
+
class MultiDbTest < ActiveRecord::Migration
|
100
|
+
db_magic :connection => :second_db
|
101
|
+
|
102
|
+
def self.up
|
103
|
+
create_table :test_table, :force => true do |t|
|
104
|
+
t.string :test_string
|
105
|
+
t.timestamps
|
106
|
+
end
|
107
|
+
end
|
108
|
+
|
109
|
+
def self.down
|
110
|
+
drop_table :test_table
|
111
|
+
end
|
112
|
+
end
|
113
|
+
|
114
|
+
Migration class example (block-level connection rewrite):
|
115
|
+
|
116
|
+
class MultiDbTest < ActiveRecord::Migration
|
117
|
+
def self.up
|
118
|
+
on_db :second_db do
|
119
|
+
create_table :test_table, :force => true do |t|
|
120
|
+
t.string :test_string
|
121
|
+
t.timestamps
|
122
|
+
end
|
123
|
+
end
|
124
|
+
end
|
125
|
+
|
126
|
+
def self.down
|
127
|
+
on_db :second_db { drop_table :test_table }
|
128
|
+
end
|
129
|
+
end
|
130
|
+
|
131
|
+
Migration class example (global connection rewrite, multiple connections with the same table):
|
132
|
+
(NOTE: both :connection and :connections can take an array of connections)
|
133
|
+
|
134
|
+
class MultiDbTest < ActiveRecord::Migration
|
135
|
+
db_magic :connections => [:second_db, :default]
|
136
|
+
|
137
|
+
def self.up
|
138
|
+
create_table :test_table, :force => true do |t|
|
139
|
+
t.string :test_string
|
140
|
+
t.timestamps
|
141
|
+
end
|
142
|
+
end
|
143
|
+
|
144
|
+
def self.down
|
145
|
+
drop_table :test_table
|
146
|
+
end
|
147
|
+
end
|
148
|
+
|
149
|
+
=== Default Migrations Connection
|
150
|
+
|
151
|
+
Starting with DbCharmer version 1.6.10 it is possible to call <tt>ActiveRecord::Migration.db_magic</tt>
|
152
|
+
and specify default migration connection that would be used by all migrations without
|
153
|
+
excplicitly switched connections. If you want to switch your migration to the default ActiveRecord
|
154
|
+
connection, just use <tt>db_magic :connection => :default</tt>.
|
155
|
+
|
156
|
+
=== Invalid Connection Names Handling
|
157
|
+
|
158
|
+
By default in all environments <tt>on_db</tt> and <tt>db_magic</tt> statments would fail if
|
159
|
+
specified connection does not exist in database.yml. It is possible to make +DbCharmer+
|
160
|
+
ignore such situations in non-production environments so that rails would create the tables
|
161
|
+
in your single database (especially useful in test databases).
|
162
|
+
|
163
|
+
This behaviour is controlled by the <tt>DbCharmer.connections_should_exist</tt>
|
164
|
+
configuration attribute which could be set from a rails initializer.
|
165
|
+
|
166
|
+
Warning: if in test environment you use separate connections and master-slave support
|
167
|
+
in DbCharmer, make sure you disable transactional fixtures support in Rails. Without
|
168
|
+
this change you're going to see all kinds of weird data visibility problems in your tests.
|
169
|
+
|
170
|
+
|
171
|
+
== Using Models in Master-Slave Environments
|
172
|
+
|
173
|
+
Master-slave replication is the most popular scale-out technique in a medium-sized and
|
174
|
+
large database-centric applications today. There are some rails plugins out there that help
|
175
|
+
developers to use slave servers in their models but none of them were flexible enough
|
176
|
+
for us to start using them in a huge application we work on.
|
177
|
+
|
178
|
+
So, we've been using ActsAsReadonlyable plugin for a long time and have made tons
|
179
|
+
of changes in its code over the time. But since that plugin has been abandoned
|
180
|
+
by its authors, we've decided to collect all of our master-slave code in one plugin
|
181
|
+
and release it for rails 2.2+. +DbCharmer+ adds the following features to Rails models:
|
182
|
+
|
183
|
+
|
184
|
+
=== Auto-Switching all Reads to the Slave(s)
|
185
|
+
|
186
|
+
When you create a model, you could use <tt>db_magic :slave => :blah</tt> or
|
187
|
+
<tt>db_magic :slaves => [ :foo, :bar ]</tt> commands in your model to set up reads
|
188
|
+
redirection mode when all your find/count/exist/etc methods will be reading data
|
189
|
+
from your slave (or a bunch of slaves in a round-robin manner). Here is an example:
|
190
|
+
|
191
|
+
class Foo < ActiveRecord::Base
|
192
|
+
db_magic :slave => :slave01
|
193
|
+
end
|
194
|
+
|
195
|
+
class Bar < ActiveRecord::Base
|
196
|
+
db_magic :slaves => [ :slave01, :slave02 ]
|
197
|
+
end
|
198
|
+
|
199
|
+
|
200
|
+
=== Default Connection Switching
|
201
|
+
|
202
|
+
If you have more than one master-slave cluster (or simply more than one database)
|
203
|
+
in your database environment, then you might want to change the default database
|
204
|
+
connection of some of your models. You could do that by using
|
205
|
+
<tt>db_magic :connection => :foo</tt> call from your models. Example:
|
206
|
+
|
207
|
+
class Foo < ActiveRecord::Base
|
208
|
+
db_magic :connection => :foo
|
209
|
+
end
|
210
|
+
|
211
|
+
Sample model on a separate master-slave cluster (so, separate main connection +
|
212
|
+
a slave connection):
|
213
|
+
|
214
|
+
class Bar < ActiveRecord::Base
|
215
|
+
db_magic :connection => :bar, :slave => :bar_slave
|
216
|
+
end
|
217
|
+
|
218
|
+
=== Per-Query Connection Management
|
219
|
+
|
220
|
+
Sometimes you have select queries that you know you want to run on the master.
|
221
|
+
This could happen for example when you have just added some data and need to read
|
222
|
+
it back and not sure if it made it all the way to the slave yet or no. For this
|
223
|
+
situation and a few others there is a set of methods we've added to ActiveRecord models:
|
224
|
+
|
225
|
+
1) +on_master+ - this method could be used in two forms: block form and proxy form.
|
226
|
+
In the block form you could force connection switch for a block of code:
|
227
|
+
|
228
|
+
User.on_master do
|
229
|
+
user = User.find_by_login('foo')
|
230
|
+
user.update_attributes!(:activated => true)
|
231
|
+
end
|
232
|
+
|
233
|
+
In the proxy form this method could be used to force one query to be performed on
|
234
|
+
the master database server:
|
235
|
+
|
236
|
+
Comment.on_master.last(:limit => 5)
|
237
|
+
User.on_master.find_by_activation_code(code)
|
238
|
+
User.on_master.exists?(:login => login, :password => password)
|
239
|
+
|
240
|
+
2) +on_slave+ - this method is used to force a query to be run on a slave even in
|
241
|
+
situations when it's been previously forced to use the master. If there is more
|
242
|
+
than one slave, one would be selected randomly. Tis method has two forms as
|
243
|
+
well: block and proxy.
|
244
|
+
|
245
|
+
3) <tt>on_db(connection)</tt> - this method is what makes two previous methods
|
246
|
+
possible. It is used to switch a model's connection to some db for a short block
|
247
|
+
of code or even for one statement (two forms). It accepts the same range of values
|
248
|
+
as the +switch_connection_to+ method does. Example:
|
249
|
+
|
250
|
+
Comment.on_db(:olap).count
|
251
|
+
Post.on_db(:foo).find(:first)
|
252
|
+
|
253
|
+
By default in development and test environments you could use non-existing connections in your
|
254
|
+
<tt>on_db</tt> calls and rails would send all your queries to a single default database. In
|
255
|
+
production <tt>on_db</tt> won't accept non-existing names.
|
256
|
+
|
257
|
+
This behaviour is controlled by the <tt>DbCharmer.connections_should_exist</tt>
|
258
|
+
configuration attribute which could be set from a rails initializer.
|
259
|
+
|
260
|
+
|
261
|
+
=== Forced Slave Reads
|
262
|
+
|
263
|
+
In some cases we could have models that are too important to be used in default "send all
|
264
|
+
reads to the slave" mode, but we still would like to be able to switch them to this mode
|
265
|
+
sometimes. For example, you could have +User+ model, which you would like to keep from
|
266
|
+
lagging with your slaves because users do not like to see outdated information about their
|
267
|
+
accounts. But in some cases (like logged-out profile page views, etc) it would be perfectly
|
268
|
+
reasonable to switch all reads to the slave.
|
269
|
+
|
270
|
+
For this use-case starting with +DbCharmer+ release 1.7.0 we have a feature called forced
|
271
|
+
slave reads. It consists of a few separate small features that together make it really
|
272
|
+
powerful:
|
273
|
+
|
274
|
+
1) <tt>:force_slave_reads => false</tt> option for +ActiveRecord+'s <tt>db_magic</tt> method.
|
275
|
+
This option could be used to disable automated slave reads on your models so that you could
|
276
|
+
call <tt>on_slave</tt> or use other methods to enable slave reads when you need it. Example:
|
277
|
+
|
278
|
+
class User < ActiveRecord::Base
|
279
|
+
db_magic :slave => slave01, :forced_slave_reads => false
|
280
|
+
end
|
281
|
+
|
282
|
+
2) <tt>force_slave_reads</tt> +ActionController+ class method. This method could be used to
|
283
|
+
enable per-controller (when called with no arguments), or per-action (<tt>:only</tt> and
|
284
|
+
<tt>:except</tt> params) forced reads from slaves. This is really useful for actions in
|
285
|
+
which you know you could tolerate some slave lag so all your models with slaves defined will
|
286
|
+
send their reads to slaves. Example:
|
287
|
+
|
288
|
+
class ProfilesController < Application
|
289
|
+
force_slave_reads :except => [ :login, :logout ]
|
290
|
+
...
|
291
|
+
end
|
292
|
+
|
293
|
+
3) <tt>force_slave_reads!</tt> +ActionController+ instance method, that could be used within
|
294
|
+
your actions or in controller filters to temporarily switch your models to forced slave reads
|
295
|
+
mode. This method could be useful for cases when the same actions could be called by logged-in
|
296
|
+
and anonymous users. Then you could authorize users in <tt>before_filter</tt> and call
|
297
|
+
<tt>force_slave_reads!</tt> method for anonymous page views.
|
298
|
+
|
299
|
+
class ProfilesController < Application
|
300
|
+
before_filter do
|
301
|
+
force_slave_reads! unless current_user
|
302
|
+
end
|
303
|
+
...
|
304
|
+
end
|
305
|
+
|
306
|
+
Notice: Before using this method you need to enable +ActionController+ support in +DbCharmer+.
|
307
|
+
You need to call <tt>DbCharmer.enable_controller_magic!</tt> method from your project
|
308
|
+
initialization code.
|
309
|
+
|
310
|
+
4) <tt>DbCharmer.force_slave_reads</tt> method that could be used with a block of ruby code
|
311
|
+
and would enable forced slave reads mode until the end of the block execution. This is really
|
312
|
+
powerful feature allowing high granularity in your control of forced slave reads mode. Example:
|
313
|
+
|
314
|
+
DbCharmer.force_slave_reads do
|
315
|
+
...
|
316
|
+
total_users = User.count
|
317
|
+
...
|
318
|
+
end
|
319
|
+
|
320
|
+
Notice: At this point the feature considered beta and should be used with caution. It is fully covered
|
321
|
+
with tests, but there still could be unexpected issues when used in real-world applications.
|
322
|
+
|
323
|
+
|
324
|
+
=== Associations Connection Management
|
325
|
+
|
326
|
+
ActiveRecord models can have an associations with each other and since every model has its
|
327
|
+
own database connections, it becomes pretty hard to manage connections in a chained calls
|
328
|
+
like <tt>User.posts.count</tt>. With a class-only connection switching methods this call
|
329
|
+
would look like the following if we'd want to count posts on a separate database:
|
330
|
+
|
331
|
+
Post.on_db(:olap) { User.posts.count }
|
332
|
+
|
333
|
+
Apparently this is not the best way to write the code and we've implemented an <tt>on_*</tt>
|
334
|
+
methods on associations as well so you could do things like this:
|
335
|
+
|
336
|
+
@user.posts.on_db(:olap).count
|
337
|
+
@user.posts.on_slave.find(:title => 'Hello, world!')
|
338
|
+
|
339
|
+
Notice: Since ActiveRecord associations implemented as proxies for resulting
|
340
|
+
objects/collections, it is possible to use our connection switching methods even without
|
341
|
+
chained methods:
|
342
|
+
|
343
|
+
@post.user.on_slave - would return post's author
|
344
|
+
@photo.owner.on_slave - would return photo's owner
|
345
|
+
|
346
|
+
|
347
|
+
Starting with +DbCharmer+ release 1.4 it is possible to use prefix notation for has_many
|
348
|
+
and HABTM associations connection switching:
|
349
|
+
|
350
|
+
@user.on_db(:foo).posts
|
351
|
+
@user.on_slave.posts
|
352
|
+
|
353
|
+
|
354
|
+
=== Named Scopes Support
|
355
|
+
|
356
|
+
To make it easier for +DbCharmer+ users to use connections switching methods with named scopes,
|
357
|
+
we've added <tt>on_*</tt> methods support on the scopes as well. All the following scope chains
|
358
|
+
would do exactly the same way (the query would be executed on the :foo database connection):
|
359
|
+
|
360
|
+
Post.on_db(:foo).published.with_comments.spam_marked.count
|
361
|
+
Post.published.on_db(:foo).with_comments.spam_marked.count
|
362
|
+
Post.published.with_comments.on_db(:foo).spam_marked.count
|
363
|
+
Post.published.with_comments.spam_marked.on_db(:foo).count
|
364
|
+
|
365
|
+
And now, add this feature to our associations support and here is what we could do:
|
366
|
+
|
367
|
+
@user.on_db(:archive).posts.published.all
|
368
|
+
@user.posts.on_db(:olap).published.count
|
369
|
+
@user.posts.published.on_db(:foo).first
|
370
|
+
|
371
|
+
|
372
|
+
=== Bulk Connection Management
|
373
|
+
|
374
|
+
Sometimes you want to run code where a large number of tables may be used, and you'd like
|
375
|
+
them all to use an alternate database. You can now do this:
|
376
|
+
|
377
|
+
DbCharmer.with_remapped_databases(:logs => :big_logs_slave) { ... }
|
378
|
+
|
379
|
+
Any model whose default database is +:logs+ (e.g., <tt>db_charmer :connection => :logs</tt>)
|
380
|
+
will now have its connection switched to +:big_logs_slave+ in that block. This is lower
|
381
|
+
precedence than any other +DbCharmer+ method, so <tt>Model.on_db(:foo).find(...)</tt> and
|
382
|
+
such things will still use the database they specify, not the one that model was remapped
|
383
|
+
to.
|
384
|
+
|
385
|
+
You can specify any number of remappings at once, and you can also use +:master+ as a database
|
386
|
+
name that matches any model that has not had its connection set by +DbCharmer+ at all.
|
387
|
+
|
388
|
+
*Note*: +DbCharmer+ works via +alias_method_chain+ in model classes. It is very careful
|
389
|
+
to only patch the models it needs to. However, if you use +with_remapped_databases+ and
|
390
|
+
remap the default database (+:master+), then it has no choice but to patch all subclasses
|
391
|
+
of +ActiveRecord::Base+. This should not cause any serious problems or any big performance
|
392
|
+
impact, but it is worth noting.
|
393
|
+
|
394
|
+
|
395
|
+
== Simple Sharding Support
|
396
|
+
|
397
|
+
Starting with the release 1.6.0 of +DbCharmer+ we have added support for simple database sharding
|
398
|
+
to our ActiveRecord extensions. Even though this feature is tested in production, we do not recommend
|
399
|
+
using it in your applications without complete understanding of the principles of its work.
|
400
|
+
|
401
|
+
At this point we support four sharding methods:
|
402
|
+
|
403
|
+
1) +range+ - really simple sharding method that allows you to take a table, slice is to a set of
|
404
|
+
smaller tables with pre-defined ranges of primary keys and then put those smaller tables to
|
405
|
+
different databases/servers. This could be useful for situations where you have a huge table that
|
406
|
+
is slowly growing and you just want to keep it simple and split the table load into a few servers
|
407
|
+
without building any complex sharding schemes.
|
408
|
+
|
409
|
+
2) +hash_map+ - pretty simple sharding method that allows you to take a table and slice it to a set
|
410
|
+
of smaller tables by some key that has a pre-defined key of values. For example, list of US mailing
|
411
|
+
addresses could be sharded by states, where you'd be able to define which states are stored in which
|
412
|
+
databases/servers.
|
413
|
+
|
414
|
+
3) +db_block_map+ - this is a really complex sharding method that allows you to shard your table into a
|
415
|
+
set of small fixed-size blocks that then would be assigned to a set of shards (databases/servers).
|
416
|
+
Whenever you would need an additional blocks they would be allocated automatically and then balanced
|
417
|
+
across the shards you have defined in your database. This method could be used to scale out huge
|
418
|
+
tables with hundreds of millions to billions of rows and allows relatively easy re-sharding techniques
|
419
|
+
to be implemented on top.
|
420
|
+
|
421
|
+
4) +db_block_group_map+ - really similar to the +db_block_map+ method with a single difference: this method
|
422
|
+
allows you to have a set of databases (table groups) on each server and every group would be handled as a
|
423
|
+
separate shard of data. This approach is really useful for pre-sharding of your data before scaling your
|
424
|
+
application out. You can easily start with one server, having 10-20-50 separate databases, and then
|
425
|
+
move those databases to different servers as you see your database outgrow one machine.
|
426
|
+
|
427
|
+
|
428
|
+
=== How to enable sharding?
|
429
|
+
|
430
|
+
To enable sharding extensions you need to take a few things:
|
431
|
+
|
432
|
+
1) Create a Rails initializer (on run this code when you initialize your script/application) with a
|
433
|
+
set of sharded connections defined. Each connection would have a name, sharding method and an optional
|
434
|
+
set of parameters to initialize the sharding method of your choice.
|
435
|
+
|
436
|
+
2) Specify sharding connection you want to use in your models.
|
437
|
+
|
438
|
+
3) Specify the shard you want to use before doing any operations on your models.
|
439
|
+
|
440
|
+
For more details please check out the following documentation sections.
|
441
|
+
|
442
|
+
|
443
|
+
=== Sharded Connections
|
444
|
+
|
445
|
+
Sharded connection is a simple abstractions that allows you to specify all sharding parameters for a
|
446
|
+
cluster in one place and then use this centralized configuration in your models. Here are a few examples
|
447
|
+
of sharded connections initizlization calls:
|
448
|
+
|
449
|
+
1) Sample range-based sharded connection:
|
450
|
+
|
451
|
+
TEXTS_SHARDING_RANGES = {
|
452
|
+
0...100 => :shard1,
|
453
|
+
100..200 => :shard2,
|
454
|
+
:default => :shard3
|
455
|
+
}
|
456
|
+
|
457
|
+
DbCharmer::Sharding.register_connection(
|
458
|
+
:name => :texts,
|
459
|
+
:method => :range,
|
460
|
+
:ranges => TEXTS_SHARDING_RANGES
|
461
|
+
)
|
462
|
+
|
463
|
+
2) Sample hash map sharded connection:
|
464
|
+
|
465
|
+
SHARDING_MAP = {
|
466
|
+
'US' => :us_users,
|
467
|
+
'CA' => :ca_users,
|
468
|
+
:default => :other_users
|
469
|
+
}
|
470
|
+
|
471
|
+
DbCharmer::Sharding.register_connection(
|
472
|
+
:name => :users,
|
473
|
+
:method => :hash_map,
|
474
|
+
:map => SHARDING_MAP
|
475
|
+
)
|
476
|
+
|
477
|
+
3) Sample database block map sharded connection:
|
478
|
+
|
479
|
+
DbCharmer::Sharding.register_connection(
|
480
|
+
:name => :social,
|
481
|
+
:method => :db_block_map,
|
482
|
+
:block_size => 10000, # Number of keys per block
|
483
|
+
:map_table => :event_shards_map, # Table with blocks to shards mapping
|
484
|
+
:shards_table => :event_shards_info, # Shards connection information table
|
485
|
+
:connection => :social_shard_info # What connection to use to read the map
|
486
|
+
)
|
487
|
+
|
488
|
+
After your sharded connection is defined, you could use it in your models:
|
489
|
+
|
490
|
+
class Text < ActiveRecord::Base
|
491
|
+
db_magic :sharded => {
|
492
|
+
:key => :id,
|
493
|
+
:sharded_connection => :texts
|
494
|
+
}
|
495
|
+
end
|
496
|
+
|
497
|
+
class Event < ActiveRecord::Base
|
498
|
+
set_table_name :timeline_events
|
499
|
+
|
500
|
+
db_magic :sharded => {
|
501
|
+
:key => :to_uid,
|
502
|
+
:sharded_connection => :social
|
503
|
+
}
|
504
|
+
end
|
505
|
+
|
506
|
+
|
507
|
+
=== Switching connections in sharded models
|
508
|
+
|
509
|
+
Every time you need to perform an operation on a sharded model, you need to specify on which shard
|
510
|
+
you want to do it. We have a method for this which would look familiar for the people that use
|
511
|
+
+DbCharmer+ for non-sharded environments since it looks and works just like those per-query
|
512
|
+
connection management methods:
|
513
|
+
|
514
|
+
Event.shard_for(10).find(:conditions => { :to_uid => 123 }, :limit => 5)
|
515
|
+
Text.shard_for(123).find_by_id(123)
|
516
|
+
|
517
|
+
There is another method that could be used with range and hash_map sharding methods, this method
|
518
|
+
allows you to switch to the default shard:
|
519
|
+
|
520
|
+
Text.on_default_shard.create(:body => 'hello', :user_id => 123)
|
521
|
+
|
522
|
+
And finally, there is a method that allows you to run your code on each shard in the system (at this
|
523
|
+
point the method is supported in db_block_map and db_block_group_map methods only):
|
524
|
+
|
525
|
+
Event.on_each_shard { |event| event.delete_all }
|
526
|
+
|
527
|
+
|
528
|
+
=== Defining your own sharding methods
|
529
|
+
|
530
|
+
It is possible with +DbCharmer+ for the users to define their own sharding methods. You need to do a
|
531
|
+
few things to implement your very own sharding scheme:
|
532
|
+
|
533
|
+
1) Create a class with a name <tt>DbCharmer::Sharding::Method::YourOwnName</tt>
|
534
|
+
|
535
|
+
2) Implement at least a constructor <tt>initialize(config)</tt> and a lookup instance
|
536
|
+
method <tt>shard_for_key(key)</tt> that would return either a connection name from <tt>database.yml</tt>
|
537
|
+
file or just a hash of connection parameters for rails connection adapters.
|
538
|
+
|
539
|
+
3) Register your sharded connection using the following call:
|
540
|
+
|
541
|
+
DbCharmer::Sharding.register_connection(
|
542
|
+
:name => :some_name,
|
543
|
+
:method => :your_own_name, # your sharding method class name in lower case
|
544
|
+
... some additional parameters if needed ...
|
545
|
+
)
|
546
|
+
|
547
|
+
4) Use your sharded connection as any standard one.
|
548
|
+
|
549
|
+
|
550
|
+
=== Adding support for default shards in your custom sharding methods
|
551
|
+
|
552
|
+
If you want to be able to use +on_default_shard+ method on your custom-sharded models, you
|
553
|
+
need to do two things:
|
554
|
+
|
555
|
+
1) implement <tt>support_default_shard?</tt> instance method on your sharded class that
|
556
|
+
would return +true+ if you do support default shard specification and +false+ otherwise.
|
557
|
+
|
558
|
+
2) implement <tt>:default</tt> symbol support as a key in your +shard_for_key+ method.
|
559
|
+
|
560
|
+
|
561
|
+
=== Adding support for shards enumeration in your custom sharding methods
|
562
|
+
|
563
|
+
To add shards enumeration support to your custom-sharded models you need to implement
|
564
|
+
an instance method +shard_connections+ on your class. This method should return an array of
|
565
|
+
sharding connection names or connection configurations to be used to establish connections in a loop.
|
566
|
+
|
567
|
+
|
568
|
+
== Documentation/Questions
|
569
|
+
|
570
|
+
For more information about the library, please visit our site at http://kovyrin.github.com/db-charmer.
|
571
|
+
If you need more defails on DbCharmer internals, please check out the source code. All the plugin's
|
572
|
+
code is ~100% covered with tests (located in a separate staging rails project
|
573
|
+
at http://github.com/kovyrin/db-charmer-sandbox). The project has unit tests for all or at
|
574
|
+
least the most actively used code paths.
|
575
|
+
|
576
|
+
If you have any questions regarding this project, you could contact the author using
|
577
|
+
the DbCharmer Users Group mailing list:
|
578
|
+
|
579
|
+
- Group Info: http://groups.google.com/group/db-charmer
|
580
|
+
- Subscribe using the info page or by sending an email to mailto:db-charmer-subscribe@googlegroups.com
|
581
|
+
|
582
|
+
|
583
|
+
== What Ruby and Rails implementations does it work for?
|
584
|
+
|
585
|
+
We have a continuous integration setup for this gem on with Rails 2.2, 2.3 and 3.0 using a few
|
586
|
+
different versions of Ruby. For more information about the build matrix please visit
|
587
|
+
http://github.com/kovyrin/db-charmer-sandbox web site.
|
588
|
+
|
589
|
+
In addition to CI testing, we use this gem in production on Scribd.com (one of the largest RoR
|
590
|
+
sites in the world) with Ruby Enterprise Edition and Rails 2.2, Rails 2.3, Sinatra and plain
|
591
|
+
Rack applications.
|
592
|
+
|
593
|
+
Starting with version 1.7.0 we support Rails 3.0, but until further notice we consider it a
|
594
|
+
beta-quality feature since we do not run any production software on this version of Rails. If you
|
595
|
+
run your application on Rails 3.0, please contact the author.
|
596
|
+
|
597
|
+
|
598
|
+
== Who are the authors?
|
599
|
+
|
600
|
+
This plugin has been created in Scribd.com for our internal use and then the sources were opened for
|
601
|
+
other people to use. Most of the code in this package has been developed by Oleksiy Kovyrin for
|
602
|
+
Scribd.com and is released under the MIT license. For more details, see the LICENSE file.
|
603
|
+
|
604
|
+
Other contributors who have helped with the development of this library are (alphabetically ordered):
|
605
|
+
* Allen Madsen
|
606
|
+
* Andrew Geweke
|
607
|
+
* Ashley Martens
|
608
|
+
* Dmytro Shteflyuk
|
609
|
+
* Eric Lindvall
|
610
|
+
* Gregory Man
|
611
|
+
* Michael Birk
|
612
|
+
* Tyler McMullen
|