yam-db-charmer 1.7.01

Sign up to get free protection for your applications and to get access to all the features.
Files changed (40) hide show
  1. data/.gitignore +4 -0
  2. data/CHANGES +184 -0
  3. data/LICENSE +21 -0
  4. data/Makefile +2 -0
  5. data/README.rdoc +612 -0
  6. data/Rakefile +4 -0
  7. data/db-charmer.gemspec +29 -0
  8. data/init.rb +1 -0
  9. data/lib/db_charmer/action_controller/force_slave_reads.rb +69 -0
  10. data/lib/db_charmer/active_record/association_preload.rb +23 -0
  11. data/lib/db_charmer/active_record/class_attributes.rb +101 -0
  12. data/lib/db_charmer/active_record/connection_switching.rb +81 -0
  13. data/lib/db_charmer/active_record/db_magic.rb +85 -0
  14. data/lib/db_charmer/active_record/migration/multi_db_migrations.rb +71 -0
  15. data/lib/db_charmer/active_record/multi_db_proxy.rb +77 -0
  16. data/lib/db_charmer/active_record/sharding.rb +40 -0
  17. data/lib/db_charmer/connection_factory.rb +76 -0
  18. data/lib/db_charmer/connection_proxy.rb +27 -0
  19. data/lib/db_charmer/core_extensions.rb +23 -0
  20. data/lib/db_charmer/force_slave_reads.rb +36 -0
  21. data/lib/db_charmer/rails2/abstract_adapter/log_formatting.rb +24 -0
  22. data/lib/db_charmer/rails2/active_record/master_slave_routing.rb +49 -0
  23. data/lib/db_charmer/rails2/active_record/named_scope/scope_proxy.rb +26 -0
  24. data/lib/db_charmer/rails3/abstract_adapter/connection_name.rb +38 -0
  25. data/lib/db_charmer/rails3/active_record/log_subscriber.rb +23 -0
  26. data/lib/db_charmer/rails3/active_record/master_slave_routing.rb +46 -0
  27. data/lib/db_charmer/rails3/active_record/relation/connection_routing.rb +147 -0
  28. data/lib/db_charmer/rails3/active_record/relation_method.rb +28 -0
  29. data/lib/db_charmer/sharding/connection.rb +31 -0
  30. data/lib/db_charmer/sharding/method/db_block_group_map.rb +257 -0
  31. data/lib/db_charmer/sharding/method/db_block_map.rb +211 -0
  32. data/lib/db_charmer/sharding/method/hash_map.rb +23 -0
  33. data/lib/db_charmer/sharding/method/range.rb +33 -0
  34. data/lib/db_charmer/sharding/method.rb +10 -0
  35. data/lib/db_charmer/sharding/stub_connection.rb +60 -0
  36. data/lib/db_charmer/sharding.rb +18 -0
  37. data/lib/db_charmer/version.rb +10 -0
  38. data/lib/db_charmer.rb +192 -0
  39. data/lib/tasks/databases.rake +82 -0
  40. metadata +178 -0
data/.gitignore ADDED
@@ -0,0 +1,4 @@
1
+ /doc
2
+ /pkg
3
+ .DS_Store
4
+ /_site
data/CHANGES ADDED
@@ -0,0 +1,184 @@
1
+ 1.7.0 (2011-08-29):
2
+
3
+ Beta feature: Rails 3 support
4
+
5
+ Beta feature: Added force_slave_reads functionality. Now we could have models with slaves
6
+ that are not used by default, but could be turned on globally (per-controller or per-action).
7
+
8
+ Heavily reorganized the source code to match Rails code structure (class names, etc). This should
9
+ make it much easier for other contributors to work with the code.
10
+ Added smarter environment detection (using Rails.env, RAILS_ENV or RACK_ENV).
11
+
12
+ Changed dependencies a bit: instead of depending on rails, we now require specific components
13
+ (ActiveRecord, ActiveSupport, etc) + we do not require blankslate gem anymore.
14
+
15
+ Bugfixes: Fix for N+1 queries when accessing shard_info for db_block_group_map sharding method.
16
+
17
+ ----------------------------------------------------------------------------------------
18
+ 1.6.17-19 (2011-04-25):
19
+
20
+ Bugfixes: Do not touch database for sharded models until we really need to. Before 1.6.17
21
+ if a database server was dead and there were any other connection issues, class loading
22
+ in Ruby would be broken and sharded model class would not be initialized.
23
+
24
+ ----------------------------------------------------------------------------------------
25
+ 1.6.14 (2011-01-09):
26
+
27
+ Bugfixes: We do not support Rails 3, and now we prohibit any versions but 2.2 and 2.3 from
28
+ being used with db-charmer gem.
29
+
30
+ ----------------------------------------------------------------------------------------
31
+ 1.6.13 (2010-08-17):
32
+
33
+ Starting with this version we use Rails.env instead of RAILS_ENV to auto-detect rails
34
+ environment. If you use DbCharmer in non-rails project, please set DbCharmer.env manually.
35
+
36
+ Bugfixes: Thanks to Eric Lindvall we now allow connection names that have symbols ruby
37
+ wouldn't like for class names.
38
+
39
+ ----------------------------------------------------------------------------------------
40
+ 1.6.12 (2010-05-09):
41
+
42
+ Starting with this version we use Rails.cache (memcache or whatever you use in your project)
43
+ to cache sharding blocks information.
44
+
45
+ Bugfixes: Thanks to Allen Madsen (github user blatyo) we've fixed a few minor issues in
46
+ database connections handling.
47
+
48
+ ----------------------------------------------------------------------------------------
49
+ 1.6.11 (2010-04-16):
50
+
51
+ Bugfix: Change the way we allocate sharding blocks in block map sharding method to
52
+ prevent race-conditions from happening on block to shard assignments.
53
+ Breaking change: We require connections to exist by default in all connection factory
54
+ methods. If you need old behavior, pass should_exist=false explicitly.
55
+
56
+ ----------------------------------------------------------------------------------------
57
+ 1.6.10 (2010-04-09):
58
+
59
+ Multi-Db migrations changed. Now it is possible to call ActiveRecord::Migration.db_magic
60
+ and specify default migration connection that would be used by all migrations without
61
+ excplicitly switched connections.
62
+
63
+ ----------------------------------------------------------------------------------------
64
+ 1.6.9 (2010-04-08):
65
+
66
+ Bugfix release: now DbCharmer works without Rails.
67
+
68
+ ----------------------------------------------------------------------------------------
69
+ 1.6.7 (2010-04-07):
70
+
71
+ Changed the way we handle associations in on_db(:foo).find(:include) calls. Now we
72
+ switch association's connection only if its default connection is the same as the
73
+ master model's connection (not more "table does not exist" problems I hope).
74
+
75
+ ----------------------------------------------------------------------------------------
76
+ 1.6.5 (2010-04-05):
77
+
78
+ Bugfix release: Fixed :connection vs :slave in db_magic behaviour. Model.on_master should
79
+ run queries on the master, not on AR's default connection.
80
+
81
+ ----------------------------------------------------------------------------------------
82
+ 1.6.4 (2010-04-05):
83
+
84
+ Default behaviour changed: DbCharmer.connections_should_exist is true in all environments
85
+ by default. Old default behaviour was too misleading for many developers.
86
+
87
+ ----------------------------------------------------------------------------------------
88
+ 1.6.3 (2010-04-03):
89
+
90
+ Bugfix release: Modified stub connection initialization code to set default connections
91
+ for sharded models using shards enumeration or default shard features of sharding methods.
92
+
93
+ ----------------------------------------------------------------------------------------
94
+ 1.6.2 (2010-04-03):
95
+
96
+ Bugfix release: Modified our stub connection used on sharded models to fail on db-calling
97
+ methods only. Proxy the rest to a real shard connection. Another bug fixed in db_block_map
98
+ sharding method: we didn't increment block counters when assigning blocks to shards.
99
+
100
+ ----------------------------------------------------------------------------------------
101
+ 1.6.1 (2010-03-31):
102
+
103
+ Breaking change from now on all connection-switching methods (both in migrations and in
104
+ models) are controlled by a single option DbCharmer.connections_should_exist. This
105
+ option is false by default in all non-production environments. Check out README for
106
+ more details.
107
+
108
+ ----------------------------------------------------------------------------------------
109
+ 1.6.0 (2010-03-31):
110
+
111
+ The major (and arguably the only noticeable) change in this version is our simple database
112
+ sharding support. The feature is still in alpha stage and should not be used in production
113
+ without complete understanding of the principles of its work.
114
+
115
+ ----------------------------------------------------------------------------------------
116
+ 1.5.5 (2010-03-15):
117
+
118
+ Thanks to ngmoco.com (http://github.com/ngmoco) now DbCharmer supports one more use-case
119
+ for multi-db migrations. Now you can run the same migration on many databases at once.
120
+ For example, the following migration would create test_table on all three shard databases:
121
+
122
+ class MultiDbTest < ActiveRecord::Migration
123
+ db_magic :connections => [ :shard01, :shard02, :shard03 ]
124
+
125
+ def self.up
126
+ create_table :test_table do |t|
127
+ t.string :test_string
128
+ t.timestamps
129
+ end
130
+ end
131
+
132
+ def self.down
133
+ drop_table :test_table
134
+ end
135
+ end
136
+
137
+ ----------------------------------------------------------------------------------------
138
+ 1.5.4 (2010-03-12):
139
+
140
+ Added DbCharmer.with_remapped_databases, so that you can change the connection for
141
+ many models simultaneously, and implicitly. Very useful for work where you want to use
142
+ a particular slave for a whole range of database access.
143
+
144
+ ----------------------------------------------------------------------------------------
145
+ 1.5.3 (2010-03-10):
146
+
147
+ Few changes:
148
+ * Colorized connection names in the logs for development mode
149
+ * We do not log connection names when connection does not exist
150
+
151
+ ----------------------------------------------------------------------------------------
152
+ 1.5.1 (2010-03-06):
153
+
154
+ In this version we've added support for connection names logging in Rails queries log.
155
+ New log records have [connection_name] prefix for all queries that are executed on
156
+ non-standard connections:
157
+
158
+ [logs] LogRecord Columns (1.1ms) SHOW FIELDS FROM `log_records`
159
+ [logs] User Delete all (0.1ms) DELETE FROM `users`
160
+ [slave01] User Load (0.2ms) SELECT * FROM `users` WHERE (`users`.`login` = 'foo')
161
+
162
+ ----------------------------------------------------------------------------------------
163
+ 1.4.6 -> 1.5.0 (2010-03-05):
164
+
165
+ Major change in this version of DbCharmer is association preload support. For example,
166
+ let's say we have a schema:
167
+
168
+ class Post < ActiveRecord::Base
169
+ belongs_to :user
170
+ end
171
+
172
+ class User < ActiveRecord::Base
173
+ has_many :posts
174
+ end
175
+
176
+ Now, if we have the following call in our code:
177
+
178
+ User.on_db(:foo).all(:include => :posts)
179
+
180
+ In 1.4.6 it would load the users from connection :foo and posts from the
181
+ default connection, which is not what we would expect from this line of code.
182
+ So, starting 1.5.0 all finder calls on models having :include parameter would
183
+ switch associated models' connections to the same connection as the main model
184
+ in the call.
data/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License
2
+
3
+ Copyright (c) 2011, Oleksiy Kovyrin
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
data/Makefile ADDED
@@ -0,0 +1,2 @@
1
+ doc/files/README_rdoc.html: README.rdoc
2
+ rdoc README.rdoc
data/README.rdoc ADDED
@@ -0,0 +1,612 @@
1
+ = DB Charmer - ActiveRecord Connection Magic Plugin
2
+
3
+ +DbCharmer+ is a simple yet powerful plugin for ActiveRecord that significantly extends its ability to work with
4
+ multiple databases and/or database servers. The major features we add to ActiveRecord are:
5
+
6
+ 1. Simple management for AR model connections (+switch_connection_to+ method)
7
+ 2. Switching of default AR model connections to separate servers/databases
8
+ 3. Ability to easily choose where your query should go (<tt>Model.on_*</tt> methods family)
9
+ 4. Automated master/slave queries routing (selects go to a slave, updates handled by the master).
10
+ 5. Multiple database migrations with very flexible query routing controls.
11
+ 6. Simple database sharding functionality with multiple sharding methods (value, range, mapping table).
12
+
13
+ For more information on the project, you can check out our web site at http://kovyrin.github.com/db-charmer.
14
+
15
+
16
+ == Installation
17
+
18
+ There are two options when approaching +DbCharmer+ installation:
19
+ * using the gem (recommended and the only way of using it with Rails 3.0+)
20
+ * install as a Rails plugin (works in Rails 2.x only)
21
+
22
+ To install as a gem, add this to your Gemfile:
23
+
24
+ gem 'db-charmer', :require => 'db_charmer'
25
+
26
+ To install +DbCharmer+ as a Rails plugin use the following command:
27
+
28
+ ./script/plugin install git://github.com/kovyrin/db-charmer.git
29
+
30
+ _Notice_: If you use +DbCharmer+ in a non-rails project, you may need to set <tt>DbCharmer.env</tt> to a correct value
31
+ before using any of its connection management methods. Correct value here is a valid <tt>database.yml</tt>
32
+ first-level section name.
33
+
34
+
35
+ == Easy ActiveRecord Connection Management
36
+
37
+ As a part of this plugin we've added +switch_connection_to+ method that accepts many different kinds
38
+ of db connections specifications and uses them on a model. We support:
39
+
40
+ 1. Strings and symbols as the names of connection configuration blocks in database.yml.
41
+ 2. ActiveRecord models (we'd use connection currently set up on a model).
42
+ 3. Database connections (<tt>Model.connection</tt>)
43
+ 4. Nil values to reset model to default connection.
44
+
45
+ Sample code:
46
+
47
+ class Foo < ActiveRecord::Model; end
48
+
49
+ Foo.switch_connection_to(:blah)
50
+ Foo.switch_connection_to('foo')
51
+ Foo.switch_connection_to(Bar)
52
+ Foo.switch_connection_to(Baz.connection)
53
+ Foo.switch_connection_to(nil)
54
+
55
+ Sample <tt>database.yml</tt> configuration:
56
+
57
+ production:
58
+ blah:
59
+ adapter: mysql
60
+ username: blah
61
+ host: blah.local
62
+ database: blah
63
+
64
+ foo:
65
+ adapter: mysql
66
+ username: foo
67
+ host: foo.local
68
+ database: foo
69
+
70
+ The +switch_connection_to+ method has an optional second parameter +should_exist+ which is true
71
+ by default. This parameter is used when the method is called with a string or a symbol connection
72
+ name and there is no such connection configuration in the database.yml file. If this parameter
73
+ is +true+, an exception would be raised, otherwise, the error would be ignored and no connection
74
+ change would happen.
75
+
76
+ This is really useful when in development mode or in a tests you do not want to create many different
77
+ databases on your local machine and just want to put all your tables in a single database.
78
+
79
+ *Warning*: All the connection switching calls would switch connection *only* for those classes the
80
+ method called on. You can't call the +switch_connection_to+ method and switch connection for a
81
+ base class in some hierarchy (for example, you can't switch AR::Base connection and see all your
82
+ models switched to the new connection, use the classic +establish_connection+ instead).
83
+
84
+
85
+ == Multiple DB Migrations
86
+
87
+ In every application that works with many databases, there is need in a convenient schema migrations mechanism.
88
+
89
+ All Rails users already have this mechanism - rails migrations. So in +DbCharmer+, we've made it possible
90
+ to seamlessly use multiple databases in Rails migrations.
91
+
92
+ There are two methods available in migrations to operate on more than one database:
93
+
94
+ 1. Global connection change method - used to switch whole migration to a non-default database.
95
+ 2. Block-level connection change method - could be used to do only a part of a migration on a non-default db.
96
+
97
+ Migration class example (global connection rewrite):
98
+
99
+ class MultiDbTest < ActiveRecord::Migration
100
+ db_magic :connection => :second_db
101
+
102
+ def self.up
103
+ create_table :test_table, :force => true do |t|
104
+ t.string :test_string
105
+ t.timestamps
106
+ end
107
+ end
108
+
109
+ def self.down
110
+ drop_table :test_table
111
+ end
112
+ end
113
+
114
+ Migration class example (block-level connection rewrite):
115
+
116
+ class MultiDbTest < ActiveRecord::Migration
117
+ def self.up
118
+ on_db :second_db do
119
+ create_table :test_table, :force => true do |t|
120
+ t.string :test_string
121
+ t.timestamps
122
+ end
123
+ end
124
+ end
125
+
126
+ def self.down
127
+ on_db :second_db { drop_table :test_table }
128
+ end
129
+ end
130
+
131
+ Migration class example (global connection rewrite, multiple connections with the same table):
132
+ (NOTE: both :connection and :connections can take an array of connections)
133
+
134
+ class MultiDbTest < ActiveRecord::Migration
135
+ db_magic :connections => [:second_db, :default]
136
+
137
+ def self.up
138
+ create_table :test_table, :force => true do |t|
139
+ t.string :test_string
140
+ t.timestamps
141
+ end
142
+ end
143
+
144
+ def self.down
145
+ drop_table :test_table
146
+ end
147
+ end
148
+
149
+ === Default Migrations Connection
150
+
151
+ Starting with DbCharmer version 1.6.10 it is possible to call <tt>ActiveRecord::Migration.db_magic</tt>
152
+ and specify default migration connection that would be used by all migrations without
153
+ excplicitly switched connections. If you want to switch your migration to the default ActiveRecord
154
+ connection, just use <tt>db_magic :connection => :default</tt>.
155
+
156
+ === Invalid Connection Names Handling
157
+
158
+ By default in all environments <tt>on_db</tt> and <tt>db_magic</tt> statments would fail if
159
+ specified connection does not exist in database.yml. It is possible to make +DbCharmer+
160
+ ignore such situations in non-production environments so that rails would create the tables
161
+ in your single database (especially useful in test databases).
162
+
163
+ This behaviour is controlled by the <tt>DbCharmer.connections_should_exist</tt>
164
+ configuration attribute which could be set from a rails initializer.
165
+
166
+ Warning: if in test environment you use separate connections and master-slave support
167
+ in DbCharmer, make sure you disable transactional fixtures support in Rails. Without
168
+ this change you're going to see all kinds of weird data visibility problems in your tests.
169
+
170
+
171
+ == Using Models in Master-Slave Environments
172
+
173
+ Master-slave replication is the most popular scale-out technique in a medium-sized and
174
+ large database-centric applications today. There are some rails plugins out there that help
175
+ developers to use slave servers in their models but none of them were flexible enough
176
+ for us to start using them in a huge application we work on.
177
+
178
+ So, we've been using ActsAsReadonlyable plugin for a long time and have made tons
179
+ of changes in its code over the time. But since that plugin has been abandoned
180
+ by its authors, we've decided to collect all of our master-slave code in one plugin
181
+ and release it for rails 2.2+. +DbCharmer+ adds the following features to Rails models:
182
+
183
+
184
+ === Auto-Switching all Reads to the Slave(s)
185
+
186
+ When you create a model, you could use <tt>db_magic :slave => :blah</tt> or
187
+ <tt>db_magic :slaves => [ :foo, :bar ]</tt> commands in your model to set up reads
188
+ redirection mode when all your find/count/exist/etc methods will be reading data
189
+ from your slave (or a bunch of slaves in a round-robin manner). Here is an example:
190
+
191
+ class Foo < ActiveRecord::Base
192
+ db_magic :slave => :slave01
193
+ end
194
+
195
+ class Bar < ActiveRecord::Base
196
+ db_magic :slaves => [ :slave01, :slave02 ]
197
+ end
198
+
199
+
200
+ === Default Connection Switching
201
+
202
+ If you have more than one master-slave cluster (or simply more than one database)
203
+ in your database environment, then you might want to change the default database
204
+ connection of some of your models. You could do that by using
205
+ <tt>db_magic :connection => :foo</tt> call from your models. Example:
206
+
207
+ class Foo < ActiveRecord::Base
208
+ db_magic :connection => :foo
209
+ end
210
+
211
+ Sample model on a separate master-slave cluster (so, separate main connection +
212
+ a slave connection):
213
+
214
+ class Bar < ActiveRecord::Base
215
+ db_magic :connection => :bar, :slave => :bar_slave
216
+ end
217
+
218
+ === Per-Query Connection Management
219
+
220
+ Sometimes you have select queries that you know you want to run on the master.
221
+ This could happen for example when you have just added some data and need to read
222
+ it back and not sure if it made it all the way to the slave yet or no. For this
223
+ situation and a few others there is a set of methods we've added to ActiveRecord models:
224
+
225
+ 1) +on_master+ - this method could be used in two forms: block form and proxy form.
226
+ In the block form you could force connection switch for a block of code:
227
+
228
+ User.on_master do
229
+ user = User.find_by_login('foo')
230
+ user.update_attributes!(:activated => true)
231
+ end
232
+
233
+ In the proxy form this method could be used to force one query to be performed on
234
+ the master database server:
235
+
236
+ Comment.on_master.last(:limit => 5)
237
+ User.on_master.find_by_activation_code(code)
238
+ User.on_master.exists?(:login => login, :password => password)
239
+
240
+ 2) +on_slave+ - this method is used to force a query to be run on a slave even in
241
+ situations when it's been previously forced to use the master. If there is more
242
+ than one slave, one would be selected randomly. Tis method has two forms as
243
+ well: block and proxy.
244
+
245
+ 3) <tt>on_db(connection)</tt> - this method is what makes two previous methods
246
+ possible. It is used to switch a model's connection to some db for a short block
247
+ of code or even for one statement (two forms). It accepts the same range of values
248
+ as the +switch_connection_to+ method does. Example:
249
+
250
+ Comment.on_db(:olap).count
251
+ Post.on_db(:foo).find(:first)
252
+
253
+ By default in development and test environments you could use non-existing connections in your
254
+ <tt>on_db</tt> calls and rails would send all your queries to a single default database. In
255
+ production <tt>on_db</tt> won't accept non-existing names.
256
+
257
+ This behaviour is controlled by the <tt>DbCharmer.connections_should_exist</tt>
258
+ configuration attribute which could be set from a rails initializer.
259
+
260
+
261
+ === Forced Slave Reads
262
+
263
+ In some cases we could have models that are too important to be used in default "send all
264
+ reads to the slave" mode, but we still would like to be able to switch them to this mode
265
+ sometimes. For example, you could have +User+ model, which you would like to keep from
266
+ lagging with your slaves because users do not like to see outdated information about their
267
+ accounts. But in some cases (like logged-out profile page views, etc) it would be perfectly
268
+ reasonable to switch all reads to the slave.
269
+
270
+ For this use-case starting with +DbCharmer+ release 1.7.0 we have a feature called forced
271
+ slave reads. It consists of a few separate small features that together make it really
272
+ powerful:
273
+
274
+ 1) <tt>:force_slave_reads => false</tt> option for +ActiveRecord+'s <tt>db_magic</tt> method.
275
+ This option could be used to disable automated slave reads on your models so that you could
276
+ call <tt>on_slave</tt> or use other methods to enable slave reads when you need it. Example:
277
+
278
+ class User < ActiveRecord::Base
279
+ db_magic :slave => slave01, :forced_slave_reads => false
280
+ end
281
+
282
+ 2) <tt>force_slave_reads</tt> +ActionController+ class method. This method could be used to
283
+ enable per-controller (when called with no arguments), or per-action (<tt>:only</tt> and
284
+ <tt>:except</tt> params) forced reads from slaves. This is really useful for actions in
285
+ which you know you could tolerate some slave lag so all your models with slaves defined will
286
+ send their reads to slaves. Example:
287
+
288
+ class ProfilesController < Application
289
+ force_slave_reads :except => [ :login, :logout ]
290
+ ...
291
+ end
292
+
293
+ 3) <tt>force_slave_reads!</tt> +ActionController+ instance method, that could be used within
294
+ your actions or in controller filters to temporarily switch your models to forced slave reads
295
+ mode. This method could be useful for cases when the same actions could be called by logged-in
296
+ and anonymous users. Then you could authorize users in <tt>before_filter</tt> and call
297
+ <tt>force_slave_reads!</tt> method for anonymous page views.
298
+
299
+ class ProfilesController < Application
300
+ before_filter do
301
+ force_slave_reads! unless current_user
302
+ end
303
+ ...
304
+ end
305
+
306
+ Notice: Before using this method you need to enable +ActionController+ support in +DbCharmer+.
307
+ You need to call <tt>DbCharmer.enable_controller_magic!</tt> method from your project
308
+ initialization code.
309
+
310
+ 4) <tt>DbCharmer.force_slave_reads</tt> method that could be used with a block of ruby code
311
+ and would enable forced slave reads mode until the end of the block execution. This is really
312
+ powerful feature allowing high granularity in your control of forced slave reads mode. Example:
313
+
314
+ DbCharmer.force_slave_reads do
315
+ ...
316
+ total_users = User.count
317
+ ...
318
+ end
319
+
320
+ Notice: At this point the feature considered beta and should be used with caution. It is fully covered
321
+ with tests, but there still could be unexpected issues when used in real-world applications.
322
+
323
+
324
+ === Associations Connection Management
325
+
326
+ ActiveRecord models can have an associations with each other and since every model has its
327
+ own database connections, it becomes pretty hard to manage connections in a chained calls
328
+ like <tt>User.posts.count</tt>. With a class-only connection switching methods this call
329
+ would look like the following if we'd want to count posts on a separate database:
330
+
331
+ Post.on_db(:olap) { User.posts.count }
332
+
333
+ Apparently this is not the best way to write the code and we've implemented an <tt>on_*</tt>
334
+ methods on associations as well so you could do things like this:
335
+
336
+ @user.posts.on_db(:olap).count
337
+ @user.posts.on_slave.find(:title => 'Hello, world!')
338
+
339
+ Notice: Since ActiveRecord associations implemented as proxies for resulting
340
+ objects/collections, it is possible to use our connection switching methods even without
341
+ chained methods:
342
+
343
+ @post.user.on_slave - would return post's author
344
+ @photo.owner.on_slave - would return photo's owner
345
+
346
+
347
+ Starting with +DbCharmer+ release 1.4 it is possible to use prefix notation for has_many
348
+ and HABTM associations connection switching:
349
+
350
+ @user.on_db(:foo).posts
351
+ @user.on_slave.posts
352
+
353
+
354
+ === Named Scopes Support
355
+
356
+ To make it easier for +DbCharmer+ users to use connections switching methods with named scopes,
357
+ we've added <tt>on_*</tt> methods support on the scopes as well. All the following scope chains
358
+ would do exactly the same way (the query would be executed on the :foo database connection):
359
+
360
+ Post.on_db(:foo).published.with_comments.spam_marked.count
361
+ Post.published.on_db(:foo).with_comments.spam_marked.count
362
+ Post.published.with_comments.on_db(:foo).spam_marked.count
363
+ Post.published.with_comments.spam_marked.on_db(:foo).count
364
+
365
+ And now, add this feature to our associations support and here is what we could do:
366
+
367
+ @user.on_db(:archive).posts.published.all
368
+ @user.posts.on_db(:olap).published.count
369
+ @user.posts.published.on_db(:foo).first
370
+
371
+
372
+ === Bulk Connection Management
373
+
374
+ Sometimes you want to run code where a large number of tables may be used, and you'd like
375
+ them all to use an alternate database. You can now do this:
376
+
377
+ DbCharmer.with_remapped_databases(:logs => :big_logs_slave) { ... }
378
+
379
+ Any model whose default database is +:logs+ (e.g., <tt>db_charmer :connection => :logs</tt>)
380
+ will now have its connection switched to +:big_logs_slave+ in that block. This is lower
381
+ precedence than any other +DbCharmer+ method, so <tt>Model.on_db(:foo).find(...)</tt> and
382
+ such things will still use the database they specify, not the one that model was remapped
383
+ to.
384
+
385
+ You can specify any number of remappings at once, and you can also use +:master+ as a database
386
+ name that matches any model that has not had its connection set by +DbCharmer+ at all.
387
+
388
+ *Note*: +DbCharmer+ works via +alias_method_chain+ in model classes. It is very careful
389
+ to only patch the models it needs to. However, if you use +with_remapped_databases+ and
390
+ remap the default database (+:master+), then it has no choice but to patch all subclasses
391
+ of +ActiveRecord::Base+. This should not cause any serious problems or any big performance
392
+ impact, but it is worth noting.
393
+
394
+
395
+ == Simple Sharding Support
396
+
397
+ Starting with the release 1.6.0 of +DbCharmer+ we have added support for simple database sharding
398
+ to our ActiveRecord extensions. Even though this feature is tested in production, we do not recommend
399
+ using it in your applications without complete understanding of the principles of its work.
400
+
401
+ At this point we support four sharding methods:
402
+
403
+ 1) +range+ - really simple sharding method that allows you to take a table, slice is to a set of
404
+ smaller tables with pre-defined ranges of primary keys and then put those smaller tables to
405
+ different databases/servers. This could be useful for situations where you have a huge table that
406
+ is slowly growing and you just want to keep it simple and split the table load into a few servers
407
+ without building any complex sharding schemes.
408
+
409
+ 2) +hash_map+ - pretty simple sharding method that allows you to take a table and slice it to a set
410
+ of smaller tables by some key that has a pre-defined key of values. For example, list of US mailing
411
+ addresses could be sharded by states, where you'd be able to define which states are stored in which
412
+ databases/servers.
413
+
414
+ 3) +db_block_map+ - this is a really complex sharding method that allows you to shard your table into a
415
+ set of small fixed-size blocks that then would be assigned to a set of shards (databases/servers).
416
+ Whenever you would need an additional blocks they would be allocated automatically and then balanced
417
+ across the shards you have defined in your database. This method could be used to scale out huge
418
+ tables with hundreds of millions to billions of rows and allows relatively easy re-sharding techniques
419
+ to be implemented on top.
420
+
421
+ 4) +db_block_group_map+ - really similar to the +db_block_map+ method with a single difference: this method
422
+ allows you to have a set of databases (table groups) on each server and every group would be handled as a
423
+ separate shard of data. This approach is really useful for pre-sharding of your data before scaling your
424
+ application out. You can easily start with one server, having 10-20-50 separate databases, and then
425
+ move those databases to different servers as you see your database outgrow one machine.
426
+
427
+
428
+ === How to enable sharding?
429
+
430
+ To enable sharding extensions you need to take a few things:
431
+
432
+ 1) Create a Rails initializer (on run this code when you initialize your script/application) with a
433
+ set of sharded connections defined. Each connection would have a name, sharding method and an optional
434
+ set of parameters to initialize the sharding method of your choice.
435
+
436
+ 2) Specify sharding connection you want to use in your models.
437
+
438
+ 3) Specify the shard you want to use before doing any operations on your models.
439
+
440
+ For more details please check out the following documentation sections.
441
+
442
+
443
+ === Sharded Connections
444
+
445
+ Sharded connection is a simple abstractions that allows you to specify all sharding parameters for a
446
+ cluster in one place and then use this centralized configuration in your models. Here are a few examples
447
+ of sharded connections initizlization calls:
448
+
449
+ 1) Sample range-based sharded connection:
450
+
451
+ TEXTS_SHARDING_RANGES = {
452
+ 0...100 => :shard1,
453
+ 100..200 => :shard2,
454
+ :default => :shard3
455
+ }
456
+
457
+ DbCharmer::Sharding.register_connection(
458
+ :name => :texts,
459
+ :method => :range,
460
+ :ranges => TEXTS_SHARDING_RANGES
461
+ )
462
+
463
+ 2) Sample hash map sharded connection:
464
+
465
+ SHARDING_MAP = {
466
+ 'US' => :us_users,
467
+ 'CA' => :ca_users,
468
+ :default => :other_users
469
+ }
470
+
471
+ DbCharmer::Sharding.register_connection(
472
+ :name => :users,
473
+ :method => :hash_map,
474
+ :map => SHARDING_MAP
475
+ )
476
+
477
+ 3) Sample database block map sharded connection:
478
+
479
+ DbCharmer::Sharding.register_connection(
480
+ :name => :social,
481
+ :method => :db_block_map,
482
+ :block_size => 10000, # Number of keys per block
483
+ :map_table => :event_shards_map, # Table with blocks to shards mapping
484
+ :shards_table => :event_shards_info, # Shards connection information table
485
+ :connection => :social_shard_info # What connection to use to read the map
486
+ )
487
+
488
+ After your sharded connection is defined, you could use it in your models:
489
+
490
+ class Text < ActiveRecord::Base
491
+ db_magic :sharded => {
492
+ :key => :id,
493
+ :sharded_connection => :texts
494
+ }
495
+ end
496
+
497
+ class Event < ActiveRecord::Base
498
+ set_table_name :timeline_events
499
+
500
+ db_magic :sharded => {
501
+ :key => :to_uid,
502
+ :sharded_connection => :social
503
+ }
504
+ end
505
+
506
+
507
+ === Switching connections in sharded models
508
+
509
+ Every time you need to perform an operation on a sharded model, you need to specify on which shard
510
+ you want to do it. We have a method for this which would look familiar for the people that use
511
+ +DbCharmer+ for non-sharded environments since it looks and works just like those per-query
512
+ connection management methods:
513
+
514
+ Event.shard_for(10).find(:conditions => { :to_uid => 123 }, :limit => 5)
515
+ Text.shard_for(123).find_by_id(123)
516
+
517
+ There is another method that could be used with range and hash_map sharding methods, this method
518
+ allows you to switch to the default shard:
519
+
520
+ Text.on_default_shard.create(:body => 'hello', :user_id => 123)
521
+
522
+ And finally, there is a method that allows you to run your code on each shard in the system (at this
523
+ point the method is supported in db_block_map and db_block_group_map methods only):
524
+
525
+ Event.on_each_shard { |event| event.delete_all }
526
+
527
+
528
+ === Defining your own sharding methods
529
+
530
+ It is possible with +DbCharmer+ for the users to define their own sharding methods. You need to do a
531
+ few things to implement your very own sharding scheme:
532
+
533
+ 1) Create a class with a name <tt>DbCharmer::Sharding::Method::YourOwnName</tt>
534
+
535
+ 2) Implement at least a constructor <tt>initialize(config)</tt> and a lookup instance
536
+ method <tt>shard_for_key(key)</tt> that would return either a connection name from <tt>database.yml</tt>
537
+ file or just a hash of connection parameters for rails connection adapters.
538
+
539
+ 3) Register your sharded connection using the following call:
540
+
541
+ DbCharmer::Sharding.register_connection(
542
+ :name => :some_name,
543
+ :method => :your_own_name, # your sharding method class name in lower case
544
+ ... some additional parameters if needed ...
545
+ )
546
+
547
+ 4) Use your sharded connection as any standard one.
548
+
549
+
550
+ === Adding support for default shards in your custom sharding methods
551
+
552
+ If you want to be able to use +on_default_shard+ method on your custom-sharded models, you
553
+ need to do two things:
554
+
555
+ 1) implement <tt>support_default_shard?</tt> instance method on your sharded class that
556
+ would return +true+ if you do support default shard specification and +false+ otherwise.
557
+
558
+ 2) implement <tt>:default</tt> symbol support as a key in your +shard_for_key+ method.
559
+
560
+
561
+ === Adding support for shards enumeration in your custom sharding methods
562
+
563
+ To add shards enumeration support to your custom-sharded models you need to implement
564
+ an instance method +shard_connections+ on your class. This method should return an array of
565
+ sharding connection names or connection configurations to be used to establish connections in a loop.
566
+
567
+
568
+ == Documentation/Questions
569
+
570
+ For more information about the library, please visit our site at http://kovyrin.github.com/db-charmer.
571
+ If you need more defails on DbCharmer internals, please check out the source code. All the plugin's
572
+ code is ~100% covered with tests (located in a separate staging rails project
573
+ at http://github.com/kovyrin/db-charmer-sandbox). The project has unit tests for all or at
574
+ least the most actively used code paths.
575
+
576
+ If you have any questions regarding this project, you could contact the author using
577
+ the DbCharmer Users Group mailing list:
578
+
579
+ - Group Info: http://groups.google.com/group/db-charmer
580
+ - Subscribe using the info page or by sending an email to mailto:db-charmer-subscribe@googlegroups.com
581
+
582
+
583
+ == What Ruby and Rails implementations does it work for?
584
+
585
+ We have a continuous integration setup for this gem on with Rails 2.2, 2.3 and 3.0 using a few
586
+ different versions of Ruby. For more information about the build matrix please visit
587
+ http://github.com/kovyrin/db-charmer-sandbox web site.
588
+
589
+ In addition to CI testing, we use this gem in production on Scribd.com (one of the largest RoR
590
+ sites in the world) with Ruby Enterprise Edition and Rails 2.2, Rails 2.3, Sinatra and plain
591
+ Rack applications.
592
+
593
+ Starting with version 1.7.0 we support Rails 3.0, but until further notice we consider it a
594
+ beta-quality feature since we do not run any production software on this version of Rails. If you
595
+ run your application on Rails 3.0, please contact the author.
596
+
597
+
598
+ == Who are the authors?
599
+
600
+ This plugin has been created in Scribd.com for our internal use and then the sources were opened for
601
+ other people to use. Most of the code in this package has been developed by Oleksiy Kovyrin for
602
+ Scribd.com and is released under the MIT license. For more details, see the LICENSE file.
603
+
604
+ Other contributors who have helped with the development of this library are (alphabetically ordered):
605
+ * Allen Madsen
606
+ * Andrew Geweke
607
+ * Ashley Martens
608
+ * Dmytro Shteflyuk
609
+ * Eric Lindvall
610
+ * Gregory Man
611
+ * Michael Birk
612
+ * Tyler McMullen