yam-db-charmer 1.7.01

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (40) hide show
  1. data/.gitignore +4 -0
  2. data/CHANGES +184 -0
  3. data/LICENSE +21 -0
  4. data/Makefile +2 -0
  5. data/README.rdoc +612 -0
  6. data/Rakefile +4 -0
  7. data/db-charmer.gemspec +29 -0
  8. data/init.rb +1 -0
  9. data/lib/db_charmer/action_controller/force_slave_reads.rb +69 -0
  10. data/lib/db_charmer/active_record/association_preload.rb +23 -0
  11. data/lib/db_charmer/active_record/class_attributes.rb +101 -0
  12. data/lib/db_charmer/active_record/connection_switching.rb +81 -0
  13. data/lib/db_charmer/active_record/db_magic.rb +85 -0
  14. data/lib/db_charmer/active_record/migration/multi_db_migrations.rb +71 -0
  15. data/lib/db_charmer/active_record/multi_db_proxy.rb +77 -0
  16. data/lib/db_charmer/active_record/sharding.rb +40 -0
  17. data/lib/db_charmer/connection_factory.rb +76 -0
  18. data/lib/db_charmer/connection_proxy.rb +27 -0
  19. data/lib/db_charmer/core_extensions.rb +23 -0
  20. data/lib/db_charmer/force_slave_reads.rb +36 -0
  21. data/lib/db_charmer/rails2/abstract_adapter/log_formatting.rb +24 -0
  22. data/lib/db_charmer/rails2/active_record/master_slave_routing.rb +49 -0
  23. data/lib/db_charmer/rails2/active_record/named_scope/scope_proxy.rb +26 -0
  24. data/lib/db_charmer/rails3/abstract_adapter/connection_name.rb +38 -0
  25. data/lib/db_charmer/rails3/active_record/log_subscriber.rb +23 -0
  26. data/lib/db_charmer/rails3/active_record/master_slave_routing.rb +46 -0
  27. data/lib/db_charmer/rails3/active_record/relation/connection_routing.rb +147 -0
  28. data/lib/db_charmer/rails3/active_record/relation_method.rb +28 -0
  29. data/lib/db_charmer/sharding/connection.rb +31 -0
  30. data/lib/db_charmer/sharding/method/db_block_group_map.rb +257 -0
  31. data/lib/db_charmer/sharding/method/db_block_map.rb +211 -0
  32. data/lib/db_charmer/sharding/method/hash_map.rb +23 -0
  33. data/lib/db_charmer/sharding/method/range.rb +33 -0
  34. data/lib/db_charmer/sharding/method.rb +10 -0
  35. data/lib/db_charmer/sharding/stub_connection.rb +60 -0
  36. data/lib/db_charmer/sharding.rb +18 -0
  37. data/lib/db_charmer/version.rb +10 -0
  38. data/lib/db_charmer.rb +192 -0
  39. data/lib/tasks/databases.rake +82 -0
  40. metadata +178 -0
data/.gitignore ADDED
@@ -0,0 +1,4 @@
1
+ /doc
2
+ /pkg
3
+ .DS_Store
4
+ /_site
data/CHANGES ADDED
@@ -0,0 +1,184 @@
1
+ 1.7.0 (2011-08-29):
2
+
3
+ Beta feature: Rails 3 support
4
+
5
+ Beta feature: Added force_slave_reads functionality. Now we could have models with slaves
6
+ that are not used by default, but could be turned on globally (per-controller or per-action).
7
+
8
+ Heavily reorganized the source code to match Rails code structure (class names, etc). This should
9
+ make it much easier for other contributors to work with the code.
10
+ Added smarter environment detection (using Rails.env, RAILS_ENV or RACK_ENV).
11
+
12
+ Changed dependencies a bit: instead of depending on rails, we now require specific components
13
+ (ActiveRecord, ActiveSupport, etc) + we do not require blankslate gem anymore.
14
+
15
+ Bugfixes: Fix for N+1 queries when accessing shard_info for db_block_group_map sharding method.
16
+
17
+ ----------------------------------------------------------------------------------------
18
+ 1.6.17-19 (2011-04-25):
19
+
20
+ Bugfixes: Do not touch database for sharded models until we really need to. Before 1.6.17
21
+ if a database server was dead and there were any other connection issues, class loading
22
+ in Ruby would be broken and sharded model class would not be initialized.
23
+
24
+ ----------------------------------------------------------------------------------------
25
+ 1.6.14 (2011-01-09):
26
+
27
+ Bugfixes: We do not support Rails 3, and now we prohibit any versions but 2.2 and 2.3 from
28
+ being used with db-charmer gem.
29
+
30
+ ----------------------------------------------------------------------------------------
31
+ 1.6.13 (2010-08-17):
32
+
33
+ Starting with this version we use Rails.env instead of RAILS_ENV to auto-detect rails
34
+ environment. If you use DbCharmer in non-rails project, please set DbCharmer.env manually.
35
+
36
+ Bugfixes: Thanks to Eric Lindvall we now allow connection names that have symbols ruby
37
+ wouldn't like for class names.
38
+
39
+ ----------------------------------------------------------------------------------------
40
+ 1.6.12 (2010-05-09):
41
+
42
+ Starting with this version we use Rails.cache (memcache or whatever you use in your project)
43
+ to cache sharding blocks information.
44
+
45
+ Bugfixes: Thanks to Allen Madsen (github user blatyo) we've fixed a few minor issues in
46
+ database connections handling.
47
+
48
+ ----------------------------------------------------------------------------------------
49
+ 1.6.11 (2010-04-16):
50
+
51
+ Bugfix: Change the way we allocate sharding blocks in block map sharding method to
52
+ prevent race-conditions from happening on block to shard assignments.
53
+ Breaking change: We require connections to exist by default in all connection factory
54
+ methods. If you need old behavior, pass should_exist=false explicitly.
55
+
56
+ ----------------------------------------------------------------------------------------
57
+ 1.6.10 (2010-04-09):
58
+
59
+ Multi-Db migrations changed. Now it is possible to call ActiveRecord::Migration.db_magic
60
+ and specify default migration connection that would be used by all migrations without
61
+ excplicitly switched connections.
62
+
63
+ ----------------------------------------------------------------------------------------
64
+ 1.6.9 (2010-04-08):
65
+
66
+ Bugfix release: now DbCharmer works without Rails.
67
+
68
+ ----------------------------------------------------------------------------------------
69
+ 1.6.7 (2010-04-07):
70
+
71
+ Changed the way we handle associations in on_db(:foo).find(:include) calls. Now we
72
+ switch association's connection only if its default connection is the same as the
73
+ master model's connection (not more "table does not exist" problems I hope).
74
+
75
+ ----------------------------------------------------------------------------------------
76
+ 1.6.5 (2010-04-05):
77
+
78
+ Bugfix release: Fixed :connection vs :slave in db_magic behaviour. Model.on_master should
79
+ run queries on the master, not on AR's default connection.
80
+
81
+ ----------------------------------------------------------------------------------------
82
+ 1.6.4 (2010-04-05):
83
+
84
+ Default behaviour changed: DbCharmer.connections_should_exist is true in all environments
85
+ by default. Old default behaviour was too misleading for many developers.
86
+
87
+ ----------------------------------------------------------------------------------------
88
+ 1.6.3 (2010-04-03):
89
+
90
+ Bugfix release: Modified stub connection initialization code to set default connections
91
+ for sharded models using shards enumeration or default shard features of sharding methods.
92
+
93
+ ----------------------------------------------------------------------------------------
94
+ 1.6.2 (2010-04-03):
95
+
96
+ Bugfix release: Modified our stub connection used on sharded models to fail on db-calling
97
+ methods only. Proxy the rest to a real shard connection. Another bug fixed in db_block_map
98
+ sharding method: we didn't increment block counters when assigning blocks to shards.
99
+
100
+ ----------------------------------------------------------------------------------------
101
+ 1.6.1 (2010-03-31):
102
+
103
+ Breaking change from now on all connection-switching methods (both in migrations and in
104
+ models) are controlled by a single option DbCharmer.connections_should_exist. This
105
+ option is false by default in all non-production environments. Check out README for
106
+ more details.
107
+
108
+ ----------------------------------------------------------------------------------------
109
+ 1.6.0 (2010-03-31):
110
+
111
+ The major (and arguably the only noticeable) change in this version is our simple database
112
+ sharding support. The feature is still in alpha stage and should not be used in production
113
+ without complete understanding of the principles of its work.
114
+
115
+ ----------------------------------------------------------------------------------------
116
+ 1.5.5 (2010-03-15):
117
+
118
+ Thanks to ngmoco.com (http://github.com/ngmoco) now DbCharmer supports one more use-case
119
+ for multi-db migrations. Now you can run the same migration on many databases at once.
120
+ For example, the following migration would create test_table on all three shard databases:
121
+
122
+ class MultiDbTest < ActiveRecord::Migration
123
+ db_magic :connections => [ :shard01, :shard02, :shard03 ]
124
+
125
+ def self.up
126
+ create_table :test_table do |t|
127
+ t.string :test_string
128
+ t.timestamps
129
+ end
130
+ end
131
+
132
+ def self.down
133
+ drop_table :test_table
134
+ end
135
+ end
136
+
137
+ ----------------------------------------------------------------------------------------
138
+ 1.5.4 (2010-03-12):
139
+
140
+ Added DbCharmer.with_remapped_databases, so that you can change the connection for
141
+ many models simultaneously, and implicitly. Very useful for work where you want to use
142
+ a particular slave for a whole range of database access.
143
+
144
+ ----------------------------------------------------------------------------------------
145
+ 1.5.3 (2010-03-10):
146
+
147
+ Few changes:
148
+ * Colorized connection names in the logs for development mode
149
+ * We do not log connection names when connection does not exist
150
+
151
+ ----------------------------------------------------------------------------------------
152
+ 1.5.1 (2010-03-06):
153
+
154
+ In this version we've added support for connection names logging in Rails queries log.
155
+ New log records have [connection_name] prefix for all queries that are executed on
156
+ non-standard connections:
157
+
158
+ [logs] LogRecord Columns (1.1ms) SHOW FIELDS FROM `log_records`
159
+ [logs] User Delete all (0.1ms) DELETE FROM `users`
160
+ [slave01] User Load (0.2ms) SELECT * FROM `users` WHERE (`users`.`login` = 'foo')
161
+
162
+ ----------------------------------------------------------------------------------------
163
+ 1.4.6 -> 1.5.0 (2010-03-05):
164
+
165
+ Major change in this version of DbCharmer is association preload support. For example,
166
+ let's say we have a schema:
167
+
168
+ class Post < ActiveRecord::Base
169
+ belongs_to :user
170
+ end
171
+
172
+ class User < ActiveRecord::Base
173
+ has_many :posts
174
+ end
175
+
176
+ Now, if we have the following call in our code:
177
+
178
+ User.on_db(:foo).all(:include => :posts)
179
+
180
+ In 1.4.6 it would load the users from connection :foo and posts from the
181
+ default connection, which is not what we would expect from this line of code.
182
+ So, starting 1.5.0 all finder calls on models having :include parameter would
183
+ switch associated models' connections to the same connection as the main model
184
+ in the call.
data/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License
2
+
3
+ Copyright (c) 2011, Oleksiy Kovyrin
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
data/Makefile ADDED
@@ -0,0 +1,2 @@
1
+ doc/files/README_rdoc.html: README.rdoc
2
+ rdoc README.rdoc
data/README.rdoc ADDED
@@ -0,0 +1,612 @@
1
+ = DB Charmer - ActiveRecord Connection Magic Plugin
2
+
3
+ +DbCharmer+ is a simple yet powerful plugin for ActiveRecord that significantly extends its ability to work with
4
+ multiple databases and/or database servers. The major features we add to ActiveRecord are:
5
+
6
+ 1. Simple management for AR model connections (+switch_connection_to+ method)
7
+ 2. Switching of default AR model connections to separate servers/databases
8
+ 3. Ability to easily choose where your query should go (<tt>Model.on_*</tt> methods family)
9
+ 4. Automated master/slave queries routing (selects go to a slave, updates handled by the master).
10
+ 5. Multiple database migrations with very flexible query routing controls.
11
+ 6. Simple database sharding functionality with multiple sharding methods (value, range, mapping table).
12
+
13
+ For more information on the project, you can check out our web site at http://kovyrin.github.com/db-charmer.
14
+
15
+
16
+ == Installation
17
+
18
+ There are two options when approaching +DbCharmer+ installation:
19
+ * using the gem (recommended and the only way of using it with Rails 3.0+)
20
+ * install as a Rails plugin (works in Rails 2.x only)
21
+
22
+ To install as a gem, add this to your Gemfile:
23
+
24
+ gem 'db-charmer', :require => 'db_charmer'
25
+
26
+ To install +DbCharmer+ as a Rails plugin use the following command:
27
+
28
+ ./script/plugin install git://github.com/kovyrin/db-charmer.git
29
+
30
+ _Notice_: If you use +DbCharmer+ in a non-rails project, you may need to set <tt>DbCharmer.env</tt> to a correct value
31
+ before using any of its connection management methods. Correct value here is a valid <tt>database.yml</tt>
32
+ first-level section name.
33
+
34
+
35
+ == Easy ActiveRecord Connection Management
36
+
37
+ As a part of this plugin we've added +switch_connection_to+ method that accepts many different kinds
38
+ of db connections specifications and uses them on a model. We support:
39
+
40
+ 1. Strings and symbols as the names of connection configuration blocks in database.yml.
41
+ 2. ActiveRecord models (we'd use connection currently set up on a model).
42
+ 3. Database connections (<tt>Model.connection</tt>)
43
+ 4. Nil values to reset model to default connection.
44
+
45
+ Sample code:
46
+
47
+ class Foo < ActiveRecord::Model; end
48
+
49
+ Foo.switch_connection_to(:blah)
50
+ Foo.switch_connection_to('foo')
51
+ Foo.switch_connection_to(Bar)
52
+ Foo.switch_connection_to(Baz.connection)
53
+ Foo.switch_connection_to(nil)
54
+
55
+ Sample <tt>database.yml</tt> configuration:
56
+
57
+ production:
58
+ blah:
59
+ adapter: mysql
60
+ username: blah
61
+ host: blah.local
62
+ database: blah
63
+
64
+ foo:
65
+ adapter: mysql
66
+ username: foo
67
+ host: foo.local
68
+ database: foo
69
+
70
+ The +switch_connection_to+ method has an optional second parameter +should_exist+ which is true
71
+ by default. This parameter is used when the method is called with a string or a symbol connection
72
+ name and there is no such connection configuration in the database.yml file. If this parameter
73
+ is +true+, an exception would be raised, otherwise, the error would be ignored and no connection
74
+ change would happen.
75
+
76
+ This is really useful when in development mode or in a tests you do not want to create many different
77
+ databases on your local machine and just want to put all your tables in a single database.
78
+
79
+ *Warning*: All the connection switching calls would switch connection *only* for those classes the
80
+ method called on. You can't call the +switch_connection_to+ method and switch connection for a
81
+ base class in some hierarchy (for example, you can't switch AR::Base connection and see all your
82
+ models switched to the new connection, use the classic +establish_connection+ instead).
83
+
84
+
85
+ == Multiple DB Migrations
86
+
87
+ In every application that works with many databases, there is need in a convenient schema migrations mechanism.
88
+
89
+ All Rails users already have this mechanism - rails migrations. So in +DbCharmer+, we've made it possible
90
+ to seamlessly use multiple databases in Rails migrations.
91
+
92
+ There are two methods available in migrations to operate on more than one database:
93
+
94
+ 1. Global connection change method - used to switch whole migration to a non-default database.
95
+ 2. Block-level connection change method - could be used to do only a part of a migration on a non-default db.
96
+
97
+ Migration class example (global connection rewrite):
98
+
99
+ class MultiDbTest < ActiveRecord::Migration
100
+ db_magic :connection => :second_db
101
+
102
+ def self.up
103
+ create_table :test_table, :force => true do |t|
104
+ t.string :test_string
105
+ t.timestamps
106
+ end
107
+ end
108
+
109
+ def self.down
110
+ drop_table :test_table
111
+ end
112
+ end
113
+
114
+ Migration class example (block-level connection rewrite):
115
+
116
+ class MultiDbTest < ActiveRecord::Migration
117
+ def self.up
118
+ on_db :second_db do
119
+ create_table :test_table, :force => true do |t|
120
+ t.string :test_string
121
+ t.timestamps
122
+ end
123
+ end
124
+ end
125
+
126
+ def self.down
127
+ on_db :second_db { drop_table :test_table }
128
+ end
129
+ end
130
+
131
+ Migration class example (global connection rewrite, multiple connections with the same table):
132
+ (NOTE: both :connection and :connections can take an array of connections)
133
+
134
+ class MultiDbTest < ActiveRecord::Migration
135
+ db_magic :connections => [:second_db, :default]
136
+
137
+ def self.up
138
+ create_table :test_table, :force => true do |t|
139
+ t.string :test_string
140
+ t.timestamps
141
+ end
142
+ end
143
+
144
+ def self.down
145
+ drop_table :test_table
146
+ end
147
+ end
148
+
149
+ === Default Migrations Connection
150
+
151
+ Starting with DbCharmer version 1.6.10 it is possible to call <tt>ActiveRecord::Migration.db_magic</tt>
152
+ and specify default migration connection that would be used by all migrations without
153
+ excplicitly switched connections. If you want to switch your migration to the default ActiveRecord
154
+ connection, just use <tt>db_magic :connection => :default</tt>.
155
+
156
+ === Invalid Connection Names Handling
157
+
158
+ By default in all environments <tt>on_db</tt> and <tt>db_magic</tt> statments would fail if
159
+ specified connection does not exist in database.yml. It is possible to make +DbCharmer+
160
+ ignore such situations in non-production environments so that rails would create the tables
161
+ in your single database (especially useful in test databases).
162
+
163
+ This behaviour is controlled by the <tt>DbCharmer.connections_should_exist</tt>
164
+ configuration attribute which could be set from a rails initializer.
165
+
166
+ Warning: if in test environment you use separate connections and master-slave support
167
+ in DbCharmer, make sure you disable transactional fixtures support in Rails. Without
168
+ this change you're going to see all kinds of weird data visibility problems in your tests.
169
+
170
+
171
+ == Using Models in Master-Slave Environments
172
+
173
+ Master-slave replication is the most popular scale-out technique in a medium-sized and
174
+ large database-centric applications today. There are some rails plugins out there that help
175
+ developers to use slave servers in their models but none of them were flexible enough
176
+ for us to start using them in a huge application we work on.
177
+
178
+ So, we've been using ActsAsReadonlyable plugin for a long time and have made tons
179
+ of changes in its code over the time. But since that plugin has been abandoned
180
+ by its authors, we've decided to collect all of our master-slave code in one plugin
181
+ and release it for rails 2.2+. +DbCharmer+ adds the following features to Rails models:
182
+
183
+
184
+ === Auto-Switching all Reads to the Slave(s)
185
+
186
+ When you create a model, you could use <tt>db_magic :slave => :blah</tt> or
187
+ <tt>db_magic :slaves => [ :foo, :bar ]</tt> commands in your model to set up reads
188
+ redirection mode when all your find/count/exist/etc methods will be reading data
189
+ from your slave (or a bunch of slaves in a round-robin manner). Here is an example:
190
+
191
+ class Foo < ActiveRecord::Base
192
+ db_magic :slave => :slave01
193
+ end
194
+
195
+ class Bar < ActiveRecord::Base
196
+ db_magic :slaves => [ :slave01, :slave02 ]
197
+ end
198
+
199
+
200
+ === Default Connection Switching
201
+
202
+ If you have more than one master-slave cluster (or simply more than one database)
203
+ in your database environment, then you might want to change the default database
204
+ connection of some of your models. You could do that by using
205
+ <tt>db_magic :connection => :foo</tt> call from your models. Example:
206
+
207
+ class Foo < ActiveRecord::Base
208
+ db_magic :connection => :foo
209
+ end
210
+
211
+ Sample model on a separate master-slave cluster (so, separate main connection +
212
+ a slave connection):
213
+
214
+ class Bar < ActiveRecord::Base
215
+ db_magic :connection => :bar, :slave => :bar_slave
216
+ end
217
+
218
+ === Per-Query Connection Management
219
+
220
+ Sometimes you have select queries that you know you want to run on the master.
221
+ This could happen for example when you have just added some data and need to read
222
+ it back and not sure if it made it all the way to the slave yet or no. For this
223
+ situation and a few others there is a set of methods we've added to ActiveRecord models:
224
+
225
+ 1) +on_master+ - this method could be used in two forms: block form and proxy form.
226
+ In the block form you could force connection switch for a block of code:
227
+
228
+ User.on_master do
229
+ user = User.find_by_login('foo')
230
+ user.update_attributes!(:activated => true)
231
+ end
232
+
233
+ In the proxy form this method could be used to force one query to be performed on
234
+ the master database server:
235
+
236
+ Comment.on_master.last(:limit => 5)
237
+ User.on_master.find_by_activation_code(code)
238
+ User.on_master.exists?(:login => login, :password => password)
239
+
240
+ 2) +on_slave+ - this method is used to force a query to be run on a slave even in
241
+ situations when it's been previously forced to use the master. If there is more
242
+ than one slave, one would be selected randomly. Tis method has two forms as
243
+ well: block and proxy.
244
+
245
+ 3) <tt>on_db(connection)</tt> - this method is what makes two previous methods
246
+ possible. It is used to switch a model's connection to some db for a short block
247
+ of code or even for one statement (two forms). It accepts the same range of values
248
+ as the +switch_connection_to+ method does. Example:
249
+
250
+ Comment.on_db(:olap).count
251
+ Post.on_db(:foo).find(:first)
252
+
253
+ By default in development and test environments you could use non-existing connections in your
254
+ <tt>on_db</tt> calls and rails would send all your queries to a single default database. In
255
+ production <tt>on_db</tt> won't accept non-existing names.
256
+
257
+ This behaviour is controlled by the <tt>DbCharmer.connections_should_exist</tt>
258
+ configuration attribute which could be set from a rails initializer.
259
+
260
+
261
+ === Forced Slave Reads
262
+
263
+ In some cases we could have models that are too important to be used in default "send all
264
+ reads to the slave" mode, but we still would like to be able to switch them to this mode
265
+ sometimes. For example, you could have +User+ model, which you would like to keep from
266
+ lagging with your slaves because users do not like to see outdated information about their
267
+ accounts. But in some cases (like logged-out profile page views, etc) it would be perfectly
268
+ reasonable to switch all reads to the slave.
269
+
270
+ For this use-case starting with +DbCharmer+ release 1.7.0 we have a feature called forced
271
+ slave reads. It consists of a few separate small features that together make it really
272
+ powerful:
273
+
274
+ 1) <tt>:force_slave_reads => false</tt> option for +ActiveRecord+'s <tt>db_magic</tt> method.
275
+ This option could be used to disable automated slave reads on your models so that you could
276
+ call <tt>on_slave</tt> or use other methods to enable slave reads when you need it. Example:
277
+
278
+ class User < ActiveRecord::Base
279
+ db_magic :slave => slave01, :forced_slave_reads => false
280
+ end
281
+
282
+ 2) <tt>force_slave_reads</tt> +ActionController+ class method. This method could be used to
283
+ enable per-controller (when called with no arguments), or per-action (<tt>:only</tt> and
284
+ <tt>:except</tt> params) forced reads from slaves. This is really useful for actions in
285
+ which you know you could tolerate some slave lag so all your models with slaves defined will
286
+ send their reads to slaves. Example:
287
+
288
+ class ProfilesController < Application
289
+ force_slave_reads :except => [ :login, :logout ]
290
+ ...
291
+ end
292
+
293
+ 3) <tt>force_slave_reads!</tt> +ActionController+ instance method, that could be used within
294
+ your actions or in controller filters to temporarily switch your models to forced slave reads
295
+ mode. This method could be useful for cases when the same actions could be called by logged-in
296
+ and anonymous users. Then you could authorize users in <tt>before_filter</tt> and call
297
+ <tt>force_slave_reads!</tt> method for anonymous page views.
298
+
299
+ class ProfilesController < Application
300
+ before_filter do
301
+ force_slave_reads! unless current_user
302
+ end
303
+ ...
304
+ end
305
+
306
+ Notice: Before using this method you need to enable +ActionController+ support in +DbCharmer+.
307
+ You need to call <tt>DbCharmer.enable_controller_magic!</tt> method from your project
308
+ initialization code.
309
+
310
+ 4) <tt>DbCharmer.force_slave_reads</tt> method that could be used with a block of ruby code
311
+ and would enable forced slave reads mode until the end of the block execution. This is really
312
+ powerful feature allowing high granularity in your control of forced slave reads mode. Example:
313
+
314
+ DbCharmer.force_slave_reads do
315
+ ...
316
+ total_users = User.count
317
+ ...
318
+ end
319
+
320
+ Notice: At this point the feature considered beta and should be used with caution. It is fully covered
321
+ with tests, but there still could be unexpected issues when used in real-world applications.
322
+
323
+
324
+ === Associations Connection Management
325
+
326
+ ActiveRecord models can have an associations with each other and since every model has its
327
+ own database connections, it becomes pretty hard to manage connections in a chained calls
328
+ like <tt>User.posts.count</tt>. With a class-only connection switching methods this call
329
+ would look like the following if we'd want to count posts on a separate database:
330
+
331
+ Post.on_db(:olap) { User.posts.count }
332
+
333
+ Apparently this is not the best way to write the code and we've implemented an <tt>on_*</tt>
334
+ methods on associations as well so you could do things like this:
335
+
336
+ @user.posts.on_db(:olap).count
337
+ @user.posts.on_slave.find(:title => 'Hello, world!')
338
+
339
+ Notice: Since ActiveRecord associations implemented as proxies for resulting
340
+ objects/collections, it is possible to use our connection switching methods even without
341
+ chained methods:
342
+
343
+ @post.user.on_slave - would return post's author
344
+ @photo.owner.on_slave - would return photo's owner
345
+
346
+
347
+ Starting with +DbCharmer+ release 1.4 it is possible to use prefix notation for has_many
348
+ and HABTM associations connection switching:
349
+
350
+ @user.on_db(:foo).posts
351
+ @user.on_slave.posts
352
+
353
+
354
+ === Named Scopes Support
355
+
356
+ To make it easier for +DbCharmer+ users to use connections switching methods with named scopes,
357
+ we've added <tt>on_*</tt> methods support on the scopes as well. All the following scope chains
358
+ would do exactly the same way (the query would be executed on the :foo database connection):
359
+
360
+ Post.on_db(:foo).published.with_comments.spam_marked.count
361
+ Post.published.on_db(:foo).with_comments.spam_marked.count
362
+ Post.published.with_comments.on_db(:foo).spam_marked.count
363
+ Post.published.with_comments.spam_marked.on_db(:foo).count
364
+
365
+ And now, add this feature to our associations support and here is what we could do:
366
+
367
+ @user.on_db(:archive).posts.published.all
368
+ @user.posts.on_db(:olap).published.count
369
+ @user.posts.published.on_db(:foo).first
370
+
371
+
372
+ === Bulk Connection Management
373
+
374
+ Sometimes you want to run code where a large number of tables may be used, and you'd like
375
+ them all to use an alternate database. You can now do this:
376
+
377
+ DbCharmer.with_remapped_databases(:logs => :big_logs_slave) { ... }
378
+
379
+ Any model whose default database is +:logs+ (e.g., <tt>db_charmer :connection => :logs</tt>)
380
+ will now have its connection switched to +:big_logs_slave+ in that block. This is lower
381
+ precedence than any other +DbCharmer+ method, so <tt>Model.on_db(:foo).find(...)</tt> and
382
+ such things will still use the database they specify, not the one that model was remapped
383
+ to.
384
+
385
+ You can specify any number of remappings at once, and you can also use +:master+ as a database
386
+ name that matches any model that has not had its connection set by +DbCharmer+ at all.
387
+
388
+ *Note*: +DbCharmer+ works via +alias_method_chain+ in model classes. It is very careful
389
+ to only patch the models it needs to. However, if you use +with_remapped_databases+ and
390
+ remap the default database (+:master+), then it has no choice but to patch all subclasses
391
+ of +ActiveRecord::Base+. This should not cause any serious problems or any big performance
392
+ impact, but it is worth noting.
393
+
394
+
395
+ == Simple Sharding Support
396
+
397
+ Starting with the release 1.6.0 of +DbCharmer+ we have added support for simple database sharding
398
+ to our ActiveRecord extensions. Even though this feature is tested in production, we do not recommend
399
+ using it in your applications without complete understanding of the principles of its work.
400
+
401
+ At this point we support four sharding methods:
402
+
403
+ 1) +range+ - really simple sharding method that allows you to take a table, slice is to a set of
404
+ smaller tables with pre-defined ranges of primary keys and then put those smaller tables to
405
+ different databases/servers. This could be useful for situations where you have a huge table that
406
+ is slowly growing and you just want to keep it simple and split the table load into a few servers
407
+ without building any complex sharding schemes.
408
+
409
+ 2) +hash_map+ - pretty simple sharding method that allows you to take a table and slice it to a set
410
+ of smaller tables by some key that has a pre-defined key of values. For example, list of US mailing
411
+ addresses could be sharded by states, where you'd be able to define which states are stored in which
412
+ databases/servers.
413
+
414
+ 3) +db_block_map+ - this is a really complex sharding method that allows you to shard your table into a
415
+ set of small fixed-size blocks that then would be assigned to a set of shards (databases/servers).
416
+ Whenever you would need an additional blocks they would be allocated automatically and then balanced
417
+ across the shards you have defined in your database. This method could be used to scale out huge
418
+ tables with hundreds of millions to billions of rows and allows relatively easy re-sharding techniques
419
+ to be implemented on top.
420
+
421
+ 4) +db_block_group_map+ - really similar to the +db_block_map+ method with a single difference: this method
422
+ allows you to have a set of databases (table groups) on each server and every group would be handled as a
423
+ separate shard of data. This approach is really useful for pre-sharding of your data before scaling your
424
+ application out. You can easily start with one server, having 10-20-50 separate databases, and then
425
+ move those databases to different servers as you see your database outgrow one machine.
426
+
427
+
428
+ === How to enable sharding?
429
+
430
+ To enable sharding extensions you need to take a few things:
431
+
432
+ 1) Create a Rails initializer (on run this code when you initialize your script/application) with a
433
+ set of sharded connections defined. Each connection would have a name, sharding method and an optional
434
+ set of parameters to initialize the sharding method of your choice.
435
+
436
+ 2) Specify sharding connection you want to use in your models.
437
+
438
+ 3) Specify the shard you want to use before doing any operations on your models.
439
+
440
+ For more details please check out the following documentation sections.
441
+
442
+
443
+ === Sharded Connections
444
+
445
+ Sharded connection is a simple abstractions that allows you to specify all sharding parameters for a
446
+ cluster in one place and then use this centralized configuration in your models. Here are a few examples
447
+ of sharded connections initizlization calls:
448
+
449
+ 1) Sample range-based sharded connection:
450
+
451
+ TEXTS_SHARDING_RANGES = {
452
+ 0...100 => :shard1,
453
+ 100..200 => :shard2,
454
+ :default => :shard3
455
+ }
456
+
457
+ DbCharmer::Sharding.register_connection(
458
+ :name => :texts,
459
+ :method => :range,
460
+ :ranges => TEXTS_SHARDING_RANGES
461
+ )
462
+
463
+ 2) Sample hash map sharded connection:
464
+
465
+ SHARDING_MAP = {
466
+ 'US' => :us_users,
467
+ 'CA' => :ca_users,
468
+ :default => :other_users
469
+ }
470
+
471
+ DbCharmer::Sharding.register_connection(
472
+ :name => :users,
473
+ :method => :hash_map,
474
+ :map => SHARDING_MAP
475
+ )
476
+
477
+ 3) Sample database block map sharded connection:
478
+
479
+ DbCharmer::Sharding.register_connection(
480
+ :name => :social,
481
+ :method => :db_block_map,
482
+ :block_size => 10000, # Number of keys per block
483
+ :map_table => :event_shards_map, # Table with blocks to shards mapping
484
+ :shards_table => :event_shards_info, # Shards connection information table
485
+ :connection => :social_shard_info # What connection to use to read the map
486
+ )
487
+
488
+ After your sharded connection is defined, you could use it in your models:
489
+
490
+ class Text < ActiveRecord::Base
491
+ db_magic :sharded => {
492
+ :key => :id,
493
+ :sharded_connection => :texts
494
+ }
495
+ end
496
+
497
+ class Event < ActiveRecord::Base
498
+ set_table_name :timeline_events
499
+
500
+ db_magic :sharded => {
501
+ :key => :to_uid,
502
+ :sharded_connection => :social
503
+ }
504
+ end
505
+
506
+
507
+ === Switching connections in sharded models
508
+
509
+ Every time you need to perform an operation on a sharded model, you need to specify on which shard
510
+ you want to do it. We have a method for this which would look familiar for the people that use
511
+ +DbCharmer+ for non-sharded environments since it looks and works just like those per-query
512
+ connection management methods:
513
+
514
+ Event.shard_for(10).find(:conditions => { :to_uid => 123 }, :limit => 5)
515
+ Text.shard_for(123).find_by_id(123)
516
+
517
+ There is another method that could be used with range and hash_map sharding methods, this method
518
+ allows you to switch to the default shard:
519
+
520
+ Text.on_default_shard.create(:body => 'hello', :user_id => 123)
521
+
522
+ And finally, there is a method that allows you to run your code on each shard in the system (at this
523
+ point the method is supported in db_block_map and db_block_group_map methods only):
524
+
525
+ Event.on_each_shard { |event| event.delete_all }
526
+
527
+
528
+ === Defining your own sharding methods
529
+
530
+ It is possible with +DbCharmer+ for the users to define their own sharding methods. You need to do a
531
+ few things to implement your very own sharding scheme:
532
+
533
+ 1) Create a class with a name <tt>DbCharmer::Sharding::Method::YourOwnName</tt>
534
+
535
+ 2) Implement at least a constructor <tt>initialize(config)</tt> and a lookup instance
536
+ method <tt>shard_for_key(key)</tt> that would return either a connection name from <tt>database.yml</tt>
537
+ file or just a hash of connection parameters for rails connection adapters.
538
+
539
+ 3) Register your sharded connection using the following call:
540
+
541
+ DbCharmer::Sharding.register_connection(
542
+ :name => :some_name,
543
+ :method => :your_own_name, # your sharding method class name in lower case
544
+ ... some additional parameters if needed ...
545
+ )
546
+
547
+ 4) Use your sharded connection as any standard one.
548
+
549
+
550
+ === Adding support for default shards in your custom sharding methods
551
+
552
+ If you want to be able to use +on_default_shard+ method on your custom-sharded models, you
553
+ need to do two things:
554
+
555
+ 1) implement <tt>support_default_shard?</tt> instance method on your sharded class that
556
+ would return +true+ if you do support default shard specification and +false+ otherwise.
557
+
558
+ 2) implement <tt>:default</tt> symbol support as a key in your +shard_for_key+ method.
559
+
560
+
561
+ === Adding support for shards enumeration in your custom sharding methods
562
+
563
+ To add shards enumeration support to your custom-sharded models you need to implement
564
+ an instance method +shard_connections+ on your class. This method should return an array of
565
+ sharding connection names or connection configurations to be used to establish connections in a loop.
566
+
567
+
568
+ == Documentation/Questions
569
+
570
+ For more information about the library, please visit our site at http://kovyrin.github.com/db-charmer.
571
+ If you need more defails on DbCharmer internals, please check out the source code. All the plugin's
572
+ code is ~100% covered with tests (located in a separate staging rails project
573
+ at http://github.com/kovyrin/db-charmer-sandbox). The project has unit tests for all or at
574
+ least the most actively used code paths.
575
+
576
+ If you have any questions regarding this project, you could contact the author using
577
+ the DbCharmer Users Group mailing list:
578
+
579
+ - Group Info: http://groups.google.com/group/db-charmer
580
+ - Subscribe using the info page or by sending an email to mailto:db-charmer-subscribe@googlegroups.com
581
+
582
+
583
+ == What Ruby and Rails implementations does it work for?
584
+
585
+ We have a continuous integration setup for this gem on with Rails 2.2, 2.3 and 3.0 using a few
586
+ different versions of Ruby. For more information about the build matrix please visit
587
+ http://github.com/kovyrin/db-charmer-sandbox web site.
588
+
589
+ In addition to CI testing, we use this gem in production on Scribd.com (one of the largest RoR
590
+ sites in the world) with Ruby Enterprise Edition and Rails 2.2, Rails 2.3, Sinatra and plain
591
+ Rack applications.
592
+
593
+ Starting with version 1.7.0 we support Rails 3.0, but until further notice we consider it a
594
+ beta-quality feature since we do not run any production software on this version of Rails. If you
595
+ run your application on Rails 3.0, please contact the author.
596
+
597
+
598
+ == Who are the authors?
599
+
600
+ This plugin has been created in Scribd.com for our internal use and then the sources were opened for
601
+ other people to use. Most of the code in this package has been developed by Oleksiy Kovyrin for
602
+ Scribd.com and is released under the MIT license. For more details, see the LICENSE file.
603
+
604
+ Other contributors who have helped with the development of this library are (alphabetically ordered):
605
+ * Allen Madsen
606
+ * Andrew Geweke
607
+ * Ashley Martens
608
+ * Dmytro Shteflyuk
609
+ * Eric Lindvall
610
+ * Gregory Man
611
+ * Michael Birk
612
+ * Tyler McMullen