replicate 1.2 → 1.3
Sign up to get free protection for your applications and to get access to all the features.
- data/README.md +129 -57
- data/bin/replicate +20 -8
- data/lib/replicate/dumper.rb +7 -0
- data/lib/replicate/status.rb +4 -1
- data/test/dumper_test.rb +10 -0
- data/test/dumpscript.rb +1 -0
- metadata +36 -58
data/README.md
CHANGED
@@ -12,30 +12,90 @@ concepts.
|
|
12
12
|
Synopsis
|
13
13
|
--------
|
14
14
|
|
15
|
-
Installing
|
15
|
+
### Installing
|
16
16
|
|
17
17
|
$ gem install replicate
|
18
18
|
|
19
|
-
Dumping objects
|
19
|
+
### Dumping objects
|
20
20
|
|
21
|
-
|
22
|
-
==> dumped 4 total objects:
|
23
|
-
Profile 1
|
24
|
-
User 1
|
25
|
-
UserEmail 2
|
26
|
-
|
27
|
-
Loading objects:
|
21
|
+
Evaluate a Ruby expression, dumping all resulting to standard output:
|
28
22
|
|
29
|
-
$ replicate -r config/environment -
|
30
|
-
==>
|
23
|
+
$ replicate -r ./config/environment -d "User.find(1)" > user.dump
|
24
|
+
==> dumped 4 total objects:
|
31
25
|
Profile 1
|
32
26
|
User 1
|
33
27
|
UserEmail 2
|
34
28
|
|
35
|
-
|
29
|
+
The `-r ./config/environment` option is used to require environment setup and
|
30
|
+
model instantiation code needed by the ruby expression.
|
31
|
+
|
32
|
+
### Dumping many objects with a dump script
|
33
|
+
|
34
|
+
Dump scripts are normal ruby source files evaluated in the context of the
|
35
|
+
dumper. The `dump(object)` method is used to put objects into the dump stream.
|
36
|
+
|
37
|
+
```ruby
|
38
|
+
# config/replicate/dump-stuff.rb
|
39
|
+
require 'config/environment'
|
40
|
+
|
41
|
+
%w[rtomayko/tilt rtomayko/bcat].each do |repo_name|
|
42
|
+
repo = Repository.find_by_name_with_owner(repo_name)
|
43
|
+
dump repo
|
44
|
+
dump repo.commit_comments
|
45
|
+
dump repo.issues
|
46
|
+
end
|
47
|
+
```
|
48
|
+
|
49
|
+
Run the dump script:
|
50
|
+
|
51
|
+
$ replicate -d config/replicate/dump-stuff.rb > repos.dump
|
52
|
+
==> dumped 1479 total objects:
|
53
|
+
AR::Habtm 101
|
54
|
+
CommitComment 95
|
55
|
+
Issue 101
|
56
|
+
IssueComment 427
|
57
|
+
IssueEvent 308
|
58
|
+
Label 5
|
59
|
+
Language 19
|
60
|
+
LanguageName 1
|
61
|
+
Milestone 3
|
62
|
+
Organization 4
|
63
|
+
Profile 82
|
64
|
+
PullRequest 44
|
65
|
+
PullRequestReviewComment 8
|
66
|
+
Repository 20
|
67
|
+
Team 4
|
68
|
+
TeamMember 6
|
69
|
+
User 89
|
70
|
+
UserEmail 162
|
71
|
+
|
72
|
+
### Loading many objects:
|
73
|
+
|
74
|
+
$ replicate -r ./config/environment -l < repos.dump
|
75
|
+
==> loaded 1479 total objects:
|
76
|
+
AR::Habtm 101
|
77
|
+
CommitComment 95
|
78
|
+
Issue 101
|
79
|
+
IssueComment 427
|
80
|
+
IssueEvent 308
|
81
|
+
Label 5
|
82
|
+
Language 19
|
83
|
+
LanguageName 1
|
84
|
+
Milestone 3
|
85
|
+
Organization 4
|
86
|
+
Profile 82
|
87
|
+
PullRequest 44
|
88
|
+
PullRequestReviewComment 8
|
89
|
+
Repository 20
|
90
|
+
Team 4
|
91
|
+
TeamMember 6
|
92
|
+
User 89
|
93
|
+
UserEmail 162
|
94
|
+
|
95
|
+
### Dumping and loading over ssh
|
36
96
|
|
37
97
|
$ remote_command="replicate -r /app/config/environment -d 'User.find(1234)'"
|
38
|
-
$ ssh example.org "$remote_command" |replicate -r config/environment -l
|
98
|
+
$ ssh example.org "$remote_command" |replicate -r ./config/environment -l
|
39
99
|
|
40
100
|
ActiveRecord
|
41
101
|
------------
|
@@ -47,8 +107,10 @@ MRI 1.8.7 as well as under MRI 1.9.2.
|
|
47
107
|
To use customization macros in your models, require the replicate library after
|
48
108
|
ActiveRecord (in e.g., `config/initializers/libraries.rb`):
|
49
109
|
|
50
|
-
|
51
|
-
|
110
|
+
```ruby
|
111
|
+
require 'active_record'
|
112
|
+
require 'replicate'
|
113
|
+
```
|
52
114
|
|
53
115
|
ActiveRecord support works sensibly without customization so this isn't strictly
|
54
116
|
necessary to use the `replicate` command. The following sections document the
|
@@ -66,12 +128,14 @@ database being sucked in. It can be useful to mark specific associations for
|
|
66
128
|
automatic inclusion using the `replicate_associations` macro. For instance,
|
67
129
|
to always include `EmailAddress` records belonging to a `User`:
|
68
130
|
|
69
|
-
|
70
|
-
|
71
|
-
|
131
|
+
```ruby
|
132
|
+
class User < ActiveRecord::Base
|
133
|
+
belongs_to :profile
|
134
|
+
has_many :email_addresses
|
72
135
|
|
73
|
-
|
74
|
-
|
136
|
+
replicate_associations :email_addresses
|
137
|
+
end
|
138
|
+
```
|
75
139
|
|
76
140
|
### Natural Keys
|
77
141
|
|
@@ -81,18 +145,20 @@ exists with matching attributes. To update existing records instead of
|
|
81
145
|
creating new ones, define a natural key for the model using the `replicate_natural_key`
|
82
146
|
macro:
|
83
147
|
|
84
|
-
|
85
|
-
|
86
|
-
|
148
|
+
```ruby
|
149
|
+
class User < ActiveRecord::Base
|
150
|
+
belongs_to :profile
|
151
|
+
has_many :email_addresses
|
87
152
|
|
88
|
-
|
89
|
-
|
90
|
-
|
153
|
+
replicate_natural_key :login
|
154
|
+
replicate_associations :email_addresses
|
155
|
+
end
|
91
156
|
|
92
|
-
|
93
|
-
|
94
|
-
|
95
|
-
|
157
|
+
class EmailAddress < ActiveRecord::Base
|
158
|
+
belongs_to :user
|
159
|
+
replicate_natural_key :user_id, :email
|
160
|
+
end
|
161
|
+
```
|
96
162
|
|
97
163
|
Multiple attribute names may be specified to define a compound key. Foreign key
|
98
164
|
column attributes (`user_id`) are often included in natural keys.
|
@@ -109,14 +175,16 @@ instance, you might want to create files on disk or load information into a
|
|
109
175
|
separate data store any time an object enters the database. The best way to go
|
110
176
|
about this currently is to override the model's `load_replicant` class method:
|
111
177
|
|
112
|
-
|
113
|
-
|
114
|
-
|
115
|
-
|
116
|
-
|
117
|
-
|
118
|
-
|
119
|
-
|
178
|
+
```ruby
|
179
|
+
class User < ActiveRecord::Base
|
180
|
+
def self.load_replicant(type, id, attrs)
|
181
|
+
id, object = super
|
182
|
+
object.register_in_redis
|
183
|
+
object.some_other_callback
|
184
|
+
[id, object]
|
185
|
+
end
|
186
|
+
end
|
187
|
+
```
|
120
188
|
|
121
189
|
This interface will be improved in future versions.
|
122
190
|
|
@@ -132,15 +200,17 @@ The dump side calls `#dump_replicant(dumper)` on each object. The method must
|
|
132
200
|
call `dumper.write()` with the class name, id, and hash of primitively typed
|
133
201
|
attributes for the object:
|
134
202
|
|
135
|
-
|
136
|
-
|
137
|
-
|
203
|
+
```ruby
|
204
|
+
class User
|
205
|
+
attr_reader :id
|
206
|
+
attr_accessor :name, :email
|
138
207
|
|
139
|
-
|
140
|
-
|
141
|
-
|
142
|
-
|
143
|
-
|
208
|
+
def dump_replicant(dumper)
|
209
|
+
attributes { 'name' => name, 'email' => email }
|
210
|
+
dumper.write self.class, id, attributes
|
211
|
+
end
|
212
|
+
end
|
213
|
+
```
|
144
214
|
|
145
215
|
### load_replicant
|
146
216
|
|
@@ -148,15 +218,17 @@ The load side calls `::load_replicant(type, id, attributes)` on the class to
|
|
148
218
|
load each object into the current environment. The method must return an
|
149
219
|
`[id, object]` tuple:
|
150
220
|
|
151
|
-
|
152
|
-
|
153
|
-
|
154
|
-
|
155
|
-
|
156
|
-
|
157
|
-
|
158
|
-
|
159
|
-
|
221
|
+
```ruby
|
222
|
+
class User
|
223
|
+
def self.load_replicant(type, id, attributes)
|
224
|
+
user = User.new
|
225
|
+
user.name = attributes['name']
|
226
|
+
user.email = attributes['email']
|
227
|
+
user.save!
|
228
|
+
[user.id, user]
|
229
|
+
end
|
230
|
+
end
|
231
|
+
```
|
160
232
|
|
161
233
|
How it works
|
162
234
|
------------
|
@@ -169,7 +241,7 @@ combined, `attributes` must consist of only string keys and simply typed values.
|
|
169
241
|
Relationships between objects in the stream are managed as follows:
|
170
242
|
|
171
243
|
- An object's attributes may encode references to objects that precede it
|
172
|
-
in the stream using a simple tuple format: [:id, 'User', 1234]
|
244
|
+
in the stream using a simple tuple format: `[:id, 'User', 1234]`.
|
173
245
|
|
174
246
|
- The dump side ensures that objects are written to the dump stream in
|
175
247
|
"reference order" such that when an object A includes a reference attribute
|
@@ -177,7 +249,7 @@ Relationships between objects in the stream are managed as follows:
|
|
177
249
|
|
178
250
|
- The load side maintains a mapping of ids from the dumping system to the newly
|
179
251
|
replicated objects on the loading system. When the loader encounters a
|
180
|
-
reference value [:id, 'User', 1234] in an object's attributes, it converts it
|
252
|
+
reference value `[:id, 'User', 1234]` in an object's attributes, it converts it
|
181
253
|
to the load side id value.
|
182
254
|
|
183
255
|
Dumping and loading happens in a streaming fashion. There is no limit on the
|
data/bin/replicate
CHANGED
@@ -1,9 +1,15 @@
|
|
1
1
|
#!/usr/bin/env ruby
|
2
|
-
#/
|
3
|
-
#/
|
4
|
-
#/
|
5
|
-
#/
|
6
|
-
|
2
|
+
#/ Usage: replicate --dump dumpscript.rb > objects.dump
|
3
|
+
#/ replicate [-r <lib>] --dump "<ruby>" > objects.dump
|
4
|
+
#/ replicate [-r <lib>] --load < objects.dump
|
5
|
+
#/ Dump and load objects between environments.
|
6
|
+
#
|
7
|
+
#/ The --dump form writes to stdout the objects dumped by the script or
|
8
|
+
#/ ruby expression(s) given. Dump scripts are normal Ruby source files but
|
9
|
+
#/ must call dump(object) on one or more objects. When a Ruby expression is
|
10
|
+
#/ given, all resulting objects are dumped automatically.
|
11
|
+
#/
|
12
|
+
#/ The --load form reads dump data from stdin and creates objects under the
|
7
13
|
#/ current environment.
|
8
14
|
#/
|
9
15
|
#/ Mode selection:
|
@@ -46,7 +52,7 @@ end
|
|
46
52
|
# load replicate lib and setup AR
|
47
53
|
require 'replicate'
|
48
54
|
if defined?(ActiveRecord::Base)
|
49
|
-
|
55
|
+
require 'replicate/active_record'
|
50
56
|
ActiveRecord::Base.replicate_id = keep_id
|
51
57
|
ActiveRecord::Base.connection.enable_query_cache!
|
52
58
|
end
|
@@ -55,11 +61,17 @@ end
|
|
55
61
|
# stdout. the database should not be modified at all by this operation.
|
56
62
|
if mode == :dump
|
57
63
|
usage.call if ARGV.empty? || ARGV[0].empty?
|
58
|
-
objects = eval(ARGV[0])
|
59
64
|
Replicate::Dumper.new do |dumper|
|
60
65
|
dumper.marshal_to out
|
61
66
|
dumper.log_to $stderr, verbose, quiet
|
62
|
-
|
67
|
+
ARGV.each do |code|
|
68
|
+
if File.exist?(code)
|
69
|
+
dumper.load_script code
|
70
|
+
else
|
71
|
+
objects = dumper.instance_eval(code, '<argv>', 0)
|
72
|
+
dumper.dump objects
|
73
|
+
end
|
74
|
+
end
|
63
75
|
end
|
64
76
|
|
65
77
|
# load mode means we're reading objects from stdin and creating them under
|
data/lib/replicate/dumper.rb
CHANGED
@@ -49,6 +49,13 @@ module Replicate
|
|
49
49
|
use Replicate::Status, 'dump', out, verbose, quiet
|
50
50
|
end
|
51
51
|
|
52
|
+
# Load a dump script. This just evals the source of the file in the context
|
53
|
+
# of the dumper. Dump scripts are useful when you want to dump a lot of
|
54
|
+
# stuff.
|
55
|
+
def load_script(file)
|
56
|
+
instance_eval File.read(file), file, 0
|
57
|
+
end
|
58
|
+
|
52
59
|
# Dump one or more objects to the internal array or provided dump
|
53
60
|
# stream. This method guarantees that the same object will not be dumped
|
54
61
|
# more than once.
|
data/lib/replicate/status.rb
CHANGED
@@ -27,7 +27,10 @@ module Replicate
|
|
27
27
|
end
|
28
28
|
|
29
29
|
def normal_log(type, id, attrs, object)
|
30
|
-
|
30
|
+
message = " %sing: %4d objects" % [@prefix, @count]
|
31
|
+
dots = '.' * (@count % 50)
|
32
|
+
dots = ' ' * 50 if dots.empty?
|
33
|
+
@out.write "#{message} #{dots}\r"
|
31
34
|
end
|
32
35
|
|
33
36
|
def complete
|
data/test/dumper_test.rb
CHANGED
@@ -69,4 +69,14 @@ class DumperTest < Test::Unit::TestCase
|
|
69
69
|
end
|
70
70
|
assert called
|
71
71
|
end
|
72
|
+
|
73
|
+
def test_loading_dump_scripts
|
74
|
+
called = false
|
75
|
+
@dumper.listen do |type, id, attrs, obj|
|
76
|
+
assert !called
|
77
|
+
called = true
|
78
|
+
end
|
79
|
+
@dumper.load_script File.expand_path('../dumpscript.rb', __FILE__)
|
80
|
+
assert called
|
81
|
+
end
|
72
82
|
end
|
data/test/dumpscript.rb
ADDED
@@ -0,0 +1 @@
|
|
1
|
+
dump Replicate::Object.new('message' => 'hello')
|
metadata
CHANGED
@@ -1,59 +1,45 @@
|
|
1
|
-
--- !ruby/object:Gem::Specification
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
2
|
name: replicate
|
3
|
-
version: !ruby/object:Gem::Version
|
4
|
-
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
version: '1.3'
|
5
5
|
prerelease:
|
6
|
-
segments:
|
7
|
-
- 1
|
8
|
-
- 2
|
9
|
-
version: "1.2"
|
10
6
|
platform: ruby
|
11
|
-
authors:
|
7
|
+
authors:
|
12
8
|
- Ryan Tomayko
|
13
9
|
autorequire:
|
14
10
|
bindir: bin
|
15
11
|
cert_chain: []
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
- !ruby/object:Gem::Dependency
|
12
|
+
date: 2011-09-10 00:00:00.000000000Z
|
13
|
+
dependencies:
|
14
|
+
- !ruby/object:Gem::Dependency
|
20
15
|
name: activerecord
|
21
|
-
|
22
|
-
requirement: &id001 !ruby/object:Gem::Requirement
|
16
|
+
requirement: &70344083570000 !ruby/object:Gem::Requirement
|
23
17
|
none: false
|
24
|
-
requirements:
|
18
|
+
requirements:
|
25
19
|
- - ~>
|
26
|
-
- !ruby/object:Gem::Version
|
27
|
-
|
28
|
-
segments:
|
29
|
-
- 3
|
30
|
-
- 1
|
31
|
-
version: "3.1"
|
20
|
+
- !ruby/object:Gem::Version
|
21
|
+
version: '3.1'
|
32
22
|
type: :development
|
33
|
-
version_requirements: *id001
|
34
|
-
- !ruby/object:Gem::Dependency
|
35
|
-
name: sqlite3
|
36
23
|
prerelease: false
|
37
|
-
|
24
|
+
version_requirements: *70344083570000
|
25
|
+
- !ruby/object:Gem::Dependency
|
26
|
+
name: sqlite3
|
27
|
+
requirement: &70344083569500 !ruby/object:Gem::Requirement
|
38
28
|
none: false
|
39
|
-
requirements:
|
40
|
-
- -
|
41
|
-
- !ruby/object:Gem::Version
|
42
|
-
|
43
|
-
segments:
|
44
|
-
- 0
|
45
|
-
version: "0"
|
29
|
+
requirements:
|
30
|
+
- - ! '>='
|
31
|
+
- !ruby/object:Gem::Version
|
32
|
+
version: '0'
|
46
33
|
type: :development
|
47
|
-
|
34
|
+
prerelease: false
|
35
|
+
version_requirements: *70344083569500
|
48
36
|
description: Dump and load relational objects between Ruby environments.
|
49
37
|
email: ryan@github.com
|
50
|
-
executables:
|
38
|
+
executables:
|
51
39
|
- replicate
|
52
40
|
extensions: []
|
53
|
-
|
54
41
|
extra_rdoc_files: []
|
55
|
-
|
56
|
-
files:
|
42
|
+
files:
|
57
43
|
- COPYING
|
58
44
|
- HACKING
|
59
45
|
- README.md
|
@@ -68,42 +54,34 @@ files:
|
|
68
54
|
- lib/replicate/status.rb
|
69
55
|
- test/active_record_test.rb
|
70
56
|
- test/dumper_test.rb
|
57
|
+
- test/dumpscript.rb
|
71
58
|
- test/loader_test.rb
|
72
59
|
- test/replicate_test.rb
|
73
60
|
homepage: http://github.com/rtomayko/replicate
|
74
61
|
licenses: []
|
75
|
-
|
76
62
|
post_install_message:
|
77
63
|
rdoc_options: []
|
78
|
-
|
79
|
-
require_paths:
|
64
|
+
require_paths:
|
80
65
|
- lib
|
81
|
-
required_ruby_version: !ruby/object:Gem::Requirement
|
66
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
82
67
|
none: false
|
83
|
-
requirements:
|
84
|
-
- -
|
85
|
-
- !ruby/object:Gem::Version
|
86
|
-
|
87
|
-
|
88
|
-
- 0
|
89
|
-
version: "0"
|
90
|
-
required_rubygems_version: !ruby/object:Gem::Requirement
|
68
|
+
requirements:
|
69
|
+
- - ! '>='
|
70
|
+
- !ruby/object:Gem::Version
|
71
|
+
version: '0'
|
72
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
91
73
|
none: false
|
92
|
-
requirements:
|
93
|
-
- -
|
94
|
-
- !ruby/object:Gem::Version
|
95
|
-
|
96
|
-
segments:
|
97
|
-
- 0
|
98
|
-
version: "0"
|
74
|
+
requirements:
|
75
|
+
- - ! '>='
|
76
|
+
- !ruby/object:Gem::Version
|
77
|
+
version: '0'
|
99
78
|
requirements: []
|
100
|
-
|
101
79
|
rubyforge_project:
|
102
80
|
rubygems_version: 1.8.6
|
103
81
|
signing_key:
|
104
82
|
specification_version: 2
|
105
83
|
summary: Dump and load relational objects between Ruby environments.
|
106
|
-
test_files:
|
84
|
+
test_files:
|
107
85
|
- test/active_record_test.rb
|
108
86
|
- test/dumper_test.rb
|
109
87
|
- test/loader_test.rb
|