mongo-oplogreplayer 0.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/.gitignore ADDED
@@ -0,0 +1,18 @@
1
+ *.gem
2
+ *.rbc
3
+ .bundle
4
+ .config
5
+ .yardoc
6
+ *.iml
7
+ Gemfile.lock
8
+ InstalledFiles
9
+ _yardoc
10
+ coverage
11
+ doc/
12
+ lib/bundler/man
13
+ pkg
14
+ rdoc
15
+ spec/reports
16
+ test/tmp
17
+ test/version_tmp
18
+ tmp
data/Gemfile ADDED
@@ -0,0 +1,4 @@
1
+ source 'https://rubygems.org'
2
+
3
+ # Specify your gem's dependencies in oplogreplayer.gemspec
4
+ gemspec
data/LICENSE.txt ADDED
@@ -0,0 +1,22 @@
1
+ Copyright (c) 2014 brettcave
2
+
3
+ MIT License
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining
6
+ a copy of this software and associated documentation files (the
7
+ "Software"), to deal in the Software without restriction, including
8
+ without limitation the rights to use, copy, modify, merge, publish,
9
+ distribute, sublicense, and/or sell copies of the Software, and to
10
+ permit persons to whom the Software is furnished to do so, subject to
11
+ the following conditions:
12
+
13
+ The above copyright notice and this permission notice shall be
14
+ included in all copies or substantial portions of the Software.
15
+
16
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
17
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
18
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
19
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
20
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
21
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
22
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,63 @@
1
+ # Oplogreplayer
2
+
3
+ Oplogreplayer connects to a replica set and monitors write operations by monitoring the oplog, replaying them
4
+ onto another replica set / mongo instance.
5
+
6
+ ## Installation
7
+
8
+ Add this line to your application's Gemfile:
9
+
10
+ gem 'oplogreplayer'
11
+
12
+ And then execute:
13
+
14
+ $ bundle
15
+
16
+ Or install it yourself as:
17
+
18
+ $ gem install oplogreplayer
19
+
20
+ ## Configuration
21
+
22
+ Here is a sample configuration file that can be used.
23
+
24
+ resume: true
25
+
26
+ source:
27
+ replicaSet: rs_name
28
+ host: "rs_host_1:27017,rs_host_2:27017"
29
+ username: rsUser
30
+ password: rsPass
31
+
32
+ dest:
33
+ host: "localhost:27017"
34
+ onlyDbs: "foo,bar"
35
+
36
+ The example above shows how use a replicaset with authentication enabled as a source, and a single instance with no
37
+ authentication as a target.
38
+
39
+ `resume` - determines if the oplog replay is to be resumed from the last known point. A timestamp is stored at `dest` in local.oplog.status that is used for resumes.
40
+
41
+ `source` and `dest` are the "from" and "to" servers. The source should be a replicaset (for the oplog). The example above shows all the used keys. The following affects how the connection to mongo is made, in both source and dest
42
+
43
+ * If there is no username and password, then authentication is disabled
44
+ * the presence of `replicaSet` in the config is used to determine whether to use a regular client or replica set client. Removing `replicaSet` but leaving a comma-seperated list of hosts may produce unpredictable results.
45
+
46
+ `onlyDbs` will enable filtering on the replay, so that only the databases in the list will have operations executed on them.
47
+
48
+ ## Usage
49
+
50
+ `oplogreplayer mongo2mongo -c path/to/oplogConf.yaml`
51
+
52
+ ### Options
53
+
54
+ `-t / --timestamp` - Specifies a timestamp to resume from. Note that this resume, regardless of configuration file setting, and it will overwrite the persisted timestamp for future resumes.
55
+
56
+
57
+ ## Contributing
58
+
59
+ 1. Fork it ( http://github.com/brettcave/oplogreplayer/fork )
60
+ 2. Create your feature branch (`git checkout -b my-new-feature`)
61
+ 3. Commit your changes (`git commit -am 'Add some feature'`)
62
+ 4. Push to the branch (`git push origin my-new-feature`)
63
+ 5. Create new Pull Request
data/Rakefile ADDED
@@ -0,0 +1 @@
1
+ require "bundler/gem_tasks"
data/bin/oplogreplayer ADDED
@@ -0,0 +1,4 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require 'oplogreplayer/cli'
4
+ Oplogreplayer::CLI.start
@@ -0,0 +1,9 @@
1
+ Feature: Replayer
2
+ In order to replay the oplog
3
+ As a CLI
4
+ I want to provide a mechanism to tail and replay the oplog
5
+
6
+ Scenario: Mongo to mongo replay
7
+ When I run `oplogreplayer mongo2mongo --test -c ../../sampleConf.yaml`
8
+ Then the output should contain "Test mode enabled"
9
+
@@ -0,0 +1 @@
1
+ require 'aruba/cucumber'
@@ -0,0 +1,14 @@
1
+ require 'thor'
2
+ require 'oplogreplayer'
3
+
4
+ module Oplogreplayer
5
+ class CLI < Thor
6
+ desc "mongo2mongo", "Replays the oplog from 1 mongo replica set to another instance or RS."
7
+ option :config, :type => :string, :required => true, :aliases => "-c", :desc => "The configuration for oplog source"
8
+ option :timestamp, :type => :numeric, :aliases => "-t"
9
+ def mongo2mongo()
10
+ Oplogreplayer::Replayer.m2m(options)
11
+ end
12
+
13
+ end
14
+ end
@@ -0,0 +1,10 @@
1
+ module Oplogreplayer
2
+ module Logging
3
+ def log
4
+ @@logger ||= Log4r::Logger.new("Oplogreplayer")
5
+ end
6
+ end
7
+
8
+ class NoConfigFileError < StandardError; end
9
+
10
+ end
@@ -0,0 +1,126 @@
1
+ require 'mongoriver'
2
+
3
+ module Oplogreplayer
4
+ class Mongobridge < Mongoriver::AbstractOutlet
5
+
6
+ @_mongoClient
7
+ @log
8
+ @filterDbs = nil
9
+
10
+
11
+ def initialize(config)
12
+
13
+ setupLogger
14
+
15
+ @log.info("Mongo bridge being configured")
16
+
17
+ connect_uri = "mongodb://"
18
+ connect_uri += "#{config["username"]}:#{config["password"]}@" if config["username"] and config["password"]
19
+ connect_uri += "#{config["host"]}"
20
+ connect_uri += "?replicaSet=#{config["replicaSet"]}" if config["replicaSet"]
21
+
22
+ if config["replicaSet"]
23
+ @_mongoClient = Mongo::MongoReplicaSetClient.from_uri(connect_uri)
24
+ @log.info("Target connection configured. Target is a replica set.")
25
+ else
26
+ @_mongoClient = Mongo::MongoClient.from_uri(connect_uri)
27
+ @log.info("Target connection configured. Target is a single instance.")
28
+ end
29
+
30
+ if config["onlyDbs"]
31
+ @filterDbs = config["onlyDbs"].split(",")
32
+ @log.info("Filter Dbs has been configured: #{@filterDbs.to_s}")
33
+ end
34
+ end
35
+
36
+ def getOplogTimestamp()
37
+ findStamp = @_mongoClient.db("local").collection("oplog.tracker").find_one()
38
+ if findStamp
39
+ @log.debug("Stamp found: #{findStamp.inspect}")
40
+ # We return timestamp+1 as "timestamp" was already replayed before shutdown.
41
+ findStamp["timestamp"]+1
42
+ else
43
+ @log.debug("No stamp found. That should mean full replay.")
44
+ nil
45
+ end
46
+ end
47
+
48
+ # This is potentially bad - a find and update for every oplog....
49
+ def update_optime(timestamp)
50
+ # track what's been written with this - write optime to the fs, perhaps periodically.
51
+ @log.debug("Optime: #{timestamp}")
52
+ @_mongoClient.db("local").collection("oplog.tracker").update({}, {'timestamp' => timestamp}, {:upsert => true})
53
+ end
54
+
55
+ def insert(db_name, collection_name, document)
56
+ if @filterDbs and @filterDbs.include? db_name
57
+ @log.debug("insert #{db_name}.#{collection_name} : #{document.inspect}")
58
+ @_mongoClient.db(db_name).collection(collection_name).insert(document)
59
+ end
60
+ end
61
+
62
+ def remove(db_name,collection_name, document)
63
+ if @filterDbs and @filterDbs.include? db_name
64
+ @log.debug("remove #{db_name}.#{collection_name} : #{document.inspect}")
65
+ @_mongoClient.db(db_name).collection(collection_name).remove(document)
66
+ end
67
+ end
68
+
69
+ def update(db_name, collection_name,selector,update)
70
+ if @filterDbs and @filterDbs.include? db_name
71
+ @log.debug("update #{db_name}.#{collection_name} : #{selector} : #{update}")
72
+ @_mongoClient.db(db_name).collection(collection_name).update(selector,update)
73
+ end
74
+ end
75
+
76
+
77
+ def create_index(db_name, collection_name, index_key, options)
78
+ if @filterDbs and @filterDbs.include? db_name
79
+ @log.debug("create_index #{db_name}.#{collection_name} : #{index_key} : #{options}")
80
+ @_mongoClient.db(db_name).collection(collection_name).create_index(index_key,options)
81
+ end
82
+ end
83
+
84
+ def drop_index(db_name, collection_name, index_name)
85
+ if @filterDbs and @filterDbs.include? db_name
86
+ @log.debug("drop_index #{db_name}.#{collection_name} : #{index_name}")
87
+ @_mongoClient.db(db_name).collection(collection_name).drop_index(index_name)
88
+ end
89
+ end
90
+
91
+ def create_collection(db_name, collection_name, options)
92
+ if @filterDbs and @filterDbs.include? db_name
93
+ @log.debug("create_collection #{db_name} : #{collection_name} : #{options}")
94
+ @_mongoClient.db(db_name).create_collection(collection_name,options)
95
+ end
96
+ end
97
+ def drop_collection(db_name, collection_name)
98
+ if @filterDbs and @filterDbs.include? db_name
99
+ @log.debug("drop_collection #{db_name} : #{collection_name}")
100
+ @_mongoClient.db(db_name).drop_collection(collection_name)
101
+ end
102
+ end
103
+
104
+ def rename_collection(db_name, old_collection_name, new_collection_name)
105
+ if @filterDbs and @filterDbs.include? db_name
106
+ @log.debug("rename_collection #{db_name} : from #{old_collection_name} to #{new_collection_name}")
107
+ @_mongoClient.db(db_name).rename_collection(old_collection_name,new_collection_name)
108
+ end
109
+ end
110
+
111
+ def drop_database(db_name)
112
+ if @filterDbs and @filterDbs.include? db_name
113
+ @log.debug("drop_database #{db_name}")
114
+ @_mongoClient.drop_database(db_name)
115
+ end
116
+ end
117
+
118
+ private
119
+
120
+ def setupLogger()
121
+ @log = Log4r::Logger.new("Oplogreplay::Mongobridge")
122
+ @log.outputters = Log4r::StdoutOutputter.new(STDERR)
123
+ @log.level = Log4r::INFO
124
+ end
125
+ end
126
+ end
@@ -0,0 +1,71 @@
1
+ require 'mongoriver'
2
+ require 'oplogreplayer/mongobridge'
3
+ require 'yaml'
4
+
5
+ module Oplogreplayer
6
+ class Replayer
7
+
8
+ @_config = {}
9
+
10
+ def self.m2m(options)
11
+
12
+ config = options[:config]
13
+ timestamp = options[:timestamp]
14
+
15
+ log = Log4r::Logger.new('Oplogreplayer::Replayer')
16
+ log.outputters = Log4r::StdoutOutputter.new(STDERR)
17
+ log.level = Log4r::INFO
18
+ self.parseConfig(@_config, config)
19
+
20
+ sourceConfig = @_config["source"]
21
+
22
+ connect_uri = "mongodb://"
23
+ connect_uri += "#{sourceConfig["username"]}:#{sourceConfig["password"]}@" if sourceConfig["username"] and sourceConfig["password"]
24
+ connect_uri += "#{sourceConfig["host"]}"
25
+ connect_uri += "?replicaSet=#{sourceConfig["replicaSet"]}"
26
+
27
+ log.info("Setting up client for source")
28
+ rsClient = Mongo::MongoReplicaSetClient.from_uri(connect_uri)
29
+
30
+ log.info("Configuring tailer")
31
+ tailer = Mongoriver::Tailer.new([rsClient], :existing)
32
+
33
+ log.info("Creating mongo2mongo bridge")
34
+ bridge = Oplogreplayer::Mongobridge.new(@_config["dest"])
35
+
36
+ log.info("Creating a stream between tailer and bridge")
37
+ stream = Mongoriver::Stream.new(tailer, bridge)
38
+
39
+ # If a timestamp is supplied as an argument, override.
40
+ if timestamp
41
+ log.info("Replaying. Timestamp provided, overriding and starting at #{timestamp}")
42
+ stream.run_forever(timestamp)
43
+ elsif @_config["resume"]
44
+ log.info("Replaying. No timestamp provided but resume is enabled, will resume based on target's last timestamp")
45
+ # otherwise, try and resume.
46
+ # We need persistence of the oplog. Not sure whether to use local fs or destination mongo
47
+ # destination mongo seems better suited though. I'm going to use the local db for now.
48
+ stream.run_forever(bridge.getOplogTimestamp)
49
+ else
50
+ # No timestamp and no resume.... we're doing a full replay.
51
+ log.info("Replaying. No resume and no timestamp - full oplog replay.")
52
+ stream.run_forever()
53
+ end
54
+ end
55
+
56
+ def self.parseConfig(target, configFile)
57
+ if ! ::File.exists?(configFile)
58
+ raise NoConfigFileError, "Config file not found: #{configFile}"
59
+ end
60
+
61
+ conf = YAML::load_file(configFile)
62
+ target.merge! conf
63
+ end
64
+
65
+ def self.setupLogging()
66
+
67
+ end
68
+
69
+ end
70
+
71
+ end
@@ -0,0 +1,3 @@
1
+ module Oplogreplayer
2
+ VERSION = "0.0.1"
3
+ end
@@ -0,0 +1,9 @@
1
+ require "mongo"
2
+ require "log4r"
3
+
4
+ module Oplogreplayer
5
+ end
6
+
7
+ require "oplogreplayer/version"
8
+ require "oplogreplayer/log"
9
+ require "oplogreplayer/replayer"
@@ -0,0 +1,30 @@
1
+ # coding: utf-8
2
+ lib = File.expand_path('../lib', __FILE__)
3
+ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
+ require 'oplogreplayer/version'
5
+
6
+ Gem::Specification.new do |spec|
7
+ spec.name = "mongo-oplogreplayer"
8
+ spec.version = Oplogreplayer::VERSION
9
+ spec.authors = ["brettcave"]
10
+ spec.email = ["brett@cave.za.net"]
11
+ spec.summary = %q{Replays the oplog from a mongo replica set}
12
+ spec.description = %q{Reads the oplog from a replica set and replays it to another mongo instance / rs.}
13
+ spec.homepage = "https://github.com/brettcave/mongo-oplogreplay"
14
+ spec.license = "MIT"
15
+
16
+ spec.files = `git ls-files -z`.split("\x0")
17
+ spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
18
+ spec.test_files = spec.files.grep(%r{^(test|spec|features)/})
19
+ spec.require_paths = ["lib"]
20
+
21
+ spec.add_development_dependency "bundler", "~> 1.5"
22
+ spec.add_development_dependency "rake", "~> 10.3"
23
+ spec.add_development_dependency "rspec", "~> 2.6"
24
+ spec.add_development_dependency "cucumber"
25
+ spec.add_development_dependency "aruba"
26
+
27
+ spec.add_dependency "mongoriver", "~> 0.3"
28
+ spec.add_dependency "mongo", "~> 1.10"
29
+ spec.add_dependency "thor", "~> 0.18"
30
+ end
@@ -0,0 +1,7 @@
1
+ require 'oplogreplayer'
2
+
3
+ describe Oplogreplayer::Replayer do
4
+ it "replays oplog" do
5
+ # TODO: test functionality
6
+ end
7
+ end
metadata ADDED
@@ -0,0 +1,196 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: mongo-oplogreplayer
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.0.1
5
+ prerelease:
6
+ platform: ruby
7
+ authors:
8
+ - brettcave
9
+ autorequire:
10
+ bindir: bin
11
+ cert_chain: []
12
+ date: 2014-07-10 00:00:00.000000000 Z
13
+ dependencies:
14
+ - !ruby/object:Gem::Dependency
15
+ name: bundler
16
+ requirement: !ruby/object:Gem::Requirement
17
+ none: false
18
+ requirements:
19
+ - - ~>
20
+ - !ruby/object:Gem::Version
21
+ version: '1.5'
22
+ type: :development
23
+ prerelease: false
24
+ version_requirements: !ruby/object:Gem::Requirement
25
+ none: false
26
+ requirements:
27
+ - - ~>
28
+ - !ruby/object:Gem::Version
29
+ version: '1.5'
30
+ - !ruby/object:Gem::Dependency
31
+ name: rake
32
+ requirement: !ruby/object:Gem::Requirement
33
+ none: false
34
+ requirements:
35
+ - - ~>
36
+ - !ruby/object:Gem::Version
37
+ version: '10.3'
38
+ type: :development
39
+ prerelease: false
40
+ version_requirements: !ruby/object:Gem::Requirement
41
+ none: false
42
+ requirements:
43
+ - - ~>
44
+ - !ruby/object:Gem::Version
45
+ version: '10.3'
46
+ - !ruby/object:Gem::Dependency
47
+ name: rspec
48
+ requirement: !ruby/object:Gem::Requirement
49
+ none: false
50
+ requirements:
51
+ - - ~>
52
+ - !ruby/object:Gem::Version
53
+ version: '2.6'
54
+ type: :development
55
+ prerelease: false
56
+ version_requirements: !ruby/object:Gem::Requirement
57
+ none: false
58
+ requirements:
59
+ - - ~>
60
+ - !ruby/object:Gem::Version
61
+ version: '2.6'
62
+ - !ruby/object:Gem::Dependency
63
+ name: cucumber
64
+ requirement: !ruby/object:Gem::Requirement
65
+ none: false
66
+ requirements:
67
+ - - ! '>='
68
+ - !ruby/object:Gem::Version
69
+ version: '0'
70
+ type: :development
71
+ prerelease: false
72
+ version_requirements: !ruby/object:Gem::Requirement
73
+ none: false
74
+ requirements:
75
+ - - ! '>='
76
+ - !ruby/object:Gem::Version
77
+ version: '0'
78
+ - !ruby/object:Gem::Dependency
79
+ name: aruba
80
+ requirement: !ruby/object:Gem::Requirement
81
+ none: false
82
+ requirements:
83
+ - - ! '>='
84
+ - !ruby/object:Gem::Version
85
+ version: '0'
86
+ type: :development
87
+ prerelease: false
88
+ version_requirements: !ruby/object:Gem::Requirement
89
+ none: false
90
+ requirements:
91
+ - - ! '>='
92
+ - !ruby/object:Gem::Version
93
+ version: '0'
94
+ - !ruby/object:Gem::Dependency
95
+ name: mongoriver
96
+ requirement: !ruby/object:Gem::Requirement
97
+ none: false
98
+ requirements:
99
+ - - ~>
100
+ - !ruby/object:Gem::Version
101
+ version: '0.3'
102
+ type: :runtime
103
+ prerelease: false
104
+ version_requirements: !ruby/object:Gem::Requirement
105
+ none: false
106
+ requirements:
107
+ - - ~>
108
+ - !ruby/object:Gem::Version
109
+ version: '0.3'
110
+ - !ruby/object:Gem::Dependency
111
+ name: mongo
112
+ requirement: !ruby/object:Gem::Requirement
113
+ none: false
114
+ requirements:
115
+ - - ~>
116
+ - !ruby/object:Gem::Version
117
+ version: '1.10'
118
+ type: :runtime
119
+ prerelease: false
120
+ version_requirements: !ruby/object:Gem::Requirement
121
+ none: false
122
+ requirements:
123
+ - - ~>
124
+ - !ruby/object:Gem::Version
125
+ version: '1.10'
126
+ - !ruby/object:Gem::Dependency
127
+ name: thor
128
+ requirement: !ruby/object:Gem::Requirement
129
+ none: false
130
+ requirements:
131
+ - - ~>
132
+ - !ruby/object:Gem::Version
133
+ version: '0.18'
134
+ type: :runtime
135
+ prerelease: false
136
+ version_requirements: !ruby/object:Gem::Requirement
137
+ none: false
138
+ requirements:
139
+ - - ~>
140
+ - !ruby/object:Gem::Version
141
+ version: '0.18'
142
+ description: Reads the oplog from a replica set and replays it to another mongo instance
143
+ / rs.
144
+ email:
145
+ - brett@cave.za.net
146
+ executables:
147
+ - oplogreplayer
148
+ extensions: []
149
+ extra_rdoc_files: []
150
+ files:
151
+ - .gitignore
152
+ - Gemfile
153
+ - LICENSE.txt
154
+ - README.md
155
+ - Rakefile
156
+ - bin/oplogreplayer
157
+ - features/replayer.feature
158
+ - features/support/setup.rb
159
+ - lib/oplogreplayer.rb
160
+ - lib/oplogreplayer/cli.rb
161
+ - lib/oplogreplayer/log.rb
162
+ - lib/oplogreplayer/mongobridge.rb
163
+ - lib/oplogreplayer/replayer.rb
164
+ - lib/oplogreplayer/version.rb
165
+ - oplogreplayer.gemspec
166
+ - spec/oplogreplayer_spec.rb
167
+ homepage: https://github.com/brettcave/mongo-oplogreplay
168
+ licenses:
169
+ - MIT
170
+ post_install_message:
171
+ rdoc_options: []
172
+ require_paths:
173
+ - lib
174
+ required_ruby_version: !ruby/object:Gem::Requirement
175
+ none: false
176
+ requirements:
177
+ - - ! '>='
178
+ - !ruby/object:Gem::Version
179
+ version: '0'
180
+ required_rubygems_version: !ruby/object:Gem::Requirement
181
+ none: false
182
+ requirements:
183
+ - - ! '>='
184
+ - !ruby/object:Gem::Version
185
+ version: '0'
186
+ requirements: []
187
+ rubyforge_project:
188
+ rubygems_version: 1.8.24
189
+ signing_key:
190
+ specification_version: 3
191
+ summary: Replays the oplog from a mongo replica set
192
+ test_files:
193
+ - features/replayer.feature
194
+ - features/support/setup.rb
195
+ - spec/oplogreplayer_spec.rb
196
+ has_rdoc: