mongoriver 0.3.1 → 0.4.0
Sign up to get free protection for your applications and to get access to all the features.
- data/.travis.yml +7 -0
- data/README.md +8 -1
- data/Rakefile +4 -1
- data/UPGRADING.md +35 -0
- data/bin/watch-oplog +1 -1
- data/lib/mongoriver.rb +1 -0
- data/lib/mongoriver/abstract_persistent_tailer.rb +66 -24
- data/lib/mongoriver/persistent_tailer.rb +21 -10
- data/lib/mongoriver/stream.rb +14 -26
- data/lib/mongoriver/tailer.rb +90 -11
- data/lib/mongoriver/toku.rb +123 -0
- data/lib/mongoriver/version.rb +1 -1
- data/test/test_mongoriver.rb +1 -1
- data/test/test_mongoriver_connected.rb +44 -9
- data/test/test_persistent_tailers.rb +74 -0
- data/test/test_toku.rb +127 -0
- metadata +9 -2
data/.travis.yml
ADDED
data/README.md
CHANGED
@@ -1,4 +1,4 @@
|
|
1
|
-
# Mongoriver
|
1
|
+
# Mongoriver [![Build Status](https://travis-ci.org/stripe/mongoriver.svg?branch=master)](https://travis-ci.org/stripe/mongoriver)
|
2
2
|
|
3
3
|
mongoriver is a library to monitor updates to your Mongo databases in
|
4
4
|
near-realtime. It provides a simple interface for you to take actions
|
@@ -51,3 +51,10 @@ stream.run_forever(starting_timestamp)
|
|
51
51
|
|
52
52
|
`starting_timestamp` here is the time you want the tailer to start at. We use
|
53
53
|
this to resume interrupted tailers so that no information is lost.
|
54
|
+
|
55
|
+
|
56
|
+
## Version history
|
57
|
+
|
58
|
+
### 0.4
|
59
|
+
|
60
|
+
Add support for [tokumx](http://www.tokutek.com/products/tokumx-for-mongodb/). Backwards incompatible changes to persistent tailers to accomodate that. See [UPGRADING.md](UPGRADING.md).
|
data/Rakefile
CHANGED
@@ -8,7 +8,10 @@ Rake::TestTask.new do |t|
|
|
8
8
|
end
|
9
9
|
|
10
10
|
Rake::TestTask.new(:'test-unit') do |t|
|
11
|
-
t.test_files = FileList[
|
11
|
+
t.test_files = FileList[
|
12
|
+
'test/test_mongoriver.rb',
|
13
|
+
'test/test_toku.rb',
|
14
|
+
'test/test_persistent_tailers.rb']
|
12
15
|
end
|
13
16
|
|
14
17
|
Rake::TestTask.new(:'test-connected') do |t|
|
data/UPGRADING.md
ADDED
@@ -0,0 +1,35 @@
|
|
1
|
+
# Version 0.4
|
2
|
+
|
3
|
+
## Outline
|
4
|
+
|
5
|
+
This version adds support for tailing TokuMX, which uses a different oplog format from Mongo, separating the concepts of time and ordering of oplog entries. This sadly breaks most applications currently using mongoriver and will require invasive changes to persistant tailers.
|
6
|
+
|
7
|
+
More specifically - TokuMX uses `BSON::Binary` instead of `BSON::Timestamp` for keeping track of the database oplog position.
|
8
|
+
|
9
|
+
## Changes
|
10
|
+
|
11
|
+
### Tailers
|
12
|
+
|
13
|
+
Tailers gained a few new methods: `database_type` either returns `:toku` or `:mongo` based on which type of database is connected to the tailer.
|
14
|
+
|
15
|
+
Tailers also have a `most_recent_position` method, which returns the most recent oplog position in the database. It has an optional `before_time` parameter of type `Time`, which if passed, finds the latest position before (or at) that time. It is useful for converting timestamps to positions to tail from in tailers.
|
16
|
+
|
17
|
+
### Persistent tailers
|
18
|
+
|
19
|
+
In persistent tailers, the `read_timestamp` and `write_timestamp` methods were replaced with `read_state` and `write_state`, which read/write state hashes from/to the database. `read_state` should return `nil` if not state has been stored in the database yet.
|
20
|
+
|
21
|
+
In addition, there are convenience methods `read_timestamp` and `read_position` (already implemented) which extract the appropriate field from the `read_state` results. Both will return `nil` if `read_state` returns `nil`.
|
22
|
+
|
23
|
+
## State hash
|
24
|
+
|
25
|
+
State hashes have two fields - `'time'` which is a ruby Time object (used for humans, logs) and `'position'` of type `BSON::Binary` or `BSON::Timestamp` which the tailer uses to keep track of its position.
|
26
|
+
|
27
|
+
## How to upgrade
|
28
|
+
|
29
|
+
Persistent tailers need to define both `read_state` and `write_state` methods, which deserialize and serialize state hashes according to the database type.
|
30
|
+
|
31
|
+
Also, calls to `read_timestamp` and other methods mentioned above will have to be updated.
|
32
|
+
|
33
|
+
Also, if your application has custom filters for namespaces or [other things tokumx has updated](http://www.tokutek.com/2014/03/comparing-a-tokumx-and-mongodb-oplog-entry/), those need to be updated.
|
34
|
+
|
35
|
+
Note that to do seamless upgrades, some more mangling might be needed (e.g. updating database tables, figuring out how to get the position from an existing timestamp). The following file should be a good example of what this might entail: https://github.com/stripe/mosql/blob/karl-mongoriver-support-for-tokumx/lib/mosql/tailer.rb
|
data/bin/watch-oplog
CHANGED
data/lib/mongoriver.rb
CHANGED
@@ -1,67 +1,109 @@
|
|
1
1
|
module Mongoriver
|
2
2
|
|
3
3
|
# A variant of Tailer that automatically loads and persists the
|
4
|
-
# "last
|
4
|
+
# "last position processes" state. See PersistentTailer for a
|
5
5
|
# concrete subclass that uses the same mongod you are already
|
6
6
|
# tailing.
|
7
7
|
|
8
8
|
class AbstractPersistentTailer < Tailer
|
9
|
+
attr_reader :last_saved, :last_read
|
10
|
+
|
9
11
|
def initialize(upstream, type, opts={})
|
10
12
|
raise "You can't instantiate an AbstractPersistentTailer -- did you want PersistentTailer? " if self.class == AbstractPersistentTailer
|
11
13
|
super(upstream, type)
|
12
14
|
|
13
|
-
@last_saved =
|
15
|
+
@last_saved = {}
|
14
16
|
@batch = opts[:batch]
|
15
|
-
@last_read =
|
17
|
+
@last_read = {}
|
16
18
|
end
|
17
19
|
|
18
20
|
def tail(opts={})
|
19
|
-
opts[:from] ||=
|
21
|
+
opts[:from] ||= read_position
|
22
|
+
log.debug("Persistent tail options: #{opts}")
|
20
23
|
super(opts)
|
21
24
|
end
|
22
25
|
|
23
26
|
def stream(limit=nil)
|
24
|
-
start_time =
|
27
|
+
start_time = Time.at(connection_config['localTime'])
|
25
28
|
found_entry = false
|
26
29
|
|
27
|
-
|
30
|
+
# Sketchy logic - yield results from Tailer.stream
|
31
|
+
# if nothing is found and nothing in cursor, save the current position
|
32
|
+
entries_left = super(limit) do |entry|
|
28
33
|
yield entry
|
34
|
+
|
29
35
|
found_entry = true
|
30
|
-
@last_read = entry
|
31
|
-
|
36
|
+
@last_read = state_for(entry)
|
37
|
+
maybe_save_state unless @batch
|
32
38
|
end
|
33
39
|
|
34
|
-
if !found_entry && !
|
35
|
-
@last_read = start_time
|
36
|
-
|
40
|
+
if !found_entry && !entries_left
|
41
|
+
@last_read['time'] = start_time
|
42
|
+
maybe_save_state unless @batch
|
37
43
|
end
|
38
44
|
|
39
|
-
return
|
45
|
+
return entries_left
|
46
|
+
end
|
47
|
+
|
48
|
+
# state to save to the database for this record
|
49
|
+
def state_for(record)
|
50
|
+
{
|
51
|
+
'time' => Time.at(record['ts'].seconds),
|
52
|
+
'position' => position(record)
|
53
|
+
}
|
40
54
|
end
|
41
55
|
|
42
56
|
def batch_done
|
43
57
|
raise "You must specify :batch => true to use the batch-processing interface." unless @batch
|
44
|
-
|
58
|
+
maybe_save_state
|
59
|
+
end
|
60
|
+
|
61
|
+
# Get the current state from storage. Implement this!
|
62
|
+
# @returns state [Hash, nil]
|
63
|
+
# @option state [BSON::Timestamp, BSON::Binary] 'position'
|
64
|
+
# @option state [Time] 'timestamp'
|
65
|
+
def read_state
|
66
|
+
raise "read_state unimplemented!"
|
45
67
|
end
|
46
68
|
|
69
|
+
# Read the most recent timestamp of a read from storage.
|
70
|
+
# Return nil if nothing was found.
|
47
71
|
def read_timestamp
|
48
|
-
|
72
|
+
state = read_state || {}
|
73
|
+
return state['time']
|
49
74
|
end
|
50
75
|
|
51
|
-
|
52
|
-
|
76
|
+
# Read the most recent position from storage.
|
77
|
+
# Return nil if nothing was found.
|
78
|
+
def read_position
|
79
|
+
state = read_state || {}
|
80
|
+
return state['position']
|
53
81
|
end
|
54
82
|
|
55
|
-
|
56
|
-
|
57
|
-
|
58
|
-
|
83
|
+
# Persist current state. Implement this!
|
84
|
+
# @param state [Hash]
|
85
|
+
# @option state [BSON::Timestamp, BSON::Binary] 'position'
|
86
|
+
# @option state [Time] 'timestamp'
|
87
|
+
def write_state(state)
|
88
|
+
raise "write_state unimplemented!"
|
59
89
|
end
|
60
90
|
|
61
|
-
def
|
62
|
-
|
63
|
-
|
64
|
-
|
91
|
+
def save_state(state=nil)
|
92
|
+
if state.nil?
|
93
|
+
state = last_read
|
94
|
+
end
|
95
|
+
write_state(state)
|
96
|
+
@last_saved = state
|
97
|
+
log.info("Saved state: #{last_saved}")
|
98
|
+
end
|
99
|
+
|
100
|
+
def maybe_save_state
|
101
|
+
# Write position once a minute
|
102
|
+
|
103
|
+
return unless last_read['time']
|
104
|
+
if last_saved['time'].nil? || last_read['time'] - last_saved['time'] > 60.0
|
105
|
+
save_state
|
106
|
+
end
|
65
107
|
end
|
66
108
|
end
|
67
109
|
end
|
@@ -4,7 +4,9 @@ module Mongoriver
|
|
4
4
|
# tailing.
|
5
5
|
class PersistentTailer < AbstractPersistentTailer
|
6
6
|
def initialize(upstream, type, service, opts={})
|
7
|
-
|
7
|
+
if type == :slave
|
8
|
+
raise "You can't use PersistentTailer against only a slave. How am I supposed to write state?"
|
9
|
+
end
|
8
10
|
super(upstream, type, opts)
|
9
11
|
|
10
12
|
db = opts[:db] || "_mongoriver"
|
@@ -13,18 +15,27 @@ module Mongoriver
|
|
13
15
|
@state_collection = @upstream_conn.db(db).collection(collection)
|
14
16
|
end
|
15
17
|
|
16
|
-
def
|
18
|
+
def read_state
|
17
19
|
row = @state_collection.find_one(:service => @service)
|
18
|
-
|
19
|
-
end
|
20
|
+
return nil unless row
|
20
21
|
|
21
|
-
|
22
|
-
row
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
22
|
+
# Try to do seamless upgrades from old mongoriver versions
|
23
|
+
case row['v']
|
24
|
+
when nil
|
25
|
+
log.warn("Old style timestamp found in database. Converting!")
|
26
|
+
ts = Time.at(row['timestamp'].seconds)
|
27
|
+
return {
|
28
|
+
'position' => most_recent_position(ts),
|
29
|
+
'time' => ts
|
30
|
+
}
|
31
|
+
when 1
|
32
|
+
return row['state']
|
27
33
|
end
|
28
34
|
end
|
35
|
+
|
36
|
+
def write_state(state)
|
37
|
+
@state_collection.update({:service => @service},
|
38
|
+
{:service => @service, :state => state, :v => 1}, :upsert => true)
|
39
|
+
end
|
29
40
|
end
|
30
41
|
end
|
data/lib/mongoriver/stream.rb
CHANGED
@@ -21,12 +21,14 @@ module Mongoriver
|
|
21
21
|
@stats
|
22
22
|
end
|
23
23
|
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
@tailer.
|
24
|
+
# @param position [BSON::Timestamp, BSON::Binary, Time] position to start
|
25
|
+
# following the oplog from. @see Tailer#most_recent_position
|
26
|
+
def run_forever(position=nil)
|
27
|
+
if position.is_a?(Time)
|
28
|
+
position = @tailer.most_recent_position(position)
|
29
29
|
end
|
30
|
+
log.debug("Start position: #{position.inspect}")
|
31
|
+
@tailer.tail(:from => position)
|
30
32
|
|
31
33
|
until @stop
|
32
34
|
@tailer.stream do |op|
|
@@ -42,19 +44,6 @@ module Mongoriver
|
|
42
44
|
|
43
45
|
private
|
44
46
|
|
45
|
-
def optime_from_ts(timestamp)
|
46
|
-
if timestamp.is_a?(Integer)
|
47
|
-
if timestamp >= 0
|
48
|
-
BSON::Timestamp.new(timestamp, 0)
|
49
|
-
else
|
50
|
-
raise "Invalid optime: #{timestamp}"
|
51
|
-
end
|
52
|
-
else
|
53
|
-
raise "Unrecognized type #{timestamp.class} (#{timestamp.inspect}) " \
|
54
|
-
"for start_timestamp"
|
55
|
-
end
|
56
|
-
end
|
57
|
-
|
58
47
|
def trigger(name, *args)
|
59
48
|
signature = "#{name}(" + args.map { |arg| arg.inspect }.join(', ') + ")"
|
60
49
|
log.debug("triggering #{signature}")
|
@@ -114,13 +103,13 @@ module Mongoriver
|
|
114
103
|
|
115
104
|
def handle_create_index(spec)
|
116
105
|
db_name, collection_name = parse_ns(spec['ns'])
|
117
|
-
index_key = spec['key'].map
|
106
|
+
index_key = spec['key'].map do |field, dir|
|
118
107
|
if dir.is_a?(Numeric)
|
119
108
|
[field, dir.round]
|
120
109
|
else
|
121
110
|
[field, dir]
|
122
111
|
end
|
123
|
-
|
112
|
+
end
|
124
113
|
options = {}
|
125
114
|
|
126
115
|
spec.each do |key, value|
|
@@ -147,7 +136,7 @@ module Mongoriver
|
|
147
136
|
index_name = data['index']
|
148
137
|
trigger(:drop_index, db_name, deleted_from_collection, index_name)
|
149
138
|
elsif created_collection = data['create']
|
150
|
-
handle_create_collection(db_name, data)
|
139
|
+
handle_create_collection(db_name, created_collection, data)
|
151
140
|
elsif dropped_collection = data['drop']
|
152
141
|
trigger(:drop_collection, db_name, dropped_collection)
|
153
142
|
elsif old_collection_ns = data['renameCollection']
|
@@ -161,12 +150,11 @@ module Mongoriver
|
|
161
150
|
end
|
162
151
|
end
|
163
152
|
|
164
|
-
def handle_create_collection(db_name, data)
|
165
|
-
collection_name = data.delete('create')
|
166
|
-
|
153
|
+
def handle_create_collection(db_name, collection_name, data)
|
167
154
|
options = {}
|
168
|
-
data.each do |
|
169
|
-
|
155
|
+
data.each do |key, value|
|
156
|
+
next if key == 'create'
|
157
|
+
options[key.to_sym] = (key == 'size') ? value.round : value
|
170
158
|
end
|
171
159
|
|
172
160
|
trigger(:create_collection, db_name, collection_name, options)
|
data/lib/mongoriver/tailer.rb
CHANGED
@@ -1,9 +1,11 @@
|
|
1
1
|
module Mongoriver
|
2
2
|
class Tailer
|
3
3
|
include Mongoriver::Logging
|
4
|
+
include Mongoriver::Assertions
|
4
5
|
|
5
6
|
attr_reader :upstream_conn
|
6
7
|
attr_reader :oplog
|
8
|
+
attr_reader :database_type
|
7
9
|
|
8
10
|
def initialize(upstreams, type, oplog = "oplog.rs")
|
9
11
|
@upstreams = upstreams
|
@@ -14,13 +16,53 @@ module Mongoriver
|
|
14
16
|
|
15
17
|
@cursor = nil
|
16
18
|
@stop = false
|
19
|
+
@streaming = false
|
17
20
|
|
18
21
|
connect_upstream
|
22
|
+
@database_type = Mongoriver::Toku.conversion_needed?(@upstream_conn) ? :toku : :mongo
|
19
23
|
end
|
20
24
|
|
21
|
-
|
22
|
-
|
23
|
-
|
25
|
+
# Return a position for a record object
|
26
|
+
#
|
27
|
+
# @return [BSON::Timestamp] if mongo
|
28
|
+
# @return [BSON::Binary] if tokumx
|
29
|
+
def position(record)
|
30
|
+
return nil unless record
|
31
|
+
case database_type
|
32
|
+
when :mongo
|
33
|
+
return record['ts']
|
34
|
+
when :toku
|
35
|
+
return record['_id']
|
36
|
+
end
|
37
|
+
end
|
38
|
+
|
39
|
+
# Find the most recent entry in oplog and return a position for that
|
40
|
+
# position. The position can be passed to the tail function (or run_forever)
|
41
|
+
# and the tailer will start tailing after that.
|
42
|
+
# If before_time is given, it will return the latest position before (or at) time.
|
43
|
+
def most_recent_position(before_time=nil)
|
44
|
+
position(latest_oplog_entry(before_time))
|
45
|
+
end
|
46
|
+
|
47
|
+
def latest_oplog_entry(before_time=nil)
|
48
|
+
query = {}
|
49
|
+
if before_time
|
50
|
+
case database_type
|
51
|
+
when :mongo
|
52
|
+
ts = BSON::Timestamp.new(before_time.to_i + 1, 0)
|
53
|
+
when :toku
|
54
|
+
ts = before_time + 1
|
55
|
+
end
|
56
|
+
query = { 'ts' => { '$lt' => ts } }
|
57
|
+
end
|
58
|
+
|
59
|
+
case database_type
|
60
|
+
when :mongo
|
61
|
+
record = oplog_collection.find_one(query, :sort => [['$natural', -1]])
|
62
|
+
when :toku
|
63
|
+
record = oplog_collection.find_one(query, :sort => [['_id', -1]])
|
64
|
+
end
|
65
|
+
record
|
24
66
|
end
|
25
67
|
|
26
68
|
def connect_upstream
|
@@ -71,14 +113,18 @@ module Mongoriver
|
|
71
113
|
@upstream_conn.db('local').collection(oplog)
|
72
114
|
end
|
73
115
|
|
116
|
+
# Start tailing the oplog.
|
117
|
+
#
|
118
|
+
# @param [Hash]
|
119
|
+
# @option opts [BSON::Timestamp, BSON::Binary] :from Placeholder indicating
|
120
|
+
# where to start the query from. Binary value is used for tokumx.
|
121
|
+
# The timestamp is non-inclusive.
|
122
|
+
# @option opts [Hash] :filter Extra filters for the query.
|
123
|
+
# @option opts [Bool] :dont_wait(false)
|
74
124
|
def tail(opts = {})
|
75
125
|
raise "Already tailing the oplog!" if @cursor
|
76
126
|
|
77
|
-
query = opts
|
78
|
-
if ts = opts[:from]
|
79
|
-
# Maybe if ts is old enough, just start from the beginning?
|
80
|
-
query['ts'] = { '$gte' => ts }
|
81
|
-
end
|
127
|
+
query = build_tail_query(opts)
|
82
128
|
|
83
129
|
mongo_opts = {:timeout => false}.merge(opts[:mongo_opts] || {})
|
84
130
|
|
@@ -87,7 +133,7 @@ module Mongoriver
|
|
87
133
|
oplog.add_option(Mongo::Constants::OP_QUERY_OPLOG_REPLAY) if query['ts']
|
88
134
|
oplog.add_option(Mongo::Constants::OP_QUERY_AWAIT_DATA) unless opts[:dont_wait]
|
89
135
|
|
90
|
-
log.
|
136
|
+
log.debug("Starting oplog stream from #{opts[:from] || 'start'}")
|
91
137
|
@cursor = oplog
|
92
138
|
end
|
93
139
|
end
|
@@ -98,14 +144,28 @@ module Mongoriver
|
|
98
144
|
tail(opts)
|
99
145
|
end
|
100
146
|
|
101
|
-
def
|
147
|
+
def tailing
|
148
|
+
!@stop || @streaming
|
149
|
+
end
|
150
|
+
|
151
|
+
def stream(limit=nil, &blk)
|
102
152
|
count = 0
|
153
|
+
@streaming = true
|
103
154
|
while !@stop && @cursor.has_next?
|
104
155
|
count += 1
|
105
156
|
break if limit && count >= limit
|
106
157
|
|
107
|
-
|
158
|
+
record = @cursor.next
|
159
|
+
|
160
|
+
case database_type
|
161
|
+
when :mongo
|
162
|
+
blk.call(record)
|
163
|
+
when :toku
|
164
|
+
converted = Toku.convert(record, @upstream_conn)
|
165
|
+
converted.each(&blk)
|
166
|
+
end
|
108
167
|
end
|
168
|
+
@streaming = false
|
109
169
|
|
110
170
|
return @cursor.has_next?
|
111
171
|
end
|
@@ -119,5 +179,24 @@ module Mongoriver
|
|
119
179
|
@cursor = nil
|
120
180
|
@stop = false
|
121
181
|
end
|
182
|
+
|
183
|
+
private
|
184
|
+
def build_tail_query(opts = {})
|
185
|
+
query = opts[:filter] || {}
|
186
|
+
return query unless opts[:from]
|
187
|
+
|
188
|
+
case database_type
|
189
|
+
when :mongo
|
190
|
+
assert(opts[:from].is_a?(BSON::Timestamp),
|
191
|
+
"For mongo databases, tail :from must be a BSON::Timestamp")
|
192
|
+
query['ts'] = { '$gt' => opts[:from] }
|
193
|
+
when :toku
|
194
|
+
assert(opts[:from].is_a?(BSON::Binary),
|
195
|
+
"For tokumx databases, tail :from must be a BSON::Binary")
|
196
|
+
query['_id'] = { '$gt' => opts[:from] }
|
197
|
+
end
|
198
|
+
|
199
|
+
query
|
200
|
+
end
|
122
201
|
end
|
123
202
|
end
|
@@ -0,0 +1,123 @@
|
|
1
|
+
module Mongoriver
|
2
|
+
# This module deals with converting TokuMX oplog records into mongodb oplogs.
|
3
|
+
class Toku
|
4
|
+
# @returns true if conn is a TokuMX database and the oplog records need to
|
5
|
+
# be converted
|
6
|
+
def self.conversion_needed?(conn)
|
7
|
+
conn.server_info.has_key? "tokumxVersion"
|
8
|
+
end
|
9
|
+
|
10
|
+
def self.operations_for(record, conn=nil)
|
11
|
+
if record["ops"]
|
12
|
+
return record["ops"]
|
13
|
+
end
|
14
|
+
refs_coll = conn.db('local').collection('oplog.refs')
|
15
|
+
mongo_opts = {:sort => [['seq', 1]], :timeout => false}
|
16
|
+
|
17
|
+
refs = refs_coll.find({"_id.oid" => record['ref']}, mongo_opts).to_a
|
18
|
+
refs.map { |r| r["ops"] }.flatten
|
19
|
+
end
|
20
|
+
|
21
|
+
# Convert hash representing a tokumx oplog record to mongodb oplog records.
|
22
|
+
#
|
23
|
+
# Things to note:
|
24
|
+
# 1) Unlike mongo oplog, the timestamps will not be monotonically
|
25
|
+
# increasing
|
26
|
+
# 2) h fields (unique ids) will also not be unique on multi-updates
|
27
|
+
# 3) operations marked by 'n' toku are ignored, as these are non-ops
|
28
|
+
# @see http://www.tokutek.com/2014/03/comparing-a-tokumx-and-mongodb-oplog-entry/
|
29
|
+
# @returns Array<Hash> List of mongodb oplog records.
|
30
|
+
def self.convert(record, conn=nil)
|
31
|
+
result = []
|
32
|
+
operations_for(record, conn).each do |operation|
|
33
|
+
case operation["op"]
|
34
|
+
when 'i'
|
35
|
+
result << insert_record(operation, record)
|
36
|
+
when 'ur'
|
37
|
+
result << update_record(operation, record, true)
|
38
|
+
when 'u'
|
39
|
+
result << update_record(operation, record, false)
|
40
|
+
when 'c'
|
41
|
+
result << command_record(operation, record)
|
42
|
+
when 'd'
|
43
|
+
result << remove_record(operation, record)
|
44
|
+
when 'n'
|
45
|
+
# keepOplogAlive requests - safe to ignore
|
46
|
+
else
|
47
|
+
raise "Unrecognized op: #{operation["op"]} (#{record.inspect})"
|
48
|
+
end
|
49
|
+
end
|
50
|
+
|
51
|
+
result
|
52
|
+
end
|
53
|
+
|
54
|
+
def self.timestamp(full_record)
|
55
|
+
# Note that this loses the monotonically increasing property, if not
|
56
|
+
# lost before.
|
57
|
+
BSON::Timestamp.new(full_record["ts"].to_i, 0)
|
58
|
+
end
|
59
|
+
|
60
|
+
def self.insert_record(operation, full_record)
|
61
|
+
{
|
62
|
+
"_id" => full_record["_id"],
|
63
|
+
# Monotonically increasing timestamp in mongodb in oplog.
|
64
|
+
# e.g. <BSON::Timestamp:0x000000100a81c8 @increment=1, @seconds=1408995488>
|
65
|
+
"ts" => timestamp(full_record),
|
66
|
+
# Unique ID for this operation
|
67
|
+
# Note that not so unique anymore
|
68
|
+
"h" => full_record["h"],
|
69
|
+
# Ignoring v ("version") for now
|
70
|
+
# "v" => 1,
|
71
|
+
"op" => "i",
|
72
|
+
# namespace being updated. in form of database-name.collection.name
|
73
|
+
"ns" => operation["ns"],
|
74
|
+
# operation being done.
|
75
|
+
# e.g. {"_id"=>BSON::ObjectId('53fb8f6b9e126b2106000003')}
|
76
|
+
"o" => operation["o"]
|
77
|
+
}
|
78
|
+
end
|
79
|
+
|
80
|
+
def self.remove_record(operation, full_record)
|
81
|
+
{
|
82
|
+
"_id" => full_record["_id"],
|
83
|
+
"ts" => timestamp(full_record),
|
84
|
+
"h" => full_record["h"],
|
85
|
+
# "v" => 1,
|
86
|
+
"op" => "d",
|
87
|
+
"ns" => operation["ns"],
|
88
|
+
# "b" => true, # TODO: what does this signify?
|
89
|
+
"o" => { "_id" => operation["o"]["_id"] }
|
90
|
+
}
|
91
|
+
end
|
92
|
+
|
93
|
+
def self.command_record(operation, full_record)
|
94
|
+
{
|
95
|
+
"_id" => full_record["_id"],
|
96
|
+
"ts" => timestamp(full_record),
|
97
|
+
"h" => full_record["h"],
|
98
|
+
# "v" => 1,
|
99
|
+
"op" => "c",
|
100
|
+
"ns" => operation["ns"],
|
101
|
+
"o" => operation["o"]
|
102
|
+
}
|
103
|
+
end
|
104
|
+
|
105
|
+
|
106
|
+
def self.update_record(operation, full_record, is_ur_record)
|
107
|
+
({
|
108
|
+
"_id" => full_record["_id"],
|
109
|
+
"ts" => timestamp(full_record),
|
110
|
+
"h" => full_record["h"],
|
111
|
+
# "v" => 1,
|
112
|
+
"op" => "u",
|
113
|
+
"ns" => operation["ns"],
|
114
|
+
# { _id: BSON::ObjectId } what object was updated
|
115
|
+
"o2" => { "_id" => operation["o"]["_id"] },
|
116
|
+
"o" => operation[is_ur_record ? "m" : "o2"]
|
117
|
+
})
|
118
|
+
end
|
119
|
+
|
120
|
+
private_class_method :timestamp, :insert_record, :update_record
|
121
|
+
private_class_method :remove_record, :command_record
|
122
|
+
end
|
123
|
+
end
|
data/lib/mongoriver/version.rb
CHANGED
data/test/test_mongoriver.rb
CHANGED
@@ -10,7 +10,7 @@ describe 'Mongoriver::Stream' do
|
|
10
10
|
end
|
11
11
|
|
12
12
|
before do
|
13
|
-
conn = stub(:db => nil)
|
13
|
+
conn = stub(:db => nil, :server_info => {})
|
14
14
|
@tailer = Mongoriver::Tailer.new([conn], :existing)
|
15
15
|
@outlet = Mongoriver::AbstractOutlet.new
|
16
16
|
@stream = Mongoriver::Stream.new(@tailer, @outlet)
|
@@ -16,6 +16,12 @@ def connect
|
|
16
16
|
end
|
17
17
|
end
|
18
18
|
|
19
|
+
def run_stream(stream, start)
|
20
|
+
Timeout::timeout(30) do
|
21
|
+
@stream.run_forever(start)
|
22
|
+
end
|
23
|
+
end
|
24
|
+
|
19
25
|
describe 'connected tests' do
|
20
26
|
before do
|
21
27
|
@mongo = connect
|
@@ -29,13 +35,12 @@ describe 'connected tests' do
|
|
29
35
|
|
30
36
|
@stream = Mongoriver::Stream.new(@tailer, @outlet)
|
31
37
|
|
32
|
-
@tail_from = @tailer.
|
33
|
-
sleep(1)
|
38
|
+
@tail_from = @tailer.most_recent_position
|
34
39
|
end
|
35
40
|
|
36
41
|
it 'triggers the correct ops in the correct order' do
|
37
|
-
db = '
|
38
|
-
collection = '
|
42
|
+
db = '_test_mongoriver'
|
43
|
+
collection = '_test_mongoriver'
|
39
44
|
doc = {'_id' => 'foo', 'bar' => 'baz'}
|
40
45
|
updated_doc = doc.dup.merge('bar' => 'qux')
|
41
46
|
index_keys = [['bar', 1]]
|
@@ -67,17 +72,47 @@ describe 'connected tests' do
|
|
67
72
|
@mongo[db].drop_collection(collection+'_foo')
|
68
73
|
@mongo.drop_database(db)
|
69
74
|
|
70
|
-
@stream
|
75
|
+
run_stream(@stream, @tail_from)
|
71
76
|
end
|
72
77
|
|
73
78
|
it 'passes options to create_collection' do
|
74
|
-
@outlet.expects(:create_collection).once.with('
|
79
|
+
@outlet.expects(:create_collection).once.with('_test_mongoriver', '_test_mongoriver', {:capped => true, :size => 10}) { @stream.stop }
|
75
80
|
@outlet.expects(:update_optime).at_least_once.with(anything) { @stream.stop }
|
76
81
|
|
77
|
-
@mongo['
|
78
|
-
@mongo.drop_database('
|
82
|
+
@mongo['_test_mongoriver'].create_collection('_test_mongoriver', :capped => true, :size => 10)
|
83
|
+
@mongo.drop_database('_test_mongoriver')
|
84
|
+
|
85
|
+
run_stream(@stream, @tail_from)
|
86
|
+
end
|
87
|
+
|
88
|
+
it 'ignores everything before the operation passed in' do
|
89
|
+
name = '_test_mongoriver'
|
90
|
+
|
91
|
+
@mongo[name][name].insert(:a => 5)
|
92
|
+
|
93
|
+
@outlet.expects(:insert).never
|
94
|
+
@outlet.expects(:drop_database).with(anything) { @stream.stop }
|
95
|
+
|
96
|
+
start = @tailer.most_recent_position
|
97
|
+
@mongo.drop_database(name)
|
98
|
+
run_stream(@stream, start)
|
99
|
+
end
|
100
|
+
|
101
|
+
it 'allows passing in a timestamp for the stream following as well' do
|
102
|
+
name = '_test_mongoriver2'
|
103
|
+
|
104
|
+
@outlet.expects(:insert).with do |db_name, col_name, value|
|
105
|
+
db_name != name || value['record'] == 'value'
|
106
|
+
end
|
107
|
+
|
108
|
+
@outlet.expects(:update).with do |db_name, col_name, selector, update|
|
109
|
+
@stream.stop if update['record'] == 'newvalue'
|
110
|
+
db_name != name || update['record'] == 'newvalue'
|
111
|
+
end
|
79
112
|
|
80
|
-
@
|
113
|
+
@mongo[name][name].insert('record' => 'value')
|
114
|
+
@mongo[name][name].update({'record' => 'value'}, {'record' => 'newvalue'})
|
115
|
+
run_stream(@stream, Time.now-3)
|
81
116
|
end
|
82
117
|
end
|
83
118
|
end
|
@@ -0,0 +1,74 @@
|
|
1
|
+
require 'mongoriver'
|
2
|
+
require 'mongo'
|
3
|
+
require 'minitest/autorun'
|
4
|
+
require 'mocha/setup'
|
5
|
+
|
6
|
+
|
7
|
+
def mocked_mongo()
|
8
|
+
mongo_connection = stub()
|
9
|
+
db = stub()
|
10
|
+
collection = stub()
|
11
|
+
|
12
|
+
mongo_connection.expects(:db).with('_mongoriver').returns(db)
|
13
|
+
db.expects(:collection).with('oplog-tailers').returns(collection)
|
14
|
+
|
15
|
+
# mongodb
|
16
|
+
mongo_connection.expects(:server_info).at_least_once.returns({})
|
17
|
+
|
18
|
+
[mongo_connection, collection]
|
19
|
+
end
|
20
|
+
|
21
|
+
describe 'Mongoriver::PersistentTailer' do
|
22
|
+
before do
|
23
|
+
@service_name = "_persistenttailer_test"
|
24
|
+
|
25
|
+
db, @state_collection = mocked_mongo
|
26
|
+
|
27
|
+
@tailer = Mongoriver::PersistentTailer.new(
|
28
|
+
[db], :existing, @service_name)
|
29
|
+
@state = {
|
30
|
+
'time' => Time.now,
|
31
|
+
'position' => 'foobar'
|
32
|
+
}
|
33
|
+
end
|
34
|
+
|
35
|
+
describe 'reading and storing state' do
|
36
|
+
it 'should be able to read the written state' do
|
37
|
+
|
38
|
+
@state_collection.expects(:update)
|
39
|
+
@tailer.save_state(@state)
|
40
|
+
|
41
|
+
@state_collection.expects(:find_one).returns({
|
42
|
+
'state' => @state,
|
43
|
+
'v' => 1
|
44
|
+
})
|
45
|
+
assert_equal(@state, @tailer.read_state)
|
46
|
+
end
|
47
|
+
|
48
|
+
it 'should update gracefully' do
|
49
|
+
ts = BSON::Timestamp.new(77, 0)
|
50
|
+
|
51
|
+
@state_collection.expects(:find_one).returns('timestamp' => ts)
|
52
|
+
@tailer.expects(:most_recent_position).with(Time.at(77))
|
53
|
+
|
54
|
+
assert_equal(Time.at(77), @tailer.read_state['time'])
|
55
|
+
end
|
56
|
+
end
|
57
|
+
|
58
|
+
it 'helper methods for timestamps/positions' do
|
59
|
+
@state_collection.expects(:find_one).returns({
|
60
|
+
'state' => @state,
|
61
|
+
'v' => 1
|
62
|
+
}).at_least_once
|
63
|
+
|
64
|
+
assert_equal(@state['time'], @tailer.read_timestamp)
|
65
|
+
assert_equal(@state['position'], @tailer.read_position)
|
66
|
+
end
|
67
|
+
|
68
|
+
it 'should tail from position' do
|
69
|
+
@tailer.expects(:read_position).returns('foo')
|
70
|
+
Mongoriver::Tailer.any_instance.expects(:tail).with(:from => 'foo')
|
71
|
+
|
72
|
+
@tailer.tail
|
73
|
+
end
|
74
|
+
end
|
data/test/test_toku.rb
ADDED
@@ -0,0 +1,127 @@
|
|
1
|
+
require 'mongoriver'
|
2
|
+
require 'mongo'
|
3
|
+
require 'minitest/autorun'
|
4
|
+
require 'mocha/setup'
|
5
|
+
|
6
|
+
describe 'Mongoriver::Toku' do
|
7
|
+
def create_op(ops)
|
8
|
+
ts = Time.now
|
9
|
+
ops = ops.map { |op| op['ns'] ||= 'foo.bar'; op }
|
10
|
+
{
|
11
|
+
'_id' => BSON::Binary.new,
|
12
|
+
'ts' => ts,
|
13
|
+
'h' => 1234,
|
14
|
+
'a' => true,
|
15
|
+
'ops' => ops
|
16
|
+
}
|
17
|
+
end
|
18
|
+
|
19
|
+
def convert(*ops)
|
20
|
+
record = create_op(ops)
|
21
|
+
result = Mongoriver::Toku.convert(record)
|
22
|
+
assert_equal(result.length, 1)
|
23
|
+
result.first
|
24
|
+
end
|
25
|
+
|
26
|
+
describe 'conversions sent to stream' do
|
27
|
+
before do
|
28
|
+
conn = stub(:db => nil, :server_info => {'tokumxVersion' => '1'})
|
29
|
+
@tailer = Mongoriver::Tailer.new([conn], :existing)
|
30
|
+
@outlet = Mongoriver::AbstractOutlet.new
|
31
|
+
@stream = Mongoriver::Stream.new(@tailer, @outlet)
|
32
|
+
|
33
|
+
@outlet.expects(:update_optime).at_least_once
|
34
|
+
end
|
35
|
+
|
36
|
+
it 'triggers insert' do
|
37
|
+
# collection.insert({a: 5})
|
38
|
+
@outlet.expects(:insert).once.with('foo', 'bar', {'_id' => 'baz', 'a' => 5})
|
39
|
+
@stream.send(:handle_op, convert({'op'=>'i', 'o'=>{'_id'=>'baz', 'a' => 5}}))
|
40
|
+
end
|
41
|
+
|
42
|
+
it 'triggers update (ur)' do
|
43
|
+
# collection.update({a:true}, {'$set' => {c: 7}}, multi: true)
|
44
|
+
@outlet.expects(:update).once.with('foo', 'bar', {'_id' => 'baz'}, {'$set' => {'c' => 7}})
|
45
|
+
|
46
|
+
@stream.send(:handle_op, convert({
|
47
|
+
'op' => 'ur',
|
48
|
+
'pk' => {'' => 'baz'},
|
49
|
+
'o' => {'_id' => 'baz', 'a' => true},
|
50
|
+
'm' => {'$set' => {'c' => 7}}
|
51
|
+
}))
|
52
|
+
end
|
53
|
+
|
54
|
+
it 'triggers update (u)' do
|
55
|
+
# collection.update({a:true}, {'b' => 2})
|
56
|
+
@outlet.expects(:update).once.with('foo', 'bar', {'_id' => 'baz'}, {'_id' => 'baz', 'b' => 2})
|
57
|
+
|
58
|
+
@stream.send(:handle_op, convert({
|
59
|
+
'op' => 'u',
|
60
|
+
'pk' => {'' => 'baz'},
|
61
|
+
'o' => {'_id' => 'baz', 'a' => true},
|
62
|
+
'o2' => {'_id' => 'baz', 'b' => 2}
|
63
|
+
}))
|
64
|
+
end
|
65
|
+
|
66
|
+
it 'triggers remove' do
|
67
|
+
# collection.delete({a:5})
|
68
|
+
@outlet.expects(:remove).once.with('foo', 'bar', {'_id' => 'baz'})
|
69
|
+
|
70
|
+
@stream.send(:handle_op, convert({
|
71
|
+
'op' => 'd',
|
72
|
+
'o' => {'_id' => 'baz', 'a' => 5}
|
73
|
+
}))
|
74
|
+
end
|
75
|
+
|
76
|
+
it 'triggers create_index' do
|
77
|
+
# collection.create_index([["baz", 1]])
|
78
|
+
@outlet.expects(:create_index).once.with('foo', 'bar', [['baz', 1]], {:name => 'baz_1'})
|
79
|
+
|
80
|
+
@stream.send(:handle_op, convert({
|
81
|
+
'op' => 'i',
|
82
|
+
'ns' => 'foo.system.indexes',
|
83
|
+
'o' => {'key' => {"baz" => 1}, 'ns' => 'foo.bar', 'name' => 'baz_1'}
|
84
|
+
}))
|
85
|
+
end
|
86
|
+
|
87
|
+
it 'triggers commands (create_collection)' do
|
88
|
+
# db.create_collection('bar', :capped => true, :size => 10)
|
89
|
+
@outlet.expects(:create_collection).once.with('foo', 'bar', {:capped => true, :size => 10})
|
90
|
+
|
91
|
+
@stream.send(:handle_op, convert({
|
92
|
+
'op' => 'c',
|
93
|
+
'ns' => 'foo.$cmd',
|
94
|
+
'o' => {'create' => 'bar', 'capped' => true, 'size' => 10}
|
95
|
+
}))
|
96
|
+
end
|
97
|
+
end
|
98
|
+
|
99
|
+
describe 'large transactions are joined by convert' do
|
100
|
+
it 'should yield the same result as separate ops' do
|
101
|
+
operations = [
|
102
|
+
{'op'=>'i', 'o'=>{'_id'=>'baz', 'a' => 5}},
|
103
|
+
{'op'=>'i', 'o'=>{'_id'=>'zoo', 'b' => 6}}
|
104
|
+
]
|
105
|
+
|
106
|
+
refs = [
|
107
|
+
{'_id' => {'_oid' => 'refref'}, 'ops' => [operations[0]]},
|
108
|
+
{'_id' => {'_oid' => 'refref'}, 'ops' => [operations[1]]}
|
109
|
+
]
|
110
|
+
|
111
|
+
collection = stub()
|
112
|
+
conn = stub(:db => stub(:collection => collection))
|
113
|
+
collection.expects(:find).returns(refs)
|
114
|
+
|
115
|
+
expected = Mongoriver::Toku.convert(create_op(operations))
|
116
|
+
got = Mongoriver::Toku.convert({
|
117
|
+
'_id' => BSON::Binary.new,
|
118
|
+
'ts' => Time.now,
|
119
|
+
'h' => 1234,
|
120
|
+
'a' => true,
|
121
|
+
'ref' => 'refref'
|
122
|
+
}, conn)
|
123
|
+
|
124
|
+
assert_equal(expected, got)
|
125
|
+
end
|
126
|
+
end
|
127
|
+
end
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: mongoriver
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.4.0
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2014-
|
12
|
+
date: 2014-09-04 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: mongo
|
@@ -116,10 +116,12 @@ extensions: []
|
|
116
116
|
extra_rdoc_files: []
|
117
117
|
files:
|
118
118
|
- .gitignore
|
119
|
+
- .travis.yml
|
119
120
|
- Gemfile
|
120
121
|
- LICENSE
|
121
122
|
- README.md
|
122
123
|
- Rakefile
|
124
|
+
- UPGRADING.md
|
123
125
|
- bin/watch-oplog
|
124
126
|
- lib/mongoriver.rb
|
125
127
|
- lib/mongoriver/abstract_outlet.rb
|
@@ -129,10 +131,13 @@ files:
|
|
129
131
|
- lib/mongoriver/persistent_tailer.rb
|
130
132
|
- lib/mongoriver/stream.rb
|
131
133
|
- lib/mongoriver/tailer.rb
|
134
|
+
- lib/mongoriver/toku.rb
|
132
135
|
- lib/mongoriver/version.rb
|
133
136
|
- mongoriver.gemspec
|
134
137
|
- test/test_mongoriver.rb
|
135
138
|
- test/test_mongoriver_connected.rb
|
139
|
+
- test/test_persistent_tailers.rb
|
140
|
+
- test/test_toku.rb
|
136
141
|
homepage: ''
|
137
142
|
licenses: []
|
138
143
|
post_install_message:
|
@@ -160,4 +165,6 @@ summary: monogdb oplog-tailing utilities.
|
|
160
165
|
test_files:
|
161
166
|
- test/test_mongoriver.rb
|
162
167
|
- test/test_mongoriver_connected.rb
|
168
|
+
- test/test_persistent_tailers.rb
|
169
|
+
- test/test_toku.rb
|
163
170
|
has_rdoc:
|