ruster 0.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (7) hide show
  1. checksums.yaml +7 -0
  2. data/.gitignore +2 -0
  3. data/LICENSE +21 -0
  4. data/README.md +99 -0
  5. data/bin/ruster +309 -0
  6. data/ruster.gemspec +16 -0
  7. metadata +50 -0
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 0fa83c07c41cf0cf7ffa6803b875bd04ec3a7358
4
+ data.tar.gz: 8dc5ac87df198e559cb9e98424e69850af449a46
5
+ SHA512:
6
+ metadata.gz: d3d19a0ce3e961cb6ed869cc658d5378ad94a3ad2137d7ab92d8622517dd48c21e257b84438e1203a267d74a46a9185c6342fda8c5dc78716c11b952a1b461e6
7
+ data.tar.gz: 64b72f3a6821d0ca2e3eb48d53300da5545797d5d25c99c564a24be0509cc0dbc5bd37bdb7a9d6cde368a45d733d4b3ae245bb63b334cf19d35df976df5e8924
data/.gitignore ADDED
@@ -0,0 +1,2 @@
1
+ .gs/
2
+ tmp/
data/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2014 Leandro López
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,99 @@
1
+ # ruster - a simple Redis Cluster Administration tool
2
+
3
+ Control your [Redis][redis] [Cluster][redis-cluster] from the command
4
+ line.
5
+
6
+ ## Usage
7
+
8
+ `ruster` relies on [redic][redic], the lightweight Redis client. It
9
+ currently allows to create a cluster, add and remove nodes, and
10
+ execute a command in all nodes in a cluster.
11
+
12
+ ### Create a cluster
13
+
14
+ ```
15
+ $ ruster create ip:port [ip:port...]
16
+ ```
17
+
18
+ Creates a cluster with all the indicated nodes, and automatically
19
+ shards Redis Cluster 16,384 slots evenly among all of them.
20
+
21
+ ### Add a node
22
+
23
+ ```
24
+ $ ruster add cluster_ip:port ip:port
25
+ ```
26
+
27
+ Adds `ip:port` to the cluster. `cluster_ip:port` must be one of the
28
+ nodes that are already part of the cluster.
29
+
30
+ ### Remove a node
31
+
32
+ ```
33
+ $ ruster remove cluster_ip:port ip:port
34
+ ```
35
+
36
+ Removes `ip:port` from the cluster. `cluster_ip:port` must be one of the
37
+ nodes that are already part of the cluster. The only requirement is
38
+ that `ip:port` isn't the same as `cluster_ip:port`.
39
+
40
+ **NOTE**: removing a node that has slots assigned leaves the cluster
41
+ in a broken state. These slots should be resharded before removing the
42
+ node.
43
+
44
+ ### Execute a command in all nodes
45
+
46
+ ```
47
+ $ ruster call ip:port [CMD ...]
48
+ ```
49
+
50
+ Executes the [Redis command][redis-commands] in all nodes, displaying
51
+ it's result in STDOUT.
52
+
53
+ ### Reshard
54
+
55
+ ```
56
+ $ ruster reshard cluster_ip:port slots target_ip:port source_ip:port [...]
57
+ ```
58
+
59
+ Reshards the cluster at `cluster_ip:port`, by moving `slots` slots
60
+ from several `source_ip:port` to `target_ip:port`.
61
+
62
+ ## TODO
63
+
64
+ * documentation
65
+ * resharding
66
+ * add interactive interface
67
+ * add REPL?
68
+ * fix cluster
69
+ * check cluster state
70
+ * cluster information
71
+ * ASSERTIONS
72
+
73
+ ## Thanks
74
+
75
+ This work wouldn't have been possible without [@antirez][@antirez]
76
+ awesome work on Redis, and [@soveran][@soveran] and [@cyx][@cyx] for
77
+ their super lightweight Redis client.
78
+
79
+ Thank you to my dear friends [@lucasefe][@lucasefe], [@pote][@pote]
80
+ and [@elcuervo][@cuerbot], who joined the
81
+ [conversation on Twitter][nameme] while I was looking for a name.
82
+
83
+ Also, I'd like to thank to [Eruca Sativa][eruca] and [Cirse][cirse]
84
+ for the music that's currently blasting my speakers while I write
85
+ this.
86
+
87
+ [redis]: http://redis.io/
88
+ [redis-cluster]: http://redis.io/topics/cluster-tutorial
89
+ [redic]: https://github.com/amakawa/redic
90
+ [@antirez]: https://twitter.com/antirez
91
+ [@soveran]: https://twitter.com/soveran
92
+ [@cyx]: https://twitter.com/cyx
93
+ [eruca]: https://twitter.com/ErucaSativa
94
+ [cirse]: https://twitter.com/cirsemusic
95
+ [redis-commands]: http://redis.io/commands
96
+ [@cuerbot]: https://twitter.com/cuerbot
97
+ [@pote]: https://twitter.com/poteland
98
+ [@lucasefe]: https://twitter.com/lucasefe
99
+ [nameme]: https://twitter.com/inkel/status/444638064393326592
data/bin/ruster ADDED
@@ -0,0 +1,309 @@
1
+ #! /usr/bin/env ruby
2
+
3
+ require "redic"
4
+ require "clap"
5
+
6
+ $verbose = 0
7
+ $timeout = 10_000_000
8
+
9
+ action, *args = Clap.run ARGV, {
10
+ "-v" => -> { $verbose += 1 },
11
+ "-t" => ->(ms) { $timeout = ms }
12
+ }
13
+
14
+ abort "Usage: #{File.basename($0)} <action> ip:port [...]" if action.nil? or args.nil? or args.empty?
15
+
16
+ module UI
17
+ def err msg
18
+ $stderr.puts msg
19
+ end
20
+
21
+ def abort msg, backtrace=[]
22
+ err msg
23
+ backtrace.each { |line| err line } if $verbose > 1
24
+ exit 1
25
+ end
26
+
27
+ def info *args
28
+ $stdout.puts args.join(" ")
29
+ end
30
+
31
+ def log *args
32
+ $stdout.puts args.join(" ") if $verbose > 0
33
+ end
34
+
35
+ def debug *args
36
+ $stdout.puts args.join(" ") if $verbose > 1
37
+ end
38
+
39
+ extend self
40
+ end
41
+
42
+ class Node
43
+ [:id, :ip_port, :flags, :master_id, :ping, :pong, :config, :state, :slots].each do |a|
44
+ attr a
45
+ end
46
+
47
+ def initialize(ip_port)
48
+ @ip_port = ip_port
49
+ load_info!
50
+ end
51
+
52
+ def cluster_enabled?
53
+ call("INFO", "cluster").include?("cluster_enabled:1")
54
+ end
55
+
56
+ def only_node?
57
+ call("CLUSTER", "INFO").include?("cluster_known_nodes:1")
58
+ end
59
+
60
+ def empty?
61
+ call("INFO", "keyspace").strip == "# Keyspace"
62
+ end
63
+
64
+ def load_info!
65
+ call("CLUSTER", "NODES").split("\n").each do |line|
66
+ parts = line.split
67
+ next unless parts[2].include?("myself")
68
+ set_info!(*parts)
69
+ end
70
+ end
71
+
72
+ def set_info!(id, ip_port, flags, master_id, ping, pong, config, state, *slots)
73
+ @id = id
74
+ @flags = flags.split(",")
75
+ @master_id = master_id
76
+ @ping = ping
77
+ @pong = pong
78
+ @config = config
79
+ @state = state
80
+ @slots = slots
81
+ @ip_port = ip_port unless flags.include?("myself")
82
+ end
83
+
84
+ def ip
85
+ @ip_port.split(":").first
86
+ end
87
+
88
+ def port
89
+ @ip_port.split(":").last
90
+ end
91
+
92
+ def client
93
+ @client ||= Redic.new("redis://#{@ip_port}")
94
+ end
95
+
96
+ def call(*args)
97
+ UI.debug ">", *args
98
+ client.call(*args)
99
+ end
100
+
101
+ def dead?
102
+ %w{ disconnected fail noaddr }.any? do |flag|
103
+ flags.include?(flag)
104
+ end
105
+ end
106
+
107
+ def alive?
108
+ p [ip_port, flags, state]
109
+ !dead?
110
+ end
111
+
112
+ def to_s
113
+ "#{@id} [#{@ip_port}]"
114
+ end
115
+
116
+ def slots
117
+ return @_slots if @_slots
118
+
119
+ slots = { slots: [], migrating: {}, importing: {} }
120
+
121
+ @slots.each do |data|
122
+ if data[0] == /\[(\d+)-([<>])-(\d+)\]/
123
+ if $2 == ">"
124
+ slots[:migrating][$1] = $2
125
+ else
126
+ slots[:importing][$1] = $2
127
+ end
128
+ elsif data =~ /(\d+)-(\d+)/
129
+ b, e = $1.to_i, $2.to_i
130
+ (b..e).each { |slot| slots[:slots] << slot }
131
+ else
132
+ slots[:slots] << data.to_i
133
+ end
134
+ end
135
+
136
+ @_slots = slots
137
+ end
138
+ end
139
+
140
+ class Cluster
141
+ SLOTS = 16384
142
+
143
+ def initialize(addrs)
144
+ @addrs = Array(addrs)
145
+ end
146
+
147
+ def nodes
148
+ @nodes ||= @addrs.map { |addr| Node.new(addr) }
149
+ end
150
+
151
+ def allocate_slots(node, slots)
152
+ UI.log "Allocating #{slots.size} slots (#{slots.first}..#{slots.last}) in node #{node}"
153
+
154
+ UI.debug "> CLUSTER ADDSLOTS #{slots.first}..#{slots.last}"
155
+
156
+ res = node.client.call("CLUSTER", "ADDSLOTS", *slots)
157
+
158
+ UI.abort res.message if res.is_a?(RuntimeError)
159
+ end
160
+
161
+ def add_node(node)
162
+ default = nodes.first
163
+
164
+ UI.log "Joining node #{node} to node #{default}"
165
+
166
+ ip, port = node.ip_port.split(":")
167
+
168
+ res = default.call("CLUSTER", "MEET", ip, port)
169
+
170
+ UI.abort res.message if res.is_a?(RuntimeError)
171
+ end
172
+
173
+ def create!
174
+ nodes.each do |node|
175
+ raise ArgumentError, "Redis Server at #{node.ip_port} not running in cluster mode" unless node.cluster_enabled?
176
+ raise ArgumentError, "Redis Server at #{node.ip_port} already exists in a cluster" unless node.only_node?
177
+ raise ArgumentError, "Redis Server at #{node.ip_port} is not empty" unless node.empty?
178
+ end
179
+
180
+ UI.log "Allocating #{SLOTS} slots in #{nodes.length} nodes"
181
+
182
+ available_slots = 0.upto(SLOTS - 1).each_slice((SLOTS.to_f / nodes.length).ceil)
183
+
184
+ nodes.each do |node|
185
+ slots = available_slots.next.to_a
186
+
187
+ allocate_slots(node, slots)
188
+ end
189
+
190
+ nodes.each { |node| add_node node }
191
+ end
192
+
193
+ def remove_node(node)
194
+ default = nodes.first
195
+
196
+ UI.log "Removing node #{node} from cluster"
197
+
198
+ res = default.call("CLUSTER", "FORGET", node.id)
199
+
200
+ UI.abort res.message if res.is_a?(RuntimeError)
201
+ end
202
+
203
+ def nodes!
204
+ node = nodes.sample
205
+
206
+ node.call("CLUSTER", "NODES").split("\n").map do |line|
207
+ _, ip_port, flags, _ = line.split
208
+
209
+ if flags.include?("myself")
210
+ node
211
+ else
212
+ Node.new(ip_port)
213
+ end
214
+ end
215
+ end
216
+
217
+ def call(*args)
218
+ nodes!.each do |node|
219
+ UI.info "#{node}: #{args.join(' ')}"
220
+
221
+ res = node.call(*args)
222
+ UI.info res
223
+ UI.info "--"
224
+ end
225
+ end
226
+
227
+ def reshard(target_addr, slots, sources)
228
+ target = Node.new(target_addr)
229
+
230
+ from = sources.map{ |addr| Node.new(addr) } \
231
+ .sort{ |a, b| b.slots[:slots].size <=> a.slots[:slots].size }
232
+
233
+ total_slots = from.inject(0) do |sum, source|
234
+ sum + source.slots[:slots].size
235
+ end
236
+
237
+ from.each do |source|
238
+ # Proportional number of slots, based on current assigned slots
239
+ node_slots = (slots.to_f / total_slots * source.slots[:slots].size).to_i
240
+
241
+ UI.info "Moving #{node_slots} slots from #{source} to #{target}"
242
+
243
+ source.slots[:slots].take(node_slots).each do |slot|
244
+ UI.log " Moving slot #{slot} from #{source} to #{target}"
245
+
246
+ target.call("CLUSTER", "SETSLOT", slot, "IMPORTING", source.id)
247
+ source.call("CLUSTER", "SETSLOT", slot, "MIGRATING", target.id)
248
+
249
+ done = false
250
+
251
+ until done
252
+ keys = source.call("CLUSTER", "GETKEYSINSLOT", slot, 10)
253
+
254
+ done = keys.empty?
255
+
256
+ keys.each do |key|
257
+ res = source.call("MIGRATE", target.ip, target.port, key, 0, 5000)
258
+
259
+ UI.abort res.message if res.is_a?(RuntimeError)
260
+ end
261
+ end
262
+
263
+ nodes!.each do |node|
264
+ res = node.call("CLUSTER", "SETSLOT", slot, "NODE", target.id)
265
+
266
+ UI.err res.message if res.is_a?(RuntimeError)
267
+ end
268
+ end
269
+ end
270
+ end
271
+ end
272
+
273
+ begin
274
+ case action
275
+ when "create"
276
+ cluster = Cluster.new(args)
277
+ cluster.create!
278
+ when "add"
279
+ cluster = Cluster.new(args.shift)
280
+
281
+ args.each do |addr|
282
+ cluster.add_node(Node.new(addr))
283
+ end
284
+ when "remove"
285
+ cluster = Cluster.new(args.shift)
286
+
287
+ args.each do |addr|
288
+ cluster.remove_node(Node.new(addr))
289
+ end
290
+ when "call"
291
+ cluster = Cluster.new(args.shift)
292
+
293
+ cluster.call(*args)
294
+ when "reshard"
295
+ cluster = Cluster.new(args.shift)
296
+
297
+ slots = Integer(args.shift)
298
+
299
+ target = args.shift
300
+
301
+ sources = args
302
+
303
+ cluster.reshard(target, slots, sources)
304
+ else
305
+ UI.abort "Unrecognized action #{action}"
306
+ end
307
+ rescue => ex
308
+ UI.abort ex.message, ex.backtrace
309
+ end
data/ruster.gemspec ADDED
@@ -0,0 +1,16 @@
1
+ # encoding: utf-8
2
+
3
+ Gem::Specification.new do |s|
4
+ s.name = "ruster"
5
+ s.version = "0.0.1"
6
+ s.summary = "A simple Redis Cluster Administration tool"
7
+ s.description = "Control your Redis Cluster from the command line."
8
+ s.authors = ["Leandro López"]
9
+ s.email = ["inkel.ar@gmail.com"]
10
+ s.homepage = "http://inkel.github.com/ruster"
11
+ s.license = "MIT"
12
+
13
+ s.executables.push("ruster")
14
+
15
+ s.files = `git ls-files`.split("\n")
16
+ end
metadata ADDED
@@ -0,0 +1,50 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: ruster
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.0.1
5
+ platform: ruby
6
+ authors:
7
+ - Leandro López
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2014-03-15 00:00:00.000000000 Z
12
+ dependencies: []
13
+ description: Control your Redis Cluster from the command line.
14
+ email:
15
+ - inkel.ar@gmail.com
16
+ executables:
17
+ - ruster
18
+ extensions: []
19
+ extra_rdoc_files: []
20
+ files:
21
+ - .gitignore
22
+ - LICENSE
23
+ - README.md
24
+ - bin/ruster
25
+ - ruster.gemspec
26
+ homepage: http://inkel.github.com/ruster
27
+ licenses:
28
+ - MIT
29
+ metadata: {}
30
+ post_install_message:
31
+ rdoc_options: []
32
+ require_paths:
33
+ - lib
34
+ required_ruby_version: !ruby/object:Gem::Requirement
35
+ requirements:
36
+ - - '>='
37
+ - !ruby/object:Gem::Version
38
+ version: '0'
39
+ required_rubygems_version: !ruby/object:Gem::Requirement
40
+ requirements:
41
+ - - '>='
42
+ - !ruby/object:Gem::Version
43
+ version: '0'
44
+ requirements: []
45
+ rubyforge_project:
46
+ rubygems_version: 2.0.3
47
+ signing_key:
48
+ specification_version: 4
49
+ summary: A simple Redis Cluster Administration tool
50
+ test_files: []