flare-tools 0.1.4 → 0.4.5.1

data/History.txt

@@ -1,4 +1,116 @@
-=== 0.0.1 2010-11-13
+=== 0.4.5 / 2012-10-17
+* Minor Enhancements
+  * Fixed install target of Makefile.
 
-* 1 major enhancement:
+=== 0.4.4 / 2012-08-01
+* Major Enhancements
+  * Added 'summary' subcommand
+
+=== 0.4.3 / 2012-07-04
+* Minor Enhancements
+  * Added --timeout option
+  * Fixed uptime column of 'stats' subcommand.
+
+=== 0.4.2 / 2012-05-23
+* Minor Enhancements
+  * Fixed option attributes for optparse.
+
+=== 0.4.1 / 2012-05-15
+* Minor Enhancements
+  * Added FLARE_INDEX_SERVER environment variable support.
+  * Added --cluster option to specify a cluster by name.
+
+=== 0.4.0 / 2012-05-08
+* Major Enhancements
+  * Added LICENSE file.
+
+=== 0.3.5 / 2012-05-04
+* Minor Enhancements
+  * Added --version option.
+  * Added --raw option to 'dump' subcommand.
+
+=== 0.3.4 / 2012-04-26
+* Major Enhancements
+  * Added 'restore' sub command.
+  * Fixed --bwlimit option of 'dump' subcommand.
+
+=== 0.3.3 / 2012-04-23
+* Minor Enhancements
+  * Suppressed vefiry error message of verify subcommand for each key.
+  * Added --meta option to 'verify' subcommand.
+  * Removed prebuilt debian package files.
+
+=== 0.3.2 / 2012-04-16
+* Major Enhancements
+  * Added 'verify' subcommand for checking dump results.
+* Minor Enhancements
+  * Fixed uptime column of 'stats' subcommand.
+  * Fixed 'stats' subcommand not to block for down nodes.
+
+=== 0.3.1 / 2012-02-25
+* Minor Enhancements
+  * Fixed 'threads' subcommand not to work for flare-1.0.9 or earlier.
+
+=== 0.3.0 / 2012-02-24
+* Major Enhancements
+  * Incremented the version number for internal release.
+* Minor Enhancements
+  * Fixed flare-part command to activate master nodes.
+  * Fixed node listing functions to sort by partition number.
+  * Added experimental options to flare-deploy command.
+
+=== 0.2.7 / 2012-01-20
+* Major Enhancements
+  * Added flare-keydump command for checking key distribution among nodes.
+* Minor Enhancements
+  * Added bandwidth limit function to Flare::Net::Connection class.
+
+=== 0.2.6 / 2012-01-06
+* Major Enhancements
+  * Added 'dump' subcommand for dumping data.
+  * Added 'dumpkey' subcommand for dumping keys.
+* Minor Enhancements
+  * Added experimental list operations to flare/tools/node.rb.
+
+=== 0.2.5 / 2011-12-29
+* Major Enhancements
+  * Added various options to 'deploy' subcommand.
+* Minor Enhancements
+  * Fixed 'master' subcommand to retry role change commands in case of failure.
+  * Changed default retry limit to 10 times for safety.
+  * Modified 'reconstruct' subcommand to ask before restoring a balance value.
+
+=== 0.2.4 / 2011-12-22
+* Major Enhancements
+  * Added 'activate' subcommand to flare-admin for activating a master node in ready state.
+* Minor Enhancements
+  * Removed --keep-ready option from 'master' subcommand. Now it's the default behavior.
+  * Added --activate option to activate a new master after its construction.
+
+=== 0.2.3 / 2011-11-30
+* Major Enhancements
+  * Added 'remove' subcommand to flare-admin for removing nodes from a cluster.
+* Minor Enhancements
+  * Aded --keep-ready option to 'master' subcommand, which is used for keeping 
+    the constructed master in ready state.
+  * Modified 'reconstruct' subcommand to skip when a node isn't ready for reconstruction.
+  * Added --clean option to 'slave' subcommand to rebuild database from scratch.
+  * Removed --transitive option from 'index' subcommand because it's the default behavior now.
+
+=== 0.2.2 / 2011-11-24
+* Minor Enhancements
+  * Introduced log4r.
+  * Implemented a reconnection routine of Connection class.
+  * Modified cli/reconstruct.rb to get out of confirmation skip mode when Ctrl+C is pressed.
+
+=== 0.2.1 / 2011-11-18
+* Minor Enhancements
+  * Added --all option to the 'reconstruct' subcommand to flush and reconstruct all 
+    the nodes in a cluster one by one.
+  * Improved the confirmation step of the reconstruct sumcommand.
+
+=== 0.2.0 / 2011-11-15
+
+* Major enhancements
   * Initial release
+

data/LICENSE

@@ -0,0 +1,21 @@
+The MIT License
+-------------------------------------
+Copyright (c) 2011 GREE, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/Manifest.txt

@@ -1,15 +1,72 @@
+Flare-tools.txt
 History.txt
+LICENSE
 Manifest.txt
 PostInstall.txt
-README.rdoc
+README.txt
 Rakefile
-bin/flare-partition-setting
+Tutorial.txt
+bin/flare-admin
+bin/flare-argv0
+bin/flare-deploy
+bin/flare-keychecker
+bin/flare-part
+bin/flare-ping
 bin/flare-stats
-lib/flare
-lib/flare/tools
+bin/flare-zkadmin
+lib/flare/net/connection.rb
+lib/flare/test/cluster.rb
+lib/flare/test/daemon.rb
+lib/flare/test/node.rb
 lib/flare/tools.rb
-lib/flare/tools/cli
-lib/flare/tools/cli/partition_setting.rb
+lib/flare/tools/cli.rb
+lib/flare/tools/cli/activate.rb
+lib/flare/tools/cli/balance.rb
+lib/flare/tools/cli/cli_util.rb
+lib/flare/tools/cli/deploy.rb
+lib/flare/tools/cli/down.rb
+lib/flare/tools/cli/dump.rb
+lib/flare/tools/cli/dumpkey.rb
+lib/flare/tools/cli/flare_admin.rb
+lib/flare/tools/cli/flare_argv0.rb
+lib/flare/tools/cli/flare_keychecker.rb
+lib/flare/tools/cli/flare_zkadmin.rb
+lib/flare/tools/cli/index.rb
+lib/flare/tools/cli/list.rb
+lib/flare/tools/cli/master.rb
+lib/flare/tools/cli/part.rb
+lib/flare/tools/cli/ping.rb
+lib/flare/tools/cli/reconstruct.rb
+lib/flare/tools/cli/remove.rb
+lib/flare/tools/cli/restore.rb
+lib/flare/tools/cli/slave.rb
 lib/flare/tools/cli/stats.rb
-lib/flare/tools/core.rb
-lib/flare/tools/logger.rb
+lib/flare/tools/cli/sub_command.rb
+lib/flare/tools/cli/summary.rb
+lib/flare/tools/cli/threads.rb
+lib/flare/tools/cli/verify.rb
+lib/flare/tools/client.rb
+lib/flare/tools/cluster.rb
+lib/flare/tools/common.rb
+lib/flare/tools/index_server.rb
+lib/flare/tools/node.rb
+lib/flare/tools/stats.rb
+lib/flare/tools/zk_util.rb
+lib/flare/util.rb
+lib/flare/util/bwlimit.rb
+lib/flare/util/command_line.rb
+lib/flare/util/conf.rb
+lib/flare/util/constant.rb
+lib/flare/util/conversion.rb
+lib/flare/util/default_logger.rb
+lib/flare/util/exception.rb
+lib/flare/util/filesystem.rb
+lib/flare/util/flared_conf.rb
+lib/flare/util/flarei_conf.rb
+lib/flare/util/hash_function.rb
+lib/flare/util/interruption.rb
+lib/flare/util/key_resolver.rb
+lib/flare/util/log4r_logger.rb
+lib/flare/util/logger.rb
+lib/flare/util/logging.rb
+lib/flare/util/result.rb

data/README.txt

@@ -0,0 +1,356 @@
+:title: README
+
+= flare-tools
+
+http://labs.gree.jp/Top/OpenSource/Flare-en.html
+
+Flare-tools is a collection of command line tools to maintain a flare cluster.
+
+* flare-stats
+  * flare-stats is a command used for aquring statistics of nodes in a flare cluster.
+
+* flare-admin
+  * flare-admin is a command used for maintaining your flare clusters.
+
+== DESCRIPTION:
+
+Management Tools for Flare
+
+Authors::   Kiyoshi Ikehara <kiyoshi.ikehara@gree.net>
+Copyright:: Copyright (C) GREE, Inc. 2011.
+License::   MIT-style
+
+== SYNOPSIS:
+
+=== flare-stats
+
+ $ flare-stats [options] [hostname:port] ...
+
+=== flare-admin
+
+ $ flare-admin [subcommand] [options] [arguments]
+
+== REQUIREMENTS:
+
+* Flare >= 1.0.14
+
+== INSTALL:
+
+* Install Flare
+
+Please see the web pages about {Flare}[http://labs.gree.jp/Top/OpenSource/Flare-en.html] in GREE Labs.
+
+* Install flare-tools
+ # gem install flare-tools
+
+== AUTHORS:
+
+* Kiyoshi Ikehara
+
+== USAGE:
+
+=== SPECIFYING YOUR INDEX SERVER
+
+All the commands in flare-tools have common ways to specify an index server.
+
+If your flare index server doesn't listen to the defalut port (12120), you have to specify 
+its hostname (or IP address) and port of your index node specifically.
+
+This is done by setting options, --index-server and --index-server-port.
+(other tools can also accept these two options.)
+
+ $ flare-stats --index-server=your.index.server --index-server-port=13130
+
+You can also set or FLARE_INDEX_SERVER and FLARE_INDEX_SERVER_PORT environment variables.
+These variables dosen't override the index node specified by options.
+
+ $ export FLARE_INDEX_SERVER=your.index.server
+ $ export FLARE_INDEX_SERVER_PORT=13130
+ $ flare-stats
+
+or 
+
+ $ export FLARE_INDEX_SERVER=your.index.server:13130
+ $ flare-stats
+
+=== flare-stats
+
+Flare-stats is a dedicated command line tool used for aquiring statistics of flare nodes.
+This tool shows a list of nodes in a flare cluster and their summarized information.
+
+ $ flare-stats --index-server=flare1.example.com
+   hostname:port             state   role partition balance    items conn behind hit size uptime version
+   flare1.example.com:12121 active master         0       1    10000  111      0 100   10    12d  1.0.10
+   flare2.example.com:12121 active  slave         0       1    10000  111      0 100   10    12d  1.0.10
+   flare3.example.com:12121 active master         1       1    10001  111      0 100   10    12d  1.0.10
+   flare4.example.com:12121 active  slave         1       1    10001  111      0 100   10    12d  1.0.10
+
+Flare-stats is a short-cut version of stats subcommand of flare-admin.
+Please see the stats subcommand section of flare-admin for further detail.
+
+==== flare-stats's options
+
+ Usage: flare-stats [options]
+     -h, --help                       shows this message
+     -d, --debug                      enables debug mode
+     -w, --warn                       turns on warnings
+     -i, --index-server=[HOSTNAME]    index server hostname(default:127.0.0.1)
+     -p, --index-server-port=[PORT]   index server port(default:12120)
+     -q, --qps                        show qps
+         --wait=[SECOND]              wait time for repeat(second)
+     -c, --count=[REPEATTIME]         repeat count
+         --delimiter=[CHAR]           delimiter
+
+=== flare-stdmin
+
+Flare-admin consists of a battery of subcommands used for maintaining a flare cluster.
+You must specify the hostname and port number of the index node by telling them as options 
+or environment variables.
+
+==== MASTER subcommand
+
+'master' subcommand is used for creating a new partition in a cluster.
+You must specify a hostname, a port number, an intitial balance and a partition number 
+for each partition in "hostname:port:balance:partition" format. The specified node will 
+be changed from 'proxy' state to 'master' state and given the specified balance after 
+the construction.
+
+If the node isn't a proxy or the partition already exists, this subcommand fails.
+
+ $ flare-admin master newmaster1:12131:1:1
+ (confirmation)
+ (construction)
+
+This subcommand asks you before committing the change to make sure that you know the 
+result. If you're convinced of the effect of this subcommand and doesn't want to be
+bothered, you can specify --force option to skip the confirmation steps.
+
+ $ flare-admin master --force newmaster:12131:1:1
+
+==== RECONSTRUCT subcommand
+
+'reconstruct' subcommand flushes the existing data and reconstruct new database from 
+another node. This subcommand can be applied to both slaves and masters. 
+If you reconstruct a master, the node will be turned down and one of slave nodes will 
+take over master's role.
+
+ $ flare-admin reconstruct node1:12121
+ (confirmation)
+ (construction)
+
+If you want to reconstruct all nodes in a cluster, you can specify --all option.
+
+ $ flare-admin reconstruct --all
+ (confirmation)
+ (construction)
+ ...
+
+This subcommand asks you before committing the change to make sure that you know the 
+result. If you're convinced of the effect of this subcommand and doesn't want to be
+bothered, you can specify --force option to skip the confirmation steps.
+
+ $ flare-admin reconstruct --force node1:12121
+ (construction)
+
+==== PING subcommand
+
+'ping' subcommand checks if the nodes specified as arguments are alive or not by
+connecting each node and sending a ping request. If the response is positive (OK),
+flare-admin exits with a status code 0 (otherwise 1). You can check the soundness
+of a node by this subcommand in scripts.
+
+ $ flare-admin ping
+ alive
+
+If you want to wait for a node to start up, you should specify --wait option and 
+the command repeatedly sends ping requests until the node is available.
+
+ $ flare-admin ping --wait
+
+==== DEPLOY subcommand (experimental)
+
+'deploy' subcommand generates node settings with control scripts.
+This subcommand is mainly used for experiments.
+
+==== SLAVE subcommand
+
+'slave' subcommand makes a node a slave state and construct the slave database.
+Before changing the state, this confirms the current state and the resulted state.
+After that, duplication of data starts and the progress is put on your console.
+
+ $ flare-admin slave newslave1:12132:1:0 newslave2:12132:1:0
+ (confirmation)
+ (construction)
+
+==== BALANCE subcommand
+
+'balance' subcommand sets balance parameters of nodes.
+
+ $ flare-admin balance node1:12131:3 node2:12131:2
+ (confirmation)
+ (setting balance)
+
+==== LIST subcommand
+
+'list' subcommand shows a node list in a cluster. 
+
+ $ flare-admin list
+ node                             partition   role  state balance
+ server1:12121                            -  proxy active       0
+
+This subcommand just refers the cluster information the index node in a cluster.
+If 'stats' subcommand fails, please check the node status by this subcommand.
+
+==== THREADS subcommand (experimental)
+
+'threads' subcommand shows the thread status of the specified node.
+
+ $ flare-admin threads node1:12121
+
+==== STATS subcommand
+
+'stats' subcommand shows the status of nodes in a cluster.
+
+ $ flare-admin stats
+ hostname:port               state   role partition balance    items conn behind hit size uptime version
+ server1:12121              active master         0       1       30   11      0  70    0     1d  1.0.14
+ server2:12121              active  slave         0       1       35   19      0  68    0     1h  1.0.14
+
+This subcommand also display qps (query per second) statistics of each node.
+
+ $ flare-admin stats --qps --count=100
+ ...
+ hostname:port               state   role partition balance    items conn behind hit size uptime version   qps qps-r qps-w
+ server1:12121              active master         0       1       30   11      0  70    0     1d  1.0.14 100.3  50.1  50.2
+ server2:12121              active  slave         0       1       35   19      0  68    0     1h  1.0.14  52.1     0  52.1
+ ...
+
+==== INDEX subcommand (experimental)
+
+'index' subcommand generates the index file of a cluster.
+
+ $ flare-admin index
+ <?xml version="1.0" encoding="UTF-8" standalone="yes" ?>
+ <!DOCTYPE boost_serialization>
+ <boost_serialization signature="serialization::archive" version="4">
+ <node_map class_id='0' tracking_level='0' version='0'> ... </node_map>
+
+==== DOWN subcommand
+
+'down' subcommand turn nodes down.
+
+ $ flare-admin down node1:12121 node2:12121
+
+==== flare-admin's options
+
+Usage: flare-admin [subcommand] [options] [arguments]
+    -h, --help                       show this message
+    -d, --debug                      enable debug mode
+    -w, --warn                       turn on warnings
+    -n, --dry-run                    dry run
+    -i, --index-server=[HOSTNAME]    index server hostname(default:)
+    -p, --index-server-port=[PORT]   index server port(default:)
+        --log-file=[LOGFILE]         output log to LOGFILE
+subcommands:
+
+[dumpkey] dump key from nodes.
+  Usage: flare-admin dumpkey [hostname:port] ...
+    -o, --output=[FILE]              output to file
+    -f, --format=[FORMAT]            output format [csv]
+    -p, --partition=[NUMBER]         partition number
+    -s, --partition-size=[SIZE]      partition size
+        --bwlimit=[BANDWIDTH]        bandwidth limit (bps)
+        --all                        dump form all partitions
+
+[master] construct a partition with a proxy node for master role.
+  Usage: flare-admin master [hostname:port:balance:partition] ...
+        --force                      commit changes without confirmation
+        --retry=[COUNT]              specify retry count (default:10)
+        --activate                   change node's state from ready to active
+
+[balance] set the balance values of nodes.
+  Usage: flare-admin balance [hostname:port:balance] ...
+        --force                      commit changes without confirmation
+
+[down] turn down nodes and move them to proxy state.
+  Usage: flare-admin down [hostname:port] ...
+        --force                      commit changes without confirmation
+
+[restore] restore data to nodes. (experimental)
+  Usage: flare-admin restore [hostname:port]
+    -i, --input=[FILE]               input from file
+    -f, --format=[FORMAT]            input format [tch]
+        --bwlimit=[BANDWIDTH]        bandwidth limit (bps)
+        --include=[PATTERN]          include pattern
+        --prefix-include=[STRING]    prefix string
+        --exclude=[PATTERN]          exclude pattern
+        --print-keys                 enables key dump to console
+
+[stats] show the statistics of a flare cluster.
+  Usage: flare-admin stats [hostname:port] ...
+    -q, --qps                        show qps
+    -w, --wait=[SECOND]              specify wait time for repeat(second)
+    -c, --count=[REPEATTIME]         specify repeat count
+    -d, --delimiter=[CHAR]           spedify delimiter
+
+[verify] verify the cluster. (experimental)
+  Usage: flare-admin verify
+        --key-hash-algorithm=[TYPE]  key hash algorithm
+        --use-test-data              store test data
+        --debug                      use debug mode
+        --64bit                      (experimental) 64bit mode
+        --verbose                    use verbose mode
+        --meta                       use meta command
+        --quiet                      use quiet mode
+
+[remove] remove a node. (experimental)
+  Usage: flare-admin remove
+        --force                      commit changes without confirmation
+        --wait=[SECOND]              specify the time to wait node for getting ready (default:30)
+        --retry=[COUNT]              retry count(default:5)
+        --connection-threshold=[COUNT]
+                                     specify connection threashold (default:2)
+
+[dump] dump data from nodes. (experimental)
+  Usage: flare-admin dump [hostname:port] ...
+    -o, --output=[FILE]              output to file
+    -f, --format=[FORMAT]            specify output format [default,csv,tch]
+        --bwlimit=[BANDWIDTH]        specify bandwidth limit (bps)
+        --all                        dump from all master nodes
+        --raw                        raw dump mode (for debugging)
+
+[threads] show the list of threads in a flare cluster.
+  Usage: flare-admin threads [hostname:port]
+
+[slave] construct slaves from proxy nodes.
+  Usage: flare-admin slave [hostname:port:balance:partition] ...
+        --force                      commit changes without confirmation
+        --retry=[COUNT]              specify retry count(default:10)
+        --clean                      clear datastore before construction
+
+[list] show the list of nodes in a flare cluster.
+  Usage: flare-admin list
+        --numeric-hosts              show numerical host addresses
+
+[activate] activate 
+  Usage: flare-admin down [hostname:port] ...
+        --force                      commit changes without confirmation
+
+[ping] ping
+  Usage: flare-admin ping [hostname:port] ...
+        --wait                       wait for OK responses from nodes
+
+[index] print the index XML document from a cluster information.
+  Usage: flare-admin index
+        --output=[FILE]              output index to a file
+
+[reconstruct] reconstruct the database of nodes by copying.
+  Usage: flare-admin reconstruct [hostname:port] ...
+        --force                      commit changes without confirmation
+        --safe                       reconstruct a node safely
+        --retry=[COUNT]              specify retry count (default:10)
+        --all                        reconstruct all nodes
+
+== THANKS:
+* Masaki FUJIMOTO  (the author of Flare)
+* kgws[http://d.hatena.ne.jp/kgws] (the author of old flare-tools)