tyrantmanager 1.0.9

data/HISTORY

@@ -0,0 +1,28 @@
+= Changelog
+== Version 1.0.9 2009-08-01
+
+* First public versioned release
+
+== Version 1.0.8
+
+* start the instance in the context of its home directory
+
+== Version 1.0.7
+
+* do not attempt to stop servers where there is no pid file
+
+== Version 1.0.6 
+
+* updates to list command
+
+== Version 1.0.4
+
+* fix db name on ttserver commandline, make sure opts/mode/etc are correct 
+
+== Version 1.0.1
+
+* fix default home directory in cli
+
+== Version 1.0.0
+
+* Initial release

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2009, Jeremy Hinegardner
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted, provided that the above
+copyright notice and this permission notice appear in all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
+REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
+FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
+INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR
+OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+PERFORMANCE OF THIS SOFTWARE.

data/README

@@ -0,0 +1,273 @@
+== Tyrant Manager
+
+* Homepage[http://copiousfreetime.rubyforge.org/tokyo-manager]
+* {Rubyforge project}[http://rubyforge.org/projects/copiousfreetime/]
+* email jeremy at copiousfreetime dot org
+* git clone git://github.com/copiousfreetime/tokyo-manager.git
+
+== INSTALLATION
+
+  sudo gem install tyrantmanager
+
+== SYNOPSIS
+
+  tyrantmanager (setup|create-instance|start|stop|status|stats|list) [options]+
+
+== DESCRIPTION
+
+A command line tool for managing Tokyo Tyrant instances.
+
+=== Setup a manager home
+
+After installing the gem, you need to setup a tyrant manager home, all the
+tyrant instances are initially relative to this location.
+
+  jeremy@playground:~ % tyrantmanager setup /tmp/tyrant
+  00:09:06  INFO : Creating directory /tmp/tyrant
+  00:09:06  INFO : Creating default config file /tmp/tyrant/config.rb
+  00:09:06  INFO : Creating directory /tmp/tyrant/instances
+  00:09:06  INFO : Creating directory /tmp/tyrant/log
+  00:09:06  INFO : Creating directory /tmp/tyrant/tmp
+
+Once the manager home is setup, all other commands need to know about it. 
+This can be achieved in 3 ways.
+
+* use the --home option on all commands
+* set the TYRANT_MANAGER_HOME environment variable
+* execute the commands when the current working directory is the tyrant home. 
+
+=== Creating a new tyrant instance
+
+Each instances is separate, yet initially relative to to the tyrant manager's
+home. 
+
+    jeremy@playground:/tmp/tyrant % tyrantmanager create-instance foo
+    00:17:55  INFO : Creating instance directory /tmp/tyrant/instances/foo
+    00:17:55  INFO : Creating directory /tmp/tyrant/instances/foo
+    00:17:55  INFO : Creating default config file /tmp/tyrant/instances/foo/config.rb
+    00:17:55  INFO : Creating directory /tmp/tyrant/instances/foo/ulog
+    00:17:55  INFO : Creating directory /tmp/tyrant/instances/foo/data
+    00:17:55  INFO : Creating directory /tmp/tyrant/instances/foo/lua
+    00:17:55  INFO : Creating directory /tmp/tyrant/instances/foo/log
+
+This creates a new tyrant instance in the instances directory.  There are
+several subdirectories created for you automatically.  
+
+* data - the actual data file location
+* ulog - where the update logs are kept
+* log  - the server logs
+* lua  - default location for lua code
+
+=== Configuring an instance
+
+Configuration of a Tyrant instances is in the 'config.rb' file in the instances
+home directory.  This all the configuration items that are specific to that
+instance.   Take a look through the file, or just execute it to see a nicely
+formatted output of all the current configuration parameters, descriptions,
+and current values.  This example just highlights a few of the options.
+
+    jeremy@playground:/tmp/tyrant/instances/foo % ruby config.rb 
+    The directory holding the database file.  By default this is relative
+    to the instance directory.  If you want to put it somewhere else, then
+    put the full path here.
+      - data_dir                   => "data"
+
+    The hostname this instance will listen on. The default is all
+      - host                       => "0.0.0.0"
+
+    [...]
+
+    Mode of the server in relation to the database.  It can be a combination of 
+    the following:
+         w - it is a database writer
+         r - it is a database reader
+         c - it creates the database if it does not exist
+         t - it truncates the database if it does exist
+         e - do not lock the database when doing operations
+         f - use a non-blocking lock when doing operations
+      - mode                       => "wc"
+
+    The options for data stored in the database.  They include one of the 
+    compression options:
+         d - Deflate
+         b - BZIP2
+         t - TCBS 
+     and 
+         l - allow the database to be larger than 2GB
+      - opts                       => "ld"
+
+    [...]
+    
+    The type of database this instance will be.  Options are:
+         memory-hash - an in memory hash database
+         memory-tree - an in memory B+ tree database
+         hash        - an on disk hash database
+         tree        - an on disk B+ tree database
+         fixed       - an on disk fixed length database
+         table       - an on disk table database
+      - type                       => "hash"
+
+There are also configuration items in the 'config.rb' in the Manager's home
+directory.  These are global defaults for all instances the manager knows about,
+and may be overridden in the instance's 'config.rb' file.  Again, just execute
+the configuration file to see all the options, their current set values and 
+the documentation.
+
+    jeremy@playground:/tmp/tyrant % ruby config.rb 
+    The top directory of the manager directory.  This is set with the 
+    --directory option or it can be set here"
+      - home                               => nil
+
+    Default settings for tyrant instances.  If a tyrant does not explicitly set one of 
+    these configurations then the default listed here is used.  This does not list all
+    of the configuration variables, only those that make sense to have defaults for.
+      - instance_defaults
+
+    [...]   
+
+     Run daemonized
+      - instance_defaults.daemonize        => true
+
+    [...]
+    
+    The list of locations that contain Tyrant instances.  Each element in this array
+    either a directory containing instance directories, or an instance directory itself.
+    Tyrant Manager will figure out what the case is.  If this is unset, then the default
+    'instances' directory below the manager's home directory is used.
+      - instances                          => nil
+
+    The 'ttserver' command.  This will use the first one found in path by default
+    or you may explicitly set it here.
+      - ttserver                           => "ttserver"
+
+There are a considerable number of configuration options for Tokyo Tyrant and I've done
+my best to document all of them.
+
+If you do have more than one instance on a single machine, you'll need to make sure 
+to change the port option, otherwise only one instance will run.
+
+    desc "The port this tyrant will listen on.  The default is 1978"
+    port 1978
+
+=== Listing, Controlling instances and Seeing their status
+
+Instances may be started and stopped all at once, or individually.  On startup, 
+the ttserver command line that was used is output.  
+
+==== List known instances
+
+    jeremy@playground:/tmp/tyrant % tyrantmanager list
+                 bar : port 1979 : /tmp/tyrant/instances/bar
+                 baz : port 1980 : /tmp/tyrant/instances/baz
+                 foo : port 1978 : /tmp/tyrant/instances/foo
+
+==== Start all
+
+    jeremy@playground:/tmp/tyrant % tyrantmanager start
+    00:51:15  INFO : Starting bar : ttserver -host 0.0.0.0 -port 1979 -thnum 8 -tout 15 -dmn -pid /tmp/tyrant/instances/bar/bar.pid -log /tmp/tyrant/instances/bar/log/bar.log -le -ulog /tmp/tyrant/instances/bar/ulog -ulim 1g -rts /tmp/tyrant/instances/bar/bar.rts /tmp/tyrant/instances/bar/data/bar.tch#opts=ld#mode=wc
+    00:51:15  INFO : 
+    00:51:15  INFO : Starting baz : ttserver -host 0.0.0.0 -port 1980 -thnum 8 -tout 15 -dmn -pid /tmp/tyrant/instances/baz/baz.pid -log /tmp/tyrant/instances/baz/log/baz.log -le -ulog /tmp/tyrant/instances/baz/ulog -ulim 1g -rts /tmp/tyrant/instances/baz/baz.rts /tmp/tyrant/instances/baz/data/baz.tch#opts=ld#mode=wc
+    00:51:15  INFO : 
+    00:51:15  INFO : Starting foo : ttserver -host 0.0.0.0 -port 1978 -thnum 8 -tout 15 -dmn -pid /tmp/tyrant/instances/foo/foo.pid -log /tmp/tyrant/instances/foo/log/foo.log -le -ulog /tmp/tyrant/instances/foo/ulog -ulim 1g -rts /tmp/tyrant/instances/foo/foo.rts /tmp/tyrant/instances/foo/data/foo.tch#opts=ld#mode=wc
+    00:51:15  INFO : 
+
+As you can see the commandline used to start the tyrant server is output.
+
+    jeremy@playground:/tmp/tyrant % tyrantmanager status
+    00:52:17  INFO : bar is running as pid 28658
+    00:52:17  INFO : baz is running as pid 28670
+    00:52:17  INFO : foo is running as pid 28682
+
+==== Stop all 
+
+    jeremy@playground:/tmp/tyrant % tyrantmanager stop all
+    00:52:40  INFO : Stopping bar : pid 28658
+    00:52:40  INFO : Sent signal TERM to 28658
+    00:52:40  INFO : Stopping baz : pid 28670
+    00:52:40  INFO : Sent signal TERM to 28670
+    00:52:40  INFO : Stopping foo : pid 28682
+    00:52:40  INFO : Sent signal TERM to 28682
+
+And see that they are all done
+
+    jeremy@playground:/tmp/tyrant % tyrantmanager status
+    00:53:15  INFO : bar is not running, or its pid file is gone
+    00:53:15  INFO : baz is not running, or its pid file is gone
+    00:53:15  INFO : foo is not running, or its pid file is gone
+
+==== Start just foo and baz
+
+    jeremy@playground:/tmp/tyrant % tyrantmanager start foo,baz
+    00:53:54  INFO : Starting baz : ttserver -host 0.0.0.0 -port 1980 -thnum 8 -tout 15 -dmn -pid /tmp/tyrant/instances/baz/baz.pid -log /tmp/tyrant/instances/baz/log/baz.log -le -ulog /tmp/tyrant/instances/baz/ulog -ulim 1g -rts /tmp/tyrant/instances/baz/baz.rts /tmp/tyrant/instances/baz/data/baz.tch#opts=ld#mode=wc
+    00:53:54  INFO : 
+    00:53:54  INFO : Starting foo : ttserver -host 0.0.0.0 -port 1978 -thnum 8 -tout 15 -dmn -pid /tmp/tyrant/instances/foo/foo.pid -log /tmp/tyrant/instances/foo/log/foo.log -le -ulog /tmp/tyrant/instances/foo/ulog -ulim 1g -rts /tmp/tyrant/instances/foo/foo.rts /tmp/tyrant/instances/foo/data/foo.tch#opts=ld#mode=wc
+    00:53:54  INFO : 
+
+Foo and baz are started, bar is still stopped
+
+    jeremy@playground:/tmp/tyrant % tyrantmanager status
+    00:54:38  INFO : bar is not running, or its pid file is gone
+    00:54:38  INFO : baz is running as pid 28708
+    00:54:38  INFO : foo is running as pid 28720
+
+=== Stop just foo
+
+    jeremy@playground:/tmp/tyrant % tyrantmanager stop foo
+    00:55:33  INFO : Stopping foo : pid 28720
+    00:55:33  INFO : Sent signal TERM to 28720
+
+And baz is still running
+
+    jeremy@playground:/tmp/tyrant % tyrantmanager status
+    00:55:51  INFO : bar is not running, or its pid file is gone
+    00:55:51  INFO : baz is running as pid 28708
+    00:55:51  INFO : foo is not running, or its pid file is gone
+
+=== Look at the server statistics
+
+You can also look at the server statistics of each running instance
+
+    jeremy@playground:/tmp/tyrant % tyrantmanager stats
+    Instance bar at /tmp/tyrant/instances/bar
+        bigend.... 0
+        fd........ 7
+        libver.... 312
+        loadavg... 0.030000
+        memrss.... 1003520
+        memsize... 173633536
+        os........ Linux
+        path...... /tmp/tyrant/instances/bar/data/bar.tch
+        pid....... 28749
+        protver... 0.91
+        rnum...... 0
+        ru_real... 14.560829
+        ru_sys.... 0.000000
+        ru_user... 0.004000
+        sid....... 72615862
+        size...... 1052992
+        time...... 1249174632.207222
+        type...... hash
+        version... 1.1.29
+    [...]
+
+== CREDITS
+
+* Inspired by the {Light Cloud}[http://opensource.plurk.com/LightCloud/Tyrant_manager/} tyrant manager
+* {Tokyo Tyrant}[http://tokyocabinet.sourceforge.net/tyrantdoc/]
+
+== LICENSE
+
+Copyright (c) 2009, Jeremy Hinegardner
+
+Permission to use, copy, modify, and/or distribute this software for any
+purpose with or without fee is hereby granted, provided that the above
+copyright notice and this permission notice appear in all copies.
+
+THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
+REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
+FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
+INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM
+LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR
+OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR
+PERFORMANCE OF THIS SOFTWARE.
+

data/bin/tyrantmanager

@@ -0,0 +1,10 @@
+#!/usr/bin/env ruby
+
+#--
+# Copyright (c) 2009 
+# All rights reserved.  See LICENSE and/or COPYING for details.
+#++
+
+$:.unshift File.expand_path(File.join(File.dirname(__FILE__),"..","lib"))
+require 'tyrant_manager'
+::TyrantManager::Cli.new( ARGV, ENV ).run

data/data/config.rb

@@ -0,0 +1,94 @@
+require 'rubygems'
+require 'loquacious'
+
+Loquacious::Configuration.for( "manager" ) do
+
+  desc <<-txt
+  The 'ttserver' command.  This will use the first one found in path by default
+  or you may explicitly set it here.
+  txt
+  ttserver "ttserver"
+
+  desc <<-txt
+  The top directory of the manager directory.  This is set with the 
+  --directory option or it can be set here"
+  txt
+  home     nil
+
+  desc <<-txt
+  The list of locations that contain Tyrant instances.  Each element in this array
+  either a directory containing instance directories, or an instance directory itself.
+  Tyrant Manager will figure out what the case is.  If this is unset, then the default
+  'instances' directory below the manager's home directory is used.
+  txt
+  instances nil
+
+  desc <<-txt
+  Default settings for tyrant instances.  If a tyrant does not explicitly set one of 
+  these configurations then the default listed here is used.  This does not list all
+  of the configuration variables, only those that make sense to have defaults for.
+  txt
+  instance_defaults {
+    desc "The number of threads to use"
+    thread_count 8
+
+    desc "Number of seconds to terminate a session request if no request is made"
+    session_timeout 15
+
+    desc "The level of logging, this can be 'debug', 'info', or 'error'"
+    log_level "error"
+
+    desc "Run daemonized"
+    daemonize true
+
+    desc <<-txt
+    The size of the update log in bytes.  You may use a trailing letter to indicate
+    kilobytes(k), megabytes(m), gigabytes(g), terabytes(t), exabytes(e)
+    txt
+    update_log_size "1g"
+
+    desc "Use asychronous I/O when writing to the update log"
+    update_log_async false
+
+    desc <<-txt
+    Forbidden protocol commands, this is the list of commands that will be denied to the client.
+    The full list is:
+    |  put putkeep putcat putshl putnr out get mget vsize iterinit iternext fwmkeys addint
+    |  adddouble ext sync optimize vanish copy restore setmst rnum size stat misc repl slave
+
+    There are meta lists too:
+
+    |  all       : all commands
+    |  allorg    : all tyrant binary protocol commands
+    |  allmc     : all memecached protocol commands
+    |  allhttp   : all http commands
+    |  allread   : get mget vsiz iterinit iternext fwmkeys rnum size stat
+    |  allwrite  : put putkeep putcat putshl putnr out addint adddouble vanish misc
+    |  allmanage : sync optimize copy restore setmst
+
+    To deny optimize vanish and slave commands the configuration would be:
+      
+    |  deny_commands [ 'vanish', 'slave', 'optimizie' ]
+
+    txt
+    deny_commands nil
+
+    desc <<-txt
+    Specifically allowed protocol commands.  This will only allow these commands.  The same
+    list of commands in the deny_commands configuration is available.
+
+    Leave this as nil to use the defaults.  To only allow the read and write commands and no
+    others the configuration would be:
+
+    |  allow_commands [ 'allread', 'allwrite' ]
+
+    txt
+    allow_commands nil
+    
+  }
+end
+
+if $0 == __FILE__ then
+  help = Loquacious::Configuration.help_for("manager", :colorize => true)
+  help.show( :values => true )
+end

data/data/default_instance_config.rb

@@ -0,0 +1,244 @@
+require 'rubygems'
+require 'loquacious'
+
+# 
+# The configuration for the <%= instance_name %> tokyo tyrant instance
+#
+# The options available in the instance_defaults section of the manager
+# configuration are also able to be set here.
+#
+Loquacious::Configuration.for( "<%= instance_name %>" ) do
+
+  desc "The hostname this instance will listen on. The default is all"
+  host "0.0.0.0"
+
+  desc "The port this tyrant will listen on.  The default is 1978"
+  port 1978
+
+  desc <<-txt
+  The type of database this instance will be.  Options are:
+  |    memory-hash - an in memory hash database
+  |    memory-tree - an in memory B+ tree database
+  |    hash        - an on disk hash database
+  |    tree        - an on disk B+ tree database
+  |    fixed       - an on disk fixed length database
+  |    table       - an on disk table database
+  txt
+  type "hash"
+
+  desc <<-txt
+  Mode of the server in relation to the database.  It can be a combination of 
+  the following:
+  |    w - it is a database writer
+  |    r - it is a database reader
+  |    c - it creates the database if it does not exist
+  |    t - it truncates the database if it does exist
+  |    e - do not lock the database when doing operations
+  |    f - use a non-blocking lock when doing operations
+  txt
+  mode "wc"
+  
+
+  desc <<-txt
+  The options for data stored in the database.  They include one of the 
+  compression options:
+  |    d - Deflate
+  |    b - BZIP2
+  |    t - TCBS
+  and 
+  |    l - allow the database to be larger than 2GB
+  txt
+  opts "ld"
+
+  desc <<-txt
+  Tuning parameters.  Each of these alters a tuning parameter on the database.
+  Not all of them are applicable to all database types
+  txt
+
+  tuning_params {
+    desc <<-txt
+    The number of elements in the bucket array.  Supported in hash, B+ tree
+    and table databases.  It is also used in the in-memory hash database.
+    It should be between 1 and 4 times the number of all records to be stored.
+    txt
+    bnum nil
+
+    desc <<-txt
+    The maximum number of records to store in the database.  This is only
+    used in the in-memory databases.
+    txt
+    capnum nil
+
+    desc <<-txt
+    The maximum amount of memory to use in the server.  This is only used
+    in the in-memory databases.
+    txt
+    capsize nil
+
+    desc <<-txt
+    Aligment power.  The byte alignment of records.  This specifies a power 
+    of two.  The default value is 4 meaning that all records are aligned on 
+    16 byte boundaries.
+
+    This applies to on disk Hash, B+ tree and Table databases.
+    txt
+    apow nil
+
+    desc <<-txt
+    Free block pool power.  This is the maximum number of elements in the free
+    block pool as a power of 2.  The default is 10 meaning that there are a 
+    maximum of 1024 free blocks.
+    
+    This applies to on disk hash, B+ tree and Table databases.
+    txt
+    fpow nil
+
+    desc <<-txt
+    The maximum number of records to be cached.  If set to 0 or less then
+    the cache is disabled.  That is the default.
+
+    This applies to on disk Hash and Table databases.
+    txt
+    rcnum nil
+
+    desc <<-txt
+    The size of the memory mapped region.  By default this is 64 megabytes.
+
+    This applies to on disk Hash, B+ tree and Table databases.
+    txt
+    xmsiz
+
+    desc <<-txt
+    Auto defragmentation configuration.  This controls how the automatic
+    defragmentation of databases is preformed.  It is the 'step' 
+    number.  It is off by default.
+
+    This applies to on disk Hash, B+ tree and Table databases.
+    txt
+    dfunit nil
+
+    desc <<-txt
+    The maximum number of leaf nodes to be cached.  The default is 4096.
+
+    This applies to B+ tree and Table databses
+    txt
+    lcnum nil
+
+    desc <<-txt
+    The maximum number of non-leaf nodes to be cached.  The default is 512.
+  
+    This applies to B+ tree and Table databses
+    txt
+    ncnum nil
+
+    desc <<-txt
+    The number of members in each leaf page.  The default is 128.
+
+    This applies to B+ tree databases.
+    txt
+    lmemb nil
+
+    desc <<-txt
+    The number of members in each non-leaf page.  The default is 128.
+
+    This applies to B+ tree databases.
+    txt
+    nmemb nil
+
+    desc <<-txt
+    The width of the value of each record in a fixed length database.  
+    The default is 255.
+    txt
+    width nil
+
+    desc <<-txt
+    The file size limit of the fixed database file.  The default is 256 MB.
+    txt
+    limsiz nil
+ 
+  }
+
+  desc <<-txt
+  The directory holding the database file.  By default this is relative
+  to the instance directory.  If you want to put it somewhere else, then
+  put the full path here.
+  txt
+  data_dir "data"
+
+  desc <<-txt
+  The directory holding the update logs.  By default this is relative
+  to the instance directory.  If you want to put it somewhere else, then
+  put the full path here.
+  txt
+  ulog_dir "ulog"
+
+
+  desc <<-txt
+  The pid file location.  By default this is relative to the instance
+  directory.  If you want it place somewhere else, then put the full 
+  path here.
+  txt
+  pid_file "<%= instance_name %>.pid"
+
+
+  desc <<-txt
+  The log file location.  By default this is relative to the instance
+  directory.  If you want it place somewhere else, then put the full 
+  path here.
+  txt
+  log_file "log/<%= instance_name %>.log"
+
+
+  desc <<-txt
+  If this instance is to replicate the data from another instance then
+  set that instances host here.
+  txt
+  master_server nil
+
+  desc <<-txt
+  If this instance is to replicate the data from another instance then
+  set that instances port here.
+  txt
+  master_port nil
+
+  desc <<-txt
+  This is the server id of this instance when it is part of a replication.
+  This must be set, and it must be unique within the set of hosts doing
+  replication.
+  txt
+  server_id nil
+
+  desc <<-txt
+  If this server is replicating data from a master, then it must have
+  this set to the location of a replication timestamp file.  The 
+  default is relative to the instance directory.
+  txt
+  replication_timestamp_file "<%= instance_name %>.rts"
+
+  desc <<-txt
+  The name of a file to load as a lua extension.  By default this 
+  is relative to the root of the instance directory.  This file will
+  only be loaded if it exists.
+  txt
+  lua_extension_file "lua/<%= instance_name %>.lua"
+
+  desc <<-txt
+  You may specify a lua function and a periodicity in which it is to
+  be called.  This lua function must be available from within the
+  lua_extension_file that is loaded.
+
+  The periodicity is really the time to wait between calls.
+  txt
+  periodic_command {
+    desc "The function to call"
+    name nil
+
+    desc "how often to call"
+    period nil
+  }
+end
+
+if $0 == __FILE__ then
+  help = Loquacious::Configuration.help_for("<%= instance_name %>", :colorize => true)
+  help.show :values => true
+end