rufus-tokyo 1.0.1 → 1.0.2

data/CHANGELOG.txt

@@ -2,6 +2,34 @@
 = rufus-tokyo CHANGELOG.txt
 
 
+== rufus-tokyo - 1.0.2    released 2009/11/01
+
+- bug   : HashMethods#to_a not working right with Ruby 1.9.1. Fixed.
+- bug   : REE problem with libXXXopen, had to switch from == 1 to != 0
+- todo  : added #counter_value to Cabinet/Tyrant
+- todo  : added default_proc / default value to Cabinet/Tyrant (not tables).
+- todo  : Table#delete_keys_with_prefix now using qry#searchout.
+- todo  : cleaned up 'no transactions for Tyrant [table]' code.
+- todo  : db#ldelete(k0, k1, ...) now OK.
+- todo  : minimal warnings when running with -w
+- todo  : db#mget(k0, k1, ...) now OK.
+- added : class level open() for Table classes and Dystopia
+- bug   : the primary key is not stringified by get() for Table or TyrantTable
+- bug   : keys() crashes for Rufus::Edo::Cabinet.new('*.tcb')
+- bug   : keys() without :prefix returns duplicate keys for b-tree database
+          (Edo or Tokyo) due to iteration, while the underlying C code or even
+          calls with :prefix do not
+- bug   : specifying tuning parameters causes Rufus::Edo::Cabinet.new('*.tcf')
+          to raise an exception
+- bug   : keys() and each() don't work fixed-width database, due to fwmkeys()
+          not being supported (Edo and Tokyo)
+- added : can set built-in or custom comparison functions for b-tree databases
+          (Edo interface only)
+- todo  : reenabled #copy for Tyrant (Edo and Tokyo)
+- added : query_count() to match query_delete() for tables (Edo and Tokyo)
+- todo  : made iteration faster in most cases
+
+
 == rufus-tokyo - 1.0.1    released 2009/09/18
 
 - todo      : add #putcat to Cabinet / Tyrant (Edo and Tokyo)

data/CREDITS.txt

@@ -5,6 +5,7 @@
 == authors
 
 John Mettraux  http://jmettraux.wordpress.com
+James Edward Gray II  http://blog.grayproductions.net/
 Zev Blut  http://www.iknow.co.jp/users/zev
 Jeremy Hinegardner  http://www.copiousfreetime.org/
 
@@ -24,7 +25,6 @@ Mark 'mebaran'  http://github.com/mebaran
 
 == feedback / support
 
-James Edward Gray II  http://blog.grayproductions.net/
 Alexandre Gravem
 Philippe Monnet  http://blog.monnet-usa.com/
 Ilya Grigorik  http://igvita.com

data/MIGRATED.txt

@@ -0,0 +1 @@
+to gemcutter.org indeed :)

data/README.rdoc

@@ -8,7 +8,7 @@ The 'abstract' and the 'table' API are covered for now.
 
 == installation
 
-  sudo gem install rufus-tokyo
+  sudo gem install rufus-tokyo --source http://gemcutter.org
 
   (see after 'usage' for how to install Tokyo Cabinet (and Tyrant) if required)
 
@@ -31,11 +31,12 @@ the rdoc is at http://rufus.rubyforge.org/rufus-tokyo/
 
 this 'master' branch/version of rufus-tokyo works with
 
-  ffi 0.3.1, 0.3.5, 0.4.0, 0.5.0
+  ffi 0.3.1, 0.3.5, 0.4.0, 0.5.0, 0.5.1
 
   ruby 1.8.6
   ruby 1.9.1p129
   jruby 1.1.6, 1.2.0, 1.3.1
+  ree 1.8.7 2009.10 (thanks to http://rvm.beginrescueend.com)
 
   TC 1.4.27 / 29 and TT 1.1.29 / 33
 
@@ -272,7 +273,7 @@ On the rufus-ruby list :
 
   http://groups.google.com/group/rufus-ruby
 
-Note : Flinn Muller started an english speaking mailing list for TC / TT / TD 
+Note : Flinn Mueller started an english speaking mailing list for TC / TT / TD 
 
   http://groups.google.com/group/tokyocabinet-users
 
@@ -298,17 +299,23 @@ http://github.com/jmettraux/rufus-tokyo
 
 many thanks to the author of Tokyo Cabinet, Mikio Hirabayashi, and to the authors of ruby-ffi
 
-* http://tokyocabinet.sourceforge.net
-* http://kenai.com/projects/ruby-ffi
+* http://1978th.net/tokyocabinet/
+* http://1978th.net/tokyotyrant/
+* http://1978th.net/tokyodystopia/
+
+* http://wiki.github.com/ffi/ffi
 
 dystopia interface developed with and sponsored by Seven Scale <http://sevenscale.com/>,
 creators of Open Syslog.
 
+
 == authors
 
 * John Mettraux, jmettraux@gmail.com, http://jmettraux.wordpress.com
 * Zev Blut, http://www.iknow.co.jp/users/zev
 * Jeremy Hinegardner http://github.com/copiousfreetime (Tokyo Dystopia)
+* James Edward Gray II  http://blog.grayproductions.net/
+
 
 == the rest of Rufus
 

data/TODO.txt

@@ -1,38 +1,37 @@
 
 [o] table : bench find(1) after set_index
 [o] table : bench insert(1) after set_index
-
 [o] align edo#new on tokyo#new
-[ ] horizontal (bdb hdb fdb) tests for tokyo and edo
-
 [o] rdoc unix socket for tyrant
 [o] edo : tran
 [x] bench ux socket
-
 [o] TT (TC) call [lua] function
 [o] queryout
-
 [o] impl lget/lput/ldelete in Rufus::Edo::Cabinet
-
 [x] maybe supported different libs for migrations...
     no, tctools are here for that.
-
-[ ] document ENV['TOKYO_TYRANT_LIB'] & _CABINET_ / LD_LIBRARY_PATH
-
+[x] maybe Edo::Abort vs Tokyo::Abort
 [o] unify m/l raise trick for util map/list
-
 [o] misc calls (defrag) for Edo
-[ ] TT set master & co
-
 [o] drop 'backward compatibility' (post 0.1.14)
 [o] qgram, token, opt inverted index types
 [o] new query FTS conds
+[o] document ENV['TOKYO_TYRANT_LIB'] & _CABINET_ / LD_LIBRARY_PATH
+[o] first integration of Jeremy's work on Dystopia
+[o] implement q.metasearch(q, u)
+[o] table/delete_keys_with_prefix, use searchout
+[o] default value / default_proc for cabinets
+[o] self.open for tables
 
-[ ] table/delete_keys_with_prefix, use searchout
-
-[ ] first integration of Jeremy's work on Dystopia
+[ ] ldelete for tables ?
 
-[ ] implement q.metasearch(q, u)
 [ ] kwic ?
-[x] maybe Edo::Abort vs Tokyo::Abort
+
+[ ] TT set master & co
+
+[ ] map/reduce ?
+
+[ ] wire setmutex for Cabinet (and ?)
+
+[ ] copy for tyrant tables ?
 

data/lib/rufus/edo/cabcore.rb

@@ -25,6 +25,7 @@
 
 require 'rufus/tokyo/hmethods'
 require 'rufus/tokyo/transactions'
+require 'rufus/tokyo/openable'
 
 
 module Rufus::Edo
@@ -42,25 +43,9 @@ module Rufus::Edo
     include Rufus::Tokyo::HashMethods
     include Rufus::Tokyo::Transactions
 
-    def self.included (target_module)
-
-      target_module.module_eval do
-        #
-        # Same args as initialize, but can take a block form that will
-        # close the db when done. Similar to File.open (via Zev)
-        #
-        def self.open (name, params={})
-          db = self.new(name, params)
-          if block_given?
-            yield db
-            nil
-          else
-            db
-          end
-        ensure
-          db.close if block_given? && db
-        end
-      end
+    # Add the open() method to all Cabinet type classes.
+    def self.included(target)
+      target.extend(Rufus::Tokyo::Openable)
     end
 
     # Returns the path to this database.
@@ -193,25 +178,14 @@ module Rufus::Edo
     #
     def keys (options={})
 
-      if pref = options[:prefix]
+      if @db.respond_to? :fwmkeys
+        pref = options.fetch(:prefix, "")
 
         @db.fwmkeys(pref, options[:limit] || -1)
-
+      elsif @db.respond_to? :range
+        @db.range("[min,max]", nil)
       else
-
-        limit = options[:limit] || -1
-        limit = nil if limit < 1
-
-        l = []
-
-        @db.iterinit
-
-        while (k = @db.iternext)
-          break if limit and l.size >= limit
-          l << k
-        end
-
-        l
+        raise NotImplementedError, "Database does not support keys()"
       end
     end
 
@@ -235,9 +209,9 @@ module Rufus::Edo
     #
     # Warning : this is a naive (slow) implementation.
     #
-    def lget (keys)
+    def lget (*keys)
 
-      keys = keys.collect { |k| k.to_s }
+      keys = keys.flatten.collect { |k| k.to_s }
 
       # only ADB has the #misc method...
 
@@ -269,9 +243,9 @@ module Rufus::Edo
     #
     # Warning : this is a naive (slow) implementation.
     #
-    def ldelete (keys)
+    def ldelete (*keys)
 
-      keys = keys.collect { |k| k.to_s }
+      keys = keys.flatten.collect { |k| k.to_s }
 
       # only ADB has the #misc method...
 
@@ -287,6 +261,11 @@ module Rufus::Edo
     # Increments the value stored under the given key with the given increment
     # (defaults to 1 (integer)).
     #
+    # Warning : Tokyo Cabinet/Tyrant doesn't store counter values as regular
+    # strings (db['key'] won't yield something that replies properly to #to_i)
+    #
+    # Use #counter_value(k) to get the current value set for the counter.
+    #
     def incr (key, val=1)
 
       key = key.to_s
@@ -295,7 +274,7 @@ module Rufus::Edo
 
       raise(EdoError.new(
         "incr failed, there is probably already a string value set " +
-        "for the key '#{key}'"
+        "for the key '#{key}'. Make sure there is no value before incrementing"
       )) unless v
 
       v
@@ -305,6 +284,15 @@ module Rufus::Edo
     alias :add_double :incr
     alias :add_int :incr
 
+    # Returns the current value for a counter (a float or an int).
+    #
+    # See #incr
+    #
+    def counter_value (key)
+
+      incr(key, 0.0) rescue incr(key, 0)
+    end
+
     # Triggers a defrag (TC >= 1.4.21 only)
     #
     def defrag

data/lib/rufus/edo/cabinet/abstract.rb

@@ -87,11 +87,29 @@ module Rufus::Edo
     #   db = Rufus::Edo::Cabinet.new(
     #     'data', :type => :hash, :opts => 'ld', :mode => 'w')
     #
-    # === mode
+    # === :mode
     #
     #   * :mode    a set of chars ('r'ead, 'w'rite, 'c'reate, 't'runcate,
     #              'e' non locking, 'f' non blocking lock), default is 'wc'
     #
+    # === :default and :default_proc
+    #
+    # Much like a Ruby Hash, a Cabinet accepts a default value or a default_proc
+    #
+    #   db = Rufus::Edo::Cabinet.new('data.tch', :default => 'xxx')
+    #   db['fred'] = 'Astaire'
+    #   p db['fred'] # => 'Astaire'
+    #   p db['ginger'] # => 'xxx'
+    #
+    #   db = Rufus::Edo::Cabinet.new(
+    #     'data.tch',
+    #     :default_proc => lambda { |cab, key| "not found : '#{k}'" }
+    #   p db['ginger'] # => "not found : 'ginger'"
+    #
+    # The first arg passed to the default_proc is the cabinet itself, so this
+    # opens up interesting possibilities.
+    #
+    #
     # === other parameters
     #
     # 'On-memory hash database supports "bnum", "capnum", and "capsiz".
@@ -99,7 +117,7 @@ module Rufus::Edo
     #  Hash database supports "mode", "bnum", "apow", "fpow", "opts",
     #  "rcnum", and "xmsiz".
     #  B+ tree database supports "mode", "lmemb", "nmemb", "bnum", "apow",
-    #  "fpow", "opts", "lcnum", "ncnum", and "xmsiz".
+    #  "fpow", "opts", "lcnum", "ncnum", "xmsiz", and "cmpfunc".
     #  Fixed-length database supports "mode", "width", and "limsiz"'
     #
     #   * :opts    a set of chars ('l'arge, 'd'eflate, 'b'zip2, 't'cbs)
@@ -109,8 +127,6 @@ module Rufus::Edo
     #   * :apow    size of record alignment by power of 2 (defaults to 4)
     #   * :fpow    maximum number of elements of the free block pool by
     #              power of 2 (defaults to 10)
-    #   * :mutex   when set to true, makes sure only 1 thread at a time
-    #              accesses the table (well, Ruby, global thread lock, ...)
     #
     #   * :rcnum   specifies the maximum number of records to be cached.
     #              If it is not more than 0, the record cache is disabled.
@@ -138,6 +154,12 @@ module Rufus::Edo
     #   * :dfunit  unit step number. If it is not more than 0,
     #              the auto defragmentation is disabled. (Since TC 1.4.21)
     #
+    #   * :cmpfunc the comparison function used to order a b-tree database.  Can
+    #              be set to :lexical (default), :decimal, or a Proc object that
+    #              implements a <=>-like comparison function for the two keys it
+    #              will be passed.  Custom comparisons must be set each time the
+    #              database is opened.
+    #
     #
     # = NOTE :
     #
@@ -166,7 +188,7 @@ module Rufus::Edo
         when :abstract then nil
         when :hash then [ :bnum, :apow, :fpow, :opts ]
         when :btree then [ :lmemb, :nmemb, :bnum, :apow, :fpow, :opts ]
-        when :fixed then [ :bnum, :width, :limsiz ]
+        when :fixed then [ :width, :limsiz ]
       end
 
       @db.tune(*tuning_parameters.collect { |o| conf[o] }) \
@@ -197,6 +219,20 @@ module Rufus::Edo
       @db.setdfunit(conf[:dfunit]) \
         if @db.respond_to?(:setdfunit)
 
+      #
+      # set cmp_func
+
+      if @db.respond_to? :setcmpfunc
+        case conf[:cmpfunc]
+        when :lexical
+          @db.setcmpfunc(TokyoCabinet::BDB::CMPLEXICAL)
+        when :decimal
+          @db.setcmpfunc(TokyoCabinet::BDB::CMPDECIMAL)
+        when Proc
+          @db.setcmpfunc(conf[:cmpfunc])
+        end
+      end
+
       #
       # open
 

data/lib/rufus/edo/cabinet/table.rb

@@ -133,6 +133,11 @@ module Rufus::Edo
 
       @db.setdfunit(conf[:dfunit]) if @db.respond_to?(:setdfunit)
 
+      #
+      # no default
+
+      @default_proc = nil
+
       #
       # open
 

data/lib/rufus/edo/ntyrant/abstract.rb

@@ -44,6 +44,7 @@ module Rufus::Edo
 
     include Rufus::Edo::CabinetCore
     include Rufus::Tokyo::TyrantCommons
+    include Rufus::Tokyo::NoTransactions
 
     attr_reader :host, :port
 
@@ -68,7 +69,26 @@ module Rufus::Edo
     #
     #   t = Rufus::Edo::NetTyrant.new('127.0.0.1', 44001)
     #
-    def initialize (host, port=0)
+    #
+    # == :default and :default_proc
+    #
+    # Much like a Ruby Hash, a Tyrant accepts a default value or a default_proc
+    #
+    #   db = Rufus::Edo::NetTyrant.new('127.0.0.1', 1978, :default => 'xxx')
+    #   db['fred'] = 'Astaire'
+    #   p db['fred'] # => 'Astaire'
+    #   p db['ginger'] # => 'xxx'
+    #
+    #   db = Rufus::Edo::NetTyrant.new(
+    #     '127.0.0.1',
+    #     1978,
+    #     :default_proc => lambda { |cab, key| "not found : '#{k}'" }
+    #   p db['ginger'] # => "not found : 'ginger'"
+    #
+    # The first arg passed to the default_proc is the tyrant itself, so this
+    # opens up interesting possibilities.
+    #
+    def initialize (host, port=0, params={})
 
       @host = host
       @port = port
@@ -84,6 +104,12 @@ module Rufus::Edo
           "tyrant at #{host}:#{port} is a table, " +
           "use Rufus::Edo::NetTyrantTable instead to access it.")
       end
+
+      #
+      # default value|proc
+
+      self.default = params[:default]
+      @default_proc ||= params[:default_proc]
     end
 
     # Returns the 'weight' of the db (in bytes)
@@ -93,14 +119,18 @@ module Rufus::Edo
       self.stat['size']
     end
 
-    # isn't that a bit dangerous ? it creates a file on the server...
+    # Tells the Tyrant server to create a copy of itself at the given (remote)
+    # target_path.
     #
-    # DISABLED.
+    # Returns true when successful.
+    #
+    # Note : if you started your ttserver with a path like "tyrants/data.tch"
+    # you have to provide a target path in the same subdir, like
+    # "tyrants/data_prime.tch".
     #
     def copy (target_path)
 
-      #@db.copy(target_path)
-      raise 'not allowed to create files on the server'
+      @db.copy(target_path) || raise_error
     end
 
     # Copies the current cabinet to a new file.
@@ -130,6 +160,7 @@ module Rufus::Edo
     protected
 
     def do_stat #:nodoc#
+
       @db.stat
     end
   end