fsdb 0.5 → 0.6.0

data/{RELEASE-NOTES → History.txt}

@@ -1,3 +1,9 @@
+fsdb 0.6.0
+
+- Replaced ftools with fileutils.
+
+- Released as a gem using Mr. Bones.
+
 fsdb 0.5
 
 - If ruby is 1.8.2 or later, flock doesn't block other threads, so accessing

data/{README → README.txt}

@@ -12,8 +12,16 @@ FSDB does not yet have any indexing or querying mechanisms, and is probably miss
 
 == Installation
 
+For .gem:
+
   gem install fsdb
 
+For .tgz, unpack and then:
+
+  ruby install.rb config
+  ruby install.rb setup
+  ruby install.rb install
+
 == Synopsis
 
   require 'fsdb'
@@ -71,8 +79,8 @@ Keys in the database are path strings, which are simply strings in the usual for
     foo//bar
     foo/../foo/bar
   
-  work correctly, as do paths that denote hard or soft links, if supported on
-  the platform.
+  work correctly (they access the same objects), as do paths that denote hard
+  or soft links, if supported on the platform.
 
   Links are subject to the same naming convention as normal files with regard
   to format identification: format is determined by the path within the
@@ -136,12 +144,11 @@ There are two kinds of transactions:
   with one that does not have the <tt>+</tt> method or the <tt>[ ]</tt> method.
   The four operations are atomic by themselves, but the sequence is not.
 
-  Note that changes to a database object using this kind of transaction
-  cannot be made using destructive methods (such as <tt><<</tt>) but only by
-  assignments of the form <tt>db[<path>] = <data></tt>. Note that
-  <tt>+=</tt> and similar "assignment operators" can be used but are not atomic,
-  because
-  
+  Note that changes to a database object using this kind of transaction cannot
+  be made using destructive methods (such as <tt><<</tt>) but only by
+  assignments of the form <tt>db[<path>] = <data></tt>. Note that <tt>+=</tt>
+  and similar "assignment operators" can be used but are not atomic, because
+
     db[<path>] += 1
   
   is really
@@ -163,7 +170,7 @@ There are two kinds of transactions:
 
   This guarantees that, if the object at the path is still <tt>[1, 2, 3]</tt>
   at the time of the #edit call, the value returned by the transaction will be
-  +4+.
+  4.
 
   Simply put, #edit allows exclusive write access to the object at the path for
   the duration of the block. Other threads or processes that use FSDB methods
@@ -263,7 +270,7 @@ FSDB has been tested on the following platforms and file systems:
   
   - Windows ME (single cpu, FAT32)
 
-FSDB is currently tested with ruby-1.9.0 and ruby-1.8.1.
+FSDB is currently tested with ruby-1.9.1 and ruby-1.8.6.
 
 On windows, both the mswin32 and mingw32 builds of ruby have been used with FSDB. It has never been tested with cygwin or bccwin.
 
@@ -304,12 +311,12 @@ FSDB is not very fast. It's useful more for its safety, flexibility, and ease of
 
 - For speed, avoid using #fetch and its alias #[]. As noted in the API docs,
   these methods cannot safely return the same object that is cached, so must
-  clear out the cache's reference to the object so that is will be loaded
+  clear out the cache's reference to the object so that it will be loaded
   freshly the next time #fetch is called on the path.
   
   The performance hit of #fetch is of course greater with larger objects,
   and with objects that are loaded by a more complex procedure, such as
-  Masrshal.load.
+  Marshal.load.
   
   You can think of #fetch as a "deep copy" of the object. If you call it twice,
   you get different copies that do not share any parts. Or you can think of it
@@ -372,7 +379,7 @@ FSDB is not very fast. It's useful more for its safety, flexibility, and ease of
 - It may be fast enough for many purposes, especially using multiple processes
   rather than multiple threads.
   
-- Pure ruby. Ruby license. Free sotware.
+- Pure ruby. Ruby license. Free software.
 
 
 == Applications
@@ -396,6 +403,8 @@ is:
 
 === Features
 
+- Should the Format objects be classes instead of just instances of Format?
+
 - Default value and proc for Database, like Hash.
 
 - FSDB::Reference class:
@@ -460,6 +469,8 @@ is:
 
 - more formats
 
+  - json
+
   - tabular data, excel, xml, ascii db, csv
   
   - SOAP marshal, XML marshal
@@ -520,9 +531,7 @@ is:
   wipe the reference to the object, and could leave the contents string. But
   this would increase file size and duplicate the file system's own cache.
 
-== Version
-
-fsdb 0.5
+== Web site
 
 The current version of this software can be found at http://redshift.sourceforge.net/fsdb.
 
@@ -533,4 +542,4 @@ This software is distributed under the Ruby license. See http://www.ruby-lang.or
 == Author
 
 Joel VanderWerf, mailto:vjoel@users.sourceforge.net
-Copyright (c) 2005, Joel VanderWerf.
+Copyright (c) 2003-2009, Joel VanderWerf.

data/examples/flat.rb

@@ -0,0 +1,146 @@
+# Fast, flat storage based on Kirk Haines' technique.
+
+require 'fsdb'
+require 'digest/md5'
+
+class FlatDB < FSDB::Database
+
+  def initialize(path, depth = 2)
+    raise ArgumentError, "Invalid depth #{depth} > 32" if depth > 32
+
+    @path_from_key = {}
+    @path_pat = Regexp.new("^" + "(..)"*depth)
+    @depth = depth
+
+    super(path)
+  end
+  
+  def path_from_key(key)
+    path = @path_from_key[key]
+    unless path
+      if @depth > 0
+        path_components = Digest::MD5.hexdigest(key).scan(@path_pat).first
+        path_components << key
+        path = path_components.join("/")
+      else
+        path = key
+      end
+      @path_from_key[key] = path
+    end
+    path
+  end
+
+  def browse(key)
+    super path_from_key(key)
+  end
+
+  def edit(key)
+    super path_from_key(key)
+  end
+
+  def replace(key)
+    super path_from_key(key)
+  end
+
+  def delete(key, load=true)
+    @path_from_key.delete key ## should probably purge this hash periodically
+    super path_from_key(key), load
+  end
+
+  # don't bother with #insert, #fetch, #[], #[]= since they are
+  # inherently less efficient
+end
+
+db = FlatDB.new('/tmp/fsdb/flat', 2)
+
+db.replace 'foo.txt' do
+  "this is the foo text"
+end
+
+db.browse 'foo.txt' do |x|
+  p x
+end
+
+# key names can have '/' in them, in which case they reference deeper subdirs
+db.replace 'sub/foo.txt' do
+  "this is the subdir's foo text"
+end
+
+db.browse 'sub/foo.txt' do |x|
+  p x
+end
+
+require 'benchmark'
+
+Benchmark.bm(10) do |bm|
+  nfiles = 100
+
+  [0,1,2].each do |depth|
+    db = FlatDB.new("/tmp/fsdb/depth_#{depth}", depth)
+    
+    puts "\ndepth=#{depth}"
+
+    bm.report "create" do
+      nfiles.times do |i|
+        db.replace i.to_s do  # with a filename like that, will use marshal
+          i
+        end
+      end
+    end
+
+    bm.report "access" do
+      nfiles.times do |i|
+        db.browse i.to_s do |j|
+          raise unless i == j
+        end
+      end
+    end
+  end
+end
+
+__END__
+
+N=100_000
+                user     system      total        real
+
+depth=0
+create     72.680000 1772.030000 1844.710000 (1853.686824)
+access     55.780000  13.090000  68.870000 ( 97.170382)
+
+depth=1
+create    125.170000  24.250000 149.420000 (329.576419)
+access    143.210000  12.040000 155.250000 (759.768371)
+
+depth=2
+create    263.900000  32.570000 296.470000 (1950.482468)
+access    195.200000  17.250000 212.450000 (1562.214063)
+
+# du -sk depth_0
+804236  depth_0
+# du -sk depth_1
+804832  depth_1
+# du -sk depth_2
+1006408 depth_2
+
+Output of two successive runs, first without the db files already existing (N=10000, depth=2):
+
+$ ruby flat.rb
+                user     system      total        real
+create      5.270000   2.520000   7.790000 ( 41.684685)
+access      2.210000   0.380000   2.590000 (  2.626480)
+$ ruby flat.rb
+                user     system      total        real
+create      3.020000   1.930000   4.950000 (  5.954889)
+access      1.830000   0.280000   2.110000 (  2.157327)
+
+Output of two successive runs, first without the db files already existing (N=10000, depth=0):
+
+$ ruby flat.rb
+                user     system      total        real
+create      2.780000  18.710000  21.490000 ( 22.292284)
+access      1.800000   0.230000   2.030000 (  2.044320)
+$ ruby flat.rb
+                user     system      total        real
+create      3.280000   2.550000   5.830000 (  6.105209)
+access      1.790000   0.300000   2.090000 (  2.200052)
+

data/examples/fsdb-example.rb

@@ -0,0 +1,28 @@
+require 'fsdb' # http://redshift.sourceforge.net/fsdb
+
+db = FSDB::Database.new("~/tmp")
+
+# Tell FSDB to use YAML format when it sees ".yml" and ".yaml" suffixes.
+# It is easy to extend the recognition rules to handle other formats.
+db.formats = [FSDB::YAML_FORMAT] + db.formats
+
+# Write (atomically) the initial data to the file.
+db["config.yml"] = { "key1" => 111, "key2" => 222 }
+
+# Enter a thread-safe and process-safe transaction to
+# change the state of the file.
+db.edit("config.yml") do |cfg|
+  cfg['key1'] = 'aaa'
+  cfg['key2'] = 'bbb'
+end
+
+# Check that it works.
+puts File.read(File.expand_path("~/tmp/config.yml"))
+
+__END__
+
+Output:
+
+--- 
+key1: aaa
+key2: bbb

data/examples/rbformat.rb

@@ -0,0 +1,17 @@
+require 'fsdb'
+
+dir ='~/tmp/fsdb/rb-format'
+db = FSDB::Database.new(dir)
+
+RUBY_FORMAT =
+  FSDB::Format.new(
+    /\.rb$/i,
+    :name => "RUBY_FORMAT",
+    :load => proc {|f| eval "proc {#{f.read}}"},
+    :dump => proc {|string, f| f.syswrite(string)}
+  )
+
+db.formats = [ RUBY_FORMAT ]
+
+db['some-script.rb'] = "|x| 1+2+x"
+p db['some-script.rb'][3]

data/examples/yaml2.rb

@@ -0,0 +1,29 @@
+  require 'fsdb'
+  require 'yaml'
+
+  dir = File.join(ENV['TMPDIR'] || ENV['TMP'] || 'tmp', "my_data")
+
+  db = FSDB::Database.new dir
+  db.formats = [FSDB::YAML_FORMAT] + db.formats
+
+  3.times do |i|
+    db["Cat#{i}.yml"] = %w{
+      name1
+      name2
+      name3
+    }
+  end
+
+  path = "Cat1.yml"
+
+  puts "Here's the object:"
+  puts "=================="
+  p db[path]
+  puts "=================="
+  puts
+
+  puts "Here's the file:"
+  puts "=================="
+  puts File.read(File.join(db.dir, path))
+  puts "=================="
+  puts

data/junk/OLDRakefile

@@ -0,0 +1,98 @@
+
+require 'rake'
+require 'rake/clean'
+require 'rake/contrib/sshpublisher'
+require 'rake/gempackagetask'
+
+PKG_VERSION = IO::read 'VERSION'
+PKG_VERSION.strip!
+PKG_NAME = "fsdb"
+PKG_FILE_NAME = "#{PKG_NAME}-#{PKG_VERSION}"
+
+#-----------------------------------------------------------------------
+desc "Default Task"
+task :default => :package
+
+#-----------------------------------------------------------------------
+# cleanup
+#-----------------------------------------------------------------------
+CLOBBER.include(
+    "pkg/**",
+    "README"
+    )
+#-----------------------------------------------------------------------
+# generate README
+#
+# - maybe make a task lib out of the gen_readme.rb code ?
+#
+#-----------------------------------------------------------------------
+desc "Generate README"
+task :readme => [:clobber] do
+    File.open('README','w') do |f|
+        $stdout = f
+        tmpl = IO::read 'gen_readme.rb'
+        eval tmpl
+        $stdout = STDOUT
+    end
+end
+
+
+#-----------------------------------------------------------------------
+# Publish to website
+#-----------------------------------------------------------------------
+desc "Publish the gem on codeforpeople.com"
+
+PUBLISH_HOST = "codeforpeople.com"
+PUBLISH_ROOT = "/lib/ruby"
+task :publish => [:package] do
+    `ssh #{PUBLISH_HOST} 'mkdir -p #{PUBLISH_ROOT}/#{PKG_NAME}/#{PKG_FILE_NAME}'`
+    `scp -rq ./ #{PUBLISH_HOST}:#{PUBLISH_ROOT}/#{PKG_NAME}/#{PKG_FILE_NAME}`
+
+    # if packages should go somewhere else
+  Rake::SshFilePublisher.new(PUBLISH_HOST, "#{PUBLISH_ROOT}/pkg", "pkg", "#{PKG_FILE_NAME}.zip").upload
+  Rake::SshFilePublisher.new(PUBLISH_HOST, "#{PUBLISH_ROOT}/pkg", "pkg", "#{PKG_FILE_NAME}.tgz").upload
+  Rake::SshFilePublisher.new(PUBLISH_HOST, "#{PUBLISH_ROOT}/gems", "pkg", "#{PKG_FILE_NAME}.gem").upload
+
+end
+
+#-----------------------------------------------------------------------
+# install
+#-----------------------------------------------------------------------
+task :install => [:package] do
+  `gem install pkg/#{PKG_FILE_NAME}.gem`
+end
+
+#-----------------------------------------------------------------------
+# gem specification for :package
+#-----------------------------------------------------------------------
+task :package => [:clobber, :readme]
+
+spec = Gem::Specification.new do |s|
+  s.name = PKG_NAME
+  s.version = PKG_VERSION
+  s.platform = Gem::Platform::RUBY
+  s.summary = "better living through metaprogramming:"
+  s.description = <<DESC
+  traits.rb aims to be a better set of attr_* methods and encourages
+  better living through meta-programming and uniform access priciples.
+  traits.rb supercedes attributes.rb.  why?  the name is shorter ;-)
+DESC
+  
+  s.has_rdoc = false
+
+  # no dependencies
+  # s.add_dependency('gem', '>= version')
+
+  s.files = %w(README VERSION) + Dir['lib/**/*'] + Dir['sample/**/*']
+  s.require_path = '.'
+
+  s.author = "Ara T. Howard"
+  s.email = "ara [dot] t [dot] howard [at] noaa [dot] gov"
+
+end
+
+Rake::GemPackageTask.new(spec) do |p|
+  p.gem_spec = spec
+  p.need_tar = true
+  p.need_zip = true
+end