dbmlite3 1.0.0 → 2.0.0.pre.alpha.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3fdf8e20ae5e85f35d83d1d5e6926864e130614b292f8003a2b895e0c15f10e2
-  data.tar.gz: beb87e9421bcfdb2fb14caaa7a56b27e88b5c8f61239b28ed083477a15431e11
+  metadata.gz: 39bb2916bde4cc6ad424d33a41d04bc361f647727b360dce68a8d1244cbbee6f
+  data.tar.gz: 9fadaf78ad2309b47e7c29d32855757e0a6636805e3021b2e8c9ab99b2e99f46
 SHA512:
-  metadata.gz: 715dcf9851a5a95f301560798aeba2216fdebf9fc20f4fa993d586b3d45b47f6b2a9ad3610e648e7afec685a232d202e13e35cb5eb316bf6e8499b2869379a36
-  data.tar.gz: b77d404a72410cdbd29bca42fdd7ca7d4490858e2ea80b0ce681438b347d8ab6e768134f4793743d14137d6025cc882aade4872591aa0e85b0a8466e0fea08ae
+  metadata.gz: 852902328d1b2ea1ef6c9a95b1cb9cbf4a152a30f9530ee38533f7a56c50759d018c15a11cd70b425056307e3b4839cc3f3809b9bef73b714c9e14ba40a3e5a8
+  data.tar.gz: 6e94518bb9371e45dd195868ce14e3654c1d6263f36099324551dd907df72971b1337d83657647fbae5abccb36b82326fc91282155313d2e0ecd6c7b3a73d51f

data/README.md

@@ -23,7 +23,7 @@ settings = Lite3::DBM.new("config.sqlite3", "settings")
 
 # You use it like a hash
 settings["speed"] = 88
-settings["date"] = Date.new(1955, 11, 5)  # Normal Ruby values are allowed
+settings["date"] = Date.new(1955, 11, 5)  # Most Ruby types are allowed
 settings["power_threshold"] = 2.2
 
 puts settings['power_threshold']
@@ -65,7 +65,7 @@ Alternately, you can fetch the source code from GitLab and build it yourself:
     $ cd dbmlite3
     $ rake
 
-Obviously, it depends on the gem `sqlite3`.
+It depends on the gem `sequel`; previously, it used `sqlite3`.
 
 ## Quirks and Hints
 
@@ -95,6 +95,30 @@ read-modify-write cycle in a transaction:
 Or, of course, you could just design your script or program so that
 only one program accesses the table at a time.
 
+### Keys must be strings
+
+While values may be any serializable type, keys *must* be strings.  As
+a special exception, Symbols are also allowed but are transparently
+converted to Strings first. This means that while something like this
+will work:
+
+    db[:foo] = 42
+
+a subseqent
+
+    db.keys.include?(:foo) or raise AbjectFailure.new
+
+will raise an exception because the key `:foo` was turned into a
+string before being used.  You will need to do this instead:
+
+    db.keys.include?('foo') or raise AbjectFailure.new
+
+However, this
+
+    db.has_key?(:foo)
+
+will work because `has_key?` does the conversion for us.
+
 
 ### Transactions and performance
 
@@ -107,9 +131,9 @@ do these in batches in one or more transactions.
 `Lite3::DBM` stores Ruby data by first serializing values using the
 `Marshal` or `Psych` modules.  This can pose a security risk if an
 untrusted third party has direct access to the underlying SQLite3
-database.  This tends to be pretty rare for most use-cases but if it
-is a concern, you can always configure `Lite3::DBM` to store its
-values as plain strings.
+database.  This tends to be pretty rare most of the time but if it is
+a concern, you can always configure `Lite3::DBM` to store its values
+as plain strings.
 
 ### Forking safely
 
@@ -125,20 +149,47 @@ also lets the child and parent use the same `Lite3::DBM` objects.
 
 ### `Lite3::DBM` objects act like file handles but are not
 
-While it is generally safe to treat `Lite3::DBM` as a wrapper
-around file handle (i.e. `open` and `close` work as expected), you
-should be aware that this is not precisely the way things
-actually work.  Instead, the gem maintains a pool of database
-handles, one per file, and associates them with `Lite3::DBM`
-instances as needed.  This is necessary for transactions to work
-correctly.
-
-See the reference doc for `Lite3::SQL` for more details.
-
-Mostly, you don't need to worry about this but certain types of
-bugs could behave in unexpected ways and knowing this may help you
-make sense of them.
-
+While it is generally safe to treat `Lite3::DBM` as a wrapper around a
+file handle (i.e. `open` and `close` work as expected), you should be
+aware that this is not precisely the way things actually work.
+Instead, the gem maintains a pool of database handles, one per file,
+and associates them with `Lite3::DBM` instances as needed.  This is
+necessary for transactions to work correctly.
+
+Mostly, you don't need to care about this.  However, it affects
+you in the following ways:
+
+1. Transactions are done at the file level and not the table level.
+   This means that you can access separate tables in the same
+   transaction, which is a Very Good Thing.
+
+2. You can safely fork the current process and keep using existing
+   `DBM` objects in both processes, provided you've invoked
+   `Lite3::SQL.close_all` before the fork.  This will have closed the
+   actual database handles (which can't tolerate being carried across
+   a fork) and opens new ones the next time they're needed.
+
+`DBM` objects that go out of scope without first being closed **will**
+eventually have their underlying resources cleaned up.  However, given
+that *when* when that happens depends on the vagaries of the garbage
+collector and various library internals, it's almost always a bad idea
+to not explicitly call `close` first.
+
+### Under the hood
+
+Currently, `Lite3::DBM` uses [Sequel](https://sequel.jeremyevans.net)
+to access the `sqlite3` library.  On JRuby, it goes through the `jdbc`
+interface.  The previous version (1.0.0) used
+[sqlite3](https://github.com/sparklemotion/sqlite3-ruby) and only
+worked on MRI.  However, you should make no assumptions about the
+underlying database libraries this gem uses.  It may change in a
+future release.
+
+All tables created by `Lite3::DBM` will have names beginning with
+`dbmlite3_` and you should not modify them directly.  It **might** be
+safe to put other tables in the same database file (e.g. via `Sequel`)
+provided that you don't make global changes or mix transactions across
+interfaces.  However, I make no guarantees.
 
 
 

data/Rakefile

@@ -7,15 +7,16 @@ RSpec::Core::RakeTask.new(:test) do |t|
 end
 
 YARD::Rake::YardocTask.new(:docs_via_yard) do |t|
-  t.files = ['lib/*.rb']
+  t.files = ['lib/*.rb', 'lib/internal_lite3/*.rb']
 end
 
-task :gem do
-  `gem build dbmlite3`
+task :gem => [:doc] do
+  sh "gem build dbmlite3"
+  sh "JRUBY_GEM=yes gem build dbmlite3"
 end
 
 task :clean do
-  gems = Dir.glob("dbmlite3-*.gem")
+  gems = Dir.glob("dbmlite3*.gem")
   rm gems if gems.size > 0
   rm_rf "doc"
 end

data/dbmlite3.gemspec

@@ -1,10 +1,27 @@
 
+JRUBY_GEM = (ENV['JRUBY_GEM'] == "yes")
+
+if JRUBY_GEM
+  SUFFIX = '_jruby'
+  DESC_EXTRA = <<EOF
+
+    This is the JRuby version of dbmlite3; it is identical to the MRI
+    version except for its dependencies.
+EOF
+  SUMMARY_EXTRA = " (JRuby version)"
+else
+  SUFFIX = ''
+  DESC_EXTRA = ''
+  SUMMARY_EXTRA = ''
+end
+
+
 Gem::Specification.new do |s|
-  s.name        = 'dbmlite3'
-  s.version     = '1.0.0'
+  s.name        = "dbmlite3#{SUFFIX}"
+  s.version     = '2.0.0-alpha.4'
   s.date        = '2022-02-21'
-  s.summary     = "A DBM-style key-value store using SQLite3"
-  s.description = <<-EOF
+  s.summary     = "A DBM-style key-value store using SQLite3#{SUMMARY_EXTRA}"
+  s.description = <<-EOF + DESC_EXTRA
     Lite3::DBM is an object that behaves like a Ruby Hash but stores
     its data in a SQLite3 database table.  It is a drop-in replacement
     for DBM.
@@ -19,15 +36,22 @@ EOF
 
   # I'm just going to add everything so that if you've got the gem,
   # you've also got the source distribution.  Yay! Open source!
-  s.files       = ["README.md", "LICENSE.txt", "dbmlite3.gemspec",
-                   "Rakefile", ".yardopts"] +
-                  Dir.glob('doc/**/*') +
-                  Dir.glob('{spec,lib}/*.rb')
+  s.files       = `git ls-files`
+    .split
+    .reject {|f| f =~ /\.org$/}     # Skip local developer notes
 
-  s.required_ruby_version = '>= 2.2.0'
-  s.requirements << "sqlite3 gem, Ruby MRI (required by sqlite3)"
+  s.required_ruby_version = '>= 2.7.0'
+  s.requirements << (JRUBY_GEM                          ?
+                     "Sequel, jdbc-sqlite3, JRuby"  :
+                     "Sequel, sqlite3, Ruby MRI")
+  
+  s.add_runtime_dependency "sequel", '~> 5.65.0'
 
-  s.add_runtime_dependency "sqlite3", '~> 1.4'
+  if JRUBY_GEM
+    s.add_runtime_dependency "jdbc-sqlite3", "~> 3.32.3.3"
+  else
+    s.add_runtime_dependency "sqlite3", "~> 1.6.1"
+  end
   
   s.add_development_dependency "rspec", '~> 3.10', '>= 3.10.0'
   s.add_development_dependency "yard", '~> 0.9.25', '>= 0.9.25'

data/extras/benchmark.rb

@@ -0,0 +1,172 @@
+#!/usr/bin/env ruby
+
+# Low-effort benchmark comparing Lite3::DBM in various modes against
+# other equivalent Hash-like Ruby storage mechanisms.
+
+require 'dbmlite3'
+require 'yaml/dbm' unless RUBY_PLATFORM == "java"
+require 'fileutils'
+require 'optparse'
+
+COUNT = 10000
+
+Opts = proc do
+  opt_defaults = {
+    count: COUNT,
+    transaction_only: true,
+  }
+
+  opts = Struct.new(*opt_defaults.keys).new(*opt_defaults.values)
+
+  OptionParser.new do |opo|
+    opo.banner = "Usage: bench_1.rb [options]"
+
+    opo.on("-t", "--multi-transaction",
+           "Don't batch Lite3 accesses in one big transaction.") {
+      opts.transaction_only = false
+    }
+
+    opo.on("-c", "--count N", Integer, "Set test count.") { |c|
+      opts.count = c
+    }
+  end.parse!
+
+  next opts
+end.call
+
+
+module Tmp
+  @root = File.join( File.dirname(__FILE__), "tmpdata")
+  @count = 0
+
+  def self.file
+    FileUtils.mkdir(@root) unless File.directory?(@root)
+
+    file = "testfile_#{@count}_#{$$}.sqlite3"
+    @count += 1
+
+    return File.join(@root, file)
+  end
+
+  def self.cleanup
+    return unless File.directory?(@root)
+    FileUtils.rm_rf(@root)
+  end
+end
+
+
+
+def insert(count, db, offset)
+  for n in 0 .. count
+    db["k_#{n}"] = "#{n + offset}"
+  end
+end
+
+
+def lookup(count, db)
+  rnd = Random.new(69_420)
+  sz = db.size
+
+  for _ in 0 .. count * 3
+    idx = rnd.rand(sz)
+    v = db["k_#{idx}"]
+    raise "Invalid value: #{v}" unless v.to_s == "#{idx + 1}"
+  end
+end
+
+
+Times = {}
+
+def time_it(type, task, db, use_tr, &block)
+  print "#{type} #{task} - "
+  STDOUT.flush
+
+  start = Time.now
+
+  if use_tr && db.respond_to?(:transaction)
+    db.transaction { block.call }
+  else
+    block.call
+  end
+
+  finish = Time.now
+
+  elapsed = (finish - start).to_f
+  Times[type] = Times.fetch(type, 0) + elapsed
+
+  puts "#{elapsed.round(4)}"
+end
+
+def bench(count, desc, db, use_tr)
+  time_it(desc, "insert", db, use_tr) {
+    insert(count, db, 0)
+  }
+  time_it(desc, "upsert", db, use_tr) {
+    insert(count, db, 1)
+  }
+  time_it(desc, "lookup", db, use_tr) {
+    lookup(count, db)
+  }
+  time_it(desc, "delete_if", db, use_tr) {
+    rnd = Random.new(69_420)
+    db.delete_if{|k,v| rnd.rand(2) == 0}
+  }
+  puts
+end
+
+
+
+def main
+  puts "Count = #{Opts.count}\n"
+
+  Tmp.cleanup
+
+  bench(Opts.count, "hash", {}, false)
+
+  if RUBY_PLATFORM != "java"
+    DBM.open(Tmp.file) {|dbm|
+      bench(Opts.count, "DBM", dbm, false)
+    }
+  
+    YAML::DBM.open(Tmp.file) {|dbm|
+      bench(Opts.count, "YAML::DBM", dbm, false)
+    }
+  end
+
+  Lite3::DBM.open(Tmp.file, "benchmark", :yaml) { |dbm|
+    bench(Opts.count, "Lite3::DBM(yaml)", dbm, true)
+  }
+
+  Lite3::DBM.open(Tmp.file, "benchmark", :marshal) { |dbm|
+    bench(Opts.count, "Lite3::DBM(marshal)", dbm, true)
+  }
+
+  Lite3::DBM.open(Tmp.file, "benchmark", :string) { |dbm|
+    bench(Opts.count, "Lite3::DBM(string)", dbm, true)
+  }
+
+  if !Opts.transaction_only
+    Lite3::DBM.open(Tmp.file, "benchmark", :yaml) { |dbm|
+      bench(Opts.count, "Lite3::DBM(yaml, single-trans)", dbm, false)
+    }
+
+    Lite3::DBM.open(Tmp.file, "benchmark", :marshal) { |dbm|
+      bench(Opts.count, "Lite3::DBM(marshal, single-trans)", dbm, false)
+    }
+
+    Lite3::DBM.open(Tmp.file, "benchmark", :string) { |dbm|
+      bench(Opts.count, "Lite3::DBM(string, single-trans)", dbm, false)
+    }
+  end
+
+
+  puts
+  puts "Totals:"
+  Times.each{|k,v|
+    puts "    #{k} - #{v.round(4)}"
+  }
+
+  Tmp.cleanup
+end
+
+main