dbmlite3 1.0.b1 → 2.0.0.pre.alpha.3

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 39c4154795b23715473f861838619458c749ce46a9c5e30abfecde4dee406554
4
- data.tar.gz: efb8323e84cd7b7b45afdbe8c5b8e5611d8ebd8aa315796ea9e7e65341fdac85
3
+ metadata.gz: dc6777e1faeaa79a0ec9b34c5bd5d12cc7568949eea71ae18845f8ff24045796
4
+ data.tar.gz: 909bc62a04f0585355b867af912660b65926665ebbb9b69404d7d461aea6b5d4
5
5
  SHA512:
6
- metadata.gz: 118cb508287926fd7850b832c130468275cac2f4a56a8ca889d3e82255882f196653026059e94b474d43cfdfadbbb0e45ca9cc68681440ee08c7782ef4e80896
7
- data.tar.gz: 31f04ecd6da0d00d684e483c19276425a0b05db40d40e2bcabc8bff69e3da1bb9a6efa452472f3153992cf0c0b4d587b35c0e69e9a34e154ce616721a75f41b9
6
+ metadata.gz: d388662969b8f9cf2d44fe697ee54bd0f58e1feba77bfafeb0061341c03b048d05fa6e877cedf56cc545e98678f02016142a0b1ccfae477abce2ccd53ebbe787
7
+ data.tar.gz: 98ecd13107c15f3db41b0093c4acdfb81f23caa2424d5163ddd2d7e626e08b5cedf7cefb01589ec19717ba14275b94d6825fb87525b7bf270104eca14f5e147c
data/README.md CHANGED
@@ -23,7 +23,7 @@ settings = Lite3::DBM.new("config.sqlite3", "settings")
23
23
 
24
24
  # You use it like a hash
25
25
  settings["speed"] = 88
26
- settings["date"] = Date.new(1955, 11, 5) # Normal Ruby values are allowed
26
+ settings["date"] = Date.new(1955, 11, 5) # Most Ruby types are allowed
27
27
  settings["power_threshold"] = 2.2
28
28
 
29
29
  puts settings['power_threshold']
@@ -65,7 +65,7 @@ Alternately, you can fetch the source code from GitLab and build it yourself:
65
65
  $ cd dbmlite3
66
66
  $ rake
67
67
 
68
- Obviously, it depends on the gem `sqlite3`.
68
+ It depends on the gem `sequel`; previously, it used `sqlite3`.
69
69
 
70
70
  ## Quirks and Hints
71
71
 
@@ -95,6 +95,30 @@ read-modify-write cycle in a transaction:
95
95
  Or, of course, you could just design your script or program so that
96
96
  only one program accesses the table at a time.
97
97
 
98
+ ### Keys must be strings
99
+
100
+ While values may be any serializable type, keys *must* be strings. As
101
+ a special exception, Symbols are also allowed but are transparently
102
+ converted to Strings first. This means that while something like this
103
+ will work:
104
+
105
+ db[:foo] = 42
106
+
107
+ a subseqent
108
+
109
+ db.keys.include?(:foo) or raise AbjectFailure.new
110
+
111
+ will raise an exception because the key `:foo` was turned into a
112
+ string before being used. You will need to do this instead:
113
+
114
+ db.keys.include?('foo') or raise AbjectFailure.new
115
+
116
+ However, this
117
+
118
+ db.has_key?(:foo)
119
+
120
+ will work because `has_key?` does the conversion for us.
121
+
98
122
 
99
123
  ### Transactions and performance
100
124
 
@@ -107,9 +131,9 @@ do these in batches in one or more transactions.
107
131
  `Lite3::DBM` stores Ruby data by first serializing values using the
108
132
  `Marshal` or `Psych` modules. This can pose a security risk if an
109
133
  untrusted third party has direct access to the underlying SQLite3
110
- database. This tends to be pretty rare for most use-cases but if it
111
- is a concern, you can always configure `Lite3::DBM` to store its
112
- values as plain strings.
134
+ database. This tends to be pretty rare most of the time but if it is
135
+ a concern, you can always configure `Lite3::DBM` to store its values
136
+ as plain strings.
113
137
 
114
138
  ### Forking safely
115
139
 
@@ -125,20 +149,47 @@ also lets the child and parent use the same `Lite3::DBM` objects.
125
149
 
126
150
  ### `Lite3::DBM` objects act like file handles but are not
127
151
 
128
- While it is generally safe to treat `Lite3::DBM` as a wrapper
129
- around file handle (i.e. `open` and `close` work as expected), you
130
- should be aware that this is not precisely the way things
131
- actually work. Instead, the gem maintains a pool of database
132
- handles, one per file, and associates them with `Lite3::DBM`
133
- instances as needed. This is necessary for transactions to work
134
- correctly.
135
-
136
- See the reference doc for `Lite3::SQL` for more details.
137
-
138
- Mostly, you don't need to worry about this but certain types of
139
- bugs could behave in unexpected ways and knowing this may help you
140
- make sense of them.
141
-
152
+ While it is generally safe to treat `Lite3::DBM` as a wrapper around a
153
+ file handle (i.e. `open` and `close` work as expected), you should be
154
+ aware that this is not precisely the way things actually work.
155
+ Instead, the gem maintains a pool of database handles, one per file,
156
+ and associates them with `Lite3::DBM` instances as needed. This is
157
+ necessary for transactions to work correctly.
158
+
159
+ Mostly, you don't need to care about this. However, it affects
160
+ you in the following ways:
161
+
162
+ 1. Transactions are done at the file level and not the table level.
163
+ This means that you can access separate tables in the same
164
+ transaction, which is a Very Good Thing.
165
+
166
+ 2. You can safely fork the current process and keep using existing
167
+ `DBM` objects in both processes, provided you've invoked
168
+ `Lite3::SQL.close_all` before the fork. This will have closed the
169
+ actual database handles (which can't tolerate being carried across
170
+ a fork) and opens new ones the next time they're needed.
171
+
172
+ `DBM` objects that go out of scope without first being closed **will**
173
+ eventually have their underlying resources cleaned up. However, given
174
+ that *when* when that happens depends on the vagaries of the garbage
175
+ collector and various library internals, it's almost always a bad idea
176
+ to not explicitly call `close` first.
177
+
178
+ ### Under the hood
179
+
180
+ Currently, `Lite3::DBM` uses [Sequel](https://sequel.jeremyevans.net)
181
+ to access the `sqlite3` library. On JRuby, it goes through the `jdbc`
182
+ interface. The previous version (1.0.0) used
183
+ [sqlite3](https://github.com/sparklemotion/sqlite3-ruby) and only
184
+ worked on MRI. However, you should make no assumptions about the
185
+ underlying database libraries this gem uses. It may change in a
186
+ future release.
187
+
188
+ All tables created by `Lite3::DBM` will have names beginning with
189
+ `dbmlite3_` and you should not modify them directly. It **might** be
190
+ safe to put other tables in the same database file (e.g. via `Sequel`)
191
+ provided that you don't make global changes or mix transactions across
192
+ interfaces. However, I make no guarantees.
142
193
 
143
194
 
144
195
 
data/Rakefile CHANGED
@@ -7,15 +7,16 @@ RSpec::Core::RakeTask.new(:test) do |t|
7
7
  end
8
8
 
9
9
  YARD::Rake::YardocTask.new(:docs_via_yard) do |t|
10
- t.files = ['lib/*.rb']
10
+ t.files = ['lib/*.rb', 'lib/internal_lite3/*.rb']
11
11
  end
12
12
 
13
- task :gem do
14
- `gem build dbmlite3`
13
+ task :gem => [:doc] do
14
+ sh "gem build dbmlite3"
15
+ sh "JRUBY_GEM=yes gem build dbmlite3"
15
16
  end
16
17
 
17
18
  task :clean do
18
- gems = Dir.glob("dbmlite3-*.gem")
19
+ gems = Dir.glob("dbmlite3*.gem")
19
20
  rm gems if gems.size > 0
20
21
  rm_rf "doc"
21
22
  end
data/dbmlite3.gemspec CHANGED
@@ -1,10 +1,25 @@
1
1
 
2
+ JRUBY_GEM = (ENV['JRUBY_GEM'] == "yes")
3
+
4
+ if JRUBY_GEM
5
+ SUFFIX = '_jruby'
6
+ DESC_EXTRA = <<EOF
7
+
8
+ This is the JRuby version of dbmlite3; it is identical to the MRI
9
+ version except for its dependencies.
10
+ EOF
11
+ else
12
+ SUFFIX = ''
13
+ DESC_EXTRA = ''
14
+ end
15
+
16
+
2
17
  Gem::Specification.new do |s|
3
- s.name = 'dbmlite3'
4
- s.version = '1.0.b1'
18
+ s.name = "dbmlite3#{SUFFIX}"
19
+ s.version = '2.0.0-alpha.3'
5
20
  s.date = '2022-02-21'
6
21
  s.summary = "A DBM-style key-value store using SQLite3"
7
- s.description = <<-EOF
22
+ s.description = <<-EOF + DESC_EXTRA
8
23
  Lite3::DBM is an object that behaves like a Ruby Hash but stores
9
24
  its data in a SQLite3 database table. It is a drop-in replacement
10
25
  for DBM.
@@ -19,15 +34,22 @@ EOF
19
34
 
20
35
  # I'm just going to add everything so that if you've got the gem,
21
36
  # you've also got the source distribution. Yay! Open source!
22
- s.files = ["README.md", "LICENSE.txt", "dbmlite3.gemspec",
23
- "Rakefile", ".yardopts"] +
24
- Dir.glob('doc/**/*') +
25
- Dir.glob('{spec,lib}/*.rb')
37
+ s.files = `git ls-files`
38
+ .split
39
+ .reject {|f| f =~ /\.org$/} # Skip local developer notes
26
40
 
27
- s.required_ruby_version = '>= 2.2.0'
28
- s.requirements << "sqlite3 gem, Ruby MRI (required by sqlite3)"
41
+ s.required_ruby_version = '>= 2.7.0'
42
+ s.requirements << (JRUBY_GEM ?
43
+ "Sequel, jdbc-sqlite3, JRuby" :
44
+ "Sequel, sqlite3, Ruby MRI")
45
+
46
+ s.add_runtime_dependency "sequel", '~> 5.65.0'
29
47
 
30
- s.add_runtime_dependency "sqlite3", '~> 1.4'
48
+ if JRUBY_GEM
49
+ s.add_runtime_dependency "jdbc-sqlite3", "~> 3.32.3.3"
50
+ else
51
+ s.add_runtime_dependency "sqlite3", "~> 1.6.1"
52
+ end
31
53
 
32
54
  s.add_development_dependency "rspec", '~> 3.10', '>= 3.10.0'
33
55
  s.add_development_dependency "yard", '~> 0.9.25', '>= 0.9.25'
@@ -0,0 +1,172 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ # Low-effort benchmark comparing Lite3::DBM in various modes against
4
+ # other equivalent Hash-like Ruby storage mechanisms.
5
+
6
+ require 'dbmlite3'
7
+ require 'yaml/dbm' unless RUBY_PLATFORM == "java"
8
+ require 'fileutils'
9
+ require 'optparse'
10
+
11
+ COUNT = 10000
12
+
13
+ Opts = proc do
14
+ opt_defaults = {
15
+ count: COUNT,
16
+ transaction_only: true,
17
+ }
18
+
19
+ opts = Struct.new(*opt_defaults.keys).new(*opt_defaults.values)
20
+
21
+ OptionParser.new do |opo|
22
+ opo.banner = "Usage: bench_1.rb [options]"
23
+
24
+ opo.on("-t", "--multi-transaction",
25
+ "Don't batch Lite3 accesses in one big transaction.") {
26
+ opts.transaction_only = false
27
+ }
28
+
29
+ opo.on("-c", "--count N", Integer, "Set test count.") { |c|
30
+ opts.count = c
31
+ }
32
+ end.parse!
33
+
34
+ next opts
35
+ end.call
36
+
37
+
38
+ module Tmp
39
+ @root = File.join( File.dirname(__FILE__), "tmpdata")
40
+ @count = 0
41
+
42
+ def self.file
43
+ FileUtils.mkdir(@root) unless File.directory?(@root)
44
+
45
+ file = "testfile_#{@count}_#{$$}.sqlite3"
46
+ @count += 1
47
+
48
+ return File.join(@root, file)
49
+ end
50
+
51
+ def self.cleanup
52
+ return unless File.directory?(@root)
53
+ FileUtils.rm_rf(@root)
54
+ end
55
+ end
56
+
57
+
58
+
59
+ def insert(count, db, offset)
60
+ for n in 0 .. count
61
+ db["k_#{n}"] = "#{n + offset}"
62
+ end
63
+ end
64
+
65
+
66
+ def lookup(count, db)
67
+ rnd = Random.new(69_420)
68
+ sz = db.size
69
+
70
+ for _ in 0 .. count * 3
71
+ idx = rnd.rand(sz)
72
+ v = db["k_#{idx}"]
73
+ raise "Invalid value: #{v}" unless v.to_s == "#{idx + 1}"
74
+ end
75
+ end
76
+
77
+
78
+ Times = {}
79
+
80
+ def time_it(type, task, db, use_tr, &block)
81
+ print "#{type} #{task} - "
82
+ STDOUT.flush
83
+
84
+ start = Time.now
85
+
86
+ if use_tr && db.respond_to?(:transaction)
87
+ db.transaction { block.call }
88
+ else
89
+ block.call
90
+ end
91
+
92
+ finish = Time.now
93
+
94
+ elapsed = (finish - start).to_f
95
+ Times[type] = Times.fetch(type, 0) + elapsed
96
+
97
+ puts "#{elapsed.round(4)}"
98
+ end
99
+
100
+ def bench(count, desc, db, use_tr)
101
+ time_it(desc, "insert", db, use_tr) {
102
+ insert(count, db, 0)
103
+ }
104
+ time_it(desc, "upsert", db, use_tr) {
105
+ insert(count, db, 1)
106
+ }
107
+ time_it(desc, "lookup", db, use_tr) {
108
+ lookup(count, db)
109
+ }
110
+ time_it(desc, "delete_if", db, use_tr) {
111
+ rnd = Random.new(69_420)
112
+ db.delete_if{|k,v| rnd.rand(2) == 0}
113
+ }
114
+ puts
115
+ end
116
+
117
+
118
+
119
+ def main
120
+ puts "Count = #{Opts.count}\n"
121
+
122
+ Tmp.cleanup
123
+
124
+ bench(Opts.count, "hash", {}, false)
125
+
126
+ if RUBY_PLATFORM != "java"
127
+ DBM.open(Tmp.file) {|dbm|
128
+ bench(Opts.count, "DBM", dbm, false)
129
+ }
130
+
131
+ YAML::DBM.open(Tmp.file) {|dbm|
132
+ bench(Opts.count, "YAML::DBM", dbm, false)
133
+ }
134
+ end
135
+
136
+ Lite3::DBM.open(Tmp.file, "benchmark", :yaml) { |dbm|
137
+ bench(Opts.count, "Lite3::DBM(yaml)", dbm, true)
138
+ }
139
+
140
+ Lite3::DBM.open(Tmp.file, "benchmark", :marshal) { |dbm|
141
+ bench(Opts.count, "Lite3::DBM(marshal)", dbm, true)
142
+ }
143
+
144
+ Lite3::DBM.open(Tmp.file, "benchmark", :string) { |dbm|
145
+ bench(Opts.count, "Lite3::DBM(string)", dbm, true)
146
+ }
147
+
148
+ if !Opts.transaction_only
149
+ Lite3::DBM.open(Tmp.file, "benchmark", :yaml) { |dbm|
150
+ bench(Opts.count, "Lite3::DBM(yaml, single-trans)", dbm, false)
151
+ }
152
+
153
+ Lite3::DBM.open(Tmp.file, "benchmark", :marshal) { |dbm|
154
+ bench(Opts.count, "Lite3::DBM(marshal, single-trans)", dbm, false)
155
+ }
156
+
157
+ Lite3::DBM.open(Tmp.file, "benchmark", :string) { |dbm|
158
+ bench(Opts.count, "Lite3::DBM(string, single-trans)", dbm, false)
159
+ }
160
+ end
161
+
162
+
163
+ puts
164
+ puts "Totals:"
165
+ Times.each{|k,v|
166
+ puts " #{k} - #{v.round(4)}"
167
+ }
168
+
169
+ Tmp.cleanup
170
+ end
171
+
172
+ main