slugdb 0.1.0 → 0.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2395e205029cd093867c8593abab596f7565a4429c4aad35e5382f15d06fd32a
-  data.tar.gz: a7ba3d30db63551af9ccdaa602662071102a0cdc974e48a9943663396d6b044c
+  metadata.gz: f61f717430a8d97efaee087d193660aa6424a4f6743d9a0c4123a191d428a335
+  data.tar.gz: 8f8427c80d1d8a78e02863cea58d1938c319cbc1b2919169c822e64090175826
 SHA512:
-  metadata.gz: 9ddf6f107a4a90c2e355330927e755804805b1c5a3b817a6423a5476ab54fe66af104a6d9e2255b32089618dca653285c9a7080a3fd79314fcdcb40478104599
-  data.tar.gz: 9c3009a21fff95f7d3bd472d55db792c6a2188ebf654826b8d74b573057edfa6f9e56c1e63fc3eede79973cf6e8eff54b55df106af4da27439a2a110b491c60f
+  metadata.gz: 507cc4a08fd33abb68c0003ea71bf61867ffc64eb9b5b2930df0c6fd8d1c31952256d6b074f4adbe4c206dd5d7ebaf630da066b3dad012336533594ca0ebfa24
+  data.tar.gz: 6c7660bad22e4df6fcb043b06d86555b026f4d5142e3cec96ef9c30a1500fa45d97f7764565afc6d1583c7254c3649e14ccd39e782797542ef0188a83bd606f0

data/README.md

@@ -1,6 +1,6 @@
-# Slugdb
+# SlugDB
 
-A completely Ruby, lightweight, zero dependency, NoSQL, file based databased.
+A completely native Ruby, lightweight, zero dependency, NoSQL, file based database.
 
 I wanted a tiny database for an embedded project that could follow the advanced data modelling techniques for NoSQL. This is basically a very slimmed down SQLite but for NoSQL without any C bindings.
 
@@ -647,6 +647,44 @@ sdb.query(
 #   :sk=>"2021-03-04T10:30:29Z"}]
 ```
 
+## Performance
+
+Using Ubuntu 20.04 on an i7-10710U with Ruby `3.0.0p0 (2020-12-25 revision 95aff21468) [x86_64-linux]` I got the following:
+
+```
+john@devbox:~/work/ruby-slugdb$ bundle exec bin/benchmark
+Rehearsal -------------------------------------------------------------------------------
+put_item 5 partitions, 1000 items           124.536929   0.864719 125.401648 (125.411962)
+put_item 50 partitions, 100 items           114.684999   0.332007 115.017006 (115.025352)
+put_item 500 partitions, 10 items           118.303101   0.288004 118.591105 (118.601603)
+put_item 5000 partitions, 1 items           160.530284   0.883942 161.414226 (161.427220)
+2 indexes put_item 5 partitions, 1000 items 251.300281   1.031866 252.332147 (252.364579)
+2 indexes put_item 50 partitions, 100 items 235.390541   0.963895 236.354436 (236.383481)
+2 indexes put_item 500 partitions, 10 items 241.453317   0.999958 242.453275 (242.479636)
+2 indexes put_item 5000 partitions, 1 items 310.027647   1.291865 311.319512 (311.353699)
+put_item, get_item 5 partitions, 1000 items 201.732843   0.499968 202.232811 (202.251035)
+put_item, get_item 50 partitions, 100 items 191.160082   0.427998 191.588080 (191.604090)
+put_item, get_item 500 partitions, 10 items 197.149766   0.419959 197.569725 (197.586483)
+put_item, get_item 5000 partitions, 1 items 266.727467   0.575973 267.303440 (267.323639)
+------------------------------------------------------------------- total: 2421.577411sec
+
+                                                  user     system      total        real
+put_item 5 partitions, 1000 items           120.888413   0.347965 121.236378 (121.245881)
+put_item 50 partitions, 100 items           113.471668   0.268009 113.739677 (113.747636)
+put_item 500 partitions, 10 items           117.166616   0.319974 117.486590 (117.492926)
+put_item 5000 partitions, 1 items           158.001298   0.427986 158.429284 (158.434901)
+2 indexes put_item 5 partitions, 1000 items 251.204792   0.719985 251.924777 (251.937582)
+2 indexes put_item 50 partitions, 100 items 235.023671   0.715972 235.739643 (235.755838)
+2 indexes put_item 500 partitions, 10 items 241.072163   0.575991 241.648154 (241.662752)
+2 indexes put_item 5000 partitions, 1 items 310.193882   1.339952 311.533834 (311.556014)
+put_item, get_item 5 partitions, 1000 items 201.417776   0.467966 201.885742 (201.898954)
+put_item, get_item 50 partitions, 100 items 190.772848   0.523983 191.296831 (191.308661)
+put_item, get_item 500 partitions, 10 items 197.940498   0.475973 198.416471 (198.428618)
+put_item, get_item 5000 partitions, 1 items 267.390145   0.715933 268.106078 (268.127310)
+```
+
+A key take away is that PStore kinda sucks with large data sets. My initial use case is hosting maybe a few hundred items. I wanted NoSQL funcitonality that didn't have a third part dep. Making this performant could be a thing but I'm not interested in that right now. PStore serializes the entire hash to disk on every write, keep that in mind.
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
@@ -655,7 +693,7 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/slugdb. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/[USERNAME]/slugdb/blob/master/CODE_OF_CONDUCT.md).
+Bug reports and pull requests are welcome on GitHub at https://github.com/watmin/Ruby-slugdb. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/watmin/Ruby-slugdb/blob/master/CODE_OF_CONDUCT.md).
 
 ## License
 
@@ -663,4 +701,4 @@ The gem is available as open source under the terms of the [MIT License](https:/
 
 ## Code of Conduct
 
-Everyone interacting in the Slugdb project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/slugdb/blob/master/CODE_OF_CONDUCT.md).
+Everyone interacting in the SlugDB project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/watmin/Ruby-slugdb/blob/master/CODE_OF_CONDUCT.md).

data/bin/benchmark

@@ -0,0 +1,159 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'bundler/setup'
+require 'benchmark'
+require 'slugdb'
+require 'pry'
+require 'tempfile'
+
+##
+# Factory to make benchmark inputs
+module BenchmarkFactory
+  module_function
+
+  def partitions(count, size)
+    Array.new(count) { |p| items("p#{p}#", size) }
+  end
+
+  def items(partition, size)
+    Array.new(size) do |x|
+      { pk: partition, sk: "s#{x}#", ipk: partition, isk: "s#{x}#" }
+    end
+  end
+end
+
+five_partitions = BenchmarkFactory.partitions(5, 1000)
+fifty_partitions = BenchmarkFactory.partitions(50, 100)
+five_hundred_partitions = BenchmarkFactory.partitions(500, 10)
+five_thousand_partitions = BenchmarkFactory.partitions(5000, 1)
+
+Benchmark.bmbm do |reporter| # rubocop:disable Metrics/BlockLength
+  reporter.report('put_item 5 partitions, 1000 items') do
+    Tempfile.create do |db_file|
+      db = SlugDB.new(db_file)
+      five_partitions.each do |partition|
+        partition.each { |item| db.put_item(**item) }
+      end
+    end
+  end
+
+  reporter.report('put_item 50 partitions, 100 items') do
+    Tempfile.create do |db_file|
+      db = SlugDB.new(db_file)
+      fifty_partitions.each do |partition|
+        partition.each { |item| db.put_item(**item) }
+      end
+    end
+  end
+
+  reporter.report('put_item 500 partitions, 10 items') do
+    Tempfile.create do |db_file|
+      db = SlugDB.new(db_file)
+      five_hundred_partitions.each do |partition|
+        partition.each { |item| db.put_item(**item) }
+      end
+    end
+  end
+
+  reporter.report('put_item 5000 partitions, 1 items') do
+    Tempfile.create do |db_file|
+      db = SlugDB.new(db_file)
+      five_thousand_partitions.each do |partition|
+        partition.each { |item| db.put_item(**item) }
+      end
+    end
+  end
+
+  reporter.report('2 indexes put_item 5 partitions, 1000 items') do
+    Tempfile.create do |db_file|
+      db = SlugDB.new(db_file)
+      db.add_index(name: :ski, pk: :sk, sk: :pk)
+      db.add_index(name: :i, pk: :ipk, sk: :isk)
+      five_partitions.each do |partition|
+        partition.each { |item| db.put_item(**item) }
+      end
+    end
+  end
+
+  reporter.report('2 indexes put_item 50 partitions, 100 items') do
+    Tempfile.create do |db_file|
+      db = SlugDB.new(db_file)
+      db.add_index(name: :ski, pk: :sk, sk: :pk)
+      db.add_index(name: :i, pk: :ipk, sk: :isk)
+      fifty_partitions.each do |partition|
+        partition.each { |item| db.put_item(**item) }
+      end
+    end
+  end
+
+  reporter.report('2 indexes put_item 500 partitions, 10 items') do
+    Tempfile.create do |db_file|
+      db = SlugDB.new(db_file)
+      db.add_index(name: :ski, pk: :sk, sk: :pk)
+      db.add_index(name: :i, pk: :ipk, sk: :isk)
+      five_hundred_partitions.each do |partition|
+        partition.each { |item| db.put_item(**item) }
+      end
+    end
+  end
+
+  reporter.report('2 indexes put_item 5000 partitions, 1 items') do
+    Tempfile.create do |db_file|
+      db = SlugDB.new(db_file)
+      db.add_index(name: :ski, pk: :sk, sk: :pk)
+      db.add_index(name: :i, pk: :ipk, sk: :isk)
+      five_thousand_partitions.each do |partition|
+        partition.each { |item| db.put_item(**item) }
+      end
+    end
+  end
+
+  reporter.report('put_item, get_item 5 partitions, 1000 items') do
+    Tempfile.create do |db_file|
+      db = SlugDB.new(db_file)
+      five_partitions.each do |partition|
+        partition.each { |item| db.put_item(**item) }
+      end
+      five_partitions.each do |partition| # rubocop:disable Style/CombinableLoops
+        partition.each { |item| db.get_item(**item) }
+      end
+    end
+  end
+
+  reporter.report('put_item, get_item 50 partitions, 100 items') do
+    Tempfile.create do |db_file|
+      db = SlugDB.new(db_file)
+      fifty_partitions.each do |partition|
+        partition.each { |item| db.put_item(**item) }
+      end
+      fifty_partitions.each do |partition| # rubocop:disable Style/CombinableLoops
+        partition.each { |item| db.get_item(**item) }
+      end
+    end
+  end
+
+  reporter.report('put_item, get_item 500 partitions, 10 items') do
+    Tempfile.create do |db_file|
+      db = SlugDB.new(db_file)
+      five_hundred_partitions.each do |partition|
+        partition.each { |item| db.put_item(**item) }
+      end
+      five_hundred_partitions.each do |partition| # rubocop:disable Style/CombinableLoops
+        partition.each { |item| db.get_item(**item) }
+      end
+    end
+  end
+
+  reporter.report('put_item, get_item 5000 partitions, 1 items') do
+    Tempfile.create do |db_file|
+      db = SlugDB.new(db_file)
+      five_thousand_partitions.each do |partition|
+        partition.each { |item| db.put_item(**item) }
+      end
+      five_thousand_partitions.each do |partition| # rubocop:disable Style/CombinableLoops
+        partition.each { |item| db.get_item(**item) }
+      end
+    end
+  end
+end

data/lib/slugdb.rb

@@ -36,7 +36,7 @@ class SlugDB
     end
     reindex!
 
-    { name: { pk: pk, sk: sk } }
+    { name => { pk: pk, sk: sk } }
   end
 
   def list_indexes
@@ -47,7 +47,7 @@ class SlugDB
     @pstore.transaction { |db| db[:main].keys }
   end
 
-  def get_item(pk:, sk:) # rubocop:disable Naming/MethodParameterName
+  def get_item(pk:, sk:, **_) # rubocop:disable Naming/MethodParameterName
     @pstore.transaction do |db|
       next if db[:main][pk].nil? || db[:main][pk][sk].nil?
 
@@ -56,41 +56,32 @@ class SlugDB
   end
 
   def put_item(pk:, sk:, **attributes) # rubocop:disable Naming/MethodParameterName
-    item = attributes.merge(pk: pk, sk: sk)
+    old_item = get_item(pk: pk, sk: sk)
+    new_item = attributes.merge(pk: pk, sk: sk)
     indexes = list_indexes
 
     @pstore.transaction do |db|
+      perform_delete(db, indexes, pk, sk, old_item)
+
       db[:main][pk] ||= {}
-      db[:main][pk][sk] = item
-      indexes.each { |name, schema| index_item(db, item, name, schema) }
+      db[:main][pk][sk] = new_item
+      indexes.each { |name, schema| index_item(db, new_item, name, schema) }
     end
 
-    item
+    new_item
   end
 
-  # rubocop:disable Naming/MethodParameterName,Metrics/AbcSize
+  # rubocop:disable Naming/MethodParameterName
   def delete_item(pk:, sk:, **_)
     item = get_item(pk: pk, sk: sk)
     return if item.nil?
 
     indexes = list_indexes
-    @pstore.transaction do |db|
-      db[:main][pk].delete(sk)
-      db[:main].delete(pk) if db[:main][pk].empty?
-
-      indexes.each do |name, schema|
-        next unless item.key?(schema[:pk]) && item.key?(schema[:sk])
-
-        db[name][item[schema[:pk]]][item[schema[:sk]]][item[:pk]].delete(item[:sk])
-        delete_if_empty?(db[name][item[schema[:pk]]][item[schema[:sk]]], item[:pk])
-        delete_if_empty?(db[name][item[schema[:pk]]], item[schema[:sk]])
-        delete_if_empty?(db[name], item[schema[:pk]])
-      end
-    end
+    @pstore.transaction { |db| perform_delete(db, indexes, pk, sk, item) }
 
     item
   end
-  # rubocop:enable Naming/MethodParameterName,Metrics/AbcSize
+  # rubocop:enable Naming/MethodParameterName
 
   # rubocop:disable Lint/UnusedBlockArgument,Naming/MethodParameterName
   # rubocop:disable Metrics/PerceivedComplexity,Metrics/MethodLength
@@ -136,6 +127,22 @@ class SlugDB
     db[name][item[schema[:pk]]][item[schema[:sk]]][item[:pk]][item[:sk]] = item
   end
 
+  def perform_delete(db, indexes, pk, sk, item) # rubocop:disable Naming/MethodParameterName,Metrics/AbcSize
+    return if item.nil?
+
+    db[:main][pk].delete(sk)
+    db[:main].delete(pk) if db[:main][pk].empty?
+
+    indexes.each do |name, schema|
+      next unless item.key?(schema[:pk]) && item.key?(schema[:sk])
+
+      db[name][item[schema[:pk]]][item[schema[:sk]]][item[:pk]].delete(item[:sk])
+      delete_if_empty?(db[name][item[schema[:pk]]][item[schema[:sk]]], item[:pk])
+      delete_if_empty?(db[name][item[schema[:pk]]], item[schema[:sk]])
+      delete_if_empty?(db[name], item[schema[:pk]])
+    end
+  end
+
   def delete_if_empty?(hash, key)
     hash.delete(key) if hash[key].empty?
   end

data/lib/slugdb/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 class SlugDB
-  VERSION = '0.1.0'
+  VERSION = '0.1.1'
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: slugdb
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.1.1
 platform: ruby
 authors:
 - John Shields <watmin>
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2021-03-04 00:00:00.000000000 Z
+date: 2021-03-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: gem-release
@@ -41,6 +41,7 @@ files:
 - LICENSE.txt
 - README.md
 - Rakefile
+- bin/benchmark
 - bin/console
 - bin/setup
 - lib/slugdb.rb