extralite 1.5 → 1.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ad8337aebac5ef75132340d8336389fe0c927ed454b5792d57f80efa4f362664
-  data.tar.gz: 602ad3648312e810398fac215353b0fac5aa916ff7a3a7901ba1990c9e3fb145
+  metadata.gz: e2924788672c7c3960a5baeaeacf14d297cfe461cfb1d20db828d7a67b5536c8
+  data.tar.gz: fb4da5551c3b1ddca95ef05520eee477bfb7a29383b9c7e5f5c0638392eba3e9
 SHA512:
-  metadata.gz: a5831721c27d2eb27c8433bcd286ba826861946b7894eb2c1564f449d321d2eaeec7cce13a4166b10db799f9e9e391bc2ba65961edc7b144592d1994767e69b0
-  data.tar.gz: ce77ee584ec89bae2819a2c0384ca7cc08daf00f405d0bf379f322d6182b3bf55bceb0ac0136d52fddd3fe5a43e4e48eb23cbfc00eef74a0439ce25eeeb529f7
+  metadata.gz: 63b73c0d05cb294b75d79bef5c1362e2c422f16b0e7ae76c3d2b577af20f612007a4fe272942958b38d545efabe4d975c1a0475ee8dec4a68e957d61800ffca7
+  data.tar.gz: b79c50cb5c696cf1eb505bbd126b197f1160f56af70e669cfc432e2e16d5423848c139ff9e942fb26c06c3d15f94e1a43f22ab4c317dae4fb0c2290f8d431aa6

data/CHANGELOG.md

@@ -1,3 +1,7 @@
+## 1.6 2021-12-13
+
+- Release GVL while fetching rows
+
 ## 1.5 2021-12-13
 
 - Release GVL while preparing statements

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    extralite (1.5)
+    extralite (1.6)
 
 GEM
   remote: https://rubygems.org/

data/README.md

@@ -6,18 +6,23 @@
 
 ## What is Extralite?
 
-Extralite is an extra-lightweight (less than 400 lines of C-code) SQLite3 wrapper for
-Ruby. It provides a single class with a minimal set of methods to interact with
-an SQLite3 database.
+Extralite is an extra-lightweight (less than 430 lines of C-code) SQLite3
+wrapper for Ruby. It provides a single class with a minimal set of methods to
+interact with an SQLite3 database.
 
 ## Features
 
-- A variety of methods for different data access patterns: row as hash, row as
-  array, single single row, single column, single value.
+- A variety of methods for different data access patterns: rows as hashes, rows
+  as arrays, single row, single column, single value.
+- Super fast - [up to 12.5x faster](#performance) than the
+  [sqlite3](https://github.com/sparklemotion/sqlite3-ruby) gem (see also
+  [comparison](#why-not-just-use-the-sqlite3-gem).)
+- Improved [concurrency](#concurrency) for multithreaded apps: the Ruby GVL is
+  released while preparing SQL statements and while iterating over results.
 - Iterate over records with a block, or collect records into an array.
 - Parameter binding.
-- Correctly execute strings with multiple semicolon-separated queries (handy for
-  creating/modifying schemas).
+- Automatically execute SQL strings containing multiple semicolon-separated
+  queries (handy for creating/modifying schemas).
 - Get last insert rowid.
 - Get number of rows changed by last query.
 - Load extensions (loading of extensions is autmatically enabled. You can find
@@ -80,20 +85,55 @@ db.closed? #=> true
 ## Why not just use the sqlite3 gem?
 
 The sqlite3-ruby gem is a popular, solid, well-maintained project, used by
-thousands of developers. I've been doing a lot of work with SQLite3 lately, and
-wanted to have a simpler API that gives me query results in a variety of ways.
-Thus extralite was born.
+thousands of developers. I've been doing a lot of work with SQLite3 databases
+lately, and wanted to have a simpler API that gives me query results in a
+variety of ways. Thus extralite was born.
+
+Here's a table summarizing the differences between the two gems:
+
+| |sqlite3-ruby|Extralite|
+|-|-|-|
+|API design|multiple classes|single class|
+|Query results|row as hash, row as array, single row, single value|row as hash, row as array, single column, single row, single value|
+|execute multiple statements|separate API (#execute_batch)|integrated|
+|custom functions in Ruby|yes|no|
+|custom collations|yes|no|
+|custom aggregate functions|yes|no|
+|Multithread friendly|no|[yes](#concurrency)|
+|Code size|~2650LoC|~500LoC|
+|Performance|1x|1.5x to 12.5x (see [below](#performance))|
 
 ## What about concurrency?
 
-Extralite currently does not release the GVL. This means that even if queries
-are executed on a separate thread, no other Ruby threads will be scheduled while
-SQLite3 is busy fetching the next record.
+Extralite releases the GVL while making blocking calls to the sqlite3 library,
+that is while preparing SQL statements and fetching rows. Releasing the GVL
+allows other threads to run while the sqlite3 library is busy compiling SQL into
+bytecode, or fetching the next row. This does not seem to hurt Extralite's
+performance:
 
-In the future Extralite might be changed to release the GVL each time
-`sqlite3_step` is called.
+## Performance
+
+A benchmark script is
+[included](https://github.com/digital-fabric/extralite/blob/main/test/perf.rb),
+creating a table of various row counts, then fetching the entire table using
+either `sqlite3` or `extralite`. This benchmark shows Extralite to be up to 12.5
+times faster than `sqlite3` when fetching a large number of rows. Here are the
+results (using the `sqlite3` gem performance as baseline):
+
+|Row count|sqlite3-ruby (baseline)|Extralite (relative - rounded)|
+|-:|-:|-:|
+|10|1x|1.5x|
+|1K|1x|7x|
+|100K|1x|12.5x|
+
+(If you're interested in checking this yourself, just run the script and let me
+know if your results are different.)
 
 ## Can I use it with an ORM like ActiveRecord or Sequel?
 
-Not yet, but you are welcome to contribute adapters for those projects. I will
-be releasing my own not-an-ORM tool in the near future.
+Not yet, but you are welcome to contribute adapters for those projects.
+
+## Contributing
+
+Contributions in the form of issues, PRs or comments will be greatly
+appreciated!

data/ext/extralite/extralite.c

@@ -228,10 +228,21 @@ inline void prepare_multi_stmt(sqlite3 *db, sqlite3_stmt **stmt, VALUE sql) {
   }
 }
 
-inline int stmt_iterate(sqlite3_stmt *stmt, sqlite3 *db) {
+struct step_ctx {
+  sqlite3_stmt *stmt;
   int rc;
-  rc = sqlite3_step(stmt);
-  switch (rc) {
+};
+
+void *stmt_iterate_without_gvl(void *ptr) {
+  struct step_ctx *ctx = (struct step_ctx *)ptr;
+  ctx->rc = sqlite3_step(ctx->stmt);
+  return NULL;
+}
+
+inline int stmt_iterate(sqlite3_stmt *stmt, sqlite3 *db) {
+  struct step_ctx ctx = {stmt, 0};
+  rb_thread_call_without_gvl(stmt_iterate_without_gvl, (void *)&ctx, RUBY_UBF_IO, 0);
+  switch (ctx.rc) {
     case SQLITE_ROW:
       return 1;
     case SQLITE_DONE:
@@ -241,7 +252,7 @@ inline int stmt_iterate(sqlite3_stmt *stmt, sqlite3 *db) {
     case SQLITE_ERROR:
       rb_raise(cSQLError, "%s", sqlite3_errmsg(db));
     default:
-      rb_raise(cError, "Invalid return code for sqlite3_step: %d (please open an issue on https://github.com/digital-fabric/extralite)", rc);
+      rb_raise(cError, "Invalid return code for sqlite3_step: %d (please open an issue on https://github.com/digital-fabric/extralite)", ctx.rc);
   }
 
   return 0;

data/lib/extralite/version.rb

@@ -1,3 +1,3 @@
 module Extralite
-  VERSION = '1.5'
+  VERSION = '1.6'
 end

data/test/perf.rb

@@ -13,36 +13,39 @@ require 'benchmark/ips'
 require 'fileutils'
 
 DB_PATH = '/tmp/extralite_sqlite3_perf.db'
-COUNT = 10000
 
-def prepare_database
+def prepare_database(count)
   FileUtils.rm(DB_PATH) rescue nil
   db = Extralite::Database.new(DB_PATH)
   db.query('create table foo ( a integer primary key, b text )')
   db.query('begin')
-  COUNT.times { db.query('insert into foo (b) values (?)', "hello#{rand(1000)}" )}
+  count.times { db.query('insert into foo (b) values (?)', "hello#{rand(1000)}" )}
   db.query('commit')
 end
 
-def sqlite3_run
+def sqlite3_run(count)
   db = SQLite3::Database.new(DB_PATH, :results_as_hash => true)
   results = db.execute('select * from foo')
-  raise unless results.size == COUNT
+  raise unless results.size == count
 end
 
-def extralite_run
+def extralite_run(count)
   db = Extralite::Database.new(DB_PATH)
   results = db.query('select * from foo')
-  raise unless results.size == COUNT
+  raise unless results.size == count
 end
 
-prepare_database
+[10, 1000, 100000].each do |c|
+  puts; puts; puts "Record count: #{c}"
 
-Benchmark.ips do |x|
-  x.config(:time => 3, :warmup => 1)
+  prepare_database(c)
 
-  x.report("sqlite3") { sqlite3_run }
-  x.report("extralite") { extralite_run }
+  Benchmark.ips do |x|
+    x.config(:time => 3, :warmup => 1)
 
-  x.compare!
+    x.report("sqlite3") { sqlite3_run(c) }
+    x.report("extralite") { extralite_run(c) }
+
+    x.compare!
+  end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: extralite
 version: !ruby/object:Gem::Version
-  version: '1.5'
+  version: '1.6'
 platform: ruby
 authors:
 - Sharon Rosner