extralite 1.2 → 1.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a6bb308b421895a192cdbc83d022e105ec1fbd4060d8f8898a71f9f13741bd18
-  data.tar.gz: b044f3d91692b19f7c0cc5912896cec1458c3101ebb5f058fe09c147a3cf00ea
+  metadata.gz: e2924788672c7c3960a5baeaeacf14d297cfe461cfb1d20db828d7a67b5536c8
+  data.tar.gz: fb4da5551c3b1ddca95ef05520eee477bfb7a29383b9c7e5f5c0638392eba3e9
 SHA512:
-  metadata.gz: 4bb40f58b77290b083466563465232cc3aadfa8cb68d305d08edc7fd4b1344d256af45f84f7efe8765fe00b604010a071b5966fd15abb63896b7d843337ce683
-  data.tar.gz: b29ee3ff34f307d9f45515971ecd0b147fe236f6c2d95e4b48cdd1a91b3d9847d477fee2c667836f83db5c68f559a0f4f2129d12b8d5d1fe5f075ca2cdd6d05c
+  metadata.gz: 63b73c0d05cb294b75d79bef5c1362e2c422f16b0e7ae76c3d2b577af20f612007a4fe272942958b38d545efabe4d975c1a0475ee8dec4a68e957d61800ffca7
+  data.tar.gz: b79c50cb5c696cf1eb505bbd126b197f1160f56af70e669cfc432e2e16d5423848c139ff9e942fb26c06c3d15f94e1a43f22ab4c317dae4fb0c2290f8d431aa6

data/CHANGELOG.md

@@ -1,3 +1,20 @@
+## 1.6 2021-12-13
+
+- Release GVL while fetching rows
+
+## 1.5 2021-12-13
+
+- Release GVL while preparing statements
+- Use `sqlite3_prepare_v2` instead of deprecated `sqlite_prepare`
+
+## 1.4 2021-08-25
+
+- Fix possible segfault in cleanup_stmt
+
+## 1.3 2021-08-17
+
+- Pin error classes (for better compatibility with `GC.compact`)
+
 ## 1.2 2021-06-06
 
 - Add support for big integers

data/Gemfile.lock

@@ -1,24 +1,17 @@
 PATH
   remote: .
   specs:
-    extralite (1.2)
+    extralite (1.6)
 
 GEM
   remote: https://rubygems.org/
   specs:
-    ansi (1.5.0)
     ast (2.4.2)
-    builder (3.2.4)
     coderay (1.1.3)
     docile (1.4.0)
     json (2.5.1)
     method_source (1.0.0)
     minitest (5.14.4)
-    minitest-reporters (1.4.2)
-      ansi
-      builder
-      minitest (>= 5.0)
-      ruby-progressbar
     parallel (1.20.1)
     parser (3.0.1.1)
       ast (~> 2.4.1)
@@ -56,7 +49,6 @@ PLATFORMS
 DEPENDENCIES
   extralite!
   minitest (= 5.14.4)
-  minitest-reporters (= 1.4.2)
   pry (= 0.13.1)
   rake-compiler (= 1.1.1)
   rubocop (= 0.85.1)

data/README.md

@@ -6,18 +6,23 @@
 
 ## What is Extralite?
 
-Extralite is an extra-lightweight (less than 400 lines of C-code) SQLite3 wrapper for
-Ruby. It provides a single class with a minimal set of methods to interact with
-an SQLite3 database.
+Extralite is an extra-lightweight (less than 430 lines of C-code) SQLite3
+wrapper for Ruby. It provides a single class with a minimal set of methods to
+interact with an SQLite3 database.
 
 ## Features
 
-- A variety of methods for different data access patterns: row as hash, row as
-  array, single single row, single column, single value.
+- A variety of methods for different data access patterns: rows as hashes, rows
+  as arrays, single row, single column, single value.
+- Super fast - [up to 12.5x faster](#performance) than the
+  [sqlite3](https://github.com/sparklemotion/sqlite3-ruby) gem (see also
+  [comparison](#why-not-just-use-the-sqlite3-gem).)
+- Improved [concurrency](#concurrency) for multithreaded apps: the Ruby GVL is
+  released while preparing SQL statements and while iterating over results.
 - Iterate over records with a block, or collect records into an array.
 - Parameter binding.
-- Correctly execute strings with multiple semicolon-separated queries (handy for
-  creating/modifying schemas).
+- Automatically execute SQL strings containing multiple semicolon-separated
+  queries (handy for creating/modifying schemas).
 - Get last insert rowid.
 - Get number of rows changed by last query.
 - Load extensions (loading of extensions is autmatically enabled. You can find
@@ -80,20 +85,55 @@ db.closed? #=> true
 ## Why not just use the sqlite3 gem?
 
 The sqlite3-ruby gem is a popular, solid, well-maintained project, used by
-thousands of developers. I've been doing a lot of work with SQLite3 lately, and
-wanted to have a simpler API that gives me query results in a variety of ways.
-Thus extralite was born.
+thousands of developers. I've been doing a lot of work with SQLite3 databases
+lately, and wanted to have a simpler API that gives me query results in a
+variety of ways. Thus extralite was born.
+
+Here's a table summarizing the differences between the two gems:
+
+| |sqlite3-ruby|Extralite|
+|-|-|-|
+|API design|multiple classes|single class|
+|Query results|row as hash, row as array, single row, single value|row as hash, row as array, single column, single row, single value|
+|execute multiple statements|separate API (#execute_batch)|integrated|
+|custom functions in Ruby|yes|no|
+|custom collations|yes|no|
+|custom aggregate functions|yes|no|
+|Multithread friendly|no|[yes](#concurrency)|
+|Code size|~2650LoC|~500LoC|
+|Performance|1x|1.5x to 12.5x (see [below](#performance))|
 
 ## What about concurrency?
 
-Extralite currently does not release the GVL. This means that even if queries
-are executed on a separate thread, no other Ruby threads will be scheduled while
-SQLite3 is busy fetching the next record.
+Extralite releases the GVL while making blocking calls to the sqlite3 library,
+that is while preparing SQL statements and fetching rows. Releasing the GVL
+allows other threads to run while the sqlite3 library is busy compiling SQL into
+bytecode, or fetching the next row. This does not seem to hurt Extralite's
+performance:
 
-In the future Extralite might be changed to release the GVL each time
-`sqlite3_step` is called.
+## Performance
+
+A benchmark script is
+[included](https://github.com/digital-fabric/extralite/blob/main/test/perf.rb),
+creating a table of various row counts, then fetching the entire table using
+either `sqlite3` or `extralite`. This benchmark shows Extralite to be up to 12.5
+times faster than `sqlite3` when fetching a large number of rows. Here are the
+results (using the `sqlite3` gem performance as baseline):
+
+|Row count|sqlite3-ruby (baseline)|Extralite (relative - rounded)|
+|-:|-:|-:|
+|10|1x|1.5x|
+|1K|1x|7x|
+|100K|1x|12.5x|
+
+(If you're interested in checking this yourself, just run the script and let me
+know if your results are different.)
 
 ## Can I use it with an ORM like ActiveRecord or Sequel?
 
-Not yet, but you are welcome to contribute adapters for those projects. I will
-be releasing my own not-an-ORM tool in the near future.
+Not yet, but you are welcome to contribute adapters for those projects.
+
+## Contributing
+
+Contributions in the form of issues, PRs or comments will be greatly
+appreciated!

data/ext/extralite/extralite.c

@@ -1,5 +1,6 @@
 #include <stdio.h>
 #include "ruby.h"
+#include "ruby/thread.h"
 #include <sqlite3.h>
 
 VALUE cError;
@@ -109,7 +110,7 @@ inline VALUE get_column_value(sqlite3_stmt *stmt, int col, int type) {
 }
 
 static inline void bind_parameter_value(sqlite3_stmt *stmt, int pos, VALUE value) {
-  switch (TYPE(value)) { 
+  switch (TYPE(value)) {
     case T_NIL:
       sqlite3_bind_null(stmt, pos);
       return;
@@ -143,7 +144,7 @@ static inline void bind_all_parameters(sqlite3_stmt *stmt, int argc, VALUE *argv
 
 static inline VALUE get_column_names(sqlite3_stmt *stmt, int column_count) {
   VALUE arr = rb_ary_new2(column_count);
-  for (int i = 0; i < column_count; i++) { 
+  for (int i = 0; i < column_count; i++) {
     VALUE name = ID2SYM(rb_intern(sqlite3_column_name(stmt, i)));
     rb_ary_push(arr, name);
   }
@@ -168,36 +169,80 @@ static inline VALUE row_to_ary(sqlite3_stmt *stmt, int column_count) {
   return row;
 }
 
-inline void prepare_multi_stmt(sqlite3 *db, sqlite3_stmt **stmt, VALUE sql) {
-  const char *rest = 0;
-  const char *ptr = RSTRING_PTR(sql);
-  const char *end = ptr + RSTRING_LEN(sql);
+struct multi_stmt_ctx {
+  sqlite3 *db;
+  sqlite3_stmt **stmt;
+  const char *str;
+  int len;
+  int rc;
+};
+
+void *prepare_multi_stmt_without_gvl(void *ptr) {
+  struct multi_stmt_ctx *ctx = (struct multi_stmt_ctx *)ptr;
+  const char *rest = NULL;
+  const char *str = ctx->str;
+  const char *end = ctx->str + ctx->len;
   while (1) {
-    int rc = sqlite3_prepare(db, ptr, end - ptr, stmt, &rest);
-    if (rc) {
-      sqlite3_finalize(*stmt);
-      rb_raise(cSQLError, "%s", sqlite3_errmsg(db));
+    ctx->rc = sqlite3_prepare_v2(ctx->db, str, end - str, ctx->stmt, &rest);
+    if (ctx->rc) {
+      sqlite3_finalize(*ctx->stmt);
+      return NULL;
     }
 
-    if (rest == end) return;
-    
+    if (rest == end) return NULL;
+
     // perform current query, but discard its results
-    rc = sqlite3_step(*stmt);
-    sqlite3_finalize(*stmt);
-    switch (rc) {
+    ctx->rc = sqlite3_step(*ctx->stmt);
+    sqlite3_finalize(*ctx->stmt);
+    switch (ctx->rc) {
     case SQLITE_BUSY:
-      rb_raise(cBusyError, "Database is busy");
     case SQLITE_ERROR:
-      rb_raise(cSQLError, "%s", sqlite3_errmsg(db));
+    case SQLITE_MISUSE:
+      return NULL;
     }
-    ptr = rest;
+    str = rest;
   }
+  return NULL;
 }
 
-inline int stmt_iterate(sqlite3_stmt *stmt, sqlite3 *db) {
+/*
+This function prepares a statement from an SQL string containing one or more SQL
+statements. It will release the GVL while the statements are being prepared and
+executed. All statements excluding the last one are executed. The last statement
+is not executed, but instead handed back to the caller for looping over results.
+*/
+inline void prepare_multi_stmt(sqlite3 *db, sqlite3_stmt **stmt, VALUE sql) {
+  struct multi_stmt_ctx ctx = {db, stmt, RSTRING_PTR(sql), RSTRING_LEN(sql), 0};
+  rb_thread_call_without_gvl(prepare_multi_stmt_without_gvl, (void *)&ctx, RUBY_UBF_IO, 0);
+  RB_GC_GUARD(sql);
+
+  switch (ctx.rc) {
+  case 0:
+    return;
+  case SQLITE_BUSY:
+    rb_raise(cBusyError, "Database is busy");
+  case SQLITE_ERROR:
+    rb_raise(cSQLError, "%s", sqlite3_errmsg(db));
+  default:
+    rb_raise(cError, "Invalid return code for prepare_multi_stmt_without_gvl: %d (please open an issue on https://github.com/digital-fabric/extralite)", ctx.rc);
+  }
+}
+
+struct step_ctx {
+  sqlite3_stmt *stmt;
   int rc;
-  rc = sqlite3_step(stmt);
-  switch (rc) {
+};
+
+void *stmt_iterate_without_gvl(void *ptr) {
+  struct step_ctx *ctx = (struct step_ctx *)ptr;
+  ctx->rc = sqlite3_step(ctx->stmt);
+  return NULL;
+}
+
+inline int stmt_iterate(sqlite3_stmt *stmt, sqlite3 *db) {
+  struct step_ctx ctx = {stmt, 0};
+  rb_thread_call_without_gvl(stmt_iterate_without_gvl, (void *)&ctx, RUBY_UBF_IO, 0);
+  switch (ctx.rc) {
     case SQLITE_ROW:
       return 1;
     case SQLITE_DONE:
@@ -207,7 +252,7 @@ inline int stmt_iterate(sqlite3_stmt *stmt, sqlite3 *db) {
     case SQLITE_ERROR:
       rb_raise(cSQLError, "%s", sqlite3_errmsg(db));
     default:
-      rb_raise(cError, "Invalid return code for sqlite3_step: %d", rc);
+      rb_raise(cError, "Invalid return code for sqlite3_step: %d (please open an issue on https://github.com/digital-fabric/extralite)", ctx.rc);
   }
 
   return 0;
@@ -222,7 +267,7 @@ typedef struct query_ctx {
 
 VALUE cleanup_stmt(VALUE arg) {
   query_ctx *ctx = (query_ctx *)arg;
-  sqlite3_finalize(ctx->stmt);
+  if (ctx->stmt) sqlite3_finalize(ctx->stmt);
   return Qnil;
 }
 
@@ -292,7 +337,7 @@ VALUE safe_query_ary(VALUE arg) {
     row = row_to_ary(ctx->stmt, column_count);
     if (yield_to_block) rb_yield(row); else rb_ary_push(result, row);
   }
-  
+
   RB_GC_GUARD(row);
   RB_GC_GUARD(result);
   return result;
@@ -453,14 +498,14 @@ void Init_Extralite() {
   rb_define_method(cDatabase, "initialize", Database_initialize, 1);
   rb_define_method(cDatabase, "close", Database_close, 0);
   rb_define_method(cDatabase, "closed?", Database_closed_p, 0);
-  
+
   rb_define_method(cDatabase, "query", Database_query_hash, -1);
   rb_define_method(cDatabase, "query_hash", Database_query_hash, -1);
   rb_define_method(cDatabase, "query_ary", Database_query_ary, -1);
   rb_define_method(cDatabase, "query_single_row", Database_query_single_row, -1);
   rb_define_method(cDatabase, "query_single_column", Database_query_single_column, -1);
   rb_define_method(cDatabase, "query_single_value", Database_query_single_value, -1);
-  
+
   rb_define_method(cDatabase, "last_insert_rowid", Database_last_insert_rowid, 0);
   rb_define_method(cDatabase, "changes", Database_changes, 0);
   rb_define_method(cDatabase, "filename", Database_filename, -1);
@@ -470,6 +515,9 @@ void Init_Extralite() {
   cError = rb_define_class_under(mExtralite, "Error", rb_eRuntimeError);
   cSQLError = rb_define_class_under(mExtralite, "SQLError", cError);
   cBusyError = rb_define_class_under(mExtralite, "BusyError", cError);
+  rb_gc_register_mark_object(cError);
+  rb_gc_register_mark_object(cSQLError);
+  rb_gc_register_mark_object(cBusyError);
 
   ID_STRIP = rb_intern("strip");
 }

data/extralite.gemspec

@@ -23,7 +23,6 @@ Gem::Specification.new do |s|
 
   s.add_development_dependency  'rake-compiler',        '1.1.1'
   s.add_development_dependency  'minitest',             '5.14.4'
-  s.add_development_dependency  'minitest-reporters',   '1.4.2'
   s.add_development_dependency  'simplecov',            '0.17.1'
   s.add_development_dependency  'rubocop',              '0.85.1'
   s.add_development_dependency  'pry',                  '0.13.1'

data/lib/extralite/version.rb

@@ -1,3 +1,3 @@
 module Extralite
-  VERSION = '1.2'
+  VERSION = '1.6'
 end

data/test/helper.rb

@@ -3,8 +3,3 @@
 require 'bundler/setup'
 require 'extralite'
 require 'minitest/autorun'
-require 'minitest/reporters'
-
-Minitest::Reporters.use! [
-  Minitest::Reporters::SpecReporter.new
-]

data/test/perf.rb

@@ -0,0 +1,51 @@
+# frozen_string_literal: true
+
+require 'bundler/inline'
+
+gemfile do
+  source 'https://rubygems.org'
+  gem 'sqlite3'
+  gem 'extralite', path: '..'
+  gem 'benchmark-ips'
+end
+
+require 'benchmark/ips'
+require 'fileutils'
+
+DB_PATH = '/tmp/extralite_sqlite3_perf.db'
+
+def prepare_database(count)
+  FileUtils.rm(DB_PATH) rescue nil
+  db = Extralite::Database.new(DB_PATH)
+  db.query('create table foo ( a integer primary key, b text )')
+  db.query('begin')
+  count.times { db.query('insert into foo (b) values (?)', "hello#{rand(1000)}" )}
+  db.query('commit')
+end
+
+def sqlite3_run(count)
+  db = SQLite3::Database.new(DB_PATH, :results_as_hash => true)
+  results = db.execute('select * from foo')
+  raise unless results.size == count
+end
+
+def extralite_run(count)
+  db = Extralite::Database.new(DB_PATH)
+  results = db.query('select * from foo')
+  raise unless results.size == count
+end
+
+[10, 1000, 100000].each do |c|
+  puts; puts; puts "Record count: #{c}"
+
+  prepare_database(c)
+
+  Benchmark.ips do |x|
+    x.config(:time => 3, :warmup => 1)
+
+    x.report("sqlite3") { sqlite3_run(c) }
+    x.report("extralite") { extralite_run(c) }
+
+    x.compare!
+  end
+end

data/test/test_database.rb

@@ -4,7 +4,7 @@ require_relative 'helper'
 
 class DatabaseTest < MiniTest::Test
   def setup
-    @db = Extralite::Database.new('/tmp/extralite.db')
+    @db = Extralite::Database.new(':memory:')
     @db.query('create table if not exists t (x,y,z)')
     @db.query('delete from t')
     @db.query('insert into t values (1, 2, 3)')
@@ -55,7 +55,7 @@ class DatabaseTest < MiniTest::Test
   def test_query_single_column
     r = @db.query_single_column('select y from t')
     assert_equal [2, 5], r
-  
+
     r = @db.query_single_column('select y from t where x = 2')
     assert_equal [], r
 end
@@ -82,6 +82,17 @@ end
     assert_equal [1, 4, 'a', 'd'], @db.query_single_column('select x from t order by x')
   end
 
+  def test_multiple_statements_with_error
+    error = nil
+    begin
+      @db.query("insert into t values foo; insert into t values ('d', 'e', 'f');")
+    rescue => error
+    end
+
+    assert_kind_of Extralite::SQLError, error
+    assert_equal 'near "foo": syntax error', error.message
+  end
+
   def test_empty_sql
     r = @db.query(' ')
     assert_nil r
@@ -97,7 +108,75 @@ end
 
     assert_equal @db, @db.close
     assert_equal true, @db.closed?
-    
+
     assert_raises(Extralite::Error) { @db.query_single_value('select 42') }
   end
 end
+
+class ScenarioTest < MiniTest::Test
+  def setup
+    @db = Extralite::Database.new('/tmp/extralite.db')
+    @db.query('create table if not exists t (x,y,z)')
+    @db.query('delete from t')
+    @db.query('insert into t values (1, 2, 3)')
+    @db.query('insert into t values (4, 5, 6)')
+  end
+
+  def test_concurrent_transactions
+    done = false
+    t = Thread.new do
+      db = Extralite::Database.new('/tmp/extralite.db')
+      db.query 'begin immediate'
+      sleep 0.01 until done
+
+      while true
+        begin
+          db.query 'commit'
+          break
+        rescue Extralite::BusyError
+          sleep 0.01
+        end
+      end
+    end
+
+    sleep 0.1
+    @db.query 'begin deferred'
+    result = @db.query_single_column('select x from t')
+    assert_equal [1, 4], result
+
+    assert_raises(Extralite::BusyError) do
+      @db.query('insert into t values (7, 8, 9)')
+    end
+
+    done = true
+    sleep 0.1
+
+    assert_raises(Extralite::BusyError) do
+      @db.query('insert into t values (7, 8, 9)')
+    end
+
+    assert_equal true, @db.transaction_active?
+
+    # the thing to do in this case is to commit the read transaction, allowing
+    # the other thread to commit its write transaction, and then we can
+    # "upgrade" to a write transaction
+
+    @db.query('commit')
+
+    while true
+      begin
+        @db.query('begin immediate')
+        break
+      rescue Extralite::BusyError
+        sleep 0.1
+      end
+    end
+
+    @db.query('insert into t values (7, 8, 9)')
+    @db.query('commit')
+
+    result = @db.query_single_column('select x from t')
+    assert_equal [1, 4, 7], result
+  end
+end
+

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: extralite
 version: !ruby/object:Gem::Version
-  version: '1.2'
+  version: '1.6'
 platform: ruby
 authors:
 - Sharon Rosner
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2021-06-06 00:00:00.000000000 Z
+date: 2021-12-13 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rake-compiler
@@ -38,20 +38,6 @@ dependencies:
     - - '='
       - !ruby/object:Gem::Version
         version: 5.14.4
-- !ruby/object:Gem::Dependency
-  name: minitest-reporters
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - '='
-      - !ruby/object:Gem::Version
-        version: 1.4.2
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - '='
-      - !ruby/object:Gem::Version
-        version: 1.4.2
 - !ruby/object:Gem::Dependency
   name: simplecov
   requirement: !ruby/object:Gem::Requirement
@@ -118,6 +104,7 @@ files:
 - lib/extralite.rb
 - lib/extralite/version.rb
 - test/helper.rb
+- test/perf.rb
 - test/test_database.rb
 homepage: https://github.com/digital-fabric/extralite
 licenses:
@@ -146,7 +133,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.1.4
+rubygems_version: 3.1.6
 signing_key: 
 specification_version: 4
 summary: Extra-lightweight SQLite3 wrapper for Ruby