activerecord-enhancedsqlite3-adapter 0.2.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f138b7e075e546691856c8e45f22413a1e381df3024c6ef30e467dbde830c896
-  data.tar.gz: 0e6bde9f85883cacc12893f8ab2b1ca5820deea6d6a1357a262fb24401ed4980
+  metadata.gz: fd134c96488a861c3fdc36c32396547a31ea596c11243aff923995acff8ae4b5
+  data.tar.gz: d2029b2162c66df5689e65cb4e281f73d21049adafaecf9bbf78490570a10321
 SHA512:
-  metadata.gz: dafd2a35c78565843b9aa384fb5988818c48bdc6852da55363c7e18e447a752b8194ef08ea4a40c9b482f30317cd7afc8e9501e22cb38abbab51dd16f3d6c7a7
-  data.tar.gz: 5c0d6b07eb0b3e90c4fd6f8859f90d944e33d90e100ee96981f24b4d1cb1212266045f81abe935b4ad1913caf22d43cb3386a73bdb0670b15f8686e680619a77
+  metadata.gz: 6803168a5f067839e6ea7416f2f067bbc044202d14fd21a9536d068bebe35d7d89a481cbfcf5e230d4063705a7b5241aeb76d8f92846e2ecf5cdc2afa8b805f0
+  data.tar.gz: 2845ea0983129dd0e6e66b66e8f10fca3538192dcdaec62ec55f497ebb8f82f52bdc2352eb1fdf85b859bbd7436e86b57d15da4ce1135c00e23f8996db5cfc59

data/CHANGELOG.md

@@ -1,5 +1,22 @@
 ## [Unreleased]
 
+## [0.4.0] - 2023-12-10
+
+- Ensure transactions are IMMEDIATE and not DEFERRED
+- Ensure that our `busy_handler` is the very first configuration to be set on a connection
+- Simplify and speed up our `busy_handler` implementation
+
+## [0.3.0] - 2023-12-06
+
+- Added a more performant implementation of the the `timeout` mechanism
+
+## [0.2.0] - 2023-09-28
+
+- Added support for deferrable constraints
+
 ## [0.1.0] - 2023-09-28
 
 - Initial release
+- Added support for virtual columns
+- Added support setting PRAGMA statements via the `config/database.yml` file
+- Added support for loading extensions via the `config/database.yml` file

data/README.md

@@ -1,24 +1,85 @@
-# Activerecord::Enhanced::Sqlite3::Adapter
+# ActiveRecord Enhanced SQLite3 Adapter
 
-TODO: Delete this and the text below, and describe your gem
+Enhance ActiveRecord's 7.1 SQLite3 adapter. Adds support for:
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/activerecord/enhanced/sqlite3/adapter`. To experiment with that code, run `bin/console` for an interactive prompt.
+* generated columns,
+* deferred foreign keys,
+* `PRAGMA` tuning,
+* and extension loading
 
 ## Installation
 
-TODO: Replace `UPDATE_WITH_YOUR_GEM_NAME_PRIOR_TO_RELEASE_TO_RUBYGEMS_ORG` with your gem name right after releasing it to RubyGems.org. Please do not do it earlier due to security reasons. Alternatively, replace this section with instructions to install your gem from git if you don't plan to release to RubyGems.org.
-
 Install the gem and add to the application's Gemfile by executing:
 
-    $ bundle add UPDATE_WITH_YOUR_GEM_NAME_PRIOR_TO_RELEASE_TO_RUBYGEMS_ORG
+```shell
+$ bundle add activerecord-enhancedsqlite3-adapter
+```
 
-If bundler is not being used to manage dependencies, install the gem by executing:
+## Usage
 
-    $ gem install UPDATE_WITH_YOUR_GEM_NAME_PRIOR_TO_RELEASE_TO_RUBYGEMS_ORG
+This gem hooks into your Rails application to enhance the `SQLite3Adapter` automatically. No setup required!
 
-## Usage
+Once installed, you can take advantage of the added features.
+
+### Generated columns
+
+You can now create `virtual` columns, both stored and dynamic. The [SQLite docs](https://www.sqlite.org/gencol.html) explain the difference:
+
+> Generated columns can be either VIRTUAL or STORED. The value of a VIRTUAL column is computed when read, whereas the value of a STORED column is computed when the row is written. STORED columns take up space in the database file, whereas VIRTUAL columns use more CPU cycles when being read.
+
+The default is to create dynamic/virtual columns.
+
+```ruby
+create_table :virtual_columns, force: true do |t|
+  t.string :name
+  t.virtual :upper_name, type: :string, as: "UPPER(name)", stored: true
+  t.virtual :lower_name, type: :string, as: "LOWER(name)", stored: false
+  t.virtual :octet_name, type: :integer, as: "LENGTH(name)"
+end
+```
+
+### Deferred foreign keys
+
+You can now specify whether or not a foreign key should be deferrable, whether `:deferred` or `:immediate`.
+
+`:deferred` foreign keys mean that the constraint check will be done once the transaction is committed and allows the constraint behavior to change within transaction. `:immediate` means that constraint check is immediate and allows the constraint behavior to change within transaction. The default is `:immediate`.
+
+```ruby
+add_reference :person, :alias, foreign_key: { deferrable: :deferred }
+add_reference :alias, :person, foreign_key: { deferrable: :deferred }
+```
+
+### `PRAGMA` tuning
+
+Pass any [`PRAGMA` key-value pair](https://www.sqlite.org/pragma.html) under a `pragmas` list in your `config/database.yml` file to ensure that these configuration settings are applied to all database connections.
+
+```yaml
+default: &default
+  adapter: sqlite3
+  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
+  pragmas:
+    # level of database durability, 2 = "FULL" (sync on every write), other values include 1 = "NORMAL" (sync every 1000 written pages) and 0 = "NONE"
+    # https://www.sqlite.org/pragma.html#pragma_synchronous
+    synchronous: "FULL"
+```
+
+### Extension loading
+
+There are a number of [SQLite extensions available as Ruby gems](https://github.com/asg017/sqlite-ecosystem). In order to load the extensions, you need to install the gem (`bundle add {extension-name}`) and then load it into the database connections. In order to support the latter, this gem enhances the `config/database.yml` file to support an `extensions` array. For example, to install and load [an extension](https://github.com/asg017/sqlite-ulid) for supporting [<abbr title="Universally Unique Lexicographically Sortable Identifiers">ULIDs</abbr>](https://github.com/ulid/spec), we would do:
+
+```shell
+$ bundle add sqlite_ulid
+```
+
+then
 
-TODO: Write usage instructions here
+```yaml
+default: &default
+  adapter: sqlite3
+  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
+  extensions:
+    - sqlite_ulid
+```
 
 ## Development
 
@@ -28,7 +89,7 @@ To install this gem onto your local machine, run `bundle exec rake install`. To
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/activerecord-enhancedsqlite3-adapter.
+Bug reports and pull requests are welcome on GitHub at https://github.com/fractaledmind/activerecord-enhancedsqlite3-adapter.
 
 ## License
 

data/lib/enhanced_sqlite3/adapter.rb

@@ -10,6 +10,27 @@ require "enhanced_sqlite3/supports_deferrable_constraints"
 
 module EnhancedSQLite3
   module Adapter
+    # Setup the Rails SQLite3 adapter instance.
+    #
+    # extends  https://github.com/rails/rails/blob/main/activerecord/lib/active_record/connection_adapters/sqlite3_adapter.rb#L90
+    def initialize(...)
+      super
+      # Ensure that all connections default to immediate transaction mode.
+      # This is necessary to prevent SQLite from deadlocking when concurrent processes open write transactions.
+      # By default, SQLite opens transactions in deferred mode, which means that a transactions acquire
+      # a shared lock on the database, but will attempt to upgrade that lock to an exclusive lock if/when
+      # a write is attempted. Because SQLite is in the middle of a transaction, it cannot retry the transaction
+      # if a BUSY exception is raised, and so it will immediately raise a SQLITE_BUSY exception without calling
+      # the `busy_handler`. Because Rails only wraps writes in transactions, this means that all transactions
+      # will attempt to acquire an exclusive lock on the database. Thus, under any concurrent load, you are very
+      # likely to encounter a SQLITE_BUSY exception.
+      # By setting the default transaction mode to immediate, SQLite will instead attempt to acquire
+      # an exclusive lock as soon as the transaction is opened. If the lock cannot be acquired, it will
+      # immediately call the `busy_handler` to retry the transaction. This allows concurrent processes to
+      # coordinate and linearize their transactions, avoiding deadlocks.
+      @connection_parameters.merge!(default_transaction_mode: :immediate)
+    end
+
     # Perform any necessary initialization upon the newly-established
     # @raw_connection -- this is the place to modify the adapter's
     # connection settings, run queries to configure any application-global
@@ -18,10 +39,10 @@ module EnhancedSQLite3
     # Implementations may assume this method will only be called while
     # holding @lock (or from #initialize).
     #
-    # extends https://github.com/rails/rails/blob/main/activerecord/lib/active_record/connection_adapters/sqlite3_adapter.rb#L691
+    # overrides https://github.com/rails/rails/blob/main/activerecord/lib/active_record/connection_adapters/sqlite3_adapter.rb#L691
     def configure_connection
-      super
-
+      configure_busy_handler_timeout
+      check_version
       configure_pragmas
       configure_extensions
 
@@ -31,8 +52,60 @@ module EnhancedSQLite3
 
     private
 
+    def configure_busy_handler_timeout
+      return unless @config.key?(:timeout)
+
+      timeout = self.class.type_cast_config_to_integer(@config[:timeout])
+      timeout_seconds = timeout.fdiv(1000)
+      retry_interval = 6e-5 # 60 microseconds
+
+      @raw_connection.busy_handler do |count|
+        timed_out = false
+        # keep track of elapsed time every 100 iterations (to lower load)
+        if (count % 100).zero?
+          # fail if we exceed the timeout value (captured from the timeout config option, converted to seconds)
+          timed_out = (count * retry_interval) > timeout_seconds
+        end
+        if timed_out
+          false # this will cause the BusyException to be raised
+        else
+          sleep(retry_interval)
+          true
+        end
+      end
+    end
+
     def configure_pragmas
-      @config.fetch(:pragmas, []).each do |key, value|
+      defaults = {
+        # Enforce foreign key constraints
+        # https://www.sqlite.org/pragma.html#pragma_foreign_keys
+        # https://www.sqlite.org/foreignkeys.html
+        "foreign_keys" => "ON",
+        # Impose a limit on the WAL file to prevent unlimited growth
+        # https://www.sqlite.org/pragma.html#pragma_journal_size_limit
+        "journal_size_limit" => 64.megabytes,
+        # Set the local connection cache to 2000 pages
+        # https://www.sqlite.org/pragma.html#pragma_cache_size
+        "cache_size" => 2000
+      }
+      unless @memory_database
+        defaults.merge!(
+          # Journal mode WAL allows for greater concurrency (many readers + one writer)
+          # https://www.sqlite.org/pragma.html#pragma_journal_mode
+          "journal_mode" => "WAL",
+          # Set more relaxed level of database durability
+          # 2 = "FULL" (sync on every write), 1 = "NORMAL" (sync every 1000 written pages) and 0 = "NONE"
+          # https://www.sqlite.org/pragma.html#pragma_synchronous
+          "synchronous" => "NORMAL",
+          # Set the global memory map so all processes can share some data
+          # https://www.sqlite.org/pragma.html#pragma_mmap_size
+          # https://www.sqlite.org/mmap.html
+          "mmap_size" => 128.megabytes
+        )
+      end
+      pragmas = defaults.merge(@config.fetch(:pragmas, {}))
+
+      pragmas.each do |key, value|
         execute("PRAGMA #{key} = #{value}", "SCHEMA")
       end
     end

data/lib/enhanced_sqlite3/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module EnhancedSQLite3
-  VERSION = "0.2.0"
+  VERSION = "0.4.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: activerecord-enhancedsqlite3-adapter
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.4.0
 platform: ruby
 authors:
 - Stephen Margheim
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-10-09 00:00:00.000000000 Z
+date: 2023-12-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord