activerecord-turso 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0c8e05a11274a315a434eb615873a9402e8afd76ff296b8efc030d9bd4d212d9
-  data.tar.gz: 0fa05a0646df999aba45e14fc31a5e2066ad471964d7501fa3994df9882cbca8
+  metadata.gz: 69e6be67a21fd809792bf867f81f6205bb06a6cc457aa05c2b80080cf272fd9b
+  data.tar.gz: d1a18f2e763cb6b5781521f1e8bd12f8a7159d3350c2fe5e91fe02ee00a50f1f
 SHA512:
-  metadata.gz: ee153e58a81fb89d2a1578cbbd4841a5720cac7c942916c1c256c234e3c0726f6bc0a2fb500cbcd7733a357efb43b1da37e9a8774a7d2eecfd98561cf9cdda76
-  data.tar.gz: 9d5ec6478407aa7c168c29dfbd64b6dc0661e84fbbb1167041ef8b2615fb5ffab899474787c2ceb946430f2aa92bac8684adaf6ffe2b6cf6188a953eb9ced36b
+  metadata.gz: defa80378418aabb5b0b619392e180102f633ce7f8f6a13d6778f34a47c47f14c2739dc5248aaf0011e8f40788002ae42fea373789e3fc8b5e6e1ea5ebac8b35
+  data.tar.gz: 573a7f4f59c98621b9c4f95af1daa2a8fe133b3e487ef83c7d73af51c8c65655cbff080ef2d3813d7e64868273b186f121d69653499bee67f1180463fc946f9a

data/README.md

@@ -1,23 +1,23 @@
-# activerecord-turso
+# Activerecord-turso
 
-ActiveRecord adapter for [Turso](https://turso.tech) (SQLite compatible).
+ActiveRecord adapter for [Turso](https://turso.tech) (SQLite compatible db).
 
 ## Status
 
-This adapter is under active development. It can run basic ActiveRecord operations against Turso today, but several production features are still being hardened. See [Limitations and Risks](#limitations-and-risks) before using it in production.
+This adapter is under **active** development. It can run basic ActiveRecord operations against Turso today, but several production features are still being hardened. See [Limitations and Risks](#limitations-and-risks) before using it in production.
 
 ## Requirements
 
 - Ruby >= 3.0
 - ActiveRecord >= 8.0, < 8.2
-- The `turso` Ruby gem (local bindings at `~/Projects/turso/bindings/ruby` during development)
+- The `turso` Ruby gem
 
 ## Installation
 
 Add to your Gemfile:
 
 ```ruby
-gem "activerecord-turso", path: "~/Projects/activerecord-turso"
+gem "activerecord-turso"
 ```
 
 Then configure `database.yml`:
@@ -94,13 +94,24 @@ class AddFtsToPosts < ActiveRecord::Migration[8.1]
 end
 ```
 
-Query with Tantivy functions:
+Query with the adapter helpers:
 
-```sql
-SELECT * FROM posts WHERE fts_match(title, body, 'database');
+```ruby
+match = ActiveRecord::Base.connection.fts_match(:posts, [:title, :body], "database")
+Post.where(match)
+
+score = ActiveRecord::Base.connection.fts_score(:posts, [:title, :body], "database")
+Post.select(:id, :title, score.as("rank")).where(match).order("rank ASC")
 ```
 
-Note: `fts5` virtual tables are not supported. Use `USING fts` indexes instead.
+Note: `fts5` virtual tables are not supported. Use `USING fts` indexes instead. This requires the `index_method` experimental feature:
+
+```yaml
+production:
+  adapter: turso
+  database: db/production.sqlite3
+  experimental_features: "index_method"
+```
 
 ### Concurrent transactions
 
@@ -112,19 +123,39 @@ development:
   database: db/dev.sqlite3
   journal_mode: mvcc
   busy_timeout: 5000
+
+  # Enable experimental Turso features when needed (e.g., custom index methods for FTS).
+  # Pass a comma-separated string or an array of feature names.
+  experimental_features: "index_method"
 ```
 
 Use `concurrent: true` inside the same pinned connection:
 
 ```ruby
 ActiveRecord::Base.connection.transaction(concurrent: true) do
-  # read-modify-write
+  # read-modify-write using raw SQL or bulk operations
 end
 ```
 
 Caveats:
 - The connection must stay checked out for the whole transaction; avoid work that yields the ActiveRecord connection back to the pool.
 - `busy_timeout` is used for lock waits, not MVCC snapshot conflicts. Snapshot conflicts retry with bounded backoff.
+- Normal ActiveRecord model persistence (`create!`, `save!`, `update!`, `touch`) is **not supported** inside a concurrent transaction because ActiveRecord opens its own internal transaction for each model change. Use raw SQL (`execute`, `exec_query`) or bulk operations (`insert_all`, `update_all`, `update_columns`) instead.
+- FTS custom index modules are not supported in MVCC mode.
+
+## Recommended production configuration
+
+```yaml
+production:
+  adapter: turso
+  database: db/production.sqlite3
+  journal_mode: wal            # Use mvcc only if you need concurrent writers and understand the caveats above
+  pool: 5
+  timeout: 5000                # busy_timeout default in milliseconds
+  query_timeout: 30000           # maximum time a single query may run
+  busy_timeout: 5000           # explicit busy timeout (falls back to timeout)
+  experimental_features: "index_method"
+```
 
 ## Limitations and Risks
 
@@ -138,19 +169,20 @@ The adapter builds column type maps from `column_decltype` metadata. This works
 
 The adapter's batch execution path splits multi-statement SQL on semicolons. This means SQL containing semicolons inside string literals, triggers, or stored expressions may be split incorrectly. Avoid relying on multi-statement strings other than simple schema dumps.
 
-### 3. MVCC requires opt-in and has pooling caveats
+### 3. MVCC requires opt-in and has ActiveRecord compatibility caveats
 
 `BEGIN CONCURRENT` is powerful but breaks ActiveRecord's default assumptions:
 
 - ActiveRecord expects transactions to commit unless the database returns an error. With `BEGIN CONCURRENT`, the commit can fail with a snapshot conflict and must be retried.
 - The retry loop must run on the same connection. Rails' connection pool is not MVCC-aware and may return the connection to the pool between retries.
 - Do not combine concurrent transactions with pessimistic locking (`lock!`, `with_lock`, `lock_version`).
+- Normal model persistence (`create!`, `save!`, `update!`, `touch`) is unsupported inside a concurrent transaction because ActiveRecord opens an internal transaction for each model change. The error is retried a few times, then raised as `ActiveRecord::StatementInvalid`.
 
-Only use `transaction(concurrent: true)` after testing it under your app's concurrency patterns.
+Only use `transaction(concurrent: true)` after testing it under your app's concurrency patterns, and prefer raw SQL or bulk operations inside the block.
 
-### 4. Prepared statement cache is disabled
+### 4. Prepared statement cache is enabled with a bounded pool
 
-The adapter returns `false` for `default_prepared_statements` and uses a no-op statement pool. Each query is prepared and finalized individually. This is slower than the upstream SQLite3 adapter for high-volume repeated queries, but avoids correctness issues until the Turso bindings expose stable prepared-statement reuse.
+The adapter uses a bounded statement pool with a default limit inherited from Rails (typically `1000`). Statements are evicted with an LRU policy and finalized against the underlying Turso connection. Set `statement_limit` in `database.yml` to tune the pool size.
 
 ### 5. ActiveRecord 8.0 support is CI-tested, not locally tested
 
@@ -175,7 +207,19 @@ bundle install
 bundle exec rake test
 ```
 
-To run against the official Rails Active Record test suite, see the CI workflow in `.github/workflows/test.yml` and the project design doc in `docs/superpowers/specs/`.
+Run with MVCC mode:
+
+```bash
+TURSO_TEST_JOURNAL_MODE=mvcc bundle exec rake test
+```
+
+Run with generated-columns experimental feature:
+
+```bash
+TURSO_TEST_EXPERIMENTAL_FEATURES=generated_columns bundle exec rake test
+```
+
+The CI workflow in `.github/workflows/ci.yml` runs all three configurations automatically.
 
 ## License
 

data/lib/active_record/connection_adapters/turso_adapter/connection_management.rb

@@ -9,14 +9,16 @@ module ActiveRecord
         end
 
         module ClassMethods
-          def new_client(config)
-            db_config = config.symbolize_keys.merge(:timeout => config[:timeout] || 5000)
-            ::Turso::AR::Connection.new(db_config)
-          end
+        def new_client(config)
+          db_config = config.symbolize_keys.merge(:timeout => config[:timeout] || 5000)
+          db_config[:experimental_features] = Array(db_config[:experimental_features])
+          ::Turso::AR::Connection.new(db_config)
+        end
         end
 
         def active?
-          @raw_connection && !@raw_connection.closed?
+          return false unless @raw_connection
+          !@raw_connection.closed?
         rescue ::Turso::Error
           false
         end
@@ -57,14 +59,11 @@ module ActiveRecord
 
           execute("PRAGMA foreign_keys = ON")
 
-          if @config[:timeout] || @config[:busy_timeout]
-            timeout = @config[:busy_timeout] || @config[:timeout]
-            @raw_connection.busy_timeout = timeout
-          end
+          timeout = @config[:busy_timeout] || @config[:timeout] || 5000
+          @raw_connection.busy_timeout = timeout
 
-          if @config[:query_timeout]
-            @raw_connection.query_timeout = @config[:query_timeout]
-          end
+          query_timeout = @config[:query_timeout] || 30_000
+          @raw_connection.query_timeout = query_timeout
         end
       end
     end

data/lib/active_record/connection_adapters/turso_adapter/database_statements.rb

@@ -9,41 +9,6 @@ module ActiveRecord
           raw_execute(sql, name)
         end
 
-        def exec_query(sql, name = "SQL", binds = [], prepare: false)
-          materialize_transactions
-
-          type_casted_binds = type_casted_binds(binds)
-          log(sql, name, binds, type_casted_binds) do
-            with_raw_connection do |conn|
-              stmt = conn.prepare(sql)
-              stmt.bind_positional(type_casted_binds) unless type_casted_binds.empty?
-              result = build_result(stmt)
-              stmt.finalize
-              result
-            end
-          end
-        end
-
-        def exec_insert(sql, name = nil, binds = [], pk = nil, sequence_name = nil, returning = nil)
-          if returning && supports_insert_returning?
-            return exec_query(sql, name, binds)
-          end
-
-          materialize_transactions
-
-          type_casted_binds = type_casted_binds(binds)
-          log(sql, name, binds, type_casted_binds) do
-            with_raw_connection do |conn|
-              stmt = conn.prepare(sql)
-              stmt.bind_positional(type_casted_binds) unless type_casted_binds.empty?
-              stmt.execute
-              last_id = pk ? last_inserted_id(sql) : nil
-              stmt.finalize
-              last_id
-            end
-          end
-        end
-
         def last_inserted_id(sql)
           @raw_connection.last_insert_rowid
         end
@@ -85,22 +50,37 @@ module ActiveRecord
             return ActiveRecord::Result.empty(affected_rows: 0)
           end
 
-          if write_query?(sql)
-            raw_connection.execute(sql, type_casted_binds)
-            affected_rows = raw_connection.changes
-            verified!
-            notification_payload[:affected_rows] = affected_rows
-            notification_payload[:row_count] = 0
-            ActiveRecord::Result.empty(affected_rows: affected_rows)
+          stmt = if prepare
+            @statements[sql] ||= raw_connection.prepare(sql)
           else
-            result = raw_connection.query(sql, type_casted_binds)
-            columns = result.column_names
-            rows = result.map(&:values)
-            affected_rows = raw_connection.changes
-            verified!
-            notification_payload[:affected_rows] = affected_rows
-            notification_payload[:row_count] = rows.length
-            ActiveRecord::Result.new(columns, rows, nil, affected_rows: affected_rows)
+            raw_connection.prepare(sql)
+          end
+
+          stmt.reset if prepare
+          stmt.bind_positional(type_casted_binds) unless type_casted_binds.empty?
+
+          begin
+            if write_query?(sql)
+              affected_rows = stmt.execute
+              verified!
+              notification_payload[:affected_rows] = affected_rows
+              notification_payload[:row_count] = 0
+              ActiveRecord::Result.empty(affected_rows: affected_rows)
+            else
+              columns = (0...stmt.column_count).map { |i| stmt.column_name(i) }
+              rows = []
+              while stmt.step == 1
+                rows << stmt.row.to_a
+              end
+              affected_rows = raw_connection.changes
+              verified!
+              notification_payload[:affected_rows] = affected_rows
+              notification_payload[:row_count] = rows.length
+              type_map = build_type_map(stmt)
+              ActiveRecord::Result.new(columns, rows, type_map, affected_rows: affected_rows)
+            end
+          ensure
+            stmt.finalize unless prepare
           end
         end
 
@@ -122,16 +102,6 @@ module ActiveRecord
 
         private
 
-        def build_result(stmt)
-          columns = (0...stmt.column_count).map { |i| stmt.column_name(i) }
-          type_map = build_type_map(stmt)
-          rows = []
-          while stmt.step == 1
-            rows << stmt.row.to_a
-          end
-          ActiveRecord::Result.new(columns, rows, type_map)
-        end
-
         def build_type_map(stmt)
           type_map = {}
           count = stmt.column_count

data/lib/active_record/connection_adapters/turso_adapter/error_translation.rb

@@ -16,6 +16,14 @@ module ActiveRecord
           when ::Turso::BusyError
             ActiveRecord::Deadlocked.new(message, sql: sql, binds: binds)
           when ::Turso::ReadonlyError
+            ActiveRecord::ReadOnlyRecord.new(message, sql: sql, binds: binds)
+          when ::Turso::IoError, ::Turso::CorruptError
+            ActiveRecord::StatementInvalid.new(message, sql: sql, binds: binds)
+          when ::Turso::DatabaseFullError
+            ActiveRecord::StatementInvalid.new(message, sql: sql, binds: binds)
+          when ::Turso::InterruptError
+            ActiveRecord::QueryCanceled.new(message, sql: sql, binds: binds)
+          when ::Turso::MisuseError
             ActiveRecord::StatementInvalid.new(message, sql: sql, binds: binds)
           else
             super

data/lib/active_record/connection_adapters/turso_adapter/schema_statements.rb

@@ -75,6 +75,16 @@ module ActiveRecord
           SQL
         end
 
+        def fts_match(_table_name, columns, query)
+          cols = Array(columns).map { |c| quote_table_name(c) }.join(", ")
+          Arel.sql("fts_match(#{cols}, #{quote(query)})")
+        end
+
+        def fts_score(_table_name, columns, query)
+          cols = Array(columns).map { |c| quote_table_name(c) }.join(", ")
+          Arel.sql("fts_score(#{cols}, #{quote(query)})")
+        end
+
         private
 
         def options_to_fts_options(options)

data/lib/active_record/connection_adapters/turso_adapter/statement_pool.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+module ActiveRecord
+  module ConnectionAdapters
+    class TursoAdapter < SQLite3Adapter
+      class StatementPool < ConnectionAdapters::StatementPool
+        alias reset clear
+
+        private
+
+        def dealloc(stmt)
+          stmt.finalize
+        rescue ::Turso::Error
+        end
+      end
+    end
+  end
+end

data/lib/active_record/connection_adapters/turso_adapter/transaction_management.rb

@@ -43,35 +43,62 @@ module ActiveRecord
         private
 
         def transaction_with_mvcc(options, &block)
+          connect if @raw_connection.nil?
+
           unless @mvcc_enabled
             raise ActiveRecord::AdapterError,
                   "transaction(concurrent: true) requires journal_mode: 'mvcc' in database.yml"
           end
 
+          if options[:requires_new] == false
+            raise ActiveRecord::AdapterError,
+                  "transaction(concurrent: true) is incompatible with nested transactions"
+          end
+
           max_retries = @config.fetch(:concurrent_retry_limit, 50)
           base_delay_ms = @config.fetch(:concurrent_retry_base_ms, 2)
           retries = 0
+          pinned_connection_id = @raw_connection.object_id
 
           loop do
             begin
-              raw_execute("BEGIN CONCURRENT", "TRANSACTION")
-              yield
-              raw_execute("COMMIT", "TRANSACTION")
-              return
+              @raw_connection.execute("BEGIN CONCURRENT")
+              result = yield
+              @raw_connection.execute("COMMIT")
+              return result
             rescue ActiveRecord::StatementInvalid => e
-              raw_execute("ROLLBACK", "TRANSACTION") rescue nil
+              rollback_if_active
               raise unless concurrent_conflict?(e) && retries < max_retries
 
               retries += 1
-              sleep(base_delay_ms * retries / 1000.0)
+              verified!
+              backoff(base_delay_ms, retries)
+
+              unless @raw_connection.object_id == pinned_connection_id
+                raise ActiveRecord::AdapterError,
+                      "MVCC retry detected a different database connection. " \
+                      "transaction(concurrent: true) must run on a pinned connection."
+              end
             end
           end
         end
 
         def concurrent_conflict?(exception)
-          exception.cause.is_a?(::Turso::BusySnapshotError) ||
-            exception.cause.is_a?(::Turso::BusyError) ||
-            /snapshot conflict|busy snapshot|database is locked/i.match?(exception.message)
+          cause = exception.cause
+          conflict_classes = [::Turso::BusySnapshotError, ::Turso::BusyError]
+          conflict_classes << ::Turso::WriteWriteConflict if defined?(::Turso::WriteWriteConflict)
+
+          conflict_classes.any? { |klass| cause.is_a?(klass) } ||
+            /snapshot conflict|busy snapshot|database is locked|cannot start a transaction within a transaction/i.match?(exception.message)
+        end
+
+        def rollback_if_active
+          return unless @raw_connection && !@raw_connection.closed?
+          @raw_connection.execute("ROLLBACK") rescue nil
+        end
+
+        def backoff(base_delay_ms, retries)
+          sleep(base_delay_ms * retries / 1000.0)
         end
 
         def savepoint_name(name)

data/lib/active_record/connection_adapters/turso_adapter.rb

@@ -2,7 +2,9 @@
 
 require "active_record/connection_adapters/sqlite3_adapter"
 
-Dir[File.expand_path("turso_adapter/*.rb", __dir__)].each { |f| require f }
+require_relative "turso_adapter/statement_pool"
+
+Dir[File.expand_path("turso_adapter/*.rb", __dir__)].sort.each { |f| require f }
 
 module ActiveRecord
   module ConnectionAdapters
@@ -35,14 +37,14 @@ module ActiveRecord
         true
       end
 
-      def explain(arel, binds = [])
+      def explain(arel, binds = [], _options = [])
         sql = "EXPLAIN QUERY PLAN " + to_sql(arel, binds)
         result = exec_query(sql, "EXPLAIN", binds)
         SQLite3::ExplainPrettyPrinter.new.pp(result)
       end
 
-      def default_prepared_statements
-        false
+      def build_statement_pool
+        StatementPool.new(self.class.type_cast_config_to_integer(@config[:statement_limit]))
       end
 
       def database_file_exists?
@@ -69,6 +71,7 @@ module ActiveRecord
       def initialize_type_map(m)
         super
         m.register_type(%r(boolean)i, Type::Boolean.new)
+        m.register_type(%r(json)i, Type::Json.new)
       end
     end
   end

data/lib/activerecord-turso/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module ActiveRecordTurso
+  VERSION = "0.2.0"
+end

data/lib/turso/ar/connection.rb

@@ -9,11 +9,17 @@ module Turso
 
       def_delegators :@db, :close, :closed?, :changes, :total_changes
 
+      DEFAULT_BUSY_TIMEOUT_MS = 5000
+      DEFAULT_QUERY_TIMEOUT_MS = 30_000
+
       def initialize(config)
         @config = config
-        @db = ::Turso::DB.new(config[:database].to_s,
-          busy_timeout: config[:busy_timeout] || config[:timeout],
-          query_timeout: config[:query_timeout])
+        db_opts = {
+          busy_timeout: config[:busy_timeout] || config[:timeout] || DEFAULT_BUSY_TIMEOUT_MS,
+          query_timeout: config[:query_timeout] || DEFAULT_QUERY_TIMEOUT_MS
+        }
+        db_opts[:experimental_features] = config[:experimental_features] if config[:experimental_features]
+        @db = ::Turso::DB.new(config[:database].to_s, **db_opts)
       end
 
       def last_insert_rowid
@@ -42,14 +48,13 @@ module Turso
       end
 
       def execute_batch(sql)
-        sql.split(";").each do |stmt|
-          s = stmt.strip
-          @db.execute(s) unless s.empty?
+        split_batch(sql).each do |stmt|
+          @db.execute(stmt)
         end
       end
 
       def prepare(sql)
-        @db.instance_variable_get(:@database).connection.prepare(sql)
+        @db.prepare(sql)
       end
 
       def busy_timeout=(ms)
@@ -66,6 +71,84 @@ module Turso
 
       private
 
+      def split_batch(sql)
+        statements = []
+        current = +""
+        in_string = false
+        in_line_comment = false
+        in_block_comment = false
+        i = 0
+        while i < sql.length
+          char = sql[i]
+          next_char = sql[i + 1]
+
+          if in_line_comment
+            if char == "\n"
+              in_line_comment = false
+            end
+            i += 1
+            next
+          end
+
+          if in_block_comment
+            if char == "*" && next_char == "/"
+              in_block_comment = false
+              i += 2
+            else
+              i += 1
+            end
+            next
+          end
+
+          if in_string
+            if char == "'" && next_char == "'"
+              current << char << next_char
+              i += 2
+              next
+            elsif char == "'"
+              in_string = false
+              current << char
+              i += 1
+              next
+            end
+            current << char
+            i += 1
+            next
+          end
+
+          case char
+          when "'"
+            in_string = true
+            current << char
+          when "-"
+            if next_char == "-"
+              in_line_comment = true
+              i += 2
+              next
+            end
+            current << char
+          when "/"
+            if next_char == "*"
+              in_block_comment = true
+              i += 2
+              next
+            end
+            current << char
+          when ";"
+            stmt = current.strip
+            statements << stmt unless stmt.empty?
+            current = +""
+          else
+            current << char
+          end
+          i += 1
+        end
+
+        stmt = current.strip
+        statements << stmt unless stmt.empty?
+        statements
+      end
+
       def normalize_binds(binds)
         binds.map do |value|
           case value

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: activerecord-turso
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - Ben D'Angelo
@@ -43,6 +43,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '0.1'
+- !ruby/object:Gem::Dependency
+  name: irb
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 executables: []
 extensions: []
 extra_rdoc_files: []
@@ -53,8 +67,10 @@ files:
 - lib/active_record/connection_adapters/turso_adapter/database_statements.rb
 - lib/active_record/connection_adapters/turso_adapter/error_translation.rb
 - lib/active_record/connection_adapters/turso_adapter/schema_statements.rb
+- lib/active_record/connection_adapters/turso_adapter/statement_pool.rb
 - lib/active_record/connection_adapters/turso_adapter/transaction_management.rb
 - lib/activerecord-turso.rb
+- lib/activerecord-turso/version.rb
 - lib/turso/ar/connection.rb
 licenses:
 - MIT
@@ -66,7 +82,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: 3.0.0
+      version: 3.2.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="