sequel-hexspace 2.0.0 → 2.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b4421ed6430cdfa0ede39d8b4956c01de6e65317af4c9ab72238e06aa98d0e18
-  data.tar.gz: b01b91610baad3b154d7a44c193ad29e88fc940de34473158e60521af958161e
+  metadata.gz: 8ce1fd4d7fed71e9678ed12c286b3a1b1e0bff80d52ef2d78c222a52e7a09713
+  data.tar.gz: 9ae18e5f352c95e8390d6d26ea09d217f63dad5c105a2bd30b2290c87483d566
 SHA512:
-  metadata.gz: bfb4bbe3fddb1be528f0cdd382e78dfcf651edd36de086248742cc4d31d23dfa293b4619ba70acf6066681fe58cb6b0661373badd187c80fd8df54935513e3cd
-  data.tar.gz: 5a979505d337170a4d41bf96e135118284b34461a21611167856d0e95b79b63bf11eb71f4adae733c7ea579347d40091fba7ab038bc1efa5d10ccdf701801f5e
+  metadata.gz: f35426bc639c4dcf948f425629ef4ec895cc0c0334106c60461b18acb28ee4cf7b0067f6f36196a57d6ba8535476b018b6c1b9b210f1f5e06c2af397cb2760ce
+  data.tar.gz: d308e6d823bf60ddacfc2578e3b09a1892e376863ec5101573f4cc6a7c3fd7b496362288c63a0e8ec5ec595f95a981713fe6c4fca5588b9872ce95152d44d8d9

data/CHANGELOG.md

@@ -1,8 +1,37 @@
 # Changelog
 
-## 2.0.0 (unreleased)
+## [2.0.1] - 2026-10-07
 
-### Breaking
+### Fixed
+
+- Emulated `DELETE` and `UPDATE` no longer drop the original table. Spark has
+  neither statement, so the adapter rebuilds the table under a temporary name and
+  swaps it in. The swap used to drop the original and rename the copy into place.
+  A process killed between those two steps lost the table, with its rows left
+  under a generated name. The swap is now an `INSERT OVERWRITE`, so the original
+  table always exists.
+- A temporary table left behind by an interrupted delete or update no longer
+  breaks every later delete or update on that table with
+  `TABLE_OR_VIEW_ALREADY_EXISTS`. The adapter now tolerates it and cleans it up.
+- The temporary table for a schema-qualified table (`sch.items`) is now created
+  in that schema. It used to be created as a single dotted identifier in the
+  default database.
+- `Sequel.date_add` / `Sequel.date_sub` now honour the `date_arithmetic`
+  extension's cast: the `:cast` option, or the generic timestamp type by
+  default. Both were silently ignored.
+
+### Changed
+
+- Require `thrift >= 0.23` (was `>= 0.18`), and delete the `Thrift::Bytes`
+  backport that older versions needed. The upper bound stays `< 0.25`.
+- Bound `hexspace` to `>= 0.2.1, < 0.4` (was `>= 0.2.1`). hexspace is a 0.x gem,
+  so a new minor version may break its API. 0.3.0 is its newest release, so the
+  bound excludes nothing that exists.
+- Internal simplifications with no change in behaviour.
+
+## [2.0.0] - 2026-10-01
+
+### Breaking changes
 
 - Declare `required_ruby_version >= 3.3`. Ruby 3.2 reached end of life in March
   2026 and is no longer tested. Installing on an older Ruby now fails with a
@@ -11,10 +40,11 @@
 
 ### Note on the gap since 1.0.0
 
-1.0.0 (April 2024) is the only version ever published; 1.0.1 was tagged in the
-repository but never released. Everything below has therefore been unavailable
-to users until now, and this is the first release to carry any of it. The list is
-reconstructed from git history.
+1.0.0 (April 2024) was the previous published release. There was never a 1.0.1
+release and never a v1.0.1 tag — 1.0.1 was only a gemspec version bump, at
+`ce8b23e`, never tagged and never published. Everything below had therefore been
+unavailable to users until this release, which is the first to carry any of it.
+The list is reconstructed from git history.
 
 ### Added
 
@@ -35,3 +65,6 @@ reconstructed from git history.
 - Route prepared `DELETE` statements through the emulated delete path.
 - Date and timestamp casting and literalisation, including Ruby 4.0
   compatibility.
+
+[2.0.0]: https://github.com/outcomesinsights/sequel-hexspace/compare/v1.0.0...v2.0.0
+[2.0.1]: https://github.com/outcomesinsights/sequel-hexspace/compare/v2.0.0...v2.0.1

data/README

@@ -1,3 +1,92 @@
 This is a hexspace adapter for Sequel, designed to be used with Spark (not
 Hive). You can use the hexspace:// protocol in the Sequel connection URL
 to use this adapter.
+
+== Emulated DELETE and UPDATE
+
+Spark has no DELETE and no UPDATE, so Sequel::Dataset#delete and
+Sequel::Dataset#update are emulated. Both run as:
+
+1. CREATE TABLE <table>__sequel_delete_emulate AS SELECT <the rows the
+   operation should leave behind>
+2. INSERT OVERWRITE TABLE <table> SELECT ... FROM
+   <table>__sequel_delete_emulate
+3. check that <table> now holds as many rows as the temp table did, then DROP
+   the temp table
+
+The suffix goes on the table's own name, so the temp table is always in the
+same schema -- and catalog -- as your table: deleting from schema.items builds
+schema.items__sequel_delete_emulate there. Before 2026-10-02 the name was
+assembled by stripping the quoting out of the literalized target, which
+collapsed `schema`.`items` into the single identifier
+"schema.items__sequel_delete_emulate", and Spark rejects that outright ("not a
+valid name for tables/databases. Valid names only contain alphabet characters,
+numbers and _"). So an emulated DELETE or UPDATE against a schema-qualified
+dataset used to fail at step 1, without touching your table. If you are
+upgrading past that there is nothing to clean up: the temp table could never be
+created, so there is none to find.
+
+This is not a transaction, and this adapter has none to offer you:
+Database#transaction just yields. The adapter's own source says this path is
+"designed to minimize the changes to the tests, and is not recommended for
+production use". The rest of this section is what that disclaimer means in
+practice, so that you can decide rather than discover.
+
+=== What is guaranteed
+
+* Your table is never dropped. There is no window in which reading it fails
+  with table-or-view-not-found. (Before 2026-10-02 the swap dropped your table
+  and renamed the temp table into its place, and a process that died in between
+  left the table GONE.)
+
+* The rows the operation intends to keep exist under
+  <table>__sequel_delete_emulate, in your table's own schema, from step 1 until
+  step 3. If the process dies before step 3, or if the row-count check in step
+  3 disagrees, that table is left in place on purpose -- it is the recovery
+  copy, and it is the only name your data may be sitting under. Nothing in your
+  own source will name it, so if an emulated delete or update died, look there
+  first.
+
+* That name is derived from your table's name and nothing else. It carries no
+  timestamp, process id or random suffix, and that is deliberate: the deaths
+  recovery exists for -- a SIGKILL, a lost connection -- raise nothing, so a
+  generated name would appear in no error, in no log, and nowhere in your
+  source, and the rows would be unfindable by anyone who had not been watching.
+  What that choice costs is the concurrency limitation below.
+
+* Step 3 does not drop the temp table on the strength of step 2 merely
+  returning. Until the row count has been checked, the temp table is treated as
+  the only intact copy.
+
+=== What is NOT guaranteed
+
+* Atomicity. INSERT OVERWRITE is unambiguously atomic only on an ACID table
+  such as Delta. This adapter emits USING <format> only when you pass :using to
+  create_table, and the temp table above passes none, so unless your metastore
+  defaults to an ACID format both tables are not ACID. Against a default Spark
+  thriftserver (Provider hive, TextInputFormat, LazySimpleSerDe) the write was
+  measured to stage and then commit -- a job that failed after writing 150,000
+  of 200,000 rows left the table holding its full pre-operation contents, and
+  SIGKILLing the client at seven points across the statement never left the
+  table absent, empty or partial. What is left is the commit step: replacing a
+  multi-file Hive table in place is not one atomic filesystem operation, so a
+  death of the SERVER, or a storage failure, inside that step can leave the
+  table PRESENT but empty or partial. A plain SELECT then returns the wrong
+  rows SILENTLY rather than failing loudly. That window has not been
+  reproduced and is not quantified; it is accepted, not closed.
+
+* Re-running is not free. Step 1 is preceded by a DROP TABLE IF EXISTS of
+  <table>__sequel_delete_emulate, because Spark's CREATE TABLE is not
+  idempotent and a stranded temp table otherwise makes every later delete or
+  update on the same table fail. That means re-running a delete or update that
+  died DESTROYS the recovery copy. Recover your data first, then re-run.
+
+* Concurrency. The temp table name is derived from the target table name and is
+  not made unique per operation, so two emulated deletes or updates against the
+  same table at the same time will fight over it. Serialize them. This is not
+  fixed, and it is not going to be: a unique name is what would fix it, and the
+  recovery guarantee above is why that is not on offer.
+
+If any of that is unacceptable for your workload, do not use emulated DELETE or
+UPDATE -- build the replacement table yourself, under a name you control, and
+swap it in on terms you have chosen.

data/lib/sequel/adapters/hexspace/thrift_compat.rb

@@ -0,0 +1,54 @@
+# rubocop:disable all
+# Restore Thrift::Client#handle_exception and #reply_seqid for thrift >= 0.24.
+#
+# hexspace's generated Thrift client (lib/hexspace/tcli_service.rb) calls both,
+# 21 times each. thrift 0.24.0 removed them, folding their work into the new
+# #validate_message_begin -- so on thrift 0.24 every Spark connection dies in
+# recv_OpenSession with NoMethodError, surfacing as
+# Sequel::DatabaseConnectionError.
+#
+# Upstream will not fix this for us: hexspace's generated client is untouched
+# since 2023-05-08, the latest release (0.3.0, Apr 2025) predates thrift 0.24,
+# and no issue or PR there mentions thrift. Regenerating that client against
+# 0.24 would drop support for thrift < 0.24, which is a maintainer's decision
+# rather than ours to force.
+#
+# Both methods are restored verbatim from thrift 0.23.0 and are implemented
+# purely in terms of primitives thrift 0.24 still provides and still uses in
+# validate_message_begin. The dequeue accounting is unchanged: exactly one
+# dequeue_pending_seqid per call on both the reply and the exception path.
+#
+# Note this keeps 0.23 semantics -- it does NOT add 0.24's stricter checks for
+# invalid message type and wrong method name. Remove once hexspace regenerates
+# its client against thrift >= 0.24.
+if defined?(Thrift::Client)
+  # Both method bodies below are copied verbatim from thrift 0.23.0 and
+  # deliberately not restyled: keeping them byte-comparable with upstream is
+  # what makes them auditable. That is enforced by the `rubocop:disable all` on
+  # this file's first line, which rubocop honours however it is invoked -- a
+  # .rubocop.yml Exclude is not, because rubocop ignores Exclude for a path named
+  # on the command line and treefmt names them. This file holds the shim and
+  # nothing else so the rest of the adapter stays linted.
+  #
+  # reply_seqid also cannot be renamed to a predicate -- hexspace's generated
+  # client calls it by that exact name.
+  module Thrift
+    module Client
+      unless method_defined?(:handle_exception) || private_method_defined?(:handle_exception)
+        def handle_exception(mtype)
+          if mtype == MessageTypes::EXCEPTION
+            dequeue_pending_seqid
+            raise_application_exception
+          end
+        end
+      end
+
+      unless method_defined?(:reply_seqid) || private_method_defined?(:reply_seqid)
+        def reply_seqid(rseqid)
+          expected_seqid = dequeue_pending_seqid
+          !expected_seqid.nil? && rseqid == expected_seqid
+        end
+      end
+    end
+  end
+end

data/lib/sequel/adapters/hexspace.rb

@@ -1,72 +1,6 @@
-require_relative 'shared/hexspace'
-require 'hexspace'
-
-# Backport upstream fix (THRIFT-5909, PR #3270) for thrift 0.22.0 on Ruby 3.4+.
-# Thrift::Bytes.empty_byte_buffer calls force_encoding on string literals,
-# which triggers "literal string will be frozen in the future" warnings.
-# Remove this once thrift > 0.22.0 is released with the fix.
-if defined?(Thrift::Bytes) && (spec = Gem.loaded_specs['thrift']) && spec.version < Gem::Version.new('0.23')
-  module Thrift
-    module Bytes
-      def self.empty_byte_buffer(size=nil)
-        if size&.positive?
-          "\0".b * size
-        else
-          ''.b
-        end
-      end
-    end
-  end
-end
-
-# Restore Thrift::Client#handle_exception and #reply_seqid for thrift >= 0.24.
-#
-# hexspace's generated Thrift client (lib/hexspace/tcli_service.rb) calls both,
-# 21 times each. thrift 0.24.0 removed them, folding their work into the new
-# #validate_message_begin -- so on thrift 0.24 every Spark connection dies in
-# recv_OpenSession with NoMethodError, surfacing as
-# Sequel::DatabaseConnectionError.
-#
-# Upstream will not fix this for us: hexspace's generated client is untouched
-# since 2023-05-08, the latest release (0.3.0, Apr 2025) predates thrift 0.24,
-# and no issue or PR there mentions thrift. Regenerating that client against
-# 0.24 would drop support for thrift < 0.24, which is a maintainer's decision
-# rather than ours to force.
-#
-# Both methods are restored verbatim from thrift 0.23.0 and are implemented
-# purely in terms of primitives thrift 0.24 still provides and still uses in
-# validate_message_begin. The dequeue accounting is unchanged: exactly one
-# dequeue_pending_seqid per call on both the reply and the exception path.
-#
-# Note this keeps 0.23 semantics -- it does NOT add 0.24's stricter checks for
-# invalid message type and wrong method name. Remove once hexspace regenerates
-# its client against thrift >= 0.24.
-# rubocop:disable-next Style/GuardClause, Naming/PredicateMethod
-if defined?(Thrift::Client)
-  # Both methods below are copied verbatim from thrift 0.23.0 and deliberately
-  # not restyled: keeping them byte-comparable with upstream is what makes them
-  # auditable. reply_seqid also cannot be renamed to a predicate -- hexspace's
-  # generated client calls it by that exact name.
-  module Thrift
-    module Client
-      unless method_defined?(:handle_exception) || private_method_defined?(:handle_exception)
-        def handle_exception(mtype)
-          if mtype == MessageTypes::EXCEPTION
-            dequeue_pending_seqid
-            raise_application_exception
-          end
-        end
-      end
-
-      unless method_defined?(:reply_seqid) || private_method_defined?(:reply_seqid)
-        def reply_seqid(rseqid)
-          expected_seqid = dequeue_pending_seqid
-          !expected_seqid.nil? && rseqid == expected_seqid
-        end
-      end
-    end
-  end
-end
+require_relative "shared/hexspace"
+require "hexspace"
+require_relative "hexspace/thrift_compat"
 
 module Sequel
   module Hexspace
@@ -80,7 +14,7 @@ module Sequel
       def connect(server)
         opts = server_opts(server)
         opts[:username] = opts[:user]
-        opts.select!{|k,v| v.to_s != '' && ALLOWED_CLIENT_KEYWORDS.include?(k)}
+        opts.select! { |k, v| v.to_s != "" && ALLOWED_CLIENT_KEYWORDS.include?(k) }
         ::Hexspace::Client.new(**opts)
       end
 
@@ -105,9 +39,9 @@ module Sequel
         end
       end
 
-      def execute(sql, opts=OPTS)
+      def execute(sql, opts = OPTS)
         synchronize(opts[:server]) do |conn|
-          res = log_connection_yield(sql, conn){conn.execute(sql, result_object: true)}
+          res = log_connection_yield(sql, conn) { conn.execute(sql, result_object: true) }
         rescue => e
           raise_error(e)
         else
@@ -115,7 +49,7 @@ module Sequel
         end
       end
 
-      def execute_insert(sql, opts=OPTS)
+      def execute_insert(sql, opts = OPTS)
         execute(sql, opts)
 
         # Return nil instead of empty array.
@@ -140,26 +74,20 @@ module Sequel
           types = result.column_types
           column_info = columns.map.with_index do |name, i|
             conversion_proc = case types[i]
-            when 'binary'
+            when "binary"
               Sequel.method(:blob)
-            when 'timestamp'
+            when "timestamp"
               db.method(:to_application_timestamp)
             end
 
-            [i, name, conversion_proc]
+            [ i, name, conversion_proc ]
           end
 
           result.rows.each do |row|
             h = {}
             column_info.each do |i, name, conversion_proc|
               value = row[i]
-              h[name] = if value.nil?
-                nil
-              elsif conversion_proc
-                conversion_proc.call(value)
-              else
-                value
-              end
+              h[name] = conversion_proc && !value.nil? ? conversion_proc.call(value) : value
             end
             yield h
           end

data/lib/sequel/adapters/shared/hexspace.rb

@@ -1,25 +1,12 @@
 # frozen_string_literal: true
 
-require_relative 'spark'
+require_relative "spark"
 
 module Sequel
-
   module Hexspace
-
-    module DatabaseMethods
-
-      include Sequel::Spark::DatabaseMethods
-
-    end
-
-    module DatasetMethods
-
-      include Sequel::Spark::DatasetMethods
-
-    end
-
+    DatabaseMethods = Sequel::Spark::DatabaseMethods
+    DatasetMethods = Sequel::Spark::DatasetMethods
   end
 
   Sequel::Database.set_shared_adapter_scheme(:hexspace, Sequel::Hexspace)
-
 end

data/lib/sequel/adapters/shared/spark.rb

@@ -1,6 +1,6 @@
 # frozen-string-literal: true
 
-require 'sequel/adapters/utils/unmodified_identifiers'
+require "sequel/adapters/utils/unmodified_identifiers"
 
 module Sequel
   module Spark
@@ -9,7 +9,7 @@ module Sequel
     module DatabaseMethods
       include UnmodifiedIdentifiers::DatabaseMethods
 
-      def create_schema(schema_name, opts=OPTS)
+      def create_schema(schema_name, opts = OPTS)
         run(create_schema_sql(schema_name, opts))
       end
 
@@ -17,7 +17,7 @@ module Sequel
         :spark
       end
 
-      def drop_schema(schema_name, opts=OPTS)
+      def drop_schema(schema_name, opts = OPTS)
         run(drop_schema_sql(schema_name, opts))
       end
 
@@ -26,28 +26,28 @@ module Sequel
       def serial_primary_key_options
         # We could raise an exception here instead of just
         # ignoring the primary key setting.
-        {:type=>Integer}
+        { type: Integer }
       end
 
       def supports_create_table_if_not_exists?
         true
       end
 
-      def tables(opts=OPTS)
+      def tables(opts = OPTS)
         _mangle_tables(_tables("TABLES", :tableName, opts) - _views(opts), opts)
       end
 
       # Spark does not support transactions.
-      def transaction(opts=nil)
+      def transaction(_opts = nil)
         yield
       end
 
       # Use an inline VALUES table.
       def values(v)
-        @default_dataset.clone(:values=>v)
+        @default_dataset.clone(values: v)
       end
 
-      def views(opts=OPTS)
+      def views(opts = OPTS)
         _mangle_tables(_views(opts), opts)
       end
 
@@ -56,10 +56,10 @@ module Sequel
       def _tables(type, column, opts)
         sql = String.new
         sql << "SHOW " << type
-        if schema = opts[:schema]
+        if (schema = opts[:schema])
           sql << " IN " << literal(schema)
         end
-        if like = opts[:like]
+        if (like = opts[:like])
           sql << " LIKE " << literal(like)
         end
 
@@ -68,7 +68,7 @@ module Sequel
         # Always internally qualify, so that if a table name in a schema
         # has the same name as a temporary view, it will not exclude
         # the table name.
-        ds.map([:namespace, column]).map do |ns, name|
+        ds.map([ :namespace, column ]).map do |ns, name|
           if ns && !ns.empty?
             Sequel::SQL::QualifiedIdentifier.new(ns, name)
           else
@@ -85,32 +85,32 @@ module Sequel
         if opts[:qualify]
           tables
         else
-          tables.map{|t| t.is_a?(Sequel::SQL::QualifiedIdentifier) ? t.column.to_sym : t}
+          tables.map { |t| t.is_a?(Sequel::SQL::QualifiedIdentifier) ? t.column.to_sym : t }
         end
       end
 
       def create_schema_sql(schema_name, opts)
         sql = String.new
-        sql << 'CREATE SCHEMA '
-        sql << 'IF NOT EXISTS ' if opts[:if_not_exists]
+        sql << "CREATE SCHEMA "
+        sql << "IF NOT EXISTS " if opts[:if_not_exists]
         sql << literal(schema_name)
 
-        if comment = opts[:comment]
-          sql << ' COMMENT '
+        if (comment = opts[:comment])
+          sql << " COMMENT "
           sql << literal(comment)
         end
 
-        if location = opts[:location]
-          sql << ' LOCATION '
+        if (location = opts[:location])
+          sql << " LOCATION "
           sql << literal(location)
         end
 
-        if properties = opts[:properties]
-          sql << ' WITH DBPROPERTIES ('
+        if (properties = opts[:properties])
+          sql << " WITH DBPROPERTIES ("
           properties.each do |k, v|
             sql << literal(k.to_s) << "=" << literal(v.to_s)
           end
-          sql << ')'
+          sql << ")"
         end
 
         sql
@@ -118,11 +118,11 @@ module Sequel
 
       def create_table_prefix_sql(name, options)
         sql = String.new
-        sql << 'CREATE '
-        sql << 'EXTERNAL ' if options[:external]
-        sql << 'TEMPORARY ' if options[:temp]
-        sql << 'TABLE '
-        sql << 'IF NOT EXISTS ' if options[:if_not_exists]
+        sql << "CREATE "
+        sql << "EXTERNAL " if options[:external]
+        sql << "TEMPORARY " if options[:temp]
+        sql << "TABLE "
+        sql << "IF NOT EXISTS " if options[:if_not_exists]
         sql << quote_schema_table(name)
         sql
       end
@@ -146,7 +146,7 @@ module Sequel
         end
 
         sql = String.new
-        sql << create_view_sql_append_columns("CREATE #{'OR REPLACE 'if options[:replace]}#{'TEMPORARY ' if options[:temp]}VIEW#{' IF NOT EXISTS' if options[:if_not_exists]} #{quote_schema_table(name)}", options[:columns])
+        sql << create_view_sql_append_columns("CREATE #{'OR REPLACE ' if options[:replace]}#{'TEMPORARY ' if options[:temp]}VIEW#{' IF NOT EXISTS' if options[:if_not_exists]} #{quote_schema_table(name)}", options[:columns])
 
         if source
           source = source.sql if source.is_a?(Dataset)
@@ -157,7 +157,7 @@ module Sequel
       end
 
       def _append_table_view_options_sql(sql, options)
-        if like = options[:like]
+        if (like = options[:like])
           sql << " LIKE " << literal(like)
         end
 
@@ -165,7 +165,7 @@ module Sequel
           sql << " USING " << options[:using].to_s
         end
 
-        if location = options[:location]
+        if (location = options[:location])
           sql << " LOCATION " << literal(location)
         end
 
@@ -187,28 +187,28 @@ module Sequel
         end
 
         if options[:options]
-          sql << ' OPTIONS ('
+          sql << " OPTIONS ("
           options[:options].each do |k, v|
             sql << literal(k.to_s) << "=" << literal(v.to_s)
           end
-          sql << ')'
+          sql << ")"
         end
 
         sql
       end
 
       def _append_column_list_sql(sql, columns)
-        sql << '('
+        sql << "("
         schema_utility_dataset.send(:identifier_list_append, sql, Array(columns))
-        sql << ')'
+        sql << ")"
       end
 
       def drop_schema_sql(schema_name, opts)
         sql = String.new
-        sql << 'DROP SCHEMA '
-        sql << 'IF EXISTS ' if opts[:if_exists]
+        sql << "DROP SCHEMA "
+        sql << "IF EXISTS " if opts[:if_exists]
         sql << literal(schema_name)
-        sql << ' CASCADE' if opts[:cascade]
+        sql << " CASCADE" if opts[:cascade]
         sql
       end
 
@@ -222,7 +222,7 @@ module Sequel
         m = output_identifier_meth(opts[:dataset])
         im = input_identifier_meth(opts[:dataset])
         metadata_dataset.with_sql("DESCRIBE #{"#{im.call(opts[:schema])}." if opts[:schema]}#{im.call(table)}").map do |row|
-          [m.call(row[:col_name]), {:db_type=>row[:data_type], :type=>schema_column_type(row[:data_type])}]
+          [ m.call(row[:col_name]), { db_type: row[:data_type], type: schema_column_type(row[:data_type]) } ]
         end
       end
 
@@ -230,36 +230,55 @@ module Sequel
         true
       end
 
-      def type_literal_generic_file(column)
-        'binary'
+      def type_literal_generic_file(_column)
+        "binary"
       end
 
-      def type_literal_generic_float(column)
-        'float'
+      def type_literal_generic_float(_column)
+        "float"
       end
 
-      def type_literal_generic_string(column)
-        'string'
+      def type_literal_generic_string(_column)
+        "string"
       end
     end
 
     module DatasetMethods
       include UnmodifiedIdentifiers::DatasetMethods
 
-      Dataset.def_sql_method(self, :select, [['if opts[:values]', %w'values'], ['else', %w'with select distinct columns from join where group having compounds order limit']])
+      Dataset.def_sql_method(self, :select, [ [ "if opts[:values]", %w[values] ], [ "else", %w[with select distinct columns from join where group having compounds order limit] ] ])
 
+      # Handle the date_arithmetic extension's DateAdd expressions using
+      # Spark's make_ym_interval/make_dt_interval functions.
+      #
+      # The extension lets a caller name the result type with the :cast
+      # option (DateAdd#cast_type), defaulting to the generic timestamp type,
+      # and the result is always cast to it here.
+      #
+      # The cast is unconditional because Spark's interval arithmetic does
+      # not settle on one result type -- it depends on both the input type
+      # and which interval function is used. Measured against a live server
+      # on the 3.5.0 that ci.yml pins: DATE + make_dt_interval and TIMESTAMP
+      # + either function yield a TIMESTAMP, but DATE + make_ym_interval
+      # yields a DATE. So there is no subset of cases in which the cast can
+      # be skipped and still be honoured, and an interval of all zeroes adds
+      # nothing to cast in the first place.
+      #
+      # Spark's CAST is narrower than other databases': types it does not
+      # have (:timestamptz) or cannot convert to (:interval) are rejected by
+      # the server with UNSUPPORTED_DATATYPE or DATATYPE_MISMATCH, raising
+      # Sequel::DatabaseError. That is deliberately not pre-screened here --
+      # an allowlist of castable types in the adapter would go stale against
+      # the server, and the server's own message names the offending type.
       def date_add_sql_append(sql, da)
         expr = da.expr
-        cast_type = da.cast_type || Time
 
         h = Hash.new(0)
         da.interval.each do |k, v|
           h[k] = v || 0
         end
 
-        if h[:weeks]
-          h[:days] += h[:weeks] * 7
-        end
+        h[:days] += h[:weeks] * 7
 
         if h[:years] != 0 || h[:months] != 0
           expr = Sequel.+(expr, Sequel.function(:make_ym_interval, h[:years], h[:months]))
@@ -269,14 +288,14 @@ module Sequel
           expr = Sequel.+(expr, Sequel.function(:make_dt_interval, h[:days], h[:hours], h[:minutes], h[:seconds]))
         end
 
-        literal_append(sql, expr)
+        literal_append(sql, Sequel.cast(expr, da.cast_type || Time))
       end
 
       # Route prepared statement / bound variable deletes through the
       # emulated delete path, since Spark does not support native DELETE.
-      def call(type, bind_variables=OPTS, *values, &)
+      def call(type, bind_variables = OPTS, *values, &)
         if type == :delete
-          ps = to_prepared_statement(type, values, :extend=>send(:bound_variable_modules))
+          ps = to_prepared_statement(type, values, extend: send(:bound_variable_modules))
           ps.bind(bind_variables).delete
         else
           super
@@ -284,11 +303,13 @@ module Sequel
       end
 
       # Emulate delete by selecting all rows except the ones being deleted
-      # into a new table, drop the current table, and rename the new
-      # table to the current table name.
+      # into a new table, then overwriting the current table with that new
+      # table's contents.
       #
       # This is designed to minimize the changes to the tests, and is
-      # not recommended for production use.
+      # not recommended for production use. It is not atomic and it has a
+      # residual failure window even after sequel-hexspace-rho hardened it --
+      # see _with_temp_table and the README section on emulated DELETE/UPDATE.
       def delete
         _with_temp_table
       end
@@ -299,21 +320,184 @@ module Sequel
         updated_vals = columns.values
 
         _with_temp_table do |tmp_name|
-          db.from(tmp_name).insert([*updated_cols, *other_cols], select(*updated_vals, *other_cols))
+          db.from(tmp_name).insert([ *updated_cols, *other_cols ], select(*updated_vals, *other_cols))
         end
       end
 
+      # Build `<table>__sequel_delete_emulate` -- in the target's own schema,
+      # see _temp_table_name -- holding the rows the operation should leave
+      # behind, let the caller adjust it, then INSERT OVERWRITE those rows back
+      # into the original table.
+      #
+      # The original table object is never dropped. That is deliberate: until
+      # 2026-10-02 this method dropped the original and renamed the temp table
+      # into its place, so a process that died in between left the named table
+      # GONE, with its rows sitting under a runtime-generated name that appears
+      # nowhere in the caller's source. See sequel-hexspace-rho.
+      #
+      # WHAT THIS STILL DOES NOT GUARANTEE -- read this before relying on it.
+      # INSERT OVERWRITE is unambiguously atomic only on an ACID table. This
+      # adapter emits USING <format> only when a caller passes :using, and the
+      # create_table below passes none, so the temp table -- and typically the
+      # caller's own table -- is whatever the metastore defaults to. Measured
+      # against this gem's test server on 2026-10-02: Provider `hive`,
+      # TextInputFormat, LazySimpleSerDe. Not ACID.
+      #
+      # It nonetheless behaved better than "not ACID" implies, and the
+      # measurements are recorded here because they bound the risk
+      # that is left:
+      #
+      #   * A job failure part way through the write does NOT truncate the
+      #     target. raise_error fired after 150k of 200k rows and the table was
+      #     left holding its FULL pre-operation contents, so the old files are
+      #     not removed up front -- the write stages and then commits.
+      #   * SIGKILLing the client at seven points across the statement never
+      #     left the table absent, empty or partial. The thriftserver carries
+      #     the statement on server-side after the client vanishes, so the table
+      #     held either its pre- or its post-operation rows every time. The same
+      #     harness against the old drop/rename sequence destroyed the table on
+      #     the first try (400,000 rows stranded under the temp name), so it was
+      #     landing in the window.
+      #
+      # What is left is the commit itself: replacing a multi-file Hive table in
+      # place is not one atomic filesystem operation, so a death of the SERVER
+      # -- not the client -- inside that commit, or a storage failure there, can
+      # still leave the table PRESENT but empty or partial, which a plain SELECT
+      # reports silently rather than loudly. That window was NOT reproduced: it
+      # needs the thriftserver killed, and this gem's test server is shared.
+      # Treat it as unquantified, not as absent.
+      #
+      # The trade is therefore deliberate and it is not free. A silently wrong
+      # read is in some ways worse than a loud table-not-found. What buys it is
+      # recoverability: an absent table is unrecoverable by anyone reading their
+      # own source, because the name holding the data appears nowhere in it,
+      # whereas the rows to keep are recoverable here at every instant -- see
+      # the ensure below. The same residual window is written down in the
+      # README, because a consumer cannot see this comment.
+      #
+      # The temp table is dropped in exactly two states, both of which make it a
+      # discardable copy:
+      #
+      #   * the overwrite never started, so the original still holds its
+      #     pre-operation contents; or
+      #   * the overwrite finished and its row count was verified against the
+      #     temp table, so the original holds the intended contents.
+      #
+      # In any other state -- a raise inside the overwrite, or a row count that
+      # disagrees with the temp table -- the temp table holds the only intact
+      # copy of the rows to keep and is left alone. That is sequel-hexspace-c5q's
+      # rule, carried over from the drop/rename sequence it was written for.
+      #
+      # Both the leading drop and the ensure exist because this sequence is not
+      # atomic and Spark's CREATE TABLE is not idempotent, so a process that
+      # dies part way through strands the temp table and every later delete or
+      # update on the same table then fails with TABLE_OR_VIEW_ALREADY_EXISTS --
+      # an error that names a table nothing in the caller's code refers to.
+      # This gem's own suite was poisoned that way on 2026-10-01: one aborted
+      # run left `a__sequel_delete_emulate` behind and the next run reported 51
+      # errors in tests that never touch table `a`.
+      #
+      # The ensure handles a raise or an interrupt, which still unwinds; the
+      # leading drop handles the rest (SIGKILL, a lost connection, a crash),
+      # where nothing of ours gets to run at all. Each names exactly the one
+      # table this method creates and owns -- neither enumerates the server,
+      # which may be shared with tables belonging to nobody here.
       private def _with_temp_table
         n = count
         table_name = first_source_table
-        tmp_name = literal(table_name).gsub('`', '') + "__sequel_delete_emulate"
-        db.create_table(tmp_name, :as=>select_all.invert)
-        yield tmp_name if defined?(yield)
-        db.drop_table(table_name)
-        db.rename_table(tmp_name, table_name)
+        tmp_name = _temp_table_name(table_name)
+        db.drop_table?(tmp_name)
+        db.create_table(tmp_name, as: select_all.invert)
+        overwrite_started = false
+        overwrite_verified = false
+        begin
+          yield tmp_name if defined?(yield)
+
+          # Counted before the overwrite rather than after, so that what the
+          # overwrite is checked against is the intended content itself and not
+          # a number derived from the same statement being verified.
+          expected = db.from(tmp_name).count
+
+          overwrite_started = true
+          _overwrite_from_temp_table(table_name, tmp_name)
+
+          actual = db.from(table_name).count
+          unless actual == expected
+            raise Sequel::Error, "emulated delete/update of #{literal(table_name)} left #{actual} rows, " \
+                                 "expected #{expected}; the rows to keep remain in #{literal(tmp_name)}, " \
+                                 "which has deliberately not been dropped"
+          end
+
+          overwrite_verified = true
+        ensure
+          db.drop_table?(tmp_name) if overwrite_verified || !overwrite_started
+        end
         n
       end
 
+      # The recovery table's name: the target's own name with
+      # `__sequel_delete_emulate` appended, IN THE TARGET'S OWN SCHEMA. The
+      # suffix goes on the LAST component only, so any qualifier -- a schema,
+      # or a catalog and a schema -- survives intact.
+      #
+      # Until 2026-10-02 this was `literal(table_name).gsub("`", "") +
+      # "__sequel_delete_emulate"`, which stripped the backticks that were
+      # keeping a qualified name's parts apart: `sch`.`items` collapsed to the
+      # single String "sch.items__sequel_delete_emulate". Sequel does not split
+      # a String on the dot -- Dataset#schema_and_table returns [nil, the whole
+      # string] for one -- so that went to the server as ONE identifier
+      # containing a dot, naming the default database rather than `sch`.
+      #
+      # That did not merely misplace the recovery copy, it broke the operation
+      # outright, which is worth recording because the bug report assumed
+      # otherwise. Spark refuses the name at CREATE TABLE:
+      #
+      #   `sch.items__sequel_delete_emulate` is not a valid name for
+      #   tables/databases. Valid names only contain alphabet characters,
+      #   numbers and _.
+      #
+      # Measured against this gem's test server on 2026-10-02. So emulated
+      # DELETE and UPDATE against a schema-qualified dataset never worked at
+      # all, and no such temp table can ever have existed anywhere for a
+      # consumer to depend on. Appending to the last component is therefore a
+      # strict widening of the `<table>__sequel_delete_emulate` convention the
+      # README documents, not a change to it -- the unqualified case below is
+      # byte-for-byte what it was.
+      #
+      # A UNIQUE per-operation suffix was considered and rejected, even though
+      # it would close the concurrent-operation collision the README documents
+      # as a hazard. It would put the rows to keep under a runtime-generated
+      # name that appears nowhere in the caller's source, and that is the exact
+      # property sequel-hexspace-rho removed from this method. It is also
+      # unrecoverable in the cases recovery exists for: a SIGKILL or a lost
+      # connection raises nothing, so there is no error message for a generated
+      # name to be reported in, and the leading drop_table? above cannot find
+      # it on the next run either. A predictable name is what makes the README's
+      # "look there first" instruction executable. Concurrency stays the
+      # caller's problem, documented rather than solved.
+      private def _temp_table_name(table_name)
+        case table_name
+        when SQL::QualifiedIdentifier
+          SQL::QualifiedIdentifier.new(table_name.table, "#{table_name.column}__sequel_delete_emulate")
+        else
+          literal(table_name).gsub("`", "") + "__sequel_delete_emulate"
+        end
+      end
+
+      # Replace the whole contents of +table_name+ with the rows in +tmp_name+,
+      # keeping the original table object. The column list is taken from the
+      # TARGET, because INSERT OVERWRITE matches columns by position: naming them
+      # explicitly means a column order difference between the two tables would
+      # be a visible error rather than silently transposed data.
+      #
+      # Extracted so a test can stand in for it, which is the only way to pin
+      # the "temp table survives an unfinished overwrite" rule without killing a
+      # process in the middle of a statement.
+      private def _overwrite_from_temp_table(table_name, tmp_name)
+        rows = db.from(tmp_name).select(*db.from(table_name).columns)
+        db.run("INSERT OVERWRITE TABLE #{literal(table_name)} #{rows.sql}")
+      end
+
       protected def compound_clone(type, dataset, opts)
         dataset = dataset.from_self if dataset.opts[:with]
         super
@@ -343,7 +527,7 @@ module Sequel
       end
 
       def quoted_identifier_append(sql, name)
-        sql << '`' << name.to_s.gsub('`', '``') << '`'
+        sql << "`" << name.to_s.gsub("`", "``") << "`"
       end
 
       def requires_sql_standard_datetimes?
@@ -355,7 +539,7 @@ module Sequel
       end
 
       def literal_blob_append(sql, v)
-        sql << "to_binary('" << [v].pack("m*").gsub("\n", "") << "', 'base64')"
+        sql << "to_binary('" << [ v ].pack("m0") << "', 'base64')"
       end
 
       # Spark requires DATE 'YYYY-MM-DD' syntax instead of plain quoted strings
@@ -388,7 +572,7 @@ module Sequel
         "true"
       end
 
-      def supports_cte?(type=:select)
+      def supports_cte?(type = :select)
         type == :select
       end
 
@@ -418,16 +602,16 @@ module Sequel
 
       # Handle forward references in existing CTEs in the dataset by inserting this
       # dataset before any dataset that would reference it.
-      def with(name, dataset, opts=OPTS)
-        opts = Hash[opts].merge!(:name=>name, :dataset=>dataset).freeze
+      def with(name, dataset, opts = OPTS)
+        opts = Hash[opts].merge!(name: name, dataset: dataset).freeze
         references = ReferenceExtractor.references(dataset)
 
-        if with = @opts[:with]
+        if (with = @opts[:with])
           with = with.dup
           existing_references = @opts[:with_references]
 
-          if referencing_dataset = existing_references[literal(name)]
-            unless i = with.find_index{|o| o[:dataset].equal?(referencing_dataset)}
+          if (referencing_dataset = existing_references[literal(name)])
+            unless (i = with.find_index { |o| o[:dataset].equal?(referencing_dataset) })
               raise Sequel::Error, "internal error finding referencing dataset"
             end
 
@@ -447,32 +631,38 @@ module Sequel
           # Assume we will insert the dataset at the end, so existing references have priority
           references = references.merge(existing_references)
         else
-          with = [opts]
+          with = [ opts ]
         end
 
-        clone(:with=>with.freeze, :with_references=>references.freeze)
+        clone(with: with.freeze, with_references: references.freeze)
       end
 
       private def select_values_sql(sql)
-        sql << 'VALUES '
+        sql << "VALUES "
         expression_list_append(sql, opts[:values])
       end
     end
 
     # ReferenceExtractor extracts references from datasets that will be used as CTEs.
     class ReferenceExtractor < ASTTransformer
-      TABLE_IDENTIFIER_KEYS = [:from, :join].freeze
-      COLUMN_IDENTIFIER_KEYS = [:select, :where, :having, :order, :group, :compounds].freeze
+      TABLE_IDENTIFIER_KEYS = [ :from, :join ].freeze
+      COLUMN_IDENTIFIER_KEYS = [ :select, :where, :having, :order, :group, :compounds ].freeze
 
       # Returns a hash of literal string identifier keys referenced by the given
       # dataset with the given dataset as the value for each key.
       def self.references(dataset)
-        new(dataset).tap{|ext| ext.transform(dataset)}.references
+        new(dataset).tap { |ext| ext.transform(dataset) }.references
       end
 
       attr_reader :references
 
       def initialize(dataset)
+        # `super()` with explicit empty parens, NOT bare `super`. ASTTransformer
+        # defines no #initialize, so the inherited one is Object's, which takes
+        # no arguments -- bare `super` would forward `dataset` to it and raise
+        # ArgumentError on every instantiation. The call is a no-op today and is
+        # here so the chain still works if ASTTransformer ever gains state.
+        super()
         @dataset = dataset
         @references = {}
       end
@@ -504,10 +694,10 @@ module Sequel
         case o
         when Sequel::Dataset
           # Special case FROM/JOIN, because identifiers inside refer to tables and not columns
-          TABLE_IDENTIFIER_KEYS.each{|k| o.opts[k]&.each{|jc| table_identifier_extract(jc)}}
+          TABLE_IDENTIFIER_KEYS.each { |k| o.opts[k]&.each { |jc| table_identifier_extract(jc) } }
 
           # Look in other keys that may have qualified references or subqueries
-          COLUMN_IDENTIFIER_KEYS.each{|k| v(o.opts[k])}
+          COLUMN_IDENTIFIER_KEYS.each { |k| v(o.opts[k]) }
         when SQL::QualifiedIdentifier
           # If a qualified identifier has a qualified identifier as a key,
           # such as schema.table.column, ignore it, because CTE identifiers shouldn't

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: sequel-hexspace
 version: !ruby/object:Gem::Version
-  version: 2.0.0
+  version: 2.0.1
 platform: ruby
 authors:
 - Jeremy Evans
@@ -31,6 +31,9 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: 0.2.1
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '0.4'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -38,13 +41,16 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: 0.2.1
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '0.4'
 - !ruby/object:Gem::Dependency
   name: thrift
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0.18'
+        version: '0.23'
     - - "<"
       - !ruby/object:Gem::Version
         version: '0.25'
@@ -54,24 +60,10 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0.18'
+        version: '0.23'
     - - "<"
       - !ruby/object:Gem::Version
         version: '0.25'
-- !ruby/object:Gem::Dependency
-  name: rake
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
 - !ruby/object:Gem::Dependency
   name: minitest
   requirement: !ruby/object:Gem::Requirement
@@ -90,30 +82,36 @@ dependencies:
   name: minitest-hooks
   requirement: !ruby/object:Gem::Requirement
     requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.5'
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 1.5.4
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.5'
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 1.5.4
 - !ruby/object:Gem::Dependency
   name: minitest-global_expectations
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '1.0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ">="
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '1.0'
 - !ruby/object:Gem::Dependency
   name: simplecov
   requirement: !ruby/object:Gem::Requirement
@@ -142,12 +140,14 @@ files:
 - LICENSE
 - README
 - lib/sequel/adapters/hexspace.rb
+- lib/sequel/adapters/hexspace/thrift_compat.rb
 - lib/sequel/adapters/shared/hexspace.rb
 - lib/sequel/adapters/shared/spark.rb
 homepage: https://github.com/outcomesinsights/sequel-hexspace
 licenses:
 - MIT
-metadata: {}
+metadata:
+  rubygems_mfa_required: 'true'
 rdoc_options:
 - "--quiet"
 - "--line-numbers"